A leading insurance company needed to accelerate its data processing of to ensure immediate access to visualizations each morning and facilitate multi-level data availability for their mortgage and health / precautions sectors.
Project Overview
Our client’s challenges
The prospect client data collection process involved two primary sources
- Insurance brokers filled out client admission forms during sales calls.
- Prospects submitted forms through comparison websites generating raw data in JSON format.
The existing data management system encountered
significant challenges in handling the prospect client data efficiently:
- Slow data transformations led to delays, restricting access to important dashboards until mid-morning, rather than at the start of the day.
- Data scientists expressed a need for access to raw JSON data for comprehensive analysis, which the current system could not adequately provide.
- The system’s heavy maintenance requirements diverted resources and attention away from the development of new features
IVY Partners solution
Implement a strategic modernisation of the data architecture, reducing dependency on Talend and migrating from SQL Server to Snowflake. This approach aimed to streamline data transformation from JSON to SQL.
Streamlined Data Transformation: Enhanced the efficiency of converting JSON to SQL, thus speeding up the processing.
Automated Conversions: Introduced scripts to decrease manual efforts in data conversions.
Snowflake Transition: Executed strategic migration planning, team training on cloud data management, and integrated Snowflake for its advanced semi-structured data processing features.
Revised Data Architecture: In Snowflake, data transformation is now accomplished using SQL views rather than Talend components, with Talend’s role reduced to initializing raw data ingestion and view generation.
Comparison of Data Management Solutions: Snowflake vs. SQL Server
Outcomes
- Successfully reduced processing time by 3
- Optimized architecture: Shift from ETL to ELT for improved data integration.
- Early access to data: Dashboards available at the start of the day instead of midday.
- Full data load: Complete import of raw data, outperforming incremental loading.
- Advanced privacy: Enhanced anonymization techniques for personal data.
- Increased BI developer productivity: Quicker debugging and issue resolution.
- Improved talent acquisition: enhanced tech stack helped to attract top talents
The shift towards a modernized data architecture significantly enhanced the company’s operational efficiency, data processing speed, and system maintainability. By embracing cloud solutions like Snowflake and optimizing data handling processes, the company not only solved its initial challenges but also gained additional benefits that contribute to its ongoing success and appeal as an employer.
Is your company facing similar data processing obstacles? Contact our IVY Partners Data Expertise Center, specialized in data management and AI and cloud technologies. Our expert team guides you through every phase of your project, from assessing your needs and providing recommendations to developing tailored solutions to leverage your data effectively and make informed decisions that drive your business’s growth.