Data Engineering & Leadership Experience in financial Services
We partnered with a US-based Data Consulting firm to address modernization and innovation challenges at one of the largest consumer financial institutions in the United States, specializing in personal loans and insurance.
Two key initiatives were
Data Architecture: We were involved in defining a new, scalable data architecture (Medallion on AWS stack with Airflow/dbt) to replace manual, obsolete processes.
Credit Card Loan Validations: This meant building an in-house microservice using a Drools-based rules engine, integrated with Data Architecture, to replace a costly third-party vendor.
The outcome was a new, scalable data platform with established best practices, reduced annual vendor expenses by over $1M, and increased organizational adoption, positioning our team as an internal "platform team" to guide future innovation.
Client & context background
General Context: In conjunction with one of our American partners, we have worked for one of the largest and oldest consumer financial institutions in the United States. Their core business is to offer accessible personal loans and insurance.
As a large, highly digitized company, they have a strong cross-functional engineering team focused on maintaining core-business applications and an innovation team focused on new initiatives. Innovation seeks to generate a mapping between the current state and the target state, which involves an architecture, a goal, and a responsible team.
LoopStudio Team Context: We embedded a strong, high-performing squad in the Innovation team of the client. One of our senior leaders took on the role of Principal Architect, leading various initiatives. Some of them related purely to data, others to the use of that data.
We formed a team with our Principal Engineer, CTO, Tech Lead, and a Sr. Developer. They took the roles of Engineering Manager, Tech Lead, and Sr. Developer, respectively, and we joined one of the existing teams composed of about 30 people, and responsible for 5 to 6 initiatives. Collaboration with other teams and units was critical to the success of the initiatives.
Initiative 1
Data Architecture
Business Problem: The organization’s data processes are very manual, using obsolete technologies for which the teams are not trained. Generating ELT pipelines for multiple consumers that scale is a pain for the business.
Solution & Approach: The Principal Architect defined the vision for the new architecture, a transversal initiative that feeds the target state and innovation projects with relevant data. This involved some key technologies in which to train other teams in the future, based mainly on the AWS stack:
Technical Challenges & Overcoming them
Results & Impacts
Performance: Managed to execute 10 critical DAGs for the organization incrementally with execution times between 3 and 15 minutes.
Business Enablement: Generated data marts for the data science, vehicle loans, and credit cards teams, enabling them to make better decisions.
Resilience: Enabled safe performance of backfillings without duplicating data in the event of data errors, code errors, or upstream changes.
Adoption: Due to the success in utilization and adoption by interested stakeholders, Data Architecture became the de facto platform for initiatives requiring key data on loans and credit information.
Initiative 2: Credit Card Loan Validations
Business Problem: The client was paying $1 million a year for a provider that, based on information from the loan creditor, determined whether the loan could be granted or not and what the reasons were. For example: age, salary, geographic location, outstanding debts, etc.
Upper management defined the goal of developing this in-house. The final objective was to remove the dependence on the third party and thus optimize costs.
Solution & Approach: Our Engineering Manager worked closely with the credit cards team to understand their current system architecture and expectations for interacting with the new microservice.
In parallel, the team developed a rules engine system (using Drools, an open source tool), which allowed for visually or programmatically generating complex business rules and their reasons for loan rejection.
A microservice in Java was developed, designed to be highly resilient to cards traffic.
This service would:
Technical Challenges & Overcoming them
Results & Impact
Economy: The final objective was achieved, cutting high costs for the organization (a $1M annual expense) and removing dependence on a vendor, making it one of the first successful cases of in-house development in this stage of innovation.
Collaboration & Leadership: The success was noted by upper management, leading to the promotion of the idea that our team should define guidelines, POCs, and initial architecture to guide others, solidifying a pattern of internal platform team mentorship.