MergeX

Providing simulation environments for data integration.

Comment

Business Problem

The inspiration for this project stemmed from the complexities and risks associated with merging data from two separate banking institutions. In real-world scenarios, financial institutions often face challenges when consolidating their databases during mergers or acquisitions. A notable example is PNC's most recent major merger, which involved the acquisition of BBVA USA in 2021, which involved integrating massive amounts of customer and transactional data. This merger faced significant challenges, including mismatched account formats, duplicate records, and inconsistent data structures.

Such difficulties can lead to operational disruptions and compliance risks. Our team aimed to create a robust simulation testing platform that could identify potential errors early in the process, ensuring a seamless data migration. The project was motivated by the need to reduce manual interventions, enhance data integrity, and streamline the database merging process.

Learning Progress

Lets try to walk through the challenges in the development process of creating a feature full web application that allows uusers to connect their internal databasse to external databases that might have been aquired in a merger. Our goal for this project was to ease the merging of databases by creating a tool that crosss examss the external databases with the our internal one, to better spot the cchangess that may need to be made in order to make a sseemlessss merge.

Letss begin with the development process of setting up the backend. We wanted to makke ssure that our app can be mobile and distrubuted to a host service, so we needed to containerize our application from the begining. Normally that wouuldnt be the case, but since we ussed live databasess running on external servers, we needed to build cloudflared access tunnels into our container so we can route traffic from our database to our application.

We started with a simple python script that conencted to our mysql database with the input of some credintials for our external merging database. Once we developed the ability to connect to a an external database, we needed to migrate our simple script to a web app so we intregrated the orginal backend into flask. This process took quite a bit of configuration due to internal networking from the cloud flare tunnels and futher more we couldnt stop there. We wanted to make our app accessable fromt the web once deployed.

We used another cloudflare zero trust tunnel to tunnel into our application and access the flask web page to access our external database. This process ate up hours from debguggin the intial internal netoworking of the application, but nonetheless, we made a deployable and accessible web application that allows us to connect to an external database. Our next mission become integrating a react front end, while simulatenously building out the logic that provides the features to our application

Development Process

1) Research and Ideation: The project was inspired by the complexities and risks of merging data from two separate banking institutions, such as PNC Bank's acquisition of BBVA USA in 2021. We began by identifying the core business problem and conducting extensive background research. To refine our approach, we gathered feedback from PNC employees, which helped shape our solution and scope.

2) Defining Functional Requirements: Once we solidified our understanding of the problem and goals, we drafted the functional requirements for the application. These requirements were collaboratively reviewed and finalized by all team members.

3) Team Division and Development: With clear requirements in place, the team divided into two groups:

Backend Team: Focused on creating and establishing database connections, as well as developing API calls to detect errors and discrepancies in the simulated environment. Frontend Team: Responsible for designing and implementing the user interface for the webpage and dashboard.

4) Integration: After the frontend and backend components were individually developed and verified, we began the integration process, ensuring seamless communication between the two layers.

5) Finalization and Presentation: Once the software prototype started taking shape, we created a demo video and prepared presentation slides for the expo. These materials highlighted the key features of our solution, showcasing its ability to simulate and address challenges in database mergers.

Core Challenges

Requirements Draft: Initially, our team faced challenges due to the broad scope of the project, making it difficult to define a clear vision or solution for the business problem.

Integration: Significant development hurdles arose while integrating the backend with the frontend, requiring additional debugging and adjustments to ensure smooth communication between the two components.

Database Communication: Establishing a connection to the database via remote host access proved challenging, with numerous errors and technical issues that needed to be resolved to enable seamless data retrieval and updates.

Data Analytics: Creating accurate and visually compelling graphs and charts presented difficulties, particularly in determining which data points to prioritize and how to structure them effectively for insights.

Share this project:

Updates