Senior Data Engineer

• Designed and developed scalable ETL/ELT pipelines using Azure Databricks (PySpark, Spark SQL), processing 10M+ records per batch to enable reliable analytics delivery. • Engineered end-to-end data migration from On-Prem SQL Server to ADLS Gen2, reducing manual data handling by 40% and improving ingestion reliability. • Developed curated Delta Silver tables and designed high-performance Snowflake Gold schemas, improving Power BI dashboard query performance by 30%. • Migrated legacy .NET and SQL Server stored procedure based logic to cloud-native Spark workflows, enhancing scalability and reducing infrastructure dependency by 50%. • Translated complex insurance business rules into scalable PySpark transformation frameworks, improving data accuracy and validation coverage by 20%.

Reduced data processing time by 50% by designing and optimizing ETL pipelines using Azure Data Factory. • Developed an end-to-end MDM ingestion pipeline to process and transform SAP master data into structured JSON payloads for seamless Reltio integration. • Designed and managed Apache Airflow (Astronomer) DAGs to orchestrate end-to-end ETL pipelines across Databricks, ADLS, and Snowflake, improving pipeline reliability by 30%. • Integrated Airflow with Databricks Jobs API and Snowflake tasks, enabling seamless cross-platform orchestration within the Azure cloud architecture. • Implemented robust CI/CD pipelines using Azure DevOps, reducing deployment time and improving overall project reliability. • Actively contributed to Agile project management through sprint planning, daily stand-ups, and retrospectives, improved project completion rate by 25% and reduced team bottlenecks by 30%.