ANKIT DAVE

Senior Data Engineer | Spark · SQL· Scala · Databricks · .NET Core · C# · AWS

About

Senior Data Engineer and technical lead with 14+ years of experience — 6+ years in large-scale ETL/ELT (Spark, Scala, Databricks, Airflow, Delta Lake, Kafka) and 8+ years in backend development (. NET Core, C#, SQL, AWS). Currently leading batch data processing at S&P Global: delivered a 3M+ record . NET Core application containerized on AWS EKS, reducing manual intervention by 90%. Prior: enhanced and maintained Spark/Scala pipelines processing 10–20M records daily in Capital Markets. Seeking Lead / Senior Data Engineering roles at product or GCC companies.

Experience

S&P Global

Senior Data Engineer — Technical Lead

S&P Global

Apr 2025 – Present

▸ Batch Orchestrator & Data Sync Application Tech: . NET Core, C#, SQL, REST API, Docker, AWS (EKS, S3, CloudWatch, Secrets Manager) Designed and delivered a 3M+ record . NET Core batch processing application — from architecture to production release, containerized via Docker and deployed on AWS EKS Reduced manual data intervention by 90%, improving data delivery velocity and pipeline reliability Pipeline stages: Extraction & Batching → API payload construction → Response matching → Transformation → Validation → Ingestion Built AWS observability stack: CloudWatch dashboards and alarms, Secrets Manager for credential lifecycle, S3 for intermediate staging Applied security best practices across all environments: IAM least-privilege policies, container vulnerability scanning and remediation, secret rotation ▸ ETL Batch Data Pipelines — Financial Data Tech: Apache Spark 3.0, Scala, Hive, Airflow, Databricks, Delta Lake, AWS S3 Enhanced and maintained ETL batch pipelines for initial and incremental financial dataset loads Optimized Spark execution plans and partitioning strategies, improving daily incremental pipeline throughput Performed data quality validation and reconciliation across Hive and Delta Lake environments

.NET CoreApache SparkAWS

Sr. Specialist Software Engineer

LTIMindtree (Client: S&P Global Inc.)

Nov 2019 – Feb 2025

▸ Large-Scale ETL, Streaming & Data Quality — Capital Markets Tech: Apache Spark, Scala, SQL, Hive, Airflow, Databricks, Delta Lake, AWS S3, Confluent Kafka 10 - 20M records daily Enhanced and maintained ETL pipelines processing of company portfolio and nonportfolio data 100 - 200M records and large-scale backfill operations Optimized parallel workflows and intermittent failures via Spark optimization, reducing execution time and improving SLA adherence Developed complex Spark SQL transformations for Hive and Delta Lake; implemented Kafka producer-consumer applications for real-time streaming integration Led effort estimation, grooming, and sprint planning sessions; acted as coordination point across upstream and downstream pipeline teams Provided L3 production support for critical pipeline failures; participated in Scrum, PI planning, and technical estimation

Apache SparkScalaKafka
SAN Health Soft Solutions

Sr. Software Engineer

SAN Health Soft Solutions

Feb 2019 – Oct 2019

Tech: ASP.NET 4.6, MVC 5, WCF, Web API, C#, SQL Server 2017 Automated 835 EDI claim file processing; maintained health capitation revenue management applications across multiple health plans

ASP.NETC#SQL Server

Sr. Software Engineer

Ontos Information Systems (Client: BCD Travels)

Jul 2016 – Jan 2019

Tech: ASP.NET 4.6, Web API, C#, VB.NET, XML, JSON, JavaScript, jQuery, SQL Server 2008 R2 Integrated GDS (Global Distribution System) APIs into corporate travel booking platform; delivered product enhancements and release strategies for global corporate traveler application

ASP.NETC#JavaScript
Odysseus Solutions

Sr. Software Engineer

Odysseus Solutions

Apr 2015 – Jun 2016

Tech: ASP.NET 4.0, C#, JavaScript, jQuery, LINQ, SQL Server 2014 Developed and enhanced features for an online multi-booking engine; contributed across full SDLC; improved architecture through systematic refactoring

ASP.NETC#JavaScript

Software Engineer

SNDJ Consultants (Client: Sheridan Healthcare)

Feb 2012 – Jun 2014

Tech: ASP.NET 4.0, C#, WCF, LINQ, jQuery, SQL Server 2008 Integrated EPSDT health module; developed and maintained patient management healthcare application; managed deployments across Agile sprints

ASP.NETC#SQL Server

Jr. Software Developer

i-Engineering Software

Feb 2011 – Dec 2011

Tech: ASP.NET 3.5, C#, SQL Server, JavaScript Built web scraping applications and stored procedures for automated insurance data extraction and premium computation

ASP.NETC#SQL Server

Education

SVIT, Gujarat University, India

Bachelor of Engineering · Information Technology

2010

Gujarat University, V.V. Nagar, India

Information Technology

2007

Skills

GitHubJavaScriptSQL ServerDockerKubernetesAWSWPFWeb APIWCFASP.NETC#.NET CoreKafkaDelta LakeDatabricksAirflowHivePySparkScalaApache Spark