Senior Data Engineer
Designed and delivered scalable cloud data engineering solutions using AWS and Microsoft Azure for large-scale data processing, analytics, and enterprise data integration. Developed batch and streaming data pipelines using Apache Spark, PySpark, Kafka, Apache Flink, Python, and cloud-based data services to ingest, transform, validate, and process high-volume datasets Developed and maintained Azure Data Factory pipelines for data ingestion, transformation, orchestration, and movement between enterprise data sources and cloud data platforms Developed AWS Glue ETL jobs and workflows to extract, transform, validate, and load large-scale datasets into Amazon S3 and Amazon Redshift. Built cloud-based ETL/ELT workflows across AWS and Azure to support batch processing, data transformation, validation, and downstream analytics. Worked with Hadoop, HDFS, Hive, Spark, and PySpark to process distributed datasets and support large-scale analytics and machine learning workloads Implemented event-driven data processing using Apache Kafka and Apache Flink for real-time ingestion, transformation, and streaming analytics Developed and optimized complex SQL queries, joins, aggregations, stored procedures, and data transformations using PostgreSQL for enterprise data processing and analytical workloads. Integrated Apache Iceberg with Apache Spark/PySpark for scalable data processing and optimized lakehouse table operations Built AWS data lake pipelines using Amazon S3, AWS Glue, and Amazon EMR for scalable ingestion, transformation, and processing of large datasets Integrated Airflow workflows with AWS services including S3, EMR, Glue, Lambda, and CloudWatch to automate data ingestion and processing workflows Developed real-time streaming pipelines using Apache Kafka and Amazon Kinesis for high-volume event ingestion and downstream data processing. Optimized Redshift query performance through data distribution, sort keys, compression, and query optimization techniques Developed and maintained Apache Airflow DAGs to orchestrate end-to-end data pipelines across Amazon S3, AWS Glue, EMR, Lambda, and Redshift, including scheduling, dependency management, monitoring, and failure handling. Performed enterprise-scale data migration activities across on-premises and cloud environments, developing data extraction, transformation, validation, and loading pipelines using Python, SQL, AWS Glue, Amazon S3, and Amazon EMR. Designed and implemented enterprise-scale cloud data migration pipelines using Python, SQL, AWS S3, AWS Glue, EMR, and Redshift to extract, transform, validate, and migrate large volumes of structured and semi-structured data. Developed Java-based data processing components and REST APIs to integrate enterprise applications with AWS/Azure-based data platforms and downstream services Built data integration workflows across SQL Server, Azure SQL, MongoDB, MongoDB Atlas, and cloud-based data platforms, supporting reliable data movement and analytics Leveraged Google Cloud Platform (GCP), BigQuery, Dataflow, Dataproc, and Cloud Storage for scalable data processing, analytics, and enterprise data engineering workloads Integrated Amazon Redshift with Amazon S3 and AWS Glue to support scalable data ingestion, transformation, and analytics workflows Developed and optimized Spark workloads on Amazon EMR for distributed batch processing and large-scale ETL pipelines Implemented data exchange and serialization workflows using JSON, XML, CSV, and Avro, including serialization and deserialization for batch and streaming pipelines Implemented secure enterprise integrations using JWT authentication, Kerberos, API security, and access controls across cloud and distributed data-processing environments Contributed to Data Mesh initiatives by developing reusable and governed data products across enterprise platforms, databases, and cloud environments Prepared and curated datasets for Data Science and ML teams, including data profiling, cleansing, transformation, aggregation, validation, and feature preparation. Developed automated testing for data pipelines and REST APIs and supported System Testing, UAT, integration testing, and end-to-end testing within CI/CD environments Used Linux/UNIX environments and Bash shell scripting to automate data-processing tasks, execute batch jobs, manage files, and support cloud data pipeline operations. Developed and optimized Amazon Redshift data warehouse solutions for large-scale analytical workloads using complex SQL queries, joins, aggregations, and transformations Collaborated with architects, data engineers, data scientists, and business stakeholders to resolve complex technical challenges and deliver scalable AWS/Azure data solutions Built ETL/ELT pipelines to load and transform structured and semi-structured data into Amazon Redshift from AWS data sources
United Airlines