Lead Data Engineer

Job Title: Lead Data Engineer

Location: Columbus, OH (On-site)
Duration: 6+Months

Job Description

We are seeking an experienced Lead Data Engineer to join our team and lead the design, development, and implementation of large-scale data engineering solutions. The ideal candidate will have strong hands-on experience with Python, PySpark, AWS data services, ETL pipelines, data lakes, and data warehouses.

The role involves building and optimizing data integration solutions, supporting marketing data initiatives, and bringing data from multiple internal and external sources into enterprise data platforms.

The candidate should have strong technical leadership skills, hands-on development experience, and the ability to drive engineering best practices across the team.

Key Responsibilities

• Lead the design, development, and implementation of large-scale ETL/ELT data pipelines.
• Build and maintain data ingestion, transformation, validation, and integration processes.
• Develop and optimize distributed data processing solutions using Apache Spark and PySpark.
• Design and implement data lake and data warehouse architectures.
• Work with AWS data services, including AWS Glue and other related services.
• Integrate data from external vendors and various internal systems.
• Perform performance tuning and optimization of data pipelines and distributed workloads.
• Implement data quality, governance, security, and monitoring practices.
• Develop and maintain Java-based services and components supporting data workflows.
• Build REST APIs and integration components for downstream data consumption.
• Establish coding standards, automated testing, CI/CD, and operational best practices.
• Provide technical leadership, mentor team members, and collaborate with cross-functional teams.
• Leverage approved AI-assisted development tools to improve engineering productivity and code quality.

Required Skills & Qualifications

• 10+ years of overall software engineering experience, with strong experience in Data Engineering.
• Strong hands-on experience with Python and PySpark.
• Extensive experience building and maintaining ETL/ELT pipelines.
• Strong knowledge of Data Lakes and Data Warehouses.
• Hands-on experience with AWS Glue and other AWS data services.
• Strong experience with Apache Spark and distributed data processing.
• Understanding of data storage formats such as Parquet and Avro.
• Knowledge of data lakehouse concepts and Medallion architecture.
• Basic to working knowledge of Java and Java-based application development.
• Experience with data integration from multiple internal and external sources.
• Strong problem-solving, communication, and technical leadership skills.

Preferred Qualifications

• Experience with Snowflake or other cloud data warehouse platforms.
• Knowledge of workflow orchestration tools such as Airflow, Dagster, or Control-M.
• Exposure to Kafka, Spark Structured Streaming, and real-time data processing.
• Experience with Delta Lake, Apache Iceberg, or Hudi.
• Exposure to Generative AI, Agentic AI, and AI-assisted development tools.
• Experience in enterprise or regulated environments.
• Knowledge of CI/CD, monitoring, performance optimization, and production support.