Lead Data Engineer

Openkyber

Atlanta, GA

Role : Sr. Data Engineer

Location: Pasadena, CA 100% Onsite

Work Arrangement: Hybrid 3 Days/Week Onsite

Candidate Requirement: Local candidates only within 50 Miles.

Job Summary

We are seeking an experienced Data Engineer with strong hands-on expertise in AWS, Databricks, PySpark, and Python. The ideal candidate will have experience designing and developing scalable data pipelines and data processing solutions using AWS cloud services and Databricks.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and AWS.
Develop robust ETL/ELT pipelines for ingesting and transforming large volumes of data.
Build and optimize PySpark/Spark SQL jobs within Databricks.
Work with AWS S3 for data storage and data lake solutions.
Develop data processing workflows using Databricks Workflows and related AWS services.
Work with Delta Lake for reliable data storage, incremental processing, and data transformation.
Develop reusable Python frameworks and utilities for data engineering processes.
Perform data validation, quality checks, error handling, and reconciliation.
Troubleshoot production pipeline issues and perform Root Cause Analysis (RCA).
Optimize Spark jobs for performance, scalability, and cost efficiency.
Integrate data from databases, APIs, files, and other enterprise data sources.
Implement CI/CD and source-control practices for data engineering applications.
Collaborate with Data Architects, Data Scientists, Analysts, and business stakeholders.
Participate in design, development, testing, deployment, and production support.

Required Skills

10+ years of overall Data Engineering experience preferred.
Strong hands-on experience with AWS.
Strong experience with Databricks.
Strong hands-on experience with PySpark / Apache Spark.
Strong Python programming experience.
Strong SQL skills.
Experience developing enterprise-scale ETL/ELT pipelines.
Experience with AWS S3 and AWS data services.
Experience with Delta Lake.
Experience working with large-scale datasets.
Strong understanding of data lake/lakehouse architecture.
Experience with Spark performance tuning and optimization.
Experience with Git and CI/CD.

AWS Skills

Candidates should have hands-on experience with AWS services such as: Amazon S3 AWS Glue AWS Lambda Amazon Redshift Amazon EMR AWS CloudWatch AWS IAM

Preferred Skills

Databricks certification
Experience with Unity Catalog
Experience with Airflow
Experience with Kafka
Experience with AWS Glue/Airflow orchestration
Experience with Terraform
Experience with data governance and data quality frameworks
Experience with streaming and real-time data processing

For applications and inquiries, contact:View email address on us.fitly.work

Apply

Vacancy posted 2 days ago

Similar jobs that could be interesting for youBased on the Lead Data Engineer in Atlanta, GA vacancy

Sr. Lead Data Engineer
$205.32k
...USAJob Family GroupEngineering / Product DevelopmentJob ProfileSr Lead Data EngineerManagement LevelSr Manager - Non People LeaderFlexible... ..., LLCSr. Lead Data EngineerJob Description: Sr. Lead Data Engineer positions offered by Cox Automotive Corporate Services, LLC (Atlanta...
Suggested
Full time
Work at office
Local area
Remote work
Flexible hours
Cox Enterprises
Atlanta, GA
10 hours ago
Lead Data Engineer
...Description: Senior Data Engineer - Data Platforms Role Overview We are seeking a Senior Data Engineer to design, build, and maintain scalable data engineering solutions that power analytics, reporting, and data products across the organization. This role works closely...
Suggested
Worldwide
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...details about the role. Position: Senior Technology Consultant - Data / AI Native Engineer Location: Arlington, VA || New York, NY || St. Louis, MO (... ...strong Software Engineering practices and has experience leading complex enterprise data initiatives. The role requires...
Suggested
Long term contract
Local area
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
Minimum 10-12 years of hands-on engineer for Amazon Flink and Kafka streaming data engineer For applications and inquiries, contact:****@*****.***
Suggested
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Senior Data Engineer Remote prefer candidates form Charleston, SC Long Term W2 The Senior Data Engineer, leads the design and implementation of robust data solutions across multiple domains, driving technical excellence and scalability. This role mentor others, shape best...
Suggested
Remote work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
$65 - $70 per hour
...Title: Senior Data Engineer Location: Miramar, FL Duration: 12 months Compensation: $65-$70/hr Work Requirements: , Holders or Authorized to Work in the U.S. Overview The Data Engineer designs, builds, and maintains scalable data pipelines and infrastructure that...
Local area
Flexible hours
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...We are seeking a Data Engineer with experience in projects emphasizing complex data solutions. This person will be responsible for supporting data transformation and distribution to end consumers. Candidate will assist with requirements analysis, design/develop process...
For contractors
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...We have an opening for Senior Data Engineer at Jersey City, NJ . Kindly apply with your updated resume if you are interested. Please find the job details below: Position: Senior Data Engineer Location: Jersey City, NJ (In-Person Interview) Duration:...
Contract work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Title: W2 Sr. Data Engineer Location: Onsite ( Covington, KY) Visa- NO OPT/CPT/TN & H1B The Role: This Senior Data Engineer role requires 5+ years of technology experience, including at least 3 years developing and deploying data solutions. The primary...
H1b
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...An enterprise technology company is seeking a Technology and Data Engineer 5 for a hybrid opportunity in Raleigh, North Carolina . This role is centered on strengthening Infrastructure as Code security across cloud environments and supporting secure software delivery practices...
Full time
Contract work
Work at office
Local area
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...We are seeking a highly experienced Senior AWS Data Engineer with AI/GenAI experience to design, develop, and support scalable cloud-based data platforms and intelligent data solutions. The ideal candidate should have extensive experience in AWS data services, Python,...
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Required Qualifications: Bachelor's degree in Information Systems, Information Technology, Computer Science, Engineering or equivalent experience Strong experience with SQL and PL/SQL development Experience working with relational databases such as SQL Server...
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Job Title: Data Engineer ( DATA ENGINEERING | DATA MODELING | PERFORMANCE) Location: Remote & hybrid, alternating weeks at the nearest office Duration: 4 months+ contract Number of Positions: 2 roles Responsibilities: DATA ENGINEERING | DATA MODELING | PERFORMANCE Senior...
Contract work
Work at office
Remote work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Description: Senior Data Engineer - Data Platforms Role Overview We are seeking a Senior Data Engineer to design, build, and maintain scalable data engineering solutions that power analytics, reporting, and data products across the organization. This role works closely...
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Senior Data Engineer Job Description Job Title: Senior Data Engineer Experience: 12+ Years Job Type: W2 Contract Job Summary We are seeking a highly experienced Senior Data Engineer with 12+ years of experience in designing, developing, and maintaining...
Contract work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Data/Database Administration - Data Engineer - Montvale, NJ - Contract W2/C2C - $55-65 per HR Join our team as a Data Engineer in a dynamic contract role based in Montvale, NJ. This position involves managing data platforms, supporting application operations, and troubleshooting...
Contract work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...you to discuss about this opportunity in details. Role 1: Sr. Data Engineer with Databricks, Python, Salesforce Automation Role 2: Sr. Data... ...source system analysis. Thanks & Regards, OpenKyber LEAD TALENT ACQUISITION SPECIALIST Direct Number:- Fax : Email: Website...
Long term contract
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Role: Senior Data Engineer Location: Dallas, TX (Onsite) Duration: 6-12 Months Responsibilities: Design and Implement scalable solutions for ever-increasing data volumes, using big data/cloud technologies like Pyspark, Kafka, etc. Collaborate...
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
$180k - $190k
...Location: Tustin, CA Salary: $180,000.00 USD Annually - $190,000.00 USD Annually Description: Lead Data Engineer (EDO Platform Engineering Lead) Location: Irvine, CA Salary: $180,000-$190,000 About the Role We are seeking an experienced Lead Data Engineer...
Work at office
Immediate start
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Role : Sr. Data Engineer with strong SQL and Databricks experience Location : Irving, TX (Hybrid 3 days' Work from Office) Type : Contract Client Interview in-person in Irving, TX. Seeking a Senior Data Engineer with deep expertise in SQL Server, Azure SQL, and Databricks...
Contract work
Work at office
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Position: Senior Data Engineer Location: Boston, MA / Providence RI (Hybrid) Job Description: Required Qualifications 8... ...accountability, urgency, and continuous improvement. Experience leading technical initiatives and influencing teams without relying...
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Job Description: We are seeking an experienced Senior Data Engineer to design, develop, and optimize scalable data platforms that power enterprise analytics, machine learning, and AI-driven solutions. The ideal candidate will bring deep expertise in modern data engineering...
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
$40 - $50 per hour
...job summary: We are seeking an experienced Databricks Data Engineer to join our high-performing Data & AI practice in Phoenix, AZ. In this role, you will be responsible for designing and deploying enterprise-grade Lakehouse solutions on Google Cloud Platform (Google Cloud...
Hourly pay
Contract work
Temporary work
Work experience placement
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
$67 per hour
...Role : Senior Data Engineer(Medicaid or Healthcare Experience(Required)) Client: Government of Nebrasaka(NeDHHS) Work Type : Remote Duration : 12 Months Max Payrate : 67$ Per Hour OpenKyber is a fast-growing Systems Integrator based...
Hourly pay
Local area
Remote work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Data Engineer and Lead with Denodo Exp. At Newark NJ Denodo Development (3+ years) Most critical skill Virtual Data Models View development and data virtualization Joins, relationships, transformations, aggregations, unions Query optimization and...
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Senior/ Lead Data Engineer- Snowflake Cotex+ AWS Bedrock Location : Remote C2C or W2 Independent Contractors We are looking for an experienced Site Reliability Engineer (SRE) to support and operate highly available Snowflake and AWS-based data and AI platforms, with hands...
For contractors
Remote work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Role: Lead Data Engineer (Azure, Snowflake, DB) Location: Remote Duration: Long Term Skill Focus : Azure , Snowflake, DB, ETL, Python, Pyspark, SnowSQL Key Responsibilities: Design and develop ETL/ELT pipelines using Azure Data Factory...
Remote work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Senior Data Engineer Malvern, PA (Hybrid -3 day's onsite): Duration: Long term uties Build out an extensive transaction stitching application to helps track the journey of money flowing through a client's account over the lifetime of the funds. Skills Advanced...
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Seeking a data engineer for hybrid contract role in Westmont IL. If interested , share your word document resume, work authorization, expected... ...enterprise data for AI/agentic consumption. Track record of leading workstreams and driving technical decisions across teams as a...
Hourly pay
Contract work
For contractors
Monday to Friday
Shift work
2 days per week
3 days per week
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Data Engineer IV Location: Remote (PST Preferred) Duration: 7-Month Contract (Through April 16, 2027) Employment Type: Contract | 40 Hours... ...rate: $110/hr. - $118/hr. on W2 About the Opportunity A Global Leading Technology Company is seeking an experienced Data Engineer IV...
Contract work
Immediate start
Remote work
Openkyber
Atlanta, GA
2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!