Lead Data Engineer

Role : Sr. Data Engineer

Location: Pasadena, CA 100% Onsite

Work Arrangement: Hybrid 3 Days/Week Onsite

Candidate Requirement: Local candidates only within 50 Miles.

Job Summary

We are seeking an experienced Data Engineer with strong hands-on expertise in AWS, Databricks, PySpark, and Python. The ideal candidate will have experience designing and developing scalable data pipelines and data processing solutions using AWS cloud services and Databricks.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and AWS.
• Develop robust ETL/ELT pipelines for ingesting and transforming large volumes of data.
• Build and optimize PySpark/Spark SQL jobs within Databricks.
• Work with AWS S3 for data storage and data lake solutions.
• Develop data processing workflows using Databricks Workflows and related AWS services.
• Work with Delta Lake for reliable data storage, incremental processing, and data transformation.
• Develop reusable Python frameworks and utilities for data engineering processes.
• Perform data validation, quality checks, error handling, and reconciliation.
• Troubleshoot production pipeline issues and perform Root Cause Analysis (RCA).
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Integrate data from databases, APIs, files, and other enterprise data sources.
• Implement CI/CD and source-control practices for data engineering applications.
• Collaborate with Data Architects, Data Scientists, Analysts, and business stakeholders.
• Participate in design, development, testing, deployment, and production support.

Required Skills

• 10+ years of overall Data Engineering experience preferred.
• Strong hands-on experience with AWS.
• Strong experience with Databricks.
• Strong hands-on experience with PySpark / Apache Spark.
• Strong Python programming experience.
• Strong SQL skills.
• Experience developing enterprise-scale ETL/ELT pipelines.
• Experience with AWS S3 and AWS data services.
• Experience with Delta Lake.
• Experience working with large-scale datasets.
• Strong understanding of data lake/lakehouse architecture.
• Experience with Spark performance tuning and optimization.
• Experience with Git and CI/CD.

AWS Skills

Candidates should have hands-on experience with AWS services such as: Amazon S3 AWS Glue AWS Lambda Amazon Redshift Amazon EMR AWS CloudWatch AWS IAM

Preferred Skills

• Databricks certification
• Experience with Unity Catalog
• Experience with Airflow
• Experience with Kafka
• Experience with AWS Glue/Airflow orchestration
• Experience with Terraform
• Experience with data governance and data quality frameworks
• Experience with streaming and real-time data processing

For applications and inquiries, contact:View email address on us.fitly.work