Lead Data Engineer
Openkyber
Role : Sr. Data Engineer
Location: Pasadena, CA 100% Onsite
Work Arrangement: Hybrid 3 Days/Week Onsite
Candidate Requirement: Local candidates only within 50 Miles.
Job Summary
We are seeking an experienced Data Engineer with strong hands-on expertise in AWS, Databricks, PySpark, and Python. The ideal candidate will have experience designing and developing scalable data pipelines and data processing solutions using AWS cloud services and Databricks.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and AWS.
- Develop robust ETL/ELT pipelines for ingesting and transforming large volumes of data.
- Build and optimize PySpark/Spark SQL jobs within Databricks.
- Work with AWS S3 for data storage and data lake solutions.
- Develop data processing workflows using Databricks Workflows and related AWS services.
- Work with Delta Lake for reliable data storage, incremental processing, and data transformation.
- Develop reusable Python frameworks and utilities for data engineering processes.
- Perform data validation, quality checks, error handling, and reconciliation.
- Troubleshoot production pipeline issues and perform Root Cause Analysis (RCA).
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Integrate data from databases, APIs, files, and other enterprise data sources.
- Implement CI/CD and source-control practices for data engineering applications.
- Collaborate with Data Architects, Data Scientists, Analysts, and business stakeholders.
- Participate in design, development, testing, deployment, and production support.
Required Skills
- 10+ years of overall Data Engineering experience preferred.
- Strong hands-on experience with AWS.
- Strong experience with Databricks.
- Strong hands-on experience with PySpark / Apache Spark.
- Strong Python programming experience.
- Strong SQL skills.
- Experience developing enterprise-scale ETL/ELT pipelines.
- Experience with AWS S3 and AWS data services.
- Experience with Delta Lake.
- Experience working with large-scale datasets.
- Strong understanding of data lake/lakehouse architecture.
- Experience with Spark performance tuning and optimization.
- Experience with Git and CI/CD.
AWS Skills
Candidates should have hands-on experience with AWS services such as: Amazon S3 AWS Glue AWS Lambda Amazon Redshift Amazon EMR AWS CloudWatch AWS IAM
Preferred Skills
- Databricks certification
- Experience with Unity Catalog
- Experience with Airflow
- Experience with Kafka
- Experience with AWS Glue/Airflow orchestration
- Experience with Terraform
- Experience with data governance and data quality frameworks
- Experience with streaming and real-time data processing
For applications and inquiries, contact:View email address on us.fitly.work
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer in Oregon, OH vacancy
- ...Title: W2 Sr. Data Engineer Location: Onsite ( Covington, KY) Visa- NO OPT/CPT/TN & H1B The Role: This Senior Data Engineer role requires 5+ years of technology experience, including at least 3 years developing and deploying data solutions. The primary...SuggestedH1b
Openkyber
Oregon, OH1 day ago - ...individual will help evaluate how organic social content performs across global channels, identify trends and opportunities, and translate data into practical recommendations that improve future content, campaigns, and stakeholder decision-making. In addition, this person...SuggestedContract work
Openkyber
Oregon, OH1 day ago $130k - $170k
...Senior Mechanical Engineer - Data Centers Contract Type: Full Time Job Locations US-TX-Austin US-GA-Atlanta US... ...the most successful years in our history. Along with industry‑leading salary compensation, you'll be eligible to purchase shares in...SuggestedFull timeContract workFor contractorsLocal areaRemote workFlexible hoursSSOE
Toledo, OH1 day ago$18 - $50 per hour
...career with us at Siemens Digital Industries Software! We are a leading global software company dedicated to the world of computer... ...: Siemens Digital Industries Software is seeking an AI & Engineering Data Intern to support emerging Artificial Intelligence and...SuggestedRemote jobHourly payFull timeInternshipLocal areaSiemens
Toledo, OH2 days ago$20.91 - $45.04 per hour
...and sensor solutions enable the distribution of power, signal and data to advance next-generation transportation, renewable energy,... ...our interns with complete access to our internal library of self-lead courses. These courses include academies like leadership to help...SuggestedRemote jobWork experience placementSummer workInternshipLocal areaTE Connectivity
Toledo, OH4 days ago- ...responsible for designing, building, and maintaining scalable data platforms and pipelines that support enterprise reporting, analytics... ...2-5 years’ relevant experience requiredProficiency with data engineering technologies such as Databricks, Spark, Python, SQL, and ETL/...Full timeWork at officeLocal area
The Andersons
Maumee, OH4 days ago - ...we are looking for entry-level software programmers, Java Full stack developers, Python/Java developers, data analysts/data scientists, machine learning engineers for full-time positions with clients. Who should apply? Recent computer science/engineering/mathematics/statistics...Full timeH1bRemote work
$107.66k - $161.7k
...working on our Poe product. About the Team and Role: Our small engineering team works on challenging problems every day. We have a culture... ...end ownership of Machine Learning systems -- from prototyping, data pipelines and training, to realtime LLM application at scale...Remote jobFull timeWork experience placementInternshipQuora
Toledo, OH3 days ago- ...approach to where we work. About the Role: As a Machine Learning Engineer, you will have the opportunity to collaborate closely with... ...problems executing on a high throughput system with dynamic data. About the Job: Design, develop, and deploy machine learning...Permanent employmentFull timeWork at officeRemote workWork from homeFlexible hours
Instacart
Toledo, OH3 days ago $73.5k - $212.28k
...At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions... ...to the success of our Firm. You are expected to lead with integrity and authenticity, articulating our purpose...H1bPwC
Toledo, OH17 hours ago- ...EY is seeking an AI Security Engineer to own the security posture of EY's Agentic AI platform end to end. This role focuses on defending... ...define agent authority, implement secure-by-default contracts, and lead red-teaming and adversarial testing to meet regulator...
$102.5k - $213.4k
...digital future. Our elite team of product leaders, data scientists, designers, and software engineers enable our clients to solve their most complex product... ...ambiguity with a drive for clarity Experience leading and growing technical talent (for software engineering...Full timeSummer holidayFlexible hours$45.76k - $52k
...experience and skillset. Basic Job Scope: ~ Leading the world’s sustainable energy future,... ...development, information technology, engineering, energy services and much more. When you... ...or Master's degree in Computer Science, Data Science, Statistics, Electrical/Computer...Summer workInternshipWork at office- ...opportunity We are seeking an AI Security Engineer to own the security posture of EY's... ...invocation, delegated authority, model and data supply chain, multi-tenancy, and every deployment... ...authorization of inter-agent calls. Lead AI red teaming and adversarial testing :...Contract work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!
