Sr. Data Engineer
Techgene Solutions, LLC
Role : Sr. Data Engineer
Location: Pasadena, CA
Work Arrangement: Hybrid
Job Summary
We are looking for an experienced Data Engineer with strong expertise in Databricks, PySpark, and Python to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have hands-on experience building ETL/ELT pipelines, data processing frameworks, and data lake/lakehouse solutions using Databricks and cloud technologies.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
- Develop ETL/ELT workflows to ingest, transform, cleanse, and integrate data from multiple sources.
- Build and optimize data processing jobs using PySpark and Spark SQL.
- Work extensively with Databricks Lakehouse, Delta Lake, notebooks, workflows, and clusters.
- Develop reusable Python modules and frameworks for data processing and automation.
- Implement data quality checks, validation, error handling, and monitoring within data pipelines.
- Optimize Spark jobs, including partitioning, caching, joins, and performance tuning.
- Work with Delta Lake for data storage, transformation, versioning, and incremental processing.
- Integrate data from relational databases, APIs, files, cloud storage, and other enterprise data sources.
- Collaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders to understand data requirements.
- Implement CI/CD and source-control practices for data engineering code.
- Troubleshoot production data pipeline failures and perform root cause analysis (RCA).
- Ensure data security, governance, lineage, and compliance requirements are followed.
- Participate in design discussions, code reviews, testing, deployment, and production support.
Required Skills
- Strong hands-on experience with Databricks
- Strong PySpark / Apache Spark experience
- Strong Python programming skills
- Experience developing ETL/ELT pipelines
- Strong SQL skills
- Experience with Delta Lake
- Experience with data lake/lakehouse architecture
- Experience with Spark performance tuning and optimization
- Experience working with large-volume datasets
- Strong understanding of data modeling and data engineering concepts
- Experience with Git and CI/CD
- Experience with cloud platforms such as AWS, Azure, or GCP
Preferred Skills
- Databricks certification
- Experience with Azure Data Factory / AWS Glue / Airflow
- Experience with Azure Data Lake / Amazon S3
- Experience with Unity Catalog
- Experience with Kafka or other streaming technologies
- Experience with Terraform
- Experience with data governance and data quality frameworks
- Experience with Power BI, Tableau, or other BI platforms
Typical Technology Stack
Databricks | PySpark | Python | Spark SQL | Delta Lake | SQL | AWS/Azure | Data Lake | Git | CI/CD | Airflow/ADF/Glue
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Sr. Data Engineer in Pasadena, CA vacancy
- Job Title: Sr Data Engineer Location: Glendale, CA Onsite ( 91201 ) Job Duration: Contract 12 Months Required Skills: 5+ years of data engineering experience specifically developing large-scale data pipelines Spark, Airflow, Databricks or Snowflake, SQL, Python. 5+ years...SeniorContract work
- Sr Data Engineer Glendale, California, United States About the Job Sr Data Engineer Job Title: Sr. Data Engineer Location: Glendale, CA Onsite Job Type: C2C Job Responsibilities: Maintain, update, and expand existing core data platform pipelines. Build and maintain APIs...Senior
$143.07k - $241.6k
...vendors, and contribute to the communities where we operate. (link to The Re:Build Way principles) Who we are looking for The Data Engineer contributes to designing and implementing scalable technical solutions while supporting the broader engineering team. They...SeniorFor contractorsWork at officeRemote work$75 - $95 per hour
Sr Data Engineer Glendale, California, United States $ 75.00 - 95.00 (US Dollar) About the job Sr Data Engineer Job Title: Sr Data Engineer Location: Glendale, CA Onsite 3-4 days a week The Company Headquartered in Los Angeles, this leader in the Entertainment & Media...Senior3 days per week- ...and should be open 4 days a week Tech Stack: Airflow, Spark, Data Brick, Snowflake, Deltalake, Kafka Team member will support the... ...Studio Data and AI Platform Basic Qualifications 7+ years of data engineering experience developing large data pipelines Proficiency in at...SeniorFull time2 days per week
- Sr Data Engineer Disney Entertainment and ESPN Product & Technology Technology is at the heart of Disney's past, present, and future. Disney Entertainment and ESPN Product & Technology is a global organization of engineers, product developers, designers, technologists,...Senior
$138.9k - $186.2k
Senior Data Engineer Disney Entertainment and ESPN Product & Technology is a global organization of engineers, product developers, designers, technologists, data scientists, and more - all working to build and advance the technological backbone for Disney's media business...Senior$144.77k - $209.11k
...Department Business Intelligence. Position Overview CGS Federal (Contact Government Services) is seeking a passionate and driven Data Engineer to support a rapidly growing Data Analytics and Business Intelligence platform focused on providing solutions that empower our...SeniorFull timeFlexible hours$127k - $150k
Position Summary Healthcare Data Engineer Sr. Consultant Deloitte Consulting's technology professionals help clients identify and solve their most critical information and technological challenges. We provide advisory through end-to-end implementation services as...SeniorLocal areaVisa sponsorshipDeloitte
Los Angeles, CA11 hours ago- ...Data Engineer Direct Hire Onsite Pasadena (local are encouraged to apply) W2 Only (No Sponsorship/C2C) We are looking for a Data Engineer to design, optimize, and manage scalable data architecture. In this role, you will build critical ETL/ELT pipelines...Full timeLocal area
Mantek Solutions
Pasadena, CA2 days ago $105k - $200k
...Job Description Job Description Description We are currently seeking highly motivated Data Engineers at various levels. This role will report to the Director - Data Engineering and work closely with data analysts, data engineers, data scientists, and clinical leaders...Work experience placementRemote workShift workAstrana Health, Inc.
Alhambra, CAa month ago$90k - $150k
...Description Your role at GEI. The Data Engineer is responsible for designing, building, and maintaining the data pipelines and integrations that power GEI's AI solutions and digital initiatives. This role focuses on ensuring enterprise data is...Work at officeFlexible hoursGEI Consultants
Pasadena, CA3 days ago$11k - $20k
...Job Description Job Description Job Title: Senior Databricks Data Engineer Location-Type: Onsite, Santa Monica, CA (office relocation to Century City anticipated within approximately one year) Start Date: ASAP Duration: Permanent Compensation Range: $2...SeniorPermanent employmentWork at officeImmediate startRelocationMondo
Los Angeles, CAa month ago$110.4k - $165.6k
...that matter – both for our audiences and our employees – and aim to leave a positive mark on culture. Overview As a Senior Data Engineer, you will play a pivotal role in driving our data strategy and standard methodologies for data collection, pipelines, usage,...SeniorWork experience placementLos Angeles, CAmore than 2 months ago$110.4k - $165.6k
...both for our audiences and our employees – and aim to leave a positive mark on culture. In This Role You’ll: The Senior Data Engineer should possess a deep sense of curiosity and a passion for building data pipelines, data structures and data products and the...SeniorWorldwideBurbank, CAmore than 2 months ago- ...Rocket is a Detroit-based company delivering digital solutions for complex transactions. The Staff Data Engineer leads the design and evolution of enterprise data platforms, building robust pipelines to power analytics and operations. As the senior-most engineer, you...Senior
- Data Engineer Location: Pasadena 91101 CA Must have: Good proficiency in Python programming Exposure to Data virtualization tools like Denodo Experience working with ETL framework on python and creating data pipelines Good understanding of SQL, PL/SQL Experience working...
$106k - $143k
The Data Engineer role will be responsible for implementing and managing the business's data infrastructure to support operations and strategic initiatives. This position plays a crucial part in developing and maintaining a robust, scalable data architecture that ensures...Temporary workWork at officeMonday to Friday- Astrana Health seeks Data Engineers at various levels to join our Data - Analytics team. You will report to the Director - Data Engineering and collaborate across analysts, engineers, data scientists, and clinical leaders to produce deliverables for internal and external...Remote job
- Crexi, located in Playa Vista, California, seeks a Senior Data Engineer to own the design and delivery of robust, scalable data pipelines, models, and infrastructure powering Crexi's data and analytics. You will drive data quality and reliability, mentor engineers, and...Senior
- Coinbase is building the future of institutional trading on the Everything Exchange. As a Senior Data Engineer, Market Data on the Institutional Markets team, you will create and operate real-time market data systems, from feed handlers to distribution layers, enabling...Senior
$150k - $200k
...or high-stakes. The firm's work is distinguished by a unique combination of precision and vision. Based in the US, the Senior Data Engineer will be responsible for designing, building, and operating the data platforms that power enterprise reporting, analytics, and...SeniorLocal areaImmediate startFlexible hoursGibson Dunn & Crutcher
Los Angeles, CA3 days ago- Tavant in Glendale, CA seeks a Senior Data Engineer to own and operate the Core Data platform on Databricks, delivering reliable batch and streaming Spark pipelines and governance across AWS, Kubernetes, and Airflow. You will collaborate with technologists to translate...Senior
- EITACIES Inc. in California seeks a Senior Data Engineer (Level II) or (Level III) to support a large Oracle-to-Snowflake migration. You will develop dbt models, transform data, validate results, and optimize Snowflake performance while collaborating with onsite and offshore...SeniorRemote job
$118.8k - $186.2k
...This means it's possible to pursue challenges you are passionate about, in industries you care about. Workday PSA & Financials Data Engineer Armanino is proud to be among the top 20 Largest Accounting and Consulting Firms in the Nation and one of the Best Places to...Full timeContract workLocal areaFlexible hours- Senior Data Engineer (Level II & Level III) Level II is Remote work. Level III is Onsite at Los Angeles, CA. We are looking for experienced Data Engineers to support a large-scale Oracle to Snowflake migration initiative. This role involves dbt development, data transformation...SeniorRemote work
$166.55k - $266.48k
As a Senior Data Engineer, you will contribute to a CG-designed and operated Quantitative Portfolio Construction Platform that generates trade recommendations and associated analytics to align an investment vehicle portfolio with a reference portfolio. This alignment is...Senior- Senior Data Engineer This is a W2 role. Skill sets needed: ETL/ELT, Python/SQL, Big Data, Docker, Gitlab, Airflow, Snowflake, Looker and PostgreSQL, AWS, S3. Interview Schedule/Availability: Zoom followed by zoom panel interview. This is an onsite role in Burbank, CA. The...Senior
- ...opportunity for you to be a part of this journey!! The Senior Software Engineer - Business Intelligence is responsible for designing, developing... ...across business intelligence initiatives by contributing to data architecture, reporting optimization, performance tuning, and...Senior
- Job Title: Senior Data Engineer Location: Glendale, California, USA Job Type: Full-Time Experience: 9-13 Years Domain: Information Technology (IT) Job Summary We are seeking an experienced Senior Data Engineer to design, develop, and optimize scalable data solutions that...SeniorFull time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Sr. Data Engineer. Be the first to apply!
Related searches
- data engineering intern summer Pasadena, CA
- senior customer service Pasadena, CA
- senior manager Pasadena, CA
- senior living Pasadena, CA
- senior vmware engineer Pasadena, CA
- senior consulting engineer Pasadena, CA
- senior sas developer Pasadena, CA
- senior real estate accountant Pasadena, CA
- remote senior project manager Pasadena, CA
- senior Pasadena, CA