Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Data Engineer/ Databricks, PySpark, Python

EPAM Systems

We are looking for an experienced Senior Data Engineer / Databricks Engineer to join a project focused on collecting, processing, and analyzing vehicle telemetry data for a major European automotive manufacturer. The platform operates at a significant scale, processing tens of terabytes of telemetry data in Azure Databricks and running approximately 200 Databricks jobs. The solution is currently in the production maintenance and evolution phase and is deployed across multiple regions, including Europe, North America, and China, with an additional deployment in China planned in the near future. A key focus of the role will be ensuring that the system can scale reliably while keeping infrastructure and operational costs predictable. Responsibilities Monitor the health, stability, and performance of the production data platform Investigate and resolve performance issues and production incidents Analyze Databricks workloads and identify performance bottlenecks Optimize PySpark workloads and Databricks jobs for performance, reliability, and cost efficiency Support and improve approximately 200 existing Databricks jobs and associated data pipelines Prepare the platform for increasing data volumes and workloads as additional vehicle platforms are onboarded Design and implement scaling strategies with a strong focus on predictable infrastructure and operational costs Track and optimize Databricks and Azure resource consumption Improve platform observability, monitoring, alerting, and operational processes Ensure consistency across multiple regional deployments in Europe, North America, and China Identify opportunities for technical improvements, automation, and reduction of operational overhead Requirements 3+ years of hands-on experience in data engineering Expertise in Azure Databricks Proficiency in PySpark and distributed data processing Advanced skills in Python development Understanding of performance optimization for large-scale Spark workloads Background in operating and troubleshooting production data platforms Knowledge of monitoring, observability, and capacity planning combined with cloud cost optimization Ability to analyze existing systems, identify bottlenecks, and propose pragmatic improvements English proficiency at an Upper-Intermediate level (B2) or higher Nice to have Familiarity with Microsoft Azure services and cloud infrastructure Experience building and maintaining CI/CD pipelines using GitHub Actions Exposure to large-scale telemetry, IoT, or automotive data Background in managing Databricks environments with a large number of scheduled jobs and pipelines Showcase of multi-region or geographically distributed cloud deployments

Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Senior Data Engineer/ Databricks, PySpark, Python in United States vacancy
  • We are looking for a Senior Data Software Engineer to join a team that builds and maintains...  ...data exchange solutions, Databricks-based data sharing...  ...services and integrations using Python Develop and support data pipelines using PySpark and Apache Airflow (MWAA) Deploy... 
    Databricks
    Senior
    Python

    EPAM Systems

    United States
    1 day ago
  •  ...разработки масштабируемых конвейеров в Databricks с использованием Python и PySpark. Вы будете работать с бизнес- и...  ...Требуется минимум 3 года опыта в data engineering, знание Python, Databricks и...  ...английским языком. Предпочтительно senior/tech lead уровень; удаленная... 
    Databricks
    Senior
    Python
    Remote job

    emagine Group

    New York, NY
    5 days ago
  • $55 - $62 per hour

     ...Overview: Req ID: 394117 NTT DATA strives to hire...  ...Client is currently seeking a Senior Data Engineering Support Engineer to join their...  ...expert-level proficiency in Databricks and PySpark. In this role, you will be...  ...frameworks and scripts in Python/PySpark to automate data... 
    Databricks
    Senior
    Python
    Temporary work
    Remote work
    Flexible hours

    NTT DATA, Inc.

    Addison, TX
    1 day ago
  •  ...0567Reference26-01842Senior PySpark and Delta Data Engineer Job Summary This is the deep build role. The Senior PySpark and Delta Data Engineer...  ...Advanced SQL and strong Python software engineering fundamentals...  ...orchestration through Databricks Workflows, Airflow, or an equivalent... 
    Databricks
    Senior
    Python
    Contract work

    ConsultNet

    Riverton, UT
    4 days ago
  •  ...Job Family : Data Science Consulting Travel Required :...  ...Guidehouse is seeking a Data Engineer to join our Defense and Security...  ...sources tools such as SQL, Python, and PySpark to formulate or recognize...  ...quality of data ingested into Databricks Organizing and leading... 
    Databricks
    Senior
    Python

    Guidehouse

    Washington DC
    18 hours ago
  • $110k - $130k

     ...We are looking for an experienced Senior Data Engineer with strong expertise in Databricks, Python, PySpark, Apache Spark, and large-scale data engineering. The ideal candidate will be responsible... 
    Databricks
    Senior
    Python
    Full time

    Neshent Technologies

    Atlanta, GA
    8 days ago
  •  ...Senior AWS Data Engineer | Onsite, Washington DC Area | TS/SCI Required About...  ...Develop processing jobs in Python, Spark, and SQL that are...  ...and SQL, plus working Spark/PySpark experience ~ Track record...  ...orchestration Exposure to Databricks, Snowflake, or Informatica... 
    Databricks
    Senior
    Python
    Full time
    For contractors

    Oxenham Group

    Washington DC
    4 days ago
  •  ...Senior Data Engineer – PySpark & Databricks Location: Charlotte, North Carolina (Hybrid)// Need only locals Experience: 10+ Years Job Type: Contract...  ...strong hands-on expertise in PySpark, Databricks, Python, and SQL to design, develop, and maintain scalable... 
    Databricks
    Senior
    Python
    Contract work
    Local area

    Gemburg

    Charlotte, NC
    4 days ago
  •  ...investments in its enterprise data and AI capabilities....  ...an opportunity for a senior-level engineer to help shape the...  ...including Databricks, Unity Catalog, AI-ready...  ...Delta Lake, Spark, and PySpark Creating robust data...  ...PySpark Delta Lake Python SQL Azure Data... 
    Databricks
    Senior
    Python

    Robert Half

    Raleigh, NC
    19 days ago
  •  ...Responsibilities & Skills End-to-End Data QA Source-to-Target Mapping...  ...Schema Drift Validation Pyspark Python Azure Databricks Delta Lake DLT Devops...  ...Science, Statistics, Mathematics, Engineering, Business Analytics. Preferred: Master... 
    Databricks
    Senior
    Python

    Thinkbyte Consulting, Inc.

    United States
    2 days ago
  • We are seeking a Senior Data Software Engineer with Databricks to lead the design, development, and maintenance of...  ...Databricks, Delta Lake and Apache Spark (PySpark) Skills in dbt and Databricks...  ...Catalog Advanced SQL and strong Python skills Solid data modeling skills,... 
    Databricks
    Senior
    Python

    EPAM Systems

    United States
    35 minutes ago
  •  ...Our partner is looking for a Senior Data Engineer based in the United States....  ...data loaders using Go and Python. Design and evolve...  ...idempotent data pipelines using Databricks to process billions of...  ...with Databricks, including PySpark, Delta Lake, and Unity Catalog... 
    Databricks
    Senior
    Python
    Full time
    Flexible hours

    Jobgether

    Remote
    5 days ago
  •  ...Strong PySpark experience is required. Candidates must be visa-...  ...local to Texas Job Title: Data Engineer – PySpark Location: Dallas...  ...hands-on experience with Python and PySpark . ~ Extensive...  ...Experience with technologies such as Databricks, Delta Lake, Hive, or... 
    Databricks
    Python
    Local area

    Highbrow Technology Inc

    Raleigh, NC
    1 day ago
  •  ...Platform Manager is a senior technical...  ...scalable enterprise data platforms within a...  ...driving enterprise data engineering, analytics,...  ...architecting scalable Azure Databricks solutions, and...  ...pipelines using PySpark, Scala, SQL, and Azure...  ...in SQL, Python/PySpark, Scala, and... 
    Databricks
    Senior
    Python
    Remote work

    Confidentiel

    Pittsburgh, PA
    13 days ago
  • $110k - $156k

     ...About the Company The Senior Developer, Data Engineering will lead the design,...  ...with Microsoft Fabric or Databricks. ~ Proven experience leading...  ...platform team. ~ Strong Python skills for data engineering...  ...development. ~ Proficiency in PySpark, Pandas, or Polars... 
    Databricks
    Senior
    Python

    Evention

    Chicago, IL
    24 days ago
  • $60 - $65 per hour

     ...help build analytics that engineers can trust. This role focuses on designing data pipelines and...  ...qualifications Advanced SQL Python plus data analysis and...  ...Tooling experience with Databricks and AWS Observability and...  ...frameworks such as Pandas / PySpark Machine learning or... 
    Databricks
    Senior
    Python
    Hourly pay

    Cloud Destinations LLC.

    Santa Clara, CA
    5 days ago
  •  ..., and governance mechanisms across enterprise data platforms. Key Responsibilities Develop...  ...handling. Required Technical Skills Data Engineering Python SQL PySpark Scala Data Platforms Databricks Delta Lake Integration REST APIs Git... 
    Databricks
    Senior
    Python

    Ampcus Incorporated

    New York, NY
    19 days ago
  •  ...Description Job Description Sr. Data Engineer for Cincinnati OH Will...  ...cloud technologies like PySpark, Spark SQL, Azure Data Lake...  ...Azure Analytical Services, Databricks and Azure Data Explorer. Will...  ...applications. using C#.NET, Python, VB.NET, SQL, KQL, PowerShell... 
    Databricks
    Senior
    Python

    Smart Data

    Cincinnati, OH
    more than 2 months ago
  • $11k - $20k

     ...Job Description Job Description Job Title: Senior Databricks Data Engineer Location-Type: Onsite, Santa Monica, CA (office relocation...  ...experience in production environments. •Strong proficiency in Python (PySpark) and Spark SQL for large-scale data transformation. •... 
    Databricks
    Senior
    Python
    Permanent employment
    Work at office
    Immediate start
    Relocation

    Mondo

    Los Angeles, CA
    20 days ago
  •  ...solutions to our clients world-wide. Job Title: Senior Analytic Data Engineer (Python/Power BI) Location: Remote Duration: 6 Months...  ...tools (e.g., Apache Spark(Dataproc), GCP Tools, Databricks, PySpark). 3 Years of experience building ETL and Data Integration... 
    Databricks
    Senior
    Python
    Temporary work
    Remote work
    Shift work

    Tekwissen

    United States
    1 day ago
  •  ...Advisory.KPMG is currently seeking a Sr. Associate, Data Science Engineer to join our Consulting practice.Responsibilities:Translate...  ..., deployment, and MLOps, with hands-on expertise in Python, PySpark, SQL, Databricks, and Snowflake, while partnering with clients and... 
    Databricks
    Senior
    Python
    H1b
    Local area

    KPMG

    Atlanta, GA
    5 days ago
  •  ...Job Description Job Description . Job Title: Senior Azure Data Engineer Location: Redmond, WA 98052 Work Setup:...  ...engineering and strong hands-on expertise in Python, SQL, Apache Spark, PySpark, and Azure Databricks . The candidate will be responsible for developing... 
    Databricks
    Senior
    Python

    eTeam Inc.

    Redmond, WA
    19 days ago
  •  ...revenue-generating software and data engineering solutions for clients. With...  ...We are looking for a Senior Databricks Data Engineer to design, build...  ..., you will leverage PySpark, SQL, and Unity Catalog to...  ...Spaces. Familiarity with Python API development using FastAPI... 
    Databricks
    Senior
    Python
    Full time
    Remote work

    Growth Acceleration Partners

    Remote
    14 days ago
  • $140k - $155k

     ...Digital, Cloud & DevOps, Data, and Software Engineering, servicing an array of...  ...and experienced Senior Data Engineer to join our...  ...and business users. Python/SQL Server/Snowflake/Databricks application development...  ..., ETL, CI/CD. Plus: PySpark, Financial systems/capital... 
    Databricks
    Senior
    Python
    Full time
    Temporary work
    Flexible hours

    Synechron

    New York, NY
    1 day ago
  • $197.3k - $225.1k

    Data Engineer 4 (Python, PySpark, AWS) Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems...  ..., and cloud-based data warehousing platforms including Databricks and Snowflake Share your passion for staying on top... 
    Databricks
    Python
    Full time
    Part time
    Internship
    H1b
    Local area

    Capital One

    Mc Lean, VA
    44 minutes ago
  •  ...Description Job Description Senior Architect Data Engineering Role Overview We are...  ...modern stack centered on Databricks, dbt, and Apache Airflow....  ...processing patterns using PySpark, Delta Live Tables (DLT),...  ...: Elite proficiency in Python (PySpark) and advanced SQL... 
    Databricks
    Senior
    Python

    ClifyX, INC

    Irvine, CA
    19 days ago
  • $77.6k - $176k

     ...Job Number: R0252244 Databricks Data Engineer, Senior The Opportunity: As a senior Databricks Data Engineer, you will be hands-on designing...  ..., and Unity Catalog ~3+ years of experience writing Python and PySpark code for ETL scripts and custom transformations ~... 
    Databricks
    Senior
    Python
    Full time
    Contract work
    Part time
    Local area
    Remote work

    Booz Allen Hamilton

    Bethesda, MD
    3 days ago
  • $146.74k - $233.66k

     ...Job Description: Senior Manager, Data Engineering Drive Operational Excellence. At Ashurst Perkins Coie...  ...-on technical excellence across Databricks, dimensional modeling, pipelines,...  ...Advanced, working knowledge of SQL, Python, and PySpark for data engineering and... 
    Databricks
    Senior
    Python

    Perkins Coie

    Anchorage, AK
    2 days ago
  • $129k - $192k

     ...The Team The Ads Data Engineering team's purpose is to design...  ...is growing. As a Senior Ads Data Engineer, you...  ...own that platform on Databricks: making the existing pipelines...  ...with Apache Spark (PySpark and/or Scala),...  ...~ Strong SQL and Python (or Scala) skills for... 
    Databricks
    Senior
    Python
    Full time
    Summer work
    Remote work
    Flexible hours

    Life360

    Remote
    8 days ago
  •  ...Job Duties:-• Experience: 8–10 years in Data Engineering and Data Analysis. • Informatica...  ...design, development, and optimization. • PySpark Development: Advanced skills in PySpark...  ...• Programming Skills: Proficiency in Python and Kafka for streaming and batch data... 
    Senior
    Python

    ThoughtStorm

    Cleveland, OH
    25 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Data Engineer/ Databricks, PySpark, Python. Be the first to apply!