Senior Spark Data Engineer - Databricks & ML Pipelines

We are seeking a Senior Data Engineer with deep expertise in Apache Spark and hands‑on experience delivering production‑grade solutions on Databricks. This role centers on designing, optimizing, and operating large‑scale Spark data platforms that power analytics, machine learning (ML), and artificial intelligence (AI) use cases. The ideal candidate is a Spark practitioner who understands how to build efficient distributed data pipelines, optimize performance and cost, and enable ML workflows at scale. You will lead Spark pipeline design, collaborate closely with data scientists and analysts, and help operationalize ML and AI solutions across the enterprise. Key Responsibilities Design, build, and maintain distributed data pipelines using Apache Spark on Databricks Lead development of high‑performance Spark jobs for batch and near‑real‑time processing Implement complex data transformations using PySpark (DataFrame & Spark SQL APIs) Apply Spark best practices for partitioning, caching, joins, and shuffles Tune Spark jobs for performance, scalability, and cost efficiency Required Qualifications 12+ years of experience in Data Engineering. Strong hands‑on experience building and operating Apache Spark applications. Proven experience running Spark workloads in Databricks (production environments). Advanced proficiency in PySpark, Spark SQL, and Python. Experience working with large‑scale, distributed datasets in cloud environments (AWS preferred). Demonstrated experience supporting ML or AI pipelines on Spark. Strong understanding of data modeling, distributed systems, and data architecture. Experience designing reliable, automated, and observable Spark pipelines. About ValueMomentum ValueMomentum is a product development company. We have 25+ years in the market and we have products like Ifoundry, rating engine and Biz dynamics. We have been partnered with Guidewire, Duck Creek and Insurity . We have headquartered in NJ and PA. we have specialised in P&C insurance, we have 75+ clients implementing and delivering. we do have offshore development centres India, Canada, and UK. we have overall 6000+ employees overall the globe. #J-18808-Ljbffr ValueMomentum