Data Engineer - Databricks / Spark / AI
Innova software Services Inc
Location: San Francisco, CA
Work Model: Hybrid 2 Days/Week Onsite
Duration: 12+ Months
Employment Type: Contract
Experience: 5+ Years
Work Authorization: Visa-independent candidates required Job Summary
We are seeking a highly skilled Senior Data Engineer with strong, recent hands-on experience in Databricks, Apache Spark, SQL, and Python .
This is primarily a Data Engineering role with additional exposure to AI and Generative AI capabilities within the Databricks ecosystem. The ideal candidate will have extensive experience designing and developing scalable data pipelines and data processing solutions using Databricks and Spark.
Candidates should also have hands-on experience with Databricks Genie and an understanding of how AI capabilities can be applied to enterprise data and analytics solutions.
Strong, recent Databricks experience is essential for this position.
Key Responsibilities-
Design, develop, and maintain scalable data engineering solutions using Databricks, Apache Spark, SQL, and Python .
-
Build and optimize high-volume ETL/ELT data pipelines and data-processing workflows.
-
Develop complex SQL queries and transformations for large-scale datasets.
-
Build distributed data-processing solutions using PySpark/Spark .
-
Design reliable and scalable data architectures within the Databricks ecosystem.
-
Work hands-on with Databricks Genie to enable AI-powered conversational data and analytics capabilities.
-
Integrate AI/GenAI capabilities with enterprise data platforms and analytics workflows.
-
Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
-
Perform data transformation, cleansing, validation, and quality checks.
-
Troubleshoot performance issues across Spark jobs, SQL workloads, pipelines, and Databricks environments.
-
Work with structured and semi-structured datasets from multiple enterprise sources.
-
Collaborate with Data Engineering, Analytics, AI/ML, Product, and business teams.
-
Translate business and analytical requirements into scalable data solutions.
-
Participate in technical design, code reviews, testing, deployment, and production support.
-
Maintain engineering standards, documentation, and data-development best practices.
-
Participate in Agile/Scrum development activities using Jira and Confluence.
Candidates should demonstr
- Senior Data Engineer Databricks / Spark / AI Location: San Francisco, CA Work Model: Hybrid 2 Days/Week Onsite Duration: 12+ Months Employment Type: Contract Experience: 5+ Years Work Authorization: Visa-independent candidates required...DatabricksContract workWork visa2 days per week
Innova software Services Inc
San Francisco, CA2 days ago - ...build the foundational data infrastructure that powers cutting-edge AI applications leveraging... ...simple hypotheses into engineered features, manage secure... ...using modern frameworks (Spark, dbt, LangChain)A strong... ...commonly used data platforms (Databricks, Snowflake, BigQuery,...DatabricksApprenticeshipWork at officeLocal areaEasy work
McKinsey & Company
San Francisco, CA2 days ago - ...locations for team members for our Data Platform roles are our San... ...now part of Superhuman, the AI productivity platform on a... ...grows its business? As a Data Engineer on the Finance & Revenue team... ...own scalable data pipelines (Spark/Databricks) that ingest and model billing...DatabricksWorldwideHome officeFlexible hours
Superhuman
San Francisco, CA3 days ago $86.7k - $170.9k
Position Summary Join Deloitte’s Core AI & Data practice and help organizations... ...artificial intelligence capabilities. As a Databricks Data Engineer, you will support the design, build,... ...solutions using Databricks, Apache Spark, Python, and Structured Query Language...DatabricksLocal areaVisa sponsorshipDeloitte
San Francisco, CA1 day ago$50 per hour
...DescriptionJoin Lockheed Martin’s Data & AI Enablement organization and... .... As a lead data engineer, you will design, build, and... ...to validate designs.Optimize Databricks performance with auto‑scaling... ...Python and SQL; proficient with Spark (PySpark/Scala) and deep experience...DatabricksFull timeTemporary workWork experience placementCasual workFlexible hoursLockheed Martin
San Francisco, CA2 days ago$159.2k - $301.6k
About the roleAdobe’s Security Data Platform team builds and... ....As a Staff Security Data Engineer, you will own full-system designs... ...architecture on Databricks and AWS, using Spark, Delta Lake, and Unity Catalog... ...platform innovations, including AI-assisted engineering and...DatabricksFull timeTemporary workLocal areaWorldwideAdobe Systems
San Francisco, CA3 hours ago$246.5k - $339k
...using the power of tech, data, and machine learning... ...ours.About this role:Our Engineering organization owns the... ...involves Fivetran, Kafka, Spark, and Airflow landing data in Snowflake and Databricks. It works, but it grew... ...the latest enterprise AI tools, to help you work...DatabricksWork experience placementWork at officeLocal areaRemote workMonday to FridayFlexible hours3 days per weekFaire
San Francisco, CA21 hours ago- ...program with a degree an engineering discipline.Who you'll... ...build the foundational data infrastructure that powers cutting-edge AI applications leveraging... ...commonly used data platforms (Databricks, Snowflake, BigQuery,... ...engineering tools (Pandas, Spark, dbt, LangChain, etc.)...DatabricksApprenticeshipInternshipWork at officeLocal areaEasy work
McKinsey & Company
San Francisco, CA2 days ago - ...have built a platform that uses AI to transform publicly available data into real time operational insights... ...looking to hire a Senior Data Engineer, who will be an integral part of... ...Engineering and building pipelines using Databricks and Spark Can demonstrate experience in...DatabricksImmediate start
- About the Role As a Data Engineer, you will design, develop, and maintain... ..., machine learning, and AI applications. Write efficient... ...Snowflake, BigQuery, Redshift, or Databricks . Experience with AWS or... ...with dbt . Experience with Spark or other distributed data-processing...Databricks
$180k - $220k
...com, we are building the AI Search Infrastructure... ...useful. Our team includes engineers, researchers, product... ...looking for a hands-on Data Engineer to help build... ...using technologies like Databricks, AWS, kafka and several... ...such as Databricks, Spark, Kafka, and AWS services...Databricks- ...The Role Senior IC role on a data innovation team responsible... ...architecture that powers analytics, ML/AI initiatives, and business... ...at the intersection of data engineering, data science, and data... ...data platforms — Snowflake, Databricks, Apache Spark, Kafka, Airflow Cloud...DatabricksLocal area
$140k - $180k
Data Engineer, Data Platform About The Role We are building out our Data... ...architecture across Snowflake and Databricks, as well as building and... ...package. About Us Sigma is the AI Apps and agentic analytics... ...K5 Global, NewView Capital, Spark Capital, Sutter Hill Ventures...DatabricksFull timeWork at officeFlexible hours- Lead Data Engineer With Martech Location: SFO, CA (Hybrid 2 days a week) Key Responsibilities... ...big-data applications using Scala, Databricks, Spark SQL, Spark Streaming, and Python. Build... ...scalability across distributed systems. Drive AI-assisted engineering practices...Databricks2 days per week
- Sr Data Engineer Disney Entertainment and ESPN Product & Technology Technology... ...the power of data and AI. We design and build innovative... ...(e.g., Kafka, Flink, Spark, Kinesis). Experience with embedding... ...Experience with: Python, Java, Databricks, LangChain, Vector Stores(...Databricks
$170k - $200k
...through the intelligent application of AI and machine learning, Tatari helps... ...to advertise on TV. As a Senior Data Engineer in our Reporting and Measure pillar,... ...SQL; deep hands-on experience with Spark/PySpark. We use Databricks and Delta Lake here Experience with...DatabricksWork at office2 days per week$75k - $115k
Data Engineer — Entry to 3 Years Location: United States (Onsite / Hybrid / Remote... ...platforms and tools such as Apache Spark, Apache Airflow, Kafka, or Databricks Cloud platforms: AWS, Microsoft... ...Power BI or Tableau Exposure to AI/ML data pipelines and machine-learning...DatabricksFull timeInternshipH1bRemote work$185k - $218k
...Checkr is building the data platform to power safe... ...people rely on Checkr for AI verification in the... ...product. As a Senior Data Engineer, you'll lead the... ...technologies such as Kafka, Spark, Iceberg, and data lakes... ...platforms such as Databricks or Snowflake. Pay Transparency...DatabricksWork at officeLocal areaRemote workRelocationFlexible hours3 days per week- Monaco is building an AI-native revenue platform that replaces the fragmented... ...software. The Role We're looking for a Data Platform Engineer to help build Monaco's data and ML... ...ElasticSearch or Turbopuffer, Kafka, Spark, Databricks or Snowflake, etc. Ability to lead...DatabricksWork at office
- ...stuff done” end‑to‑end. You use AI to work smarter and solve... ...what’s next. About the team The Data & AI org is at the heart of... ...managing our data infrastructure (Databricks, Spark, Kafka, etc.), the technology... ...do We’re looking for a Data Engineering Manager to lead a team within...DatabricksWorldwide
$190k - $240k
Senior Data Platform Engineer San Francisco, California, United States Tatari is... ...intelligent application of AI and machine learning, Tatari... ...Distributed data processing (Spark, PySpark, or similar big... ...similar) Managed data platforms (Databricks or similar — administering...DatabricksWork at office2 days per week- ...London, and Amsterdam. Making data-driven decisions is key to... ...and tooling to teams across engineering, product, and business and help... ...lakes; Redshift, Snowflake, Databricks. ~ You have experience... ...pipelines using technologies like Spark, Kafka. ~ You appreciate...DatabricksWork experience placementLocal areaFlexible hours
Carbon Data Solutions
San Francisco, CA2 hours ago $269k - $314k
...growth-stage technology companies, across AI, bio + healthcare, consumer, crypto,... ...like Anduril, Airbnb, Coinbase, Cursor, Databricks, Deel, Figma, GitHub, Roblox, SpaceX, and... ...Partners (“LPs”). The Global Partnerships Data Engineer owns and advances critical data...Databricksa16z
San Francisco, CA1 day ago- ...Talentify in San Francisco, CA is seeking a Data Engineer to build core data systems powering batch and streaming data and ML applications. You will design pipelines, work with Hadoop, Spark, Databricks, Snowflake and AWS, and collaborate with cross-functional teams to...Databricks
- ...For more information, please visit Role Overview As a Data Engineer at Gridware, you’ll help build and maintain the pipelines... ...-time and batch ingestion processes using Python, SQL, Databricks, and Spark Implementing data quality checks, validation rules, and...DatabricksLocal area
Gridware
San Francisco, CAa month ago $85 - $95 per hour
...healthcare technology company to hire a Data Engineer, Principal for its Data Services team. This... ...platform that powers automation and AI deployment across core health plan operations... ...on cloud platforms such as Snowflake, Databricks, and Synapse, with a focus on...DatabricksContract workReleady
San Francisco, CAa month ago$180k - $220k
...on behalf of an early stage start-up who are leveraging AI to modernize the data engineering industry. They're looking for a Senior Data Engineer to... ...stage start-ups Strong exposure to Snowflake or Databricks and query optimization Commercial exposure to Airflow,...Databricks- ...Our mission is to architect AI that learns from and interacts... ...model innovation and systems engineering paired with a design-minded product... ...General Catalyst, SV Angel, Databricks and others. We're fortunate... ..., marketing, and operational data into reliable models and...DatabricksWork at officeVisa sponsorshipFlexible hours
Cartesia AI, Inc.
San Francisco, CA3 days ago $180k - $220k
...the Position We are looking for a Data Analytics Engineer to build and scale the data models,... ...Operations. You'll also leverage modern AI tools to accelerate development and help... ...best practices. Experience with Databricks, Snowflake, BigQuery, or other cloud data...DatabricksWork from homeFlexible hoursParafin Inc
San Francisco, CA4 days ago- ...Job Title : Data Engineer Location: San Francisco CA Hybrid – Monday/Friday Remote +... ...modern data technologies such as Hadoop, Spark, DBT, Dagster/Airflow, Atlan, Trino,... ...etc., modern data platforms such as Databricks and Snowflake and cloud technologies across...DatabricksWork experience placementRemote workMonday to Friday
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Data Engineer - Databricks / Spark / AI. Be the first to apply!
- senior data center engineer San Francisco, CA
- senior cloud data engineer San Francisco, CA
- senior data integration developer San Francisco, CA
- data developer San Francisco, CA
- entry level data engineer San Francisco, CA
- data engineer San Francisco, CA
- big data cloud engineer San Francisco, CA
- data platform engineer San Francisco, CA
- data engineer intern San Francisco, CA
- data engineering intern summer San Francisco, CA