Data Engineer - Databricks / Spark / AI
Innova software Services Inc
Location: San Francisco, CA
Work Model: Hybrid 2 Days/Week Onsite
Duration: 12+ Months
Employment Type: Contract
Experience: 5+ Years
Work Authorization: Visa-independent candidates required Job Summary
We are seeking a highly skilled Senior Data Engineer with strong, recent hands-on experience in Databricks, Apache Spark, SQL, and Python .
This is primarily a Data Engineering role with additional exposure to AI and Generative AI capabilities within the Databricks ecosystem. The ideal candidate will have extensive experience designing and developing scalable data pipelines and data processing solutions using Databricks and Spark.
Candidates should also have hands-on experience with Databricks Genie and an understanding of how AI capabilities can be applied to enterprise data and analytics solutions.
Strong, recent Databricks experience is essential for this position.
Key Responsibilities-
Design, develop, and maintain scalable data engineering solutions using Databricks, Apache Spark, SQL, and Python .
-
Build and optimize high-volume ETL/ELT data pipelines and data-processing workflows.
-
Develop complex SQL queries and transformations for large-scale datasets.
-
Build distributed data-processing solutions using PySpark/Spark .
-
Design reliable and scalable data architectures within the Databricks ecosystem.
-
Work hands-on with Databricks Genie to enable AI-powered conversational data and analytics capabilities.
-
Integrate AI/GenAI capabilities with enterprise data platforms and analytics workflows.
-
Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
-
Perform data transformation, cleansing, validation, and quality checks.
-
Troubleshoot performance issues across Spark jobs, SQL workloads, pipelines, and Databricks environments.
-
Work with structured and semi-structured datasets from multiple enterprise sources.
-
Collaborate with Data Engineering, Analytics, AI/ML, Product, and business teams.
-
Translate business and analytical requirements into scalable data solutions.
-
Participate in technical design, code reviews, testing, deployment, and production support.
-
Maintain engineering standards, documentation, and data-development best practices.
-
Participate in Agile/Scrum development activities using Jira and Confluence.
Candidates should demonstr
$170k - $200k
...Senior Data EngineerSan Francisco, California, United StatesTatari... ...intelligent application of AI and machine learning,... ...major era.As a Senior Data Engineer in our Reporting and... ...deep hands-on experience with Spark/PySpark. We use Databricks and Delta Lake hereExperience...Databricks2 days per weekTatari
San Francisco, CA5 days ago$180k - $220k
...Data EngineerSan Francisco (Hybrid)At You.com, we are building the AI Search Infrastructure that powers modern... ...useful.Our team includes engineers, researchers, product... ...technologies like Databricks, AWS, Kafka and several... ...such as Databricks, Spark, Kafka, and AWS servicesBuild...DatabricksFull timeImmediate startRemote workWork from homeY.O.U.
San Francisco, CA2 days ago- ...have built a platform that uses AI to transform publicly available data into real time operational insights... ...looking to hire a Senior Data Engineer, who will be an integral part of... ...Engineering and building pipelines using Databricks and Spark Can demonstrate experience in...DatabricksImmediate start
$157k - $245k
...locations for team members for our Data Platform roles are our San... ...now part of Superhuman, the AI productivity platform on a... ...grows its business? As a Data Engineer on the Finance & Revenue team... ...own scalable data pipelines (Spark/Databricks) that ingest and model...DatabricksFull timeWorldwideHome officeFlexible hours- About the Role As a Data Engineer, you will design, develop, and maintain... ..., machine learning, and AI applications. Write efficient... ...Snowflake, BigQuery, Redshift, or Databricks . Experience with AWS or... ...with dbt . Experience with Spark or other distributed data-processing...Databricks
$140k - $180k
Data Engineer, Data Platform About The Role We are building out our Data... ...architecture across Snowflake and Databricks, as well as building and... ...package. About Us Sigma is the AI Apps and agentic analytics... ...K5 Global, NewView Capital, Spark Capital, Sutter Hill Ventures...DatabricksFull timeWork at officeFlexible hours$75k - $115k
Data Engineer — Entry to 3 Years Location: United States (Onsite / Hybrid / Remote... ...platforms and tools such as Apache Spark, Apache Airflow, Kafka, or Databricks Cloud platforms: AWS, Microsoft... ...Power BI or Tableau Exposure to AI/ML data pipelines and machine-learning...DatabricksFull timeInternshipH1bRemote work$185k - $218k
...Checkr is building the data platform to power safe... ...people rely on Checkr for AI verification in the... ...product. As a Senior Data Engineer, you'll lead the... ...technologies such as Kafka, Spark, Iceberg, and data lakes... ...platforms such as Databricks or Snowflake. Pay Transparency...DatabricksWork at officeLocal areaRemote workRelocationFlexible hours3 days per week$180k - $250k
...Verse? The race to AI has become the race to... ...the deployment of the data centers that will power... ...This role supports Core Engineering needs within the organization... ...such as Hadoop, Spark, Airflow, DBT, Dagster,... ...data solutions such as Databricks, Snowflake, ClickHouse,...DatabricksRemote workFlexible hoursVerse
San Francisco, CA25 days ago$224k - $264k
...Checkr is building the data platform to power safe... ...people rely on Checkr for AI verification in the... ...Company. As a Staff Data Engineer on the People Data team... ...technologies such as Kafka, Spark, Iceberg, data lakes,... ...platforms such as Databricks or Snowflake. An A-player...DatabricksWork at officeLocal areaRemote workRelocationFlexible hours3 days per week- ...stuff done” end‑to‑end. You use AI to work smarter and solve... ...what’s next. About the team The Data & AI org is at the heart of... ...managing our data infrastructure (Databricks, Spark, Kafka, etc.), the technology... ...do We’re looking for a Data Engineering Manager to lead a team within...DatabricksWorldwide
$180k - $220k
...on behalf of an early stage start-up who are leveraging AI to modernize the data engineering industry. They're looking for a Senior Data Engineer to... ...growth stage start-ups Strong exposure to Snowflake or Databricks and query optimization Commercial exposure to Airflow,...DatabricksEvolve Group, Inc.
San Francisco, CA2 days ago- ...share me your updated resume. Sr. Data Scientist/Data Engineer SF/Bay Area OR Seattle/Bellevue, WA(... ...and rock-solid SQL skills. 7+ yrs Databricks and its APIs 5+ yrs building scalable data pipelines in Spark using Airflow scheduler/executor framework...DatabricksLong term contractLocal area
Stellent IT LLC
San Francisco, CA2 days ago - ...Data EngineerDevelopment Sciences Informatics is seeking... ...and experienced Data Engineer to build core... ...of the platforms (e.g. Databricks, Snowflake, Redshift, Athena... ...systemsProficiency in Spark and SQL queriesStrong experience... ...elements of AI/ML projects Drive the development...Databricks
Software Technology Inc
South San Francisco, CA1 day ago $60 - $70 per hour
...healthcare technology company to hire a Data Engineer, Senior for its Data Services team. This... ...governed platform that powers automation and AI deployment across core health plan... ...data stack tools such as dbt, Snowflake, Databricks, Airflow, or Tidal. Solid knowledge of...DatabricksContract workReleady
San Francisco, CAa month ago- ...For more information, please visit Role Overview As a Data Engineer at Gridware, you’ll help build and maintain the pipelines... ...-time and batch ingestion processes using Python, SQL, Databricks, and Spark Implementing data quality checks, validation rules, and...DatabricksLocal area
Gridware
San Francisco, CAa month ago - ...Data EngineerServal is an AI-native automation platform transforming how enterprises operate. We build... ..., HR, Finance, Security, Legal, and Engineering. Our mission is to eliminate... ...lakehouse from scratch (Snowflake, Databricks, or comparable) and building the ETL...Databricks
Serval
San Francisco, CA4 days ago - Superhuman is hiring a Data Engineer for the Data Foundations team to design and operate large-scale data pipelines and data lakes using Spark/Databricks. You will help ingest billions of events daily and shape the data platform that powers product and analytics. You'll...Databricks
$170.4 per hour
...P-1381 At Databricks, we are passionate about enabling Data & AI teams to solve the world's toughest problems. We do... ...About the Team The Backline Engineering Team serves as the critical bridge... ...Conduct deep-dive forensics into Spark core internals and the broader...DatabricksCacheflow
San Francisco, CA1 day ago- ...in San Francisco is seeking a hands-on Data Engineer to build and scale a modern data platform... ...across batch and real-time processing using Databricks, AWS, Kafka, and third-party data,... ...and availability for analytics, BI, and AI-driven applications. #J-18808-Ljbffr Y....Databricks
$190k - $220k
...Senior Data EngineerGusto is on a mission to grow the small business... ...our Total Rewards philosophy.AI is a fundamental part of how... ...interview process.The Data Engineering team builds tools and systems... ..., Redshift, BigQuery, or Databricks.Experience implementing CI/CD...DatabricksFull timeWork at officeLocal area2 days per week3 days per weekGusto
San Francisco, CA3 days ago$133.1k - $236.4k
...connect creativity, collaboration, and AI. Our teams are reimagining what professional... ...craft.We are looking for a Senior Data Engineer who will build a strong and scalable foundation... ...data architecture and ETL/pipeline on Databricks across Pro Design products, spanning...DatabricksTemporary workLocal areaWorldwideAdobe
San Francisco, CA2 days ago- ...This Role We are looking for a Senior Data Engineer to join our Data Team and help build reliable... ...technologies such as dbt, Airflow, Spark , or similar frameworks. You have... ...worked with technologies such as Snowflake, Databricks, BigQuery, Redshift, Iceberg, Delta Lake...DatabricksWork at officeFlexible hours3 days per week
User Interviews
San Francisco, CA3 days ago - ...About Omni Omni is the AI analytics platform that turns company data into a trusted source of truth for... ...platforms, backend systems, workflow engines, or data / ML infrastructure... ...systems such as Snowflake, Databricks, BigQuery, dbt, Spark, Airflow, Dagster, or similar...DatabricksFull time
Omni
San Francisco, CAmore than 2 months ago $180k - $220k
...the Position We are looking for a Data Analytics Engineer to build and scale the data models,... ...Operations. You'll also leverage modern AI tools to accelerate development and help... ...best practices. Experience with Databricks, Snowflake, BigQuery, or other cloud data...DatabricksWork from homeFlexible hoursParafin Inc
San Francisco, CA3 days ago- ...Fanatic | Foodie Akkodis is seeking a Data Security Engineer for an FTE role with a growing and... ...company: developing a state-of-the-art AI platform that handles significant amounts... ...(RBAC). ○ Experience with Azure Databricks ○ Deploy Data Loss Prevention (DLP)...DatabricksFull timeTemporary workLocal area
Akkodis
San Francisco, CA3 days ago - Proximity Works is seeking a Senior Data Engineer for its AdTech team in San Francisco to design, build, and optimize scalable data platforms... .... The right candidate will have hands-on experience with Spark, Scala, ETL pipelines, and cloud-native data services, collaborating...
- ...industry through technology, automation, and data-driven decision making. This position... ...to data governance initiatives, engineering standards, and scalable analytics practices... ...and modern scripting languages dbt, Spark, Databricks, or comparable data tooling Experience...Databricks
- We are seeking a highly motivated Data Engineer with experience in Databricks on Microsoft Azure to join a fast-paced team responsible for modernizing enterprise... ...SQL ETL processes and migrate them to Databricks using Spark SQL and/or PySpark. Design, develop, test, deploy, and...Databricks
- We are seeking a Data Engineer with expertise in Azure Databricks to support the modernization of enterprise data platforms... ...to Azure Databricks using Spark SQL and PySpark. Design, develop,... ...Power BI or Tableau. Familiarity with AI or machine learning-enabled data engineering...Databricks
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Data Engineer - Databricks / Spark / AI. Be the first to apply!
- data engineer contract San Francisco, CA
- big data cloud engineer San Francisco, CA
- staff data engineer San Francisco, CA
- big data developer San Francisco, CA
- finance data engineer San Francisco, CA
- software data engineer San Francisco, CA
- data engineer intern San Francisco, CA
- data systems engineer San Francisco, CA
- hadoop big data developer San Francisco, CA
- data engineer machine learning San Francisco, CA