Data Engineer - Databricks, ADF, SQL - Technical Lead

Job Title: Databricks Data Engineer ( Databricks, AWS/Azure, Snowflake ) - St. Louis

Primary Location: St. Louis, MO

Work Model: Hybrid

Security Clearance: Visa Independent (Green Card holder or US Citizen)

Supplier Comments / MSP Notes

• Secondary Locations: Richardson, TX; Phoenix, AZ; Tampa, FL
• 5+ years hands-on experience required

Must Haves

• Visa Independent: Green Card holder or US Citizen
• 5+ years of enterprise-scale data engineering experience
• Hands-on implementation experience (not just theoretical knowledge)
• Ability to work hybrid model in St. Louis, MO (or relocation to secondary locations)
• Strong problem-solving and analytical skills
• Experience with both batch and streaming data processing
• Strong communication and collaboration skills

Description / Responsibilities / Skills

Overview

• Design and implement data engineering solutions using Databricks, Delta Lake, Spark, and cloud platforms (AWS/Azure)
• Hands-on expertise in Snowflake integration and data warehouse optimization for the Health Insurance account

Experience & Skill Set Requirements

Required Skills & Experience

• 5+ years of hands-on Databricks and/or Apache Spark development/implementation experience
• Strong proficiency with PySpark and Scala (or Python)
• Deep understanding of Delta Lake and ACID transactions
• Experience designing and implementing data lakehouses
• Proficiency with SQL and data transformation logic
• Cloud platform expertise: AWS or Azure (hands-on implementation)
• Snowflake experience: Integration, optimization, and ETL pipeline development
• Understanding of distributed computing and optimization
• Git and CI/CD pipeline experience

Nice-to-Have

• Healthcare or insurance industry experience
• Databricks certification or advanced training
• AWS or Azure cloud certifications
• Data governance and metadata management experience
• Experience with ML pipelines and feature engineering
• Agile/Scrum experience
• Experience with data quality frameworks

Key Responsibilities

• Design and architect data solutions using Databricks and Delta Lake
• Develop scalable data pipelines using Apache Spark and PySpark
• Integrate Snowflake data warehouse solutions with Databricks platforms
• Optimize cloud platform performance on AWS or Azure
• Implement data governance and quality frameworks
• Optimize Databricks cluster configurations and performance
• Build and maintain data lakehouses and analytics solutions
• Ensure data security, lineage tracking, and compliance
• Collaborate with data scientists and analytics teams
• Support cloud migrations and modernization initiatives