Data Platform Support Engineer

Job Title: Databricks Data Engineer ( Databricks, AWS/Azure, Python, PySpark, UDP) - Chicago

Location: Chicago, IL

Secondary Location(s): Richardson, TX; Phoenix, AZ; Tampa, FL; Indianapolis, IN; Hartford, CT

Work Model: Hybrid

Security Clearance: Visa Independent

Must Haves

• Visa Independent: Green Card holder or US Citizen
• 5+ years of enterprise-scale data engineering implementation experience
• Hands-on Python and PySpark development (production-level code)
• UDP or modern data platform architecture experience
• Ability to work hybrid model in Chicago, IL (or relocation to secondary locations)
• Experience in both batch and streaming data processing
• Strong problem-solving and analytical skills
• Excellent communication and collaboration abilities

Description / Responsibilities / Skills

Overview

• Lead the design and implementation of advanced data engineering solutions using Databricks, Delta Lake, Apache Spark, and cloud platforms (AWS/Azure).
• This role combines Databricks expertise with strong Python/PySpark development skills and Unified Data Platform (UDP) architecture experience for the Health Insurance account.

Key Responsibilities

• Build and maintain Delta Lake data lakehouses with advanced performance optimization
• Implement cloud platform solutions on AWS or Azure
• Develop reusable Python frameworks and libraries for data engineering
• Optimize performance, cost, and resource utilization across platforms
• Implement data governance, quality, and security frameworks
• Lead technical discussions and mentor junior data engineers

Deliverables

• Support cloud migrations and data platform modernization initiatives
• Design and implement Unified Data Platform (UDP) architectures
• Develop and optimize complex data pipelines using PySpark and Python

Experience & Skill Set Requirements

Required Skills & Experience

• 5+ years of hands-on Databricks and Apache Spark development/implementation
• Advanced Python proficiency with focus on data engineering applications
• Strong PySpark expertise with hands-on optimization and performance tuning
• Deep understanding of Delta Lake and ACID transactions
• Unified Data Platform (UDP) architecture experience and design patterns
• Cloud platform expertise: AWS or Azure (hands-on implementation)
• Experience designing and implementing data lakehouses
• SQL proficiency and data transformation logic
• Understanding of distributed computing and optimization
• Git and CI/CD pipeline experience

Nice-to-Have

• Healthcare or insurance industry experience
• Databricks certification or advanced training
• AWS or Azure cloud certifications (Developer, Solutions Architect)
• Machine learning pipeline development experience
• Data governance and metadata management expertise
• Experience with data quality frameworks and testing
• Agile/Scrum and enterprise software delivery experience
• Open-source contribution experience (Spark, Delta Lake, etc.)