Lead Software Engineer - Databricks, Spark - Chicago

Salary: $61,000 - 101,000 per year Requirements:

• 7+ years of data or platform engineering experience, including hands-on Databricks experience in production environments.
• Strong proficiency in Spark (PySpark or Scala) and SQL, including performance tuning and troubleshooting.
• Experience building and operating data platforms, including batch and streaming ingestion, transformation frameworks, orchestration, and curated data layers.
• Experience with data lake table design and optimization, including partitioning, Z-ORDER, and file sizing, using Databricks, Snowflake, or AWS.
• Familiarity with Unity Catalog or comparable governance tools, including permissions, catalogs and schemas, lineage, and auditing.
• Solid software engineering practices, including Git, CI/CD, automated testing, code reviews, modular design, and documentation.
• Experience implementing observability, data quality checks, and monitoring for pipelines and SQL workloads.
• Strong communication skills and demonstrated ability to lead technical decisions across teams.
• Experience leading effective use of approved AI-assisted development tools for activities such as coding, code review, testing, and troubleshooting, and setting expectations for validating outputs for correctness, performance, and security.
• Understanding of responsible AI use in engineering workflows, including data sensitivity, secure handling of inputs and outputs, and security and resiliency expectations; ability to coach engineers on safe, compliant adoption.
• Preferred: Experience with Databricks Workflows, Delta Live Tables, Structured Streaming, or Databricks SQL Warehouses.
• Preferred: Experience with transformation frameworks such as dbt and semantic layer patterns.
• Preferred: Experience with infrastructure as code, such as Terraform, and automated environment provisioning.
• Preferred: Experience in regulated-data environments, privacy controls, and enterprise data governance programs.

• Define and lead the technical roadmap for our Databricks lakehouse and database platform, covering ingestion, storage, modeling, and serving, and establish standards and reference patterns.
• Design curated datasets, including medallion architecture and dimensional or semantic models where appropriate; establish consistent naming, partitioning, and performance practices.
• Build reliable pipelines and SQL workloads, and establish service-level objectives, runbooks, alerting, incident response, and operational practices.
• Implement and maintain data governance, least-privilege access, auditing, data classification, and lifecycle management, including use of Unity Catalog.
• Tune Spark and SQL workloads, optimize clusters and warehouses, manage caching and storage patterns, and implement cost observability or chargeback as needed.
• Set and uphold standards for code quality, testing, CI/CD, branching, documentation, and review; create reusable libraries and templates and promote consistency across teams.
• Coach engineers, lead design reviews, and work with stakeholders to turn business needs into scalable data platform capabilities.
• Encourage adoption of enterprise-authorized AI-assisted engineering practices to improve quality, delivery speed, and operational outcomes; establish validation standards such as secure coding, peer review, and automated testing, and share effective patterns across the team.
• Use software development lifecycle tools, including authorized AI-assisted development and automation capabilities, to increase the value of automation.

• AI
• AWS
• CI/CD
• Databricks
• Git
• Support
• PySpark
• SQL
• Scala
• Security
• Snowflake
• Spark
• Terraform
• Unity
• dbt
• Cloud
• GameDev
• Copilot
• GitHub
• Marketing

More:

We are hiring a Lead Software Engineer for our Corporate Employee Platforms organization, where you will contribute to an agile team building secure, stable, scalable technology products and advancing our Databricks data platform. We use AI-assisted development tools in our daily work alongside engineering controls and review practices. JPMorganChase is a longstanding financial institution serving consumers, businesses, and prominent corporate, institutional, and government clients, with businesses spanning banking, transaction processing, and asset management. Our corporate teams support the success of our businesses, clients, customers, and employees. We offer a competitive total rewards package, with compensation based on role, experience, skills, and location; eligible employees may receive incentive compensation. Benefits vary by eligibility and include health coverage, wellness resources, retirement savings, backup childcare, tuition reimbursement, mental health support, and financial coaching. We value diversity and inclusion and provide reasonable accommodations for applicants and employees.

last updated 42 week of 2026