Lead Software Engineer - Databricks AWS

Salary: $61,000 - 101,000 per year Requirements:

• We expect formal training or certification in software engineering concepts, plus at least 5 years of applied experience.
• We require 10+ years of professional software or data engineering experience, including significant production work with Spark on Databricks or EMR.
• We need strong proficiency in Python and/or Java for data processing, platform tooling, and automation.
• We are looking for hands-on Databricks expertise, including Delta Lake, Unity Catalog, Workflows, Repos/notebooks, and SQL Warehouses.
• We require solid AWS experience with S3, IAM, Glue, CloudWatch, Kinesis or MSK, and DynamoDB.
• We seek a proven track record architecting and operating ETL/ELT pipelines, both batch and streaming, with schema design and evolution, SLAs, and reliability engineering.
• We need deep skills in Spark performance tuning and Databricks cluster setup and optimization.
• We require strong SQL and analytics data modeling knowledge, including dimensional/star schema and lakehouse best practices.
• We expect CI/CD and automation tooling for data, including Git workflows, artifact management, and testing frameworks such as pytest and JUnit.
• We are looking for a security-first mindset, including roles and instance profiles, secret management, encryption at rest and in transit, and network controls.
• We want demonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship, along with excellent stakeholder communication.
• Preferred experience includes Delta Live Tables and advanced governance in Databricks.
• Preferred experience includes AWS networking knowledge such as VPCs, subnets, routing, security groups, and data egress controls.
• Preferred experience includes Terraform for infrastructure deployments.
• Preferred experience includes cost optimization techniques such as autoscaling strategies, spot versus on-demand, auto-termination, storage layouts, and compaction.
• Preferred familiarity with Kafka or MSK, or Kinesis Data Streams or Firehose, for real-time ingestion.
• Preferred experience with observability for data systems, including freshness and completeness metrics, lineage, SLAs, and alerting.
• Preferred experience in financial services or other regulated industries.

• We lead the architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark.
• We establish lakehouse patterns with Delta Lake and ensure performance at scale.
• We own Databricks cluster strategy and configuration, including runtime selection, autoscaling, sizing, Spark settings, unit scripts, cluster policies, pools, and instance profiles.
• We orchestrate jobs with Databricks Workflows and integrate with AWS eventing and orchestration where needed.
• We design secure ingestion and transformation frameworks using AWS services such as S3, Glue, IAM, Secrets Manager, CloudWatch, Lambda, and Kinesis or Kafka/MSK.
• We enforce data quality, lineage, and governance through Unity Catalog and/or Glue Catalog, embedding validation and expectations into pipelines.
• We drive Spark performance engineering to improve cost and efficiency through partitioning, file sizing, AQE, broadcast joins, shuffle tuning, caching, spill and memory control, and right-sizing.
• We build reusable libraries, frameworks, and APIs in Python and/or Java, and oversee unit, integration, and data validation testing.
• We implement CI/CD for data projects, infrastructure deployments, environment promotion, and automated releases, while promoting engineering standards and code reviews.
• We partner with security and networking teams to enforce encryption, network controls, least-privilege access, and compliance with organizational policies.
• We lead incident response and root-cause analysis, define SLAs, observability, and runbooks, and drive continuous improvement in reliability and cost efficiency.
• We mentor engineers, guide technical design reviews, contribute to roadmaps and backlog prioritization, and collaborate with product owners and cross-functional partners.

• AWS
• Lambda
• CI/CD
• CloudWatch
• Databricks
• ETL
• Git
• IAM
• Support
• JUnit
• Java
• Kafka
• Network
• Python
• REST
• SQL
• Security
• Spark
• Terraform
• Unity
• pytest
• Cloud
• GameDev
• Serverless

More:

We are JPMorgan Chase, one of the worlds oldest financial institutions, providing innovative financial solutions to millions of consumers, small businesses, and many of the worlds most prominent corporate, institutional, and government clients through the J.P. Morgan and Chase brands. Our history spans more than 200 years, and we are a leader in investment banking, consumer and small business banking, commercial banking, financial transaction processing, and asset management. We offer a competitive total rewards package, including base salary determined by role, experience, skill set, and location, with eligible roles potentially receiving commission and/or discretionary incentive compensation. Our benefits and programs may include comprehensive health coverage, on-site health and wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, financial coaching, and more. We are committed to diversity, inclusion, and equal opportunity, and we value the diverse talents that strengthen our global workforce. This role sits within our Corporate Technology team, which develops applications and provides technology support across corporate functions including Global Finance, Corporate Treasury, Risk Management, Human Resources, Compliance, Legal, and the Corporate Administrative Office.

last updated 37 week of 2026