Lead Software Engineer - Databricks/Spark/AWS

Full-time

JPMorgan Chase & Co.

{"description": " Job Description

This is your chance to change the path of your career and guide multiple teams to success at one of the world's leading financial institutions.

As a Lead Software Engineer at JPMorgan Chase within Corporate Sector, Chief Technology Office, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm's business objectives.

Job Responsibilities:

Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark (Core, SQL, Structured Streaming).

Establish lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) and ensure performance at scale.

Own Databricks cluster strategy and setup: runtime selection, autoscaling, driver/executor sizing, Spark configs, unit scripts, cluster policies, pools, and instance profiles.

Orchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.

Design secure data ingestion and transformation frameworks leveraging AWS services:

S3 for data lake storage and lifecycle management

Glue for catalog/metadata and ETL jobs

IAM and Secrets Manager for role-based access and credential management

CloudWatch for logging, metrics, and alerting

Lambda for serverless utilities

Kinesis and/or Kafka/MSK for streaming ingestion

Enforce data quality, lineage, and governance using Unity Catalog and/or Glue Catalog; embed expectations and validation into pipelines.

Drive Spark performance engineering: partitioning strategies, file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, and job right-sizing to optimize cost.

Build reusable libraries, frameworks, and APIs in Python and/or Java; oversee unit, integration, and data validation testing.

Implement CI/CD for data projects (Git-based workflows), Terraform Infrastructure deployments environment promotion, and automated deployments; champion engineering standards and code reviews.

Partner with platform security and networking teams to enforce encryption, network controls, and least-privilege access; ensure compliance with organizational policies.

Lead incident response and root-cause analysis; establish SLAs, observability, and runbooks; drive continuous improvement in reliability and cost efficiency.

Mentor engineers, guide technical design reviews, and contribute to roadmaps and backlog prioritization; collaborate with product owners and cross-functional teams.

Required qualifications, capabilities, and skills:

Formal training or certification on software engineering concepts and 5+ years applied experience.

10+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR.

Strong proficiency in Python and/or Java for data processing, platform tooling, and automation.

Hands-on Databricks expertise (Delta Lake, Unity Catalog, Workflows, Repos/notebooks, SQL Warehouses).

Solid AWS experience: S3, IAM, Glue, CloudWatch, Kinesis / MSK, DynamoDB

Proven track record architecting and operating ETL/ELT pipelines (batch and streaming), with schema design/evolution, SLAs, and reliability engineering.

Deep skills in Spark performance tuning and Databricks cluster setup/optimization.

Strong SQL and analytics data modeling (dimensional/star schema; lakehouse best practices).

CI/CD and automation tooling for data (Git workflows, artifact management) and testing frameworks (pytest, JUnit).

Security-first mindset: roles/instance profiles, secret management, encryption-at-rest/in-transit, and network controls.

Demonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship; excellent communication with stakeholders.

Preferred qualifications, capabilities, and skills:

Experience with Delta Live Tables and advanced governance (catalogs, grants, auditing) in Databricks.

AWS networking knowledge (VPC, subnets, routing, security groups) and data egress controls.

Experience with Terraform for Infra deployments

Cost optimization experience: autoscaling strategies, spot vs on-demand, auto-termination, storage layouts and compaction.

Familiarity with Kafka/MSK or Kinesis Data Streams/Firehose for real-time ingestion.

Observability for data systems (freshness/completeness metrics, lineage, SLAs, alerting).

Experience in financial services or other regulated industries.

About Us

JPMorganChase, one of the oldest financial institutions, offers innovative financial solutions to millions of consumers, small businesses and many of the world's most prominent corporate, institutional and government clients under the J.P. Morgan and Chase brands. Our history spans over 200 years and today we are a leader in investment banking, consumer and small business banking, commercial banking, financial transaction processing and asset management.

We offer a competitive total rewards package including base salary determined based on the role, experience, skill set and location. Those in eligible roles may receive commission-based pay and/or discretionary incentive compensation, paid in the form of cash and/or forfeitable equity, awarded in recognition of individual achievements and contributions. We also offer a range of benefits and programs to meet employee needs, based on eligibility. These benefits include comprehensive health care coverage, on-site health and wellness centers, a retirement savings plan, backup childcare, tuition reimbursement, mental health support, financial coaching and more. Additional details about total compensation and benefits will be provided during the hiring process.

We recognize that our people are our strength and the diverse talents they bring to our global workforce are directly linked to our success. We are an equal opportunity employer and place a high value on diversity and inclusion at our company. We do not discriminate on the basis of any protected attribute, including race, religion, color, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy or disability, or any other basis protected under applicable law. We also make reasonable accommodations for applicants' and employees' religious practices and beliefs, as well as mental health or physical disability needs. Visit our FAQs for more information about requesting an accommodation.

JPMorgan Chase & Co. is an Equal Opportunity Employer, including Disability/Veterans

About the Team

Our Corporate Technology team relies on smart, driven people like you to develop applications and provide tech support for all our corporate functions across our network. Your efforts will touch lives all over the financial spectrum and across all our divisions: Global Finance, Corporate Treasury, Risk Management, Human Resources, Compliance, Legal, and within the Corporate Administrative Office. You'll be part of a team specifically built to meet and exceed our evolving technology needs, as well as our technology controls agenda.", "salary_raw": "Row(double=None, string=None)"}

Apply

Vacancy posted 10 hours ago

Similar jobs that could be interesting for youBased on the Lead Software Engineer - Databricks/Spark/AWS in Columbus, OH vacancy

Lead Software Engineer - Java/AWS/Agentic
As a Senior Lead Software Engineer at JPMorganChase within Employee Experience, you are an integral... ...and optimize data pipelines using Databricks and Apache Spark to enable analytics and machine... ....g., Terraform, CloudFormation, or AWS CDK)Experience with...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
4 days ago
Python PySpark/ Data Engineer
...migrating/developing data solutions in the AWS cloud is required.2+ years of experience... .../implementing data pipelines using Databricks or similar cloud database.Expert level knowledge... ...building real-time data streams using Spark and Experience in Spark.Knowledge or experience...
Databricks
Amazon Web Service
Work experience placement
HAN Staffing
Columbus, OH
1 day ago
Lead Software Engineer - Java, Python preferred
You will raise engineering standards through hands-on development, thoughtful design, and... ...that helps the team grow.As a Lead Software Engineer at JPMorganChase within Asset... ...tools such as PySpark, Snowflake, or Databricks.Financial services knowledge, with an...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
1 day ago
Python Data Engineer
...Python Data Engineer W2 Only Skills Required: (Python, Spark, Pyspark, AWS, SQL), Location - Columbus, OH and Jersey City, NJ... ...- rate range 2- Senior Level Lead Developer Note- 3-5 years will... .... Need- (Python, Pyspark, Databricks and AWS 4 skills required primarily...
Databricks
Amazon Web Service
Syntricate Technologies
Columbus, OH
3 days ago
Lead Software Engineer - Home Lending Technology
...responsibilities Executes software solutions, design,... ...Contributes to software engineering communities of practice... ...Pyspark ~ Proficient in AWS Technologies ~ Experience... ...S3, Lambdas, Snowflake/Databricks About Us Chase is a leading financial services firm,...
Databricks
Amazon Web Service
JPMorgan Chase & Co.
Delaware, OH
6 days ago
Sr. Databricks Developer
...Job Title: Sr. Databricks Developer (5) Location: Columbus... ...We are looking for a Lead Databricks Developer... ...experience in Python and Apache Spark to design, develop, and... ..., SQL, and modern data engineering practices. Key... ...Experience with AWS, Azure, or GCP- Familiarity...
Databricks
Amazon Web Service
Fixed term contract
TalentOla
Columbus, OH
3 days ago
Machine Learning / Data Science Engineer
...DescriptionCapTech Machine Learning Engineers are responsible for... ...data platforms (e.g., AWS, Azure, GCP).Designing,... ...events, and leading teams of junior data scientists... ...leveraging SQL, Spark, NoSQL, and/or cloud data... ...environments such as Snowflake, Databricks, Azure SQL, Amazon...
Databricks
Amazon Web Service
Work at office
Remote work
Visa sponsorship
Work visa
Flexible hours
CapTech
Columbus, OH
20 hours ago
Data Engineer (AWS, Azure, GCP)
...the country.Job DescriptionCapTech Data Engineering consultants enable clients to build and... ...products using Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform... ..., Jenkins Additional Technologies: Spark, Databricks, Kafka, Kinesis, Hadoop, Lambda, EMR Popular...
Databricks
Amazon Web Service
Work at office
Remote work
Visa sponsorship
Work visa
Flexible hours
CapTech
Columbus, OH
4 days ago
Forward Deployed Engineer - Databricks - Forensic Discovery & Financial Crime
$134.5k - $265.1k
...Forward Deployed Engineer (FDE) - DatabricksForward... ...role combines software engineering, data... ...Engineer (FDE) - Databricks on the Forensic, Discovery... ...maintainability.Lead implementation... ...Web Services (AWS), Microsoft Azure,... ...casesExperience with Spark, Airflow, dbt,...
Databricks
Amazon Web Service
Local area
Deloitte
Columbus, OH
3 days ago
Sr Databricks Data Engineer
$116.2k - $229.1k
...Join Deloitte’s AI & Engineering practice and help organizations... ...innovation. As a Databricks Engineer in our AI & Data... ...• Team and Technology Lead: Evaluate, pilot, and... ...such as Azure DevOps, AWS Code Pipeline, Jenkins,... ...Lakehouse architecture, Apache Spark, Delta Lake, cloud-...
Databricks
Amazon Web Service
Local area
Visa sponsorship
Deloitte
Columbus, OH
4 days ago
Practice Lead - Data
...through modern Data, AI, and engineering solutions. Impower designs... ...Role Summary The Practice Lead - Data & Analytics is a technical... ...such as Snowflake, Databricks, dbt, Azure, AWS, and GCP Architect scalable... ...architecture, Delta Lake, Spark, and modern data dbt and modern...
Databricks
Amazon Web Service
Impower.ai
Columbus, OH
4 days ago
Business Banking Data Owner Senior Associate - Insurance
...data governance, data analytics, data engineering, product data, business analysis, or... ....Experience with Snowflake, AWS, Databricks, Hadoop/Spark, Alteryx, Tableau, or similar data platforms... ...or AI/ML data readiness.Chase is a leading financial services firm, helping...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
3 days ago
Business Banking Data Owner Lead - Insurance
...ensure successful delivery of data outcomes.Lead through influence across direct and... ...business domains.Experience with data engineering, analytics, or governance tools such as SQL, Tableau, Alteryx, Spark, AWS, Snowflake, Databricks, Python, or related platforms.Knowledge...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
3 days ago
Lead Software Engineer
...Lead Software Engineer As a Lead Software Engineer at JPMorganChase within CCB, you are an integral part of an agile... ...Expert-level implementation skills with Java, AWS, Database technologies, Python, Scala, Spark, Snowflake. Broad and deep understanding of multiple...
Amazon Web Service
Hackajob
Columbus, OH
2 days ago
Data Engineer
...pipelines using PySpark and Spark in Azure Databricks Designs and implements... ...languages. Perform feature engineering and data preparation for machine... ...storing data (e.g. Azure , AWS, Spark , Pyspark, Scala ,... ...edge technology programs of leading OEMS and Tier 1s in automotive...
Databricks
Amazon Web Service
Full time
Flexible hours
KPIT Technologies
Columbus, OH
1 day ago
Software Engineer
$90k - $115k
...ll Be Doing: Evolent is seeking a Software Engineer to join an Agile team building... ...procedures. ~ Experience with PySpark, Databricks, or other Spark-based frameworks for large-scale data... ...cloud platforms (Azure preferred; AWS or GCP is a plus). ~ Knowledge of...
Databricks
Amazon Web Service
Full time
Immediate start
Work from home
Evolent
Columbus, OH
3 days ago
Consultant, Data Engineer | Databricks
$118k - $222k
...Consultant, Data Engineer will join the Altoids... ...and build Databricks and Delta Lake data... ...SummaryNationwide’s industry leading workforce is... ...the full lifecycle software engineering... ...technologies - Azure and AWS (i.e. Redshift, S3... ..., SQL, R, Python, Spark, UNIX Shell...
Databricks
Amazon Web Service
Full time
Temporary work
Part time
Casual work
Work at office
2 days per week
Nationwide
Columbus, OH
20 hours ago
Databricks Data Engineer
...Databricks Data Engineer Experience Required: 8-10 Dublin, OH ContractAs a Databricks... ...background, you will lead the design of enterprise-scale... ...requires strong expertise in Spark, Delta Lake, Unity Catalog,... ...operational excellence across Azure, AWS, or GCP environments....
Databricks
Amazon Web Service
Diverse Lynx
Dublin, OH
1 day ago
Data Product Manager
$100.71k - $157.63k
...Product Manager , you will lead and inspire cross-... ...Work closely with data engineering, data science, data governance... ...with cloud platforms (AWS, Azure), modern data platforms (Snowflake, Databricks), and data processing frameworks (Hadoop, Spark). Experience with BI...
Databricks
Amazon Web Service
Full time
Work experience placement
Work at office
Local area
Remote work
Flexible hours
UMB Bank
Columbus, OH
3 days ago
AWS Big Data Lead Software Engineer
...applications utilizing leading edge technologies and modernize... ...batch & real-time software components consistent... ...as Apache flink or Spark on very large datasetsDesign... ...that leverage the AWS infrastructure deploying... ...certification on software engineering concepts and 5+ years...
Amazon Web Service
Shift work
JP Morgan Chase
Columbus, OH
20 hours ago
Corporate Technology Team - Lead Data Architect
...the target state architecture goals.As a Lead Data Architect, within the Corporate Technology... ...DB, or data lake platforms such as Databricks or SnowflakeProven ability to set standards... ...classification needsCloud fundamentals, AWS (preferred) or other public cloud...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
4 days ago
AVP, Data Engineering and AI Innovation
...Brief OverviewThe AVP, Data Engineering and AI Innovation will provide... ...integrations and technologies.Lead the transition to new data... ...warehouses (e.g., Snowflake, Databricks, BigQuery) and relational databases... ...tools (e.g., dbt, AWS Glue, Talend). (High proficiency...
Databricks
Amazon Web Service
Full time
Local area
Safelite AutoGlass
Columbus, OH
3 days ago
Lead Software Engineer
$119k - $187k
About this role:Wells Fargo is seeking a Lead Software Engineer (LSE) to serve as a key technical contributor for Customer Account Servicing (CAS... ...architectureExperience working in cloud environments (AWS, Azure, or GCP)Knowledge of observability practices, including...
Amazon Web Service
Full time
Work experience placement
Relocation package
Wells Fargo
Columbus, OH
18 hours ago
GenAI Python Systems Engineer - Senior Associate
$77k - $202k
...Description & SummaryThe OpportunityAs a GenAI Python Systems Engineer - Senior Associate, you will play a pivotal role in... ..., machine learning, and cloud platforms, including AWS, Google Cloud, Microsoft Azure, Databricks, Snowflake, or related data and AI credentials-...
Databricks
Amazon Web Service
Full time
H1b
PwC
Columbus, OH
20 hours ago
Tech Lead/Manager, Software Engineering
$101.9k - $150k
...that help learners achieve their goals and lead a choice-filled life. Our culture... ...see . What you'll do here: As the Software Engineering Manager, you will lead a team dedicated... ...Comfortable designing and deploying solutions on AWS, GCP, or Azure, including containers,...
Amazon Web Service
Full time
Work experience placement
Live in
Local area
Remote work
Worldwide
Cengage Group
Columbus, OH
4 days ago
AI Agentic Sr Lead Software Engineer
...build, and deliver top-notch technology products.As a Senior Lead Software Engineer at JPMorganChase within the Consumer & Community Banking... ...or mobile) with distributed systems and cloud technologies (AWS, GCP, Azure, etc.)Deep expertise with enterprise design patterns...
Amazon Web Service
JP Morgan Chase
Columbus, OH
20 hours ago
Press Brake Operator
$130k - $140k
...DevOps Manager – Lead Databricks, AWS & Azure Cloud Strategy for a High-Growth Data-Driven Organization (Hybrid TX) This Jobot Job is hosted... .... They are seeking a DevOps Manager (Data & Cloud Engineering) to lead the team responsible for modernizing and expanding...
Databricks
Amazon Web Service
Local area
Jobot
Columbus, OH
4 days ago
Data Engineer - Senior Manager
$124k - $280k
...people in data and analytics engineering focus on leveraging advanced... ...and experiences you need to lead and deliver value at this level... ...Certification in Cloud Platforms [e.g., AWS Solutions Architect, AWS Data... ...Core, Snowflake Architect, Databricks Data Engineer Associate] is a...
Databricks
Amazon Web Service
Full time
H1b
PwC
Columbus, OH
20 hours ago
Lead Software Engineer
...adventure where you can push the limits of what's possible.As a Lead Software Engineer at JPMorgan Chase within the Consumer & Community Banking (... .../Flask), with typed contracts, testing, and packaging.Solid AWS background with working knowledge of ECS or EKS,...
Amazon Web Service
JP Morgan Chase
Columbus, OH
2 days ago
Lead Forward Deployed Engineer, ServiceNow
...Deloitte, Forward Deployed Engineers (FDE) don’t just build... ...-scale impact, pairing leading class engineering with... ...cloud environments (AWS, Azure, and/or Google Cloud... ...years of experience in software engineering, data... ...engineering experience with Spark, Airflow/dbt, streaming...
Amazon Web Service
Local area
Shift work
Deloitte
Columbus, OH
4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Software Engineer - Databricks/Spark/AWS. Be the first to apply!