Lead Software Engineer - Databricks/Spark/AWS

Full-time

JPMorgan Chase & Co.

{"description": " Job Description

This is your chance to change the path of your career and guide multiple teams to success at one of the world's leading financial institutions.

As a Lead Software Engineer at JPMorgan Chase within Corporate Sector, Chief Technology Office, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm's business objectives.

Job Responsibilities:

Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark (Core, SQL, Structured Streaming).

Establish lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) and ensure performance at scale.

Own Databricks cluster strategy and setup: runtime selection, autoscaling, driver/executor sizing, Spark configs, unit scripts, cluster policies, pools, and instance profiles.

Orchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.

Design secure data ingestion and transformation frameworks leveraging AWS services:

S3 for data lake storage and lifecycle management

Glue for catalog/metadata and ETL jobs

IAM and Secrets Manager for role-based access and credential management

CloudWatch for logging, metrics, and alerting

Lambda for serverless utilities

Kinesis and/or Kafka/MSK for streaming ingestion

Enforce data quality, lineage, and governance using Unity Catalog and/or Glue Catalog; embed expectations and validation into pipelines.

Drive Spark performance engineering: partitioning strategies, file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, and job right-sizing to optimize cost.

Build reusable libraries, frameworks, and APIs in Python and/or Java; oversee unit, integration, and data validation testing.

Implement CI/CD for data projects (Git-based workflows), Terraform Infrastructure deployments environment promotion, and automated deployments; champion engineering standards and code reviews.

Partner with platform security and networking teams to enforce encryption, network controls, and least-privilege access; ensure compliance with organizational policies.

Lead incident response and root-cause analysis; establish SLAs, observability, and runbooks; drive continuous improvement in reliability and cost efficiency.

Mentor engineers, guide technical design reviews, and contribute to roadmaps and backlog prioritization; collaborate with product owners and cross-functional teams.

Required qualifications, capabilities, and skills:

Formal training or certification on software engineering concepts and 5+ years applied experience.

10+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR.

Strong proficiency in Python and/or Java for data processing, platform tooling, and automation.

Hands-on Databricks expertise (Delta Lake, Unity Catalog, Workflows, Repos/notebooks, SQL Warehouses).

Solid AWS experience: S3, IAM, Glue, CloudWatch, Kinesis / MSK, DynamoDB

Proven track record architecting and operating ETL/ELT pipelines (batch and streaming), with schema design/evolution, SLAs, and reliability engineering.

Deep skills in Spark performance tuning and Databricks cluster setup/optimization.

Strong SQL and analytics data modeling (dimensional/star schema; lakehouse best practices).

CI/CD and automation tooling for data (Git workflows, artifact management) and testing frameworks (pytest, JUnit).

Security-first mindset: roles/instance profiles, secret management, encryption-at-rest/in-transit, and network controls.

Demonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship; excellent communication with stakeholders.

Preferred qualifications, capabilities, and skills:

Experience with Delta Live Tables and advanced governance (catalogs, grants, auditing) in Databricks.

AWS networking knowledge (VPC, subnets, routing, security groups) and data egress controls.

Experience with Terraform for Infra deployments

Cost optimization experience: autoscaling strategies, spot vs on-demand, auto-termination, storage layouts and compaction.

Familiarity with Kafka/MSK or Kinesis Data Streams/Firehose for real-time ingestion.

Observability for data systems (freshness/completeness metrics, lineage, SLAs, alerting).

Experience in financial services or other regulated industries.

About Us

JPMorganChase, one of the oldest financial institutions, offers innovative financial solutions to millions of consumers, small businesses and many of the world's most prominent corporate, institutional and government clients under the J.P. Morgan and Chase brands. Our history spans over 200 years and today we are a leader in investment banking, consumer and small business banking, commercial banking, financial transaction processing and asset management.

We offer a competitive total rewards package including base salary determined based on the role, experience, skill set and location. Those in eligible roles may receive commission-based pay and/or discretionary incentive compensation, paid in the form of cash and/or forfeitable equity, awarded in recognition of individual achievements and contributions. We also offer a range of benefits and programs to meet employee needs, based on eligibility. These benefits include comprehensive health care coverage, on-site health and wellness centers, a retirement savings plan, backup childcare, tuition reimbursement, mental health support, financial coaching and more. Additional details about total compensation and benefits will be provided during the hiring process.

We recognize that our people are our strength and the diverse talents they bring to our global workforce are directly linked to our success. We are an equal opportunity employer and place a high value on diversity and inclusion at our company. We do not discriminate on the basis of any protected attribute, including race, religion, color, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy or disability, or any other basis protected under applicable law. We also make reasonable accommodations for applicants' and employees' religious practices and beliefs, as well as mental health or physical disability needs. Visit our FAQs for more information about requesting an accommodation.

JPMorgan Chase & Co. is an Equal Opportunity Employer, including Disability/Veterans

About the Team

Our Corporate Technology team relies on smart, driven people like you to develop applications and provide tech support for all our corporate functions across our network. Your efforts will touch lives all over the financial spectrum and across all our divisions: Global Finance, Corporate Treasury, Risk Management, Human Resources, Compliance, Legal, and within the Corporate Administrative Office. You'll be part of a team specifically built to meet and exceed our evolving technology needs, as well as our technology controls agenda.", "salary_raw": "Row(double=None, string=None)"}

Apply

Vacancy posted 1 day ago

Similar jobs that could be interesting for youBased on the Lead Software Engineer - Databricks/Spark/AWS in Columbus, OH vacancy

Lead Software Engineer - Java/AWS/Agentic
As a Senior Lead Software Engineer at JPMorganChase within Employee Experience, you are an integral... ...and optimize data pipelines using Databricks and Apache Spark to enable analytics and machine... ....g., Terraform, CloudFormation, or AWS CDK)Experience with...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
1 day ago
Lead Software Engineer: Data Engineering
...of what's possible. As an Engineering Lead at JPMorganChase within Employee... ...engineering solutions on AWS and Databricks. As a core technical... ...responsibilitiesExecutes creative software solutions, design,... ...on AWS and Databricks using Spark and Delta.Builds reusable Python...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
2 days ago
Lead Software Engineer - Databricks AWS
$61k - $101k
...training or certification in software engineering concepts, plus at least 5... ...significant production work with Spark on Databricks or EMR. We need strong... .... We require solid AWS experience with S3, IAM, Glue... ...Responsibilities: We lead the architecture and delivery...
Databricks
Amazon Web Service
Full time
Work at office
J.P. Morgan
Columbus, OH
10 days ago
Python PySpark/ Data Engineer
...migrating/developing data solutions in the AWS cloud is required.2+ years of experience... .../implementing data pipelines using Databricks or similar cloud database.Expert level knowledge... ...building real-time data streams using Spark and Experience in Spark.Knowledge or experience...
Databricks
Amazon Web Service
Work experience placement
HAN Staffing
Columbus, OH
2 days ago
Big Data Architect
$100k - $150k
...technology consulting and software development company... ...to design and lead enterprise-scale data... ...expertise in Snowflake, Databricks, BigQuery, Redshift, Spark, Kafka, lakehouse... ...with data engineering, analytics, ML, and... ...Snowflake, Databricks, AWS, Azure, or GCP) and...
Databricks
Amazon Web Service
Full time
H1b
Local area
Remote work
Visa sponsorship
Bright Vision Technologies
Columbus, OH
3 days ago
Lead Software Engineer - Home Lending Technology
...responsibilities Executes software solutions, design,... ...Contributes to software engineering communities of practice... ...Pyspark ~ Proficient in AWS Technologies ~ Experience... ...S3, Lambdas, Snowflake/Databricks About Us Chase is a leading financial services firm,...
Databricks
Amazon Web Service
JPMorgan Chase & Co.
Delaware, OH
27 days ago
Sr DataBricks developer
Sr Databricks Developer Location: Columbus, OH, Fulltime Required: • Experience with design,... ...and implementation of data pipeline using Spark/Python Applications. • Good knowledge and... ...setup of CI/CD pipeline. • Experience with AWS cloud platform to perform big data...
Databricks
Amazon Web Service
Full time
Syntricate Technologies
Columbus, OH
2 days ago
Machine Learning / Data Science Engineer
...DescriptionCapTech Machine Learning Engineers are responsible for... ...data platforms (e.g., AWS, Azure, GCP).Designing,... ...events, and leading teams of junior data scientists... ...leveraging SQL, Spark, NoSQL, and/or cloud data... ...environments such as Snowflake, Databricks, Azure SQL, Amazon...
Databricks
Amazon Web Service
Work at office
Remote work
Visa sponsorship
Work visa
Flexible hours
CapTech
Columbus, OH
1 day ago
Lead Software Engineer
As a Lead Software Engineer at JPMorganChase within CCB, you are an integral part of an agile team that works to enhance,... ...starter. Expert-level implementation skills with Java, AWS, Database technologies, Python, Scala, Spark, Snowflake. Broad and deep understanding of...
Amazon Web Service
JP Morgan Chase
Columbus, OH
3 days ago
Solutions Architect
...Architect - AI to design, lead, and deliver... ...across cloud platforms (AWS, Azure, GCP) Evaluate... ...with data scientists, engineers, and developers to ensure... ...consulting, software architecture, or solution... ...data engineering tools (Spark, Kafka, Databricks, Snowflake) AI/ML...
Databricks
Amazon Web Service
Work at office
Local area
Remote work
Visa sponsorship
Work visa
Flexible hours
CapTech
Columbus, OH
5 days ago
MLOps Engineer (Databricks/AWS)
Overview Inabia is seeking an MLOps Engineer (Databricks/AWS) to design, develop, and maintain end-to-end machine learning operations pipelines... ...automation using Terraform. Strong understanding of Apache Spark architecture, optimization, and distributed data processing...
Databricks
Amazon Web Service
Inabia Solutions and Consulting, Inc.
Columbus, OH
2 days ago
Data Engineer (AWS, Azure, GCP)
...the country.Job DescriptionCapTech Data Engineering consultants enable clients to build and... ...products using Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform... ..., Jenkins Additional Technologies: Spark, Databricks, Kafka, Kinesis, Hadoop, Lambda, EMR Popular...
Databricks
Amazon Web Service
Work at office
Remote work
Visa sponsorship
Work visa
Flexible hours
CapTech
Columbus, OH
5 days ago
Forward Deployed Engineer - Databricks - Forensic Discovery & Financial Crime
$134.5k - $265.1k
...Forward Deployed Engineer (FDE) - DatabricksForward... ...role combines software engineering, data... ...Engineer (FDE) - Databricks on the Forensic, Discovery... ...maintainability.Lead implementation... ...Web Services (AWS), Microsoft Azure,... ...casesExperience with Spark, Airflow, dbt,...
Databricks
Amazon Web Service
Local area
Deloitte
Columbus, OH
4 days ago
Business Banking Data Owner Senior Associate - Insurance
...data governance, data analytics, data engineering, product data, business analysis, or... ....Experience with Snowflake, AWS, Databricks, Hadoop/Spark, Alteryx, Tableau, or similar data platforms... ...or AI/ML data readiness.Chase is a leading financial services firm, helping...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
4 days ago
Data Platform Engineer
$100k - $150k
...technology consulting and software development company... ...Job Title: Data Platform Engineer Location: 100% Remote... ...on expertise with Apache Spark (Scala, Python, or Java)... ...cloud platforms such as AWS EMR, Azure HDInsight, or Databricks. Familiarity with modern...
Databricks
Amazon Web Service
Full time
H1b
Local area
Immediate start
Remote work
Visa sponsorship
Bright Vision Technologies
Powell, OH
a month ago
Business Banking Data Owner Lead - Insurance
...ensure successful delivery of data outcomes.Lead through influence across direct and... ...business domains.Experience with data engineering, analytics, or governance tools such as SQL, Tableau, Alteryx, Spark, AWS, Snowflake, Databricks, Python, or related platforms.Knowledge...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
4 days ago
Investment Data Analyst
$100k - $168k
...with Snowflake and/or Databricks, including data storage... ...Key Responsibilities Engineer batch and streaming data... ...Python, SQL, OpenFlow, Spark, or comparable... ...Nationwide’s industry leading workforce is passionate... ...technologies – Azure and AWS (i.e., Redshift, S3, EC...
Databricks
Amazon Web Service
Full time
Temporary work
Part time
Casual work
Work at office
Remote work
Nationwide
Columbus, OH
7 hours ago
Lead Software Engineer DevOps & Infrastructure
$61k - $101k
...Requirements: We expect formal software engineering training or certification, plus 5+ years... ...production data solutions with Databricks, Apache Spark or PySpark, Delta Lake, Lakeflow pipelines... ...applications and systems. We lead evaluation sessions with external vendors...
Databricks
Full time
J.P. Morgan
Columbus, OH
7 days ago
Consultant, Data Engineer | Databricks
$118k - $222k
...Consultant, Data Engineer will join the Altoids... ...and build Databricks and Delta Lake data... ...SummaryNationwide’s industry leading workforce is... ...the full lifecycle software engineering... ...technologies - Azure and AWS (i.e. Redshift, S3... ..., SQL, R, Python, Spark, UNIX Shell...
Databricks
Amazon Web Service
Full time
Temporary work
Part time
Casual work
Work at office
2 days per week
Nationwide
Columbus, OH
2 days ago
Consultant, Data Engineer | Snowflake Data Engineer (Python / Snowpark)
$118k - $222k
...program.Snowflake Data Engineer (Python / Snowpark)... ...graphs and scheduling.Databricks, Apache Spark, PySpark, Apache... ...cloud environments like AWS, Azure, or Google Cloud... ...’s industry-leading workforce is passionate... ...the full lifecycle of software engineering development...
Databricks
Amazon Web Service
Full time
Temporary work
Part time
Casual work
Work at office
2 days per week
Nationwide
Columbus, OH
5 days ago
Senior Data Engineer
...big data architecture using modern data engineering technologies. The Senior Data Engineer... ...maintain new cloud data pipelines using Databricks and PySpark. Required... ...years working with Cloud Technologies (AWS/Azure ecosystem). ~ Proficiency with...
Databricks
Amazon Web Service
3B Staffing LLC
Columbus, OH
3 days ago
AI Sales Director
...the entire sales process — from lead generation to contract closing... ...with ecosystem partners like Databricks and Snowflake for co-selling... ...Delivery Director and pre-sales engineers on solution strategies.•... ...Databricks, Snowflake, Microsoft, AWS, and Google.Communication & Relationship...
Databricks
Amazon Web Service
Contract work
Gables Search Group
Columbus, OH
2 days ago
We are hiring: Sr. MLOps Engineer at Columbus, OH
...Job Title: Sr. MLOps Engineer Location: Columbus, OH 5 Days onsite role Duration: Long Term Project... ...Description: DABS pipeline, I/CD, MLOPs principles, AWS MLOPs (Sagemaker, ECS, Lambda Databricks, Genie one, Unity catalog, SQL warehouse, compute...
Databricks
Amazon Web Service
Realtech Services
Columbus, OH
3 days ago
Corporate Technology Team - Lead Data Architect
...the target state architecture goals.As a Lead Data Architect, within the Corporate Technology... ...DB, or data lake platforms such as Databricks or SnowflakeProven ability to set standards... ...classification needsCloud fundamentals, AWS (preferred) or other public cloud...
Databricks
Amazon Web Service
JP Morgan Chase
Columbus, OH
5 days ago
DevOps Engineer
...DevOps Engineer This role is W2 only and cannot work corp to corp/1099. Role Overview As an Analytics Platform Engineer, you will support Azure and AWS Databricks from a DevOps perspective. This role requires a hybrid skill set spanning cloud infrastructure management...
Databricks
Amazon Web Service
Apex Systems
Columbus, OH
2 days ago
Press Brake Operator
$130k - $140k
...DevOps Manager – Lead Databricks, AWS & Azure Cloud Strategy for a High-Growth Data-Driven Organization (Hybrid TX) This Jobot Job is hosted... .... They are seeking a DevOps Manager (Data & Cloud Engineering) to lead the team responsible for modernizing and expanding...
Databricks
Amazon Web Service
Local area
Jobot
Columbus, OH
5 days ago
AVP, Data Engineering and AI Innovation
...Brief OverviewThe AVP, Data Engineering and AI Innovation will provide... ...integrations and technologies.Lead the transition to new data... ...warehouses (e.g., Snowflake, Databricks, BigQuery) and relational databases... ...tools (e.g., dbt, AWS Glue, Talend). (High proficiency...
Databricks
Amazon Web Service
Full time
Local area
Safelite AutoGlass
Columbus, OH
4 days ago
Senior Data Scientist
$170k - $215k
.... Together, via our market-leading brands, we help our clients... ...collaboration with content and product engineering teams Implement rigorous... ...(Docker, Kubernetes, AWS/GCP) Experience moving AI... ...Big Data tools like Apache Spark, Hadoop, or Databricks Benefits: Medical and...
Databricks
Amazon Web Service
Full time
Contract work
Temporary work
Work experience placement
Local area
Remote work
Flexible hours
Norstella
Columbus, OH
3 days ago
Head of Delivery Management
$155.6k - $306.8k
...Work you'll do In this role, you will lead core delivery execution for industrial... ...platform depth to review offshore data engineering output, identify architectural risks, and... ...in engagements, you may coordinate with Databricks and AWS alliance solution architects for...
Databricks
Amazon Web Service
Local area
Deloitte
Columbus, OH
1 day ago
Lead Software Engineer
$119k - $187k
About this role:Wells Fargo is seeking a Lead Software Engineer (LSE) to serve as a key technical contributor for Customer Account Servicing (CAS... ...architectureExperience working in cloud environments (AWS, Azure, or GCP)Knowledge of observability practices, including...
Amazon Web Service
Full time
Work experience placement
Relocation package
Wells Fargo
Columbus, OH
1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Software Engineer - Databricks/Spark/AWS. Be the first to apply!