Lead Software Engineer - ML Platform (AI/ML)

Salary: $61,000 - 101,000 per year Requirements:

• Formal training or certification in software engineering concepts, with at least 5 years of applied experience.
• Experience running machine learning training in cloud environments and diagnosing issues across infrastructure and code.
• Strong Python skills and sound engineering practices, including testing, code reviews, modular design, and dependency management.
• Experience automating CI workflows for machine learning codebases, including build, test, release, and deployment or promotion processes.
• Hands-on experience with deep learning training workflows and at least one major framework, such as PyTorch or TensorFlow.
• Understanding of training performance and stability, including data-loading bottlenecks, mixed precision, checkpointing, reproducibility, and evaluation methods.
• Experience with distributed training concepts and tools, such as DDP, FSDP, or DeepSpeed, as well as collective communication, scaling, and bottleneck analysis.
• Ability to profile and optimize training systems across CPU and GPU utilization, memory, I/O throughput, networking, and scheduling.
• Experience with Kubernetes fundamentals for compute-intensive workloads and AWS services such as EKS, ECR, S3, IAM, VPC/networking, CloudWatch, and EC2.
• Experience leading effective use of approved AI-assisted software development tools, and establishing expectations for validating generated output for correctness, performance, and security.
• Understanding of responsible AI use in engineering, including data sensitivity, secure handling of inputs and outputs, and security and resiliency requirements; experience coaching engineers in safe, compliant adoption.
• Preferred: Experience running training workloads across multiple cloud platforms and managing portability, performance, and governance across environments.
• Preferred: Familiarity with cloud-native networking and storage patterns for high-throughput training and artifact management.
• Preferred: Experience optimizing training input pipelines, working with large datasets, and using approaches such as sharding, prefetching, caching, and formats such as Parquet or WebDataset.
• Preferred: Familiarity with distributed compute frameworks such as Spark, Ray, or Dask for feature or dataset generation.
• Preferred: Familiarity with workflow orchestration tools and patterns, such as Airflow-like systems or Argo Workflows, and with model registry concepts.
• Preferred: Experience improving training cost and performance through right-sizing, scheduling, interruptible capacity strategies where applicable, budget guardrails, and quota planning.
• Preferred: Strong observability practices for training systems, including metrics, logs, traces, GPU telemetry, dashboards, and alert tuning.

• Design, develop, and maintain an end-to-end machine learning training platform.
• Operate and optimize single-node and distributed GPU training workloads to improve throughput, utilization, and reproducibility.
• Build and run training infrastructure on Kubernetes platforms, including EKS, and manage resources and workload troubleshooting.
• Enable generative AI and large language model training and fine-tuning workflows, including supervised fine-tuning, evaluation harnesses, artifact and version governance, and scalable GPU execution aligned with enterprise controls.
• Establish observability for training systems through metrics, logs, dashboards, alerting, and operational runbooks.
• Work with data engineering and platform teams to establish interfaces, standards, and safeguards for security, access, and cost controls.
• Improve the training developer experience with standardized containers, CI/CD, templates, documentation, and self-service workflows.
• Encourage team adoption of enterprise-authorized AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes, while maintaining validation standards such as secure coding, peer review, and automated testing.
• Apply software development lifecycle tools, including authorized AI-assisted development and automation capabilities, to increase the value of automation.

• AI
• Airflow
• AWS
• CI/CD
• Cloud
• CloudWatch
• EC2
• Fine-tuning
• IAM
• Support
• Kubernetes
• Machine Learning
• Marketing
• Model Training
• PyTorch
• Python
• Security
• Spark
• TensorFlow
• NodeJS
• LLM

More:

We are JPMorganChase, a long-established financial institution serving consumers, small businesses, and corporate, institutional, and government clients through the J.P. Morgan and Chase brands. We operate across investment banking, consumer and small business banking, commercial banking, financial transaction processing, and asset management. This Lead Software Engineer opportunity is within our AI/ML Data Platforms work, where our agile teams build and operate scalable machine learning training systems across AWS and other cloud platforms. Our Corporate Functions professionals work across areas including finance, risk, human resources, and marketing to support our businesses, clients, customers, and employees. We offer competitive total rewards, with compensation based on role, experience, skills, and location; eligible employees may also receive incentive compensation. Benefits, subject to eligibility, include health coverage, on-site wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, and financial coaching. We are an equal opportunity employer, value diversity and inclusion, and provide reasonable accommodations. This position is subject to Section 19 of the Federal Deposit Insurance Act, and an offer is contingent on our review of criminal conviction history, including pretrial diversions or program entries.

last updated 42 week of 2026