Lead Software Engineer - ML Platform

Salary: $61,000 - 101,000 per year Requirements:

• We require formal training or certification in software engineering concepts, plus 5+ years of hands-on experience.
• We need demonstrated experience running ML training workloads in cloud environments and diagnosing issues across both infrastructure and code.
• We require strong Python skills with sound engineering practices, including testing, code reviews, modular design, and dependency management.
• We need experience building automation and CI for ML codebases, including build, test, release, and deployment or promotion workflows.
• We require practical experience with deep learning training workflows and at least one major framework such as PyTorch or TensorFlow.
• We need a solid understanding of training performance and stability, including data loading bottlenecks, mixed precision, checkpointing, reproducibility, and evaluation methods.
• We require experience with distributed training concepts such as DDP, FSDP, or DeepSpeed, plus familiarity with collective communication and scaling bottlenecks.
• We need the ability to profile and tune training systems across CPU, GPU, memory, I/O, networking, and scheduling.
• We require experience with Kubernetes fundamentals for compute-intensive workloads and with AWS services such as EKS or ECR, S3, IAM, VPC networking, CloudWatch, and EC2.
• We need demonstrated experience leading the effective use of approved AI-assisted software development tools, with clear expectations for validating outputs for correctness, performance, and security.
• We require a strong understanding of responsible AI use in engineering workflows, including data sensitivity, secure handling of inputs and outputs, and resilience and security expectations.

• We will design, build, and maintain an end-to-end ML training platform.
• We will run and optimize GPU training workloads, both single-node and distributed, to improve throughput, utilization, and reproducibility.
• We will build and operate training infrastructure on Kubernetes, including resource management and workload troubleshooting.
• We will support Gen AI and LLM training and fine-tuning workflows, including evaluation harnesses, artifact and version governance, and scalable GPU execution aligned with enterprise controls.
• We will implement observability for training systems through metrics, logs, dashboards, alerting, and operational runbooks.
• We will partner with data engineering and platform teams to define interfaces, standards, and guardrails for security and cost control.
• We will improve the developer experience for training through standardized containers, CI/CD, templates, documentation, and self-service workflows.
• We will drive adoption of enterprise-authorized AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes.
• We will apply knowledge of SDLC tooling, including enterprise-authorized AI-assisted development and automation capabilities, to increase the value delivered by automation.

• AI
• AWS
• CI/CD
• Cloud
• CloudWatch
• EC2
• Fine-tuning
• IAM
• Support
• Kubernetes
• LLM
• Marketing
• PyTorch
• Python
• Security
• TensorFlow
• NodeJS
• Airflow
• Spark

More:

hackajob is collaborating with J.P. Morgan to connect them with exceptional professionals for this role. We are offering an opportunity to make an impact on your career in an agile environment focused on building scalable, reliable ML training systems and pipelines on AWS and other cloud platforms. This Lead Software Engineer role sits within AI/ML Data Platforms and is centered on productionizing training workloads, improving performance and cost efficiency, and enabling repeatable, well-governed training across environments. We offer a competitive total rewards package, with base salary based on role, experience, skill set, and location, and eligible roles may also receive commission or discretionary incentive compensation. Our benefits and programs may include comprehensive health care coverage, on-site health and wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, financial coaching, and more. We are committed to diversity and inclusion and provide reasonable accommodations where needed. This role is subject to Section 19 of the Federal Deposit Insurance Act. Our Corporate Functions teams support finance, risk, human resources, marketing, and other essential business areas that help set our company, clients, customers, and employees up for success.

last updated 36 week of 2026