Principal Software Engineer - LLM Optimization

Salary: $61,000 - 101,000 per year Requirements:

• We want someone with formal training or certification in software engineering and 7+ years of applied experience.
• We need deep, hands-on experience with LLM inference systems such as vLLM, TensorRT-LLM, SGLang, LLM-D, or comparable production serving engines.
• We expect strong knowledge of GPU memory architecture, including KV cache sizing and behavior, memory-bandwidth versus compute bottlenecks, and the practical effects of inference-time quantization.
• We need experience with quantization methods and their real-world tradeoffs at scale.
• We value familiarity with speculative decoding and the factors that influence acceptance rates in production workloads.
• We require rigorous benchmarking discipline using tools such as GuideLLM, custom harnesses, or similar approaches, with every claim backed by data.
• We need comfort operating at scale in cloud GPU environments, including AWS, EKS, and managed inference services.
• We value demonstrated awareness of the LLM inference competitive landscape, with experience applying industry benchmarks to improve platform performance and communicating technical trade-offs clearly to senior engineering and business stakeholders.
• We need experience designing and leading adoption of agentic AI-enabled development practices across teams, using enterprise-authorized tools, with standards for human-in-the-loop validation, auditability, traceability, and secure handling of sensitive data.
• We expect a strong understanding of responsible AI use and control expectations in engineering workflows, including security and resiliency implications, data sensitivity, and risk-based governance, with the ability to influence senior technical leaders on safe scaling patterns and reuse.
• Preferred: experience with disaggregated prefill/decode serving architectures, GPU hardware diagnostics such as DCGM, NVML, and XID event tracking, and ML observability and production monitoring.

• We will have you own systematic benchmarking and performance analysis across all production LLM workloads, setting reproducible baselines, detecting regressions early, and measuring the effect of every configuration change before production release.
• We will have you design and run quantization experiments across formats such as FP8, INT8, and INT4 approaches like GPTQ and AWQ, as well as emerging precision formats, while assessing accuracy impact, throughput gains, memory savings, and cost per token.
• We will have you drive speculative decoding strategy across the model portfolio, including draft model, n-gram, and multi-token prediction methods, while managing acceptance-rate measurement and workload-specific recommendations.
• We will have you build and maintain a GPU efficiency scorecard that tracks utilization, memory headroom, cost per 1K tokens, and identified waste to give leadership a clear view of platform efficiency.
• We will have you benchmark our platform against external providers and published industry numbers to identify gaps and drive improvements.
• We will have you lead evaluations of inference engine upgrades, including new scheduler designs, async tensor parallelism, disaggregated prefill/decode, and advanced speculative decoding, with systematic validation before production rollout.
• We will have you partner with the EKS and disaggregated serving teams on KV-cache optimization, prefix caching strategies, and multi-node serving architecture.
• We will have you design and run GPU chaos engineering exercises, including induced failure scenarios, hardware diagnostic monitoring, and measurement of detection and recovery.
• We will have you architect and oversee agentic AI-enabled engineering workflows using enterprise-authorized tools to improve delivery speed, code quality, and operational outcomes, while establishing guardrails for validation, security, resiliency, and reuse.
• We will have you apply knowledge of SDLC tooling, including enterprise-authorized AI-assisted development and automation capabilities, to increase the value delivered by automation at scale.

• Agentic AI
• AI
• AWS
• Architect
• Cloud
• Hardware
• Support
• LLM
• Marketing
• Security
• vLLM
• NodeJS
• Machine Learning

More:

We are JPMorganChase, and hackajob is collaborating with us to connect exceptional professionals to this Principal Software Engineer opportunity within our AI/ML Data Platform team. We are building the infrastructure behind the next generation of enterprise AI, with a strong focus on LLM inference performance, optimization strategy, benchmarking rigor, and efficiency at scale. This is a high-visibility individual contributor role where you will work closely with senior engineering leadership to shape how our platform evolves and to ensure our models are fast, cost-efficient, and production-ready. We offer a competitive total rewards package, with base pay determined by role, experience, skill set, and location, and eligible roles may include commission or discretionary incentive compensation. Our benefits and programs may include comprehensive health coverage, wellness centers, a retirement savings plan, backup childcare, tuition reimbursement, mental health support, and financial coaching, with additional compensation and benefits details shared during hiring. We are committed to diversity, inclusion, and equal opportunity, and our Corporate Functions team plays an essential role in supporting our business, clients, customers, and employees across finance, risk, human resources, marketing, and more.

last updated 36 week of 2026