Principal Software Engineer - LLM Optimization - Jersey City

Salary: $61,000 - 101,000 per year Requirements:

• Formal training or certification in software engineering concepts, with at least 7 years of applied experience
• Extensive hands-on experience with production LLM inference systems such as vLLM, TensorRT-LLM, SGLang, LLM-D, or equivalent engines
• Strong knowledge of GPU memory architecture, including KV-cache sizing and behavior, memory-bandwidth versus compute bottlenecks, and the practical effects of inference-time quantization
• Experience with quantization methods and their real-world trade-offs at scale
• Familiarity with speculative decoding and the factors affecting acceptance rates in production workloads
• Strong benchmarking skills using GuideLLM, custom harnesses, or comparable tools, with quantitative evidence supporting performance claims
• Experience operating large-scale cloud GPU infrastructure, including AWS, EKS, and managed inference services
• Awareness of the LLM inference competitive landscape and a record of applying industry benchmarks to improve platforms
• Ability to explain technical trade-offs clearly to senior engineering and business stakeholders
• Experience designing and leading the adoption of agentic AI-enabled development practices across teams, using enterprise-authorized tools and establishing standards for human review, traceability, and secure handling of sensitive data
• Understanding of responsible AI use and engineering-workflow controls, including security, resiliency, data sensitivity, and risk-based governance, with the ability to influence senior technical leaders on safe scaling and reuse
• Employment offers for this position are contingent on our review of criminal conviction history, including pretrial diversions or program entries, as required under Section 19 of the Federal Deposit Insurance Act
• Preferred: Experience with disaggregated prefill/decode serving, GPU diagnostics such as DCGM, NVML, or XID event tracking, ML observability, and production monitoring

• Own systematic benchmarking and performance analysis across production LLM workloads; establish reproducible baselines, identify regressions, and quantify configuration changes before production deployment
• Plan and run quantization experiments, including FP8, INT8/INT4 methods such as GPTQ and AWQ, and newer precision formats; assess accuracy, throughput, memory use, and cost per token
• Set the speculative decoding strategy across our model portfolio, evaluating draft models, n-gram methods, and multi-token prediction, and recommend configurations based on measured acceptance rates and workloads
• Create and maintain a GPU efficiency scorecard covering utilization, memory headroom, cost per 1,000 tokens, and identified waste
• Compare our platform with external providers and published industry benchmarks, then drive improvements to close performance gaps
• Lead evaluations of inference engine upgrades, including scheduler designs, asynchronous tensor parallelism, disaggregated prefill/decode, and advanced speculative decoding, validating changes before production rollout
• Work with our EKS and disaggregated serving teams on KV-cache optimization, prefix caching, and multi-node serving architecture
• Design and conduct GPU chaos engineering exercises, including failure simulations and measurement of hardware diagnostics, detection, and recovery
• Design and govern agentic AI-enabled engineering workflows using enterprise-authorized tools to improve delivery speed, code quality, and operational results; establish guardrails for validation, security, resiliency, and reuse across teams
• Apply knowledge of the software development lifecycle toolchain, including authorized AI-assisted development and automation capabilities, to increase the value of automation at scale

• Agentic AI
• AI
• AWS
• Cloud
• Hardware
• Support
• LLM
• Security
• vLLM
• NodeJS
• Architect
• Machine Learning
• Marketing

More:

We are partnering with hackajob to connect exceptional professionals with this opportunity at JPMorganChase. As a Principal Software Engineer on our AI/ML Data Platform team, you will be a senior individual contributor shaping LLM inference performance and the evolution of infrastructure that supports enterprise AI at scale. Our firm serves consumers, small businesses, and corporate, institutional, and government clients through the J.P. Morgan and Chase brands, with businesses spanning banking, financial transaction processing, and asset management. We offer a competitive total rewards package, with compensation based on role, experience, skills, and location; eligible employees may also receive incentive compensation. Benefits may include health coverage, wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, and financial coaching, subject to eligibility. We value diversity and inclusion, are an equal opportunity employer, and provide reasonable accommodations. Our Corporate Functions teams support the success of our businesses, clients, customers, and employees.

last updated 42 week of 2026