Principal AI Inference Systems Software Architect

Job Description

Job Description

Principal AI Inference Systems Software Architect

Job Title: Principal AI Inference Systems Software Architect
Location: San Jose, CA or Irvine, CA
Compensation: $250K - $300K base plus equity
Requirements: LLM Inference, AI Runtime Architecture, KV Cache Management, Distributed Inference, Memory Systems Architecture, CXL, Hardware/Software Co-Design, AI Infrastructure, GPU Performance Optimization

Company Brief  

We are a leading provider of high-performance modular memory subsystems and next-generation storage-class memory solutions. We specialize in cutting-edge technologies that enable innovation in AI, cloud computing, and enterprise infrastructure. Our product portfolio includes DDR4/DDR5 DIMMs, NAND-based SSDs, and advanced interconnect technologies such as CXL and PCIe . Join our world-class engineering team and be part of shaping the future of memory and storage architecture.

Role  

This is a highly technical role for an individual who deeply understands AI data center and edge AI inference workloads, model execution, systems software, memory management, and hardware/software co-design. The successful candidate will work closely with the Chief AI Systems Architect, AI Product Strategist, AI Hardware / Memory Systems Architect, and R&D engineering teams to understand how AI workloads execute on current infrastructure, identify software bottlenecks, and architect software that enables new hardware and memory technologies to deliver measurable customer value. We are particularly interested in investigating opportunities involving AI workload, software stack, and inference runtime usage of the hardware to deliver the required performance and efficiency.

Key Responsibilities

• Analyze and optimize AI inference workloads including LLMs, GenAI, Agentic AI, RAG, MoE, multimodal, and edge AI.
• Architect AI runtime and systems software for efficient utilization of compute, memory, and accelerator resources.
• Design software for advanced memory architectures, including HBM, DDR, CXL, memory pooling, disaggregated memory, and KV-cache management.
• Collaborate with hardware architects on CXL, memory systems, AI accelerators, compression engines, and near-memory computing technologies.
• Develop performance optimization strategies focused on throughput, latency, memory efficiency, power, and total cost of ownership.
• Define software architectures enabling practical deployment of CXL-based memory expansion, pooling, and disaggregation.
• Build prototypes, performance models, benchmarks, and emulation environments to validate new architectural concepts before silicon availability.

Required Qualifications

• BS/MS in Computer Science, Computer Engineering, Electrical Engineering, or related field (PhD preferred).
• 10+ years of experience in systems software, AI infrastructure, computer architecture, inference systems, runtimes, or compilers.
• Deep expertise in AI/ML inference workloads and large-scale AI systems.
• Strong programming skills in C/C++ and/or Python.
• Strong understanding of computer architecture, operating systems, memory systems, and data movement.
• Experience translating AI workload requirements into scalable system architectures.
• Expertise in performance analysis, modeling, and benchmarking.

Preferred Experience

• LLM inference, transformers, KV-cache optimization, vLLM, PyTorch, CUDA, Triton, TensorRT, ROCm.
• Quantization, FlashAttention, speculative decoding, distributed inference, and MoE architectures.
• GPU systems, AI accelerators, HBM, DDR, CXL, NUMA, and memory disaggregation.
• Hardware/software co-design for AI infrastructure and data center systems.

#Enghp

Benefits

• Vacation/PTO
• Medical
• Dental
• Vision
• Equity
• 401k w/ match
  
  

- For this position, you must be currently authorized to work in the United States without the need for sponsorship for a non-immigrant visa. CyberCoders will consider for Employment in the City of Los Angeles qualified Applicants with Criminal Histories in a manner consistent with the requirements of the Los Angeles Fair Chance Initiative for Hiring (Ban the Box) Ordinance.This job was first posted by CyberCoders on 10/16/2026 and applications will be accepted on an ongoing basis until the position is filled or closed. Everforth CyberCoders is proud to be an Equal Opportunity Employer

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, sexual orientation, gender identity or expression, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, status as a crime victim, disability, protected veteran status, or any other characteristic protected by law. Our hiring process includes AI screening for keywords and minimum qualifications, and a virtual recruiter as part of the application process. A human recruiter reviews all results. Click here for details on our virtual recruiter .  Everforth CyberCoders will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable state and local law, including but not limited to the Los Angeles County Fair Chance Ordinance, the San Francisco Fair Chance Ordinance, and the California Fair Chance Act. Everforth CyberCoders is committed to working with and providing reasonable accommodation to individuals with physical and mental disabilities. Individuals needing special assistance or an accommodation while seeking employment can contact a member of our Human Resources team at  View email address on ziprecruiter.com to make arrangements.

Salary

$250000 - $300000 USD per year

#ENGT2