Principal AI Inference Systems Software Architect
$250k - $300kCyberCoders
Job Title: Principal AI Inference Systems Software Architect
Location: San Jose, CA or Irvine, CA
Compensation: $250K - $300K base plus equity
Requirements: LLM Inference, AI Runtime Architecture, KV Cache Management, Distributed Inference, Memory Systems Architecture, CXL, Hardware/Software Co-Design, AI Infrastructure, GPU Performance Optimization
We are a leading provider of high-performance modular memory subsystems and next-generation storage-class memory solutions. We specialize in cutting-edge technologies that enable innovation in AI, cloud computing, and enterprise infrastructure. Our product portfolio includes DDR4/DDR5 DIMMs, NAND-based SSDs, and advanced interconnect technologies such as CXL and PCIe . Join our world-class engineering team and be part of shaping the future of memory and storage architecture.
Role This is a highly technical role for an individual who deeply understands AI data center and edge AI inference workloads, model execution, systems software, memory management, and hardware/software co-design. The successful candidate will work closely with the Chief AI Systems Architect, AI Product Strategist, AI Hardware / Memory Systems Architect, and R&D engineering teams to understand how AI workloads execute on current infrastructure, identify software bottlenecks, and architect software that enables new hardware and memory technologies to deliver measurable customer value. We are particularly interested in investigating opportunities involving AI workload, software stack, and inference runtime usage of the hardware to deliver the required performance and efficiency.Key Responsibilities
- Analyze and optimize AI inference workloads including LLMs, GenAI, Agentic AI, RAG, MoE, multimodal, and edge AI.
- Architect AI runtime and systems software for efficient utilization of compute, memory, and accelerator resources.
- Design software for advanced memory architectures, including HBM, DDR, CXL, memory pooling, disaggregated memory, and KV-cache management.
- Collaborate with hardware architects on CXL, memory systems, AI accelerators, compression engines, and near-memory computing technologies.
- Develop performance optimization strategies focused on throughput, latency, memory efficiency, power, and total cost of ownership.
- Define software architectures enabling practical deployment of CXL-based memory expansion, pooling, and disaggregation.
- Build prototypes, performance models, benchmarks, and emulation environments to validate new architectural concepts before silicon availability.
Required Qualifications
- BS/MS in Computer Science, Computer Engineering, Electrical Engineering, or related field (PhD preferred).
- 10+ years of experience in systems software, AI infrastructure, computer architecture, inference systems, runtimes, or compilers.
- Deep expertise in AI/ML inference workloads and large-scale AI systems.
- Strong programming skills in C/C++ and/or Python.
- Strong understanding of computer architecture, operating systems, memory systems, and data movement.
- Experience translating AI workload requirements into scalable system architectures.
- Expertise in performance analysis, modeling, and benchmarking.
Preferred Experience
- LLM inference, transformers, KV-cache optimization, vLLM, PyTorch, CUDA, Triton, TensorRT, ROCm.
- Quantization, FlashAttention, speculative decoding, distributed inference, and MoE architectures.
- GPU systems, AI accelerators, HBM, DDR, CXL, NUMA, and memory disaggregation.
- Hardware/software co-design for AI infrastructure and data center systems.
Benefits
- Vacation/PTO
- Medical
- Dental
- Vision
- Equity
- 401k w/ match
$250k - $300k
Principal AI Inference Systems Software Architect Job Title: Principal AI Inference Systems Software Architect Location: San Jose, CA or Irvine, CA Compensation: $250K - $300K base plus equity Requirements: LLM Inference, AI Runtime Architecture, KV Cache...PrincipalOngoing contractLocal area$138.84k - $208k
...our world. Across enterprise, cloud and AI, and carrier architectures, our innovative... ...performance response for thousands of daily system usersOwn and continuously improve the... ...may require access to technology and/or software subject to U.S. export control laws and regulations...PrincipalPermanent employmentFull timeInternshipWork from home- ...Summary Celestica is seeking a Principal Hardware Systems Architect for Compute Platforms to provide senior... ...and development of next-generation AI, accelerated computing, and high-performance... ...GPU-based AI training or inference platforms. Experience with NVIDIA...PrincipalLocal area
- NVIDIA Corporation in Santa Clara is seeking a Principal LPU Technical Product Marketing Manager to shape the go-to-market for Groq-based LPUs, drive low-latency inference, and highlight NVIDIA’s leadership in AI acceleration. You will work with engineering and product...Principal
$250k - $300k
...Job Description Job Description Principal AI Memory Systems Architect Job Title: Principal AI Memory... ...Architecture, Memory Hierarchy, CXL, HBM, AI Inference, Computer Architecture, Memory... ..., AI Product Strategist, AI Software Architect, and R&D teams to develop...PrincipalOngoing contractLocal area$200k - $300k
..., CASoftware - Platform Software /Full-time /On-siteRole... ...OverviewWe are looking for a Principal AI SoC Runtime Software Architect to own the software... ...ingest, preprocessing, AI inference, postprocessing, and... ...combines deep runtime and systems-software expertise with...PrincipalFull timeContract workFlexible hours- ...Matrix and help build the future of AI inference. We’ve pioneered a memory-centric architecture... ...on a global scale. From chips and software to rack-scale systems, our team is tackling some of AI... ...3 days per week.The role: Principal System Software Engineer, AI Inference...Principal3 days per week
$231.44k - $282.88k
...computing platforms for the 21st century. As a datacenter system software architect, you’ll play a leading role in designing and... ...application workloads, including distributed databases, AI/ML training/inference, agentic AI, and high-throughput microservices. Experience...Full timeWork experience placement$224k - $356.5k
We are looking for a Senior System Software Architect to help build NVIDIA Halos, our full-stack safety platform for physical AI: autonomous vehicles, humanoids, industrial robots, and intelligent machines that sense, decide, and act in the real world. This role sits at...Full time$152k - $241.5k
...amazing people. NVIDIA is looking for an experienced Senior Software and System Architect to join our Networking Software Architecture group. Nvidia'... ...2026.This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes.NVIDIA is committed to fostering...Full timeRemote work$186.2k - $273.07k
...standards for technologies like AI, data centers, automotive and... ..., metrology, and analytics systems detect and monitor defects across... ...(MACH) division is KLA’s software powerhouse, responsible for developing... ...role you will design and architect next-generation High Performance...Minimum wageFull timeWork experience placement- ...Overview We are looking for a Principal AI SoC Runtime Software Architect to own the software architecture that... ...sensor ingest, preprocessing, AI inference, postprocessing, and delivery of... ...candidate combines deep runtime and systems-software expertise with a strong...PrincipalFull time
$124k - $195.5k
NVIDIA is looking for a passionate and dedicated SW & System Architect to join our SW Architecture group. The position includes researching new... ...24, 2026.This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes.NVIDIA is committed to...Full timeRemote work$180k - $225k
...and density challenges for datacenter products serving the most critical AI customers. You will work directly with customers defining roadmaps aligned with their platform guided by compelling system level outcomes ensuring Renesas remains dominant in this space. You will...PrincipalFull timeRemote work$272k - $431.25k
...tapping into the unlimited potential of AI to define the next era of computing. An... ...legacy of innovation! We are seeking a Principal SW Architect, Networking to build the future of AI... ...development of AI networks alongside hardware, software, and applicationsGuaranteeing...PrincipalFull time- Cerebras Systems builds the world's largest AI chip, 56 times larger than GPUs... ...training and inference speeds; over 10 times... ...a Network Systems Architect, you will define the... ...experience, including principal-level technical... ...make decisions across software, accelerator I/O, topology...Principal
$272k - $431.25k
We are now looking for a CPU System Software Architect!Do you want to be a part of the Artificial Intelligence Revolution? Would you like to work... ...architectures that are at the forefront of accelerating agentic AI, building data centers, and enabling other high-performance...PrincipalFull time- ...The Principal AI Solutions Architect at DataDirect Networks (DDN) combines technical mastery with outcome driven strategic selling... ...Large Language Models (LLMs), neural networks, RAG, Inference, and recommendation systems, to drive innovation and client success....PrincipalFull time
$169.47k - $242.36k
...devices, non-silicon technology, have kept systems in sync, but they struggle in harsher,... ...applications, including high-growth ones in AI datacenters, automated driving,... ...visit: .Job SummaryThe Networking System Architect serves as the resident networking systems...Principal$100k
...leading the industry on cutting-edge AI technology, revolutionizing performance... ...must evolve to unify innovations in software models, compilers, platforms,... ....We are seeking a highly experienced Principal Power Systems Architect to define and deliver next-generation...PrincipalPermanent employment$124.5k - $272k
...security and networking for the cloud and AI era. We secure and accelerate cloud, data... ...Machine Learning Scientist, you own the inference and optimization layer that makes AI in agentic... ...as the platform matures.Partner with the systems and backend engineers to ship...Principal- Company DescriptionAxiado is an AI-enhanced security processor... ...and management of every digital system. The company was founded in 20... ...Summary As a Lead System Architect at Axiado, you will define the... ...platforms, and low-level system software/firmware. Leveraging your deep...Principal
$155.8k - $224.2k
...digitizing, energy-intensive world. From revolutionizing power for AI-driven data centers to ensuring resilience for hospitals,... ...challenges of the 21st century. We are looking for a Principal IT Systems Architect to join our team in one of today’s most exciting technologies...PrincipalFull timeWork at officeWorldwide$202k - $343k
...and advance faster than ever.SummaryThe Principal System Architect is a senior technical individual... ...reliability domains. Have experience using AI.End-to-end technical ownership: Lead... ...mechanical, thermal, reliability, firmware, software, ASIC, media, validation, operations,...PrincipalFull timeLocal areaImmediate start$233.9k - $330.4k
...design and heterogeneous system integration. Our... ...Impact We are seeking a Principal Hardware Engineer to serve... ...practices. Work with hardware, software and test engineering to... ...across customer architects, software teams, and Cisco... ...organizations in the AI era - and beyond. We’ve...PrincipalFull timeTemporary workWork at officeLocal areaFlexible hours- ...generation computing experiences—from AI and data centers, to PCs, gaming and embedded systems. Grounded in a culture of... .... THE ROLEWe are seeking a Principal GenAI Inference Optimization Engineer to join... ...models, working across the software-hardware stack.THE PERSONThe...Principal
$184k - $287.5k
We are looking for an architect that wants to change how the AI inference industry works. With agent adoption taking off and GPUs in shortage, NVIDIA has a... ...datacenter developers how to scale their businesses, systems and infrastructure. This role covers a wide range of...Full time$184k - $287.5k
...ambitious and forward-thinking solution architect to help in the enablement of Network Industry Software Vendors (ISVs). These ISVs are... ...a network stack for distributed inference which will be used to... ...impact on the world by applying AI inference aware technology to existing...Full timeWork experience placement- ...the next generation of AI infrastructure: a large-scale inference platform powered by... ...datacenter scale. As a Principal Software Engineer you will define... ...into a unified inference system. You will drive the... ...Distributed Inference Runtime Architect runtime systems...PrincipalFull timeLocal area
$184k - $287.5k
...dedicated Senior Networking Software Architect to join our team in Santa Clara... ...to develop the future of AI and data-center infrastructure... ...addressing AI training/inference and cloud-native workloads.Lead... ...experience.8+ years of experience in systems/platform architecture, with...Full time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal AI Inference Systems Software Architect. Be the first to apply!
- technical architect San Jose, CA
- system architect San Jose, CA
- pega system architect San Jose, CA
- embedded systems architect San Jose, CA
- application architect San Jose, CA
- .net software architects (remote) San Jose, CA
- software architect San Jose, CA
- principal San Jose, CA
- senior principal scientist San Jose, CA
- principal architect San Jose, CA



