Technical Director, Large-Scale AI Model Inferencing
$219k - $351kSamsung Semiconductor
Job Description
Job Description: Please Note: To provide the best candidate experience amidst our high application volumes, each candidate is limited to 10 applications across all open jobs within a 6-month period. Advancing the World's Technology Together Our technology solutions power the tools you use every day--including smartphones, electric vehicles, hyperscale data centers, IoT devices, and so much more. Here, you'll have an opportunity to be part of a global leader whose innovative designs are pushing the boundaries of what's possible and powering the future. We believe innovation and growth are driven by an inclusive culture and a diverse workforce. We're dedicated to empowering people to be their true selves. Together, we're building a better tomorrow for our employees, customers, partners, and communities. What You'll DoInference is becoming a memory-bandwidth business. As models scale past what any single GPU can hold — KV caches grow with context, MoE expert weights spill beyond HBM, and new architectures change the rules of what "model state" even means — the winners will be the companies that treat memory as the core product of AI inference, not an afterthought.We are looking for a Hands-on Principal Engineer who combines deep, first-principles knowledge of AI model architectures (dense Transformers, Mixture-of-Experts, State Space Models, and hybrids) with production-scale inference expertise, to own the requirement for full-stack AI memory solutions at scale — spanning GPU HBM, host DRAM, CXL-attached memory pools, and NVMe/SSD tiers and Samsung Cognos, AI memory software that moves model state intelligently across them.This person will be the technical authority who connects model behavior to memory-system design: someone who can explain why an MoE router's activation pattern dictates an LRU expert cache policy, why a Mamba state cache breaks the assumptions of PagedAttention, and why disaggregated prefill/decode changes the required memory bandwidth per token by an order of magnitude — and then build the products that exploit those facts.Level: Principal Engineer Team: Memory Solutions Lab / Data Fabric SolutionsReports to: Chief Technologist, Memory Solutions LabLocation: Daily onsite presence preferred at our San Jose office/headquarters in alignment with our Flexible Work policy; remote/hybrid option available. Job ID: 43027Model Architecture Expertise — The Foundation Serve as expert on how different model families consume and move memory, and translate that into memory-product requirements: Dense Transformers: MHA/MQA/GQA/MLA attention, KV-cache growth characteristics, long-context behaviors, attention sinks and prefix locality. Mixture-of-Experts: routed vs. shared experts, expert-parallel execution, routing skew and hot-expert locality, expert-weight offloading and cache-admission policies, per-token weight-read economics. State Space Models (Mamba/Mamba-2) and hybrid SSM-attention architectures: recurrent state vs. KV cache semantics, state size per sequence and per layer, cache-swapping behavior for context switching and batching, and what "cache-aware scheduling" means when the state is a fixed-size tensor instead of a token-indexed table. Emerging architectures: linear attention, sliding-window/hybrid layers, diffusion and multimodal transformers — and how each changes the memory hierarchy math. Model the memory footprint, bandwidth demand, and access patterns of frontier open-weight models (e.g., Llama/Qwen-class dense, DeepSeek/Kimi-class MoE, Jamba-class hybrids) and publish internal reference architectures for each. Track the model landscape as a roadmap input: anticipate what coming architectures (longer contexts, agentic multi-session reuse, reasoning-loop workloads, speculative decoding drafts) will demand from memory systems 12–24 months out.Large-Scale Inference Expertise Own deep expertise in production inference stacks — SGLang (HiCache), vLLM (PagedAttention, LMCache integration), NVIDIA Dynamo, TensorRT-LLM, llama.cpp-class engines — including their memory-management internals, not just their flags. Drive inference performance engineering: continuous batching, chunked prefill, disaggregated prefill/decode, prefix and radix caching, speculative decoding, CUDA Graphs, and their interactions with memory tiering. Own the latency/throughput/cost envelope: TTFT and TBT/TPOT SLOs, tokens-per-second per dollar, GPU memory utilization as the binding constraint, and the tradeoff curves between cache hit rate, memory capacity, and bandwidth. Define benchmarking and characterization methodology: realistic agentic and long-context workloads (multi-turn reuse, session persistence, RAG prefixes), KV-cache reuse-rate measurement, and bandwidth-latency profiling across the full hierarchy (Nsight, PyTorch Profiler, vendor memory tools).Full-Stack AI Memory Solutions — The Core Mandate Define engineering requirements, with proof, for tiered memory systems for inference at fleet scale: HBM as L1, host DRAM (pinned, NUMA-aware pools) as L2, CXL-attached memory pools as an elastic tier, and NVMe/SSD as capacity tier — with the policies (admission, eviction, prefetch, placement) that make the hierarchy behave like one memory. Design expert-weight offloading solutions for MoE serving: host-resident expert pools, GPU-resident expert caches with bandwidth-adaptive fill/evict policies, and CPU/CXL-execution hybrid paths — informed by the routing statistics of real models. Translate model knowledge into product: write the requirements, reference architectures, and performance models that guide memory hardware and firmware roadmaps (HBM capacity/bandwidth, CXL device behavior, SSD QoS for cache tiers), and validate with end-to-end prototypes on real inference workloads. Develop and Deliver POCs: demos and published benchmarks showing inference TCO improvement from the memory stack — e.g., context capacity multiplied at constant GPU count, or cost-per-token reduced through cache-hit-rate gains — credible to both CTOs and PhD researchers.Technical Leadership Set multi-year technical strategy for AI memory solutions; own build-vs-adopt-vs-contribute decisions across the open-source inference and caching ecosystem (vLLM, SGLang, LMCache, Cognos-style KV stores) and drive upstream contributions where strategic. Lead architecture reviews and deep-dive design sessions; write the documents that become the company's standard for how we talk about memory for AI. Represent the company with customers and partners at the deepest technical level: serve as the expert voice in CTO-to-CTO conversations, design wins, and standards discussions. Mentor senior engineers and grow a bench of architecture talent across the model-to-memory boundary. What You Bring BS in Computer/Electrical/Electronic Engineering or Computer Science, and 10+ years of relevant experience MS in Computer/Electrical/Electronic Engineering or Computer Science with 8 years of relevant experience preferred. 12+ years in systems engineering, with 4+ years hands-on in large-scale LLM inference or GPU systems performance — you have personally profiled, diagnosed, and fixed memory bottlenecks in production serving, not just read about them. First-principles understanding of transformer-class model internals: you can derive KV-cache size formulas from attention math, explain MQA/GQA/MLA tradeoffs, and reason about activation-memory peaks during prefill. Working expertise with MoE model behavior: routing, expert parallelism, load skew, and the weight-memory economics of serving models larger than GPU capacity. Direct experience with at least one major inference stack's memory-management internals (vLLM PagedAttention/block manager, SGLang HiCache/token pools, TensorRT-LLM KV manager, or llama.cpp compute buffers) — code-level, not configuration-level. Strong performance-engineering skills: bandwidth-bound vs. compute-bound analysis, NUMA and PCIe topology reasoning, RDMA basics, and fluency with GPU/CPU profilers. Track record of building systems software at the memory/storage/IO layer — caches, tiering, paging, or storage engines — with production deployments. Ability to write models and simulators, not just measure: analytical queueing, cache-hit-rate, and bandwidth models that predict system behavior before hardware exists. Excellent written and verbal communication, including executive-level technical narrative; comfort being the technical face of the company in front of customers.Preferred Experience with State Space Model or hybrid SSM-attention serving (Mamba-class state management, cache swapping for recurrent models) — rare and highly valued. Contributions to open-source inference/caching projects (vLLM, SGLang, LMCache, HiCache, Mooncake, KTransformers, llama.cpp). Experience with CXL memory pooling, CXL-attached tiering, or near-memory processing in real deployments or serious prototypes. Experience with SSD/NVMe as a KV or expert cache tier, including QoS engineering for inference-grade latency. Background in memory/storage product companies bringing hardware-software co-designed solutions to market. You're inclusive, adapting your style to the situation and diverse global norms of our people. An avid learner, you approach challenges with curiosity and resilience, seeking data to help build understanding. You're collaborative, building relationships, humbly offering support and openly welcoming approaches. Innovative and creative, you proactively explore new ideas and adapt quickly to change.#LI-SF1 What We OfferThe pay range below is for all roles at this level across all US locations and functions. Pay within this range varies by work location and may also depend on job-related knowledge, skills, and experience. We also offer incentive opportunities that reward employees based on individual and company performance. This is in addition to our diverse package of benefits centered around the wellbeing of our employees and their loved ones. In addition to the usual Medical/Dental/Vision/401k, our inclusive rewards plan empowers our people to care for their whole selves. An investment in your future is an investment in ours. Give Back With a charitable giving match and frequent opportunities to get involved, we take an active role in supporting the community.Enjoy Time Away You'll start with 4+ weeks of paid time off a year, plus holidays and sick leave, to rest and recharge.Care for Family Whatever family means to you, we want to support you along the way—including a stipend for fertility care or adoption, caregiver support, and medical travel support.Prioritize Emotional Wellness With on-demand apps and free confidential therapy sessions, you'll have support no matter where you are.Stay Fit Eating well and being active are important parts of a healthy life. Our onsite Café and gym, plus virtual classes, make it easier.Embrace Flexibility Benefits are best when you have the space to use them. That's why we facilitate a flexible environment so you can find the right balance for you. Base Pay Range $219,000—$351,000 USD Equal Opportunity Employment Policy Samsung Semiconductor takes pride in being an equal opportunity workplace dedicated to fostering an environment where all individuals feel valued and empowered to excel, regardless of race, religion, color, age, disability, sex, gender identity, sexual orientation, ancestry, genetic information, marital status, national origin, political affiliation, or veteran status. When selecting team members, we prioritize talent and qualities such as humility, kindness, and dedication. We extend comprehensive accommodations throughout our recruiting processes for candidates with disabilities, long-term conditions, neurodivergent individuals, or those requiring pregnancy-related support. All candidates scheduled for an interview will receive guidance on requesting accommodations. Our Commitment to Innovation and Fairness At Samsung Semiconductor, we use Artificial Intelligence (AI) tools in the recruitment process to enhance efficiency. However, AI is used as a support tool, not a final decision-maker. All hiring decisions are made by our human recruiting team and hiring managers to ensure every candidate is evaluated fairly and holistically. Recruiting Agency Policy We do not accept unsolicited resumes. Only authorized recruitment agencies that have a current and valid agreement with Samsung Semiconductor, Inc. are permitted to submit resumes for any job openings. Applicant AI Use Policy At Samsung Semiconductor, we support innovation and technology. However, to ensure a fair and authentic assessment, we ask that candidates rely on their own knowledge and skills throughout the process. AI tools may be used for basic preparation, grammar, and research, but should not be used to generate or assist with submitted content or live interview responses. If we determine that AI is being used outside these guidelines, we reserve the right to pause or end the interview, and your candidacy may be disqualified. Trade Secret Notice By submitting an application, you agree not to disclose to Samsung—or encourage Samsung to use—any confidential or proprietary information (including trade secrets) belonging to a current or former employer or other entity. Applicant Privacy Policy
$162k - $387.6k
...recommendation system serving a billion-scale user base across short-video,... ...and GMV. Scale recommendation models like LLMs. Push ranking models... ...Inject world knowledge. Use large models' real-world knowledge to... ...degree in Computer Science, AI, Mathematics, Statistics or a...SuggestedFull timeTemporary workLive inLocal areaWorldwide$168k - $258.75k
...looking for a highly-motivated Technical Program Manager (TPM) to join... ...the next generation of NVIDIA AI supercomputing systems. This TPM... ...of the latest AI systems at scale, from datacenter design and requirements... ...to build and deploy large scale GPU computing systems based...SuggestedFull timeRemote work- ...engineer who can help us turn powerful AI and ML models into fast, reliable production systems.... ...optimizing inference latency and throughput, scaling serving systems, and making sure our... ...environments. Have experience with large-scale model serving, multi-GPU...SuggestedFull timeRelocation package
$184k - $287.5k
...NVIDIA is redefining what is possible with AI, and the Relational Foundation Model team is helping lead that transformation. We are building a unified... ...learning lifecycle, from architecture exploration and large-scale training to post-training optimization and high-...SuggestedFull time$152k - $241.5k
We are building a team of innovators to help partners develop and adopt the next generation of Physical AI, spanning data generation, large-scale multimodal model training, robotics simulation and deployment!We are looking for a hands-on Applied Engineer with deep expertise...SuggestedFull time$174.72k - $295.68k
...forefront of innovation, integrating advanced AI and autonomous driving technologies into... .../ Research Scientist to drive the modeling and algorithmic development of XPENG’s next... ...infrastructure experts to design, train, and deploy large-scale multi-modal models that unify vision,...Full time- ...Systems builds the world's largest AI chip, 56 times larger than... ...works with the leading model labs, global enterprises, and... ...Cerebras, to deploy 750 megawatts of scale, transforming key workloads with... ....You will define the technical vision, organizational strategy...
$70 per hour
...think deeply and solve complex technical challenges in areas like... ...extraction pipeline that parses large-scale simulation logs, vehicle trajectory... ...) Develop temporal graph modeling techniques to capture time-... ...published papers in top-tier AI/ML, data mining, or computer...Hourly payFull timeInternshipSummer internshipRelocation package$193.3k - $261.5k
...running a wide range of models and supporting novel... ...of what's possible in AI acceleration.As part of... ...in spaces that are very large, yet our teams remain small... ...peak performance at scale for customers and developers... ...decisions with your technical input. You will work in...Work experience placementInternshipLocal areaFlexible hours$224k - $356.5k
...who specializes in building cutting-edge infrastructure for large-scale foundation model training in the Generalist Embodied Agent Research (GEAR)... ...multimodal foundation models, large-scale robot learning, embodied AI, and physics simulation. Our past projects includeEureka,...Full time$193.3k - $261.5k
...Trainium, Amazon's custom cloud-scale machine learning accelerators.... ...us to optimize the latest models to run really fast on the Trainium... ...inference libraries* Drive technical excellence in performance optimization... ...with Machine Learning and Large Language Model fundamentals,...InternshipLocal areaFlexible hours- ...forefront of innovation, integrating advanced AI and autonomous driving technologies into... ...with strong expertise in generative modeling and a demonstrated record of original research... ...the dynamics of the physical world from large-scale multimodal data — predicting how a scene...InternshipWork at office
$192k - $278k
...prioritized portfolio of cross-product AI initiatives, leading executive... ...feedback that inform future model development.Accelerate how... ...:Bachelor's degree in a technical field, or equivalent practical... ...priority.To turn Google’s product scale into a lasting advantage,...$150k
...About the Institute of Foundation Models We are a dedicated research... ...nurture the next generation of AI builders, and drive... ...ensuring that experimental work can scale reliably when needed. Key... ...manage distributed systems for large-scale data processing (filtering...Full timeVisa sponsorship$184.7k - $324.8k
...8-11SummaryDo you get excited by building AI applications to enhance the evaluation of... ...AI Scientist to develop cutting-edge AI/ML models for the automatic grading and quality assessment... ...evaluate various AI product quality at scale.You will apply deep expertise in prompt...Relocation$311.1k
...intelligence by developing sophisticatedfoundation models that power intelligent features across... ...a Distinguished Engineer to set the technical direction for thesystems that power our... ...running rollouts fortraining, powering large-scale model evaluation, and serving as an LLM...Relocation$141k - $221k
...directing design engineers and other technical engineers working on specific... ...are responsible for managing large projects with complex scope,... ...of accelerated compute and AI technologies . This understanding... ...– from drawing board to full-scale production and after-market...Temporary workWork at officeLocal areaRemote workRelocation$97.68k - $173k
...Description U.S. Bank is seeking the position of Quantitative Model Analyst in Cupertino California. Essential... ...SQL and SAS; and use of reporting tools Tableau and Power BI in large-scale data environments. Position may allow working from home within...Full timeTemporary workBank staffLocal areaWork from home- ...think deeply and solve complex technical challenges in areas like... ...extraction pipeline that parses large-scale simulation logs, vehicle trajectory... ...) Develop temporal graph modeling techniques to capture time-... ...published papers in top-tier AI/ML, data mining, or computer...InternshipSummer internshipLocal areaRemote work
- ...Overview TENEX is an AI-native, automation-first, built-for-scale Managed Detection and Response... ...upside. As the Senior Technical Program Manager (TPM) at... ...Kansas City, KS. Hybrid work model. Job Responsibilities... ...record of delivering large-scale AI or ML initiatives...
$120k - $150k
...Required Skills Strong Technical Program/Program Management experience. Cloud platforms... ...Drive end-to-end delivery of large-scale technical programs. Partner with Engineering... ...system-integration initiatives. Lead AI/automation programs focused on improving...$152k - $241.5k
.... NVIDIA is building the computing platforms, software, and AI systems that enable autonomous vehicles to perceive, reason,... ...integration, evaluation, and deployment of end-to-end driving models across large-scale training infrastructure, simulation environments, and...Full time$200k - $322k
...unlimited potential of AI to define the next era... ...PMO organization seeks a Technical Program Manager to... ...and release of massive-scale datasets. In this position... ...that enables NVIDIA’s models to lead in enterprise and... ...successfully delivering medium to large organizational-wide...Full time$148.2k - $239.8k
...Integrity, and Inclusion. We weave AI into the fabric of everything... ...when it’s needed. This model supports real-time problem-solving... ...Networks is seeking a Senior Technical Program Manager II to lead... ...experience leading large-scale software, cloud security, or...Full timeWork at officeVisa sponsorshipWork visaShift work$136k - $218.5k
...their surroundings. NVIDIA is seeking a Technical Program Manager on the Solutions Architecture... ...build, deploy, and support successful AI Factories. Academic and commercial... ...validation through deployment and support-in large-scale AI infrastructure. Technical depth to...Worldwide- Aquent is seeking an experienced Technical Program Manager to support a global technology... ...software development processes, including AI-enabled transformation. What You'll Do... ...complex software projects supporting a large-scale corporate real estate portfolio. Manage...2 days per week
- ...and Inclusion. We weave AI into the fabric of... ...when it’s needed. This model supports real‑time problem... ...businesses through high impact technical partnerships. This is... ...primary architect for scaling our Cloud Security (... ...results within a large‑scale organization. Preferred...Full timeWork at officeShift work
$160k - $180k
...opportunity to offer a large number of new positions... ...seeking an experienced Sr. Technical Program Manager to join... ...will own the operating model, governance, and... ...programs run on time and at scale. This is a high-... ...data center, storage, or AI infrastructure. Deep program...Worldwide$195.2k - $293.8k
Manager, Technical SEO Programs - Cupertino Cupertino, California... ...Apple's presence across AI platforms, AI-powered... ...emerging standards such as MCP (Model Context Protocol), UCP (... ...others Experience building or scaling an SEO function within a large, matrixed organization...WorldwideRelocation$60 per hour
...seeking Biology Experts and Life Science Professionals to evaluate AI-generated science and ensure compliance with scientific... ...Responsibilities include reviewing biological inquiries, validating technical claims from public databases, and critiquing experimental designs...Hourly payRemote workWork from homeFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Technical Director, Large-Scale AI Model Inferencing. Be the first to apply!
- technical director San Jose, CA
- director of technical services San Jose, CA
- technical product manager San Jose, CA
- technical coordinator San Jose, CA
- technical account manager San Jose, CA
- technical writing manager San Jose, CA
- technical services manager San Jose, CA
- senior technical director San Jose, CA
- technical integration manager San Jose, CA
- senior technical manager San Jose, CA







