Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Staff Engineer - Infrastructure Systems

Full-time

Runware

As Senior Staff Engineer, Infrastructure Systems, you will own the GPU software stack on every Runware node: the firmware, the kernel and NVIDIA driver, the CUDA runtime and libraries, the container runtime, and the way Kubernetes and our inference services see and share a GPU. When a GPU misbehaves anywhere in the fleet, you will find out why and write the software that keeps it from happening again.

This is a senior staff-level individual contributor role for someone who knows GPUs from the driver up and still writes code every day. One month that means qualifying a new driver and CUDA release against every model family we serve. The next it is the GPU part of the acceptance suite that every node must pass, a detector that catches a failing GPU from its Xid and ECC history before a customer does, the bring-up of a new GPU generation, or a CUDA error that turns out to start in firmware, a PCIe link or a kernel update. You will work directly with the node operations, Managed Kubernetes, inference performance and Sonic Inference Engine teams, and be the person each of them calls when the problem sits below the framework.

Success means that every GPU in every pod runs a known, tested combination of firmware, driver and CUDA, that upgrades roll across the fleet without failed inferences, that GPU faults are detected, classified and remediated automatically, and that a new GPU generation goes from delivery to serving traffic in days.

What you'll do

  • Own the GPU software baseline. Versioned, reproducible combinations of kernel, NVIDIA driver, firmware, CUDA runtime and libraries (cuDNN, cuBLAS, NCCL, TensorRT) and container toolkit, tested before they roll and rolled across the fleet without downtime.
  • Qualify every driver and CUDA release. Keep the compatibility matrix across GPU generations, frameworks and model families, and catch regressions in correctness, performance and memory use before they reach production.
  • Take GPU failures to root cause: Xid errors, CUDA "unknown error", cuBLAS handle failures, out-of-memory and fragmentation, GPUs that fall off the bus. Separate hardware, firmware, driver, runtime and application causes, and prove which one it is.
  • Build GPU acceptance and diagnostics: the burn-in, DCGM diagnostics, memory and interconnect tests and real-workload tests that gate every node and rack before it serves traffic, and the reports customers receive at handover.
  • Make GPU health visible and self-healing. Pull telemetry from DCGM, the driver, the kernel log and the BMC into one trusted view, classify faults, and automate the drain, reset and replacement paths.
  • Own how GPUs are exposed to workloads: the container runtime, device plugins and the GPU Operator, MIG and sharing policies, and isolation between tenants, on bare metal and on Kubernetes.
  • Bring up new GPU generations and new pods with the hardware and node operations teams: first driver and firmware, NVLink and NVSwitch validation, and the automation that makes the next bring-up take days.
  • Characterize hardware with real workloads. Benchmark image, video, audio and LLM inference on each GPU generation and software stack, and feed the results into placement, pricing and the next purchase.
  • Work with NVIDIA and upstream projects. File driver and firmware bugs with reproducible cases and see them through, track what each release changes, and contribute fixes to the open-source tools we depend on.
  • Use AI to accelerate the work: agents for log and Xid triage, test generation, release qualification and runbook automation, so that the team's output grows faster than its headcount.

Requirements

What we're looking for

  • 12+ years building and operating systems software, with several of those on GPU software running in production at scale.
  • Deep, hands-on knowledge of the NVIDIA software stack: the kernel driver and its modules, the CUDA runtime and libraries, NVML, DCGM and the Container Toolkit, and how their versions interact.
  • A record of taking GPU failures to root cause across hardware, firmware, driver, runtime and application: Xid errors, ECC and row remapping, PCIe and NVLink errors, thermal and memory faults.
  • Strong Linux internals: the kernel, device drivers and modules, PCIe, NUMA and IOMMU, cgroups and containers, and the tools (perf, eBPF, ftrace) that show what a machine is really doing.
  • You write code every day. Fluent in Python and in at least one systems language (C, C++, Rust or Go), and comfortable enough in CUDA to write a test or reproduce a bug yourself.
  • Experience running GPUs under Kubernetes and on bare metal: device plugins, the GPU Operator, MIG, image builds and provisioning at scale.
  • Working knowledge of how inference frameworks use a GPU (PyTorch, TensorRT, ONNX Runtime): memory allocation, compilation and caches, and multi-GPU execution.
  • Clear technical writing. Your design documents, postmortems and ticket updates can be acted on by someone who was not in the room.
  • You use AI to accelerate your own output every day, and you have a clear view of how a small team armed with these tools should operate differently.
  • Comfortable owning outcomes in a fast-moving, ambiguous environment, picking the problem that matters most and being accountable for the result.

Nice to have

  • Time spent building GPU drivers, CUDA libraries or GPU system software at a GPU vendor, a cloud provider or an HPC centre.
  • Experience with Blackwell-generation hardware (B200, GB200/GB300 NVL72), NVLink and NVSwitch topologies, and liquid-cooled racks.
  • Multi-GPU and multi-node execution: NCCL, GPUDirect, and RDMA over RoCE or InfiniBand.
  • Firmware qualification across BIOS, BMC, NIC and GPU, and acceptance testing with OEMs and ODMs.
  • Inference serving internals: PyTorch, TensorRT-LLM, vLLM, SGLang or custom diffusion and video pipelines, and how they load, schedule and fail.
  • Experience with non-NVIDIA accelerators (AMD or custom silicon) and their software stacks.
  • An open-source project you wrote or maintain that people run in production.
  • Multi-tenant isolation and confidential computing on shared GPU hardware.
  • Experience running a 24x7 service with published service levels.

Benefits

Why Runware

You will design the network for an AI supercomputer that is being built right now, with the budget, the hardware, and the mandate to do it properly. Few network architects get to own a fabric from the first pod to the global backbone, with a direct line to the people deciding what the next pod looks like.

We're a remote-first collective, meeting in person twice a year to plan, brainstorm, celebrate wins, and enjoy some face-to-face time. We have core hours for cooperative working and calls, but outside of that your calendar is yours. Work the hours that let you perform at your peak while also building a healthy life.

Our release cycles are fast and intense, but they're followed by real downtime. After big pushes we expect the team to unplug, recharge, and come back ready and stronger than ever for the next leap.

  • Generous paid time off: vacation, sick days, public holidays
  • Meaningful stock options: share in the upside you create
  • Flexible hours: own your schedule outside core collaboration blocks
  • Family leave: paid maternity, paternity, and caregiver time
  • Company retreats: twice-yearly gatherings in inspiring locations
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Senior Staff Engineer - Infrastructure Systems in California vacancy
  • $183k - $195k

     ...Job Description Job Description Senior Staff Engineer, Fluidic Systems Architecture, and Integration Location: San Diego, CA Job Type: Full-Time Salary: $183,000 - $195,000   Company We are a rapidly growing, stealth biotech founded by leading industry... 
    Senior
    Full time
    Work at office

    San Diego Stealth Startup

    San Diego, CA
    18 days ago
  •  ...who can be calm while working under pressure, and has strong operational experience to systemically and expediently drive issues/escalations resolutions. The Network Engineer engages with customers to understand their topology and provides suggestions, answer... 
    Senior
    Work at office
    Remote work

    Celestica International LP

    San Jose, CA
    4 days ago
  • $160.36k - $291.15k

     ...T. Rowe Price, and other leading investors.About the WorkLead system-level requirements definition, architecture design, integration...  ...or on road testing. Hands on experience in systems engineering.Ability to read code in C++ and write code in Python / MATLAB.... 
    Senior
    Odd job
    Immediate start
    Flexible hours

    Nuro

    Mountain View, CA
    3 days ago
  •  ...Descriptions & Requirements Job Description and Requirements We Are Synopsys is the leader in engineering solutions from silicon to systems, enabling customers to rapidly innovate AI-powered products. We deliver industry-leading silicon design, IP, simulation... 
    Senior

    Synopsys, Inc.

    Sunnyvale, CA
    2 days ago
  •  ...groundbreaking technology for algorithmic search, machine learning, natural language processing, and artificial intelligence. Our universal search engine powers search features across a variety of Apple products, including Siri, Spotlight, Safari, Messages and Lookup.
    Senior

    Apple

    Santa Clara, CA
    12 days ago
  •  ...It all started when engineer Fred Luddy wrote code that automated a tedious task...  ...organization is seeking a versatile Senior Staff Security Systems Engineer to support multiple security...  ...vulnerability management, cloud and infrastructure security, security data platforms, and... 
    Senior
    Permanent employment
    Full time
    Contract work
    Live in
    Immediate start

    ServiceNow

    California
    9 days ago
  • $97k - $180k

     ...space, air, land, sea and cyber domains in the interest of national security. Job Title: Senior Specialist, Software Engineer - Application Infrastructure & Embedded Systems Job Number: 45417 Job Location: Yorba Linda, CA Job Schedule: 4/10: Employees work... 
    Senior
    Contract work
    For subcontractor
    Local area
    Flexible hours
    Shift work

    Harris Corporation

    Yorba Linda, CA
    4 days ago
  • $160k - $200k

     ...largest autonomous logistics system, delivering critical supplies...  ...aircraft, droids, docks, ground infrastructure, and operational sites. RF...  ...these issues requires an engineer who can move across disciplinary...  ...-cause methods. At the Staff level, lead cross-platform investigations... 
    Senior
    Permanent employment
    Local area
    Remote work

    Zipline

    South San Francisco, CA
    3 days ago
  • $189k - $301k

     ...employees, customers, partners, and communities.About the RoleWe are seeking a Senior Staff Engineer to build and optimize the EDA design environment and large-scale compute infrastructure that supports our semiconductor design organizations, and to establish company-wide... 
    Senior
    Contract work
    Work at office
    Flexible hours

    Samsung Semiconductor

    San Jose, CA
    5 days ago
  •  ...Unlimited PTO About This Role This is a backend- and systems-heavy founding engineering role where “full-stack” means owning the product and...  ..., with the ability to work across frontend, data, infrastructure, and product Depth in one or more of the following:... 
    Senior
    Full time
    H1b
    Work at office
    Visa sponsorship

    Cephas Consultancy Services Private Limited

    San Francisco, CA
    6 days ago
  • $148.5k - $223.9k

     ...efforts. Job Category Software Engineering Job Details About...  ...immediate opportunities for senior software developers who want...  ...products over public cloud infrastructure - AWS/Azure/GCP. Proven experience...  ...and developing distributed systems at scale. Proficiency in... 
    Senior
    Full time
    Immediate start

    Salesforce

    San Francisco, CA
    11 days ago
  • $222k - $308k

     ...remove vendor constraints.Shape multi-year hardware, silicon, and system roadmaps for high-scale, air- and liquid-cooled platforms...  ...qualifications:Bachelor's degree in Computer Science, Electrical Engineering, Computer Engineering, a related technical field, or... 
    Senior

    Google

    Sunnyvale, CA
    1 day ago
  •  ...members and civilians with intelligent systems. Its products include Hivemind autonomy...  ...driving innovation through robust software engineering and practical integration work across...  ...with networking concepts and messaging infrastructure.Hands-on experience with CMake, Conan,... 
    Senior
    Full time
    Temporary work
    Part time
    Work experience placement
    Worldwide

    Shield AI

    San Diego, CA
    3 hours ago
  • Cerebras Systems builds the world's largest AI chip, 56 times larger than GPUs....  ...speed inference.Cerebras is seeking a Senior Staff Security Engineer, Distributed Systems to help build...  ...identity in k8s podsPartner with platform, infrastructure, networking, and hardware teams to... 
    Senior

    Cerebras Systems

    Sunnyvale, CA
    12 hours ago
  • $130k - $190k

     ...Senior/Staff Application Engineer Seyond is a global leader in high fidelity LiDAR solutions, powering the future of mobility across ADAS, ITS,...  ...experienced Senior/Staff Application Engineer to join our System & Application Engineering team and lead the integration,... 
    Senior
    Local area
    Visa sponsorship

    Seyond

    Sunnyvale, CA
    1 day ago
  • $150k - $170k

     ...life sciences, we pioneer cutting-edge solutions that decipher the complexities of biological data.The Senior Linux Systems Administrator / Storage Infrastructure Engineer is responsible for managing and supporting enterprise Linux environments, Dell EMC storage systems,... 
    Senior
    Full time
    Local area

    Qiagen

    Redwood City, CA
    4 days ago
  • $133.4k - $232k

     ...commercial service. Overview Joby Aviation is seeking a passionate, motivated and hands-on Senior or Staff Test Engineer on the Energy Storage and Distribution System (ESDS) Test Team to lead battery system integration testing, verifying that the battery packs,... 
    Senior
    Full time
    Temporary work

    Joby Aviation

    San Carlos, CA
    3 days ago
  •  ...members and civilians with intelligent systems. Its products include Hivemind autonomy...  ...autonomy solutions. You will join the Systems Engineering, Integration, & Test (SEIT) team in the...  ...combines developer tools, APIs, data infrastructure, simulation, AI/ML workflows, and GenAI... 
    Senior
    Temporary work
    Part time
    Worldwide

    Shield AI

    San Diego, CA
    3 hours ago
  • $185k - $255k

     ...world’s largest autonomous logistics system, delivering critical supplies...  ...You and The RoleZipline is seeking a senior-level electrical engineer to lead the design and development...  ...support our drone charging and docking infrastructure.The Ground Systems team at Zipline... 
    Senior
    Work at office
    Local area

    Zipline

    South San Francisco, CA
    1 day ago
  • $250k - $300k

     ...vertically integrated AI infrastructure company built from the...  ...our customers. As Sr. Staff Optical Network...  ...reporting into Network Engineering, with the mandate to set...  ...segments, including line system design, ROADM topology...  ...as an architect or senior technical lead designing... 
    Senior
    Full time
    Temporary work
    Flexible hours

    Crusoe

    Sunnyvale, CA
    12 days ago
  • $170.4k - $255.6k

     ...Senior Staff Network Engineer PacBio (NASDAQ: PACB) is a premier life science technology company that designs, develops, and manufactures advanced...  ...technology, and manufacturing networks; support validated-system change controls, audits, and ISO/SOX evidence.... 
    Senior

    Pacific Biosciences

    Menlo Park, CA
    1 day ago
  • $175k - $265k

     ...global scale. From chips and software to rack-scale systems, our team is tackling some of AI infrastructure’s hardest challenges — and shipping breakthrough...  ...SRE team owns the infrastructure layer that every engineering team and customer depends on — colocation facilities... 
    Senior

    d-Matrix

    Santa Clara, CA
    1 day ago
  • $188k - $356k

    Senior Staff/Principal Engineer, Communication System DesignCategoryAlgorithm & ArchitectureLocationIrvine, CA ExperienceMore than 6 Years Work Expe.EducationDoctorateJob Description- Develop advanced algorithms for 6G modems- Implement high-level modem algorithms in C/... 
    Senior
    Temporary work
    Worldwide

    MediaTek

    Irvine, CA
    3 days ago
  • $291k - $409k

     ...Autonomy Software /Full-time /HybridThe Collision Avoidance System (CAS) is Zoox's independent safety layer. It runs...  ...defense in the most challenging dense urban environments.As a Staff or Senior Staff engineer, you will be the technical lead for CAS algorithms, with a... 
    Senior
    Full time
    Temporary work
    Relocation package

    Zoox

    Foster, CA
    12 hours ago
  • $184k - $356.5k

     ...We are seeking a Linux Systems Engineer to join our Infrastructure Engineering team, dedicated to supporting high-performance EDA compute farms. You will be responsible for designing, deploying, and maintaining enterprise-grade Linux environments across RPM-based distributions... 
    Senior
    Full time

    NVIDIA

    California
    5 days ago
  • $134.6k - $217.8k

     ...matters. Job Summary Job Summary We're looking for an Infrastructure Engineer to build developer tooling that enables developer velocity...  ...platform is reproducible and version-controlled. Drive Systemic Quality: Identify and eliminate systemic bottlenecks in the... 
    Senior
    Local area
    Remote work

    PaloAlto Networks

    San Francisco, CA
    12 hours ago
  • $144k - $236k

     ...part of our world-class software engineering team, you will help build the next-generation infrastructure and platforms that power...  ...data infrastructure, storage systems, streaming platforms, traffic...  ...timeFunction: EngineeringExperience level: Mid-Senior LevelIndustry: Internet
    Senior
    For contractors
    Work experience placement
    Work at office
    Flexible hours

    Linkedin

    Mountain View, CA
    2 days ago
  • $198k - $326k

     ...world-class software engineering team, you will take the...  ...building the next-generation infrastructure and platforms for...  ..., API design and systems design, and your passion...  ...our company.As a Sr. Staff Software Engineer, you...  ...EngineeringExperience level: Mid-Senior LevelIndustry:... 
    Senior
    For contractors
    Work at office
    Flexible hours

    Linkedin

    Sunnyvale, CA
    4 days ago
  • $174k - $299k

     ...tech, and hyper-connected world. Job Overview: As a Senior Staff Backend Engineer - Cloud Infrastructure, this role will focus on Cloud Platform Infrastructure...  ...best to enhance Cloud infrastructure reliability & system efficiencies, productivity, availability,... 
    Senior
    Temporary work
    Work experience placement
    Flexible hours

    Coupang

    Mountain View, CA
    2 days ago
  • $134.6k - $217.8k

     ...along with contributing to shared Cortex infrastructures and components, to bring value to...  .... Job Summary We are seeking a Senior Software Engineer to join our Cortex backend engineering...  ...challenges to continuously improve system architecture and development processes... 
    Senior
    Full time
    Work at office

    PaloAlto Networks

    California
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Staff Engineer - Infrastructure Systems. Be the first to apply!