Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

GPU Cluster Engineer, Networking

Full-time

Sciforium

Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million-dollar funding and direct sponsorship from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack powering frontier AI models and real-time applications.

Role Overview

We are looking for a Senior Network Engineer to own the complete networking stack of our GPU clusters — from the RDMA compute fabric to the data center perimeter to cross-site and cloud connectivity. You will design the network architecture for new large-scale clusters, lead its bring-up, and operate it in production. This role owns everything from the NIC port outward: backend InfiniBand/RoCE fabrics, frontend and storage networks, out-of-band management, firewalls and edge security, and the links that connect our clusters to each other and to the cloud. You will work alongside our Hardware Operations team (physical install) and Systems & Platform team (host software stack) to deliver a fabric that never becomes the bottleneck.

Key Responsibilities

  • Fabric Architecture & Cluster Network Design

    • Greenfield Design: Architect the full network for new GPU clusters — compute/backend fabric, storage network, in-band frontend, out-of-band management (BMC/IPMI), and perimeter — including topology (fat-tree/Clos, rail-optimized), oversubscription analysis, and growth planning.

    • RDMA Fabric Selection: Design and justify InfiniBand (NDR/XDR, Quantum-class) vs. lossless Ethernet/RoCE v2 (Spectrum-X, Tomahawk-class) fabrics for training and inference workloads, including scale-up domains (NVLink/NVSwitch, NVL72-class rack systems).

    • Hardware & Physical Plant: Specify switches, NICs (ConnectX/BlueField), optics (400/800G OSFP/QSFP-DD), and cabling plans; produce the port maps and cable schedules Hardware Operations executes.

    • Logical Design: Own IP addressing plans, VLAN/VRF segmentation, and BGP/EVPN-VXLAN underlay-overlay architecture.

  • RDMA & Performance Engineering

    • Lossless Ethernet: Configure and validate RoCE v2 at scale — PFC, ECN/DCQCN congestion control, QoS/DSCP classification, buffer tuning — and prove no-drop behavior under full training load.

    • InfiniBand Operations: Run subnet managers (UFM/OpenSM), adaptive routing, SHARP in-network aggregation, partition keys, and virtual-lane QoS.

    • Fabric Validation: Benchmark and sign off new fabrics with perftest/ib_write_bw and nccl-tests/rccl-tests; verify GPUDirect RDMA paths and topology-aware placement with the Systems & Platform team.

    • Performance Debugging: Root-cause fabric-level issues — congestion trees, ECMP polarization, link flaps, degraded optics, straggler nodes caused by the network.

  • Production Network Operations & Security

    • Routing & Switching: Configure and maintain BGP, OSPF, ECMP, and EVPN-VXLAN across multi-vendor NOSes (Arista EOS, NVIDIA Cumulus, Cisco NX-OS, Junos, SONiC).

    • Perimeter & Security: Own firewalls, VPNs (WireGuard/IPsec), NAT, ACLs, and network segmentation between production, research, and management domains.

    • Network as Code: Generate configs from a source of truth (NetBox), deploy via Ansible/Nornir/NAPALM with Git-reviewed, CI-validated changes and zero-touch provisioning for new switches.

    • Telemetry: Build fabric observability — streaming telemetry/gNMI, sFlow, optics DOM, buffer/drop counters — into Prometheus/Grafana with actionable alerting.

    • Lifecycle & On-Call: Run switch/NIC firmware upgrade campaigns with minimal workload disruption, and serve as escalation point for network incidents.

  • Cross-Cluster & Cloud Connectivity

    • Inter-Site Links: Design and operate dark fiber and DWDM capacity between sites, including latency budgeting and carrier/IX peering.

    • Hybrid Cloud: Architect AWS Direct Connect / Azure ExpressRoute / GCP Interconnect links, VPC and Transit Gateway designs, and VPN failover paths.

    • Data Movement: Engineer WAN QoS and traffic paths for cross-cluster replication, dataset transfer, and checkpoint traffic.

Qualifications

  • Must-Haves:

    • 7+ years designing and operating production data center networks, including at least one large-scale HPC/AI cluster fabric (InfiniBand or RoCE v2) you designed or ran.

    • Expert-level routing and switching: BGP (including EVPN-VXLAN), OSPF, ECMP, VRF segmentation, across at least two major vendor platforms.

    • Deep RDMA expertise: lossless RoCE v2 tuning (PFC/ECN/DCQCN) and/or InfiniBand fabric management (subnet managers, adaptive routing, SHARP).

    • Hands-on experience with 100–800G optics, high-radix switching, and Clos/rail-optimized topologies.

    • Production network security experience: enterprise firewalls, site-to-site and remote-access VPNs, segmentation design.

    • Network automation proficiency: Python plus Ansible (or Nornir/NAPALM), with Git-based configuration workflows.

    • Working knowledge of the host-side RDMA stack (MOFED/DOCA, NIC tuning, GPUDirect) and how NCCL/RCCL collectives map onto physical fabric.

    • Cloud networking experience: VPC design and dedicated interconnects (Direct Connect/ExpressRoute class).

  • Nice-to-Haves:

    • Dark fiber / DWDM procurement and operations experience.

    • BlueField DPU or SmartNIC deployments.

    • Experience supporting distributed training at 1,000+ GPU scale, or multi-cluster/cross-site training.

    • Kubernetes networking for GPU serving (CNI, SR-IOV, Multus).

    • Expert certifications (CCIE/JNCIE) or NVIDIA networking certification (InfiniBand/Spectrum-X/UFM).

Benefits include

  • Medical, dental, and vision insurance

  • 401k plan

  • Daily lunch, snacks, and beverages

  • Flexible time off

  • Competitive salary and equity

Equal opportunity

Sciforium is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.

Vacancy posted 8 days ago
Similar jobs that could be interesting for youBased on the GPU Cluster Engineer, Networking in San Francisco, CA vacancy
  • $250k

     ...provider building a next-generation GPU platform designed for AI...  .../ Staff Site Reliability Engineer to support and scale large-scale...  ...frameworks for GPU compute clusters Collaborate with ML, data,...  ...providers Strong understanding of networking fundamentals (DNS, TCP/IP,... 
    Suggested
    Full time
    Remote work
    San Francisco, CA
    more than 2 months ago
  •  ...platform reliability and infrastructure engineering. You'll be instrumental in ensuring...  ...internal research and external customers' GPU cluster environments. Responsibilities include...  ...platforms (GCP preferred). Knowledge of high-performance networking and real-time systems.... 
    Suggested

    Macpower Digital Assets Edge Private Limited

    San Francisco, CA
    25 days ago
  •  ...from AMD with hands-on support from AMD engineers the team is scaling rapidly to build...  ...About the Role We are looking for a GPU Cluster Engineer to own the entire software stack...  ...~ Working knowledge of NCCL and RDMA networking (InfiniBand/RoCE, GPUDirect) and of PyTorch... 
    Suggested
    Full time
    Flexible hours

    Sciforium

    San Francisco, CA
    8 days ago
  • $170k - $250k

     ...This company is looking for an engineer to work directly with the CTO on complex GPU virtualisation challenges. The role...  ...complex issues across low-level networking, GPU drivers, and distributed...  ...checkpointing, and distributed GPU cluster management You will be... 
    Suggested
    Full time
    Visa sponsorship
    Flexible hours
    San Francisco, CA
    more than 2 months ago
  •  ...real time. Behind that sits a large GPU fleet spread across several cloud providers. Today, our inference engineers and researchers build models while also managing networking, securing capacity, and operating the underlying GPU clusters, responsibilities we want a... 
    Suggested
    Full time
    Shift work

    perplexity

    San Francisco, CA
    8 days ago
  •  ...matters to the world. The Production Engineering Team Examples of key exciting...  ...of 10s to 100s of GWs: at our scale, a GPU failure isn't a ticket. It's a throughput...  ...at construction speed: we're converting clusters across production sites simultaneously,... 
    Full time
    Local area

    fluidstack

    San Francisco, CA
    8 days ago
  • $250k - $300k

     ...demanding AI workloads. Our networking team designs and operates the...  ...of GPUs into unified compute clusters, and increasingly the optical...  ...position reporting into Network Engineering, with the mandate to set...  ...align optical capacity with GPU cluster deployments, the backbone... 
    Full time
    Temporary work
    Flexible hours

    Crusoe

    San Francisco, CA
    4 days ago
  • Senior Engineer A profitable, high growth technology company in the Bay Area is expanding the team that powers the compute behind its...  ...visible impact from day one. What you will do: Build and operate GPU clusters across public cloud, specialty GPU providers and owned data... 

    Career Movement

    San Francisco, CA
    2 days ago
  • Senior Hpc & Gpu Infrastructure Engineer Sciforium is an AI infrastructure company developing next-generation...  ..., and performance of our GPU compute cluster. You will be the primary PyTorch...  ...of the cluster. 2. Linux & Network Administration OS Management: Install... 
    Flexible hours

    Sciforium

    San Francisco, CA
    2 days ago
  •  ...About the Team The Core Network Engineering team owns the end-to-end networking stack that connects...  ...to model training efficiency, cluster reliability, or fleet expansion. They design...  ...InfiniBand, Ethernet, and high-performance GPU interconnects Define and... 
    Full time

    OpenAI

    San Francisco, CA
    12 days ago
  •  ...Systems Administrator / DevOps Engineer, you will maintain and build...  ...our large-scale training clusters and the deployments and services...  ...validation of new hardware (compute, networking, storage) Supporting and...  ...AI infrastructure, including GPU management and workload... 
    Full time
    Work at office
    Remote work
    Relocation package

    zyphra

    San Francisco, CA
    8 days ago
  •  ...'s largest AI infrastructure networks. The team owns day-to-day operations...  ...to deliver highly available GPU infrastructure for AI...  ...an Infrastructure Operations Engineer to operate and improve the large...  ...fabrics that support GPU clusters, storage systems, and management... 
    Permanent employment

    OpenAI

    San Francisco, CA
    2 days ago
  •  ...Overview As a Principal Infrastructure Engineer, AI Cluster Performance & Validation, you will be a...  ...criteria by which multi-thousand-GPU clusters are declared production-ready,...  ..., QoS and congestion control, SHARP in-network reduction, rail and topology-aware placement... 

    Nscale

    San Francisco, CA
    5 days ago
  • $246k - $271k

     ...and search products. We need engineers who can make that foundation — Kubernetes, storage, networking, deployments — fast, reliable,...  ...workloads on Kubernetes — search clusters, crawling fleets, queues, and...  ...infrastructure experience (GPU workloads, model deployment pipelines... 
    Full time
    Temporary work
    For contractors
    Remote work
    Visa sponsorship
    Flexible hours

    Firecrawl

    San Francisco, CA
    1 day ago
  •  ...superintelligence. One person, one GPU. If you'd like to build the...  ...'s high performance cloud network Work on deploying and...  ...hardware for new and existing clusters Ensure high availability of our...  ...on-call rotation for Network Engineering team You Have 10+ years of experience... 
    Work at office
    Local area
    Work from home
    Flexible hours

    Lambda

    San Francisco, CA
    2 days ago
  • $170k - $210k

     ...What We Are Looking For We are seeking an experienced Senior Network Engineer (InfiniBand / UFM) to design, deploy, automate, and operate...  ...AI Factory networking infrastructure supporting large-scale GPU clusters. This role is responsible for building and maintaining high-... 
    Work at office
    Remote work
    Work from home
    Flexible hours

    Lightning AI

    San Francisco, CA
    2 days ago
  • $180k - $225k

     ...impact. What We’re Looking For Lightning AI is seeking a Senior Network Engineer with hands-on Cumulus Linux expertise to build and scale the...  ...Engineer high-performance Ethernet fabrics supporting GPU clusters and AI workloads Build and maintain EVPN/VXLAN, BGP, and high... 
    Work at office
    Remote work
    Work from home
    Flexible hours

    Lightning AI

    San Francisco, CA
    2 days ago
  • $182k - $242k

     ...CoreWeave is seeking an experienced Senior Network Security Engineer for our network security initiatives...  ...breaches across high-density AI cluster networks. Proactively identify vulnerabilities...  ...Security: Secure high-density GPU clusters and ultra-low latency backend... 
    Full time
    Temporary work
    Casual work
    Work at office
    Flexible hours

    CoreWeave

    San Francisco, CA
    2 days ago
  • Principal Network Engineer Houston; New York; San Francisco; Seattle Nscale is the GPU cloud engineered for AI. We provide cost-effective, high-performance infrastructure...  ...the industry, supporting large-scale AI GPU clusters where network performance directly impacts customer... 
    Flexible hours

    Nscale

    San Francisco, CA
    2 days ago
  • $230k

     ...scale systems that span data centers, GPUs, networking, and more, ensuring high availability,...  ...growth.About the roleAs a software engineer on the Fleet High Performance Computing...  ..., and lifecycle events.Collaborate with clusters, networking, and infrastructure teams.Partner... 
    Work at office
    Local area
    Flexible hours

    OpenAI

    San Francisco, CA
    4 days ago
  • $229k - $299.8k

     ...largest AI training and inference fabrics in the world — GPU clusters, high-radix Ethernet back-end networks, and the optical transport connecting them. The pace...  ...'s outcome. Your Impact As a Senior Solutions Engineer, you own the technical relationship with Neocloud... 
    Full time
    Temporary work
    Work at office
    Local area
    Relocation
    Flexible hours
    Shift work
    2 days per week

    Cisco

    San Francisco, CA
    2 days ago
  • $10k - $20k

     ...services. The company is looking for a Network Automation Engineer to design and build high-speed, low-latency network fabrics for large-scale GPU-accelerated computing. The role...  ...driven deployment model. Optimising AI Clusters: You will be writing sophisticated... 
    Full time
    San Francisco, CA
    more than 2 months ago
  •  ...fabrics the frontier trains on. Lossless, non-blocking backend networks for clusters of 100k+ accelerators, re-derived for every new generation of...  ...Own the network design lifecycle from customer requirements (GPU shape, workload, scale, tenancy) through deployable,... 
    Local area

    FluidStack

    San Francisco, CA
    2 days ago
  • $225k

     ...Looking to shape the networks behind large-scale AI compute? Join...  ...compute providers. The Network Engineer will help define design standards...  ...for current and next-generation GPU fabrics across different sites...  .... Close collaboration with cluster architects and deployment engineers... 
    Full time
    Work experience placement
    San Francisco, CA
    2 days ago
  • About the Role GPU clusters are some of the most performant computers on the planet. Even smaller...  ...to shape culture, mentor junior engineers, and learn from our customers. About You...  ...hands-on management or architecture with network for at least one GPU cluster in the... 
    Work at office
    Local area
    Visa sponsorship
    3 days per week

    The San Francisco Compute Company

    San Francisco, CA
    2 days ago
  • $144.5k - $180.6k

     ...data processing, and software engineering, our office is a truly...  ...~ Knowledge of hardware and network level implications of on‑prem...  ...optimization, management, and cluster tuning in a constrained environment...  ...Experience with CUDA‑based GPU programs Security expertise... 
    Full time
    Temporary work
    For contractors
    Work at office
    Local area
    Remote work
    Worldwide
    Home office
    3 days per week

    Planet Labs PBC

    San Francisco, CA
    2 days ago
  • $180k - $220k

     ...Role:As a Staff Cloud Support Engineer, you are a technical...  ...across Customer Experience, SRE, Networking, Fleet, and Product teams.You...  ...Slurm, Terraform), and AI/ML cluster stability.Reduce MTTR and incident...  ...NCCL, IB, GPU driver/firmware issues, distributed... 
    Temporary work

    Crusoe

    San Francisco, CA
    4 days ago
  • $401k

     ...the RoleOpenAI is seeking a Principal Security Engineer to join our Infrastructure Security (InfraSec...  ...and production environments, spanning GPU supercomputing clusters, multi-cloud infrastructure, datacenters, networking, storage, and the critical services that power... 
    Work at office
    Local area
    Remote work
    Flexible hours

    OpenAI

    San Francisco, CA
    1 day ago
  •  ...ChatGPT and the OpenAI API. We work across engineering and product teams to make infrastructure...  ...other teams build on. Our work spans cluster creation and lifecycle management, upgrades...  ...platform capabilities, rather than specialized GPU optimization. We build across OCI, AWS... 
    Relocation package

    OpenAI

    San Francisco, CA
    2 days ago
  •  ...skills with hands-on knowledge of GPU-accelerated computing and the...  ...AI compute, high-speed networking, storage, and overall platform...  ...SUSE Product Management and Engineering to help inform product direction...  ...Kubernetes architecture and cluster administration experience,... 
    Full time

    SUSE

    San Francisco, CA
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to GPU Cluster Engineer, Networking. Be the first to apply!