Software Engineer, Infrastructure
$180k - $250kFal
Software Engineer, Infrastructure
San Francisco
You are a hands-on engineer who builds the software and processes that keep a large fleet of GPU servers healthy and productive. You write systems and tooling for managing 1000s of servers including provisioning, health monitoring, error detection, and recovery — and when something breaks that automation can't fix, you drive resolution with partners.
Key Responsibilities
- Build and maintain Python fleet tracking system that manages the full lifecycle of servers including contracting and procurement, target use, pricing, availability, health, RMAs, etc
- Build server management tooling that automates provisioning, health checks, GPU diagnostics, recovery and alerting
- Create and maintain metrics, dashboards, and alerting for hardware health across the fleet (GPU errors, disk failures, network issues, thermals)
- Leverage AI to an extreme level to build tools and automate alerting and recovery
- Implement and enforce OS-level security: hardening baselines, SELinux/AppArmor policies, SSH key management, vulnerability scanning, and compliance automation
- Manage and optimize distributed and local storage systems supporting model weights, checkpoints, and ephemeral scratch: NVMe arrays, NFS, parallel file systems, and object storage
- Tune Linux systems for AI workloads: kernel parameters, NUMA topology, CPU pinning, hugepages, I/O schedulers, and GPU driver stack optimization (NVIDIA drivers, CUDA, container runtimes)
- Develop a suite of automated error detection and recovery processes
- Work with partners to solve technical issues
Requirements
- 3+ years experience managing bare-metal and cloud based server fleets at scale (100+ nodes)
- Strong software engineering skills in Python; you write production tooling, not scripts
- Deep Linux systems knowledge: boot process, kernel tuning, networking, storage, systemd, cgroups, namespaces, performance profiling
- Strong experience with configuration management and infrastructure-as-code: Ansible, Terraform, cloud-init
- Solid understanding of storage technologies: LVM, RAID, NVMe, NFS, Lustre or GPFS, and Linux I/O stack tuning
- Familiarity with hardware diagnostics and failure modes (GPUs, NVMe, NICs, memory)
- Experience building internal tools or dashboards for infrastructure visibility
- Excellent communication and ability to drive technical decisions across teams
- Self-starter who executes quickly, takes ownership, and constantly seeks improvement
Nice to Have
- Familiarity with network configuration and diagnostics (VLAN, VXLAN, ECMP, BGP, tcpdump)
- Experience with NVIDIA GPU infrastructure: driver management, health monitoring, DCGM, NVLink/NVSwitch diagnostics, RDMA, InfiniBand/RoCEv2
- Experience with AMD GPUs
- Experience with bare metal and VM provisioning (PXE/iPXE, Kickstart, libvirt, Qemu/KVM)
- Experience with compliance frameworks relevant to cloud providers (SOC 2, ISO 27001)
Compensation
- $180,000-250,000 plus equity + benefits
Location
- San Francisco, CA (we are open to remote in the US for Senior and Staff levels)
What We Offer At Fal
- Interesting and challenging work
- A lot of learning and growth opportunities
- We are offering relocation assistance to San Francisco.
- Health, dental, and vision insurance (US)
- Regular team events and offsites
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Software Engineer, Infrastructure in San Francisco, CA vacancy
$255k - $405k
...About the Team The Agent Infrastructure team at OpenAI is responsible for building systems... ...execute code, debug issues, and develop software just as human SWEs do. Our training... ...world. About the Role As a Software Engineer on the Agent Infrastructure team, you will...SuggestedWork at officeWorldwideRelocation packageOpenAI
San Francisco, CA2 days ago$140k - $260k
...As an Infrastructure Engineer, you will build and scale the systems that power our rapid growth. You will ensure our infrastructure is highly available, cost-effective, and able to handle explosive traffic and compute demands. You will work closely with engineers across...SuggestedWork at officeVisa sponsorshipSlope
San Francisco, CA2 days ago- ...economy has grown 6x over the last decade, software businesses have gone from not worrying... ...In this role, you will play a Software Engineer who designs, builds, and operates the... .... Drive technical decisions about our infrastructure. Identify and fix bugs and technical debt...SuggestedWork at officeRemote workHome officeFlexible hours3 days per week
Anrok
San Francisco, CA3 days ago $168k - $213k
...and our team comes from world-class organizations like SpaceX, Google, Netflix, Stripe, Plaid and more. The Role As a Senior Infrastructure Engineer , you will own the foundation that enables us to deploy secure, compliant AI systems at major financial institutions...SuggestedFlexible hoursBretton AI
San Francisco, CA3 days ago$180k - $250k
...You are a hands-on engineer who builds the software and processes that keep a large fleet of GPU servers healthy and productive. You write systems... ...Strong experience with configuration management and infrastructure-as-code: Ansible, Terraform, cloud-init Solid understanding...SuggestedCurrently hiringLocal areaRelocationVisa sponsorshipFal
San Francisco, CA2 days ago- ...Software Engineer, Infrastructure Our mission is to automate coding. The first step in our journey is to build the best tool for professional programmers, using a combination of inventive research, design, and engineering. Our organization is very flat, and our team...Work at office
Anysphere
San Francisco, CA1 day ago $166k - $225k
...and running the world's best data and AI infrastructure platform so our customers can use deep... ...to improve their business. Founded by engineers — and customer obsessed — we leap at every... ...'re only getting started. As a Senior Software Engineer on the Infrastructure teams,...Local areaWorldwideFlexible hours- Software Engineer Intern (AI Infrastructure / Training / Inference) About the Role We are hiring Software Engineers focused on AI Infrastructure to build the systems that enable frontier multimodal AI to operate reliably at production scale. This role exists because modern...InternshipImmediate start
$325k
About the Role We are seeking a Cloud Infrastructure Engineer to help design and evolve the platforms that power OpenAI's products. In this role, you will be a hands‑on technical leader, driving the architecture, scalability, and reliability of critical infrastructure...$170k - $240k
...Orb Orb is transforming how modern AI and software companies monetize at scale. We’ve built the next-generation billing infrastructure that turns complex usage‑based pricing... ...large customer growth Partner with other engineering teams to ensure we build reliable product...Work at office3 days per week- About the Role This role broadly owns infrastructure across the stack. If it’s running in the cloud, you probably care about it. The life... ...and scaled ChatGPT and GPT‑4 to hundreds of millions of users, engineered the foundations of autonomous driving, built next‑generation...
- ...is a new cross-functional group working across engineering, product, research, and design to build the foundational... ...the Role We’re looking for an experienced Software Engineer to help build the machine learning infrastructure that powers OpenAI’s monetization and ads...
$180k - $260k
...the company: the company is the agentic infrastructure company. We free people and agents to... ...exceptional developer experience. Now, software is entering a new era, and the next generation... ...every day. As a Senior Infrastructure Engineer on the CI/CD team, you’ll take...Work from homeWorldwideFlexible hours$260k - $330k
About the Web Infrastructure Team Notion is building one place where your knowledge, projects, meetings, and AI tools live side by side,... ...modern teams think and execute. About the Role You’ll be a core engineer on the Web Infrastructure team, focusing on improving Notion’...Local area- Exa is building a search engine from scratch to serve every AI application. We build massive‑scale infrastructure to crawl the web, train state‑of‑the‑art embedding models to... ...thousands of machines Design GPU scheduling software so we max out our cluster utilization...H1b
$202k - $237k
...democratize distributed computing and make it accessible to software developers of all skill levels. We’re commercializing... .... About the role Anyscale is looking for a Software Engineer to join the Infrastructure team. Anyscale aims to provide the next generation of tools...Work at officeFlexible hours- About Flow Flow Engineering is building a requirements platform for modern engineering teams, helping them design, build... ...reliability and speed. About the role Flow is hiring a Software Engineer with an infrastructure focus to build and scale the core platform behind Flow...Flexible hours
- ...including stablecoins. You’ll help design, deploy and operate the infrastructure that makes this possible. This is a hands-on devops role... ...role sits at the intersection of infrastructure, platform engineering and security. You’ll define how Modern Treasury scales and operates...Work at officeLocal areaImmediate start
$190k - $221k
Senior Software Engineer, Infrastructure TRM Labs is a blockchain intelligence company committed to fighting crime and creating a safer world. By leveraging blockchain data, threat intelligence, and advanced analytics, our products empower governments, financial institutions...Remote work$200k - $325k
...legacy systems with adaptive, learning software. Founded in early 2024, Serval is already... ...IT, HR, Finance, Security, Legal, and Engineering. Our mission is to eliminate... ...modern enterprises. As a Software Engineer, Infrastructure, you’ll build and scale the foundational...Flexible hours$164.2k - $205.2k
RDQ427R175 At Databricks, the Compute Infrastructure organization builds and operates the foundation that runs all Data, AI and stateful... ...elasticity, reliability and cost efficiency. As a Senior Software Engineer on the Compute Infra team, you will design and build the...Local area$172.2k - $215.2k
Senior Software Engineer, Storage Infrastructure - Remote Eligible Location: Remote Compensation: $172,200 - 215,200 USD / year Job Tags: Software About The Role Planet designs, builds, and operates the largest constellation of imaging satellites in history. The constellation...Full timeTemporary workWork at officeLocal areaRemote workWorldwideHome office3 days per week$194k - $239k
Senior Software Engineer, Infrastructure Hover helps people design, improve, and protect the properties they love. With proprietary AI built on over a decade of real property data, Hover answers age-old questions like “What will it look like?” and “What will it cost?” Homeowners...Full timeFor contractorsWork at officeLocal areaFlexible hours$190k - $205k
...insightsinstantly. We are revolutionizing how businesses solve their most complex problems. As a Software Engineer in our Platform & Core Services team, or our Infrastructure team, you’ll design, build, and optimize core systems that power the Instabase platform. You...Temporary workWork at officeFlexible hoursWeekend work- ...reliability, performance, and security for multi‑tenant compute. What You’ll Do Design and operate secure, multi‑tenant container infrastructure with fast startup and smart autoscaling. Ship cloud deployments (Helm/Terraform) with SSO, network controls, and audit logging...Remote work
- About the Team OpenAI’s Privacy Engineering team sits at the intersection of Security, Privacy... .... Our mission is to build data infrastructure and systems to support our privacy, legal... ...culture. About the Role We’re looking for a Software Engineer who can design and operate...Relocation package
- ...accelerating project schedules of billion-dollar infrastructure projects and improving safety on job... ...construction veterans and world-class engineers to solve physical-world problems that... ...Onboard Infrastructure team builds the software foundation Bedrock's autonomous...Work at officeFlexible hours
$215k - $265k
...collaborate with them on scheming mitigations. We’re looking for a Software Engineer to build the platform that the rest of Apollo runs on .... ...why, and in what order. Build and maintain Apollo's cloud infrastructure . This means IaC, networking, environment management,...Full timeWork experience placementWork at officeImmediate startVisa sponsorshipFlexible hours- About Orb Orb is redefining what billing software can be, turning one of the most complex... ...the role As a senior member of the infrastructure team, you’ll play a key role in maintaining... ...customer growth Partner with other engineering teams to ensure we build reliable product...Work at officeRemote work3 days per week
$145k - $215k
...largest organizations to empower scientists, engineers, financial experts, product creators,... ...newest Snorkeler! We’re looking for a Software Engineer in Test who thrives in fast-... ...beyond traditional QA—fine tuning test infrastructure, tooling, and processes that directly...Local area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer, Infrastructure. Be the first to apply!
Related searches
- software engineer amazon San Francisco, CA
- experienced software developer San Francisco, CA
- federal - software developer San Francisco, CA
- software developer internship San Francisco, CA
- senior software engineer San Francisco, CA
- software developer fintech San Francisco, CA
- part time software developer remote San Francisco, CA
- software developer intern San Francisco, CA
- software data engineer San Francisco, CA
- fall software engineering internship San Francisco, CA