AI Systems engineer

Systems Engineer, Training Infrastructure Tilted · San Francisco, CA [(hybrid) / Remote-US] · Full-time · Founding team · 2 openings About Tilted Tilted is an agent economy. Not a chat box — an operating system where every unit of work is a typed, owned, logged job on a NATS job mesh, where agents hold wallets, call each other as services, and settle payments in real time. Job state is reconstructed from an event log, not held in process memory, which is why a job survives a backend restart and any node can read any node's work. If that architecture appeals to you, the rest of this posting will too. We're standing up an in-house pretraining program — staged, starting at a few hundred GPUs and climbing from there over the next few years. You're one of the first two infrastructure engineers on it, working directly with the Founding Pretraining Lead. They own the recipe. You own the machine that makes it. What you'll own The training stack. Distributed training across data, tensor, pipeline, and expert parallelism. Standing it up, scaling it, and keeping it correct when the topology changes underneath you. MFU. Model FLOPs utilization is a first-class tracked metric here and you are the person accountable for it. At our later phases, one point of MFU is millions of dollars — this is not a vanity number. Fault tolerance. Checkpoint and restart at scale, node-failure detection and automatic recovery, silent data corruption detection, loss-spike rollback. Runs are measured in weeks; nodes die every day. The run should not care. The MoE communication problem. Expert-parallel all-to-all is the bottleneck that defines throughput for sparse models. Owning it is most of the job at later phases. The data pipeline at scale. Tokenization, sharding, streaming loaders that keep thousands of GPUs fed without stalling. Mesh integration. Wiring the whole pipeline — ingest, filter, train, eval — as typed jobs on TiltedOS's praxis mesh, so it schedules, retries, and audits itself. Observability. The dashboards and alerts that tell us a run is diverging before we've burned a week of compute finding out. First 90 days Stand up the training monorepo inside TiltedOS with the pretraining lead. Bring up the first reserved cluster and get a real multi-node run training end to end. Ship checkpoint/restart that survives a deliberately killed node without human intervention. Get the ablation grid launching as scheduled mesh jobs, with results flowing back automatically. What we're looking for Required You've run distributed training on multi-node GPU clusters and debugged it when it broke. Any framework, any scale — what matters is that you've been the person who fixed it. Strong PyTorch internals: autograd, memory behavior, profiling, and where the time actually goes. Real experience with NCCL and collective communication — and the ability to tell a network problem from a kernel problem from a data-loader problem. Systems fundamentals: Linux, networking, storage, containers. You can read a flame graph and a nsys trace. Comfort with the on-call reality of long runs. Something will break at 3am during a two-week job, and you'll want to know why rather than just restart it. Nice to have CUDA/Triton kernel work, or custom kernel integration. FSDP, DeepSpeed, Megatron-LM, TorchTitan, or equivalent at four-figure GPU counts and above. MoE-specific systems work: expert parallelism, all-to-all optimization, capacity/dropping tradeoffs. FP8 / FP4 mixed-precision training in production. High-performance storage and checkpointing at petabyte scale. Experience with NATS, event-sourced systems, or job meshes — you'll be building on one. How you work Instrument first, guess second. You treat a 5% throughput regression as a bug, not a rounding error. You'd rather automate the recovery than be the recovery. Why this role is different Not a cog role. Two engineers, one ML lead. You'll touch every layer of the stack from the kernel to the scheduler, and your name is on the throughput number. Infrastructure that's already interesting. You're not bolting a training pipeline onto generic cloud primitives. You're building it inside an OS where every job is already typed, owned, audited, and metered — and making a frontier training program run as jobs on that mesh is a genuinely novel systems problem. A roadmap with real scale at the end. We start at hundreds of GPUs, staged and honest. The plan doesn't stop there, and neither does the difficulty. Compensation [Base band] · [Meaningful founding equity] · [Benefits] · [Hardware + conference budget] We publish our bands. Tilted is an equal opportunity employer. #J-18808-Ljbffr