Member of Technical Staff, ML Platform (Evals) [Remote]
runway-ml
- Remote job
We are building AI to simulate the world through merging art and science.
We believe that world models are at the frontier of progress in artificial intelligence. Language models alone won’t solve the world’s hardest problems – robotics, disease, scientific discovery. Real progress requires models that experience the world and learn from their mistakes, the same way that humans do. And this kind of trial and error can be massively accelerated when done in simulation, rather than in the real world. World models offer the most clear path to general-purpose simulation, changing how stories are told, how scientific progress is made and how the next frontiers of humanity are reached.Our team consists of creative, open minded, caring and ambitious people who are determined to change the world. We aspire to continuously build impossible things and our ability to do so relies on building an incredible team. If you are driven to do the same, we'd love to hear from you.
*Titles posted to our careers page are updated regularly to best advertise the role and expected seniority, but internally we have a flat titling system with two types of titles: one for individual contributors and one for people managers.
About the role
We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You'll turn it into one platform.
Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision.
You'll be embedded within research teams as a member of ML Platform, and you'll set the technical direction for evals across the company. This is an extremely high-leverage role: the quality of our models is bounded by how well we can measure them.
A peek at our technical stack
Our inference and training code is written in Python (PyTorch) and is cloud-native. We leverage multiple clouds and use a combination of Kubernetes native and home-grown tooling for efficient job orchestration.
We’re big users of agentic development and operations. We have a robust internal platform and provide broad access to the latest models and harnesses.
You’ll have access to best-in-class models, agents, GPUs, storage and cloud services you need to drive a world class evaluation pipeline that helps us deliver frontier models.
What you'll do
Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale
Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative
Work directly with research teams on video, image, audio, agents, and robotics to understand what they need to measure and build it, then generalize the result into the platform
Help set the standards for how we evaluate models at Runway: reproducibility, metric definitions, reporting formats, and when a result is trustworthy enough to act on
Support broad adoption of the platform and its integration throughout Runway’s research efforts across training, production model serving and more
Contribute broadly as a member of the ML Platform team to tools and systems that help Runway train and serve frontier models
What you'll need
5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems
Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes
Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
Comfort building internal tools end to end, from the command line to the browser
Ability to lead a broad technical area: gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap without waiting to be told
Familiarity with the full model development lifecycle: data, training, evaluation, serving
Self-starter who can work embedded with research teams and move fast
Strong systems thinking and pragmatic approach to production reliability
Humility and open mindedness; at Runway we love to learn from one another
Nice to have
Track record of building evaluation suites for generative models
Hands-on work with LLM- or VLM-as-judge pipelines
Experience with online experimentation platforms and connecting offline metrics to product outcomes
Prior work evaluating agents or robotics policies
Runway strives to recruit and retain exceptional talent from diverse backgrounds while ensuring pay equity for our team. Our salary ranges are based on competitive market rates for our size, stage and industry, and salary is just one part of the overall compensation package we provide.
There are many factors that go into salary determinations, including relevant experience, skill level and qualifications assessed during the interview process, and maintaining internal equity with peers on the team. The range shared below is a general expectation for the function as posted, but we are also open to considering candidates who may be more or less experienced than outlined in the job description. In this case, we will communicate any updates in the expected salary range.
Lastly, the provided range is the expected salary for candidates in the U.S. Outside of those regions, there may be a change in the range, which again, will be communicated to candidates.
Working at Runway
Great things come from great teams. We’d love to hear from you.
We’re committed to creating a space where our employees can bring their full selves to work and have equal opportunity to succeed. So regardless of race, gender identity or expression, sexual orientation, religion, origin, ability, age, veteran status, if joining this mission speaks to you, we encourage you to apply.
More about Runway
We're excited to be recognized as a best place to work:
- ...than $3 billion has moved through the platform to date. Founded in New York by... .... The Role We’re hiring a Member of Technical Staff – AI/ML to design, build, and deploy AI-powered... ...Have strong opinions on AI/ML evals — golden datasets, offline + online...PlatformFull timeFlexible hours
$500 per month
...delivered through a breakout consumer platform. Today's frontier models... ...hours. As our RecSys founding member, you'll own this problem end-... ...in recommendation systems, ML engineering, or applied ML in... ...Experience running recommendation evals end-to-end (offline + online)...PlatformFull timeWork at officeFlexible hours$200k - $400k
...About the Role We're looking for a Member of Technical Staff, Product Security with deep infrastructure... ...expertise to secure the Inferact platform as it moves from early deployments to... ...company. Experience securing AI / ML infrastructure, model serving, GPU workloads...PlatformFull timeRemote workVisa sponsorship$150k - $300k
...physical economy. What You'll Do As a Member of Technical Staff on our Frontier Data team, you'll... ...that close them. You'll write the platform code that lets us run hundreds of thousands... ...surface area — systems engineering, ML tooling, and research collaboration —...PlatformWork at officeRemote work- ...Technical Staff Position At 3Y Health 3Y Health is building the operating... ...practices, an AI-native platform that helps clinicians start,... ...psychiatry. About the Role Members of Technical Staff own... ...incident response Write evals for model-facing features and...PlatformRemote work
$190k - $210k
...Member of Technical Staff, Software Engineering, Mosaic Reporting Cognita Imaging Level: Senior... ...lead and partner across engineering, ML, and clinical teams to take the product... ...metadata and model outputs. Ensure platform services are reliable, secure, low-...PlatformFull timeRemote work- ...video, audio, slide generation, and more, through the large-scale platforms that we’ve built. Design Arena, our flagship product, is the... ..., Physics, or a related field. Experience building ML systems. You've trained preference models, built data pipelines,...PlatformFull timeRelocationVisa sponsorship
- ..., ICLR, or comparable physics-ML venues). You can defend the result... ..., memory and harnesses, evals, verifiers, reward models, instrumentation... ...How We Work We hold a high technical bar and give people full... ...perspectives in building platforms for AI-driven discovery....PlatformRemote jobPermanent employmentFull timeWork at office3 days per week
- ...Member of Technical Staff, Customer Tools Location: North America Remote / San Francisco, CA · Full... ...Andromeda is a market and infrastructure platform to buy, sell, and operate compute.... ...role. Strong Candidates May Have ML or HPC infrastructure: Training clusters...PlatformRemote jobFull time
$20 - $50 per hour
...Job Title: Member of Technical Staff, Robotics & Embodied AI Research Job Type: Full-time Location... ..., our human-in-the-loop annotation platform, to structure and scale robotics datasets... ...and experience with robotics or ML frameworks such as PyTorch, ROS, or simulation...PlatformFull timeLocal areaRemote work$200k - $300k
...Higgsfield AI is looking for a highly technical, hands-on MTS, ML Security to join Higgsfield Security... ...model loading. Partner with ML and Platform Engineering teams to integrate... ...in the San Francisco Bay Area . Team members are expected to work from our San Francisco...PlatformRemote jobFull timeWork at officeWorldwide3 days per week$175k - $350k
...Job Description Job Description Member of Technical Staff - Applied ML Company: Confidential — venture-backed technology company Location: New... ...applications: benchmarking, model orchestration and evals for agent behavior and reliability ~ Structured ML experimentation...Full timeWork at officeVisa sponsorship$200k - $320k
...Job Title: Member of Technical Staff, Model Training (infra & platform) Job Type: Full-time Location : Remote The Role In this role, you will take technical... .... Nice to Have Experience with distributed ML frameworks and technologies such as PyTorch...PlatformFull timeLocal areaRemote work- ...follow leads across company sites, public records, and social platforms, cross-check what they find, and score how much each answer can... ...done. New data sources and tools for the enrichment engine. Evals that tell us whether a prompt, model, or agent change made...PlatformFull timeWork at officeNight shift
- ...Member Of Technical Staff We are seeking an experienced, hands-on software engineer to build and scale an AI-native platform transforming how healthcare practices operate. In this role, you will take ownership of features from initial discovery and technical design...PlatformContract workRemote work
- ...to its limits. We're looking for an engineer to own the GPU and ML-systems layer that frontier AI teams run on: accelerator scheduling... ...handling, and sandboxes for training, RL rollouts, and evals. Make the AI stack run great out of the box : deepen integrations...Full timeRemote work
- ...Collaborate on architecture, product decisions, and technical direction. Help define product engineering practices as an early team member. Requirements ~6+ years of... ...product or infrastructure in a SaaS or platform environment. ~ Ability to consider reliability...PlatformFull timeTemporary workWork at officeRemote work
- ...help architect and scale the core systems that power Nectar's platform. You'll work on the infrastructure that ingests, processes, and... ...operate on that data in real time. This role combines deep technical execution with architectural leadership. You'll help define...PlatformFull timeWork at officeRemote workFlexible hoursShift work
- ...Member of Technical Staff, Provider Tools Location: North America Remote / San Francisco, CA · Full-Time About Andromeda Andromeda is a market and infrastructure platform to buy, sell, and operate compute. We believe demand for compute will grow exponentially...PlatformFull timeRemote work
- ...Mobile Engineer to own and define the mobile experience at Nectar. You'll architect and build the mobile platform from the ground up, making foundational technical decisions that shape how we deliver AI-powered social commerce to brand teams. This is a highly autonomous...PlatformFull timeRemote workFlexible hoursShift work
$130k - $220k
...across the United States to help them hire. Title of Role: Member of Technical Staff (AI Benchmarking) Location: San Francisco, CA (On-site, 5... ...of experience in consulting (MBB) or technical roles (SWE, ML, TPM, data roles) ~ Strong Python proficiency with recent...Work at officeRemote workVisa sponsorshipRelocation package$117k - $175.4k
...hardest problems and providing unmatched technical expertise. As the operator of a... ...What You Need to be Successful - Senior Member of the Technical Staff Minimum Requirements: ~ Master's... ...Experience with applying AI/ML to microwave design problems We offer...Full timeFor contractorsWork at officeImmediate startRemote workRelocation packageFlexible hours$200k - $250k
...and LLM systems that run inside real enterprise workflows. The staff member designs evaluations, benchmarks, and datasets, digs into why agents... ...building evaluation pipelines, experimental tooling, or ML infrastructure Ability to trace agent failures to root causes...Full timeRemote work$200k - $400k
...in production. You'll work alongside the engineers building the platform, bringing a reliability perspective to how systems are designed,... ...qualifications: Experience supporting AI inference, model serving, ML infrastructure, GPU workloads, or other latency-sensitive, high-...PlatformFull timeRemote workVisa sponsorship$500 per month
...the legal intelligence platform built for in-house legal... ...GC AI is hiring a Staff Product Manager to own... ...enterprises actually adopted. Technically deep. You can walk an... ...Familiarity with AI/ML products and how to build... ...company with team members across North America, and...PlatformRemote jobPermanent employmentFull timeCurrently hiringWork at officeImmediate startHome officeRelocation packageShift work- ...Responsibilities Own the evaluation platform end to end, including systems for generating... ...other research efforts. Contribute to ML Platform tools and systems used to train... ...the browser. ~ Ability to lead a broad technical area, gather requirements, set direction,...PlatformFull time
- ...capacity together into one unified, productive platform, making it easier to access, sell, resell... ...storage and data infrastructure for AI/ML workloads, including object storage, high... ...excellent communication skills across technical and non-technical stakeholders Proven...PlatformFull timeRemote work
$200k - $400k
...managing GPU clusters and debugging hardware issues. Ability to work across cloud platforms (AWS, GCP, Azure) and on-premise infrastructure. Preferred qualifications: Experience with ML-specific orchestration tools (Ray, Slurm). Knowledge of GPU scheduling, multi...PlatformFull timeRemote workWorldwideVisa sponsorship$200k - $320k
...built and scaled with flow, micro1's human-in-the-loop annotation platform, from egocentric video, demonstrations, and sensor data. The... ...robotics, embodied AI, robot learning, computer vision, or applied ML research Hands-on experience training or evaluating robot...PlatformFull timeRemote work$180k - $260k
...Profound is the fastest-growing AI marketing platform, used by 15% of the Fortune 500 including... ...gets better every day. As an AI and ML Engineer, you will design, build, and... ...performance teams Clear communication with both technical and non-technical partners Ownership...PlatformFull timeWork at office
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Member of Technical Staff, ML Platform (Evals) [Remote]. Be the first to apply!



