Member of Technical Staff

Responsibilities

• Own the evaluation platform end to end, including tooling and systems for generating, annotating, reviewing, and adapting model evaluations at scale.
• Design CLIs, APIs, GUIs, and storage layers for a seamless and collaborative evaluation workflow.
• Partner with research teams in video, image, audio, agents, and robotics to build and generalize evaluation capabilities.
• Set standards for reproducibility, metric definitions, reporting formats, and trustworthy evaluation results.
• Drive adoption and integration of the platform across training, production model serving, and broader research efforts.
• Contribute to ML Platform tools and systems used to train and serve frontier models.

Requirements

• 5+ years of experience building ML infrastructure or data platforms in production, including experience with evaluation, experimentation, or benchmarking systems.
• Strong Python and PyTorch skills, with hands-on experience running large-batch GPU workloads on Kubernetes.
• Experience designing data pipelines and storage for large volumes of media or model outputs, including versioning and reproducibility.
• Familiarity with experimental statistics, including paired comparisons, confidence intervals, multiple-comparison pitfalls, and inter-rater agreement.
• Experience building internal tools end to end from the command line to the browser.
• Ability to lead a broad technical area, gather requirements, set direction, make tradeoffs, and drive a roadmap.
• Familiarity with the full model development lifecycle across data, training, evaluation, and serving.
• Preferred experience building evaluation suites for generative models, LLM- or VLM-as-judge pipelines, online experimentation platforms, or evaluations for agents and robotics policies.
• Ability to work independently with research teams, move quickly, and apply strong systems thinking to production reliability.

Benefits

• The posting describes a comprehensive compensation package and an emphasis on pay equity, but does not provide a specific salary amount.
• Employees have access to models, agents, GPUs, storage, and cloud services for building evaluation pipelines.
• Runway is committed to an inclusive workplace and equal opportunity for employees from diverse backgrounds.