Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Member of Technical Staff, Evals Lead

$2,000 per month
Full-time

build-ai

About Build AI

Build AI is the data hyperscaler for Physical AI. We're vertically integrated across hardware, manufacturing, logistics, collection, and model training to scale the physical labor dataset orders of magnitude faster than anyone in the world.

Job Summary

We’re hiring an Evals Lead to benchmark video and world-model capabilities with and without our data. You need to deeply understand the philosophy of evals: what an eval is allowed to claim, contamination, leakage, construct validity, and whether a number actually corresponds to a capability. More practically, you should have pushed consequential evals before — evals that changed what a lab trained, shipped, or collected, not a weekend leaderboard.

Key Responsibilities

  • Design and run benchmarks for video and world models, with Build data and without it, under the same protocol

  • Make the comparison honest: held-out tasks, contamination and leakage checks, no cooking the numbers

  • Build the analytics layer: model performance, failure patterns, and whether scaling the dataset moves capability — and on which axes

  • Work with research and dataset/quality so collection and evals inform each other

  • Push evals that are consequential enough that people change plans when the number moves

  • Design processes that increase evaluation quality, repeatability, and scale as we add tasks and countries

You may be a good fit if you have (Must-have qualifications)

  • You have shipped or driven evals that mattered: they changed training, hiring, product, or data decisions

  • You understand eval philosophy well enough to argue about validity, not only to plot a curve

  • Ideally video, robotics, world models, or multimodal, but the bar is consequential evals more than a specific domain

  • You will not confuse a pretty dashboard with an eval that is allowed to decide things

Strong candidates may also have experience with (Nice-to-have qualifications)

  • Video, robotics, world models, or multimodal evals

  • You have designed evals used in a paper, a product launch, or a data decision

  • Background in construct validity, contamination, or leakage

  • Exposure to evaluation operations: throughput, failure taxonomy, repeatability

Benefits

  • Competitive pay

  • Medical, dental, and vision packages with generous premium coverage

  • $500 per month credit for waiving medical benefits

  • Housing subsidy of $2k per month for those living within walking distance of the office

  • Relocation support for those moving to San Francisco (Financial District) or Shenzhen (Nanshan)

  • Various wellness benefits covering fitness, mental health, and more

  • Daily lunch and dinner in our office

  • Unlimited compute budget subject to ROI justification

  • Unlimited Codex and Claude credits

  • Travel

How we're different

Build believes in the Bitter Lesson . By taking a general approach of learning from humans, our addressable market is all physical labor.

We are a fully in-person team in San Francisco (Financial District) and Shenzhen (Nanshan), and greatly value engineering skills. We do not have boundaries between engineering and research, and we expect all of our technical staff to contribute to both and work across disciplines as needed.

Build AI is an equal opportunity employer. We review every application. If you do not meet every bullet, still apply. Questions: View email address on jobs.ashbyhq.com

Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the Member of Technical Staff, Evals Lead in San Francisco, CA vacancy
  •  ...Palantir—and help build the systems that make expert human knowledge useful for advancing AI. The Role We are hiring a Member of Technical Staff, Evals to help define how frontier AI systems are measured, understood, and improved. This is a broad, high-ownership role... 
    Suggested
    Full time
    Work at office
    Flexible hours

    handshake

    San Francisco, CA
    5 days ago
  • $225k

     ...this goal. About the role The Evals team measures the performance and capabilities...  ...'s most important decisions. As a Member of Technical Staff on Evals, you will develop new...  ...subtle bugs or flawed assumptions that lead to misleading conclusions. We care deeply... 
    Suggested
    Remote job
    Full time
    Visa sponsorship
    Relocation package

    magic.dev

    San Francisco, CA
    5 days ago
  • $150k - $220k

    # Founding Member of Technical Staff (MTS)Bay Area, CAFull-time$150k-$220k + equity## About UsVizopsAI is the secure runtime for custom enterprise...  ....## What You'll Do* •Build backend services for training, evals, telemetry, and online policy updates* •Instrument... 
    Suggested

    Vizops

    San Francisco, CA
    4 days ago
  •  ...that runs order-to-cash for the world’s leading enterprises. Across order management,...  ...ai . The Role We’re hiring a Member of Technical Staff – AI/ML to design, build, and deploy...  ...calibration Have strong opinions on AI/ML evals — golden datasets, offline + online... 
    Suggested
    Full time
    Flexible hours

    stuut-ai

    San Francisco, CA
    5 days ago
  •  ...Member of Technical Staff @ Lotus AI Who we are Lotus AI is a groundbreaking primary care app that integrates your medical records, AI...  ...and fine-tuning, model tooling, data pipelines, retrieval/evals, and product workflows. You'll be close to the core system... 
    Suggested
    Full time

    lotushealth

    San Francisco, CA
    5 days ago
  •  ..., and actionable. In this role, you will build and improve the technical foundations that support Answer Quality across Perplexity. This...  ...shared evaluation infrastructure that helps teams run reliable evals, analyze results, and make product and model decisions Develop... 
    Remote job
    Full time

    perplexity

    San Francisco, CA
    5 days ago
  •  ...evaluation infrastructure alongside the world’s leading AI labs. Now, we’re bringing those...  ...experts, and devise new methods, so that evals and rubrics keep improving and so do the...  ...Degree in Computer Science or a related technical degree such as Electrical Engineering, Physics... 
    Full time
    Work at office
    Local area
    Relocation package

    mercor

    San Francisco, CA
    5 days ago
  • $180k - $200k

     ...Job Description Job Description Member of Technical Staff – Full Stack (Backend-Leaning) Location...  ...agent evaluation, LLM systems, or evals/QA. This role may not be for you if...  ...engineering, solutions, product, or management/lead role. This is a hands-on engineering... 
    Full time
    Work at office
    Immediate start

    Syndesus

    San Francisco, CA
    28 days ago
  •  ...Read more here, here, and here. The role You’ll lead the red-teaming of the world’s most used AI agents,...  ...Refine the core engine that runs all our red-teaming evals DRI end-to-end: technical scoping, reconnaissance, execution, remediation, certification... 
    Relocation
    Visa sponsorship

    Artificial Intelligence Underwriting Company

    San Francisco, CA
    2 days ago
  •  ...Our AI agents do the research a great analyst would. They follow leads across company sites, public records, and social platforms,...  ...done. New data sources and tools for the enrichment engine. Evals that tell us whether a prompt, model, or agent change made results... 
    Full time
    Work at office
    Night shift

    Sixtyfour

    San Francisco, CA
    4 days ago
  •  ...Francisco or Seattle with a hybrid work arrangement, offering a collaborative environment where technical and product decisions shape the company's future. The Member of Technical Staff position involves full-stack product engineering, working closely with clinicians and... 

    AI Voice Base

    San Francisco, CA
    4 days ago
  •  ...Member of Technical Staff – Machine Learning I’m partnering with a rapidly scaling healthtech startup that has just raised a $40M Series A to expand its engineering team. Their AI-powered platform is already helping clinicians by automating huge amounts of back-office... 
    Work at office

    Quantix Search

    San Francisco, CA
    2 days ago
  •  ...within a native office suite, and deploys them in secure environments for Fortune 500 companies. About the Role As a Member of Technical Staff, you will be part of the team responsible for the work platform. You will think end-to-end about what it means to build... 
    Work experience placement
    H1b
    Work at office
    Visa sponsorship

    Product Pulse

    San Francisco, CA
    4 days ago
  •  ...applied across dozens of biopharma partnerships. Backed by leading technology and biotechnology investors including a16z, Lux,...  ...machine learning and drug discovery. We are seeking a Member of Technical Staff, Machine Learning to join our talented, ambitious, and... 

    Inductive Bio

    San Francisco, CA
    4 days ago
  •  ...AI coding tools such as Claude Code or Codex Clear thinking about technical trade-offs, and clear written communication Experience doing research or running rigorous experiments, such as LLM evals, benchmarks or A/B tests, and drawing sound conclusions from noisy... 
    Full time

    Lightsage

    San Francisco, CA
    4 days ago
  •  ...OpenAI, Google DeepMind, Cursor, Decagon, CoreWeave, Profound, Northwood Space, Solidworks, and more. About the Role Member of Technical Staff is a generalist position where you will collaborate closely with the founders on product direction, implementation, and... 

    Ripline

    San Francisco, CA
    4 days ago
  • $227.5k - $401k

     ...Member of Technical Staff San Francisco This is Adyen Adyen provides payments, data, and financial products in a single solution for global...  ...or fixing critical performance issues. Learn and Lead: Connect with the broader AI community (including startups,... 
    Work at office
    Immediate start
    Relocation
    Flexible hours

    Adyen

    San Francisco, CA
    5 days ago
  •  ...getting started. THE ROLE As a Member of the Technical Staff, you build the system that makes our agents...  ...and remediate findings, and build the evals that tell us whether they actually got...  ...Work with customers and our security leads to turn real findings, misses, and... 
    Full time

    Tolmo

    San Francisco, CA
    9 days ago
  •  ...hyperscalers, and AI-native companies, giving our team access to technical problems spanning frontier models, production infrastructure, and emerging hardware. About the role As a Member of Technical Staff, you will build the systems that schedule, route, and... 
    Full time

    gimlet

    San Francisco, CA
    5 days ago
  •  ...Stuut is the AI coworker that runs order-to-cash for the world’s leading enterprises. Across order management, credit, collections,...  ...finance teams set rules, review decisions and steer agents. Write evals against real customer outcomes, and turn each failure into a fix... 
    Full time
    Flexible hours

    stuut-ai

    San Francisco, CA
    5 days ago
  •  ...them from idea through deployment. Collaborate on architecture, product decisions, and technical direction. Help define product engineering practices as an early team member. Requirements ~6+ years of experience with deeper architectural ownership and emphasis... 
    Full time
    Temporary work
    Work at office
    Remote work

    Pallet

    San Francisco, CA
    10 days ago
  • $200k - $300k

     ...Job Description Job Description Member of Technical Staff Company: Confidential — AI cloud on a mission to maximize intelligence per watt Location: San Francisco, CA (FiDi office, on-site 5 days per week) Compensation: $200,000 - $300,000 + 0.5% - 1% equity... 
    Full time
    Internship
    H1b
    Work at office
    Overseas
    Visa sponsorship

    Wafer AI

    San Francisco, CA
    2 days ago
  •  ...and enterprises that integrate LLMs into their products. The team is 5 people with a research and product focus. As a Member of Technical Staff on our infrastructure team, you'll own the cloud systems that serve our compression API end-to-end. You'd get to build... 
    Full time
    Visa sponsorship

    the-token-company

    San Francisco, CA
    5 days ago
  • $150k - $300k

     ...the mission. Be dependable -  Take ownership and deliver with high agency.   The Role   Nimble is looking for a Member of Technical Staff to help us advance our robotics moonshot by designing, developing, training, and implementing robotic foundation models,... 
    Local area
    Immediate start
    Flexible hours
    Weekend work

    Nimble Robotics

    San Francisco, CA
    2 days ago
  •  ...internal agents across government, Fortune 500, frontier labs and hyper-growth startups. Teams use our configurable primitives such as evals, observability, router, authentication, permissions, sandboxes, harnesses and more to deploy internal agents at scale securely. We... 
    Full time

    brainbaselabs

    San Francisco, CA
    5 days ago
  •  ...Job Description Job Description About the Role We are hiring Members of Technical Staff internally referred to as Founding Engineers. The mandate is to architect and own the core systems that power real-time AI voice and compliance workflows for top-10 US mortgage... 
    H1b
    Immediate start
    Relocation
    Relocation package

    The SeventhHire LLC

    San Francisco, CA
    3 days ago
  •  ...Member of Technical Staff, Science Our mission is to advance humanity's ability to engineer biology. We approach this by understanding and improving AI’s application to biology across basic research, drug discovery, and biosecurity. We are a team of engineers, scientists... 
    Full time
    Work at office
    Visa sponsorship
    Relocation package
    Flexible hours

    LatchBio

    San Francisco, CA
    9 days ago
  •  ...systems engineering - distributed systems, APIs, databases, cloud infra. Expert Python (Go or TypeScript a plus). Terraform, GCP, containers, multi-tenant production experience. Taste for the LLM-specific infra problems: model routing, evals, cost and latency controls.... 
    Full time

    ciridae

    San Francisco, CA
    5 days ago
  •  ...good mental models of to do good work. Great interpersonal and technical communication. Please don't use LLMs to write prose. We desk-...  ...you want to hear. # Share an online whiteboard with a team member and work through a technical problem. We spend a lot of time at... 
    Full time
    Work at office

    sail

    San Francisco, CA
    5 days ago
  • $300k

     ...scale: distributed rollouts, training orchestration, inference, evals, data pipelines, observability, and reliability. You will...  ...Experience building tools, platforms, or services used by other technical users. Strong judgment around technical tradeoffs: when to prototype... 
    Work at office
    Local area

    Vmax

    San Francisco, CA
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Member of Technical Staff, Evals Lead. Be the first to apply!