Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Research Engineer, Benchmarking

Refresh

  • Research Engineer, BenchmarkingEngineeringSan FranciscoFull-timeBuild the benchmarks frontier labs use to measure real-world coding and computer-use capability. Translate expert workflows into rigorous, verifiable evaluations, run them against frontier models, and publish numbers that hold up under adversarial scrutiny.Across every role at Refresh: you're willing to ship full-stack work on our core stack (Vercel, Supabase, Render) when it's needed, and you're comfortable with reinforcement learning and supervised fine-tuning at a high level.
  • J-18808-Ljbffr Refresh

Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Research Engineer, Benchmarking in San Francisco, CA vacancy
  • Refresh is seeking a Research Engineer focused on benchmarking, to build rigorous evaluations of real-world coding and computer-use capabilities. You will translate expert workflows into verifiable tests and run benchmarks against frontier models. In this role you’ll work... 
    Suggested

    Refresh

    San Francisco, CA
    1 day ago
  •  ...improve AI using their own expertise and standards. Our APEX benchmarks measure how AI performs on economically valuable work, while...  ...Francisco, New York, and London. About the Role As a Research Engineer at Mercor, you’ll work at the intersection of engineering and... 
    Suggested
    Full time
    Work at office
    Local area
    Relocation package

    mercor

    San Francisco, CA
    5 days ago
  •  ...capabilities Build large-scale pipelines for synthetic data generation, filtering, and quality control Develop evaluations and benchmarks that measure meaningful model improvements Why join: Work at the frontier of multimodal foundation models and world models... 
    Suggested

    Goliath Partners LP

    San Francisco, CA
    5 days ago
  • $250k - $400k

     ...Research Engineer – Post-Training, Agents, Evals Fundamental AI Research Institute San Francisco Bay Area, USA (hybrid) $250,000 –...  ...including developing agentic and evaluation harnesses Develop benchmarks and evaluations for reasoning and agentic capabilities... 
    Suggested
    Relocation package

    Storm3

    San Francisco, CA
    2 days ago
  •  ...Hedra Job OpportunityHedra is a research and infrastructure lab advancing post-training and inference for multimodal and physical AI...  ...inputs, and action sequencesEvaluate model performance using both benchmark datasets and real-world deployment metricsContributions... 
    Suggested

    Outset Capital

    San Francisco, CA
    2 days ago
  •  ...Research Engineer On Physical AI TeamHedra is a pioneering generative modeling company — first models to market — now building a Physical...  ..., and action sequencesEvaluate model performance using both benchmark datasets and real-world deployment metricsContributions research... 

    HEDRA INC

    San Francisco, CA
    2 days ago
  •  ...results. You’ll develop Taste’s internal research, evaluation design, judges and reward...  ...style). Develop frontier evaluations and benchmarks for subjective domains. Run post-...  ...Thinks like a researcher, moves like an engineer. Creative, scrappy, comfortable in ambiguity... 

    Taste Labs

    San Francisco, CA
    4 days ago
  • $175k - $195k

     ...The role Gauntlet is looking for an AI Research Engineer to help improve how we build, evaluate and use AI agents across the business....  ...capabilities scale across the business. With few established benchmarks in this space, you’ll have the freedom to explore new approaches... 
    Work at office
    Remote work
    Work from home

    Gauntlet

    San Francisco, CA
    1 day ago
  •  ...Ando Research Team MemberAndo is a messaging platform where AI agents take on work alongside their human teammates. We're rebuilding...  ...track, or intervene. Evaluating that judgment means building benchmarks where the ground truth includes silence. Existing agent benchmarks... 
    Work from home

    ASARI S.A de CV

    San Francisco, CA
    3 days ago
  •  ...starting with data centers. About the Role At Watney, research engineers decide what our models learn to do next, and a live fleet of...  ...method to try, and whether a result actually holds up outside a benchmark. What You’ll Do Decide which training method to try... 

    Watney

    San Francisco, CA
    3 days ago
  •  ...Research EngineerDatacurve provides the frontier coding data that powers the world's most...  ...create the world's first autonomous data engine, allowing us to teach the next...  ...of the same domain. You will produce the benchmarks, artifacts, and technical narratives that... 
    Shift work

    Datacurve

    San Francisco, CA
    5 days ago
  •  ...ElevenlabsElevenLabs is an AI research and product company transforming how we interact...  ...work of their lives. We are researchers, engineers, and operators. IOI medalists and ex-founders...  ...training paradigms.Designing rigorous benchmarks and evaluations that tell us whether a... 
    Immediate start
    Remote work

    Eleven Labs

    San Francisco, CA
    1 day ago
  •  ...~ Scaled from $0 to eight figures in ARR ~ Raised $30M from Benchmark, Y Combinator, Paul Graham, and Initialized Team ~ We have...  ...into production systems Stay on top of the latest research across LLMs, information retrieval, and developer tooling Qualifications... 
    Work at office

    Greptile

    San Francisco, CA
    4 days ago
  •  ...troubleshooting have become a massive tax of engineering velocity. Resolve AI is solving this by...  ...workflows end-to-end, balancing research and engineering to create production-ready...  ...and evaluationDesign and execute benchmarks to evaluate AI models, improve performance... 
    Work at office

    Resolve AI

    San Francisco, CA
    2 days ago
  •  ...frontier AI. The Impact You'll Make Our research team is expanding to keep pace with a...  ...edge of the field. As a Research Engineer, you'll take a research direction and run with it – finding the right papers, benchmarks, and prior work, reimplementing what's relevant... 
    Full time

    SuperAnnotate AI

    San Francisco, CA
    28 days ago
  •  ...you want to help change that, apply below.About The RoleAs a Research Engineer, you'll be responsible for building industry-leading...  ...models developed by our customersDesign evaluation protocols and benchmarks that measure model quality, generalization, robustness, and... 
    Work at office

    Assort Health

    San Francisco, CA
    2 days ago
  •  ...that do that coordination directly. What you'll do The research that matters most to us comes straight out of deployment: getting...  ...to end, building agents, training models, and designing the benchmarks, environments, and evals that tell us whether it's working. You... 

    Orin Labs, Inc.

    San Francisco, CA
    2 days ago
  •  ...Research EngineerNew York - Hybrid; San Francisco Bay Area - HybridAbout InvisibleInvisible...  ...organization, focused on evaluation methodology, benchmarks, and RL environments for frontier labs and enterprise clients. As a Research Engineer, you'll work on how frontier models are... 
    Full time
    Work at office
    Local area
    Remote work

    Invisible Technologies Inc. Defunct

    San Francisco, CA
    5 days ago
  • $9.7k - $19k

     ...Research Engineer InternThe Center for AI Safety (CAIS) is a leading research and advocacy organization focused on mitigating societal-...  ...areas such as AI security, machine ethics, AI alignment, and benchmarking AI risks. We will assign you a dedicated mentor throughout... 
    Full time
    Internship
    Local area

    Center for AI Safety

    San Francisco, CA
    3 days ago
  • $170k - $210k

     ...Research Engineer, Coding Agent Optimization Engine San Francisco, in person · Full-time · Entelligence Coding agents ship more code...  ...without flooding agents with noise. Evaluation. Own the benchmark harness of record: replay real agentic tasks headlessly... 
    Full time

    Entelligence

    San Francisco, CA
    2 days ago
  •  ...Research Engineer We believe that software is the foundation of modern civilization - yet vulnerabilities threaten its integrity, security...  ...with strong intuition, experience in model evaluation, and benchmarks. Reinforcement Learning experience is a plus. Your work will... 
    Full time
    Work at office

    DepthFirst

    San Francisco, CA
    5 days ago
  •  ...Role Description This is a Research Engineer role focused on post‑training and reasoning. Full‑time or part‑time; we're open to candidates...  ...experiments on reasoning traces from our expert network Building benchmarks and evals that meaningfully measure clinical and... 
    Full time
    Part time
    Remote work
    Flexible hours

    Cobalt.io

    San Francisco, CA
    1 day ago
  • $150k - $250k

     ...goods, and global social organizations.We research and deploy technologies that power AI-...  ...What We Are Looking ForAt Distyl, Research Engineers build the bridge between frontier AI...  ...why behavior changes, not just whether a benchmark improvesAI Systems Mindset: You understand... 
    Work at office
    3 days per week

    Distyl AI

    San Francisco, CA
    1 day ago
  • $200k - $225k

    About AlembicAlembic is where top engineers are solving marketing's hardest problem: proving...  ...to leverage advanced analyticsDocument research and implementation decisions for...  ...ensures new hires earn at or above real-time benchmarks. Ownership: Our generous equity program... 

    Alembic

    San Francisco, CA
    1 day ago
  •  ...recently led Google Labs.Sierra’s Applied Research team advances the quality, speed, and...  ...to applied ML research or benchmarks.Our valuesTrust: We build trust with our...  ...Francisco, CAEmployment TypeFull timeLocation TypeOn-siteDepartmentEngineeringPlatform Engineering
    Full time
    Local area
    Flexible hours

    Sierra

    San Francisco, CA
    4 days ago
  • $200k - $275k

     ...the world in a positive way. To learn more visit: As a Research Engineer focused on Generative AI Deployment, you will own the path from...  .... Own correctness as part of performance. Build benchmarks and numerical-parity tests focused on simulation realism and... 
    Full time
    Work at office
    Work from home
    Flexible hours

    Waabi

    San Francisco, CA
    6 days ago
  •  ...consequence systems problems where the edge cases matter most. We’re looking for a Research Engineer to help define how we measure and improve model quality. You’ll build the benchmarks, datasets, tooling, and evaluation loops that tell us whether our systems are... 
    Full time

    intrinsic-safety

    San Francisco, CA
    5 days ago
  • $250k - $400k

     ...frontier AI labs to generate high-quality datasets, reinforcement learning environments, and frontier research benchmarks that improve model capabilities in software engineering, enterprise knowledge work, and advanced STEM reasoning. In software engineering, Turing is the... 

    Turing

    San Francisco, CA
    25 days ago
  •  ...stealth team of elite founders and AI researchers, with backgrounds spanning Stanford, OpenAI...  ...we measure is what we want Product engineers, by instrumenting real-user behavior on...  ...to apply Send a link to an eval, benchmark, or measurement system you built — and... 
    Full time
    Relocation package

    agi-inc

    San Francisco, CA
    5 days ago
  •  ...improve AI using their own expertise and standards. Our APEX benchmarks measure how AI performs on economically valuable work, while...  ...We focus on pushing the frontier of world-building, verifier engineering, and more alongside our partners. Your goal will be to automate... 
    Full time
    Local area
    Relocation package

    mercor

    San Francisco, CA
    5 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Research Engineer, Benchmarking. Be the first to apply!