Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Research Engineer - Benchmarking

Full-time

mercor

About Mercor

Mercor’s mission is to organize human intelligence for the AI economy. We build the infrastructure that enables frontier AI labs and enterprises to train, evaluate, and improve AI on real-world professional work. More than five million experts have joined our network, and we pay out over $5 million daily for their knowledge and expertise.

We built our training and evaluation infrastructure alongside the world’s leading AI labs. Now, we’re bringing those capabilities to enterprises across industries, enabling them to train, evaluate, and improve AI using their own expertise and standards. Our APEX benchmarks measure how AI performs on economically valuable work, while our enterprise evaluations and post-training capabilities turn those insights into measurable improvements across real-world workflows.

Our work sits at the intersection of frontier AI, human expertise, and real-world application. We’re solving problems few companies have encountered, with the opportunity to influence what the world’s most advanced AI systems become capable of doing. We’re looking for people who bring different perspectives, take ownership, exercise sound judgment, and hold themselves to an exceptionally high standard. You’ll have the opportunity to do consequential work alongside talented colleagues who challenge one another, collaborate openly, and share a commitment to excellence.

Mercor is profitable and recently raised a $750 million Series D at a $20 billion valuation. We work in person five days a week in San Francisco, New York, and London.

About the Role

As a Research Engineer at Mercor, you’ll work at the intersection of engineering and applied AI research. You’ll own benchmarking pipelines, evaluation systems, and failure analysis workflows that directly inform how we train and improve frontier language models.
Your work will define how we measure tool use, agentic behavior, and real-world reasoning. You’ll design and run evals, build rubrics and scorers, and turn failure analysis into actionable improvements for post-training, RLVR, and data pipelines.

What You’ll Do

  • Benchmarking: Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning; ensure benchmarks scale with training and stay aligned with product and research goals.

  • Evaluation systems: Build and operate LLM evaluation systems end-to-end runs, scoring, dashboards, and reporting, so researchers and applied AI teams can track model performance and compare runs at scale.

  • Failure analysis: Run systematic failure analysis on model outputs (e.g., wrong tool use, reasoning errors, safety/alignment issues); categorize failure modes, quantify prevalence, and feed findings into reward design, data curation, and benchmark design.

  • Rubrics and evaluators: Create and refine rubrics, automated evaluators, and scoring frameworks that drive training and evaluation decisions; balance rigor with scalability (human vs. model-as-judge, calibration, agreement).

  • Data quality and usability: Quantify data usability, quality, and impact on key benchmarks; use evals and failure analysis to guide data generation, augmentation, and curation.

  • Cross-team collaboration: Work with AI researchers, applied AI teams, and data producers to align evals with training objectives and to prioritize benchmarks and failure analyses that matter most.

  • Ownership in a fast-paced environment: Operate in a high-iteration research setting with strong ownership of benchmarks, evals, and failure-analysis workflows.

What We’re Looking For

  • Strong applied research background, with focus on model evaluation, benchmarking, and/or failure analysis.

  • Strong coding skills and hands-on experience with ML models and evaluation code.

  • Solid grasp of data structures, algorithms, and backend systems.

  • Comfort with APIs, SQL/NoSQL, and cloud platforms for running and storing eval results.

  • Ability to reason about model behavior, experimental results, and data quality from evals and failure analyses.

  • Excitement to work in person in San Francisco five days a week in a high-intensity, high-ownership environment.

Nice To Have

  • Industry experience on a post-training or evaluation/benchmarking team (highest priority).

  • Publications at top-tier venues (NeurIPS, ICML, ACL), especially in evaluation or benchmarking.

  • Experience building or running LLM evaluations, benchmarks, or failure-analysis pipelines.

  • Experience with synthetic data generation, rubric design, or RL-style workflows that use evals for reward shaping.

  • Work samples or code (e.g., eval frameworks, benchmark suites, failure-analysis reports or tooling) that demonstrate relevant skills/

 

Benefits:

  • Generous equity grant vested over 4 years

  • Health, dental, and vision insurance

  • 401(k) with company match

  • Unlimited paid vacation and sick days

  • Parental leave for birthing and non-birthing parents

  • Free mental health services

  • $200 monthly wellness reimbursement

  • $1.5K monthly meal stipend

  • $200 monthly laundry reimbursement

  • Free Equinox membership

  • Up to $15K relocation bonus

Mercor is an Equal Opportunity Employer and is committed to providing equal opportunity to all employees and applicants. Mercor does not discriminate based on actual or perceived race, color, creed, religion, national origin, ancestry, citizenship status, age, sex or gender (including pregnancy, childbirth, related medical conditions and lactation), gender identity or gender expression, sexual orientation, marital status, military service and veteran status, disability, protected medical condition as defined by California or local law, genetic information, or any other characteristic protected by applicable federal, California, or local laws and ordinances. Further, Mercor prohibits discrimination, harassment, and retaliation based on any combination of protected characteristics. This Policy applies to all areas of employment, including recruitment, hiring, training, promotion, compensation, benefits, discipline, termination, and all other privileges, terms, and conditions of employment. This Policy and the law prohibit employment discrimination against any employee or applicant based on any legally protected status.

Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Research Engineer - Benchmarking in San Francisco, CA vacancy
  •  ...improve AI using their own expertise and standards. Our APEX benchmarks measure how AI performs on economically valuable work, while...  ...Francisco, New York, and London. About the Role As a Research Engineer at Mercor, you’ll work at the intersection of engineering and... 
    Suggested
    Internship
    Local area
    Relocation package

    mercor

    San Francisco, CA
    3 days ago
  •  ...frontier AI. The Impact You'll Make Our research team is expanding to keep pace with a...  ...edge of the field. As a Research Engineer, you'll take a research direction and run with it – finding the right papers, benchmarks, and prior work, reimplementing what's relevant... 
    Suggested
    Full time

    SuperAnnotate AI

    San Francisco, CA
    27 days ago
  • $130k - $225k

     ...Job Description Job Description About the Role Join a research and engineering team building privacy and anonymization systems for...  ...transformations suited to the data and its intended use. Develop and benchmark detection methods that combine rules, statistical models,... 
    Suggested
    Visa sponsorship

    Clera

    San Francisco, CA
    5 days ago
  •  ...starting with data centers. About the Role At Watney, research engineers decide what our models learn to do next, and a live fleet of...  ...method to try, and whether a result actually holds up outside a benchmark. What You’ll Do Decide which training method to try... 
    Suggested

    Watney

    San Francisco, CA
    2 days ago
  • $9.7k - $19k

     ...Research Engineer InternThe Center for AI Safety (CAIS) is a leading research and advocacy organization focused on mitigating societal-...  ...areas such as AI security, machine ethics, AI alignment, and benchmarking AI risks. We will assign you a dedicated mentor throughout... 
    Suggested
    Full time
    Internship
    Local area

    Center for AI Safety

    San Francisco, CA
    2 days ago
  •  ...Research EngineerNew York - Hybrid; San Francisco Bay Area - HybridAbout InvisibleInvisible...  ...organization, focused on evaluation methodology, benchmarks, and RL environments for frontier labs and enterprise clients. As a Research Engineer, you'll work on how frontier models are... 
    Full time
    Work at office
    Local area
    Remote work

    Invisible Technologies Inc. Defunct

    San Francisco, CA
    2 days ago
  •  ...you want to help change that, apply below.About The RoleAs a Research Engineer, you'll be responsible for building industry-leading...  ...models developed by our customersDesign evaluation protocols and benchmarks that measure model quality, generalization, robustness, and... 
    Work at office

    Assort Health

    San Francisco, CA
    6 days ago
  •  ...Research EngineerWe believe that software is the foundation of modern civilization - yet...  ...Infrastructure.We're seeking an experienced Research Engineer to join our effort in building and...  ..., experience in model evaluation, and benchmarks. Reinforcement Learning experience is a... 
    Full time
    Work at office

    DepthFirst

    San Francisco, CA
    4 days ago
  •  ...troubleshooting have become a massive tax of engineering velocity. Resolve AI is solving this by...  ...workflows end-to-end, balancing research and engineering to create production-ready...  ...and evaluationDesign and execute benchmarks to evaluate AI models, improve performance... 
    Work at office

    Resolve AI

    San Francisco, CA
    6 days ago
  •  ...~ Scaled from $0 to eight figures in ARR ~ Raised $30M from Benchmark, Y Combinator, Paul Graham, and Initialized Team ~ We have...  ...into production systems Stay on top of the latest research across LLMs, information retrieval, and developer tooling Qualifications... 
    Work at office

    Greptile

    San Francisco, CA
    3 days ago
  • $250k - $400k

     ...frontier AI labs to generate high-quality datasets, reinforcement learning environments, and frontier research benchmarks that improve model capabilities in software engineering, enterprise knowledge work, and advanced STEM reasoning. In software engineering, Turing is the... 

    Turing

    San Francisco, CA
    24 days ago
  • $200k - $275k

     ...the world in a positive way. To learn more visit: As a Research Engineer focused on Generative AI Deployment, you will own the path from...  .... Own correctness as part of performance. Build benchmarks and numerical-parity tests focused on simulation realism and... 
    Full time
    Work at office
    Work from home
    Flexible hours

    Waabi

    San Francisco, CA
    5 days ago
  •  ...stealth team of elite founders and AI researchers, with backgrounds spanning Stanford, OpenAI...  ...we measure is what we want Product engineers, by instrumenting real-user behavior on...  ...to apply Send a link to an eval, benchmark, or measurement system you built — and... 
    Full time
    Relocation package

    agi-inc

    San Francisco, CA
    4 days ago
  •  ...consequence systems problems where the edge cases matter most. We’re looking for a Research Engineer to help define how we measure and improve model quality. You’ll build the benchmarks, datasets, tooling, and evaluation loops that tell us whether our systems are... 
    Full time

    intrinsic-safety

    San Francisco, CA
    4 days ago
  •  ...things we have to think about as well. If you want to have your research come into contact with reality, Ando is the place for it....  ...track, or intervene. Evaluating that judgment means building benchmarks where the ground truth includes silence . Existing agent benchmarks... 
    Full time
    Work from home

    ando

    San Francisco, CA
    4 days ago
  • $75 - $105 per hour

     ...About the Opportunity Our client, a leading AI research organization, is looking for engineers and computational researchers to help evaluate how AI systems...  ...a task. Investigate reported problems in coding benchmarks, including undocumented expectations, missing... 
    Hourly pay
    Immediate start

    Braintrust

    San Francisco, CA
    2 hours ago
  • $150 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,...  ...Responsibilities Develop challenging machine learning engineering tasks for a long-horizon AI research... 
    Contract work
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    29 days ago
  • $200k - $225k

    About AlembicAlembic is where top engineers are solving marketing's hardest problem: proving...  ...to leverage advanced analyticsDocument research and implementation decisions for...  ...ensures new hires earn at or above real-time benchmarks. Ownership: Our generous equity program... 

    Alembic

    San Francisco, CA
    5 days ago
  •  ...recently led Google Labs.Sierra’s Applied Research team advances the quality, speed, and...  ...to applied ML research or benchmarks.Our valuesTrust: We build trust with our...  ...Francisco, CAEmployment TypeFull timeLocation TypeOn-siteDepartmentEngineeringPlatform Engineering
    Full time
    Local area
    Flexible hours

    Sierra

    San Francisco, CA
    3 days ago
  • $150k - $250k

     ...goods, and global social organizations.We research and deploy technologies that power AI-...  ...What We Are Looking ForAt Distyl, Research Engineers build the bridge between frontier AI...  ...why behavior changes, not just whether a benchmark improvesAI Systems Mindset: You understand... 
    Work at office
    3 days per week

    Distyl AI

    San Francisco, CA
    5 days ago
  •  ...improve AI using their own expertise and standards. Our APEX benchmarks measure how AI performs on economically valuable work, while...  ..., New York, and London. About the Role As a Research Engineer at Mercor, you’ll work at the intersection of engineering and... 
    Internship
    Remote work
    Relocation package

    mercor

    San Francisco, CA
    4 days ago
  •  ...the AI brains behind The Path’s AI Therapist. Combine research, data science, and engineering to create models, orchestration, and evaluation systems...  ...and user outcomes, rather than chasing generic benchmarks. Can look at evals, transcripts, and metrics and quickly... 

    Mental, LLC

    San Francisco, CA
    5 days ago
  •  ...cyberthreats, this is your opportunity. The Role As an AI Research Engineer at Alaris, you'll help design and deploy autonomous agents...  ...evaluation pipelines and metrics frameworks for agent benchmarking Implement advanced reinforcement learning algorithms to... 

    Alaris Security, Inc.

    San Francisco, CA
    3 days ago
  •  ...LanceDB AI advances at the speed of its research, and research moves at the speed of its...  .... About the Role As the AI research engineer at LanceDB, you'll work with the research...  ...replicable experiments, that may include benchmarking Provide core content and work cross-... 

    LanceDB

    San Francisco, CA
    5 days ago
  • $180k - $290k

     ...computer use. Our FLUX models top public benchmarks and run in production in some of the...  ...platforms. We're a small, fast growing research lab competing with the largest...  ...cleanly separated into “research” or “engineering.” A promising architecture only matters... 
    Full time
    Work at office
    2 days per week

    Black Forest Labs

    San Francisco, CA
    8 days ago
  •  ...Job Description Job Description We are Genmo, a research lab developing the world’s most sophisticated video world models to understand...  .... About the Role We're seeking an exceptional Software Engineer to join our research team in advancing the frontiers of visual... 
    Work at office

    Genmo

    San Francisco, CA
    23 days ago
  • $122k - $215k

     ...diverse, innovative and collaborative candidates who want to impact the world in a positive way. To learn more visit: As a Research Engineer, you will be at the forefront of advancing and deploying artificial intelligence algorithms for our self-driving vehicles. You... 
    Full time
    Work at office
    Work from home
    Flexible hours

    Waabi

    San Francisco, CA
    5 days ago
  • $125k - $400k

     ...Description Job Description About idler idler is a frontier data research lab. We build the evals and environments that the world's...  ...some of our work here: About the role As a Research Engineer at idler, you’ll build the systems and infrastructure that our... 
    Work at office
    Relocation package

    Idler

    San Francisco, CA
    6 days ago
  • $140k - $250k

     ...This is the company's top hiring priority and a genuinely hard research problem. Because data flows through a decentralized marketplace...  ...scale is the single biggest bottleneck to growth. As a Research Engineer, you will build the automated systems that verify and assure... 
    Full time
    Work at office

    talentpluto

    San Francisco, CA
    7 days ago
  •  ...Description Job Description Bridge the gap between cutting-edge research and production systems. Build and implement research...  ...technical documentation Bridge communication between research and engineering teams What We're Looking For ~3+ years of experience in... 

    Engramme

    San Francisco, CA
    19 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Research Engineer - Benchmarking. Be the first to apply!