Research Engineer, Benchmarking
Refresh
- Research Engineer, BenchmarkingEngineeringSan FranciscoFull-timeBuild the benchmarks frontier labs use to measure real-world coding and computer-use capability. Translate expert workflows into rigorous, verifiable evaluations, run them against frontier models, and publish numbers that hold up under adversarial scrutiny.Across every role at Refresh: you're willing to ship full-stack work on our core stack (Vercel, Supabase, Render) when it's needed, and you're comfortable with reinforcement learning and supervised fine-tuning at a high level.
- J-18808-Ljbffr Refresh
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Research Engineer, Benchmarking in San Francisco, CA vacancy
- Refresh is seeking a Research Engineer focused on benchmarking, to build rigorous evaluations of real-world coding and computer-use capabilities. You will translate expert workflows into verifiable tests and run benchmarks against frontier models. In this role you’ll work...Suggested
- ...improve AI using their own expertise and standards. Our APEX benchmarks measure how AI performs on economically valuable work, while... ...Francisco, New York, and London. About the Role As a Research Engineer at Mercor, you’ll work at the intersection of engineering and...SuggestedFull timeWork at officeLocal areaRelocation package
- ...capabilities Build large-scale pipelines for synthetic data generation, filtering, and quality control Develop evaluations and benchmarks that measure meaningful model improvements Why join: Work at the frontier of multimodal foundation models and world models...Suggested
$250k - $400k
...Research Engineer – Post-Training, Agents, Evals Fundamental AI Research Institute San Francisco Bay Area, USA (hybrid) $250,000 –... ...including developing agentic and evaluation harnesses Develop benchmarks and evaluations for reasoning and agentic capabilities...SuggestedRelocation package- ...Hedra Job OpportunityHedra is a research and infrastructure lab advancing post-training and inference for multimodal and physical AI... ...inputs, and action sequencesEvaluate model performance using both benchmark datasets and real-world deployment metricsContributions...Suggested
- ...Research Engineer On Physical AI TeamHedra is a pioneering generative modeling company — first models to market — now building a Physical... ..., and action sequencesEvaluate model performance using both benchmark datasets and real-world deployment metricsContributions research...
- ...results. You’ll develop Taste’s internal research, evaluation design, judges and reward... ...style). Develop frontier evaluations and benchmarks for subjective domains. Run post-... ...Thinks like a researcher, moves like an engineer. Creative, scrappy, comfortable in ambiguity...
$175k - $195k
...The role Gauntlet is looking for an AI Research Engineer to help improve how we build, evaluate and use AI agents across the business.... ...capabilities scale across the business. With few established benchmarks in this space, you’ll have the freedom to explore new approaches...Work at officeRemote workWork from home- ...Ando Research Team MemberAndo is a messaging platform where AI agents take on work alongside their human teammates. We're rebuilding... ...track, or intervene. Evaluating that judgment means building benchmarks where the ground truth includes silence. Existing agent benchmarks...Work from home
- ...starting with data centers. About the Role At Watney, research engineers decide what our models learn to do next, and a live fleet of... ...method to try, and whether a result actually holds up outside a benchmark. What You’ll Do Decide which training method to try...
- ...Research EngineerDatacurve provides the frontier coding data that powers the world's most... ...create the world's first autonomous data engine, allowing us to teach the next... ...of the same domain. You will produce the benchmarks, artifacts, and technical narratives that...Shift work
- ...ElevenlabsElevenLabs is an AI research and product company transforming how we interact... ...work of their lives. We are researchers, engineers, and operators. IOI medalists and ex-founders... ...training paradigms.Designing rigorous benchmarks and evaluations that tell us whether a...Immediate startRemote work
- ...~ Scaled from $0 to eight figures in ARR ~ Raised $30M from Benchmark, Y Combinator, Paul Graham, and Initialized Team ~ We have... ...into production systems Stay on top of the latest research across LLMs, information retrieval, and developer tooling Qualifications...Work at office
- ...troubleshooting have become a massive tax of engineering velocity. Resolve AI is solving this by... ...workflows end-to-end, balancing research and engineering to create production-ready... ...and evaluationDesign and execute benchmarks to evaluate AI models, improve performance...Work at office
- ...frontier AI. The Impact You'll Make Our research team is expanding to keep pace with a... ...edge of the field. As a Research Engineer, you'll take a research direction and run with it – finding the right papers, benchmarks, and prior work, reimplementing what's relevant...Full time
- ...you want to help change that, apply below.About The RoleAs a Research Engineer, you'll be responsible for building industry-leading... ...models developed by our customersDesign evaluation protocols and benchmarks that measure model quality, generalization, robustness, and...Work at office
- ...that do that coordination directly. What you'll do The research that matters most to us comes straight out of deployment: getting... ...to end, building agents, training models, and designing the benchmarks, environments, and evals that tell us whether it's working. You...
- ...Research EngineerNew York - Hybrid; San Francisco Bay Area - HybridAbout InvisibleInvisible... ...organization, focused on evaluation methodology, benchmarks, and RL environments for frontier labs and enterprise clients. As a Research Engineer, you'll work on how frontier models are...Full timeWork at officeLocal areaRemote work
$9.7k - $19k
...Research Engineer InternThe Center for AI Safety (CAIS) is a leading research and advocacy organization focused on mitigating societal-... ...areas such as AI security, machine ethics, AI alignment, and benchmarking AI risks. We will assign you a dedicated mentor throughout...Full timeInternshipLocal area$170k - $210k
...Research Engineer, Coding Agent Optimization Engine San Francisco, in person · Full-time · Entelligence Coding agents ship more code... ...without flooding agents with noise. Evaluation. Own the benchmark harness of record: replay real agentic tasks headlessly...Full time- ...Research Engineer We believe that software is the foundation of modern civilization - yet vulnerabilities threaten its integrity, security... ...with strong intuition, experience in model evaluation, and benchmarks. Reinforcement Learning experience is a plus. Your work will...Full timeWork at office
- ...Role Description This is a Research Engineer role focused on post‑training and reasoning. Full‑time or part‑time; we're open to candidates... ...experiments on reasoning traces from our expert network Building benchmarks and evals that meaningfully measure clinical and...Full timePart timeRemote workFlexible hours
$150k - $250k
...goods, and global social organizations.We research and deploy technologies that power AI-... ...What We Are Looking ForAt Distyl, Research Engineers build the bridge between frontier AI... ...why behavior changes, not just whether a benchmark improvesAI Systems Mindset: You understand...Work at office3 days per week$200k - $225k
About AlembicAlembic is where top engineers are solving marketing's hardest problem: proving... ...to leverage advanced analyticsDocument research and implementation decisions for... ...ensures new hires earn at or above real-time benchmarks. Ownership: Our generous equity program...- ...recently led Google Labs.Sierra’s Applied Research team advances the quality, speed, and... ...to applied ML research or benchmarks.Our valuesTrust: We build trust with our... ...Francisco, CAEmployment TypeFull timeLocation TypeOn-siteDepartmentEngineeringPlatform EngineeringFull timeLocal areaFlexible hours
$200k - $275k
...the world in a positive way. To learn more visit: As a Research Engineer focused on Generative AI Deployment, you will own the path from... .... Own correctness as part of performance. Build benchmarks and numerical-parity tests focused on simulation realism and...Full timeWork at officeWork from homeFlexible hours- ...consequence systems problems where the edge cases matter most. We’re looking for a Research Engineer to help define how we measure and improve model quality. You’ll build the benchmarks, datasets, tooling, and evaluation loops that tell us whether our systems are...Full time
$250k - $400k
...frontier AI labs to generate high-quality datasets, reinforcement learning environments, and frontier research benchmarks that improve model capabilities in software engineering, enterprise knowledge work, and advanced STEM reasoning. In software engineering, Turing is the...- ...stealth team of elite founders and AI researchers, with backgrounds spanning Stanford, OpenAI... ...we measure is what we want Product engineers, by instrumenting real-user behavior on... ...to apply Send a link to an eval, benchmark, or measurement system you built — and...Full timeRelocation package
- ...improve AI using their own expertise and standards. Our APEX benchmarks measure how AI performs on economically valuable work, while... ...We focus on pushing the frontier of world-building, verifier engineering, and more alongside our partners. Your goal will be to automate...Full timeLocal areaRelocation package
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Research Engineer, Benchmarking. Be the first to apply!
Related searches
- junior machine learning research engineer San Francisco, CA
- senior research engineer San Francisco, CA
- research engineer San Francisco, CA
- deep learning research engineer San Francisco, CA
- research programmer San Francisco, CA
- research assistant engineering San Francisco, CA
- research software engineer San Francisco, CA
- ai research engineer San Francisco, CA
- research assistantship San Francisco, CA
- research writer San Francisco, CA




