Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Code Quality Specialist / Research Engineer

$75 - $105 per hour

Braintrust

About the Opportunity

Our client, a leading AI research organization, is looking for engineers and computational researchers to help evaluate how AI systems solve coding tasks.

You’ll investigate whether an unsuccessful result comes from the AI’s code, an issue in the tests, or unclear task requirements. This work combines code reading, debugging and technical judgment to help make AI evaluations more accurate and reliable.

The role is suited to people who enjoy understanding unfamiliar codebases, examining test behavior and explaining technical findings clearly.

What You’ll Do

  • Review code, tests, execution logs and the steps an AI coding agent took while attempting a task.
  • Investigate reported problems in coding benchmarks, including undocumented expectations, missing definitions and overly restrictive test assertions.
  • Determine whether a reported issue is a genuine flaw, a gap that still permits reasonable solutions to pass, or an error in the AI’s implementation.
  • Assess whether an identified issue affected the task’s pass/fail result.
  • Write concise technical explanations supported by relevant excerpts from task instructions, tests and run logs.
  • Diagnose test-harness and environment issues using Docker configuration, dependencies and execution logs.
  • Participate in daily calibration discussions with research scientists to compare findings and align evaluation decisions.

Individual claim investigations are expected to take approximately 30–60 minutes. You’ll use review tools and supporting technical resources to examine the evidence and document a clear conclusion.

What We’re Looking For

  • Either 2+ years of professional experience writing, reviewing and debugging software , or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
  • Strong Python code-reading, debugging and comprehension skills, including familiarity with complex frameworks and test assertions.
  • Reading familiarity with at least one additional language: JavaScript/TypeScript, Go, C++ or Rust.
  • Practical experience with Git and Docker.
  • Familiarity with unit-testing frameworks, such as pytest, unittest, Jest or Mocha, and software build environments.
  • Careful attention to edge cases, ambiguous requirements and discrepancies in test logic.
  • Fluent professional English, with the ability to communicate technical findings clearly in writing and conversation.

Nice to Have

  • Experience with LLM benchmarking, AI evaluation, prompt engineering or coding-agent workflows.
  • Familiarity with benchmarks such as SWE-bench or CodeContests.
  • Experience in AI red teaming, safety or reliability auditing.
  • Previous collaboration with AI or machine-learning research scientists.

Engagement Details

  • Location: United States only.
  • Expected commitment: 35 hours per week.
  • Start: Immediate.

Compensation

$75–$105 USD per hour, depending on relevant skills and experience.

Interview Process

Shortlisted candidates will be invited to a 15-minute AI interview. The hiring team will then review your profile and interview responses to determine next steps, with no additional interview rounds required.

Vacancy posted 2 hours ago
Similar jobs that could be interesting for youBased on the Code Quality Specialist / Research Engineer in San Francisco, CA vacancy
  •  ...Research Engineer, Code RL (Reinforcement Learning) About Anthropic Anthropic’s mission is to create reliable, interpretable, and steerable...  ...engineering excellence, with a deep commitment to building high-quality, scalable systems that push the boundaries of what AI can... 
    Suggested
    Full time
    Work at office
    Visa sponsorship
    Flexible hours

    Anthropic

    San Francisco, CA
    3 days ago
  •  ...company based in San Francisco, California. The Role: As a Research Engineer - Brain Computer Interface Models , you will be a core...  ...communication and collaboration skills, with clean and concise coding practices Strong implementation and prototyping skills, with... 
    Suggested
    Work at office
    Relocation package

    Zyphra

    San Francisco, CA
    6 days ago
  • $122k - $215k

     ...the world in a positive way. To learn more visit: As a Research Engineer, you will be at the forefront of advancing and deploying artificial...  ..., and a commitment to rigorous experimental validation and code quality as we strive to bring research ideas into production and... 
    Suggested
    Full time
    Work at office
    Work from home
    Flexible hours

    Waabi

    San Francisco, CA
    5 days ago
  •  ...Description Bridge the gap between cutting-edge research and production systems. Build and...  ...understanding Optimize research code for production deployment Design and...  ...Bridge communication between research and engineering teams What We're Looking For ~3+ years... 
    Suggested

    Engramme

    San Francisco, CA
    19 days ago
  • $125k - $400k

     ...idler is a frontier data research lab. We build the evals and environments...  ...the last year developing coding evals for top coding models...  ...the role As a Research Engineer at idler, you’ll build the systems...  ...to validate environment quality and fairness Work directly... 
    Suggested
    Work at office
    Relocation package

    Idler

    San Francisco, CA
    6 days ago
  • $200k - $375k

     ...understands complex enterprise systems, the tooling experience developers use every day, and the analysis and planning engine that transforms legacy code. You will work directly with customers on high-leverage projects and help define our culture, our engineering bar,... 
    Visa sponsorship

    FlowGen Labs

    San Francisco, CA
    8 days ago
  •  ...starting with data centers. About the Role At Watney, research engineers decide what our models learn to do next, and a live fleet of...  ...you are one of the people deciding it. Write production ML code (Python, PyTorch, etc.). We’re committed to building a diverse... 

    Watney

    San Francisco, CA
    2 days ago
  • $134k - $235k

     ...positive way. To learn more visit: As a Research Engineer in Neural Rendering, you will create...  ...in reading and developing production quality software, versus only creating prototypes...  ...You write efficient and maintainable code in Python and PyTorch, as well as in at... 
    Full time
    Work at office
    Work from home
    Flexible hours

    Waabi

    San Francisco, CA
    29 days ago
  •  ...months, we’ve gone from our first line of code to deploying real humanoid capabilities. The role As our VLA/WAM Research Engineer, you'll develop the foundation models...  ...learning, RL, reward/value-guided learning, and specialist-to-generalist distillation. Continual... 
    Work at office
    Relocation

    Flexion Robotics

    San Francisco, CA
    15 days ago
  •  ...Research EngineerWe believe that software is the foundation of modern civilization - yet...  ...Infrastructure.We're seeking an experienced Research Engineer to join our effort in building and...  ...learning environments to train security coding agentsWork on a Product that Solves a... 
    Full time
    Work at office

    DepthFirst

    San Francisco, CA
    4 days ago
  • $9.7k - $19k

     ...Research Engineer InternThe Center for AI Safety (CAIS) is a leading research and advocacy organization focused on mitigating societal-scale...  ...defend their impact. You will plan and run experiments, conduct code reviews, and work in a small team to create a publication... 
    Full time
    Internship
    Local area

    Center for AI Safety

    San Francisco, CA
    2 days ago
  •  ...Research EngineerNew York - Hybrid; San Francisco Bay Area - HybridAbout...  ...clients. As a Research Engineer, you'll work on how frontier...  ...design the approach and write the code that proves it out.Depending...  ...We NeedProduction-quality code written daily; this is a... 
    Full time
    Work at office
    Local area
    Remote work

    Invisible Technologies Inc. Defunct

    San Francisco, CA
    2 days ago
  •  ...Greptile is an AI code reviewer that catches bugs and anti-patterns in pull requests with complete context of the codebase. Hundreds...  ...prototypes into production systems Stay on top of the latest research across LLMs, information retrieval, and developer tooling Qualifications... 
    Work at office

    Greptile

    San Francisco, CA
    3 days ago
  •  ...Research EngineerSesame believes in a future where computers are lifelike...  ...product goals. As a Research engineer at Sesame, you are a research...  ...and live evals — surface quality regressions before users or...  ...comfortable writing production code across the stack.Hands-on... 
    Full time
    Contract work
    Shift work

    SESAME

    San Francisco, CA
    2 days ago
  •  ...Outset invented a better way to do research: AI-powered user interviews. Global brands...  ...sectional levels of analysis. The Research Engineer reports to the Head Scientist and builds...  ...from stored inputs, and that research code stays in step with the shipped product. The... 

    Outset

    San Francisco, CA
    4 days ago
  •  ...fleet of affordable, durable robots. We're looking for a research engineer to help us train increasingly capable models across...  ...system is doing Are comfortable moving between research code and production-quality systems Like running experiments, getting surprising... 

    Pantograph

    San Francisco, CA
    1 day ago
  • $250k - $400k

     ...frontier AI labs to generate high-quality datasets, reinforcement...  ...learning environments, and frontier research benchmarks that improve model capabilities in software engineering, enterprise knowledge work,...  ...for Software Engineering / coding agents  # UI-Environments... 

    Turing

    San Francisco, CA
    24 days ago
  • $200k - $275k

     ...positive way. To learn more visit: As a Research Engineer focused on Generative AI Deployment,...  ...research to production. Turn research code and checkpoints into versioned,...  ...experience reading and developing production-quality software at scale, not only prototypes.... 
    Full time
    Work at office
    Work from home
    Flexible hours

    Waabi

    San Francisco, CA
    5 days ago
  •  ...We are seeking a highly technical Forward Deployed AI Engineer – Agents & Research Systems to build and deploy advanced agent systems for large...  ...solutions. Ability to personally write and own production code. Strong understanding of evaluation, agent behavior, and... 

    MaxIT Consulting - Max Corporate Group

    San Francisco, CA
    7 days ago
  • $315k

     ...Research Engineer, Interpretability San Francisco, CA About Anthropic Anthropic’s mission is to create reliable, interpretable, and steerable...  ...candidates may also have experience with: Designing a code base so that anyone can quickly code experiments, launch them... 
    Full time
    Temporary work
    Work at office
    Remote work
    Visa sponsorship
    Flexible hours

    Anthropic

    San Francisco, CA
    12 days ago
  •  ...Job Description Job Description Research Engineer, Post-Training Data Location: San Fran...  ...opportunity to join a team building high-quality post-training data, reinforcement...  ...Debug ML pipelines, understand unfamiliar code quickly, and solve open-ended technical... 
    Full time

    NxT Level

    San Francisco, CA
    1 day ago
  •  ...and we're seeking the team of researchers that will build them....  ...We're looking for a Research Engineer to sit between our data and our...  ...from, and write the research code (libraries, dataloaders, evaluation...  ...Comprehensive, high-quality health, dental, and vision insurance... 
    Full time
    Work at office
    Relocation package

    constellation

    San Francisco, CA
    4 days ago
  •  ...to the work of semiconductor engineering. Our goal is to help...  ...design cycles. This work brings research, model training, and hardware...  ...reproducible. We value strong coding fundamentals, careful experimental...  ...the team’s hardware specialists. In this role, you will:... 
    Full time

    OpenAI

    San Francisco, CA
    12 days ago
  •  ...about as well. If you want to have your research come into contact with reality, Ando is the...  ...are not Ando's moat; execution, product quality, and customer experience are. Research here...  ...over credentials. Work samples or code that demonstrate the skills: eval frameworks... 
    Full time
    Work from home

    ando

    San Francisco, CA
    4 days ago
  •  ...is changing that. Tutor is a robotics research and deployment company, building generally...  ...States. We’re a team of researchers, engineers, technicians, operators, and sellers who...  ...Strong Python, and maintainable, tested code that runs on real hardware. Hands-on experience... 
    Hourly pay
    Full time
    Work at office

    Tutor Intelligence

    San Francisco, CA
    16 days ago
  •  ...way amounts to creating AGI. The economic engine we’re creating on the e-commerce side,...  ...to Have significant software, ML, or research engineering experience Command a deep,...  ...efforts. Quickly and iteratively ship code that is correct, robust, scalable, and maintainable... 
    Full time
    Work at office
    Local area
    Remote work
    Relocation
    Shift work
    3 days per week

    onton

    San Francisco, CA
    4 days ago
  •  ...for long-horizon tasks like autonomous research, and the full-stack platform our own research...  ...learning. We’re hiring research engineers and scientists across RL algorithms, scaling...  ...Develop and test RL methods for reasoning, coding, and long-horizon tasks, including... 
    Full time
    Work at office
    Remote work
    Visa sponsorship
    Relocation package
    Flexible hours

    PrimeIntellect

    San Francisco, CA
    4 days ago
  •  ...Devin, the first AI software engineer. Our team is extremely talent...  ...join us. Role Mission Research moves at the speed of the...  ...hundreds of thousands of concurrent coding agent rollouts in VM...  ...and evaluation. Ensure data quality, reproducibility, and efficiency... 
    Full time
    Local area

    Cognition

    San Francisco, CA
    8 days ago
  • $175k - $260k

     ...the role Were looking for a Senior Research Infrastructure Engineer to accelerate research on training...  ...in experimentation and turn research code into reliable reusable systems. Youll...  ...behavior agent failure modes and data quality issues Make experimentation faster... 
    Full time
    Work at office
    Remote work
    Relocation
    Visa sponsorship

    HUD

    San Francisco, CA
    6 days ago
  •  ...build. We look for world-class engineers who have both a vision for...  ...down and produce exceptional code. Responsibilities Develop...  ...Stay current with relevant research and apply it pragmatically...  ...waiting for at the speed and quality the market demands (or if you... 
    Full time
    For contractors

    Bobyard

    San Francisco, CA
    6 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Code Quality Specialist / Research Engineer. Be the first to apply!