Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Software Engineer - AI Research & Evaluation (US-based) [Remote]

AuraOne Human Data

Remote
  • Remote job

Software Engineer - AI Research & Evaluation (US-based) is a remote engineering review track for evaluating production code, debugging traces, and developer-facing AI outputs against real-world correctness standards. Reviewers reproduce failures, write the unit test the model should have written, and explain the fix so the modeling team can target the gap.

Why this role matters

Engineering model quality lives or dies on whether the generated code actually compiles, passes tests, and handles edge cases. AuraOne pairs experienced engineers with the modeling team to grade outputs the way a code reviewer would.

Responsibilities

  • Run and reproduce candidate code outputs in a sandboxed environment for Software Engineer - AI Research & Evaluation (US-based) assignments.
  • Grade software engineer ai research evaluation engineering review solutions for correctness, style, and edge-case handling.
  • Write minimal failing tests that demonstrate the bug a model output missed.
  • Compare paired solutions and rank them with a written rationale tied to the rubric.
  • Tag failure modes (compile error, runtime crash, off-by-one, security issue) with severity scores.
  • Document recurring code-generation failures so the modeling team can target them.
  • Calibrate against gold-standard reviews to keep inter-rater agreement above target.

Qualifications

  • Strong day-job engineering experience — you can read, run, and debug unfamiliar code for Software Engineer - AI Research & Evaluation (US-based) work.
  • Comfort writing concise unit tests that capture a single failure mode.
  • Familiarity with a testing framework. pytest, Jest, or JUnit. Go test, RSpec, or whatever you use.
  • Clear written reasoning — your review note has to convince another senior engineer.
  • Reliable async availability for at least 10 hours per week.
  • Prior code-review or technical-interview-grading experience is a plus.

Example tasks

  • Reproduce a generated engineering solution to a coding task, run the test suite, and grade it.
  • Write the smallest failing test that demonstrates a model's edge-case bug.
  • Compare two paired solutions and rank them with a written rationale tied to the rubric.
  • Triage a security issue surfaced by a model and document the patch the model should have produced.

Nice to have

  • Open-source contributions or a public portfolio that demonstrates production-quality code.
  • Experience with the target language's standard tooling, linters, and idiomatic style guides.
  • Familiarity with security-review checklists (OWASP, CWE) and AppSec patterns.

Skills

  • Code review
  • Debugging
  • Unit testing
  • Software engineering judgment
  • Software Engineer AI Research Evaluation engineering review
  • Python
  • JavaScript
  • Java
  • C++
  • Rust
  • Software development

Work model

Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.

Compensation

Hourly rate confirmed after the interview process.

Application process

Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.

Vacancy posted 20 hours ago
Similar jobs that could be interesting for youBased on the Software Engineer - AI Research & Evaluation (US-based) [Remote] in Remote vacancy
  •  ...About Us Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global...  ...researchers who specialize in software engineering, logical reasoning,...  ...Software Engineering evaluator, you will create... 
    Suggested
    Contract work
    For contractors
    Flexible hours

    Turing

    Remote
    12 days ago
  •  ...About Us Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global...  ...who specialize in software engineering, logical reasoning, STEM...  ...Software Engineering evaluator, you will create... 
    Suggested
    Contract work
    For contractors
    Flexible hours

    Turing

    Remote
    12 days ago
  • $125k - $180k

     ...power. We design, engineer, manufacture, install...  ...a highly capable Software Engineer (Full Stack...  ...and implementation AI Experience & Application...  ...Understanding of evaluating AI outputs for...  ...Location Remote U.S.-based role with occasional...  ...to work in the US without requiring any... 
    Suggested
    Remote job
    Temporary work
    Casual work
    Work at office

    Torus

    New York, NY
    2 days ago
  • $145.4k - $188.1k

     ...personalized guidance to AI tools and a best-...  ...you, join us. Imagine what we’ll...  ...We empower product engineering teams by providing...  ...Challenge As a Software Engineer on the ML...  ..., deployment, and evaluation tooling. Work hands...  ...vary and will be based on various factors... 
    Suggested
    Remote job
    Full time
    Seasonal work
    Local area

    thumbtack

    Remote
    8 days ago
  • $184k - $287.5k

     ...for an experienced engineer to join our Planning...  ...on metrics and evaluation. In this role you will...  ...Autonomous Vehicle software.Build compelling, data...  ...) or another Unix based systemAbility and...  ...vacancy. NVIDIA uses AI tools in its...  ...law.SummaryLocation: US, CA, Santa Clara; US... 
    Suggested
    Full time
    Remote work

    Nvidia

    Santa Clara, CA
    1 day ago
  • $144.7k - $221.4k

     ...unprecedented scale. Join us to help deliver...  ...Organization The Evaluation team builds and...  ...feedback for engineering and leadership, and...  ...autonomous driving software performance at interfaces...  ...leveraging AI-assisted development...  ...estimate only. It is based on what a... 
    Full time
    Work at office
    Local area
    Work from home
    Relocation
    Relocation package
    Flexible hours

    General Motors

    Sunnyvale, CA
    2 days ago
  • $90 per hour

     ...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours a week. Pay...  ...software engineering experience; you live in the US or Canada and can work as an independent contractor... 
    Hourly pay
    Temporary work
    For contractors
    Live in
    Remote work
    Visa sponsorship
    Flexible hours

    rustLabs

    New York, NY
    21 days ago
  •  ...Mars and beyond. Join us on this extraordinary...  ...Senior Image Analyst and Software Engineer to provide software...  ...support the processing and evaluation of imagery collected...  ...applications, web-based applications, websites...  ...artificial intelligence (AI) and machine learning... 
    Hourly pay
    Contract work
    Work at office
    Local area
    Work from home
    Monday to Friday

    Amentum

    Huntsville, AL
    2 days ago
  • $57.2k - $108.8k

     ...America Location: US-MA-CAMBRIDGE-BBN06 10 &...  ...Corporation, is the research company behind many revolutionary...  ..., BBN's scientists and engineers are working to advance...  ...projects, focusing on AI based planning and scheduling...  ...and develop prototype software using the latest open-... 
    Full time
    Temporary work
    Work experience placement
    Internship
    Work at office
    Remote work
    Relocation
    Flexible hours

    RTX

    Cambridge, MA
    3 days ago
  •  ...to democratize good AI. We are building...  ...'re looking for an engineer to own the data and...  ...coordinated activity, graph-based approaches,...  ...feedback loops: labeling, evaluation, false-positive...  ...workloads (CI, research crawlers, integrations...  ..., it just tells us a real person is on... 
    Full time
    Work at office
    Remote work
    Flexible hours

    Hugging Face

    Remote
    1 day ago
  • $153.2k - $234.1k

     ...the Autonomous Vehicle software by designing,...  ...robust processes for evaluation and validation of our...  ...will work closely with AI/ML Engineers, Simulation Engineers,...  ...performance of our ML based stack leveraging simulation...  ...employment, emailus or call us at (***) ***-****. In... 
    Full time
    Local area
    Work from home
    Flexible hours

    General Motors

    Sunnyvale, CA
    21 hours ago
  • $92.1k

     ...744 Position Title Research Software Engineer Hiring Range Minimum...  ...community interactions as part of the evaluation. Demonstrated efficiency with agentic AI systems for software...  ...funded by a grant. Continuation is based upon available funds.... 
    Full time
    Remote work
    Worldwide
    Monday to Friday

    Dartmouth College

    Hanover, NH
    1 day ago
  • $76.5k - $93.5k

     ...Research Software Engineer The University of Chicago Research Computing Center...  ...storage, visualization, and AI infrastructure, along with scientific...  ...local, national, and cloud-based computational resources....  ...standards are met. Evaluates design alternatives for development... 
    Work experience placement
    Work at office
    Local area

    University of Chicago

    Chicago, IL
    1 day ago
  • $180k - $240k

     ...thinkers. System builds software to help the world see...  ...this role As Senior Research Engineer, you will lead the definition...  ...scaling of our LLM evaluation architecture and...  ...reports to the Head of AI & Engineering. What you...  ...as-a-judge, reference-based metrics, human-in-the-... 
    Full time
    Remote work
    Flexible hours

    System

    Remote
    3 days ago
  • $146.8k - $272.6k

     ...are seeking a Lead Research Engineer who will bring expertise in AI and ML and is interested...  ...you? Come join us!   About the...  ...Deliver: Applying modern software development...  ...research scientists to evaluate, prototype and...  ...environment for our office-based roles while... 
    Full time
    Work at office
    Local area
    Remote work
    Flexible hours

    Thomson Reuters

    New York, NY
    3 days ago
  • $118.4k - $219.8k

     ...requiring fiduciary-grade AI. The Thomson LLM...  ...release. Our research engineers build the systems that train, evaluate and run the Thomson...  ..., please tell us which one fits you...  ...experience. ~ Strong software engineering...  ...for a position. The base compensation range... 
    Full time
    Local area
    Flexible hours

    Thomson Reuters

    Remote
    3 days ago
  •  ...About IRIS IRIS Software Group is one of the UK's largest...  ...You’ll work on cloud-based applications and data...  ...skills across scalable SaaS engineering. We’re also evolving...  ...towards becoming an AI-first organisation, you...  ...the process, let us know. We’re committed to... 
    Permanent employment
    Full time
    Work at office
    Remote work

    IRIS

    Remote
    10 days ago
  • $50.64k - $171.85k

     ...including analytics, AI, intelligent automation...  ...scalable, and reliable software. Maintain existing...  ...business needs change. Research new technologies and...  ...Adhere to and ensure engineering best practices using...  ...standards For this US-based position, the base pay... 
    Full time
    Local area

    R1 RCM

    Remote
    2 days ago
  •  ...About IRIS IRIS Software Group is one of the UK's largest...  ...is a hands-on senior engineering role where you’ll solve...  ..., build scalable cloud-based software and have...  ...technologies, helping us create secure, reliable...  ...journey towards becoming an AI-first organisation, you... 
    Permanent employment
    Full time
    Work at office
    Remote work

    IRIS

    Remote
    12 days ago
  •  ...Health Bask Health is a software platform that enables...  ..., product-focused engineering team that values ownership...  ...at Bask is AI-first and builder-led....  ...Engineer, AI Quality & Evaluation to build the engineering...  ...deterministic checks and model-based evaluations.... 
    Full time

    Bask Health

    Remote
    9 days ago
  • $195k - $204.75k

     ...artificial intelligence (“AI”) solutions, and other...  ...is what makes us smarter, drives better...  ...Design, implement, and evaluate novel machine learning...  ...Collaborate with other research engineers to prototype and validate...  ...We do not discriminate based upon race, color, religion... 
    Local area
    Worldwide
    Visa sponsorship

    Archer Aviation

    San Jose, CA
    4 days ago
  •  ...you on our team! join us in reshaping the future...  ...Date: 08/03/2026The US base salary range for this full...  ...posting date.product engineering: bridge the gap between...  ...platform engineering: develop software platforms that enable...  ...and applying AI/ML to develop scalable... 
    Full time
    H1b
    Remote work

    BNSF Railway

    Fort Worth, TX
    21 hours ago
  • $106k - $176.6k

     ...Help Build a New AI-Native...  ...build will support researchers tackling some of...  ...and MLOps AI Evaluation and Benchmarking...  ...experts in AI, engineering, computational...  ...continuously improve based on real-world...  ...-quality software and AI solutions...  ...like to contact us regarding the accessibility... 
    Permanent employment
    H1b
    Local area
    Visa sponsorship
    Work visa
    Relocation package
    2 days per week

    Pfizer

    Cambridge, MA
    2 days ago
  • $197.3k - $225.1k

     ...Job Description AI Engineer 4 (AI Foundations: Benchmarking, Evaluation, and...  ...learning — position us to be at the forefront...  ...team of engineers, research scientists, technical...  ...deploy, and support AI software components...  ...will be prorated based upon the agreed upon... 
    Full time
    Part time
    Local area

    Capital One

    McLean, VA
    18 days ago
  • $121.2k - $181.8k

     ...customers, and the world around us. As a Fortune 500 company and a leading AI platform for managing...  ...Product Managers and QA Engineers on functional design and...  ...development or SaaS (Software as a Service) technologies...  ...Statement The annualized base salary ranges for the primary... 
    Work experience placement
    Work at office
    Remote work
    Home office
    Flexible hours

    Workday

    Reston, VA
    2 days ago
  • $200k - $250k

     ...About Dropzone AI Dropzone’s mission...  ...human security engineer/analyst with an...  .... Join us if you want to...  ...Principal-level AI Research Engineer to...  ...engineering Robust evaluation and...  ...semantic, retrieval-based, hybrid) Build...  ...~5+ years in software engineering,... 
    Full time
    Temporary work
    Work at office
    Remote work
    Work from home
    Night shift

    Dropzone AI

    Remote
    9 days ago
  • $250k - $400k

     ...Dscout AI Training & Evals Dscout AI...  ...Evals points that engine at AI: consented human...  ...work. We serve AI researchers, the product...  ...check our work. Join us to do rigorous science...  ...collaboration, and behavioral evaluation of agents. Co-...  ...-as-judge, rubric-based evaluation, and... 
    Internship
    Remote work
    Flexible hours

    dscout

    United States
    1 day ago
  • $237.6k - $401.7k

     ...5Team: Machine Learning and AI, Software and ServicesLocation: CupertinoWeekly...  ...As part of the Siri Agentic Evaluation Engineering organization, you will help...  ...harnesses that let us evaluate Siri reliably and at...  ...maturesPay & BenefitsAt Apple, base pay is one part of our total... 
    Relocation

    Apple

    Cupertino, CA
    2 days ago
  • $120.75k - $161k

     ...generation of Agentic AI products that help millions...  .... We’re looking for a Software Engineer to design and build...  ...-grade software, not research or experimentation-...  ...LLMs, NLP, and agent-based workflowsDevelop backend...  ...engineers to integrate, evaluate, and productionize... 
    Work at office
    Remote work
    Flexible hours

    Eightfold

    Santa Clara, CA
    3 days ago
  • $103.2k

     ...opportunity for an AI Engineer to join their team...  ...for an AI Research Engineer with strengths...  ...AI integration. Evaluate the effectiveness...  ...offerings and iterate based on participant...  ...collaborative open source software development....  ...material? Tell us about an AI or data... 
    Full time
    Temporary work
    H1b
    Work at office
    Remote work
    Visa sponsorship
    Shift work
    Day shift
    3 days per week

    University of Washington

    Seattle, WA
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Software Engineer - AI Research & Evaluation (US-based) [Remote]. Be the first to apply!