Software Engineer - AI Research & Evaluation (US-based) [Remote]
AuraOne Human Data
- Remote job
Software Engineer - AI Research & Evaluation (US-based) is a remote engineering review track for evaluating production code, debugging traces, and developer-facing AI outputs against real-world correctness standards. Reviewers reproduce failures, write the unit test the model should have written, and explain the fix so the modeling team can target the gap.
Why this role matters
Engineering model quality lives or dies on whether the generated code actually compiles, passes tests, and handles edge cases. AuraOne pairs experienced engineers with the modeling team to grade outputs the way a code reviewer would.
Responsibilities
- Run and reproduce candidate code outputs in a sandboxed environment for Software Engineer - AI Research & Evaluation (US-based) assignments.
- Grade software engineer ai research evaluation engineering review solutions for correctness, style, and edge-case handling.
- Write minimal failing tests that demonstrate the bug a model output missed.
- Compare paired solutions and rank them with a written rationale tied to the rubric.
- Tag failure modes (compile error, runtime crash, off-by-one, security issue) with severity scores.
- Document recurring code-generation failures so the modeling team can target them.
- Calibrate against gold-standard reviews to keep inter-rater agreement above target.
Qualifications
- Strong day-job engineering experience — you can read, run, and debug unfamiliar code for Software Engineer - AI Research & Evaluation (US-based) work.
- Comfort writing concise unit tests that capture a single failure mode.
- Familiarity with a testing framework. pytest, Jest, or JUnit. Go test, RSpec, or whatever you use.
- Clear written reasoning — your review note has to convince another senior engineer.
- Reliable async availability for at least 10 hours per week.
- Prior code-review or technical-interview-grading experience is a plus.
Example tasks
- Reproduce a generated engineering solution to a coding task, run the test suite, and grade it.
- Write the smallest failing test that demonstrates a model's edge-case bug.
- Compare two paired solutions and rank them with a written rationale tied to the rubric.
- Triage a security issue surfaced by a model and document the patch the model should have produced.
Nice to have
- Open-source contributions or a public portfolio that demonstrates production-quality code.
- Experience with the target language's standard tooling, linters, and idiomatic style guides.
- Familiarity with security-review checklists (OWASP, CWE) and AppSec patterns.
Skills
- Code review
- Debugging
- Unit testing
- Software engineering judgment
- Software Engineer AI Research Evaluation engineering review
- Python
- JavaScript
- Java
- C++
- Rust
- Software development
Work model
Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
- ...About Us Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global... ...researchers who specialize in software engineering, logical reasoning,... ...Software Engineering evaluator, you will create...SuggestedContract workFor contractorsFlexible hours
- ...About Us Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global... ...who specialize in software engineering, logical reasoning, STEM... ...Software Engineering evaluator, you will create...SuggestedContract workFor contractorsFlexible hours
$125k - $180k
...power. We design, engineer, manufacture, install... ...a highly capable Software Engineer (Full Stack... ...and implementation AI Experience & Application... ...Understanding of evaluating AI outputs for... ...Location Remote U.S.-based role with occasional... ...to work in the US without requiring any...SuggestedRemote jobTemporary workCasual workWork at office$145.4k - $188.1k
...personalized guidance to AI tools and a best-... ...you, join us. Imagine what we’ll... ...We empower product engineering teams by providing... ...Challenge As a Software Engineer on the ML... ..., deployment, and evaluation tooling. Work hands... ...vary and will be based on various factors...SuggestedRemote jobFull timeSeasonal workLocal area$184k - $287.5k
...for an experienced engineer to join our Planning... ...on metrics and evaluation. In this role you will... ...Autonomous Vehicle software.Build compelling, data... ...) or another Unix based systemAbility and... ...vacancy. NVIDIA uses AI tools in its... ...law.SummaryLocation: US, CA, Santa Clara; US...SuggestedFull timeRemote work$144.7k - $221.4k
...unprecedented scale. Join us to help deliver... ...Organization The Evaluation team builds and... ...feedback for engineering and leadership, and... ...autonomous driving software performance at interfaces... ...leveraging AI-assisted development... ...estimate only. It is based on what a...Full timeWork at officeLocal areaWork from homeRelocationRelocation packageFlexible hours$90 per hour
...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours a week. Pay... ...software engineering experience; you live in the US or Canada and can work as an independent contractor...Hourly payTemporary workFor contractorsLive inRemote workVisa sponsorshipFlexible hours- ...Mars and beyond. Join us on this extraordinary... ...Senior Image Analyst and Software Engineer to provide software... ...support the processing and evaluation of imagery collected... ...applications, web-based applications, websites... ...artificial intelligence (AI) and machine learning...Hourly payContract workWork at officeLocal areaWork from homeMonday to Friday
$57.2k - $108.8k
...America Location: US-MA-CAMBRIDGE-BBN06 10 &... ...Corporation, is the research company behind many revolutionary... ..., BBN's scientists and engineers are working to advance... ...projects, focusing on AI based planning and scheduling... ...and develop prototype software using the latest open-...Full timeTemporary workWork experience placementInternshipWork at officeRemote workRelocationFlexible hours- ...to democratize good AI. We are building... ...'re looking for an engineer to own the data and... ...coordinated activity, graph-based approaches,... ...feedback loops: labeling, evaluation, false-positive... ...workloads (CI, research crawlers, integrations... ..., it just tells us a real person is on...Full timeWork at officeRemote workFlexible hours
$153.2k - $234.1k
...the Autonomous Vehicle software by designing,... ...robust processes for evaluation and validation of our... ...will work closely with AI/ML Engineers, Simulation Engineers,... ...performance of our ML based stack leveraging simulation... ...employment, emailus or call us at (***) ***-****. In...Full timeLocal areaWork from homeFlexible hours$92.1k
...744 Position Title Research Software Engineer Hiring Range Minimum... ...community interactions as part of the evaluation. Demonstrated efficiency with agentic AI systems for software... ...funded by a grant. Continuation is based upon available funds....Full timeRemote workWorldwideMonday to Friday$76.5k - $93.5k
...Research Software Engineer The University of Chicago Research Computing Center... ...storage, visualization, and AI infrastructure, along with scientific... ...local, national, and cloud-based computational resources.... ...standards are met. Evaluates design alternatives for development...Work experience placementWork at officeLocal area$180k - $240k
...thinkers. System builds software to help the world see... ...this role As Senior Research Engineer, you will lead the definition... ...scaling of our LLM evaluation architecture and... ...reports to the Head of AI & Engineering. What you... ...as-a-judge, reference-based metrics, human-in-the-...Full timeRemote workFlexible hours$146.8k - $272.6k
...are seeking a Lead Research Engineer who will bring expertise in AI and ML and is interested... ...you? Come join us! About the... ...Deliver: Applying modern software development... ...research scientists to evaluate, prototype and... ...environment for our office-based roles while...Full timeWork at officeLocal areaRemote workFlexible hours$118.4k - $219.8k
...requiring fiduciary-grade AI. The Thomson LLM... ...release. Our research engineers build the systems that train, evaluate and run the Thomson... ..., please tell us which one fits you... ...experience. ~ Strong software engineering... ...for a position. The base compensation range...Full timeLocal areaFlexible hours- ...About IRIS IRIS Software Group is one of the UK's largest... ...You’ll work on cloud-based applications and data... ...skills across scalable SaaS engineering. We’re also evolving... ...towards becoming an AI-first organisation, you... ...the process, let us know. We’re committed to...Permanent employmentFull timeWork at officeRemote work
$50.64k - $171.85k
...including analytics, AI, intelligent automation... ...scalable, and reliable software. Maintain existing... ...business needs change. Research new technologies and... ...Adhere to and ensure engineering best practices using... ...standards For this US-based position, the base pay...Full timeLocal area- ...About IRIS IRIS Software Group is one of the UK's largest... ...is a hands-on senior engineering role where you’ll solve... ..., build scalable cloud-based software and have... ...technologies, helping us create secure, reliable... ...journey towards becoming an AI-first organisation, you...Permanent employmentFull timeWork at officeRemote work
- ...Health Bask Health is a software platform that enables... ..., product-focused engineering team that values ownership... ...at Bask is AI-first and builder-led.... ...Engineer, AI Quality & Evaluation to build the engineering... ...deterministic checks and model-based evaluations....Full time
$195k - $204.75k
...artificial intelligence (“AI”) solutions, and other... ...is what makes us smarter, drives better... ...Design, implement, and evaluate novel machine learning... ...Collaborate with other research engineers to prototype and validate... ...We do not discriminate based upon race, color, religion...Local areaWorldwideVisa sponsorship- ...you on our team! join us in reshaping the future... ...Date: 08/03/2026The US base salary range for this full... ...posting date.product engineering: bridge the gap between... ...platform engineering: develop software platforms that enable... ...and applying AI/ML to develop scalable...Full timeH1bRemote work
$106k - $176.6k
...Help Build a New AI-Native... ...build will support researchers tackling some of... ...and MLOps AI Evaluation and Benchmarking... ...experts in AI, engineering, computational... ...continuously improve based on real-world... ...-quality software and AI solutions... ...like to contact us regarding the accessibility...Permanent employmentH1bLocal areaVisa sponsorshipWork visaRelocation package2 days per week$197.3k - $225.1k
...Job Description AI Engineer 4 (AI Foundations: Benchmarking, Evaluation, and... ...learning — position us to be at the forefront... ...team of engineers, research scientists, technical... ...deploy, and support AI software components... ...will be prorated based upon the agreed upon...Full timePart timeLocal area$121.2k - $181.8k
...customers, and the world around us. As a Fortune 500 company and a leading AI platform for managing... ...Product Managers and QA Engineers on functional design and... ...development or SaaS (Software as a Service) technologies... ...Statement The annualized base salary ranges for the primary...Work experience placementWork at officeRemote workHome officeFlexible hours$200k - $250k
...About Dropzone AI Dropzone’s mission... ...human security engineer/analyst with an... .... Join us if you want to... ...Principal-level AI Research Engineer to... ...engineering Robust evaluation and... ...semantic, retrieval-based, hybrid) Build... ...~5+ years in software engineering,...Full timeTemporary workWork at officeRemote workWork from homeNight shift$250k - $400k
...Dscout AI Training & Evals Dscout AI... ...Evals points that engine at AI: consented human... ...work. We serve AI researchers, the product... ...check our work. Join us to do rigorous science... ...collaboration, and behavioral evaluation of agents. Co-... ...-as-judge, rubric-based evaluation, and...InternshipRemote workFlexible hours$237.6k - $401.7k
...5Team: Machine Learning and AI, Software and ServicesLocation: CupertinoWeekly... ...As part of the Siri Agentic Evaluation Engineering organization, you will help... ...harnesses that let us evaluate Siri reliably and at... ...maturesPay & BenefitsAt Apple, base pay is one part of our total...Relocation$120.75k - $161k
...generation of Agentic AI products that help millions... .... We’re looking for a Software Engineer to design and build... ...-grade software, not research or experimentation-... ...LLMs, NLP, and agent-based workflowsDevelop backend... ...engineers to integrate, evaluate, and productionize...Work at officeRemote workFlexible hours$103.2k
...opportunity for an AI Engineer to join their team... ...for an AI Research Engineer with strengths... ...AI integration. Evaluate the effectiveness... ...offerings and iterate based on participant... ...collaborative open source software development.... ...material? Tell us about an AI or data...Full timeTemporary workH1bWork at officeRemote workVisa sponsorshipShift workDay shift3 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer - AI Research & Evaluation (US-based) [Remote]. Be the first to apply!
- junior software developer Remote
- senior software design engineer Remote
- software web developer Remote
- senior software engineer Remote
- labview software developer Remote
- software sales engineer Remote
- software developer fintech Remote
- software support engineer Remote
- security software engineer Remote
- software developer night shift Remote





