Applied AI Researcher, Agent Systems & Evaluation

$193.93k - $352.29k

Full-time

Nuro

Who We Are
Nuro is a self-driving technology company on a mission to make autonomy accessible to all. Founded in 2016, Nuro is building the world’s most scalable driver, combining cutting-edge AI with automotive-grade hardware. Nuro licenses its core technology, the Nuro Driver™, to support a wide range of applications, from robotaxis and commercial fleets to personally owned vehicles. With technology proven over years of self-driving deployments, Nuro gives automakers and mobility platforms a clear path to AVs at commercial scale, empowering a safer, richer, and more connected future.
About The Team
Frontier models are fungible. Any team can rent the same intelligence we can, and the model we build on today will be replaced within a month. What is not fungible is the infrastructure that decides whether an autonomous system's output can be trusted — evaluation, verification, and the discipline to gate on evidence instead of impressions. Nuro has spent a decade building exactly that discipline for a robot that drives on public roads, and this team turns it inward: we build the platform that lets AI agents operate autonomously inside Nuro's own engineering organization, under the same standard of proof we apply to the vehicle.
Our mandate is to amplify the output of every engineer and researcher at Nuro by 100x. Not a better IDE, not a faster build — a change in what a single person can attempt. That number is a target, not a claim, and reaching it depends on one thing above all: autonomous work has to be trustworthy enough to run unattended. So our central ambition is to build the most rigorous closed-loop evaluation system for AI work anywhere. Leverage follows from trust, and trust follows from measurement.
We operate as a startup inside a company that has already shipped a hard thing. Small team, no established playbook, direct access to compute and to the systems we are automating. You will work directly with engineering leadership and the CEO, and the decisions you make will be yours to make rather than yours to implement.
About The Role
Most teams building agents make design decisions by intuition and anecdote. Someone tries a new memory scheme, it feels better, it ships. We think that is the central failure of the field right now, and we are building this team to work the other way: every decision about how our agent systems are constructed should be settled by evidence.
You would not be starting from zero. We already operate a substantial agent system in production, a fleet of agents with an extensive library of skills and plugins, integrated into the tools our engineers use daily, serving real users with real work. So every hypothesis you form can be tested against genuine production traffic from your first month. And the system is now complex enough that intuition has stopped being sufficient to improve it, which is precisely why this role exists.
Everything this team builds is centered on frontier-lab transformer models. We are not inventing architectures. We are extracting the maximum from the best models that exist, and adapting them ourselves in the narrow places where our data gives us an advantage nobody else has.
Your charter has two halves. Make the system perform against real-world data, not public benchmarks or tasks we invented to look good, but our codebase, our infrastructure, and our engineers' actual requests, with all the ambiguity that implies. The gap between benchmark performance and real-task performance is where most agent systems quietly fail. And turn any task into a closed loop : for any workflow an agent takes on, you should be able to say what success looks like, where the evaluation data comes from, how signal is collected, and how results feed the next iteration.
Own the evaluation pipeline end to end. Three stages, and the value is in owning all three:
Eval data collection. Where ground truth comes from. Mining production traces for labeled outcomes, capturing human accept/reject/edit signal as it happens, building task sets that reflect the real distribution of work rather than the tasks that are easy to score, and knowing when a model-based judge is trustworthy and when it is laundering an assumption.
Eval loop construction. Turning a fuzzy objective into a measurement that runs on every change. Noise floors, statistical standards for acceptance, task suites that resist gaming, and experiments designed for production settings where clean randomization is not always available.
Automated hill climbing. The payoff. Once a task has a trustworthy loop, improvement can be searched rather than hand-crafted — prompts, context strategies, tool sets, routing, reasoning budgets, eventually model choice. This only works if the first two stages are sound; done wrong, it optimizes hard against a metric that means nothing. Post-train models on data nobody else has. This role includes hands-on model work: supervised fine-tuning and RL on open-source vision-language models, using the proprietary driving data Nuro has collected across years of real-world autonomous operation. You would have a labeling workforce available to you, which means you can specify the data you need rather than making do with what exists. Very few researchers get to run this loop, form a hypothesis about model behavior, commission the exact data to test it, post-train, and evaluate against real driving performance. The fungibility of frontier models is precisely why this matters: the weights are rentable, the data and the labeling capacity behind them are not.
Read the frontier and convert it into experiments. The research frontier moves weekly, and most of it is noise. You are the person who reads it, separates results that will hold from those that will not replicate, and turns the ones that matter into live experiments against our workloads. This is a deliverable, not background reading — our architecture should reflect the best of what is publicly known within weeks of it becoming known.
Test-time scaling. We believe the largest near-term gains come from how inference compute is spent, not from which model is called: reasoning budgets, sampling and search strategies, verifier-guided selection, when to escalate and when to stop. Each buys accuracy at a price, and the exchange rate differs by task. Mapping where additional inference compute pays and where a verifier beats a bigger model is a core agenda here, and it directly determines how we spend a real budget.
Across all of this, we want someone fluent at four levels: frontier models and how they are built; frontier model evaluation and where measurements lie; frontier AI system evaluation, an agent is a model plus tools, memory, retries, verifiers, and a human who accepts or rejects the output, and the compound system fails in ways none of its parts do; and real-world impact, quantified , with the integrity to say when the numbers do not support the story.
About The Work
You will work in close partnership with the engineer who builds the platform this runs on: they make it run safely at scale, you determine what it should be doing and whether it worked.
What You Might Own in Your First Two Quarters
Establish the evaluation foundation for the agent fleet we already run: eval data sources, task suites, noise floors, and the statistical standard the team uses to accept or reject a change.
Take one high-volume workflow from unmeasured to automatically hill-climbing, end to end, as the template the rest of the system follows.
Run a first post-training experiment on an open-weight VLM against our driving data, and establish whether the result justifies the pipeline.
Put a defensible number on what the platform is worth: which workflows improved, by how much, with what confidence.
About You
The person we are looking for probably has an engineering background and strong research taste, in that order of surprise. Researchers who can ship are common enough to name; researchers who can ship, judge a literature, and care more about scaled impact than authorship are not. If you have been frustrated by how long it takes good research to reach anything real, this role is the correction.
Graduate degree in CS, ML, statistics, or a related field, or equivalent research experience. We care about demonstrated research judgment, not credentials.
Fluent in the current literature and able to judge it. You read papers continuously, can tell a real result from a well-marketed one, and have opinions about which recent directions are overrated.
Deep understanding of how LLMs work — pretraining through the post-training stack, and what actually happens at inference. You reason from mechanism, not just from published numbers.
Firm grasp of the full evaluation pipeline: sourcing eval data, constructing the loop, automating the climb. Having done all three for a real system, rather than one in isolation, is the strongest signal for this role.
Rigorous experimentalist. You design experiments that can fail, you understand variance and power, and you are comfortable saying an intervention didn't work.
Hands-on post-training experience — SFT and RL, ideally on open-weight models — including the data curation and evaluation work required to know whether it actually helped. Vision-language model experience is a strong plus.
A real engineering background. Strong Python, comfortable with production systems and data, able to stand up the infrastructure your own experiment needs.
Direct experience with LLM agent systems — building them, evaluating them, or studying why they fail.
You measure yourself in impact and in weeks. You want your work in front of hundreds of engineers this quarter.
Bonus Points
Published or applied work in agent evaluation, reasoning, test-time compute, RL, or verification.
Experience with multimodal or vision-language models, and with data curation at scale.
Online experimentation in production: A/B testing, causal inference from observational data, offline-to-online correlation.
Experience building evaluation harnesses, task suites, or LLM-as-judge systems, including their failure modes.
Familiarity with autonomous systems, safety cases, or verification-gated deployment.
At Nuro, your base pay is one part of your total compensation package. For this position, the reasonably expected base pay range is between $193,930 and $352,290 for the level at which this job has been scoped. Your base pay will depend on several factors, including your experience, qualifications, education, location, and skills. In the event that you are considered for a different level, a higher or lower pay range would apply. This position is also eligible for an annual performance bonus, equity, and a competitive benefits package.
At Nuro, we celebrate differences and are committed to a diverse workplace that fosters inclusion and psychological safety for all employees. Nuro is proud to be an equal opportunity employer and expressly prohibits any form of workplace discrimination based on race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, veteran status, or any other legally protected characteristics.

Apply

Vacancy posted 1 day ago

Similar jobs that could be interesting for youBased on the Applied AI Researcher, Agent Systems & Evaluation in California vacancy

Applied AI Researcher, System Discovery
$150k - $250k
About Distyl AI Distyl is an applied AI technology company partnering with the... ...social organizations.We research and deploy technologies that... ...into self-constructing systems, the development of the most... ...pipelines, reasoning agents, evaluation harnesses, multimodal integrations...
Suggested
Work at office
3 days per week
Distyl AI
San Francisco, CA
4 days ago
Staff Software Engineer, Systems Infrastructure - Agent Evaluation
$175k - $287k
...the business needs of the team. LinkedIn’s Core AI is building the Evaluation Operating System (EOS), a foundational Agent Evaluation platform that defines how all AI... ...Reasonable Accommodation to search for a job opening, apply for a position, or participate in the interview...
Suggested
For contractors
Work experience placement
Work at office
Flexible hours
Linkedin
Mountain View, CA
4 days ago
Applied AI/ML Researcher - Vice President
...Applied AI/ML ResearcherAs an Applied AI/ML Researcher within the AI4Tech Team at JPMorgan Chase, you will lead technology research and development initiatives... ...and operational resilience.Design, prototype, and evaluate AI-driven tools and frameworks that streamline...
Suggested
Hackajob
Palo Alto, CA
4 days ago
Applied AI Engineer, Codex Core Agent
...Team The Codex Core Agent team builds the kernel... ...better, accelerating research, and making those improvements... ...working across the systems that make Codex... ...We’re looking for applied AI engineers to help bring... ...better real-task data into evaluation and research. Work...
Suggested
Full time
OpenAI
San Francisco, CA
23 hours ago
Applied Researcher I (AI Foundations)
$218.7k - $249.6k
...Overview Applied Researcher I (AI Foundations) Overview: At Capital One, we are creating trustworthy and reliable AI systems, changing banking for good. For years, Capital One has been... ..., from design through training, evaluation, validation, and implementation....
Suggested
Full time
Part time
Local area
Flexible hours
Capital One
San Jose, CA
10 days ago
Applied Researcher I (AI Foundations, LLM Customization, Finetuning, Reinforcement Learning)
$218.7k - $249.6k
...Overview Applied Researcher I (AI Foundations, LLM Customization, Finetuning, Reinforcement Learning... ...creating trustworthy and reliable AI systems, changing banking for good. For years... ..., from design through training, evaluation, validation, and implementation. Engage...
Full time
Part time
Local area
Flexible hours
Capital One
San Jose, CA
1 day ago
Machine Learning Systems Research Engineer, Agent Post-training - Enterprise GenAI
$264.8k - $331k
...AI is becoming vitally important in every... ...for complex agents in enterprises around... ...The Enterprise ML Research Lab works on the front... ...optimize our ML system. Your customer... ...training algorithms and applying them to client use... ...fair and thorough evaluation of all applicants....
Full time
United States Digital Space LLC
San Francisco, CA
1 day ago
Machine Learning Systems Research Engineer, Agent Post-training - Enterprise GenAI
$264.8k - $331k
...Machine Learning Systems Research Engineer, Agent Post-training - Enterprise GenAI AI is becoming vitally important in every function... ...the training algorithms and applying them to client use-cases... ...to ensure a fair and thorough evaluation of all applicants. About Us: At...
Full time
Contract work
For contractors
For subcontractor
Work at office
Scale LLP
San Francisco, CA
23 hours ago
Applied AI Engineer (LLMs, Agents, and Healthcare AI)
...healthcare. We believe AI will fundamentally... ...an exceptional Applied AI Engineer who... ...and agentic systems that help clinicians... ...Generation (RAG), agent frameworks, and healthcare... ...processes. Evaluate and integrate... ...impact over academic research. Are energized...
Human Longevity
South San Francisco, CA
4 days ago
Materials Science Researcher for AI Model Evaluation
$70 - $110 per hour
...Help advance frontier AI systems by bringing rigorous materials science... ...engineering judgment to the evaluation, design, and improvement of... ...work closely with an AI research team to define what high quality... ..., chemical engineering, applied physics, or metallurgy. A master...
Hourly pay
Full time
Live in
Relocation
Relocation package
SaidGig
California
23 hours ago
Applied AI Engineer, Advertising Agents (New Grad)
$135k - $155k
...information powered by advanced AI, recommendation systems, and adtech.... ...recent graduate excited to apply LLMs and Agent technology to real advertising... ..., and offline/online evaluation pipelines that keep optimization... ...; relevant internships, research, coursework, or personal...
Full time
Work experience placement
Internship
Local area
Work from home
NewsBreak
Mountain View, CA
9 days ago
Frontier Agents Engineer (Applied AI)
$180k - $225k
About Scale AIScale AI is the data... ...through frontier AI systems that solve real business... ...production AI agents that automate complex... ..., and research papers emerge every... ...Frontier Agent Engineer (Applied AI), you'll bridge... ...customers to design, evaluate, and deploy...
Full time
Scale AI
San Francisco, CA
6 days ago
Manager, SAP AI & Agent Governance
$114.1k - $268.18k
...currently seeking a Manager, SAP AI & Agent Governance to join our... ...other enterprise functions Apply SAP functional and business process... ...oversight requirements Evaluate risks associated with AI-... ...sdegree in business, information systems, engineering, technology, or...
Full time
H1b
Local area
KPMG
San Francisco, CA
2 days ago
User Researcher, AI Evaluations
$196k - $230k
...is the collaborative AI workspace where teams and agents think together. We're... ...building a business’s system of record to making... ...seeking an experienced UX Researcher to define and scale how we evaluate Notion’s AI-powered... ...and data science can apply consistently.This...
Local area
Shift work
Notion Labs
San Francisco, CA
4 days ago
Member of Technical Staff, Applied AI Engineer, Agents
$300 per month
...Scientific builds and deploys AI scientist agents to accelerate science and... ..., tool integrations, and evaluation frameworks. Develop reusable... ...insights and internal research into product direction – help... ...production experience shipping systems that real users depend on....
Full time
Work at office
Edison Scientific Inc.
San Francisco, CA
23 hours ago
Senior Software Engineer, Full Stack, Applied AI Agent Studio
$174k - $252k
...developers to seamlessly build, evaluate, deploy, monitor, and... ...enterprise-grade generative AI agents.Architect and implement intuitive... ...distributed computing, large-scale system design, networking and data... ...goal oriented team.The Cloud Applied AI (AAI) powers business...
Google
Sunnyvale, CA
5 days ago
Enterprise Agent Systems Engineer
...seeking a highly technical Enterprise Agent Systems Engineer to build and deploy... ...combines strong software engineering with applied AI research judgment. You will work directly with... ...pipelines, post-training workflows, and evaluation infrastructure required for...
MaxIT Consulting - Max Corporate Group
San Francisco, CA
10 days ago
Product Manager, AI Agents
...Manager SuperDial is building AI agents that do real work inside... ...centered, and excited to learn how applied AI can solve real operational... ...post-launch monitoring. Evaluate agent performance across... ...production, inside complex healthcare systems, for customers who need...
SuperDial
Burlingame, CA
2 days ago
Software Engineer, API Agents
...About the Team API Agents builds the shared... ...models into systems that can reliably... ...software engineering, research, finance, healthcare... ...infrastructure, and applied product teams, we... ..., and evaluations. This is software... ...OpenAI OpenAI is an AI research and deployment...
Full time
Internship
OpenAI
San Francisco, CA
23 hours ago
Software Engineer, Agents
$161.3k - $241.9k
...combining frontier agentic AI, an enterprise-grade... ...Software Engineer, Agents, you'll build the systems that make our AI... ...for engineers and researchers who are immersed in... ...in using practical evaluations to drive task completion... ...of agents on applying a standard legal "playbook...
Full time
Harvey
San Francisco, CA
23 hours ago
Software Engineer, Agent - Healthcare
...customer experiences with AI. We are primarily... ...industries—our agents already reach over... ...with the healthcare system. Our healthcare... ...and collaborate with research, product, and... ...We encourage you to apply even if your experience... .... We strive to evaluate all applicants consistently...
Full time
Flexible hours
Sierra Limited
San Francisco, CA
23 hours ago
Software Engineer, Agent - Retail
...experiences with AI. We are primarily... ...The Retail team on Agent Engineering supports... ..., scalable systems running in production... ...collaborate with research, product, and platform... ...frontier of enterprise applied AI. What you'll... .... We strive to evaluate all applicants...
Full time
Flexible hours
Sierra Limited
San Francisco, CA
23 hours ago
Software Engineer, Agent - Healthcare Integrations
...customer experiences with AI. We are primarily an in... ...team Our healthcare agents already reach over 50%... ...with the healthcare system. Our healthcare customers... .... We encourage you to apply even if your experience... .... We strive to evaluate all applicants consistently...
Full time
Flexible hours
Sierra Limited
San Francisco, CA
23 hours ago
Software Engineer, Voice Agents / AI - Deepgram for Restaurants
...trillion-dollar Voice AI economy, providing... ...-grade voice agents at scale. More than... ...effectively AI is applied to deliver results... ...alongside Deepgram’s core research teams. We’ll be... ...backend systems for our automated... ...will continue to evaluate where we can create...
Full time
Home office
Flexible hours
Deepgram
San Francisco, CA
23 hours ago
Senior Software Engineer, Agents - Hiring Sprint
...the data and action layer for AI agents. We give agents fast,... ...context, and take action in the systems they read from, not just observe... ...Interview process: # Apply # Technical Take-Home (Java... ...orchestration, routing, retrieval, evaluation, evidence generation, permissions...
Full time
Work at office
Local area
Flexible hours
Airbyte
San Francisco, CA
23 hours ago
Software Engineer, Agent - Travel & Hospitality
...customer experiences with AI. We are primarily... ...of owning an agent end to end. As a part... ...powerful, scalable systems running in... ...and collaborate with research, product, and platform... ...We encourage you to apply even if your experience... .... We strive to evaluate all applicants consistently...
Full time
Flexible hours
Sierra Limited
San Francisco, CA
23 hours ago
Software Engineer, Agent Architecture
...human customer experiences with AI. We are primarily an in-... ...you'll do Build the core systems that power agents. You’ll work on the Agent SDK... ...using knowledge. Build evaluation systems. You’ll design frameworks... ...team. We encourage you to apply even if your experience...
Full time
Flexible hours
Sierra Limited
San Francisco, CA
23 hours ago
Applied Scientist / Machine Learning Engineer
$311.85k - $370k
...developer of Embodied AI technology. Our advanced... ...safety of automated driving systems. Our vision is to... ...we curate, enrich, and evaluate the real-world driving... ...are hiring at either Applied Scientist or Machine Learning... ...of taking ML from research into production systems...
Full time
Work at office
Work from home
Shift work
Wayve
California
12 days ago
Senior Applied Scientist, Data Processing Agents Science (AWS)
$192.2k - $260k
...Senior Applied Scientist, Data Processing Agents Science Job ID: 10488736 | Amazon Development Center... ...a strong background in agentic AI/ML applied to systems problems in databases, programming... ...to define science features; researching the state of the art and creating...
Local area
Flexible hours
Amazon
Santa Clara, CA
3 days ago
AI Agent Evaluation Engineer - Remote
...Engineers to support an AI training project by... ...learning environments that evaluate AI models on complex... ...deterministic verification systems and golden reference... ...solutions. Evaluate AI agents' ability to reason... ...submission requirements may apply. Availability...
Remote job
For contractors
YO AI Labs
San Jose, CA
17 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Applied AI Researcher, Agent Systems & Evaluation. Be the first to apply!