AI Evaluation Engineer
$130k - $220kAurora Jobs ApS
This position is at Aurora Jobs ApS
The selection process will be fully managed by Aurora Jobs ApS.
--
**Sede:** San Francisco, CA - Full-time, On-site **Salario:** $130K - $220K **Equity:** $60K - $120K options/year **Sponsorizzazione Visto:** Disponibile caso per caso **About Artificial Analysis** Artificial Analysis is the leading independent AI benchmarking and insights company. They help engineers, enterprises, investors, media, and policymakers understand AI capabilities and make critical decisions about AI strategy. Their benchmarks are trusted by hundreds of thousands of users and are the go-to reference for leading AI labs including OpenAI, Google, Meta, NVIDIA, and Anthropic. Their work is also cited by the Wall Street Journal, Bloomberg, Financial Times, and The Economist. Their benchmarks do not just measure frontier AI — they actively shape it. The company was founded in 2024, has raised $3.5M, and has a team of 30+ employees across San Francisco, Sydney, and Melbourne, with plans to double by mid-year. They are backed by Nat Friedman, Daniel Gross, Andrew Ng, Adam D’Angelo, Clem Delangue, and other leading figures in AI. **About Aurora** Aurora helps top engineers discover opportunities at some of the most ambitious startups worldwide. We work closely with companies to identify exceptional talent and match them with roles where they can have real impact. We are currently helping Artificial Analysis expand their team. **Position: Member of Technical Staff** **What This Role Actually Is** This role is best described as an AI Evaluation Engineer / Technical Generalist. It is not a traditional software engineering role and not a pure ML research position. It combines AI benchmarking, technical product work, strategic analysis, customer-facing collaboration, and startup execution. **About the Role** Artificial Analysis is hiring Members of Technical Staff to design the evaluations that define how AI is measured, produce analysis that shapes how companies and the broader market understand AI, and work directly with the leading AI labs and enterprise customers who rely on their insights. You will develop new benchmarking methodologies, manage relationships with some of the most important AI labs and enterprise customers in the world, and help drive the product direction of the platform. The success bar for this role is becoming a world expert in modern AI technologies. This is a unique combination of product, research, technical, and client-facing work suited to highly driven technical generalists who thrive on breadth and ownership. You will work directly with the founders and across the full team, with outsized impact on both the company and product. **What You’ll Do** * Design, structure, and execute projects evaluating AI systems and technologies. Develop new benchmarking methodologies and datasets that improve evaluation capabilities. * Build reports, strategic analysis, and data visualizations that help enterprises shape their AI strategy. * Collaborate with leading AI companies to benchmark agentic AI applications, foundation models, and hardware systems. * Identify opportunities to improve the benchmarking platform and work with developers to ship those improvements. * Operate in an AI-native workflow using cutting-edge AI tools to create leverage and maintain speed. * Contribute to company strategy and lead major initiatives that support the company’s goal of becoming the leading AI benchmarking platform. **Requirements** * 2–10 years of experience in MBB consulting or technical roles such as TPM, ML Engineer, or similar. * Strong analytical ability, technical fluency in modern AI systems, excellent communication skills, ownership mindset, and comfort operating in a fast-moving startup environment. **Tech Stack** * Python **Why Join** This is a rare chance to get a front-row seat to frontier AI and evaluate cutting-edge systems before they are publicly released. You’ll work directly with the world’s top AI labs while helping define what best-in-class AI performance actually means. The company is financially strong, fast-growing, highly connected, and offers strong ownership and visibility from day one.
--
[#J-ONE] [#J-MIN]
[#J-INTERNAL]
- ...Forward Deployed AI Engineer The opportunity We are looking for a Forward Deployed AI Engineer to serve as the critical bridge... ...domains. You understand the unique data challenges and evaluation paradigms of biological modelling. You have contributed to...SuggestedShift work
- ...exceptional problem solvers, combining expert engineering talent with deep experience in National Security. We build AI for the most important problem in defense -... ...data exploration and feature engineering to evaluation and production monitoring. Data & Systems...Suggested
- ...Founding Engineers: Build the AI That Builds AI Full-time • Founding Team • Engineering • On-site (India or US) What We're Building... ...on repetitive loops of data curation, prompt refinement, evaluation, and fine-tuning-turning what should be a month of work into...SuggestedFull time
$150k - $250k
...Max AI – Stripe for Healthcare Max AI is the World’s first human-free, fully-autonomous... ...for over 10 years. And our Head of Engineering was one of the earliest engineers at Figma... ...Responsibilities Build, experiment, and evaluate AI agents and ML models in the NLP domain...Suggested$120k - $180k
...About the job Founding AI Engineer Job Title: Founding AI Engineer Location: San Francisco, CA (On-site, 5 days/week)... ...monitoring conversational agents. Develop scalable pipelines and evaluation methods for LLMs and conversational AI. Conduct research...SuggestedInternshipH1bVisa sponsorship$150k - $250k
...Distyl AI Job Posting Distyl is an applied AI technology company partnering with... ...business. What We Are Looking For AI Engineers build and operate production AI systems that... ...and continuously improve systems through evaluation, feedback, integration, and production...Work at office3 days per week$180k - $250k
...AI Engineer Location: San Francisco, CA Company Stage of Funding: Seed Stage AI Startup ($6M Raised) Office Type: Onsite... ...application layer of the platform-building production AI agents, evaluation systems, and LLM-powered features that customers rely on...H1bWork at officeVisa sponsorship- ...AI Engineer Opportunity at Goodfin Goodfin is an AI-native investment platform giving accredited investors access to pre-IPO and alternative... ...in the real world. Implement and improve RAG pipelines, evaluations, and reliability mechanisms. Monitor live AI systems,...
- ...demonstrated track record of turning ambitious AI ideas into products people actually use;... ...move effortlessly between research and engineering; have shipped something extraordinary,... ...of your work by regularly running evaluations and tests. Analyze production traces to...Relocation package
$180k - $300k
...About the role You'll own the core AI systems that power Gamma: the models, prompts... ...scale. Your job is to elevate quality, evaluate new frontier models, and push into new... ...across our AI stack. You'll work closely with engineering and product to ship improvements that...Full timeWork at officeImmediate startWork from home$120k - $200k
...Founding AI Engineer San Francisco | Engineering | In office | Full-time Mason AI has cracked a research problem that went unsolved for... ...run, run those experiments, and then analyze them. You’ll be evaluated on your speed at running thoughtful experiments. We do reference...Full timeWork at office$263.9k - $301.2k
...we are creating responsible and reliable AI systems, changing banking for good. For... ...to build world-class applied science and engineering teams to deliver our industry leading... ...inference, similarity search, guardrails, model evaluation, experimentation, governance, and...Full timePart timeLocal area- ...Accellor is an AI-native services firm purpose-built for the post-ChatGPT era. Free... ...outcomes through advanced AI, data, and engineering capabilities. Our mission is to operationalize... ...: Design, build, train, and evaluate AI/ML models using Python, TensorFlow, PyTorch...
- ...GenAI Engineer Location - (Hybrid from Altimetrik office) Altimetrik's office locations... ...function calling, versioning with evals) Evaluation & observability (ground truth setup,... ...providers (Azure OpenAI, AWS Bedrock, Vertex AI) Workflow orchestration (Airflow,...Work at office
- ...Agentic Ai Engineer At Kargo, our mission is to build a connective tissue between the physical world of freight and the digital ecosystem... ..., and internal systems. Improve agent reliability through evaluation, testing, monitoring, and guardrails. Optimize agent...Local area
$180k - $250k
...AI Engineer We're hiring a full-time AI Engineer to own the prompts, agents, evals, and pipelines behind user-facing features that ship... ...turn them into working prompts, agents, and pipelines. You'll evaluate them rigorously, iterate until they're production-ready, and...Full timeWork at officeRemote workRelocation- ...AI Engineer at Fathom As an AI Engineer at Fathom, you'll be hands-on with LLMs, prototyping and refining our features. Your work... ...experience with the latest available models. Improved or created new evaluations for our existing features. By 90 days, you will have...Work at officeRemote work3 days per week
$800 per month
...autonomy. Vooma's mission is to build the AI orchestration platform for America's $800... ...you choose to join us, will be an AI Engineer on our founding team. You're the... ...world tasks Designing and implementing evaluation frameworks to rigorously measure model performance...Shift work- ...you "get stuff done" end-to-end. You use AI to work smarter and solve problems... ...tooling across the spectrum: from prompt engineering and in-context learning to fine-tuned models... ...systems. ~ Understanding of monitoring, evaluation, and iteration in production AI systems....Worldwide
$150k - $250k
...A cutting-edge healthcare tech company is seeking an AI Engineer to build and evaluate AI agents, particularly in medical billing. The role requires a strong background in ML, NLP, and Python packages such as TensorFlow and PyTorch, with at least 6 years of industry experience...$160k - $207k
...Semgrep gets smarter as you build, with AI that learns your context to cut false positives... ...at semgrep.dev. About the role As an AI engineer, you’ll apply LLM technologies throughout... ...of AI-powered solutions and rigorously evaluate the efficacy of different prompts and...Currently hiringLocal areaRemote workWeekend work3 days per week- ...the data and infrastructure layer for taste. Our goal is to end AI slop. To make AI feel right, not just be correct. We raised $18.... .... The work skews backend (synthetic data, embeddings, crawling, evaluation systems) but you’ll also ship front‑end tooling and gamified experiences...
$70.35k - $235.1k
...thinking services company at the forefront of AI-native innovation. We partner with... ...next-generation, agent-powered workflows engineered to scale in real-world settings. Our... ...policy-based routing, tool invocation, evaluation harnesses, and lifecycle observability....Work experience placementLive inWork at officeLocal area$225k - $280k
...We’re partnering with a startup building AI powered systems to modernize a complex legacy... .... We are looking for a Senior AI Engineer open to working 3-4 days/week in San Francisco... ...pipelines, memory/context systems, evaluation layers, and workflow automation Building...Local area3 days per week- ...Problem We're building an agentic AI caregiver advocate that lives in your text... ...team, and needs over time. The AI engineering challenge: build an autonomous agent that... ...RAG pipelines, tool-use frameworks, and evaluation infrastructure that make it effective, reliable...Immediate startRemote workFlexible hours
$124.9k - $228.9k
...to meet you. Whatwe do You'll join a software engineering team building internal AI solutions that support teams across The Trade Desk. We... ...through automated testing, monitoring, and data-driven evaluation of success criteria, user adoption, and operational efficiency...Full timeTemporary workLocal areaWorldwide$194k - $239k
...the properties they love. With proprietary AI built on over a decade of real property... ...validated the direction. Now we're looking for engineers who want to help build the systems that... ...production AI systems evaluation and reliability engineering and enterprise...Full timeFor contractorsWork at officeLocal areaFlexible hours- ...AI/ML Software Engineer At Gallatin, we are rebuilding logistics infrastructure for the national security missions of the United States... ...full AI/ML lifecycle—from defining and building models to evaluating and deploying large scale ML pipelines and real-time inference...Local area
$190k - $286k
...Amplitude is the leading AI analytics platform, helping over 4,700 customers-including... ...Role We're looking for a Senior AI Engineer to help advance Amplitude's AI products and... ...Amplitude's AI systems: orchestration, evaluation, retrieval, context management, and integrations...Home officeFlexible hours$210k - $270k
...teams across the United States to help them hire. Senior AI Engineer Location - San Francisco, CA (Hybrid / Onsite) Compensation... ..., building the systems responsible for model accuracy, evaluation, inference, monitoring, and production AI workflows. This...Remote workVisa sponsorship
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Evaluation Engineer. Be the first to apply!
- ai engineer remote San Francisco, CA
- ai developer San Francisco, CA
- machine learning ai engineer San Francisco, CA
- ai ml engineer San Francisco, CA
- ai engineer San Francisco, CA
- senior ai engineer San Francisco, CA
- ai research engineer San Francisco, CA
- ai prompt engineer San Francisco, CA
- ai solutions engineer
- ai agent engineer

