Evaluations Lead — AI Model Progress & Metrics
Cartesia
Cartesia is looking for an Evaluations Lead to design frameworks that measure AI model interactions, focusing on understanding, naturalness, and adaptability in real-world settings. The role blends research, product, and infrastructure, requiring a candidate with a scientific mindset and technical skills to create impactful evaluation systems. Join our in-person team in San Francisco and dive into pioneering AI methodologies while enjoying competitive benefits. #J-18808-Ljbffr Cartesia
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Evaluations Lead — AI Model Progress & Metrics in San Francisco, CA vacancy
$200k - $400k
Simile is seeking a Member of Technical Staff, Model Evaluations, to develop measurement systems that gauge the accuracy of simulations of human behavior. You will design evaluation metrics and collaborate with modeling teams to ensure models are effective. Ideal candidates...Suggested- A leading AI solutions company in San Francisco is seeking an ML Eval Engineer to design evaluation benchmarks and improve model performance. This role involves working with unstructured enterprise... ...teams. You will develop metrics, conduct evaluations, and contribute...Suggested
$180k - $270k
...building the world’s most trusted AI work companion for professionals... ..., defensible, and automated metrics that researchers and leadership... ...distributed systems, data pipelines, or evaluation harnesses that can run at scale against live model checkpoints. Can deeply partner...SuggestedFull timeWork at officeWorldwide- Software Engineer (Model Evaluation & Benchmarking) About the Role We are hiring Engineers focused on AI Model Evaluation to build the systems that ensure multimodal AI behaves... ...across versions. Develop evaluation metrics for realism, consistency, and performance. Integrate...Suggested
- Anthropic is seeking a Research Lead for the Training Insights team to shape the evaluation of model capabilities. This hands-on leadership role involves developing innovative... ...You will play a crucial role in transforming how AI capabilities are assessed, working...SuggestedRemote work
$176k - $253k
Harper Group, an AI-native commercial insurance company in San Francisco, is looking for a Senior Member of Technical... ...agent quality assessments from vague estimations to concrete metrics, ensuring agents are evaluated, tested, and monitored effectively. Candidates should...$50 - $75 per hour
A leading tech company based in Australia is seeking an AI Model Evaluator on a contract basis. The role involves evaluating AI-generated responses, writing prompts, and providing justifications based on specific criteria. Ideal candidates will hold a Master's degree in...Hourly payContract work$25 per hour
Prolific is seeking AI Training Experts to assist in training and evaluating cutting-edge AI models. The role involves completing tasks such as analyzing and writing annotations, and judging AI performance. Candidates should have professional experience as an AI Trainer...Remote jobHourly payWork from homeFlexible hours- Welocalize is seeking a Data Quality Associate to evaluate AI model outputs and provide structured feedback. This is a full-time, onsite role located in San Francisco. The ideal candidate possesses a Bachelor's degree and has 1-2 years of professional writing experience...Full time
- .... is seeking a Technical Business Development professional in San Francisco, CA. You will work directly with partners on AI infrastructure and model launches, serving as their primary contact and ensuring successful integration. Your role involves significant collaboration...
- Welocalize is seeking a detail-oriented Data Quality Associate in San Francisco. This role involves evaluating AI model outputs, providing feedback, and ensuring data quality. Candidates must have native-level language proficiency and a Bachelor's degree. The position requires...
- ...seeking an innovative Quality Engineer for their AI products. This role blends ops, strategy, and... ...to shape how AI behaves, work with partners in leading labs, and ensure user satisfaction through effective evaluation baselines. Competitive salary and benefits offered...
- A fast-growing AI company seeks a Software Engineer to focus on Model Evaluation & Benchmarking. This role involves building evaluation systems for multimodal AI, ensuring reliable performance. The ideal candidate will possess strong Python programming skills, familiarity...
$146.2k - $261.4k
...Position Description RAND’s Center on AI, Security, and Technology (CAST), part... ...research and policy analysis projects, and leading multidisciplinary teams of policy... ...scientists. Your team will build systems to evaluate how AI models perform across the full attack...Fixed term contractWork experience placementRemote workWork from homeDormont Manufacturing Company
San Francisco, CA2 days ago- ...world. Backed by leading investors like PayPal... ...a future where AI is embedded into how... ...communicate progress, risks, and outcomes... ...each org’s operating model Establish Measurement... ...Define success metrics for each... ...Standardize how we build, evaluate, and scale AI solutions...Work at officeRemote work2 days per week
$90k - $110k
Common Sense Media is seeking an Evaluations Partner Manager in San Francisco, California. This role involves managing the operational execution of the Youth AI Safety Institute's evaluation work, with responsibilities such as coordinating workflow between internal teams...Full time$192k - $250k
...for a builder to help lead our ‘AI for Work’ efforts. Together... ...That means building, evaluating vendors, and... ...Integrate large language models, agentic frameworks, and... ...signals that demonstrate progress, partnering with... ...experiments, interpret metrics, and distinguish meaningful...Hourly payImmediate startFlexible hoursShift work$180k - $260k
A tech-focused investment firm in San Francisco is seeking a Model Behavior Architect to enhance their answer engine by collaborating with... ...teams. Candidates should demonstrate a strong passion for AI, be familiar with Python, and possess a high-level understanding...Full time- About the Role We’re looking for a Model Behavior Architect to help build the company’s AI products and evaluations. You’ll sit within our AI team and collaborate closely... ...Experience designing evaluations, benchmarks, or metrics for AI systems. Strong written and verbal...
$315k
...interpretable, and steerable AI systems. We want AI to be safe... ...the role Anthropic's production models undergo sophisticated post-... ...pipelines for model fine-tuning and evaluation Develop tools to measure and... ...navigate ambiguity and make progress in fast-moving research environments...Work at officeVisa sponsorshipFlexible hours- Xcede is looking for a Member of Technical Staff focused on AI Safety to lead red-teaming efforts and ensure the robustness of next-generation AI systems. The selected candidate will design scalable safety frameworks, partner with researchers to define production safety...
$405k
...interpretable, and steerable AI systems. We want AI to... ...systems, tooling, and evaluation infrastructure that... ...frameworks that measure model capabilities across... ...our research roadmap Lead the design of infrastructure... ...and how we measure progress against them Mentor and...Visa sponsorship$207k - $285k
OpenAI is seeking a Technical Program Manager in San Francisco to lead initiatives that ensure the safety and robustness of its AI models. The role involves collaborating with diverse teams to turn risks into actionable plans. Ideal candidates will have experience in technical...$100 per hour
Turing, based in San Francisco, is seeking finance experts to improve AI model performance. Candidates with a background in capital markets, portfolio management, or related fields are invited to apply. This role offers flexible hours (10-30 hrs/week) and allows you to...Remote jobFlexible hours- ...Designing and running experiments to improve model capabilities Developing training methods, architectures, and evaluation metrics Iterating quickly on research ideas using large... ...on ambitious problems at the intersection of AI, research, and science , we’d love to hear...
$301.75k - $355k
...the only vertically integrated AI infrastructure company built... ...The Senior Director for the Model LifeCycle team will undertake... ...management: versioning, lineage, evaluation, and reproducible fine‑tuning... ...years of industry experience leading and driving impactful projects...Temporary work$85 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco,... ...frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving cloud platforms...Contract workSummer workRemote work- ...Lead Product Manager, Agentic AI As the Lead Product Manager for Agentic AI on Hinge Health's Intelligent... ...outcomes. Own company-level metrics tied to Robin's success: including engagement... ...and iteration. Own evaluation framework: LLM-as-a-judge, golden datasets...Work at officeLocal area3 days per week
Hinge Health
San Francisco, CA2 days ago $50 - $60 per hour
...committed to creating high-quality AI. Enjoy the flexibility of... ...area – teaching AI Assistant models to be a more useful tool for finance... ...’s or PhD (completed or in progress) is strongly preferred.... ...diverse and complex problems and evaluate their outputs Evaluate the...Hourly payContract workFor contractorsWork experience placementRemote workData Annotation
San Francisco, CAmore than 2 months ago$300 per month
...the only vertically integrated AI infrastructure company built... ...research teams to create industry-leading technical content. This means... ...they care about, how they evaluate tools, and what content actually... ...content performance and analyze metrics to understand what's...Temporary workFor contractorsDay shift
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Evaluations Lead — AI Model Progress & Metrics. Be the first to apply!
