LLM Benchmark Engineer — Elevate Model Evaluation
Vals AI, Inc.
Vals AI, Inc. is seeking engineers to own and expand our leaderboards, evaluating new LLM models against benchmarks across law, tax, coding, finance, and more. You will analyze error modes, compare strengths and weaknesses, and collaborate with a communications team to share results. The role emphasizes strong Python engineering, sprint-based work, and close collaboration with both open-source and foundation model labs. Relocation and in-person SF work are supported as needed. #J-18808-Ljbffr Vals AI, Inc.
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the LLM Benchmark Engineer — Elevate Model Evaluation in San Francisco, CA vacancy
- ...’s voice-native foundation models are accessed through cloud... ...looking for a Senior Software Engineer - Model Evaluation & AI Systems to join the... ...to-speech, and increasingly LLM- and multimodal-powered systems... ...grounded in Research benchmarks, and build the monitoring that...Suggested
- Mercor is hiring PhD and Master's scientists to author AI evaluation tasks (Sci Code). You will author original, executable research problems that today's frontier models cannot solve. The role emphasizes material sourcing, prompt design, and robust grading criteria against...SuggestedPart timeImmediate start
- ...Member Of Technical Staff (Language Model Evaluations) Location: San Francisco (... ...Analysis is the leading independent AI benchmarking company. We support labs, engineers and enterprises to understand AI... ...Deep familiarity with the LLM evaluation landscape: the major benchmarks...Suggested
$70 - $85 per hour
...We are building and evaluating state-of-the-art large language models (LLMs) and are looking for experienced software engineers to join our evaluation and... ...outputs against rigorous benchmarks, identify failure modes,... ...problems). Evaluate LLM outputs for code generation...SuggestedContract workPart timeRemote workBraintrust
San Francisco, CA21 days ago- ...working with frontier AI labs to accelerate model development through high-quality training data, evaluations, and engineering talent. About the Role We are... ...and engineering teams to design coding benchmarks and improve LLM evaluation methodologies. Required Qualifications...SuggestedContract workPart timeFor contractors
Turing
San Francisco, CA21 days ago - Deepgram is seeking a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech, audio, and multilingual models before they reach customers. You will define evaluation methodologies, build scalable pipelines...
- ...We are looking for strong engineers to join our team and own the... ...responsible for testing new models against our benchmarks as they're released;... ...analyze error modes of models, evaluate their strengths and weaknesses... ...You’ll Do Evaluate new LLM model releases across the Vals...Work experience placementRelocationRelocation packageShift work
- Obsidian is seeking a candid evaluator to assess the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train frontier AI labs. You will review... ...concrete, rubric-based assessments to guide model training and #J-18808-Ljbffr Obsidian
- Obsidian is seeking an independent contractor for a remote role focused on benchmark dataset projects that evaluate AI models on visual document understanding in the Energy/Utilities domain. The role requires authoring complex tasks and providing objective rubrics to assess...Remote jobFor contractorsFlexible hours
$1,000 per month
...million dollar trial? Today's models can contribute to these... ...decisions are made, and how to evaluate complex, multiplayer systems... ...the role You'll join our engineering team and report to an Engineering... ...there isn't an established benchmark, distinguish evidence from noise...Work at officeLocal areaRemote workHome officeFlexible hoursElicit
Oakland, CA4 days ago$170k - $245k
...250+ million raised to date.About the roleAs a Distributed LLM Inference Engineer, you will help systems and optimizations that push the boundaries... ...Inc. is an Equal Opportunity Employer. Candidates are evaluated without regard to age, race, color, religion, sex,...Work at officeAnyscale
San Francisco, CA4 days ago- .... About the role You’ll take a new model and get it running — correctly — on our ASIC... .... Take a new architecture — a frontier LLM, an MoE, a multimodal model — from... ...agent loops or LLM-driven tooling that did real engineering work, not demos. #J-18808-Ljbffr...Live in
General Compute
San Francisco, CA1 day ago $217.8k - $326.8k
...Platform Infrastructure Engineering, and more —... ...safeguard customer data, elevate security... ...workflows that pair LLM reasoning with deterministic... ...Build the evals, benchmarks, and calibration... ...discussions, evaluating the security and reliability... ...about where model judgment belongs...Local areaRemote workFlexible hoursBlock
San Francisco, CA1 day ago$136.44k - $265.11k
...structured data, and run AI agents and models directly in their workflows. Over 200,0... ...our work here.You’ll build the datasets, evaluations, and systems that help close that gap.... ...work at the intersection of software engineering, biology, and frontier AI: finding tasks...Work at officeLocal areaMonday to FridayShift workBenchling
San Francisco, CA2 days ago$208k - $312k
...role: We are looking for a Software Engineer, Trust & Safety, to design, build, and... ...detection systems, and large language model (LLM) tooling that disrupt abuse at internet... ...prevention, and build the datasets and evaluation pipelines that measure them Build and...Work at officeWork from homeWorldwideFlexible hours3 days per weekVercel Inc.
San Francisco, CA1 day ago- ...roadmap and go-to-market for an enterprise product that evaluates which foundation model best fits a customer's use case. You will run customer discovery... ..., shape pricing and packaging, and collaborate with engineering, design and GTM teams to drive activation and recurring...
- GMI Cloud, Inc is hiring world-class Machine Learning Engineers to advance LLM inference optimization leveraging GPUs and cutting-edge techniques... ...to open-source projects, and help define recipes and benchmarks for industry-leading inference performance. #J-18808-Ljbffr...
- Clera in San Francisco, CA, is seeking a founding engineer to build evaluation environments and benchmarks for AI agents performing company work. This role sits at the intersection of agent evaluation and production readiness, with direct influence on how agent competence...Visa sponsorship
$300 per month
...strategies, and be part of a high-performing team that believes in each other, come build with us at Crusoe.The RoleAs an Engineer for Load Integration & Model Development at Crusoe, you will play a critical role in expanding our energy infrastructure and integrating our...Temporary workWork at officeCrusoe
San Francisco, CA2 days ago$45 - $60 per hour
Mercor maintains a standing humanities pool for AI training and evaluation. This listing is not a single opening but a continuously open... ...experience, you can join. Work involves writing questions, grading model outputs, and flagging misinterpretations, with #J-18808-Ljbffr...Hourly pay- ...of AI to bring cutting-edge models into production. We're growing... ...and help build the platform engineers turn to ship AI products.... ...) and Large Language Model (LLM) engineering. You will not just... .... RESPONSIBILITIES Benchmarking : Evaluate, run and automate standard LLM...Flexible hours
BaseTen
San Francisco, CA1 day ago - Build AI in San Francisco seeks an Evals Lead to benchmark video and world-model capabilities with and without our data. You must deeply understand evals: what an eval is allowed to claim, contamination, leakage, construct validity, and whether a number actually corresponds...Weekend work
$150k - $250k
...shape of AI: System One Models. Based in San... ...Jev. While others chase benchmarks and academic puzzles,... ...quietly rethinking the LLM stack from first principles... ...mid-2024, we’ve been engineering the foundation for what... ...interfaces, datasets, and evaluation tooling you build,...Work at office- Benchling is building an AI-powered platform for biotech R&D in San Francisco. You will design datasets, evaluate frontier AI models, and partner with scientists to translate complex biology into rigorous tasks that models can learn from. This role sits at the intersection...
- ...open-weight foundation models with a mission to make... ...includes researchers, engineers, and operators from... ...advanced AI systems are evaluated, stress-tested, and safely... ...dynamic safety benchmarks that evolve alongside... ...required: Deep expertise in LLM safety, red teaming,...
- Turing Enterprises, Inc. in San Francisco seeks a Software Engineering evaluator to create cutting-edge datasets for training and benchmarking large language models. You will curate code examples and provide precise Rust solutions, plus other languages, while refining AI...Contract work
$220k - $320k
...specialized language models for companies that need... ...distillation, training, evaluation, and planet-scale... ...funded ten-person team of engineers who work in-person in... ...vLLM, SGLang, TensorRT-LLM) and underlying... ...Build tooling and benchmarks to measure and track inference...Work at officeInference
San Francisco, CA4 days ago- ...Program Manager to lead initiatives that test the safety and robustness of its models through creative experimentation and structured evaluation. You will coordinate across research and engineering teams to transform ambiguous risks into concrete programs and influence...Work at officeRelocation package
- ...Reflection is building model evals and safety from the ground up, and this RPM will be... ...center of that effort. You will shape the evaluation frameworks, operational infrastructure... ...ecosystem, coordinating across research and engineering to accelerate frontier model progress...
- ...customize, and build on. We build open models that let anyone control their intelligence... ...makes it real. That means defining the evaluation frameworks, building the operational infrastructure... ...lifecycle. Partner with research and engineering leads across pre-training, mid-training,...Work at officeVisa sponsorship
Sierra Ventures
San Francisco, CA5 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to LLM Benchmark Engineer — Elevate Model Evaluation. Be the first to apply!
