LLM Evaluation Scientist — Benchmarks & Failure Analysis
Scale AI, Inc.
Scale AI, Inc. is seeking Research Scientists and Research Engineers to join the GenAI Research Organization. You will focus on evaluation of LLMs and multimodal models, building benchmarks, and diagnosing failure modes to guide improvements. You will collaborate with researchers and engineers, apply post-training techniques like SFT and RLHF, publish results at top conferences, and help shape the next generation of generative AI models, with competitive compensation and equity. #J-18808-Ljbffr Scale AI, Inc.
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the LLM Evaluation Scientist — Benchmarks & Failure Analysis in Seattle, WA vacancy
- ...Scale AI, Inc. is seeking Research Scientists and Research Engineers to join the GenAI Research Organization. You will focus on evaluation of LLMs and multimodal models, building benchmarks, and diagnosing failure modes to guide improvements. You will collaborate with...Suggested
Scale AI
Seattle, WA4 days ago $136k - $184k
...Applied Scientist, Artificial General Intelligence, Foundational... ..., and automated evaluation systems to ensure the... ...performances on benchmarks. The Applied Scientist... ...developing and maintaining LLM-as-a-Judge systems,... ...design, conduct root cause analysis on data quality issues...Suggested16 hoursFlexible hoursAmazon
Bellevue, WA1 day ago- Scale AI is hiring for roles focused on evaluating and benchmarking frontier LLMs and Agents within the GenAI Research Organization. You will develop... ...to shape evaluation-driven AI development and translate failure analyses into strategic input for next-gen models. The...Suggested
- ...developing a proprietary AI-assisted story evaluation system for film, television and literature. I need an experienced Python/LLM evaluation engineer to implement and... ...framework, evaluation protocol and initial global benchmark manifest are already prepared....Suggested
House of Scribes
Seattle, WA1 day ago $142.3k - $263.3k
...Sr. Applied Scientist, AI Evaluation & Quality Systems Seattle, Washington, United... ...(ASE) powers the AI and LLM features behind experiences... ...truth generation, to root-cause analysis that turns disagreements... ...judge design, meta-evaluation, failure mode analysis, and...SuggestedRelocationShift workApple Inc.
Seattle, WA7 hours ago- ...Large Language Model (LLM) service platforms that... ...resume. Design evaluation systems for LLM-based... ...reliability. Build benchmarks and automated judging... ...feedback to identify failure patterns and turn them... ...annotation and statistical analysis, etc. Familiarity with...Internship
ByteDance
Seattle, WA2 days ago $167.1k - $226.1k
...models, and for the benchmarks that determine... ...language model (LLM)-as-a-Judge is the... ...strategy and error analysis- Communicate... ...science, mentor junior scientists, and present... ...of datasets and evaluation benchmarks, and document... ...- Diagnose their failures and improve the...Flexible hoursAmazon
Bellevue, WA3 days ago$142.8k - $193.2k
...Services (TS) is seeking an Applied Scientist to be based in our Seattle... ...* Apply your expertise in LLM models to design, develop,... ...solutions.* Conduct thorough data analysis to gain insights, identify... ...marketplaces.* Continuously explore and evaluate state-of-the-art modeling...Work at officeWorldwideFlexible hoursAmazon
Seattle, WA1 day ago$180.6k - $225.75k
...looking for Research Scientists and Research... ...with expertise in LLM post-training (SFT,... ...reward modeling) and evaluation. This role is on the... ...focus on building benchmarks and diagnosing model failure modes in both text... ...translate failure analysis into technical and...Full time$167.1k - $226.1k
...for an seasoned Applied Scientist to design, build, and... ...detection, and LLM-based reasoning — all... ...inference & root cause analysis:** Build models that decompose... ...(site, service, failure mode, time)- **Dose-response... ...rationales- **LLM evaluation:** Build evaluation frameworks...Flexible hoursAmazon
Seattle, WA5 days ago$142.8k - $193.2k
...science-driven pipeline where LLM-powered classification,... ...process tens of millions of cases, evaluate hundreds of initiatives... ...seeking a world-class Applied Scientist to help define and build the... ...large scale data analyses, model benchmarking, model validation and model implementation...Flexible hoursAmazon
Seattle, WA2 days ago$136k - $184k
..., and highly skilled Applied Scientist with expertise in AI, Agentic... ...Learning, and NLP to help build LLM-powered solutions for Amazon’... ..., RAG, MCP, and automated benchmarking to improve pre-purchase, in-purchase... ...will:Invent, implement, and evaluate state-of-the-art models and...Local areaFlexible hoursAmazon
Seattle, WA3 days ago$142.8k - $193.2k
...looking for an Applied Scientist to help shape the science of evaluating agent-driven builder experiences... ...problem. At Amazon Benchmarking, we are developing new... ...idea generation, business analysis and scientific research,... .../tool-using systems, LLM-based agents, or the evaluation...Work experience placementFlexible hoursAmazon
Seattle, WA1 day ago$142.8k - $193.2k
...'re hiring an Applied Scientist to push the boundaries... ...deployment to metric evaluation.Key job responsibilities... ...scoring, and LLM-based summarization.2.... ...customer impact, not just benchmark scores.4. Ship production... ...experiments and statistical analysis of results- Experience...Flexible hoursAmazon
Seattle, WA1 day ago- ...Data Services, Inc. seeks an Applied Scientist with a strong ML and statistics background... ...framework SOPs, and develop automated evaluation for LLM-based judging. Collaboration with... ...teams is essential to improve Nova benchmark performance. The role involves audits,...
Amazon
Bellevue, WA1 day ago $172.4k - $223.4k
We are looking for an Applied Scientist II to build the AI behind... ...the datasets that train and evaluate these agents, from campaign and... ...optimization to performance analysis and customer insights.We are... ...Experience designing and deploying LLM-based agents in production....Flexible hoursShift workAmazon
Seattle, WA4 days ago- ...serving as the foundation for intelligent e-commerce agents across diverse applications.5. Agent Evaluation, Safety & Compliance:Designing evaluation metrics and benchmarks aligned with real-world business scenarios, ensuring robustness, safety, and compliance under highly...
TikTok
Seattle, WA1 day ago $142.8k - $193.2k
...We are looking for an Applied Scientist II to push the science behind... ...Build the training data and evaluation methods that make these models... ...with sharp experiment analysis: diagnosing where search breaks... ...fixes that resolve stubborn failure pattern to broad modeling changes...WorldwideFlexible hoursAmazon
Seattle, WA5 days ago$142.8k - $193.2k
.... We are seeking a world-class Applied Scientist to help define and build the next generation... ...for large scale data analyses, model benchmarking, model validation and model... ...craft effective prompts and critically evaluate AI-generated outputs in a professional...WorldwideFlexible hoursAmazon
Seattle, WA4 days ago$187k
...everywhere.Senior Machine Learning Scientist - Personalization... ...engineering, model development, evaluation, and iteration, with strong attention... ...methods, statistical analysis, experimentation, and data-driven... ...models, semantic IDs, hybrid LLM-recommender systems, two-...Full timeExpedia
Seattle, WA2 days ago$142.8k - $193.2k
Do you want to join an innovative team of scientists who use state of art AI & ML techniques to help... ...- Innovate with the latest GenAI/LLM/VLM technology to build highly automated solutions for efficient risk evaluation and automated operations- Design, develop and...Flexible hoursAmazon
Seattle, WA1 day ago$142.8k - $193.2k
...alongside world-class scientists to build AI that transforms... ...will design train and evaluate systems that reason... ...combines expertise in LLM reasoning and agentic... ...harnesses and benchmarks to measure agent reliability... ...reliability faithfulness and failure modes in high-stakes...Flexible hoursAmazon
Seattle, WA4 days ago$159.75k - $255.6k
...and innovative Machine Learning Scientists to join our AI team, focusing on AI applications (LLM and Computer Vision) in Cloud,... ...MLLM models for inference and analysis of multimodal data.Explore novel... ...specific hardware architectures.Evaluate the performance of LLM, MLLM,...Work experience placementWork at officeRemote workAxon
Seattle, WA4 days ago$142.8k - $193.2k
...CXBT) is looking for an Applied Scientist to join their team. CXBT's... ...user privacy.- Create robust evaluation metrics and test sets to assess... ...that experience. Through our analysis and insights, we inform... ...as customers of new GenAI and LLM technologies.About the teamCXBT...Flexible hoursAmazon
Seattle, WA2 days ago- ...be you?As a Principal Applied Scientist, you will lead the... ...ML and AI systems, covering LLM fine-tuning, multimodal pipelines... ...frameworks, and metrics for evaluating agentic systems, including offline... ...and cost, latency, and quality analysis.• Serve as a technical leader...Work at officeImmediate startRemote work
UiPath
Bellevue, WA4 days ago $105k - $120k
...decades, we've connected the lessons of past failures with tomorrow's solutions to advise... ...We are currently seeking a Human Factors Scientist for our Human Factors Practice in... ...standardsAssisting with the design and evaluation of safety management practices and safety...Work at officeRelocationFlexible hoursExponent
Bellevue, WA5 days ago$142.8k - $193.2k
...work.We're hiring an Applied Scientist to own causal inference at the... ...bridge between econometric analysis (owned by our economists) and... ...for pricing — Design, train, evaluate, and deploy end-to-end causal... ...offline evaluation framework to benchmark competing optimization...Flexible hoursAmazon
Seattle, WA2 days ago- Title- Adult Salmonid Trapping and Evaluation Biologist Classification- Fish & Wildlife Biologist 3 Job Status- Full-Time / Permanent... ...sampling, and spawning protocols. You’ll coordinate the collection, analysis, and sharing of summer steelhead data with co-managers and...Permanent employmentFull timeContract workTemporary workSummer workCasual workWork at officeLocal areaRemote workMonday to FridayAfternoon shift
- ...Scale AI, Inc. is seeking Research Scientists and Research Engineers to advance LLM post-training techniques for text and multimodal data. You will develop... ...data-driven best practices in data curation and evaluation. This role involves publishing results at top AI conferences...
Scale AI
Seattle, WA3 days ago - ...Immunohematology Laboratory Scientist IV position. Principal responsibilities... ...Perform warm autoantibody evaluations using matched cells and... ...unusual circumstances or failure to follow the written... ...presentations/teaching, problem analysis, negotiation, mentoring, the...Interim roleWork at officeRemote workRelocation packageMonday to FridayShift workWeekend workAfternoon shift
Bloodworks Northwest
Seattle, WA5 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to LLM Evaluation Scientist — Benchmarks & Failure Analysis. Be the first to apply!
Related searches
- research scientist machine learning deep learning Seattle, WA
- image scientist Seattle, WA
- health scientist Seattle, WA
- regulatory scientist Seattle, WA
- scientist biology Seattle, WA
- drug safety scientist Seattle, WA
- lead scientist Seattle, WA
- principal applied scientist Seattle, WA
- machine learning research scientist Seattle, WA
- cell culture scientist Seattle, WA