AI Benchmark Quality Engineer for Task Evaluation
Obsidian
Obsidian is seeking a candid evaluator to assess the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train frontier AI labs. You will review repository-level tasks, reference patches, test harnesses, and grading integrity, and provide rubric-based written feedback. Responsibilities include identifying gaps in tooling, ensuring task reproducibility across environments, and delivering concrete, rubric-based assessments to guide model training and #J-18808-Ljbffr Obsidian
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the AI Benchmark Quality Engineer for Task Evaluation in San Francisco, CA vacancy
$172.55k - $203k
...Senior Software Test Engineer Hybrid - San Francisco... ...readiness, activity, and sleep quality by using their Oura... ...Improvement Evaluate and introduce new tools... ...Explore the use of AI-driven testing approaches... ...products meet performance benchmarks. This is a Hybrid...SuggestedWork at officeLocal areaFlexible hoursOura
San Francisco, CA2 days ago$115k - $188k
...We AreHP IQ is HP’s new AI innovation lab.... ...diverse, world-class team—engineers, designers, researchers... ...solutions that make complex tasks feel effortless,... ...experiences with exceptional quality and developer efficiency... ...pipelines, or model evaluation workflows.Understanding...SuggestedFull timeTemporary workLocal areaFlexible hours$80k - $100k
...'s first accessible X-Ray CT scanner for engineers. Our easy-to-use scanner and cloud-based... ...unprecedented visibility into products, as well as AI-driven tools that highlight problems and... ...a QA Test Engineer to help ensure the quality and reliability of our cutting-edge,...SuggestedFull timeWork at officeWork visaFlexible hoursLumafield
San Francisco, CA10 hours ago$152.9k - $210.2k
...experiences and devices.As a Staff QA Engineer in the Dolby Cinema QA group,... ...risk assessment, and release-quality reporting across complex... ...and help advance practical AI-assisted QA tools that... ...including calibration, image evaluation, video playback, color processing...SuggestedFull timeWork at officeLocal areaWorldwide$113.4k - $221.75k
...performance- first pipeline. As a Senior Quality Engineer working on the rendering pipeline , you... ...editing, color, and export workflows. Evaluate user impact, functional ity ,... ...****@*****.*** or call (***) ***-****. AI Use Guidelines for Interviews: Our...SuggestedTemporary workLocal areaRelocationAdobe
San Francisco, CA4 days ago$160k - $220k
...Lead Quality EngineerLocation: Remote (US or Canada) Company Stage... ...leading enterprise conversational AI platforms, powering AI voice... ...'ll help define how quality engineering operates for modern... ...experienceExperience with LLM evaluation, RAG testing, or agentic AI systemsBuilt...H1bWork at officeRemote workRecruiting from Scratch
San Francisco, CA5 days ago$180k - $230k
...its approach to software quality. We're moving from a... ...supported by embedded Quality Engineers. We need a Principal-... ...and own Dealpath's AI evals practice, creating... ...AI to accelerate evaluation workflows.Drive the production... ...for each AI feature — task completion rate, tool-...Full timeWork at office3 days per week$110k - $160k
...: At Nonstop Health, we are engineering the backbone of modern healthcare... ...and cutting-edge enterprise AI platforms. We are looking for... ...and building our global quality control plane from the ground... ...software quality by developing evaluation methods for multi-agent LLM architectures...Contract workImmediate startNonstop Administration and Insurance Services Inc
San Francisco, CA2 days ago$90k - $150k
...industry, join our team as we help shape a brighter way forward. QA/QC Engineer (MEP/Cx) Data center project field QA/QC role that is... ...Daily PayAt JLL, we harness the power of artificial intelligence (AI) to efficiently accelerate meaningful connections between candidates...Full timeWork experience placementFor subcontractorLocal area$140k - $200k
...Description At Ouster, we are pioneering the future of Physical AI. Our advanced vision algorithms and cutting-edge sensor... ...and more efficient world. The Sr. Data Infrastructure / Quality Engineering role will play a crucial part in architecting, building, and...Work experience placementLocal areaOuster
San Francisco, CA9 days ago$98k - $140k
Notion in San Francisco is looking for a Quality Engineer to own the quality standards of its AI products. This role blends operational strategy and product... ...and a keen interest in LLMs, contributing to the evaluation and launch of new models. Notion offers a competitive...$140k - $190k
...Software Test Engineer San Francisco Company Overview Echo Neurotechnologies is an... ...through advanced hardware engineering and AI solutions. Our mission is to deliver... ...living with disabilities and improve their quality of life. Team Culture Join a small,...Echo Neurotechnologies
San Francisco, CA1 day ago- ...Sr Quality Assurance Engineer Location: San Francisco, California (Partial Remote) Duration: 12 months Role Overview We are seeking a... ...details. By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from Everforth...Remote work
Apex Systems
San Francisco, CA1 day ago - ...leading investors, we're building the category-defining AI workflow automation platform that healthcare teams rely... ...the Role We're looking for a midlevel or Senior QA Engineer who's passionate about quality engineering, automation, and building scalable testing practices...Full timeWork at officeRemote workFlexible hours2 days per week
Plenful
San Francisco, CA4 days ago $155k - $185k
...Senior Software Quality Engineer San Francisco Company Overview Echo Neurotechnologies is an exciting new startup in the Brain-Computer... ...driving innovation through advanced hardware engineering and AI solutions. Our mission is to deliver cutting-edge technologies...Echo Neurotechnologies
San Francisco, CA4 days ago- ...Location Type On-site Department Engineering Our Mission Reflection’s... ...nation states. Our team of AI researchers and company builders... ...models meets a high bar for quality, reliability, and downstream... ...data pipelines, QA systems, or evaluation workflows for pre‑training...Full timeRelocation package
- ...of their readiness, activity, and sleep quality by using their Oura Ring and its connected... ...for a Hardware Signal Quality Test Engineer to join our San Francisco team. As Oura’... ...executing defined test activities and helping evaluate signal quality across relevant sensor...Contract workWork experience placementWork at officeLocal areaRemote workFlexible hours
- Midjourney Inc. is seeking a QA-focused professional to join our hybrid SF/NY team. You’ll partner with product, design, and engineering to ensure quality across our core initiatives and explore edge cases early in development. We expect you to think critically about how...
$140k - $153k
...hardware, software, and increasingly AI, coupled with the patient-... ...are looking for a Senior QA Engineer I to join our team! You will... ...responsible for ensuring the quality of a suite of tools for providers... ...the ground up. Analyze and evaluate business requirements with...Local areaImmediate startRemote workFlexible hours- Heartflow in San Francisco is advancing AI-driven cardiac care with SaMD. As Principal Device Quality Engineer, you will be the global SME for software quality and design excellence, aligning Agile development with medical device regulations across the SDLC. Lead design...
- ...Responsibilities: Ensures that suppliers deliver quality parts, materials, and services.... ...practices (GMP) and quality standards are met. Evaluates suppliers' internal functions to assess... ...operation. Provides Pre-Market Quality Engineering support to New Product Development (NPD)...
- ...on a mission to create the world's first AI-powered Personal & Entrepreneurial Resource... .... The Role As a QA Automation Engineer at Air Apps, you will play a crucial role... ...ensure our applications meet the highest quality standards. You will work closely with...Temporary workWorldwide
Air Apps
San Francisco, CA4 days ago - ...like application development and AI/ML, and our people-first... ...are looking for a QA/Support Engineer to author and QA AI-driven automations... ...-volume repetitive automation tasks with accuracy, and maintain... ...room. - A deep sense of quality and attention to detail....Contract workLocal areaRemote workVisa sponsorshipWork visaFlexible hours
- B Capital is seeking a data engineer focused on ensuring high-quality data for AI model training. You will work closely with top researchers to develop metrics and standards for data quality. The ideal candidate has solid experience in Python and data pipelines and is detail...
- Visa Hunt is seeking a data-focused engineer to own upstream data quality for LLM pre-training. You will design QA workflows, build reusable pipelines, and translate quality signals into concrete actions across languages and modalities. Collaborate with researchers and...
- ...navigate the complexities of risk-based contracting. We’re looking for a QA Automation Engineer to lead and elevate quality across our SaaS platform and its expanding suite of AI-driven capabilities. In this role, you help drive quality across our engineering teams by...Flexible hoursShift work
Arbital Health
San Francisco, CA10 days ago - Ōura seeks a Senior Software Test Engineer to lead end-to-end testing across iOS, Android, and cloud services. You will design test strategies... ...Pytest and Appium, and drive CI/CD integration to ensure high-quality product releases. The role emphasizes embedded device testing,...
- B Capital is seeking a data engineer to ensure high data quality for training AI models. You will own the upstream data quality for LLM post-training and design automated QA methods in a collaborative environment. Ideal candidates will have strong engineering skills, a...
- ...Quality Systems EngineerAt Nudge, our mission is to develop the best technology for interfacing... ...from each other.As a quality systems engineer at Nudge, you will:Ensure that new and... ...clinical trialsEffectively manage multiple tasks and drive execution of change ordersInitiate...
Nudge Inc.
San Francisco, CA5 days ago - ...Obsidian is hiring expert Evaluators in Real estate, hospitality, and events to review AI-generated work for accuracy, rigor, and domain quality. This remote position requires deep expertise and involves grading outputs like documents and presentations. Applicants...Work at officeRemote work
Obsidian
San Francisco, CA4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Benchmark Quality Engineer for Task Evaluation. Be the first to apply!
Related searches
- ai ml engineer San Francisco, CA
- ai prompt engineer San Francisco, CA
- senior ai engineer San Francisco, CA
- ai research engineer San Francisco, CA
- machine learning ai engineer San Francisco, CA
- ai developer San Francisco, CA
- ai engineer San Francisco, CA
- ai engineer remote San Francisco, CA
- operations quality engineer San Francisco, CA
- advanced quality engineer San Francisco, CA