AI Prompt Quality Rater
$35 - $55 per hourClaru
Pay: $35–55/hour
You'll evaluate prompt-response pairs generated by large language models, scoring them across dimensions like factual accuracy, instruction-following, coherence, and appropriate refusal behavior. Each session involves reviewing batches of 20–50 pairs using a structured rubric inside a web-based annotation platform, flagging edge cases, and writing brief justifications for non-obvious scores.
Beyond surface-level ratings, you'll identify subtle failure modes — responses that are technically accurate but misleading, answers that follow the letter of a prompt while violating its intent, and outputs that pass a quick read but contain embedded errors. You'll work asynchronously, with a typical batch taking 60–90 minutes, and you're expected to maintain inter-annotator agreement scores above 0.75 kappa.
Feedback you submit feeds directly into preference datasets used to fine-tune and align production models. Calibration sessions run bi-weekly, where you'll review disagreements with a senior evaluator and update your scoring approach. Strong performance can lead to specialized tracks covering domain-specific evaluation (legal, medical, code).
Skills & requirements
- Prompt evaluation
- Rubric-based scoring
- LLM output analysis
- Instruction-following assessment
- Factual accuracy verification
- Annotation platform proficiency
Category: RLHF
- ...Job Description Job Description Quality Assurance Analyst Data Integrity General Description... ...and correct data issues from Generative AI by reviewing and annotating datasets... ...mindset and execute necessary changes promptly and accurately. Key Qualifications:...Quality
$262k - $364k
Own roadmap execution, turn quality strategy into Objectives and Key Results (OKRs), quarterly... ...distributed computing systems for AI/ML workloads.Experience with Large Language... ...conversation is and owns the full quality loop: prompts, model turn-ups, evals and autoraters,...Quality$183.7k - $319.8k
..., and QA teams across Apple to define what a high-quality camera experience looks like on our platforms.Description... ...or early hardware bring-upExperience applying AI-assisted development and testing tools — including prompt construction for test authoring, log analysis, and...QualityRelocation$120.9k - $249k
...believe Machine Learning and AI can change how people experience... ...organization. We build high-quality ML datasets at scale to train... ...evaluation programs, including rater calibration, gold sets, and ongoing... ...designing and iterating on prompts for quality checks assisted by...QualityRelocationShift work$190.9k - $334.1k
...meaningful work. Today, ServiceNow is the AI control tower for business reinvention.... ...specifications, authoring the instructions and prompts that govern voice agent behavior, proving... ...class engineering activity rather than a quality-assurance afterthought. The engineer owns...QualityFull timeWork at officeLocal areaImmediate startRemote workFlexible hours- ...Welo Global is seeking a Data Quality Analyst to ensure quality and execution across teams in Sunnyvale, CA. This role involves auditing outputs, providing feedback, and supporting day-to-day operations. Ideal candidates will have a university degree and 2–4 years of relevant...Quality
- ...forefront of the Artificial Intelligence (AI) revolution. Our mission is to create intelligent... .... We are seeking a talented and motivated Prompt Engineer to join our dynamic team and play... ...for designing and developing high-quality prompts that drive our AI models and algorithms...Quality
$115k - $165k
...technology consulting and software development company delivering cloud, AI, data, and enterprise solutions across the United States. This... ...is seeking a Test Automation Lead to improve software quality through structured testing, automation, defect analysis, and release...QualityFull timeH1bLocal areaImmediate startRemote workVisa sponsorship$20 - $25 per hour
...Digital Advertising Quality Reviewer Overview We are seeking detail-oriented digital advertising professionals to support a specialized... ...and civil liability. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of our...QualityContract workTemporary workRemote work- ...provider of network automation and multi-tenancy solutions for AI infrastructure, supporting high-growth neoclouds, sovereign AI cloud... ...This opportunity is for a Director of QA to build and lead the quality function as the company scales its product and engineering...QualityFull timeWork at office
$123k - $135k
...Role: Ooma is seeking a motivated and detail-oriented Software Quality Assurance Engineer to join our SQA team. In this role, you will... ...opportunity to work with modern test automation tools, including AI-assisted testing technologies. What You’ll Do: Design, develop,...QualityTemporary workLocal areaWork from home$140k - $170k
...intelligence platform for digital infrastructure. By adopting an AI,ML-based analytics approach, the platform provides actionable... ...improve test coverage, accelerate delivery, and enhance product quality. Key Responsibilities Design, develop, and maintain robust...Quality- ...At Rhoda AI, we’re building the next generation of generalist intelligent robots. We own the full robotics stack from high-performance... ...test, collect and analyze logs, and report results. Track quality for your program: escaped defects, regression trends, and release...Quality
- ...* Validate the reliability, performance, and functionality of AI/ML workloads across the Apple Neural Engine (ANE) software stack... ..., compiler, driver, and firmware layers, ensuring comprehensive quality coverage. * Analyze test results, investigate failures, and collaborate...QualityFull time
$144k - $198k
...create and service is essential to advancing AI and accelerating the commercialization of... ...join our winning team that is focused on Quality Engineering and Reliability Engineering.... ...77-612-7547, option 1, and following the prompts to speak to an HR Advisor. This contact...QualityFull timeWorldwide$207k - $300k
...Staff Software Engineer, Eval Infra TL, Quality Platform, Gemini Cloud Operator, Agentic AI Google's software engineers develop the next-generation technologies that change how billions of users connect, explore, and interact with information and one another. Our products...Quality$120k - $140k
...Engineer (Agent Security)ResponsibilityBuild AI security solutions for enterprise... ...debugging skills and a focus on delivering high-quality solutions. Strong PlusHands-on experience... ...and agent security, including areas such as prompt injection, tool poisoning, jailbreaks, and...QualityFull timeWorldwide- ...Job Description Job Description About Otter.ai Join us at Otter.ai in transforming the way remote teams work and communicate... ...you will Support testing of multiple features and drive the quality via extensive hands-on functional testing of user-facing features...QualityRemote work
$38 per hour
...experienced, detail-oriented professionals to join our team as Data Quality Analysts. This role sits at the center of execution and quality... ..., but done right. You’ll work closely with both people and AI systems — auditing outputs, supporting day-to-day execution, and...QualityFull timeContract workRemote workVisa sponsorship- ...Join d-Matrix and help build the future of AI inference. We’ve pioneered a memory-centric architecture that brings compute directly... ...our Santa Clara, CA, HQ 3 days per week. Title: Director, Quality and Reliability Engineering – AI Hardware The Role: d-...QualityFull timeContract work3 days per week
$45 - $60 per hour
...monetization platforms. Our core mission is to empower the QA (Quality Assurance) and RD (Research and Development) teams by developing... ...process. We encourage the exploration and application of cutting-edge AI tools and are committed to integrating advanced technologies...QualityHourly payInternshipLocal area$120k - $135k
...Job Description Job Description Kodiak AI, Inc. (Nasdaq: KDK), founded in 2018, is a leader in Physical AI, developing driverless... ...General Dynamics Land Systems. We are looking for a Software Quality Triage Specialist to join our rapidly growing Software Quality team...QualityTemporary workWork at officeVisa sponsorshipFlexible hours- ...guidelines across the team. Responsibilities: Operational & Quality Oversight Oversee verification and quality of processed cases... ..., QA/QC exposure We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing...Quality
- ...AreAccenture is recognized as a global leader in AI and cloud transformation, helping... ...with LLM APIs (Claude/Anthropic, GPT-4), prompt engineering, and agentic frameworks (LangChain... ...context and where it does not. Own AI quality and observability: Apply LLM tracing, failure...QualityWork experience placementLive inWork at officeLocal area
$190k - $234k
...billions in simulation across 15+ U.S. states. The Software Quality Operations (SWQOps) team is at the heart of ensuring the safety,... ...build an adaptable and scalable operation, increasingly powered by AI, to deliver the crucial insights necessary to confidently deploy...QualityOdd jobFull timeTemporary workWork experience placementLocal areaRemote work- ...circuit board manufacturing and assembly, specializing in high-quality, quick-turn and complex prototypes. With over 35 years of experience... ...product testing, identifying and addressing quality issues promptly. Process Improvement: Analyze quality data to identify...QualityWork at officeFlexible hours
$150k - $250k
...economies, protect space and our planet and enhance our overall quality of life. We are hiring an Engineering Lead in IoT Systems Integration... ...• Paid time off We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing...QualityFull timeImmediate startVisa sponsorshipNight shift$144.6k - $218k
...Marketing team within Worldwide Product Marketing is looking for a Quality Assurance Engineer to support Apple's world-class product... ...with precision, triaging issues under launch timelines, and using AI tools to accelerate root-cause investigation. This role is well...QualityWorldwideRelocationFlexible hoursShift work- ...Join d-Matrix and help build the future of AI inference. We’ve pioneered a memory-centric architecture that brings compute directly... ...execution and energized by hard problems. Director, Software Quality Engineering Location: Santa Clara, CA (Hybrid on-site 3 days...QualityFull time3 days per week
$120k - $135k
...Job Description Job Description Kodiak AI, Inc. (Nasdaq: KDK), founded in 2018, is a leader in Physical AI, developing driverless... ...General Dynamics Land Systems. We are looking for a Software Quality Triage Specialist to join our rapidly growing Software Quality team...QualityTemporary workWork at officeVisa sponsorshipFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Prompt Quality Rater. Be the first to apply!
- supplier quality Sunnyvale, CA
- quality assurance quality control engineer Sunnyvale, CA
- rn quality Sunnyvale, CA
- quality Sunnyvale, CA
- quality manufacturing Sunnyvale, CA
- quality inspection Sunnyvale, CA
- quality equipment Sunnyvale, CA
- quality improvement rn Sunnyvale, CA
- quality intern Sunnyvale, CA
- quality lead Sunnyvale, CA





