Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Prompt Quality Rater

$35 - $55 per hour
Temporary

Claru

Pay: $35–55/hour

You'll evaluate prompt-response pairs generated by large language models, scoring them across dimensions like factual accuracy, instruction-following, coherence, and appropriate refusal behavior. Each session involves reviewing batches of 20–50 pairs using a structured rubric inside a web-based annotation platform, flagging edge cases, and writing brief justifications for non-obvious scores.

Beyond surface-level ratings, you'll identify subtle failure modes — responses that are technically accurate but misleading, answers that follow the letter of a prompt while violating its intent, and outputs that pass a quick read but contain embedded errors. You'll work asynchronously, with a typical batch taking 60–90 minutes, and you're expected to maintain inter-annotator agreement scores above 0.75 kappa.

Feedback you submit feeds directly into preference datasets used to fine-tune and align production models. Calibration sessions run bi-weekly, where you'll review disagreements with a senior evaluator and update your scoring approach. Strong performance can lead to specialized tracks covering domain-specific evaluation (legal, medical, code).

Skills & requirements

  • Prompt evaluation
  • Rubric-based scoring
  • LLM output analysis
  • Instruction-following assessment
  • Factual accuracy verification
  • Annotation platform proficiency

Category: RLHF

Vacancy posted a month ago
Similar jobs that could be interesting for youBased on the AI Prompt Quality Rater in Sunnyvale, CA vacancy
  •  ...Job Description Job Description Quality Assurance Analyst Data Integrity General Description...  ...and correct data issues from Generative AI by reviewing and annotating datasets...  ...mindset and execute necessary changes promptly and accurately. Key Qualifications:... 
    Quality

    Mindlance

    Cupertino, CA
    21 days ago
  • $262k - $364k

    Own roadmap execution, turn quality strategy into Objectives and Key Results (OKRs), quarterly...  ...distributed computing systems for AI/ML workloads.Experience with Large Language...  ...conversation is and owns the full quality loop: prompts, model turn-ups, evals and autoraters,... 
    Quality

    Google

    Mountain View, CA
    2 days ago
  • $183.7k - $319.8k

     ..., and QA teams across Apple to define what a high-quality camera experience looks like on our platforms.Description...  ...or early hardware bring-upExperience applying AI-assisted development and testing tools — including prompt construction for test authoring, log analysis, and... 
    Quality
    Relocation

    Apple

    Cupertino, CA
    3 days ago
  • $120.9k - $249k

     ...believe Machine Learning and AI can change how people experience...  ...organization. We build high-quality ML datasets at scale to train...  ...evaluation programs, including rater calibration, gold sets, and ongoing...  ...designing and iterating on prompts for quality checks assisted by... 
    Quality
    Relocation
    Shift work

    Apple

    Cupertino, CA
    3 days ago
  • $190.9k - $334.1k

     ...meaningful work. Today, ServiceNow is the AI control tower for business reinvention....  ...specifications, authoring the instructions and prompts that govern voice agent behavior, proving...  ...class engineering activity rather than a quality-assurance afterthought. The engineer owns... 
    Quality
    Full time
    Work at office
    Local area
    Immediate start
    Remote work
    Flexible hours

    ServiceNow

    Santa Clara, CA
    12 days ago
  •  ...Welo Global is seeking a Data Quality Analyst to ensure quality and execution across teams in Sunnyvale, CA. This role involves auditing outputs, providing feedback, and supporting day-to-day operations. Ideal candidates will have a university degree and 2–4 years of relevant... 
    Quality

    Welo Global

    Sunnyvale, CA
    1 day ago
  •  ...forefront of the Artificial Intelligence (AI) revolution. Our mission is to create intelligent...  .... We are seeking a talented and motivated Prompt Engineer to join our dynamic team and play...  ...for designing and developing high-quality prompts that drive our AI models and algorithms... 
    Quality

    Forhyre

    Sunnyvale, CA
    4 days ago
  • $115k - $165k

     ...technology consulting and software development company delivering cloud, AI, data, and enterprise solutions across the United States. This...  ...is seeking a Test Automation Lead to improve software quality through structured testing, automation, defect analysis, and release... 
    Quality
    Full time
    H1b
    Local area
    Immediate start
    Remote work
    Visa sponsorship

    Bright Vision Technologies

    Sunnyvale, CA
    2 days ago
  • $20 - $25 per hour

     ...Digital Advertising Quality Reviewer Overview We are seeking detail-oriented digital advertising professionals to support a specialized...  ...and civil liability. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of our... 
    Quality
    Contract work
    Temporary work
    Remote work
    Menlo Park, CA
    8 days ago
  •  ...provider of network automation and multi-tenancy solutions for AI infrastructure, supporting high-growth neoclouds, sovereign AI cloud...  ...This opportunity is for a Director of QA to build and lead the quality function as the company scales its product and engineering... 
    Quality
    Full time
    Work at office
    Santa Clara, CA
    more than 2 months ago
  • $123k - $135k

     ...Role: Ooma is seeking a motivated and detail-oriented Software Quality Assurance Engineer to join our SQA team. In this role, you will...  ...opportunity to work with modern test automation tools, including AI-assisted testing technologies. What You’ll Do: Design, develop,... 
    Quality
    Temporary work
    Local area
    Work from home

    ooma

    Sunnyvale, CA
    1 day ago
  • $140k - $170k

     ...intelligence platform for digital infrastructure. By adopting an AI,ML-based analytics approach, the platform provides actionable...  ...improve test coverage, accelerate delivery, and enhance product quality. Key Responsibilities Design, develop, and maintain robust... 
    Quality

    MetAntz

    Santa Clara, CA
    3 days ago
  •  ...At Rhoda AI, we’re building the next generation of generalist intelligent robots. We own the full robotics stack from high-performance...  ...test, collect and analyze logs, and report results. Track quality for your program: escaped defects, regression trends, and release... 
    Quality

    RHODA

    Mountain View, CA
    1 day ago
  •  ...* Validate the reliability, performance, and functionality of AI/ML workloads across the Apple Neural Engine (ANE) software stack...  ..., compiler, driver, and firmware layers, ensuring comprehensive quality coverage. * Analyze test results, investigate failures, and collaborate... 
    Quality
    Full time

    Applab Systems Inc

    Sunnyvale, CA
    21 days ago
  • $144k - $198k

     ...create and service is essential to advancing AI and accelerating the commercialization of...  ...join our winning team that is focused on Quality Engineering and Reliability Engineering....  ...77-612-7547, option 1, and following the prompts to speak to an HR Advisor. This contact... 
    Quality
    Full time
    Worldwide

    Applied Materials

    Santa Clara, CA
    23 hours ago
  • $207k - $300k

     ...Staff Software Engineer, Eval Infra TL, Quality Platform, Gemini Cloud Operator, Agentic AI Google's software engineers develop the next-generation technologies that change how billions of users connect, explore, and interact with information and one another. Our products... 
    Quality

    Google

    Sunnyvale, CA
    3 days ago
  • $120k - $140k

     ...Engineer (Agent Security)ResponsibilityBuild AI security solutions for enterprise...  ...debugging skills and a focus on delivering high-quality solutions. Strong PlusHands-on experience...  ...and agent security, including areas such as prompt injection, tool poisoning, jailbreaks, and... 
    Quality
    Full time
    Worldwide

    Fortinet

    Sunnyvale, CA
    1 day ago
  •  ...Job Description Job Description About Otter.ai Join us at Otter.ai in transforming the way remote teams work and communicate...  ...you will Support testing of multiple features and drive the quality via extensive hands-on functional testing of user-facing features... 
    Quality
    Remote work

    Hired Recruiters

    Los Altos, CA
    17 days ago
  • $38 per hour

     ...experienced, detail-oriented professionals to join our team as Data Quality Analysts. This role sits at the center of execution and quality...  ..., but done right. You’ll work closely with both people and AI systems — auditing outputs, supporting day-to-day execution, and... 
    Quality
    Full time
    Contract work
    Remote work
    Visa sponsorship

    Welo Global

    Sunnyvale, CA
    2 days ago
  •  ...Join d-Matrix and help build the future of AI inference. We’ve pioneered a memory-centric architecture that brings compute directly...  ...our Santa Clara, CA, HQ 3 days per week.   Title: Director, Quality and Reliability Engineering – AI Hardware The Role: d-... 
    Quality
    Full time
    Contract work
    3 days per week

    d-Matrix

    Santa Clara, CA
    4 days ago
  • $45 - $60 per hour

     ...monetization platforms. Our core mission is to empower the QA (Quality Assurance) and RD (Research and Development) teams by developing...  ...process. We encourage the exploration and application of cutting-edge AI tools and are committed to integrating advanced technologies... 
    Quality
    Hourly pay
    Internship
    Local area

    TikTok

    San Jose, CA
    1 day ago
  • $120k - $135k

     ...Job Description Job Description Kodiak AI, Inc. (Nasdaq: KDK), founded in 2018, is a leader in Physical AI, developing driverless...  ...General Dynamics Land Systems. We are looking for a Software Quality Triage Specialist to join our rapidly growing Software Quality team... 
    Quality
    Temporary work
    Work at office
    Visa sponsorship
    Flexible hours

    Kodiak

    Mountain View, CA
    26 days ago
  •  ...guidelines across the team. Responsibilities: Operational & Quality Oversight Oversee verification and quality of processed cases...  ..., QA/QC exposure We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing... 
    Quality

    TSMG

    Mountain View, CA
    7 days ago
  •  ...AreAccenture is recognized as a global leader in AI and cloud transformation, helping...  ...with LLM APIs (Claude/Anthropic, GPT-4), prompt engineering, and agentic frameworks (LangChain...  ...context and where it does not. Own AI quality and observability: Apply LLM tracing, failure... 
    Quality
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    Mountain View, CA
    23 hours ago
  • $190k - $234k

     ...billions in simulation across 15+ U.S. states. The Software Quality Operations (SWQOps) team is at the heart of ensuring the safety,...  ...build an adaptable and scalable operation, increasingly powered by AI, to deliver the crucial insights necessary to confidently deploy... 
    Quality
    Odd job
    Full time
    Temporary work
    Work experience placement
    Local area
    Remote work

    Waymo

    Mountain View, CA
    8 days ago
  •  ...circuit board manufacturing and assembly, specializing in high-quality, quick-turn and complex prototypes. With over 35 years of experience...  ...product testing, identifying and addressing quality issues promptly.  Process Improvement: Analyze quality data to identify... 
    Quality
    Work at office
    Flexible hours

    Sierra Circuits

    Sunnyvale, CA
    3 days ago
  • $150k - $250k

     ...economies, protect space and our planet and enhance our overall quality of life. We are hiring an Engineering Lead in IoT Systems Integration...  ...• Paid time off We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing... 
    Quality
    Full time
    Immediate start
    Visa sponsorship
    Night shift

    E-Space

    Santa Clara, CA
    28 days ago
  • $144.6k - $218k

     ...Marketing team within Worldwide Product Marketing is looking for a Quality Assurance Engineer to support Apple's world-class product...  ...with precision, triaging issues under launch timelines, and using AI tools to accelerate root-cause investigation. This role is well... 
    Quality
    Worldwide
    Relocation
    Flexible hours
    Shift work

    Apple

    Cupertino, CA
    3 days ago
  •  ...Join d-Matrix and help build the future of AI inference. We’ve pioneered a memory-centric architecture that brings compute directly...  ...execution and energized by hard problems. Director, Software Quality Engineering Location: Santa Clara, CA (Hybrid on-site 3 days... 
    Quality
    Full time
    3 days per week

    d-Matrix

    Santa Clara, CA
    4 days ago
  • $120k - $135k

     ...Job Description Job Description Kodiak AI, Inc. (Nasdaq: KDK), founded in 2018, is a leader in Physical AI, developing driverless...  ...General Dynamics Land Systems. We are looking for a Software Quality Triage Specialist to join our rapidly growing Software Quality team... 
    Quality
    Temporary work
    Work at office
    Visa sponsorship
    Flexible hours

    Kodiak

    Mountain View, CA
    28 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Prompt Quality Rater. Be the first to apply!