AI Prompt Quality Evaluator & Feedback Analyst
Claru
AI Trainer Jobs is seeking a detail-oriented evaluator to score prompt-response pairs from large language models. You will use structured rubrics to rate factual accuracy, instruction-following, and coherence across batches, typically 20–50 pairs per session. You will flag edge cases, document subtle failure modes, and contribute to fine-tuning data. Sessions are asynchronous with calibration meetings every two weeks, offering potential tracks in legal, medical, or coding domains. #J-18808-Ljbffr Claru
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Prompt Quality Evaluator & Feedback Analyst in New York, NY vacancy
- AuraOne, Inc. seeks a remote Historical Reasoning AI Evaluator to review prompts and responses, applying AuraOne's quality rubric. You will label edge cases, provide structured feedback, and inform retraining efforts through precise rationale. The role supports a contractor...QualityRemote jobFor contractors
- AuraOne is seeking an AI Agent Evaluator to remotely review and label ai agent evaluation prompts and responses against AuraOne's quality rubric. You will compare outputs, identify edge cases, and provide structured feedback to support retraining. The role focuses on auditing...QualityRemote job
- AI Trainer is a remote evaluation track for reviewing AI trainer evaluation prompts and responses against AuraOne’s quality rubric. Reviewers compare paired outputs, label edge cases, and write structured feedback for retraining the model. Responsibilities include evaluating...QualityRemote jobHourly payFor contractorsFlexible hours
$35 - $40 per hour
A leading evaluation organization seeks a Generalist Evaluator Expert for a remote, flexible... ...contract role. This position involves designing prompts for language model evaluations, defining... ...thinking skills, familiarity with AI tools like ChatGPT, and a BS or BA degree...QualityRemote jobContract workFlexible hours- ...Join a 16-week research study evaluating AI assistants from the perspective of everyday... ..., Instinct, or Muse and can assess prompts and responses in English. A part-time... ...everyday user. Assess response quality and provide feedback following the study’s guidelines....QualityFull timePart timeImmediate start
Braintrust
New York, NY3 days ago - ...Job-Senior Business Analyst & Prompt Engineer (AI Skills) Job Location - NYC, NY Job Type- Full Time... ...prompts, instructions and evaluation sets that let AI agents do work across... ...Skills with clear inputs, outputs, quality criteria and escalation points. Write...QualityFull time3 days per week
Maveric Systems Limited
New York, NY15 hours ago - Thomson Reuters seeks a Sr. Legal AI Prompter / Evaluator to advance CoCounsel Legal through prompts, skills, and agent configurations. You will experiment with Thomson... ..., and data scientists to improve product quality. The ideal candidate is an experienced lawyer with...Quality
- AuraOne, Inc. is seeking a remote Browser Automation Task Evaluator to review prompts and responses against our quality rubric. This contractor role focuses on labeling edge cases, writing structured feedback, and helping retrain our models. You will compare paired outputs...QualityRemote jobFor contractors10 hours per week
- AuraOne, Inc. is seeking an AI Data Reviewer for the Misinformation Context AI Evaluator track. This remote contractor... ..., and providing structured feedback to retrain prompts. You will compare paired responses... ...to maintain high reviewer quality. Applicants must be able to...QualityRemote jobFor contractors
- AI Trainer Jobs seeks a Hindi Language Expert for a remote evaluation track. Review Hindi prompts and responses, compare paired outputs, label edge... ..., and provide structured feedback to retrain models. You will... ...to maintain reviewer quality. This contractor role offers...QualityRemote jobFor contractorsFlexible hours
- Misinformation Context AI Evaluator is a remote evaluation track for reviewing... ...misinformation context ai evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired... ...and write the kind of structured feedback the modeling team can use to...QualityHourly payFor contractorsRemote work10 hours per week
- AuraOne, Inc. seeks a Revenue Recognition AI Reviewer to remotely assess evaluation prompts and responses against a published quality rubric. You will compare paired outputs, label edge cases, and produce structured feedback to retrain the model. As a contractor, you’ll...QualityRemote jobFor contractors10 hours per week
- Create challenging prompts and evaluate large language model responses for accuracy... ...where they fall short. Your feedback will focus on whether... ...execution to assess how well AI handles management-related topics... ..., relevance, and overall quality. Find factual errors,...QualityContract workPart timeFor contractorsRemote workWorldwide
- AuraOne, Inc. is seeking a diligent remote contractor to evaluate at-work video recorder outputs against a versioned rubric. You will assign... ...tags, write rationales, and help identify edge cases for quality control. Responsibilities include comparing responses, labeling...QualityRemote jobFor contractors
$70 per hour
...and technical talent with leading AI research labs. Headquartered in San... ...Remote Role Responsibilities Evaluate AI-generated responses to strengthen... .... Provide structured written feedback to improve training data quality and downstream performance. Develop...QualityContract workSummer workRemote work- AI Trainer Jobs is seeking remote contractors to review Application... ...Photoshop on Windows - Generalist prompts and responses, evaluating them against AuraOne's quality rubric. Reviewers compare paired... ...cases, and provide structured feedback to retrain the model. As a reviewer...QualityRemote jobFor contractors
- ...is seeking a Luganda Language Expert for a remote evaluation track to review Luganda prompts and model outputs against a quality rubric. Reviewers compare paired outputs, label edge cases, and draft structured feedback for retraining. As a remote independent contractor...QualityRemote jobFor contractors
- HELP SHAPE THE FUTURE OF AI BY EVALUATING HOW WELL AI ASSISTANTS UNDERSTAND... ...Anyone AI, we create high-quality training and evaluation data... ...compare your experiences. Your feedback will help assess how... ...needs. WHAT YOU'LL DO Write prompts about questions, interests,...QualityPart timeFor contractorsRemote work
- ...Title: Gmail & Google Calendar AI Assistant Evaluator Role Type: Contractor,... ...project enhancing AI assistant quality through real-world task... ...responses, providing structured feedback that improves model... ...projects. Familiarity with prompt writing and a practical understanding...QualityContract workFor contractorsRemote work
$137.9k - $256.1k
Sr. Legal AI Prompter / EvaluatorJob DescriptionSr... .... Legal AI Prompters / Evaluators are responsible for... ...by writing and refining prompts, skills, and agent configurations... ...into practical, high-quality AI outputs.Key... ...and provide strategic feedback to software engineers and...QualityFull timeWork at officeLocal areaFlexible hoursThomson Reuters
New York, NY2 days ago- Appsierra Group is seeking an AI Session Annotator for a fully remote, independent contractor role. You will review recorded AI sessions... ..., compare responses to visuals, and provide precise written feedback using a detailed rubric. Strong English, attention to detail,...Remote jobWeekly payFor contractorsFlexible hours
$35 per hour
AI Trainer Jobs in the United States is seeking detail oriented analysts to evaluate and rank AI generated responses to text prompts. You will judge answers on accuracy, relevance, clarity, conciseness, safety, localization, and how well they follow instructions, then...Remote jobHourly payFor contractorsFlexible hours$14.5 per hour
...Join to apply for the AI Web Search Evaluator role at Welo Data Welo Data works with technology... ...to provide datasets that are high-quality, ethically sourced, relevant, diverse,... ...they meet user expectations. Offer feedback and recommendations to improve algorithm...QualityHourly payPart timeImmediate startRemote workWork from home10 hours per weekFlexible hoursWelo Data
New York, NY4 days ago- ...is seeking a Conlang Language Expert to remotely review language evaluation prompts and responses against our quality rubric. You will compare paired outputs, label edge cases, and write structured feedback used to retrain models. As a contractor, you will work...QualityRemote jobFor contractors10 hours per weekFlexible hours
$65 - $70 per hour
...DESCRIPTION Business Analyst is responsible for... ...applications, AI-supported... ...owners, gather their feedback, and ensure business... ...and to develop and evaluate the technical requirements... ...implemented. - Quality assurance... ...user story issues promptly, and follow up on...QualityHourly payContract workFor contractors- AuraOne is seeking a remote evaluation contractor to review Gmail and Google Calendar AI Assistant outputs against our quality rubric. You will label issues, compare responses, and contribute structured feedback to help retrain the model. This role requires strong attention...QualityRemote jobHourly payFor contractors10 hours per week
- Anyone AI Inc. is seeking multilingual coding experts to evaluate AI-generated responses on a part-time remote contract lasting... ...by providing precise feedback and evaluations. You will review... ...content for clarity, correctness, and quality, validating JSON structure and giving...QualityRemote jobContract workPart time
- ...thinking tech company in the United States is seeking an individual to evaluate AI model responses for personalization quality. Responsibilities include analyzing AI outputs, designing multi-turn prompts, and collaborating with teams to enhance model performance. Ideal...QualityRemote jobFull time
- OpenTrain AI, Inc. is seeking an Education Professional AI Response Evaluator. This remote, contract, part-time role involves creating education-focused prompts and evaluating AI-generated responses for accuracy... ...contexts to judge language quality and real-world task...QualityRemote jobContract workPart time
- Anyone AI is seeking multilingual coding experts to evaluate AI-generated responses. This part-time remote contract lasts up to 3 months... ...content for clarity, correctness, and quality. Validate JSON structure and provide clear feedback based on project guidelines. What we're...QualityRemote jobContract workPart timeFor contractors
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Prompt Quality Evaluator & Feedback Analyst. Be the first to apply!
Related searches
- evaluator New York, NY
- work from home web search evaluator New York, NY
- program evaluator New York, NY
- clinical evaluator New York, NY
- nurse evaluator New York, NY
- work from home social media evaluator New York, NY
- ai evaluator New York, NY
- education evaluator New York, NY
- quality evaluator New York, NY
- social media evaluator New York, NY