AI Evaluation Specialist
Crossing Hurdles
A leading AI organization in Australia is seeking individuals with strong writing and analytical skills to evaluate and improve AI outputs. The ideal candidate must possess the ability to assess emotional nuances and detail while adhering to structured guidelines. Responsibilities include ensuring clarity and empathy in evaluations, identifying weaknesses in AI reasoning, and contributing to a safe AI environment. Immediate availability post-onboarding is required for this role. #J-18808-Ljbffr
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Evaluation Specialist in New York, NY vacancy
- ...develop and map analytic cases to defined taxonomies across multiple evaluation areas. You will author complex test cases to evaluate model... ...on source handling and analytic rigor. You will assess and grade AI-generated responses based on reasoning, source use, and...Suggested
- AI Trainer Jobs is seeking an English (India) Recording Specialist to remotely evaluate prompts and responses against AuraOne’s rubric. You will compare paired outputs, label edge cases, and document structured feedback to help retrain the model. As a contractor, you will...SuggestedRemote jobFor contractors
- ...remote contract work. You will craft prompts, run side-by-side evaluations using ChatGPT and Claude, and provide thorough verbal feedback... ...emphasizes consistent evaluation standards and collaboration with AI model improvement efforts. Ideal candidates have strong English...SuggestedRemote jobContract work
$25 per hour
...seeking a Remote Personal Account Generalist Rater (Mobile) on a contract basis. The role involves reviewing AI-generated prompts and responses in English, evaluating accuracy and relevance, and providing structured feedback to support AI system improvements. The position...SuggestedRemote jobHourly payContract workFlexible hours- OpenTrain AI, Inc. is seeking individuals with deep physics knowledge to help improve AI systems through rigorous question design and evaluation. You will craft challenging physics questions and assess AI-generated answers for accuracy and validity. The role is an asynchronous...SuggestedRemote jobHourly payContract workFlexible hours
- ...tests that isolate model capabilities and create robust rubrics for scoring. You will develop reference answers, edge cases, and evaluate model outputs against consistent standards to improve benchmarks. The role emphasizes collaboration, documentation, and trend analysis...Remote jobFlexible hours
- ...Experts in project management to design Excel-based tasks that test AI's ability to perform real PM work. You bring 3+ years in PMO... ...infrastructure, or IT projects. You will craft tasks, write precise evaluation rubrics, and work as an independent contractor on a fully remote...Remote jobWeekly payFor contractors
$30 per hour
Evaluate and rank AI-generated responses in Singaporean Malay, explain your choices, and provide written feedback. This remote, part-time project is open to candidates in the United States and pays up to $30 per hour. You will review AI-generated responses and help improve...Hourly payPart timeFor contractorsLocal areaRemote work- Anyone AI is seeking remote independent contractors to help create and evaluate data for frontier AI models. You will interact with AI models and compare responses to ensure quality and reliability. Responsibilities include assessing accuracy, helpfulness, and instruction...Remote jobFor contractors
- AuraOne seeks a video-related professional to remotely evaluate AI training outputs against AuraOne's quality rubric. You will review video prompts and responses, compare paired outputs, label edge cases, and write structured feedback for the modeling team. Independent...Remote jobHourly payPart timeFor contractors
- AuraOne, Inc. seeks an AI Image Specialist (US Based) for remote image evaluation. You will review prompts and responses, apply a versioned rubric, and provide structured feedback to guide retraining. The role emphasizes careful labeling, rationale, and consistency across...Remote jobFor contractors10 hours per week
- AuraOne, Inc. is seeking a remote evaluation specialist to review agentic task completion model outputs, apply a structured rubric, and provide rationales for retraining. The role focuses on tagging edge cases, formatting auditable feedback, and maintaining calibration...Remote jobFor contractors10 hours per week
- ...American English. You will also interpret guidelines to ensure consistent evaluation and contribute to ongoing model improvement. The role requires strong English proficiency, familiarity with AI evaluation practices, and a reliable technical setup for screen #J-18808-...Remote jobContract workPart time
- AuraOne, Inc. is seeking a Senior AI Trainer evaluation contractor based in the United States to support remote evaluation tasks. You will review senior AI trainer prompts and responses, label edge cases, and provide structured feedback to retrain models. Responsibilities...Remote jobFor contractors10 hours per week
- AI Image Specialist (US Based) - Remote Image Evaluation is a remote evaluation track for reviewing ai image evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback...Remote jobHourly payFor contractors10 hours per week
- OpenTrain AI, Inc. is hiring for a worldwide remote, part-time contract role focused on creating challenging prompts and evaluating language model responses. Candidates should bring strong English writing, analytic thinking, and team-management experience to assess accuracy...Remote jobContract workPart timeWorldwideFlexible hours
- Mercor is seeking experienced music producers and audio engineers to evaluate generative music AI models. You will assess AI-generated music across genres and rate it against detailed quality standards, working in Portuguese and English. Start date is immediate with duration...Hourly payImmediate start
- AuraOne, Inc. is seeking a remote contractor to review Japanese generalist evaluation prompts and responses, applying AuraOne's quality rubric. You'll compare outputs, label issues, and generate structured feedback for model retraining. This role requires careful attention...Remote jobFor contractors10 hours per week
- Synthires seeks analytical thinkers to contribute to advanced AI research and evaluation projects. In this role, you will evaluate AI-generated responses, assess quality and reasoning, and provide structured, detailed written feedback to improve next-generation AI systems...Remote job
- Agency is seeking a linguist with deep Indonesian language expertise to assist with advanced language model evaluation and data refinement. You will analyze grammar, semantics, and pragmatics, and help shape prompts and evaluation metrics as a contractor. This remote role...For contractorsRemote work
- AuraOne, Inc. is seeking an AI Trainer (Life Sciences) for a remote evaluation track. You will review ai trainer prompts and responses, label edge cases, and provide structured feedback to help retrain models. This is a remote contractor role focused on QA of trainer outputs...Remote jobFor contractors10 hours per week
$30 per hour
HINDI AI EVALUATION SPECIALIST (SINGAPOREAN DIALECT) Up to $30/hr Depending on the project Bachelor's Remote Part-time Flexible hours Opportunity Overview Handshake is looking for Hindi speakers familiar with the Singaporean‑Hindi dialect to help make AI responses more...Part timeLocal areaRemote workFlexible hours$60 per hour
AuraOne is seeking a Consulting Expert for a remote evaluation track focused on Mandarin Chinese (mainland China). You will review model outputs against a versioned rubric, label edge cases, and provide structured feedback the modeling team can use to retrain. This independent...Remote jobHourly payFor contractors10 hours per week- Synthires is seeking evaluators to review AI-generated responses, judging quality and reasoning while maintaining high standards of accuracy. You will craft clear, well-supported rationales in a fully remote setting. The role emphasizes independent work, strict adherence...Remote jobHourly payFlexible hours
- Gaddr is seeking a detail-oriented AI Image Evaluation Specialist for a remote, part-time contractor role. You will upload your own images and compare them to AI-generated versions to identify visual differences and artifacts. Strong writing skills and patience are essential...Remote jobPart timeFor contractors
- Mercor is hiring experienced music producers and audio engineers to evaluate generative music AI models, in partnership with a leading AI lab. You will assess AI-generated music across genres and rate it against detailed quality standards, working in Spanish (MX) and English...
- Aitrainergigs is seeking independent Audio Evaluation Specialists for an AI benchmark project. You will autonomously design evaluation frameworks, conduct role-play-based simulations, and audit AI outputs to generate high-quality audio datasets. Flexibility, attention...Remote job
- AuraOne is seeking a remote Consumer Reviewer to evaluate prompts and responses against our quality rubric. You will compare outputs, label edge cases, and craft structured feedback to help retrain the model. This contractor role requires strong written reasoning, attention...Remote jobFor contractors10 hours per week
- Invisible Technologies is seeking independent Audio Evaluation Specialists for an AI benchmark project evaluating agentic audio models. You will create and execute evaluation tasks, audit outputs, and help generate clean datasets based on realistic customer support scenarios...Remote jobHourly payFreelance
- DataAnnotation is seeking remote contributors to evaluate Icelandic translations and localization quality across real projects. Flexible,... ...identify errors, explain failures, and draft better answers to improve AI performance. No degree is required; you can demonstrate...Remote jobFor contractorsFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Evaluation Specialist. Be the first to apply!
