Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Evaluation Specialist

Crossing Hurdles

A leading AI organization in Australia is seeking individuals with strong writing and analytical skills to evaluate and improve AI outputs. The ideal candidate must possess the ability to assess emotional nuances and detail while adhering to structured guidelines. Responsibilities include ensuring clarity and empathy in evaluations, identifying weaknesses in AI reasoning, and contributing to a safe AI environment. Immediate availability post-onboarding is required for this role. #J-18808-Ljbffr

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Evaluation Specialist in New York, NY vacancy
  •  ...develop and map analytic cases to defined taxonomies across multiple evaluation areas. You will author complex test cases to evaluate model...  ...on source handling and analytic rigor. You will assess and grade AI-generated responses based on reasoning, source use, and... 
    Suggested

    Crossing Hurdles

    New York, NY
    1 day ago
  • AI Trainer Jobs is seeking an English (India) Recording Specialist to remotely evaluate prompts and responses against AuraOne’s rubric. You will compare paired outputs, label edge cases, and document structured feedback to help retrain the model. As a contractor, you will... 
    Suggested
    Remote job
    For contractors

    Auraone

    New York, NY
    2 days ago
  •  ...remote contract work. You will craft prompts, run side-by-side evaluations using ChatGPT and Claude, and provide thorough verbal feedback...  ...emphasizes consistent evaluation standards and collaboration with AI model improvement efforts. Ideal candidates have strong English... 
    Suggested
    Remote job
    Contract work

    Crossing Hurdles

    New York, NY
    2 days ago
  • $25 per hour

     ...seeking a Remote Personal Account Generalist Rater (Mobile) on a contract basis. The role involves reviewing AI-generated prompts and responses in English, evaluating accuracy and relevance, and providing structured feedback to support AI system improvements. The position... 
    Suggested
    Remote job
    Hourly pay
    Contract work
    Flexible hours

    Crossing Hurdles

    New York, NY
    2 days ago
  • OpenTrain AI, Inc. is seeking individuals with deep physics knowledge to help improve AI systems through rigorous question design and evaluation. You will craft challenging physics questions and assess AI-generated answers for accuracy and validity. The role is an asynchronous... 
    Suggested
    Remote job
    Hourly pay
    Contract work
    Flexible hours

    OpenTrain AI, Inc.

    New York, NY
    1 day ago
  •  ...tests that isolate model capabilities and create robust rubrics for scoring. You will develop reference answers, edge cases, and evaluate model outputs against consistent standards to improve benchmarks. The role emphasizes collaboration, documentation, and trend analysis... 
    Remote job
    Flexible hours

    Clera

    New York, NY
    1 day ago
  •  ...Experts in project management to design Excel-based tasks that test AI's ability to perform real PM work. You bring 3+ years in PMO...  ...infrastructure, or IT projects. You will craft tasks, write precise evaluation rubrics, and work as an independent contractor on a fully remote... 
    Remote job
    Weekly pay
    For contractors

    Aptura

    New York, NY
    3 days ago
  • $30 per hour

    Evaluate and rank AI-generated responses in Singaporean Malay, explain your choices, and provide written feedback. This remote, part-time project is open to candidates in the United States and pays up to $30 per hour. You will review AI-generated responses and help improve... 
    Hourly pay
    Part time
    For contractors
    Local area
    Remote work

    OpenTrain AI, Inc.

    New York, NY
    2 days ago
  • Anyone AI is seeking remote independent contractors to help create and evaluate data for frontier AI models. You will interact with AI models and compare responses to ensure quality and reliability. Responsibilities include assessing accuracy, helpfulness, and instruction... 
    Remote job
    For contractors

    Anyone AI Inc.

    New York, NY
    16 hours ago
  • AuraOne seeks a video-related professional to remotely evaluate AI training outputs against AuraOne's quality rubric. You will review video prompts and responses, compare paired outputs, label edge cases, and write structured feedback for the modeling team. Independent... 
    Remote job
    Hourly pay
    Part time
    For contractors

    Auraone

    New York, NY
    2 days ago
  • AuraOne, Inc. seeks an AI Image Specialist (US Based) for remote image evaluation. You will review prompts and responses, apply a versioned rubric, and provide structured feedback to guide retraining. The role emphasizes careful labeling, rationale, and consistency across... 
    Remote job
    For contractors
    10 hours per week

    AuraOne Human Data

    New York, NY
    3 days ago
  • AuraOne, Inc. is seeking a remote evaluation specialist to review agentic task completion model outputs, apply a structured rubric, and provide rationales for retraining. The role focuses on tagging edge cases, formatting auditable feedback, and maintaining calibration... 
    Remote job
    For contractors
    10 hours per week

    AuraOne Human Data

    New York, NY
    2 days ago
  •  ...American English. You will also interpret guidelines to ensure consistent evaluation and contribute to ongoing model improvement. The role requires strong English proficiency, familiarity with AI evaluation practices, and a reliable technical setup for screen #J-18808-... 
    Remote job
    Contract work
    Part time

    Crossing Hurdles

    New York, NY
    2 days ago
  • AuraOne, Inc. is seeking a Senior AI Trainer evaluation contractor based in the United States to support remote evaluation tasks. You will review senior AI trainer prompts and responses, label edge cases, and provide structured feedback to retrain models. Responsibilities... 
    Remote job
    For contractors
    10 hours per week

    AuraOne Human Data

    New York, NY
    3 days ago
  • AI Image Specialist (US Based) - Remote Image Evaluation is a remote evaluation track for reviewing ai image evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback... 
    Remote job
    Hourly pay
    For contractors
    10 hours per week

    AuraOne Human Data

    New York, NY
    3 days ago
  • OpenTrain AI, Inc. is hiring for a worldwide remote, part-time contract role focused on creating challenging prompts and evaluating language model responses. Candidates should bring strong English writing, analytic thinking, and team-management experience to assess accuracy... 
    Remote job
    Contract work
    Part time
    Worldwide
    Flexible hours

    OpenTrain AI, Inc.

    New York, NY
    4 days ago
  • Mercor is seeking experienced music producers and audio engineers to evaluate generative music AI models. You will assess AI-generated music across genres and rate it against detailed quality standards, working in Portuguese and English. Start date is immediate with duration... 
    Hourly pay
    Immediate start

    Mercor

    New York, NY
    1 day ago
  • AuraOne, Inc. is seeking a remote contractor to review Japanese generalist evaluation prompts and responses, applying AuraOne's quality rubric. You'll compare outputs, label issues, and generate structured feedback for model retraining. This role requires careful attention... 
    Remote job
    For contractors
    10 hours per week

    AuraOne Human Data

    New York, NY
    1 day ago
  • Synthires seeks analytical thinkers to contribute to advanced AI research and evaluation projects. In this role, you will evaluate AI-generated responses, assess quality and reasoning, and provide structured, detailed written feedback to improve next-generation AI systems... 
    Remote job

    Synthires

    New York, NY
    3 days ago
  • Agency is seeking a linguist with deep Indonesian language expertise to assist with advanced language model evaluation and data refinement. You will analyze grammar, semantics, and pragmatics, and help shape prompts and evaluation metrics as a contractor. This remote role... 
    For contractors
    Remote work

    Agency

    New York, NY
    2 days ago
  • AuraOne, Inc. is seeking an AI Trainer (Life Sciences) for a remote evaluation track. You will review ai trainer prompts and responses, label edge cases, and provide structured feedback to help retrain models. This is a remote contractor role focused on QA of trainer outputs... 
    Remote job
    For contractors
    10 hours per week

    AuraOne Human Data

    New York, NY
    3 days ago
  • $30 per hour

    HINDI AI EVALUATION SPECIALIST (SINGAPOREAN DIALECT) Up to $30/hr Depending on the project Bachelor's Remote Part-time Flexible hours Opportunity Overview Handshake is looking for Hindi speakers familiar with the Singaporean‑Hindi dialect to help make AI responses more... 
    Part time
    Local area
    Remote work
    Flexible hours

    Handshake

    New York, NY
    2 days ago
  • $60 per hour

    AuraOne is seeking a Consulting Expert for a remote evaluation track focused on Mandarin Chinese (mainland China). You will review model outputs against a versioned rubric, label edge cases, and provide structured feedback the modeling team can use to retrain. This independent... 
    Remote job
    Hourly pay
    For contractors
    10 hours per week

    Auraone

    New York, NY
    3 days ago
  • Synthires is seeking evaluators to review AI-generated responses, judging quality and reasoning while maintaining high standards of accuracy. You will craft clear, well-supported rationales in a fully remote setting. The role emphasizes independent work, strict adherence... 
    Remote job
    Hourly pay
    Flexible hours

    Synthires

    New York, NY
    2 days ago
  • Gaddr is seeking a detail-oriented AI Image Evaluation Specialist for a remote, part-time contractor role. You will upload your own images and compare them to AI-generated versions to identify visual differences and artifacts. Strong writing skills and patience are essential... 
    Remote job
    Part time
    For contractors

    Gaddr

    New York, NY
    2 days ago
  • Mercor is hiring experienced music producers and audio engineers to evaluate generative music AI models, in partnership with a leading AI lab. You will assess AI-generated music across genres and rate it against detailed quality standards, working in Spanish (MX) and English... 

    Mercor

    New York, NY
    1 day ago
  • Aitrainergigs is seeking independent Audio Evaluation Specialists for an AI benchmark project. You will autonomously design evaluation frameworks, conduct role-play-based simulations, and audit AI outputs to generate high-quality audio datasets. Flexibility, attention... 
    Remote job

    Meridial

    New York, NY
    1 day ago
  • AuraOne is seeking a remote Consumer Reviewer to evaluate prompts and responses against our quality rubric. You will compare outputs, label edge cases, and craft structured feedback to help retrain the model. This contractor role requires strong written reasoning, attention... 
    Remote job
    For contractors
    10 hours per week

    Auraone

    New York, NY
    2 days ago
  • Invisible Technologies is seeking independent Audio Evaluation Specialists for an AI benchmark project evaluating agentic audio models. You will create and execute evaluation tasks, audit outputs, and help generate clean datasets based on realistic customer support scenarios... 
    Remote job
    Hourly pay
    Freelance

    Invisible Technologies Inc.

    New York, NY
    2 days ago
  • DataAnnotation is seeking remote contributors to evaluate Icelandic translations and localization quality across real projects. Flexible,...  ...identify errors, explain failures, and draft better answers to improve AI performance. No degree is required; you can demonstrate... 
    Remote job
    For contractors
    Flexible hours

    DataAnnotation

    New York, NY
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Evaluation Specialist. Be the first to apply!