Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Trainer & Evaluator

Temporary

AuraOne

Category: Frontier Model Evaluation · Pay: Hourly rate confirmed after the interview process · Location: Remote — US-eligible · Contractor

AI Trainer & Evaluator is a remote evaluation track for reviewing ai trainer evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain.

Why this role matters

AI data reviewers help turn ai trainer evaluation outputs into auditable labels, rationales, and regression cases for AuraOne Human Data.

Responsibilities

  • Evaluate ai trainer evaluation model outputs against a versioned rubric and assign severity tags for AI Trainer & Evaluator assignments.
  • Compare paired responses and pick the stronger answer with a written rationale.
  • Label hallucinations, instruction-following failures, and unsafe content with structured tags.
  • Capture ambiguous prompts and route them back to the program team for rubric updates.
  • Maintain reviewer-quality scores by calibrating against gold-standard examples each week.
  • Document recurring failure modes so the modeling team can target them in the next training run.

Qualifications

  • Prior evaluation, annotation, or human-rater experience on ai trainer evaluation or adjacent content for AI Trainer & Evaluator work.
  • Comfort applying multi-page rubrics consistently across long batches.
  • Clear written reasoning that names the issue and the rubric clause being applied.
  • Strong attention to detail and the ability to flag when a prompt itself is the problem.
  • Reliable async availability for at least 10 hours per week.

Example tasks

  • Compare two ai trainer evaluation model responses to the same prompt and pick the stronger one with rationale.
  • Tag an unsafe response with the correct policy category and severity.
  • Audit a 50-row batch for rubric consistency and report drift to the program lead.
  • Propose a rubric clarification after spotting a recurring failure mode.

Nice to have

  • Background in linguistics, content moderation, or trust & safety review.
  • Experience with inter-rater agreement metrics and calibration cycles.
  • Domain expertise that lets you spot subject-matter errors automated checks miss.

Skills

  • Model output evaluation
  • Rubric-based annotation
  • Severity tagging
  • Inter-rater calibration
  • AI Trainer evaluation
  • AI model evaluation

Work model

Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.

Compensation

Hourly rate confirmed after the interview process.

Application process

Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.

Vacancy posted a month ago
Similar jobs that could be interesting for youBased on the AI Trainer & Evaluator in San Jose, CA vacancy
  • $80 - $120 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Jack Dorsey . Position: Data analysis / quantitative readouts Evaluator Type: Contract Compensation: $80–$120/hour... 
    Suggested
    Contract work
    Summer work
    Work at office
    Remote work

    Mercor

    San Jose, CA
    3 days ago
  • $75 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...mathematics , or biology to enhance AI model training and evaluation. Apply structured reasoning and quantitative methods to improve... 
    Suggested
    Contract work
    Summer work
    Internship
    Remote work
    Flexible hours

    Mercor

    San Jose, CA
    a month ago
  • CXG, a global leader in customer experience, seeks luxury brand evaluators to discreetly assess shopping experiences across premium brands. Choose assignments, visit stores or evaluate online from home, and provide detailed feedback to help brands refine service. The role... 
    Suggested
    Flexible hours

    CXG

    Santa Clara, CA
    2 days ago
  • $170.6k

     ...Job Details: Job Description: The Role and Impact As an AI Software Engineering PhD Intern, you will contribute to the design, development, and optimization of cutting-edge AI software solutions across domains such as computer vision, machine learning, and deep learning... 
    Suggested
    Internship
    Local area
    Immediate start
    Worldwide
    Shift work

    Intel

    Santa Clara, CA
    1 day ago
  • $40 - $125 per hour

     ...Job Description About the role \n DataAnnotation is looking for experienced drug discovery scientists to evaluate how frontier AI models handle real discovery and preclinical work: SAR analysis, screening cascades, DMPK and PK/PD interpretation, biomarker strategy... 
    Suggested
    Contract work
    Work at office
    Remote work
    10 hours per week
    Flexible hours

    DataAnnotation

    Mountain View, CA
    4 days ago
  • $20 - $55 per hour

     ...Inworld is a research lab and inference provider focused on realtime AI for consumer-facing applications. We build first-party speech...  ...findings that engineers can act on For multilingual roles: evaluate pronunciation, dialect accuracy, and naturalness in your native... 
    Hourly pay
    For contractors
    Remote work

    Inworld, Inc.

    Mountain View, CA
    5 days ago
  • DataAnnotation seeks an experienced Radiologist or credentialed imaging professional to help train AI models by reviewing de-identified studies and writing structured reports documenting every finding. You will provide clear impressions and recommend next steps, such as... 

    DataAnnotation

    Mountain View, CA
    3 days ago
  • $125 per hour

    We’re looking for an experienced Radiologist to help train AI models. You’ll measure how well frontier models interpret medical images, check their reasoning, and help raise the quality of each new version. This is a new and growing area: teaching AI to read diagnostic... 
    Full time
    Remote work
    Flexible hours

    DataAnnotation

    Sunnyvale, CA
    4 days ago
  •  ...Job DescriptionJob Description Indent : PSL307525-6Role : Senior Data Scientist / AI EngineerLocation : RemoteSalary: $120k to $125kIMP: - Skills - Agentic AI (A2A , MCP) , Azure Cloud , healthcare -payer space experience.Exp : 10 Job description:-We're looking for a... 

    PeopleNTech

    Santa Clara, CA
    2 days ago
  • The CompanyPayPal has been revolutionizing commerce globally for more than 25 years. Creating innovative experiences that make moving money, selling, and shopping simple, personalized, and secure, PayPal empowers consumers and businesses in approximately 200 markets to ...
    Full time
    Work at office
    Local area
    Immediate start
    Remote work
    Flexible hours

    PayPal

    San Jose, CA
    3 days ago
  • Contractor is expected to support a range of activities associated with dashboarding and analytics. The nature of the work tends to fall into two categories: Complementary detailed analytical and/or critical documentation development work supporting an inflight project...
    For contractors

    TechDigital Group

    San Jose, CA
    4 days ago
  • Job DescriptionJob Description Primary Skill : DataBricks, NumPy, Python Job Description : Job Requirements Role : Data Scientist Experience : 4 Years Job Description:" 4 years of primary, hands-on experience in Data Science and Predictive Modeling, with a strong foundation...
    Work experience placement

    Diversity Nexus

    San Jose, CA
    1 day ago
  • Job DescriptionJob Description We are seeking a highly skilled Data Scientist for a hybrid contract role. This position requires a professional who can bridge the gap between complex data modeling and business strategy. The ideal candidate will be present in the office...
    Contract work
    Work at office
    3 days per week

    Macpower Digital Assets Edge Private Limited

    San Jose, CA
    2 days ago
  • Data ScientistLocation: RemoteDuration: Long Term10 years of experience and 5 years of work experience in data scientist discipline in FMCG/CPG/Retail industry.Hands-on expertise in building advanced analytics/machine learning models for price elasticity, promo optimization...
    Work experience placement

    Rootshell Inc

    Santa Clara, CA
    1 day ago
  • Data ScientistWe are seeking a seasoned Data Scientist to collaborate closely with cross-functional teams. Your role will be pivotal in shaping our product strategy using robust scientific methodologies. Leveraging your expertise in statistical analysis, machine learning...

    Aven, Inc.

    Campbell, CA
    1 day ago
  • $145k - $180k

     ...systems (UAS), aviation-related physical artificial intelligence (AI) solutions, and other technologies to customers worldwide across...  ...are not directly observable in the raw telemetry. Build, evaluate, and iterate on models across diverse modalities, including dense... 
    Local area
    Worldwide
    Visa sponsorship
    Night shift

    Archer

    San Jose, CA
    2 days ago
  • $248k - $379.5k

     ...experience even at high resolutions and fps and at industry leading low latencies. Our team is building the Diagnostic, Prescriptive and AI-augmented Analytics solutions that encompass processing, visualization, anomaly detection, root cause and predictive modeling for the... 
    Full time
    Remote work

    NVIDIA

    Santa Clara, CA
    2 days ago
  •  ...Company Overview TENEX is an AI-native, automation-first, built-for-scale Managed Detection and Response (MDR) provider. We are...  ...cyber threats. Design and implement A/B testing and evaluation frameworks to measure the impact and performance of deployed models... 
    Work at office

    TenEx

    San Jose, CA
    5 days ago
  • $152k - $264.5k

     ...NVIDIA is hiring a Senior Data Scientist for its Finance Data Science and AI team in Santa Clara, CA (onsite) . In this role, you will integrate Data Science and AI solutions into financial workflows, helping translate analytics into measurable outcomes for both... 

    NVIDIA

    Santa Clara, CA
    3 days ago
  • $184.7k - $324.8k

     ...Role Number: 200631850 Team: Machine Learning and AI Location: Cupertino; Seattle Posted: 2025-11-12 Summary Do you get excited by driving product impact via measurement and evaluation, for products and services used by hundreds of millions of people globally? Within... 
    Work experience placement
    Relocation

    Apple

    Cupertino, CA
    1 day ago
  • $131.7k - $219.5k

     ...Scientist to join our AMI team. This hands‑on role combines Machine Learning, statistical modeling, Generative AI, Retrieval‑Augmented Generation (RAG), model evaluation, and applied AI experimentation for enterprise AI applications. AMI supports commercial AI providers and... 

    BMC Software

    Santa Clara, CA
    3 days ago
  • $170k - $225k

     ...outliers, and inconsistencies. Labeling datasets appropriately for classification problems. Machine Learning: Build, train, and evaluate custom machine learning models for security use-cases such as threat/vulnerability detection, including traditional algorithms... 
    Work at office
    Remote work
    Flexible hours

    Versa Networks

    Santa Clara, CA
    5 days ago
  • $156k - $204.23k

     ...support isnt available at this time. About this opportunity The AI Innovation & Incubation (AIII) team in Silicon Valley is a group...  ...fast-moving field. What you will do Design, train, and evaluate deep neural networks — including vision-language models (VLMs)... 
    Temporary work
    Work at office
    Relocation package

    Ericsson

    Santa Clara, CA
    3 days ago
  •  ...scrap.Develop and apply statistical, predictive, and prescriptive models using complex, high-volume manufacturing data.Design and evaluate experiments, including DOE and controlled analyses, to understand process drivers and validate improvement actions.Apply regression... 

    Opportunities

    San Jose, CA
    1 day ago
  • Santa Clara University is seeking an On-Call Field Supervisor to support, evaluate, and guide teacher candidates within the SCU credential program. The role involves coordinating with cooperating teachers, reviewing lesson plans, and ensuring alignment with program expectations... 

    Santa Clara University

    Santa Clara, CA
    4 days ago
  • Job Description Job Description Benefits: Bonus based on performance Opportunity for advancement Full-time Real Estate Appraiser position available at a small Silicon Valley firm. Must be licensed or certified appraiser. Can work while attaining California...
    Full time
    Work at office
    Work from home
    Flexible hours

    Accurate Appraisal & Review

    San Jose, CA
    more than 2 months ago
  •  ...Job Title AI Software Engineer Intern Date posted 14-Oct-2026 Job ID 137214 City / Township / Village POUGHKEEPSIE, DURHAM, San Jose, New York State / Province New York, North Carolina, California Country United States Work arrangement... 
    Contract work
    Fixed term contract
    Internship
    Shift work

    IBM

    San Jose, CA
    2 days ago
  •  ...Introduction Join the pioneering AI Foundations Research team at IBM Research and contribute to shaping the future foundations of artificial intelligence. Our group of scientists, engineers, and designers is dedicated to conducting end-to-end research that delivers real... 
    Full time
    Contract work
    Part time
    Fixed term contract
    Internship
    Shift work

    IBM

    San Jose, CA
    4 days ago
  • $38 - $94 per hour

     ...of our many internship opportunities. NVIDIA pioneered accelerated computing to tackle challenges no one else can solve. Our work in AI and digital twins is transforming the world's largest industries and profoundly impacting society - from gaming to robotics, self-... 
    Hourly pay
    Internship
    Summer internship

    NVIDIA

    Santa Clara, CA
    1 day ago
  • $55.56 per hour

     ...so they could focus on meaningful work. Today, ServiceNow is the AI control tower for business reinvention. Our ServiceNow AI...  ...use AI coding assistants to accelerate delivery while critically evaluating AI-generated output for correctness and security. You’ll also collaborate... 
    Full time
    Summer work
    Internship
    Work at office
    Immediate start
    Remote work
    Flexible hours

    ServiceNow

    Santa Clara, CA
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Trainer & Evaluator. Be the first to apply!