Remote AI Data Annotation Evaluator & Rubric Expert
AuraOne
New York, NY
- Remote job
AuraOne is seeking an AI Data Annotation Expert for a remote contractor role focused on evaluating AI data annotation outputs against a versioned rubric. You will compare paired responses, label edge cases, and provide structured feedback to retrain the model. Responsibilities include tagging inconsistencies, capturing ambiguous prompts, calibrating reviewer scores weekly, and reporting recurring failure modes to improve rubrics and training runs. #J-18808-Ljbffr AuraOne
Vacancy posted 9 hours ago
Similar jobs that could be interesting for youBased on the Remote AI Data Annotation Evaluator & Rubric Expert in New York, NY vacancy
- AuraOne is seeking a Remote Social Engineering Security Evaluator to review prompts and responses against our quality rubric, producing auditable labels, rationales, and regression cases... ...with the team to improve training data. Strong written reasoning and reliable weekly...Remote jobFor contractors
$80 - $120 per hour
...technical talent with leading AI research labs. Headquartered in... .../ vendor management Evaluator Type: Contract Compensation... ...$80–$120/hour Location: Remote Role Responsibilities Evaluate... ...domain-specific quality rubrics. Identify factual, aesthetic...Remote workContract workSummer workWork at officeMercor
Dallas, TX4 days ago$80 - $120 per hour
Role Description ~Evaluate AI-generated artifacts against domain-specific quality rubrics. ~Identify factual, aesthetic, and presentation errors in documents, spreadsheets... ...~Compensation: $80–$120/hour ~Location: Remote Application Process ~Upload resume ~...Remote workPart timeWork at officeMercor
Remotea month ago- ...zone is seeking a Senior AI data annotator to perform data labeling and evaluation for NLP tasks, RLHF... ...performance. This is a US-based remote, full-time role aligned... ...You will apply strict rubrics, document edge cases,... ...background in data annotation and familiarity with RLHF...Remote jobFull time
- AuraOne is seeking a Remote Receipt and Invoice Understanding Model Evaluator to review outputs against rubrics and provide structured feedback. You will compare paired responses, label edge cases, and contribute to retraining with rationale. Ideal candidates have prior...Remote jobFor contractors10 hours per week
$20 - $80 per hour
...Overview Train and evaluate next-generation AI systems by... ...model outputs, annotating real-world content... ...micro1 is an AI data lab that converts... ...frontier models. Experts contribute... ...using well-defined rubrics and metrics across... ...collaboratively in a fully remote environment....Remote workHourly payFor contractorsSaidGig
United States14 days ago- AuraOne is seeking a Rubrics Expert to remotely review AI outputs and evaluate them against established rubrics, policies, and workflows. You will identify gaps, rate tone and escalation logic, and document next steps to guide model training. As a remote contractor eligible...Remote jobFor contractors10 hours per week
- AuraOne is seeking a remote Capability Elicitation Risk Evaluator to review prompts and model outputs. You will apply a versioned rubric to label edge cases, assign severity, and provide structured feedback for retraining. You will compare paired responses, select the stronger...Remote job
- AuraOne is seeking a remote Video Understanding AI Evaluator contractor to review model outputs, apply a structured rubric, and provide rationale. You will compare responses, tag issues, and help refine prompts to improve reliability across batches. The role emphasizes...Remote jobFor contractors
- AuraOne is seeking an Educational Safety AI Evaluator for a remote, contractor-based role. You will... ...safety AI outputs, apply the quality rubric, and produce auditable labels, rationales... ...and regression cases for AuraOne Human Data. Responsibilities include comparing responses...Remote jobFor contractors
- A tech company focusing on AI research is looking for experienced Krita users for a flexible... .... This role allows you to earn while evaluating AI-generated content related to digital... ...communication skills. Work is fully remote and scheduled flexibly, making it ideal for...Remote jobContract workFlexible hours
- Handshake AI is seeking experienced CAD professionals with 2+ years of hands-on experience using SolidWorks and related CAD software,... ...contract work. This ongoing, project-based opportunity is fully remote and asynchronous, with no minimum hour requirement; many contributors...Remote jobHourly payContract workFlexible hours
$150 per hour
...technical talent with leading AI research labs. Headquartered... ...: Investment Banking Expert Type: Contract Compensation... ...0–$220/hour Location: Remote Role Responsibilities... ...experience with AI training , evaluation, or human-data projects. Application...Remote workContract workSummer workMercor
Dallas, TX4 days ago$80 - $150 per hour
...technical talent with leading AI research labs.... ...Position: UI / UX Design Expert Type: Contract... ...0/hour Location: Remote Role Responsibilities... ...and asynchronously to evaluate and improve AI model performance... ..., evaluation, or human-data projects....Remote workContract workSummer workMercor
Pennsauken, NJ4 days ago$80 - $120 per hour
...technical talent with leading AI research labs. Headquartered... ...Position: Software / AI / IT / data Evaluator Type: Contract... ...80–$120/hour Location: Remote Role Responsibilities Evaluate... ...domain-specific quality rubrics. Identify factual, aesthetic...Remote workContract workSummer workWork at office$40 per hour
Prolific is seeking an AI Trainer - Neuroscience Specialist for a remote advisory role. This flexible position pays $40+/h and requires deep expertise in neuroscience, along with advanced academic credentials. Your main tasks include reviewing scientific papers and ensuring...Remote jobFlexible hours$30 per hour
...Prolific is not just another player in the AI space – we are building the biggest pool of quality human data in the world. About Prolific... ...Graphic and Visual Designers to act as Domain Experts for a high-level AI evaluation project. AI models are evolving beyond simple...Remote workWork from homeFlexible hoursProlific
New York, NY2 days ago$50 - $100 per hour
DataAnnotation is seeking an experienced Legal Expert to help train AI models. You will tackle diverse legal problems, measure chatbot reasoning, and improve model quality from home on a flexible schedule. This independent contractor role pays hourly from $50 to $100+ per...Remote jobHourly payFor contractorsFlexible hours$20 per hour
...A leading data annotation firm is seeking a Web Designer to evaluate AI-generated designs and enhance their understanding of design principles. This remote position offers flexibility in project selection and schedule. Ideal candidates will have a strong background in...Remote workDataAnnotation
Wyoming, OH4 days ago$60 - $70 per hour
...and technical talent with leading AI research labs. Headquartered in San... ...Compensation: $60–$70/hour Location: Remote Role Responsibilities Evaluate AI-generated responses for safety... .... Apply and refine evaluation rubrics for RLHF , SFT , and AI safety...Remote workContract workSummer work- ...Description Job Title: AI Software Engineering Domain Remote Job Type:... ...Engineering Domain Experts to contribute their... ...role, you will evaluate, review, and refine... ...outputs. Conduct rubric-based evaluations... ...Interpret and annotate technical data to support AI model...Remote workPart timeFor contractors
YO AI Labs
Dallas, TX4 days ago $80 - $120 per hour
...technical talent with leading AI research labs. Headquartered in... ...Position: Legal / compliance Evaluator Type: Contract... ...$80–$120/hour Location: Remote Role Responsibilities Evaluate... ...against domain-specific quality rubrics. Identify factual, aesthetic...Remote workContract workSummer workWork at office$120 - $175 per hour
...technical talent with leading AI research labs. Headquartered in... .... Position: CNC Machining Expert Type: Contract... ...120–$175/hour Location: Remote Role Responsibilities Contribute... ...text-only, objective, verifiable rubrics for grading CNC machining...Remote workContract workSummer work$80 - $120 per hour
...technical talent with leading AI research labs. Headquartered in... .../ environmental science Evaluator Type: Contract Compensation... ...$80–$120/hour Location: Remote Role Responsibilities... ...against domain-specific quality rubrics. Identify factual, aesthetic...Remote workContract workSummer workWork at office- ...Location: Remote | Work from Home Employment Type: Project-based | Contract We are looking for detail-oriented Image Quality Evaluator for a multilingual AI data Annotation and Transcription Specialists with strong proficiency in English. In this role, you will support...Remote workContract workWork from homeMonday to FridayDay shift
iMerit Technologies
San Jose, CA29 days ago - ...Assessment Rubric AI Evaluator is a remote review track for evaluating AI outputs across assessment rubric ai specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document the right next...Remote jobHourly payFor contractorsWork experience placement10 hours per week
- ...Position Summary This is a fully remote, hourly contractor role supporting AI data and language projects on a... ...learning across diverse topics. Data annotation: labeling, classifying, and... ...support AI training datasets. LLM evaluation: reviewing AI-generated...Remote workHourly payFor contractorsFlexible hours
CNTXT AI
Brooklyn, NY7 days ago - ...Join Sigma.AI – Shaping the Future of Artificial Intelligence... ...company specializing in data collection and annotation for Artificial Intelligence... ...– Transcription – Evaluation – Conversational interactions... ...words. All tasks are remote, performed through an online...Remote workContract workFreelance
Sigma AI
United States19 hours ago - CNTXT AI is seeking a fully remote, hourly contractor to support AI data and language projects on a flexible, project-based schedule. The role involves content generation, data annotation, LLM evaluation, and localization QA across diverse topics. Ideal candidates are native...Remote jobHourly payFor contractorsFlexible hours
- ...Description Job Title: AI Consulting Domain Remote Job Type:... ...AI Consulting Domain Experts to contribute their... ...this role, you will evaluate, review, and refine... ...quality assurance and rubric-based evaluations of... ...generated outputs. Annotate data, interpret findings,...Remote jobPart timeFor contractors
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Remote AI Data Annotation Evaluator & Rubric Expert. Be the first to apply!
Related searches
- program evaluator New York, NY
- work from home web search evaluator New York, NY
- clinical evaluator New York, NY
- evaluator New York, NY
- ai evaluator New York, NY
- work from home social media evaluator New York, NY
- quality evaluator New York, NY
- education evaluator New York, NY
- social media evaluator New York, NY
- data specialist New York, NY