AI Benchmark and Task Designer [Remote]
AuraOne Human Data
Remote
- Remote job
AI Benchmark and Task Designer is a remote review track for evaluating AI outputs across ai benchmark and task designer creative review workflows. Reviewers grade craft, constraints, and tooling adherence; flag production-readiness issues; and document the corrected approach so the modeling team can train on it.
Why this role matters
AI Benchmark and Task Designer creative review AI is only useful if a working professional would actually ship the output. AuraOne uses portfolio-credentialed reviewers to grade outputs the way a senior creative would — with craft, constraints, and tooling all in scope.
Responsibilities
- Review AI outputs against current ai benchmark and task designer creative review craft and tooling standards for AI Benchmark and Task Designer assignments.
- Grade composition, constraint adherence, and production-readiness on a structured rubric.
- Flag tooling-specific failures (file format, scale, naming, layer hygiene) with severity tags.
- Capture the corrected approach or reference so the modeling team can train on it.
- Adjudicate disputed creative judgments against published references or firm style guides.
- Maintain reviewer-quality scores in inter-rater calibration cycles.
Qualifications
- Portfolio or production experience in ai benchmark and task designer creative review or an adjacent creative discipline for AI Benchmark and Task Designer work.
- Hands-on familiarity with the standard tooling for the discipline.
- Comfort applying multi-page rubrics consistently across long batches.
- Clear written reasoning that names the craft issue and the standard being applied.
- Reliable async availability for at least 10 hours per week.
Example tasks
- Grade a model's ai benchmark and task designer creative review output against the brief and rate craft, constraints, and production-readiness.
- Flag a tooling-specific failure (incorrect file format, unsupported feature, naming convention) with severity.
- Adjudicate a disputed creative judgment between two reviewers using a firm style guide.
- Audit a 25-asset batch for rubric consistency and report drift to the program lead.
Nice to have
- Studio, agency, or in-house team experience shipping production work.
- Prior experience reviewing AI-assisted creative tooling and its failure modes.
- Familiarity with AAA or feature-grade pipelines and asset standards.
Skills
- Creative review
- Production-grade craft
- Tooling fluency
- Asset evaluation
- AI Benchmark and Task Designer creative review
- Cross-cutting evaluation roles
- AI evaluation
- Rubric writing
- Expert review
Work model
Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
Vacancy posted 22 days ago
Similar jobs that could be interesting for youBased on the AI Benchmark and Task Designer [Remote] in Remote vacancy
- Obsidian is seeking an Electrical Engineering & RF/Circuit Design Expert to design graduate-level problems for benchmarking AI research. You will build challenging tasks using scikit-rf or ngspice, run simulations, interpret results, and craft experiments that reveal system...SuggestedRemote work
- ...high-quality academic assessment content for an AI research initiative. You will write and verify... ...quality, and help establish gold-standard benchmarks used to advance AI capabilities. You will be assigned one of two task types: question authoring or question verification...SuggestedRemote jobPart time10 hours per week
- ...strong English comprehension to broken down complex problems for evaluation tasks. You will collaborate with LLM researchers on benchmarks spanning undergraduate to PhD topics, enabling cutting-edge AI projects while working independently in a fully remote #J-18808-Ljbffr...SuggestedRemote jobContract work
- OpenTrain AI is seeking a Finance and Accounting Task Author to design expert-level accounting reconciliation tasks that train and evaluate advanced AI agents. You will create numerically rigorous scenarios, write scoring rubrics, set up task environments, and validate...SuggestedRemote jobPart timeFor contractors
$50 per hour
A leading AI research accelerator seeks PhDs in Chemistry or related fields for a fully remote role. You will design advanced chemistry problems and evaluate AI performance. The position... ...and collaborating with researchers on benchmarks. No AI experience is required. Join a...SuggestedRemote jobHourly payFlexible hours- OpenTrain AI is seeking a CFD AI Task Designer to create compact, self-contained engineering challenges for training and evaluating AI models. You will develop problems involving boundary-condition selection, mesh quality assessment, flow-regime identification, and simulation...Remote jobPart timeFor contractorsFreelanceFlexible hours
$50 - $70 per hour
...firms. Responsibilities include building digital workspaces for environmental data and collaborating with experts to refine AI research benchmarks. Candidates with a MS/PhD in environmental science or related fields are preferred. Strong analytical and writing skills...Remote jobHourly pay$80 - $110 per hour
About OpenTrain OpenTrain AI is the hiring and contracting organization for this role and... ...technical expertise Use flexible, task-based work to build experience in a fast-growing... ...Role OpenTrain AI is seeking a CFD AI Task Designer to create compact, self-contained...Hourly payOngoing contractPart timeFor contractorsFreelanceRemote workFlexible hours- Alignerr is building a dataset of expert tasks to train and evaluate AI agents in real enterprise support. As a Task Author for Customer Support / Success, you will design end-to-end support triage tasks—from reading a ticket to deciding between fix and escalation—and...Remote jobHourly payContract workFlexible hours
- Mercor is seeking a Computational Chemistry & Electronic Structure Expert to design graduate-level problems for AI benchmarks. You will use real scientific software (e.g., PySCF) to run simulations, interpret results, and design experiments that reveal hidden information...Remote work
$50 per hour
A leading AI research accelerator is seeking remote contractors with PhDs in Chemistry or related fields to work on cutting-edge AI projects. Responsibilities include designing advanced chemistry problems for AI models, developing solutions, and collaborating with researchers...Remote jobFor contractorsFlexible hours- Mercor is seeking a Particle & Nuclear Physics Expert for a remote contract role. You will design and refine AI-evaluative problems in Particle & Nuclear Physics, using scikit-hep and related Python tools for data analysis and simulations. Applicants should have graduate...Remote jobContract work
$60 - $90 per hour
24-MAG LLC seeks professionals to design research-grade computational problems in astrophysics and cosmology. You will craft tasks using astropy, cosmological calculations, and... ...and develop Python-based validation and benchmark workflows. The role is a fully remote, independent...Remote jobHourly payFor contractors$50 per hour
A leading AI research company is seeking remote PhDs in Chemistry to work on cutting-edge projects with top AI labs. The role involves designing tests for AI performance and collaborating with researchers. Candidates should have strong problem-solving skills and the ability...Remote jobFlexible hours$50 per hour
A leading AI research accelerator in San Francisco is looking for PhD candidates in Chemistry or related fields for a fully remote contract role. You will design and evaluate advanced chemistry problems to enhance AI performance, collaborating with researchers. Strong problem...Remote jobContract workFlexible hours$50 per hour
...accelerator is seeking remote experts with a PhD in Biology, Biotechnology, Biochemistry, or related fields. You will help fine-tune AI systems by designing advanced biology questions and evaluating AI-generated answers. This role offers a pay rate of $50+/hour, flexible working...Remote jobFlexible hours$50 per hour
A leading AI research accelerator in the United States is seeking PhDs in Chemistry or related fields for a remote role. You will help fine-tune large language models, designing advanced problems, evaluating AI outputs, and collaborating with researchers. Ideal candidates...Remote jobHourly payFlexible hours$50 per hour
A leading research accelerator for AI is offering a remote contract for PhDs in Chemistry or related fields. The role involves fine-tuning large language models using chemistry knowledge, designing advanced problems, and evaluating AI outputs. Ideal candidates possess...Remote jobContract workFlexible hours$50 per hour
A leading research accelerator for AI labs is seeking skilled PhDs in Mathematics or related fields for a fully remote position. You will help fine-tune AI models by designing advanced math problems and evaluating AI performance. The role offers a pay rate of $50+/hour...Remote jobFlexible hours- A leading AI research company is searching for highly skilled Biomedical Informatics Subject Matter Experts to design rigorous evaluation questions for advanced AI models. Ideal candidates must have deep expertise and experience in biomedical data processing, knowledge...Remote jobContract work
$50 per hour
A leading AI research accelerator is seeking remote PhD candidates in Mathematics or related fields to design math problems and evaluate AI performance. The role involves collaboration with researchers and offers flexible hours at a pay rate of $50+/hour. Ideal candidates...Remote jobHourly payFlexible hours- Alignerr bouwt een dataset van expert taken die AI-agenten trainen en evalueren op echt bedrijfswerk. Deze volledig remote, flexibele contractrol is open voor ervaren support engineers en customer success professionals in Amsterdam en heel Nederland die willen vormgeven...Remote jobFreelance
$50 per hour
...remote PhD candidates in Biology, Biotechnology, or related fields. You will help fine-tune large language models by designing problems, evaluating AI outputs, and collaborating with researchers. The position offers flexibility and competitive pay of $50+/hour based on...Remote jobFlexible hours$50 per hour
A leading AI research accelerator is seeking PhDs in Mathematics or related fields for remote contract roles. You will work on cutting-edge AI projects, designing math problems and evaluating AI performance. Strong analytical skills and clear communication are essential...Remote jobContract workFlexible hours$50 per hour
A leading research accelerator for AI is seeking PhDs in Biology, Biotechnology, or related fields for a remote contract role. You will help fine-tune large language models, design advanced biology questions, and collaborate with researchers on cutting-edge AI projects....Remote jobContract workFlexible hours- ...company is seeking PhD candidates in Chemistry or related fields for a remote role. Responsibilities include designing advanced chemistry problems to test AI performance and evaluating outputs for accuracy. This position offers flexible work hours and the opportunity...Remote jobFlexible hours
$60 - $90 per hour
...creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam... .../week Role Responsibilities Design tasks by transforming real ML research ideas into...Hourly payWeekly payFull timeContract workFor contractorsSummer workRemote workMercor
Remote10 hours ago- ...UX Designer You will be responsible for the design and development... ..., and User Testing Conduct benchmarking against similar types of businesses... ..., wireframes, use cases, and task flows,) to be used by the team... ...iHorizons has evolved into an AI-centric company, investing in...Work experience placementRemote work
IHorizons
United States3 days ago $169k - $303k
...creatives and builders on a mission to make design accessible to all. Figma’s platform helps... ...designs into code, or iterating with AI. From idea to product, Figma empowers teams... ...a job, but successfully perform their job tasks to the same extent as people without disabilities...Minimum wageFull timeLocal areaRemote workFlexible hours$169k - $303k
...creatives and builders on a mission to make design accessible to all. Figma’s platform helps... ...designs into code, or iterating with AI. From idea to product, Figma empowers teams... ...linesExperience using AI models for design tasks, either coding or image modelsProven track...Minimum wageFull timeFor contractorsLocal areaRemote workFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Benchmark and Task Designer [Remote]. Be the first to apply!
