Data Annotator
AuraOne
About the role
Data Annotator is a remote evaluation track for reviewing data annotator evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain.
AI data reviewers help turn data annotator evaluation outputs into auditable labels, rationales, and regression cases for AuraOne Human Data.
Review frontier model outputs. Judge benchmark failures and calibrate other evaluators.
Responsibilities
- Evaluate data annotator evaluation model outputs against a versioned rubric and assign severity tags for Data Annotator assignments.
- Compare paired responses and pick the stronger answer with a written rationale.
- Label hallucinations, instruction-following failures, and unsafe content with structured tags.
- Capture ambiguous prompts and route them back to the program team for rubric updates.
- Maintain reviewer-quality scores by calibrating against gold-standard examples each week.
What you should bring
- Prior evaluation, annotation, or human-rater experience on data annotator evaluation or adjacent content for Data Annotator work.
- Comfort applying multi-page rubrics consistently across long batches.
- Clear written reasoning that names the issue and the rubric clause being applied.
- Strong attention to detail and the ability to flag when a prompt itself is the problem.
- Reliable async availability for at least 10 hours per week.
Role signals
Example tasks
- Compare two data annotator evaluation model responses to the same prompt and pick the stronger one with rationale.
- Tag an unsafe response with the correct policy category and severity.
- Audit a 50-row batch for rubric consistency and report drift to the program lead.
- Propose a rubric clarification after spotting a recurring failure mode.
Useful experience
- Background in linguistics, content moderation, or trust & safety review.
- Experience with inter-rater agreement metrics and calibration cycles.
- Domain expertise that lets you spot subject-matter errors automated checks miss.
Compensation and schedule
Hourly rate confirmed after the interview process.
Expected arrangement: contractor , with program-defined task volume and review pacing. Placement depends on current program demand and reviewer confirmation.
Skills used in matching
- Model output evaluation
- Rubric-based annotation
- Severity tagging
- Inter-rater calibration
- Data Annotator evaluation
Application boundary
Creating a specialist profile records your experience and preferences. Starting role intake is a separate action that attaches this role to your candidate record.
Specialist intake
The intake preserves your chosen role, the visible terms, and source attribution for reviewer context.
- 01 Confirm profile and eligibility details.
- 02 Attach this role deliberately.
- 03 Receive a human review decision or follow-up.
Placement timing depends on program demand and reviewer confirmation.
$30 per hour
...URGENTLY HIRING AI Audio Data Annotators | $30/hr | Start ASAP Looking to break into the AI industry? This is a great opportunity to gain hands-on experience with a leading AI company while working on technology that helps train next-generation voice and language...SuggestedWeekly payContract workTemporary workImmediate startDay shiftAfternoon shift$30 per hour
...URGENTLY HIRING 40-50 AI Audio Data Annotators | $30/hr | Start ASAP Looking to break into the AI industry? This is a great opportunity to gain hands-on experience with a leading AI company while working on technology that helps train next-generation voice and language...SuggestedWeekly payContract workTemporary workImmediate startDay shiftAfternoon shift$2,500 per month
...Deploying next-generation inference hardware at scale requires more than great chips - it demands world-class physical infrastructure. As a Data Center Engineer at Etched, you will own the end-to-end lifecycle of our data center and hardware lab environments: from facility...SuggestedWork at officeRelocation packageShift work$131.7k - $219.5k
...business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage. We're looking for an experienced Data...Suggested- ...users, groups, and memberships for the purpose of access control. Dashboard/Metric development. Additional services include: Validate data integrity and maintain projects, action items, checklists, and/or issues data attributes within the client's tool ecosystem. Lead...SuggestedFor contractors
- ...of ourselves, each other, and our communities. Job Summary: This job will lead the development and implementation of advanced data science models and algorithms. You will work with stakeholders to understand requirements and deliver solutions. Your role involves...Work at officeLocal areaImmediate startRemote workFlexible hours
- ...floor as we build the next-generation cybersecurity platform. We are expanding our engineering organization and seeking a Senior Data Scientist to design, build, and deploy the machine learning models that power our AI-driven cybersecurity platform. This role is...Work at office
$103k - $125k
...Job Description Job Description Data Center Infrastructure Engineer - (NVD) Locations: San Jose (NVD) Business Units: CESBG Job Classification: Exempt Summary Foxconn is seeking a highly skilled and motivated Data Center Infrastructure Engineer...Full timeTemporary workFor contractorsLocal area- ...Principal Data Scientist It's fun to work in a company where people truly BELIEVE in what they're doing! We're committed to bringing passion and customer focus to the business. If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy...
$129.2k - $209k
..., stronger relationships, and the kind of precision that drives great outcomes. Job Summary We are seeking a versatile Senior Data Scientist to join our team and help shape the future of our business operations. This role is a unique hybrid of classical machine learning...Full timeWork at office3 days per week$60 per hour
...Data Scientist V-6355 itD is seeking a Data Scientist V to drive data-informed decision-making by translating large-scale datasets into actionable insights, defining business objectives and success metrics, and developing analytical models that identify trends...Contract workTemporary workWork experience placement3 days per week$70 - $77 per hour
...Product Data ScientistOverview We are seeking a highly analytical and business-savvy Product Data Scientist to support high-impact digital payments teams. This role will use product and merchant data to identify growth drivers, evaluate performance across global markets...Contract workTemporary workH1b3 days per week$183k - $247.6k
...automation. It lets employees find answers across their company's data, apps, and the web, turn insights into action, and automate... ...programs from start to finish: write guidelines, measure inter-annotator agreement, and control data quality Analyze model and agent...Local areaFlexible hours- ...evaluation, for products and services used by hundreds of millions of people globally? Within the evaluation organization, the mission of Data Science and Insights team is to guide product development, decisions and directions through principled evaluation and rigorous use...Work experience placement
- ...scale, the analysis that turns them into judgments about autonomy behavior — including where on the map that behavior changes — the data and tooling that gate software releases, and the agentic workflows that take a detected issue from triage to a proposed fix. You will...
$152k - $230k
...inspired to do their best work. Come join the team and see how you can make a lasting impact on the world. Join NVIDIA as a Senior Data Scientist on our Finance Data Science and AI team, where you will work alongside Data Scientists, Data Engineers, AI developers to lead...Full time$72.96k - $115.2k
...What this Job Entails The Data Center Engineer III will be heavily involved in the design of the datacenter collocations, infrastructure hardware, and provide technical consultations. The incumbent will build/create all drawings for datacenter POP’s and develop/write...Temporary workWork at officeFlexible hours- ...Rootshell Enterprise Technologies Inc. is a recognized provider of professional IT Consulting services in the US. We are actively seeking Data Engineer for one of our client, Please share your resume with current location & full contact info Role:Data Engineer...
- ...Data Engineer Who You Are: Data Engineers are focused on enabling a data-driven approach to optimization by sourcing, maintaining and ensuring the availability of data used to drive full lifecycle marketing insights to optimize CX's marketing investments and...
$157k - $175.5k
...LatentView Analytics is a global dataanalytics and AI solutions company that partners with Fortune 500 andhigh-growth clients to turn data into competitive advantage. Our teams combinedeep technical expertise with business context to ship measurable outcomes -not just...$184.7k - $324.8k
...advertisers of all sizes, from small app developers to big global brands. Because when advertising is done right, it benefits everyone. The Data Products Engineering team is seeking versatile data engineers to build, extend, and operate the near-real-time and batch pipelines...Relocation$75 - $85 per hour
...Position: Battery Data Engineer III Location: Cupertino, California Duration: Contract Job ID: 171364 Job Description: This will be a data engineer role for processing battery testing data to facilitate battery algorithm delivery and support battery algorithm simulations...Full timeContract work- ...Data Engineer Comtech LLC is a woman-owned small business focused on delivering end-to-end solutions and products. Since 1998, we have successfully serviced enterprises across the public and private sectors, and the Department of Defense. Our services span all aspects...
- ...Data Engineer II Rootshell Enterprise Technologies Inc. is a recognized provider of professional IT Consulting services in the US. We are actively seeking Data Engineer II for one of our client. Role: Data Engineer II Location: Santa Clara, CA Duration: Long Term...
- ...Data Engineer Minimum 10 years of experience Responsibilities: Data Pipeline Development: Design, build, and maintain scalable, efficient, and reliable ETL/ELT data pipelines for batch and real-time processing using GCP services (e.g., Dataflow, Dataproc,...
$184.7k - $324.8k
...motivated by curiosity to keep making our products better. Apple Services Engineering's Device Compute Analytics team provides the data foundation for Apple's internal device cloud, which supports all of Apple's hardware, software, and services. Our work helps Apple ship...Relocation$135k - $180k
...autonomy, Plus is looking for talented individuals to join its fast-growing teams. Responsibilities Create and maintain optimal data processing pipeline architecture Build the infrastructure required for optimal extraction, transformation, and loading of data...- ...Roles and Responsibilities Designed, developed, and maintained scalable data pipelines to extract, transform, and load data from multiple sources. Developed ETL/ELT workflows using Python, SQL, PySpark, and Azure Data Factory. Built and optimized data processing...Local area
$180k - $330k
...shapes what we build next. SR. ANALYTICS ENGINEER (X) We are seeking a skilled Senior Analytics Engineer to build and maintain robust data systems that enable high-impact quantitative analysis and business decision-making. This role combines strong software engineering...Temporary work$55.1k - $115k
...and foster an inclusive environment, empowering our employees to be their authentic selves. We are actively seeking an experienced Data Center Engineer for our Infrastructure Reliability team. This person will be responsible for building and installing multiple...Temporary workWork at officeRemote workWorldwide3 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Data Annotator. Be the first to apply!




