Remote Evaluation Harness Failure Analyst
AuraOne
New York, NY
- Remote job
AuraOne is seeking an Evaluation Harness Failure Analysis Reviewer to assess model outputs against our quality rubric. This is a remote contractor role, US-eligible, with flexible hours and a focus on edge-case labeling and structured feedback. You will compare paired responses, tag severity, and document recurring failure modes to help retrain the model. Prior annotation experience and strong written reasoning are valued, with an estimated hourly rate discussed after interview. #J-18808-Ljbffr AuraOne
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Remote Evaluation Harness Failure Analyst in New York, NY vacancy
- ...AI Evaluation Analyst Micro1 is engaging AI Evaluation Analysts to contribute to a customer's project focused on advancing frontier language... ...knowledge of frontier LLM behaviors and common model failure patterns. # Demonstrated ability to interpret and apply highly...Remote work
micro1
United States3 days ago - AuraOne is seeking a Bilingual Writer - German (Germany) for a remote evaluation track to review German generalist prompts and responses... ...You will evaluate outputs, tag issues, and document recurring failure modes, with a focus on consistency and clear reasoning. This...Remote jobFor contractorsFlexible hours
- ...Evaluation Harness Failure Analysis Reviewer is a remote evaluation track for reviewing evaluation harness failure analysis evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured...Remote jobHourly payFor contractors10 hours per week
$60 per hour
...advocate. If you thrive in ambiguity, enjoy remote asynchronous work, and want to learn how modern AI systems are tested and evaluated, we want to hear from you. Project... ...with QA or test‑case thinking (edge cases, failure modes, "what could go wrong"). Some understanding...Remote workFreelanceFlexible hours$92.14k - $134.8k
...St, ST Cloud, MN (MN2608). How You Will Be EvaluatedYou will be evaluated for this job based on how well you meet the qualifications... ...information to support your qualifications for this position; failure to provide sufficient evidence in your resume may result in a "...Remote workPermanent employmentFull timeWork experience placementRelocationMonday to Friday$30 - $70 per hour
...Overview Help shape the next generation of AI productivity tools by evaluating, critiquing, and improving AI-generated business documents. You... ...consulting or Fortune 500 quality standards. This is a fully remote, contract role with a preference for candidates based in the...Remote workHourly payContract workWork at officeSaidGig
Remotemore than 2 months ago$230k - $270k
...for enforcing our policies, protecting users, and ensuring our platform is not misused. As a Safeguards Enforcement Analyst focused on Safety Evaluations, you'll play a central role in ensuring our models meet safety and policy standards before and after launch. You'll...Remote workWork at officeVisa sponsorshipFlexible hoursShift workAnthropic
United States4 days ago$20 - $70 per hour
...Role Overview Evaluate and improve AI-generated business documents in Portuguese, using advanced Office Suite skills and business judgment... ..., and Fortune 500 or consulting-level quality. In this remote contractor role, you will work directly with AI systems and cross...Remote workHourly payContract workFor contractorsWork at officeWork visaSaidGig
Indianamore than 2 months ago- ...contribution to society.Job Summary The Senior Well-Being Data & Evaluations Analyst will build, maintain, and evaluate AI-assisted database... ...savingsModes of WorkPositions that are eligible for hybrid or mobile/remote work mode are at the discretion of the hiring department....Remote workTraineeshipWork at office
- UN News seeks an Evaluation and Research Analyst (ERA) to support independent country programme evaluations and broader IEO research with strong data analytics and evaluation methods. The role emphasizes applying AI-enabled analytics (Python/R) to strengthen evidence and...Remote jobWork from home
- ...transfers at this time. Position Summary The REMS Business Analyst serves as the critical liaison between business stakeholders,... ...sponsors, and IT teams to ensure the successful delivery of Risk Evaluation and Mitigation Strategy (REMS) programs and supporting REMS...Remote workH1bVisa sponsorship
MMS
Austin, TX5 days ago - ...AI research company is seeking a detail-oriented Business Analyst to support evaluation and annotation workflows with clear tasks around quality and... ...compensation, flexible working hours, and a fully remote environment, committing to at least 30 hours per week. #J-...Remote jobFlexible hours
- Turing, remote with a global reach, seeks a detail-oriented Business Analyst to support evaluation and annotation workflows focused on quality, accuracy, and structured reasoning. You will perform side-by-side evaluations, conduct fact-checking, and provide clear justifications...Remote jobContract workFlexible hours
- ...leading AI research firm is seeking a detail-oriented Business Analyst to support evaluation and annotation workflows. The role involves performing... ...and attention to detail, and be comfortable working in a remote environment. The position offers competitive compensation...Remote jobFlexible hours
$57.2k - $108.8k
...successful Component Analyst to perform any combination... ...in support of the evaluation and disposition of various... ...assemblies, cable/harness assemblies, electronics... ...Knowledge of component failure modes by commodity... ...as on-site, hybrid or remote. The salary range for...Remote workContract workTemporary workWork experience placementWork at officeWorldwideRelocationFlexible hours$60 per hour
...Austin is looking for QA experts to validate and improve AI systems. This remote, freelance role requires strong analytical skills and attention to detail. Candidates will review AI evaluation tasks, identify inconsistencies, and define expected behaviors for agents....Remote workFreelanceMind Rift
Austin, TX1 day ago- ...nonprofit health IT organization enabling data-driven evaluation across programs. The Program Evaluation Analyst partners with leaders to design, implement, and... ...functional teams, and delivering clear, actionable findings in a remote work environment. #J-18808-Ljbffr OCHINRemote job
$100k - $115k
Join to apply for the CX Research and Evaluation Analyst role at Bixal 2 weeks ago Be among the first 25 applicants Join to apply for the CX... ...155,000.00 11 months ago Customer Experience Administrator - Remote(Should be able to work CST hours0 Transformation Train Product...Remote workFull timeContract workTemporary workLocal areaWork from homeWeekend work- ...AI Evaluation Analyst is a remote review track for evaluating AI outputs across ai evaluation analyst specialist operations workflows. Reviewers... ...Familiarity with AI-assisted workflow tooling and its failure modes. Bilingual experience for cross-region operations...Remote jobHourly payFor contractorsWork experience placement10 hours per week
- As a management analyst for DPI, you will lead teams of specialists... ...conduct in-depth studies and evaluations of issues and proposals... ...flexibilities, which may include remote work or telework options, and... ...Department of Education. Failure to provide all of the required...Remote workPermanent employmentFull timeTemporary workPart timeFor contractorsWork at officeLocal areaWork from homeRelocationMonday to FridayFlexible hours1 day per week
$20 - $30 per hour
...Role Overview Help train and evaluate next-generation large language models by authoring... ...of frontier LLM behavior and common model failure patterns. Experience with data annotation... ...Role type, independent contractor, remote work. Output-based assignments, work independently...Remote jobHourly payFor contractorsSaidGig
Canada2 days ago$80 per hour
...-thinking AI company is seeking Quality Analysts for autonomous AI agents. This project-based... .... Responsibilities include reviewing evaluation tasks, identifying inconsistencies, and... ...With rates up to $80/hour, this flexible remote position allows candidates to influence...Remote jobFlexible hours$80 per hour
...autonomous AI agents to validate complex task structures and improve evaluation frameworks. The role requires excellent analytical skills and... .... Offering up to $80/hour, this position allows for flexible, remote work while contributing to advanced AI projects that positively...Remote jobFlexible hours- 3D Spatial Failure Analyst is a remote review track for evaluating AI outputs across 3d design workflows. Reviewers grade craft, constraints, and tooling adherence; flag production-readiness issues; and document the corrected approach so the modeling team can train on it...Remote jobHourly payFor contractors10 hours per week
- MC² Education is seeking a versatile Research & Evaluation Analyst to conduct quantitative analyses, contribute to surveys, and support multiple applied research projects on a flexible, remote basis. The role emphasizes rigorous programming in R or Stata, collaboration...Remote jobFlexible hours
- ...- NNSA Management & Program Analyst - Program Management Support... ...near CONUS - Washington, DC (Remote) to support requirements for... ...analysts and advisors on the evaluation of the effectiveness of government... ...grace in success and failure while doing meaningful work where...Remote workFull timeTemporary workFor contractorsWork at officeFlexible hours
$122.53k - $159.29k
...primary collector, organizer, and analyst of performance measurement... ....Develops, implements, evaluates, and monitors training for liaisons... ...Department of Education. Failure to provide all of the... ...arrangement (e.g., telework, remote work, etc.) as of November 22...Remote workPermanent employmentFull timeTemporary workPart timeWork at officeLocal areaRelocationRelocation packageShift work- ...DAY TO DAY As a Management Analyst, you will use your knowledge... ...experience by: Analyzing and evaluating the management practices and... ...include the required information, failure to provide this information... ...arrangement (e.g., telework, remote work, etc.), subject to...Remote workPermanent employmentFull timeTemporary workPart timeWork experience placementWork at officeLocal areaRelocationRelocation packageFlexible hours
- ...doing: As the Business Systems Analyst, you will create and share... ...health and well-being; offering a remote work environment and flexible... ...were made, or for failure to fully disclose applicable... ...information. Applicants are evaluated on the basis of job qualifications...Remote workWork experience placementWork at officeWork from homeFlexible hours3 days per week
$100 - $125 per hour
...translate insurance workflows into AI tasks. Responsibilities include evaluating AI outputs, documenting processes, and working on insurance... ...as Underwriters or Actuaries, and must be comfortable in remote work settings. The role offers flexible scheduling requiring a...Remote workHourly payTemporary workFreelance10 hours per weekFlexible hoursCrossing Hurdles
United States3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Remote Evaluation Harness Failure Analyst. Be the first to apply!
Related searches
- nonprofit analyst New York, NY
- provisioning analyst New York, NY
- senior contracts analyst New York, NY
- aml kyc analyst New York, NY
- allocation analyst New York, NY
- kyc analyst New York, NY
- behavior analyst bcba New York, NY
- sharepoint analyst New York, NY
- corporate strategy analyst New York, NY
- reimbursement analyst New York, NY