AI Evaluation Analyst [Remote]

Contribute to the advancement of frontier language model capabilities as an AI Evaluation Analyst. In this role, you will leverage your domain expertise to help train next-generation AI systems, shaping how models learn, reason, and perform through high-quality, real-world input. This position offers a unique opportunity to work at the forefront of AI, focusing on written and verbal clarity, multi-turn conversation design, and in-depth analysis of model behavior. Key Responsibilities

• Author detailed, task-based multi-turn conversations and rubrics aligned with project specifications.
• Test and refine conversation drafts against frontier large language models, iterating to meet quality and difficulty requirements.
• Deliver comprehensive evaluation assets including transcripts, target behaviors, binary rubrics, and supporting evidence.
• Ensure strict fidelity to evolving project specs while maintaining high throughput and attention to detail.
• Validate and calibrate outputs with team leads and quality control as guidelines change.
• Work independently and consistently, meeting expected output rates for deliverable completion.

• Native-level written English with exceptional clarity, structure, and attention to detail.
• Prior experience in data annotation, RLHF, SFT, evaluation, or prompt engineering for AI systems.
• Working knowledge of frontier LLM behaviors and common model failure patterns.
• Demonstrated ability to interpret and apply highly detailed specifications without supervision.
• Strong critical thinking and analytical skills in writing-heavy or analysis-heavy domains.
• Experience authoring evaluation items, rubrics, or conducting deep analysis of technology outputs.
• Background in research, editorial, technical writing, or quality assurance is a plus.

This is a contractor position with remote work flexibility. Compensation is output-based, with experts paid per task that meets project specifications. The time required to complete work may vary depending on the expert’s experience and workflow. Minimum submission requirements apply, and experts must submit a minimum number of tasks per week.

Hourly pay ranges from $20 to $30, depending on experience and task completion.

We typically fill roles within 48 hours and are looking for experts ready to start immediately. If selected, you are expected to begin your first tasks within 24, 48 hours of completing onboarding.