AI Workflow Evaluator - AI Trainer
$50 - $190 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: MCP & Plug In Connectors Expert
Type: Contract
Compensation: $50–$190/hour
Location: Remote
Commitment: 20+ hours/week
Role Responsibilities
- Evaluate AI systems on complex personal workflows, including personal health , travel , and activity planning .
- Design realistic prompts for complex personal-life tasks using Google Drive , Expedia , Notion , and other plugins/connectors.
- Execute tasks while recording your screen and using personal plugins/connectors.
- Write clear explanations of AI successes and failures to improve AI model performance .
- Judge AI outputs for practicality, personalization, and reasoning.
- Identify where models miss context, overreach, or fail to use tools correctly.
Qualifications
Must-Have
- US-based only.
- Strong MCP experience and plugin/connector usage.
- Experience using LLM plugins/connectors like Google Drive , Expedia , and Notion .
- Heavy personal usage of LLM products.
- Active LLM account with 6+ months of history.
- Willingness to sign a data-share consent form via DocuSign .
- Experience using AI for multi-step planning, research, and decision-making.
- Strong written judgment and attention to detail.
- Ability to evaluate AI outputs and write against rubrics.
Preferred
- Extensive rubric experience with 100+ hours on prior projects.
Compensation & Legal
- Hourly contractor .
- Experts added to the project will begin in a trial period.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
#hiringmercorVacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the AI Workflow Evaluator - AI Trainer in New York, NY vacancy
$60 per hour
A leading technology company is seeking proficient programmers to advance AI development in a fully remote setting. As part of this team, you'll engage in tasks like solving coding problems and building applications, with a competitive pay rate of up to $60/hour based...SuggestedRemote workFlexible hoursDataAnnotation
New York, NY2 days ago- YO IT Consulting is seeking an AI Trainer & Evaluator for a remote contract role. You will apply your domain expertise to train next-generation AI systems, shaping how models learn, reason, and perform through high-quality real-world input. Responsibilities include scoring...SuggestedRemote jobContract work
$60 per hour
A leading AI development company is seeking quantitative professionals to evaluate AI-generated outputs and solve complex problems. This fully remote role allows flexibility and offers competitive pay of up to $60 USD/hour. Ideal candidates should have at least 2 years...SuggestedRemote workFlexible hoursDataAnnotation
New York, NY1 day ago$40 per hour
A leading AI training company in the United States is seeking a Software Engineer to evaluate and improve AI chatbots. The role offers flexibility to choose projects and work hours, as well as payment starting at $40 per hour. Candidates should be proficient in Python or...SuggestedRemote jobHourly payContract work$60 per hour
A leading AI development company is seeking quantitative professionals to evaluate AI-generated work and contribute to advancing AI systems. This remote role allows you to choose your projects and working hours, with competitive pay of up to $60 per hour. Ideal candidates...SuggestedRemote jobHourly pay$20 per hour
A leading AI model training company is seeking a Pyxis Technician to enhance AI models by providing healthcare-related challenges, ensuring logical and accurate outputs from AI chatbots. Candidates should hold a current or in-progress medical or healthcare degree and be...Remote jobHourly payFor contractorsFlexible hours$30 per hour
...About the Company Ajaia is a rapidly growing AI consultancy and product studio dedicated to helping organizations move from experimentation... ...exactly how to integrate AI into their specific professional workflows. We are seeking "smart, well-presenting" individuals with a...For contractorsImmediate startRemote workShift workAJAIA
New York, NY2 days ago$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... .... Position: User/customer research and feedback synthesis Evaluator Type: Contract Compensation: $80–$120/hour Location...Contract workSummer workWork at officeRemote work- ...looking for skilled 3D Slicer users to support AI research through flexible, hourly... ...MRI segmentation, or surgical planning to evaluate AI-generated content and provide feedback... ...medical imaging tasks and clinical research workflows. This is an ongoing, project-based...Hourly payFull timeContract workFor contractorsRemote workFlexible hours
Handshake
New York, NY4 days ago $73 per hour
...intelligence to ethically shape the future of AI. What We Do The Mindrift platform... ...how modern AI systems are tested and evaluated? This is a flexible, project‑based opportunity... ...and engaging examples, cases, or workflows based on your domain knowledge. Excellent...Permanent employmentPart timeFreelanceRemote workFlexible hoursMind Rift
New York, NY1 day ago- ...States for part-time, fully remote projects related to AI research. Individuals will evaluate AI-generated content based on their architectural design... ...a good understanding of architectural design and BIM workflows. The role offers flexible hours and can be combined with...Part timeRemote workFlexible hours
Handshake
New York, NY4 days ago $8 - $65 per hour
...Specialist (Fluent in Thai) - Freelance AI Trainer Project World Wide - Remote Are you a... ...troubleshooting, and collaborative editing workflows, documenting every failure mode so we... ...to our prompt engineering and evaluation metrics. A bachelor’s or master’s degree...Hourly payContract workFor contractorsFreelanceImmediate startRemote work$30 per hour
...: 10-40 hours/week Role Responsibilities Evaluate outputs from large language models and autonomous... ...standards. Review multi-step agent workflows, including screenshots and reasoning... ...Requirements Strong experience in LLM evaluation, AI output analysis, QA/testing, UX research,...Remote jobHourly payContract work$20 - $30 per hour
A remote-focused technology firm is looking for an individual proficient in evaluating large language model outputs. The role involves assessing AI systems, reviewing workflows, and providing actionable feedback to enhance product quality. Ideal candidates will have strong...Remote jobHourly pay- ...looking for skilled Inkscape users to support AI research through flexible, hourly... ...logo creation, or print and web graphics to evaluate AI-generated content and provide feedback... ...better understand visual tasks and creative workflows. This is an ongoing, project‑based...Hourly payFull timeContract workFor contractorsFreelanceRemote workFlexible hours
Handshake
New York, NY4 days ago - ...looking for skilled Krita users to support AI research through flexible, hourly... ...painting, concept art, or illustration to evaluate AI‑generated content and provide feedback... ...better understand visual tasks and creative workflows. This is an ongoing, project‑based opportunity...Hourly payFull timeContract workFor contractorsFreelanceRemote workFlexible hours
$65 per hour
Freelance Cybersecurity Analyst - AI Trainer 3 days ago Be among the first 25 applicants This opportunity is only for candidates currently... ...model's understanding of Microsoft Defender products and SOC workflows Provide expert feedback on AI performance in identifying and...Part timeFreelanceWork at officeRemote work- ...skilled KiCad and LibrePCB users to support AI research through flexible, hourly... ...PCB layout, or embedded hardware design to evaluate AI-generated content and provide feedback... ...understand electronics design tasks and EDA workflows. This is an ongoing, project-based opportunity...Hourly payFull timeContract workFor contractorsFreelanceRemote workFlexible hours
GrabJobs
New York, NY2 days ago - ...looking for skilled LMMS users to support AI research through flexible, hourly... ...composition, or electronic music production to evaluate AI-generated content and provide feedback... ...understand music tasks and creative production workflows. This is an ongoing, project-based...Hourly payFull timeContract workFor contractorsFreelanceRemote workFlexible hours
Handshake
New York, NY4 days ago $23 per hour
...Annotators connects individuals with Generative AI projects from leading tech innovators.... ...such as rating AI-generated content, evaluating factual accuracy, or comparing responses... ...expertise required. On this project, AI trainers earn up to $23 per hour equivalent. Requirements...Hourly payPart timeFreelanceRemote workMindrift Data Annotation Projects
New York, NY3 days ago$23 per hour
...Toloka is seeking a part-time, remote freelance annotator to participate in online projects such as rating AI-generated content, evaluating factual accuracy, or comparing responses. Projects are available based on your profile and opportunities. Compensation varies...Hourly payPart timeFreelanceRemote workMindrift Data Annotation Projects
New York, NY10 hours ago$100 - $150 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...exfiltration , ransomware , worms , and exploits . Evaluate POC exploit development to determine boundaries between security...Hourly payWeekly payFull timeContract workFor contractorsSummer workRemote workMercor
New York, NY1 day ago$139.51k
...build a better working world.Consulting - Technology Consulting - AI and Quantitative Modelling - Artificial Intelligence Data... ...and pipelines as well as designing and deploying end to end ML workflows on at least one major cloud computing platform. Must have 1 year...Full timeWork experience placementSummer holidayImmediate startMonday to Friday- Mid-level Applied AI Data Scientist (Tableau Focus)Location: New York, NY (Hybrid - 3... ...trends, and anomaliesBuild and support NLP workflows, including named entity recognition (NER)... ...senior team members, contributing to evaluation and refinementCreate Tableau dashboards and...Work at office
$40 per hour
Overview Back End Developer role at DataAnnotation . We train AI models, measure progress, evaluate logic, and solve problems to improve model quality.... ...with algorithms, data structures, and debugging workflows A current, in progress, or completed Bachelor’s degree...Hourly payFull timeContract workPart timeRemote work$40 per hour
...Full Stack Developer to join our team to train AI models. You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the... ...with algorithms, data structures, and debugging workflows A current, in progress, or completed Bachelor’s...Hourly payFull timeContract workPart timeRemote work$40 per hour
...analytics company is seeking quantitative professionals to assist in AI development. The role is fully remote, allowing you to work from... ..., Ireland, Australia, or New Zealand. Responsibilities include evaluating AI-generated quantitative analyses and solving complex problems....Remote jobHourly payFlexible hours$40 per hour
A leading AI development company seeks quantitative professionals to evaluate AI-generated work and contribute to the development of advanced AI systems. The role is fully remote, allowing for flexible scheduling and a competitive pay rate starting at $40+ USD per hour....Remote jobHourly payFlexible hours$40 per hour
A leading AI development firm is seeking experienced quantitative professionals to evaluate AI-generated quantitative analysis and provide feedback to shape AI models. The role offers fully remote work with flexible scheduling and competitive pay starting at $40+ per hour...Remote jobHourly payFlexible hours$40 per hour
A forward-thinking AI development company is seeking experienced quantitative professionals in Canada to evaluate AI-generated analyses and provide valuable feedback. This fully remote role offers competitive pay starting at $40+ per hour, a flexible schedule, and the...Remote jobHourly payFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Workflow Evaluator - AI Trainer. Be the first to apply!
Related searches
- program evaluator New York, NY
- work from home web search evaluator New York, NY
- clinical evaluator New York, NY
- evaluator New York, NY
- ai evaluator New York, NY
- work from home social media evaluator New York, NY
- quality evaluator New York, NY
- education evaluator New York, NY
- social media evaluator New York, NY
- ai trainer New York, NY