AI Workflow Evaluator - Fully Remote | Upto $190/hr
$190 per hourMercor
New York, NY
- Remote job
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: MCP & Plug In Connectors Expert
Type: Contract
Compensation: $50–$190/hour
Location: Remote
Commitment: 20+ hours/week
Role Responsibilities
- Evaluate AI systems on complex personal workflows, including personal health , travel , and activity planning .
- Design realistic prompts for complex personal-life tasks using Google Drive , Expedia , Notion , and other plugins/connectors.
- Execute tasks while recording your screen and using personal plugins/connectors.
- Write clear explanations of AI successes and failures to improve AI model performance .
- Judge AI outputs for practicality, personalization, and reasoning.
- Identify where models miss context, overreach, or fail to use tools correctly.
Qualifications
Must-Have
- US-based only.
- Strong MCP experience and plugin/connector usage.
- Experience using LLM plugins/connectors like Google Drive , Expedia , and Notion .
- Heavy personal usage of LLM products.
- Active LLM account with 6+ months of history.
- Willingness to sign a data-share consent form via DocuSign .
- Experience using AI for multi-step planning, research, and decision-making.
- Strong written judgment and attention to detail.
- Ability to evaluate AI outputs and write against rubrics.
Preferred
- Extensive rubric experience with 100+ hours on prior projects.
Compensation & Legal
- Hourly contractor .
- Experts added to the project will begin in a trial period.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the AI Workflow Evaluator - Fully Remote | Upto $190/hr in New York, NY vacancy
$30 per hour
...Compensation: $20-$30/hour Location: Remote Commitment: 10-40 hours/week Role Responsibilities Evaluate outputs from large language... .... Review multi-step agent workflows, including screenshots and reasoning... ...experience in LLM evaluation, AI output analysis, QA/testing, UX...Remote jobHourly payContract work$20 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Contract Compensation $15-$20/hour Location Remote Role Responsibilities Conduct fact-checking... ...tools. Generate high-quality human evaluation data by identifying response strengths, areas...Remote jobContract workPart timeSummer work$20 per hour
A tech company is seeking a Receptionist to assist in training AI models. This role involves evaluating the logic of AI chatbots and ensuring quality in their outputs. Applicants should be fluent in English, detail-oriented, and possess strong writing skills. This position...Remote workHourly payFlexible hoursDataAnnotation
New York, NY1 day ago- ...users based in the United States for part-time, fully remote projects related to AI research. Individuals will evaluate AI-generated content based on their... ...understanding of architectural design and BIM workflows. The role offers flexible hours and can be combined...Remote workPart timeFlexible hours
Handshake
New York, NY3 days ago $70 per hour
Mercor is seeking a Generalist Expert to improve AI model performance by evaluating AI-generated responses and providing structured feedback. This remote position requires strong analytical and written communication skills, along with a Bachelor's degree from a top-500...Remote jobContract work$50 - $60 per hour
...in Minnesota seeks a Registered Nurse to help improve AI models. In this role, you will evaluate chatbot logic and measure their progress while solving... ...independent contract position with the flexibility to work remotely. Competitive pay ranges from $50-$60 per hour, with...Remote workHourly payContract workFlexible hoursDataAnnotation
Saint Paul, MN17 hours ago$120 per hour
...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco... .... Position: Software / AI / IT / data Evaluator Type: Contract Compensation: $80–$120/hour Location: Remote Role Responsibilities Evaluate AI-...Remote jobContract workSummer workWork at office$120 per hour
...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco... ...: Media / journalism / communications Evaluator Type: Contract Compensation: $80–$120/hour Location: Remote Role Responsibilities Evaluate AI-generated...Remote jobContract workSummer workWork at office$50 per hour
...creative and technical talent with leading AI research labs. Headquartered in San Francisco... ...Dorsey . Position: Dutch Audio Generalist Evaluator Expert Type: Contract Compensation: $50/hour Location: Remote Duration: 3–6 months Role...Remote jobContract workSummer work$120 per hour
...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco... ...: Investment analysis / valuation / credit Evaluator Type: Contract Compensation: $80–$120/hour Location: Remote Role Responsibilities Evaluate AI-...Remote jobContract workSummer workWork at office$17 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Compensation: $17/hour Location: Remote Duration: Up to 6 months Commitment... ...+ hours/week Role Responsibilities Evaluate AI model output lyrics , voice generation...Remote jobContract workSummer workImmediate start- ...Management Consultant — AI Workflow Evaluator is a remote evaluation track for reviewing management consultant ai workflow evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured...Remote jobHourly payFor contractors10 hours per week
$20 per hour
A tech company focused on AI and design is seeking a Digital Designer to evaluate AI-generated designs and enhance AI's understanding... ...-centered design. This role is fully remote and offers flexibility in... ...rates starting from $20/hr for general projects and $40/hr for...Remote workDataAnnotation
Honolulu, HI23 hours ago$60 per hour
...A leading AI development company seeks experienced quantitative professionals to evaluate AI-generated analyses and solve quantitative problems. This fully remote position offers flexibility, competitive pay of up to $60 USD/hour, and the opportunity to shape the future...Remote workDataAnnotation
Phoenix, AZ17 hours ago$40 per hour
...A leading AI development firm is looking for experienced quantitative professionals to evaluate AI-generated work and design problems for AI training. This fully remote position allows for a flexible schedule, offering competitive pay starting at $40+ per hour. Ideal...Remote workHourly payFlexible hoursDataAnnotation
Bismarck, ND17 hours ago$40 per hour
A leading AI development company is seeking experienced quantitative professionals to evaluate AI-generated analyses and design quantitative problems for AI training. This fully remote role offers flexibility in project selection and scheduling, with competitive pay starting...Remote workHourly payDataAnnotation
Juneau, AK17 hours ago$40 per hour
...A forward-thinking AI solutions company is seeking experienced quantitative professionals to evaluate AI-generated analyses and contribute to the development of cutting-edge AI systems. This fully remote role offers flexible scheduling and competitive pay starting at...Remote workHourly payFlexible hoursDataAnnotation
Lincoln, NE17 hours ago$40 per hour
...A leading AI development company is seeking experienced quantitative professionals to evaluate AI-generated quantitative work and design problems for AI training. Enjoy the flexibility of fully remote work with a competitive pay starting at $40+ per hour. Ideal candidates...Remote workHourly payDataAnnotation
Washington DC17 hours ago$60 per hour
...contribute to developing cutting-edge AI systems, while enjoying the flexibility of remote work and setting your own... ...-the-art AI models on tasks like evaluating AI-generated quantitative analysis... ...on our platform. Benefits Fully remote: work from anywhere in the...Remote workHourly payFull timeFlexible hoursDataAnnotation
Kansas City, MO1 day ago$20 - $30 per hour
A remote-focused technology firm is looking for an individual proficient in evaluating large language model outputs. The role involves assessing AI systems, reviewing workflows, and providing actionable feedback to enhance product quality. Ideal candidates will have strong...Remote jobHourly pay- A leading cybersecurity firm is looking for experienced professionals to evaluate AI-generated security content and solve technical cybersecurity problems. The role is fully remote and allows you to choose projects and work on your schedule. Candidates should have at least...Remote job
$40 per hour
...firm is looking for experienced cybersecurity professionals to evaluate AI-generated security content and solve technical problems.... ...This role offers flexibility with a full-time or part-time, fully remote schedule and projects that pay hourly starting at $40 USD. Strong...Remote jobHourly payFull timePart time- Alignerr is seeking a Population Health Informaticist to evaluate and improve AI-driven health analyses from a population health perspective. This fully remote, hourly contract role offers 10-40 hours per week with flexible scheduling. You will review data pipelines, dashboards...Remote jobHourly payContract work10 hours per weekFlexible hours
$80 - $120 per hour
Mercor is seeking a Healthcare / Clinical Evaluator to connect elite creative and technical talent with leading AI research labs. This fully remote role includes evaluating AI-generated artifacts against quality standards, identifying errors, and providing feedback to enhance...Remote jobHourly payWork at office$60 per hour
A leading AI development company seeks quantitative professionals to contribute to the development of advanced AI systems. This fully remote position allows for a flexible schedule and offers competitive pay up to $60 per hour. Candidates should have at least 2 years of...Remote jobHourly payFlexible hours$40 per hour
A leading AI development company in the United States is seeking experienced quantitative... ...of AI systems. The role involves evaluating AI-generated quantitative work, designing... ...strong coding and analytical skills. Work is fully remote and offers competitive pay starting at $...Remote jobHourly pay$40 per hour
A leading AI development firm is seeking experienced quantitative professionals to evaluate AI-generated quantitative analysis and provide impactful feedback. This fully remote role allows for flexible scheduling and competitive pay starting at $40 per hour. Candidates...Remote jobHourly payFlexible hours$40 per hour
A leading AI development firm is seeking experienced quantitative professionals to evaluate AI-generated analysis and solve complex technical problems. Ideal candidates... ...and experience with analytical coding. This fully remote position allows for flexible scheduling and...Remote jobHourly payFlexible hours$60 per hour
A leading AI development firm is seeking experienced quantitative professionals to contribute to developing AI systems. The role is fully remote and offers a flexible schedule, where you'll evaluate AI-generated analyses and provide feedback. Candidates should possess...Remote jobHourly payFlexible hours$40 per hour
A leading AI development firm is seeking experienced quantitative professionals to evaluate AI-generated analytics and provide technical feedback for model improvement. The role offers fully remote work from multiple countries and a flexible schedule to choose your projects...Remote jobHourly payFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Workflow Evaluator - Fully Remote | Upto $190/hr. Be the first to apply!
Related searches
- program evaluator New York, NY
- work from home web search evaluator New York, NY
- clinical evaluator New York, NY
- evaluator New York, NY
- ai evaluator New York, NY
- work from home social media evaluator New York, NY
- quality evaluator New York, NY
- education evaluator New York, NY
- social media evaluator New York, NY
- operations coordinator remote New York, NY