Gmail & Google Calendar AI Assistant Evaluator

Help improve AI assistants by evaluating how they handle the everyday coordination people rely on, including email, scheduling, bookings, and shared plans. This remote contract role uses your practical experience with Gmail, Google Calendar, and AI tools to assess real-world task performance. Key Responsibilities

• Complete standardized, repeatable tasks involving email, scheduling, bookings, and coordination using AI assistant tools.
• Provide clear task instructions to an AI assistant, then verify its work for accuracy and completeness.
• Apply detailed grading guidelines to evaluate, rank, and annotate assistant responses, delivering structured feedback to improve model performance.
• Test email threads, calendar invitations, online bookings, and multi-person coordination scenarios, including varied situations and edge cases.
• Record findings and submit evaluation data through AI training tools in line with project standards and milestones.
• Work with trainers and other contributors to clarify ambiguities, refine guidelines, and maintain consistent evaluations.
• Take part in quality reviews and incorporate feedback to uphold rigorous standards.

• Daily, hands-on experience using Gmail and Google Calendar for sending emails, accepting invitations, scheduling, and reorganizing plans.
• Regular experience coordinating with others by email or text, such as arranging meetups, following up on conversations, or confirming shared plans and payments.
• Ability to independently book restaurants, appointments, travel, deliveries, or similar services online.
• Practical experience using AI tools for real tasks, including instructing an AI assistant and checking its work.
• Exceptional attention to detail and accuracy when reviewing AI-generated content.
• Strong written communication skills for clear reporting and timely progress updates.
• Strong time-management and self-organization skills for independent remote work.

• Experience with AI evaluation, data labeling, user testing, or similar feedback-focused projects.
• Familiarity with prompt writing and an applied understanding of AI assistant behavior and limitations.
• Experience reviewing or grading AI outputs for accuracy, helpfulness, and safety.

• Remote, independent contractor engagement.
• Applicants must be located in the United States.

$15 to $30 per hour.

• Must have an iPhone with iMessage.
• Must have a Facebook or Instagram account.