Telephone Operator for AI Model Evaluation

Telephone Operators apply their hands-on experience with call handling and switchboard systems to evaluate AI-generated outputs for telephone operator tasks. In remote, project-based assignments you will create realistic prompts, review model responses for accuracy and usefulness, and deliver clear, structured feedback that improves model understanding of telephone operator language and procedures.

• Develop prompts that reflect real-world telephone operator scenarios and workflows, then evaluate AI responses to those prompts.
• Assess AI-produced content for correctness, clarity, and usefulness in the context of telephone operator duties.
• Provide structured feedback that highlights errors, ambiguous phrasing, missing context, and better alternatives.
• Observe signal lights on switchboards, and dial or press buttons to make connections, as part of assessing scenario fidelity.
• Operate telephone switchboards and systems to advance and complete connections, including local, long distance, pay telephone, mobile, person-to-person, and emergency calls, when evaluating model behavior.
• Listen to customer requests, use alphabetical or geographical directories to answer questions, and evaluate whether model outputs replicate this behavior.
• Update directory information, and judge model suggestions for directory updates.
• Suggest and check alternate spellings, locations, or listing formats for customers lacking complete information, and assess model handling of incomplete data.
• Perform clerical duties such as typing, proofreading, and sorting mail, and evaluate the model against these tasks where applicable.
• Offer special assistance to persons unable to dial or who are in emergency situations, and evaluate model responses for appropriate escalation and assistance.
• Operate paging systems or other notification systems such as bells or buzzers to alert recipients of incoming calls, and assess model familiarity with these procedures.
• Monitor automated systems for placing collect calls and intervene for callers needing assistance, and judge model intervention suggestions.
• Interrupt busy lines if an emergency warrants, and evaluate whether model recommendations respect safety and priority rules.
• Provide assistance for customers with special billing requests and assess the accuracy of model-generated billing guidance.
• Provide relay service for users who are deaf or hard of hearing, and evaluate model handling of accessibility needs.
• Keep records of calls placed and received, and of related toll charges, and assess model outputs for proper recordkeeping language and format.
• Promote company products, services, and savings plans when appropriate, and evaluate model approach to customer-facing promotions.

• At least 4 years of professional experience in telephone operator work or closely related roles that included the responsibilities listed above.
• Experience with switchboards, directories, customer assistance, or call monitoring is directly relevant.
• Ability to participate in asynchronous work with research partners, completing assignments and feedback on a flexible schedule.

• Remote, asynchronous work that can be completed from any location.
• Part-time, flexible hours with no minimum commitment, most contributors average 5 to 20 hours per week.
• Ongoing, project-based engagements that can be done alongside primary employment.
• Placement into specific projects depends on project availability, projects run year-round, and new opportunities open periodically across different areas of expertise.
• Work typically involves creating prompts, reviewing AI outputs, and submitting structured evaluations and comments.

Up to $60/hr (depending on the project)

• F-1 students who are eligible for CPT or OPT may qualify for projects. You must consult your Designated School Official to confirm eligibility.
• If your school requires a CPT course enrollment, these projects may not meet that specific requirement.
• STEM OPT is not supported for this program.
• Refer to the program help resources for additional details about supported work authorizations.

• Create an account with the program and complete the initial profile information.
• Verify your identity as required for participation.
• Apply to or join relevant projects and complete any required project onboarding steps or assessments.
• Begin assigned work once placed on a project, submit evaluations according to project instructions, and receive payment as specified for each project.