Attorney, Evaluation Operations (Contract)

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for an Attorney, Evaluation Operations (Contract) based in the United States. The Attorney, Evaluation Operations will play an important role in assessing and improving the quality, accuracy, and reliability of AI-powered legal solutions. In this position, you will leverage your legal expertise to evaluate AI-generated content, test new product features, and provide structured feedback before their release to customers. Working closely with legal product specialists, you will help shape the development of innovative tools designed to transform professional legal workflows. Your responsibilities will include reviewing legal research, contracts, memoranda, and other AI-generated outputs against established evaluation criteria. This is a flexible, project-based opportunity that allows you to contribute to cutting-edge legal technology while working remotely on your own schedule. The role is particularly suited to experienced attorneys interested in the intersection of legal practice, artificial intelligence, and product innovation. Accountabilities - AI Output Evaluation: Compare multiple AI-generated responses to identical legal tasks, including legal memoranda, contract redlines, and research answers, assessing their accuracy, relevance, completeness, and overall quality against established evaluation rubrics. - Product Testing and Feedback: Test new legal technology features and workflows in pre-release environments, providing detailed qualitative feedback based on professional legal experience. - Evaluation Rubric Review: Review and assess grading rubrics to ensure they are accurate, comprehensive, consistent, and appropriate for evaluating legal AI outputs. - Dataset Development: Create and contribute to evaluation datasets containing realistic legal queries, contract markups, and synthetic legal documents to support the testing and improvement of AI systems. - Legal Research and Analysis: Apply practical legal knowledge and analytical judgment to assess AI-generated legal content and identify potential inaccuracies, gaps, or areas requiring improvement. - Cross-Functional Collaboration: Work closely with legal product specialists and evaluation teams to support research initiatives and provide insights that inform product development and key technical decisions.

- Project-Based Delivery: Complete assigned evaluation projects within agreed deadlines, often involving short turnaround times of 24–48 hours. Projects generally require between 2 and 20 hours over several days, with the flexibility to accept or decline individual assignments. Requirements - Juris Doctor (JD) degree and an active license to practice law in at least one U. S. jurisdiction. - At least 3 years of professional legal experience at an AmLaw 100 or comparable law firm, or within an in-house legal department. - Professional experience in either commercial litigation or transactional law. - Familiarity with artificial intelligence tools, including practical experience using AI applications in a professional or personal context. - Excellent legal writing skills, with the ability to communicate complex legal concepts and reasoning clearly, concisely, and accurately. - Strong analytical and critical-thinking abilities, particularly when evaluating the quality, reliability, and completeness of legal information. - Excellent attention to detail and the ability to apply structured evaluation criteria consistently. - Ability to work independently, manage project deadlines, and deliver high-quality feedback in a remote environment. - Availability to accept occasional short-notice assignments and work on weekends when project deadlines require it. - Flexibility to manage a variable workload, as projects are assigned on an as-needed basis without a minimum hours commitment.