Evaluation Scenario Writer - QA
$55 per hourMindrift
Overview
At Mindrift, we connect domain experts with cutting-edge AI projects. We are seeking an Evaluation Scenario Writer - QA for a project focused on ensuring the quality and correctness of evaluation scenarios created for LLM agents. This is a flexible, project-based opportunity that blends manual scenario validation, automated test thinking, and collaboration with writers and engineers.
Responsibilities
- Reviewing and validating test scenarios from Evaluation Writers
- Spotting logical inconsistencies, ambiguities, or missing checks
- Suggesting improvements to structure, edge cases, or scoring logic
- Collaborating with infrastructure and tool developers to automate parts of the review
- Creating clean and testable examples for others to follow
Qualifications
- Strong QA background (manual or automation), preferably in complex testing environments
- Understanding of test design, regression testing, and edge case detection
- Ability to evaluate logic and structure of test scenarios (even if written by others)
- Experience reviewing and debugging structured test case formats (JSON, YAML)
- Familiarity with Python and JS scripting for test automation or validation
- Clear communication and documentation skills
- Willingness to occasionally write or refactor test scenarios
- Experience testing AI-based systems or NLP applications (valued)
- Familiarity with scoring systems and behavioral evaluation
- Git/GitHub workflow familiarity (PR review, versioning of test cases)
- Experience using test management systems or tracking tools
Benefits
- Get paid for your expertise, with rates that can go up to $55/hour depending on your skills, experience, and project needs
- Take part in a flexible, remote, freelance project that fits around your commitments
- Participate in an advanced AI project and gain valuable experience for your portfolio
- Influence how future AI models understand and communicate in your field of expertise
How To Get Started
Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule.
Employment details
- Seniority level: Internship
- Employment type: Part-time
Vacancy posted 10 hours ago
Similar jobs that could be interesting for youBased on the Evaluation Scenario Writer - QA in New York, NY vacancy
- AuraOne seeks a contractor to join a remote evaluation track for reviewing writer evaluation prompts and responses against its quality rubric. You will compare outputs, label edge cases, and write structured feedback for model retraining. Responsibilities include evaluating...SuggestedRemote jobFor contractors10 hours per weekFlexible hours
- Mercor is hiring experienced musicians to evaluate generative music AI models in partnership with a leading AI lab. You will assess AI-generated music across a wide range of genres and rate it against detailed quality standards, working in Bengali and English. Key responsibilities...SuggestedFlexible hours
- ...Job Title: Copywriter/QA Content Specialist (Legal and Compliance Team) Location: Remote Duration: 12 Months to start with - W2 Our... ...professional experience as a Copywriter, QA Content Specialist or Content Writer, or in a related role, demonstrated by a strong portfolio of work...SuggestedRemote work
Intelliswift
New York, NY1 day ago - ...smart, skeptical people with copy that passes the “No BS” test? Do you use AI like a power tool - fast drafts, faster research, ruthless QA? Do you obsessively say the words “we should test that”? Join us. We’re hiring a Senior Conversion Copywriter to write and optimize...SuggestedRemote jobFreelanceWork from home
- YO IT Consulting is seeking a Copywriting & Content SME for an hourly remote contractor role to review AI-generated outputs and provide expert feedback on reasoning, clarity, and brand alignment. You will assess accuracy, tone, and audience fit across formats such as landing...SuggestedRemote jobHourly payPart timeFor contractorsFlexible hours
- DataAnnotation is a platform that pays contributors to validate accounting outputs. As an Accountant you will grade AI-generated accounting responses against real-world practice and craft the precise, well-reasoned answers a professional would provide. This is a remote,...Remote jobFlexible hours
$55 per hour
...Title: SAR Writer Office Status: In-Office Pay Rate: $55 Contract Length: Not specified Overview A financial services... ...including preparation and analysis of wire transaction data. Evaluate activity to determine whether cases represent continuing or non...Contract workWork at officeMadison-Davis, LLC
New York, NY4 days ago- ...script adaptation and localization workflows executed by Junior Writers. Ensure all scripts meet editorial standards, cultural... ...targets. Participate in weekly reviews with the business team to evaluate show progress and define action plans. Communicate insights, risks...
Remote Jobs
New York, NY5 days ago - ...destinations, culture, and experiences. If you're a driven writer who stays on top of travel trends and loves watching your work... ...could be a match for this role. Please note that a writing evaluation may be required as part of our application process. TheTravel...FreelanceRemote workFlexible hours
Valnet Inc.
New York, NY4 days ago - ...Overview YPC is seeking an experienced and thoughtful Grant Writer to join our Development Department on a part-time basis,... ...reports. ~ Strong research skills and the ability to identify and evaluate prospective funders. ~ Ability to translate programmatic...Part timeInterim roleWork at office3 days per week
Young People's Chorus of NYC
New York, NY3 days ago $60k - $66k
...Title: Grant Writer Reports To: Senior Grants Manager Location: 305 7th Avenue, New York, NY 10001 Hours... ...funders and managing site visits, liaising with our Impact and Evaluation department to secure and analyze program outcomes, and...Interim roleWork at officeWork from homeAfternoon shiftGood Shepherd Services
New York, NY3 days ago- ...world. We are seeking a talented and passionate freelance travel writer to join our team and help us share unique and captivating... ...experience could be a match for this role. Please note that a writing evaluation may be required as part of our application process. TheTravel...FreelanceRemote workFlexible hours
valnetconcept
New York, NY4 days ago - ...# Perform prospect research on foundations and corporations to evaluate for grant opportunities # Responsible for conducting the full... ...stewardship to current grantors, including working with a fellow grant writer to provide regular written updates (newsletters, etc.) to...
The Citizen Project
New York, NY15 hours ago $100k
...holistic success of our vibrant academic community. The Grants Writer will report to the Director of Corporate Relations Foundations... ...and knowledge of information sources, and the ability to evaluate and match opportunities to the mission and goals of the organization...Work at officeLocal areaFlexible hoursAfternoon shiftThe Research Foundation of the City University of New York
New York, NY5 days ago$35 per hour
...digital product improvement company is seeking a Remote Social Media Writer to engage in app reviews. This entry-level role offers $35 per... ...5 hours weekly. You will test various mobile apps, evaluate their performance, and write concise feedback to assist in enhancing...Hourly payRemote workFlexible hoursReview Pays
New York, NY1 day ago- ...A mobile app evaluation company is looking for an Entry Level App Feedback Writer to test and analyze mobile games and applications. This remote role allows you to provide insightful feedback that can enhance future updates while working at your pace. Ideal candidates...Remote workFlexible hours
Review Pays
New York, NY1 day ago - A dynamic mobile technology company seeks a Junior Review Writer to test mobile apps and provide engaging reviews that will help improve... ...only a smartphone or tablet with internet access. You'll evaluate app design and performance, sharing your findings and insights...Remote workFlexible hours
Review Pays
New York, NY1 day ago - ...great cause? When you work with us in the role of Grant/Proposal Writer , this could be your life, while ensuring we bring in diverse,... ...publications for government RFP’s (Requests for Proposals), evaluates them for relevance to Minds Beyond Measure’s mission and programs...Remote work
Minds Beyond Measure
New York, NY5 days ago $45 - $65 per hour
...Linguists, Instructional Designers, Technical Writers, and AI Content Specialists who excel at... ..., and actionable guidance for human evaluators. In this role, you will play a critical... ..., exceptions, and complex evaluation scenarios while minimizing reviewer escalation. Required...Hourly payFull timeContract workFor contractorsRemote workFlexible hoursWeekday work- ...translate complex concepts into clear, effective documentation plans. Evaluate hardware/software prototypes and updated builds to ensure an... ...on information design best practices. Partner with fellow writers to provide creative support and participate in editorial, peer,...Temporary workWork at officeFlexible hours
Biamp Systems
New York, NY5 days ago - ...20 hours/week Team & Position Details: As a Contract Technical Writer, you will report to a Solutions Architect on Virtru 's Standards... ...virtual and in-person standards writing sessions. Prepare test scenario descriptions and technical reports for multinational...Contract workLocal areaFlexible hoursShift work
Virtru
New York, NY16 hours ago - ...Technical Writer | Developer Infrastructure | New York City (Onsite) About the company: This is a rare greenfield opportunity: there... ...familiarity with developer communities and how engineers discover and evaluate content Comfortable interviewing engineers and translating...Visa sponsorship
EBAS Group
New York, NY15 hours ago $125k - $140k
...About the Role: We're seeking a Senior Technical Writer to create clear, accurate, and developer-friendly technical documentation... ...drafts from code, OpenAPI specs, and code samples Evaluate and refine AI-generated content for accuracy, completeness, clarity...Work at officeRemote workPubmatic
New York, NY15 hours ago$45 - $95 per hour
...About the job AML Investigator and SAR Writer -FinTrust Connect Talent Community... ...surge desks across transaction monitoring scenarios and help eliminate backlogs. Requirements... ...targets and quality thresholds Partner with QA and tuning teams to improve precision and...Hourly payContract workRemote workFinTrust Connect LLC
New York, NY1 day ago$80k - $105k
...Slate is hiring a staff writer for its life section, covering culture, social trends, political life, relationships, religion, education... ...in our hiring process and given equal opportunities to be evaluated based on their qualifications and skills. If you require a reasonable...Full timeLocal areaRemote workSlate
New York, NY2 days ago- ...knowledge of Fantasy content in your cover letter. You will not be considered for the role if this information is not included . Please note that the next step in the hiring process involves a writing evaluation. We look forward to hearing from you! Powered by JazzHR...FreelanceRemote work
Valnet Freelance
New York, NY3 days ago - ...Technical Writer / Editor Role Type: Contractor Location: United States, Canada, United Kingdom We are seeking experienced... ...Writers / Editors to contribute to a customer project focused on evaluating and improving advanced AI systems. You’ll use your expertise...For contractorsRemote work
YO AI Labs
New York, NY1 day ago - ...At APV, we Always Provide Value POSITION SUMMARY The Technical Writer/Editor will develop, edit, and quality-review clear, accurate,... ...audience-appropriate materials. The writer-editor will support evaluation and modernization of existing courses and development of new courses...Contract workWork at officeRemote work
Jimmy Jazz
New York, NY17 hours ago - ...cybersecurity challenges. Position Summary We are seeking a Technical Writer to take ownership of the Blackpoint knowledge base and turn it... ...an agentic support workflow, including chunking, metadata, and evaluating retrieval quality. Familiarity with compliance frameworks...Permanent employmentFull timeLocal areaWorldwide
Blackpoint Cyber
New York, NY16 hours ago - Freelance Writer Position This is a paid freelance, remote position MovieWeb is a top destination for fans of television and movies,... ...experience could be a match for this role. Please note that a writing evaluation may be required as part of our application process. MovieWeb is...FreelanceRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Evaluation Scenario Writer - QA. Be the first to apply!
Related searches