Evaluation QA Lead (Agentic and RAG Evaluation)
$60 - $90 per hourVettedlance
Experienced QA leads run quality for AI evaluation projects: rubric design, reviewer calibration and agreement tracking for agentic and RAG systems. What you will do Own QA for agentic-workflow and RAG/conversational evaluation projects. Write and refine rubrics and gold sets. Calibrate reviewers and track inter-annotator agreement (kappa, alpha). Audit samples and report quality trends. What you need 4+ years owning QA on annotation, evaluation or software testing. Hands-on experience evaluating LLM, agent or RAG systems. Comfortable with agreement metrics and spreadsheets. Fluent written English. Contract Remote, hourly contract (independent contractor) Part-time, flexible: 10-40 hrs/week Remote Pay: $60 - $90 / hour (USD). Paid in USD, weekly. Rate set after a short paid assessment. #J-18808-Ljbffr Vettedlance
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Evaluation QA Lead (Agentic and RAG Evaluation) in New York, NY vacancy
- Vettedlance is seeking an experienced QA lead to run quality for AI evaluation projects, including rubric design, reviewer calibration and tracking agreement for agentic and RAG systems. You will own QA for agentic-workflow, write and refine rubrics and gold sets, calibrate...SuggestedRemote jobHourly payContract workPart time
$136k - $170k
...partnering with engineers and researchers to evaluate large language models, improve AI system... ..., and continuously improving. You'll lead initiatives spanning LLM evaluation frameworks... ...products around retrieval-augmented generation (RAG), AI agents, MCP, or model orchestration...Suggested$150k - $190k
...the quality bar across our platform. This is not a traditional QA role. We are looking for a senior, pragmatic engineer who ensures... ...improving QA processes in a fast-paced startup environmentComfort evaluating and integrating AI tools into test automation and quality...SuggestedFull timeLive inWork at officeLocal areaWork from homeKIRA
New York, NY1 day ago- SME Careers is seeking a Physics Quality Assurance Lead to oversee quality and trainer performance across physics AI training projects... ...hourly contractor role reviews AI-generated physics content, evaluates output against guidelines, and provides precise written...SuggestedRemote jobHourly payFor contractors
$71 - $76 per hour
iSoftStone, Inc. is Seeking an eCommerce Website QA Lead to Join our Team! *THIS IS A REMOTE CONTRACT POSITION* **Applicants must be... ...automated regression tests into CI/CD pipelines where appropriate. Evaluate testing tools and frameworks, such as Playwright, Selenium,...SuggestedContract workRemote workWorldwideWork visaShift work- AI Trainer Jobs is seeking a Chemistry Quality Assurance Lead to oversee quality, consistency, and trainer performance across chemistry... ...projects. You will review AI-generated chemistry content, evaluate output quality, and provide precise written feedback. This remote...Remote jobFor contractors
- ...Careers is seeking a Chinese (Traditional) Taiwan / Mandarin QA Lead to oversee quality across Traditional Chinese AI training projects... ...focuses on reviewing AI-generated content and trainer output, evaluating against project guidelines, and providing precise feedback to...Remote jobFor contractors
- SME Careers is seeking an Economics Quality Assurance Lead (QAL) to oversee quality, consistency, and trainer performance across economics... ...role. You will review AI-generated economics content, evaluate output against guidelines, and provide precise written feedback...Remote jobHourly payFor contractors
$65 per hour
...Data Services company, seeks an Anthropology Quality Assurance Lead (QAL) to oversee quality across anthropology-focused AI training... ...hour and involves reviewing AI-generated anthropology content, evaluating against guidelines, and providing precise feedback to trainers...Remote jobFor contractors- ...subsidiary of SuperAnnotate, seeks a Gujarati Quality Assurance Lead to oversee quality and trainer performance across Gujarati AI training projects. You will review AI-generated Gujarati content, evaluate output against rubrics, and provide precise written feedback to...Remote job
- ...Careers is seeking a Slovak Quality Assurance Lead for a remote hourly contractor role to... ...generated Slovak content and trainer outputs, evaluate against guidelines, and provide precise... ...to detail, and the ability to manage QA workflows across remote teams. Selection includes...Remote jobHourly payFor contractors
- AI Trainer Jobs seeks a Maltese Quality Assurance Lead (QAL) to oversee quality and trainer performance across Maltese AI training... ...contractor role. You will review Maltese content and trainer/QA work, evaluate output against rubrics, and provide precise written feedback...Remote jobFor contractors
- ...subsidiary of SuperAnnotate, seeks an Estonian Quality Assurance Lead to oversee quality and trainer performance across Estonian AI... ...contractor role. You will review AI-generated Estonian content, evaluate against guidelines, provide precise feedback, and ensure contributors...Remote jobFor contractors
- ...is seeking a Ukrainian Quality Assurance Lead to oversee quality and trainer performance... ...will review AI-generated Ukrainian content, evaluate output against guidelines, and provide... ...adherence to standards. You will manage remote QA workflows, communicate updates to trainers...Remote job
- ...growing AI Data Services company, seeks a Spanish Quality Assurance Lead (QAL) for a remote hourly contractor role to oversee quality... ...Spanish AI training projects. You will review AI-generated content, evaluate output against guidelines, and provide precise written feedback,...Remote jobHourly payFor contractors
- Vettedlance is seeking a part-time, remote QA lead for Taiwanese Mandarin AI training data. You will evaluate outputs and reviewer work for grammar, tone, meaning, and cultural alignment, and track recurring issues to elevate critical ones. You will maintain style guides...Remote jobPart time
- HarZa Group is seeking a QA Team Lead to own the quality standard for Customer Experience across two brands and to lead a small team of evaluators. You will define what quality means, recalibrate scoring, and ensure coverage across voice, email, and chat. You will drive...Remote job
- ...to 12pm EST collaboration window. The Role We are hiring a QA Team Lead to own the quality standard for Customer Experience across both... ...prove that it reflects reality, and then run a small team of evaluators against it. Your job is to make the quality number something...For contractorsImmediate startRemote work
- Pavago is hiring a QA & Training Supervisor (Bilingual - English/Spanish) to lead quality assurance, coaching, and training across the Patient Concierge team in a remote capacity. The role focuses on evaluating calls, developing onboarding materials, and delivering actionable...Remote job
$105 per hour
...building artificial intelligence. Specialists review examples, evaluate model responses, and provide structured feedback so AI systems... ...content The Role OpenTrain AI is seeking a Civil Engineering QA Lead to review AI-generated civil engineering content and trainer QA...Hourly payPart timeFor contractorsRemote work$80k - $90k
OVERVIEW Reporting to the Manager, IT Quality Assurance, the QA Lead owns the quality assurance strategy and execution for a new enterprise... ...who understand artificial intelligence concepts and can evaluate, communicate, and apply AI-enabled approaches within a business...Remote jobContract workHome officeFlexible hours$136k - $163k
...intelligence, measurement, and agentic AI to show enterprises what... ...people who want to help shape how leading brands show up in an AI world.... ...skilled hands‑on QA Manager to join our engineering... ...scenarios Define testing and evaluation strategies for AI-powered features...Remote workWork from home$70 per hour
...role, you will work as a Political Science Quality Assurance Lead (QAL) to oversee quality, consistency, and trainer performance... ...review AI-generated political science content and trainer/QA work, evaluate output quality against project guidelines, provide precise written...Hourly payFor contractorsImmediate startRemote work$45 per hour
...a Chinese (Traditional) Taiwan / Mandarin Quality Assurance Lead (QAL) to oversee quality, consistency, and trainer performance... ...generated Traditional Chinese Taiwan / Mandarin content and trainer/QA work, evaluate output quality against project guidelines, provide precise...Hourly payFor contractorsLocal areaImmediate startRemote work$75 per hour
...artificial intelligence. Specialists review examples, evaluate model responses, and provide precise feedback that... ...OpenTrain is hiring a Biology Quality Assurance Lead to review AI-generated biology content and trainer QA work across remote biology training projects. You...Hourly payContract workPart timeRemote workFlexible hours$90 per hour
...work as a Neuroscience / Cognitive Science Quality Assurance Lead (QAL) to oversee quality, consistency, and trainer performance... ...generated neuroscience/cognitive science content and trainer/QA work, evaluate output quality against project guidelines, provide precise...Hourly payFor contractorsImmediate startRemote work$35 per hour
...contractor role, you will work as a Portuguese Quality Assurance Lead (QAL) to oversee quality, consistency, and trainer... ...You will review AI-generated Portuguese content and trainer/QA work, evaluate output quality against project guidelines, provide precise written...Hourly payFor contractorsImmediate startRemote work$45 per hour
...contractor role, you will work as an Italian Quality Assurance Lead (QAL) to oversee quality, consistency, and trainer... ...projects. You will review AI-generated Italian content and trainer/QA work, evaluate output quality against project guidelines, provide precise written...Hourly payFor contractorsImmediate startRemote work$35 per hour
...contractor role, you will work as an English Quality Assurance Lead (QAL) to oversee quality, consistency, and trainer... ...projects. You will review AI-generated English content and trainer/QA work, evaluate output quality against project guidelines, provide precise written...Hourly payFor contractorsImmediate startRemote work$75 per hour
...contractor role, you will work as a Chemistry Quality Assurance Lead (QAL) to oversee quality, consistency, and trainer... .... You will review AI-generated chemistry content and trainer/QA work, evaluate output quality against project guidelines, provide precise written...Hourly payFor contractorsRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Evaluation QA Lead (Agentic and RAG Evaluation). Be the first to apply!
Related searches
- quality assurance lead New York, NY
- pharmaceutical quality assurance New York, NY
- quality assurance auditor New York, NY
- qa technician New York, NY
- qa tech New York, NY
- quality assurance monitor New York, NY
- quality assurance test manager New York, NY
- qa software testing New York, NY
- software quality assurance New York, NY
- software qa New York, NY