Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Evaluation Analyst

$55 per hour

Mindrift

Overview 2 days ago Be among the first 25 applicants Get AI-powered advice on this job and more exclusive features. This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency. At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. What We Do The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe. Who we\'re looking for We\'re looking for curious and intellectually proactive contributors, the kind of person who double-checks assumptions and plays devil\'s advocate. Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated? This is a flexible, project-based opportunity well-suited for: Analysts, researchers, or consultants with strong critical thinking skills Students (senior undergrads / grad students) looking for an intellectually interesting gig People open to a part-time and non-permanent opportunity About the project We\'re on the hunt for QAs for autonomous AI agents for a new project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. Throughout the project, you\'ll have to balance quality assurance, research, and logical problem-solving. This project opportunity is ideal for people who enjoy looking at systems holistically and thinking through scenarios, implications, and edge cases. You do not need a coding background, but you must be curious, intellectually rigorous, and capable of evaluating the soundness and consistency of complex setups. If you\'ve ever excelled in things like consulting, CHGK, Olympiads, case solving, or systems thinking — you might be a great fit. What you\'ll be doing Reviewing evaluation tasks and scenarios for logic, completeness, and realism Identifying inconsistencies, missing assumptions, or unclear decision points Helping define clear expected behaviors (gold standards) for AI agents Annotating cause-effect relationships, reasoning paths, and plausible alternatives Thinking through complex systems and policies as a human would to ensure agents are tested properly Working closely with QA, writers, or developers to suggest refinements or edge case coverage How to get started Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule. Shape the future of AI while building tools that benefit everyone. Requirements Excellent analytical thinking: Can reason about complex systems, scenarios, and logical implications Strong attention to detail: Can spot contradictions, ambiguities, and vague requirements Familiarity with structured data formats: Can read, not necessarily write JSON/YAML Ability to assess scenarios holistically: What\'s missing, what\'s unrealistic, what might break? Good communication and clear writing (in English) to document your findings We also value applicants who have Experience with policy evaluation, logic puzzles, case studies, or structured scenario design Background in consulting, academia, olympiads (e.g. logic/math/informatics), or research Exposure to LLMs, prompt engineering, or AI-generated content Familiarity with QA or test-case thinking (edge cases, failure modes, "what could go wrong") Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.) Benefits Get paid for your expertise, with rates that can go up to $55/hour depending on your skills, experience, and project needs Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments Participate in an advanced AI project and gain valuable experience to enhance your portfolio Influence how future AI models understand and communicate in your field of expertise We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI. #J-18808-Ljbffr Mindrift

Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluation Analyst in Raleigh, NC vacancy
  • $80 per hour

     ...-time opportunity focused on quality assurance for autonomous AI agents. You will analyze complex systems, review tasks for logic, and...  ...analytical and detail-oriented skills, with experience in policy evaluation or logic puzzles preferred. Compensation can reach up to $80/... 
    Suggested
    Remote job
    Part time
    Flexible hours

    Mindrift

    Raleigh, NC
    2 days ago
  • $112.5k - $147.5k

     ...looking for an experienced Senior Analyst, IT Internal Controls & SOX...  ...role will be responsible for evaluating the design and operating effectiveness...  ...assess risks associated with AI-enabled processes and...  ...use of AI-enabled solutions, agents, automations, or productivity... 
    Suggested
    Flexible hours

    Circle

    Raleigh, NC
    4 days ago
  •  ...institutions. We specialize in leveraging advanced technologies such as AI, cloud, and data-led innovation to help our clients accelerate...  ...findings with preventive recommendations Participate in evaluation of client IT infrastructure, prepare actionable assessment reports... 
    Suggested
    Full time
    Temporary work
    Relocation

    Infosys Technologies

    Raleigh, NC
    2 days ago
  •  ...solutions firm is seeking an Online Data Analyst for a fully remote part-time position. Candidates...  ...map content through online research and evaluation tasks. This entry-level role offers...  ...team making a difference in the world of AI and data solutions. #J-18808-Ljbffr TELUS... 
    Suggested
    Remote job
    Part time
    Flexible hours

    TELUS Digital AI Data Solutions

    Raleigh, NC
    3 days ago
  • $88k - $107.5k

     ...Impact You’ll Make As an Enterprise Risk Analyst - AI Risk, you'll help shape how BECU...  ...the organization to identify, assess, and evaluate risks associated with AI, machine learning...  ...BECU's inventory of AI models, tools, and agents, including registration, risk classification... 
    Suggested
    Work experience placement
    Remote work

    BECU

    Raleigh, NC
    1 day ago
  •  ...home. A Day in the Life of an Online Data Analyst: In this role, you will be working on a...  ...worldwide Completing research and evaluation tasks in a web-based environment such as...  ...difference in the world! TELUS Digital AI Community Our global AI Community is a vibrant... 
    Part time
    Freelance
    Local area
    Worldwide

    TELUS Digital

    Raleigh, NC
    3 days ago
  • Overview Join our team as a Data Governance Analyst and assist the Data Governance Manager...  ...spreadsheet imports, web interface updates & AI enhancements. Collaborate with business...  ...the organization. Technology and Tools: Evaluate and implement data governance tools and... 
    Work at office
    Flexible hours

    Kimley-Horn

    Raleigh, NC
    2 days ago
  •  ...institutions. We specialize in leveraging advanced technologies such as AI, cloud, and data-led innovation to help our clients accelerate...  ...findings with preventive recommendations Participate in evaluation of client IT infrastructure, prepare actionable assessment reports... 
    Full time
    Temporary work
    Relocation

    Infosys

    Raleigh, NC
    2 days ago
  • We're seeking a hands-on AI Product Analyst who's excited to dive deep into building AI systems...  ...specialist managing prompt engineering, evaluations, and AI performance optimization...  ...everything from crafting and testing agent prompts and tools, to building evaluation... 

    Sproutai

    Raleigh, NC
    3 days ago
  • $125.65k - $180k

     ...data standards and policies, ensuring fitness for use in downstream systems.Implement AI/ML-driven automation for enhanced data quality and standardization.Consult data consumers, analysts and data scientists on navigating data sources and interpreting data definitions to... 
    Contract work
    Work experience placement
    Work at office
    Remote work
    Flexible hours

    Red Hat

    Raleigh, NC
    2 days ago
  •  ...more complex, and more impactful. And as communications shifts to AI, we're the infrastructure layer making it possible.We call...  ...dev cycle is slow or repetitive, your instinct is to script it, agent it, or eliminate it. You don't just build the product — you accelerate... 
    Shift work

    Bandwidth

    Raleigh, NC
    2 days ago
  • $73.2k - $95k

     ...Microsoft Teams, Power Platform capabilities, AI-enabled tools, and other technologies to...  ...executive reporting processes. Identify, evaluate, and implement artificial intelligence (...  ...banking industry. Experience using AI agents and agentic workflows, with the ability to... 
    Full time
    Temporary work
    Part time
    Work at office
    Shift work
    Day shift

    Truist Inc

    Raleigh, NC
    2 days ago
  •  ...institutions. We specialize in leveraging advanced technologies such as AI, cloud, and data‑led innovation to help our clients accelerate...  ...and consulting initiatives focusing on domain‑specific product evaluation, benchmarking studies, and domain‑specific system configurations... 
    Temporary work
    Relocation

    Infosys Limited

    Raleigh, NC
    3 days ago
  • $76k - $125.3k

     ...tax compliance filings and other business analysis Compile and evaluate moderately complex data, computations, documentation and various...  ...planet, while building trust in capital markets. Enabled by data, AI and advanced technology, EY teams help clients shape the future... 
    Summer holiday
    Flexible hours

    EY

    Raleigh, NC
    4 days ago
  •  ...Technology|UI & Markup Language|TypeScript Technical Skills 3 Technology|Java|Core Java Technical Skills 4 Technology|Agentic AI|Agent Engineering Overview The Infosys Engineering unit is dedicated to amplifying product-centric value delivery through innovative engineering... 
    Full time
    Temporary work
    Relocation

    Infosys Limited

    Raleigh, NC
    1 day ago
  •  ...friendly, enabling customers to buy now and pay later without hidden fees. The Senior Product Manager for the Agent Enablement pod will shape the tools, workflows, and AI-powered capabilities used by agents to support millions of customers with speed and care. You will... 

    Affirm

    Raleigh, NC
    2 days ago
  •  ...is currently seeking a Senior Associate, Corporate Affairs Data Analyst to join our Corporate Affairs team.Responsibilities:Elevate Corporate...  ...data analysis and dashboards with all teamsDevelop and oversee AI-enabled data workflows, with a human-in-the-loop mindset and... 
    H1b
    Local area

    KPMG

    Raleigh, NC
    4 days ago
  •  ...Provisioning Point / Authorized Data Source program within the AI & Data Organization Support the definition of a data domain based...  ...including training and change management operations Experience evaluating and integrating AI/ML data governance considerations (e.g.,... 

    Jobtailor

    Raleigh, NC
    4 days ago
  • $100k - $175k

     ...employees, and communities.The RoleThis is not a traditional curriculum developer role - please do not apply if you have dabbled with AI agents. In this unique role, we are accepting applications for two different personas: The AI Solutions Engineer turned Mentor/Coach -... 
    Work at office
    Local area
    Remote work
    Work from home
    3 days per week

    Veeva Systems

    Raleigh, NC
    16 hours ago
  •  ...Job title : Analyst – Data and Analytics (DNA) Office/region : Raleigh, NC; Dallas, TX Eligibility : May 2026 through June 202...  ...multiple live data feeds, and developing and applying a bespoke AI based risk methodology – click here for more information Developing... 
    Work at office
    Flexible hours
    Raleigh, NC
    7 days ago
  • $80k - $115k

     ...advancements in underlying technology have finally made it possible for AI to impact clinical care in a meaningful way. Tempus' proprietary...  ...at the right time. We're looking for an experienced Senior Data Analyst to help us design and execute on clinical evidence studies with... 
    Work experience placement
    Remote work
    Shift work

    Tempus AI

    Raleigh, NC
    16 hours ago
  •  ...Encouraging and collaborativeenvironment Growth Open Positions ERP Report Analyst Reports to: Director of Informer CustomerSuccess Candidates must...  ...maintaining reports and dashboards using Informer and Informer AI Assistants across Banner ERP and other higher education ERP... 
    Temporary work

    Entrinsik Inc.

    Raleigh, NC
    3 days ago
  • M3 Wake Research is seeking a Participant Recruitment Analyst to analyze medical records and evaluate outreach opportunities for upcoming clinical trials. The role focuses on reviewing documentation, coordinating pre-screening, and supporting recruitment across study sites... 

    Doist

    Raleigh, NC
    3 days ago
  • $200k - $322k

     ...with some of the most dedicated people in the world working for us! The Agent Policy Fabric (APF) team is building durable, signed policy authority and the supporting supply chain for governed AI-agent actions. What You'll Be Doing Program Foundations: Establish the roadmap... 

    NVIDIA

    Raleigh, NC
    16 hours ago
  • $105.4k - $207.8k

     ...to build the data foundations that power the next generation of AI-enabled cyber defense?If yes, then Deloitte’s Cyber team could be...  ..., vector or hybrid retrieval, tool or function calling, evaluation or monitoring, prompt-injection defenses, and secure access patterns... 
    Local area
    Visa sponsorship

    Deloitte

    Raleigh, NC
    4 days ago
  •  ...Position: We are seeking a detail-oriented D365 F&O Functional Analyst to join our Information Systems team. This is a functional,...  ...Support D365 F&O version updates, including regression testing and evaluating new features for business value.Identify, troubleshoot, and... 
    Work experience placement
    Work at office

    Sumitomo Electric

    Raleigh, NC
    1 day ago
  •  ...Job Title: Senior Business Analyst (Data Modernization Initiative) Client: NC Department of Health and Human Services (NC DHHS)...  ...Assist in drafting procurement-related documentation and evaluation criteria. Conduct market research and analysis to support... 
    Contract work
    Work at office
    Local area

    Expert Technology Services

    Raleigh, NC
    3 days ago
  •  ...transportation.Your OpportunityWe're looking for a Senior Noise Analyst to support our Mid-Atlantic Traffic Noise team. The position...  ...and final traffic noise analysis performed for highway projects. Evaluates and designs feasible and reasonable noise abatement in compliance... 
    Full time
    Contract work
    Temporary work
    Part time
    Casual work
    Work at office
    Local area
    Flexible hours

    Stantec

    Raleigh, NC
    2 days ago
  •  ...an owned communications cloud that delivers advanced automation, AI integrations, global reach, and premium human support.Bandwidth...  ...selves. #jointheband What We Are Looking For: The Product Support Analyst(Voice) supports the development and implementation of proactive... 
    Full time

    Bandwidth

    Raleigh, NC
    2 days ago
  • $40k - $45k

     ...Department: NCCE‑Sampson County Working Title: Assistant Extension Agent, 4‑H Appointment: 12‑Month Recurring Job Type: Full‑Time -...  ...This position provides leadership in planning, implementing, and evaluating research‑based youth programs that develop life skills,... 
    Permanent employment
    Full time
    Work at office
    Local area
    Flexible hours
    Night shift
    Weekend work
    Afternoon shift

    NC State University

    Raleigh, NC
    16 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Evaluation Analyst. Be the first to apply!