Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Evaluation Analyst

$55 per hour

Mindrift

Overview 2 days ago Be among the first 25 applicants Get AI-powered advice on this job and more exclusive features. This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency. At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. What We Do The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe. Who we\'re looking for We\'re looking for curious and intellectually proactive contributors, the kind of person who double-checks assumptions and plays devil\'s advocate. Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated? This is a flexible, project-based opportunity well-suited for: Analysts, researchers, or consultants with strong critical thinking skills Students (senior undergrads / grad students) looking for an intellectually interesting gig People open to a part-time and non-permanent opportunity About the project We\'re on the hunt for QAs for autonomous AI agents for a new project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. Throughout the project, you\'ll have to balance quality assurance, research, and logical problem-solving. This project opportunity is ideal for people who enjoy looking at systems holistically and thinking through scenarios, implications, and edge cases. You do not need a coding background, but you must be curious, intellectually rigorous, and capable of evaluating the soundness and consistency of complex setups. If you\'ve ever excelled in things like consulting, CHGK, Olympiads, case solving, or systems thinking — you might be a great fit. What you\'ll be doing Reviewing evaluation tasks and scenarios for logic, completeness, and realism Identifying inconsistencies, missing assumptions, or unclear decision points Helping define clear expected behaviors (gold standards) for AI agents Annotating cause-effect relationships, reasoning paths, and plausible alternatives Thinking through complex systems and policies as a human would to ensure agents are tested properly Working closely with QA, writers, or developers to suggest refinements or edge case coverage How to get started Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule. Shape the future of AI while building tools that benefit everyone. Requirements Excellent analytical thinking: Can reason about complex systems, scenarios, and logical implications Strong attention to detail: Can spot contradictions, ambiguities, and vague requirements Familiarity with structured data formats: Can read, not necessarily write JSON/YAML Ability to assess scenarios holistically: What\'s missing, what\'s unrealistic, what might break? Good communication and clear writing (in English) to document your findings We also value applicants who have Experience with policy evaluation, logic puzzles, case studies, or structured scenario design Background in consulting, academia, olympiads (e.g. logic/math/informatics), or research Exposure to LLMs, prompt engineering, or AI-generated content Familiarity with QA or test-case thinking (edge cases, failure modes, "what could go wrong") Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.) Benefits Get paid for your expertise, with rates that can go up to $55/hour depending on your skills, experience, and project needs Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments Participate in an advanced AI project and gain valuable experience to enhance your portfolio Influence how future AI models understand and communicate in your field of expertise We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI. #J-18808-Ljbffr Mindrift

Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluation Analyst in Raleigh, NC vacancy
  • $80 per hour

     ...-time opportunity focused on quality assurance for autonomous AI agents. You will analyze complex systems, review tasks for logic, and...  ...analytical and detail-oriented skills, with experience in policy evaluation or logic puzzles preferred. Compensation can reach up to $80/... 
    Suggested
    Remote job
    Part time
    Flexible hours

    Mindrift

    Raleigh, NC
    2 days ago
  • $112.5k - $147.5k

     ...looking for an experienced Senior Analyst, IT Internal Controls & SOX...  ...role will be responsible for evaluating the design and operating effectiveness...  ...assess risks associated with AI-enabled processes and...  ...use of AI-enabled solutions, agents, automations, or productivity... 
    Suggested
    Flexible hours

    Circle

    Raleigh, NC
    4 days ago
  •  ...institutions. We specialize in leveraging advanced technologies such as AI, cloud, and data-led innovation to help our clients accelerate...  ...findings with preventive recommendations Participate in evaluation of client IT infrastructure, prepare actionable assessment reports... 
    Suggested
    Full time
    Temporary work
    Relocation

    Infosys Technologies

    Raleigh, NC
    2 days ago
  •  ...Online Data Analyst – Fully Remote Position Are you a detail-oriented individual with...  ...people worldwide. Completing research and evaluation tasks in a web‑based environment such as...  ...difference in the world! TELUS Digital AI Community Our global AI Community is a... 
    Suggested
    Part time
    Freelance
    Local area
    Remote work
    Worldwide

    TELUS Digital AI Data Solutions

    Raleigh, NC
    4 days ago
  • Sprout AI is seeking a hands-on AI Product Analyst to dive into building AI systems across multiple ventures. You'll be the go-to expert for prompt engineering, evaluations, and performance optimization, crafting prompts, testing tools, and designing evaluation pipelines... 
    Suggested

    Sproutai

    Raleigh, NC
    5 days ago
  •  ...solutions firm is seeking an Online Data Analyst for a fully remote part-time position. Candidates...  ...map content through online research and evaluation tasks. This entry-level role offers...  ...team making a difference in the world of AI and data solutions. #J-18808-Ljbffr TELUS... 
    Remote job
    Part time
    Flexible hours

    TELUS Digital AI Data Solutions

    Raleigh, NC
    3 days ago
  • We're seeking a hands-on AI Product Analyst who's excited to dive deep into building AI systems...  ...specialist managing prompt engineering, evaluations, and AI performance optimization...  ...everything from crafting and testing agent prompts and tools, to building evaluation... 

    Sproutai

    Raleigh, NC
    3 days ago
  • Overview Join our team as a Data Governance Analyst and assist the Data Governance Manager...  ...spreadsheet imports, web interface updates & AI enhancements. Collaborate with business...  ...the organization. Technology and Tools: Evaluate and implement data governance tools and... 
    Work at office
    Flexible hours

    Kimley-Horn

    Raleigh, NC
    2 days ago
  • $125.65k - $180k

     ...data standards and policies, ensuring fitness for use in downstream systems.Implement AI/ML-driven automation for enhanced data quality and standardization.Consult data consumers, analysts and data scientists on navigating data sources and interpreting data definitions to... 
    Contract work
    Work experience placement
    Work at office
    Remote work
    Flexible hours

    Red Hat

    Raleigh, NC
    2 days ago
  •  ...more complex, and more impactful. And as communications shifts to AI, we're the infrastructure layer making it possible.We call...  ...dev cycle is slow or repetitive, your instinct is to script it, agent it, or eliminate it. You don't just build the product — you accelerate... 
    Shift work

    Bandwidth

    Raleigh, NC
    2 days ago
  • $73.2k - $95k

     ...Microsoft Teams, Power Platform capabilities, AI-enabled tools, and other technologies to...  ...executive reporting processes. Identify, evaluate, and implement artificial intelligence (...  ...banking industry. Experience using AI agents and agentic workflows, with the ability to... 
    Full time
    Temporary work
    Part time
    Work at office
    Shift work
    Day shift

    Truist Inc

    Raleigh, NC
    1 day ago
  • $76k - $125.3k

     ...tax compliance filings and other business analysis Compile and evaluate moderately complex data, computations, documentation and various...  ...planet, while building trust in capital markets. Enabled by data, AI and advanced technology, EY teams help clients shape the future... 
    Summer holiday
    Flexible hours

    EY

    Raleigh, NC
    4 days ago
  •  ...institutions. We specialize in leveraging advanced technologies such as AI, cloud, and data‑led innovation to help our clients accelerate...  ...and consulting initiatives focusing on domain‑specific product evaluation, benchmarking studies, and domain‑specific system configurations... 
    Temporary work
    Relocation

    Infosys Limited

    Raleigh, NC
    3 days ago
  •  ...is currently seeking a Senior Associate, Corporate Affairs Data Analyst to join our Corporate Affairs team.Responsibilities:Elevate Corporate...  ...data analysis and dashboards with all teamsDevelop and oversee AI-enabled data workflows, with a human-in-the-loop mindset and... 
    H1b
    Local area

    KPMG

    Raleigh, NC
    4 days ago
  •  ...friendly, enabling customers to buy now and pay later without hidden fees. The Senior Product Manager for the Agent Enablement pod will shape the tools, workflows, and AI-powered capabilities used by agents to support millions of customers with speed and care. You will... 

    Affirm

    Raleigh, NC
    2 days ago
  •  ...Manufacturing Data Analyst / Senior Data AnalystAscentt is building cutting-edge data analytics & AI/ML solutions for global automotive and manufacturing leaders. We turn enterprise data into real-time decisions using advanced machine learning and GenAI. Our team solves... 

    Ascentt

    Raleigh, NC
    3 days ago
  • $100k - $175k

     ...employees, and communities.The RoleThis is not a traditional curriculum developer role - please do not apply if you have dabbled with AI agents. In this unique role, we are accepting applications for two different personas: The AI Solutions Engineer turned Mentor/Coach -... 
    Work at office
    Local area
    Remote work
    Work from home
    3 days per week

    Veeva Systems

    Raleigh, NC
    5 days ago
  •  ...Job title : Analyst – Data and Analytics (DNA) Office/region : Raleigh, NC; Dallas, TX Eligibility : May 2026 through June 202...  ...multiple live data feeds, and developing and applying a bespoke AI based risk methodology – click here for more information Developing... 
    Work at office
    Flexible hours
    Raleigh, NC
    7 days ago
  • Software Technology Inc. in Raleigh, NC is seeking a Data Analyst to join the Data Analytics team within the NC DIT. The role focuses...  ..., systems analysis, and project management to support program evaluation, transparency, and safety. #J-18808-Ljbffr Software... 
    Local area

    Software Technology Inc

    Raleigh, NC
    3 days ago
  •  ...Encouraging and collaborativeenvironment Growth Open Positions ERP Report Analyst Reports to: Director of Informer CustomerSuccess Candidates must...  ...maintaining reports and dashboards using Informer and Informer AI Assistants across Banner ERP and other higher education ERP... 
    Temporary work

    Entrinsik Inc.

    Raleigh, NC
    3 days ago
  • $80k - $115k

     ...advancements in underlying technology have finally made it possible for AI to impact clinical care in a meaningful way. Tempus' proprietary...  ...at the right time. We're looking for an experienced Senior Data Analyst to help us design and execute on clinical evidence studies with... 
    Work experience placement
    Remote work
    Shift work

    Tempus AI

    Raleigh, NC
    5 days ago
  • $200k - $322k

     ...with some of the most dedicated people in the world working for us! The Agent Policy Fabric (APF) team is building durable, signed policy authority and the supporting supply chain for governed AI-agent actions. What You'll Be Doing Program Foundations: Establish the roadmap... 

    NVIDIA

    Raleigh, NC
    5 days ago
  •  ...global digital solutions company is seeking a remote Online Data Analyst to enhance digital maps used worldwide. As a part-time position,...  ...a reliable internet connection and computer. Join a dynamic team committed to building better AI models. #J-18808-Ljbffr... 
    Part time
    Remote work
    Worldwide

    TELUS Digital AI Data Solutions

    Raleigh, NC
    4 days ago
  • A financial services firm is seeking a Senior Product Analyst to enhance its virtual assistant platform. This role involves analyzing data...  ...management skills are essential, as well as familiarity with AI technologies. Benefits include comprehensive health plans and generous... 

    Truist

    Raleigh, NC
    2 days ago
  •  ...opportunityHitachi Energy is looking for a Strategic Marketing Analyst to join our Transformers Sales Operations team in North America...  ...offerings fit into the delivery of our customer’s products/services.Evaluate and derive insights from customer and competitor regulatory... 

    Hitachi

    Raleigh, NC
    2 days ago
  •  ...Job Title: Senior Business Analyst (Data Modernization Initiative) Client: NC Department of Health and Human Services (NC DHHS)...  ...Assist in drafting procurement-related documentation and evaluation criteria. Conduct market research and analysis to support... 
    Contract work
    Work at office
    Local area

    Expert Technology Services

    Raleigh, NC
    3 days ago
  •  ...Position: We are seeking a detail-oriented D365 F&O Functional Analyst to join our Information Systems team. This is a functional,...  ...Support D365 F&O version updates, including regression testing and evaluating new features for business value.Identify, troubleshoot, and... 
    Work experience placement
    Work at office

    Sumitomo Electric

    Raleigh, NC
    1 day ago
  • $105.4k - $207.8k

     ...to build the data foundations that power the next generation of AI-enabled cyber defense?If yes, then Deloitte’s Cyber team could be...  ..., vector or hybrid retrieval, tool or function calling, evaluation or monitoring, prompt-injection defenses, and secure access patterns... 
    Local area
    Visa sponsorship

    Deloitte

    Raleigh, NC
    4 days ago
  •  ...transportation.Your OpportunityWe're looking for a Senior Noise Analyst to support our Mid-Atlantic Traffic Noise team. The position...  ...and final traffic noise analysis performed for highway projects. Evaluates and designs feasible and reasonable noise abatement in compliance... 
    Full time
    Contract work
    Temporary work
    Part time
    Casual work
    Work at office
    Local area
    Flexible hours

    Stantec

    Raleigh, NC
    2 days ago
  • Harris Computer is seeking a Quality Assurance Analyst in the United States (North Carolina) to join the Mobile Team. You will create and maintain comprehensive test plans and scripts for new features, bug fixes, and integrations, and validate customer scenarios to ensure... 
    Remote job

    Harris Computer

    Raleigh, NC
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Evaluation Analyst. Be the first to apply!