Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Evaluation Specialist

Metas Solutions LLC

Metas Solutions excels in providing strategic consulting, program management, and data-driven analytics to federal public health agencies. Our expertise enhances health systems and delivers measurable outcomes, helping agencies achieve their mission. We collaborate with federal partners to implement agile solutions for national public health initiatives, driving impactful changes through cutting-edge programs that strengthen communities and promote well-being. See for further details about us and careers with Metas.

Job Description:

Metas Solutions has an "immediate opening" for a talented AI Evaluation Specialist to evaluate enterprise AI tools and capabilities to determine their effectiveness, safety, usability, and readiness for business adoption. In this Remote position, you'll partner with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms, including tools such as OpenAI and life sciences models like Rosalind. You will design repeatable evaluation frameworks, test real-world use cases, analyze model and product behavior, and translate findings into clear recommendations. You will be responsible for comparing AI solutions across quality, accuracy, reliability, safety, privacy and security.


Responsibilities:
  • Plan and execute hands-on evaluations of AI tools such as OpenAI and life sciences models like Rosalind using representative business scenarios, workflows, prompts, datasets, and user personas
  • Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness
  • Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications
  • Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders
  • Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost
  • Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities where applicable
  • Stay current with advancements in generative AI, model evaluation, AI safety, responsible AI, and emerging enterprise AI products and incorporate relevant methods into the evaluation program
  • Engage in knowledge sharing and contribute to building a strong, collaborative team environment
Qualifications:
  • 5+ years of experience in model evaluation, model quality assurance with hands-on experience evaluating AI-enabled products or services
  • Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria
  • Experience with hands-on generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes
  • Experience assessing AI systems across dimensions such as accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost
  • Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, retrieval-augmented generation (RAG), AI agents, APIs, enterprise search, and tool or connector integrations
  • Bachelor's degree
  • Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder)
Additional Qualifications:
  • Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models like Rosalind or comparable generative AI solutions
  • Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing
  • Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows
  • Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results
  • Ability to work with datasets and evaluation results using tools such as Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings
  • Possession of an industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI
  • Master's degree preferred
Security Requirements:
  • Must be US Citizen and with the ability to obtain a US Government security clearance (Public Trust 5) within a reasonable period
  • Market competitive salary, commensurate with experience and education
  • Comprehensive benefits package available, Medical, Dental, Vision and Life Insurance, Paid Time Off (PTO), 401K with company match, growth, and promotion opportunities

Notice on Candidate AI Usage

Metas Solutions is committed to maintaining a fair and competitive interview process for all candidates, based on their experience, skills, and education. To preserve the integrity and fairness of our interview process, candidates are expected to respond to all interview questions based on their own knowledge, skills, and experience. The use of artificial intelligence (AI) tools or other third-party resources to generate, assist with, or provide interview responses during phone, virtual, or in-person interviews is not permitted. Any use of AI-assisted responses during the interview process may result in disqualification from further consideration for employment.

We are an Equal Opportunity Employer/Veterans/Disabled
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Evaluation Specialist in Decatur, GA vacancy
  • $155k

     ...Job Title: AI Evaluation Specialist US Citizenship or Green Card required per government contract Clearance: Must be able to obtain Public Trust Salary: $155K Job Description In this position, you'll evaluate enterprise AI tools and capabilities to determine... 
    Suggested
    Permanent employment
    Contract work

    System One

    Atlanta, GA
    4 days ago
  •  ...Job Description Job Description Job description AI Evaluation Specialist Location: Continental United States Position Overview We are looking for an experienced AI Evaluation Specialist to test and evaluate AI tools and help determine whether they are... 
    Suggested
    Permanent employment

    Onsite It Advisors LLC

    Atlanta, GA
    3 days ago
  •  ...AI Evaluation EngineerDeposco is seeking an experienced AI Evaluation Engineer to join our innovative quality assurance team. This role focuses on ensuring the accuracy, reliability, and performance of AI-driven applications, models, and automation workflows. The ideal... 
    Suggested
    Full time
    Work at office

    Deposco

    Alpharetta, GA
    1 day ago
  •  ...Contribute to the design, build, deployment, and optimization of AI/GenAI/Agentic AI solutions for customers as well as internal TCS...  ...into applications and workflows. • Conduct experiments, evaluate model performance, and present findings with clear metrics and visualizations... 
    Suggested
    Full time
    Internship

    Tata Consultancy Services

    Atlanta, GA
    13 hours ago
  • $100k - $130k

     ...130K ### Top Skills & Technologies Python | SQL | Generative AI | Data Engineering | Machine Learning | Git | Azure ### Job Summary...  .... Sound judgment when handling confidential information, evaluating AI-generated output, and documenting assumptions or limitations.... 
    Suggested
    Permanent employment
    Full time
    Internship
    Shift work

    Thinking Minds Inc.

    Atlanta, GA
    2 days ago
  • $110k

     ...topics including Digital Transformation, Data & Digital Platforms, AI at Scale, Agile, Cybersecurity and Digitizing the Tech Function....  ...ERP and similar solutions. Solution Architecture & Design Evaluating an organization's IT, enterprise applications, data, analytics,... 
    Work at office
    Local area

    Boston Consulting Group

    Atlanta, GA
    13 hours ago
  • $150k

     ...innovative client solutions. Tech Advisory & Delivery Senior AI Tech Consultants at BCG Platinion are: Technical experts....  ...automation use cases within a functional domain. Supporting vendor evaluations and platform assessments for AI product tooling. Assisting... 
    Work at office
    Local area

    Boston Consulting Group

    Atlanta, GA
    13 hours ago
  • $55 - $80 per hour

     ...jobMercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...high-quality data by developing difficult problems in your domain.Evaluate and improve AI model performance through rigorous testing and... 
    Contract work
    Summer work

    Mercor Inc

    Atlanta, GA
    2 days ago
  •  ...)Please review the following job description:Designs and builds AI models and data architectures to improve business processes and...  ...capabilities into business applications.Conducts experiments and evaluates the performance, effectiveness, and business value of AI and data... 
    Full time
    Part time
    Shift work
    Day shift

    Truist

    Atlanta, GA
    3 days ago
  • $162.8k - $203.5k

     ...generations. Role Summary Join the Enterprise Applied AI team to build and ship production AI experiences and platforms...  ...stack, from data pipelines and retrieval to prompt/agent logic, evaluation/guardrails, and serving. You will collaborate closely with partners... 
    Full time
    Contract work
    Temporary work
    Part time
    Local area
    Shift work

    Rivian

    Atlanta, GA
    13 hours ago
  •  ...United States Experience: Previous internship experience in AI, machine learning, or related fields Job Type: Permanent ###...  ...performance, and reliability. Build orchestration, retrieval, evaluation, and safety guardrails into production AI applications. Create... 
    Permanent employment
    Internship

    Thinking Minds Inc.

    Atlanta, GA
    2 days ago
  •  ...you're in the right place. Job Details: Job Title: Senior AI Engineer Location: Atlanta, GA Work Mode: 5 days onsite (Let...  ...chatbot holds up under thousands of concurrent users. Build evaluation suites for answer quality, groundedness and regression testing... 
    Long term contract
    Relocation

    Staffworxs LLC

    Atlanta, GA
    20 days ago
  •  ...Description About the Role: Ensisinfo Inc is looking for a talented AI Engineer to join our innovative team in Atlanta, GA. This is an...  ...business requirements into scalable AI solutions Monitor, evaluate, and optimize model performance in production environments... 

    Ensisinfo Inc

    Atlanta, GA
    13 days ago
  • $100k - $250k

     ...also be part of something bigger. Position Purpose: The Senior AI Engineer is responsible for designing, building, scaling, and optimizing...  ...: Knowledge and experience in establishing Responsible AI, evaluation frameworks, reliability engineering, and AI governance... 
    Full time
    Work experience placement
    Remote work
    Night shift

    Home Depot

    Atlanta, GA
    1 day ago
  •  ...About the role About the role We are looking for a Senior AI Engineer to own and scale the AI layer of one of our core D2C subscription...  ...around them end to end – prompt systems, structured outputs, evaluation, observability, and model selection – from first experiment to... 
    For contractors
    Remote work
    Flexible hours

    Ruby Labs Ltd

    Atlanta, GA
    3 days ago
  • $195k - $325k

     ...tomorrow’s health today, we want to hear from you.The Principal AI Engineer provides the highest level of individual-contributor technical...  ...-augmented generation (RAG), tool/connector ecosystems, and evaluation/governance tooling — across every team that builds on the AI... 
    Full time

    CoverMyMeds

    Atlanta, GA
    1 day ago
  • $20 - $80 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Remote Commitment: 5+ hours/week Role Responsibilities Evaluate the accuracy and depth of AI-generated content in STEM fields... 
    Contract work
    Summer work
    Remote work

    Mercor

    Atlanta, GA
    28 days ago
  •  ...capability that uses advanced data, analytics, AI, and GenAI to help Coca-Cola and its...  ...leads AI engineers and technical specialists while partnering closely with Product, Data...  ...into scalable production solutions ~ Evaluate, design, and implement generative AI and... 

    The Coca-Cola Company

    Atlanta, GA
    3 days ago
  • $160k - $210k

     ...Compensation: $160K - $210K ### Top Skills & Technologies Applied AI | Python | JavaScript | Ruby | REST APIs | Customer Engineering...  ...Ruby . Ability to learn unfamiliar technologies quickly and evaluate architectural trade-offs. Proficiency integrating systems... 
    Permanent employment

    Thinking Minds Inc.

    Atlanta, GA
    2 days ago
  •  ...Guidance & Assistance, Inc., (SGA), is searching for an Agentic AI Engineer / Architect for a contract assignment with one of...  ...recursive reasoning across multiple contexts or sessions. Evaluate and integrate frameworks such as LangGraph, CrewAI, AutoGen, or... 
    Contract work
    Remote work

    Software Guidance & Assistance

    Atlanta, GA
    1 day ago
  •  ...About the Role MiniMed is building a lean, high-leverage AI & Data Science team. We are looking for a Principal AI Engineer...  ...sanitized sandbox. Take it to production and drive adoption. Build, evaluate, and deploy on the MIA stack (Databricks, LangGraph/LangSmith,... 

    USA-MiniMed Distribution Corp. 1018

    Atlanta, GA
    13 hours ago
  •  ...and redefine markets, you’ve come to the right place. Principal AI Engineer is a senior technical leader responsible for designing,...  ...engineering standards and best practices. Research & Innovation: Evaluate emerging AI technologies and integrate cutting-edge methods into... 
    Full time
    Work experience placement
    Flexible hours

    Manhattan Associates

    Atlanta, GA
    3 days ago
  • $204.44k - $324.99k

     ...test Prompt Builder templates and grounded AI experiences using Salesforce data, Data...  ...(RAG) patterns Implement agent testing, evaluation, observability, and guardrails,...  ...Developer I and Salesforce Certified Agentforce Specialist are required. Platform Developer II,... 
    H1b
    Local area

    KPMG

    Atlanta, GA
    13 hours ago
  •  ...Develop and maintain the voice of Adventure Ted using cutting-edge technology and AI tools to ensure consistent character portrayal across animations, virtual events, and multimedia projects. Collaborate with Childhood Cancer Society’s creative and production teams... 
    Remote job

    Childhood Cancer Society

    Atlanta, GA
    4 days ago
  •  ...an IT Services and Consultant company, is looking for a Senior AI/ML Engineer for their Atlanta, GA/Remote location. Responsibilities...  ..., and statistical modelling on large-scale datasets. Evaluate model performance using appropriate metrics and validation techniques... 
    Remote work

    ICONMA, LLC

    Atlanta, GA
    20 days ago
  • $60 - $70 per hour

     ...Job Description Job Description Business Consultant – AI/ML Engineer Location: Atlanta/Chicago Employment Type: Contract...  ...Develop data preprocessing, feature engineering, model training, and evaluation pipelines. Build and integrate AI/ML services with enterprise... 
    Hourly pay
    Permanent employment
    Contract work
    Remote work

    NTT DATA

    Atlanta, GA
    20 days ago
  •  ...About the Role We are seeking an experienced Lead Python AI Engineer to design and build scalable, AI-powered backend services...  ...systems, vector databases, and external tools. Implement AI evaluation, monitoring, observability, fallback mechanisms, and cost optimization... 

    SALT

    Atlanta, GA
    1 day ago
  • $131k - $218.3k

     ...from you. We are seeking a highly skilled and experienced Senior AI Full Stack Software Engineer to join our innovative team at CMM....  ...data (e.g., clinical notes, imaging, lab results). Model Evaluation and Compliance: Apply rigorous evaluation metrics (e.g., sensitivity... 
    Full time

    CoverMyMeds

    Atlanta, GA
    3 days ago
  • $155k

     ...Job Title: AI Engineer Location: Atlanta, GA or the Washington, DC / Maryland / Virginia area Type: Direct Hire Compensation...  ..., web frameworks, containers, and Azure services. Test, evaluate, and monitor AI applications to improve quality, reliability, latency... 
    Permanent employment
    Full time
    Contract work
    Local area
    Remote work

    System One

    Atlanta, GA
    19 days ago
  •  ...Phillips, a premier international labor and employment law firm is seeking a naturally inquisitive AI Solutions Engineer with legal applications experience to help evaluate, configure, integrate, and build technology solutions that support legal work and business... 
    Full time
    Relocation

    Fisher & Phillips

    Atlanta, GA
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Evaluation Specialist. Be the first to apply!