AI Evaluation Specialist
Metas Solutions LLC
Metas Solutions excels in providing strategic consulting, program management, and data-driven analytics to federal public health agencies. Our expertise enhances health systems and delivers measurable outcomes, helping agencies achieve their mission. We collaborate with federal partners to implement agile solutions for national public health initiatives, driving impactful changes through cutting-edge programs that strengthen communities and promote well-being. See for further details about us and careers with Metas. Job Description: Metas Solutions has an "immediate opening" for a talented AI Evaluation Specialist to evaluate enterprise AI tools and capabilities to determine their effectiveness, safety, usability, and readiness for business adoption. In this Remote position, you'll partner with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms, including tools such as OpenAI and life sciences models like Rosalind. You will design repeatable evaluation frameworks, test real-world use cases, analyze model and product behavior, and translate findings into clear recommendations. You will be responsible for comparing AI solutions across quality, accuracy, reliability, safety, privacy and security.
Responsibilities:
Notice on Candidate AI Usage Metas Solutions is committed to maintaining a fair and competitive interview process for all candidates, based on their experience, skills, and education. To preserve the integrity and fairness of our interview process, candidates are expected to respond to all interview questions based on their own knowledge, skills, and experience. The use of artificial intelligence (AI) tools or other third-party resources to generate, assist with, or provide interview responses during phone, virtual, or in-person interviews is not permitted. Any use of AI-assisted responses during the interview process may result in disqualification from further consideration for employment. We are an Equal Opportunity Employer/Veterans/Disabled
Responsibilities:
- Plan and execute hands-on evaluations of AI tools such as OpenAI and life sciences models like Rosalind using representative business scenarios, workflows, prompts, datasets, and user personas
- Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness
- Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications
- Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders
- Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost
- Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities where applicable
- Stay current with advancements in generative AI, model evaluation, AI safety, responsible AI, and emerging enterprise AI products and incorporate relevant methods into the evaluation program
- Engage in knowledge sharing and contribute to building a strong, collaborative team environment
- 5+ years of experience in model evaluation, model quality assurance with hands-on experience evaluating AI-enabled products or services
- Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria
- Experience with hands-on generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes
- Experience assessing AI systems across dimensions such as accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost
- Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, retrieval-augmented generation (RAG), AI agents, APIs, enterprise search, and tool or connector integrations
- Bachelor's degree
- Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder)
- Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models like Rosalind or comparable generative AI solutions
- Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing
- Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows
- Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results
- Ability to work with datasets and evaluation results using tools such as Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings
- Possession of an industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI
- Master's degree preferred
- Must be US Citizen and with the ability to obtain a US Government security clearance (Public Trust 5) within a reasonable period
- Market competitive salary, commensurate with experience and education
- Comprehensive benefits package available, Medical, Dental, Vision and Life Insurance, Paid Time Off (PTO), 401K with company match, growth, and promotion opportunities
Notice on Candidate AI Usage Metas Solutions is committed to maintaining a fair and competitive interview process for all candidates, based on their experience, skills, and education. To preserve the integrity and fairness of our interview process, candidates are expected to respond to all interview questions based on their own knowledge, skills, and experience. The use of artificial intelligence (AI) tools or other third-party resources to generate, assist with, or provide interview responses during phone, virtual, or in-person interviews is not permitted. Any use of AI-assisted responses during the interview process may result in disqualification from further consideration for employment. We are an Equal Opportunity Employer/Veterans/Disabled
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Evaluation Specialist in Decatur, GA vacancy
$155k
...Job Title: AI Evaluation Specialist US Citizenship or Green Card required per government contract Clearance: Must be able to obtain Public Trust Salary: $155K Job Description In this position, you'll evaluate enterprise AI tools and capabilities to determine...SuggestedPermanent employmentContract work- ...Job Description Job Description Job description AI Evaluation Specialist Location: Continental United States Position Overview We are looking for an experienced AI Evaluation Specialist to test and evaluate AI tools and help determine whether they are...SuggestedPermanent employment
- ...AI Evaluation EngineerDeposco is seeking an experienced AI Evaluation Engineer to join our innovative quality assurance team. This role focuses on ensuring the accuracy, reliability, and performance of AI-driven applications, models, and automation workflows. The ideal...SuggestedFull timeWork at office
- ...Contribute to the design, build, deployment, and optimization of AI/GenAI/Agentic AI solutions for customers as well as internal TCS... ...into applications and workflows. • Conduct experiments, evaluate model performance, and present findings with clear metrics and visualizations...SuggestedFull timeInternship
$100k - $130k
...130K ### Top Skills & Technologies Python | SQL | Generative AI | Data Engineering | Machine Learning | Git | Azure ### Job Summary... .... Sound judgment when handling confidential information, evaluating AI-generated output, and documenting assumptions or limitations....SuggestedPermanent employmentFull timeInternshipShift work$110k
...topics including Digital Transformation, Data & Digital Platforms, AI at Scale, Agile, Cybersecurity and Digitizing the Tech Function.... ...ERP and similar solutions. Solution Architecture & Design Evaluating an organization's IT, enterprise applications, data, analytics,...Work at officeLocal area$150k
...innovative client solutions. Tech Advisory & Delivery Senior AI Tech Consultants at BCG Platinion are: Technical experts.... ...automation use cases within a functional domain. Supporting vendor evaluations and platform assessments for AI product tooling. Assisting...Work at officeLocal area$55 - $80 per hour
...jobMercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...high-quality data by developing difficult problems in your domain.Evaluate and improve AI model performance through rigorous testing and...Contract workSummer work- ...)Please review the following job description:Designs and builds AI models and data architectures to improve business processes and... ...capabilities into business applications.Conducts experiments and evaluates the performance, effectiveness, and business value of AI and data...Full timePart timeShift workDay shift
$162.8k - $203.5k
...generations. Role Summary Join the Enterprise Applied AI team to build and ship production AI experiences and platforms... ...stack, from data pipelines and retrieval to prompt/agent logic, evaluation/guardrails, and serving. You will collaborate closely with partners...Full timeContract workTemporary workPart timeLocal areaShift work- ...United States Experience: Previous internship experience in AI, machine learning, or related fields Job Type: Permanent ###... ...performance, and reliability. Build orchestration, retrieval, evaluation, and safety guardrails into production AI applications. Create...Permanent employmentInternship
- ...you're in the right place. Job Details: Job Title: Senior AI Engineer Location: Atlanta, GA Work Mode: 5 days onsite (Let... ...chatbot holds up under thousands of concurrent users. Build evaluation suites for answer quality, groundedness and regression testing...Long term contractRelocation
- ...Description About the Role: Ensisinfo Inc is looking for a talented AI Engineer to join our innovative team in Atlanta, GA. This is an... ...business requirements into scalable AI solutions Monitor, evaluate, and optimize model performance in production environments...
$100k - $250k
...also be part of something bigger. Position Purpose: The Senior AI Engineer is responsible for designing, building, scaling, and optimizing... ...: Knowledge and experience in establishing Responsible AI, evaluation frameworks, reliability engineering, and AI governance...Full timeWork experience placementRemote workNight shift- ...About the role About the role We are looking for a Senior AI Engineer to own and scale the AI layer of one of our core D2C subscription... ...around them end to end – prompt systems, structured outputs, evaluation, observability, and model selection – from first experiment to...For contractorsRemote workFlexible hours
$195k - $325k
...tomorrow’s health today, we want to hear from you.The Principal AI Engineer provides the highest level of individual-contributor technical... ...-augmented generation (RAG), tool/connector ecosystems, and evaluation/governance tooling — across every team that builds on the AI...Full time$20 - $80 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Remote Commitment: 5+ hours/week Role Responsibilities Evaluate the accuracy and depth of AI-generated content in STEM fields...Contract workSummer workRemote work- ...capability that uses advanced data, analytics, AI, and GenAI to help Coca-Cola and its... ...leads AI engineers and technical specialists while partnering closely with Product, Data... ...into scalable production solutions ~ Evaluate, design, and implement generative AI and...
$160k - $210k
...Compensation: $160K - $210K ### Top Skills & Technologies Applied AI | Python | JavaScript | Ruby | REST APIs | Customer Engineering... ...Ruby . Ability to learn unfamiliar technologies quickly and evaluate architectural trade-offs. Proficiency integrating systems...Permanent employment- ...Guidance & Assistance, Inc., (SGA), is searching for an Agentic AI Engineer / Architect for a contract assignment with one of... ...recursive reasoning across multiple contexts or sessions. Evaluate and integrate frameworks such as LangGraph, CrewAI, AutoGen, or...Contract workRemote work
- ...About the Role MiniMed is building a lean, high-leverage AI & Data Science team. We are looking for a Principal AI Engineer... ...sanitized sandbox. Take it to production and drive adoption. Build, evaluate, and deploy on the MIA stack (Databricks, LangGraph/LangSmith,...
- ...and redefine markets, you’ve come to the right place. Principal AI Engineer is a senior technical leader responsible for designing,... ...engineering standards and best practices. Research & Innovation: Evaluate emerging AI technologies and integrate cutting-edge methods into...Full timeWork experience placementFlexible hours
$204.44k - $324.99k
...test Prompt Builder templates and grounded AI experiences using Salesforce data, Data... ...(RAG) patterns Implement agent testing, evaluation, observability, and guardrails,... ...Developer I and Salesforce Certified Agentforce Specialist are required. Platform Developer II,...H1bLocal area- ...Develop and maintain the voice of Adventure Ted using cutting-edge technology and AI tools to ensure consistent character portrayal across animations, virtual events, and multimedia projects. Collaborate with Childhood Cancer Society’s creative and production teams...Remote job
- ...an IT Services and Consultant company, is looking for a Senior AI/ML Engineer for their Atlanta, GA/Remote location. Responsibilities... ..., and statistical modelling on large-scale datasets. Evaluate model performance using appropriate metrics and validation techniques...Remote work
$60 - $70 per hour
...Job Description Job Description Business Consultant – AI/ML Engineer Location: Atlanta/Chicago Employment Type: Contract... ...Develop data preprocessing, feature engineering, model training, and evaluation pipelines. Build and integrate AI/ML services with enterprise...Hourly payPermanent employmentContract workRemote work- ...About the Role We are seeking an experienced Lead Python AI Engineer to design and build scalable, AI-powered backend services... ...systems, vector databases, and external tools. Implement AI evaluation, monitoring, observability, fallback mechanisms, and cost optimization...
$131k - $218.3k
...from you. We are seeking a highly skilled and experienced Senior AI Full Stack Software Engineer to join our innovative team at CMM.... ...data (e.g., clinical notes, imaging, lab results). Model Evaluation and Compliance: Apply rigorous evaluation metrics (e.g., sensitivity...Full time$155k
...Job Title: AI Engineer Location: Atlanta, GA or the Washington, DC / Maryland / Virginia area Type: Direct Hire Compensation... ..., web frameworks, containers, and Azure services. Test, evaluate, and monitor AI applications to improve quality, reliability, latency...Permanent employmentFull timeContract workLocal areaRemote work- ...Phillips, a premier international labor and employment law firm is seeking a naturally inquisitive AI Solutions Engineer with legal applications experience to help evaluate, configure, integrate, and build technology solutions that support legal work and business...Full timeRelocation
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Evaluation Specialist. Be the first to apply!





