Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Evaluation Engineer

Sedgwick

Telecommuter FL

R80152

By joining Sedgwick, you'll be part of something truly meaningful. It’s what our 33,000 colleagues do every day for people around the world who are facing the unexpected. We invite you to grow your career with us, experience our caring culture, and enjoy work-life balance. Here, there’s no limit to what you can achieve.

Newsweek Recognizes Sedgwick as America’s Greatest Workplaces National Top Companies

Certified as a Great Place to Work®

Fortune Best Workplaces in Financial Services & Insurance

AI Evaluation Engineer

PRIMARY PURPOSE: The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle.

Fully remote role, open to U.S. based candidates.

ESSENTIAL FUNCTIONS AND RESPONSIBILITIES

  • Establishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release.

  • Deploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency.

  • Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., faithfulness, groundedness, relevance, task completion, tool-use correctness, safety).

  • Build reusable evaluation frameworks, templates, and libraries that pods can adopt with minimal effort.

  • Partner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds.

  • Develop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes.

  • Implement offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts).

  • Support A/B and shadow testing for model, prompt, and architecture changes.

  • Design, validate, and maintain LLM-as-a-judge evaluators, including rubric design, bias mitigation (position, verbosity, self-preference), and calibration against human labels.

  • Establish human annotation workflows, labeling guidelines, and inter-annotator agreement practices with SMEs.

  • Track judge reliability over time and re-calibrate as models and use cases change.

  • Build and curate golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches.

  • Apply approved data-handling practices (de-identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data.

  • Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing.

  • Produce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews.

  • Contribute to responsible AI standards and release-readiness criteria.

  • Embed with pods to onboard them to evaluation practices, then coach teams toward ownership of their own eval suites.

  • Build dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends.

  • Evaluate and recommend eval tooling and platforms; document standards and best practices.

ADDITIONAL FUNCTIONS and RESPONSIBILITIES

  • Performs other duties as assigned.

  • Travel as required.

QUALIFICATIONS

Education & Licensing

Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred.

Experience

4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market).

Skills & Knowledge

  • Professional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs

  • A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance

  • Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review).

  • Practical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics.

  • Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation.

  • Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement.

  • Familiarity with at least one eval or observability tool (e.g., Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent).

  • Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services.

  • Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders.

  • Excellent oral and written communication skills, including presentation skills

  • Ability to work in a team environment

  • Ability to meet or exceed Performance Competencies

Preferred

  • Experience evaluating agentic systems (multi-step reasoning, tool calling, trajectory evaluation).

  • Background in ML evaluation, search/relevance, NLP, or QA automation.

  • Experience with LLMOps, monitoring and observability, and production feedback loops.

  • Familiarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments.

  • Experience with data annotation workflows and working with SMEs.

  • Knowledge of enterprise controls: SSO, RBAC, private networking, logging, and encryption.

SUCCESS MEASURES (first 12 months)

  • Standard evaluation framework adopted by pods.

  • Automated quality gates in CI/CD for production-bound solutions.

  • Calibrated LLM judges with documented agreement against human labels.

  • Reduction in regressions reaching production and in time to validate model or prompt changes.

WORK ENVIRONMENT

When applicable and appropriate, consideration will be given to reasonable accommodations.

Mental: Clear and conceptual thinking ability; excellent judgment, troubleshooting, problem solving, analysis, and discretion; ability to handle work-related stress; ability to handle multiple priorities simultaneously; and ability to meet deadlines

Physical: Computer keyboarding, travel as required

Auditory/Visual: Hearing, vision and talking

The statements contained in this document are intended to describe the general nature and level of work being performed by a colleague assigned to this description. They are not intended to constitute a comprehensive list of functions, duties, or local variances. Management retains the discretion to add or to change the duties of the position at any time.

Sedgwick is an Equal Opportunity Employer and a Drug-Free Workplace.

If you're excited about this role but your experience doesn't align perfectly with every qualification in the job description, consider applying for it anyway! Sedgwick is building a diverse, equitable, and inclusive workplace and recognizes that each person possesses a unique combination of skills, knowledge, and experience. You may be just the right candidate for this or other roles.

Sedgwick is the world’s leading risk and claims administration partner, which helps clients thrive by navigating the unexpected. The company’s expertise, combined with the most advanced AI-enabled technology available, sets the standard for solutions in claims administration, loss adjusting, benefits administration, and product recall. With over 33,000 colleagues and 10,000 clients across 80 countries, Sedgwick provides unmatched perspective, caring that counts, and solutions for the rapidly changing and complex risk landscape. For more, see sedgwick.com

Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the AI Evaluation Engineer in Tampa, FL vacancy
  •  ...Key Responsibilities Design, develop, test, and deploy AI/ML and Generative AI solutions for enterprise...  ...end-to-end ML pipelines covering data ingestion, feature engineering, model training, evaluation, deployment, and monitoring. Develop RAG solutions and... 
    Suggested
    Contract work

    PB consulting

    Tampa, FL
    a month ago
  •  ...OpenSystem Technical Skills 2 Technology|Agentic AI|Agent Engineering Technical Skills 3 Technology|AI...  ...solutions that meet complex business needs. The ability to evaluate and adopt emerging technologies, translating them into practical... 
    Suggested
    Full time
    Temporary work
    Relocation

    Infosys

    Tampa, FL
    4 days ago
  •  ...MANTECH seeks a motivated, career and customer-oriented Lead AI Engineer to join our team in Tampa, FL. The Lead AI Engineer...  ...architectures, neural network optimization, and automated model evaluation. Demonstrated experience executing and supporting projects... 
    Suggested
    Work at office
    Local area
    Immediate start

    ManTech International Corporation

    Tampa, FL
    1 day ago
  •  ...Distinguished Engineer This role requires associates to be in-office 3 days per week, fostering...  ...and deliver scalable, resilient AI solutions leveraging technologies such as...  ...practices for AI systems, including monitoring, evaluation, and responsible AI. Collaborate with... 
    Suggested
    Work at office
    3 days per week

    ANTHEM INSURANCE COMPANIES INC

    Tampa, FL
    1 day ago
  • $254.58k - $381.88k

     ...Distinguished AI Engineer Location: This role requires associates to be in-office 3 days per week, fostering collaboration and connectivity...  ..., and best practices for AI systems, including monitoring, evaluation, and responsible AI. • Collaborate with UX, product... 
    Suggested
    Temporary work
    Work experience placement
    Work at office
    Local area
    3 days per week
    1 day per week

    Elevance Health

    Tampa, FL
    5 days ago
  • $122k - $240.5k

     ...significant impact on our clients’ success. We are hiring an AI Engineer to build and operate the data, features, and GenAI foundations...  .../chunking, embeddings, vector/hybrid search, and retrieval/evaluation telemetry.Deliver governed datasets and feature engineering/serving... 
    Local area
    Visa sponsorship

    Deloitte

    Tampa, FL
    5 days ago
  • $137k - $263k

     ...#VTeamLife.What you'll be doing…As a Distinguished Forward Deployed AI Engineer within Verizon’s Transformation Command Center (TCC), you...  ..., and serverless compute across AWS, GCP, or Azure).Evaluation Harness Design: Architect offline and online evaluation frameworks... 
    Full time
    Temporary work
    Part time
    Work experience placement
    Work at office
    Work from home
    Shift work
    3 days per week

    Verizon

    Tampa, FL
    2 days ago
  • $10k

     ...Investments! We are expanding our Generative AI capabilities to support our international growth. We are looking for a Software Engineer who will design, build, and improve...  ...application and model performance using evaluation frameworks, observability tools, and iterative... 
    Work at office
    Work from home

    Fisher Investments

    Tampa, FL
    3 days ago
  • $91.1k - $179.5k

     ...significant impact on our clients’ success. We are hiring an AI Engineer to build and operate the data, features, and GenAI foundations...  .../chunking, embeddings, vector/hybrid search, and retrieval/evaluation telemetry.Deliver governed datasets and feature engineering/serving... 
    Local area

    Deloitte

    Tampa, FL
    5 days ago
  • DescriptionThe AI Workflow & Process Design Engineer evaluates and redesigns clinical and operational workflows before AI-enabled solutions are built at Tampa General Hospital. The role maps current-state processes, identifies inefficiencies and failure points, and defines... 

    The Tampa General Hospital Foundation

    Tampa, FL
    2 days ago
  •  ...Job Description Job Description Bridgeway is seeking an AI Engineer to help build and operate large language model and intelligent...  ...analytics layer — embedding and chunking choices, retrieval evaluation, semantic search, summarization, anomaly narration, and contextual... 
    Work at office
    Remote work

    Bridgeway Benefit Technologies

    Tampa, FL
    8 days ago
  • $113.84k - $170.76k

     ...840.00 - $170,760.00Category: TechnologyCompany: CitiThe Gen Ai Engineer is a senior level position responsible for establishing and implementing...  ...variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry... 
    Full time

    Citigroup

    Tampa, FL
    4 days ago
  •  ...PositionWe are looking for an innovative engineer for the GTM Product Team to strengthen the...  ...(ODM). Along the way you will bring AI into the product to enable smarter lead qualification...  ...into a product: structured outputs, evaluation discipline, confidence thresholds,... 
    Contract work
    Interim role
    Local area

    OPSWAT

    Tampa, FL
    3 days ago
  • $129.3k - $177.8k

     ...Become a part of our caring community We're seeking a Lead AI Software Engineer to help build the next generation of AI-powered products...  ..., microservices, data pipelines, orchestration layers, and evaluation frameworks. Develop production-grade applications integrating... 
    Full time
    Temporary work
    Apprenticeship
    Remote work

    Humana

    Tampa, FL
    5 days ago
  • $113.84k - $170.76k

     ...- $170,760.00Category: ProfessionalCompany: CitiAs a Senior AI Engineer specializing in Python and agentic systems, you will architect...  ...Agent protocol, along with experience in agent observability and evaluation.Strong Python skills across data preprocessing, document... 
    Full time
    Work at office
    Remote work

    Citigroup

    Tampa, FL
    3 days ago
  • $55k - $151.47k

     ...SummaryAt PwC, our people in data and analytics engineering focus on leveraging advanced technologies...  ...team you will develop and implement AI solutions that enhance product offerings....  ...closely with team members. We evaluate these factors thoughtfully to establish a... 
    Full time
    H1b

    PwC

    Tampa, FL
    3 days ago
  • $77k - $202k

     ...LevelSenior AssociateJob Description & SummaryThe OpportunityAs an AI Engineer - Senior Associate - Commercial Technology & Innovation, you...  ...assets, or collaborating closely with team members. We evaluate these factors thoughtfully to establish a secure and trusted workplace... 
    Full time
    H1b

    PwC

    Tampa, FL
    3 days ago
  • $102.5k - $210.6k

    Position Summary About UsThe AI Transformation Team, part of the Applied AI Strategic...  ...of business and technology, rapidly evaluating, prototyping, and scaling AI-powered...  ...Transformation Team is seeking a Applied AI Software Engineer III to join our intake solutioning team... 
    Visa sponsorship

    Deloitte

    Tampa, FL
    1 day ago
  • $73.5k - $212.28k

     ...SummaryAt PwC, our people in data and analytics engineering focus on leveraging advanced technologies...  ...cross-functional teams to incorporate AI into various applications- Drive...  ...collaborating closely with team members. We evaluate these factors thoughtfully to establish a... 
    Full time
    H1b

    PwC

    Tampa, FL
    1 day ago
  • $99k - $232k

     ...AIManagement LevelManagerJob Description & SummaryThe OpportunityAs an AI Engineer specializing in AML and Sanctions, you will leverage data and...  ...assets, or collaborating closely with team members. We evaluate these factors thoughtfully to establish a secure and trusted... 
    Full time
    H1b

    PwC

    Tampa, FL
    4 days ago
  •  ...and citizen services. We’re a nonprofit engineering, applied research, and advanced technology...  ...developing, deploying, and transitioning AI/ML-enabled capabilities in support of complex...  ...to identify emerging challenges, evaluate advanced technologies, and transition mission... 
    Full time
    Work experience placement
    Internship
    Local area

    Mitre

    Tampa, FL
    2 days ago
  • $91k - $321.5k

     ...LevelSenior ManagerJob Description & SummaryThe OpportunityAs a CTIO - AI Engineer- Senior Manager, you will play a pivotal role in transforming...  ...assets, or collaborating closely with team members. We evaluate these factors thoughtfully to establish a secure and trusted... 
    Full time
    H1b

    PwC

    Tampa, FL
    4 days ago
  • $124k - $280k

     ...SummaryThe OpportunityAs a Senior Manager, Agentic Engineering Lead, you will guide the design and delivery of agentic and generative AI solutions for clients within our Advisory...  ..., including operational monitoring- Evaluating models and agent behavior through testing,... 
    Full time
    H1b

    PwC

    Tampa, FL
    3 days ago
  • $113.84k - $170.76k

     ...growth and progress together.About the TeamThe Senior Agentic AI Engineer is a high-impact technical professional who sits at the...  ...-agent collaboration.Experience with agent observability and evaluation (e.g., tracing, OpenTelemetry-based tooling) for production agent... 
    Full time

    Citigroup

    Tampa, FL
    4 days ago
  • $8.42k

     ...SRI Tech SolutionsRole: Senior Full Stack AI EngineerLocation: Tampa, FL (3 Days...  ...seeking a highly skilled and motivated AI Engineer with a strong focus on Generative AI and...  ...preprocessing, feature engineering, model training, evaluation, and deployment.• Generative AI... 
    Full time
    3 days per week

    SRI Tech

    Tampa, FL
    3 days ago
  • $99k - $232k

     ...ScienceManagement LevelManagerJob Description & SummaryThe OpportunityAs an AI Engineer- Manager- Commercial Technology & Innovation, you will play a...  ...assets, or collaborating closely with team members. We evaluate these factors thoughtfully to establish a secure and trusted... 
    Full time
    H1b

    PwC

    Tampa, FL
    4 days ago
  • $116.2k - $229.1k

     ...027The Work You’ll DoThe Knowledge Based Engineer Senior Consultant executes hands-on work...  ...executable rules and models used by engineers and AI-enabled workflows. This role is expected...  ...LLM/RAG application work (retrieval + evaluation/validation)Bachelor’s DegreeAbility to... 
    Local area

    Deloitte

    Tampa, FL
    4 days ago
  •  ...recruitment team will provide more details.Senior AI Security EngineerRole OverviewMUFG is seeking an experienced AI Security engineer to establish and scale the Enterprise AI...  ...and compliant adoption of AI solutions.Evaluate and incorporate emerging Microsoft AI and Agentic... 
    Full time
    Work experience placement
    Work at office
    Local area
    Remote work
    1 day per week

    MUFG

    Tampa, FL
    16 hours ago
  • $113.84k - $170.76k

     ...00 - $170,760.00Category: ProfessionalCompany: CitiAs a Lead AI Engineer, you will design and architect agentic AI frameworks and knowledge...  ...accelerationPractical experience with unit testing, agent evaluation benchmarks, security/guardrail enforcement, and standard SDLC... 
    Full time
    Work at office
    Remote work
    2 days per week

    Citigroup

    Tampa, FL
    16 hours ago
  • $127 per hour

     ...Position SummaryMUFG is seeking a Senior Power Platform & Agentic AI Engineer to serve as a core technical member of AI Productivity &...  ...Copilot, citizen development, and enterprise agent capabilities.Evaluate new Microsoft capabilities for architecture, security, governance... 
    Full time
    Work at office
    Local area
    Remote work
    1 day per week

    MUFG

    Tampa, FL
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Evaluation Engineer. Be the first to apply!