Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Evaluation Engineer

Sedgwick

Telecommuter FL

R80152

By joining Sedgwick, you'll be part of something truly meaningful. It’s what our 33,000 colleagues do every day for people around the world who are facing the unexpected. We invite you to grow your career with us, experience our caring culture, and enjoy work-life balance. Here, there’s no limit to what you can achieve.

Newsweek Recognizes Sedgwick as America’s Greatest Workplaces National Top Companies

Certified as a Great Place to Work®

Fortune Best Workplaces in Financial Services & Insurance

AI Evaluation Engineer

PRIMARY PURPOSE: The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle.

Fully remote role, open to U.S. based candidates.

ESSENTIAL FUNCTIONS AND RESPONSIBILITIES

  • Establishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release.

  • Deploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency.

  • Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., faithfulness, groundedness, relevance, task completion, tool-use correctness, safety).

  • Build reusable evaluation frameworks, templates, and libraries that pods can adopt with minimal effort.

  • Partner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds.

  • Develop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes.

  • Implement offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts).

  • Support A/B and shadow testing for model, prompt, and architecture changes.

  • Design, validate, and maintain LLM-as-a-judge evaluators, including rubric design, bias mitigation (position, verbosity, self-preference), and calibration against human labels.

  • Establish human annotation workflows, labeling guidelines, and inter-annotator agreement practices with SMEs.

  • Track judge reliability over time and re-calibrate as models and use cases change.

  • Build and curate golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches.

  • Apply approved data-handling practices (de-identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data.

  • Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing.

  • Produce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews.

  • Contribute to responsible AI standards and release-readiness criteria.

  • Embed with pods to onboard them to evaluation practices, then coach teams toward ownership of their own eval suites.

  • Build dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends.

  • Evaluate and recommend eval tooling and platforms; document standards and best practices.

ADDITIONAL FUNCTIONS and RESPONSIBILITIES

  • Performs other duties as assigned.

  • Travel as required.

QUALIFICATIONS

Education & Licensing

Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred.

Experience

4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market).

Skills & Knowledge

  • Professional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs

  • A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance

  • Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review).

  • Practical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics.

  • Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation.

  • Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement.

  • Familiarity with at least one eval or observability tool (e.g., Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent).

  • Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services.

  • Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders.

  • Excellent oral and written communication skills, including presentation skills

  • Ability to work in a team environment

  • Ability to meet or exceed Performance Competencies

Preferred

  • Experience evaluating agentic systems (multi-step reasoning, tool calling, trajectory evaluation).

  • Background in ML evaluation, search/relevance, NLP, or QA automation.

  • Experience with LLMOps, monitoring and observability, and production feedback loops.

  • Familiarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments.

  • Experience with data annotation workflows and working with SMEs.

  • Knowledge of enterprise controls: SSO, RBAC, private networking, logging, and encryption.

SUCCESS MEASURES (first 12 months)

  • Standard evaluation framework adopted by pods.

  • Automated quality gates in CI/CD for production-bound solutions.

  • Calibrated LLM judges with documented agreement against human labels.

  • Reduction in regressions reaching production and in time to validate model or prompt changes.

WORK ENVIRONMENT

When applicable and appropriate, consideration will be given to reasonable accommodations.

Mental: Clear and conceptual thinking ability; excellent judgment, troubleshooting, problem solving, analysis, and discretion; ability to handle work-related stress; ability to handle multiple priorities simultaneously; and ability to meet deadlines

Physical: Computer keyboarding, travel as required

Auditory/Visual: Hearing, vision and talking

The statements contained in this document are intended to describe the general nature and level of work being performed by a colleague assigned to this description. They are not intended to constitute a comprehensive list of functions, duties, or local variances. Management retains the discretion to add or to change the duties of the position at any time.

Sedgwick is an Equal Opportunity Employer and a Drug-Free Workplace.

If you're excited about this role but your experience doesn't align perfectly with every qualification in the job description, consider applying for it anyway! Sedgwick is building a diverse, equitable, and inclusive workplace and recognizes that each person possesses a unique combination of skills, knowledge, and experience. You may be just the right candidate for this or other roles.

Sedgwick is the world’s leading risk and claims administration partner, which helps clients thrive by navigating the unexpected. The company’s expertise, combined with the most advanced AI-enabled technology available, sets the standard for solutions in claims administration, loss adjusting, benefits administration, and product recall. With over 33,000 colleagues and 10,000 clients across 80 countries, Sedgwick provides unmatched perspective, caring that counts, and solutions for the rapidly changing and complex risk landscape. For more, see sedgwick.com

Vacancy posted 23 hours ago
Similar jobs that could be interesting for youBased on the AI Evaluation Engineer in Jacksonville, FL vacancy
  • $122k - $240.5k

     ...significant impact on our clients’ success. We are hiring an AI Engineer to build and operate the data, features, and GenAI foundations...  .../chunking, embeddings, vector/hybrid search, and retrieval/evaluation telemetry.Deliver governed datasets and feature engineering/serving... 
    Suggested
    Local area
    Visa sponsorship

    Deloitte

    Jacksonville, FL
    23 hours ago
  • $91.1k - $179.5k

     ...significant impact on our clients’ success. We are hiring an AI Engineer to build and operate the data, features, and GenAI foundations...  .../chunking, embeddings, vector/hybrid search, and retrieval/evaluation telemetry.Deliver governed datasets and feature engineering/serving... 
    Suggested
    Local area

    Deloitte

    Jacksonville, FL
    23 hours ago
  • $110k - $190k

    Fortegra is seeking an Internal AI Forward Deployed Software Engineer to stay ahead of cutting-edge AI developments, rapidly prototype proofs-of-concept...  ...for continuously validating AI-driven processes and evaluating their outputs for accuracy, effectiveness, and... 
    Suggested
    Full time
    Temporary work
    Work visa
    Shift work

    Fortegra Financial

    Jacksonville, FL
    1 day ago
  • $102.5k - $210.6k

    Position Summary About UsThe AI Transformation Team, part of the Applied AI Strategic...  ...of business and technology, rapidly evaluating, prototyping, and scaling AI-powered...  ...Transformation Team is seeking a Applied AI Software Engineer III to join our intake solutioning team... 
    Suggested
    Visa sponsorship

    Deloitte

    Jacksonville, FL
    2 days ago
  • $140,000 - $170,000 per day

    Fortegra is building AI-enabled systems for commercial insurance. Our document-intelligence...  ...and parsing tools change.We're hiring an engineer to build and improve these production...  ...through planning, implementation, evaluation, and production readiness. You'll surface... 
    Suggested
    Full time
    Temporary work
    Work visa

    Fortegra Financial

    Jacksonville, FL
    3 days ago
  • What You Must Have 8-10+ years of software engineering experience Strong experience with cloud-...  ...) Experience building and deploying AI/LLM-based systems in production (agents,...  ...function invocation Policy-based routing Build evaluation frameworks for accuracy, latency, and... 

    Saxon Global

    Jacksonville, FL
    1 day ago
  • $50 - $60 per hour

     ...Job Description We are seeking a hands-on Senior AI/ML Engineer to design, build, and deploy next-generation Agentic AI solutions in...  ...Agent memory patterns Multi-agent systems LLM evaluation frameworks ~ Experience deploying and monitoring production... 
    Hourly pay
    Permanent employment
    Local area
    Remote work

    Kavaliro

    Neptune Beach, FL
    2 days ago
  •  ...agile, adaptable, and ready to act quickly.The Senior Engineer is a full-stack engineer with strong AI and large language model (LLM) expertise, embedded...  ...ticketing, and deployment systems, with the security, evaluation, and auditability required for privileged access.... 

    Intercontinental Exchange

    Jacksonville, FL
    3 days ago
  •  ...Tuesday9-12 months We are looking for an 8+ years of experience AI Solution Engineer to design, build, and deploy enterprise-grade AI...  ...with Microsoft Entra ID, OAuth2, JWT, or SAML.Establish AI evaluation frameworks, including regression tests, benchmark datasets,... 

    Experis

    Jacksonville, FL
    2 days ago
  •  ...Senior AI Platform Engineer SS&C Advent's financial solutions are going Agentic! And we're growing the team to get us there. Our best-of-...  ...secure, reliable, and multi-tenant deployment of AI agents. Evaluate new agentic AI, LLM, and orchestration technologies and... 
    Ongoing contract
    Casual work
    Flexible hours

    Blue Prism Group

    Jacksonville, FL
    23 hours ago
  •  ...FIS? About the Role We are seeking a highly technical Senior AI Engineer to lead the development of our next-generation banking customer...  ...multi-region resilience, and secure DevOps practices. Build evaluation, experimentation, and observability frameworks to measure... 
    Full time

    FIS

    Jacksonville, FL
    1 day ago
  • $113.84k - $170.76k

     ...improvementsResolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry...  ...at least one major cloud platform (AWS, Azure, or Google Cloud AI). Hands-on experience with cloud-native AI/ML services (e.g.,... 
    Full time

    Citigroup

    Jacksonville, FL
    4 days ago
  •  ...Job Description Job Description SR GENERATIVE AI PLATFORM ENGINEER | HYBRID – JACKSONVILLE, FL ARC Group has an immediate opportunity...  .... MLOps, DevSecOps & Observability: Familiarity with LLM evaluation frameworks, CI/CD pipelines targeting OpenShift/AWS, and enterprise... 
    Permanent employment
    Contract work
    Work experience placement
    Local area
    Immediate start
    Remote work
    2 days per week
    3 days per week

    ARC Group

    Jacksonville, FL
    3 days ago
  •  ...expertise with core AWS infrastructure and AI/ML services (e.g., IAM, VPC, Amazon...  ...tasks, asynchronous programming, and data engineering. Java: Deep experience building distributed...  ...& Observability: Familiarity with LLM evaluation frameworks, CI/CD pipelines targeting... 
    Hourly pay
    Temporary work
    Work experience placement
    Local area
    Flexible hours

    Computer Merchant, Ltd., The

    Jacksonville, FL
    1 day ago
  • $105.4k - $207.8k

    Position Summary Deloitte is seeking an AI Cloud Security Engineer, Senior Consultant to design, build, and secure next-generation AI-enabled...  ...modeling, security gates, model and prompt versioning, evaluation datasets, approval workflows, release criteria, and... 
    Work experience placement
    Local area
    Visa sponsorship

    Deloitte

    Jacksonville, FL
    2 days ago
  • $134.5k - $265.1k

    Position Summary Deloitte is seeking an AI Cloud Security Engineer, Manager to lead the design, implementation, and security of artificial...  ...modeling, security gates, model and prompt versioning, evaluation datasets, approval workflows, release criteria, and testing... 
    Work experience placement
    Local area
    Visa sponsorship

    Deloitte

    Jacksonville, FL
    2 days ago
  •  ...code reviews and maintenance of internally developed applications, evaluation and performance testing of purchased products. Responsible for...  ...application. Key Responsibilities: * Design and develop AI agents using Model Context Protocol (MCP) standards * Build and... 
    Work experience placement
    Remote work
    Flexible hours

    Spectraforce Technologies

    Jacksonville, FL
    2 days ago
  • We are looking for an AI DevOps Engineer to help build and scale secure, production-ready AI platform capabilities in Jacksonville, Florida. This Long-term Contract position focuses on enabling application teams with reliable Azure-based infrastructure, automation, and... 
    Long term contract

    Robert Half

    Jacksonville, FL
    1 day ago
  •  ...Head of Applied AI Engineering About the Company Profitable AI platform turning healthcare data into measurable enterprise action Industry Computer Software Type Privately Held, VC-backed About the Role The Company is seeking a Head of Applied... 

    Confidential

    Jacksonville, FL
    2 days ago
  • $125.5k - $230.2k

     ...optimized for XOps processes. Additionally, you will monitor and evaluate learning processes to continuously improve high-performance...  .... Your key responsibilities As a Manager in AI Native Engineering, you will play a pivotal role in delivering innovative solutions... 
    Full time
    Work experience placement
    Summer holiday
    Flexible hours

    EY

    Jacksonville, FL
    4 days ago
  • Job Description Oracle EBS / Fusion Cloud Project Manager - Remote Experience: 8-12 years Focus: Oracle EBS to Oracle Fusion Cloud Migration Job Summary We are seeking an experienced Oracle EBS/Fusion Project Manager with strong expertise leading Oracle EBS to Oracle...
    Remote job

    Simple Solutions

    Jacksonville, FL
    1 day ago
  •  ...not have been described for purposes of ADA reasonable accommodation. All reasonable accommodation requests will be reviewed and evaluated on a case-by-case basis. Sourcing Model Recruitment at FIS works primarily on a direct sourcing model; a relatively small... 

    FIS

    Jacksonville, FL
    more than 2 months ago
  •  ...Job Description Job Description Sr. AI Engineer (Agentic AI & AWS Bedrock) Remote/Hybrid with Travel The key is time zone coverage...  ...AgentCore, Claude Agent SDK, Langfuse, AI Observability & Evaluation Job Summary We are seeking an experienced AI Engineer... 
    Remote job

    Simple Solutions

    Jacksonville, FL
    29 days ago
  •  ..., FL, no remote options are available. Summary The AI Solutions Engineer partners directly with AMPORTS operational and corporate teams...  ...manual effort, errors, delays, and operating costs. Evaluates potential AI use cases based on business value, technical feasibility... 
    Full time
    Work at office
    Remote work
    Afternoon shift

    AMPORTS Inc

    Jacksonville, FL
    20 days ago
  • $159.1k - $224.1k

    Position Summary The Team This role sits within Zora AI by Deloitte, Deloitte's product-focused asset business for back office...  ...& Delivery Transformation (I&DT) practice. I&DT brings an engineering- and innovation-led mindset to how Deloitte builds, delivers, and... 
    Local area

    Deloitte

    Jacksonville, FL
    3 days ago
  • Are you passionate about building scalable cloud platforms that power next-generation AI solutions? We're looking for an AI Platform Engineer to help design, automate, and support a modern Azure-based platform used to deploy and manage AI applications, agents, and services... 

    Robert Half

    Jacksonville, FL
    3 days ago
  •  ...As a Principal Artificial Intelligence and Machine Learning Engineer, you will set technical direction for generative artificial intelligence...  ..., context management, memory, tool use, orchestration and evaluation. Resolve complex challenges involving scale, latency,... 
    Full time

    FIS Global

    Jacksonville, FL
    2 days ago
  • $102.5k - $187.9k

     ...world’s most ambitious organizations. Using our product-driven, AI-centric approach, we empower organizations to build for the intelligent...  ...of product leaders, data scientists, designers, and software engineers enable our clients to solve their most complex product... 
    Summer holiday
    Flexible hours

    EY

    Jacksonville, FL
    1 day ago
  •  ...maintenance of planned and unplanned internally developed applications, evaluation and performance testing of purchased products. Responsible...  ..., to ensure timely processing, you agree to be contacted by our AI recruiter via email, text, or phone. Message frequency varies... 
    Contract work
    Work experience placement
    Remote work
    Flexible hours

    Software Guidance & Assistance

    Jacksonville, FL
    1 day ago
  •  ...cloud technologies, APIs, automation, and AI-assisted software development tools. The...  ...requirements. Participates in the evaluation and implementation of software...  ...application monitoring, and other modern software engineering practices. Experience working with... 
    Work at office
    Remote work

    McGuireWoods

    Jacksonville, FL
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Evaluation Engineer. Be the first to apply!