Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Evaluation Engineer

Sedgwick

Fortune Best Workplaces in Financial Services & Insurance AI Evaluation Engineer PRIMARY PURPOSE: The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle. Fully remote role, open to U.Establishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release. Deploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency. Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., Partner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds. Develop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes. Implement offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts). Support A/B and shadow testing for model, prompt, and architecture changes. Build and curate golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches. Apply approved data-handling practices (de-identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data. Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing. Produce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews. Contribute to responsible AI standards and release-readiness criteria. Build dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends. Evaluate and recommend eval tooling and platforms; document standards and best practices. Travel as required. Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred. 4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market). A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review). Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement. Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services. Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders. Ability to meet or exceed Performance Competencies Background in ML evaluation, search/relevance, NLP, or QA automation. Experience with LLMOps, monitoring and observability, and production feedback loops. Familiarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments. Experience with data annotation workflows and working with SMEs. SSO, RBAC, private networking, logging, and encryption. Automated quality gates in CI/CD for production-bound solutions. Calibrated LLM judges with documented agreement against human labels. excellent judgment, troubleshooting, problem solving, analysis, and discretion; Computer keyboarding, travel as required Auditory/Visual: Hearing, vision and talking The statements contained in this document are intended to describe the general nature and level of work being performed by a colleague assigned to this description. They are not intended to constitute a comprehensive list of functions, duties, or local variances. Sedgwick is building a diverse, equitable, and inclusive workplace and recognizes that each person possesses a unique combination of skills, knowledge, and experience.

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Evaluation Engineer in Florida, FL vacancy
  • $110k

     ...smarter and safer communities with exciting AI, cloud, IoT, and cyber-physical security...  ...shape the future of IT solutions. AI Engineer The AI Engineer is responsible for...  ...ingestion, chunking, embeddings, retrieval, and evaluation Help configure deployments to meet... 
    Suggested

    Claro

    Florida, FL
    1 day ago
  • $73.59k - $81.77k

     ...Job Posting ID: KT_[273]_JOB Position: Software Engineer FT/PT Status: Full-Time Location: Remote within the following locations: DC, FL, GA, MD, TX, VA Education: Bachelor's degree in Computer Science, Software Engineering or a related field required.... 
    Suggested
    Full time
    Work at office
    Remote work

    Kreative Technologies

    Florida, FL
    5 days ago
  • $106.61k - $284.28k

     .... Position Summary The Lead Software Engineer for MinuteClinic will play a pivotal role...  ...our entire technology stack and embrace AI-enabled engineering practices. In this...  ...will proactively adopt new technologies, evaluate emerging tools—including AI-driven development... 
    Suggested
    Hourly pay
    Full time
    Temporary work
    Local area

    CVS Health

    Florida, FL
    1 day ago
  • $120k - $125k

    Senior Site Reliability Engineer & Duties: The Senior Site Reliability Engineer (Sr. SRE) ensures the reliability, performance, and...  ...and availability reviews with product and engineering teams to evaluate and develop system health. Run regular Game Day and disaster recovery... 
    Suggested
    Full time
    Part time
    For contractors
    Casual work
    Work at office
    Local area
    Remote work
    Work from home
    Flexible hours

    Foundation Finance Company, LLC

    Florida, FL
    18 hours ago
  •  ...Voyages is looking for a Principal Data Engineer to set the technical direction for how we...  ...quality, and you'll lead the way in making AI-assisted development the everyday...  ...RAG, embeddings, and vector search, plus evaluating and monitoring AI systems in production.... 
    Suggested
    Full time
    Work at office
    Remote work

    Virgin Voyages

    Florida, FL
    3 days ago
  •  ...predictive models to define process performance. Collaboration: Work closely with cross-functional teams, including chemists, engineers, etc., to support data-driven decision-making. Machine Learning: Apply machine learning algorithms to optimize process development... 

    SGS Consulting

    Florida, FL
    a month ago
  •  ...complex challenges in healthcare. We’re searching for a Software Engineer to help us build and scale our platform. You’ll own the...  ...ownership of our results. We also embrace new technology, including AI-powered tools, to improve both the quality and the speed of what... 
    Work experience placement
    Work at office
    Remote work
    Relocation
    Flexible hours

    Quest Analytics

    Florida, FL
    3 days ago
  • $86.6k - $144.4k

     ...opportunity to serve as the technical anchor of a cross-functional engineering squad, driving architecture decisions, mentoring peers, and...  ...generation healthcare data platform? Would you like to work in an AI-accelerated development environment where you can leverage... 
    Local area

    Nextens

    Florida, FL
    1 day ago
  • $110k - $130k

     ...maintaining CI/CD pipelines and infrastructure as code to deploy solutions across AWS, OCI, Azure, and Kubernetes environments Using AI-assisted development tools to improve the speed and quality of delivery What you’ll bring Think you could make a difference... 
    Remote work
    Flexible hours

    Boldyn Networks

    Florida, FL
    13 hours ago
  • $15 per hour

     ...What if your knowledge Spanish language could help improve the AI.   WHAT YOU’LL DO Review, evaluate, and validate AI-generated content and data according to project guidelines. Compare AI-generated outputs and identify the most accurate, relevant, or high-quality... 
    Hourly pay
    Part time
    Freelance
    Work at office
    Remote work

    Welo Data

    Florida, FL
    more than 2 months ago
  • $90k - $130k

     ...SecOps Engineer Location: Remote CO, TX, MO, NC, FL, or GA Department: Data Science & Engineering Employment Type: Full-Time Salary Range: $90,000 - 130,000 About Shiplify We are a data provider serving the Less-Than-Truckload (LTL) freight industry... 
    Full time
    Remote work

    Shiplify

    Florida, FL
    18 hours ago
  • $106.61k - $284.28k

     ...bigger – helping to simplify health care one person, one family and one community at a time. Position Summary As a Staff Software Engineer, you'll work at the boundary between individual service teams and the larger system — identifying the architectural problems that... 
    Hourly pay
    Full time
    Contract work
    Temporary work
    Local area

    CVS Health

    Florida, FL
    2 days ago
  •  ...***CANDIDATE MUST BE US Citizen (due to contractual/access requirements)*** This job is an integral member of an agile software engineer team responsible for building scalable software applications across all digital channels to serve our customers. Responsible for taking... 
    Temporary work
    For contractors
    Work at office
    Local area

    Highmark Health

    Florida, FL
    1 day ago
  •  ...reinvent what a pharmacy and a health care company can be in the digital world. Currently, we are seeking a Staff Software Development Engineer with deep, hands‑on experience building and operating backend, platform, and integration services at scale. In this role, you will... 
    Hourly pay
    Full time
    Temporary work
    Local area

    CVS Health

    Florida, FL
    2 days ago
  •  ...SS&C is a leading provider of mission-critical, AI-powered technology and services empowering financial services and healthcare organizations...  ..., scale, and technology. Job Description Senior Data Platform Engineer Role Summary We are seeking a Senior Data Platform Engineer... 
    Ongoing contract
    Worldwide

    SS&C Technologies

    Florida, FL
    13 hours ago
  • $100.13k - $140.77k

     ...and code reviews; addresses findings. Required Qualifications: ~ Bachelor Degree in Computer Science, Information Technology, Engineering or related field ~ Ability to obtain and maintain a Public Trust Clearance ~ Must be willing and able to work EST hours ~2+... 
    Local area
    Immediate start
    Flexible hours

    Groundswell

    Florida, FL
    3 days ago
  • Position Summary The Application Developer configures, develops, and sustains Salesforce applications for a large enterprise Salesforce platform. This position carries a mix of configuration, development, and data remediation on the live platform, and owns assigned work...
    Full time
    Temporary work
    Remote work
    Relocation
    Flexible hours

    Tripoint Solutions LLC

    Florida, FL
    2 days ago
  • $90k

     ...submitted through Workday is reviewed by a real person — not an AI. We value your time and take each submission seriously. Who are...  ...Element, a New Charter Company, we are seeking a remote Application Engineer, in other companies this person could hold the title of (... 
    Remote work

    Element Technologies

    Florida, FL
    1 day ago
  •  ...Must live in Florida, Georgia, Nevada, Ohio, Tennessee or Texas to be considered. Do you find that you have a creative ability to engineer elegant solutions to complex problems or that you love to solve puzzles with code? We are looking for smart problem solvers who are... 
    Full time
    Live in
    Local area
    Remote work
    Work from home
    Flexible hours

    Summit Racing Equipment

    Florida, FL
    2 days ago
  • $127.8k - $135.9k

     ...work, from members of the public who rely on benefit programs to government agency staff. Through human-centered design and modern engineering best practices, we help our government partners understand user needs and deliver on their missions more effectively. This focus... 
    Contract work
    Temporary work
    For contractors
    Work at office
    Local area
    Remote work
    Home office
    Visa sponsorship
    Flexible hours

    Nava PBC

    Florida, FL
    13 hours ago
  • $130k - $150k

     ...architecture and modernization efforts, and drive engineering best practices across the team. What...  ...enhancement. Drive the adoption of AI-assisted development tools, prompt...  ...software development lifecycle. Ability to evaluate, validate, and refine AI-generated code... 
    Work experience placement
    Work at office
    Local area

    Karsun Solutions, LLC

    Florida, FL
    13 hours ago
  • $15 per hour

     ...The Wikimedia Foundation is looking for a Senior Site Reliability Engineer to support and develop the platform serving the world’s favorite...  ...), through critical thinking and expertise, with support of AI tooling Implementing and utilizing configuration management... 
    Permanent employment
    For contractors
    Currently hiring
    Local area
    Remote work

    Wikimedia Foundation

    Florida, FL
    2 days ago
  • $146k - $183k

     ...people leader for a small, high-ownership Site Reliability team (3–4 engineers) focused on building reliable, scalable, and secure environments...  ...high standards for quality and reliability Model effective AI-assisted development in your own hands-on work and set the... 
    Full time
    Temporary work
    Currently hiring
    Work at office
    Immediate start
    Remote work
    Work from home
    Home office
    Relocation package
    Flexible hours

    Niche

    Florida, FL
    18 hours ago
  • $170k - $275k

    Software Engineer, Backend and Infrastructure Title of Role: Software Engineer, Backend and Infrastructure Location: Florida, hybrid Company...  ...a dynamic company that is revolutionizing healthcare through AI technology. This organization focuses on developing innovative solutions... 
    Work at office

    Recruiting from Scratch

    Homestead, FL
    3 days ago
  •  ...customers include some of the world’s largest airlines and ground handling providers. For more information, visit . As a Senior Data Engineer at AeroVect, you will design, build, and scale the data infrastructure powering our autonomous ground support equipment (AGSE)... 

    AeroVect

    Florida, FL
    3 days ago
  • $140k - $175k

     ...generation solutions that patients and healthcare partners need today and into the future. Job Description The Senior Data Engineer will be responsible for building and maintaining the platform that services timely and accurate data for operational teams and... 
    Full time
    Temporary work
    Work experience placement
    Remote work
    Work visa
    Flexible hours

    Fuze Health

    Florida, FL
    1 day ago
  • $83.43k - $222.48k

     ...and data lakes by working with stakeholders to understand their requirements and translate them into scalable and efficient data engineering solutions Build, optimize, and deploy forecasting models using time-series modeling, machine learning, and MLOps practices Apply... 
    Hourly pay
    Full time
    Temporary work
    Work experience placement
    Local area

    CVS Health

    Florida, FL
    18 hours ago
  • $85.9k - $118.1k

     ...Power Global Supply Chains: Database Developer (Remote, US) Shape the future of global logistics by engineering high-availability database solutions that connect freight networks worldwide. If you thrive in a fast-paced, collaborative space turning complex data into seamless... 
    Full time
    Local area
    Remote work
    Worldwide

    Trimble Inc.

    Florida, FL
    1 day ago
  •  ...connectivity at 4G/5G speeds everywhere on the planet – on land, at sea and in flight. Position Summary: The Lead Manufacturing Engineer / Manufacturing Engineering Manager will oversee and optimize manufacturing processes for the Cover Glass Integrated Solar Cell (... 
    Work at office
    Shift work

    AST SpaceMobile

    Homestead, FL
    6 days ago
  •  ...support to stakeholders across the federal government, with established and growing capabilities across Intelligence, Analytics, Engineering, Mission Support, and Communications disciplines. Founded in 2008, our mission is to transform the way our customers approach constantly... 
    For contractors
    Work at office

    Barbaricum

    Homestead, FL
    13 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Evaluation Engineer. Be the first to apply!