Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Evaluation Engineer

Sedgwick

By joining Sedgwick, you'll be part of something truly meaningful. It’s what our 33,000 colleagues do every day for people around the world who are facing the unexpected. We invite you to grow your career with us, experience our caring culture, and enjoy work-life balance. Here, there’s no limit to what you can achieve.

Newsweek Recognizes Sedgwick as America’s Greatest Workplaces National Top CompaniesCertified as a Great Place to WorkFortune Best Workplaces in Financial Services & InsuranceAI Evaluation Engineer

PRIMARY PURPOSE:

The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle.
Fully remote role, open to U.S. based candidates.

ESSENTIAL FUNCTIONS AND RESPONSIBILITIES
  • Establishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release.
  • Deploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency.
  • Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., faithfulness, groundedness, relevance, task completion, tool‑use correctness, safety).
  • Build reusable evaluation frameworks, templates, and libraries that pods can adopt with minimal effort.
  • Partner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds.
  • Develop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes.
  • Implement offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts).
  • Support A/B and shadow testing for model, prompt, and architecture changes.
  • Design, validate, and maintain LLM‑as‑a‑judge evaluators, including rubric design, bias mitigation (position, verbosity, self‑preference), and calibration against human labels.
  • Establish human annotation workflows, labeling guidelines, and inter‑annotator agreement practices with SMEs.
  • Track judge reliability over time and re‑calibrate as models and use cases change.
  • Build and curate golden datasets, adversarial and edge‑case suites, and synthetic test data generation approaches.
  • Apply approved data‑handling practices (de‑identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data.
  • Implement red‑teaming, hallucination, toxicity, PII leakage, prompt‑injection, and jailbreak testing.
  • Produce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews.
  • Contribute to responsible AI standards and release‑readiness criteria.
  • Embed with pods to onboard them to evaluation practices, then coach teams toward ownership of their own eval suites.
  • Build dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends.
  • Evaluate and recommend eval tooling and platforms; document standards and best practices.
ADDITIONAL FUNCTIONS AND RESPONSIBILITIES
  • Performs other duties as assigned.
  • Travel as required.
QUALIFICATIONS
  • Education & Licensing
  • Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred.
  • Experience
  • 4+ years in software, ML, or data engineering, with at least 1-2 years working hands‑on with LLM‑based applications (adjust to your market).
  • Skills & Knowledge
  • Professional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs
  • A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance
  • Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review).
  • Practical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics.
  • Hands‑on experience with LLM‑as‑a‑judge design, prompt engineering, and judge validation.
  • Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter‑rater agreement.
  • Familiarity with at least one eval or observability tool (e.g., Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent).
  • Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services.
  • Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders.
  • Excellent oral and written communication skills, including presentation skills
  • Ability to work in a team environment
  • Ability to meet or exceed Performance Competencies
Preferred
  • Experience evaluating agentic systems (multi‑step reasoning, tool calling, trajectory evaluation).
  • Background in ML evaluation, search/relevance, NLP, or QA automation.
  • Experience with LLMOps, monitoring and observability, and production feedback loops.
  • Familiarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments.
  • Experience with data annotation workflows and working with SMEs.
  • Knowledge of enterprise controls: SSO, RBAC, private networking, logging, and encryption.
SUCCESS MEASURES

(first 12 months)
Standard evaluation framework adopted by pods.
Automated quality gates in CI/CD for production‑bound solutions.
Calibrated LLM judges with documented agreement against human labels.
Reduction in regressions reaching production and in time to validate model or prompt changes.

WORK ENVIRONMENT

When applicable and appropriate, consideration will be given to reasonable accommodations.

Mental: Clear and conceptual thinking ability; excellent judgment, troubleshooting, problem solving, analysis, and discretion; ability to handle work‑related stress; ability to handle multiple priorities simultaneously; and ability to meet deadlines.

Physical: Computer keyboarding, travel as required

Auditory/Visual: Hearing, vision and talking

Sedgwick is an Equal Opportunity Employer and a Drug-Free Workplace.

Sedgwick is building a diverse, equitable, and inclusive workplace and recognizes that each person possesses a unique combination of skills, knowledge, and experience. You may be just the right candidate for this or other roles.

#J-18808-Ljbffr
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the AI Evaluation Engineer in New York, NY vacancy
  • $229.9k - $262.4k

    AI Engineer 5 (Gen AI SDK: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One has been an industry leader in using machine learning to create real-time, personalized... 
    Suggested
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    1 day ago
  • $197.3k - $225.1k

    AI Engineer 4 (AI Foundations: Benchmarking, Evaluation, and Explainability) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One has been an industry leader in using machine learning to create real-time,... 
    Suggested
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    1 day ago
  • $229.9k - $262.4k

    Sr. Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One has been an industry leader in using machine learning to create... 
    Suggested
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    1 day ago
  •  ...Job Description Job Description Job Title AI Evaluation Engineer Location Hybrid / Remote Employment Type Full-time Job Summary We are seeking an AI Evaluation Engineer to design, implement, and maintain evaluation frameworks for AI and machine learning... 
    Suggested
    Full time
    Remote work

    Ova Technologies

    New York, NY
    25 days ago
  • $176k

     ...Current PhD, AI Engineering Internship Program - Summer 2027 Key Role Details This is a full-time paid internship program spanning twelve...  ...model , for the duration of the program. Interns will be evaluated for a full-time position with their team and will work with... 
    Suggested
    Full time
    Part time
    Summer work
    Internship
    Local area

    Capital One

    New York, NY
    2 days ago
  • $170k - $300k

     ...AI Engineer | $10B+ Consumer FinTech | NYC $170k–$300k Base + Bonus + Equity New York City | 5 Days Onsite I'm partnering with...  ...products. Improving LLM performance, reliability, evaluation frameworks and latency at scale. Building intelligent experiences... 
    Local area

    Orbis Group

    New York, NY
    17 days ago
  •  ...AI Engineer (GenAI Workflows for OTC Derivatives) Location: New York City (SoHo) — In-Person About Our Client Our client is...  ...the OTC derivatives lifecycle. You’ll own key parts of the evaluation and context engineering infrastructure—ensuring model outputs... 
    Full time

    Nxt Level

    New York, NY
    2 days ago
  •  ...The Senior AI Engineer reports to the Lead AI Engineer and serves as their primary technical partner in designing and delivering secure...  ...established by the Lead AI Engineer. Build and apply evaluation frameworks for quality, groundedness, safety, latency, reliability... 
    Full time

    Atlantic Partners Corporation

    New York, NY
    10 hours ago
  • $210k - $250k

     ...both clients and candidates. Title of Role: Founding AI Engineer Location: New York City, NY (Flatiron District) Company...  ...end, from architecture and implementation through deployment, evaluation, and ongoing improvement. Build reliable agent systems... 
    Full time
    Work at office

    Recruiting from Scratch

    New York, NY
    14 days ago
  •  ...Title: AI Engineer Location: New York, NY (Hybrid) Duration: 12 month Contracts with possibility of extension or contract to hire...  ...accuracy, consistency, explainability, and user trust. • Design evaluation frameworks to test AI outputs for accuracy, completeness,... 
    Contract work
    Local area

    Russell Tobin

    New York, NY
    17 hours ago
  • $160k - $200k

     ...AI Engineer [Senior Associate / AVP] Founded in 1992, Cerberus is a global leader in alternative investing with approximately $71...  ...or similar services, including tool use, structured outputs, evaluation and monitoring considerations. Agent frameworks and observability... 

    Cerberus Capital Management

    New York, NY
    3 days ago
  • $180k - $250k

     ...AI Engineer – Agentic AI / LLM Systems New York | Series A Startup $180–$250k + Equity In office (5 days) No visa sponsorship...  ..., multi-agent systems, and learning workflows Build evaluation and confidence frameworks to determine when to automate vs escalate... 
    Full time
    Work at office
    Visa sponsorship

    Stott and May

    New York, NY
    1 day ago
  •  ...Founding Engineer (AI) NYC | In-Person | Founding Team We're looking for a rare builder who can ship products, build AI systems from...  ...-on experience with AI agents, tool calling, memory, RAG, evaluations, and prompt systems Proven track record of building and shipping... 
    Full time

    Stable10

    New York, NY
    1 day ago
  •  ...Senior AI Engineer Overview This well-funded, growth-stage healthcare technology company pairs healthcare expertise with AI to...  ...Python and engineering discipline, plus a rigorous approach to evaluation Nice to Haves GPU and distributed training experience... 
    Full time

    Discover International

    New York, NY
    1 day ago
  •  ...Bachelor’s degree in computer science, Engineering, Mathematics, Finance, or a related field...  ...output validation. Experience building AI or data-driven applications that use...  ...intelligence. Experience designing AI evaluation frameworks, including test datasets, quality... 
    H1b
    Local area

    Robotic Technologies Automation

    New York, NY
    5 days ago
  • $75 - $105 per hour

     ...Review Python code, tests, and execution logs from AI coding tasks to determine why a solution failed and...  ...’ll work with a leading AI research organization to evaluate how coding agents perform on software engineering tasks. A failing test might point to a bug in the agent... 
    Hourly pay
    Immediate start
    Remote work

    Braintrust

    New York, NY
    4 days ago
  • $180k - $250k

     ...About Distyl AI Distyl is an applied AI technology company partnering with the...  ...What We Are Looking For AI Engineers build and operate production AI systems that...  ...and continuously improve systems through evaluation, feedback, integration, and production iteration... 
    Work at office
    Flexible hours
    3 days per week

    Distyl AI

    New York, NY
    5 days ago
  •  ...Build and accelerate the firmwide AI ecosystem and infrastructure to support AI...  ...systems and automation Design, deploy, and evaluate production-grade AI applications and advanced...  ...in Python alongside solid software engineering principles and a deep understanding of the... 

    Radix Trading LLC

    New York, NY
    1 day ago
  • $143k - $210k

     ...CoreWeave is The Essential Cloud for AI. Built for pioneers by pioneers, CoreWeave delivers...  .... Learn more at.What You'll Do:The Field Engineering organization at CoreWeave supports the...  ...which the system should stop.- Build and evaluate retrieval over a large body of historical... 
    Full time
    Casual work
    Work at office

    CoreWeave

    New York, NY
    3 days ago
  •  ...Job Title: Agentic AI Engineer Location: NYC, NY (Hybrid) Job Summary: We are seeking an Agentic AI Engineer with hands-on experience...  ..., AutoGen, CrewAI, Strands or equivalent technologies. Evaluate, test, and improve agent performance, including prompt quality... 

    Jobs via Dice

    New York, NY
    4 days ago
  •  ...uncertainty for developers nationwide. We combine AI-powered technology with an in-house team of registered architects, engineers, and city planners to deliver the fastest,...  .../Code-Interpreter where appropriate. Evaluation & safety harness: Stand up task-level and... 
    Full time
    Summer work
    Work at office
    Flexible hours

    GreenLite Technologies, Inc.

    New York, NY
    2 days ago
  •  ...new category of enterprise software: an AI platform that changes how the world's largest...  ...We don't. Tessera is a transformation engine: a governed, multi-agent platform that understands...  ...search, reranking, grounding, and the evaluation that tells you whether any of it helped.... 

    Tessera Labs

    New York, NY
    5 days ago
  •  ...AI Engineer We're hiring for our first AI Engineer. Someone who's excited to own and build all things AI at Coast. You'll be responsible...  ...to or built internal tooling for prompt testing and LLM evaluation A strong product sense and love building intuitive UI/UX alongside... 

    Coast

    New York, NY
    5 days ago
  •  ...Role : AI Engineer Location : New York Its day onsite role Skills - AI, AWS, TypeScript, JavaScript, Python, Go...  ...pipelines • LLM infrastructure, inference, and model gateways • Evaluation, observability, and safety tooling for autonomous systems •... 

    Fisec Global

    New York, NY
    1 day ago
  •  ...AI Engineer Stripe is a financial infrastructure platform for businesses. Millions of companies—from the world's largest enterprises...  ...translate ambiguous business needs into well-defined deliverables Evaluate and integrate AI capabilities (LLMs, agents, workflow... 
    Immediate start

    CapitalG

    New York, NY
    2 days ago
  •  ...AI Engineer Most AI systems are built as one-size-fits-all solutions. Powerful in benchmarks, brittle in production. In a real workplace...  ..., acceleration, and agent pipeline optimization Build evaluation frameworks and benchmarks to measure agent performance... 
    Live in

    Stochastic

    New York, NY
    4 days ago
  • $180k - $280k

     ...AI Engineer Title of Role: AI Engineer Location: New York, hybrid Company Stage of Funding: Venture-Backed — Healthcare, Fintech...  ...handling denials and unpaid claims. Conduct experiments to evaluate model effectiveness and iterate based on findings.... 
    Work at office

    Recruiting from Scratch

    New York, NY
    2 days ago
  • $120k - $145k

     ...00.00 To $145,000.00 Annually Purpose of Position: The AI Engineer is responsible for developing, testing, deploying, and supporting...  ...integration, and agent orchestration activities. Model Evaluation, Monitoring, and Optimization: • Support the creation and... 
    Work experience placement
    Work at office
    Remote work
    Visa sponsorship
    Work visa

    Amalgamated Bank

    New York, NY
    4 days ago
  •  ...AI Engineer StatusNeo is a global AI-native transformation firm helping enterprises design, engineer, and govern AI-led systems with...  ...automation, or healthcare knowledge systems. Experience evaluating LLM performance using frameworks such as Ragas, DeepEval, or... 

    Statusneo

    New York, NY
    2 days ago
  • $65k - $80k

     ...Join our team as a Junior Generative AI Engineer and help develop, test, and improve generative AI solutions that address real-world business...  ...Prepare, process, and organize data used by AI applications. Evaluate AI-generated responses for quality, accuracy, and relevance.... 
    H1b
    Remote work
    Visa sponsorship

    BeaconFire Inc.

    New York, NY
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Evaluation Engineer. Be the first to apply!