AI Evaluation Engineer
Sedgwick
Fortune Best Workplaces in Financial Services & Insurance AI Evaluation Engineer PRIMARY PURPOSE: The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle. Fully remote role, open to U.Establishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release. Deploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency. Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., Partner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds. Develop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes. Implement offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts). Support A/B and shadow testing for model, prompt, and architecture changes. Build and curate golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches. Apply approved data-handling practices (de-identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data. Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing. Produce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews. Contribute to responsible AI standards and release-readiness criteria. Build dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends. Evaluate and recommend eval tooling and platforms; document standards and best practices. Travel as required. Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred. 4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market). A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review). Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement. Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services. Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders. Ability to meet or exceed Performance Competencies Background in ML evaluation, search/relevance, NLP, or QA automation. Experience with LLMOps, monitoring and observability, and production feedback loops. Familiarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments. Experience with data annotation workflows and working with SMEs. SSO, RBAC, private networking, logging, and encryption. Automated quality gates in CI/CD for production-bound solutions. Calibrated LLM judges with documented agreement against human labels. excellent judgment, troubleshooting, problem solving, analysis, and discretion; Computer keyboarding, travel as required Auditory/Visual: Hearing, vision and talking The statements contained in this document are intended to describe the general nature and level of work being performed by a colleague assigned to this description. They are not intended to constitute a comprehensive list of functions, duties, or local variances. Sedgwick is building a diverse, equitable, and inclusive workplace and recognizes that each person possesses a unique combination of skills, knowledge, and experience.
$110k
...smarter and safer communities with exciting AI, cloud, IoT, and cyber-physical security... ...shape the future of IT solutions. AI Engineer The AI Engineer is responsible for... ...ingestion, chunking, embeddings, retrieval, and evaluation Help configure deployments to meet...Suggested$73.59k - $81.77k
...Job Posting ID: KT_[273]_JOB Position: Software Engineer FT/PT Status: Full-Time Location: Remote within the following locations: DC, FL, GA, MD, TX, VA Education: Bachelor's degree in Computer Science, Software Engineering or a related field required....SuggestedFull timeWork at officeRemote work$106.61k - $284.28k
.... Position Summary The Lead Software Engineer for MinuteClinic will play a pivotal role... ...our entire technology stack and embrace AI-enabled engineering practices. In this... ...will proactively adopt new technologies, evaluate emerging tools—including AI-driven development...SuggestedHourly payFull timeTemporary workLocal area$120k - $125k
Senior Site Reliability Engineer & Duties: The Senior Site Reliability Engineer (Sr. SRE) ensures the reliability, performance, and... ...and availability reviews with product and engineering teams to evaluate and develop system health. Run regular Game Day and disaster recovery...SuggestedFull timePart timeFor contractorsCasual workWork at officeLocal areaRemote workWork from homeFlexible hours- ...Voyages is looking for a Principal Data Engineer to set the technical direction for how we... ...quality, and you'll lead the way in making AI-assisted development the everyday... ...RAG, embeddings, and vector search, plus evaluating and monitoring AI systems in production....SuggestedFull timeWork at officeRemote work
- ...predictive models to define process performance. Collaboration: Work closely with cross-functional teams, including chemists, engineers, etc., to support data-driven decision-making. Machine Learning: Apply machine learning algorithms to optimize process development...
- ...complex challenges in healthcare. We’re searching for a Software Engineer to help us build and scale our platform. You’ll own the... ...ownership of our results. We also embrace new technology, including AI-powered tools, to improve both the quality and the speed of what...Work experience placementWork at officeRemote workRelocationFlexible hours
$86.6k - $144.4k
...opportunity to serve as the technical anchor of a cross-functional engineering squad, driving architecture decisions, mentoring peers, and... ...generation healthcare data platform? Would you like to work in an AI-accelerated development environment where you can leverage...Local area$110k - $130k
...maintaining CI/CD pipelines and infrastructure as code to deploy solutions across AWS, OCI, Azure, and Kubernetes environments Using AI-assisted development tools to improve the speed and quality of delivery What you’ll bring Think you could make a difference...Remote workFlexible hours$15 per hour
...What if your knowledge Spanish language could help improve the AI. WHAT YOU’LL DO Review, evaluate, and validate AI-generated content and data according to project guidelines. Compare AI-generated outputs and identify the most accurate, relevant, or high-quality...Hourly payPart timeFreelanceWork at officeRemote work$90k - $130k
...SecOps Engineer Location: Remote CO, TX, MO, NC, FL, or GA Department: Data Science & Engineering Employment Type: Full-Time Salary Range: $90,000 - 130,000 About Shiplify We are a data provider serving the Less-Than-Truckload (LTL) freight industry...Full timeRemote work$106.61k - $284.28k
...bigger – helping to simplify health care one person, one family and one community at a time. Position Summary As a Staff Software Engineer, you'll work at the boundary between individual service teams and the larger system — identifying the architectural problems that...Hourly payFull timeContract workTemporary workLocal area- ...***CANDIDATE MUST BE US Citizen (due to contractual/access requirements)*** This job is an integral member of an agile software engineer team responsible for building scalable software applications across all digital channels to serve our customers. Responsible for taking...Temporary workFor contractorsWork at officeLocal area
- ...reinvent what a pharmacy and a health care company can be in the digital world. Currently, we are seeking a Staff Software Development Engineer with deep, hands‑on experience building and operating backend, platform, and integration services at scale. In this role, you will...Hourly payFull timeTemporary workLocal area
- ...SS&C is a leading provider of mission-critical, AI-powered technology and services empowering financial services and healthcare organizations... ..., scale, and technology. Job Description Senior Data Platform Engineer Role Summary We are seeking a Senior Data Platform Engineer...Ongoing contractWorldwide
$100.13k - $140.77k
...and code reviews; addresses findings. Required Qualifications: ~ Bachelor Degree in Computer Science, Information Technology, Engineering or related field ~ Ability to obtain and maintain a Public Trust Clearance ~ Must be willing and able to work EST hours ~2+...Local areaImmediate startFlexible hours- Position Summary The Application Developer configures, develops, and sustains Salesforce applications for a large enterprise Salesforce platform. This position carries a mix of configuration, development, and data remediation on the live platform, and owns assigned work...Full timeTemporary workRemote workRelocationFlexible hours
$90k
...submitted through Workday is reviewed by a real person — not an AI. We value your time and take each submission seriously. Who are... ...Element, a New Charter Company, we are seeking a remote Application Engineer, in other companies this person could hold the title of (...Remote work- ...Must live in Florida, Georgia, Nevada, Ohio, Tennessee or Texas to be considered. Do you find that you have a creative ability to engineer elegant solutions to complex problems or that you love to solve puzzles with code? We are looking for smart problem solvers who are...Full timeLive inLocal areaRemote workWork from homeFlexible hours
$127.8k - $135.9k
...work, from members of the public who rely on benefit programs to government agency staff. Through human-centered design and modern engineering best practices, we help our government partners understand user needs and deliver on their missions more effectively. This focus...Contract workTemporary workFor contractorsWork at officeLocal areaRemote workHome officeVisa sponsorshipFlexible hours$130k - $150k
...architecture and modernization efforts, and drive engineering best practices across the team. What... ...enhancement. Drive the adoption of AI-assisted development tools, prompt... ...software development lifecycle. Ability to evaluate, validate, and refine AI-generated code...Work experience placementWork at officeLocal area$15 per hour
...The Wikimedia Foundation is looking for a Senior Site Reliability Engineer to support and develop the platform serving the world’s favorite... ...), through critical thinking and expertise, with support of AI tooling Implementing and utilizing configuration management...Permanent employmentFor contractorsCurrently hiringLocal areaRemote work$146k - $183k
...people leader for a small, high-ownership Site Reliability team (3–4 engineers) focused on building reliable, scalable, and secure environments... ...high standards for quality and reliability Model effective AI-assisted development in your own hands-on work and set the...Full timeTemporary workCurrently hiringWork at officeImmediate startRemote workWork from homeHome officeRelocation packageFlexible hours$170k - $275k
Software Engineer, Backend and Infrastructure Title of Role: Software Engineer, Backend and Infrastructure Location: Florida, hybrid Company... ...a dynamic company that is revolutionizing healthcare through AI technology. This organization focuses on developing innovative solutions...Work at office- ...customers include some of the world’s largest airlines and ground handling providers. For more information, visit . As a Senior Data Engineer at AeroVect, you will design, build, and scale the data infrastructure powering our autonomous ground support equipment (AGSE)...
$140k - $175k
...generation solutions that patients and healthcare partners need today and into the future. Job Description The Senior Data Engineer will be responsible for building and maintaining the platform that services timely and accurate data for operational teams and...Full timeTemporary workWork experience placementRemote workWork visaFlexible hours$83.43k - $222.48k
...and data lakes by working with stakeholders to understand their requirements and translate them into scalable and efficient data engineering solutions Build, optimize, and deploy forecasting models using time-series modeling, machine learning, and MLOps practices Apply...Hourly payFull timeTemporary workWork experience placementLocal area$85.9k - $118.1k
...Power Global Supply Chains: Database Developer (Remote, US) Shape the future of global logistics by engineering high-availability database solutions that connect freight networks worldwide. If you thrive in a fast-paced, collaborative space turning complex data into seamless...Full timeLocal areaRemote workWorldwide- ...connectivity at 4G/5G speeds everywhere on the planet – on land, at sea and in flight. Position Summary: The Lead Manufacturing Engineer / Manufacturing Engineering Manager will oversee and optimize manufacturing processes for the Cover Glass Integrated Solar Cell (...Work at officeShift work
- ...support to stakeholders across the federal government, with established and growing capabilities across Intelligence, Analytics, Engineering, Mission Support, and Communications disciplines. Founded in 2008, our mission is to transform the way our customers approach constantly...For contractorsWork at office
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Evaluation Engineer. Be the first to apply!


