Artificial Intelligence Engineer

Job Description

Senior AI Evaluations Engineer (AWS)

We are seeking a highly skilled Senior AI Evaluations Engineer to work closely with AWS Professional Services (ProServe) teams, designing and operating evaluation frameworks that determine whether agentic AI systems are accurate, safe, reliable, and ready for production.

This position supports large enterprise customers, primarily within financial services, and focuses on building sophisticated evaluation capabilities for multi-agent AI workflows leveraging Amazon Bedrock .

The ideal candidate combines advanced generative AI engineering expertise, strong Python development skills, and a rigorous, metrics-driven approach to evaluating production-grade AI systems.

Key Responsibilities

\n
• AI Evaluation Frameworks: Architect, develop, and maintain end-to-end evaluation frameworks for agentic AI systems, including offline regression testing, online monitoring, live-traffic evaluation, and A/B testing.
\n
• Rubrics & Scoring: Design quality rubrics, scoring architectures, and evaluation standards across complex multi-agent workflows.
\n
• LLM-as-Judge / Agent-as-Judge: Develop automated evaluation pipelines assessing task completion, tool-call accuracy, agent trajectories, groundedness, and safety.
\n
• Python Engineering: Build scalable, production-grade Python applications and evaluation services supporting AI model testing, benchmarking, and performance monitoring.
\n
• Evaluation Infrastructure: Own evaluation infrastructure from architecture through production deployment, integrating CI/CD pipelines, automated regression testing, observability, and monitoring.
\n
• Agentic AI Development: Work with frameworks such as LangGraph, Strands, and CrewAI to evaluate complex agent interactions and multi-step workflows.
\n
• AWS Integration: Leverage Amazon Bedrock, Guardrails, CloudWatch, Lambda, S3, and IAM to support secure, scalable evaluation systems.
\n
• Technical Leadership: Collaborate with AWS architects, engineering teams, and enterprise customers, providing guidance on AI evaluation methodologies and production readiness.
\n

Required Qualifications

\n
• Advanced Python: Expert-level Python programming with demonstrated experience building production-grade AI applications, backend services, or evaluation frameworks.
\n
• Generative AI: Expert-level (Level 400) experience with LLMs, generative AI systems, and agentic AI architectures.
\n
• AI Evaluations: Hands-on experience designing and implementing evaluation frameworks for LLM-powered agents, including offline/on-demand regression testing and online/continuous live-traffic scoring.
\n
• LLM-as-Judge: Strong understanding of automated scoring, evaluation rubrics, and assessment at the session, response, and tool-call levels.
\n
• Evaluation Metrics: Experience measuring task completion, tool-call accuracy, trajectory quality, groundedness, and safety.
\n
• Agentic Frameworks: Hands-on experience with LangGraph, Strands, CrewAI, or similar agent orchestration frameworks.
\n
• AWS Expertise: Practical experience with Amazon Bedrock, Bedrock Guardrails, CloudWatch, Lambda, S3, and IAM.
\n
• Production Engineering: Proven ability to architect, deploy, and support scalable, reliable AI applications in enterprise environments.
\n
• AWS Certification: Must hold an active AWS Certified Generative AI Developer – Professional or AWS Certified Machine Learning Engineer – Associate certification.
\n

Preferred Qualifications

\n
• Amazon Bedrock AgentCore Evaluations: Experience with on-demand SDK evaluation runners, online monitoring, and A/B testing.
\n
• Advanced Evaluation Techniques: Agent-as-a-judge architectures, human-in-the-loop (HITL) evaluation, and automated scoring systems.
\n
• Benchmarking: Familiarity with tau-bench, SWE-bench, GAIA, and custom benchmark dataset development.
\n
• RAG Evaluation: Experience evaluating retrieval-augmented generation systems using faithfulness, context precision, recall, and related quality metrics.
\n
• CI/CD & Observability: Automated evaluation regression testing, distributed tracing, structured logging, and production monitoring.
\n
• Legacy Modernization: Familiarity with Java/Spring, Sybase, DB2, or COBOL in the context of AI-driven modernization initiatives.
\n
• Financial Services: Experience supporting highly regulated financial services environments, including compliance, trade surveillance, data governance, and audit requirements.
\n
• Enterprise Scale: Experience delivering reusable AI evaluation frameworks across large application portfolios, including 50+ applications.
\n
• Consulting Experience: Previous experience working with AWS Professional Services or delivering complex AI engineering solutions to enterprise clients.
\n

Ideal Candidate Profile

The ideal candidate is a hands-on Senior AI Engineer with deep expertise in Python, agentic AI, LLM evaluation frameworks, and AWS technologies .

This individual should have experience building and deploying production-grade AI evaluation systems, rather than solely developing LLM prototypes or proof-of-concept applications.

Strong candidates will bring a combination of advanced Python engineering, Amazon Bedrock expertise, LLM-as-judge methodologies, agent orchestration, automated evaluation infrastructure, and enterprise-scale AI delivery experience .

\n
• The role offers the opportunity to work closely with AWS Professional Services on cutting-edge agentic AI initiatives for some of the largest enterprise organizations in the financial services industry.
\n