Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Artificial Intelligence Engineer

Full-time

Streetid

Senior AI Evaluations Engineer (AWS)

We are seeking a highly skilled Senior AI Evaluations Engineer to work closely with AWS Professional Services (ProServe) teams, designing and operating evaluation frameworks that determine whether agentic AI systems are accurate, safe, reliable, and ready for production.

This position supports large enterprise customers, primarily within financial services, and focuses on building sophisticated evaluation capabilities for multi-agent AI workflows leveraging Amazon Bedrock .

The ideal candidate combines advanced generative AI engineering expertise, strong Python development skills, and a rigorous, metrics-driven approach to evaluating production-grade AI systems.

Key Responsibilities

  • AI Evaluation Frameworks: Architect, develop, and maintain end-to-end evaluation frameworks for agentic AI systems, including offline regression testing, online monitoring, live-traffic evaluation, and A/B testing.
  • Rubrics & Scoring: Design quality rubrics, scoring architectures, and evaluation standards across complex multi-agent workflows.
  • LLM-as-Judge / Agent-as-Judge: Develop automated evaluation pipelines assessing task completion, tool-call accuracy, agent trajectories, groundedness, and safety.
  • Python Engineering: Build scalable, production-grade Python applications and evaluation services supporting AI model testing, benchmarking, and performance monitoring.
  • Evaluation Infrastructure: Own evaluation infrastructure from architecture through production deployment, integrating CI/CD pipelines, automated regression testing, observability, and monitoring.
  • Agentic AI Development: Work with frameworks such as LangGraph, Strands, and CrewAI to evaluate complex agent interactions and multi-step workflows.
  • AWS Integration: Leverage Amazon Bedrock, Guardrails, CloudWatch, Lambda, S3, and IAM to support secure, scalable evaluation systems.
  • Technical Leadership: Collaborate with AWS architects, engineering teams, and enterprise customers, providing guidance on AI evaluation methodologies and production readiness.

Required Qualifications

  • Advanced Python: Expert-level Python programming with demonstrated experience building production-grade AI applications, backend services, or evaluation frameworks.
  • Generative AI: Expert-level (Level 400) experience with LLMs, generative AI systems, and agentic AI architectures.
  • AI Evaluations: Hands-on experience designing and implementing evaluation frameworks for LLM-powered agents, including offline/on-demand regression testing and online/continuous live-traffic scoring.
  • LLM-as-Judge: Strong understanding of automated scoring, evaluation rubrics, and assessment at the session, response, and tool-call levels.
  • Evaluation Metrics: Experience measuring task completion, tool-call accuracy, trajectory quality, groundedness, and safety.
  • Agentic Frameworks: Hands-on experience with LangGraph, Strands, CrewAI, or similar agent orchestration frameworks.
  • AWS Expertise: Practical experience with Amazon Bedrock, Bedrock Guardrails, CloudWatch, Lambda, S3, and IAM.
  • Production Engineering: Proven ability to architect, deploy, and support scalable, reliable AI applications in enterprise environments.
  • AWS Certification: Must hold an active AWS Certified Generative AI Developer – Professional or AWS Certified Machine Learning Engineer – Associate certification.

Preferred Qualifications

  • Amazon Bedrock AgentCore Evaluations: Experience with on-demand SDK evaluation runners, online monitoring, and A/B testing.
  • Advanced Evaluation Techniques: Agent-as-a-judge architectures, human-in-the-loop (HITL) evaluation, and automated scoring systems.
  • Benchmarking: Familiarity with tau-bench, SWE-bench, GAIA, and custom benchmark dataset development.
  • RAG Evaluation: Experience evaluating retrieval-augmented generation systems using faithfulness, context precision, recall, and related quality metrics.
  • CI/CD & Observability: Automated evaluation regression testing, distributed tracing, structured logging, and production monitoring.
  • Legacy Modernization: Familiarity with Java/Spring, Sybase, DB2, or COBOL in the context of AI-driven modernization initiatives.
  • Financial Services: Experience supporting highly regulated financial services environments, including compliance, trade surveillance, data governance, and audit requirements.
  • Enterprise Scale: Experience delivering reusable AI evaluation frameworks across large application portfolios, including 50+ applications.
  • Consulting Experience: Previous experience working with AWS Professional Services or delivering complex AI engineering solutions to enterprise clients.

Ideal Candidate Profile

The ideal candidate is a hands-on Senior AI Engineer with deep expertise in Python, agentic AI, LLM evaluation frameworks, and AWS technologies .

This individual should have experience building and deploying production-grade AI evaluation systems, rather than solely developing LLM prototypes or proof-of-concept applications.

Strong candidates will bring a combination of advanced Python engineering, Amazon Bedrock expertise, LLM-as-judge methodologies, agent orchestration, automated evaluation infrastructure, and enterprise-scale AI delivery experience .

  • The role offers the opportunity to work closely with AWS Professional Services on cutting-edge agentic AI initiatives for some of the largest enterprise organizations in the financial services industry.

Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Artificial Intelligence Engineer in United States vacancy
  •  ..., Inc. Job Area: Interns Group, Interns Group Interim Engineering Intern - SW General Summary: About Qualcomm Qualcomm...  ...we're committed to engineering human progress by delivering intelligent computing everywhere. We're passionate about harnessing the power... 
    Suggested
    Interim role
    Summer work
    Internship
    Summer internship

    Qualcomm

    Raleigh, NC
    1 day ago
  • $240k - $340k

     ...Job Description AI Engineering Lead – Agentic AI & LLM Systems Location: New York, NY – Financial District (5 days onsite) Employment...  .... The platform uses large language models (LLMs), intelligent classification systems, and autonomous AI agents to analyse millions... 
    Suggested
    Permanent employment
    Full time
    Visa sponsorship

    Surge Group

    Mundelein, IL
    3 days ago
  •  ...Confidential AI Leadership Opportunity | Enterprise AI Architecture & Engineering Locations: Dallas, TX | Seattle, WA | Atlanta, GA | Chicago...  ...enterprise-grade AI applications, agentic systems, and intelligent automation solutions that solve real business problems.... 
    Suggested
    Full time
    3 days per week

    Harnham

    Chicago, IL
    12 days ago
  • $150k - $190k

     ...Gridiron IT is seeking an AI Engineer Senior to support a federal program in Chantilly, VA Chantilly, VA Bachelors required 8 years experience minimum Designs, develops, and deploys scalable, production-ready AI, machine learning (ML), and generative AI systems... 
    Suggested
    Full time
    Part time
    Work experience placement

    Gridiron IT

    Chantilly, Fairfax County, VA
    19 days ago
  •  ...Token Specialist will support the U.S. Cyber Command Chief Artificial Intelligence Officer by providing technical expertise in large language...  ...guidance. Advise leadership on model selection, prompt engineering, context-window management, token budgeting, and AI... 
    Suggested
    Full time

    Parsons Corporation

    United States
    4 days ago
  •  ...bringing together the urgency of a startup with a commitment to building reliable AI. ABOUT THE ROLE We are seeking an AI Engineer to build the AI, machine learning, and agentic systems behind a product tackling complex regulatory challenges. This is an opportunity... 
    Full time

    Opportunity Engine, Inc.

    Chicago, IL
    8 hours ago
  •  ...capabilities, they are seeking an experienced AI Engineer to help design, develop, and scale innovative generative AI and intelligent automation solutions across the enterprise....  ..., helping to develop and scale innovative artificial intelligence solutions across the business.... 
    Full time

    Qorali

    Baltimore, MD
    8 hours ago
  •  ...Top tier financial investment firm seeks an AI Engineer who will lead the design, build, and end-to-end delivery of business-facing AI applications and automation across the Firm. As a member of the AI team within Business Technology, the AI Engineer will work directly... 
    Full time
    Temporary work

    PRI Technology

    New York, NY
    3 days ago
  •  ...Robert Half Technology is seeking a talented Senior Applied AI Engineer for an exciting opportunity with a mission-driven organization...  ...innovative AI concepts into production-ready applications and intelligent automation solutions. The ideal candidate has extensive... 

    Robert Half

    San Diego, CA
    12 days ago
  •  ...Administration, Information Systems, Data Analytics, Industrial Engineering, or a related field, or equivalent experience. • PMP, CBAP,...  ...1+ years leading or contributing to AI, machine learning, or intelligent automation initiatives in a business setting; experience... 
    Casual work
    Work at office

    FIT Solutions, LLC

    San Marcos, CA
    5 days ago
  •  ...actions remain traceable and reviewable in a regulated environment Qualifications: · Bachelor's degree in Computer Science, Engineering, or a related technical field; Master's preferred ·4-7 years of software engineering experience, with production AI/LLM systems... 
    Permanent employment
    Full time
    Remote work

    Request Technology, LLC

    Chicago, IL
    a month ago
  • $99k - $225k

     ...Job Number: R0252233 Artificial Intelligence Engineer The Opportunity: As an AI Engineer, you will bring advanced AI capabilities into production systems by integrating models, inference endpoints, agent-orchestration frameworks, and intelligent workflows into... 
    Full time
    Contract work
    Part time
    Local area
    Remote work

    Booz Allen Hamilton

    San Antonio, TX
    4 days ago
  •  ...with a leading accounting and advisory firm, is seeking an AI Engineer to design, develop, and implement agentic AI solutions that enhance...  ...Generative AI and agent-based technologies to build intelligent solutions that improve automation, efficiency, and business operations... 

    Optomi

    New York, NY
    8 hours ago
  •  ...collaborating closely with business stakeholders, data scientists, and engineering teams. Key Responsibilities: Design and architect...  ...and implementation of scalable AI applications and intelligent automation solutions. Architect end-to-end RAG solutions,... 
    Full time
    Local area

    Covetus

    United States
    3 days ago
  •  ...Job Description Job Description Job Description acuCyber is seeking an AI Engineer to design and deploy secure AI systems that connect foundation models with enterprise tools, data, and services. You will build agent architectures, advanced retrieval capabilities... 
    Full time
    Temporary work

    acuCyber

    Reston, VA
    1 day ago
  •  ...About the Opportunity We're partnering with a growing technology company looking for a Senior AI Engineer to build and scale AI-powered software applications. This is a hands-on engineering role focused on developing production-ready AI solutions, with significant... 
    Full time

    Colossus Technologies Group

    New York, NY
    3 days ago
  •  ...Reporting to the AI Business Systems leadership team, the AI Engineer / Architect will help establish scalable development...  ...Microsoft technology stack using traditional automation and artificial intelligence methods. Define and standardize agentic development practices... 
    Local area

    The Phoenix Group

    New York, NY
    12 days ago
  •  ...We're building a full AI engineering team for a federal agency in Washington, DC. These are long-term contracts (3-4 years) with a hybrid schedule: on-site in Washington, DC 3 days per week. Requirement: US Citizens and Green Card holders only (government requirement... 
    Hourly pay
    Long term contract
    3 days per week

    Synergy Business Consulting, Inc.

    Washington DC
    4 days ago
  • $82.08k - $193.44k

     ...At Capgemini Engineering, the world leader in engineering services, we bring together a global team of engineers, scientists, and architects to help the world’s most innovative companies unleash their potential. From autonomous cars to life-saving robots, our digital and... 
    Full time
    Local area
    Night shift

    Capgemini Engineering

    Denver, CO
    22 days ago
  •  ...home to 230,000+ people across 60 countries, supercharging progress through industry-leading capabilities centered around Digital, Engineering and Cloud. About the Role Design and implement "loop engineering" workflows using Claude Code and similar tools, defining... 
    Contract work
    Local area

    HCLTech

    Hoboken, NJ
    8 days ago
  • $150k - $200k

     ...sell to, that turns their retirement plan business into a growth engine. We're a small team split across the US and Spain. We...  ...cost through scoped permissions, validation, prompt caching, and intelligent model routing. WHO YOU ARE You want your work to... 
    Work at office
    Flexible hours

    TIFIN

    Boulder, CO
    12 days ago
  •  ...Job Title: AI Engineer Location: Charlotte, NC (Remote) Type: W2 Contract What are the top 3 skills required for this role? 1) Agentic AI, Gen AI, Prompt Designing, Advanced RAG Architecture, LLM 2) AWS core services including Bedrock 3) Python, PySpark... 
    Contract work
    Remote work

    New York Technology Partners

    Charlotte, NC
    14 hours ago
  • $99k - $225k

     ...Job Number: R0247598 Artificial Intelligence Engineer, Senior The Opportunity: Great AI solutions start with understanding the problem-then designing so met hing that actually works in the real world. As an Artifi cia l Intelligence Engineer on our team, you'll... 
    Full time
    Contract work
    Part time
    Work at office
    Local area
    Remote work
    Day shift

    Booz Allen Hamilton

    El Segundo, CA
    8 hours ago
  • $77.6k - $176k

     ...Job Number: R0251799 Business Intelligence Engineer The Opportunity: Do you thrive by using your analytical expertise to convert large...  ...process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity... 
    Full time
    Contract work
    Part time
    Local area
    Remote work

    Booz Allen Hamilton

    McLean, VA
    1 day ago
  • $70 per hour

     ...Agentic AI Engineer / Applied AI Engineer Job Description We are seeking a highly skilled **Agentic AI Engineer / Applied AI...  ..., risk scoring, anomaly detection, compliance automation, and intelligent business operations**. * Establish evaluation frameworks and... 

    iT Resource Solutions.net,inc

    New York, NY
    12 days ago
  •  ...Hiring Fulltime AI Engineer, Englewood Cliffs, NJ- Onsite 5 days a week. NO C2C Job Description: This position will identify high-impact business opportunities, design scalable AI-enabled solutions, and build production-ready capabilities that improve how teams... 
    Full time

    Marlabs

    Englewood, NJ
    19 days ago
  • $150k

     ...AI Engineer | Charlotte | Onsite 4 days per week. $150,000 Take ownership of a live conversational AI system used every day by real customers, and shape where it goes next. My client is an established fleet management software company. Their AI assistant answers... 

    Accelero

    Charlotte, NC
    4 days ago
  •  ...Role: Artificial Intelligence Engineer Location: Charlotte, NC We are At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading... 
    Full time
    Temporary work
    Flexible hours

    Synechron

    Charlotte, NC
    3 days ago
  • $62k - $80k

     ...is based in Central NJ, specializing in Software Development, Web Development, and Business Intelligence. About the Role We are looking for motivated Software Engineers with a strong programming and object-oriented foundation to build scalable, production-ready... 
    Full time
    Visa sponsorship

    BeaconFire Inc.

    New York, NY
    1 day ago
  •  ...making remote work more accessible and straightforward for both talent and employers. Role Description The Artificial Intelligence Engineer role at Remolity is a full-time remote position focused on designing, developing, and deploying AI models that improve... 
    Full time
    Remote work

    Remolity

    United States
    4 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Artificial Intelligence Engineer. Be the first to apply!