Artificial Intelligence Engineer
Streetid
Senior AI Evaluations Engineer (AWS)
We are seeking a highly skilled Senior AI Evaluations Engineer to work closely with AWS Professional Services (ProServe) teams, designing and operating evaluation frameworks that determine whether agentic AI systems are accurate, safe, reliable, and ready for production.
This position supports large enterprise customers, primarily within financial services, and focuses on building sophisticated evaluation capabilities for multi-agent AI workflows leveraging Amazon Bedrock .
The ideal candidate combines advanced generative AI engineering expertise, strong Python development skills, and a rigorous, metrics-driven approach to evaluating production-grade AI systems.
Key Responsibilities
- AI Evaluation Frameworks: Architect, develop, and maintain end-to-end evaluation frameworks for agentic AI systems, including offline regression testing, online monitoring, live-traffic evaluation, and A/B testing.
- Rubrics & Scoring: Design quality rubrics, scoring architectures, and evaluation standards across complex multi-agent workflows.
- LLM-as-Judge / Agent-as-Judge: Develop automated evaluation pipelines assessing task completion, tool-call accuracy, agent trajectories, groundedness, and safety.
- Python Engineering: Build scalable, production-grade Python applications and evaluation services supporting AI model testing, benchmarking, and performance monitoring.
- Evaluation Infrastructure: Own evaluation infrastructure from architecture through production deployment, integrating CI/CD pipelines, automated regression testing, observability, and monitoring.
- Agentic AI Development: Work with frameworks such as LangGraph, Strands, and CrewAI to evaluate complex agent interactions and multi-step workflows.
- AWS Integration: Leverage Amazon Bedrock, Guardrails, CloudWatch, Lambda, S3, and IAM to support secure, scalable evaluation systems.
- Technical Leadership: Collaborate with AWS architects, engineering teams, and enterprise customers, providing guidance on AI evaluation methodologies and production readiness.
Required Qualifications
- Advanced Python: Expert-level Python programming with demonstrated experience building production-grade AI applications, backend services, or evaluation frameworks.
- Generative AI: Expert-level (Level 400) experience with LLMs, generative AI systems, and agentic AI architectures.
- AI Evaluations: Hands-on experience designing and implementing evaluation frameworks for LLM-powered agents, including offline/on-demand regression testing and online/continuous live-traffic scoring.
- LLM-as-Judge: Strong understanding of automated scoring, evaluation rubrics, and assessment at the session, response, and tool-call levels.
- Evaluation Metrics: Experience measuring task completion, tool-call accuracy, trajectory quality, groundedness, and safety.
- Agentic Frameworks: Hands-on experience with LangGraph, Strands, CrewAI, or similar agent orchestration frameworks.
- AWS Expertise: Practical experience with Amazon Bedrock, Bedrock Guardrails, CloudWatch, Lambda, S3, and IAM.
- Production Engineering: Proven ability to architect, deploy, and support scalable, reliable AI applications in enterprise environments.
- AWS Certification: Must hold an active AWS Certified Generative AI Developer – Professional or AWS Certified Machine Learning Engineer – Associate certification.
Preferred Qualifications
- Amazon Bedrock AgentCore Evaluations: Experience with on-demand SDK evaluation runners, online monitoring, and A/B testing.
- Advanced Evaluation Techniques: Agent-as-a-judge architectures, human-in-the-loop (HITL) evaluation, and automated scoring systems.
- Benchmarking: Familiarity with tau-bench, SWE-bench, GAIA, and custom benchmark dataset development.
- RAG Evaluation: Experience evaluating retrieval-augmented generation systems using faithfulness, context precision, recall, and related quality metrics.
- CI/CD & Observability: Automated evaluation regression testing, distributed tracing, structured logging, and production monitoring.
- Legacy Modernization: Familiarity with Java/Spring, Sybase, DB2, or COBOL in the context of AI-driven modernization initiatives.
- Financial Services: Experience supporting highly regulated financial services environments, including compliance, trade surveillance, data governance, and audit requirements.
- Enterprise Scale: Experience delivering reusable AI evaluation frameworks across large application portfolios, including 50+ applications.
- Consulting Experience: Previous experience working with AWS Professional Services or delivering complex AI engineering solutions to enterprise clients.
Ideal Candidate Profile
The ideal candidate is a hands-on Senior AI Engineer with deep expertise in Python, agentic AI, LLM evaluation frameworks, and AWS technologies .
This individual should have experience building and deploying production-grade AI evaluation systems, rather than solely developing LLM prototypes or proof-of-concept applications.
Strong candidates will bring a combination of advanced Python engineering, Amazon Bedrock expertise, LLM-as-judge methodologies, agent orchestration, automated evaluation infrastructure, and enterprise-scale AI delivery experience .
- The role offers the opportunity to work closely with AWS Professional Services on cutting-edge agentic AI initiatives for some of the largest enterprise organizations in the financial services industry.
- ..., Inc. Job Area: Interns Group, Interns Group Interim Engineering Intern - SW General Summary: About Qualcomm Qualcomm... ...we're committed to engineering human progress by delivering intelligent computing everywhere. We're passionate about harnessing the power...SuggestedInterim roleSummer workInternshipSummer internship
$240k - $340k
...Job Description AI Engineering Lead – Agentic AI & LLM Systems Location: New York, NY – Financial District (5 days onsite) Employment... .... The platform uses large language models (LLMs), intelligent classification systems, and autonomous AI agents to analyse millions...SuggestedPermanent employmentFull timeVisa sponsorship- ...Confidential AI Leadership Opportunity | Enterprise AI Architecture & Engineering Locations: Dallas, TX | Seattle, WA | Atlanta, GA | Chicago... ...enterprise-grade AI applications, agentic systems, and intelligent automation solutions that solve real business problems....SuggestedFull time3 days per week
$150k - $190k
...Gridiron IT is seeking an AI Engineer Senior to support a federal program in Chantilly, VA Chantilly, VA Bachelors required 8 years experience minimum Designs, develops, and deploys scalable, production-ready AI, machine learning (ML), and generative AI systems...SuggestedFull timePart timeWork experience placement- ...Token Specialist will support the U.S. Cyber Command Chief Artificial Intelligence Officer by providing technical expertise in large language... ...guidance. Advise leadership on model selection, prompt engineering, context-window management, token budgeting, and AI...SuggestedFull time
- ...bringing together the urgency of a startup with a commitment to building reliable AI. ABOUT THE ROLE We are seeking an AI Engineer to build the AI, machine learning, and agentic systems behind a product tackling complex regulatory challenges. This is an opportunity...Full time
- ...capabilities, they are seeking an experienced AI Engineer to help design, develop, and scale innovative generative AI and intelligent automation solutions across the enterprise.... ..., helping to develop and scale innovative artificial intelligence solutions across the business....Full time
- ...Top tier financial investment firm seeks an AI Engineer who will lead the design, build, and end-to-end delivery of business-facing AI applications and automation across the Firm. As a member of the AI team within Business Technology, the AI Engineer will work directly...Full timeTemporary work
- ...Robert Half Technology is seeking a talented Senior Applied AI Engineer for an exciting opportunity with a mission-driven organization... ...innovative AI concepts into production-ready applications and intelligent automation solutions. The ideal candidate has extensive...
- ...Administration, Information Systems, Data Analytics, Industrial Engineering, or a related field, or equivalent experience. • PMP, CBAP,... ...1+ years leading or contributing to AI, machine learning, or intelligent automation initiatives in a business setting; experience...Casual workWork at office
- ...actions remain traceable and reviewable in a regulated environment Qualifications: · Bachelor's degree in Computer Science, Engineering, or a related technical field; Master's preferred ·4-7 years of software engineering experience, with production AI/LLM systems...Permanent employmentFull timeRemote work
$99k - $225k
...Job Number: R0252233 Artificial Intelligence Engineer The Opportunity: As an AI Engineer, you will bring advanced AI capabilities into production systems by integrating models, inference endpoints, agent-orchestration frameworks, and intelligent workflows into...Full timeContract workPart timeLocal areaRemote work- ...with a leading accounting and advisory firm, is seeking an AI Engineer to design, develop, and implement agentic AI solutions that enhance... ...Generative AI and agent-based technologies to build intelligent solutions that improve automation, efficiency, and business operations...
- ...collaborating closely with business stakeholders, data scientists, and engineering teams. Key Responsibilities: Design and architect... ...and implementation of scalable AI applications and intelligent automation solutions. Architect end-to-end RAG solutions,...Full timeLocal area
- ...Job Description Job Description Job Description acuCyber is seeking an AI Engineer to design and deploy secure AI systems that connect foundation models with enterprise tools, data, and services. You will build agent architectures, advanced retrieval capabilities...Full timeTemporary work
- ...About the Opportunity We're partnering with a growing technology company looking for a Senior AI Engineer to build and scale AI-powered software applications. This is a hands-on engineering role focused on developing production-ready AI solutions, with significant...Full time
- ...Reporting to the AI Business Systems leadership team, the AI Engineer / Architect will help establish scalable development... ...Microsoft technology stack using traditional automation and artificial intelligence methods. Define and standardize agentic development practices...Local area
- ...We're building a full AI engineering team for a federal agency in Washington, DC. These are long-term contracts (3-4 years) with a hybrid schedule: on-site in Washington, DC 3 days per week. Requirement: US Citizens and Green Card holders only (government requirement...Hourly payLong term contract3 days per week
$82.08k - $193.44k
...At Capgemini Engineering, the world leader in engineering services, we bring together a global team of engineers, scientists, and architects to help the world’s most innovative companies unleash their potential. From autonomous cars to life-saving robots, our digital and...Full timeLocal areaNight shift- ...home to 230,000+ people across 60 countries, supercharging progress through industry-leading capabilities centered around Digital, Engineering and Cloud. About the Role Design and implement "loop engineering" workflows using Claude Code and similar tools, defining...Contract workLocal area
$150k - $200k
...sell to, that turns their retirement plan business into a growth engine. We're a small team split across the US and Spain. We... ...cost through scoped permissions, validation, prompt caching, and intelligent model routing. WHO YOU ARE You want your work to...Work at officeFlexible hours- ...Job Title: AI Engineer Location: Charlotte, NC (Remote) Type: W2 Contract What are the top 3 skills required for this role? 1) Agentic AI, Gen AI, Prompt Designing, Advanced RAG Architecture, LLM 2) AWS core services including Bedrock 3) Python, PySpark...Contract workRemote work
$99k - $225k
...Job Number: R0247598 Artificial Intelligence Engineer, Senior The Opportunity: Great AI solutions start with understanding the problem-then designing so met hing that actually works in the real world. As an Artifi cia l Intelligence Engineer on our team, you'll...Full timeContract workPart timeWork at officeLocal areaRemote workDay shift$77.6k - $176k
...Job Number: R0251799 Business Intelligence Engineer The Opportunity: Do you thrive by using your analytical expertise to convert large... ...process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity...Full timeContract workPart timeLocal areaRemote work$70 per hour
...Agentic AI Engineer / Applied AI Engineer Job Description We are seeking a highly skilled **Agentic AI Engineer / Applied AI... ..., risk scoring, anomaly detection, compliance automation, and intelligent business operations**. * Establish evaluation frameworks and...- ...Hiring Fulltime AI Engineer, Englewood Cliffs, NJ- Onsite 5 days a week. NO C2C Job Description: This position will identify high-impact business opportunities, design scalable AI-enabled solutions, and build production-ready capabilities that improve how teams...Full time
$150k
...AI Engineer | Charlotte | Onsite 4 days per week. $150,000 Take ownership of a live conversational AI system used every day by real customers, and shape where it goes next. My client is an established fleet management software company. Their AI assistant answers...- ...Role: Artificial Intelligence Engineer Location: Charlotte, NC We are At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading...Full timeTemporary workFlexible hours
$62k - $80k
...is based in Central NJ, specializing in Software Development, Web Development, and Business Intelligence. About the Role We are looking for motivated Software Engineers with a strong programming and object-oriented foundation to build scalable, production-ready...Full timeVisa sponsorship- ...making remote work more accessible and straightforward for both talent and employers. Role Description The Artificial Intelligence Engineer role at Remolity is a full-time remote position focused on designing, developing, and deploying AI models that improve...Full timeRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Artificial Intelligence Engineer. Be the first to apply!
- bi developer United States
- senior business intelligence developer United States
- bi data engineer United States
- bi etl developer United States
- business intelligence bi developer United States
- sql bi developer United States
- artificial intelligence engineer United States
- business intelligence engineer United States
- junior business intelligence developer United States
- business intelligence developer United States




