AgentCore - Evaluation & Observability / AgentOps Architect
VDart Inc
Job role : AgentCore - Evaluation & Observability / AgentOps Architect Duration : 3 months contract Location : Remote AWS Certification - Mandatory Active AWS certification is required. Core Skills : AWS AgentCore, AgentOps, AI/ML, Evaluation, Observability, Generative AI We are seeking a senior AgentCore / AgentOps Architect to design and establish enterprise-grade evaluation, observability, and operational standards for AI agents. The ideal candidate will have strong hands-on experience with AWS AgentCore, Amazon Bedrock, AI/ML, LLM evaluation, agent tracing, observability, and production AI systems. Key Responsibilities: Architect and establish AgentOps, Evaluation, and Observability frameworks for AI agents. Define standards for monitoring: Agent execution paths Model and tool calls Token usage Latency Failures and errors Infrastructure and model costs Establish evaluation frameworks for: Task completion Correctness Faithfulness Safety Tool-selection accuracy Agent performance and reliability Design automated evaluation and regression-testing frameworks for AI agents and LLM applications. Build a continuous Observe → Evaluate → Improve → Validate → Deploy lifecycle. Establish production monitoring, tracing, alerting, and performance dashboards. Integrate AgentOps and evaluation capabilities into CI/CD and MLOps pipelines. Define quality gates and validation standards before AI agents are promoted to production. Partner with AI/ML, cloud, and engineering teams to operationalize agentic AI solutions. Drive best practices for AI reliability, safety, governance, and responsible AI. Required Skills: Strong hands-on experience with AWS AgentCore. Strong AI/ML and Generative AI background. Experience with Amazon Bedrock and LLM-based applications. Strong experience in AgentOps / AI Agent Observability. Experience designing LLM/AI evaluation frameworks. Experience with agent tracing, telemetry, monitoring, and performance analysis. Strong understanding of LLM/agent metrics including correctness, faithfulness, safety, latency, cost, and tool-selection accuracy. Experience with automated regression testing for AI/LLM applications. Strong Python and API development experience. Experience with cloud architecture, CI/CD, and MLOps. VDart
- ...Must have skills/Matrix AWS Bedrock / Bedrock AgentCore Agentic-AI Architecture & Multi-Agent Orchestration Tool-Use / Function... ...-Authority Models Systems of Record Integration (ERP, HRIS, ITSM) Observability & Distributed Tracing (OpenTelemetry, CloudWatch)...Suggested
Intelliswift - An LTTS Company
New York, NY4 days ago $101.14k - $182.62k
...Observability architect Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you’d like, where you’ll be supported and inspired by a collaborative community of colleagues around the world, and where you’ll be able to...SuggestedPermanent employmentFull timeContract workLocal areaCapgemini
New York, NY4 days ago- ...AWS AI Architect – Agentic AI, Knowledge Graphs Contract | United... ...production code, establish evaluation and quality standards, and lead... ...the existing Amazon Bedrock AgentCore and Strands Agents... ...DataZone. Knowledge of LLM observability, tracing, evaluation, and monitoring...SuggestedContract work
EazyML
New York, NY2 days ago $230k
...You'll DoPrincipal AI Tech Architects at BCG Platinion are:Collaborative... ..., and agent orchestration.Evaluate emerging AI frameworks and... ...Oversee system performance, observability, and governance as solutions... ...platforms: AWS (Bedrock, AgentCore), Microsoft (Azure AI Foundry...SuggestedWork at officeLocal areaThe Boston Consulting Group
New York, NY6 days ago- Job Title: GenAI /Bedrock Architect Location: Remote Duration: / Term: FTE... ...Implement and manage Bedrock Agents, AgentCore, Guardrails, and model evaluation frameworks . Develop reusable... ...guardrails, and enterprise AI governance. Observability: Model latency, token consumption...SuggestedRemote work
- ...action. Endless possibilities.We’re hiring AgentCore Platform Engineers to build the agent-... ...adding, versioning and reviewing tools.Evaluation and Guardrails: Build in evaluation,... ...agent systems in production, including observability and failure handling.Experience integrating...Contract workTemporary work
Appnovation
New York, NY4 days ago $213k - $263k
An autonomous technology company is seeking a Senior Software Engineer to design and implement core evaluation APIs in a hybrid role. You will drive technical architecture and collaborate with cross-functional teams to ensure reliability and efficiency. The role requires...$55 per hour
A leading technology firm is seeking an Evaluation Scenario Writer - QA to ensure the quality of evaluation scenarios for AI projects. This part-time, flexible role involves reviewing tests, spotting inconsistencies, and collaborating on automation. Ideal for those with...Part timeRemote workFlexible hoursMindrift
New York, NY6 hours ago- Job Title : Principal Observability Architect Duration : Up to 1 Year Location : Remote Position Summary: Client is seeking a Principal Observability Architect to establish and lead the observability strategy for a globally distributed payments platform. This individual...Remote work
- Principal Architect - Gen AI The Gen AI Principal Architecture within... ...using platforms such as AWS AgentCore and Azure AI Agent Service.... ...human-in-the-loop controls, observability, fail-safes, and model/agent... ...stores, context management, evaluation, and performance tuning....Work at officeLocal area
- 24-MAG LLC is seeking experienced legal researchers to design evaluation frameworks for AI systems operating in legal domains. The role involves developing benchmark suites, methodologies, and datasets to assess legal reasoning and analytical quality. You will collaborate...Remote job
- ...scalable, user-centric AI capabilities that automate workflows and improve service delivery. You will integrate AI agents with observability and monitoring platforms, optimize model performance, govern production stability and compliance, and continually improve agent-...Remote job
- ...Title: AI / Agentic AI ArchitectRole: Architect - Agentic AI Platforms & ApplicationsLocation... ..., Sales, Support, and Growth.• Evaluate foundation models and enterprise AI platforms... ...and knowledge repositories.• Implement observability, evaluation, guardrails, monitoring,...Remote work
Brillio
New York, NY7 days ago $184.9k - $250.2k
Amazon Bedrock AgentCore is building the foundational services developers use to create, deploy, and operate production... ...of AgentCore Runtime, Memory, Gateway, Identity, Observability, Browser, Code Interpreter, Evaluations, and other services together into a coherent...InternshipLocal areaImmediate startFlexible hoursAmazon
New York, NY3 days ago$158.1k - $213.8k
Amazon Bedrock AgentCore is building the foundational services developers use to create,... ...session isolation, identity, networking, observability, and deployment lifecycle. This allows... ..., Browser, Code Interpreter, and Evaluations. You will collaborate with engineers across...InternshipLocal areaFlexible hoursAmazon
New York, NY4 days ago- micro1 is seeking a Member of Technical Staff, Law Research, to lead the development of evaluation frameworks for AI-enabled legal reasoning. You will design benchmarks and collaborate with researchers to shape enterprise AI workflows, focused on robust evaluation across...Remote job
- ...Understand the current legacy system and architect, scale Agentic AI systems capable of... ...hybrid ML/LLM systems, and standardized evaluation frameworks. Evaluate and select the... ...(models, orchestration frameworks, and observability tools) based on cost, latency, and performance...Work at office
Diversity Nexus
New York, NY5 days ago - ...AI outputs against your rubric to ensure publication-ready quality. Preferred candidates bring hands-on journalism, editing, writing, or video production experience and comfort evaluating professional-grade work to benchmark AI performance. #J-18808-Ljbffr BAM Ventures
$70k - $75k
...Job Title: Intermediate Architect / Technical Designer Corporate Interiors Job Description... ...specifications. Conduct site observations during construction, provide layout approvals... ..., including sourcing, screening, and evaluating candidates. AI helps assess...Permanent employmentContract workFor contractorsCasual workWork at officeRemote workMonday to FridayActalent
New York, NY5 days ago- ...Applied AI Architect (Data, Systems & Governance) | Brillio Architect the Foundations... ...knowledge layers, and AI data pipelines Evaluate and select AI models based on use case,... ..., CI/CD, model lifecycle management, observability, monitoring, and evaluation Help...Full time
Brillio
New York, NY3 days ago - ...Job Title- Gen AI Architect Location - NYC, NY Job Type-Full Time Technical Skills Deep expertise in AI/ML engineering and GenAI platforms. LLM Evaluation & Observability – Exposure to tools such as LangSmith, LLM as a Judge, RAGAS, DeepEval or equivalent...Full time
Maveric Systems Limited
New York, NY6 hours ago - Alice is seeking a seasoned researcher-leader to helm AI safety benchmarks and evaluations. You will own the taxonomy, harness, and release cadence, coordinating researchers and freelancers while reporting to the CTO office. You will influence the public research agenda...FreelanceWork at office
$215.2k - $244.7k
...impact?West Monroe is seeking a Senior Architect to join our thriving Technology and Experience... ...management expertise to our clients in evaluating requirements and developing data... ...feature engineering, model monitoring, observability, and governanceLead technical discussions...Local areaImmediate startFlexible hours2 days per week1 day per weekWest Monroe Partners
New York, NY4 days ago$176.3k - $374.2k
...build As a Principal Forward Deployed Architect, you will provide technical leadership... ...engineers and architects. Drive security, observability, performance, compliance, and... ..., vector databases, MCP, tool calling, evaluation frameworks, and AI observability. Extensive...Permanent employmentFull timeWork at officeWorldwideFlexible hoursSAP
New York, NY7 days ago$187.6k - $220.7k
...you ready to make an impact?Agentic AI Architect-Google Cloud West Monroe is seeking an... ..., tool integration, memory management, observability, governance, and secure deployment patterns... ...clients, exercising sound judgment in evaluating outputs. Apply AI technologies (e.g.,...Local areaImmediate startFlexible hoursWest Monroe Partners
New York, NY4 days ago$226k - $311k
...belongs to you.The OpportunityWe are looking for a Product Architect, AI to help define how Auth0 and Okta lead the market in... ..., agent harnesses, AI frameworks, orchestration tools, evaluation and observability platforms, and rapidly evolving developer practices. You...Full timeLocal areaWorldwideFlexible hoursShift workEarly shiftOkta
New York, NY5 days ago$215k - $275k
...safely at enterprise scale. As an AI/ML Architect, you’ll design and deliver production-... ...model orchestration, agentic workflows, evaluation, deployment, monitoring, optimization,... ...loop controls.* Establish AI evaluation, observability, and reliability standards, including...Temporary workLocal areaSlalom
New York, NY3 days ago$200k - $280k
...engagements. Act as a trusted senior technical architect and advisor to client leadership,... ...: Define multi-year GenAI roadmaps by evaluating structural architecture trade-offs... ...depending on enterprise complexity. Advanced Observability & Monitoring: Design deeply integrated...- ...visit: Website or LinkedIn Page. Role: Architect Machine Learning Engineer Experience... ...optimization, and latency analysis Design novel evaluation frameworks to assess multi-step agentic... ..., and accuracy Utilize advanced observability tools (LangSmith, Arize AI, or custom...
- Contract Architect Fabric Platform Engineering Hands-on architecture role across reusable... ...for reusability, maintainability, observability, cost optimisation and technical debt control... ...retrieval-augmented generation, prompt evaluation, automated testing and AI governance...Contract workImmediate startRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AgentCore - Evaluation & Observability / AgentOps Architect. Be the first to apply!
