Principal, Agentic Reliability Engineer

Agentic Reliability Engineer (ARE)Slalom · Agentic Managed Services · Remote (US) with possible client travel · Principal or Senior Principal level ABOUT THE ROLEAI agents are moving into production faster than most organizations can operate them reliably and closing that gap is exactly why Slalom built its Agentic Managed Services (AMS) practice. As an Agentic Reliability Engineer, you are the technical heart of that work: the engineer who makes AI agents dependable in the real world.Where a traditional Site Reliability Engineer keeps software services healthy, you own the reliability of AI agents. These are systems that reason, make decisions, and act on a client's behalf, but can fail in ways classical engineering never had to consider. You'll help design and stand up agentic solutions with clients, prove they hold up under real conditions, and then keep them performing in production: watching how they behave, diagnosing why they drift, tuning and evaluating them, and safely rolling out changes as they evolve.Your day to day will look like a small senior delivery pod working shoulder-to-shoulder with the client and with Slalom's platform team. It's a collaborative, high-ownership environment: you are the final technical authority on how the agents behave, and the person a client trusts to tell them their systems are reliable, well-governed, and getting better over time. Because our agentic reliability engineering group is a new and fast-growing practice, you'll also help shape how the work is done. We expect you to provide input in to the patterns, tooling, and standards that make running production agents repeatable.This is a role for engineers who want to be at the frontier of what AI systems actually do in production. You thrive on work with real stakeholders, measured outcomes, and sustained impacts. WHAT YOU'LL DODesign & deploy agentic solutions* Partner with clients to shape the opportunity and design the agent workflow — what the agents should do, how their performance will be judged, and what guardrails keep them safe.* Build the first working agents: stand up the initial workflow, evaluation harness, and guardrails, and demonstrate that the solution meets the client's acceptance criteria under real conditions.* Define the reliability baseline the solution must hold in production including the SLOs, evaluation criteria, and autonomy posture that govern it.Run & continuously improve production agents* Own root-cause analysis and ongoing performance of the agent workflow itself: why it behaves as it does, and how to make it measurably better over time.* Own the evaluation harness: configure, run, and interpret agent evaluations against workflow-specific acceptance criteria.* Monitor drift signals (e.g. output-quality degradation, latency trends, evaluation pass-rate decline, cost anomalies) and remediate before they breach SLOs.* Run A/B and shadow evaluations when agent behavior changes are proposed* Capture operational findings, incident lessons, and evaluation-calibration notes so the pod's knowledge compounds over time rather than resetting.Measure, analyze & communicate performance* Define and track the KPIs that describe how well the agents are performing and investigate when they move in unexpected directions.* Translate telemetry into narratives the team and client can act on.* Develop and communicate clear plans to improve agent performance and reporting on their impact once shipped.* Serve as the final technical authority for AI agent incidents including owning root-cause analysis of the agent workflow, remediation, and postmortem documentation.* Determine whether an incident reflects a transient error, agent drift, workflow-logic failure, or an underlying model-behavior change.Safe change & controlled rollout* Manage change control for agent workflow updates: version the change, run the evaluation suite, document the expected behavioral delta, and propose the go-live schedule.* Stage rollouts with shadow-deployment patterns before promoting to full release; monitoring the stability window and rolling back per runbook if thresholds are breached.* Manage identity, access, and integrations: app registrations and credentials, access control and permissions, and the authentication, authorization, and contracts that govern how agents communicate with other tools and systems. * Manage cost and infrastructure: right-sizing the compute agents run on, setting token and budget caps per agent, optimizing workflow steps for efficient resource use, and reporting spend — and the cost impact of the changes you make — to the team and client.Build reusable tooling & help shape the practice* Contributing reusable agents, evaluation templates, and runbook patterns so each new engagement starts further ahead than the last.* Acting as a primary production signal for Slalom's AMS platform team — surfacing the gaps whose closure most improves how the work gets done.* Helping define the standards and playbooks for a new and fast-growing engineering discipline.CORE SKILLS* Deep analytical & debugging rigor — defines what "correct" looks like for an agent decision and builds the test to prove it.* Critical thinking — approaches problems with a problem solving mindset.* Communication & translation — turns telemetry and metrics into clear narratives and reports a mixed team-and-client audience can act on; can explain why an agent is behaving as it is, and what the plan is to change it.* Solutioning & client fluency — can shape a solution with a client, translate a business problem into an agentic workflow design, and make the value case.* Workflow root-cause & performance ownership — owns RCA of the agent workflow itself and drives its improvement.* Production engineering — strong Python/TypeScript; reads and modifies agent orchestration code.* AI / agent reliability — evaluations, drift detection, controlled rollout, observability.* A builder's instinct for reuse — reaches for the reusable pattern over the one-off.WHAT YOU BRINGRequired* 5+ years of engineering experience in a production environment (software engineering, data engineering, platform/DevOps, or SRE).* Strong Python or TypeScript; ability to read, modify, and evaluate agent orchestration code without extensive scaffolding.* Hands-on experience with at least one production AI/ML system (model serving, pipeline orchestration, or an LLM-based application).* Familiarity with observability tooling (distributed tracing, structured logging, alerting) and incident-response workflows.* Analytical rigor: comfortable defining what correct looks like for an AI agent decision, and building a test to verify it.Preferred* Client-facing or solutioning experience — the ability to scope and shape technical engagements with clients, not only operate them.* Direct experience with LLM-based agent frameworks — LangChain, LangGraph, CrewAI, AutoGen, AWS Bedrock Agents, or equivalent.* Evaluation frameworks for AI systems (Langfuse, Braintrust, RAGAS, DeepEval, or custom harnesses).* Familiarity with agentic memory architectures — episodic, semantic, and procedural stores; vector databases; RAG.* AI governance concepts: safety guardrails, NIST AI RMF, or OWASP LLM Top 10.WHAT SUCCESS LOOKS LIKE* Agentic system stability & efficiency — the agents you own run reliably and improve over time: strong evaluation pass rates, drift caught early, fast recovery when incidents happen, clean rollouts, and steadily better workflow performance and cost efficiency.* Client outcomes — your clients trust that their agentic systems are dependable, well-governed, and advancing, reflected in SLO attainment and the growth of the outcomes the agents are responsible for.* Revenue impacted — unlike a traditional services role, you are measured on the value you drive, not hours booked.* Feedback, collaboration & partnership — you make the people around you better: giving and acting on candid feedback, sharing what you learn, and partnering effectively with Slalom teammates across capabilities, platform, and client teams to move the work forward together.WHY THIS ROLE, WHY NOWAI agents are moving from novelty to critical infrastructure, and the engineers who build the discipline of running them reliably will define the field for the next decade. Being able to say "I have run agents in production, at SLA, and debugged them", is one of the most exciting and evolving skillsets engineering today. This role puts you at the center of it: real clients, real outcomes, and a platform team investing in the tooling that makes the work tractable, at a firm building this practice as a long-term strategic bet.About UsSlalom is a fiercely human business and technology consulting company that leads with outcomes to bring more value, in all ways, always. From strategy through delivery, our agile teams across 52 offices in 12 countries partner with clients to co-create powerful customer experiences, modern ways of working, and meaningful impact. What sets us apart? We believe work should be challenging and fulfilling, not perfect, but possible. That’s why we prioritize purpose, flexibility, connection, and recognition, so our people can thrive and love what they do, most days. Compensation and BenefitsSlalom prides itself on helping team members thrive in their work and life. As a result, Slalom is proud to invest in benefits that include meaningful time off and paid holidays, parental leave, 401(k) with a match, a range of choices for highly subsidized health, dental, & vision coverage, adoption and fertility assistance, and short/long-term disability. We also offer yearly $350 reimbursement account for any well-being-related expenses, as well as discounted home, auto, and pet insurance.Slalom is committed to fair and equitable compensation practices. For this role, we are hiring at the following levels and targeted base pay salary ranges: The targeted base salary pay range for Principal level is: * Boston, Seattle, Los Angeles; $171,000-$210,000* Atlanta, Dallas, Philadelphia; $157,000-$195,000The targeted base salary range for Senior Principal level is:* Boston, Seattle, Los Angeles; $216,000-$254,000* Atlanta, Dallas, Philadelphia; $198,000-$235,000In addition, individuals may be eligible for an annual discretionary bonus.  Actual compensation will depend upon an individual’s skills, experience, qualifications, location, and other relevant factors. The salary pay range is subject to change and may be modified at any time. We will accept applicants until November 12th, or until the position is filled.We are committed to pay transparency and compliance with applicable laws. If you have questions or concerns about the pay range or other compensation information in this posting, please contact us at: [1] View email address on click.appcast.io. Please note, this recipient is not able to support recruitment inquiries beyond this purpose.EEO and AccommodationsSlalom is an equal opportunity employer and is committed to attracting, developing and retaining highly qualified talent who empower our innovative teams through unique perspectives and experiences. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability status, protected veterans’ status, or any other characteristic protected by federal, state, or local laws. Slalom will also consider qualified applications with criminal histories, consistent with legal requirements. Slalom welcomes and encourages applications from individuals with disabilities. Reasonable accommodations are available for candidates during all aspects of the selection process. Please advise the talent acquisition team or contact [2] View email address on click.appcast.io if you require accommodations during the interview process. ReferencesVisible links1. mailto:View email address on click.appcast.io. mailto:View email address on click.appcast.io