Observability Platform Architect

Observability Platform ArchitectThe Opportunity: Our team runs observability as a service for the enterprise, built on a platform that tells us whether its business services are healthy, so problems are caught before users notice. The platform supports more than 250 business services across multiple portfolios and environments. In this role, you will own the platform's architecture, instrumentation strategy, and entity model, and build the automation that lets a small team support it at this scale. More and more of that automation is driven by AI agents, so you will also decide what those agents can do on their own and how we confirm they made the right call. This is a hands-on engineering role, not an advisory one. You will set the standards, guide the engineers who build to them, and build alongside them. The ideal candidate comes from a platform engineering, observability, or site reliability engineering background and has owned an observability platform rather than just used one. Due to the nature of work performed within this facility, U.S. citizenship is required.What You'll Work On: Translate the service owner's priorities into the platform's technical plan, set technical direction through architecture decisions, design reviews, configuration governance, and mentoring engineers, and guide the team's day-to-day technical work.Architect and build the observability platform on Dynatrace, Grafana, and ServiceNow Event Management across every environment, including its integrations with ITSM, CMDB, and CI/CD, and keep environments from drifting into separately maintained estates.Keep the platform healthy, including upgrades, licensed consumption within budget, and escalation for complex platform issues, major incidents, and vendor support.Drive OpenTelemetry adoption and automate the platform itself, including collector configuration, service onboarding, monitoring agent deployment, tagging, and alert provisioning, delivered as code through CI/CD and tested in non-production before release.Build and deploy AI agents for detection, triage, and remediation, connected to the platform's APIs and tools, and own the platform's AI capability end to end, including what agents may do unattended, how we verify recovery, and how incorrect actions get caught.Own signal quality, including coverage, detection accuracy, and false positive rate, along with the entity model, dashboards, and SLOs leadership depends on, and drive down cost per monitored service.Join us. The world can’t wait.You Have: 8+ years of engineering experience3+ years of experience owning an enterprise observability or APM platform, including its architecture, upgrades, standards, cost, and production support2+ years of experience setting technical direction for other engineers, including architecture decisions, code review, and mentoringExperience writing and optimizing complex queries against observability data such as DQL or PromQLExperience replacing recurring manual platform work with production automation, including automated alert correlation, triage, or remediation, and owning whether it stayed correct over timeExperience using AI in operations, including vendor AI assistants, LLM tools, or agentsExperience with infrastructure as code and CI/CD such as Terraform, GitHub Actions, Azure DevOps, or GitLab, and deploying monitoring agents or collectors at scale across cloud, Kubernetes, and hybrid environmentsAbility to write, review, and ship code in Python, Go, or JavaScript against REST APIsBachelor's degree in CS, Information Systems, or EngineeringNice If You Have: Experience with Dynatrace in depth, including Grail, DQL, OpenPipeline, Workflows, Site Reliability Guardian, Davis, Dashboards, and the Dynatrace Operator on Kubernetes, with configuration managed through the Terraform providerExperience with Grafana and its broader stack such as Prometheus, Loki, Tempo, or Mimir and what it costs to run at scaleExperience with ServiceNow Event Management and CMDB, including event rules, alert correlation, and closed-loop incident creation from an observability platformExperience building LLM-based agents and the tooling they use such as agent frameworks or Model Context Protocol (MCP) serversExperience with OpenTelemetry, including collectors, instrumentation, semantic conventions, and the practical limits of vendor-neutral instrumentationExperience operating one platform across more than one environment or security boundary, and managing the drift and duplication that createsITIL 5 Foundation or ITIL 4 Practitioner: Monitoring and Event Management CertificationCompensationAt Booz Allen, we celebrate your contributions, provide you with opportunities and choices, and support your total well-being. Our offerings include health, life, disability, financial, and retirement benefits, as well as paid leave, professional development, tuition assistance, work-life programs, and dependent care. Our recognition awards program acknowledges employees for exceptional performance and superior demonstration of our values. Full-time and part-time employees working at least 20 hours a week on a regular basis are eligible to participate in Booz Allen’s benefit programs. Individuals that do not meet the threshold are only eligible for select offerings, not inclusive of health benefits. We encourage you to learn more about our total benefits by visiting the Resource page on our Careers site and reviewing Our Employee Benefits page.Salary at Booz Allen is determined by various factors, including but not limited to location, the individual’s particular combination of education, knowledge, skills, competencies, and experience, as well as contract-specific affordability and organizational requirements. The projected compensation range for this position is $86,800.00 to $198,000.00 (annualized USD). The estimate displayed represents the typical salary range for this position and is just one component of Booz Allen’s total compensation package for employees. This posting will close within 90 days from the Posting Date.Identity StatementAs part of the hiring process, we will ask you to complete an identity verification process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity fraud. You are expected to be on camera during interviews and assessments. We reserve the right to take your picture to verify your identity and prevent fraud.Candidate AI Usage PolicyAI is a part of our daily work at Booz Allen, and we are committed to the responsible and ethical use of AI tools. However, we want to ensure a fair candidate process based on your own skills and knowledge. As part of this commitment, the use of artificial intelligence (AI) or other tools to assist with responses during interviews (whether in-person or virtual) is prohibited unless permission is explicitly provided. Work ModelOur people-first culture prioritizes the benefits of collaboration. To support engagement and effective communication, employees working virtually are generally expected to have their cameras on during meetings.Remote: If this position is listed as remote, there may still be occasions when you are required to work in person at a Booz Allen or customer facility.Hybrid: If this position is listed as hybrid, you will be expected to work from a Booz Allen or customer facility, in alignment with your leadership's expectations and the needs of the role.Onsite: If this position is listed as onsite, work will primarily be performed full-time at a customer facility, where employees will collaborate directly with colleagues and customers as required by the role.Commitment to Non-DiscriminationAll qualified applicants will receive consideration for employment without regard to disability, status as a protected veteran or any other status protected by applicable federal, state, local, or international law.SummaryLocation: McLean, VAType: Full time