Platform Engineer (AI Gateway, Identity & Observability)

Job Title: Senior Platform Engineer (AI Gateway, Identity & Observability)
Job Location: Oaks,PA
Job Type: Contract

Job Description:

Deliver the AI gateway that routes model traffic across cloud-hosted models, other providers and on-premises inference, presenting one consistent, provider-agnostic interface to every consuming agent.

Implement gateway policy: routing and backend pools, token-based rate limiting and quotas, retries, failover and circuit breaking, caching, and request and response transformation.

Harden an existing service for production and run a controlled migration to the long-term gateway, moving consumers across without changing the contract they already integrated against.

Build the agent identity and registration path: workload or service identity for agents, registration on deployment, and the promotion gate that governs what reaches production. Full automation is not assumed; part of the work is automating what can be automated and designing a clean, documented path for the steps that require human approval.

Design and implement the telemetry schema and the OpenTelemetry pipelines that carry it, spanning agents in every runtime, so traces are consistent and attributable to an owning team.

Build the fleet view showing each agent, its owner, version, environment, health and consumption, including the identifiers that make usage chargeable back to consuming teams.

Own the reliability view of the platform: defining what good telemetry looks like, verifying that traces are complete and correctly attributed across every runtime, setting service level indicators and objectives for the gateway and the agent fleet, and building the alerting that surfaces failures, latency regressions and cost anomalies before consuming teams report them.

Apply guardrails at the gateway, including content safety policy and a callout path to custom filtering services.

Establish the network and integration paths the control plane depends on, including private connectivity or controlled egress to observability backends and third-party services, DNS and certificate handling, and the security review each path requires.

Load test and prove the gateway under realistic production volume, and produce the runbooks and operational documentation the client will run it from.

5+ years hands-on engineering across API platforms, identity or observability, with production depth in at least one and working competence in the others. All three areas are in scope for this position.

4+ years building and operating API gateways in production: routing, policy authoring, rate limiting and quotas, authentication, transformation and versioning. Azure API Management is strongly preferred; comparable depth in Apigee, Kong or an equivalent enterprise gateway is acceptable.

3+ years enterprise identity and access engineering: OAuth2, OpenID Connect and JWT validation, service principals and workload or managed identity, and secrets management using an enterprise vault.

3+ years hands-on OpenTelemetry or equivalent distributed tracing: instrumentation, collectors, resource attributes, span design, and trace, metric and log pipelines.

3+ years in a site reliability, production engineering or platform operations role, or equivalent hands-on responsibility for a service other teams depend on: service level objectives and error budgets, alerting design, on-call and incident response, and post-incident analysis. This position owns whether the telemetry is trustworthy, not only whether it is being collected.

2+ years working with LLM or AI workloads in production, including model routing across providers, token-based limits and quotas, streaming responses, content filtering, and how inference cost accrues and is attributed.

Experience with LLM observability tooling, such as Langfuse, LangSmith, Arize or an equivalent platform, including how agent traces differ from conventional application traces.

3+ years production API operations: failover, circuit breaking, caching, load testing to prove capacity, and incident response for a service other teams depend on.

2+ years integrating third-party or self-hosted services into an enterprise network, including private connectivity or controlled egress, DNS resolution, TLS and certificate management, and working the firewall and security review needed to get each path approved.

Working knowledge of cloud infrastructure on at least one major hyperscaler: networking, identity, container platforms and infrastructure-as-code.

Programming competence in Python, C# or an equivalent language, sufficient to build and maintain policy extensions, callout services and instrumentation libraries.

Familiarity with Azure AI Foundry observability and Azure Monitor or Application Insights, including agent tracing, continuous evaluation and how a managed observability plane compares with a self-hosted one such as Langfuse. This platform may run one, the other, or both, and the choice is still open.

Azure API Management depth, including policy expressions, reusable policy fragments and the AI or LLM policy set.

Microsoft Entra ID experience, including managed identity, app registrations and conditional access.

Experience with reliability engineering for AI or machine learning systems, including quality regression detection, drift monitoring or evaluation-driven alerting.

Familiarity with emerging agent identity models and how agent-to-service authentication differs from conventional service-to-service patterns.

Experience implementing chargeback or showback for a shared platform service.

Content safety or guardrail services, such as Azure AI Content Safety, or custom filtering and moderation services.

Enterprise workflow integration, such as ServiceNow APIs for approval-driven provisioning.

Experience in financial services or another regulated industry, where security review governs the pace of change.

Familiarity with agent frameworks and how agents consume model endpoints, tools and memory.