Principal Engineer (San Mateo)

Company DescriptionOrganizations everywhere struggle under the crushing costs and complexities of “solutions” that promise to simplify their lives. To create a better experience for their customers and employees. To help them grow. Software is a choice that can make or break a business. Create better or worse experiences. Propel or throttle growth. Business software has become a blocker instead of ways to get work done.There’s another option. Freshworks. With a fresh vision for how the world works.Freshworks Inc. builds uncomplicated service software that delivers exceptional employee and customer experiences. Our people-first approach to AI eliminates friction, helping businesses reduce complexity, lower cost-to-serve, and deliver faster, more human support through enterprise-grade yet easy-to-use CX and IT solutions. Nearly 75,000 companies, including Bridgestone, New Balance, Nucor, S&P Global, and Sony Music, trust Freshworks to power their Employee Experience (EX) and Customer Experience (CX) operations.Fresh vision. Real impact. Come build it with us.Job DescriptionWe are seeking a Sr. Staff / Principal AI Systems Engineer to build, scale, and operate the AI Agent Platform that powers reasoning-driven assistants and autonomous agents across Freshworks — and to make that platform operate itself. You will own the systems architecture and engineering of a multi-tenant agent runtime that serves agentic workloads at high throughput and low latency, and you will pioneer an Agentic AIOps approach where autonomous agents monitor, diagnose, and remediate the platform in production.This is a hands-on systems engineering role at the staff/principal level, with a strong platform-scale and agentic-operations center of gravity. You'll design how thousands of agents are orchestrated, served, observed, and kept healthy at enterprise scale, and you'll set the technical direction which the broader engineering organization builds against. The level (Sr. Staff vs. Principal) will be calibrated to your scope of technical ownership and organizational impact.Key Responsibilities:Scalable Agent Platform SystemsArchitect and build the core AI Agent Platform — agent runtime, orchestration, tool/API invocation, state and memory management, and the retrieval/knowledge services agents reason overDesign for scale and efficiency: high-throughput multi-tenant serving, concurrency and queueing for agent workloads, model/inference routing, caching, and cost-aware executionBuild the control plane and systems primitives other teams use to define, deploy, version, and operate agents safelyDrive latency, throughput, and cost optimization across the agentic request path (planning retrieval tool calls generation)Agentic AIOps & Autonomous OperationsArchitect agentic operations workflows where autonomous agents observe platform telemetry, reason about anomalies, perform root-cause analysis, and execute remediation — shifting operations from human-driven to agent-drivenDesign multi-agent operational loops (detection, diagnosis, remediation) that collaborate, escalate, and hand off to on-call humans with clear rationale and audit trailsBuild closed-loop self-healing for the platform: auto-detection and repair of failing agents, degraded tools/connectors, stale knowledge, failed ingestion, and retrieval/index driftDefine guardrails, confidence thresholds, and human-in-the-loop controls that make autonomous remediation safe at multi-tenant scaleApply LLMs to operations directly — incident summarization, runbook generation, on-call copilots, and natural-language querying of platform telemetryObservability for Agentic SystemsInstrument the platform end to end: distributed tracing across planning-retrieval-tool-generation loops, metrics, structured logging, and event correlation so multi-agent behavior is explainable and debuggableDefine golden signals for both system health and agent quality — task success rate, tool-call accuracy, grounding/hallucination rates, latency, cost-per-task, throughput — as first-class telemetry the operating agents act onEstablish SLOs/SLIs and error budgets for agent workflows, with alerting that feeds the agentic-ops layerReliability, SRE & Distributed SystemsEngineer the platform as a resilient, event-driven, cloud-native distributed system (Kubernetes, streaming pipelines, microservices) across regions and tenantsDrive SRE practices — capacity planning, graceful degradation, failover, chaos/resilience testing, blameless incident response — and progressively automate them through agentsBuild for operability first: every component designed to be observed, diagnosed, and acted on autonomouslyKnowledge & Retrieval (the domain agents operate over)Guide the RAG/retrieval and knowledge services agents depend on, ensuring health, freshness, and quality are continuously monitored and remediated by the agentic-ops layerOversee enterprise content ingestion and sync (Confluence, SharePoint, Google Drive, Salesforce KB, ServiceNow KB) and multi-modal retrieval at platform scaleTechnical LeadershipSet technical direction for the agent platform and agentic operations; mentor engineers and drive cross-team systems architecture decisions.Partner with product and platform leadership on the long-term strategy for self-operating, enterprise-grade agentic AIQualificationsRequired10+ years building production software, with deep experience designing and operating large-scale distributed systems and platformsProven experience building scalable AI/agent platforms or high-throughput ML serving systems in production — orchestration, multi-tenancy, latency/cost optimizationHands-on experience designing agentic or autonomous workflows — multi-agent reasoning, tool/API invocation, planning loops — applied to real production problemsStrong AIOps and SRE background: observability tooling (OpenTelemetry, Prometheus, Grafana, distributed tracing), SLOs/error budgets, anomaly detection, incident management, and closed-loop automation with human-in-the-loop safeguardsHands-on experience applying LLMs to production workflows (reasoning, decision support, summarization)Strong proficiency in Python and a systems language (Go/Java); cloud-native architecture (Kubernetes, event-driven microservices, streaming pipelines).Working familiarity with RAG and knowledge systems (retrieval, embeddings, knowledge graphs) sufficient to architect over them — depth here is a plus, not the primary barFor Principal level: demonstrated org-wide technical influence — setting architecture direction across teams and driving large, ambiguous systems initiatives end to endPreferredML-driven anomaly detection, alert correlation, and predictive operations at scaleSRE leadership operating AI/ML or data-intensive platformsFamiliarity with agentic frameworks (LangChain, LangGraph) and vector/graph storesAI safety and governance grounding for autonomous enterprise systems.Contributions to open-source agentic, observability, or AIOps frameworksAdditional InformationPlease note this is a hybrid role with onsite expectations of 3x/week (Tues - Thurs) from our San Mateo, CA headquarters. The annual base salary range for this position is $216,000- $298,000. This role is also eligible for a target bonus. Compensation is based on a variety of factors, including but not limited to location, experience, job-related skills, and level. Freshworks offers multiple options for dental, medical, vision, disability, and life insurance. Equity + ESPP, flexible PTO, flexible spending, commuter benefits, and wellness benefits are also offered. Freshworks also offers adoption and parental leave benefits. At Freshworks, we have fostered an environment that enables everyone to find their true potential, purpose, and passion, welcoming colleagues of all backgrounds, genders, sexual orientations, religions, and ethnicities. We are committed to providing equal opportunity and believe that diversity in the workplace creates a more vibrant, richer environment that boosts the goals of our employees, communities, and business. Fresh vision. Real impact. Come build it with us.SummaryType: Full-timeFunction: OtherExperience level: DirectorIndustry: Information Technology And Services