Principal Software Engineer

Principal Software Engineer Engineering Remote - USA POS-5690 About the Team The Observability team owns the internal platform that gives every HubSpot engineer real visibility into how their systems behave in production. We build and operate the distributed tracing, metrics, alerting, and logging infrastructure that spans hundreds of microservices, billions of daily events, and thousands of engineers who depend on that signal to ship reliably. We are now investing in the next generation of this platform. As HubSpot deploys AI agents and ML-powered features across the product, the team is building the tracing and telemetry primitives that make it possible to understand, debug, and trust what those systems are doing in production. This is greenfield, technically interesting work at a scale few companies operate at and we are looking for a Principal Engineer to help lead it. About the Role We are seeking a Principal Software Engineer to be the technical anchor for HubSpot’s Observability platform. This role sits at the intersection of large-scale distributed systems, developer platform design, and AI observability. A big part of this role is working horizontally across a large engineering org and setting patterns and standards that make it easier for teams to instrument, alert on, and reason about their services. You will also shape how we trace and understand our growing fleet of AI agents and ML systems in production: a technically distinct and increasingly critical problem. Key Expectations Observability Platform Architecture: Define the patterns and evolution of HubSpot’s core telemetry platform — distributed tracing, metrics, and structured logging — at a scale that spans hundreds of services and billions of daily events. Set the standards for how instrumentation is done across a large, polyglot engineering organization. AI & Agentic Observability: Lead the technical strategy for tracing and understanding AI agents and ML-powered systems in production. Define the primitives, telemetry standards, and debugging workflows that help product engineers understand what their models and agents are doing — and build trust in those systems over time. This is greenfield and consequential work. High-Cardinality, High-Throughput Systems: Architect telemetry pipelines and storage systems that handle high-cardinality data at high throughput without blowing up cost or query latency. Make principled tradeoffs between sampling, fidelity, retention, and developer ergonomics. Hands-on, High-Leverage Builder: Ship production code. Lead design reviews and take high-impact initiatives end-to-end, from prototype to production system at scale. Stay close to the systems you build and be the person who can debug the hardest problems when they surface. Developer Experience & Adoption: Design the instrumentation APIs and libraries that product engineers reach for, making correct observability the path of least resistance. Drive OpenTelemetry adoption across a large, polyglot codebase. Build the tooling that turns raw telemetry into actionable signal for teams operating at speed. Production Intelligence & Reliability Patterns: Define patterns for SLO/SLI design, alerting philosophy, and how teams graduate from reactive to proactive incident response. Push for simplicity in a domain that wants to get complicated, and consistency where tooling can drift across a large organization. Technical Leadership & Influence: Partner with infrastructure, platform, and product engineering teams to understand their signal gaps and close them. Influence technical strategy alongside engineering leadership, translating observability constraints and opportunities into product and operational decisions. Mentor senior engineers and tech leads, driving thoughtful design decisions and capturing learnings from major incidents and large-scale migrations. What You Bring Platform-Builder Experience: Proven experience building observability or telemetry tooling for internal engineering teams, rather than simply consuming it. You understand how to architect developer platforms that serve thousands of engineers across a large organization, backed by deep operational instincts and hard-earned expertise. Telemetry Systems Depth: Deep expertise navigating trade-offs in telemetry pipeline design across high-cardinality data, dynamic sampling, query latency, retention economics, and data fidelity. Strong technical fluency with OpenTelemetry, distributed tracing engines, metric backends, and large-scale log ingestion infrastructure. Incident Automation & Operational Excellence: Proven track record linking telemetry signals directly to automated operational workflows. You have designed architecture for real-time telemetry triggers that power automated remediation, dynamic runbooks, or AI-assisted root-cause diagnosis across microservices environments. Why This Role, Why Now HubSpot is scaling fast — more engineers, more microservices, more AI systems running in production — and the Observability platform is at an inflection point. The foundations are solid, but the next phase requires a different kind of investment: rethinking cardinality economics, making OpenTelemetry the default across a large polyglot org, and building an entirely new layer of AI tracing that doesn’t yet exist. This Principal Engineer will have a direct line of sight from the architecture they design to the engineering outcomes we measure: incident response times, adoption rates, developer satisfaction, and the trust that product teams place in their production signal. The AI observability layer in particular is greenfield — there is no playbook to follow, which is exactly what makes this the right moment for the right person. If you want to build the platform that helps thousands of engineers understand what their systems are doing — including systems powered by AI that are genuinely hard to see inside — this is the role. Pay & Benefits The cash compensation below includes base salary, on-target commission for employees in eligible roles, and annual bonus targets under HubSpot’s bonus plan for eligible roles. In addition to cash compensation, some roles are eligible to participate in HubSpot’s equity plan to receive restricted stock units (RSUs). Some roles may also be eligible for overtime pay. Individual compensation packages are tailored to your skills, experience, qualifications, and other job-related reasons. This resource will help guide how we recommend thinking about the range you see. Learn more about HubSpot’s compensation philosophy. Benefits are also an important piece of your total compensation package. Explore the benefits and perks HubSpot offers to help employees grow better. At HubSpot, fair compensation practices aren’t just about checking off the box for legal compliance. It’s about living out our value of transparency with our employees, candidates, and community. Annual Cash Compensation Range: $313,800 — $502,080 USD About HubSpot HubSpot (NYSE: HUBS) is an AI-powered customer platform with all the software, integrations, and resources customers need to connect marketing, sales, and service. HubSpot's connected platform enables businesses to grow faster by focusing on what matters most: customers. At HubSpot, bold is our baseline. Our employees around the globe move fast, stay customer-obsessed, and win together. Our culture is grounded in four commitments: Solve for the Customer, Be Bold, Learn Fast, Align, Adapt & Go!, and Deliver with HEART. These commitments shape how we work, lead, and grow. We’re building a company where people can do their best work . We focus on brilliant work, not badge swipes. By combining clarity, ownership, and trust, we create space for big thinking and meaningful progress. And we know that when our employees grow, our customers do too. Recognized globally for our award-winning culture by Comparably, Glassdoor, Fortune, and more, HubSpot is headquartered in Cambridge, MA, with employees and offices around the world. Explore more: HubSpot Careers Life at HubSpot on Instagram HubSpot, Inc. is an equal opportunity employer. As a federal contractor, we take affirmative action to ensure equal opportunity and all candidates are considered without regard to race, color, religion, national origin, age, sex, sexual orientation, gender identity, marital status, ancestry, physical or mental disability, veteran status, or any other legally protected characteristics. To assist HubSpot in its commitments and compliance with federal and state record keeping, reporting and other legal requirements, you are invited to provide the following information. In accordance with applicable law, we may also use the information in an aggregated, anonymous form to help us improve diversity at HubSpot. Completion of the form is entirely voluntary. Whatever your decision, it will not be considered in the hiring process or affect your candidacy in any way. Any information that you do provide will be recorded and maintained in a confidential file. What is your gender? What is your race/ethnicity? Please mark the one box that describes the race/ethnicity category with which you primarily identify. Race & Ethnicity Definitions Protected Veteran Status Voluntary Self-Identification of Disability -Form CC-305OMB Control Number 1250-0005 Expires 04/30/2026 Why are you being asked to complete this form? We are a federal contractor or subcontractor. The law requires us to provide equal employment opportunity to qualified people with disabilities. We have a goal of having at least 7% of our workers as people with disabilities. The law says we must measure our progress towards this goal. To do this, we must ask applicants and employees if they have a disability or have ever had one. People can become disabled, so we need to ask this question at least every five years. Completing this form is voluntary, and we hope that you will choose to do so. Your answer is confidential. No one who makes hiring decisions will see it. Your decision to complete the form and your answer will not harm you in any way. If you want to learn more about the law or this form, visit the U.S. Department of Labor’s Office of Federal Contract Compliance Programs (OFCCP) website at How do you know if you have a disability? A disability is a condition that substantially limits one or more of your “major life activities.” If you have or have ever had such a condition, you are a person with a disability. Disabilities include, but are not limited to: Alcohol or other substance use disorder (not currently using drugs illegally) Autoimmune disorder, for example, lupus, fibromyalgia, rheumatoid arthritis, HIV/AIDS Blind or low vision Cancer (past or present) Cardiovascular or heart disease Celiac disease Cerebral palsy Deaf or serious difficulty hearing Diabetes Disfigurement, for example, disfigurement caused by burns, wounds, accidents, or congenital disorders Epilepsy or other seizure disorder Gastrointestinal disorders, for example, Crohn's Disease, irritable bowel syndrome Intellectual or developmental disability Mental health conditions, for example, depression, bipolar disorder, anxiety disorder, schizophrenia, PTSD Missing limbs or partially missing limbs Mobility impairment, benefiting from the use of a wheelchair, scooter, walker, leg brace(s) and/or other supports Nervous system condition, for example, migraine headaches, Parkinson’s disease, multiple sclerosis (MS) Neurodivergence, for example, attention‑deficit/hyperactivity disorder (ADHD), autism spectrum disorder, dyslexia, dyspraxia, other learning disabilities Partial or complete paralysis (any cause) Pulmonary or respiratory conditions, for example, tuberculosis, asthma, emphysema Short stature (dwarfism) Traumatic brain injury PUBLIC BURDEN STATEMENT: According to the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. This survey should take about 5 minutes to complete. Please select one of the options below: Are you currently based in the country this position is advertised for or are you prepared to relocate yourself for this position? This position requires a proficient level of written and spoken English. Are you proficient in English? Are you proficient in an object oriented programming language? Were you previously employed at HubSpot Inc or any of its subsidiaries? If Yes, please select which entity below. (Please note this question is only applicable for past employees not current employees) Which time zone would you be working from in this role? Please note many of our teams are aligned to ET/CT in North America, which may impact the suitability of your application. Do you have experience building observability or telemetry tooling that other engineers use as a platform (not just operating it for your own team's services)? Have you worked with distributed tracing systems at scale (e.g., Tempo, Jaeger, Zipkin, or equivalent)? Are you comfortable working in a hands‑on individual contributor role that requires shipping production code alongside cross‑org technical leadership? Location Massachusetts Applicants: It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability. Germany Applicants: (m/f/d) - link to HubSpot's Career Diversitypage here . India Applicants: link to HubSpot India's equal opportunity policy here . HubSpot may use AI to help screen or assess candidates, but all hiring decisions are always human. More information can be found here . By submitting your application, you agree that HubSpot may collect your personal data for recruiting, global organization planning, and related purposes. We may use CLEAR ID Verification during the hiring process to confirm your identity and help maintain a safe, secure, and trusted experience for all candidates. Refer to HubSpot's Recruiting Privacy Notice for details on data processing and your rights. #J-18808-Ljbffr