Principal Software Engineer - SRE, Incident Response & Operational Excellence
$247.5k - $335kIntuit Inc
Intuit is the global financial technology platform that powers prosperity for the people and communities we serve. Our public-facing products — TurboTax, Credit Karma, QuickBooks, and Mailchimp — serve approximately 100 million customers who depend on us at the moments that matter most: filing a return before the deadline, running payroll, checking their credit, or sending a campaign. When something breaks, minutes matter, and how quickly and how well we recover is a direct expression of our customer promise. We are hiring Principal Software Engineers to serve as Technical Duty Officers (TDOs) — the leaders who take command of Intuit’s most critical incidents. During our most critical incidents you hold ultimate authority on decisions and enforcement of our incident principles, partnering with the Business Unit Driver and the Incident Operations Center (IOC) to direct recovery, actions, and communications on the bridge. Reporting to the Director of Cloud Engineering & Operations, this is a dedicated, full-time role: when you are not on the bridge, your mandate is to make the next incident shorter, rarer, and better understood — partnering across every business unit on effective observability, clarity of customer workflows and their real-time impact signals, high-quality runbooks, and validated escalation and notification readiness. This role is equal parts operator, engineer, and leader. You bring the calm, decisive command presence to lead senior leaders through high-stakes ambiguity, the deep systems intuition to cut to the most likely fault and the fastest safe path to recovery, and the engineering skills to implement the improvements you identify rather than hand them off. As a Principal engineer, you will not just execute this work manually but direct, orchestrate, and govern AI agents to accelerate detection, triage, impact assessment, and post-incident learning, accountable for the quality and reliability of everything you and those agents produce. Responsibilities
- Lead Intuit’s most critical incidents, serve as the “Technical Duty Officer” role on all significant incidents, holding ultimate authority on decisions and enforcement of incident principles; direct recovery, actions, and communications on the bridge in partnership with the BU Driver and IOC Leader
- Drive fast recovery, bias the bridge toward mitigation over diagnosis, make timely and decisive calls on rollbacks, failovers, traffic shifts, and feature disablement, and keep responders focused on the shortest safe path to restoring the customer experience
- Get the priority right early, challenge and correct incident priority based on customer impact during active response rather than after recovery, and ensure the right leaders, escalations, and communications are engaged at the right time
- Establish clear customer-impact signals, partner with each business unit to define, instrument, and continuously report real-time customer impact for the highest-value workflows across top revenue-generating products, and drive readiness validation through the company-wide Customer Impact Gameday
- Raise the observability bar, partner with platform and product teams on SLOs, dashboards, alerting, and tracing so responders can assess the problem and its impact in minutes; feed tool and product-gap findings from live incidents back to platform teams as part of the RCA
- Own runbook and workflow clarity, ensure critical customer workflows have accurate, exercised runbooks and clear service ownership, and drive the recurring validation of notification channels, escalation policies, bridge creation, and executive notification workflows
- Make communications actionable, work with the IOC Leader so that executive and customer-facing updates consistently convey customer impact, recovery progress, and next steps with enough clarity for confident decision-making
- Lead post-incident learning, drive blameless RCAs for the incidents you command, identify systemic and cross-BU remediation, and track high-leverage actions to closure
- Continuously improve incident and observability capabilities, bring an SRE mindset to the incident-management and observability platforms, implement solutions yourself, and encode triage, impact-assessment, and communication knowledge into agentic and self-service tooling so standards travel with the automation
- Provide 24x7 coverage, participate in the follow-the-sun TDO rotation across the US and IDC, including off-hours and peak-season (e.g., tax deadline, payroll) readiness windows, with clean shift handoffs
- Coach and scale incident leadership, contribute to Incident Commander training for Directors, Distinguished Engineers, and Vice Presidents, mentor BU Drivers and responders on the bridge, and raise the incident-leadership bar across the company
- BS in Computer Science or equivalent work-related experience; 12+ years of engineering experience, including significant time operating large-scale, high-traffic production systems
- Proven track record commanding large-scale incidents in an enterprise-level or financial-services environment, with the ability to lead senior leaders, make decisive calls with incomplete information, and remain calm and clear under pressure
- Deep operational excellence and SRE expertise, including SLOs and error budgets, observability (metrics, logs, traces), alerting design, on-call and escalation practices (e.g., PagerDuty), and formal incident-management frameworks and change management
- Broad, hands-on distributed-systems intuition across the full stack — cloud infrastructure (AWS), Kubernetes and service meshes, networking and edge (CDN, DNS, TLS), data stores, and messaging — with the ability to reason quickly about failure modes, blast radius, and the fastest safe path to recovery
- Demonstrated ability to translate technical state into crisp customer-impact and business-impact terms for executives and customer-facing teams, in writing and live on a bridge
- A software engineer who can build, not just operate: proficient in scripting and development (e.g., Python, Golang, Java), Infrastructure as Code, and automation, able to implement or oversee the observability, runbook, and escalation improvements you identify
- Fluent in AI agent development and orchestration, able to build agentic and self-service tooling (e.g., MCP servers and skills) for triage, impact assessment, and post-incident analysis, and to review AI-generated output to the same quality bar as peer work
- Experience driving blameless post-incident reviews and systemic remediation across organizational boundaries, influencing Director- and VP-level leaders without direct authority
- Willingness to participate in a 24x7 follow-the-sun on-call rotation, including off-hours and peak-season coverage
- Principal-level expectations: align on the technical vision for incident response, observability, and operational readiness across multiple business units; define and drive the company-wide standards, guardrails, and GameDays that measurably reduce time-to-action and time-to-recovery; and be recognized as the authority others turn to during Intuit’s most critical moments
- ...Principal Engineer, Ai/Ml & Cloud Company description... .... You will operate as a force multiplier... ...management. Responsibilities The position's... ..., cross-cutting incidents. Work with... ...architecture. Cloud & software engineering... ...hardest problems to. Excellent experience...SuggestedWorldwideFlexible hours
- ...Network and Positioning Engine deliver centimeter... ...will build and operate the infrastructure... ...on. This role is responsible for keeping it... ...rotation and lead incident response, drive root... ...years) in DevOps, SRE, infrastructure, or... ...’s outcomes with excellence, the path to greater...SuggestedImmediate start
$134k - $180k
...Senior DevOps Engineer At AoPS, we are world... ...same standard of excellence into new subjects... ...Production: Help operate and evolve our container... ...Break: Lead incident response, find root causes,... ...experience in DevOps, SRE, platform engineering, or software engineering, with...SuggestedFull timeCasual workH1bRelocation packageFlexible hours$142.2k - $213.4k
...looking to add a Senior Principal Embedded Software Engineer to join our team of qualified... ...businesses. You will be responsible for the design,... ...products to our customers Operate in an embedded technology... ...integration and testing Excellent communication, interpersonal...SuggestedRelocation packageShift work- ...Sr or Principal Embedded SW Engineer – Our client is a well known developer... ...SW Engineer will be responsible for implementing embedded software that forms part of... ...thinking skills. • Excellent verbal and written... ...platforms using Real-time Operating Systems. • Must...SuggestedWork experience placementWork at officeRelocation package
$195k - $235k
...passionate team as a Senior Principal Robotics Software Engineer. As we scale, you will... ...and growth. Your key responsibilities are as follows: ~ Build... ...discipline, and passion for excellence to assure accuracy of... ...~ Fluent developing and operating software on Windows,...Full timeLocal areaImmediate start- ...Responsibilities Support software engineering, development, testing, integration, and implementation of submarine... ...equipment configurations to meet operational requirements. Strong... ...Suite, including Visio, Project, Excel, Outlook, PowerPoint, and Word....Full timeWork at office
- ...NASDAQ: POWW) owns and operates GunBroker.com, the... ...seeking a hands-on Senior Software Developer to lead the... ...shape how our engineering team leverages AI-powered... ...systems, leading incident response efforts, and identifying root causes. ~ Excellent written and verbal communication...
$261.5k - $353.5k
...We are seeking a Principal Software Engineer, Data Engineering to... ...successful candidate will operate with a Builder... ...-term impact. Responsibilities Technical Craft... ...approaches for how to avoid incidents and position for '... ...people. Excellent interpersonal and communication...Full timeTemporary workWork experience placement$107.5k - $204.5k
...leading businesses, world-class operations and investments in research... ...of experience and renowned engineering expertise to meet the needs... ...-discipline organization responsible for designing, building and... ...systems to help support tactical software development, testing, and...Temporary workWork experience placementInterim roleWork at officeRemote workRelocationFlexible hours- We are:Accenture’s Global Responsible AI team within the Global Data... ...if you’re an experienced RAI Engineer with a Responsible AI... ...better serve their customers and operate always-on enterprises. We’re... ...in a fast-paced environment.Excellent documentation and presentation...Work experience placementLive inWork at officeLocal area
$69.55k - $125.73k
...opportunity for a Java Software Engineer in our INTEL Sector... ..., Computer Network Operations (CNO), Mission... ...disciplines. Primary Responsibilities: The Java... ...environment and has excellent written and communication... ...enforcement and report the incident to the ....Local areaImmediate startFlexible hours$195k - $235k
...fast-paced and passionate team as a Principal Software Engineer on our Robotic Automation team. As... ...support our mission and growth. Your key responsibilities are as follows: Lead the... ..., self-discipline, and passion for excellence to assure accuracy of execution...Full timeLocal area$168.1k - $227.4k
...people and organizations operating in locations without... ...in all phases of software development from requirements... ...control. Job Responsibilities - Develop embedded... ...lead or leading an engineering team PREFERRED QUALIFICATIONS... ...to standards of excellence despite stressful...Permanent employmentInternshipLocal areaFlexible hours$130k - $150k
Job TitleLead Operations EngineerLocationSan Diego, CA 22400 US... ...data.The Lead Operations Engineer (LOE) is responsible for the end-to-end lifecycle... ...requirements, integrating software and hardware solutions for... ...MS Word, PowerPoint, and Excel, supporting system design,...- ...and empowered to excel, take informed risks... ...platform. As a Software Engineer, you must possess... ...on our roadmap. Responsibilities and Accountabilities... ...for production incident management... ...Reliability Engineering (SRE) concepts and... ...experience building and operating Kubernetes...Full timeTemporary workWork experience placementLive inRemote workFlexible hours
$109.65k
...your playground. Our engineers work on scalable systems... ...facing products that operate at global scale. This... ...at Amazon means real responsibility from day one. You'll... ...to operational excellence, ensures we deliver the... ...systems Knowledge of software development lifecycle...Full timeSummer workInternshipSeasonal workSummer internshipWork at officeLocal areaWorldwideMonday to Friday$114k - $171k
...Aeronautics Systems is looking to add a Principal Software Engineer - Java to join our team of... ...settings across all levels Duties and Responsibilities include, but are not limited to:... ...Learning Model development. • Excellent communication, interpersonal skills...Full timeRemote workRelocation packageShift work$132.4k - $251.6k
...businesses, world-class operations and investments in... ...experience and renowned engineering expertise to meet the... ...and defense. The Software organization develops... ...across the country, responsible for all aspects of the... ...solutions. Our Senior Principal Embedded Real-Time Software...Temporary workWork experience placementWork at officeRemote workRelocationFlexible hours$100k - $140k
...2026-11-12 Category: Software Subcategory: SW Testing... ...SAIC is seeking a Principal Software Test Engineer to support advanced submarine... ...and testing. This is an excellent opportunity for a motivated... ...configurations to meet operational requirements Exceptional...Full timeWork at officeRemote workShift work- ...Interactive, Technology, and Operations services, all powered by... ...dependable services at scale.Key Responsibilities:Design and implement... ...capacity planning, monitoring, incident management, reporting, and... ...management, reliability engineering, change management, and performance...Work experience placementLive inWork at officeLocal area
- ...Senior Site Reliability Engineer ensures the... ...performance of large-scale software systems through a... .... Key responsibilities involve automating operational tasks, improving observability... ...contributing to incident management, while also... ...model where the SRE team collaborates with...Permanent employment
$180k - $195k
...a Senior Director of Software Engineering to lead the design, delivery... ...systems that operate at scale across financial... ...velocity and quality. Responsibilities Own engineering... ...practices across CI/CD, incident response, and operational excellence Leverage modern AI‑...Work at officeFlexible hours$132.4k - $251.6k
...businesses, world-class operations and investments in... ...experience and renowned engineering expertise to meet the... ...and defense. Software Security Sensors and... ...looking to hire a Sr Principal Software Security Engineer... ...function and associated responsibilities, a candidate's work...Temporary workWork experience placementInterim roleWork at officeRemote workFlexible hours$160k - $239.95k
...Job Summary This role leads a software engineering team focused on delivering... ...platform solutions. The Manager is responsible for execution, team leadership, operational excellence, and delivery outcomes while... .... Own system stability and incident response, including...Full time$150.4k - $277.6k
Role Number: 200687074Team: Software and Services, HardwareLocation... ...Group is hiring a software engineer to build automation and... ...that connect to it, and our operations partners that control it.You... ...DescriptionYou will join the team responsible for developing, deploying,...WorldwideRelocation$190k - $285k
...Hivemind autonomy software, V-BAT and X-BAT... ...actively supports operations worldwide. For... ...for an experienced SRE lead to help drive... ...work with Cloud Engineering and product teams... ...turn lessons from incidents into engineering... ...Lead technical response to complex incidents...Full timeTemporary workPart timeWorldwide$96.8k - $121k
...Field Application Engineer II At TE, you will unleash... ...Sales, Engineering, Operations, Quality, and Program... ...background, excellent analytical skills, and... ...effectively support the responsibilities of the position remotely... ..., and to report the incident to your local...Contract workLocal areaRemote work$261.5k - $353.5k
...We’re seeking a Principal Software Engineer to shape how AI transforms everyday... ...decisions across teams Responsibilities Define the technical direction... ..., monitoring, and incident practices. Balance task... ...security, performance, and operating cost. ~ Sound judgment...Full time$160k - $200k
...6-11-03 Category: Engineering and Sciences Subcategory... ...a Digital Engineer (Software) to serve as the lead... ...Diego, CA. Responsibilities Serve as the lead... ..., cybersecurity, and operations teams to deliver... ...communication skills Excellent writing skills for...Full timeFor contractorsRemote workShift work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Software Engineer - SRE, Incident Response & Operational Excellence. Be the first to apply!
- principal software engineer San Diego, CA
- principal San Diego, CA
- principal scientist San Diego, CA
- principal applied scientist San Diego, CA
- senior principal scientist San Diego, CA
- senior principal cloud computing engineer San Diego, CA
- principal cloud computing engineer San Diego, CA
- software intern San Diego, CA
- bank software San Diego, CA
- software sales executive San Diego, CA








