Principal Site Reliability Engineer

At Early Warning, we've powered and protected the U.S. financial system for over thirty years with cutting-edge solutions like Zelle , Paze , and so much more. As a trusted name in payments, we partner with thousands of institutions to increase access to financial services and protect transactions for hundreds of millions of consumers and small businesses.

Positions located in Scottsdale, San Francisco, Chicago, or New York follow a hybrid work model to allow for a more collaborative working environment.

Candidates responding to this posting must independently possess the eligibility to work in the United States, for any employer, at the date of hire. This position is ineligible for employment Visa sponsorship.

Role Summary

The Principal Site Reliability Engineer applies software engineering and systems engineering practices to improve the reliability, resilience, scalability, and operational health of production services. The role partners with Software Engineering and other technology teams to ensure reliability, observability, recoverability, performance, and operational readiness are engineered into systems throughout their lifecycle.

The role operates at enterprise scope, establishing technical direction and applying evidence-driven engineering, technical rigor, sound judgment, automation, and broad systems expertise across organizational boundaries.

Core Responsibilities

• Use software engineering, automation, and DevOps principles and practices to continually improve how services are built, tested, deployed, observed, operated, and recovered.

• Use data, evidence, experimentation, and rigorous engineering analysis appropriate to the level to identify reliability risks, test assumptions, and guide technical decisions.

• Define, implement, or improve SLIs, SLOs, error budgets, and other service-health measures appropriate to the scope of responsibility.

• Improve observability through metrics, logging, tracing, monitoring, alerting, dashboards, and service-health instrumentation.

• Drive continuous improvement across CI/CD, observability, deployment practices, Infrastructure as Code, automation, testing, incident response, capacity management, resilience, and operational readiness.

• Identify recurring or systemic production issues and translate operational experience into improvements in code, architecture, automation, tooling, and engineering practices.

• Partner with Software Engineering teams to incorporate reliability, resiliency, scalability, performance, observability, recoverability, and operational readiness throughout the development lifecycle.

• Participate in or lead incident response, troubleshooting, service restoration, and blameless post-incident learning appropriate to the level.

• Provides enterprise-level technical leadership for critical production incidents and establishes or influences engineering practices that improve incident response, escalation, service restoration and sustainable on-call operations across the organization.

• Reduce operational toil and unnecessary manual intervention through software, automation, reusable patterns, and better engineering practices.

Leveling Intent

Principal represents enterprise domain-level technical leadership and organizational impact. Deep individual expertise is expected, but Principal-level impact comes from identifying systemic risk, establishing technical direction, influencing engineering practices and architecture, and multiplying the capability of the broader engineering organization.

Level Expectations

• Acts as an enterprise force multiplier, raising the effectiveness and technical capability of engineers and teams across the organization while building sustainable organizational capability rather than individual dependency.

• Demonstrates software engineering, systems thinking, troubleshooting, and production reliability capabilities appropriate to the level.

• Applies evidence-driven reasoning and technical rigor to distinguish observed facts from assumptions and make defensible engineering recommendations.

• Shares knowledge and contributes to sustainable engineering capability rather than creating dependency on individual expertise.

• Operates with significant autonomy across the organization's most consequential reliability challenges.

• Establishes enterprise technical direction, develops senior technical leaders, and demonstrates impact well beyond systems personally touched.

Minimum Qualifications

• Typically 15+ years of relevant professional experience in Software Engineering, Site Reliability Engineering, Systems Engineering, Cloud/Platform Engineering, DevOps, Infrastructure Engineering, Architecture where applicable, or a comparable technical discipline.

• Experience with software development or scripting using one or more modern programming languages.

• Experience with software engineering principles, distributed systems, production troubleshooting, automation, and observability appropriate to the level.

• Experience with public cloud technologies and architectures, preferably AWS, along with infrastructure, networking, Linux/Unix, and modern application architectures appropriate to the level.

• Demonstrated analytical, problem-solving, communication, and collaboration skills appropriate to the scope of the role.

Preferred Qualifications

• Hands-on experience with AWS is preferred, or comparable experience with another major cloud platform such as Microsoft Azure, Google Cloud Platform (Google Cloud Platform), or Oracle Cloud Infrastructure (OCI).

• Experience developing, deploying, operating, or improving highly available production software or distributed systems.

• Experience with CI/CD, Infrastructure as Code, containers or orchestration, observability, monitoring, alerting, and software-delivery automation.

• Experience with SLIs, SLOs, error budgets, incident management, performance analysis, capacity management, resilience testing, disaster recovery, or operational readiness appropriate to the level.

• Experience creating reusable automation, tooling, platforms, patterns, or practices that improve engineering effectiveness.

• Bachelor's degree in Computer Science, Software Engineering, Computer Engineering, Information Systems, or a related technical field, or equivalent practical experience.

The base pay scale for this position in:
Phoenix, AZ in USD per year is: $173,000 - $230,000.
San Francisco, CA in USD per year is: $207,000 - $276,000.
Additionally, candidates are eligible for a discretionary incentive plan and benefits.

This pay scale is subject to change and is not necessarily reflective of actual compensation that may be earned, nor a promise of any specific pay for any specific candidate, which is always dependent on legitimate factors considered at the time of job offer. Early Warning Services takes into consideration a variety of factors when determining a competitive salary offer, including, but not limited to, the job scope, market rates and geographic location of a position, candidate's education, experience, training, and specialized skills or certification(s) in relation to the job requirements and compared with internal equity (peers). The business actively supports and reviews wage equity to ensure that pay decisions are not based on gender, race, national origin, or any other protected classes.

Physical Requirements

Early Warning works together in a highly collaborative office environment.Working conditions consist of a normal office environment. Work is primarily sedentary and requires extensive use of a computer and involves sitting for periods of approximately four hours. Work may require occasional standing, walking, kneeling, and reaching. Must be able to lift 10 pounds occasionally and/or negligible amount of force frequently. Requires visual acuity and dexterity to view, prepare, and manipulate documents and office equipment including personal computers. Requires the ability to communicate with internal and/or external customers.

Employee must be able to perform essential functions and physical requirements of position with or without reasonable accommodation.

Candidates responding to this posting must independently possess the eligibility to work in the United States at the date of hire.

Some of the Ways We Prioritize Your Health and Happiness

• Healthcare Coverage -Competitive medical (PPO/HDHP), dental, and vision plans as well as company contributions to your Health Savings Account (HSA) or pre-tax savings through flexible spending accounts (FSA) for commuting, health & dependent care expenses.
• 401(k) Retirement Plan -Featuring a 100% Company Safe Harbor Match on your first 6% deferral immediately upon eligibility.
• Paid Time Off - Flexible Time Off for Exempt (salaried) employees, as well as generous PTO for Non-Exempt (hourly) employees, plus 11 paid company holidays and a paid volunteer day.
• 12 weeks of Paid Parental Leave
• Maven Family Planning - provides support through your Parenting journey including egg freezing, fertility, adoption, surrogacy, pregnancy, postpartum, early pediatrics, and returning to work.

And SO much more! We continue to enhance our program, so be sure to check our Benefits page here for the latest. Our team can share more during the interview process!

Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.

Early Warning Services, LLC ("Early Warning") considers for employment, hires, retains and promotes qualified candidates on the basis of ability, potential, and valid qualifications without regard to race, religious creed, religion, color, sex, sexual orientation, genetic information, gender, gender identity, gender expression, age, national origin, ancestry, citizenship, protected veteran or disability status or any factor prohibited by law, and as such affirms in policy and practice to support and promote equal employment opportunity and affirmative action, in accordance with all applicable federal, state, and municipal laws. The company also prohibits discrimination on other bases such as medical condition, marital status or any other factor that is irrelevant to the performance of our employees.