Principal Site Reliability Engineer

Job Title

Delivers services at high scale and high availability with resilience by using automation and Infrastructure Code. Builds reliability into the ecosystem by applying best practices in resiliency engineering and observability by developing resiliency tools and capabilities for observability and chaos testing of pipelines. Combines systems and software engineering techniques with site reliability engineering practices to create reliable user experiences to support workplace investing, healthcare and defined benefits organizations. Assists teams scale through production insights, operational automation, developer guidance, real-time metrics, and automation. Provides training, support, and alignment to ensures Site Reliability Engineers have the skills, tools, and opportunities to accomplish engineering reliable systems. Provides Product and Platform teams with engineering expertise that enables them to clarify and achieve their system reliability goals. Partners with our key stakeholders in defining and adopting policies, processes and practices that lead to reliable Information Technology systems and measures compliance with those policies.

Primary Responsibilities:

• Provides cloud support and enhances cloud capabilities according to Site Reliability Engineering principles -- observability, automation, and resiliency.
• Develops and enhances internal chaos framework to streamline chaos executions and reporting.
• Facilitates the adoption of chaos engineering by application teams, helping to perform chaos testing, and to analyze business-critical applications to understand the weaknesses and increase application resiliency.
• Develops and designs products created around the Site Reliability Engineering domain to improve stability and improve platform availability.
• Collaborates with business and technology teams to scale the products and automation across business units.
• Minimizes the impact of operational problems by developing strategies and tools to remediate the issues.
• Provides technical leadership on chaos testing for cloud and on-premises based applications.
• Develops scripts and applications to automate repeatable business processes.
• Advises senior management on technical strategy and tools.
• Mentors team members to build core competencies required in Site Reliability Engineering space.

Education and Experience:

Bachelor's degree in Computer Science, Engineering, Information Technology, Information Systems, or a closely related field (or foreign education equivalent) and five (5) years of experience as a Principal Site Reliability Engineer (or closely related occupation) designing and automating container and Cloud-based platform products and infrastructure solutions within a production environment.

Or, alternatively, Master's degree in Computer Science, Engineering, Information Technology, Information Systems or a closely related field (or foreign education equivalent) and three (3) years of experience as a Principal Site Reliability Engineer (or closely related occupation) designing and automating container and Cloud-based platform products and infrastructure solutions within a production environment.

Skills and Knowledge:

• Demonstrated Expertise ("DE") developing and designing products created around Site Reliability Engineering pillars to improve stability and improve platform availability for containerized workloads and on-premises services using Kubernetes; managing and interpreting datasets using query languages; and developing reports using PowerBi and Grafana.
• DE managing Cloud and on-premises systems using infrastructure-as-code tools -- Azure ARM and Terraform; and building, operating, monitoring, logging, and alerting services of distributed systems at scale and utilizing modern monitoring tools – Datadog and Splunk.
• DE creating solutions that support DevOps practice for delivery and operations of services using Jenkins and Azure DevOps, Team Foundation Version Control, and Cloud Formation Template.
• DE maintaining scalability and resiliency of applications deployed on Amazon Web Services (AWS) and Azure using LAMBDA, API Gateway, Fault Injection Service (FIS), and Azure Chaos Studio; and developing scripts and applications to automate repeatable business processes and providing solutions to program needs for applications hosted on Windows and Linux using Python.