Director of Platform Engineering and Observability

Job Description

***Position is bonus eligible***

Prestigious Financial Institution is currently seeking a Director of Platform Engineering with strong SRE and Observability leadership experience. Candidate will lead and manage this team to drive reliability, observability, and cloud platform engineering excellence across a large, complex cloud-based computing environment. The ideal candidate is a hands-on, data-driven technical leader, who can personally raise the bar on SRE and observability practices, reduce waste, optimize cloud efficiency, and improve performance, in close partnership with a dedicated SRE/monitoring team and a centralized architecture function.

Responsibilities:

Management Responsibilities:

\n
• Manage and develop a team of engineers and managers focused on SRE, observability, and cloud platform engineering, growing and retaining talent
\n
• Lead hiring and staffing for the team, including sourcing, interviewing, and selecting engineers and managers to build out the organization
\n
• Meet with team members regularly to provide coaching and feedback on performance
\n
• Perform evaluations and deal effectively with staff problems and corrective actions as needed
\n
• Develop employee career development plans to assist with team member career growth and development
\n
• Manage and participate in the implementation of production changes during defined maintenance windows and support on-call rotation
\n
• Serve as a point of escalation within the team for reliability, observability, and platform support issues
\n
• Foster an atmosphere of trust, respect, and high performance while displaying strong ethics and integrity
\n
• Manage project and daily work task planning and prioritization, meeting project deadlines while maintaining a high quality of work
\n
• Ensure team compliance with all appropriate policies and procedures, and institute corrective actions to address audit and other regulatory or compliance findings
\n
• Operate within budget; establish and assure adherence to schedules, work plans, and performance requirements
\n

Technical & Platform Responsibilities:

\n
• Bring deep, hands-on SRE and observability expertise to Platform Engineering, elevating the maturity, rigor, and technical depth of the organization’s observability, monitoring, and reliability practices in close partnership with the dedicated SRE and monitoring team
\n
• Serve as a technical authority on reliability engineering, mentoring engineers and managers and raising the bar on SLOs/SLIs, error budgets, incident response, and postmortem discipline across the platform organization
\n
• Own the definition, governance, and lifecycle of SLOs and SLAs across Platform Engineering, codified as SLO-as-code (e.g., OpenSLO, Sloth, or Nobl9) and version-controlled in the repo rather than only living in a vendor UI
\n
• Drive resilience engineering as a discipline across the platform, including chaos engineering (e.g., AWS FIS, Gremlin) and load/performance testing, to proactively validate and improve system resilience ahead of production incidents
\n
• Deliver golden paths and templates (Terraform modules, Helm charts, pipeline templates) that ship with logging, metrics, tracing, dashboards, alerts, and SLO defaults out of the box, reducing toil and accelerating safe onboarding for engineering teams
\n
• Act as product owner for observability within the platform organization, defining the roadmap and requirements for observability capabilities while the dedicated SRE and monitoring team retains ownership of the underlying tooling and operations
\n
• Own and drive platform engineering delivery and reliability metrics, including DORA metrics (deployment frequency, lead time for changes, change failure rate, MTTR), cycle time, and throughput, holding the organization accountable to continuous improvement
\n
• Define and report on a rounded set of platform and observability metrics: platform adoption (percentage of services on paved roads, service catalog completeness, time to first deploy, onboarding time), observability coverage (percentage of tier-1 services with SLOs, instrumented traces, and runbooks), observability health (MTTD, MTTA, alert actionability ratio, toil percentage, and paging load per engineer), observability economics (telemetry cost per service, cardinality and ingestion growth, and retention tiers), and developer experience (SPACE/DevEx measures alongside DORA)
\n
• Champion the Golden Signals, RED, and USE methods as the platform’s common language for monitoring and alerting
\n
• Ensure audit-grade telemetry practices, including log retention and immutability, PII and sensitive-data scrubbing, and access controls on observability data, mapped to relevant compliance frameworks (e.g., CIS, NIST, and SIFMU-specific resilience and reporting expectations)
\n
• Drive reliability across four core pillars: a data-driven culture (“data junkies”) that uses telemetry and metrics, not intuition, to make decisions; waste reduction through elimination of toil and inefficient processes; cloud measures such as utilization, rightsizing, and FinOps metrics; and continuous performance optimization
\n
• Translate reliability signals (SLOs, error budgets, incident learnings) surfaced by the SRE organization into concrete cloud platform investments and roadmap priorities
\n
• Drive results in the cloud platform by building a reliable, scalable, secure technology stack in collaboration with engineers and leaders, using leading industry practices
\n
• Champion infrastructure-as-code (IaC) practices across the cloud platform, ensuring provisioning, configuration, and environment management are automated, version-controlled, and consistently applied
\n
• Assess and plan for capacity needs within the cloud platform and forecast accordingly
\n
• Collaborate with the centralized architecture function on platform architecture decisions, ensuring cloud platform engineering execution aligns with broader architectural standards and direction
\n
• Implement and manage initiatives within your assigned area of responsibility with accountability for results and compliance with all controls and security requirements
\n
• Lead in the development of technology roadmaps and end-of-life technology plans
\n
• Effectively communicate project and operational service issues to senior management promptly with observations, decisions, and recommendations for corrective measures
\n
• Other duties as assigned
\n
• Manage a team of engineers and engineering managers focused on SRE, observability, and cloud platform engineering. Full people-management responsibilities, including hiring, performance management, coaching, career development, and corrective action.
\n

Qualifications:

\n
• [Required] 5+ years of demonstrated experience leading engineering teams, with an emphasis on developing key talent and cultivating positive, high-performing cultures
\n
• [Required] 10+ years of progressive, hands-on experience in software engineering with an understanding of large-scale computing solutions (primarily AWS), including software design and development, database architectures, IP networking, security, cloud operations, and performance tuning
\n
• [Required] Demonstrated, hands-on expertise building and maturing SRE and observability practices at scale, able to personally raise the technical bar for a dedicated SRE/monitoring team, not just consume their output
\n
• [Required] Demonstrated track record owning the definition and governance of SLOs and SLAs for mission-critical systems, and personally driving resilience engineering practices (chaos engineering, load/performance testing) to validate reliability ahead of incidents
\n
• [Required] Demonstrated track record driving reliability through a data-driven culture, waste/toil reduction, cloud efficiency measures, and performance optimization, treating metrics as the primary lens for every decision
\n
• [Required] Experience defining, instrumenting, and acting on software delivery performance metrics (DORA metrics, cycle time, deployment frequency, lead time, etc.) to drive engineering improvement initiatives
\n
• [Required] Strong consultative, communication, team player, and analytical skills, with the ability to regularly interact between various teams distributed across the US
\n
• [Required] Strong technical team leadership and technical project management skills
\n
• [Required] Relevant experience leading highly technical team members through adopting new technologies while maintaining highly available, mission-critical systems, with a proven track record of success
\n
• [Required] Ability to clearly communicate verbally and in writing to business and technology leaders, architects, developers, and team members
\n
• [Required] Must be able to collaborate effectively with a group of high-performing, technical individuals
\n
• [Required] Experience acting as a product owner, defining roadmap, requirements, and priorities, for a platform capability such as observability, ideally in partnership with a separate team that owns the underlying tooling and operations
\n
• [Required] Experience with architecting, implementing, and maintaining highly available mission-critical environments for 24x7 availability
\n
• [Required] Demonstrated history of working within deadlines and ability to work well under pressure
\n
• [Required] Experience managing work tasks using Agile methodology/scrum desired
\n
• [Preferred] Comfort with ambiguity and demonstrated ability to lead complex programs in a decentralized environment
\n
• [Preferred] Experience working in an environment with a defined production change control process; experience working with audits and compliance or in a regulated environment a plus
\n
• [Preferred] Experience in organizations with a mature, centralized SRE function, avoiding role/scope overlap
\n
• [Required] Deep expertise in OpenTelemetry, including instrumentation standards, auto-instrumentation, semantic conventions, and the OTel Collector, as the foundation for a vendor-neutral, paved-road instrumentation strategy
\n
• [Required] Hands-on experience with metrics engines and time-series databases, including Prometheus and PromQL, plus scale-out options such as Mimir, Thanos, VictoriaMetrics, or Amazon Managed Prometheus, including cardinality management
\n
• [Required] Hands-on experience with tracing and logging backends such as Tempo/Jaeger, Loki/Elastic/Splunk (Splunk is common in financial services), and AWS X-Ray
\n
• [Required] Experience with Kubernetes and Kafka observability specifically, including EKS metrics, kube-state-metrics, consumer lag, and broker health
\n
• [Required] Experience with alerting and incident tooling such as PagerDuty or Opsgenie, including ServiceNow integration, alert routing, and noise reduction
\n
• [Required] Hands-on experience with SLO-as-code frameworks such as OpenSLO, Sloth, or Nobl9, defining and governing SLOs in the repo rather than only in a vendor UI
\n
• [Required] Experience delivering golden paths and templates (Terraform modules, Helm charts, pipeline templates) that ship with logging, metrics, tracing, dashboards, alerts, and SLO defaults out of the box
\n
• [Required] Experience with resilience validation practices, including chaos engineering (e.g., AWS FIS, Gremlin) and load/performance testing
\n
• [Required] Deep, hands-on mastery of observability tooling and practices (metrics, distributed tracing, centralized logging, dashboards, alerting), e.g., Datadog, Prometheus/Grafana, CloudWatch, or equivalent, sufficient to elevate, not just consume, a dedicated SRE team’s capability
\n
• [Required] Deep understanding of SRE principles including SLOs/SLIs, error budgets, incident management, and postmortem culture, with a track record of driving adoption and maturity
\n
• [Required] Fluency in the Golden Signals, RED, and USE methods as applied frameworks for monitoring and alerting design
\n
• [Required] Hands-on experience with: Terraform, Kubernetes, Jenkins or other CI/CD tooling, Kafka, Github, and configuration management tools such as Puppet, Chef, or Ansible
\n
• [Required] Deep, hands-on expertise with infrastructure-as-code (IaC) tools and practices (e.g., Terraform, CloudFormation, CDK, Pulumi), with a track record of driving IaC adoption at scale across a cloud platform organization
\n
• [Required] Relevant experience with configuration and implementation of IaaS, Infrastructure as Code, AWS, Azure, etc.
\n
• [Required] Expert working knowledge of infrastructure design and components, such as servers, operating systems, networks, and storage
\n
• [Required] Basic understanding of good delivery practices and continual integration and improvement; Agile/Lean background for projects and project delivery
\n
• [Preferred] Experience with telemetry pipeline tools such as Fluent Bit, Vector, or Cribl for routing, sampling, redaction, and cost control
\n
• [Preferred] Experience with synthetic monitoring, real user monitoring (RUM), and eBPF-based observability
\n
• [Preferred] Familiarity with audit-grade telemetry practices (log retention/immutability, PII and sensitive-data scrubbing, access controls on observability data) and mapping observability controls to compliance frameworks such as CIS, NIST, or SIFMU-specific resilience requirements
\n
• [Preferred] Familiarity with engineering metrics/analytics platforms (e.g., LinearB, Jellyfish, Sleuth, Haystack, or internal equivalents) used to track DORA metrics and delivery performance
\n
• [Preferred] Competent in all phases of application development and implementation, including SDLC; hands-on scripting/development skills in Python, Ruby, Go, Java, etc. in a corporate environment strongly desired
\n
• [Preferred] Experience establishing IaC governance and standards (module libraries, policy-as-code, drift detection) across multiple teams or business units
\n
• [Preferred] Experience building a metrics-driven engineering culture, including scorecards, dashboards, or leadership reporting on delivery performance
\n
• [Required] Bachelor’s degree, preferably in a technical discipline (Computer Science, Mathematics, etc.), or equivalent combination of education and experience required; Master’s degree and relevant experience also considered
\n
• [Required] 10+ years’ experience in IT systems installation, operations, administration, and maintenance of cloud systems / virtualized servers, including 5+ years in a technical leadership