Observability engineer

Role: Observability Engineer

Location: Irvine, CA (Hybrid)

Duration: 12+Months

Description:

• As a Senior Observability Engineer, you will be responsible for designing, implementing, migrating, and optimizing end‑to‑end monitoring and observability solutions to ensure the reliability, performance, and resilience of distributed systems and application services for our clients. You will play a critical role in advancing observability maturity across complex legacy, distributed and hybrid environments by enabling proactive detection, rapid diagnosis, efficient resolution of incidents and using AI capabilities.
• This role requires deep expertise in real‑time monitoring, intelligent alerting, anomaly detection, and automated remediation, with a strong focus on reducing operational risk, minimizing alert fatigue, and improving overall service reliability.

What You Will Do:

• Assess the current state of monitoring and observability across applications and systems, including identifying alert fatigue, monitoring gaps, and coverage deficiencies.
• Define and execute strategies to incrementally improve the monitoring and observability maturity of platforms, applications, and infrastructure.
• Design and implement end‑to‑end observability solutions that provide comprehensive visibility into business transactions, service dependencies, and underlying technical components.
• Establish and promote monitoring best practices focused on noise reduction, controlled metric cardinality, and the prevention of duplicate or redundant telemetry.
• Define and implement automated alerting strategies aligned with Service Level Objectives (SLOs) and Service Level Agreements (SLAs) to ensure actionable and meaningful alerts.
• Develop and enforce monitoring audit standards to support governance, compliance, and regulatory requirements.
• Act as an escalation point for complex or critical monitoring‑related incidents and provide strategic guidance and recommendations to engineering and operations teams.
• Automate monitoring configurations, policy management, and telemetry collection using CI/CD pipelines and Infrastructure as Code (IaC) practices with tools such as Helm, Ansible, and Terraform.
• Build reusable automation frameworks and standardized reporting solutions to support consistent monitoring rollouts, configuration management, and operational insights.
• Leverage AI and machine learning techniques to enhance observability outcomes, including intelligent anomaly detection, alert noise reduction, predictive incident identification, automated root‑cause analysis, and data‑driven insights to improve service reliability and operational efficiency.

Required Qualifications:

• Overall 10+ years of experience out of which, 7+ years of solid experience with APM, monitoring, observability and event management tools including Dynatrace/AppDynamics, Splunk, Cortex, Prometheus, Grafana, and Netcool.
• Experience with ITSM, ticketing tools and their integration with monitoring tools.
• Proficiency in Application Workloads (Binary, Java, Python, .NET, Batch Jobs).
• Experience in Python, Bash, PowerShell or JavaScript for automation of tasks.
• Exposure to CI/CD pipelines and IaC (Infrastructure as Code).
• Strong in analytical and problem-solving skills for diagnosing complex issues.
• Effective in communication, individual leadership, and cross-functional team collaboration.
• Ability to think outside the box, sensitivity towards business impacts, and self-awareness to refine processes.
• Bachelor’s degree in computer science or engineering field.

Preferred Qualifications:

• Proficiency in broader aspects of monitoring and observability (APM, System Monitoring, Logs, Tracing, Visualization, Reporting and Integration).
• Experience in automation/programming/coding to an extent that can instrument monitoring solutions for a given platform/tooling/practice.
• Certified professional in Dynatrace/AppDynamics, Splunk, ITIL or AI.