Splunk Observability Engineer

Location: Chicago, IL onsite


To design, implement, and optimize a full-stack observability strategy using the Splunk Observability Cloud (formerly SignalFx) and Splunk Enterprise/Cloud. You will ensure that engineering teams have 360-degree visibility into system health, moving the organization from reactive "firefighting" to proactive "pattern-based" incident prevention.


Key Responsibilities
Data Orchestration: Architect the ingestion of the "Three Pillars" (Metrics, Logs, Traces) usingOpenTelemetry (OTel) collectors.
Aggregation Strategy: Develop logic to aggregate high-cardinality data to reduce "noise" while maintaining "signal" for troubleshooting.
Analytical Modeling: Use SPL (Search Processing Language) and SignalFlow to perform pattern analysis, detecting anomalies before they trigger traditional threshold alerts.
Visual Storytelling: Build executive and technical dashboards that correlate disparate data points (e.g., showing how a spike in 500-errors in Logs relates to a specific span in a Trace).
Required Hands on Technical Skills
1. Telemetry & Data Specialization
Logs: Proficiency in "Logging-in-Context." You must be able to link logs directly to trace IDs so developers can jump from a failing trace to the specific line of code in the logs.
Metrics: Expertise in SignalFlow (Splunk’s background streaming analytics language). You should know how to calculate percentiles ($P95, P99$), rates of change, and historical averages.
Traces: Deep understanding of Distributed Tracing. You must know how to instrument applications (Java, Python, Go) to capture spans and identify bottlenecks in microservices.
2. Pattern Analysis & Aggregation
Anomaly Detection: Ability to configure Metric Finder and MDetector using standard deviations or "Mean Absolute Deviation" to find outliers.
Data Scrubbing: Skills in using Splunk Ingest Actions or Edge Processors to filter, mask, or aggregate data at the edge to save on license costs and improve search speed.
Pattern Discovery: Using Splunk’s machine learning commands (e.g., findkeywords, cluster) to group millions of log events into a few dozen "patterns" for faster root cause analysis.
3. Hands on - Dashboards & Visualization
High-Cardinality Handling: Designing dashboards that don’t "break" when viewing thousands of containers.
Contextual Drill-downs: Building "Glass Tables" (in ITSI) or Unified Dashboards that allow a user to click a metric and immediately see the associated logs.
Frameworks: Familiarity with the Dashboard Studio and JSON-based dashboard definitions for version control (GitOps).
Preferred Qualifications & Certifications
DevOps & IAC skills
Splunk Cloud Certified Metrics User: Focuses on the metrics and alerting side.
Splunk Core Certified Power User: Essential for mastering complex SPL for log analysis.
OpenTelemetry Expert: Knowledge of the OTel Collector configuration (receivers, processors, exporters) is currently the most "in-demand" skill for this role.
Job Specification: Splunk Observability Engineer
Chicago, IL onsite


To design, implement, and optimize a full-stack observability strategy using the Splunk Observability Cloud (formerly SignalFx) and Splunk Enterprise/Cloud. You will ensure that engineering teams have 360-degree visibility into system health, moving the organization from reactive "firefighting" to proactive "pattern-based" incident prevention.


Key Responsibilities
Data Orchestration: Architect the ingestion of the "Three Pillars" (Metrics, Logs, Traces) usingOpenTelemetry (OTel) collectors.
Aggregation Strategy: Develop logic to aggregate high-cardinality data to reduce "noise" while maintaining "signal" for troubleshooting.
Analytical Modeling: Use SPL (Search Processing Language) and SignalFlow to perform pattern analysis, detecting anomalies before they trigger traditional threshold alerts.
Visual Storytelling: Build executive and technical dashboards that correlate disparate data points (e.g., showing how a spike in 500-errors in Logs relates to a specific span in a Trace).
Required Hands on Technical Skills
1. Telemetry & Data Specialization
Logs: Proficiency in "Logging-in-Context." You must be able to link logs directly to trace IDs so developers can jump from a failing trace to the specific line of code in the logs.
Metrics: Expertise in SignalFlow (Splunk’s background streaming analytics language). You should know how to calculate percentiles ($P95, P99$), rates of change, and historical averages.
Traces: Deep understanding of Distributed Tracing. You must know how to instrument applications (Java, Python, Go) to capture spans and identify bottlenecks in microservices.
2. Pattern Analysis & Aggregation
Anomaly Detection: Ability to configure Metric Finder and MDetector using standard deviations or "Mean Absolute Deviation" to find outliers.
Data Scrubbing: Skills in using Splunk Ingest Actions or Edge Processors to filter, mask, or aggregate data at the edge to save on license costs and improve search speed.
Pattern Discovery: Using Splunk’s machine learning commands (e.g., findkeywords, cluster) to group millions of log events into a few dozen "patterns" for faster root cause analysis.
3. Hands on - Dashboards & Visualization
High-Cardinality Handling: Designing dashboards that don’t "break" when viewing thousands of containers.
Contextual Drill-downs: Building "Glass Tables" (in ITSI) or Unified Dashboards that allow a user to click a metric and immediately see the associated logs.
Frameworks: Familiarity with the Dashboard Studio and JSON-based dashboard definitions for version control (GitOps).
Preferred Qualifications & Certifications
DevOps & IAC skills
Splunk Cloud Certified Metrics User: Focuses on the metrics and alerting side.
Splunk Core Certified Power User: Essential for mastering complex SPL for log analysis.
OpenTelemetry Expert: Knowledge of the OTel Collector configuration (receivers, processors, exporters) is currently the most "in-demand" skill for this role.