Senior Data Engineer (Snowflake & Observability Implementation)

Senior Data Engineer – Data Observability (Snowflake, dbt, DMF)

We are looking for a highly skilled Senior Data Engineer with strong experience in Data Observability to help operationalize and scale a robust data reliability framework. This role will focus on implementing end-to-end observability across dbt, Snowflake Data Metric Functions (DMFs), Splunk, and OpsGenie, ensuring proactive detection and resolution of data quality issues across critical data assets. The mission is to establish a self-healing, highly visible data reliability layer that eliminates silent data failures and enables faster incident response.

Key Responsibilities

• Unified Metadata Collection & Persistence
• Standardize and automate dbt metadata capture across all model runs.
• Build and maintain a hardened dbt-to-Snowflake logging pipeline to persist run_results.json and manifest.json into an Observability schema.
• Implement automated cleanup and retention policies to manage storage efficiently.
• Apply data observability rules at scale by pushing dbt checks into Snowflake DMFs.
• Data Quality & Observability Framework
• Implement and manage observability rules across key dimensions:
• Validity (data types, formats)
• Freshness (timeliness, latency)
• Volume (record count reconciliation)
• Schema & Values (structural and value changes)
• Distribution (anomaly detection, trend deviations)
• Ensure data quality monitoring for high-priority and Tier-1 tables.
• DMF Thresholding & Performance Optimization
• Design and implement targeted Snowflake DMFs instead of blanket monitoring.
• Define dynamic thresholds (e.g., standard deviation-based) to reduce alert fatigue.
• Analyze and optimize DMF credit consumption, keeping monitoring costs within 5–10% of total compute.
• Observability & Alerting (Splunk Integration)
• Build a single pane of glass for data observability using Splunk.
• Create high-performance alerts correlating dbt job failures with DMF violations.
• Ensure alerts include contextual payloads such as:
• Failing dbt model or code link
• Table owner
• Downstream BI impact
• Incident Management & SLA Enforcement (OpsGenie)
• Integrate Splunk alerts with OpsGenie for actionable incident management.
• Configure:
• Priority-based alert routing (Warning vs Critical)
• Auto-resolution of alerts when issues self-heal
• SLA tracking for MTTD (Mean Time to Detect) and MTTR (Mean Time to Resolve)
• Reporting & Executive Visibility
• Build a Data Reliability Executive Dashboard (Splunk or Snowflake/Streamlit) to provide:
• Overall Data Health Score
• Volume and Freshness trends
• Top offending models/tables
• Month-over-month MTTD and MTTR improvements
• Operational Standards & Documentation
• Create detailed Runbooks / SOPs for on-call engineers.
• Implement Monitoring-as-Code using version control (Terraform, dbt project files).
• Maintain a weekly Observability Health Dashboard to identify noisy or unstable models.

Required Skills & Experience

• Strong hands-on experience with Snowflake, especially Data Metric Functions (DMFs)
• Advanced experience with dbt (metadata, testing, orchestration)
• Proven experience integrating observability tools like Splunk
• Experience with incident management platforms (OpsGenie preferred)
• Strong SQL and data modeling skills
• Experience building scalable, production-grade data pipelines
• Familiarity with cost optimization and performance tuning in Snowflake

Nice to Have

• Experience with Streamlit dashboards
• Infrastructure-as-Code (Terraform)
• Background in data governance or data reliability engineering
• Experience supporting on-call or production data platforms