Senior Data Engineer

Job Description

Job Description

SiloSmashers is hiring for a Senior Data Engineer to support a government customer. This position will be responsible for architecting, developing, optimizing, and maintaining high‑performance data ingestion, transformation, streaming, and analytical pipelines within Databricks‑enabled Lakehouse environments.

The selected candidate will serve as a senior technical expert for Databricks Lakehouse engineering, advanced Delta Lake optimization, governed data design (Unity Catalog), medallion‑architecture enforcement, structured streaming, and platform‑integrated cloud data engineering across enterprise platforms. This role is expected to collaborate closely with the Project Manager / Contractor Lead, federal technical teams, and cross‑task platform, SRE, governance, privacy, and development personnel.

Key Responsibilities

• Serve as the principal technical authority for Databricks data engineering, pipeline optimization, streaming, and Lakehouse standards across programs.
• Lead design reviews, technical roadmaps, and architectural decisions in partnership with federal SMEs and platform/security teams.
• Build and optimize batch and streaming pipelines using PySpark, Lakeflow Spark Declarative Pipelines, Delta Lake, Auto Loader, and Workflows with robust error handling, retries, alerting, and monitoring.
• Lead and enforce medallion‑architecture ingestion and refinement patterns, data lifecycle management, and governed data storage tiering.
• Implement advanced performance tuning and cost‑efficiency strategies, including cluster/sizing optimization, serverless compute adoption, workload health monitoring, and resource governance.
• Provide root cause analysis, disruption/outage reporting, and production support for Databricks jobs, pipelines, applications, and platform features.
• Coordinate with Privacy & Governance teams to enforce Unity Catalog RBAC/ABAC, row/column‑level security, governed tags, lineage, and compliance controls.
• Mentor mid‑level/junior engineers, perform code reviews, and uphold high engineering quality through testing strategies, quality gates, reproducible builds, and DevSecOps best practices.

Required Skills

• Demonstrated expertise designing and implementing Databricks Lakehouse solutions, including:
  • Delta Lake table design, schema evolution, expectations, and medallion‑architecture (bronze/silver/gold).
  • Databricks Workflows, Jobs, Auto Loader, COPY INTO, Structured Streaming, Databricks SQL Warehouses, and Unity Catalog governance models.
  • Performance and cost optimization (Photon, file sizing, Z‑ORDER, clustering/partitioning, caching, AQE, shuffle/skew mitigation).
• Hands‑on experience building distributed data pipelines using Python, PySpark, and cloud‑native services across AWS and/or Microsoft Azure.
• Demonstrated experience administering and optimizing relational and NoSQL data systems (e.g., PostgreSQL, Aurora, DynamoDB) and designing efficient, secure schemas.
• Experience developing and maintaining CI/CD pipelines using Git‑based workflows (e.g., GitHub Actions, Azure DevOps), Databricks Repos, Asset/Bundles, Terraform/IaC, and automated testing frameworks.
• Experience supporting federal environments and compliance‑aligned development.
• Experience applying agile development methodologies, including backlog management, sprint planning, and iterative delivery.
• Experience in AI/ML DevSecOps or agentic‑AI engineering, including Databricks Model Serving, Genie, Agent Bricks, or integration with AWS Kiro or other government‑approved platforms.

Mandatory Qualifications & Requirements

• Ability to obtain and maintain a Public Trust clearance.
• Eight (8) or more years of experience (7 of which must be within the last 10 years) in data engineering, ETL/ELT pipeline development, or distributed data processing within enterprise cloud environments.
• Bachelor’s or master’s degree in computer science, information systems, engineering, or related field; or twelve (12) total years of equivalent experience in lieu of a degree.
• Databricks Data Engineer Professional Certification
• At least one cloud-focused certification (e.g. AWS Developer Associate, Solutions Architect Associate/Professional, or equivalent).
• Preferred but not required: Databricks Lakehouse Architect, Generative AI Associate/Professional, or related ML/AI certifications.
• This position may require regular, physical attendance at customer sites.