Senior Databricks Engineer

Role Description

Koniag Emerging Technologies, LLC, a Koniag Government Services company, is seeking a Senior Databricks Engineer with a Secret security clearance to support KET and our government customer in Washington, DC. This is a remote position.

The Senior Databricks Engineer will be responsible for designing, developing, and deploying scalable data infrastructure while driving key data engineering initiatives. By leveraging the Databricks platform and deep expertise in modern data warehousing and the Medallion Architecture, the engineer will build data pipelines supporting the Department of State’s “4Ps” data lake. The ideal candidate will be able to support the following functions:

• Cross-Functional Collaboration: Work closely with project managers, team members, and business owners to gather requirements, communicate progress, and ensure data meets customer needs.
• Architecture & Design: Architect and implement scalable, highly available data pipelines using Databricks, adhering strictly to the Medallion Architecture (Bronze, Silver, Gold layers) to ensure data quality, consistency, and optimized reporting.
• Data Quality Governance & Monitoring: Establish comprehensive data quality standards and implement monitoring dashboards to enable business owners and responsible parties to track and proactively address data issues.
• Pipeline Development: Write clean, efficient, and production-ready code using PySpark and SQL for complex data extraction, transformation, and loading (ETL/ELT) processes. Expertise in DLT and structured streaming pipelines.
• API Integration: Design and build robust data ingestion pipelines to reliably pull from various third-party external systems using RESTful API integrations.
• Automated Deployments (CI/CD): Implement infrastructure and pipelines as code. Utilize Databricks Asset Bundles (DABs) integrated with Azure DevOps to manage and automate deployments across Development, Staging, and Production environments safely and reliably.
• Code Reusability: Develop, package, and maintain custom Python libraries and modules to establish common frameworks used across multiple data projects and teams.
• Data Warehousing: Apply foundational and advanced data warehousing concepts (e.g., dimensional modeling, slowly changing dimensions) to optimize data storage and retrieval for downstream analytics and business intelligence.
• Performance Tuning: Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs to ensure maximum efficiency and cost-effectiveness.
• External BI & Native Dashboarding: Securely share data with external BI tools (such as PowerBI or SAP BusinessObjects) and develop dashboards and capabilities natively within the Databricks environment.
• Mentorship & Leadership: Serve as a technical SME for the data engineering team, conducting code reviews, establishing best practices, and mentoring other staff members.

Qualifications

• Knowledge of Object Oriented Programming (OOP) concepts and proficiency in one or more programming or scripting languages (Java, C#, Python, JavaScript, PowerShell).
• Experience: 8+ years of dedicated Data Engineering experience, with at least 3+ years functioning in a senior capacity focused on the Databricks ecosystem.
• Core Languages: Expert-level proficiency in PySpark and complex SQL.
• Python Library Management: Proven ability to create, package, and manage custom Python libraries (e.g., building Wheel files) for scalable reuse across multiple data pipelines and projects.
• API Data Ingestion: Deep understanding of REST methods (GET, POST, PUT, etc.) with hands-on experience extracting, paginating, and processing data from external API sources.
• Frameworks & Methodology: Deep theoretical and practical understanding of modern Data Warehousing concepts and hands-on experience building out the Medallion Architecture.
• DevOps & CI/CD: Proven experience automating Databricks deployments. Hands-on experience configuring and deploying Databricks Asset Bundles (DABs) using Azure DevOps (creating YAML pipelines, managing service principals, and handling multi-workspace deployments).
• Cloud Platforms: Strong familiarity with the broader cloud ecosystem surrounding Databricks (e.g., Azure Data Lake Storage Gen2, Azure Key Vault).
• Version Control: Proficiency with Git-based version control workflows.
• Concise and comprehensive written and oral communication skills for both technically understanding and customer usage support.

Requirements

• Secret Security clearance.

Desired Skills

• Experience with a wide range of data products, BI tools, and visualization libraries such as Tableau, PowerBI, Jupyter Notebooks, d3js, and Analytics Search Tools such as Elasticsearch, SOLR, Lucene.
• Experience working with HR application (PeopleSoft) data for reporting purposes.
• 3+ years of .NET, Java or similar OOP experience.
• Experience writing Oracle PL/SQL queries, packages, and procedures.
• Experience architecting Data Vault 2.0 solutions.

Benefits

• Competitive compensation.
• Extraordinary benefits package including health, dental and vision insurance.
• 401K with company matching.
• Flexible spending accounts.
• Paid holidays.
• Three weeks paid time off.
• And more.