Databricks Lead/Architecture

Role Description

The Databricks Lead/Architecture position involves the following responsibilities:

• Architecture & Design:
  • Define and design end-to-end data Lakehouse architecture leveraging Databricks, Delta Lake, and cloud-native services.
  • Create reference architectures for batch, real-time, and streaming data pipelines.
  • Architect data ingestion, curation, storage, and governance frameworks.
  • Ensure the platform is scalable, secure, and optimized for performance and cost.
  • Establish standards for data lineage, metadata management, and compliance.
  • Work with enterprise architects to align Databricks solutions with the overall cloud strategy (AWS).
• Leadership & Delivery:
  • Lead and mentor a team of data engineers in building robust data pipelines.
  • Collaborate with data scientists, BI teams, and business stakeholders to enable advanced analytics and AI/ML use cases.
  • Drive adoption of DevOps and CI/CD practices for data engineering.
  • Review designs and solutions to ensure adherence to architectural principles.
• Implementation & Optimization:
  • Build, optimize, and manage large-scale PySpark/SQL pipelines in Databricks.
  • Enable real-time data processing through Kafka, Kinesis, or Event Hubs.
  • Implement security best practices, including RBAC, data masking, and encryption.

Qualifications

• Experience with Databricks, Delta Lake, and cloud-native services.
• Strong understanding of data architecture and data engineering principles.
• Proven leadership skills in managing data engineering teams.
• Familiarity with DevOps and CI/CD practices.
• Experience with real-time data processing technologies.

Requirements

• Ability to work remotely with quarterly in-office meetings in Washington, DC.
• Long-term commitment to the position.