Senior Data Architect - Databricks

Essentially need a resource who has led Databricks implementations previously

Senior Data Architect – Databricks

Position Overview

We are seeking an experienced Senior Data Architect with deep expertise in Databricks architecture and implementations to lead the design, development, and modernization of enterprise data platforms.

The ideal candidate will have a strong background in data architecture, cloud-based data engineering, and hands-on experience implementing Databricks solutions from initial design through production deployment.

This individual will play a critical role in defining data architecture strategies, implementing scalable data solutions, and helping organizations maximize the value of their data through modern cloud technologies.

We are looking for someone who can operate at both the strategic and technical levels—designing enterprise solutions while remaining hands-on throughout implementation.

Key Responsibilities

Data Architecture & Strategy

• Design and implement scalable, secure, and high-performing enterprise data architectures leveraging Databricks.
• Lead end-to-end Databricks implementations, including architecture design, environment setup, data integration, migration, and production deployment.
• Define architectural standards, best practices, and technical roadmaps for modern data platforms.
• Design and implement Lakehouse architectures supporting analytics, business intelligence, and advanced data processing.
• Evaluate existing data environments and recommend modernization strategies.
• Partner with business stakeholders, engineering teams, and technology leadership to translate business requirements into technical solutions.

Databricks Implementation & Engineering

• Lead the design, configuration, deployment, and optimization of Databricks environments.
• Architect and implement data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL.
• Design and implement Delta Lake solutions for reliable, scalable data storage and processing.
• Establish data ingestion, transformation, orchestration, and integration patterns.
• Implement medallion architecture (Bronze, Silver, Gold) and other appropriate data modeling frameworks.
• Configure and optimize Databricks workflows, compute resources, and data processing performance.
• Support migration of legacy data warehouses, ETL processes, and data platforms into Databricks.
• Implement CI/CD processes, infrastructure automation, and deployment best practices.

Data Governance, Security & Optimization

• Design and implement data governance frameworks utilizing Databricks Unity Catalog.
• Establish data quality, lineage, access control, and security standards.
• Ensure solutions meet enterprise security, compliance, and data management requirements.
• Optimize platform performance, scalability, reliability, and cloud infrastructure costs.
• Develop architectural documentation, technical standards, and implementation guidelines.

Technical Leadership & Collaboration

• Serve as the subject matter expert for Databricks architecture and implementation.
• Provide technical leadership and guidance to data engineers, developers, and cross-functional teams.
• Lead architecture discussions, design reviews, and technical decision-making.
• Collaborate with cloud infrastructure, DevOps, analytics, and business intelligence teams.
• Identify opportunities to improve data accessibility, platform efficiency, and overall data capabilities.

Required Qualifications

• 8+ years of experience in data architecture, data engineering, data warehousing, or related technical disciplines.
• 3+ years of hands-on experience with Databricks, including designing and implementing enterprise solutions.
• Demonstrated experience leading at least one end-to-end Databricks implementation, from architecture and initial setup through production deployment.
• Strong understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and Apache Spark.
• Advanced proficiency in Python, PySpark, and SQL.
• Experience designing and implementing scalable ETL/ELT pipelines and data integration solutions.
• Strong experience with at least one major cloud platform: Azure, AWS, or Google Cloud Platform.
• Experience with cloud-based data storage, data warehouses, and distributed processing technologies.
• Strong understanding of data modeling, data governance, data security, and performance optimization.
• Experience migrating or modernizing legacy data platforms into cloud-based architectures.
• Ability to develop architecture diagrams, technical documentation, and implementation roadmaps.
• Strong communication skills with the ability to collaborate with technical teams and business stakeholders.

Preferred Qualifications

• Databricks Certified Data Engineer Professional or Databricks Certified Data Engineer Associate.
• Databricks Certified Data Architect (if applicable to current certification offerings) or other advanced Databricks credentials.
• Experience implementing Databricks in multiple enterprise environments.
• Experience with Azure Data Factory, Azure Data Lake Storage, AWS Glue, Amazon S3, or similar cloud services.
• Experience with Terraform, infrastructure as code, and automated deployment pipelines.
• Familiarity with dbt, Airflow, or other orchestration and transformation technologies.
• Experience integrating Databricks with Power BI, Tableau, or other analytics platforms.
• Understanding of streaming data architectures and real-time processing.
• Experience supporting machine learning, AI, or advanced analytics workloads.
• Experience with enterprise data platform modernization initiatives.

What Success Looks Like

• Successfully architect and implement a production-ready Databricks environment.
• Establish scalable, repeatable, and well-governed data architecture standards.
• Deliver reliable data pipelines and integration frameworks that support enterprise reporting and analytics.
• Improve data processing performance, accessibility, and operational efficiency.
• Provide technical leadership that enables engineering teams to effectively develop and maintain solutions within Databricks.

Ideal Candidate Profile

The ideal candidate is a hands-on architect who has personally designed, built, and implemented Databricks environments, rather than someone whose experience is limited to maintaining or developing within an established platform.

They should be comfortable owning technical architecture decisions, guiding implementation teams, solving complex data challenges, and translating business needs into scalable enterprise data solutions.