Databricks Architect

Databricks Architect


Experience: 15+ Years


Location: Remote


Visa: Any Visa

Job Summary

We are seeking an experienced Databricks Architect with 15+ years of overall IT experience and strong expertise in Lakehouse Architecture, Apache Spark, Databricks, and cloud data platforms. The ideal candidate will be responsible for designing scalable, secure, and high-performance modern data architectures and leading the implementation of enterprise-scale data solutions.

The candidate should have strong hands-on experience with Databricks and Spark, combined with deep knowledge of cloud-based data engineering, data architecture, governance, and optimization.

Key Responsibilities


Design and architect enterprise-scale Databricks Lakehouse solutions.


Define modern data architecture strategies leveraging Databricks, Apache Spark, Delta Lake, and cloud platforms.


Lead the design and implementation of scalable data ingestion, transformation, processing, and analytics solutions.


Develop and optimize large-scale Spark workloads using PySpark/Scala and SQL.


Design data pipelines and workflows using Databricks and cloud-native services.


Establish best practices for Lakehouse architecture, data modeling, security, governance, and performance optimization.


Work closely with data engineers, data scientists, architects, and business stakeholders to translate requirements into technical solutions.


Design solutions for batch and near-real-time/streaming data processing.


Implement and optimize Delta Lake capabilities, including ACID transactions, schema evolution, partitioning, and data optimization.


Drive cloud data platform architecture across AWS, Azure, and/or GCP.


Provide technical leadership and mentorship to data engineering teams.


Evaluate emerging technologies and recommend solutions aligned with enterprise data strategies.


Troubleshoot complex performance, scalability, reliability, and data quality issues.

Required Skills


15+ years of experience in data engineering, data architecture, or related technology roles.


Strong hands-on experience with Databricks and Lakehouse Architecture.


Expert-level knowledge of Apache Spark, including PySpark and/or Scala.


Strong experience with Delta Lake and modern data lake/lakehouse concepts.


Extensive experience with at least one major cloud platform:


Microsoft Azure


AWS


Google Cloud Platform (GCP)


Strong SQL and data modeling skills.


Experience designing enterprise-scale data pipelines and ETL/ELT frameworks.


Strong understanding of distributed computing and big-data architecture.


Experience with cloud data services, storage, compute, networking, and security.


Knowledge of data governance, security, lineage, metadata management, and access controls.


Experience with CI/CD, DevOps, Git, and infrastructure/application deployment practices.


Excellent architecture, problem-solving, communication, and stakeholder-management skills.

Preferred Qualifications


Databricks certifications such as Databricks Certified Data Engineer, Data Engineer Professional, or related certifications.


Experience with Unity Catalog and Databricks governance capabilities.


Experience with streaming technologies such as Kafka, Structured Streaming, or equivalent.


Experience with orchestration tools such as Databricks Workflows, Airflow, Azure Data Factory, or similar.


Experience integrating Databricks with enterprise BI and analytics platforms.


Experience implementing data mesh, data governance, or enterprise data modernization initiatives.


Candidate Profile