Databricks Architect

Role: Databricks Architect

Location: Bolingbrook, IL Remote

Duration: 12 Months

Job Summary:

• We are seeking an experienced Databricks Architect to design, implement, and govern enterprise-scale data platforms using Databricks. The ideal candidate will have strong expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, cloud platforms, data lakehouse architecture, and real-time data processing.
• The Databricks Architect will provide technical leadership across data architecture, cloud integration, data engineering, governance, security, analytics enablement, and platform optimization.

Key Responsibilities:

• Design and implement scalable data lakehouse architectures using Databricks.
• Define enterprise data models, architecture standards, governance frameworks, and security controls.
• Lead the migration of legacy data warehouses, ETL processes, and data workloads to Databricks.
• Architect and implement batch and real-time data processing solutions using Apache Spark and PySpark.
• Design producer/consumer architectures for streaming and event-driven data platforms.
• Evaluate and define appropriate streaming technologies, including Apache Kafka, GCP Pub/Sub, Azure Event Hubs, and AWS Kinesis.
• Optimize data platforms for performance, scalability, reliability, and cost efficiency.
• Develop and implement data integration and orchestration solutions.
• Establish CI/CD, DataOps, DevOps, monitoring, logging, and operational best practices.
• Implement data security, access management, governance, and compliance controls.
• Collaborate with business stakeholders, data engineers, data scientists, cloud architects, and analytics teams.
• Provide technical leadership and mentorship to data engineering teams.
• Support analytics and BI initiatives using platforms such as Power BI and Tableau.
• Evaluate emerging Databricks and cloud data technologies and recommend appropriate solutions.

Required Skills & Qualifications:

• 8+ years of experience in data engineering, data architecture, or enterprise data platforms.
• 3+ years of hands-on experience designing and architecting Databricks solutions.

Strong expertise in:

• Databricks
• Apache Spark
• PySpark
• SQL
• Delta Lake
• Hands-on experience with at least one major cloud platform:
• Azure / Azure Databricks
• AWS / AWS Databricks
• GCP / GCP Databricks
• Strong experience with real-time and streaming data architectures.
• Experience with one or more streaming technologies:
• Apache Kafka
• GCP Pub/Sub
• Azure Event Hubs
• AWS Kinesis

• Strong understanding of producer/consumer architecture, scalability, reliability, and technology trade-offs.
• Experience designing data lakehouse architectures and enterprise data governance.
• Hands-on experience with data integration tools and orchestration frameworks.
• Strong understanding of data security, identity/access management, compliance, and governance.
• Experience with CI/CD and automated deployment processes for data platforms.
• Strong communication, architecture documentation, problem-solving, and technical leadership skills.

Preferred Skills:

• Databricks Certified Data Engineer Professional certification or equivalent experience.
• Experience with Unity Catalog.
• Experience with Delta Live Tables (DLT).
• Experience with Databricks Workflows.
• Experience with Power BI, Tableau, or other BI platforms.
• Knowledge of DataOps, DevOps, and MLOps practices.
• Experience with cloud-native data services and enterprise data modernization initiatives.

Education:

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.
• Master's degree preferred.