Databricks Data Architect

Role Description

The Databricks Data Architect is a senior technical lead responsible for building and optimizing a robust data platform in a financial services environment. You will lead a team of 10+ data engineers and own the end-to-end architecture and implementation of the Databricks Lakehouse platform. You will collaborate closely with application development and analytics teams to design scalable data solutions that drive business insights. This position demands deep expertise in Databricks (Azure), hands-on experience with PySpark and Delta Lake, and strong leadership to ensure best practices in data engineering, performance tuning, and governance.

Key Responsibilities

• Lead Data Engineering Team: Lead, mentor, and manage a team of 10+ data engineers, providing technical guidance, code reviews, and career development to foster a high-performing team.
• Databricks Platform Ownership: Own the Databricks platform architecture and implementation, ensuring the environment is secure, scalable, and optimized for the organization’s data processing needs. Design and oversee the Lakehouse architecture leveraging Delta Lake and Apache Spark.
• Unity Catalog Implementation: Implement and manage Databricks Unity Catalog for unified data governance. Ensure fine-grained access controls and data lineage tracking are in place to secure sensitive financial data and comply with industry regulations.
• Cluster Provisioning & Policies: Provision and administer Databricks clusters (in Azure), including configuring cluster sizes, auto-scaling, and auto-termination settings. Set up and enforce cluster policies to standardize configurations, optimize resource usage, and control costs across different teams and projects.
• Databricks SQL Optimization: Collaborate with analytics teams to develop and optimize Databricks SQL queries and dashboards. Tune SQL workloads and caching strategies for faster performance and ensure efficient use of the query engine.
• Performance Tuning: Lead performance tuning initiatives for Spark jobs and ETL pipelines. Profile data processing code (PySpark/Scala) to identify bottlenecks and refactor for improved throughput and lower latency. Implement best practices for incremental data processing with Delta Lake, and ensure compute cost efficiency (e.g., by optimizing cluster utilization and job scheduling).
• Data Solutions Collaboration: Work closely with application developers, data analysts, and data scientists to understand requirements and translate them into robust data pipelines and solutions. Ensure that data architectures support analytics, reporting, and machine learning use cases effectively.
• DevOps Integration: Integrate Databricks workflows into the CI/CD pipeline using Azure DevOps and Git. Develop automated deployment processes for notebooks, jobs, and clusters (infrastructure-as-code) to promote consistent releases. Manage source control for Databricks code (using Git integration) and collaborate with DevOps engineers to implement continuous integration and delivery for data projects.
• Data Governance & Security: Collaborate with security and compliance teams to uphold data governance standards. Implement data masking, encryption, and audit logging as needed, leveraging Unity Catalog and Azure security features to protect sensitive financial data.
• Platform Innovation: Stay up-to-date with the latest Databricks features and industry best practices. Proactively recommend and implement improvements (such as new performance optimization techniques or cost-saving configurations) to continuously enhance the platform’s reliability and efficiency.

Qualifications

• Bachelor’s degree in Computer Science, Information Systems, or a related field.
• 7+ years of experience in data engineering, data architecture, or related roles, with a track record of designing and deploying data pipelines and platforms at scale.
• Significant hands-on experience with Databricks (preferably Azure Databricks) and the Apache Spark ecosystem.
• Proficient in building data pipelines using PySpark/Scala and managing data in Delta Lake format.
• Strong experience working with cloud data platforms (Azure preferred, or AWS/GCP).
• Familiarity with Azure data services (such as Azure Data Lake Storage, Azure Blob Storage, etc.) and managing resources in an Azure environment.
• Advanced SQL skills with the ability to write and optimize complex queries.
• Solid understanding of data warehousing concepts and performance tuning for SQL engines.
• Proven ability to optimize ETL jobs and Spark processes for performance and cost efficiency.
• Demonstrated experience implementing data security and governance measures.
• Experience leading and mentoring engineering teams.
• Excellent analytical and problem-solving skills.

Requirements

• Databricks Certified Data Engineer Professional (highly preferred) or Databricks Certified Data Engineer Associate.
• Prior experience in the financial services industry or other highly regulated industries.
• Master’s degree in Computer Science, Data Engineering, or related field is a plus.
• Exposure to related big data and streaming tools such as Apache Kafka/Event Hubs, Apache Airflow or Azure Data Factory for orchestration.
• Experience implementing CI/CD pipelines for data projects.
• Proven strategies or projects where you significantly improved performance or reduced cloud costs in a Databricks/Spark environment.

Benefits

• Competitive salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance.
• Retirement savings plan with company matching.
• Flexible working hours and remote work options.
• Professional development opportunities and training.