Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Databricks Data Architect

$140 - $150 per hour
Full-time

NewVison

The Databricks Data Architect is a senior technical lead responsible for building and optimizing a robust data platform in a financial services environment. You will lead a team of 10 data engineers and own the end-to-end architecture and implementation of the Databricks Lakehouse platform. You will collaborate closely with application development and analytics teams to design scalable data solutions that drive business insights. This position demands deep expertise in Databricks (Azure) hands-on experience with PySpark and Delta Lake and strong leadership to ensure best practices in data engineering performance tuning and governance.

Key Must have skills

  • Lead Data Engineering Team: Lead mentor and manage a team of 10 data engineers providing technical guidance code reviews and career development to foster a high-performing team.
  • Databricks Platform Ownership: Own the Databricks platform architecture and implementation ensuring the environment is secure scalable and optimized for the organizations data processing needs. Design and oversee the Lakehouse architecture leveraging Delta Lake and Apache Spark.
  • Unity Catalog Implementation: Implement and manage Databricks Unity Catalog for unified data governance. Ensure fine-grained access controls and data lineage tracking are in place to secure sensitive financial data and comply with industry regulations.
  • Cluster Provisioning & Policies: Provision and administer Databricks clusters (in Azure) including configuring cluster sizes auto-scaling and auto-termination settings. Set up and enforce cluster policies to standardize configurations optimize resource usage and control costs across different teams and projects.
  • Databricks SQL Optimization: Collaborate with analytics teams to develop and optimize Databricks SQL queries and dashboards. Tune SQL workloads and caching strategies for faster performance and ensure efficient use of the query engine.
  • Performance Tuning: Lead performance tuning initiatives for Spark jobs and ETL pipelines. Profile data processing code (PySpark/Scala) to identify bottlenecks and refactor for improved throughput and lower latency. Implement best practices for incremental data processing with Delta Lake and ensure compute cost efficiency (e.g. by optimizing cluster utilization and job scheduling).
  • Data Solutions Collaboration: Work closely with application developers data analysts and data scientists to understand requirements and translate them into robust data pipelines and solutions. Ensure that data architectures support analytics reporting and machine learning use cases effectively.
  • DevOps Integration: Integrate Databricks workflows into the CI/CD pipeline using Azure DevOps and Git. Develop automated deployment processes for notebooks jobs and clusters (infrastructure-as-code) to promote consistent releases. Manage source control for Databricks code (using Git integration) and collaborate with DevOps engineers to implement continuous integration and delivery for data projects.
  • Data Governance & Security: Collaborate with security and compliance teams to uphold data governance standards. Implement data masking encryption and audit logging as needed leveraging Unity Catalog and Azure security features to protect sensitive financial data.
  • Platform Innovation: Stay up-to-date with the latest Databricks features and industry best practices. Proactively recommend and implement improvements (such as new performance optimization techniques or cost-saving configurations) to continuously enhance the platforms reliability and efficiency.

Minimum Qualifications

  • Education & Experience: Bachelors degree in Computer Science Information Systems or a related field. 7 years of experience in data engineering data architecture or related roles with a track record of designing and deploying data pipelines and platforms at scale.
  • Databricks & Spark Expertise: Significant hands-on experience with Databricks (preferably Azure Databricks) and the Apache Spark ecosystem. Proficient in building data pipelines using PySpark/Scala and managing data in Delta Lake format.
  • Cloud Platform: Strong experience working with cloud data platforms (Azure preferred or AWS/GCP). Familiarity with Azure data services (such as Azure Data Lake Storage Azure Blob Storage etc.) and managing resources in an Azure environment.
  • SQL and Data Warehousing: Advanced SQL skills with the ability to write and optimize complex queries. Solid understanding of data warehousing concepts and performance tuning for SQL engines.
  • Job Optimization & Performance: Proven ability to optimize ETL jobs and Spark processes for performance and cost efficiency. Experience tuning cluster configurations parallelism and caching to improve job runtimes and resource utilization.
  • Unity Catalog & Security: Demonstrated experience implementing data security and governance measures. Comfortable configuring Unity Catalog or similar data catalog tools to manage schemas tables and fine-grained access controls. Able to ensure compliance with data security standards and manage user/group access to data assets.
  • Leadership Skills: Experience leading and mentoring engineering teams. Excellent project leadership abilities to coordinate multiple projects and priorities. Strong communication skills to effectively collaborate with cross-functional teams and present architectural plans or results to stakeholders.
  • Problem-Solving: Excellent analytical and problem-solving skills with the ability to troubleshoot complex data pipeline issues quickly. Attention to detail in maintaining data quality and reliability across the platform.

Preferred Nice to have

  • Certification: Databricks Certified Data Engineer Professional (highly preferred) or Databricks Certified Data Engineer Associate. Equivalent certifications in cloud data engineering or architecture (e.g. Azure Data Engineer Azure Solutions Architect) are a plus.
  • Financial Services Experience: Prior experience in the financial services industry or other highly regulated industries. Familiarity with financial data types privacy regulations and compliance requirements (e.g. handling PII PCI data) can be beneficial.
  • Advanced Education: Masters degree in Computer Science Data Engineering or related field is a plus.
  • Additional Big Data Technologies: Exposure to related big data and streaming tools such as Apache Kafka/Event Hubs Apache Airflow or Azure Data Factory for orchestration and BI/analytics tools (e.g. Power BI) is advantageous.
  • CI/CD & DevOps : Experience implementing CI/CD pipelines for data projects. Familiarity with Databricks Repos Jenkins or other CI tools for automated testing and deployment of data pipelines.
  • Performance and Cost Management : Proven strategies or projects where you significantly improved performance or reduced cloud costs (e.g. using spot instances job scheduling or improved query design) in a Databricks/Spark environment.

Tools & Technologies

  • Databricks Lakehouse Platform : Databricks Workspace Apache Spark Delta Lake Databricks SQL MLflow (for model tracking).
  • Data Governance: Databricks Unity Catalog for data cataloging and access control; Azure Active Directory integration for identity management.
  • Programming & Data Processing: PySpark and Python for building data pipelines and Spark Jobs; SQL for querying and analytics;
  • Cloud Services (Azure-focused): Azure Databricks Azure Data Lake Storage (ADLS Gen2) Azure Blob Storage Azure Synapse or SQL Database Azure Key Vault (for secrets).
  • DevOps & CI/CD: Azure DevOps (Azure Pipelines) for build/release pipelines Git for version control (GitHub or Azure Repos); experience with Terraform or ARM templates for infrastructure-as-code is a plus.
    • Other Tools: Project and workflow management tools (JIRA or Azure Boards) monitoring tools (Azure Log Analytics Spark UI or Databricks performance monitoring) and collaboration tools for documentation and design (Figma Visio Lucidcharts etc.) with strong leadership and collaboration skills to drive innovative data solutions.

Required Experience:

Staff IC

Vacancy posted 7 days ago
Similar jobs that could be interesting for youBased on the Databricks Data Architect in San Francisco, CA vacancy
  •  ...: San Francisco, United StatesCategory: AI & Data EngineeringData ArchitectPrimary SkillsETL Fundamentals...  ...substantial hands-on expertise with Databricks and modern data platformsKey Responsibilities: 1. Design and architect scalable, high-performance data solutions... 
    Databricks
    Work experience placement

    Brillio

    San Francisco, CA
    16 hours ago
  •  ...responsibilities As a Business Transformation Consultant specializing in Data Platforms, you will plan, design, manage, and implement core...  ...and implementing core components of data platforms, Snowflake, Databricks and Digital Insights Platforms, to deliver significant business... 
    Databricks
    Worldwide

    IBM

    San Francisco, CA
    3 days ago
  • $50 per hour

     ...ServicesStandard Job DescriptionJoin Lockheed Martin’s Data and AI Enablement organization and...  ...the work experience of our team’s architects and engineers, and enable them to grow...  ...technical training.Experience with Databricks and/or medallion data architectures.Experience... 
    Databricks
    Full time
    Temporary work
    Work experience placement
    Casual work
    Flexible hours

    Lockheed Martin

    San Francisco, CA
    4 days ago
  • Job Description: Data Architect, MarTech Data Architect, MarTech our mission is to empower individuals on their journey to better financial...  ...at scale, with hands-on experience in Spark (7+ years), Databricks, and/or Snowflake Skilled in building and supporting MarTech... 
    Databricks
    Work experience placement

    ESR Healthcare

    San Francisco, CA
    3 days ago
  • $220k - $300k

     ...principles to see it in full.How you'll make impactYou'll architect the governed, reusable data, knowledge, and skills layers that power analytics, AI...  ...and global data localisation strategies within Databricks, balancing regulatory requirements with a unified global... 
    Databricks
    Temporary work
    Work at office
    Local area
    3 days per week

    Airwallex

    San Francisco, CA
    3 days ago
  •  ...practice across the entire company.We are seeking a Staff AI Data Transformation Architect to lead as the senior-most individual contributor driving...  ...to-end: integration of data and access models within the Databricks platform, multi-agent workflows, token management, and... 
    Databricks
    Contract work
    Work at office
    Local area
    Remote work
    Flexible hours

    Oura

    San Francisco, CA
    5 days ago
  • $124k - $280k

     ...Description & SummaryThe Opportunity As a Applied AI Health Data Architect you will shape data architecture solutions that support health...  ...platforms, pipelines, and integration patterns across AWS, Azure, Databricks, and Snowflake - Developing data models, warehouse structures... 
    Databricks
    Full time
    H1b

    PwC

    San Francisco, CA
    5 days ago
  • We Are:The beginning of a new Data & AI decade that will reshape work and society has...  ...Ecosystem. You Are:You are a data/senior data architect who designs enterprise data fabric...  ...concepts, cloud data platforms (Snowflake, Databricks, Azure Synapse, BigQuery), and streaming... 
    Databricks
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    San Francisco, CA
    1 day ago
  • $50 per hour

     ...ServicesStandard Job DescriptionJoin Lockheed Martin’s Data & AI Enablement organization and...  ...’s AI transformation. As a lead data architect, you will drive the creation of logical...  ...Ecosystem (EDE). The role operates in a Databricks‑centric environment, integrating source... 
    Databricks
    Full time
    Temporary work
    Work experience placement
    Casual work
    Flexible hours

    Lockheed Martin

    San Francisco, CA
    2 days ago
  • $148.19k - $231.98k

     ...and you are the future of Salesforce.About the TeamWe are a Data & AI Architects team focused on helping customers understand what this new...  ...service experiences.Modern data platforms such as Snowflake, Databricks, BigQuery, or RedShift, and the hyperscaler ecosystems... 
    Databricks
    Full time
    Work experience placement
    Immediate start
    Remote work

    Salesforce

    San Francisco, CA
    2 days ago
  • We Are  The beginning of a new Data & AI decade that will reshape work and society has...  ...Power Ecosystem.  You are A strong Data Architect / Data Engineering Lead who can provide...  ...validation.Guide the team on proper use of Databricks, Delta Lake, Unity Catalog,... 
    Databricks
    Temporary work
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    San Francisco, CA
    1 day ago
  • $159.8k - $188k

     ...Are you ready to make an impact? As a Data Architect at West Monroe, you will lead the design, implementation, and optimization of...  ...development. Extensive experience with modern data platforms: Databricks and/or Snowflake preferred Experience with data modeling... 
    Databricks
    Work at office
    Local area
    Immediate start
    Flexible hours

    West Monroe

    San Francisco, CA
    5 days ago
  • $166k - $210.25k

     ...Applied ML Engineer on the Applied AI team at Databricks, you will use machine learning,...  ...leaders across Databricks.Drive Deployment: Architect, train, and deploy state-of-the-art...  ...Infrastructure: Build robust ML pipelines, data processing layers, model serving components... 
    Databricks
    Local area
    Worldwide

    DataBricks

    San Francisco, CA
    1 day ago
  •  ...Machine Learning Engineer, you will report to the Director of Data Science and Machine Learning in the Center for Data and Insights...  ...such as MLflow, Docker, and cloud services like AWS SageMaker, Databricks MLFlow, etc. Experience with cloud computing infra and tooling... 
    Databricks
    Work at office
    Flexible hours
    3 days per week

    Crunchyroll

    San Francisco, CA
    3 days ago
  • $131.4k - $236k

     ...the intersection of ML architecture, IoT data systems, and product impact, shaping the...  ...intelligence across MaintainX products.Architect end-to-end ML systems—from data ingestion...  ..., TensorFlow, and cloud ML stacks (AWS, Databricks, or similar).Proven experience deploying... 
    Databricks

    MaintainX

    San Francisco, CA
    2 days ago
  • $200.35k - $260.91k

     ....What You’ll DoOwn machine learning problems end to end — from data exploration and modeling through deployment, monitoring, and iteration...  ...distributed ML pipelines on cloud infrastructure (AWS, Databricks, or equivalent).Solid grasp of feature engineering, model evaluation... 
    Databricks
    Full time

    6sense

    San Francisco, CA
    3 days ago
  •  ...enterprise systems is hard. The moment an agent accesses customer data, executes a workflow, or makes changes on behalf of a user,...  ...Morgan Stanley and Wipro. Our earlier investors have also backed Databricks, Clickhouse, MongoDB, Perplexity, Cohere, ScaleAI, Confluent,... 
    Databricks
    Shift work

    Arcade

    San Francisco, CA
    3 days ago
  •  ...electrification for all ⚡ XFRA by SPAN is the first distributed data center using existing infrastructure and underutilized power...  ...electrical grid The Role We are looking for a Network Architect to own the design, deployment, and operation of the network connecting... 
    Full time
    Work at office
    Remote work
    Flexible hours

    span

    San Francisco, CA
    9 days ago
  •  ...Machine Learning Engineer, iOS and Android, data scientist, Developer solutions to...  ...Key Responsibilities: ·        Architect and refine sophisticated ML models and algorithms...  ...learning solutions on platforms such as AWS, Databricks, Azure, Google Cloud and OpenAI.... 
    Databricks
    Remote work
    Visa sponsorship
    Relocation package

    Adidev Technologies Inc

    Daly City, CA
    1 day ago
  •  ...frameworks• At least 6 years of experience in open source big data frameworks (Kafka, Spark streaming, SCALA,solr)Skills • At least...  ...data.• Should have knowledge and understanding of integration architect, ETL tools and features• Should be able to manage IT and business... 
    Permanent employment
    Full time
    H1b

    Sonsoft

    San Francisco, CA
    2 days ago
  • $227.33k - $312.58k

    We’re looking for a Staff ML Data Engineer to join Procore’s AI & Frontier Models organization...  ..., applied ML engineers, and system architects to turn ambiguous research needs into...  ...reproducibility toolingData Platforms: Databricks, Spark, lakehouse architectures, cloud data... 
    Databricks
    Full time
    Work at office
    Local area
    Immediate start
    3 days per week

    Procore Technologies

    San Francisco, CA
    2 days ago
  • $100k - $300k

     ...scale. Cogent's "AI Taskforce" assesses petabytes of enterprise data to remediate these issues before critical breaches occur. To...  ..., Waterloo Unicorn, high-growth companies like Scale AI, Databricks, Stripe, Tesla, Coinbase World class cybersecurity experts from... 
    Databricks

    COGENT SECURITY

    San Francisco, CA
    3 days ago
  •  ...Talent, with advanced capabilities in AI and Data, Industry and Process, and Technology....  ...analytics and AI applications.The Work:Architect, develop, and implement production...  ...Big Data & Production EngineeringLeverage Databricks, Apache Spark, or comparable technologies... 
    Databricks
    Live in
    Work at office
    Local area

    Accenture

    San Francisco, CA
    3 days ago
  • $100k - $300k

     ...scale. Cogent’s "AI Taskforce" assesses petabytes of enterprise data to remediate these issues before critical breaches occur....  ...Mellon, Waterloo Unicorn, high-growth companies like Scale AI, Databricks, Stripe, Tesla, Coinbase World class cybersecurity experts from... 
    Databricks

    COGENT SECURITY

    San Francisco, CA
    3 days ago
  • $190k - $285k

    P-1504The Applied AI team at Databricks sits at the forefront of advancing GenAI-powered products. Over the past years, we’ve launched Databricks...  ...(e.g., Databricks Assistant and AI/BI Genie).Develop novel data collection, fine-tuning, and LLM technologies that achieve... 
    Databricks
    Work at office
    Local area
    Worldwide

    DataBricks

    San Francisco, CA
    5 days ago
  •  ...Head of Data About the Company Innovative provider of online legal services Industry Internet Type Privately Held...  ...in modern cloud data platforms and tools such as Snowflake or Databricks, dbt, Spark, SQL, orchestration, and streaming, as well as experience... 
    Databricks

    Confidential

    San Francisco, CA
    3 days ago
  • $240k - $270k

     ...just adding AI—we’re building the future of how people work with data. Our platform already lets users explore billions of rows of...  ...round was led by Princeville Capital, with new strategic investors Databricks Ventures, ServiceNow Ventures, and Workday Ventures... 
    Databricks
    Full time
    Work at office

    Sigma Computing

    San Francisco, CA
    5 days ago
  •  ...power of change to create value and shared success for our clients, people, shareholders, partners, and communities.You Are:A senior data architecture leader with deep experience in enterprise data platforms, insurance data ecosystems, and policy administration... 
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    San Francisco, CA
    1 day ago
  • $135.1k - $231.6k

     ...Science (IAS) is a global technology and data company that builds verification,...  ...initiatives from inception to global deployment.Architect at Scale: Design and build large-scale...  ...CUDA, NCCL, RDMA) and big data ecosystems (Databricks, Spark, Kubernetes, Kafka, Prometheus).... 
    Databricks
    Full time

    Integral Ad Science

    San Francisco, CA
    4 days ago
  •  ...the listed location(s) are encouraged for this role, candidates in other locations will be considered.At Databricks, our mission is to simplify and democratize data and AI so that every organization can put them to work. Design is central to that mission. We turn powerful... 
    Databricks
    Worldwide

    DataBricks

    San Francisco, CA
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Databricks Data Architect. Be the first to apply!