Databricks Data Architect
$140 - $150 per hourNewVison
The Databricks Data Architect is a senior technical lead responsible for building and optimizing a robust data platform in a financial services environment. You will lead a team of 10 data engineers and own the end-to-end architecture and implementation of the Databricks Lakehouse platform. You will collaborate closely with application development and analytics teams to design scalable data solutions that drive business insights. This position demands deep expertise in Databricks (Azure) hands-on experience with PySpark and Delta Lake and strong leadership to ensure best practices in data engineering performance tuning and governance.
Key Must have skills
- Lead Data Engineering Team: Lead mentor and manage a team of 10 data engineers providing technical guidance code reviews and career development to foster a high-performing team.
- Databricks Platform Ownership: Own the Databricks platform architecture and implementation ensuring the environment is secure scalable and optimized for the organizations data processing needs. Design and oversee the Lakehouse architecture leveraging Delta Lake and Apache Spark.
- Unity Catalog Implementation: Implement and manage Databricks Unity Catalog for unified data governance. Ensure fine-grained access controls and data lineage tracking are in place to secure sensitive financial data and comply with industry regulations.
- Cluster Provisioning & Policies: Provision and administer Databricks clusters (in Azure) including configuring cluster sizes auto-scaling and auto-termination settings. Set up and enforce cluster policies to standardize configurations optimize resource usage and control costs across different teams and projects.
- Databricks SQL Optimization: Collaborate with analytics teams to develop and optimize Databricks SQL queries and dashboards. Tune SQL workloads and caching strategies for faster performance and ensure efficient use of the query engine.
- Performance Tuning: Lead performance tuning initiatives for Spark jobs and ETL pipelines. Profile data processing code (PySpark/Scala) to identify bottlenecks and refactor for improved throughput and lower latency. Implement best practices for incremental data processing with Delta Lake and ensure compute cost efficiency (e.g. by optimizing cluster utilization and job scheduling).
- Data Solutions Collaboration: Work closely with application developers data analysts and data scientists to understand requirements and translate them into robust data pipelines and solutions. Ensure that data architectures support analytics reporting and machine learning use cases effectively.
- DevOps Integration: Integrate Databricks workflows into the CI/CD pipeline using Azure DevOps and Git. Develop automated deployment processes for notebooks jobs and clusters (infrastructure-as-code) to promote consistent releases. Manage source control for Databricks code (using Git integration) and collaborate with DevOps engineers to implement continuous integration and delivery for data projects.
- Data Governance & Security: Collaborate with security and compliance teams to uphold data governance standards. Implement data masking encryption and audit logging as needed leveraging Unity Catalog and Azure security features to protect sensitive financial data.
- Platform Innovation: Stay up-to-date with the latest Databricks features and industry best practices. Proactively recommend and implement improvements (such as new performance optimization techniques or cost-saving configurations) to continuously enhance the platforms reliability and efficiency.
Minimum Qualifications
- Education & Experience: Bachelors degree in Computer Science Information Systems or a related field. 7 years of experience in data engineering data architecture or related roles with a track record of designing and deploying data pipelines and platforms at scale.
- Databricks & Spark Expertise: Significant hands-on experience with Databricks (preferably Azure Databricks) and the Apache Spark ecosystem. Proficient in building data pipelines using PySpark/Scala and managing data in Delta Lake format.
- Cloud Platform: Strong experience working with cloud data platforms (Azure preferred or AWS/GCP). Familiarity with Azure data services (such as Azure Data Lake Storage Azure Blob Storage etc.) and managing resources in an Azure environment.
- SQL and Data Warehousing: Advanced SQL skills with the ability to write and optimize complex queries. Solid understanding of data warehousing concepts and performance tuning for SQL engines.
- Job Optimization & Performance: Proven ability to optimize ETL jobs and Spark processes for performance and cost efficiency. Experience tuning cluster configurations parallelism and caching to improve job runtimes and resource utilization.
- Unity Catalog & Security: Demonstrated experience implementing data security and governance measures. Comfortable configuring Unity Catalog or similar data catalog tools to manage schemas tables and fine-grained access controls. Able to ensure compliance with data security standards and manage user/group access to data assets.
- Leadership Skills: Experience leading and mentoring engineering teams. Excellent project leadership abilities to coordinate multiple projects and priorities. Strong communication skills to effectively collaborate with cross-functional teams and present architectural plans or results to stakeholders.
- Problem-Solving: Excellent analytical and problem-solving skills with the ability to troubleshoot complex data pipeline issues quickly. Attention to detail in maintaining data quality and reliability across the platform.
Preferred Nice to have
- Certification: Databricks Certified Data Engineer Professional (highly preferred) or Databricks Certified Data Engineer Associate. Equivalent certifications in cloud data engineering or architecture (e.g. Azure Data Engineer Azure Solutions Architect) are a plus.
- Financial Services Experience: Prior experience in the financial services industry or other highly regulated industries. Familiarity with financial data types privacy regulations and compliance requirements (e.g. handling PII PCI data) can be beneficial.
- Advanced Education: Masters degree in Computer Science Data Engineering or related field is a plus.
- Additional Big Data Technologies: Exposure to related big data and streaming tools such as Apache Kafka/Event Hubs Apache Airflow or Azure Data Factory for orchestration and BI/analytics tools (e.g. Power BI) is advantageous.
- CI/CD & DevOps : Experience implementing CI/CD pipelines for data projects. Familiarity with Databricks Repos Jenkins or other CI tools for automated testing and deployment of data pipelines.
- Performance and Cost Management : Proven strategies or projects where you significantly improved performance or reduced cloud costs (e.g. using spot instances job scheduling or improved query design) in a Databricks/Spark environment.
Tools & Technologies
- Databricks Lakehouse Platform : Databricks Workspace Apache Spark Delta Lake Databricks SQL MLflow (for model tracking).
- Data Governance: Databricks Unity Catalog for data cataloging and access control; Azure Active Directory integration for identity management.
- Programming & Data Processing: PySpark and Python for building data pipelines and Spark Jobs; SQL for querying and analytics;
- Cloud Services (Azure-focused): Azure Databricks Azure Data Lake Storage (ADLS Gen2) Azure Blob Storage Azure Synapse or SQL Database Azure Key Vault (for secrets).
- DevOps & CI/CD: Azure DevOps (Azure Pipelines) for build/release pipelines Git for version control (GitHub or Azure Repos); experience with Terraform or ARM templates for infrastructure-as-code is a plus.
- Other Tools: Project and workflow management tools (JIRA or Azure Boards) monitoring tools (Azure Log Analytics Spark UI or Databricks performance monitoring) and collaboration tools for documentation and design (Figma Visio Lucidcharts etc.) with strong leadership and collaboration skills to drive innovative data solutions.
Required Experience:
Staff IC
- ...: San Francisco, United StatesCategory: AI & Data EngineeringData ArchitectPrimary SkillsETL Fundamentals... ...substantial hands-on expertise with Databricks and modern data platformsKey Responsibilities: 1. Design and architect scalable, high-performance data solutions...DatabricksWork experience placement
- ...responsibilities As a Business Transformation Consultant specializing in Data Platforms, you will plan, design, manage, and implement core... ...and implementing core components of data platforms, Snowflake, Databricks and Digital Insights Platforms, to deliver significant business...DatabricksWorldwide
$50 per hour
...ServicesStandard Job DescriptionJoin Lockheed Martin’s Data and AI Enablement organization and... ...the work experience of our team’s architects and engineers, and enable them to grow... ...technical training.Experience with Databricks and/or medallion data architectures.Experience...DatabricksFull timeTemporary workWork experience placementCasual workFlexible hours- Job Description: Data Architect, MarTech Data Architect, MarTech our mission is to empower individuals on their journey to better financial... ...at scale, with hands-on experience in Spark (7+ years), Databricks, and/or Snowflake Skilled in building and supporting MarTech...DatabricksWork experience placement
$220k - $300k
...principles to see it in full.How you'll make impactYou'll architect the governed, reusable data, knowledge, and skills layers that power analytics, AI... ...and global data localisation strategies within Databricks, balancing regulatory requirements with a unified global...DatabricksTemporary workWork at officeLocal area3 days per week- ...practice across the entire company.We are seeking a Staff AI Data Transformation Architect to lead as the senior-most individual contributor driving... ...to-end: integration of data and access models within the Databricks platform, multi-agent workflows, token management, and...DatabricksContract workWork at officeLocal areaRemote workFlexible hours
$124k - $280k
...Description & SummaryThe Opportunity As a Applied AI Health Data Architect you will shape data architecture solutions that support health... ...platforms, pipelines, and integration patterns across AWS, Azure, Databricks, and Snowflake - Developing data models, warehouse structures...DatabricksFull timeH1b- We Are:The beginning of a new Data & AI decade that will reshape work and society has... ...Ecosystem. You Are:You are a data/senior data architect who designs enterprise data fabric... ...concepts, cloud data platforms (Snowflake, Databricks, Azure Synapse, BigQuery), and streaming...DatabricksWork experience placementLive inWork at officeLocal area
$50 per hour
...ServicesStandard Job DescriptionJoin Lockheed Martin’s Data & AI Enablement organization and... ...’s AI transformation. As a lead data architect, you will drive the creation of logical... ...Ecosystem (EDE). The role operates in a Databricks‑centric environment, integrating source...DatabricksFull timeTemporary workWork experience placementCasual workFlexible hours$148.19k - $231.98k
...and you are the future of Salesforce.About the TeamWe are a Data & AI Architects team focused on helping customers understand what this new... ...service experiences.Modern data platforms such as Snowflake, Databricks, BigQuery, or RedShift, and the hyperscaler ecosystems...DatabricksFull timeWork experience placementImmediate startRemote work- We Are The beginning of a new Data & AI decade that will reshape work and society has... ...Power Ecosystem. You are A strong Data Architect / Data Engineering Lead who can provide... ...validation.Guide the team on proper use of Databricks, Delta Lake, Unity Catalog,...DatabricksTemporary workWork experience placementLive inWork at officeLocal area
$159.8k - $188k
...Are you ready to make an impact? As a Data Architect at West Monroe, you will lead the design, implementation, and optimization of... ...development. Extensive experience with modern data platforms: Databricks and/or Snowflake preferred Experience with data modeling...DatabricksWork at officeLocal areaImmediate startFlexible hours$166k - $210.25k
...Applied ML Engineer on the Applied AI team at Databricks, you will use machine learning,... ...leaders across Databricks.Drive Deployment: Architect, train, and deploy state-of-the-art... ...Infrastructure: Build robust ML pipelines, data processing layers, model serving components...DatabricksLocal areaWorldwide- ...Machine Learning Engineer, you will report to the Director of Data Science and Machine Learning in the Center for Data and Insights... ...such as MLflow, Docker, and cloud services like AWS SageMaker, Databricks MLFlow, etc. Experience with cloud computing infra and tooling...DatabricksWork at officeFlexible hours3 days per week
$131.4k - $236k
...the intersection of ML architecture, IoT data systems, and product impact, shaping the... ...intelligence across MaintainX products.Architect end-to-end ML systems—from data ingestion... ..., TensorFlow, and cloud ML stacks (AWS, Databricks, or similar).Proven experience deploying...Databricks$200.35k - $260.91k
....What You’ll DoOwn machine learning problems end to end — from data exploration and modeling through deployment, monitoring, and iteration... ...distributed ML pipelines on cloud infrastructure (AWS, Databricks, or equivalent).Solid grasp of feature engineering, model evaluation...DatabricksFull time- ...enterprise systems is hard. The moment an agent accesses customer data, executes a workflow, or makes changes on behalf of a user,... ...Morgan Stanley and Wipro. Our earlier investors have also backed Databricks, Clickhouse, MongoDB, Perplexity, Cohere, ScaleAI, Confluent,...DatabricksShift work
- ...electrification for all ⚡ XFRA by SPAN is the first distributed data center using existing infrastructure and underutilized power... ...electrical grid The Role We are looking for a Network Architect to own the design, deployment, and operation of the network connecting...Full timeWork at officeRemote workFlexible hours
- ...Machine Learning Engineer, iOS and Android, data scientist, Developer solutions to... ...Key Responsibilities: · Architect and refine sophisticated ML models and algorithms... ...learning solutions on platforms such as AWS, Databricks, Azure, Google Cloud and OpenAI....DatabricksRemote workVisa sponsorshipRelocation package
- ...frameworks• At least 6 years of experience in open source big data frameworks (Kafka, Spark streaming, SCALA,solr)Skills • At least... ...data.• Should have knowledge and understanding of integration architect, ETL tools and features• Should be able to manage IT and business...Permanent employmentFull timeH1b
$227.33k - $312.58k
We’re looking for a Staff ML Data Engineer to join Procore’s AI & Frontier Models organization... ..., applied ML engineers, and system architects to turn ambiguous research needs into... ...reproducibility toolingData Platforms: Databricks, Spark, lakehouse architectures, cloud data...DatabricksFull timeWork at officeLocal areaImmediate start3 days per week$100k - $300k
...scale. Cogent's "AI Taskforce" assesses petabytes of enterprise data to remediate these issues before critical breaches occur. To... ..., Waterloo Unicorn, high-growth companies like Scale AI, Databricks, Stripe, Tesla, Coinbase World class cybersecurity experts from...Databricks- ...Talent, with advanced capabilities in AI and Data, Industry and Process, and Technology.... ...analytics and AI applications.The Work:Architect, develop, and implement production... ...Big Data & Production EngineeringLeverage Databricks, Apache Spark, or comparable technologies...DatabricksLive inWork at officeLocal area
$100k - $300k
...scale. Cogent’s "AI Taskforce" assesses petabytes of enterprise data to remediate these issues before critical breaches occur.... ...Mellon, Waterloo Unicorn, high-growth companies like Scale AI, Databricks, Stripe, Tesla, Coinbase World class cybersecurity experts from...Databricks$190k - $285k
P-1504The Applied AI team at Databricks sits at the forefront of advancing GenAI-powered products. Over the past years, we’ve launched Databricks... ...(e.g., Databricks Assistant and AI/BI Genie).Develop novel data collection, fine-tuning, and LLM technologies that achieve...DatabricksWork at officeLocal areaWorldwide- ...Head of Data About the Company Innovative provider of online legal services Industry Internet Type Privately Held... ...in modern cloud data platforms and tools such as Snowflake or Databricks, dbt, Spark, SQL, orchestration, and streaming, as well as experience...Databricks
$240k - $270k
...just adding AI—we’re building the future of how people work with data. Our platform already lets users explore billions of rows of... ...round was led by Princeville Capital, with new strategic investors Databricks Ventures, ServiceNow Ventures, and Workday Ventures...DatabricksFull timeWork at office- ...power of change to create value and shared success for our clients, people, shareholders, partners, and communities.You Are:A senior data architecture leader with deep experience in enterprise data platforms, insurance data ecosystems, and policy administration...Work experience placementLive inWork at officeLocal area
$135.1k - $231.6k
...Science (IAS) is a global technology and data company that builds verification,... ...initiatives from inception to global deployment.Architect at Scale: Design and build large-scale... ...CUDA, NCCL, RDMA) and big data ecosystems (Databricks, Spark, Kubernetes, Kafka, Prometheus)....DatabricksFull time- ...the listed location(s) are encouraged for this role, candidates in other locations will be considered.At Databricks, our mission is to simplify and democratize data and AI so that every organization can put them to work. Design is central to that mission. We turn powerful...DatabricksWorldwide
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Databricks Data Architect. Be the first to apply!
- sr data modeler San Francisco, CA
- data center architect San Francisco, CA
- database designer San Francisco, CA
- senior data architect San Francisco, CA
- senior data modeler San Francisco, CA
- data integration architect San Francisco, CA
- data architect San Francisco, CA
- databricks San Francisco, CA
- data intern San Francisco, CA
- data collection researcher San Francisco, CA




