Databricks Architect/Admin

Title : Databricks Architect/Admin
Location : Hartford, CT (Hybrid)
Job Type : Contract-to-Hire (6 Month Contract period)
Compensation : $50 -62 p/h
Industry: Insurance & Financial Services
---
About the Role
Our client is a well-established, highly regulated organization in the insurance and financial services space, operating at enterprise scale and investing heavily in modern data and analytics capabilities. They are seeking a senior Databricks Architect/Admin to act as a hands-on technical consultant for their Databricks platform—focused on platform governance, pipeline architecture, automation, and usage/cost forecasting.
This is not an end-to-end people management role. You’ll partner with an existing Databricks team (including a lead) and provide deep expertise to strengthen platform reliability, scalability, and operational efficiency across a multi-platform data ecosystem. Job Description
In this role, you will serve as the technical authority across the Databricks environment—driving architectural standards, governance, automation, compute strategy, and platform optimization. You’ll collaborate closely with data engineering, analytics, infrastructure/cloud engineering, and security teams to ensure the platform is cost-effective, secure, and built for enterprise performance. A strong Unix/Linux systems administration background is essential, as you’ll engage directly with the underlying compute infrastructure, automation, and diagnostics that support Databricks operations.
What you’ll do:

• Architect and govern the enterprise Databricks environment, including workspace topology, Unity Catalog design, and access control frameworks.
• Define and enforce standards for clusters/compute, including runtime versioning, instance pools, autoscaling policies, and SQL warehouse practices.
• Design scalable and performant data pipeline patterns using Delta Live Tables, Databricks Workflows, and structured streaming.
• Establish Delta Lake standards, including partitioning strategies, Z-ordering, and OPTIMIZE/VACUUM scheduling.
• Lead integration design with upstream ingestion/ETL tools (including Fivetran and Ab Initio) to ensure reliable, governed data delivery.
• Administer and troubleshoot Unix/Linux environments supporting Databricks compute, init scripts, and cluster lifecycle management.
• Build and maintain automation using Bash and Python for monitoring, log aggregation, maintenance tasks, and operational workflows.
• Support compute-level diagnostics and tuning (e.g., EC2/VM-level troubleshooting) in partnership with cloud/infrastructure teams.
• Own DBU consumption tracking and reporting, proactively identifying opportunities to optimize job performance and platform costs.
• Implement cost attribution models to enable chargeback/showback reporting across teams or business units.
• Partner on capacity planning, contract utilization forecasting, and long-range commitment management.
• Design and implement governance practices within Unity Catalog (e.g., lineage, tagging, auditing) and support compliance requirements.
• Collaborate with cybersecurity on platform hardening (e.g., secrets management, network isolation, encryption, and auditing).
• Design automation frameworks for platform operations (job scheduling, alerting, self-healing workflows, and reliability improvements).
• Support ML/AI enablement on Databricks (e.g., MLflow, AutoML, Model Serving) and evaluate emerging capabilities, including GenAI/LLM workflows.
• Serve as an escalation point for Databricks platform issues, contributing to incident/change management via ServiceNow and delivery tracking via Jira.
• Maintain platform documentation including runbooks, architectural diagrams, standards, and onboarding materials.

Qualifications
Required Qualifications

• 7+ years in data engineering, data platform, or related roles, including 4+ years hands-on Databricks implementation experience.
• Deep expertise in Databricks platform capabilities, including: Unity Catalog, Delta Lake, Databricks Workflows, Delta Live Tables, and SQL Warehouses.
• Strong Unix/Linux proficiency (shell scripting, process management, file systems, permissions, cron scheduling, environment configuration).
• Proficiency in Python and PySpark for distributed processing, pipeline development, and platform automation.
• Experience with cloud infrastructure (AWS, Azure, or GCP) including compute, storage, networking, and IAM/security concepts.
• Proven ability to design platforms for scale, cost efficiency, reliability, and operational excellence in an enterprise environment.
• Experience building automation frameworks for platform operations (orchestration, monitoring, alerting, and self-healing pipelines).
• Familiarity with Databricks ML/AI tooling (MLflow, AutoML, Model Serving); exposure to generative AI/LLM-integrated workflows is a plus.
• Experience with Oracle environments (SQL development, schema concepts, extraction/integration patterns).
• Strong experience with Git-based version control and CI/CD practices (branching, PR workflows, repository management, pipeline integration).
• Working knowledge of ITSM and delivery tools such as ServiceNow and Jira.
• Strong communication skills—able to translate complex architecture into clear guidance for technical and non-technical stakeholders.

Preferred Qualifications

• Hands-on experience with MLflow experiment tracking, model registry, and deployment patterns in Databricks.
• Exposure to GenAI frameworks (e.g., LangChain, LlamaIndex) or building RAG/LLM-integrated data pipelines.
• Experience with enterprise-scale orchestration tools (e.g., Apache Airflow, Databricks Workflows, or similar).
• Experience integrating Databricks with Fivetran and/or Ab Initio (Ab Initio development/admin experience is a strong plus).
• Familiarity with enterprise catalog/governance tools and frameworks (e.g., Collibra, Alation, or advanced Unity Catalog features).
• Experience supporting data platforms in regulated industries with audit/compliance requirements (insurance or financial services preferred).
• Working knowledge of Infrastructure-as-Code tooling (e.g., Terraform, Ansible) for provisioning and configuration management.
• Background in disaster recovery and resiliency planning for cloud-hosted data platforms.

Benefits
Dahl Consulting is proud to offer a comprehensive benefits package to eligible employees that will allow you to choose the best coverage to meet your family’s needs. For details, please review the DAHL Benefits Summary:  How to Apply
Take the first step on your new career path!  To submit yourself for consideration for this role, simply click the apply button and complete our mobile-friendly online application. Once we’ve reviewed your application details, a recruiter will reach out to you with next steps! Equal Opportunity Statement
As an equal opportunity employer, Dahl Consulting welcomes candidates of all backgrounds and experiences to apply. If this position sounds like the right opportunity for you, we encourage you to take the next step and connect with us. We look forward to meeting you!

#LI-CF1
#LI-Hybrid