Databricks Architect / Admin (W2 only C2H)

Position Title: Databricks Architect / Admin (W2 only C2H)

Employment Type:

Location: Hartford, CT (Hybrid – 3 Days Onsite / 2 Days Remote)

Schedule: Monday–Friday, 8:00 AM – 5:00 PM

Duration: 6-month Contract-to-Hire

Team Size: 10–12 members (Onshore + Offshore)

Work Environment: Agile, daily stand-up at 9:00 AM, ServiceNow workflow management

Interview Process: 1–2 rounds (possible third round), panel with hiring manager and team members; technical + behavioral focus, no formal assessment.

Travel: Minimal (commute only)

A senior Databricks architect with deep platform administration experience who can function as a strategic advisor rather than a day-to-day operator. The ideal candidate combines strong Databricks architecture knowledge with AWS infrastructure expertise, cost optimization skills, Linux administration, and enterprise-scale data platform experience.

Position Overview

The Databricks Architect/Admin is a senior individual contributor responsible for architecting, governing, optimizing, and supporting the enterprise Databricks platform. This role is primarily consultative with selective hands‑on responsibilities and serves as the subject matter expert for platform architecture, administration, governance, cost optimization, and enterprise integrations.

The environment is already established; the selected candidate will provide strategic guidance, optimization recommendations, and ongoing support rather than building from scratch. This position is a backfill for a retiring resource and requires deep expertise to guide engineering teams and platform direction.

The role requires a strong combination of:

• Advanced administration knowledge
• Cloud platform experience (AWS)
• Cost management and optimization skills
• Data engineering and integration experience
• Strategic advisory capability

Key Responsibilities

• Architect and govern enterprise Databricks environments including:
  • Unity Catalog structures
  • Access control frameworks
• Define and enforce:
  • Cluster standards
  • Runtime versions
  • Instance pool strategies
• Design scalable and performant pipelines using:
  • Databricks Workflows
  • Structured Streaming
  • Z‑ordering
  • OPTIMIZE/VACUUM strategies
• Lead integration architecture with upstream platforms:
  • Fivetran
  • Ab Initio
• Serve as the primary Databricks consultant across teams
• Provide guidance on platform best practices
• Act as the escalation point for Databricks platform issues
• Mentor teams on platform usage and optimization
• Recommend new Databricks capabilities and roadmap enhancements

Cost Management & Optimization

A major focus area during the first 3–6 months.

Responsibilities include:

• Own DBU consumption reporting and tracking
• Identify optimization opportunities across:
  • Jobs
  • SQL Warehouses
• Detect inefficiencies such as:
  • Excessive DBU consumption
  • Misconfigured workloads
• Support:
  • Capacity planning
  • Contract forecasting
• Collaborate on monthly cost analyses
• Administer and troubleshoot Linux environments supporting Databricks
• Manage:
  • Init scripts
  • Shell scripting
  • Build Bash and Python automation solutions
• Support:
  • Monitoring
  • Maintenance workflows
• Manage:
  • File systems
  • Permissions
  • Data movement operations
• Perform EC2/VM diagnostics with infrastructure teams

Governance, Security & Compliance

• Design governance frameworks in Unity Catalog
• Implement:
  • Partner with cybersecurity teams
• Support:
  • Secrets management
  • Network isolation
  • Encryption
• Maintain audit documentation and compliance artifacts
• Scheduling
• Alerting
• Leverage:
  • AutoML
  • MLflow
  • Model Serving
• Integrate:
  • Databricks Assistant
• Support scalable feature engineering and ML deployment pipelines
• AI capabilities
• Generative AI tools
• LLM-integrated workflows

Required Qualifications

• 7+ years in data engineering or platform engineering
• Minimum 4+ years of hands‑on Databricks implementation experience
• Deep expertise in:
  • Unity Catalog
  • Databricks Workflows
  • SQL Warehouses
  • Shell scripting
  • Process management
  • File systems
  • Cron
  • Python and PySpark expertise
• AWS or cloud platform experience:
  • EC2
  • Storage
  • Networking
  • IAM/security
• SQL expertise for reporting and analysis
• Experience designing scalable and cost-efficient enterprise platforms
• Experience with automation frameworks
• Familiarity with AI/ML ecosystem
• Git and CI/CD knowledge
• ServiceNow and Jira exposure

Preferred Qualifications

• MLflow deployment and model registry experience
• Generative AI tools:
  • LangChain
• Workflow orchestration:
  • Databricks Workflows
• Ab Initio administration or development
• Data governance platforms:
  • Collibra
  • Alation
• Experience in regulated industries:
  • Insurance
• Disaster recovery and resiliency planning

Core Technical Skills

Databricks Ecosystem

• Databricks
• Unity Catalog
• Databricks Workflows
• SQL Warehouses
• MLflow
• AutoML
• AWS
• EC2
• S3
• Networking
• Private Links
• Firewalls

Programming & Automation

• Python
• PySpark
• Bash/Shell scripting
• SQL
• Git

Integration Technologies

• Fivetran
• Ab Initio
• Oracle

Platforms & Operations

• ServiceNow
• Jira

Primary focus areas:

• Cost optimization and Databricks usage tracking
• Stabilization of Databricks onboarding application (currently MVP)
• Identification of billing discrepancies and inefficiencies
• Strategic capacity planning and vendor support
• Providing platform guidance and consultation to teams

Hiring Notes

• Hartford candidates strongly preferred
• Exceptional candidates from alternate locations may be considered
• Budget is below current market expectations
• Hiring manager noted limited flexibility but open to strong profiles
• For information on benefits, equal opportunity employment, and location-specific applicant notices, click here

At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position's starting pay is: $45.00/daily.