DIRECTOR, DATA & AI ENGINEERING -- DATABRICKS LEAD

Job Description

About AbarVa

\n

AbarVa is an AI-native enterprise transformation company that helps organizations move from strategy to execution. Our Nexus platform connects business, technology, data, vendors, programs, risk and value into one shared enterprise context so leaders can prioritize the right initiatives and deliver measurable outcomes. We build the company and the product alongside real enterprise transformations, currently in healthcare.

\n

\n

The role

\n

We are hiring a Director, Data & AI Engineering to lead our US Databricks practice — a player-coach role, not architecture from a distance and not slideware. You will build and lead the engineering team and still own the hardest engineering yourself: architect a modern Databricks environment, set the engineering patterns, review and write code, and personally lead the implementations that matter.

\n

You will operate at Databricks Solutions-Architect / Champion caliber, and you’ll be judged on whether the architecture becomes real — and on whether the engineers around you get better and ship faster because of you.

\n

\n

The work is a mix of greenfield build and forensic recovery. You will stand up ingestion from core operational, transactional and financial systems; assess and migrate existing SQL / ETL / data-warehouse estates — often undocumented — using Databricks Lakebridge to inventory code, score complexity and reconcile parity; and build the governance that makes the result trustworthy enough to act on.

\n

\n

This is an AI-native engineering role in both directions. You build with AI — Claude Code and Codex are part of the normal development loop, and you set the team standard for where to trust them and where to review hard. And you build for AI — everything you model must be consumable by Databricks Genie and Claude on AWS Bedrock, so semantics, metric definitions and column descriptions are architecture decisions, not documentation chores.

\n

\n

What you’ll do

\n

\n
• Lead and build: build and lead the US Databricks engineering team — hire, set standards, review code, mentor and grow engineers — while staying hands-on in the hardest problems yourself.
\n
• Ingestion framework: design and build a metadata-driven ingestion framework so a new source is onboarded by configuration, not another hand-written pipeline — and hand it to the client’s team as a working asset.
\n
• Estate assessment & migration: assess existing SQL, ETL, stored-procedure and warehouse estates with Databricks Lakebridge — inventory code, score complexity and dependencies, accelerate conversion, reconcile legacy-vs-Databricks parity — then turn the assessment into a sized, wave-based build plan.
\n
• Lineage & recovery: trace lineage end to end and reconstruct undocumented business logic from analytics estates being decommissioned; validate against source code, job metadata and owners; record the unknowns.
\n
• Medallion & data products: design scalable Bronze/Silver/Gold architectures and data products — named owner, contract, quality thresholds, lineage, documented consumers — run as a backlog and release cadence, not project phases.
\n
• Governance: design and implement Unity Catalog governance — lineage, security, access control, tenant isolation, data-product patterns.
\n
• Engineering velocity: stand up CI/CD and the toolchain for speed — Databricks Asset Bundles, Git workflows, automated testing, environment promotion, IaC, and observability that catches a broken pipeline before the business does.
\n
• Data for AI: model and certify Gold data for natural-language consumption through Databricks Genie and Claude on Bedrock — clear semantics, defined metrics, described columns, certified tables.
\n
• Partner & scale: partner directly with client CTO, CDAO, architecture, security and engineering leaders, and help build AbarVa’s Data & AI engineering capability as we scale.
\n

\n

\n

What you bring

\n

\n
• Certification (required): an active Databricks certification — Data Engineer Professional, Solutions Architect, or an equivalent current credential. MVP or Champion status is a strong plus.
\n
• Team leadership: proven experience building and leading data engineering teams — hiring, standards, code review, mentoring — while remaining personally hands-on.
\n
• Hands-on depth: 12–18+ years of hands-on data engineering across modern data architecture, modeling and delivery, with exceptional depth in Databricks, Spark, SQL and Python.
\n
• Ingestion engineering: demonstrated experience building metadata-driven ingestion frameworks — Auto Loader / Lakeflow, CDC, schema evolution, data-quality expectations and reconciliation.
\n
• Lakebridge / migration: hands-on Databricks Lakebridge experience (or directly equivalent assessment/transpile/reconcile tooling) to analyze existing data loads, size migrations and prove parity. Familiarity with legacy platforms — Netezza, Teradata, Exadata, SQL Server, SAS — is a strong plus.
\n
• Governance & quality: deep Unity Catalog, medallion and data-product architecture; strong command of reconciliation, data quality, identity, master and reference data.
\n
• Delivery discipline: real CI/CD practice for data and the conviction that shipping weekly beats shipping quarterly; strong cloud architecture experience, preferably AWS.
\n
• AI-native: you use AI coding assistants daily and can set the team’s standard for using them well — including where they’re unreliable.
\n
• Presence: executive presence that holds a room with senior client leaders while keeping technical credibility with engineers.
\n

\n

\n

What makes this role different

\n

We want someone who does both: leads and grows a team, and stays close enough to the engineering to make the architecture real. If your ideal role is architecture in slides, program management, or leading entirely through others, this is not it. If you want to lead the hardest engineering, set patterns other engineers scale, work directly with senior decision-makers, and help build a young technology company — we should talk.

\n

\n

Applicants must be authorized to work in the United States on a permanent basis without employer sponsorship, now or in the future. W2 only — no C2C, no 1099, no third-party staffing firms or agency submissions; direct applicants only.

\n

AbarVa is an equal opportunity employer. We consider all qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, citizenship status, age, disability, veteran status, or any other protected characteristic.