Databricks Architect and Engineering Lead (United States)

About AbarVa

AbarVa is building Nexus , an AI-native enterprise execution platform that helps organizations turn strategic decisions into governed action, measurable outcomes, and sustained business value.

We are building Nexus alongside complex enterprise Data & AI transformations, particularly in regulated industries.

The Opportunity

We are looking for an experienced Databricks Architect and Engineering Lead to lead the architecture and engineering execution of large-scale data modernization and migration programs.

This is a hands-on architecture and engineering leadership role . We are looking for someone who has spent significant time personally building modern data platforms, has deep Databricks expertise, and can lead engineering teams through complex enterprise migrations.

This is not a strategy-only or architecture-only position. The successful candidate must be able to design, build, review code, troubleshoot complex engineering issues, establish engineering standards, and lead teams through production delivery.

Required Experience

• 10+ years of hands-on data engineering experience
• 5+ years of hands-on Databricks experience
• Current Databricks certification is mandatory
• Significant experience as a Databricks Architect, Data Engineering Lead, Principal Data Engineer or equivalent
• Deep hands-on expertise with Databricks, Apache Spark, PySpark and SQL
• Strong experience with Delta Lake, Unity Catalog, Databricks Workflows and modern Databricks ingestion and transformation patterns
• Demonstrated experience designing and implementing enterprise-scale Bronze, Silver and Gold architectures
• Proven experience leading large-scale migrations from legacy data platforms to Databricks
• Experience assessing existing data workloads, ETL pipelines, data models, reports, KPIs and downstream dependencies
• Strong experience designing metadata-driven ingestion and transformation frameworks
• Strong knowledge of data quality, reconciliation, lineage, observability, security and governance
• Strong cloud platform experience; AWS experience is highly desirable
• Proven ability to lead and mentor teams of data engineers while remaining technically hands-on
• Strong communication skills and experience working directly with senior client and technology stakeholders
  

What You’ll Do

• Own the architecture and engineering execution of large-scale Databricks modernization and migration programs
• Assess complex legacy data environments and develop practical modernization strategies
• Determine which workloads should be migrated, refactored, redesigned, consolidated or retired
• Design scalable enterprise Bronze / Silver / Gold architectures
• Lead migration of legacy data warehouses, ETL platforms, analytics environments and reporting ecosystems to Databricks
• Architect reusable, metadata-driven ingestion, transformation and orchestration frameworks
• Lead enterprise data modeling and development of reusable data products
• Establish engineering standards for performance, security, governance, lineage, observability and data quality
• Lead reconciliation and parallel-run activities to prove Databricks outputs against trusted legacy platforms
• Define migration waves, cutover plans and production stabilization approaches
• Conduct architecture, design and code reviews
• Troubleshoot complex engineering, scalability and performance issues
• Lead and mentor distributed engineering teams
• Partner directly with business and technology leaders to translate business outcomes into scalable data solutions
• Introduce modern AI-assisted engineering approaches to improve development productivity while maintaining appropriate governance and human review
• Help develop reusable AbarVa engineering methods, accelerators and intellectual property
  

Large-Scale Migration Experience

Candidates should have personally led significant migrations involving platforms such as:

• Netezza
• Teradata
• Oracle / Exadata
• SQL Server
• Hadoop ecosystems
• Traditional enterprise data warehouses
• Legacy ETL and integration platforms
• Externally hosted analytics platforms
• Enterprise reporting and BI environments

Candidates should be able to clearly explain how they:

• Assessed the existing data landscape
• Identified workload and reporting dependencies
• Rationalized legacy pipelines and reports
• Designed the target Databricks architecture
• Sequenced migration waves
• Reconciled legacy and Databricks results
• Managed parallel operations
• Executed production cutover
• Stabilized and optimized the new environment
  

Architecture & Leadership Expectations

This role requires someone who can lead from the front technically .

You will be expected to:

• Own critical architecture decisions
• Lead engineering teams through complex delivery programs
• Establish architecture and engineering standards
• Mentor architects and data engineers
• Conduct detailed design and code reviews
• Make difficult architecture and engineering trade-offs
• Work directly with senior client technology and business leaders
• Translate complex technical decisions into clear business implications
• Take accountability from architecture through successful production implementation

We are not looking for someone who only manages engineers or produces architecture diagrams. You must be comfortable getting deeply hands-on when required.

Industry Experience

Experience in healthcare, financial services, insurance or another highly regulated industry is a significant advantage .

Healthcare experience is particularly valuable, including exposure to:

• Health-plan / payer data
• Claims
• Membership, enrollment and eligibility
• Medicare and Medicaid
• Clinical and provider data
• Epic / Epic Clarity
• Healthcare financial and operational analytics
• PHI and other regulated data environments

Deep healthcare expertise is not mandatory, but experience designing platforms where security, privacy, governance, auditability and data quality are critical is strongly preferred.

Work Authorization

Candidates must be legally authorized to work in the United States on a W-2 basis without current or future visa sponsorship .

AbarVa is not able to provide employment visa sponsorship for this role.

What Makes This Role Different

You will not simply create an architecture and hand it to another team.

You will be expected to lead discovery, architecture, engineering, migration, reconciliation, cutover and production delivery .

We are looking for a Databricks architect, hands-on engineer and engineering leader who has successfully delivered large-scale modernization programs and can build and lead high-performing teams.