Azure Databricks & Agentic AI Architect

Role Summary:

We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.

The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.

Key Responsibilities

Agentic Data Engineering Leadership

Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.

Define autonomous workflows using AI Agents for:

Data ingestion

Data mapping

Schema evolution

Data quality remediation

Metadata Enrichment

Pipeline optimization

Root cause analysis

Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.

AI-Driven Data Lifecycle (AI-DLC)

Lead architecture for AI-enabled Data Development Lifecycle across:

Requirement analysis

Data modeling

Pipeline generation

Automated testing

Code review

Documentation

Deployment

Monitoring

Implement AI copilots to accelerate developer productivity.

Enable automated lineage creation and intelligent impact analysis.

Lakehouse & Data Platform Architecture

Design scalable Lakehouse platforms using:

Azure Databricks

Delta Lake

Unity Catalog

ADLS Gen2

Databricks Workflows

Delta Live Tables

Enterprise GenAI Integration

Architect RAG-based solutions using enterprise data assets.

Design agent orchestration frameworks using:

Azure OpenAI

LangGraph

Semantic Kernel

AutoGen

MCP-enabled architectures

Build domain-specific AI agents supporting Data Engineering and Analytics teams.

AI Governance & Responsible AI

Define guardrails for enterprise GenAI adoption.

Implement:

Prompt governance

Observability

Cost monitoring

Auditability

Explainability

Security controls

Establish governance models for autonomous AI agents.

AI-Powered Platform Optimization

Design self-healing data pipelines.

Implement AI-driven:

Incident triage

Failure prediction

Capacity planning

Cost optimization

SLA monitoring

Enable intelligent workload placement and model routing.

Technical Skills

Data Platform

Azure Databricks

Delta Lake

Unity Catalog

Azure Data Factory

AI & Agentic Frameworks

Azure OpenAI

Knowledge Graph

RAG Architecture

LangChain

LangGraph

MCP Protocol

Vector Databases

AI Agent Orchestration

Data Engineering

PySpark

Spark SQL

Python

SQL

ELT/ETL Modernization

DevOps & AI-DLC

Azure DevOps

GitHub Actions

CI/CD

MLOps

LLMOps

Evaluation Frameworks

AI Testing Frameworks

Leadership Expectations

Drive AI-First Data Engineering transformation.

Define enterprise patterns, accelerators, and reusable AI agents.

Mentor architects, data engineers, and AI engineers.

Lead executive conversations on AI adoption, ROI, and transformation roadmaps.

Preferred Certifications

Databricks Certified Data Engineer Professional

Databricks Certified Solution Architect

Microsoft Azure Solution Architect (AZ-305)

Azure Data Engineer (DP-203)

Microsoft Applied Skills Azure OpenAI

Generative AI / Agentic AI Certifications

Success Metrics

30-50% Data Engineering productivity improvement.

Reduction in manual pipeline development effort.

Improved data quality and governance compliance.

Measurable ROI from Agentic AI adoption.

Expansion of reusable AI agents across programs.

Ideal Candidate Profile

A forward-looking architect with 15+ years of Data & Analytics experience, strong Azure Databricks expertise, and hands-on experience building Agentic AI platforms, AI-powered SDLC frameworks, and autonomous data engineering ecosystems. The candidate should be comfortable leading enterprise-scale AI transformation initiatives and engaging CXO stakeholders on AI strategy and business value .

Experience: - 16-20 Years

Location: - Chicago, IL(Hybrid- 3 days/week in office)

Educational Qualifications: -

• Engineering Degree BE/ME/BTech/MTech/BSc/MSc.

Technical certification in multiple technologies is desirable.

Skills: -

Mandatory skills

Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering