Azure Databricks & Agentic AI Architect

Role : Azure Databricks & Agentic AI Architect

Location: Chicago, IL(Hybrid- 3 days/week in office)

Contract role

Looking for 16-20 Years minimum experience

Make sure to apply with all the requested information, as laid out in the job overview below.

Mandatory skills

Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering

Role Summary:

We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.

The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.

Key Responsibilities

Agentic Data Engineering Leadership

· Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.

· Define autonomous workflows using AI Agents for:

· Data ingestion

· Data mapping

· Schema evolution

· Data quality remediation

· Metadata Enrichment

· Pipeline optimization

· Root cause analysis

· Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.

AI-Driven Data Lifecycle (AI-DLC)

· Lead architecture for AI-enabled Data Development Lifecycle across:

· Requirement analysis

· Data modeling

· Pipeline generation

· Automated testing

· Code review

· Documentation

· Deployment

· Monitoring

· Implement AI copilots to accelerate developer productivity.

· Enable automated lineage creation and intelligent impact analysis.

Lakehouse & Data Platform Architecture

· Design scalable Lakehouse platforms using:

· Azure Databricks

· Delta Lake

· Unity Catalog

· ADLS Gen2

· Databricks Workflows

· Delta Live Tables

Enterprise GenAI Integration

· Architect RAG-based solutions using enterprise data assets.

· Design agent orchestration frameworks using:

· Azure OpenAI

· LangGraph

· Semantic Kernel

· AutoGen

· MCP-enabled architectures

· Build domain-specific AI agents supporting Data Engineering and Analytics teams.

AI Governance & Responsible AI

· Define guardrails for enterprise GenAI adoption.

· Implement:

· Prompt governance

· Observability

· Cost monitoring

· Auditability

· Explainability

· Security controls

· Establish governance models for autonomous AI agents.

AI-Powered Platform Optimization

· Design self-healing data pipelines.

· Implement AI-driven:

· Incident triage

· Failure prediction

· Capacity planning

· Cost optimization

· SLA monitoring

· Enable intelligent workload placement and model routing.

Technical Skills

· Data Platform

· Azure Databricks

· Delta Lake

· Unity Catalog

· Azure Data Factory

AI & Agentic Frameworks

· Azure OpenAI

· Knowledge Graph

· RAG Architecture

· LangChain

· LangGraph

· MCP Protocol

· Vector Databases

· AI Agent Orchestration

Data Engineering

· PySpark

· Spark SQL

· Python

· SQL

· ELT/ETL Modernization

DevOps & AI-DLC

· Azure DevOps

· GitHub Actions

· CI/CD

· MLOps

· LLMOps

· Evaluation Frameworks

· AI Testing Frameworks

Leadership Expectations

· Drive AI-First Data Engineering transformation.

· Define enterprise patterns, accelerators, and reusable AI agents.

· Mentor architects, data engineers, and AI engineers. xhyhwjd

· Lead executive conversations on AI adoption, ROI, and transformation roadmaps.