Data & AI Automation Engineer (Azure Databricks)

Position: Data & AI Automation Engineer Azure Databricks

Location: Austin, TX

Duration: Long-Term Contract

Key Skills: Azure Databricks, Azure Cloud, Generative AI, LLMs

Job Summary:

We are looking for a highly experienced Data & AI Automation Engineer to design and implement intelligent automation solutions within an Azure Databricks and Azure Cloud environment. The role will focus on applying Generative AI, Large Language Models (LLMs), AI agents, and advanced automation techniques to improve data engineering productivity, software delivery, data quality, governance, and operational efficiency.

The ideal candidate will possess a strong foundation in data engineering and software development, combined with hands-on experience building AI-powered automation and agent-based solutions. This individual will collaborate with data engineers, architects, DevOps teams, and platform stakeholders to identify opportunities for intelligent automation and deliver scalable enterprise solutions.

Key Responsibilities:

• Design, develop, and deploy AI-powered agents using LLMs, rule-based approaches, or hybrid methodologies to automate data engineering processes within Azure Databricks.
• Build intelligent solutions capable of generating, optimizing, reviewing, and refactoring Python, PySpark, and SQL code.
• Develop AI-driven testing and validation agents for data quality, reconciliation, validation, and QA automation.
• Implement automated data governance capabilities covering metadata management, data lineage, compliance monitoring, and policy enforcement.
• Develop autonomous CI/CD capabilities for test generation, deployment validation, release verification, and recovery or rollback processes.
• Build monitoring and diagnostic agents to identify anomalies, detect performance issues, and assist with root-cause analysis.
• Establish enterprise standards for prompt engineering, reusable prompts, agent templates, orchestration patterns, and AI automation workflows.
• Collaborate with data engineering, architecture, DevOps, and platform teams to identify and prioritize high-value automation opportunities.
• Define best practices for the complete AI agent lifecycle, including development, testing, deployment, observability, maintenance, and governance.
• Create technical documentation, architecture diagrams, workflow documentation, operating procedures, and knowledge-transfer materials.
• Research and evaluate emerging Generative AI technologies, LLM platforms, agent frameworks, and automation tools to continuously enhance the data platform.
• Ensure AI-driven automation solutions meet enterprise security, reliability, scalability, and governance requirements.

Required Experience:

• 10+ years of experience in Data Engineering, Software Engineering, AI Engineering, or a related technical discipline.
• Strong hands-on experience with Azure Databricks, Azure Cloud, or comparable modern cloud data platforms such as Snowflake.
• Proven experience developing AI-driven automation solutions using LLMs, Generative AI, prompt engineering, and agent orchestration frameworks.
• Advanced programming experience with Python and strong expertise in SQL development in enterprise production environments.
• Hands-on experience with Azure Databricks notebooks, workflows, jobs, Delta Live Tables (DLT), and related services.
• Experience implementing CI/CD and Continuous Testing (CT) pipelines using GitHub Actions, Azure DevOps, or similar technologies.
• Demonstrated experience developing automated testing frameworks, data validation solutions, reconciliation processes, or data quality automation.
• Experience working in highly regulated, compliance-focused, or risk-sensitive environments is preferred.
• Experience integrating AI capabilities into existing enterprise data engineering and DevOps workflows is highly desirable.

Technical Skills:

• Advanced Python development with strong knowledge of software engineering principles, testing frameworks, documentation, version control, and code quality practices.
• Strong expertise in SQL and PySpark, including performance tuning and optimization.
• Hands-on experience with LLM services, Generative AI APIs, prompt engineering, and AI agent frameworks.
• Experience with technologies/frameworks such as Databricks Agent Framework, LangChain, AutoGen, OpenAI, Azure OpenAI, or comparable platforms.
• Strong knowledge of Azure Databricks, including Delta Lake, Databricks Workflows, Asset Bundles, DLT, notebooks, jobs, and enterprise-scale data processing.
• Experience building workflow automation using orchestration platforms, serverless technologies, APIs, and event-driven architectures.
• Knowledge of AI agent design patterns, tool calling, agent orchestration, context management, evaluation, observability, and lifecycle management.
• Strong understanding of data engineering, data quality, data governance, CI/CD, and automated testing.
• Ability to analyze complex technical and business processes and convert them into scalable, AI-enabled automation solutions.
• Excellent communication skills with the ability to explain AI architectures, automation strategies, and technical designs to both technical and non-technical stakeholders.

Education:

• Bachelor's degree in Computer Science, Software Engineering, Data Science, Information Technology, or a related field.
• Master's degree in a relevant technical discipline is preferred.