Agent Manager

Job Description

Job Description

Description:

On April 28, 2021, Avelo took flight as America’s first new airline in nearly 15 years – ushering in a new era of affordable, convenient, and reliable air travel. Founded and led by airline industry veteran, Andrew Levy, along with a team of world-class airline executives, we endeavored to build a different and better kind of airline with one mission in mind: “To inspire travel” and we’ve done so with industry-leading reliability and a caring Soul of Service. If you are looking for the opportunity to join a new and exciting airline that offers the chance to make your mark on aviation history, keep reading!

Purpose: The Agent Manager is a hands-on technical role responsible for designing, building, deploying, and governing the AI agents that power our data and operational workflows. This is a new position on a team that is actively expanding its use of agentic AI — you will help define what good looks like. Microsoft Fabric Data Agents and Fabric Operations Agents are the core delivery focus, orchestrated through Azure AI Foundry and complemented by Claude-powered agents built via the Anthropic API, Claude Cowork, and Microsoft Copilot. You will write code, craft and refine prompts, wire agents to internal tools and data surfaces through MCP and REST APIs, and own the full agent lifecycle from prototype to production to retirement. Equally important: you will establish the governance frameworks, evaluation standards, and safety protocols that allow the organization to move quickly without losing control of what its agents do.

What Makes This Role Different: Most “AI” roles today involve using existing tools. This role involves building the tools others will use. You will be the person who decides how an agent is structured, what it can and cannot access, how it fails safely, and when it needs a human in the loop. You will ship working agents — not slide decks about agents. If you are energized by moving fast at the frontier of what enterprise AI can do, and grounded enough to know that responsible deployment matters as much as capability, this role was written for you.

Core Responsibilities

1. Fabric Data Agents — Design, Build & Own

• Design and configure Microsoft Fabric Data Agents that expose semantic models and lakehouse datasets to natural language querying via Microsoft Copilot and other surfaces.
• Define entity mappings, data surface boundaries, Q&A annotations, and grounding constraints for each Data Agent; ensure agents return accurate, source-attributed responses.
• Build evaluation harnesses that test Data Agent responses against known-good ground truth before any agent is published; establish and enforce accuracy and groundedness pass thresholds.
• Monitor deployed Data Agents for query accuracy, usage volume, and failure patterns; iterate on configurations as underlying semantic models and data assets evolve.
• Partner with the Power BI Analyst to align Data Agent data surfaces with certified semantic model definitions; ensure agents never expose uncertified or draft metrics.

2. Fabric Operations Agents — Design, Build & Own

• Design and deploy Fabric Operations Agents that monitor pipeline health, detect anomalies, trigger automated remediation steps, and escalate to humans when defined thresholds are crossed.
• Define the event surfaces, alert conditions, and action schemas that operations agents act upon; document decision logic and escalation trees for each deployed agent.
• Implement human-in-the-loop checkpoints for any operations agent action that modifies data, restarts pipelines, or affects production systems; no autonomous destructive action without an approved override protocol.
• Maintain an operations agent registry that documents purpose, scope, data access, action permissions, deployment status, and review schedule for every active agent.
• Partner with the Data Engineering team to instrument pipelines with the observability hooks that operations agents depend on.

3. Azure AI Foundry — Orchestration & Deployment

• Use Azure AI Foundry as the primary platform for building, evaluating, and deploying multi-agent workflows; manage model deployments, prompt flow definitions, and agent evaluation runs.
• Design multi-agent architectures using task decomposition, agent handoff patterns, parallel execution, and shared memory stores; document orchestration logic in version-controlled repositories.
• Configure and manage AI Foundry evaluation pipelines that score agent outputs on accuracy, groundedness, safety, coherence, and latency; own quality gates between Dev, Test, and Production.
• Manage compute, quota, and cost for AI Foundry deployments; flag anomalies and optimize configurations to stay within budget targets.
• Stay current on Azure AI Foundry feature releases; evaluate new capabilities (e.g., new model deployments, agent SDK updates) and recommend adoption where there is clear operational value.

4. Claude & Anthropic Platform

• Build production-quality agents using the Claude API: design system prompts, configure tool use schemas, manage context window strategy, and implement multi-turn conversation patterns.
• Deploy and administer Claude Cowork for desktop and file-based automation workflows; define task scope, validate outputs, and govern the boundaries of autonomous action for each workflow.
• Configure and extend MCP (Model Context Protocol) servers to securely expose internal tools, data APIs, and Fabric surfaces to Claude agents; design tool schemas that are reliable, safe, and appropriately scoped.
• Apply prompt engineering best practices — system prompt structure, few-shot examples, chain-of-thought elicitation, output format constraints — and maintain a versioned prompt library for all production Claude agents.
• Evaluate Claude model versions and configurations for specific use cases; document reasoning for model selection decisions and review when new Claude versions are released.

5. Copilot Studio & Microsoft Copilot

• Build and configure Copilot Studio agents for business user-facing workflows: define topics, actions, Power Automate integrations, and escalation paths.
• Connect Copilot Studio agents to Fabric Data Agents, internal REST APIs, and SharePoint knowledge sources as appropriate; ensure connection security and data access is scoped to minimum necessary permissions.
• Advise business teams on appropriate Copilot (M365) use cases; document what Copilot can and cannot do reliably in the context of specific business workflows.
• Test Copilot Studio agents with representative end-user scenarios before deployment; maintain feedback loops that surface accuracy issues post-launch.

6. Agent Governance, Safety & Lifecycle Management

• Own the agent governance framework: define and enforce policies covering data access scope, action permissions, output validation requirements, human-in-the-loop triggers, and audit logging for all deployed agents.
• Classify each agent by risk tier based on the actions it can take and the data it can access; apply proportionate oversight controls for each tier.
• Establish and maintain the agent registry as the authoritative record of every agent in production: purpose, owner, data access, action scope, deployment date, last review date, and retirement plan.
• Define a structured agent review cadence — at minimum quarterly — to assess whether each agent’s configuration, data access, and behavior remain appropriate as the underlying platform and business context evolve.
• Design and enforce incident response procedures for agent failures, unexpected outputs, or security events; conduct post-incident reviews and publish findings to the broader team.
• Maintain awareness of evolving responsible AI standards (Microsoft RAI, Anthropic’s guidelines) and incorporate relevant principles into team practices.

7. Collaboration & Enablement

• Partner with the Power BI Analyst on Data Agent design and semantic model alignment; partner with the Data Engineering team on Operations Agent instrumentation and pipeline data surfaces.
• Act as the team’s subject matter expert on agentic AI capabilities and limitations — translate what agents can realistically do into scoped, deliverable proposals for business stakeholders.
• Produce clear technical documentation for every agent: architecture diagrams, prompt rationale, tool schemas, evaluation results, and known limitations.
• Train and support team members on effective use of AI tools; build internal literacy around prompt engineering, agent interaction patterns, and responsible AI usage.
• Contribute to and review pull requests for agent code, prompt definitions, and evaluation scripts; uphold code quality and safety standards across the team’s shared repositories.

Requirements:

Education & Certification:

• Bachelor’s degree in Computer Science, Engineering, Data Science, Information Systems, or a related field; or equivalent practical experience.
• Microsoft Certified: Azure AI Engineer Associate or Azure AI Foundry-related certification strongly preferred.
• Additional relevant certifications (e.g., Microsoft Fabric, Anthropic prompt engineering, Copilot Studio) are a plus and speak to genuine engagement with the tooling.

Experience:

• 3–5 years of hands-on experience designing, building, and deploying AI-powered applications or agent-based systems in a production environment.
• Demonstrated experience with at least one enterprise agentic platform — Azure AI Foundry, Copilot Studio, LangChain, Semantic Kernel, or equivalent.
• Practical experience with the Anthropic Claude API: system prompt design, tool use configuration, multi-turn conversation management, and output evaluation.
• Experience building and consuming REST APIs; comfort writing Python code for agent logic, tool definitions, and integration scripts.
• Demonstrated experience establishing evaluation and testing practices for AI systems — not just building agents but knowing whether they work.
• Exposure to enterprise data platforms (Microsoft Fabric, Azure Synapse, Databricks, or equivalent) and an understanding of how agents access and query structured data.
• Experience working within Git-based workflows and Agile delivery environments.

Skills & Attributes:

• Strong engineering instincts — writes clean, maintainable code and documents decisions in a way that teammates can build on.
• Comfortable operating at the frontier: the tooling in this space moves fast, and you need to distinguish genuine capability from vendor hype.
• Rigorous about safety and governance — equally motivated by what agents can do and what they should be prevented from doing.
• Clear communicator: able to explain agent architecture, risk, and limitations to non-technical stakeholders without oversimplifying.
• Collaborative by default — this role sits at the intersection of data engineering, BI, and business operations and requires active partnership across all three.
• Intellectually curious and self-directed — identifies emerging capabilities relevant to the team’s work without waiting to be asked.

X Factors: 

Strong candidates will stand out in one or more of the following areas:

• Prior experience specifically with Microsoft Fabric Data Agents or Fabric Operations Agents in a production deployment.
• Hands-on experience with MCP (Model Context Protocol) server configuration and tool schema design.
• Background in AI safety, responsible AI policy, or enterprise AI governance frameworks.
• Experience deploying Semantic Kernel-based agents in a Python environment with persistent memory and multi-agent orchestration.
• Familiarity with LLM evaluation frameworks (Azure AI Foundry Evaluation, Ragas, ROUGE, or similar) and experience designing domain-specific evaluation datasets.
• Experience in a startup, high-growth company, or operationally critical industry where agent failures have real consequences.
• Contributions to open-source agent frameworks, MCP tooling, or published writing on agentic AI system design.

This is a new role. The Agent Manager will have significant influence over how agentic AI is adopted and governed across the organization. The right candidate will treat that influence as both an opportunity and a responsibility.

May perform other responsibilities as assigned. Responsibilities and duties may change when circumstances dictate (e.g., emergencies, changes in workload, etc.).

Avelo is an equal opportunity employer.