Sr. AI Application Engineer

Salary: $92,000 - 118,000 per year Requirements:

• Bachelors degree in Computer Science, Engineering, Information Systems, or a related discipline.
• 5 to 10 years of experience in software engineering, QA engineering, test automation, business systems analysis, or another technical function.
• At least 1 year of hands-on experience with generative AI, LLMs, prompt engineering, or conversational AI solutions.
• Experience using Azure DevOps or a comparable Agile delivery platform.
• Experience building and running structured test plans.
• Strong analytical ability and troubleshooting skills.
• Experience working with APIs and JSON data formats.
• Excellent written and verbal communication skills.
• Hands-on experience in AI and GenAI areas such as prompt engineering, agent design, evaluation frameworks, RAG concepts, LLM testing, AI quality assurance, AI observability, and prompt optimization.
• Familiarity with tools such as Arize AI, Azure DevOps, Azure OpenAI, Git, Postman, and optionally Jira or Power BI.
• Development experience with Python, SQL, REST APIs, JSON, and test automation frameworks.
• Preferred experience with Arize AI or similar observability tools, Azure OpenAI or other major LLM platforms, RAG architectures, human and automated AI scoring, vector databases, semantic search, automated test frameworks, responsible AI principles, and prompt versioning or experimentation practices.

• Design, build, and refine AI agent logic for business use cases.
• Create, test, and improve prompts, system instructions, tools, and workflows to boost agent performance.
• Run prompt experiments and A/B tests to evaluate effectiveness.
• Review agent outputs and adjust prompting approaches for accuracy, consistency, groundedness, and user satisfaction.
• Document prompt patterns, reasoning methods, reusable templates, and agent design choices.
• Work with engineering teams to release prompt and agent updates through established SDLC processes.
• Develop and execute test plans for AI agents, web applications, APIs, and backend services.
• Build evaluation frameworks and scoring methods for AI output quality.
• Create test cases focused on accuracy, hallucination detection, safety, tone and style adherence, instruction following, context retention, and response consistency.
• Perform regression testing for prompt and model updates.
• Track, prioritize, validate, and retest defects in Azure DevOps.
• Monitor model and agent performance using Arize or similar observability platforms.
• Analyze user interactions and evaluation data to identify trends and improvement opportunities.
• Build quality metrics and dashboards covering accuracy, relevance, latency, user satisfaction, defect rates, and prompt performance.
• Investigate production issues and recommend prompt, workflow, or architecture improvements.
• Support integration of AI capabilities into enterprise applications and workflows.
• Collaborate with developers to validate API integrations, retrieval pipelines, and agent workflows.
• Assist with CI/CD deployment and release validation activities in Azure DevOps.
• Create and maintain automated testing assets where appropriate.
• Maintain documentation for prompts, evaluation methods, testing procedures, and agent configurations.
• Participate in sprint planning, backlog refinement, and Agile ceremonies.
• Communicate findings and recommendations to technical and non-technical stakeholders.
• Share best practices and lessons learned across the team.

• AI
• AI Agents
• API
• Azure
• Backend
• CI/CD
• CRM
• DevOps
• Git
• Support
• JIRA
• JSON
• LLM
• Postman
• Power BI
• Python
• RAG
• REST
• SQL
• Salesforce
• Web
• Cloud
• C#
• JavaScript

More:

We are seeking a Sr. AI Application Engineer for a contract, right-to-hire, or direct placement opportunity with one of our premier infrastructure services clients. This is a 100% remote role focused on designing, testing, optimizing, and supporting AI-powered agent experiences across customer and internal business workflows. We use Salesforce in our CRM stack, and experience with AgentForce or Einstein would be highly valued, though it is not required. The role offers the chance to work closely with product managers, software engineers, architects, and business stakeholders while contributing to a diverse team built on customer service, employee development, and quality and integrity. We are a women-owned technology and resource solutions provider with a boutique approach, matching consultants to more than 1,000 engagements each year.

last updated 36 week of 2026