Application Engineer

Application Engineer
100 % Remote
Pay: $50.00 - $54.00 per hour
26-09365

Job Summary

The AI Engineer will design, test, optimize, and support AI-powered agent experiences across customer-facing and internal business workflows. This role combines application development, prompt engineering, agent orchestration, AI quality assurance, evaluation framework development, and observability to improve agent reliability, accuracy, consistency, and user experience. The ideal candidate has a strong application development background, hands-on experience with Generative AI and Large Language Models (LLMs), and the ability to quickly understand complex business challenges and deliver effective technical solutions. This is a fully remote role requiring strong communication skills, independent execution, and the ability to contribute effectively with minimal ramp-up time.

Job Responsibilities

• Design, develop, and iterate on AI agent logic and workflows for business use cases.
• Create, test, and optimize prompts, system instructions, tools, and workflows to improve agent performance.
• Conduct prompt experiments and A/B testing to measure effectiveness and identify improvement opportunities.
• Analyze agent responses and refine prompting strategies to improve accuracy, consistency, groundedness, and user satisfaction.
• Document prompt patterns, reasoning frameworks, reusable templates, and agent design decisions.
• Collaborate with engineering teams to deploy prompt and agent updates through established software development lifecycle processes.
• Design and execute comprehensive test plans for AI agents, web applications, APIs, and backend services.
• Develop evaluation frameworks and scoring methodologies for AI output quality.
• Create and maintain test cases covering accuracy, hallucination detection, safety, tone and style adherence, instruction following, context retention, and response consistency.
• Perform regression testing for prompt, model, application, and workflow changes.
• Track, prioritize, validate, and retest defects using Azure DevOps or similar platforms.
• Utilize AI observability platforms to monitor model and agent performance.
• Analyze user interactions and evaluation results to identify trends and improvement opportunities.
• Develop and monitor quality metrics related to accuracy, relevance, latency, user satisfaction, defect rates, and prompt performance.
• Investigate production issues and recommend prompt, workflow, application, or architectural improvements.
• Support the integration of AI capabilities into enterprise applications and business workflows.
• Collaborate with developers to validate API integrations, retrieval pipelines, and agent workflows.
• Support CI/CD deployment, release validation, and automated testing activities.
• Create and maintain automated testing assets and frameworks where appropriate.
• Maintain documentation for prompts, evaluation methodologies, testing procedures, and agent configurations.
• Participate in sprint planning, backlog refinement, and other Agile ceremonies.
• Communicate technical findings and recommendations to technical and non-technical stakeholders.
• Share best practices, technical knowledge, and lessons learned across the team.

Job Requirements

• Minimum of 4-5 years of relevant experience in application development, software engineering, QA engineering, automation testing, business systems analysis, or a related technical role.
• At least 1 year of hands-on experience with Generative AI, LLMs, prompt engineering, conversational AI, or AI-powered agent solutions.
• Strong application development and software engineering background.
• Experience designing, developing, testing, and supporting enterprise applications.
• Experience designing and executing structured test plans and testing methodologies.
• Experience working with AI agents and agent-based workflows.
• Experience working with Azure technologies and Azure DevOps or similar Agile delivery platforms.
• Experience working with APIs and JSON data structures.
• Strong analytical, troubleshooting, and problem-solving skills.
• Strong written and verbal communication skills.
• Ability to quickly understand complex business challenges and contribute effectively with minimal ramp-up time.
• Ability to work independently and collaboratively in a remote team environment.

Education

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field is required.

Technical Skills

• Prompt engineering and prompt optimization.
• AI agent design and orchestration.
• AI evaluation frameworks and quality assurance.
• LLM testing and evaluation.
• Retrieval-Augmented Generation (RAG) concepts.
• AI observability and performance monitoring.
• Application development and software engineering practices.
• REST APIs and JSON.
• Python.
• SQL.
• Test automation frameworks.
• Git.
• Azure DevOps.
• Azure AI and LLM platforms.
• API testing tools.

Preferred Qualifications

• Experience with AI observability platforms.
• Experience with Azure OpenAI or other LLM platforms.
• Knowledge of Retrieval-Augmented Generation (RAG) architectures.
• Experience evaluating AI systems using human and automated scoring methods.
• Familiarity with vector databases and semantic search.
• Experience creating automated test frameworks.
• Understanding of responsible AI principles and AI governance.
• Experience with Python, SQL, JavaScript, or C#.
• Knowledge of prompt versioning and experimentation practices.
• Salesforce application development or integration experience is preferred for one available opening.

Additional Job Details (Commute, Attendance, etc.)

• This is a 100% remote position.
• Standard business hours are generally 8:00 AM to 5:00 PM local time, with some flexibility.
• This role is intended for experienced professionals and is not an entry-level opportunity.

#ED

About Equiliem

Equiliem believes in empowering success. It's our job to cultivate relationships that connect people and employers in a way that is inclusive, intelligent, and allows both to thrive.

Across the U.S., leading companies in healthcare, government, engineering, manufacturing, professional services, and energy rely on us for their workforce solutions. Our recruiting and HR services include contract and direct hire staffing, Payrolling/EOR, Independent Contractor Compliance, and Managed Services.

For almost 30 years, we've helped shape our industry. Today, we continue to research, ask questions, and continuously enhance the candidate journey and client experience.

Benefits

We offer a comprehensive benefits package to our employees, which includes:

• Medical Insurance
• Vision & Dental Insurance
• Life Insurance
• Matching 401(k)
• Commuter Benefits
• Employee Discounts & Rewards
• Payroll Payment Options

By applying, you are providing consent for Equiliem to engage with you via phone calls, AI-generated calls, AI-generated text messages, standard text messages, and/or emails to share job opportunities. Consistent with our Privacy Policy, information obtained will not be shared with third parties for marketing or promotional purposes. Message frequency may vary for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You may reply STOP to cancel or HELP for assistance.

Equiliem is an equal opportunity employer. We do not discriminate or allow discrimination based on race, color, religion, creed, sex, age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Equiliem will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Employee Services Department at [emailprotected] or View phone number on click.appcast.io.