AI Agent Evaluation Engineer CI/CD Quality & Metrics
$142.65k - $213.98kComcast Corporation
Comcast in Washington, D.C. is seeking an experienced professional to design and manage agent evaluation pipelines ensuring AI solutions meet user expectations. Responsibilities include developing evaluation metrics, conducting A/B testing, and mentoring junior engineers. Candidates should possess a Bachelor's degree and 5-7 years of experience, preferably in customer support AI. The position offers competitive compensation ranging from $142,651.46 to $213,977.19, along with best-in-class benefits. #J-18808-Ljbffr
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer CI/CD Quality & Metrics in Washington DC vacancy
- ...Senior AI Engineer — Inference & Agent Systems Title: Applied AI Engineer — Inference... ...architecture, and an evaluation harness we use to measure... ...where TTFT was the primary metric and you moved it meaningfully... ...unsolved at production quality. Small team. High ownership...Quality
Arcana Analytics
Washington DC2 days ago - ...Northramp LLC is seeking a detail-oriented Senior QA Automation Engineer to deliver high-quality software solutions in federal environments. The role... ...with various teams to integrate testing into CI/CD pipelines. Ideal candidates will have over 7 years in QA...QualityRemote workFlexible hours
Northramp
Washington DC1 day ago - ...Machine Learning Test Engineer Location:... ...define and uphold quality standards for ML systems... ..., repeatable evaluation of ML models and data... ...suites, metrics, and test datasets... ...using Python and CI/CD (e.g., GitHub Actions... ...engineering or QA for ML/AI systems ~ Strong...QualityFor contractorsWork at officeRemote work
Autodesk
Washington DC3 days ago - ...Everforth ECS is seeking a CI/CD Pipeline Engineer to work in the National Capital... .... Department of War's (DoW) AI-First strategy introduced in... .... • Executes test and evaluation workflow integrations, incorporating... ...Monitors pipeline execution metrics, job durations, failure...SuggestedContract work
ECS Limited
Falls Church, VA4 days ago - ...Everforth ECS is seeking a Senior CI/CD Pipeline Engineer to work in the National... .... Department of War's (DoW) AI-First strategy introduced in... .... • Integrates test and evaluation gates into deployment... ...operational diagrams, and reliability metrics that strengthen model-...SuggestedContract work
ECS Limited
Falls Church, VA2 days ago $110k - $230k
...brand by offering quality coverage to millions... ...The Staff Security Engineer owns the end-to-end... ...control performance metrics, and telemetry... ...internal systems. Evaluate and guide the responsible use of AI capabilities within... ...native services, and CI/CD environments to support...QualityHourly payWork experience placementLocal areaRemote workFlexible hoursGEICO
Bethesda, MD2 days ago- ...Sr. QA Automation Engineer Job ID#: 26-2250 Clearance: Must... ...framework development, and quality engineering efforts... ...libraries integrated with CI/CD pipelines. Execute functional... ...and automation maturity. Evaluate and implement new automation and AI‑driven testing tools....QualityLocal area
Navitas Business Consulting
Hyattsville, MD4 days ago - ...We are seeking a QA Automation Engineer / SDET to support quality engineering for our Enterprise Data... ...~ Support tracking of data quality metrics and trends ~ Develop and... ...Integrate and execute tests within CI/CD pipelines (e.g., Azure DevOps)...QualityRemote work
Priwils, Inc
Hyattsville, MD1 day ago $160k - $180k
...Position Summary The Engineer, AI & Process Automation sits... ...of throughput and quality. We are building Carlyle... ...solutions using modern coding agents and developer tools —... ...blocks (components, evaluation harnesses, prompt and... ...— version control, CI/CD, testing,...QualityWork at officeCarlyle
Washington DC4 days ago- ...minimum. The AV Safety Engineering Analytics team is seeking an AI/ML Engineer with... ...-related data and evaluate driving performance... ...Pilot and develop metrics for monitoring development... ...the efficiency, quality and transparency of... ...& MLOps: CI/CD, MLflow, Model Monitoring...QualityH1bRemote workRelocationRelocation package
- ...BaseCamp Consulting & Solutions is seeking a QA Engineer for a hybrid role requiring 2 days onsite in Washington, DC. The successful candidate... ...test automation scripting alongside strong DevOps and CI/CD integration. Key qualifications include a minimum of 12 years of...
BaseCamp Consulting & Solutions
Washington DC4 days ago $90k
...Software Engineer, Intelligent Systems Job no: 5... ...systems, 2) engineering AI systems, and 3) integrated... ...) • Thorough test, evaluation, and documentation of... ...Continuous Development (CI/CD) processes for code development... ...that enhances the quality of life in Virginia and...QualityVirginia Tech
Arlington, VA2 days ago- ...Quality Engineer Seeking a highly responsible experienced quality engineer... ...integrate automated tests into CI/CD pipelines. Provide clear test documentation, metrics, and quality reports to... ...scrum teams. Continuously evaluate and improve testing methodologies...Quality
InstantServe LLC
Arlington, VA5 days ago - ...data analyst and systems engineer to supplement an... ...traditional ML and generative AI solutions within... ...support of delegated work, quality assurance reviews, project... ...business intelligence metrics to assist stakeholders... ...Learning solutions, including CI/CD implementation for data...Quality
Technomics
Arlington, VA5 days ago $115.4k - $192.3k
...Solr, or similar search engines and are looking to... ...graph-powered agentic AI with advanced search technologies... ...to ensure software quality and maintainability.... ...driven development and CI/CD practices. Ability to independently... ...data systems (e.g., metrics, logs, alerts)....QualityLocal areaWorldwideRELX
Alexandria, VA5 days ago- ...Quality Assurance Engineer At IntraFi, we do more than innovate—we empower. Our services help banks provide... ...with artificial intelligence (AI) tools to optimize workflows, problem-solving... ...and testing. Familiarity with CI/CD pipelines and related tools, including...QualityFlexible hours
Promontory Interfinancial Network
Arlington, VA2 days ago $141.2k - $211.8k
...Lead Partner, Sales Engineer (Public Sector) Black... ...organizations build secure, high-quality software, minimizing... ...source components, and AI generated code... ..., DAST, DevSecOps, and CI/CD integration Advise partners... ...government customers evaluate, adopt, and scale...QualityLocal areaRemote workBlack Duck Software
Washington DC1 day ago- ...autonomous systems, and agentic AI can be built and... ...tied to a gitlab CI/CD infrastructure. Day to... ...building out profiling metrics, and securing overall cloud... ...Working with security engineers to ensure NIST, STIG compliance... .../QA automation Code quality and security scanning...QualityWork at officeLocal areaRemote work3 days per week
Swoop Technologies, LLC
Washington DC4 days ago $335k
...powered by Lattice OS, an AI-powered operating... ...a Director of AI Engineering & Research to build and... ...across the team: code quality, model evaluation gates, reproducibility... ...registries, and CI/CD for models targeting... ...training, and deployment of agents, VLA models, and...QualityFull timeWork experience placementNeura Market
Washington DC5 days ago$117k - $135k
...highly responsible experienced quality engineer with strong background in test... ...automated tests into CI/CD pipelines. Provide clear test documentation, metrics, and quality reports to stakeholders... ...all scrum teams. Continuously evaluate and improve testing methodologies...QualityFull timeImmediate startRemote workRogueThink
Arlington, VA5 days ago- ...A technology solutions provider is seeking a Senior Quality Assurance Engineer – Automated Test Specialist in Washington, DC. This critical role involves designing automated testing frameworks and executing robust test strategies for mission-critical systems. The ideal...Quality
Diverse Agile Solutions LLC
Washington DC4 days ago - ...currently seeking an Agentic AI Systems Engineer with 6+ years of... ...and implement AI agents that perform drafting,... ...standards. Design evaluation frameworks and test harnesses to assess quality, factuality, accessibility... ...continuous deployment (CI/CD) practices. Ability...QualityFull timeTemporary workWork experience placementWork at officeLocal areaImmediate startAfternoon shift
Corner Alliance
Washington DC4 days ago - Forward Deployed Engineer Job Description Join... ..., often involving AI/automation, into... ...performance and data quality. Participate in... ..., model evaluation, and safety/guardrails... ...containers, microservices, CI/CD, IaC (Terraform/... ...(logging/metrics/tracing). Familiarity...QualityLocal area
$142.8k - $261.8k
...build a better working world. AI & Data - Physical AI Engineering Consultant – Manager The... ..., you will monitor and evaluate learning processes to continuously... .... Consistently deliver quality client services focusing... ...MLFlow. Experience with CI/CD practices to automate the...QualityFull timeWork experience placementSummer holidayFlexible hoursErnst & Young Oman
Arlington, VA5 days ago$105.8k - $174.8k
...Data and Decision Science – AI Native Engineering Physical AI Engineering... ...backgrounds. Consistently deliver quality client services focusing on... ..., transforming data and evaluating results to make meaningful... ...MLFlow. Experience with CI/CD practices to automate the...QualityFull timeWork experience placementSummer holidayFlexible hoursEY
Arlington, VA2 days ago$142.65k - $213.98k
Job Summary The Agent Evaluation team is responsible... ...for testing whether AI agents return the... ...build the framework, metrics, and test cases... ...conversational AI quality (accuracy, relevance... ...evaluation into CI/CD pipelines to support... ...Partner with engineering, and product teams...QualityWork experience placement$105.8k - $174.8k
...and ambitions. As a Senior AI Native Engineer, you will be at the forefront... ...backgrounds. Consistently deliver quality client services focusing on... ..., transforming data and evaluating results to make meaningful... ...as MLFlow. Experience with CI/CD practices to automate the testing...QualityFull timeWork experience placementSummer holidayFlexible hoursErnst & Young Oman
Washington DC4 days ago$170k - $190k
The Lead Engineer, AI & Process Automation sits within Carlyle... ...of throughput and quality. We are building... ...solutions using modern coding agents and developer tools —... ...AI practices. Evaluate and select the right models... ...stack — version control, CI/CD, testing,...QualityWork at office- ...Sr. QA Automation Engineer Job ID#: 26-2250 Clearance: Must be... ...Engineer/SDET to drive comprehensive quality engineering for our Enterprise... ..., data quality, and AI/BI model accuracy. This role requires... ...tests, integrating with CI/CD pipelines, with coverage on all...QualityLocal area
Navitas Business Consulting
Hyattsville, MD4 days ago $86.5k - $142.7k
...prototypes and builds modern, AI‑enabled applications... ...‑concept, and guiding engineering teams through complex... ...prompt orchestration, evaluation and guardrails. Author... ...delivery and raise technical quality. Leverage AI coding and... ...operations. Design CI/CD pipelines, environment...QualitySummer holidayFlexible hoursErnst & Young Oman
Washington DC4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluation Engineer CI/CD Quality & Metrics. Be the first to apply!
Related searches
- signing agent Washington DC
- freight agent no experience Washington DC
- state farm agent Washington DC
- work from home chat agent Washington DC
- special agent Washington DC
- fbi agent Washington DC
- commissioning agent Washington DC
- executive protection agent Washington DC
- cruise agent Washington DC
- agent Washington DC