AI/ML Evaluation Engineer

AI/ML Evaluation Engineer The Opportunity: As an experienced engineer, you know that machine learning (ML) and AI evaluation are critical to understanding and operationalizing massive datasets in support of public health and safety missions. Your ability to evaluate, optimize, and deploy AI-driven systems makes you an integral part of delivering mission-focused solutions for scientists, analysts, and leadership teams. In this role, you’ll help define and implement scientific AI evaluation and enablement initiatives by translating advanced AI capabilities into practical, mission-specific workflows. You’ll collaborate with a large community of ML engineers, data scientists, architects, and product teams to design scalable ML and Generative AI solutions, including AI agents, retrieval-augmented generation (RAG) pipelines, and enterprise AI evaluation frameworks. You’ll apply technical expertise across AI evaluation, retrieval optimization, memory and state management, and AI agent architecture to support real-time insights and decision-making. Your work will also contribute to enterprise MLOps capabilities, data governance standards, and ethical AI practices within regulated public health environments. This position is located in Atlanta, GA. What You'll Work On: Build and maintain scalable data pipelines using PySpark and Palantir Foundry to support AI, analytics, and scientific evaluation workflows. Design and implement ML and Generative AI workflows, including AI agents, RAG pipelines, and AI evaluation frameworks. Integrate advanced AI technologies such as Codex and Claude, to support mission-specific workflows, real-time insights, and decision-making capabilities. Develop evaluation strategies covering model quality, retrieval optimization, benchmarking, and performance monitoring for deployed AI systems. Design AI architectures supporting memory, state management, structured and unstructured public health data interaction, and scalable agent orchestration. Establish data governance, privacy, anonymization, documentation, and ethical AI standards across AI/ML systems and public health data environments. Join us. The world can’t wait. You Have: 5+ years of experience with Generative AI, LLMs, AI agents, or RAG applications, and designing, developing, and deploying ML models and AI solutions using Python 3+ years of experience with AI agents and AI evaluation strategies in enterprise environments 2+ years of experience with Deep Research evaluation methodologies and AI evaluation workflows Experience with ML frameworks such as TensorFlow or PyTorch, for production-grade model development Experience with data engineering using PySpark, SQL, and Palantir Foundry, including Foundry AIP Experience with MLOps platforms such as MLflow and cloud environments, including Azure Knowledge of public health, healthcare, or government data systems and associated governance practices Ability to design and optimize AI systems involving retrieval workflows, memory or state management, and real-time decision-support capabilities Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements Bachelor's degree in CS, Engineering, or Data Science Nice If You Have: Experience working in healthcare, biomedical, or government public health AI/ML environments Experience with conversational AI, chatbot systems, or full-stack AI application development Experience with containerization, CI/CD, orchestration, and production MLOps pipelines Experience with Agile delivery environments and tools such as Jira Experience writing technical documentation and presenting AI/ML solutions to various audiences Experience integrating enterprise AI tools such as Codex, Claude, or similar AI enablement platforms Knowledge of enterprise AI governance, compliance, and ethical AI frameworks Knowledge of AI systems for retrieval, ranking, and scientific evaluation use cases Ability to collaborate effectively in matrixed, cross-functional organizations Master's degree in CS, Data Science, ML, or a related field Vetting: Applicants selected will be subject to a government investigation and may need to meet eligibility requirements of the U.S. government client. Compensation At Booz Allen, we celebrate your contributions, provide you with opportunities and choices, and support your total well-being. Our offerings include health, life, disability, financial, and retirement benefits, as well as paid leave, professional development, tuition assistance, work-life programs, and dependent care. Our recognition awards program acknowledges employees for exceptional performance and superior demonstration of our values. Full-time and part-time employees working at least 20 hours a week on a regular basis are eligible to participate in Booz Allen’s benefit programs. Individuals that do not meet the threshold are only eligible for select offerings, not inclusive of health benefits. We encourage you to learn more about our total benefits by visiting the Resource page on our Careers site and reviewing Our Employee Benefits page. Salary at Booz Allen is determined by various factors, including but not limited to location, the individual’s particular combination of education, knowledge, skills, competencies, and experience, as well as contract-specific affordability and organizational requirements. The projected compensation range for this position is $128,700.00 to $292,000.00 (annualized USD). The estimate displayed represents the typical salary range for this position and is just one component of Booz Allen’s total compensation package for employees. This posting will close within 90 days from the Posting Date. Identity Statement As part of the hiring process, we will ask you to complete an identity verification process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity fraud. You are expected to be on camera during interviews and assessments. We reserve the right to take your picture to verify your identity and prevent fraud. Candidate AI Usage Policy AI is a part of our daily work at Booz Allen, and we are committed to the responsible and ethical use of AI tools. However, we want to ensure a fair candidate process based on your own skills and knowledge. As part of this commitment, the use of artificial intelligence (AI) or other tools to assist with responses during interviews (whether in-person or virtual) is prohibited unless permission is explicitly provided. Work Model Our people-first culture prioritizes the benefits of collaboration, whether it occurs in person or virtually. To support engagement and effective communication, employees working virtually are generally expected to have their cameras on during meetings. Remote: If this position is listed as remote, there may still be occasions when you are required to work in person at a Booz Allen or customer facility. Hybrid: If this position is listed as hybrid, you will be expected to work from a Booz Allen facility frequently, in alignment with leadership expectations and the needs of the role. You may also be required to work from or visit a customer facility. Onsite: If this position is listed as onsite, work will primarily be performed at a Booz Allen office or customer facility, where employees will collaborate directly with colleagues and customers as required by the role. Commitment to Non-Discrimination All qualified applicants will receive consideration for employment without regard to disability, status as a protected veteran or any other status protected by applicable federal, state, local, or international law. Note: Microsoft Internet Explorer is not fully compatible with Workday and users may experience systems issues with this browser. We recommend that you use one of the following browsers to avoid problems: Google Chrome, Mozilla Firefox, Microsoft Edge, Apple Safari, Opera Browser or Blackberry Browser. If you continue to experience issues, it is sometimes necessary to reset your browser by clearing your cache. About Booz Allen Hamilton Booz Allen is an advanced technology company delivering outcomes with speed for America’s most critical defense, civil, and national security priorities. We build technology solutions using AI, cyber, and other cutting-edge technologies to advance and protect the nation and its citizens. By focusing on outcomes, we enable our people, clients, and their missions to succeed—accelerating the nation to realize our purpose: Empower People to Change the World®. Commitment to Non-Discrimination All qualified applicants will receive consideration for employment without regard to disability, status as a protected veteran or any other status protected by applicable federal, state, local, or international law. Know Your Rights Poster Accommodations If you are an individual with a disability and would like to request a reasonable workplace accommodation for any part of our employment process, please contact the Booz Allen Help Desk by calling View phone number on click.appcast.io or sending an email to View email address on click.appcast.io. This option is reserved only for individuals who are requesting a reasonable workplace accommodation. It is not intended for other purposes or inquiries. Data Privacy For more information on how Booz Allen uses your information, please see our Careers Privacy Policy.