Engineer, AI - AI Evaluation & Model Risk Lead

We are seeking an experienced Engineer, AI – AI Evaluation & Model Risk Lead to lead how AI models are evaluated, cleared, monitored, and documented for enterprise use. This role owns model-risk clearance, ongoing model evaluation, model cards, evaluation harnesses, and test suites to ensure AI models are measured, documented, and current.

The ideal candidate will have hands-on experience with machine learning models, AI tools, Python, cloud computing, data analysis, data modeling, model tuning, prompt engineering, and model evaluation . You will collaborate with Safety Standards, Cyber, Responsible AI, and cross-functional teams to support the safe and effective adoption of AI solutions.

Key Responsibilities

• Own initial model-risk clearance and approved-use ratings that gate AI models into use.
• Own ongoing model evaluation and re-clearance following model version changes.
• Evaluate models against live traffic and identify model drift.
• Author and maintain model cards documenting what was tested, approved use cases, and identified risks.
• Build and run AI evaluation harnesses and test suites aligned with established Safety Standards.
• Collaborate with Cyber teams responsible for security red-teaming and Responsible AI teams.
• Collaborate with cross-functional teams to ensure effective integration and functionality of AI systems.
• Develop, test, debug, document, and communicate application and component development activities.
• Validate technical results with user representatives and support overall solution integration.
• Select appropriate technical solutions by reusing, improving, reconfiguring, or creating components.
• Optimize application development for efficiency, cost, and quality .
• Review and create unit test cases, scenarios, test plans, and execution results .
• Contribute to HLD, LLD, SAD, architecture, application features, business components, and data models .
• Perform defect root-cause analysis and mitigation and identify proactive quality improvements.
• Support release processes, configuration management, documentation, project delivery, and technical presentations.
• Interface with customers and development teams to clarify requirements and provide technical guidance.

Required Qualifications

• 2+ years of professional experience in relevant AI, machine learning, software development, or data-focused roles.
• Bachelor's degree with 3 years of related work experience , OR advanced degree with 1 year of related work experience , OR equivalent combination of education and experience.
• Degree in Computer Science, Engineering, Artificial Intelligence, Data Science, or related field preferred.
• 2–4+ years of experience developing and deploying machine learning models , including fine-tuning and prompt engineering.
• 2–4+ years of experience with AI tools and software development using Python or R .
• 2–4+ years of experience collaborating with cross-functional teams to integrate AI solutions into business workflows.
• Strong knowledge of Cloud Computing .
• Strong experience in Data Analysis, Data Management, and Data Modeling .
• Knowledge of Model Tuning and Prompt Engineering .
• Strong understanding of Model Evaluation and AI model-risk concepts.
• Experience creating and maintaining Model Cards & Documentation .
• Understanding of Model Risk & Clearance processes.
• Strong analytical and problem-solving abilities with the ability to work under pressure and manage multiple tasks.

Preferred Qualifications

• Experience with AI model-risk evaluation and clearance .
• Experience building and maintaining AI evaluation harnesses and test suites .
• Experience evaluating models against production/live traffic and identifying model drift.
• Experience with model cards, model documentation, and AI governance practices .
• Experience with machine learning fine-tuning and prompt engineering .
• Preferred certifications include:
• Certified Analytics Professional (CAP)
• Machine Learning Certification
• Certified Data Scientist (CDS)

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.