Engineer, AI - AI Evaluation & Model Risk Lead

We are seeking an experienced Engineer, AI – AI Evaluation & Model Risk Lead to lead how AI models are evaluated, cleared, monitored, and documented for enterprise use. This role owns model-risk clearance, ongoing model evaluation, model cards, evaluation harnesses, and test suites to ensure AI models are measured, documented, and current.

The ideal candidate will have hands-on experience with machine learning models, AI tools, Python, cloud computing, data analysis, data modeling, model tuning, prompt engineering, and model evaluation . You will collaborate with Safety Standards, Cyber, Responsible AI, and cross-functional teams to support the safe and effective adoption of AI solutions.

Have you got the right qualifications and skills for this job Find out below, and hit apply to be considered.

Key Responsibilities

• Own initial model-risk clearance and approved-use ratings that gate AI models into use.
• Own ongoing model evaluation and re-clearance following model version changes.
• Evaluate models against live traffic and identify model drift.
• Author and maintain model cards documenting what was tested, approved use cases, and identified risks.
• Build and run AI evaluation harnesses and test suites aligned with established Safety Standards.
• Collaborate with Cyber teams responsible for security red-teaming and Responsible AI teams.
• Collaborate with cross-functional teams to ensure effective integration and functionality of AI systems.
• Develop, test, debug, document, and communicate application and component development activities.
• Validate technical results with user representatives and support overall solution integration.
• Select appropriate technical solutions by reusing, improving, reconfiguring, or creating components.
• Optimize application development for efficiency, cost, and quality
• Review and create unit test cases, scenarios, test plans, and execution results
• Contribute to HLD, LLD, SAD, architecture, application features, business components, and data models
• Perform defect root-cause analysis and mitigation and identify proactive quality improvements.
• Support release processes, configuration management, documentation, project delivery, and technical presentations.
• Interface with customers and development teams to clarify requirements and provide technical guidance.

Required Qualifications

• 2+ years of professional experience in relevant AI, machine learning, software development, or data-focused roles.
• Bachelor's degree with 3 years of related work experience , OR advanced degree with 1 year of related work experience , OR equivalent combination of education and experience.
• Degree in Computer Science, Engineering, Artificial Intelligence, Data Science, or related field preferred.
• 2–4+ years of experience developing and deploying machine learning models , including fine-tuning and prompt engineering.
• 2–4+ years of experience with AI tools and software development using Python or R
• 2–4+ years of experience collaborating with cross-functional teams to integrate AI solutions into business workflows.
• Strong knowledge of Cloud Computing
• Strong experience in Data Analysis, Data Management, and Data Modeling
• Knowledge of Model Tuning and Prompt Engineering
• Strong understanding of Model Evaluation and AI model-risk concepts.
• Experience creating and maintaining Model Cards & Documentation
• Understanding of Model Risk & Clearance processes.
• Strong analytical and problem-solving abilities with the ability to work under pressure and manage multiple tasks.

Preferred Qualifications

• Experience with AI model-risk evaluation and clearance
• Experience building and maintaining AI evaluation harnesses and test suites
• Experience evaluating models against production/live traffic and identifying model drift.
• Experience with model cards, model documentation, and AI governance practices
• Experience with machine learning fine-tuning and prompt engineering
• Preferred certifications include:
• Certified Analytics Professional (CAP)
• Machine Learning Certification
• Certified Data Scientist (CDS)

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. xhyhwjd BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.