AI Observability Engineer

AI Observability Engineer Be part of something groundbreaking at AIG. Join our new generative AI team to explore how artificial intelligence can be applied in insurance and beyond. Work alongside talented colleagues to innovate and lead projects that transform how we manage risk and serve customers. As an AI Observability Engineer, you will have the opportunity to make a meaningful impact by guiding groundbreaking AI initiatives and supporting our AI product teams. Responsibilities Model Performance Monitoring: track ML, LLM, and Agentic AI key performance indicators (e.g., context relevance, accuracy, precision, recall, F1 scores) to ensure model effectiveness. Data Quality and Bias Detection: monitor data inputs for anomalies, bias, and drift that affect fairness and accuracy of AI systems. Utilize observability tools to produce structured, trustworthy insights from detailed data sets on AI system activities. Manage ongoing operations of our AI model management platform and pipelines, evolving KPIs and observability capabilities. Review AI solutions across the business to ensure alignment with AIG’s Policies, AI Governance Framework, and Responsible‑AI principles. Provide support to AI product teams, conduct independent reviews, respond to ad‑hoc queries, and participate in workshops. Collaborate with legal, compliance, security, and other teams to coordinate review of Responsible‑AI issues and ensure a cohesive approach organization‑wide. Create executive‑level reports on Responsible‑AI and provide strategic recommendations to senior leadership. Create and update RAI training content for technical and non‑technical audiences at all levels of seniority. Contribute to the development and improvement of RAI assets (methods, tutorials, tools). Monitor and remain current on emerging global AI regulatory frameworks, standards, and policies, translating them into RAI framework updates. Required Skills and Experience Minimum 3 years’ experience and a Bachelor’s degree. Strong knowledge of machine learning and large language models, including models, algorithms, and evaluation metrics. Data pipeline and analysis skills, with ability to clean and transform data for monitoring and analysis. Passion for scaling human oversight of AI systems and familiarity with LLM development. Comfort with ambiguity and curiosity; capable of working in a fast‑moving, ownership‑driven environment. Experience and passion in Responsible‑AI or AI ethics, with understanding of AI technologies and frameworks such as generative AI. Strong collaboration skills with cross‑functional teams, driving implementation of Responsible‑AI practices, ideally within financial services or regulated industries. Working knowledge of effective policies and processes around usage of emerging AI tools and products, bringing industry best practices. Proven track record of solving complex problems, developing mitigation strategies, and influencing stakeholders on key AI initiatives. Passion for staying ahead of emerging global AI regulations and translating frameworks into practical, actionable updates. Nice to Have Prior experience building production LLM, RAG, or model inference monitoring pipelines. Experience instrumenting applications using OpenTelemetry distributed tracing or structured logs. Hands‑on experience with AI observability tools such as Fiddler, Arize, LangSmith or similar. Ability to build dashboards and alerts tied to AI model behavior beyond system uptime. Work Environment We value in‑person collaboration as a vital part of our culture; team members are expected to be primarily in the office to work effectively and support a connected environment. Benefits Our Total Rewards Program offers comprehensive benefits focused on health, wellbeing, financial security, and professional development, providing peace of mind for you and your family. Equal Opportunity Employer AIG provides equal opportunity to all qualified individuals regardless of race, color, religion, age, gender, gender expression, national origin, veteran status, disability or any other legally protected category. AIG is committed to working with and providing reasonable accommodations to job applicants and employees with disabilities. If you need a reasonable accommodation, please email View email address on click.appcast.io. #J-18808-Ljbffr