Manager; AI Evaluation Engineering
$147.76k - $240.11kCaterpillar
AI Engineering Team Leader
Your Work Shapes the World at Caterpillar Inc.
When you join Caterpillar, you're joining a global team who cares not just about the work we do but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities. We don't just talk about progress and innovation here we make it happen, with our customers, where we work and live. Together, we are building a better world, so we can all enjoy living in it.
Job Summary: Join the AI Engineering team of Cat Digital and take charge of leading a team dedicated to evaluating and validating our advanced generative AI solutionsincluding intelligent agents, digital assistants, and other innovative capabilities. You will shape the development of cutting-edge products that redefine how customers and dealers interact with technology through AI-driven experiences.
What You Will Do:
- Providing strong technical support and clear direction to ensure the team is aligned with company goals and capable of evaluating advanced AI projects.
- Overseeing the performance of both individual team members and the team as a whole, fostering a culture of learning by identifying and addressing training and development needs.
- Taking ownership of the quality of AI engineering products, making sure all solutions are robust, reliable, and meet high standards.
- Establishing and supervising the implementation of engineering best practices to maintain consistency and excellence in development processes.
What You Will Have:
- Knowledge of major products and services and product and service groups; ability to apply knowledge of product and service appropriately to diverse situations.
- Knowledge of software development tools and activities; ability to produce software products or systems in line with product requirements.
- Knowledge of software development life cycle; ability to use a structured methodology for delivering and managing new or enhanced software products to the marketplace.
- Knowledge of software quality assurance and testing; ability to apply appropriate processes, tools, and techniques for assuring a high level of quality in computer software products and systems.
Considerations For Top Candidates:
- Proven experience leading software quality, test automation, validation, and AI evaluation teams, with the ability to scale processes, tools, metrics, and engineering practices across multiple products and enterprise initiatives.
- Demonstrated success delivering enterprise-scale software and GenAI solutions across hybrid cloud and embedded/edge environments, leveraging modern software engineering practices including CI/CD, automated testing, incident management, feature flags, and progressive deployment strategies (e.g., blue/green and canary deployments).
- Deep expertise in GenAI architecture and system design, including LLMs, SLMs, multimodal, speech, and real-time models; prompt engineering; agentic systems; tool use and orchestration; RAG architectures; vector databases, embeddings, and chunking strategies; fine-tuning techniques such as LoRA; and emerging standards including MCP and A2A.
- Hands-on experience with AI development and evaluation ecosystems, including Azure, AWS, GCP, Azure AI Foundry, SageMaker, Bedrock, Snowflake Cortex, LangChain, LangGraph, Langfuse, Arize, LangSmith, Humanloop, Ragas, DeepEval, Phoenix, or similar technologies.
- Deep understanding of AI evaluation methodologies and the challenges of non-deterministic systems, including metric design, RAG quality assessment, output reliability, safety evaluation, A/B testing, human-in-the-loop validation, and balancing deterministic and probabilistic testing approaches.
- Expertise in modern AI observability and monitoring practices, including tracing, telemetry, evaluation data collection, root-cause analysis, and the use of AI technologies to enhance testing through automated test generation, synthetic data creation, scenario expansion, and LLM-assisted evaluation.
Summary Pay Range: $147,760.00 - $240,110.00
Compensation and benefits offered may vary depending on multiple individualized factors, job level, market location, job-related knowledge, skills, individual performance and experience. Please note that salary is only one component of total compensation at Caterpillar.
Benefits:
- Medical, dental, and vision benefits*
- Paid time off plan (Vacation, Holidays, Volunteer, etc.)*
- 401(k) savings plans*
- Health Savings Account (HSA)*
- Flexible Spending Accounts (FSAs)*
- Health Lifestyle Programs*
- Employee Assistance Program*
- Voluntary Benefits and Employee Discounts*
- Career Development*
- Incentive bonus*
- Disability benefits
- Life Insurance
- Parental leave
- Adoption benefits
- Tuition Reimbursement
* These benefits also apply to part-time employees
Visa Sponsorship is not available for this position.
Posting Dates: September 1, 2026 - September 20, 2026
Any offer of employment is conditioned upon the successful completion of a drug screen.
Caterpillar is an Equal Opportunity Employer, Including Veterans and Individuals with Disabilities. Qualified applicants of any age are encouraged to apply.
Not ready to apply? Join our Talent Community.
Caterpillar$73.5k - $212.28k
...ApplicableSpecialismIFS - Information Technology (IT)Management LevelManagerJob Description & SummaryAt PwC, our people in data and analytics engineering focus on leveraging advanced... ...closely with team members. We evaluate these factors thoughtfully to establish...SuggestedFull timeH1b$99k - $232k
...Description & SummaryThe OpportunityAs an AI Engineer specializing in AML and Sanctions, you... ...and enhance competitive advantage. As a Manager, you will motivate, develop, and inspire... ...collaborating closely with team members. We evaluate these factors thoughtfully to establish...SuggestedFull timeH1b$91k - $321.5k
...ApplicableSpecialismIFS - Information Technology (IT)Management LevelSenior ManagerJob Description & SummaryThe OpportunityAs a CTIO - AI Engineer- Senior Manager, you will play a pivotal... ...closely with team members. We evaluate these factors thoughtfully to establish a...SuggestedFull timeH1b- ...Accenture’s Global Responsible AI team within the Global Data &... ...if you’re an experienced RAI Engineer with a Responsible AI... ...responsible practices.Detecting, evaluating, and applying relevant RAI dimensions... ...etc)Experience with database management and data processing....SuggestedWork experience placementLive inWork at officeLocal area
$124k - $280k
...Description & SummaryThe OpportunityAs a Senior Manager, Agentic Engineering Lead, you will guide the design and delivery of agentic and generative AI solutions for clients within our... ..., including operational monitoring- Evaluating models and agent behavior through testing...SuggestedFull timeH1b$124k - $280k
...SummaryThe OpportunityAs an Applied AI Health Services Data Science-Senior Manager you will lead client-facing work... ...will draw on data and analytics engineering, artificial intelligence, and... ...deployment- Validating model outputs, evaluation methods, and feedback loops using...Full timeH1b- ...tech-agnostic and human-centered. The Opportunity As an AI Delivery Engineer on PwC's client-facing Generative AI team, you work... ...aware access through AWS IAM Write automated tests and AI evaluations, and build deployment pipelines, monitoring, and error handling...Work experience placementH1bFlexible hours
- ...the best job for you. Job Position: AI Engineer Location: Chicago, IL, USA_ Onsite... ...multi-agent collaboration patterns, memory management, and autonomous planning frameworks.... ..., compliance, risk mitigation, and evaluation guardrails across all AI touchpoints....Permanent employmentContract workRemote work
$130k - $170k
...families, and businesses. We help our clients evaluate their risks and develop solutions... ..., retirement, and private wealth management products and services. With over $5 billion... ...Position Overview HUB is seeking a Senior AI Engineer to serve as the senior technical...Contract work- ...Modernization. What are we looking? Overview: The AI Software Engineer is responsible for the technical backbone of our autonomous... ...layers, retrieval systems (RAG), and production-grade evaluation pipelines. ~ Integrate Generative AI services (e.g., OpenAI...
- ...About Menos AI Menos AI builds AI-powered research intelligence for institutional asset managers. We help some of the world’s largest investment... ...We’re looking for AI engineers who do more than write code... ...Engineer for uncertainty. Build evaluation frameworks, analyze failure...Full time
$135k - $150k
...families, and businesses. We help our clients evaluate their risks and develop solutions... ..., retirement, and private wealth management products and services. With over $5 billion... ...Position Overview HUB is seeking an AI Engineer to join the RPA & Automation team within...Work at office- Join a global engineering team reimagining insurance technology through intelligent automation... ...and deploy production-grade Agentic AI systems powered by LLMs, RAG, and document... ...performance improvement Develop robust AI evaluation frameworks encompassing testing,...
- ...AI Forward Deployed Engineer We are building out practitioners within our Song Sales practice: AI Forward Deployed Engineers who embed directly with clients to design, prototype, and deploy Agentic AI solutions that transform how our clients sell, serve, and operate...
- ...pushing boundaries with innovative AI solutions that simplify and... ...problem, scoped with product managers and data scientists, through... ...and buy-in, review other engineers' designs and code with the same... ...with LLM observability and evaluation tooling such as LangSmith, Arize...
- ...Senior AI Data Engineer (Cloud) Location: Chicago, IL Schedule: Full-time, 40 hours/week... ...role will work closely with Data Product Management and Data Science teams to design,... ...Arize. ~ Strong understanding of AI/LLM evaluation, error analysis, quality measurement,...Full timeContract work
$130k - $170k
...consultancy with nearly 200 engineers, designers, and product strategists... .... Umain is growing its AI engineering team in the US.... ...ideas into robust solutions, evaluate where AI truly adds value and... ...systems, including context management, memory, orchestration, and the...Full timeWork at officeRelocationFlexible hours3 days per week- ...pushing boundaries with innovative AI solutions that simplify and... ...problem scoped with product managers and data scientists through... ...of professional software engineering experience, including at least... ...Experience with LLM observability and evaluation tooling such as LangSmith,...
- ...-wide capability. We are looking for AI Engineers who are energized by working close to the... ...needs into well-defined deliverables Evaluate and integrate AI capabilities (LLMs,... ...automated workflows that accelerate SA and manager productivity across technical discovery,...Immediate start
$133k - $166k
...and operating secure, scalable AI platforms that power real-... ...? As an AI Infrastructure Engineer , you'll play a key role in shaping... ...Azure environments, and manage core services such as Azure OpenAI... ..., promotion, compensation, evaluation, disciplinary actions, and termination...WorldwideFlexible hours- ...thinking services company at the forefront of AI-native innovation. We partner with... ...next-generation, agent-powered workflows engineered to scale in real-world settings. Our engineers... ...You Are:A Forward-Deployed AI EngineerSr Manager who thrives at the intersection of...Work experience placementLive inWork at officeLocal area
- ...ideas into reality.We are Secure, Responsible AI & Data Protection professionals who enable... ...You AreManagers are the hands-on delivery engine of the Secure AI practice. They lead day-... ...into capable practitioners. Each Manager hire will be expected to operate across multiple...Work experience placementLive inWork at officeLocal area
- ...AI Agent Engineer Evanston, IL (On-Site) Bear Claw is an AI-powered recruiting and applicant... ...one system—from candidate and job management to sourcing, outreach, and team collaboration... ...orchestration, tool calling, memory, evaluation, observability, and appropriate human...Temporary work
- ...Description Job Description Title: Sr. AI/ML Engineer Location: Chicago, IL Duration: 6... ..., with a high degree of autonomy. Evaluate model and LLM performance using... ...retraining pipelines, and model lifecycle management practices. Design and maintain production...
$83k - $111k
...Early-Career Applied AI Role This is an early-career role designed... ...work alongside experienced engineers and external delivery partners... ...engineers and the Product Manager Contribute to building and... ...applications Help build, train, and evaluate machine-learning and LLM-based...Full timeInternshipWork at office$147k - $202k
...building what comes next. Job Title: Lead AI Engineer Company: Prologis Title: Lead AI... ...production outcomes rather than people management. The engineer partners with business, technology... ...tool calling, workflow orchestration, evaluation, and human review. Build an AI-native...Full timeWork at office- .... Job Description The Senior AI Agency Engineer designs, builds, and deploys enterprise... ...orchestration, structured outputs, evaluation frameworks, and human approval checkpoints... ..., identity federation, secrets management, data mapping, error handling, observability...Local area
- ...Job Description: The AI/ML Engineer will be responsible for designing, developing, deploying... ...Engineering, Information Security, Risk Management, Compliance, Architecture, and the AI/ML... .... The AI/ML Engineer will evaluate emerging technologies and identify opportunities...Full timeWork experience placement
$72k - $141k
...building and operationalizing AI-driven solutions that... ...This role combines hands-on engineering architecture and governance ensuring... ...and agent-based patterns Evaluate model quality latency cost safety... ...practices (OAuth JWT secrets management) Mentor engineers and guide...Full timeWork experience placement$180k - $225k
IMC is embedding agentic AI directly into the developer workflow... ...As a member of the Agentic AI engineering team, you'll help build agents, MCP servers, and evaluation loops that make AI-generated... ...best outcome in value and risk management to investors. Using our own technology...Permanent employmentFull time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Manager; AI Evaluation Engineering. Be the first to apply!






