Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Manager; AI Evaluation Engineering

$147.76k - $240.11k

Caterpillar

AI Engineering Team Leader

Your Work Shapes the World at Caterpillar Inc.

When you join Caterpillar, you're joining a global team who cares not just about the work we do but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities. We don't just talk about progress and innovation here we make it happen, with our customers, where we work and live. Together, we are building a better world, so we can all enjoy living in it.

Job Summary: Join the AI Engineering team of Cat Digital and take charge of leading a team dedicated to evaluating and validating our advanced generative AI solutionsincluding intelligent agents, digital assistants, and other innovative capabilities. You will shape the development of cutting-edge products that redefine how customers and dealers interact with technology through AI-driven experiences.

What You Will Do:

  • Providing strong technical support and clear direction to ensure the team is aligned with company goals and capable of evaluating advanced AI projects.
  • Overseeing the performance of both individual team members and the team as a whole, fostering a culture of learning by identifying and addressing training and development needs.
  • Taking ownership of the quality of AI engineering products, making sure all solutions are robust, reliable, and meet high standards.
  • Establishing and supervising the implementation of engineering best practices to maintain consistency and excellence in development processes.

What You Will Have:

  • Knowledge of major products and services and product and service groups; ability to apply knowledge of product and service appropriately to diverse situations.
  • Knowledge of software development tools and activities; ability to produce software products or systems in line with product requirements.
  • Knowledge of software development life cycle; ability to use a structured methodology for delivering and managing new or enhanced software products to the marketplace.
  • Knowledge of software quality assurance and testing; ability to apply appropriate processes, tools, and techniques for assuring a high level of quality in computer software products and systems.

Considerations For Top Candidates:

  • Proven experience leading software quality, test automation, validation, and AI evaluation teams, with the ability to scale processes, tools, metrics, and engineering practices across multiple products and enterprise initiatives.
  • Demonstrated success delivering enterprise-scale software and GenAI solutions across hybrid cloud and embedded/edge environments, leveraging modern software engineering practices including CI/CD, automated testing, incident management, feature flags, and progressive deployment strategies (e.g., blue/green and canary deployments).
  • Deep expertise in GenAI architecture and system design, including LLMs, SLMs, multimodal, speech, and real-time models; prompt engineering; agentic systems; tool use and orchestration; RAG architectures; vector databases, embeddings, and chunking strategies; fine-tuning techniques such as LoRA; and emerging standards including MCP and A2A.
  • Hands-on experience with AI development and evaluation ecosystems, including Azure, AWS, GCP, Azure AI Foundry, SageMaker, Bedrock, Snowflake Cortex, LangChain, LangGraph, Langfuse, Arize, LangSmith, Humanloop, Ragas, DeepEval, Phoenix, or similar technologies.
  • Deep understanding of AI evaluation methodologies and the challenges of non-deterministic systems, including metric design, RAG quality assessment, output reliability, safety evaluation, A/B testing, human-in-the-loop validation, and balancing deterministic and probabilistic testing approaches.
  • Expertise in modern AI observability and monitoring practices, including tracing, telemetry, evaluation data collection, root-cause analysis, and the use of AI technologies to enhance testing through automated test generation, synthetic data creation, scenario expansion, and LLM-assisted evaluation.

Summary Pay Range: $147,760.00 - $240,110.00

Compensation and benefits offered may vary depending on multiple individualized factors, job level, market location, job-related knowledge, skills, individual performance and experience. Please note that salary is only one component of total compensation at Caterpillar.

Benefits:

  • Medical, dental, and vision benefits*
  • Paid time off plan (Vacation, Holidays, Volunteer, etc.)*
  • 401(k) savings plans*
  • Health Savings Account (HSA)*
  • Flexible Spending Accounts (FSAs)*
  • Health Lifestyle Programs*
  • Employee Assistance Program*
  • Voluntary Benefits and Employee Discounts*
  • Career Development*
  • Incentive bonus*
  • Disability benefits
  • Life Insurance
  • Parental leave
  • Adoption benefits
  • Tuition Reimbursement

* These benefits also apply to part-time employees

Visa Sponsorship is not available for this position.

Posting Dates: September 1, 2026 - September 20, 2026

Any offer of employment is conditioned upon the successful completion of a drug screen.

Caterpillar is an Equal Opportunity Employer, Including Veterans and Individuals with Disabilities. Qualified applicants of any age are encouraged to apply.

Not ready to apply? Join our Talent Community.

Caterpillar
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Manager; AI Evaluation Engineering in Chicago, IL vacancy
  • $73.5k - $212.28k

     ...ApplicableSpecialismIFS - Information Technology (IT)Management LevelManagerJob Description & SummaryAt PwC, our people in data and analytics engineering focus on leveraging advanced...  ...closely with team members. We evaluate these factors thoughtfully to establish... 
    Suggested
    Full time
    H1b

    PwC

    Chicago, IL
    4 days ago
  • $99k - $232k

     ...Description & SummaryThe OpportunityAs an AI Engineer specializing in AML and Sanctions, you...  ...and enhance competitive advantage. As a Manager, you will motivate, develop, and inspire...  ...collaborating closely with team members. We evaluate these factors thoughtfully to establish... 
    Suggested
    Full time
    H1b

    PwC

    Chicago, IL
    1 day ago
  • $91k - $321.5k

     ...ApplicableSpecialismIFS - Information Technology (IT)Management LevelSenior ManagerJob Description & SummaryThe OpportunityAs a CTIO - AI Engineer- Senior Manager, you will play a pivotal...  ...closely with team members. We evaluate these factors thoughtfully to establish a... 
    Suggested
    Full time
    H1b

    PwC

    Chicago, IL
    1 day ago
  •  ...Accenture’s Global Responsible AI team within the Global Data &...  ...if you’re an experienced RAI Engineer with a Responsible AI...  ...responsible practices.Detecting, evaluating, and applying relevant RAI dimensions...  ...etc)Experience with database management and data processing.... 
    Suggested
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    Chicago, IL
    14 hours ago
  • $124k - $280k

     ...Description & SummaryThe OpportunityAs a Senior Manager, Agentic Engineering Lead, you will guide the design and delivery of agentic and generative AI solutions for clients within our...  ..., including operational monitoring- Evaluating models and agent behavior through testing... 
    Suggested
    Full time
    H1b

    PwC

    Chicago, IL
    14 hours ago
  • $124k - $280k

     ...SummaryThe OpportunityAs an Applied AI Health Services Data Science-Senior Manager you will lead client-facing work...  ...will draw on data and analytics engineering, artificial intelligence, and...  ...deployment- Validating model outputs, evaluation methods, and feedback loops using... 
    Full time
    H1b

    PwC

    Chicago, IL
    2 days ago
  •  ...tech-agnostic and human-centered. The Opportunity As an AI Delivery Engineer on PwC's client-facing Generative AI team, you work...  ...aware access through AWS IAM Write automated tests and AI evaluations, and build deployment pipelines, monitoring, and error handling... 
    Work experience placement
    H1b
    Flexible hours

    Kunai

    Chicago, IL
    14 days ago
  •  ...the best job for you. Job Position: AI Engineer Location: Chicago, IL, USA_ Onsite...  ...multi-agent collaboration patterns, memory management, and autonomous planning frameworks....  ..., compliance, risk mitigation, and evaluation guardrails across all AI touchpoints.... 
    Permanent employment
    Contract work
    Remote work

    Tekfortune Inc.

    Chicago, IL
    21 days ago
  • $130k - $170k

     ...families, and businesses. We help our clients evaluate their risks and develop solutions...  ..., retirement, and private wealth management products and services. With over $5 billion...  ...Position Overview HUB is seeking a Senior AI Engineer to serve as the senior technical... 
    Contract work

    HUB International

    Chicago, IL
    1 day ago
  •  ...Modernization. What are we looking? Overview: The AI Software Engineer is responsible for the technical backbone of our autonomous...  ...layers, retrieval systems (RAG), and production-grade evaluation pipelines. ~ Integrate Generative AI services (e.g., OpenAI... 

    Photon

    Chicago, IL
    19 days ago
  •  ...About Menos AI Menos AI builds AI-powered research intelligence for institutional asset managers. We help some of the world’s largest investment...  ...We’re looking for AI engineers who do more than write code...  ...Engineer for uncertainty. Build evaluation frameworks, analyze failure... 
    Full time

    Menos AI

    Chicago, IL
    14 hours ago
  • $135k - $150k

     ...families, and businesses. We help our clients evaluate their risks and develop solutions...  ..., retirement, and private wealth management products and services. With over $5 billion...  ...Position Overview HUB is seeking an AI Engineer to join the RPA & Automation team within... 
    Work at office

    HUB International

    Chicago, IL
    1 day ago
  • Join a global engineering team reimagining insurance technology through intelligent automation...  ...and deploy production-grade Agentic AI systems powered by LLMs, RAG, and document...  ...performance improvement Develop robust AI evaluation frameworks encompassing testing,... 

    EPAM Systems

    Chicago, IL
    4 days ago
  •  ...AI Forward Deployed Engineer We are building out practitioners within our Song Sales practice: AI Forward Deployed Engineers who embed directly with clients to design, prototype, and deploy Agentic AI solutions that transform how our clients sell, serve, and operate... 

    Hackajob

    Chicago, IL
    1 day ago
  •  ...pushing boundaries with innovative AI solutions that simplify and...  ...problem, scoped with product managers and data scientists, through...  ...and buy-in, review other engineers' designs and code with the same...  ...with LLM observability and evaluation tooling such as LangSmith, Arize... 

    CCC Intelligent Solutions

    Chicago, IL
    4 days ago
  •  ...Senior AI Data Engineer (Cloud) Location: Chicago, IL Schedule: Full-time, 40 hours/week...  ...role will work closely with Data Product Management and Data Science teams to design,...  ...Arize. ~ Strong understanding of AI/LLM evaluation, error analysis, quality measurement,... 
    Full time
    Contract work

    Forbes Technical Consulting

    Chicago, IL
    19 days ago
  • $130k - $170k

     ...consultancy with nearly 200 engineers, designers, and product strategists...  .... Umain is growing its AI engineering team in the US....  ...ideas into robust solutions, evaluate where AI truly adds value and...  ...systems, including context management, memory, orchestration, and the... 
    Full time
    Work at office
    Relocation
    Flexible hours
    3 days per week

    Umain AB

    Chicago, IL
    3 days ago
  •  ...pushing boundaries with innovative AI solutions that simplify and...  ...problem scoped with product managers and data scientists through...  ...of professional software engineering experience, including at least...  ...Experience with LLM observability and evaluation tooling such as LangSmith,... 

    CCC Intelligent Solutions

    Chicago, IL
    3 days ago
  •  ...-wide capability. We are looking for AI Engineers who are energized by working close to the...  ...needs into well-defined deliverables Evaluate and integrate AI capabilities (LLMs,...  ...automated workflows that accelerate SA and manager productivity across technical discovery,... 
    Immediate start

    FDE Times

    Chicago, IL
    4 days ago
  • $133k - $166k

     ...and operating secure, scalable AI platforms that power real-...  ...? As an AI Infrastructure Engineer , you'll play a key role in shaping...  ...Azure environments, and manage core services such as Azure OpenAI...  ..., promotion, compensation, evaluation, disciplinary actions, and termination... 
    Worldwide
    Flexible hours

    Kirkland & Ellis LLP

    Chicago, IL
    4 days ago
  •  ...thinking services company at the forefront of AI-native innovation. We partner with...  ...next-generation, agent-powered workflows engineered to scale in real-world settings. Our engineers...  ...You Are:A Forward-Deployed AI EngineerSr Manager who thrives at the intersection of... 
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    Chicago, IL
    1 day ago
  •  ...ideas into reality.We are Secure, Responsible AI & Data Protection professionals who enable...  ...You AreManagers are the hands-on delivery engine of the Secure AI practice. They lead day-...  ...into capable practitioners. Each Manager hire will be expected to operate across multiple... 
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    Chicago, IL
    1 day ago
  •  ...AI Agent Engineer Evanston, IL (On-Site) Bear Claw is an AI-powered recruiting and applicant...  ...one system—from candidate and job management to sourcing, outreach, and team collaboration...  ...orchestration, tool calling, memory, evaluation, observability, and appropriate human... 
    Temporary work

    Bear Claw Events

    Evanston, IL
    14 hours ago
  •  ...Description Job Description Title: Sr. AI/ML Engineer Location: Chicago, IL Duration: 6...  ..., with a high degree of autonomy. Evaluate model and LLM performance using...  ...retraining pipelines, and model lifecycle management practices. Design and maintain production... 

    Diverse Lynx

    Chicago, IL
    21 days ago
  • $83k - $111k

     ...Early-Career Applied AI Role This is an early-career role designed...  ...work alongside experienced engineers and external delivery partners...  ...engineers and the Product Manager Contribute to building and...  ...applications Help build, train, and evaluate machine-learning and LLM-based... 
    Full time
    Internship
    Work at office

    WORLDPAC

    Oak Brook, IL
    3 hours ago
  • $147k - $202k

     ...building what comes next. Job Title: Lead AI Engineer Company: Prologis Title: Lead AI...  ...production outcomes rather than people management. The engineer partners with business, technology...  ...tool calling, workflow orchestration, evaluation, and human review. Build an AI-native... 
    Full time
    Work at office

    Prologis

    Chicago, IL
    1 day ago
  •  ....   Job Description The Senior AI Agency Engineer designs, builds, and deploys enterprise...  ...orchestration, structured outputs, evaluation frameworks, and human approval checkpoints...  ..., identity federation, secrets management, data mapping, error handling, observability... 
    Local area

    EVERSANA INTOUCH

    Chicago, IL
    21 days ago
  •  ...Job Description: The AI/ML Engineer will be responsible for designing, developing, deploying...  ...Engineering, Information Security, Risk Management, Compliance, Architecture, and the AI/ML...  .... The AI/ML Engineer will evaluate emerging technologies and identify opportunities... 
    Full time
    Work experience placement

    Beacon Hill

    Chicago, IL
    4 days ago
  • $72k - $141k

     ...building and operationalizing AI-driven solutions that...  ...This role combines hands-on engineering architecture and governance ensuring...  ...and agent-based patterns Evaluate model quality latency cost safety...  ...practices (OAuth JWT secrets management) Mentor engineers and guide... 
    Full time
    Work experience placement

    Continental Casualty Company

    Chicago, IL
    2 days ago
  • $180k - $225k

    IMC is embedding agentic AI directly into the developer workflow...  ...As a member of the Agentic AI engineering team, you'll help build agents, MCP servers, and evaluation loops that make AI-generated...  ...best outcome in value and risk management to investors. Using our own technology... 
    Permanent employment
    Full time

    IMC

    Chicago, IL
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Manager; AI Evaluation Engineering. Be the first to apply!