Lead Research Engineer - AI Evaluation & Benchmarking
$146.8k - $272.6kThomson Reuters
Job DescriptionThomson Reuters Labs is seeking a Lead Research Engineer to lead the design and delivery of production-ready software for artificial intelligence (AI) model evaluation and benchmarking. You will build scalable evaluation infrastructure that enables research and product teams to compare large language models (LLMs), measure quality, cost, and latency, and make informed decisions about model selection and development.You will own the engineering approach from early prototypes through reliable, maintainable tools, collaborating closely with research scientists, engineering teams, and product partners while helping shape technical direction and engineering delivery.Key ResponsibilitiesLead technical delivery and engineering planning: Partner with program and research leaders to define priorities, break projects into delivery stages, estimate effort and cost, recommend staffing and required skills, identify dependencies, and adapt plans to available resources.Design and build scalable evaluation infrastructure: Extend existing shared tools and services, design new components where needed, and develop production-quality Python software, APIs, and reusable evaluation frameworks. Deliver useful initial versions, validate them with users, and improve them through iterative development.Turn research methods into working software: Collaborate with scientists to build automated evaluation pipelines connecting model APIs, test datasets, and agreed scoring methods. Implement reliable workflows for comparing LLMs and other AI models using evaluation datasets relevant to products and customer needs.Ensure reliable, reproducible model comparisons: Maintain repeatable experiments, separate evaluation data from model development, and make results traceable to evaluation inputs and methods. Measure model quality, operating cost, and latency to support product teams in validating model performance.Build practical tools for researchers and product teams: Develop clear user interfaces, documentation, evaluation reports, and diagnostic capabilities. Enable scientists and product teams to run evaluations, understand failures, interpret results, and extend tools without depending on the original developer.Own production reliability, performance, and responsible data use: Integrate automated testing, deployment, and monitoring from the beginning. Address performance, scalability, and operating costs, establish service maintenance and support ownership, and work with data specialists to implement access controls, permitted-use rules, and retention requirements.Collaborate, mentor, and guide engineering decisions: Partner with foundational research, data engineering, platform engineering, and product teams to align interfaces, responsibilities, and delivery plans. Resolve technical dependencies, mentor engineers, communicate trade-offs and delivery risks, and recommend changes to scope or technical approach when necessary.Required Qualifications8-10+ years of relevant software engineering experience, including building and maintaining production systems involving machine learning or complex data.Strong hands-on Python programming skills, including API design, automated testing, and working with structured and unstructured data.Technical leadership and end-to-end delivery experience, including architecture decisions, engineering planning, staffing and effort estimation, project execution, and adapting priorities as requirements change.Experience collaborating with research scientists or machine learning practitioners to translate experiments and research methods into working software.Experience integrating or testing machine learning models, interpreting evaluation results, and applying findings to engineering decisions. Understanding of language models and the ability to recognize unreliable comparisons or test-data leakage.Cloud engineering experience using AWS, Azure, or a comparable platform, including distributed workflows, automated deployment, and monitoring.Working knowledge of SQL and experience developing software that processes structured and unstructured data.Strong collaboration, communication, problem-solving, and mentoring skills, with experience coordinating technical delivery across research, engineering, and product stakeholders.Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.Preferred QualificationsExperience building LLM evaluation and benchmarking pipelines, including automated scoring methods and human assessment workflows.Familiarity with experiment tracking, dataset versioning, and reproducible evaluation processes.Experience in natural language processing, information retrieval, or other applications involving complex text.Experience reducing model inference or operating costs while maintaining evaluation quality and system performance.A track record of developing reusable software libraries, frameworks, or tools adopted by other engineering teams.Research contributions or open-source software contributions.#LI-TH1This posting is for proactive recruitment purposes and may be used to fill current openings or future vacancies within our organization.What’s in it For You?Hybrid Work Model: We’ve adopted a flexible hybrid working environment for our office-based roles while delivering a seamless experience that is digitally and physically connected.Flexibility & Work-Life Balance: Flex My Way is a set of supportive workplace policies designed to help manage personal and professional responsibilities, whether caring for family, giving back to the community, or finding time to refresh and reset. This builds upon our flexible work arrangements, including work from anywhere for up to 8 weeks per year, empowering employees to achieve a better work-life balance.Career Development and Growth: By fostering a culture of continuous learning and skill development, we prepare our talent to tackle tomorrow’s challenges and deliver real-world solutions. Our Grow My Way programming and skills-first approach ensures you have the tools and knowledge to grow, lead, and thrive in an AI-enabled future.Industry Competitive Benefits: We offer comprehensive benefit plans to include flexible vacation, two company-wide Mental Health Days off, access to the Headspace app, retirement savings, tuition reimbursement, employee incentive programs, and resources for mental, physical, and financial wellbeing.Culture: Globally recognized, award-winning reputation for inclusion and belonging, flexibility, work-life balance, and more. We live by our values: Obsess over our Customers, Compete to Win, Challenge (Y)our Thinking, Act Fast / Learn Fast, and Stronger Together.Social Impact: Make an impact in your community with our Social Impact Institute. We offer employees two paid volunteer days off annually and opportunities to get involved with pro-bono consulting projects and Environmental, Social, and Governance (ESG) initiatives. Making a Real-World Impact: We are one of the few companies globally that helps its customers pursue justice, truth, and transparency. Together, with the professionals and institutions we serve, we help uphold the rule of law, turn the wheels of commerce, catch bad actors, report the facts, and provide trusted, unbiased information to people all over the world.Our use of AI within the recruitment process Thomson Reuters utilizes Artificial Intelligence (AI) to support parts of our global recruitment process. Unless you opt-out, our AI system will assess the information provided by you and compare it to the requirements listed for the role, and present the result to our recruitment personnel for further review. The AI system acts as a supporting tool, but there is always a human making the decision if you will be considered for the role.In the United States, Thomson Reuters offers a comprehensive benefits package to our employees. Our benefit package includes market competitive health, dental, vision, disability, and life insurance programs, as well as a competitive 401k plan with company match. In addition, Thomson Reuters offers market leading work life benefits with competitive vacation, sick and safe paid time off, paid holidays (including two company mental health days off), parental leave, sabbatical leave. These benefits meet or exceeds the requirements of paid time off in accordance with any applicable state or municipal laws. Finally, Thomson Reuters offers the following additional benefits: optional hospital, accident and sickness insurance paid 100% by the employee; optional life and AD&D insurance paid 100% by the employee; Flexible Spending and Health Savings Accounts; fitness reimbursement; access to Employee Assistance Program; Group Legal Identity Theft Protection benefit paid 100% by employee; access to 529 Plan; commuter benefits; Adoption & Surrogacy Assistance; Tuition Reimbursement; and access to Employee Stock Purchase Plan.Thomson Reuters complies with local laws that require upfront disclosure of the expected pay range for a position. The base compensation range varies across locations. Eligible office location(s) for this role include one or more of the following: New York City, San Francisco, Los Angeles, and/or Irvine, CA; McLean, VA; Washington, DC. The base compensation range for the role in any of those locations is $146,800 USD - $272,600 USD. For any eligible US locations, unless otherwise noted, the base compensation range for this role is $127,400 USD - $236,600 USD. For Ontario, Canada, the base compensation range for this role is $140,000 CAD - $175,000 CAD. Base pay is positioned within the range based on several factors including an individual’s knowledge, skills and experience with consideration given to internal equity. Base pay is one part of a comprehensive Total Reward program which also includes flexible and supportive benefits and other wellbeing programs. This role may also be eligible for an Annual Bonus based on a combination of enterprise and individual performance. About UsThomson Reuters informs the way forward by bringing together the trusted content and technology that people and organizations need to make the right decisions. We serve professionals across legal, tax, accounting, compliance, government, and media. Our products combine highly specialized software and insights to empower professionals with the data, intelligence, and solutions needed to make informed decisions, and to help institutions in their pursuit of justice, truth, and transparency. Reuters, part of Thomson Reuters, is a world leading provider of trusted journalism and news.We are powered by the talents of 26,000 employees across more than 70 countries, where everyone has a chance to contribute and grow professionally in flexible work environments. At a time when objectivity, accuracy, fairness, and transparency are under attack, we consider it our duty to pursue them. Sound exciting? Join us and help shape the industries that move society forward.As a global business, we rely on the unique backgrounds, perspectives, and experiences of all employees to deliver on our business goals. To ensure we can do that, we seek talented, qualified employees in all our operations around the world regardless of race, color, sex/gender, including pregnancy, gender identity and expression, national origin, religion, sexual orientation, disability, age, marital status, citizen status, veteran status, or any other protected classification under applicable law. Thomson Reuters is proud to be an Equal Employment Opportunity Employer providing a drug-free workplace.Thomson Reuters makes reasonable accommodations for applicants with disabilities, including veterans with disabilities, and for sincerely held religious beliefs in accordance with applicable law. If you reside in the United States and require an accommodation in the recruiting process, you may contact our Human Resources Department at View email address on click.appcast.io. Disability accommodations in the recruiting process may include things like a sign language interpreter, making interview rooms accessible, providing assistive technology, or other relevant accommodations. Please note this email is not intended for general recruitment questions and we will promptly respond to inquiries regarding accommodations. More information on requesting an accommodation here.Learn more on how to protect yourself from fraudulent job postings here.More information about Thomson Reuters can be found on thomsonreuters.comSummaryLocation: United States of America, Eagan, Minnesota; United States of America, Frisco, Texas; United States of America, New York, New York; Canada, Toronto, Ontario; United States of America, Ann Arbor, MichiganType: Full time
$146.8k - $272.6k
## Lead Research EngineerApplyremote type: Hybridlocations: United States of America... .... We are seeking a Lead Research Engineer who will bring expertise in AI and ML and is interested in... ...collaborating with research scientists to evaluate, prototype and productionize...SuggestedWork at officeLocal areaRemote workFlexible hours2 days per week3 days per week$127.4k - $236.6k
...Job Description The Lead AI Forward Engineer designs and guides the delivery of AI-powered... ...and operationally reliable. You will evaluate emerging AI technologies, define... ...including offline and online tests, benchmarks, canaries, and A/B experiments. Work...SuggestedWork at officeLocal areaFlexible hours$96.4k - $144.7k
...looking for a Sr Innovation Engineer - Design who will lead the development,... ...RESPONSIBILITIES Lead evaluation and refinement of complex... ...of an Innovation Manager (Research Lead or Development Lead)... ...-aided drafting tools and AI enabled solutions Perform...SuggestedH1bWork at office$127k - $235k
...technology at Thomson Reuters Labs.We are seeking a Senior Research Engineer who will bring expertise in AI and ML and is interested in building data-driven... ...approach ensures you have the tools and knowledge to grow, lead, and thrive in an AI-enabled future.Industry...SuggestedFull timeWork at officeLocal areaRemote workFlexible hours$127.4k - $236.6k
...you passionate about building AI-driven software that... ...platform, CLEAR (Consolidated Lead Evaluation and Reporting), which empowers... ...global leader. As a Lead Software Engineer - AI, you will play a lead role... ...; stay current with AI research and translate insights into technical...SuggestedFull timeWork at officeLocal areaFlexible hours$127.4k - $236.6k
...legal, tax, accounting, and audit. As a Lead Software Engineer AI (Staff Engineer AI) on Cloud Audit... ...practices.Partner with AI/ML teams, researchers, designers, and product managers to... ..., including deployment, monitoring, evaluation, scaling, and ongoing operational...Full timeWork at officeLocal areaFlexible hours$124.13k - $151.71k
Job Description:Senior Research EngineerCollaborate with Innovative... ...RoleAs a Senior Research Engineer you will have the opportunity... ...laboratory concept through pilot evaluation and into manufacturing... ...and benefits, 3M regularly benchmarks with other companies that are...Full timeFlexible hours$89.6k - $157k
We are seeking a Lead Test Automation Architect to define... ...evolving our quality engineering capabilities to support Agentic AI / LLM-based systems (... ...hallucination detection, and model evaluation). This role will... ...based validationgoldens/benchmark datasets and evaluation...Full timeContract workWork at officeShift workNight shift$133.9k - $190.75k
...Dynamics 365 Technical Resource to lead configuration and... ...Background in business process re-engineering or continuous improvement initiatives... ...insights, and people-first AI, our ability to reveal unseen... ...equal opportunity employer. We evaluate qualified applicants without regard...Work at officeLocal area$158.9k - $295.1k
...Manager of Research Engineering, TR Labs Are you excited about the future of AI technology? Are you passionate about delivering value and maximizing the impact of... ...partnership with product, science, and the technical lead, prioritizing initiatives that enhance product...Full timeWork at officeLocal areaFlexible hours$91.7k - $163.7k
...Caring. Connecting. Growing together.As a Lead Software Engineer within Optum Technology, you will... ...solutions with a focus on Generative AI and engineering productivity. In this... ...other AI-powered technologiesExperience evaluating emerging AI frameworks, tools, and engineering...Minimum wageFull timeWork experience placementLocal areaRemote work$103.71k - $138.28k
...is the trusted network for the AI‑powered world, connecting... ...today. The Role The Field Engineering Quality Control Engineer is responsible... ...Key Responsibilities Lead logistics projects from... ...role. Background results will be evaluated on a case-by-case basis. Pursuant...Full timeTemporary workFor contractors- VLSI Physical Design Lead EngineerThis role has been designed as... ...analyzes, develops, modifies and evaluates VLSI components and hardware... ...for multiple project teams and engineers. Leverages recognized VLSI... ...QualificationsExperience navigating AI tools and methodologies to run...Full timeWork experience placementWork at officeLocal areaImmediate start2 days per week
$118.4k - $219.8k
Manager, Software Engineering Content Tools • CoCounsel... ...Software Engineering to lead the engineering team... ...capabilities that give AI agents precise, reliable... ..., CI/CD pipelines, evaluation infrastructure, and cloud... ...developer experience benchmarks — including tool...Full timeContract workWork at officeLocal areaFlexible hours$118.4k - $219.8k
...Thomson Reuters is investing in a dedicated security engineering capability, and we are looking for a hands-on technical lead to help build and shape it. You will design how... ...SLA-driven burndown), and champion adoption of AI-augmented security tooling, including SAST and...Full timeWork at officeLocal areaFlexible hours$127.4k - $236.6k
...Lead Software Engineer, AI What if the AI agents you architect today are empowering legal professionals at firms large and small to act more decisively... ...platform services. Design agent observability and evaluation practices (tracing, automated evals, and quality/drift...Full timeWork at officeLocal areaFlexible hoursShift work2 days per week3 days per week$124.13k - $151.71k
Job Description:Software Engineering Technical LeadCollaborate with Innovative... ...Engineering Technical Lead, you will have the... ...development resources.Proficiency with AI software development tools.Work... ...and benefits, 3M regularly benchmarks with other companies that are...Full timeH1bFlexible hours$100k - $170k
...the opportunity? The Agent Lead sits at the forefront of... ...productivity through agentic AI workflows-bridging business problems... ...ecosystem. This is not a pure engineering role-it's a product-minded builder... ...workflow opportunities and evaluate when AI is the right solution...Full timeFlexible hours- ...The beginning of a new Data & AI decade that will reshape work... ...Models, Recommendation Engines, Optimization Models, NLP ApplicationsPerform... ...engineering, model tuning, evaluation, and deployment.Implement end... ....Develop AI evaluation and benchmarking frameworks.Agentic AIBuild...Work experience placementLive inWork at officeLocal area
- ...and seeking to hire an experienced Lead Modern Workplace SharePoint Engineer to join our talented Information Technology... ..., and continuous improvement.Evaluate emerging Microsoft 365 technologies... ...recommendations regarding SharePoint, Copilot, AI-enabled collaboration, and employee...Full time
$125k - $175k
...transforming how brands grow. We partner with leading names like American Eagle, Best Buy,... ..., USA Today, Inc. 5000, and Search Engine Land.About the team:The AI OS & Agentic Execution team builds... ...Set the quality bar Define the evaluation framework the team uses for agent...Full timeWork at officeRemote workVisa sponsorshipWork visa$110k - $165k
...future of our communities. This is a Lead Cloud & Infrastructure Engineering position at Vice President Level,... ...resiliency, security, automation, and AI adoption efforts, partnering with... ..., and disposal.Lead hardware evaluations for major technology refreshes, covering...Temporary workLocal areaWeekend work- ...maintain scalable Python APIs that support platform features and AI workflows.Implement AI orchestration and agent workflows using... ...Instrument applications for monitoring, logging, tracing, and AI evaluation using Splunk and Arize AX.Write automated unit, integration, and...
$87.1k - $157.45k
Leidos is seeking Lead Systems Engineers to join the Air Traffic Business Area within the Homeland Sector, supporting the development of next-generation... ...Java, Python, React, and scripting languagesKnowledge of AI-assisted development workflows and tools (i.e. Claude Code,...Full timeRemote work$120k - $125k
...preferred. Join Our Dynamic Research Team We are looking for a Research Engineer II to join our growing Research... ...responsible for identifying and evaluating emerging and innovative energy efficiency... ..., you will be responsible for leading research projects, leading...Hourly payFull timeFor contractorsWork at officeLocal areaImmediate startRelocationFlexible hoursShift work2 days per week1 day per week$88.9k - $155.5k
...and in the future.SummaryIn this role, you will lead the day-to-day execution and delivery of AI Quality Engineering initiatives supporting Vizient's enterprise AI... ...preferred.Familiarity with AI-assisted testing, AI evaluation approaches, prompt testing, runtime...Full timeFor contractors$201.6k - $374.4k
...President, Talent Acquisition - Product Engineering will lead the recruiting organization... ...specialized capability focused on the AI, engineering, research, and other highly competitive technical... ..., interviewing, or candidate evaluation practices.Location: this is a hybrid...Full timeWork at officeLocal areaRemote workFlexible hoursShift work- ...possess strong, recent experience in: Generative AI Agentic AI RAG (Retrieval-Augmented Generation... ...Vector Databases LLM Integration AI Platform Engineering Production AI Deployment AI Evaluation Frameworks Prompt Engineering Observability &...
$188.25k - $230.08k
...Market Development Engineer Collaborate with... ...Systems Division will lead Voice of Market... ...hypotheses, apply AI-enabled tools where... ...Market/Customer research, or growth strategy... ...synthesis, opportunity evaluation, and growth... ...benefits, 3M regularly benchmarks with other...Full timeH1bFlexible hours$124k - $280k
...SummaryThe OpportunityAs a SAP Tech Lead (App dev, BTP, Integration, SAP AI) you will lead client work across... ...into SAP technical solutions by evaluating application patterns, integration... ...Computer Science/Information Systems, Engineering- Excelling in SAP S/4 HANA...Full timeH1b
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Research Engineer - AI Evaluation & Benchmarking. Be the first to apply!
- lead engineer Eagan, MN
- lead operating engineer Eagan, MN
- rn research Eagan, MN
- remote legal research Eagan, MN
- lab research Eagan, MN
- education policy research Eagan, MN
- research and development Eagan, MN
- research and development internship Eagan, MN
- research nurse Eagan, MN
- research registered nurse Eagan, MN




