Founding Engineer, RL & Evals
$150k - $200kRuntime Labs, Inc.
- Founding Engineer, RL & EvalsRuntime Labs builds Óra at ora.app.As a Founding Engineer focused on RL & Evals, you will own how Runtime Labs measures model behavior in Óra: planning, scheduling, recommendations, memory use, tool use, and long-horizon interaction. You will define what good looks like, build the datasets and harnesses that score it, and close the loop so evaluation signal drives product decisions, model selection, and—when the data justifies it—post-training and reinforcement learning.This is not generic chatbot evaluation. The domain is agentic behavior over time: whether a model uses the right context, makes good plans, chooses appropriate actions, recovers from mistakes, and improves as interaction data compounds.## About the Role* Own the eval roadmap for planning, scheduling, recommendations, memory, tool use, and long-horizon model behavior in Óra* Define task- and system-level metrics that separate useful behavior from superficially plausible outputs—including temporal reasoning, retrieval quality, action selection, constraint satisfaction, and recovery from failure* Build and version evaluation datasets from synthetic scenarios, curated examples, production failures, and longitudinal interaction; turn real failures into durable regression cases* Design dataset schemas that preserve context, expected behavior, provenance, annotations, and outcomes—with clear conventions for sampling, contamination control, and reproducible comparisons* Build developer-friendly harnesses for writing, running, inspecting, and comparing evals; integrate them into model experimentation and everyday product engineering* Support offline replay and controlled comparison across models, prompts, retrieval strategies, and agent policies; make failures traceable from interaction through retrieval, reasoning, tool calls, and final action* Establish release criteria so model and product changes are measured before they reach users* Close the loop from eval signal to shipping decision: prioritize regressions, guide prompting/retrieval/architecture choices, and develop preference or reward signals from evaluation, user behavior, and longitudinal outcomes when warranted* Explore post-training and reinforcement-learning approaches when evaluation quality and interaction data justify them* Partner with Product, Infrastructure, and Data Infrastructure so production interaction becomes structured learning signal## About the Person* Research-minded engineer who cares deeply about measurement, experimental rigor, and why model behavior changes* Strong ML foundation with experience evaluating, training, or improving language-model or agentic systems* Moves between research questions and production engineering: define the experiment, build the dataset, implement the harness, analyze failures, ship the change* Strong intuition for evaluation design and the failure modes of automated metrics, model-based graders, and human labels* Reasons precisely about noisy signals, baselines, distributions, regressions, and experimental validity* Strong software engineer who builds durable evaluation infrastructure—not one-off notebooks or manual review alone* Interested in reinforcement learning, preference optimization, post-training, agent evaluation, and learning from interaction* Excited by planning, temporal reasoning, recommendations, memory, and models that act over long-lived user context* Wants high ownership as an early long-term partner defining how Runtime Labs measures and improves intelligent behavior## Experience* Experience building evaluation systems for LLMs, agents, recommendations, search, ranking, or decision-making systems* Experience with reinforcement learning, preference optimization, reward modeling, or post-training* Experience building datasets, annotation systems, graders, replay infrastructure, or experimentation platforms* Familiarity with model tracing, tool-use evaluation, retrieval evaluation, and multi-step agent benchmarks* Experience turning production model failures into reproducible test cases and measurable improvements* Background in sequential decision-making, recommendation systems, temporal reasoning, or human-AI interaction## Location & CompensationLocation: San Francisco. We are building the founding team in person and expect to work closely together during the company's early formation.Salary: $150k–$200k depending on experience and role.Equity: meaningful early-stage ownership for long-term partners.Exceptional candidates may be considered outside this range.We encourage you to apply even if you do not meet every qualification.
- J-18808-Ljbffr Runtime Labs, Inc.
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Founding Engineer, RL & Evals in Eastern, KY vacancy
- ...is open source, because the community should be able to outlive the company that started it. About the Role This is a founding engineering role, and a senior one. You’ll own large parts of the Rust systems that make all of the above run, working directly with our...SuggestedRemote work
Blacksky Algorithms
Eastern, KY14 hours ago $180k - $250k
# Founding Engineer* New York, United States* Full-timeApply for this role## CompensationCompensation, location, and equity## About the companyOur client is building the operating system for agentic commerce: infrastructure that turns a user's intent to buy into a completed...SuggestedFull timeInternshipH1bRelocation packageRaydar Inc
Eastern, KY1 day ago- ...estimate for this role and may vary based on experience and leveling. Job Description About the Role We’re hiring a Founding Engineer to help architect and scale the core platform at Proaction. This is a hands-on, high-ownership role for someone who operates at...Suggested
Pro Action
Eastern, KY4 days ago - ...that read and act on customer data, and an engineering team shipping into all of it every day.... ...exfiltration paths through agents, and the evals that tell you whether your controls hold.... ...you can walk us through: bugs you found, incidents you ran, systems you hardened,...Suggested
- Linkup is seeking a founding Developer Relations Engineer to shape how developers discover, understand, and build with Linkup. You will create demos, SDKs, content, and experiences that inspire builders and accelerate adoption across the US. This is a hands-on, outward-...SuggestedLocal area
- Grex Defense, Inc. is seeking engineers to advance control and decision-making for autonomous systems. You will blend classical control with learning-based methods to keep systems stable and dependable in real-world scenarios. The role focuses on robotics, GNC, and autonomy...
- ...York (in-person) Reports to: Head of Sales Type: Full-time, founding team About Lighthouse Lighthouse is the fastest immigration solution... ...and Atomic Semi. The role We’re looking for a Founding Growth Engineer to join Lighthouse. This is a founding GTM role: you’ll be the...Full timeWork at officeVisa sponsorship
- ...the global leader in vehicle service equipment. It is one of the founding companies of Dover Corporation, an eight billion dollar... ...and injury to the employee and others.The Automation Controls Engineer is an individual contributor responsible for specification, design...Full timeFor contractorsWork experience placementWorldwideFlexible hours
DESTACO
Eastern, KY4 days ago $135k - $185k
...Back Senior Automation & Controls Engineer Engineering Charlottesville , Virginia Full-Time Aug 26, 2026 About the Role: Join... ...TAI are capable, well-rounded, flexible, and optimistic. We have found that the best employees are the ones who recognize the importance...Full timeTemporary workFor contractorsFlexible hoursTai Corp
Eastern, KY4 days ago- ...unique, practical approach to nuclear development leverages modern software engineering to rapidly deliver safe, factory-built microreactors that use existing, well-qualified materials. Founded in 2020, Radiant is on track to test its first reactor at the Idaho National...Full timeSummer workRemote workFlexible hoursWeekend work
Radiant Industries, Inc.
Eastern, KY4 days ago $120k - $150k
...Overview Integral Federal is seeking a Test Automation Engineer to develop and maintain automated regression, API, integration, and... ...important challenges and deliver positive outcomes. Since our founding in 1998, we have helped clients leverage existing and emerging...Temporary workImmediate startFlexible hoursShift workIntegral Federal, Inc.
Eastern, KY4 days ago$245k - $300k
...we build the platform that turns that judgment into the data, evals, and RL environments frontier models learn from. We work with leading... ...keep it healthy once it's running. You sit between Pareto's engineering team and the researchers at the labs we work with, close enough...$112k - $140k
...the generations to come.**We are looking for:**A Process Control Engineer to provide technical expertise and day-to-day controls and... ...solutions contribute to safer, cleaner, and more sustainable products found in homes, food and consumer goods, planes, cars, batteries,...Local areaRelocation packageSyensqo SA
Eastern, KY14 hours ago$116k - $145k
...future of aerospace and defense starts here. Ursa Major was founded to revolutionize how America and its allies access and apply high... .... We are seeking a Senior Instrumentation & Controls Engineer to own sensor selection and implementation of automated control...Permanent employmentFull timeTemporary workIndustrious Ventures
Eastern, KY4 days ago- Mercor, a leading AI data company, seeks a strong engineer to build agentic products that scale across backend, frontend, data, and infrastructure. You’ll work with researchers, operators, and AI companies to push frontier models forward, joining a CEO-facing but team-...
$100k - $130k
...SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where... ...ultimate goal of enabling human life on Mars. CONSTRUCTION ENGINEER As a member of the HQ construction team, you will design,...Permanent employmentTemporary workFor contractorsWork at officeWeekend workSPACE EXPLORATION TECHNOLOGIES CORP
Eastern, KY4 days ago$167k - $209k
...rapidly, and stay ahead of evolving threats. CHAOS Industries was founded in 2022 and has raised a total of $1 billion in funding from... ...: CHAOS is seeking a hands‑on Staff Mechanical Design Engineer to lead the design and integration of air‑launched and ground‑launched...Full timeWork experience placementCasual workRemote workRelocation packageChaos Inc
Eastern, KY4 days ago$115.2k - $172.8k
...leading-edge semiconductor manufacturing. As a Staff Mechanical Engineer, you will lead the design and development of precision... ...Innovation supports customers with a worldwide sales and service organization. Additional information can be found at #J-18808-LjbffrPermanent employmentWorldwideOnto Innovation
Eastern, KY3 days ago- ...monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It’s all oriented to prevention. Summary Serve as a strategic quality leader, define long-term testing...Work at officeLocal areaWorldwide3 days per week
OCLC
Eastern, KY4 days ago - ...Electrical Engineer - Central Maintenance Remote Position: No September 4th, 2026 Internship: No No Job Summary:... ...products manufactured in the US, France and Switzerland can be found in platforms manufactured by Airbus, Applied Materials, Blue Origin...InternshipRemote work
Constellium Deutschland GmbH
Eastern, KY4 days ago $156.8k - $219.52k
...spaceflight! This role is part of Advanced Concepts and Enterprise Engineering (ACE), supporting Blue Origin’s mission of millions of people... ...for your interest in working at Blue Origin! Blue Origin was founded with a vision to enable a future where millions of people are...Permanent employmentTemporary workLocal areaImmediate startBlue Origin
Eastern, KY4 days ago$160k - $210k
...this role: Robust AI is a fast-growing, early-stage startup founded in 2019 by an unsurpassed team of veterans in robotics, AI and... ...robust, safe, flexible and genuinely autonomous. As a mechanical engineer at Robust AI you will join a small fast-paced team working to...Flexible hoursRobust
Eastern, KY4 days ago- ...structural variation analysis to laboratories of all sizes. The company was founded in 2005 and is based in Providence, Rhode Island. For more information, visit Title: Principal Mechanical Engineer Nature of Role: The Principal Mechanical Engineer is a senior...
Nabsys 2.0, LLC
Eastern, KY1 day ago $148.5k - $223.9k
...Automation team is a critical part of Salesforce's infrastructure engineering organization, responsible for coordinating and securing build... ...purchasing program. More details about company benefits can be found at the following link: ### ### ### At Salesforce, we believe...Full timeWork at officeSalesforce
Eastern, KY4 days ago$100k - $130k
...SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one... ...Mars. AVIONICS MANUFACTURING The Avionics Manufacturing Engineering team is a highly interdisciplinary organization that is responsible...Permanent employmentTemporary workSPACE EXPLORATION TECHNOLOGIES CORP
Eastern, KY4 days ago- ...unique, practical approach to nuclear development leverages modern software engineering to rapidly deliver safe, factory-built microreactors that use existing, well-qualified materials. Founded in 2020, Radiant is on track to test its first reactor at the Idaho National...Full timeSummer workRemote workFlexible hoursWeekend work
Radiant
Eastern, KY1 day ago $190k - $260k
...rapidly, and stay ahead of evolving threats. CHAOS Industries was founded in 2022 and has raised a total of $1 billion in funding from... .... Role Overview: CHAOS is seeking a Principal Electrical Engineer with deep expertise in combined power and analog board design...Full timeWork experience placementCasual workRelocation packageChaos Inc
Eastern, KY1 day ago$105k - $140k
...Huntington Beach, California, United States About Mach Industries Founded in 2023, Mach Industries is a rapidly growing defense... ...Mach Industries is seeking a Forward Deployed Autonomous Systems Engineer (FDASE) to support the integration, testing, deployment, and...Permanent employmentWork experience placementWork at officeLocal areaRemote workWeekend workMach Industries
Eastern, KY2 days ago- ...Full-stack engineer who understands vibe coding tools and how recent AI frameworks work. You'll build both frontend and backend for our enterprise workflow automation platform. Responsibilities Architect and build both frontend and backend systems powering our...
Carnot
Eastern, KY2 days ago $134k - $179k
...to accelerate breakthroughs and turn compute into capability. Founded in 2017, CoreWeave became a publicly traded company (Nasdaq: CRWV... ...at . About the Role As a Senior Security Operations Engineer I, you will independently lead the investigation and response to...Permanent employmentFull timeTemporary workFor contractorsCasual workWork at officeFlexible hoursNight shiftWeekend workCoreWeave
Eastern, KY2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Founding Engineer, RL & Evals. Be the first to apply!
