Data Infra - Signals and Evaluations

Responsibilities

• Build versioned workflows for dataset collection, curation, filtering, labeling, ground truth, synthetic data, hard-negative mining, contamination controls, and train-evaluation isolation.
• Design benchmark suites for perception, multimodal reasoning, physics-informed prediction, retrieval, planning, tool use, world modeling, and customer tasks.
• Create simulation and environment infrastructure for reinforcement learning, imitation learning, offline learning, model-based learning, and agent training.
• Define repeatable evaluations covering accuracy, calibration, generalization, robustness, safety, latency, cost, and operational outcomes.
• Convert field, mission, customer, and experimental failures into versioned datasets, benchmark cases, adversarial tests, and environment scenarios.
• Build automated promotion gates and communicate tested capabilities, unresolved gaps, and release evidence.

Requirements

• Experience in ML evaluation, dataset engineering, reinforcement-learning environments, simulation, scientific ML, test infrastructure, or production AI systems.
• Strong foundations in statistics, experimental design, benchmark validity, distribution shift, calibration, reward design, and evidence-based acceptance.
• Fluency in Python and modern machine-learning frameworks, with software engineering discipline for building tested modules, data contracts, or services.
• Experience converting qualitative model or agent failures into reproducible datasets, tests, or environment scenarios.
• Ability to reason across acquisition, transformation, indexing, orchestration, and presentation in the complete data path.
• Preferred qualifications include experience with dataset curation, benchmark platforms, deep reinforcement learning, world models, synthetic data, multimodal evaluation, scientific validation, PyTorch, JAX, distributed evaluation, experiment tracking, environment frameworks, model and agent observability, uncertainty quantification, calibration, out-of-distribution detection, conformal methods, or selective prediction.
• Experience evaluating models deployed to aircraft, satellites, robotics, industrial systems, or other constrained environments is preferred.
• Applicants must be U.S. citizens or nationals, lawful permanent residents, or eligible to obtain required U.S. Department of State authorizations to satisfy ITAR requirements.

Benefits

• Competitive compensation based on experience and an early-stage equity package.
• 100% employer-paid health, dental, and vision coverage.
• Opportunity to work on novel sensing, data, and AI systems with real-world deployment paths.
• Based in San Francisco, California, with required onsite work.