Software Engineer, Data Infra

Responsibilities

• Design, build, and operate scalable, fault-tolerant infrastructure for distributed LLM research workloads across compute, orchestration, and multimodal storage.
• Develop high-throughput systems for data ingestion, processing, transformation, training-data catalogs, deduplication, quality checks, and search.
• Build traceability, reproducibility, and quality-control systems across the data lifecycle.
• Implement and maintain monitoring and alerting for platform reliability and performance.
• Collaborate with research teams to improve data quality, accelerate training cycles, and deliver new capabilities.
• Own projects end-to-end and contribute across different technology stacks and teams.

Requirements

• Bachelor’s degree or equivalent experience in computer science, engineering, or a similar field.
• Proficiency in at least one backend language, specifically Python or Rust.
• Fluency with distributed compute frameworks such as Apache Spark or Ray.
• Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines.
• Preferred hands-on experience with Kafka, dbt, Terraform, and Airflow.
• Preferred experience building a web crawler.
• Preferred extensive experience scaling deduplication, data mining, and search systems.
• Strong knowledge of file formats and storage systems such as Parquet and Delta Lake.
• Ability to work collaboratively with cross-functional partners, take initiative, and own projects end-to-end.

Benefits

• The role is based in San Francisco, California.
• Visa sponsorship is available, with the company committed to working through the visa process for the right fit, though success is not guaranteed for every candidate or role.
• Generous health, dental, and vision benefits.
• Unlimited paid time off.
• Paid parental leave.
• Relocation support as needed.