Staff Software Engineer, Search & Retrieval Infrastructure

Role Description

We are seeking a Staff Software Engineer to take ownership of our advanced search, indexing, and data scanning infrastructure as we scale to the next echelon of data volume.

• Own the constellation of scaling challenges as ingest sizes push into the multi-petabyte range.
• Continuously optimize and evolve systems to ensure hot indexes maintain sub-second latency for interactive workflows.
• Design highly concurrent, cost-effective architectures for deep scanning and vectorizing massive volumes of cold-storage data.
• Lead the design and implementation of sophisticated data tiering and retrieval strategies to maintain peak performance without inflating cloud compute costs.

Responsibilities

• Scale the Retrieval Stack: Lead the optimization and architectural evolution of our existing hybrid search infrastructure, maximizing throughput and efficiency of both lexical search (e.g., Elasticsearch, Lucene) and dense vector databases.
• Advanced Data Tiering & Scanning: Design and implement intelligent, cost-effective tiering strategies across hot, warm, and cold data states. Evolve distributed pipelines to efficiently execute asynchronous, massive-scale scans of petabytes of data.
• Relentless Optimization: Drive down latency and cost-to-serve by analyzing system bottlenecks, tuning indexing and querying algorithms, and optimizing cloud infrastructure for maximum efficiency at extreme scale.
• Technical Leadership: Act as the domain expert and owner of the indexing and search ecosystem. Set the long-term technical vision for data storage and retrieval, guiding engineering teams on best practices for high-volume data modeling and performance tuning.
• Resiliency at Scale: Ensure fault-tolerant, highly available operations during massive parallel ingest events and complex, concurrent querying across millions of documents.

Qualifications

• Extreme Scale Experience: 8+ years of software engineering experience, with a proven track record operating at the Staff/Principal level optimizing and scaling highly distributed, high-throughput systems to handle petabyte-level data.
• Search & Vector Mastery: Deep, production-level expertise tuning and scaling Lucene-based search engines (Elasticsearch, Solr) and modern vector indexing infrastructure.
• Cost-Aware Architecture: Strong history of managing the compute vs. storage trade-off, designing sophisticated cold-storage scanning solutions and hot-index architectures that are highly performant and cost-effective.
• Distributed Systems: Extensive experience managing complex data pipelines, high-throughput event streaming (Kafka, Kinesis), and distributed compute architectures handling billions of records.
• Cloud Infrastructure: Expert command of cloud primitives (GCP preferred), Kubernetes, and infrastructure-as-code.
• Languages: Expert-level proficiency in systems-level and backend languages (Go, Rust, Python, or Java/C++).

Requirements

• Salary Range: $190,000 - $230,000 USD plus health insurance and equity.
• Location: United States - Remote Pay Range.