Research Crawling Engineer

Job Description
The employer is a decentralized, Solana-based web-scraping network that allows users to monetize their unused internet bandwidth. By installing a browser extension, users securely share bandwidth to help AI companies crawl the web for public data, receiving Points (convertible to crypto tokens) as compensation.

They also operate a massive distributed crawler, giving them unique access to high-quality public web data at global scale.

They are hiring a Research Crawling Engineer (Full remote - USA/EU 6 hour overlap with EST)


You will join a company at the forefront of developing a web-scale crawler and knowledge graph that improves access to public web data and extends the value of AI to the people.

As a Research Crawling Engineer, you will design and operate large-scale web data acquisition systems for research and model development. You will work will span distributed systems, scraping infrastructure, and data pipelines.

This Role Involves:


- Operating at the boundary of scale and reliability

- Adapting to constantly changing web environments

- Balancing throughput, coverage, and data quality

- Owning end-to-end data acquisition pipelines

MISSIONS

• Design high-throughput, fault-tolerant systems for data collection (millions to billions of URLs/day)
• Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites
• Develop pipelines for cleaning, deduplication, filtering, and normalisation
• Construct and maintain datasets for research and model training
• Monitor crawl performance, coverage, and data quality; iterate quickly
• Collaborate with research teams to align data collection with modeling needs
• Optimize infrastructure for cost, latency, and reliability

Example Projects you could work on :


- Build a distributed crawler for a continuously updated, high-quality web project

- Design a system to classify and filter billions of pages for pretraining

- Extract structured data from dynamic, JS-heavy sites at scale

- Improve deduplication and quality scoring across multimodal datasets

Requirements

• Strong programming experience in one or more of : Go, Rust, Python, Java, or C++
• Experience working for reputable companies
• Experience building and maintaining large-scale web crawlers or large-scale data pipelines
• Experience designing high-throughput, fault-tolerant systems for data collection (millions to billions of URLs/day)
• Experience handling anti-bot systems, rate limits, and dynamic/JS-heavy sites
• Experience constructing and maintaining datasets for research and model training
• Solid understanding of networking, and browser behavior
• Familiarity with distributed systems and parallel processing
• Experience working with large datasets (TB-PB scale preferred)
• Ability to debug unstable or adversarial environments

Preferred / Bonus:

• Experience with NLP pipelines or dataset curation for ML
• Familiarity with LLM pretraining data or retrieval systems
• Experience with headless browsers (e.g., Chrome DevTools Protocol, Playwright, Puppeteer)
• Knowledge of proxy systems, IP rotation, and large-scale request orchestration
• Background in data quality evaluation or benchmarking
• Experience running workloads on cloud or bare-metal infrastructure

Main Evaluation Criteria:

• Ability to design systems that scale without degrading quality
• Practical problem-solving under real-world constraints
• Speed of iteration and ownership
• Measurable improvements in data coverage, quality, or efficiency

Benefits

• Contract : Permanent role (Full remote - USA or 6 hour overlap with EST).
• Salary : $150k to $225k based on experience and demonstrated ability to operate at scale + Equity package / tokens

Recruitment process :

• Recruiter / HR Call
• Technical Interview
• CEO Interview
• Final Interview