Research Crawling Engineer
$160k - $250kFull-time
True Talent Solutions Partners
About the Role
Join a data and infrastructure team building large-scale web data systems for AI research and model development. You will help deliver reliable, high-quality datasets by operating crawlers across complex and changing web environments.
What You'll Do
Build and maintain large-scale crawlers across diverse domains, including social media, travel, and multilingual websites.
Design high-throughput, fault-tolerant collection systems capable of processing millions to billions of URLs per day.
Adapt crawling strategies to anti-bot measures, rate limits, and dynamic or JavaScript-heavy sites.
Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
Build datasets for research and model training in collaboration with research teams.
Monitor crawl coverage, performance, and data quality, and improve systems as web environments change.
Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.
What We're Looking For
Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years.
Strong experience with Go, Rust, Python, Java, or C++, and distributed systems or parallel processing.
Experience operating data pipelines at terabyte scale and processing millions to billions of URLs per day.
Deep understanding of networking, and browser behavior, including experience handling anti-bot systems, rate limits, and JavaScript-heavy websites.
Experience with headless browsers such as Playwright or Puppeteer, proxy systems, IP rotation, and request orchestration is valuable.
Background in data quality evaluation, web data processing, NLP pipelines, or machine-learning dataset curation is a plus.
Experience running data workloads on cloud or bare-metal infrastructure; a relevant technical degree is beneficial.
Compensation & Benefits
Base salary: $160,000 to $250,000 USD annually. Visa sponsorship is not available.
Location
Remote.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Research Crawling Engineer in Remote vacancy
- ...decision-making; just a team of builders pushing to expand what’s possible for open web data and AI. Overview: As a Research Crawling Engineer, you will design and operate large-scale web data acquisition systems for research and model development. Your work will...SuggestedRemote jobFull time
- ...the development of next-generation open models, the full-time Research Engineer will find and evaluate training data, build scalable data... ...end and collaborating with researchers Relevant experience in web crawling, document extraction, or large-scale corpus curation...SuggestedFull timeRemote work
Virtual Vocations Inc
United States1 day ago $50 per hour
...organization spread across the U.S. and Canada. Sony AI is Sony's new research organization pursuing the mission to use AI to unleash human... ...Summary We are seeking a motivated and passionate Research Engineer Intern to join our interdisciplinary team working at the...SuggestedHourly payInternshipLocal areaRemote workWorldwideSony
New York, NY4 days ago- ...expectation does not apply to contractor positions About the role and team We are looking for passionate early-career Research Engineers to join the Research Group at Applied Intuition. The mission of the group is to create cutting-edge technology enabling next...SuggestedFull timeFor contractorsFor subcontractorSummer workCasual workWork at officeImmediate startRemote workDay shift
Applied Intution
Sunnyvale, CA2 days ago $81.2k - $121.8k
...automotive and intelligent mobility. This is not an IC chip design engineer role. This role combines semiconductor expertise with field... ...flag weak signals and propose initiatives. Conduct approved research through visits, interviews, conferences, demonstrations, and...SuggestedFull timeTemporary workWork experience placementRemote workRelocation packageShift workHonda Dev. and Mfg. of Am.,LLC
Raymond, OHa month ago- ...time zones) Start Date: ASAP Languages: English Team: Engineering About the Role Our client is a fast-growing AI company... ...scaling quickly to meet strong demand. We are looking for a Research Engineer, Privacy and Anonymization to build the systems that...Full timeLocal areaImmediate startRemote work
$170k - $220k
...manufacturable CAD and runs simulation, and you will lead the research practice that decides which methods we commercialize next. The... ...serve them. You will work alongside the computational geometry engineer, who owns the exact geometry and the analysis tools your models...Full timeRemote workVisa sponsorship$175k - $195k
...The role Gauntlet is looking for an AI Research Engineer to help improve how we build, evaluate and use AI agents across the business. We already have agents running in production, including Aria, our in-house generalist agent. They're used across research, engineering...Full timeWork at officeRemote workWork from home- ...Research Engineer Rootshell Enterprise Technologies Inc. is a recognized provider of professional IT Consulting services in the US. We are actively seeking a Research Engineer for one of our clients. Job Description: Bachelor's degree in Computer Science, Engineering...Remote work
Rootshell Inc
United States1 day ago - ...Elevenlabs ElevenLabs is an AI research and product company transforming how we interact with technology. We launched in January... ...builders doing the best work of their lives. We are researchers, engineers, and operators. IOI medalists and ex-founders. If you want to...Immediate startRemote work
Eleven Labs
United States1 day ago $150k - $250k
...About Us FAR.AI is a non-profit AI research institute working to ensure advanced AI is safe and beneficial for everyone. Our mission... ...Staff, with significant overlap between scientist and engineer roles. As an engineer, you will develop scalable implementations...Full timeFor contractorsWork experience placementWork at officeRemote workWork from homeVisa sponsorshipFar
United States2 days ago- ...JOB DESCRIPTION We are in search of a Research & Development Engineering Co-op to join our Power & Water Solutions business, based in Cranberry Township, PA . We want to work with people that make learning and teaching others a fundamental part of every day...Work experience placementInternshipImmediate startRemote workFlexible hours
Emerson
Pittsburgh, PA3 days ago $75k - $90k
...Research EngineerFounded in 2017 and headquartered in Rochester, NY, REMADE? is a 170+ member public-private partnership funded in part... ...addition to managing the tire recycling program, the research engineer will support the broader mission of the Institute by fostering...Permanent employmentWork at officeREMADE Institute
Rochester, NY4 days ago$122k - $215k
...Research Engineer Waabi, founded by AI visionary Raquel Urtasun, is the leader in Physical AI. With a world-class team, we're unlocking the next era of autonomous transportation with technology that's powering commercial autonomous trucks and robotaxis. Waabi is backed...Full timeWork at officeRemote workWork from homeFlexible hoursG2 Venture Partners
United States3 days ago- ...The Cloud Technology Advancement Department of the Applied Research Laboratory (ARL) at Penn State University is seeking graduate... ...integrate supporting AI/ML models, agentic workflows, and agentic engineering services Develop HPC / cloud security principals, protocols...Part timeFor contractorsInterim roleRemote work
Penn State University
United States3 days ago - ...Research EngineerNew York - Hybrid; San Francisco Bay Area - HybridAbout InvisibleInvisible Technologies makes AI work. Our end-to-end... ...environments for frontier labs and enterprise clients. As a Research Engineer, you'll work on how frontier models are measured, meaning the...Full timeWork at officeLocal areaRemote work
Invisible Technologies Inc. Defunct
New York, NY4 days ago - ...Research EngineerSesame believes in a future where computers are lifelike - with the ability to see, hear, and collaborate with us in... ...are needed to realize all of our product goals. As a Research engineer at Sesame, you are a research-oriented person with experience in...Full timeContract workShift work
SESAME
Bellevue, WA4 days ago $180k - $300k
...DatologyAI Research EngineerAs a Research Engineer, you will play a crucial role in conducting and enabling cutting-edge research and translating it into our core product pipeline. You will work closely with other members of the technical staff to develop and improve state...Work at officeWork from homeRelocation packageOutset Capital
San Mateo, CA3 days ago- ...Research Engineer, Biological Safety and Orthopedic Devices Location: Remote, US-based Preferred location: Denver, CO, with onsite work possible for local candidates We are looking for a Research Engineer to support investigation, research, and remediation...Local areaRemote work
Mantu
Denver, CO2 days ago - ...Role Description This is a Research Engineer role focused on post‑training and reasoning. Full‑time or part‑time; we're open to candidates currently pursuing a PhD or Master's or working in industry. Your core responsibilities will include conducting research in post‑training...Full timePart timeRemote workFlexible hours
Cobalt.io
San Francisco, CA2 days ago - ...students to review reinforcement learning tasks and environments used to test whether frontier AI models can carry out real research and engineering work. Your job is to make sure each one is correct, fair, and actually measures what it claims to measure before it is...Contract workRemote workFlexible hours
Cobalt
United States1 day ago $180k - $250k
...the creator of PyTorch Lightning. We develop an end‑to‑end platform that helps bring AI from research to production more efficiently. What We're Looking For Research Engineer to optimize training and inference workloads on compute accelerators and clusters via the Lightning...Work at officeHome officeFlexible hours2 days per weekVoltage Park
New York, NY2 days ago$81.33k - $90k
...Position Summary We have an exciting opportunity to join our team as a Research Engineer. The successful applicant will join a team working on exciting projects focusing on low‑field MRI development, working with a diverse group of physicists, imaging experts, and radiologists...NYU Grossman School of Medicine
New York, NY12 hours ago$90k - $120k
...Research Engineer Photonic Integrated Circuit Development Job Location: Golden (Hybrid Remote) Job Type: Full Time Salary: $90,000-$120,000 based upon education, work experience, and skill Please send your resume with a cover letter to ****@*****.***....Full timeWork experience placementRemote workFlexible hoursVescent
Golden, CO20 days ago- ...building a world-class AI team from the ground up excites you, we'd love to talk.Minimum Qualifications3+ years of combined research and engineering experience in machine learning, including hands-on work with language models. Direct experience working with Large...Internship
Nace AI
Palo Alto, CA4 days ago $62k - $141k
...Job Number: R0249745 PNT Research Engineer The Opportunity: Are you passionate about developing Positioning, Navigation, and Timing ( PNT ) solutions? Do you enjoy working where science, engineering, and mission-driven innovation meet? If you are excited about...Full timeContract workPart timeWork at officeLocal areaRemote work- ...Research Engineer Intern Pluralis Research works on Protocol Learning: training and serving large models in a fully decentralized way on small consumer-grade devices connected via the internet. Despite being dismissed as infeasible, we have made significant advances...Full timeFixed term contractInternshipRemote work
Pluralis Research
United States3 days ago - ...strategic investors: industry leaders in construction, building products manufacturing, and distribution. Higharc is hiring a Research Engineer to join our Special Projects team. In this role, you'll conduct foundational research in spatial AI for residential...Full timeTemporary workRemote workHome officeFlexible hours
Higharc
United States5 days ago - ...development, optimization, and validation.Document results, maintain accurate records, and provide technical insights to engineering and research teams.Operate general machine shop and material-fabrication tools in accordance with safe operating procedures.Required QualificationsBachelor...Local areaWork from home
General Motors
Warren, MI3 days ago $180k - $250k
...Research Engineer San Francisco, CA In-person Full-time Most vision models are either accurate or fast. We need ours to be both, because people are running them on every frame of video in realtime. That constraint makes the work genuinely hard in a way that “scale up the...Full timeRemote workRelocation packageMoondream AI
San Francisco, CA1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Research Crawling Engineer. Be the first to apply!
Related searches
- ai research engineer Remote
- deep learning research engineer Remote
- research software engineer Remote
- research programmer Remote
- research engineer Remote
- junior machine learning research engineer Remote
- research assistant engineering Remote
- senior research engineer Remote
- brain research Remote
- physics research Remote