Distributed Web Crawling Engineer for Large-Scale AI Data
Reflection
Reflection in San Francisco is seeking an engineer to build and operate web-scale systems for data collection. This role involves working closely with AI researchers to optimize crawling infrastructure and enhance data quality. The ideal candidate will have a strong background in distributed systems and web data processing. Competitive compensation and benefits are offered to ensure impactful work. #J-18808-Ljbffr Reflection
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Distributed Web Crawling Engineer for Large-Scale AI Data in San Francisco, CA vacancy
- ...shape the future of AI. Our mission:... ...About the Role The web is one of the... ...diversity of web data directly influence... ...build and operate large-scale web crawling systems that continuously... ...scheduling to distributed crawling, content... ...is ideal for engineers who love building...DataWork at officeVisa sponsorship
- causal is building a Large Physics foundation Model to enable... ...actions to alter it while scaling novel architectures across multimodal physical data. We are seeking infrastructure engineers who can design, implement, and optimize distributed training systems, profile performance...Data
- ...intelligence open and accessible. We seek an engineer to build and operate large-scale data ingestion systems for pre-training, turning open web data into reliable corpora for frontier... ...role suits those who love robust distributed systems and data-driven iteration. #J-...Data
- ...biggest Financial AI company in the... ...Staff Backend Engineer at Rogo, you'll... ...product and scaling it to massive enterprises... ...and build large-scale infrastructure to power crawling, ranking, and... ...with sensitive data. Write... ...building and scaling distributed systems. ~...DataFull time
Rogo
San Francisco, CA1 day ago - ...exceptional backend software engineer to help define and... ...Platform needs to scale.; this is a product... .... You will lead large cross-functional... ...routing, and distributed services, with strong... ...OpenAI OpenAI is an AI research and... ...possession (including the data contained therein)...DataFull timeInternship
OpenAI
San Francisco, CA1 day ago - ...that is building the AI backbone for the... ...for AI models”—not data or raw compute, but... ...a Backend Software Engineer (ML Infrastructure)... ...design, build, and scale the core systems that power large-scale model training... ...candidate will work on distributed training pipelines,...DataFull time
Rockstar
San Francisco, CA1 day ago - ...production at global scale and are used to... ...the hard part of AI-driven software... ...internal packages for engineering efficiency... ...in architecting large-scale, distributed, modular frontend... ...6+), TypeScript, Web APIs, browser internals... ...and client-side data architecture ~...DataFull timeRemote workFlexible hours
Emergent Labs
San Francisco, CA1 day ago - ...OpenAI’s Applications Engineering organization builds... ...trusted at global scale. About the Role... ..., and safety across distributed services. Implement data storage, retrieval,... ...services that handle large scale and complexity... ...OpenAI OpenAI is an AI research and deployment...DataFull timeWorldwide
OpenAI
San Francisco, CA1 day ago $100k - $130k
...startup disrupting the data privacy and... ...businesses and software engineers with a deploy-once... ...and running large scale systems in cloud native... ...about scalable distributed systems ~ Have built... ...for mobile, web and mixed environments... ...intelligence (AI) tools to support...DataFull timeKetch
San Francisco, CA1 day ago- ...scrappy product-focused engineer to improve our... ...experiment with AI. Across our product... ...some incredible data for improving developers... ...skills. We’re a distributed engineering team... ...driving large projects: 2+ past... ...decisions based on scaling or other constraints...DataWork at officeFlexible hours3 days per week
- ...EngineerSeeking a Senior Backend Engineer to architect and... ...analytics—from data processing pipelines... ...to work on complex distributed systems at enterprise scale.What You'll Do:... ...intensive applications and large-scale data... ...Work with advanced AI/ML algorithms, composite...DataCasual work
- A cutting-edge AI research firm in San Francisco is seeking talent to build and optimize GPU infrastructure for large-scale model inference and training workloads. The ideal candidate will... ..., actively contributing to synthetic data generation and reinforcement learning pipelines...Data
$220k - $240k
...OpportunityThe Forward Deployed Engineering (FDE) team tackles... ..., solve complex distributed systems challenges, and... ...to safely adopt AI at scale.This isn't consulting... ...and deploy Postman's data-path components into customer... ...production telemetry across large-scale distributed...DataWork at officeFlexible hours3 days per week$185k - $385k
...TeamOpenAI’s Applications Engineering organization builds... ...trusted at global scale.About the RoleWe’re... ..., and safety across distributed services.Implement data storage, retrieval,... ...that handle large scale and complexity... ...OpenAIOpenAI is an AI research and deployment...DataWork at officeLocal areaWorldwideFlexible hours- UCSF is seeking a Storage Systems Engineer to design, deploy, and operate large-scale storage for FAC and related platforms, ensuring seamless integration with the CoreHPC compute cluster. This role supports AI, data science, and computational research workloads across...Data
$126k
...About the team: The ML Data Engineering team powers metadata... ...operate at massive scale, supporting diverse datasets... ...data engineering, and distributed systems, collaborating... ...design and optimize large-scale data and service... ...-edge generative AI and metadata enrichment...DataFull timeLocal areaWorldwideHome officeFlexible hoursScribd, Inc.
San Francisco, CA1 day ago- ...bringing advanced AI capabilities to... ..., mobile engineers, frontend engineers... ...generation at global scale. Whether users... ...orchestration systems, and data infrastructure.... ...with modern web technologies... ..., APIs, and distributed systems. Enjoy... ...integration, and large-scale user experiences...DataFull timeWorldwide
OpenAI
San Francisco, CA1 day ago $193.8k - $285k
...enable DoorDash Backend engineers across our three... ...services at scale and with confidence... ...Your team will have a large scope of influence... ...engineering.Champion data-informed decision-making... ...and operating distributed systems and platform... ...Notice Regarding Use of AI and Automated Tools...DataHourly payWork at officeLocal areaImmediate startRemote workFlexible hours$300k - $320k
...interpretable, and steerable AI systems. We want AI... ...researchers, engineers, policy experts,... ...deploy Claude at scale, to the agentic tools... ...of complex distributed systems and user-facing... ...Claude access to their data while automatically... ...systems that large organizations require...DataFull timeWork at officeVisa sponsorshipFlexible hoursAnthropic
San Francisco, CA1 day ago- ...'re not just building AI features, we're building... ...on Rippling's unique data and workflows. The... ...the company.As a Senior Engineer on AI Platform, you'll... ...expertise in platform and distributed systems engineering —... ...reliable, observable, large-scale systemsStrong hands-on...DataWork at office3 days per week
$210k - $285k
...Description Senior Backend Engineer San Francisco, CA... ..., venture-backed AI company automating... ...still running largely on manual,... ...You’ll own the core data infrastructure behind... ...— designing and scaling the systems that ingest... ...and scale distributed backend systems —...DataFull timeH1bRelocationVisa sponsorshipDavid Joseph & Company
San Francisco, CA4 days ago- ...Senior Backend Engineer San Francisco • On... ...learning in long-horizon AI agents. The next... ...into high-quality data that can actually... ...data at production scale. This role... ...you'll be deep in distributed systems and query performance... ...scale. A single large customer should not...DataFull time
Judgment Labs
San Francisco, CA5 days ago $120k - $140k
...Ketch is a Series B data privacy B2B SaaS... ...data use for the AI era. The Ketch Data... ...businesses and software engineers with a deploy-once... ...and running large scale systems in cloud native... ...about scalable distributed systems ~ Have... ...complexities for mobile, web and mixed...DataFull timeKetch
San Francisco, CA9 days ago$200k - $220k
...high-level problems into engineering solutions. You’re able... ...to enable Finch’s data presentation layer.Lead... ...reliably and efficiently at scale.Lead large product initiatives... ...working with distributed systems.Experience working... ...artificial intelligence (AI) tools to support...DataFull timeWork at officeRemote work2 days per week- ...Infrastructure Software Engineer to help us develop AI-based autonomous... ...software platform and data pipeline that supports... ...Knowledge of distributed and micro-service systems... ...processing pipelines at scale Knowledge of fundamental... ...working with large data sources and data...DataFull timeWork at officeImmediate start
OSARO
San Francisco, CA2 days ago $123.7k - $254.67k
...Possible.At Pinterest, AI isn't just a feature,... ...well-rounded Backend engineers to join our Core, Monetization... ...managers, designers, data scientists, and... ...and experiences as we scale and evolve. You will build... ...develop, and operate large scale, distributed systems and...DataWork at officeLocal areaRelocationRelocation package$166k - $225k
...passionate about enabling data teams to solve the... ...'s best data and AI infrastructure... .... Founded by engineers — and customer obsessed... ...with data to scaling our services and infrastructure... ...learning and distributed systems.What we... ...developing large-scale distributed...DataLocal areaWorldwide- LiveRamp is the data collaboration platform... ...a Senior Software Engineer to help shape the next... ..., build, and scale core Data Marketplace... ...workflows.Lead medium-to-large technical... ...to the evolution of AI-forward marketplace... ...delivering scalable distributed systems, APIs, and...DataFull timeWork from homeFlexible hoursNight shift
$167.7k - $245.2k
...massive amounts of data from its devices... ...popularity comes with large scalability... ...you will develop, scale, and maintain the... ...different people within Engineering and throughout... ...maintain scalable, distributed, enterprise software... ...organizations in the AI era - and beyond....DataPermanent employmentFull timeTemporary workLocal areaRemote workFlexible hours- ...responsible for the reliability, scale, performance, and cost... ...Networking, DocumentDB, Distributed Caching and High assurance engineering). We build the... ...latency microservices to large-scale data pipelines and machine learning... ....Developer Productivity AI—We're responsible for...Data
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Distributed Web Crawling Engineer for Large-Scale AI Data. Be the first to apply!
Related searches
- entry web developer San Francisco, CA
- web developer specialist San Francisco, CA
- web associate San Francisco, CA
- web programmer San Francisco, CA
- work from home web developer San Francisco, CA
- junior web developer internship San Francisco, CA
- react web developer San Francisco, CA
- web developer San Francisco, CA
- remote web developer apprenticeship San Francisco, CA
- content developer San Francisco, CA