Data Ingestion Engineering Lead: Web Crawling & Pipelines
Reflection
Reflection is seeking a Data Ingestion Lead in New York to head the team building the ingestion layer—web crawl, extraction pipelines, and data lakes. You’ll mentor engineers, guide architecture, and coordinate with pre-training, data quality, and partnerships to deliver scalable corpora for AI models. You’ll stay hands-on, drive technical decisions, and manage day-to-day execution in a fast-moving environment while ensuring auditable, compliant data delivery. #J-18808-Ljbffr Reflection
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Data Ingestion Engineering Lead: Web Crawling & Pipelines in New York, NY vacancy
- ...Role Reflection's Data team builds the... ...scale. The ingestion layer is the machinery... ...turns the open web, licensed... ...Data Ingestion Lead, you'll provide... ...its pillars web crawl , data ingestion pipelines , and data lakes... ...data ingestion engineers, guide the technical...PipelineWebWork at officeVisa sponsorship
- Reflection, located in New York, is searching for a Data Engineer to build robust data ingestion systems essential for AI training. The ideal candidate will be skilled in web crawling and data acquisition, comfortable working with large datasets, and have excellent communication...PipelineWeb
- Reflection is seeking a Data Ingestion Lead to head the team building the ingestion... ...You will mentor data ingestion engineers, guide architecture across crawling, extraction, and corpus storage,... ...shaping strategy, driving scalable pipelines, and delivering timely data to...Pipeline
- ...all. About The Role Data plays a crucial... ...build and operate the ingestion systems that turn the open web and other large‑... ...to our pre‑training pipelines, working directly with... ...role is ideal for engineers who love building... ...training, including web crawling, extraction, and...PipelineWebRelocation package
- ...looking for a top-tier Senior Data Engineer to join our tech team... ..., high-throughput data pipelines powering AI-ready web search Design and... ...Develop and optimize data ingestion and transformation systems... ...pipelines for continuous crawling, deduplication, and enrichment...PipelineWebFlexible hours
Linkup Inc
New York, NY1 day ago - ...the Role As a Senior Data Acquisition Engineer , you will own and evolve... ...for large-scale web data collection. You... ...production-grade scraping and ingestion infrastructure that... ...to level up data pipelines and ensure data... ...Background in large-scale crawling, scraping, or distributed...PipelineWebWork at office
$123k - $215.25k
...benefitsJob Function: Engineering &... ...engineering (Mobile and Web), AI/ML, marketing... ..., grounded in our data technology model that... ...capabilities, and leads the end-to-end technology... ...of scalable data pipelines and data... ...workflows, including ingesting Rooam POS data from...PipelineWebWorldwideVisa sponsorship- ...Experience & Design, Engineering and Managed Services.... ...experiences, from mobile and web applications to... ...client partnerships.As a Data Architect / Engineer,... ...-RAG platform and the pipelines that get content into... ...domains, build resilient ingestion pipelines from...PipelineWebImmediate start
- ...marketing technology startup in New York, NY, is seeking a Senior Data Acquisition Engineer to lead the development of a scalable web data acquisition platform. This hybrid role focuses on enhancing data ingestion processes and system reliability. Candidates should possess...PipelineWeb
- ...Department is seeking a Senior Data Engineer with a proven background in... ...architecture, including Synapse ETL/ELT Pipelines, SQL pools, Azure Event Hub... ...and manage end-to-end data ingestion and integration processes in... ...(API endpoints, progressive web applications) for improved...PipelineWebContract workWork at officeRemote work1 day per week
Port Authority of New York and New Jersey
New York, NY4 days ago $112.29k - $161.28k
...descriptionPHM is the leading health media agency... ...meets media scale, where data becomes direction,... ...casesArchitect end-to-end data pipelines for ingestion, transformation, and... ...mentorship to data engineers and analystsAct as a... ...callsExperience with Web and Digital marketing...PipelineWebTemporary workFreelanceFlexible hours$185k
...Senior Data EngineerNew York, NY (This role... ...data lake. The Pipeline Team plays a crucial... ...product teams and engineering teams to solve... ...data-driven products.Lead complex data engineering... ...reliable data ingestion and transformation... ...with the web-based tools and insight...PipelineWebTemporary workWork at officeWork from homeMonday to FridayTrepp, LLC
New York, NY1 day ago- Publicis Health Media in New York City is seeking a Data Manager of Data Engineering to lead data projects for multiple clients. You will design scalable pipelines, ingest data from APIs, and contribute to enterprise data architecture to support governance and analytics...Pipeline
- ...looking for a Senior Data Engineer that is innovative, curious... ..., reliable data pipelines that support sell-side media monetization. Ingest, integrate, transform,... ...methodologies to build web applications that access... ...supporting or leading data platform migrations...PipelineWebWork experience placement
- ...Responsibilities Design, build, and operate the data warehouse and transformation layer,... .... Build and operate reliable data ingestion pipelines. Design and own data models... ...handling broad ownership. Interest in leading through technical direction and influence...PipelineFull timeImmediate start
Patlytics, Inc.
New York, NY1 day ago - Tabs, the leading AI-native revenue platform, seeks a Senior Software Engineer for the Data Ingest team in our Integrations org. You will own parts of the usage-to-billing pipeline, ingest real-time data from CRM systems, and help scale data flows to support AI-era billing...Pipeline
$150k - $175k
...building a governed, AI-powered data operation for FP&A, and this... ...integrate NetSuite, SAP, and open web or third-party sources (... ...will do Own the end-to-end data pipeline , including NetSuite to... ...and traceable back to source . Ingest open web data from third-party...PipelineWebWork at officeFlexible hours$100k - $175k
...all about here. Our data: We’ve assembled leading climate scientists... ...looking for a Senior Data Engineer to join our team.... ...will assist in the ingestion of climate risk and... ...partners, develop data pipelines, query geospatial... ...synchronize with APIs and web services. This...PipelineWebWork at officeLocal areaWorldwide$148.5k - $260.1k
...career at the company leading workforce... ...search, enterprise data discovery, and intelligent... ...Graph and AI Engineering team. The SMTS will... ..., semantic pipeline workflows, and AI‑... ...data pipelines to ingest and map structured... ...databases and semantic web concepts (e.g., Neo...PipelineWebContract work$170k - $216k
...ve raised $10M from leading investors in... ...a Senior Software Engineer, Data & AI to build Holly... ...data foundation Build Ingestion & Normalization Pipelines Build systems that... ...sources across the web Turn messy, heterogeneous... ...web scraping / crawling, document extraction...PipelineWebCasual workWork at officeLocal area$115.4k - $192.3k
...building trusted data products that power... ...experiences? Do you enjoy leading teams, improving... ...offers a web-based research platform... ...our processes to ingest content, extract... ...with Product and Engineering teams to make sure... ...normalization, and enrichment pipelines using appropriate...PipelineWebLocal areaImmediate startFlexible hours- ...operating models and embed data, automation, and AI... ..., data, design, and engineering move together. Siloed... ...builds and manages data pipelines, unifies customer data... ...Responsibilities Data Ingestion: Connect and ingest... ...processing) Lightning Web Components (dynamic, reusable...PipelineWebFull timeContract workLocal areaVisa sponsorshipWork visa
$111.4k - $185k
...on other high-quality sites across the web, and on millions of Kindles, tablets,... ...Enablement organization (SITE). We are a big-data focused engineering team that provides unique, pan-Amazon... ..., warehousing and building ETL pipelines- Experience with one or more query language...PipelineWebFlexible hours- ...Role: Data Engineer Location- New York, NY JD- Must have finance (investment... ..., designing and maintaining data pipelines and models that drive decision-making across... ...solutions. Experience with web-scraping. Strong problem-solving...PipelineWeb
SysMind Tech
New York, NY4 days ago $160k - $175k
...Senior Software Engineering ManagerThe New York Post is... ...ecosystem — spanning web, mobile apps, video, audio... ...technology team and lead the systems that power... ...infrastructure services, data pipelines, personalization... ...service tiers.Own data ingestion pipelines for real-time...PipelineWebPermanent employmentWork at officeFlexible hours3 days per weekNew York Post
New York, NY1 day ago- ...manages high-performance backend systems and data pipelines to process large volumes of data. Leads technical architecture, mentors engineering teams, and optimizes cloud storage and... ..., high-performance RESTful APIs and web services. Manage Cloud Infrastructure : Integrate...PipelineWeb
Recurring Decimal
New York, NY9 hours ago - ...Sr. Data EngineerAetna Resources LLC, a CVS Health company, is hiring... ...in New York, NY: Sr. Data Engineer to design, build and manage large scale data structures, pipelines and efficient Extract/Load/Transform... ...development for enterprise or web applications.Preferred...PipelineWebLocal areaRemote work
Phenom People
New York, NY2 days ago - ...Background required - Former Data Engineer who understands the business... ...the data side (inventory to web traffic to accounting) Job Description... ...strategies, building new pipelines to speed up delivery,... ...operational reviews with industry-leading engineers. fashion clothing retailer...PipelineWeb
- ...The Role: Engineering Manager – Backend Platforms We... ...Engineering Manager to lead a team responsible for... ...reliable systems that support web and mobile applications... ...systems, APIs, and data platforms. Define... ...based workflows, CI/CD pipelines, and observability frameworks...PipelineWebFull time
Next Ventures
New York, NYmore than 2 months ago $140k - $220k
GPTZero is seeking software engineers to build and run a research engine for... ...journalism on AI slop. You will develop pipelines for processing web-scale content, create polished... ...visuals, and craft search and crawl systems to collect data from the web. You will...PipelineWeb
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Data Ingestion Engineering Lead: Web Crawling & Pipelines. Be the first to apply!
Related searches
- data center engineer New York, NY
- senior cloud data engineer New York, NY
- data infrastructure engineer New York, NY
- sr information security engineer New York, NY
- hadoop big data developer New York, NY
- data visualization developer New York, NY
- associate data engineer New York, NY
- data engineer freelance New York, NY
- remote data engineer New York, NY
- etl data engineer New York, NY