Senior Data Engineer - Identity Resolution & Large-Scale Data Engineering
Sumeru Solutions
Role: Senior Data Engineer Identity Resolution & Large-Scale Data Engineering
Location: Bellevue, WA
Overview:
We are seeking an experienced Senior Data Engineer to design, build, and optimize production-grade data solutions supporting Identity Resolution (IDR) and large-scale customer data initiatives.
The ideal candidate will have proven experience delivering data solutions at billion-record scale, with strong expertise in Azure Databricks, PySpark, Python, Azure Data Factory, and distributed data processing.
The role requires strong production-scale engineering judgment-the ability to identify technical and scalability constraints early, evaluate alternatives quickly, and deliver tested, scalable, and production-ready solutions.
Experience with Identity Resolution or comparable high-throughput matching systems is highly preferred.
Key Responsibilities
Design and optimize "large-scale data processing solutions capable of operating at 1B+ record scale".
Develop scalable data pipelines using Azure Databricks, PySpark, Python, and ADF".
Optimize Spark workloads including large-scale joins, partitioning, shuffling, data skew, memory, and compute utilization.
Design and improve solutions for Identity Resolution, Entity Resolution, Record Linkage, Deduplication, or similar high-volume matching problems.
Evaluate technical approaches against production data volumes and infrastructure constraints.
Identify and communicate scalability and environment constraints early and recommend viable alternatives.
Perform performance and scalability testing using production-like workloads.
Build reliable data pipelines with appropriate validation, error handling, monitoring, and recovery mechanisms.
Collaborate with architects, engineers, data scientists, and stakeholders to deliver solutions from design through production.
Required Qualifications
7+ years of Data Engineering experience.
Proven experience working with very large-scale datasets, preferably 1B+ records.
Strong hands-on experience with Azure Databricks and PySpark.
Strong Python and SQL skills.
Experience with Azure Data Factory (ADF) and Azure data services.
Strong understanding of distributed processing and Spark performance optimization.
Experience delivering production-grade, scalable data solutions.
Ability to identify technical constraints early and make timely technical decisions.
Strong problem-solving and communication skills.
Preferred Qualifications
Experience with Identity Resolution / Entity Resolution / Record Linkage / Deduplication / Fuzzy Matching.
Experience with Cosmos DB and low-latency APIs.
Experience with Azure Functions or Azure Web Apps.
Experience with Event Hub, Kafka, or Spark Structured Streaming.
ML experience related to matching or customer data.
Exposure to Generative AI, LLMs, RAG, or Agentic AI.
Telecommunications or large-scale customer data experience.
- The Trade Desk, Inc. is seeking a Senior Identity Engineer to own and operate large-scale distributed identity systems that process petabyte-scale data across cloud and on-prem infrastructure. You will collaborate with data scientists, platform engineers, and product teams...SeniorWorldwide
- ...work gets done. We are hiring a Senior Software Engineer across our Identity, Data Security and Trust organization. These... ...the most secure platform at scale, and giving customers the tools to... ...designing, building, and operating large-scale distributed systems in production...SeniorFull time
Snowflake
Bellevue, WA16 hours ago $130k - $196.5k
LiveRamp is the data collaboration platform of choice... ..., and foundational identity, LiveRamp is setting... ...core identity, entity resolution, and measurement capabilities... ...Analyze and transform large-scale, high-dimensional, and... ...actionable insights, engineer robust features, and...SeniorFull timeWork from homeFlexible hoursNight shift- ...Senior Data EngineerBuild the data foundation that powers... ...systems that operate at scale. At the core of this system is Data Engineering. This role sets direction... ...owning and evolving large-scale production data systems... ...systems, led incident resolution, and driven long-term...Senior
Auger Services Inc
Bellevue, WA4 days ago - ...is building the identity trust infrastructure... ...the RoleThe Big Data R&D team develops... ...search, entity resolution, and identity matching... ...products.As a Senior Data Scientist I,... ...algorithms on large-scale PII datasets, own... ...partner to Product, Engineering, and Client-...Senior
Socure Inc
Seattle, WA1 day ago $153.5k - $207.7k
...environment, then come join us. As a Senior Data Engineer, you will share ownership of Redfin’s... ...purposesYou have experience building large scale Data lakesYou can add architectural insights... ..., sexual orientation, gender identity or expression, age, military or veteran...SeniorMinimum wageFull timeLocal area- Moloco is seeking a Senior Software Engineer for the Ads Applied Data team to design, develop, and optimize distributed... ...to enable feature engineering, identity resolution, and advanced user signal... ...offline pipelines, supporting large-scale ingestion and serving. #J-1880...Senior
$198k - $297k
...productive, and happy. The Role As a Data Engineer you will handle everything... ...the foundation of our large and unique datasets that... ...critical domains including identity, account access, eligibility... ...positive reduction and automated resolution. Participate in on‑call...SeniorWork at officeLocal areaRemote workFlexible hoursCash App
Seattle, WA1 day ago$124.9k - $228.9k
...businesses responsibly. The scale of our platform brings... ...Technology Operations group in Engineering at The Trade Desk and... ...well‑governed enterprise data solutions. As a Senior Software Engineer – Data Engineer... ...orientation, gender identity or expression, veteran status...SeniorFull timeTemporary workWork experience placementLocal areaWorldwideThe Trade Desk
Bellevue, WA2 days ago$200.4k - $260.5k
Senior Machine Learning Engineer, Data InfrastructureUnity Vector builds an Data platform that... ....Our systems operate at scale across batch and streaming... ...platform also supports large-scale model training, feature... ..., disability, sex, gender identity or expression, sexual...SeniorFull timeWork at officeWorldwideRelocation package$124k - $280k
...SummaryAt PwC, our people in data and analytics engineering focus on leveraging... ...future business needs. As a Senior Manager you lead large projects, innovate... ...strategies- Lead large-scale projects and innovate processes... ...orientation, and gender identity); age; disability;...SeniorFull timeH1b- ...DoorDash success. The Data Engineering team builds database solutions... ...is looking for a Senior Data Engineer to be a... ...powerhouse to help us scale our data... ...develop and implement large scale, high volume, high... ...sexual orientation, gender identity or expression, disability...SeniorHourly payWork at officeLocal areaRemote workFlexible hours
$182k - $242k
...enables innovators to build and scale AI with confidence. Trusted... ...at What You'll Do: The Data Engineering Team builds and operates the... ...the role: We're seeking a Senior Data Engineer to design and... ...sexual orientation, gender identity, national origin, veteran status...SeniorPermanent employmentFull timeTemporary workCasual workWork at officeFlexible hoursCoreWeave
Bellevue, WA5 days ago$150k - $200k
...allocated. Our team includes researchers, engineers, and investment professionals who care... ...creation. Responsibilities Build and maintain data pipelines and infrastructures that power... ...ingesting, processing, and analyzing large-scale datasets Implement ETL workflows and...SeniorFull timeRemote work$190.8k - $286.8k
...and Amsterdam. Making data‑driven decisions is key... ...support that, we need to scale our data systems while... ...to teams across engineering, product, and business... ...data pipelines on top of large datasets (in the order... ...orientation, gender, gender identity, gender expression, transgender...SeniorLocal areaFlexible hours$128k - $161k
...the world. At DigitalOcean, Data Center Engineers play a critical role in... ...deploying, maintaining, and scaling the servers and networking equipment... ...opportunities to work on large‑scale deployments, new data... ...orientation, gender, gender identity or expression, age, disability...SeniorLocal areaRemote workWorldwideFlexible hours- ...member of PNNL’s Software Engineering and Architectures (SEA)... ...software, scalable data architecture & pipelines... ...entry [Link] Develop large data pipelines to thwart... ...grid to prevent large‑scale outages [Link] Optimize... ...people of all racial/ethnic identities, women, veterans, and...SeniorWork experience placementLocal areaRemote work
- A leading infrastructure consulting firm is seeking a Senior Drainage Lead in Seattle, WA. The role involves overseeing drainage design for large scale projects, managing teams, and ensuring compliance with relevant regulations. The ideal candidate will have extensive experience...Senior
$152.2k - $243.7k
...opportunity to create impact at scale — tackling meaningful... ...entrepreneurs and engineers in 2016, Pismo is a... ...technological challenges that large banks, marketplaces,... ...and evolution of the data platform, ensuring reliability... ...orientation, gender identity, disability or protect...SeniorFull timeWork experience placementWork at officeLocal area- ...Socure is building the identity trust infrastructure... ...the RoleThe Big Data R&D team is responsible... ...graph and entity-resolution capabilities that power... ...work closely with senior data scientists and engineers while developing your skills in large-scale ML, distributed systems...
Socure Inc
Seattle, WA1 day ago $112.8k - $153.7k
...listen to your ideas. This is a senior engineering role. You will architect,... ...continuously improve cloud data and analytics platforms (e.g... ...response: drive triage, own the resolution, run the post-incident... ...command of cloud fundamentals: identity and access, resource...SeniorFull timeContract workLocal areaFlexible hours$120k - $160k
Consigli Construction is seeking a Senior Estimator in our Estimating division to lead complex, large-scale projects from concept to GMP. The role requires a BS or MS in Engineering or Construction Management with extensive construction estimating experience, including...SeniorFull time- ...Corporation in Bothell, WA seeks an experienced Senior Preconstruction Project Manager to lead preconstruction activities for large-scale commercial construction projects ranging... ..., coordinating with owners, designers, engineers, and subcontractors to develop accurate...SeniorFor subcontractor
$153k - $204k
...enables innovators to build and scale AI with confidence. Trusted... .... About the role: As a Senior Data Engineer you will own and evolve the... ..., modeling, and querying large datasets. ~ Proficiency in... ...sexual orientation, gender identity, national origin, veteran status...SeniorPermanent employmentFull timeTemporary workCasual workWork at officeFlexible hours- ...There is only one Data Cloud. Snowflake’s... ...stop there. They engineered Snowflake to power... ...with near-unlimited scale, concurrency, and... ...tomorrow. The identity & access management... ...engineers. AS A SENIOR SOFTWARE ENGINEER... ...Lead and deliver large projects,...SeniorFull time
Snowflake
Bellevue, WA16 hours ago $167.8k - $209.7k
The Allen Institute is seeking an AI Research Scientist II to develop large-scale AI/ML models for biological systems. This role entails collaborating with various scientific teams and establishing community standards for AI/ML methodologies. In addition, the successful...Senior$147k - $202.4k
Secure Every Identity, from AI to HumanIdentity is the key to... ...seeking a highly skilled Senior Site Reliability Engineer to join our team. We are a... ...specializing in securing large-scale systems. This role is a blend... ...tooling for our Snowflake data systems.Automation:...SeniorPermanent employmentWork at officeLocal areaWorldwideFlexible hoursShift work$131.5k - $243.3k
...customers and developers. We are seeking a Senior Software Engineer to join Apple Services Engineering (... ...passion for building distributed large scale data processing systems, frameworks and... ..., sex, sexual orientation, gender identity, national origin, disability, veteran...SeniorFull timeRelocationApple
Seattle, WA1 day ago$214k - $302k
Senior Staff Data Center Engineer, Mechanical Engineering $214,000.00/yr - $302,000.00/yr Preferred locations... ...in designing modular building scale assemblies, with an understanding of... ...marital status, disability, gender identity or veteran status. We also consider...SeniorFull timeWorldwide$222.5k - $257.5k
...seamlessly and scale solutions smarter... ....Observability Engineering owns the... ...Observability & Telemetry Data), you will set... ..., and mentor senior and mid-level... ...complex issue resolution and driving root... ...and operating large-scale distributed... ..., gender identity, national origin...Full timeTemporary workWork at officeLocal areaRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Data Engineer - Identity Resolution & Large-Scale Data Engineering. Be the first to apply!
- data engineer analytics Bellevue, WA
- senior cloud data engineer Bellevue, WA
- data center engineer Bellevue, WA
- big data devops engineer Bellevue, WA
- sr data engineer Bellevue, WA
- data engineer Bellevue, WA
- sr information security engineer Bellevue, WA
- data visualization developer Bellevue, WA
- data developer Bellevue, WA
- etl data engineer Bellevue, WA