Senior Data Engineer

MyHealthTeam is hiring a Senior Data Engineer (hybrid, Redwood City, CA) to own the data platform end-to-end. In this role, you will build and operate ingestion, transformation, storage, query, access, and the platform roadmap so product, analytics, and operations teams can deliver faster while meeting security and compliance needs. Role Overview You will cover both production engineering and data engineering responsibilities across the data and product services ecosystem. The work includes pipeline development for batch and streaming workloads, warehouse modeling and conventions, performance optimization at the data layer, and reliable operations when infrastructure issues arise. Key Responsibilities Own the data platform end-to-end, including ingestion, transformation, storage, query, and access, and lead its roadmap as data demands grow. Build and evolve batch and streaming pipelines using PySpark/EMR, Kinesis, Lambda, and Step Functions, ingesting from Postgres, Salesforce, third‑party vendors, and product event streams into an Iceberg-based lake. Model the warehouse by designing SCD tables, event tables, and the conventions used by engineers and analysts when adding new datasets. Collaborate with product, engineering, analytics, and operations stakeholders to scope data requests into reliable pipelines, and provide documentation and tooling for self‑serve use. Own the security and compliance backbone of data systems, including audit logging, access control, and temporary‑access workflows. Optimize backend query performance where the data layer meets product code, including read‑replica routing, indexing, caching, and IO instrumentation in Java/Spring services. Lead investigation and remediation for data infrastructure issues such as IOPS spikes, pipeline failures, schema drift, and late data, with durable fixes. Use AI to accelerate pipeline scaffolding, schema work, and ad‑hoc investigations, including shipping internal AI tooling for other teams. Mentor engineers and analysts on how to use and work effectively with the data platform. Required Qualifications 5+ years of experience building production data pipelines and platforms. Strong Python (PySpark) and SQL skills, including tuning Spark jobs at scale. Ability and willingness to work on services around the data layer, including Java and Spring Boot. Hands‑on experience with distributed compute (Spark/EMR), streaming (Kinesis), and object storage (S3). Solid fundamentals in Postgres, including query optimization, indexing, replication, replica routing, and diagnosing bottlenecks. Experience with Iceberg and Trino (or comparable tooling). Comfort working with CI/CD and Terraform. Experience building with AI, including frontier models, agentic coding tools, or practical prototypes. Proven track record of collaborating across teams with different priorities and language (for example product and operations). Technology Stack PySpark, EMR, Kinesis, Lambda, Step Functions, Postgres, Salesforce, Iceberg, Trino, S3, Java, Spring Boot, CI/CD, Terraform, SQL About the Role and Team Impact As a Senior Data Engineer, you will build the platform layer that enables other teams to understand the business, ship product features, and meet compliance obligations. The goal is for the systems you create to scale beyond individual tickets, supporting product, analytics, operations, and data science over time. The work sits at the intersection of production engineering and data engineering, with impact measured by whether teams move faster due to the platform. What Success Looks Like Own platform reliability, cost, accessibility, and compliance beyond day‑to‑day ticket delivery. Treat security and PII/PHI handling as core engineering work. Leave behind documentation, tools, and workflows that increase speed for engineers and analysts. Ship iteratively with instrumentation, favoring quick demos of rough versions over long polished releases. Prioritize effectively under urgency with good judgment. Compensation USD 205,000 - 240,000 per year Hybrid location: Redwood City, CA, US #J-18808-Ljbffr DataJobs.com