Senior Software Engineer, Spark Platform

Responsibilities

• Set multi-year technical direction for DoorDash’s in-house Spark-on-Kubernetes platform.
• Own distributed-systems problems involving runtime performance, shuffle architecture, multi-tenant scheduling, and large-scale failure modes.
• Shape architecture across the Spark runtime, remote shuffle service, cluster scheduler, and reliability tooling.
• Partner with the Engineering Manager on technical roadmap, hiring, interview design, and team shape.
• Mentor engineers through design reviews and technical guidance.
• Represent Spark Platform in cross-team architecture forums and guide how data, analytics, and ML workloads use the platform.

Requirements

• Bachelor’s, master’s, or PhD in Computer Science or equivalent.
• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep hands-on Apache Spark experience covering internals, query execution, shuffle, and the executor/driver model at platform scale.
• Experience operating Spark on Amazon EMR, Databricks, or an in-house deployment, with emphasis on platform operations rather than authoring individual Spark jobs.
• Production experience with remote or external shuffle systems, batch or big-data schedulers, or observability and SRE patterns for distributed compute platforms.
• Strong production Kubernetes experience, including operator patterns, executor pod lifecycle, network topology, and multi-tenant failure modes.
• Familiarity with data lake table formats such as Apache Iceberg or Delta Lake and with query and SQL engines that read them.
• Technical leadership experience on a platform team, including setting direction, mentoring, and partnering with management on roadmap and hiring.
• Professional experience with Scala, Java, Python, or Go, plus strong SQL.
• Located in or willing to relocate to the Bay Area, Seattle, or New York City.

Benefits

• Hybrid work arrangement requiring residence in San Francisco, Sunnyvale, Seattle, or New York City.
• 401(k) plan with employer matching.
• 16 weeks of paid parental leave.
• Medical, dental, and vision benefits.
• Wellness benefits and wellness expense reimbursement.
• Commuter benefits match.
• Flexible paid time off or vacation and paid sick time for salaried roles.
• 11 paid holidays.
• Disability and basic life insurance.
• Family-forming assistance and a mental health program.
• Equity grant opportunities in addition to base salary.