Senior Data Engineer | Scala & Spark (Cloud, Big Data)

Requirements Proficiency in Scala and Spark to build and optimize large-scale batch and streaming data pipelines

• With 5+ years of software development experience with a BS/MS Computer Science or equivalent work experience
• Who wants to take ownership. We are responsible for the entire software development lifecycle, from requirements gathering to production support
• To engage in critical and creative thinking, and constructive brainstorming. We expect engineers to present and discuss tradeoffs to solve the problems we face
• With either existing experience designing and building systems that work with large-scale data volumes and data ingestion systems at scale in a cloud-first setting, OR established technical excellence with a desire to learn and crush the data engineering world

What the job involves We are seeking someone with proficiency in Scala and Spark to build and optimize large-scale batch and streaming data pipelines

• The Core Data Engineering team sits within the data engineering group and maintains technical involvement across the applications and infrastructure that make up Magnite's data pipelines
• Our pipelines collectively handle 400+ billion events per day and generate terabytes of data per hour across both cloud-based and on-prem infrastructure
• This data underpins the various business lines - including client reporting, internal data science, account managers, and product + business teams - and as such we need to build systems that remain scalable and efficient at this volume of data while also ensuring data consistency and reliability
• We value communication, discussion, and sharing of ideas to come to the best technical solutions to our large-scale data challenges
• We are looking for people who want to get things done and value open collaboration (including constructive feedback when brainstorming)
• The team's mandate is technical development across the three platform-specific data engineering teams in the group
• As such they get exposure to all aspects of our data engineering infrastructure and applications including Spark jobs, Java-based real-time event processing, and large-scale data warehousing
• Get to work on handling internet-scale data problems
• Help architect and build systems to process our data volume to empower all consumers
• Have hands-on involvement across the group's various data pipelines and related systems for data delivery
• Be a part of and promote our culture of collaboration and mentorship

, Typical challenges we face in this role include:

• Highly-scalable infrastructure: Our traffic has patterns with peak- and off-peak times and seasonal shifts, and our data infrastructure needs to be able to respond accordingly
• Cost optimization: Improving profit margins by lowering the infrastructure cost basis through building efficient systems
• Technical architecture considerations given various SLAs for data delivery

, Our current tech stack:

• Spark (batch + streaming) data pipelines, currently based both on Databricks and on-prem Spark infrastructure, primarily using Scala
• Streaming event processing data pipeline, using Java
• Terraform, Docker, Jenkins for CI/CD / infra / application deployment
• Airflow for job orchestration
• AWS-based cloud infrastructure including RDS, EC2, S3, Kinesis, ECS

#J-18808-Ljbffr Magnite