Senior Big Data Engineer

You will Design, build, and optimize scalable data processing pipelines using Spark, Airflow, and related big data technologies to support both batch and real‑time use cases. Lead and collaborate with technical, application, and security stakeholders to deliver reliable, secure Big Data infrastructure leveraging tools and platforms such as Spark, Dataproc, Redpanda, and Temporal. Own and participate in on‑call responsibilities for Big Data platforms, triaging and resolving incidents, responding to tickets, and ensuring systems consistently meet defined SLAs for availability, performance, and data quality. Design, develop, and automate large‑scale infrastructure on Kubernetes using Terraform and other infrastructure‑as‑code patterns to support high‑performance data processing, analytics, and AI/ML workloads. Define and implement monitoring, alerting, and runbooks to provide end‑to‑end observability and drive continuous improvement in reliability and operational excellence using tools like Grafana. Onboard, train, and mentor vendor teams and external partners so they can effectively support, operate, and extend the solutions owned by the Big Data Infrastructure (BDI) team. Drive the technical roadmap for emerging data and infrastructure technologies by evaluating options, building proofs of concept (POCs), and authoring solution selection and design documents. Champion engineering best practices (code reviews, testing strategies, CI/CD, security and compliance standards) across the Big Data ecosystem to ensure maintainable, resilient, and cost‑efficient systems. Your team will Build and operate the core Big Data Infrastructure (BDI) platform that powers large‑scale data processing, analytics, and AI/ML workloads across LiveRamp using tools like Dataproc, Redpanda, Airflow and Temporal. Provide secure, self‑service, and cost‑efficient data environments that enable product and data teams to experiment, ship, and scale data collaboration applications quickly. Partner closely with application, security, SRE, and platform teams to ensure data systems meet LiveRamp’s standards for privacy, compliance, reliability, and performance. Evolve common patterns, tooling, and reference architectures that reduce friction for teams adopting new big data, streaming, and AI/ML capabilities. Continuously improve platform reliability and developer experience by automating operations, simplifying workflows, and incorporating feedback from internal customers. About you 7+ years of experience in software engineering or data engineering, with a focus on large‑scale data processing and distributed systems. Hands‑on experience building and operating pipelines with Apache Spark (batch and/or streaming) and at least one orchestration framework such as Airflow or Temporal. Practical experience running Big Data workloads on cloud platforms (for example, using Dataproc or similar managed compute services). Proficiency with Kubernetes and infrastructure‑as‑code tools such as Terraform to provision, configure, and manage production services. Experience with modern streaming and messaging technologies (for example, Redpanda, Kafka, or similar) and real‑time data processing patterns. Solid understanding of systems design, scalability, reliability, and observability for data‑intensive workloads. Experience participating in or leading on‑call rotations and incident management processes for production systems. Strong collaboration and communication skills, including working with cross‑functional partners (security, application owners, vendors) and documenting designs and decisions clearly. Ability to evaluate new technologies, build POCs, and make data‑informed recommendations that influence team and platform roadmaps. Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related technical field, or equivalent practical experience. Preferred Skills Experience working on Big Data Infrastructure, data platform, or core infrastructure teams in a cloud‑native environment. 3 years of Software Development experience in one of the languages - Python, Go, Java or Scala. 3 years of experience with Linux Operating Systems. 3 years of experience in setting up, managing and optimizing Big Data ecosystems such as Kafka, Airflow, Redpanda and Temporal. 3 years of experience using cloud‑based platforms such as AWS, GCP, Azure or similar. 3 years of experience with a Version Control System (GIT, Subversion, or similar). Experience in using AI tools, NLP and agent creation. Exposure to security and compliance considerations for data platforms (for example, access control, encryption, data governance, and auditing). Prior experience mentoring engineers, leading technical initiatives, or coordinating work with vendors and external partners. The approximate annual base compensation range is $130,00 to $196,500. The actual offer, reflecting the total compensation package and benefits, will be determined by a number of factors including the applicant's experience, knowledge, skills, and abilities, geography, as well as internal equity among our team. Benefits People: Work with talented, collaborative, and friendly people who love what they do. Fun: We host in‑person and virtual events such as game nights, happy hours, camping trips, and sports leagues. Work/Life Harmony: Flexible paid time off, paid holidays, options for working from home, and paid parental leave. Comprehensive Benefits Package: LiveRamp offers a comprehensive benefits package designed to help you be your best self in your personal and professional lives. Our benefits package offers medical, dental, vision, life and disability, an employee assistance program, voluntary benefits as well as perks programs for your healthy lifestyle, career growth and more. Savings: Our 401K matching plan—1:1 match up to 6% of salary—helps you plan ahead. LiveRamp is an affirmative action and equal opportunity employer (AA/EOE/W/M/Vet/Disabled) and does not discriminate in recruiting, hiring, training, promotion or other employment of associates or the awarding of subcontracts because of a person's race, color, sex, age, religion, national origin, protected veteran, disability, sexual orientation, gender identity, genetics or other protected status. Qualified applicants with arrest and conviction records will be considered for the position in accordance with the San Francisco Fair Chance Ordinance. We use automated decision systems (ADS) as part of our recruitment and hiring process. If you require an accommodation or believe that the use of an ADS may create a barrier to your application or participation in the hiring process due to a disability or other protected characteristic, please let us know. We are committed to providing reasonable accommodations and ensuring an equitable hiring experience for all candidates. California residents: Please see our California Personnel Privacy Policy ( for more information regarding how we collect, use, and disclose the personal information you provide during the job application process. To all recruitment agencies: LiveRamp does not accept agency resumes. Please do not forward resumes to our jobs alias, LiveRamp employees or any other company location. LiveRamp is not responsible for any fees related to unsolicited resumes. We are proud to be an equal employment opportunity and affirmative action employer. We believe in diversity and do not discriminate based on race, color, religion, sex, age, national origin, veteran status, sexual orientation, gender identity, disability, or any other basis of discrimination prohibited by law. #J-18808-Ljbffr