Senior Data Engineer

Senior Data EngineerNew York, New York, United StatesWe're TripleLift, an advertising platform on a mission to elevate digital advertising through beautiful creative, quality publishers, actionable data and smart targeting. Through over 1 trillion monthly ad transactions, we help publishers and platforms monetize their businesses. Our technology is where the world's leading brands find audiences across online video, connected television, display and native ads. Brand and enterprise customers choose us because of our innovative solutions, premium formats, and supportive experts dedicated to maximizing their performance.As part of the Vista Equity Partners portfolio, we are NMSDC certified, qualify for diverse spending goals and are committed to economic inclusion. Find out how TripleLift raises up the programmatic ecosystem at triplelift.com.OverviewTripleLift is seeking a Senior Data Engineer to join a small, influential Data Engineering team. This hire will be responsible for expanding and optimizing our data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The Senior Data Engineer will support our software engineers, product managers, business intelligence analysts and data scientists on data initiatives, and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. The right candidate will be excited by the prospect of optimizing or even re-designing our company's data architecture to support our next generation of products and data initiatives.ResponsibilitiesCreate and maintain optimal data pipeline architecture,Explore and assemble large, complex data sets that meet functional / non-functional business requirements.Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Spark, EMR, Kafka and other big data technologiesWork with stakeholders across different teams, including product managers, engineers and analysts to assist with data-related technical issues and support their data infrastructure needs.Education & RequirementsBachelor's degree in Computer Science, a related technical field, or equivalent professional experience.Minimum of 5 years of professional software engineering experience, with a strong focus on data and ML processes running in production at scale.Experience with optimizing big data and ML pipelines like Spark and Kafka at scaleFundamental knowledge of ML and AI operations and processes. Experience in MLFlow, Feast, Ray, and other AI training technologies a plusUnderstanding of a variety of database technologies such as OpenSearch, Druid, Clickhouse, InfluxDB, etcProven track record of cross-functional data modelingStrong command of SQL and expertise in one or familiarity with multiple of the following languages: Python, Scala, JavaExperience in containerization, K8s and ArgoCD is a plusUS Jobs: The base salary range represents the low and high end of the TripleLift US salary range for this position. Actual salaries will vary depending on factors including but not limited to experience and performance. The range listed is just one component of TripleLift's total compensation package for employees. Other rewards may include bonuses, an open Paid Time Off policy, and many region-specific benefits. Pay is based on various non-discriminatory factors including but not limited to experience, education, and skills. Benefits Available to Eligible Employees Include the following*: Medical, Dental & Vision Plans Flexible PTO 401k w/ employer match *Full-time employees are eligible for comprehensive benefits (subject to the terms of applicable plans/policies/agreements, which will be made available to you after commencing employment).$130,000 - $170,000 USD