Senior Data Engineer

Job Description The Data Engineering team is seeking a Senior Data Engineer to help design, build, and scale the modern data platform that powers analytics, data science, and data products across Versant brands. In this role, you’ll collaborate closely with Data Product, Data Science, Analytics, and Engineering teams to deliver reliable, high-impact data solutions used by hundreds of internal users. We’re looking for engineers who enjoy hands‑on development, take ownership of production systems, and influence implementation through collaboration and technical expertise. Key Responsibilities Design and implement lakehouse architecture using Delta Lake, including medallion pipeline patterns (Bronze/Silver/Gold), schema enforcement, and time travel. Build and operate batch and real‑time ingestion pipelines leveraging Databricks Auto Loader, Structured Streaming, and Change Data Capture patterns. Implement data governance and security using Unity Catalog, RBAC, and compliance‑driven practices for sensitive environments. Optimize performance and manage costs through FinOps strategies, including cluster sizing, workload tagging, Spark tuning, and Photon acceleration. Design, implement, and maintain CI/CD pipelines and orchestration workflows using Databricks Workflows, Delta Live Tables, and tools such as Airflow. Collaborate with Data Science teams on ML workflows, including MLflow, feature store integration, and model lifecycle management. Ensure data quality, observability, and lineage across media‑specific datasets such as streaming logs, ad impressions, and audience metrics. Provide technical mentorship through code reviews, pairing, and knowledge sharing. Qualifications Bachelor’s degree in Computer Science, Data Engineering, or equivalent practical experience. 5+ years of experience building production‑grade data pipelines in cloud environments using Spark‑based platforms (e.g., Databricks, EMR, Dataproc, open‑source Spark). Expertise in PySpark, SQL, and Spark‑based data processing, with experience operating pipelines at scale in production. Hands‑on experience building batch or streaming production data pipelines using distributed processing frameworks (e.g., Spark, Flink) and query engines such as Presto. Proficiency with orchestration tools such as Apache Airflow or Dagster, with hands‑on experience in CI/CD, monitoring, alerting, and data quality for production systems. Experience working with modern data architectures, including event‑driven and distributed systems. Proficiency with Git and collaborative development workflows. Solid understanding of infrastructure, networking, and data security fundamentals. Preferred Qualifications Experience building Lakehouse platforms and medallion pipelines in Databricks. Familiarity with Unity Catalog, data governance, and compliance frameworks (e.g., PCI). Hands‑on experience with CI/CD pipelines, orchestration tools, and infrastructure‑as‑code. Experience with Lakeflow Spark Declarative Pipelines, MLflow, feature stores, and MLOps practices. Background in media and entertainment data (e.g., video metadata, ad tech, audience analytics). Experience building data platforms within the media industry, with a strong understanding of audience analytics. Experience working with large‑scale analytical datasets (e.g., event logs, clickstream data, audience metrics). Comfortable using AI‑assisted development tools (e.g., ChatGPT). Databricks or cloud certifications (e.g., Databricks Certified Data Engineer). What Sets You Apart Passion for clean, reliable, and scalable data systems. Strong communication and collaboration skills. Ability to balance near‑term delivery with thoughtful technical design. Curiosity and a growth mindset. A team‑first attitude who supports and uplifts others. Additional Information Hybrid: This position has been designated as hybrid, generally contributing from the office a minimum of three days per week from our NYC office. Salary: $140,000 – $160,000, with company‑sponsored benefits including medical, dental and vision insurance, 401(k), paid leave, tuition reimbursement, and other perks. VERSANT is committed to equal employment opportunities for all applicants and employees, regardless of race, color, religion, creed, gender, gender identity or expression, age, national origin or ancestry, citizenship, disability, sexual orientation, marital status, pregnancy, veteran status, membership in the uniformed services, genetic information, or any other basis protected by applicable law. If you are a qualified individual with a disability or a disabled veteran and require support throughout the application or recruitment process, you have the right to request a reasonable accommodation. Submit your request to View email address on click.appcast.io. #J-18808-Ljbffr Jobr