Sr. Software Engineer II - Pinnacle

Senior Analytics Data Platform Engineer

New York (Hybrid)

You will join the Data & Analytics Platform (DAP) team within the Pinnacle engineering organization. The DAP team owns and operates two data consolidation platforms — Quantum (contract-driven, next-gen) and Analytics 2.0 (SQL-driven, legacy) — that ingest, transform, and serve billions of records daily from social platforms, measurement systems, and third-party partners. The data powers Looker dashboards, customer-facing reports, and downstream APIs used across DoubleVerify's product suite.

What you'll do:

• Platform Abstraction & Design: Design and maintain the YAML-based "Contract" system that allows users to define data entities, transformations, and SLOs without writing low-level orchestration code.
• Infrastructure as Code (IaC): Develop the translation engine that converts user contracts into automated dbt models, Airflow DAGs, and Snowflake objects.
• API Development: Transition the platform from static configuration files to a dynamic, API-first architecture, enabling programmatic creation of data artifacts.
• Self-Service Enablement: Build tooling and guardrails that allow business units to deploy their own data solutions while maintaining global standards for governance and security.
• Performance & Scale: Optimize the "translation" layer to ensure that generated jobs are efficient, cost-effective, and leverage the full power of the Snowflake/dbt stack.
• Developer Experience (DevEx): Act as the "Product Manager" for your platform, gathering feedback from internal users to simplify the data development lifecycle.
• Design and build data pipelines that process billions of records a day across consolidation, semantic, and externalization layers using the DV Internal Data Platform — a self-service, contract-driven architecture where pipelines are defined via YAML contracts and automatically deployed to Snowflake, Airflow, and Looker.
• Develop and extend the Contract Interpreter — a Python library (Pydantic, Jinja2) that reads contract driven platform based YAML and generates dbt models, Airflow DAGs, and environment configurations for each deployment environment (dev, stg, prod).
• Lead new initiatives and integrations with the world's largest social platforms (YouTube, TikTok, Meta, Snapchat, Reddit, Netflix, etc.) to measure ad performance end-to-end.
• Build and maintain the semantic layer — design LookML models, explores, and views that translate consolidated data into customer-ready analytics through Looker.
• Implement and maintain observability — build monitoring, alerting, watermarking, and data consistency checks to ensure pipeline reliability and data freshness at scale.
• Leverage AI agents and tooling — contribute to and use the team's AI agent workspace (meta-repo with AGENTS.md context files, skills, and MCP integrations) to accelerate development, automate workflows, and encode institutional knowledge for AI-assisted engineering.
• Design schema evolution and data migration strategies — manage schema versioning, backward compatibility, incremental vs. full-refresh deployments, and large-scale data backfills.
• Work in multi-functional agile teams with end-to-end responsibility for product development and delivery — from contract definition to customer-facing data.
• Collaborate directly with engineers from partner platforms on API development and data integration specifications.
• Train and mentor a team of software engineers.

Who you are:

Required

• Bachelor's degree or foreign equivalent in Computer Science, Data Engineering, or a related field.
• 5+ years of experience in a Data Engineering or related role.
• Strong SQL skills — advanced querying, performance tuning, window functions, and complex transformations at scale.
• Proficiency in Python — building libraries, data processing scripts, and automation tooling (experience with Pydantic, Jinja2, or similar templating frameworks is a plus).
• Deep experience with Snowflake — schema design, Snowpipe, streams, tasks, materialized views, clustering, and query optimization.
• Experience with dbt (data build tool) — building and maintaining models, macros, custom materializations, and incremental strategies.
• Experience with orchestration tools — Airflow / Cloud Composer, DAG design, scheduling, and monitoring.
• Experience with cloud platforms — GCP (GCS, BigQuery, Cloud Composer, Kubernetes) or equivalent.
• Strong understanding of data warehousing concepts — dimensional modeling, star/snowflake schemas, slowly changing dimensions, fact/aggregate table design, and data consistency patterns.
• Experience with CI/CD pipelines — GitLab CI, Flyway migrations, or similar deployment automation.
• Experience with AI-assisted development tools — Claude Code, Cursor, GitHub Copilot, or similar AI coding assistants. Experience building or contributing to AI agent context files (AGENTS.md), skills, or meta-repo patterns is a strong plus.

Preferred

• Experience building or working with contract-driven / configuration-driven data platforms where pipelines are generated from declarative specifications (YAML, JSON schemas).
• Experience with Looker / LookML — building semantic models, explores, aggregate awareness, and dashboard development.
• Experience with Kafka — schema registries, topic management, and streaming data integration.
• Experience with data quality and observability frameworks — automated testing, watermarking, data integrity validation, and SLA monitoring.
• Experience with Terraform or infrastructure-as-code for managing cloud resources.
• Familiarity with data mesh principles — federated data ownership, data products, and self-service platform design.

The successful candidate's starting salary will be determined based on a number of non-discriminating factors, including qualifications for the role, level, skills, experience, location, and balancing internal equity relative to peers at DV. The estimated salary range for this role based on the qualifications set forth in the job description is between $107,000- $212,000. This role will also be eligible for bonus/commission (as applicable), equity, and benefits. The range above is for the expectations as laid out in the job description; however, we are often open to a wide variety of profiles, and recognize that the person we hire may be more or less experienced than this job description as posted.

Not-so-fun fact: Research shows that while men apply to jobs when they meet an average of 60% of job criteria, women and other marginalized groups tend to only apply when they check every box. So if you think you have what it takes but you're not sure that you check every box, apply anyway!