Staff Data Engineer (Scala, Spark, & Gen AI)

Job Description Job Description DISQO’s mission is to build the world’s most trusted ad measurement platform that fuels brand growth. The world’s largest brands, agencies, and media companies trust DISQO for expert insight and AI-driven intelligence about their advertising performance across all platforms. We capture people’s sentiments and journeys, connecting them with the brands they value and the media they consume. With this identity-based approach, brands gain more accurate and authentic insight so they can create more meaningful interactions. Joining DISQO Nation means becoming part of a community that champions speed, innovation, and continuous growth. We invest deeply in our talent, empowering our teams to reach their highest potential. Together, we are shaping the future of work at DISQO—defined by performance, purpose, and impact. We show up each day with curiosity and ambition, committed to learning, accelerating growth, and making a lasting difference. Grounded in our values and principles, we lead and collaborate to elevate performance, accountability, and excellence at every level of the organization. And through it all, we make sure to have fun along the way. This is a great opportunity to join a fun, highly motivated team and lead the development of intelligent data products that directly power how brands measure advertising effectiveness. At DISQO, we use modern cloud infrastructure, Generative AI, and expert-level data engineering to solve complex, real-world problems at scale. We are looking for a visionary technical leader who is a master of distributed data processing (Scala/Spark) and passionate about the intersection of data engineering and Artificial Intelligence. You’ll serve as a force multiplier, working closely with engineering leadership, product managers, and analysts in a collaborative environment where rapid innovation and systemic impact matter. We believe the best software is built by highly aligned, autonomous teams that take ownership and move quickly. We use agile development practices, modern tooling, and strong engineering discipline to deliver early and often. We care deeply about architectural excellence, data correctness, system reliability, and building intelligent systems the right way. Position Description As a Staff Data Engineer, you will set the technical direction for DISQO’s ad measurement platform. You will architect, build, and scale our most complex data pipelines while spearheading the integration of Generative AI capabilities directly into our core data infrastructure and products. You will tackle our hardest scalability challenges, utilizing expert-level Spark and Scala to process massive datasets, while leveraging LLMs to unlock new value from unstructured and structured data. Operating with a high degree of autonomy, you will lead cross-functional technical initiatives, drive architectural decisions, and pioneer how we use AI to enrich data, automate pipelines, and improve data quality. You will mentor senior and mid-level engineers, raising the technical bar for the entire team while expanding DISQO's technical depth across big data systems, cloud infrastructure, and applied AI. What you will do: Architect and Lead: Design, build, and maintain highly scalable, fault-tolerant data pipelines using expert-level Scala and Apache Spark. Gen AI Integration: Pioneer the use of Generative AI within our data ecosystem—incorporating LLMs to enrich datasets, extract value from unstructured data, automate metadata generation, and build intelligent data products. Cross-Functional Strategy: Partner with Product and Engineering leadership to translate complex business requirements into forward-looking data and AI-augmented architectures. Optimize Systems: Architect and aggressively optimize large-scale ETL/ELT workflows. Dive deep into Spark internals to resolve complex performance bottlenecks, memory issues, and data skew. Modern AI Tooling: Implement and manage infrastructure to support AI integration, including vector databases, embeddings pipelines, and Retrieval-Augmented Generation (RAG) architectures. Set the Standard: Write clean, highly optimized, and maintainable code, while establishing standards for code quality, testing, and system architecture across the organization. Ensure Operational Excellence: Champion data quality, observability, and system health to consistently meet enterprise SLAs and customer commitments. Mentorship: Actively mentor engineers, lead technical design reviews, and foster a culture of continuous learning and technical rigor. What we're looking for: 8+ years of experience building, architecting, and supporting complex production data pipelines, distributed systems, and backend infrastructure. Expert-Level Scala & Spark: Deep, hands-on expertise in Scala and Apache Spark. You must understand Spark internals, query plans, memory management, and advanced performance tuning for massive-scale batch processing. Applied Generative AI Experience: Proven experience integrating Gen AI / LLMs (e.g., OpenAI APIs, Anthropic, Bedrock) into data products or data engineering workflows. Hands on experience developing with AI dev tools such as Claude code, etc Strong Python Skills: Proficiency in Python specifically to interface with modern AI ecosystems, data APIs, and orchestration tools. Cloud Mastery: Extensive architectural experience within the AWS ecosystem (EMR, Glue, Athena, S3, Bedrock, etc.). Core Data Foundations: Deep understanding of advanced ETL/ELT concepts, complex data modeling, and performance-tuning SQL. Orchestration: Expert-level experience with workflow orchestration tools such as Airflow. Leadership: Proven track record of leading technical initiatives, making architectural decisions, and mentoring teams in an agile, fast-moving environment. Nice to have: Experience with Snowflake or other modern cloud data warehouses. Deep exposure to streaming or real-time event processing (Kafka, Flink, Kinesis, etc.). Experience utilizing AI for automated data observability, anomaly detection, or data quality tooling. Background in ad tech, measurement, attribution modeling, or specialized analytics platforms. Why DISQO? Lead the architecture of intelligent data products that directly influence how the world's top brands measure advertising impact. Work with bleeding-edge data and Gen AI infrastructure at a highly meaningful scale. Shape the technical culture and elevate a talented engineering organization while owning massive-scale production systems. #LI-MV1 At DISQO, we pride ourselves on having a positive, performance-oriented workplace that includes a flexible hybrid approach, competitive medical benefits, and an amazing vacation policy. Read more about our culture on Glassdoor. You can learn more about what’s happening at DISQO by visiting the DISQO Company Blog. Perks & Benefits: ·100% covered Medical/Dental/Vision for employee, competitive dependent coverage ·Stock options ·401K ·Generous PTO policy ·Team offsites, social events & happy hours ·Life Insurance ·Health FSA ·Commuter FSA (for hybrid employees) ·Catered lunch and fully stocked kitchen ·Paid Maternity/Paternity leave ·Disability Insurance ·Travel Assistance Program ·24/7 Counseling Services offered to Employees Note: The benefits noted above are for full time US based employees only. DISQO is an equal opportunity employer. Discovery, innovation, and growth are possible when we open ourselves to new possibilities, perspectives, and approaches. That’s why, at DISQO, we welcome, support, and empower individuals from diverse backgrounds. Exceptional teams are rooted in extraordinary people, each with a unique story and a compelling set of skills. DISQO does not discriminate against employees based on race, color, religion, sex, national origin, gender identity or expression, age, disability, pregnancy (including childbirth, breastfeeding, or related medical condition), genetic information, protected military or veteran status, sexual orientation, or any other characteristic protected by applicable federal, state or local laws. *Recruiting firms that submit resumes to DISQO without first entering into a written contract will not be entitled to any compensation on candidates referred by that firm. We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.