Data Engineering Manager

Data Engineer Manager

As a Data Engineer Manager, you will design and maintain data platform road maps and data structures that support business and technology objectives. Naturally inquisitive and open to the deep exploration of underlying data, finding actionable insights, and working with functional competencies to drive identified actions. You also enjoy working both freely and as part of a team and have the confidence to influence and communicate with stakeholders at all levels, and to work in a fast-paced complex environment with conflicting priorities. Reporting into the delivery leader, you will deliver consumable, contemporary, and immediate data and AI solutions to support and drive business decisions. As a Manager Data Engineer, you will be a critical leader in our technology team, responsible for shaping the architectural vision and technical roadmap for our data and AI initiatives. You will work closely with business stakeholders, data scientists, engineers, and product teams to translate complex business requirements into elegant, high-performance technical solutions. This role demands a hands-on architect who can navigate complex environments, mentor teams, and drive the adoption of best practices in solution design and delivery.

Responsibilities:

Brief Description of Role:

We are looking for 6+ years of experience in architecture design, data engineering, and AI based initiatives in a customer or business facing capacity and experience in the following:

• Solution Architecture Design: Participate in the end-to-end design of complex, scalable, and fault-tolerant data and AI solutions, ensuring alignment with business strategy, enterprise architecture standards, and industry guidelines.
• Knowledge and experience of delivering CI/CD and DevOps capabilities in a data environment.
• Solution Assessments: Conduct thorough solution assessments of existing systems and proposed architectures, identifying gaps, risks, and opportunities for improvement and modernization.
• Data Mesh & Data Products: Champion and design a data mesh architectural approach, enabling the creation and management of high-quality, discoverable, and reusable data products for decentralized data ownership and consumption.
• AI/ML Integration: Define pipelines that seamlessly integrate Artificial Intelligence (AI) and Machine Learning (ML) models into operational systems, ensuring robust data for model training, inference, and monitoring.
• Cloud Ecosystem Mastery: Design and optimize solutions primarily within AWS and Azure cloud environments, making strategic choices for platform services, infrastructure, security, and cost efficiency.
• Modern Data Stack Expertise: Drive the adoption and optimal utilization of platforms like Databricks for data engineering, machine learning, and analytics, and/or Snowflake for scalable cloud data warehousing.
• Data Pipeline Development: Design and oversee the implementation of robust, automated, and observable data pipelines for batch, streaming, and real-time data integration from diverse sources.
• Solution Integration: Architect and guide the integration of new and existing systems, applications, and third-party services, ensuring data flow, API design, and security protocols are meticulously defined.
• Technical Leadership & Mentorship: Provide technical leadership, guidance, and mentorship to engineering teams, fostering a culture of innovation, excellence, and continuous improvement.
• Stakeholder Collaboration: Collaborate extensively with product owners, business analysts, data scientists, and senior leadership to gather requirements, present architectural options, and gain consensus.
• Technology Evaluation: Continuously evaluate new technologies, tools, and methodologies in data, AI, and cloud spaces, recommending strategic investments and fostering innovation.
• Security & Compliance: Ensure all architectural designs incorporate robust security measures, data governance, and compliance with relevant regulations (e.g., GDPR, HIPAA, industry-specific standards).

Qualifications:

Required Qualifications:

6+ years of progressive experience in IT with at least 5+ years in a dedicated engineering role, focusing on data, analytics, and AI/ML initiatives. Extensive experience designing and developing identity spines/graphs in both the paid and owned audience spaces. Demonstrable expertise in Solution Architecture Design, including creating architectural diagrams, design documents, and technical specifications for complex systems. Deep hands-on experience with modern data platforms, specifically Databricks and Snowflake, including data lakehouse patterns, SQL warehousing, and MLflow. Extensive experience designing and implementing solutions on major public cloud platforms, with strong proficiency in AWS (e.g., Q, S3, EC2, Lambda, Glue, SageMaker, RDS) and/or Azure (e.g., Data Lake, Data Factory, Synapse Analytics, Azure ML, Cosmos DB). Proven track record in designing and optimizing robust data pipelines using various integration patterns (ETL/ELT, streaming) and tools. Strong programming skills in Python for data engineering, scripting, and automation. Expert-level proficiency in SQL for data manipulation, querying, and optimization. Experience with Java is highly desirable, particularly in enterprise-level application integration or backend services. Solid understanding of Artificial Intelligence (AI) and Machine Learning (ML) concepts, including model development lifecycle, MLOps principles, and integrating ML models into production systems. Experience with various integration patterns (API, message queues, event-driven architectures) and technologies.

Preferred Qualifications:

Platform certifications (e.g., Data Engineer (Professional) – Databricks, Gen AI Engineer (Professional) – Databricks, etc.) Experience with other data technologies such as Kafka, Spark, Flink. Familiarity with containerization (Docker, Kubernetes) and CI/CD practices. Experience building reusable feature pipelines Experience ensuring training/serving parity Management of embedded pipelines Hands-on experience with Vector DB integrations The following skills are nice to have, and expertise is not required: Adobe (AEP, AJO, CJA, Campaign, Audience Manager, Analytics) Salesforce (Marketing Cloud, Data Cloud) Microsoft Power BI/Tableau