Data Engineer - DataBricks

Data EngineerPosition Summary: The Data Engineer will play a critical role in designing, building, and managing the data infrastructure that powers the Digital department’s Analytics functions. This position is responsible for developing and maintaining robust, scalable, and high-quality data pipelines and solutions. In this important role, you will help shape the future of data-driven decision-making at Client. By developing innovative data solutions and collaborating with diverse teams, you will directly contribute to our digital transformation and enhance customer experiences. You will join a collaborative and high-performing team that values innovation, creativity, and a passion for data.Essential Duties and Responsibilities:Data Pipeline Development: Design, build, and maintain scalable, reliable ETL/ELT pipelines using Databricks, Spark, SQL, and Python to process large volumes of data.Data Architecture & Modeling: Implement and optimize data models within the Databricks Lakehouse Platform, focusing on creating and managing Delta Tables for performance and reliability.Technical Expertise: Serve as a subject matter expert on Databricks, Spark, and Delta Lake. Provide technical guidance on data engineering best practices, including data validation, performance testing, and production readiness.Process and Workflow Optimization: Support the standardization and automation of data workflows, including requirements gathering, architecture planning, development, quality control, and production deployment.Data Quality and Governance: Implement robust data quality checks and help ensure data governance standards are consistently met throughout the data lifecycle.Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand their data needs and deliver high-quality, actionable data solutions.Documentation and Knowledge Management: Develop and maintain comprehensive documentation for data pipelines, table structures, dependencies, and business logic to ensure clarity and accessibility.Continuous Improvement and Innovation: Stay current with industry trends and proactively identify and implement enhancements to data processes, tools, and workflows.Cost Management and Optimization: Monitor and optimize data pipelines and cloud resource usage to ensure cost-effectiveness, implementing strategies to reduce compute and storage costs.CI/CD and Automation: Develop and maintain CI/CD (Continuous Integration/Continuous Deployment) pipelines for data workflows to improve deployment frequency and reliability.Security and Compliance: Partner with security teams to implement data encryption, access controls, and data masking techniques to ensure compliance with data privacy regulations.Preferred Qualifications:Bachelor’s degree in computer science, Data Science, or a related field preferred, or equivalent work experience. MBA or MS a bonus.Minimum of 3-5 years of experience in data solution design, development, and implementation.Demonstrated expertise in big data technologies, including Azure Data Factory, Databricks, AWS, as well as BI Systems and Adobe Analytics or other log-level data.Demonstrated ability in advanced performance tuning of Spark jobs and Databricks clusters.Proficiency in SQL, Python, and Agile (Scrum) methodologies.Excellent leadership, project management, and communication skills.Experience mentoring junior data engineers on technical concepts and best practices.Experience leading technical projects and supervising a team of data developers.Hands-on experience with data integration, segmentation, and campaign performance analysis.Familiarity with Agile collaboration tools like JIRA.Strong organizational and multitasking abilities.Proven ability to lead technical discussions and provide strategic direction.Ability to translate business needs into technical solutions.Demonstrated commitment to staying updated with industry trends and adopting new technologies or methodologies.