Data Engineer - DataBricks

Data Engineer

Position Summary: The Data Engineer will play a critical role in designing, building, and managing the data infrastructure that powers the Digital department’s Analytics functions. This position is responsible for developing and maintaining robust, scalable, and high-quality data pipelines and solutions. In this important role, you will help shape the future of data-driven decision-making at Client. By developing innovative data solutions and collaborating with diverse teams, you will directly contribute to our digital transformation and enhance customer experiences. You will join a collaborative and high-performing team that values innovation, creativity, and a passion for data.

Essential Duties and Responsibilities:

• Data Pipeline Development: Design, build, and maintain scalable, reliable ETL/ELT pipelines using Databricks, Spark, SQL, and Python to process large volumes of data.
• Data Architecture & Modeling: Implement and optimize data models within the Databricks Lakehouse Platform, focusing on creating and managing Delta Tables for performance and reliability.
• Technical Expertise: Serve as a subject matter expert on Databricks, Spark, and Delta Lake. Provide technical guidance on data engineering best practices, including data validation, performance testing, and production readiness.
• Process and Workflow Optimization: Support the standardization and automation of data workflows, including requirements gathering, architecture planning, development, quality control, and production deployment.
• Data Quality and Governance: Implement robust data quality checks and help ensure data governance standards are consistently met throughout the data lifecycle.
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand their data needs and deliver high-quality, actionable data solutions.
• Documentation and Knowledge Management: Develop and maintain comprehensive documentation for data pipelines, table structures, dependencies, and business logic to ensure clarity and accessibility.
• Continuous Improvement and Innovation: Stay current with industry trends and proactively identify and implement enhancements to data processes, tools, and workflows.
• Cost Management and Optimization: Monitor and optimize data pipelines and cloud resource usage to ensure cost-effectiveness, implementing strategies to reduce compute and storage costs.
• CI/CD and Automation: Develop and maintain CI/CD (Continuous Integration/Continuous Deployment) pipelines for data workflows to improve deployment frequency and reliability.
• Security and Compliance: Partner with security teams to implement data encryption, access controls, and data masking techniques to ensure compliance with data privacy regulations.

Preferred Qualifications:

• Bachelor’s degree in computer science, Data Science, or a related field preferred, or equivalent work experience. MBA or MS a bonus.
• Minimum of 3-5 years of experience in data solution design, development, and implementation.
• Demonstrated expertise in big data technologies, including Azure Data Factory, Databricks, AWS, as well as BI Systems and Adobe Analytics or other log-level data.
• Demonstrated ability in advanced performance tuning of Spark jobs and Databricks clusters.
• Proficiency in SQL, Python, and Agile (Scrum) methodologies.
• Excellent leadership, project management, and communication skills.
• Experience mentoring junior data engineers on technical concepts and best practices.
• Experience leading technical projects and supervising a team of data developers.
• Hands-on experience with data integration, segmentation, and campaign performance analysis.
• Familiarity with Agile collaboration tools like JIRA.
• Strong organizational and multitasking abilities.
• Proven ability to lead technical discussions and provide strategic direction.
• Ability to translate business needs into technical solutions.
• Demonstrated commitment to staying updated with industry trends and adopting new technologies or methodologies.