Lead Data Engineer

Lead Data Engineer

The Lead Data Engineer will lead the design, development, and administration of critical data pipelines and platform infrastructure within an Azure Databricks environment. This role is focused on building robust, scalable, and secure data solutions that enable efficient data consumption and digital transformation across the organization. The Lead Data Engineer will be responsible for catalog architecture, data quality, security compliance, and stakeholder engagement, while independently managing complex projects from concept to delivery while keeping the organization's unique needs in consideration. Employee may perform other related duties as required to meet the ongoing needs of the organization.

Essential Responsibilities:

• Build, improve, manage and administrate a complex set of mission-critical data pipelines and data sets in a Databricks environment, with a heavy emphasis on stability, best practice and long-term continual improvement. Plan out and execute major catalog redesigns and rebuilds with a focus on stability, data integrity and zero data loss.
• Troubleshoot and resolve data pipeline, quality and resource issues efficiently with a high level of urgency.
• Lead adoption of the Databricks environment and modern methods of data consumption, digital transformation and automation solutions.
• Meet with stakeholders at every level of the organization to understand needs, objectives, requirements and set expectations. Train and assist users, teams and departments with furthering their adoption and utilization of Databricks. Assess and determine the success of current platform utilization while also identifying and defining new process improvement opportunities.
• Plan the adoption of new features and technologies within Databricks, Azure and the Power Platform.
• Independently manage multiple projects (solo and team-based).
• Support data security and governance within the Unity Catalog by maintaining data access controls, including PII-management, tokenization, row level security, masking and data usage logging and monitoring. Perform security audits and establish alerts.
• Perform data analysis, quality improvements, dashboarding, reporting and visualization of data. Assist data domain owners, report builders and analysts with report development (Power BI), improvement, migration and best practices.
• Serve as final approval of releases to production with thorough code and impact reviews, dependencies and change management.
• Serve as administrator of Power BI production workspaces, evaluation of best practices and compliance within reports, and certification of Power BI reports.
• Administrate all areas of Databricks including user access, resources, cost and security.

Essential Skills:

• Expert-level proficiency in developing advanced ETL pipelines using Databricks.
• Extensive experience with Unity Catalog, Delta Live Tables, data ingestion, transformation, cleansing, and governance methodologies.
• Advanced expertise in Python and SQL within Databricks Notebooks.
• Hands-on experience working with large, complex ERP data sets such as SAP, Infor, Dynamics 365, or Oracle.
• Strong understanding of Machine Learning (ML), Large Language Models (LLMs), and related concepts.
• Experience designing and implementing IoT and real-time streaming data pipelines.
• Proficiency with Azure DevOps, including repositories, version control, production deployments, and CI/CD best practices.
• Demonstrated experience in Azure and Databricks cost management, including resource administration, cost reporting, budgeting, and alerting.
• Bachelor's degree in Computer Science, STEM, or a related field (or equivalent experience).
• Minimum 3 years of experience working in a technology-focused role.
• At least 1 year of direct, hands-on experience with Databricks in an Azure environment.
• Familiarity with Agile/Scrum methodologies and collaborative development environments.
• Willingness to travel to Brentwood site locations and attend industry conferences (approximately 10% travel).

Preferred Certifications & Skills:

• Databricks Certified Data Engineer Professional
• Microsoft Certified: Azure Data Engineer Associate
• Experience with R, Scala, or other statistical programming languages

Brentwood offers professional growth potential, pleasant work environment, and an excellent wage and benefits package including 401K w/employer match. Brentwood Industries, Inc., provides equal employment opportunities to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics.