Principal Engineer - Sr. Azure Databricks

As a Principal Engineer I embedded in the Enterprise Data & Analytics function, you will be responsible for delivery and operations of technologies and platforms required to model, transform, analyze, report, visualize data.
You'll provide SME expertise in designing, building, optimizing, streamlining and automating the Azure Databricks platform to enable analytical workloads like Machine Learning (ML), Model Development, Data Insights, Data Science etc. As a Principal Engineer I, you will partner with ML engineers, data scientists, data analysts, and enterprise architects to provide frameworks, set standards, enforce best practices, train and enable u using the Azure Databricks platform.
In this role, you'll develop technical skills of one or more junior team-members.
- Take assignments that can be worked on individually without supervision, and manage work effort from concept to completion.
- Design, build, optimize, automate and maintain the Azure --Databricks platform, ensuring scalability, security, governance and performance.
- Design, implement and manage Azure Databricks workspaces, clusters, jobs, access management.
- Design, implement and manage policies, monitoring and observability for the Azure Databricks platform.
- Implement data analytics principles aimed at business enablement, reliability practices and sound recovery procedures.
- Ensure compliance with IT policies, procedures, and industry standards, including reviewing and refining IT control enhancements.
- Work closely with business and analytics teams to ensure reliable and governed data access for data needed for -analytics.
- Troubleshoot platform issues, optimize performance, and ensure uptime for critical Databricks services.
- Stay current on emerging data analytics technologies, recommending enhancements to improve efficiency and governance.
- Act as a thought leader on Azure Databricks implementation, usage, end user enablement and training.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:


Required Skills & Experience
- 8+ years of related experience in data analytics administration and development.
- 4+ years of Databricks related experience.
- Bachelor's degree in related field required, Masters in related field preferred.
- Previous leadership experience preferred.
- Advanced proven experience in Azure Databricks (Workspace management, Clusters, Jobs, Unity Catalog, Delta Lake, User access management, Rest APIs and SDKs).
- Knowledge of MLFlow & MLOps.
- Deep understanding of Azure infrastructure and data services, including Azure Data Lake, Azure Data Factory, Azure SQL, Azure Synapse Analytics, Azure Key Vault, Azure Monitor, networking
- Experience with CI/CD pipelines (Azure DevOps preferred)
- Strong programming skills in SQL, Python, and/or PySpark.
- Advanced proven experience in leading cross-functional teams and managing multiple projects simultaneously.
- Advanced ability to see the big picture and align projects with organizational goals. Capable of leading and motivating cross-functional teams. Expertise in resolving conflicts and addressing challenges as well as skilled at identifying and mitigating risks at the project level.
- Advanced communication skills, excellent teamwork experience, ability to mentor and develop more junior engineers.


Nice to Have Skills & Experience
- Advanced to expert knowledge of general Financial Services or Banking preferred.


Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.