Databricks Platform Engineer (New York)

Azure Databricks Platform Engineer

We're looking for an Azure Databricks Platform Engineer to take ownership of the administration, governance, security, and operational health of an enterprise Databricks environment. This individual will ensure the platform remains secure, reliable, properly governed, and cost-efficient while supporting a growing portfolio of enterprise data, analytics, and AI workloads.

This is a platform administration role rather than a traditional data engineering position. The ideal candidate has hands-on experience owning Databricks environments and understands how to manage Unity Catalog, access controls, workspaces, compute resources, production jobs, and platform governance.

What You'll Do

• Administer Databricks environments supporting development, testing, pre-production, and production workloads.
• Own day-to-day platform health by monitoring workloads, scheduled jobs, compute resources, availability, and other operational activity.
• Administer Unity Catalog and maintain scalable governance standards for enterprise data, permissions, and access.
• Manage users, groups, roles, and access controls while ensuring sensitive information is appropriately protected across the platform.
• Troubleshoot Databricks platform issues involving permissions, data access, jobs, compute, workspace configuration, and connectivity.
• Configure and optimize clusters and compute resources based on workload requirements, performance, utilization, and cloud cost considerations.
• Establish consistent administrative and security standards as additional users, datasets, and business workloads are introduced to the platform.
• Support controlled promotion of platform configurations and security standards across development and production environments.

What We're Looking For

• Hands-on experience administering Databricks in a production enterprise environment , with direct responsibility for the platform rather than solely developing notebooks or data pipelines.
• Strong experience with Unity Catalog , including governance, permissions, catalogs, schemas, users/groups, and enterprise data-access controls.
• Experience administering Databricks workspaces, clusters/compute, jobs, security, permissions, and platform configurations.
• Demonstrated experience troubleshooting production Databricks issues and supporting the operational health of the platform.
• Understanding of role-based access, least-privilege security, sensitive-data protection, and scalable enterprise governance practices.
• Experience monitoring and optimizing Databricks compute for performance, utilization, reliability, and cost.
• Experience supporting multiple Databricks environments and maintaining consistent configurations and security standards between them.
• Azure Databricks experience is strongly preferred; candidates with significant Databricks administration experience in another major cloud environment may also be considered.
• Familiarity with Azure identity, storage, networking, monitoring, or security services is preferred.
• Exposure to automation, Infrastructure as Code, Terraform, CI/CD, GitHub, or Azure DevOps is beneficial.
• Knowledge of Microsoft Purview, Microsoft Fabric, Delta Lake, Lakehouse architecture, Power BI, or enterprise AI integrations is a plus.

Join us at Lumen Resources, where your expertise will contribute to our growth, and you'll have the opportunity to have fun and thrive alongside a dynamic group of professionals. Apply today and become a part of our exciting journey!

Lumen Resources:

At Lumen Resources you will be partnering with trained recruiters who will guide you through discovering a brighter future. Our recruiters are concise and efficient in all interactions, hold exceptional communication, and will support resume writing and interview preparation, so you can land any position you want!

If this role doesn't align with what you are looking for, we have a host of clients with roles that could be a match. Feel free to reach out to connect with one of our team members at View email address on apply.placed-app.com

Lumen Resources is an Equal Opportunity Employer. Don't hesitate to get in touch with our team for more information.