Databricks Platform Engineer

Databricks Platform Engineer

We are seeking a skilled Databricks Platform Engineer to design, implement, and maintain our enterprise Databricks environment. This role will focus on enabling scalable data processing, optimizing performance, and ensuring security and compliance across the platform. The ideal candidate will have strong experience with cloud technologies, data engineering principles, and automation practices.

Databricks Platform Engineer Key Responsibilities

• Platform Administration & Optimization
  • Deploy, configure, and manage Databricks workspaces, clusters, and jobs.
  • Monitor platform performance and optimize resource utilization.
  • Implement best practices for cost management and scalability.
  • Knowledge of Unity Catalog Implementation
• Security & Compliance
  • Configure identity and access management (IAM) for Databricks.
  • Ensure compliance with organizational security standards and data governance policies.
  • Manage secrets, tokens, and secure data access.
• Integration & Automation
  • Integrate Databricks with cloud storage, data lakes, and enterprise systems.
  • Automate cluster provisioning, job scheduling, and CI/CD pipelines.
  • Develop reusable scripts and templates for infrastructure as code (IaC).
• Collaboration & Support
  • Partner with Data Architecture, Enterprise Architecture, and Enterprise Infrastructure
  • Partner with data engineers, analysts, and data scientists to enable efficient workflows.
  • Provide technical support and troubleshooting for Databricks-related issues.
  • Document platform architecture, processes, and operational guidelines.

Databricks Platform Engineer Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, or related field (or equivalent experience).
• 3+ years of experience in cloud-based data platforms (AWS, Azure, or GCP).
• Hands-on experience with Databricks administration and development.
• Strong knowledge of Spark, Python, and SQL.
• Familiarity with CI/CD, Terraform, or other IaC tools.
• Understanding of data security, networking, and identity management in cloud environments.

Databricks Platform Engineer Preferred Qualifications

• Experience with AWS Databricks.
• Knowledge of Delta Lake, MLflow, GitHub, and advanced Databricks features.
• Exposure to data governance frameworks and compliance standards.
• Certifications in Databricks, AWS.

Databricks Platform Engineer Soft Skills

• Strong problem-solving and analytical skills.
• Excellent communication and collaboration abilities.
• Ability to work in a fast-paced, agile environment.