Databricks Platform Engineer

Job Title - Databricks Platform Administrator / Engineer
Location - Hybrid
Duration - 12+ Months



Job Summary: We are seeking a skilled Databricks Platform Administrator / Engineer with proven experience supporting modern data platforms in a Property & Casualty (P&C) insurance setting. The ideal candidate will be responsible for managing, configuring, optimizing, and maintaining the Databricks environment to support data engineering, analytics, and machine learning workloads in a secure and compliant manner.



Education: Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or related field.



Required Qualifications:

• 5+ years of experience in data platform engineering or system administration.
• 2+ years of hands-on experience with Databricks administration on Azure
• Solid understanding of Apache Spark , Delta Lake, and Lakehouse architecture.
• Proficiency in scripting and automation using Python, Bash, PowerShell , or Terraform .
• Familiarity with data security, audit, and compliance frameworks relevant to P&C insurance (e.g., GDPR, NYDFS, SOC 2).
• Experience with Azure Data Services , ADLS , Key Vault , and Azure DevOps
• Strong understanding of the P&C insurance data ecosystem (policy, claims, underwriting, actuarial, etc.).

Preferred Qualifications:

• Databricks Certified Associate / Professional (Administrator / Data Engineer) certification.
• Experience integrating Databricks with core systems like Guidewire, Duck Creek, or custom P&C platforms.
• Knowledge of industry standards like ACORD data models and ISO data feeds.
• Exposure to tools like Unity Catalog, Purview, Collibra, Informatica, or similar.

Soft Skills:

• Strong problem-solving and analytical skills.
• Ability to work cross-functionally in an agile team environment.
• Excellent communication and stakeholder management skills.
• Experience working in regulated enterprise environments with strong data governance standards.

Key Responsibilities:

• Platform Administration & Management:
  • Manage and administer Databricks workspaces, clusters, jobs, libraries, and notebooks.
  • Implement access controls using Unity Catalog, SCIM, and workspace-level RBAC.
  • Configure and monitor autoscaling, cluster policies, and compute usage to optimize cost and performance.
  • Support platform upgrades, patching, and operational automation.
• Security, Compliance & Governance:
  • Enforce security policies aligned with P&C insurance industry regulations.
  • Integrate with enterprise identity providers (Azure AD, Okta) and manage SSO.
  • Support data governance using tools like Unity Catalog, Immuta, or Collibra.
• DevOps & Automation:
  • Build CI/CD pipelines for notebooks, jobs, and ML workflows using tools like Azure DevOps, GitHub Actions, or Jenkins.
  • Automate platform provisioning and configurations using Infrastructure-as-Code (Terraform, ARM, etc.).
• Monitoring & Troubleshooting:
  • Set up logging, monitoring, and alerting using tools like Azure Monitor, Datadog, or Splunk.
  • Troubleshoot performance issues, job failures, and cluster errors.
• Collaboration & Enablement:
  • Work closely with Data Engineers, Data Scientists, and Business Analysts to ensure optimal platform usage.
  • Develop platform usage guidelines, onboarding documentation, and best practices.
  • Provide technical support and training to users.