Databricks Technical Lead Developer

We are looking for an experienced, hands-on Databricks Lead who will work with the client and lead the design and delivery of scalable data solutions. The role requires strong Databricks expertise, technical leadership, client communication, and the ability to translate business requirements into practical data-engineering solutions.

Key Responsibilities

• Client Engagement: Work closely with client stakeholders to understand business requirements, provide technical recommendations, and communicate solution designs, delivery progress, risks, and dependencies.
• Solution Design: Lead the architecture and design of scalable batch and streaming solutions using Databricks, PySpark, Spark SQL, and Lakeflow Declarative Pipelines (formerly Delta Live Tables).
• Data Pipeline Development: Design, build, test, and maintain reliable data-ingestion and transformation pipelines.
• Lakehouse Implementation: Design and manage Delta Lake solutions using Medallion Architecture across Bronze, Silver, and Gold layers.
• Performance Optimization: Optimize Spark jobs, queries, cluster configurations, and storage patterns to improve performance and control cloud costs.
• Security and Governance: Implement data access controls, governance, lineage, auditing, and secure data-sharing practices using Unity Catalog.
• Delivery Leadership: Own technical delivery, effort estimation, sprint planning, dependency management, issue resolution, and adherence to agreed timelines.
• DevOps and Automation: Establish Git-based development, automated testing, CI/CD pipelines, and infrastructure deployment using Terraform.
• Production Support: Lead troubleshooting, root-cause analysis, performance monitoring, and resolution of production issues.
• Team Leadership: Guide onsite and offshore teams, conduct code and design reviews, establish engineering standards, and mentor junior developers.
• Stakeholder Communication: Conduct technical workshops, solution walkthroughs, sprint demonstrations, and status discussions with business and technology stakeholders.

Required Technical Skills

• Programming: Advanced Python, PySpark, and SQL
• Databricks: Delta Lake, Lakeflow Declarative Pipelines, Unity Catalog, Databricks Workflows, MLflow, and Databricks SQL
• Data Engineering: Apache Spark, batch and streaming processing, data warehousing, dimensional modelling, and Medallion Architecture
• Cloud Platforms: Strong experience with at least one cloud platform- Azure
• Cloud Storage: ADLS
• DevOps: Git, Azure DevOps or GitHub Actions, CI/CD, and Terraform
• Security: Role-based access control, data governance, auditability, and secure data-management practices
• Operations: Performance tuning, monitoring, troubleshooting, and production support

Leadership and Client-Facing Skills

• Strong client-facing communication and stakeholder-management skills
• Ability to translate business requirements into scalable technical solutions
• Experience leading distributed onsite and offshore delivery teams
• Strong problem-solving, decision-making, and conflict-resolution abilities
• Ability to explain complex technical topics clearly to both technical and business audiences
• Experience managing delivery risks, priorities, dependencies, and client expectations

Preferred Qualifications

• Databricks certification
• Experience delivering enterprise-scale cloud data-modernization or migration programs
• Experience working in Agile delivery environments
• Exposure to data-quality frameworks, observability, and cost-governance practices
• Domain experience in healthcare or life sciences

Education

• Bachelors or Masters in Information Technology, Computer Science or relevant field.