Azure Databricks Data Engineer

Role Description

This is a remote position.

Design and implement end-to-end data pipeline using Azure-native services and Databricks. This role focuses on:

• Building scalable, high-performance data platforms
• Defining robust data frameworks
• Partnering with stakeholders to translate business requirements into reliable data solutions

• Implement data governance principles using Unity Catalog
• Create and implement data ingestion patterns and data integration patterns in medallion architecture
• Apply best practices in data engineering
• Create PySpark notebooks
• Create procedures for advanced data transformation
• Handle performance tuning aspects in Databricks and Azure SQL DBs

Qualifications

• Hands-on experience with Azure Databricks
• Hands-on expertise in data ingestion, transformation, and orchestration frameworks
• Hands-on experience in building Insurance specific data models
• Proven ability to optimize data pipelines for performance, scalability, and cost efficiency
• Solid understanding of distributed data processing concepts

Requirements

• Hands-on experience with PySpark for large-scale data processing
• Experience defining and enforcing best practices for Databricks workloads and Spark-based pipelines
• Strong analytical and problem-solving skills
• Excellent communication skills to effectively work with cross-functional stakeholders
• Prior experience in working in Insurance P&C cloud-based data & analytics projects

Benefits

• Diversity Inclusion: Exavalu promotes flexibility, adapting to the needs of employees, customers, and the business
• Part-time work, working outside normal 9-5 business hours, or working remotely
• Welcome back program to help individuals return to the mainstream after a prolonged absence due to health or family reasons
• Commitment to building a diverse and inclusive workforce
• Welcoming applications for employment from all qualified candidates, regardless of race, colour, gender, national or ethnic origin, age, disability, religion, sexual orientation, gender identity, or any other status protected by applicable law
• Nurturing a culture that embraces all individuals and promotes diverse perspectives, where you can make an impact and grow your career