Senior Data Engineer

Title: Senior Data Engineer

Mode: Fulltime

Location: Toronto, ON

Senior Data Engineer (Databricks)

Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data.

Build and optimize data solutions using Databricks, PySpark, and Spark SQL.

Implement data ingestion frameworks from various source systems including databases, APIs, files, and streaming platforms.

Develop and maintain data lakehouse architectures leveraging Databricks best practices.

Design and implement Medallion Architecture (Bronze, Silver, Gold layers) for enterprise data platforms.

Collaborate with business stakeholders, data scientists, architects, and application teams to understand data requirements.

Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.

Implement data quality, governance, security, and monitoring frameworks.

Support AI/ML initiatives by preparing and engineering datasets for model development and deployment.

Develop CI/CD pipelines and automate deployment processes for data engineering workloads.

Participate in architecture reviews and establish data engineering standards and best practices.

Mentor junior engineers and provide technical leadership across projects.

Required Qualifications

Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, or related field.

8+ years of experience in Data Engineering and Data Platform development.

Strong hands-on experience with Databricks and Apache Spark.

Proficiency in Python, PySpark, and SQL.

Experience with cloud platforms such as:

o Microsoft Azure (preferred)

o AWS

o Google Cloud Platform

Experience with Delta Lake, Unity Catalog, and Databricks workflows.

Strong understanding of Data Lake, Data Warehouse, and Lakehouse architectures.

Experience working with large-scale distributed data processing systems.

Knowledge of data modeling, partitioning, indexing, and query optimization techniques.

Experience with Git, CI/CD pipelines, and DevOps practices.

Preferred Qualifications

Databricks Certified Data Engineer Associate or Professional Certification.

Experience with Azure Data Factory, Synapse Analytics, or equivalent cloud-native services.

Experience with streaming technologies such as Kafka, Event Hubs, or Spark Streaming.

Exposure to AI/ML platforms, MLOps, and Generative AI solutions.

Experience working in Insurance, Financial Services, or Enterprise Digital Transformation programs.

Technical Skills

Category Skills

Data Engineering Databricks, Apache Spark, PySpark, Spark SQL

Programming Python, SQL, Scala (Preferred)

Cloud Azure, AWS, GCP

Data Platforms Delta Lake, Unity Catalog, Data Lakehouse

ETL/ELT Azure Data Factory, Databricks Workflows, Airflow

DevOps Git, Azure DevOps, Jenkins, CI/CD

Streaming Kafka, Spark Streaming, Event Hubs

Databases SQL Server, PostgreSQL, Oracle, Snowflake