DataOps/Cloud Data Engineer

Job Title: DataOps/Cloud Data Engineer - Senior

Location: Toronto, Ontario

Work Model: Onsite

Duration: 12 months

Extension: 1 Time

Must Haves

• Experience developing ETL processes with Informatica, Azure Data Factory and Databricks (Python, Lakeflow, and SQL Optimization)
• Experience with solution development with Databricks in Medallion Architecture.
• Experience with Cloud data engineer must be familiar and experienced with different programming languages (Python, SQL, t-SQL, PL/SQL, Informatica, ADF, SSIS. Microsoft Fabric) and be able to integrate with many different platforms to create data pipelines, jobs, automate tasks, and write scripts.
• Experience with Data pipeline and workflow development, orchestration, deployment, and automation and specializing in programming and pipelines to create and managing the dataflow, Delta Lake indexing, parallelism and movement of data.

Description / Responsibilities / Skills

Description

• To support Post Secondary Analytics (PSAE) from FY26-27 to FY27-28 to migrate from legacy infrastructure to Ontario Data Platform (ODP).
• Designing and developing data pipelines from source to end user
• Optimizing data pipelines
• Designing, developing, and optimizing Azure Data Factory and Databricks pipelines from Oracle databases to Lakehouses
• Translating Informatica ETL to Azure Data Factory and Databricks ELT
• Designing and optimizing relational data models

Technical Experience

• Extensive experience in designing/developing and implementing data conversation and migration of VLD (Very large Data) of OLAP and OLTP environments to Cloud SaaS, PaaS and IaaS environments.

Core Skills

• Demonstrates experience working with and analyzing large and diverse data sets, data visualization, KPI development and data profiling.
• Demonstrates experience creating detailed documentation for tools developed, such as business, system and technical requirements.

Project Experience and Techniques

• Demonstrates experience gathering and consolidating input into changes required to business and/or system requirements and leading requirement gathering sessions.
• Demonstrates experience communicating regarding day-to-day tasks and issue tracking, reporting and facilitating resolution of issues and risks.

Knowledge Transfer

• Transfer From DataOps/Cloud Data Engineer to Designated CSC Resource
• All deliverables, including design/supporting/release/training documents must be checked into designated version control repositories.
• Project manager and designated ministry staff must be regularly informed in writing of where documentation has been stored.
• Knowledge will be transferred through 1 on 1 sessions, emails, document updates and document review with the team.

Deliverables

• Experience managing Cloud Data services including storage, repositories, Data Lakehouse, key vault, virtual machine, Azure Storage Account, SIHR, parquet files etc.
• Designing and optimizing data models using Medallion Architecture principles
• Designing, developing and optimizing data connections from Databricks Medallion Architecture to on-premises data sources for downstream consumers.

Experience & Skill Set Requirements

Technical Experience - 40%

• Experience with Cloud data platforms, data management and data exchange tools and technologies
• Experience with Commercial and opensource data and database development and management and specializing in data storage setting (Delta Lake) and managing cloud Data As a Service (DaaS). application Database as a Service (DBaaS), Data Warehouses as a Service (DWaaS), and other storage platforms (both in the cloud and on-premise).
• Experience in Continuous Integration/Continuous Development/Deployment (CI/CD) and Data provisioning Automation
• Experience with digital product, data analysis, data exchange, data integration APIs, data provisioning, and data security
• Extensive experience in designing/developing and implementing data conversation and migration of VLD (Very large Data) of Online analytical processing (OLAP) and online transaction processing (OLTP) environments to Cloud Software-as-a-service (SaaS), Platform-as-a-service (PaaS) and Infrastructure-as-a-service (IaaS) environments.
• Experience in design, development and implementation of fact/dimension model, data mapping, data warehouse, data lakehouse for enterprise.
• Demonstrated experience developing star schema multi-dimensional models and documenting detailed design models.
• Experience managing Cloud Data services for project delivery, including storage, repositories, Data Lakehouse, key vault, virtual machine, Azure Storage Account, SIHR, parquet files etc.
• Experience with structured, semi-structured, unstructured data collection, ingestion, provisioning and exchange technological development of enterprise data warehouse and data lake and data lakehouse solutions and operational support.
• Experience with DataOPS performance monitoring and tuning
• Implementing and enforcing data quality checks and principles, including data validation, profiling, cleansing, and monitoring across pipelines
• Applying data governance practices and information architecture standards, including development of conceptual, logical, and physical data models
• Integrating and managing Microsoft Entra ID for secure authentication, authorization, and role-based access control across data platforms
• Implementing data anonymization and masking techniques to protect sensitive and regulated data in compliance with privacy requirements
• Developing and maintaining data lineage reports to provide end-to-end visibility and traceability of data movement and transformations
• Databricks and/or Microsoft Fabric certifications are considered an asset

Core Skills - 35%

• Excellent analytical, problem-solving and decision-making skills; verbal and written communication skills; presentation skills; interpersonal and negotiation skills
• Demonstrates development and solution design experience with technology, data, databases, statistics, applications, and networking techniques, tools and practices that enables the candidate to lead the design and development of business intelligence projects.
• Experience in DataOPS best practices, and Agile project development and deployment
• Demonstrates experience in working with and analyzing large and diverse data sets, data visualization, KPI development and data profiling to identify relationships and themes.
• Demonstrates experience manipulating and analyzing complex, high-volume data from structured and unstructured sources.
• Demonstrates experience creating detailed documentation for the tools that are developed, such as business requirements, system requirements and technical requirements.
• Demonstrates experience with advanced data visualization tools and techniques, designing high quality interfaces to present information in a meaningful way.
• Troubleshooting of production issues and creating fixes for any unplanned deployments.
• Defect investigation, resolution and assignment to team members using defect/issue.
• Demonstrated experience in coordinating and planning for the modernization of large, complex multi-platform, multitier information technology systems.
• Experience with Azure DevOps.
• Experience with AODA/WCAG compliance.
• A team player with a track record for meeting deadlines

Project Experience and Techniques - 15%

• Demonstrates development and design experience with SDLC processes, Agile and Waterfall methodologies.
• Demonstrates experience conducting system testing and unit testing, and conducting SIT, SAT and UAT Testing.
• Demonstrates experience communicating regarding day-to-day tasks and issue tracking, reporting and facilitating resolution of issues and risks for all project activities.
• Demonstrates experience gathering and consolidating input into changes required to business and/or system requirements, change requests, project artefacts, as well as leading requirement gathering sessions.

General Skills - 5%

• Demonstrates experience problem solving and decision making.
• Demonstrates experience communicating clearly in both written and verbal formats.
• Demonstrates experience working with and leading multiple teams representing various areas in delivering the project.
• Demonstrates ability to work collaboratively, with the ability to coordinate multiple projects with competing priorities and a track record for meeting strict deadlines.
• Demonstrates experience working collaboratively with other team members and/or groups and leading a team to ensure optimal solution integration.
• Demonstrates experience leading a team and with providing technical advice and guidance.
• Demonstrates experience working with both the business users and IT development teams to ensure business requirements are properly reflected in the system design and technical specifications.

Organization Experience - 5%

• Previous public sector experience in an organization of equivalent size.