Senior Data Engineer

Role Description

We are looking for a Senior Data Engineer to join the Innovation team as a core member of the PF-LLM programme — our initiative to build a from-scratch multivariate time-series foundation model across a fleet of ~1,000 wind and PV sites. You will be the connective tissue of the entire programme:

• Owning the data foundation that makes state-of-the-art model training possible.
• Managing the inference service that makes model outputs usable.
• Overseeing platform integration that puts those outputs in front of pilot customers.
• From production ETL through to shadow-mode validation pipelines, you will be the engineer who keeps every track moving.

This role is critical-path from day one.

Qualifications

• 6+ years of back-end and data engineering experience, with a proven track record of shipping production systems.
• Production-grade ETL/ELT pipeline design at scale: idempotency, retry logic, backfill jobs, incremental loading, and cost-controlled warehouse compute.
• Schema design and data modelling across heterogeneous sources — experience reconciling signals from disparate systems into a canonical, queryable format.
• Data quality engineering: automated quality gates (sparsity, flatline detection, outlier flagging, freshness checks), alerting pipelines, and dataset versioning for ML reproducibility.
• API design and development: RESTful inference services with contract testing, latency and throughput budgeting, and structured observability (logs, metrics, traces).
• Experience integrating ML model outputs into SaaS product surfaces: auth and authorisation, customer isolation, and feature flag management.
• Cloud infrastructure proficiency (AWS preferred), containerisation (Docker, Kubernetes), and CI/CD pipeline ownership.
• Python and SQL as core tools; hands-on experience with modern warehouse technologies (Snowflake, BigQuery, or Databricks).
• Pipeline orchestration with Airflow, Prefect, Dagster, or equivalent.
• Excellent written and verbal communication skills in English.

Requirements

• Design and build the production ETL pipeline from source systems to warehouse and feature store at fleet scale, covering thousands of wind and PV sites across multiple OEMs.
• Own canonical signal schema design across wind and PV asset classes and OEMs — the deepest technical unknown in the programme and the foundation everything else depends on.
• Implement automated data quality gates: sparsity and missingness checks, flatline detection, outlier flagging, and freshness validation, with alerting that generates tickets automatically.
• Implement dataset versioning sufficient to reproduce every trained model from scratch.
• Build and maintain backfill jobs, idempotency guarantees, and retry logic that survive mid-run failure without duplicating data.
• Govern storage and compute costs on the warehouse from day one.
• Build the batch and on-demand inference API with contract tests, sized for fleet-wide daily runs.
• Establish latency and throughput baselines; own the cold-start and model-loading strategy.
• Instrument the service with structured logs and metrics from the outset.
• Integrate forecasts into the Power Factors product platform: auth and authorisation with customer isolation, observability hooked into the existing stack, and feature flags per customer and per site.
• Build and maintain the shadow validation pipeline: run live inference in parallel with the existing forecast path, log predictions and actuals, and produce weekly validation reports broken down by asset class, OEM, and region.
• Support the pilot customer rollout: enable the product for friendly customers behind flags and own incoming data and integration tickets throughout the pilot window.
• Work closely with the ML Engineer to align on data quality requirements, feature store interfaces, and the handoff between the data platform and training pipeline.
• Partner with the Tech Lead and Frontend Engineer during platform integration to ensure a clean, maintainable integration surface.
• Contribute to architectural decisions across the programme and document data flows, schemas, and pipeline runbooks to a standard that supports the broader team.

Benefits

• Comprehensive benefits package including health, dental, and vision coverage, plus dedicated wellness support.
• Generous paid vacation policy.
• Employer RRSP matching program.
• Work-from-abroad opportunities with manager approval.
• Exposure to a global team operating across multiple countries and time zones.
• A humble cause with a clear purpose — you will help us fight climate change with code every day at work.