Azure Data Engineer

Hello;

Title: Sr. Data Engineer

Long Term

Location is Remote

product Tier 1 experience required.

8+ years experience

Must have

ETL/ELT pipeline development

Snowflake

Databricks

dbt

Azure

JD:

Principal Data Engineer Data & Intelligence

________________________________________

DATA PIPELINE DEVELOPMENT

Architect, design, and oversee development of enterprise-scale ELT/ETL pipelines for finance and revenue data (billing, revenue, GL, opex).

Define and enforce standards for batch, incremental, and streaming ingestion patterns (CDC, watermarking, event-driven ingestion).

Ensure idempotent, fault-tolerant, and highly scalable pipeline design across platforms.

Establish frameworks for error handling, retry strategies, dead-letter queue patterns, and operational resiliency.

Provide technical leadership for multi-source, high-volume data integration pipelines.

________________________________________

PLATFORM & TOOLING

Lead architecture and adoption of Snowflake and Databricks platforms for large-scale data processing and analytics.

Define best practices for:

o Snowflake (Snowpipe, streams, tasks, query optimization, cost efficiency)

o Databricks (PySpark, Delta Live Tables, Unity Catalog, job optimization)

o dbt (modular design, testing frameworks, CI/CD integration, reusable components)

Establish and govern orchestration frameworks using Airflow / Azure Data Factory, including DAG standards, dependency design, and monitoring.

Evaluate and drive tooling strategy and platform standardization across teams.

________________________________________

CLOUD INFRASTRUCTURE

Architect and optimize cloud-native data platforms on Azure (ADLS Gen2, Event Hub, ADF, Key Vault) or AWS equivalents.

Define standards for infrastructure-as-code (Terraform, Bicep) and environment provisioning.

Drive cost optimization strategies (compute sizing, storage design, partitioning, workload isolation).

Ensure platforms are scalable, secure, and production-ready.

________________________________________

LANGUAGES & FRAMEWORKS

Provide deep technical leadership in:

o Advanced SQL (query tuning, execution optimization, complex transformations)

o Python / PySpark for distributed data processing

Guide teams on best practices, reusable frameworks, and performance optimization.

Oversee development standards for Spark, Scala (where applicable), and automation scripting.

________________________________________

STREAMING & REAL-TIME

Architect real-time and near real-time data processing solutions using Kafka / Event Hub and Spark Structured Streaming.

Define patterns for stateful processing, watermarking, checkpointing, and fault tolerance.

Lead implementation of real-time finance/revenue use cases such as reconciliation, anomaly detection signals, and operational reporting.

________________________________________

DATA QUALITY & TESTING

Establish enterprise frameworks for data quality, validation, and observability.

Define standards for:

o Automated testing (unit, integration, regression)

o Data validation (completeness, accuracy, consistency)

o Data quality tools (dbt tests, Great Expectations, custom frameworks)

Ensure SLA monitoring, alerting, and data freshness tracking across all pipelines.

Drive proactive data quality and governance practices across teams.

________________________________________

DATA MODELING SUPPORT

Interpret and implement architect-defined enterprise data models (star, snowflake, data vault).

Provide guidance on:

o SCD (Type 1/2) strategies

o Partitioning, clustering, and performance optimization

Collaborate with architects to evolve scalable and reusable data models.

Support semantic layer enablement for analytics and reporting.

________________________________________

DEVOPS & ENGINEERING PRACTICES

Define and enforce CI/CD standards for data engineering (GitHub Actions, Azure DevOps).

Establish code quality, versioning, and deployment best practices (branching strategies, PR reviews, release pipelines).

Standardize environment promotion (dev QA prod) and release management.

Drive adoption of engineering excellence practices including reusable frameworks and templates.

________________________________________

SECURITY & GOVERNANCE

Lead implementation of enterprise-grade security and governance controls:

o RBAC, row/column-level security

o PII and CPNI compliance (TISS-310)

Define standards for secrets management and secure pipeline design.

Ensure data lineage, auditability, and compliance readiness across platforms.

________________________________________

FINANCE DOMAIN KNOWLEDGE

Deep understanding of finance and revenue data domains, including:

o Billing and revenue systems

o GL structures and financial reporting

o Revenue recognition and reconciliation

o Period-end close cycles

Guide engineering teams on accurate implementation of finance logic.

Ensure high data integrity standards for regulated financial data.

________________________________________

SOFT SKILLS & COLLABORATION

Act as a technical leader and escalation point across engineering teams.

Partner with architects, product managers, analysts, and business stakeholders.

Drive cross-team alignment and solution consistency.

Communicate complex technical topics clearly to both technical and non-technical audiences.

Lead incident reviews and ensure continuous improvement.

________________________________________

PRINCIPAL-LEVEL EXPECTATIONS

Own and drive enterprise-level data engineering strategy and execution.

Lead delivery of large, complex, multi-domain data platforms.

Mentor senior engineers and define technical direction for the team.

Drive tooling, architecture, and platform decisions across programs.

Identify and lead technical debt reduction and modernization initiatives.

Establish best practices, reusable components, and platform standards at scale.

Influence cross-functional teams and leadership decisions on data platform strategy.