Senior Data Engineer

We are seeking a Senior Data Engineer to design, develop, and optimize enterprise-scale data pipelines and platforms supporting Finance and Data Intelligence initiatives. The ideal candidate will have strong expertise in Snowflake, Databricks, Delta Lake, Delta Live Tables (DLT), Unity Catalog, dbt, and PySpark, along with experience building reliable, scalable, and secure data solutions.

Key Responsibilities

• Design and develop scalable ELT/ETL pipelines for finance and revenue data, including billing, general ledger (GL), and operating expenses (OPEX).
• Build batch, incremental, and streaming ingestion pipelines using CDC, watermarking, and event-driven patterns.
• Ensure pipeline reliability through idempotent design, error handling, retries, and fault-tolerance mechanisms.
• Lead development and optimization of Snowflake and Databricks platforms, including Snowpipe, streams, tasks, Delta Live Tables, and Unity Catalog.
• Develop modular, tested, and reusable data transformations using dbt and PySpark.
• Implement cloud-native data platforms using Azure or AWS, including infrastructure-as-code and cost optimization.
• Develop real-time processing solutions using Kafka, Event Hub, and Spark Structured Streaming.
• Establish data quality, validation, monitoring, testing, and observability standards.
• Implement CI/CD practices, version control, and automated deployment workflows.
• Ensure data security, governance, lineage, auditability, and compliance, including RBAC and PII/CPNI controls.
• Collaborate with architects, product managers, analysts, and business stakeholders to deliver reliable data solutions.
• Provide technical leadership, mentor engineers, and drive platform standards and continuous improvement.

Required Qualifications

• 7+ years of experience with Snowflake and Delta Lake.
• 7+ years of experience with Databricks, Delta Live Tables (DLT), and Unity Catalog.
• 5+ years of experience with dbt and PySpark.
• Strong expertise in SQL, Python, data modeling, and enterprise-scale data pipeline development.
• Experience with ETL/ELT, incremental ingestion, CDC, pipeline optimization, and fault-tolerant processing.
• Knowledge of cloud data platforms, preferably Azure or AWS.
• Experience with data quality, testing, CI/CD, security, and data governance.
• Strong problem-solving, communication, and technical leadership skills.

Preferred Qualifications

• Experience with Airflow or Azure Data Factory.
• Familiarity with Terraform or Bicep for infrastructure provisioning.
• Experience with Kafka, Event Hub, and Spark Structured Streaming.
• Knowledge of finance and revenue data, including billing, GL, revenue recognition, reconciliation, and period-end close.
• Experience with GitHub Actions, Azure DevOps, Great Expectations, and enterprise data modeling.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.