Senior Data Engineer

Senior Data Engineer

The Senior Data Engineer will help transform our cloud data systems by designing and operating architectures that drive analytical and business value from a wide range of data sources. This role partners closely with analysts, traders, product owners, and IT teams to deliver high-performance, resilient, and automated data pipelines, curated analytical datasets, and governed semantic models.

The role requires strong judgment in selecting and applying the right technologies-across Snowflake, Databricks, Azure data services, and traditional databases-based on workload characteristics, performance, and cost. The Senior Data Engineer will also apply AI enabled techniques (semantic layers, RAG, natural-language-to-data experiences) to improve data discoverability and usability while maintaining high standards for data quality, security, and lineage.

Essential Duties & Responsibilities
• Design and operate Snowflake-centric analytical architectures supporting mixed workloads, including heavy read/query patterns, reporting, downstream applications, and AI/RAG use cases.
• Evaluate and apply the appropriate platform (Snowflake, Databricks, Postgres, ADLS) based on workload requirements, performance characteristics, and cost considerations.
• Build and maintain scalable, automated data ingestion and refresh pipelines at terabyte scale using Azure Data Factory, Azure Functions, Azure Logic Apps, Databricks, Python, and Snowflake.
• Integrate data from external vendors and internal systems using APIs, streams, flat files, event feeds, and relational databases; implement robust incremental and backfill strategies.
• Design and develop analytical data models, including dimensional models (facts, dimensions), conformed dimensions, and SCD patterns that balance usability, performance, and maintainability.
• Build and maintain governed semantic models / semantic layers (business entities, measures, metrics, hierarchies) to ensure consistent data consumption across BI tools, APIs, and AI-driven interfaces.
• Optimize Snowflake performance and cost, including warehouse sizing, query tuning, clustering and pruning strategies, and SQL best practices.
• Own operational readiness for data pipelines, including monitoring, alerting, runbooks, incident response, and ongoing reliability improvements.
• Develop and implement data quality validation and testing frameworks, including schema validation, reconciliation, anomaly detection, and freshness/completeness checks.
• Plan and execute work using agile methodologies, contributing to technical design reviews, documentation, and knowledge sharing.
• Collaborate directly with analysts and business stakeholders to understand data usage, clarify requirements, and translate data needs into actionable technical designs.

Required Experience and Skills
• Bachelor's degree in computer science, Engineering, Data Science, or equivalent practical experience.
• Strong, hands-on experience with Snowflake in production environments, including data loading patterns, query optimization, and cost management.
• Advanced SQL expertise (complex ANSI-SQL, window functions, performance tuning) and solid data warehousing fundamentals.
• 6+ years of experience with relational databases (e.g., SQL Server, Postgres, MySQL, Oracle), including schema design and query optimization.
• 6+ years of experience building and operating data ingestion and transformation pipelines on large datasets (batch and incremental).
• 2+ years of experience with Spark or distributed data processing frameworks (Databricks, Hadoop/Cloudera).
• 2+ years of experience with Azure data services, including Azure Data Factory, Azure Functions, Logic Apps, ADLS Gen2, Azure SQL, and CI/CD tooling (Azure DevOps or equivalent).
• Strong experience in data modeling, including dimensional modeling, SCDs, and designing curated "gold" datasets.
• Experience working with modern data file formats and ingestion strategies (Parquet, Avro, JSON; partitioning, compression, schema evolution).
• Proven experience supporting enterprise data quality, governance, and documentation.
• Practical experience applying AI to data platforms, including semantic models, RAG pipelines, or natural-language-to-data solutions.
• Strong Python programming skills for data acquisition, orchestration, and automation.
• Excellent communication skills, with the ability to explain technical concepts clearly to both technical and non-technical stakeholders.
• Demonstrated ownership mindset, strong troubleshooting skills, and commitment to continuous improvement through automation and better platform design.

Range: $160,000 - $190,000