Senior Data Engineer

Overview Senior Data Engineer will help transform our cloud data systems by designing and operating architectures that drive analytical and business value from a wide range of data sources. This role partners closely with analysts, traders, product owners, and IT teams to deliver high-performance, resilient, and automated data pipelines, curated analytical datasets, and governed semantic models. The role requires strong judgment in selecting and applying the right technologies—across Snowflake, Databricks, Azure data services, and traditional databases—based on workload characteristics, performance, and cost. The Senior Data Engineer will also apply AI enabled techniques (semantic layers, RAG, natural-language-to-data experiences) to improve data discoverability and usability while maintaining high standards for data quality, security, and lineage. Responsibilities Design and operate Snowflake-centric analytical architectures supporting mixed workloads, including heavy read/query patterns, reporting, downstream applications, and AI/RAG use cases. Evaluate and apply the appropriate platform (Snowflake, Databricks, Postgres, ADLS) based on workload requirements, performance characteristics, and cost considerations. Build and maintain scalable, automated data ingestion and refresh pipelines at terabyte scale using Azure Data Factory, Azure Functions, Azure Logic Apps, Databricks, Python, and Snowflake. Integrate data from external vendors and internal systems using APIs, streams, flat files, event feeds, and relational databases; implement robust incremental and backfill strategies. Design and develop analytical data models, including dimensional models (facts, dimensions), conformed dimensions, and SCD patterns that balance usability, performance, and maintainability. Build and maintain governed semantic models / semantic layers (business entities, measures, metrics, hierarchies) to ensure consistent data consumption across BI tools, APIs, and AI-driven interfaces. Optimize Snowflake performance and cost, including warehouse sizing, query tuning, clustering and pruning strategies, and SQL best practices. Own operational readiness for data pipelines, including monitoring, alerting, runbooks, incident response, and ongoing reliability improvements. Develop and implement data quality validation and testing frameworks, including schema validation, reconciliation, anomaly detection, and freshness/completeness checks. Plan and execute work using agile methodologies, contributing to technical design reviews, documentation, and knowledge sharing. Collaborate directly with analysts and business stakeholders to understand data usage, clarify requirements, and translate data needs into actionable technical designs. Required Experience and Skills Bachelor’s degree in computer science, Engineering, Data Science, or equivalent practical experience. Strong, hands-on experience with Snowflake in production environments, including data loading patterns, query optimization, and cost management. Advanced SQL expertise (complex ANSI-SQL, window functions, performance tuning) and solid data warehousing fundamentals. 6+ years of experience with relational databases (e.g., SQL Server, Postgres, MySQL, Oracle), including schema design and query optimization. 6+ years of experience building and operating data ingestion and transformation pipelines on large datasets (batch and incremental). 2+ years of experience with Spark or distributed data processing frameworks (Databricks, Hadoop/Cloudera). 2+ years of experience with Azure data services, including Azure Data Factory, Azure Functions, Logic Apps, ADLS Gen2, Azure SQL, and CI/CD tooling (Azure DevOps or equivalent). Strong experience in data modeling, including dimensional modeling, SCDs, and designing curated “gold” datasets. Experience working with modern data file formats and ingestion strategies (Parquet, Avro, JSON; partitioning, compression, schema evolution). Proven experience supporting enterprise data quality, governance, and documentation. Practical experience applying AI to data platforms, including semantic models, RAG pipelines, or natural-language-to-data solutions. Strong Python programming skills for data acquisition, orchestration, and automation. Excellent communication skills, with the ability to explain technical concepts clearly to both technical and non-technical stakeholders. Demonstrated ownership mindset, strong troubleshooting skills, and commitment to continuous improvement through automation and better platform design. #J-18808-Ljbffr Boston Energy Trading and Marketing Company