Lead Data Engineer

Data Architect

6-8 years of IT experience focusing on enterprise data architecture and management.

• Experience in conceptual/logical/physical data modeling & expertise in relational and dimensional data modeling

• Experience with Databricks & on-prem, structured streaming, delta lake concepts, and delta live tables required

• Experience with Spark scala

• Data lake concepts such as time travel and schema evolution and optimization

• Structured streaming and delta live tables with Databricks a bonus

• Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation/support

• Advanced level understanding of streaming data pipelines and how they differ from batch systems

• Formalize concepts of how to handle late data, defining windows, and data freshness

• Advanced understanding of ETL and ELT and ETL/ELT tools such as Data Migration Service etc

• Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermark, etc.

• Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus

• Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness

• Advanced level SQL experience (joins, aggregation, windowing functions, common table expressions, RDBMS schema design performance optimization)

• Indexing and partitioning strategy experience

• Debug, troubleshoot, design and implement solutions to complex technical issues

• Experience with large-scale, high-performance enterprise big data application deployment and solution

• Architecture experience in AWS environment a bonus

• Familiarity working with Lambda specifically with how to push and pull data, how to use AWS tools to view data for processing massive data at scale a bonus

• Experience with Gitlabs and CloudWatch and ability to write and maintain gitlabs for supporting CI/CD pipelines

• Experience working with AWS Lambdas for configuration and optimization and experience with S3

• Familiarity with Schema Registry, message formats such as Avro, ORC, etc.