Principal Data Engineer

Principal Data EngineerResponsibilities:• Architecting, building, and maintaining modern, scalable data architectures in the cloud preferably AWS• Design & develop frameworks for increasing the overall efficiency of bringing data into the data lake, processing and delivery of data; encode best practices into reusable tools that can be shared across the team.• Automate the deployment of changes to production to improve data reliability & quality• Work closely with upstream and downstream stakeholders to collect business requirements and develop data models to satisfy those requirements.• Design and deliver event-driven solutions using a streaming platform such as Kafka.• Mentor data engineers and other stakeholders on the best practices of managing a large-scale production data platform and developing a data-driven operations culture within the data engineering team.• Evaluate new technologies and solutions for inclusion into the data platform.• Oversee the migration of data from legacy systems to new solutions.• Partner with Security and Privacy teams to ensure data is secure and in compliance with GDPR, CCPA, Data Privacy, and data retention policies.• Build instrumentation for end-to-end data observability so as to ensure accurate, timely and high quality of data.• Generate documentation on existing production data logic and its influencing business processes in order to reconcile knowledge gaps between the business, engineering, and data collection.• Experience with Machine Learning or other leading-edge technologiesThe individual will need the following skills:• Proven leadership experience in a data platform organization.• Proven business domain expertise in at least two to three areas (such as Sales & Marketing, Supply Chain, Finance, Data Science, Analytics).• 10+ years of hands-on experience building large scale data systems.• 10+ years of experience with architectural patterns, building APIs, microservices, event streams, and high throughput systems.• 5+ years of using AWS ecosystem tools preferred, including Redshift, S3, EMR,Lambda, Kinesis, ECS, Glue and/or another cloud providers equivalent stack.• Expert level proficiency with Python and SQL. Professional experience with at least one other programming language (Java, Scala, Go, Ruby, Javascript)• 5+ years of experience building ETL pipelines and working with Cloud Data Warehouses (such as Redshift, Snowflake, Azure SQL Data Warehouse, BigQuery).• Experience operating very large data warehouses or data lakes.• Experience designing and implementing data models for enterprise data warehouses.• Demonstrated Kafka, streaming experience• 7+ years of experience with ETL tools (Matillion, Informatica, DataStage, Talend)• Experience working with orchestration tools(Airflow, Luigi, Apache Nifi, Step Functions, etc.)