Sr. Data Engineer

Sr. Data Engineer

Responsibilities:

• Build resilient production ETL pipelines with data observability
• Deploy solutions to production using CICD best practices and infrastructure automation
• Build data ingestion jobs using Python from multiple types of data sources - including APIs, relational and NoSQL databases
• Work closely with business and technical partners to refine pipeline requirements and design data models to support business objectives
• Integrate new capabilities into the architecture, including streaming data, new monitoring tools, etc.
• Help establish and follow a data driven operations culture within the data team
• Help establish standard practices for building out scalable solutions, mentor other team members
• Help with proof-of-concept implementations for new technologies being tested for the data platform
• Ensure data is secure and in compliance with GDPR, CCPA, Data Privacy, and data retention policies
• Triage and debug production data issues

Skills Required:

• BS degree in Computer Science or a related technical field, or equivalent practical experience
• Proven domain expertise in at least one area (such as Sales & Marketing, Supply Chain, Finance, Data Science, Analytics)
• 5+ years of building mid to large scale data systems
• 4+ years of using AWS ecosystem tools preferred, including Redshift, S3, EMR, Lambda, Kinesis, ECS, Glue and/or another cloud providers equivalent stack
• 4+ years of experience building ETL pipelines and working with Cloud Data Warehouses (such as Redshift, Snowflake, Azure SQL Data Warehouse, BigQuery)
• Experience working with a streaming platform such as Kafka
• Experience working with Big Data frameworks such as Apache Spark or Flink
• Experience using Infrastructure as Code tools (Terraform, CloudFormation)
• 5+ years of experience writing and optimizing SQL. Experience with data modeling for data warehouses
• 4+ years of experience working with Python. Professional experience with at least one other programming language (Java, Scala, Go, Ruby, Javascript)
• 3+ years of experience with ETL tools Informatica, DataStage, SSIS, AbInitio, Talend, Matillion)
• Experience working with orchestration tools(Airflow, Luigi, Apache Nifi, Step Functions, etc.)