Data Engineer - Databricks, Python, and SQL

Data Engineer

Develop data pipelines to ingest, load, and transform data from multiple sources.

Leverage Data Platform, running on Google Cloud, to design, optimize, deploy and deliver data solutions in support of scientific discovery

Use programming languages like Java, Scala, Python and Open-Source RDBMS and NoSQL databases and Cloud based data store services such as MongoDB, DynamoDB, Elasticache, and Snowflake

The continuous delivery of technology solutions from product roadmaps adopting Agile and DevOps principles

Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences

Design and develop data pipelines, including Extract, Transform, Load (ETL) programs to extract data from various sources and transform the data to fit the target model

Test and deploy data pipelines to ensure compliance with data governance and security policies

Moving implementation to ownership of real-time and batch processing and data governance and policies

Maintain and enforce the business contracts on how data should be represented and stored

Ensures that technical delivery is fully compliant with Security, Quality and Regulatory standards

Keeps relevant technical documentation up to date in support of the lifecycle plan for audits/reviews.

Pro-actively engages in experimentation and innovation to drive relentless improvement e.g., new data engineering tools/frameworks

Implementing ETL processes, moving data between systems including S3, Snowflake, Kafka, and Spark

Work closely with our Data Scientists, SREs, and Product Managers to ensure software is high quality and meets user requirements

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• 5+ years of experience as a data engineer building ETL/ELT data pipelines.
• Experience with data engineering best practices for the full software development life cycle, including coding standards, code reviews, source control management (GIT, continuous integrations, testing, and operations)
• Experience in programming language Python and SQL good to have Java, C#, C++, Go, Ruby, and Rust
• Experience with Agile, DevOps & Automation [of testing, build, deployment, CI/CD, etc.], Airflow
• Experience with Docker, Kubernetes, Shell Scripting
• 2+ years of experience with a public cloud (AWS, Microsoft Azure, Google Cloud)
• 3+ years experience with distributed data/computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, Gurobi, or MySQL)
• 2+ years experience working on real-time data and streaming applications
• 2+ years of experience with NoSQL implementations (DynamoDB, MongoDB, Redis, Elasticache)
• 2+ years of data warehousing experience (Redshift, Snowflake, Databricks, etc.