Sr Data Engineer

Data Engineer

The Data Engineer will work iteratively on the cloud platform to design, develop and implement scalable, high performance solutions that offer measurable business value to customers.

Data Solutions: Design and implementation of data pipelines using GCP services

Manage Deployments with Helm and Kubernetes: Oversee containerized deployments of data applications on Kubernetes Engine using Helm for package management, ensuring efficient orchestration of services.

Automation with Jenkins: Implement CI/CD pipelines using Jenkins to automate data pipeline deployment, testing, and integration with other services, ensuring quick iterations and deployments.

Develop and Optimize Pipelines: Write efficient Python and SQL scripts to build data pipelines and ETL/ELT processes. Continuously monitor and optimize data workflows for performance and cost-effectiveness.

Data Integration and Orchestration: Design workflows to integrate data from various sources using GCP services, and orchestrate complex tasks with Cloud Composer (Apache Airflow).

Security and Compliance: Ensure data security and compliance by implementing IAM policies, encryption, and other security measures in GCP, adhering to best practices for handling sensitive data.

Collaborate Across Teams: Work closely with application developers, data architects, and business stakeholders to define and deliver robust data-driven solutions. Provide technical leadership and ensure alignment between business and technical objectives.

Monitoring and Logging: Set up monitoring, logging, and alerting using Cloud Monitoring (formerly Stackdriver), Datadog, or other tools to ensure visibility into pipeline performance and quickly identify and resolve issues.

Skills Google Cloud Platform (GCP) Tools:(Preferred) - BigQuery, Cloud Storage, Dataflow, Cloud Functions, Pub/Sub, Cloud Run, Cloud Composer (Airflow), Cloud Spanner, Bigtable. Container Orchestration - Kubernetes (preferred on GKE) and Helm for managing and deploying containerized applications. CI/CD and Automation- Jenkins for building CI/CD pipelines to automate deployment and testing of data pipelines. Programming Languages- Proficient in Python for data processing and automation, SQL for querying and data manipulation. Experience with Java is a plus. DevOps Tools - Familiarity with Terraform or Deployment Manager for Infrastructure as Code (IaC) to manage GCP resources. Monitoring and Logging -Experience with Cloud Monitoring, Datadog, or other monitoring solutions to track pipeline performance and ensure operational efficiency. Data Engineering Skills Expertise in ETL/ELT pipelines, data modeling, and data integration across large datasets. Strong understanding of data warehousing and real-time data processing workflows. Strong communication skills to work effectively with cross-functional teams and mentor junior developers. Proven ability to lead in an Agile environment. 3+ years of experience as a data engineer, with hands-on experience in Kubernetes, Helm, Python and Jenkins. Strong experience building and optimizing data pipelines and services in any cloud platform. Proficiency in Python and SQL. Familiarity with Java and Docker is a plus.