GCP Data Engineer

GCP Data Engineer

Mandatory: Big Query, Big Table, Google Cloud Storage, PubSub, Data Fusion, Dataflow, Dataproc Skills:

• Google Cloud Platform (GCP) Tools: (Preferred) - BigQuery, Cloud Storage, Dataflow, Cloud Functions, Pub/Sub, Cloud Run, Cloud Composer (Airflow), Cloud Spanner, Bigtable.
• Container Orchestration - Kubernetes (preferred on GKE) and Helm for managing and deploying containerized applications.
• CI/CD and Automation- Jenkins for building CI/CD pipelines to automate deployment and testing of data pipelines.
• Programming Languages- Proficient in Python for data processing and automation, SQL for querying and data manipulation. Experience with Java is a plus.
• DevOps Tools - Familiarity with Terraform or Deployment Manager for Infrastructure as Code (IaC) to manage GCP resources.
• Monitoring and Logging -Experience with Cloud Monitoring, Datadog, or other monitoring solutions to track pipeline performance and ensure operational efficiency.
• Data Engineering Skills Expertise in ETL/ELT pipelines, data modeling, and data integration across large datasets.
• Strong understanding of data warehousing and real-time data processing workflows.
• Strong communication skills to work effectively with cross-functional teams and mentor junior developers. Proven ability to lead in an Agile environment.
• 3+ years of experience as a data engineer, with hands-on experience in Kubernetes, Helm, Python and Jenkins.
• Strong experience building and optimizing data pipelines and services in any cloud platform.
• Proficiency in Python and SQL. Familiarity with Java and Docker is a plus.

Job Description:

• The GCP Data Engineer will work iteratively on the cloud platform to design, develop and implement scalable, high performance solutions that offer measurable business value to customers.
• Data Solutions: Design and implementation of data pipelines using GCP services
• Manage Deployments with Helm and Kubernetes: Oversee containerized deployments of data applications on Kubernetes Engine using Helm for package management, ensuring efficient orchestration of services.
• Automation with Jenkins: Implement CI/CD pipelines using Jenkins to automate data pipeline deployment, testing, and integration with other services, ensuring quick iterations and deployments.
• Develop and Optimize Pipelines: Write efficient Python and SQL scripts to build data pipelines and ETL/ELT processes. Continuously monitor and optimize data workflows for performance and cost-effectiveness.
• Data Integration and Orchestration: Design workflows to integrate data from various sources using GCP services, and orchestrate complex tasks with Cloud Composer (Apache Airflow).
• Security and Compliance: Ensure data security and compliance by implementing IAM policies, encryption, and other security measures in GCP, adhering to best practices for handling sensitive data.
• Collaborate Across Teams: Work closely with application developers, data architects, and business stakeholders to define and deliver robust data-driven solutions. Provide technical leadership and ensure alignment between business and technical objectives.
• Monitoring and Logging: Set up monitoring, logging, and alerting using Cloud Monitoring (formerly Stackdriver), Datadog, or other tools to ensure visibility into pipeline performance and quickly identify and resolve issues.