Senior GCP Data Engineer - Python, PySpark & BigQuery

Senior GCP Data Engineer - Python, PySpark & BigQuery

Experience: 6+ Years
Job Location: As per project requirement

Job Summary

We are looking for an experienced GCP Data Engineer with strong expertise in Python, PySpark, SQL, and Google Cloud Platform (GCP). The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and big data solutions for enterprise data processing, analytics, and data warehousing.

The candidate should have hands-on experience with GCP services such as BigQuery, Cloud Composer (Airflow), Dataproc, and Pub/Sub, along with strong knowledge of Apache Spark, Hadoop, Hive, ETL processes, and data engineering best practices.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, and GCP services.

• Build and optimize large-scale data processing solutions using Apache Spark, Hadoop, Hive, and Big Data technologies.

• Develop and manage data warehousing solutions using Google BigQuery.

• Implement data ingestion, transformation, and processing workflows using GCP services such as BigQuery, Dataproc, Pub/Sub, and Cloud Composer (Airflow).

• Design and implement event-driven data processing pipelines using Kafka or Google Cloud Pub/Sub.

• Develop and optimize complex SQL queries, data models, and database structures for performance and scalability.

• Implement workflow orchestration and scheduling using Apache Airflow.

• Build and maintain CI/CD pipelines using Jenkins and Git-based source control tools.

• Perform data validation, troubleshooting, performance tuning, and optimization of data pipelines.

• Follow coding standards, object-oriented programming principles, and software design patterns.

• Collaborate with product owners, architects, and cross-functional teams to understand requirements and deliver effective data solutions.

• Participate in Agile/Scrum ceremonies, code reviews, testing, deployment, and continuous improvement initiatives.

• Maintain technical documentation and ensure adherence to enterprise development standards.

Required Technical Skills

Programming & Big Data

• Strong programming experience in Python.

• Hands-on experience with PySpark and Apache Spark.

• Good understanding of Hadoop and Hive.

• Strong knowledge of SQL and relational databases.

• Understanding of object-oriented programming and design patterns.

Google Cloud Platform (GCP)

• Hands-on experience with Google Cloud Platform.

• Google BigQuery for data warehousing and analytics.

• Google Cloud Dataproc for distributed data processing.

• Google Cloud Pub/Sub for event-driven data ingestion and processing.

• Cloud Composer / Apache Airflow for workflow orchestration.

Data Engineering & Integration

• Strong experience in ETL/ELT development and large-scale data processing.

• Data pipeline design, development, transformation, and optimization.

• Experience with data warehousing and data analytics.

• Experience building event-processing pipelines using Kafka or Pub/Sub.

DevOps & Development Tools

• Experience with GitHub, GitLab, Bitbucket, or SVN.

• Knowledge of Jenkins and CI/CD pipeline implementation.

• Familiarity with automated testing frameworks and source code management.

• Shell scripting experience.

Preferred Qualifications

• Google Cloud Professional Data Engineer certification.

• Experience with enterprise-scale Big Data and analytics solutions.

• Knowledge of distributed systems, multi-tier architectures, and algorithms.

• Experience with automated testing and deployment frameworks.

• Understanding of Agile/Scrum and SDLC methodologies.

Required Experience

• 6+ years of hands-on software development experience in Big Data and Analytics.

• Strong practical experience with Python, PySpark, SQL, and GCP.

• Experience designing and optimizing enterprise data pipelines.

Key Competencies

• Strong analytical and problem-solving skills.

• Excellent communication and collaboration abilities.

• Ability to work independently and within cross-functional teams.

• Proactive approach to identifying technical improvements.

• Willingness to learn and adopt enterprise technologies and frameworks.