Data Engineer - GCP

Job Description

We are seeking a highly skilled GCP Data Engineer with 5+ years of experience designing and building scalable, production-grade data platforms. This role requires deep expertise in Google Cloud Platform, Python, PySpark, SQL, and distributed data processing across the complete data lifecycle—including ingestion, transformation, orchestration, storage, quality, and serving.

\n

The ideal candidate is a hands-on engineer with strong experience building batch and real-time streaming pipelines, developing reusable Python data frameworks, optimizing large-scale workloads, and designing high-performance data models in BigQuery. The engineer will help establish reliable, secure, observable, and cost-efficient data solutions using modern GCP architecture and engineering practices.

\n

\n

Key Responsibilities

\n

\n
• Design, develop, and maintain enterprise-scale batch and streaming data pipelines using Python, PySpark, Dataflow, Dataproc, Pub/Sub, BigQuery, and Cloud Composer
\n
• Build modular, reusable, and testable Python frameworks for data ingestion, transformation, validation, and pipeline automation
\n
• Develop high-performance data-processing solutions using PySpark, Spark SQL, and Apache Beam
\n
• Integrate data from relational and NoSQL databases, REST APIs, flat files, cloud storage, third-party platforms, and event streams
\n
• Build real-time streaming pipelines using Pub/Sub and Dataflow , including windowing, triggers, watermarking, late-arriving data, deduplication, and delivery semantics
\n
• Design event-driven architectures supporting real-time analytics, operational reporting, and downstream data products
\n
• Create and maintain workflow orchestration using Cloud Composer and Apache Airflow , including reusable operators, dependency management, retries, alerting, and recovery
\n
• Design scalable BigQuery schemas using dimensional, normalized, denormalized, and lakehouse modeling patterns
\n
• Optimize BigQuery workloads through effective partitioning, clustering, materialized views, query tuning, and cost-management strategies
\n
• Tune Spark workloads by resolving data skew, excessive shuffling, inefficient joins, memory constraints, partitioning issues, and small-file problems
\n
• Develop incremental loading, change data capture, and high-volume data-processing solutions
\n
• Implement automated data-quality checks, reconciliation, schema validation, lineage, logging, monitoring, and alerting
\n
• Develop Python-based data services and APIs using frameworks such as FastAPI
\n
• Apply security best practices using IAM, service accounts, encryption, secret management, and least-privilege access
\n
• Build and maintain CI/CD pipelines for data applications using Cloud Build, GitHub Actions, or equivalent tools
\n
• Provision and manage GCP data infrastructure using Terraform
\n
• Collaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into scalable data solutions
\n
• Create technical documentation, data mappings, operational runbooks, and architecture diagrams
\n
• Participate in code reviews and promote Python, SQL, GCP, and data-engineering best practices
\n

\n

\n

Required Qualifications

\n

\n
• 5+ years of hands-on experience in data engineering , including building production-grade data pipelines and platforms
\n
• Strong programming expertise in Python , including object-oriented programming, reusable modules, exception handling, logging, testing, packaging, and performance optimization
\n
• Advanced experience with PySpark, Spark SQL, and Apache Spark
\n
• Strong knowledge of GCP data services , particularly BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, and Cloud Storage
\n
• Advanced SQL skills, including complex joins, window functions, CTEs, query optimization, and large-scale data transformation
\n
• Deep expertise in BigQuery , including schema design, partitioning, clustering, performance tuning, workload management, and cost optimization
\n
• Hands-on experience building batch and streaming pipelines with Dataflow and Apache Beam using Python
\n
• Strong experience designing and operating Spark workloads on Dataproc
\n
• Experience developing and managing Airflow DAGs using Cloud Composer
\n
• Experience designing Pub/Sub topics, subscriptions, schemas, retry policies, dead-letter queues, and deduplication strategies
\n
• Strong understanding of distributed processing concepts, including partitioning, shuffling, parallelism, fault tolerance, and delivery guarantees
\n
• Solid understanding of Spark internals, including execution plans, DAGs, Catalyst optimization, memory management, and Spark UI troubleshooting
\n
• Experience with dimensional data modeling, including star and snowflake schemas and SCD Type 1 and Type 2
\n
• Experience processing large-scale partitioned datasets and implementing incremental data loads
\n
• Experience with data formats such as Parquet, Avro, ORC, JSON, and Delta
\n
• Strong understanding of streaming concepts, including event time, processing time, windowing, triggers, watermarking, and exactly-once versus at-least-once processing
\n
• Experience implementing data-quality, observability, reconciliation, and production-monitoring frameworks
\n
• Proficiency with Git, automated testing, Agile delivery, and CI/CD practices
\n

\n

\n

Preferred Qualifications

\n

\n
• Experience building Python APIs and microservices using FastAPI or Flask
\n
• Working knowledge of Terraform and infrastructure-as-code practices
\n
• Experience with CDC technologies and patterns
\n
• Familiarity with Dataform or dbt for SQL-based data transformations
\n
• Knowledge of GCP security, networking, IAM, Secret Manager, and data-governance practices
\n
• Google Cloud Professional Data Engineer certification is preferred
\n

\n