Data Engineering Manager - GCP Cloud & AIML

Greetings of the day!!

I am Arumugam Veera reaching out to you regarding an exciting career opportunity with Naveera Technology LLC. I would be happy to connect and discuss the opportunity further. You can also connect with me on LinkedIn: Naveera Technology LLC

Naveera Technology LLC is a trusted global engineering partner delivering Data Engineering Generative AI Application Development and IT Infrastructure solutions. With over 15 years of experience in IT services and consulting we help organizations transform raw data into actionable business value.

With a team of 100 employees and successful delivery of 3 global projects Naveera serves clients across multiple industries and geographies through agile delivery models and proven engineering practices.

From Digital Health and Financial Services to E-Commerce and Technology we support a diverse client base and back every engagement with proven frameworks low-attrition teams and scalable global delivery capabilities. At Naveera we empower organizations to turn challenges into opportunities data into insights and innovative ideas into enterprise-grade platforms.

Specialties
Data Engineering & Modern Data Stack Generative AI Solutions & Model Deployment Application Development (Web Mobile & Enterprise) Artificial Intelligence (Predictive Conversational Computer Vision) IT Infrastructure Services (Cloud & On-Prem) Security DR Cloud Transformation & Microservices DevOps API & Systems Integration Extended Technology Teams & Dedicated Delivery Real-Time Streaming & Analytics and BI & Data Warehousing

• Job Title: Engineering Manager GCP & AIML

• Experience: 15 Years

• Job Location: USA (Remote)

• Profile location: Anywhere in USA

• Primary Focus: GCP & AI/ML & GenAI Data Engineering Data Architecture Engineering

• Interview Levels: 3 Levels of Interview from Naveera 1 Level of client discussion

Pls find the below benefit details for the above role.

• Medical Insurance

• 10 Company Paid holidays

• Supportive culture with one-of-a-kind growth opportunities

• Ability to work from home

• Peer-to-peer recognition program

Total experience must be 15 years and the primary focus must be below:

• 5 years of strong hands-on relevant GCP Data Engineering experience is mandate

• 1-2 years of relevant experience in AIML Solutioning

Position Overview

We are looking for an experienced Engineering Manager with strong hands-on expertise in AWS and GCP Data Engineering to lead a large-scale AWS-to-GCP data platform migration.

The ideal candidate will have strong experience designing enterprise data platforms on AWS and migrating them to Google Cloud Platform (GCP). The role requires a combination of technical architecture hands-on engineering migration leadership team management stakeholder management and delivery ownership.

In addition to AWS-to-GCP data migration this role will lead the design and delivery of AI/ML Generative AI and MLOps capabilities on GCP. The candidate will work with Data Science Product Business Analytics BI and Engineering teams to build secure scalable governed and production-ready data and AI solutions using GCP-native services.

All target-state data engineering AI/ML Generative AI and MLOps architecture and implementation experience must be on GCP.

Key Responsibilities

1. GCP Data Platform Architecture

• Architect and implement scalable enterprise data platforms on GCP.

• Design Data Lake and Lakehouse architectures using Google Cloud Storage and BigQuery.

• Define Bronze Silver and Gold/Atomic data layers.

• Design scalable batch real-time and event-driven ingestion transformation and data-consumption frameworks.

• Establish standards for data modeling partitioning clustering storage metadata lineage and data access.

• Design multi-tenant and multi-location data architectures.

• Define schema-on-read and schema-on-write strategies.

• Build scalable data platforms that support analytics BI real-time reporting machine learning and Generative AI use cases.

2. AWS Data Platform Expertise

• Analyze and optimize existing AWS data platforms before migration.

• Work with Amazon S3 AWS Glue AWS Glue Data Quality Amazon Redshift Redshift Serverless Amazon Athena AWS Step Functions AWS DMS and AWS Lake Formation.

• Understand existing AWS ETL/ELT pipelines data models workloads security controls and platform dependencies.

• Identify equivalent or improved GCP services for AWS data workloads.

• Prepare technical mapping modernization recommendations migration plans and implementation roadmaps between AWS and GCP services.

3. GCP Streaming & Real-Time Data Engineering

• Architect real-time data pipelines using Google Pub/Sub Dataflow Apache Beam BigQuery and Cloud Storage.

• Design high-volume event ingestion enrichment transformation and delivery pipelines.

• Implement event-driven architectures with appropriate delivery guarantees.

• Optimize streaming pipelines for latency throughput scalability reliability and cost.

• Design BigQuery streaming-ingestion patterns.

• Implement monitoring logging alerting and operational support processes for real-time workloads.

4. ETL/ELT & Data Processing

• Design and implement scalable batch and real-time ETL/ELT pipelines on GCP.

• Migrate AWS Glue-based pipelines to appropriate GCP-native services.

• Develop transformation frameworks using Python PySpark SQL Dataflow Apache Beam BigQuery and dbt.

• Design CDC pipelines and real-time ingestion patterns.

• Build orchestration workflows using Cloud Composer and Apache Airflow.

• Optimize data-processing jobs Spark workloads pipeline execution and query performance.

• Establish coding testing documentation deployment and operational standards for data engineering workloads.

5. Data Modeling & BigQuery

• Design enterprise data models for analytics reporting operational intelligence and AI/ML workloads.

• Define dimensional normalized denormalized and multi-tenant data models.

• Design BigQuery partitioning clustering storage and query-optimization strategies.

• Optimize BigQuery SQL and query execution for performance and cost.

• Design data models that support real-time and batch workloads.

• Work closely with BI and Analytics teams to build scalable self-service consumption models.

6. AI/ML Generative AI & MLOps

• Design and implement AI/ML and Generative AI solutions on GCP using Vertex AI BigQuery Cloud Storage Dataflow Pub/Sub Cloud Run and related GCP-native services.

• Build production-grade machine learning pipelines for data preparation model training validation evaluation deployment monitoring retraining and lifecycle management.

• Develop Generative AI and Retrieval-Augmented Generation (RAG) solutions including enterprise search document intelligence AI assistants summarization semantic search embeddings vector search and knowledge-management applications.

• Design scalable ingestion transformation chunking embedding indexing and retrieval pipelines for structured and unstructured enterprise data.

• Implement MLOps practices using Vertex AI Pipelines Model Registry model endpoints Terraform GitHub Cloud Build and CI/CD pipelines.

• Establish standards for model versioning experiment tracking data and feature validation automated testing deployment approvals rollback and environment promotion.

• Implement monitoring for model performance data drift latency reliability inference cost response quality retrieval accuracy and GenAI risks such as hallucination and prompt injection.

• Ensure responsible AI data privacy security governance access control auditability and human-review processes are incorporated into AI/ML and GenAI solutions.

• Partner with Data Science Analytics Product BI Security and US-based stakeholders to identify prioritize and deliver high-value AI/ML and GenAI use cases.

7. Stakeholder & Client Management

• Act as the primary technical point of contact for US-based stakeholders.

• Work closely with Business Product Data Science BI Analytics DevOps Security and Architecture teams.

• Translate business requirements into scalable data cloud AI/ML and Generative AI solutions.

• Present architecture decisions migration strategies technical roadmaps implementation plans risks timelines and trade-offs.

• Communicate technical dependencies delivery status operational risks and mitigation plans clearly to technical and non-technical stakeholders.

• Collaborate with business teams to define operational analytical data-quality platform and AI/ML KPIs.

Required Qualifications

• 15 years of experience in Data Engineering Data Architecture Cloud Engineering AI/ML Engineering or related technology leadership roles.

• Strong hands-on experience with AWS Data Engineering and Data Architecture.

• 5 years of strong hands-on GCP Data Engineering experience.

• 1-2 yrs in AIML Solutioning.

• Strong experience designing enterprise Data Lake and Lakehouse platforms on GCP.

• Strong hands-on experience with BigQuery Google Cloud Storage Dataflow Pub/Sub Cloud Composer Dataproc IAM and Terraform.

• Experience migrating AWS data workloads pipelines and platforms to GCP.

• Strong knowledge of AWS and GCP service mapping migration patterns modernization strategies and cloud architecture best practices.

• Experience designing building and deploying AI/ML solutions on GCP using Vertex AI.

• Hands-on experience with Generative AI LLM-based applications RAG architectures embeddings vector search prompt engineering and enterprise AI assistants.