Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

System Administrator (GCP/AWS/Azure, PySpark, BigQuery, and Google Airflow)

Macpower Digital Assets Edge Private Limited

Job Description

Job Description

Job Overview: This role involves managing and optimizing Big Data environments (PySpark, BigQuery, Airflow) across Google Cloud, AWS, or Azure platforms, ensuring efficient, secure, and cost-effective operations. Key responsibilities include 24x7 support, data pipeline optimization, automation, and troubleshooting, with a strong emphasis on DevOps, CI/CD, and disaster recovery.

Roles and Responsibilities: (Google Cloud/AWS/Azure, PySpark, BigQuery, and Google Airflow)

  • Participate in 24x7x365 rotational shift support and operations for SAP environments.
  • Serve as a team lead responsible for maintaining the upstream Big Data ecosystem, handling millions of financial transactions daily using PySpark, BigQuery, Dataproc, and Google Airflow.
  • Streamline and optimize existing Big Data systems and pipelines while developing new ones, ensuring efficient and cost-effective performance.
  • Manage the operations team during your designated shift and make necessary changes to the underlying infrastructure.
  • Provide day-to-day support, improve platform functionality using DevOps practices, and collaborate with development teams to enhance database operations.
  • Architect and optimize data warehouse solutions using BigQuery to enable efficient data storage and retrieval.
  • Install, build, patch, upgrade, and configure Big Data applications.
  • Administer and configure BigQuery environments, including datasets and tables.
  • Ensure data integrity, availability, and security on the BigQuery platform.
  • Implement partitioning and clustering strategies for optimized query performance.
  • Define and enforce access policies for BigQuery datasets.
  • Set up query usage caps and alerts to control costs and prevent overages.
  • Troubleshoot issues in Linux-based systems with strong command-line proficiency.
  • Create and maintain dashboards and reports to monitor key metrics such as cost and performance.
  • Integrate BigQuery with other GCP services like Dataflow, Pub/Sub, and Cloud Storage.
  • Enable BigQuery usage through tools such as Jupyter Notebook, Visual Studio Code, and CLI utilities.
  • Implement data quality checks and validation processes to maintain data accuracy.
  • Manage and monitor data pipelines using Airflow and CI/CD tools like Jenkins and Screwdriver.
  • Collaborate with data analysts and scientists to gather data requirements and translate them into technical implementations.
  • Provide guidance and support to application development teams for database design, deployment, and monitoring.
  • Demonstrate proficiency in Unix/Linux fundamentals, scripting in Shell/Perl/Python, and using Ansible for automation.
  • Contribute to disaster recovery planning and ensure high availability, including backup and restore operations.
  • Experience with geo-redundant databases and Red Hat clustering is a plus.
  • Ensure timely delivery within defined SLAs and project milestones, adhering to best practices for continuous improvement.
  • Coordinate with support teams including DB, Google, PySpark data engineering, and infrastructure.
  • Participate in Incident, Change, Release, and Problem Management processes.

Must Have Skills, Experience:

  • 4 8 years of relevant experience.
  • Strong experience with Big Data technologies including PySpark, BigQuery, and Google Airflow.
  • Hands-on expertise in cloud platforms (Google Cloud, AWS, or Azure) and Linux system troubleshooting.
  • Proficiency in automation and DevOps tools such as Shell/Python scripting, CI/CD processes, and Ansible.
Vacancy posted 23 days ago
Similar jobs that could be interesting for youBased on the System Administrator (GCP/AWS/Azure, PySpark, BigQuery, and Google Airflow) in San Jose, CA vacancy
  •  ...designing, implementing, securing, and governing cloud based solutions across multiple cloud platforms, including AWS, Microsoft Azure, and Google Cloud Platform (GCP). This role requires a strong understanding of cloud security architecture, identity and access management,... 
    Amazon Web Service
    Google

    Conexess Group, LLC

    Milpitas, CA
    23 days ago
  • $96.28k - $125k

     ...manages the sensors, systems and other assets necessary...  ...(IT) Systems Administrator for the Defense Innovation...  ...productivity platforms including Google Workspace and/or...  ...on Google Workspace, AWS, GCP, SIEM, ACAS, XDR, and...  ...(e.g., Okta, Azure AD/Entra ID, Lastwall)... 
    Amazon Web Service
    Google
    Full time
    Work experience placement
    Local area
    Remote work
    Worldwide

    HII Mission Technologies Division

    Mountain View, CA
    3 days ago
  •  ...warehouses (Snowflake, BigQuery, or Redshift). ~...  ...g., Kubeflow, MLflow, Airflow/Dagster for orchestration...  ...a cloud environment (GCP or AWS). ~ Excellent communication...  ...time / streaming data systems (Kafka, Pub/Sub,...  ...cybersecurity technologies and Google SecOps solutions.... 
    Amazon Web Service
    Google
    Work at office

    TenEx

    San Jose, CA
    21 hours ago
  • $180k - $210k

     ...Pre-Sales Systems Engineer Onehouse is a mission...  ...including Uber, Snowflake, AWS, Linkedin, Confluent...  .../Presto, DBT, Kafka, Airflow, Avro, Hadoop. ~...  ...in one or more of AWS/GCP/Azure cloud technologies....  ...Amazon Kinesis, AWS Glue, Google BigQuery, Databricks, Snowflake... 
    Amazon Web Service
    Google
    Work at office
    Remote work

    Onehouse

    Sunnyvale, CA
    2 days ago
  • $38 - $94 per hour

     ...out from the crowd: Computer Science or Engineering undergrad degree Experience with Amazon Web Services (AWS), Google Cloud (GCP), or Microsoft Azure Experience as an sw developer, product manager, sales engineer, or customer support/eng Working knowledge... 
    Amazon Web Service
    Google
    Hourly pay
    Work experience placement
    Summer work
    Internship

    NVIDIA

    Santa Clara, CA
    3 days ago
  •  ...Deep knowledge of cloud platforms (AWS, GCP, Azure), cloud-native ecosystems (Kubernetes,...  ...infrastructure—such as Amazon SageMaker, Google Vertex AI, Azure Machine Learning, Kubeflow...  ...orchestration tools like Apache Airflow and Argo Workflows. Familiarity with... 
    Amazon Web Service
    Google

    Ai Squared

    Mountain View, CA
    4 days ago
  •  ...with Al ML search and data services within GCP ecosystem such as GCP Vertex AI Vertex AI...  ...Experience with agent frameworks such as Google ADK Langgraph CrewAI etc ~ Hands on...  ...working with cloud AI services GCP Vertex AI Azure AI AWS Bedrock as well as managed large language... 
    Amazon Web Service
    Google

    Trinity Technology Solutions

    San Jose, CA
    3 days ago
  •  ..., blending Frontier Models (Azure OpenAI/Gemini) with Cost-Efficient...  ...from OpenAI, NVIDIA, Google, , Microsoft and AWS. This role has a focus on enabling...  ...for multi-agent systems, covering hierarchical, peer...  ...integration, Vertex AI for GCP workloads). Advise clients on... 
    Amazon Web Service
    Google
    Temporary work

    HCL Technologies

    Santa Clara, CA
    3 days ago
  •  ...TimeCompany: SRI Tech SolutionsRole:GCP data EngineerLocation: Santa...  ...environments, with a strong preference for Google Cloud Platform (GCP) services, specifically: BigQuery: Expert-level skills in data...  ...tools (e.g., Apache Airflow, Cloud Composer, Airflow,Dagster... 
    Google
    Full time

    SRI Tech

    San Jose, CA
    2 days ago
  •  ...)Contact: Elankumaran VJob Title: Senior Google Cloud ArchitectLocation: Milpitas, CA (Hybrid...  ...with 6+ years in Google Cloud Platform (GCP) and proven expertise in designing,...  ...industry experience.Multi-cloud exposure (AWS / Azure) is an added advantage.Experience working... 
    Amazon Web Service
    Google
    Hourly pay

    VDart

    Milpitas, CA
    16 hours ago
  •  ...rapid global adoption, we are scaling systems designed for the world’s most demanding...  ...an experienced, hands-on Staff System Administrator (Hardware & Infrastructure) to support...  ...one major public cloud platform (AWS, GCP, or Azure).   ~ Active comfort utilizing frontier... 
    Amazon Web Service
    Full time
    Local area
    Immediate start

    upscale-ai

    Santa Clara, CA
    6 days ago
  •  ...infrastructure on a public cloud platform (we run on GCP, but AWS or Azure experience translates well)Strong distributed systems and infrastructure skills: standing up...  ...with observability tooling (we use Grafana + Google Cloud Monitoring)Experience building internal... 
    Amazon Web Service
    Google
    Home office
    Flexible hours

    Gridmatic

    Cupertino, CA
    2 days ago
  •  ...the consequences of a poorly governed AI system in production are real. What this...  ...hyperscalers that host them, Microsoft, Google and AWS. Know the strengths and the limits of each...  ...across at least one major cloud platform (Azure, GCP, or AWS) and hands-on familiarity with... 
    Amazon Web Service
    Google
    Temporary work

    VDart

    Santa Clara, CA
    3 days ago
  • $78k - $143.5k

     ...five years of relevant practical experienceDeep experience hardening, scaling, and operating live systems in production cloud environments (Azure, AWS, or Google Cloud (GCP))Deep fluency with modern AI coding harnesses, agentic tooling (for example, Claude Code, Cursor,... 
    Amazon Web Service
    Google
    Local area

    KPMG

    Santa Clara, CA
    2 days ago
  • $229k - $412k

     ...on large scale distributed systems, processing almost 3 trillion...  ...tenancy, delegated administration, cross-tenant access patterns...  ...with cloud-native platforms (AWS, GCP, or Azure) and Kubernetes ~ Proven...  ...Points: Experience with Google Zanzibar-style authorization... 
    Amazon Web Service
    Google
    Permanent employment
    Work experience placement
    Work at office
    Local area
    2 days per week
    3 days per week
    Sunnyvale, CA
    27 days ago
  • $90.4k - $168.2k

     ...tests using frontier cyber models (e.g., Mythos/Fable, Google Argon, Daybreak) against client environments....  ...engineering expertise (hardening, scaling, and operating live systems across Azure, AWS, or Google Cloud (GCP)Bachelor's degree from an accredited college/... 
    Amazon Web Service
    Google
    Local area

    KPMG

    Santa Clara, CA
    1 day ago
  • $167.27k - $278.79k

     ...ideal candidate combines deep expertise in systems programming, deep knowledge in VPP/DPDK...  .... Expertise in cloud networking across AWS, Azure, GCP and other cloud service providers...  ...networking platforms in AWS, Azure, or Google Cloud. Knowledge of SD-WAN, SASE, ZTNA,... 
    Amazon Web Service
    Google
    Work at office

    Alkira

    San Jose, CA
    1 day ago
  •  ...the IT infrastructure, systems, applications, and...  ...managing cloud environments (AWS, Azure, SAP) Proficiency...  ...(e.g., AWS, Azure, GCP). Bilingual in English...  ...’s degree in business administration, Computer Science, or...  ...II, Infrastructure, Google Cloud AI San Jose,... 
    Amazon Web Service
    Google
    Full time
    Work experience placement
    Work at office

    Intellipro, Inc.

    San Jose, CA
    2 days ago
  • $198k - $368k

     ...frontier cyber models (e.g., Mythos/Fable, Google Argon, Daybreak)Maintain deep, active coding...  ...building, hardening, and operating scalable systems in live enterprise cloud environments across Azure, AWS, or Google Cloud (GCP), with strong proficiency in modern languages... 
    Amazon Web Service
    Google
    Local area

    KPMG

    Santa Clara, CA
    3 days ago
  • $140k - $210k

     ...across microservices and monolith systems. You will own crucial components...  ...platforms like Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure.• You will need 2+ years of...  ...orchestration systems including Apache Airflow or Google Cloud Composer.•... 
    Amazon Web Service
    Google
    Full time

    Fiserv

    Sunnyvale, CA
    3 days ago
  • $162.7k - $284.7k

     ...global cloud operations across Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). This role is responsible for ensuring the reliability...  ...KLAWe provide advanced inspection tools, metrology systems, process solutions, and computational analytics that... 
    Amazon Web Service
    Google
    Minimum wage
    Full time

    KLA-Tencor

    Milpitas, CA
    2 days ago
  • $137.8k - $202.13k

     ...solutions across multiple cloud platforms — GCP, Azure, and AWS, with GCP and Azure as the preferred...  ...(primary), plus Azure Bicep/ARM, Google Config Connector, and AWS CloudFormation...  ...provide advanced inspection tools, metrology systems, process solutions, and computational... 
    Amazon Web Service
    Google
    Minimum wage
    Full time

    KLA-Tencor

    Milpitas, CA
    3 days ago
  • $124.44k - $207.4k

     ...as a point of escalation for complex systems issues. Subject matter expert in one...  ...information security, and best practice system administration. Regular availability outside of...  ...of Cloud Infrastructure and Security (GCP/AWS/Azure/OCI) is a plus In depth... 
    Amazon Web Service
    Work at office
    Local area
    Worldwide
    Flexible hours
    Night shift

    KIOXIA America, Inc.

    San Jose, CA
    2 days ago
  • $190.9k - $334.1k

     ...external integrations (e.g. AWS, Okta) into the Veza...  ...and complex data systems as you determine an efficient...  ...services like AWS, GCP, Azure and resources like Box...  ...as AWS, Azure, or Google Cloud Platform.Knowledge...  ...the U.S. Export Administration Regulations (EAR), ServiceNow... 
    Amazon Web Service
    Google
    Work at office
    Remote work
    Flexible hours

    ServiceNow

    Santa Clara, CA
    4 days ago
  • $248k - $396.75k

     ...-premises data centers and major CSPs (AWS, Azure, GCP), owning the end-to-end lifecycle from...  ...pipelines that integrate deployed storage systems with interdependent tooling — including...  ...(for example AWS S3, Azure Blob, or Google Cloud Storage), as well as on-prem infrastructureNVIDIA... 
    Amazon Web Service
    Google
    Full time
    Shift work

    Nvidia

    Santa Clara, CA
    3 days ago
  • $124k - $195.5k

     ...NVIDIA EBC), and packaging agentic AI systems. Partnering with account SAs you will co...  ...Deploys workloads at scale on public cloud (AWS, GCP, Azure, OCI) or on-premise.Strong grasp of the...  ...as MCP (Model Context Protocol) or Google A2A.Proven experience handling NVIDIA... 
    Amazon Web Service
    Google
    Full time

    Nvidia

    Santa Clara, CA
    4 days ago
  • $220k - $250k

     ...CAPosition reports to: Director of SW Systems EngineeringApplication deadline: Applications...  ...preferably on cloud platforms such as AWS, Azure, or Google Cloud.Experience in one or more of the...  ...such as Kafka, Spark, Flink, PySpark, MapReduce, or comparable technologies... 
    Amazon Web Service
    Google
    Full time
    H1b
    Work visa
    Shift work

    Extreme Networks, Inc.

    San Jose, CA
    16 hours ago
  • $160k - $225k

     ...platform veterans from Google and Instacart, we've...  ...the scalable serving systems that deliver their...  ...and data lakes (e.g., BigQuery, Snowflake, Redshift,...  ...with cloud platforms (GCP, AWS, or Azure). Strong problem-solving...  ...tools (e.g., Airflow). Familiarity with... 
    Amazon Web Service
    Google
    Full time

    mai

    Mountain View, CA
    6 days ago
  • $144k - $180k

     ...that delivers air taxis, unmanned aircraft systems (“UAS”), aviation-related physical...  ...automation.Working knowledge of SIEM platforms (Google SecOps/Chronicle, Splunk, Microsoft...  ...firewalls and network security, cloud security (AWS/Azure/GCP), and application security (AppSec)... 
    Amazon Web Service
    Google
    Local area
    Worldwide
    Visa sponsorship

    Archer Aviation

    San Jose, CA
    16 hours ago
  •  ...Job Description Job Description System Administrator This is a six month Contract role Job Description: Responsible for managing...  ...servers, VMware virtual environments, and cloud services such as AWS or Azure. The Systems Administrator applies ITIL best practices to... 
    Amazon Web Service
    Contract work
    Work experience placement

    Marathon TS, Inc.

    San Jose, CA
    21 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to System Administrator (GCP/AWS/Azure, PySpark, BigQuery, and Google Airflow). Be the first to apply!