System Administrator (GCP/AWS/Azure, PySpark, BigQuery, and Google Airflow)

Macpower Digital Assets Edge

Job Overview: This role involves managing and optimizing Big Data environments (PySpark, BigQuery, Airflow) across Google Cloud, AWS, or Azure platforms, ensuring efficient, secure, and cost-effective operations. Key responsibilities include 24x7 support, data pipeline optimization, automation, and troubleshooting, with a strong emphasis on DevOps, CI/CD, and disaster recovery.

Roles and Responsibilities: (Google Cloud/AWS/Azure, PySpark, BigQuery, and Google Airflow)

Participate in 24x7x365 rotational shift support and operations for SAP environments.
Serve as a team lead responsible for maintaining the upstream Big Data ecosystem, handling millions of financial transactions daily using PySpark, BigQuery, Dataproc, and Google Airflow.
Streamline and optimize existing Big Data systems and pipelines while developing new ones, ensuring efficient and cost-effective performance.
Manage the operations team during your designated shift and make necessary changes to the underlying infrastructure.
Provide day-to-day support, improve platform functionality using DevOps practices, and collaborate with development teams to enhance database operations.
rchitect and optimize data warehouse solutions using BigQuery to enable efficient data storage and retrieval.
Install, build, patch, upgrade, and configure Big Data applications.
dminister and configure BigQuery environments, including datasets and tables.
Ensure data integrity, availability, and security on the BigQuery platform.
Implement partitioning and clustering strategies for optimized query performance.
Define and enforce access policies for BigQuery datasets.
Set up query usage caps and alerts to control costs and prevent overages.
Troubleshoot issues in Linux-based systems with strong command-line proficiency.
Create and maintain dashboards and reports to monitor key metrics such as cost and performance.
Integrate BigQuery with other GCP services like Dataflow, Pub/Sub, and Cloud Storage.
Enable BigQuery usage through tools such as Jupyter Notebook, Visual Studio Code, and CLI utilities.
Implement data quality checks and validation processes to maintain data accuracy.
Manage and monitor data pipelines using Airflow and CI/CD tools like Jenkins and Screwdriver.
Collaborate with data analysts and scientists to gather data requirements and translate them into technical implementations.
Provide guidance and support to application development teams for database design, deployment, and monitoring.
Demonstrate proficiency in Unix/Linux fundamentals, scripting in Shell/Perl/Python, and using Ansible for automation.
Contribute to disaster recovery planning and ensure high availability, including backup and restore operations.
Experience with geo-redundant databases and Red Hat clustering is a plus.
Ensure timely delivery within defined SLAs and project milestones, adhering to best practices for continuous improvement.
Coordinate with support teams including DB, Google, PySpark data engineering, and infrastructure.
Participate in Incident, Change, Release, and Problem Management processes.

Must Have Skills, Experience:

4-8 years of relevant experience.
Strong experience with Big Data technologies including PySpark, BigQuery, and Google Airflow.
Hands-on expertise in cloud platforms (Google Cloud, AWS, or Azure) and Linux system troubleshooting.
Proficiency in automation and DevOps tools such as Shell/Python scripting, CI/CD processes, and Ansible.

Apply

Vacancy posted 4 days ago

Similar jobs that could be interesting for youBased on the System Administrator (GCP/AWS/Azure, PySpark, BigQuery, and Google Airflow) in San Jose, CA vacancy

Data Engineer -(GCP,Airflow,Pyspark) Hybrid-Sunnyvale ,CA
...development, distributed systems. Solid understanding... ...ETL tools such as Apache Airflow, digdag, oozie. Experience... ...Services such as Google GCP and Microsoft Azure, as well as Distributed Storage... ...Required? ~ GCP,Airflow,Pyspark,Tableau. What are the...
Google
Work at office
Kaav Inc.
Sunnyvale, CA
1 day ago
Google Cloud Platform (GCP) Solution Architect - Pre Sales
...cloud-native development on GCP. Solution Design: Design and... ...leveraging services such as Google Kubernetes Engine (GKE), Cloud... ...competing cloud platforms (AWS, Azure) and cross-platform migration... ...Modernization concepts (e.g., BigQuery, Cloud SQL) as they relate to...
Amazon Web Service
Google
Yantran LLC
Santa Clara, CA
2 days ago
System Administrator
...accelerator hardware. The Role The System Administrator owns the day-to-day health of OXMIQ's... ...hybrid meetings across Teams, Zoom, and Google Meet for executive and customer‑... ...cloud administration in at least one of Azure, GCP, or AWS, including identity federation and SSO...
Amazon Web Service
Google
Work at office
Oxmiq Labs
Campbell, CA
4 days ago
Systems Administrator
...We are seeking a highly skilled Systems Administrator to join our growing IT team. The ideal candidate will be responsible for managing... ....Hands-on experience with cloud services such as AWS, Azure, or Google Cloud.Familiarity with virtualization technologies (VMware...
Amazon Web Service
Google
Work at office
Remote work
Flexible hours
Radix Construction Inc
San Jose, CA
4 days ago
Senior Systems Engineer (IaaS, Linux)
...Senior Systems Engineer (IaaS, Linux) FlashGrid Inc is looking... ...-critical databases in AWS, Azure, and Google clouds using FlashGrid and... ...features from Oracle/AWS/Azure/GCP and testing new reference... ...~5+ years of Linux system administration ~ Understanding of network...
Amazon Web Service
Google
Full time
Remote work
FlashGrid Inc
Sunnyvale, CA
3 days ago
Java Developer (Azure & GCP Cloud) || Sunnyvale, CA || Contract
...Java Developer (Azure & GCP Cloud) Location: Sunnyvale, CA Duration: 6 months We... ...cloud-native architectures, distributed systems, and modern development practices. This... ...microservices deployed in Kubernetes ecosystems on Google GCP and Microsoft Azure. • Architect...
Google
Contract work
AceStack LLC
Sunnyvale, CA
1 day ago
Linux (Cloud) Systems Administrator
...We have contract Onsite opportunity for Linux (Cloud) Systems Administrator in Santa Clara, CA (Onsite) Role : Linux (Cloud) Systems... ...: Experience with cloud platforms such as AWS, Azure, or Google Cloud . Ability to deploy, configure, and maintain...
Amazon Web Service
Google
Contract work
Remote work
SolGenie
Santa Clara, CA
3 days ago
Data Engineer (AI/ML Focus)
...using tools like Apache Spark, Airflow, and Kafka to support AI and... ...(PII or PHI) within AI systems, ensuring compliance with regulations... ...within cloud environments (AWS, Azure, Google Cloud) using services like S... ...AWS (Glue, SageMaker) or GCP. Analytical Skills:...
Amazon Web Service
Google
Diverse Lynx
Sunnyvale, CA
1 day ago
Azure Lead
...Azure Lead Long term Contract Columbia, SC - Remote Job Summary: We are seeking an experienced Azure Lead to spearhead... ...templates). Familiarity with Agile methodologies. Experience with other cloud platforms (AWS, Google Cloud) is beneficial....
Amazon Web Service
Google
Long term contract
Remote work
Sparktek
San Jose, CA
4 days ago
IT Administrator
$40 - $50 per hour
...Remote IT Administrator | $40 - $50 per hour | Full-Time | RemoteReady to take charge of... ...ManagementOversee cloud services, including AWS, Azure, and Google Workspace, optimizing performance and... ...Windows, macOS, and Linux operating systems.Experience with virtualization (...
Amazon Web Service
Google
Hourly pay
Full time
Remote work
Everglade Works
San Jose, CA
1 day ago
Junior Data Analyst
...jobs at top tech firms like Google, Apple, Client, Visa, PayPal... ...Track: Python/Java, Spark, Airflow, Databricks ETL architecture, Snowflake, S3, BigQuery, Redshift Infrastructure-... ...code and cloud data platforms (AWS, Azure) Why Candidates Choose SynergisticIT...
Amazon Web Service
Google
SynergisticIT
San Jose, CA
4 days ago
Tech Lead, Data & Inference Engineer
...across platforms such as Meta, Google and YouTube, they enable... ...evolving world of intelligent systems. Location San... ...modern orchestration tools (Airflow, Dagster, Prefect). ~ Familiarity... ...Kubernetes and cloud infrastructure (AWS, GCP, or Azure). ~ Bonus: Strong Node.js...
Amazon Web Service
Google
Full time
Catalyst Labs, LLC
Cupertino, CA
10 days ago
Senior Data Scientist
...warehouses (Snowflake, BigQuery, or Redshift). ~... ...g., Kubeflow, MLflow, Airflow/Dagster for orchestration... ...a cloud environment (GCP or AWS). ~ Excellent communication... ...time / streaming data systems (Kafka, Pub/Sub,... ...cybersecurity technologies and Google SecOps solutions....
Amazon Web Service
Google
Work at office
Tenex.AI Inc
San Jose, CA
2 days ago
Backend Engineer (Senior+)
...Airbnb for over 9 years combined. Backed by Google’s Gradient Ventures because they... ...integrating data. The modern replacement for Airflow. Our mission is to make data... ...Experience working with cloud providers (e.g., AWS, GCP, Azure) Experience writing technical design...
Amazon Web Service
Google
Full time
Remote work
Monograph
Santa Clara, CA
3 days ago
Sr. Cloud Security Architect, GCP (Remote)
$125k - $180k
...Architect to serve as the primary technical authority for our Google Cloud Platform (GCP) ecosystem. Joining our Information Security Architecture... ...Strategy: Provide architectural oversight for existing AWS and OCI environments, ensuring a consistent security posture...
Amazon Web Service
Google
Work at office
Local area
Remote work
CrowdStrike
Sunnyvale, CA
2 days ago
Oracle DBA - San Jose, CA
...leverages cloud-native technologies, Big Data systems, and AI-powered automation to optimize... ...available database clusters (Oracle, Google Spanner, MySQL, CloudSQL, BigTable).... .... ~ Strong multi-cloud experience (AWS, GCP, Azure). ~ Hands-on experience with...
Amazon Web Service
Google
E-Solutions
San Jose, CA
1 day ago
Senior MLOps & AI Infrastructure Engineer
$149.1k - $215.93k
...operationalize machine learning systems at scale. This role sits... ...such as Kubeflow, MLflow, Airflow, or similar •... ...cloud ML infrastructure on AWS (SageMaker), Azure (AML), or GCP (Vertex AI) with cost and... ...such as NVIDIA, AMD, Intel, Google DeepMind, or similar AI/HPC...
Amazon Web Service
Google
Local area
Shift work
Altera
San Jose, CA
2 days ago
Principal Engineer Software (AI Ops)
$104k - $169.5k
...is powerful, featuring Golang, Java, Python, BigQuery, Spanner, Pub Sub, Kubernetes and more, all within the Google Cloud Platform. This isn't just another coding... ...with one or more cloud platforms, such as AWS, Azure, GCP, Kubernetes and their technologies (Lambda functions...
Amazon Web Service
Google
Full time
Work at office
Remote work
Palo Alto Networks
Santa Clara, CA
5 days ago
Sr Staff Software Engineer
$145k - $235.5k
...robust and efficient backend systems. Qualifications Required... ...(SQL), MongoDB, and BigQuery. Proven experience building... ...applications on cloud platforms such as Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure. Demonstrated ability as a...
Amazon Web Service
Google
Full time
Work at office
Visa sponsorship
Work visa
Palo Alto Networks
Santa Clara, CA
5 days ago
Distinguished Software Engineer (Big Data & AI)
$230k - $315k
...establish the standards and systems necessary to process and... ...and evangelism of the core BigQuery, Vertex AI, Nvidia Triton,... .../Architect ~ Mastery of Google Cloud Platform (GCP) with extensive, hands-on... ...and Vertex AI or equivalent AWS, Azure, or other Big Data & AI...
Amazon Web Service
Google
Full time
Work at office
Palo Alto Networks
Santa Clara, CA
5 days ago
(Agile1) IT- Product Specialist
...large scale PostgreSQL environments on GCP alongside AWS. This is a senior Database Reliability... ...expertise) MongoDB (critical requirement) AWS Google Cloud Platform (GCP) Terraform CI/CD... ...worked with for databases? AWS, GCP, Azure? Which AWS managed databases have you...
Amazon Web Service
Google
InterSources
Sunnyvale, CA
4 days ago
Principal Software Engineer
$126k - $204.5k
...native solutions on GCP. Lead the redesign of... ...crafting robust distributed systems that achieve both... ...platforms such as Google Cloud Platform (GCP),... ...Amazon Web Services (AWS), or Microsoft Azure and designing cloud-native... ..., Redis, BigTable, BigQuery ~ Experience with...
Amazon Web Service
Google
Full time
Temporary work
Work at office
Palo Alto Networks
Santa Clara, CA
4 days ago
Senior Remote Data Engineer - GCP & BigQuery
...Engineer with at least 10 years of experience, particularly in Google Cloud Platform. The role includes designing and optimizing analytical... ...data needs. The ideal candidate will have expert skills in BigQuery, Python, and advanced SQL, alongside experience in data...
Google
Remote work
Global Channel Management
Santa Clara, CA
4 days ago
Tech Lead - AI Engineering
...from a leading AI lab (Anthropic, OpenAI, Google DeepMind, and the like) or Big Tech (... ...Architecture & Build Define and lead system architecture (back-end, APIs, and lightweight... ...solutions across cloud environments (AWS, Azure, GCP) Design evaluation frameworks for...
Amazon Web Service
Google
T3
San Jose, CA
2 days ago
Cloud Network Architect: GCP/AWS, IaC & Global Load Balancers
...You will configure and manage VPCs, VPNs, and load balancers on Google Cloud Platform while ensuring robust security and troubleshooting... ...such as Google Cloud Professional Network Engineer or AWS Certified Advanced Networking, and possesses strong skills in Google...
Amazon Web Service
Google
Ethereum Technologies LLC
Santa Clara, CA
2 days ago
Deal Data Technology & Analytics, Senior Associate
$77k - $202k
...Management Information Systems, Engineering - 3... ...pipelines (e.g., Azure Synapse Analytics, dbt, Apache Airflow) and AI/ML platforms... ...Azure OpenAI, Claude, Google Gemini) for business... ...native AI services (AWS SageMaker, Azure ML,... ...such as AWS, Azure, GCP, etc. - Certifications...
Amazon Web Service
Google
Full time
H1b
PwC
San Jose, CA
4 days ago
Technical Program Manager (TPM) - Security Operations
$60 - $70 per hour
...metrics. • Experience managing programs across cloud platforms (AWS, GCP, Azure) and modern security stacks (Splunk, CrowdStrike, Tenable,... ...dashboards using tools like Jira, Confluence, Notion, Google Sheets, or Tableau. • Strong written and verbal communication...
Amazon Web Service
Google
Work at office
Stefanini
Santa Clara, CA
1 day ago
Staff AI/ML Full-Stack Engineer: End-to-End AI Platforms
...focused on technical depth, system design, and product impact.... ...Design & Implementation: Use GCP/Azure to design, implement, and test... ...Points Experience with Google Cloud, Microsoft Azure, or AWS. Experience with Kubeflow, Flyte, Airflow, etc. Experience with Kubernetes...
Amazon Web Service
Google
Israelvcforum
Sunnyvale, CA
2 days ago
Senior DevOps Engineer
...Job Title Skills: Must have hands-on working experience with Amazon AWS, Google GCP or MS Azure Must have hands-on experience building large scale distributed serving system Must have experiences of Linux and Shell scripting Must have good communication...
Amazon Web Service
Google
Work experience placement
Software Technology Inc
Santa Clara, CA
1 day ago
Database Engineer
...large scale PostgreSQL environments on GCP alongside AWS. This is a senior Database Reliability... ...requirement) AWS Google Cloud Platform (GCP) Terraform... ...you worked with for databases? AWS, GCP, Azure? Which AWS managed databases have...
Amazon Web Service
Google
Contract work
H1b
3B Staffing LLC
Sunnyvale, CA
1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to System Administrator (GCP/AWS/Azure, PySpark, BigQuery, and Google Airflow). Be the first to apply!