Principal Systems Engineer (Codex)

Salary: $142,000 - 194,000 per year Requirements:

• 8+ years of software operations or infrastructure automation experience, with strong Python and Bash skills.
• Expert Linux administration experience, particularly with Ubuntu and Oracle Linux in large-scale production environments.
• Strong understanding of distributed systems and peer-to-peer, node-to-node, and service-to-service communication.
• Data-center and host lifecycle experience, including provisioning, validation, repairs, hardware replacement, and fleet recovery.
• Strong troubleshooting and problem-solving abilities, with excellent communication and teamwork skills.
• Experience with observability tools for metrics, logging, dashboards, and alerting.
• Experience with AI agents and related tooling.
• Experience leading on-call operations and incident response.
• Bachelors degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• Hands-on experience with NVIDIA- or AMD-based GPU infrastructure and familiarity with GPU shapes or instance families is preferred.
• Experience operating or automating GPU, compute, or other large-scale cloud infrastructure is preferred.
• The posting also lists minimum qualifications of 11 years of relevant experience; or a related bachelors degree and 7 years of relevant experience; or a related masters degree and 5 years of relevant experience.
• Additional listed skills include knowledge of cybersecurity trends, incident management and response, training and development, technical account management, and cloud architecture.
• Preferred qualifications also include 12 years of relevant experience; or a related bachelors degree and 8 years of relevant experience; or a related masters degree and 6 years of relevant experience.

• Lead GPU repair operations, diagnosis, and resolution of complex, high-severity failures across hardware, Linux, networking, drivers, firmware, and dependent cloud services.
• Develop and maintain automation and operational tools for GPU infrastructure across multiple regions.
• Partner with software engineering, hardware, and operations teams to maintain high GPU fleet availability.
• Improve monitoring, alerting, and diagnostics for fleet health, performance, capacity, and utilization, using tools such as Grafana.
• Act as the senior escalation point for complex GPU host and repair issues.
• Respond to incidents, conduct root-cause analysis, and address issues affecting GPU capacity, availability, and regional deployments.
• Improve AI2 Ops processes, GPU fleet automation, and readiness for OCI region builds.
• Participate in on-call rotations and support critical infrastructure incidents.
• Document operating procedures, automation workflows, troubleshooting guides, and runbooks.
• Develop and improve AI agents, overseeing their safe rollout, execution, and monitoring.
• Mentor junior engineers, provide senior technical guidance, and promote continuous improvement.

• AI
• AI Agents
• Bash
• Cloud
• Firmware
• Grafana
• Hardware
• Incident Management
• Support
• Linux
• Oracle
• Python
• Ubuntu
• NodeJS
• Network

More:

We are collaborating with hackajob to recruit a Principal Systems Engineer for Oracle Cloud Infrastructures AI Infra Operations team. In this technical operations leadership role, you will support GPU infrastructure and help scale reliable operations as OCIs GPU fleets grow across regions. You will work closely with engineering and operations partners, with a focus on availability, repair efficiency, automation, observability, and data-driven operational improvement.

last updated 41 week of 2026