Principal Core Infrastructure Engineer, AI Infrastructure

Job Description

OCI (Oracle Cloud Infrastructure) AI Infrastructure is at the forefront of building a cutting-edge, ultra-high-performance GPU platform designed to support AI/ML/HPC workloads. This is your chance to be part of the AI revolution, creating systems that allow customers to scale from tens to thousands of GPUs without compromising performance.

Our team is responsible for designing and developing fundamental architectural changes for GPU delivery, health monitoring, triage automation, and diagnostic services. These are essential for running distributed AI/ML/HPC workloads across thousands of GPUs, leveraging technologies like RoCE and Infiniband.

Why Join Us?

• Innovative Projects: Build groundbreaking solutions for our customers from the ground up.
• Exciting Times: Be part of a young, fast-growing team working on ambitious new initiatives.
• Dynamic Environment: Collaborate in a vibrant, agile environment where learning and adaptability are key.

What We're Looking For:

• Adaptable Engineers: Self-motivated individuals with a quick learning ability.
• Technical Excellence: Rock-solid developers and distributed systems engineers with a deep understanding of distributed systems and algorithms. Comfortable diving deep into any part of the stack, as well as software debugging and low-level systems troubleshooting.
• Passion for Simplicity and Scale: Value simplicity and scalability in design and implementation.
• Collaborative Spirit: Comfortable working in a collaborative, agile environment and eager to learn. Ability to collaborate effectively with various dependencies, including Network and Data Center operations.

Join us and be a part of the team that's pushing the boundaries of AI technology!

Minimum Qualifications

• BS or MS degree in Computer Science or relevant technical field involving coding or equivalent practical experience
• Deep understanding of operating systems, computer networks, and high-performance applications
• 6+ years' experience delivering and operating large-scale production systems (1000+ server instances)
• Proficient in one programming language(java/python/c/c++/goLang/shell scripting)
• Systematic problem-solving approach, strong communication skills, a sense of ownership, and drive.
• Proven ability to deliver products and experience with the full software development lifecycle

Preferred Qualifications

• Strong background in Linux systems
• Familiarity with system-level architecture, data synchronization, fault tolerance, and state management.
• General enterprise storage, networking, or computing experience
• Experience with Server/GPU hardware architecture and system management.
• Experience with Infiniband or RoCE networking
• Hands-on experience designing, developing, and operating public cloud service data planes
• Good understanding of databases and SQL (MySQL) and caching technologies (Redis, Memcache etc)

Responsibilities

Key Responsibilities

System Design & Architecture - System Scalability:

• Lead the development and implementation, and begin to architect, components of scalable distributed systems that support horizontal and vertical scaling to meet system demands, including leveraging distributed state management tools.
• Optimize code and/or systems for large-scale data processing and high-throughput requirements to support hyper-scale systems.
• Define scalability requirements for owned components and ensure design and implementation requirements are met.
• Design systems to scale with elasticity (e.g., effectively scaling both up and down).
• Leverage data plane platforms to effectively handle large-scale data retrieval, storage, and processing.
• Design performance and load testing.

System Design & Architecture - System Reliability Design:

• Build and design fault-tolerant components and systems capable of withstanding in-service updates by implementing redundancy, replication, and automatic failover mechanisms.
• Design systems to effectively handle service disruptions (e.g., network partitions) by prioritizing consistency, availability, or partition tolerance.
• Implement and optimize approaches to handle network unreliability, including load-shedding, throttling, and rate-limiting.
• Design components and systems that are durable and adhere to service level objectives (SLOs), setting expectations for availability and durability of other computing services within the department.

System Design & Architecture - System Reliability Performance:

• Define key performance indicators (KPIs) and telemetry to identify gaps or issues in running systems.
• Build and customize moderately complex dashboards, telemetry systems, and alerting mechanisms to proactively monitor components and system health..

System Design & Architecture - Correctness / Availability:

• Design and implement functional and correctness requirements for feature sets and/or systems in new or existing systems.
• Design complex test scenarios (e.g., fault-injection, brown-out) to evaluate system correctness.
• Implement data replication and synchronization techniques to maintain data integrity and availability.

Operational Troubleshooting & Incident Management:

• Take a proactive role in diagnosing, debugging, and resolving issues in active components and systems to support ongoing operation, and mentor others in these processes.
• Implement strategies to prevent interruptions, ensuring no maintenance windows are required for customers and users when resolving issues.
• Maintain expertise in owned components and systems to ensure effective troubleshooting and performance.
• Meet operational readiness expectations through design and implementation.
• Serve in operational support rotations, providing guidance in incident response and root cause investigations.

Compliance & Security:

• Implement robust security measures to protect data and applications in multi-tenant environments, including encryption techniques and access controls.
• Execute remediation plans to address identified security gaps.
• Ensure cloud infrastructure is in compliance with industry standards and regulations and that documentation is up to date.

Automation & Change Management:

• Develop and maintain automation scripts and tools (e.g., Infrastructure as Code (IaC)) to manage cloud infrastructure.
• Create and adhere to change management plans for patching, updating, and rolling back applications, and begin designing systems and components to allow for automation of these processes.

Core Responsibilities

Planning & Execution:

• Manages and coordinates moderately complex tasks, monitoring timelines and deliverables to ensure timely completion and adherence to requirements for a moderately-sized project or initiative. Efficiently delegates, monitors, and prioritizes work across multiple projects, providing technical oversight and adjusting plans to address shifts in resources or timelines.

Collaboration & Partnership:

• Collaborates across the organization to align on expectations and achieve shared objectives. Leverages understanding of business leaders, stakeholders, and/or customers to ensure proposed solutions meet their needs. Supports inclusivity by actively seeking and listening to diverse perspectives, ensuring others feel heard and respected.

Problem Solving:

• Identifies and addresses moderately complex issues by analyzing a wide range of data and/or information to identify solutions in accordance with standard practices. Proactively escalates unresolved or critical issues with a thorough assessment and suggests potential solutions. Reviews, contributes to, and documents problem solving strategies.

Continuous Learning:

• Pursues learning opportunities to expand knowledge and skills and/or tools in new areas and stays abreast of the latest industry trends and best practices. Proactively seeks and leverages ongoing feedback and training to improve skills. Coaches and mentors junior team members, fostering continuous learning and knowledge sharing within and across teams.

Continuous Improvement:

• Develops ideas, recommends updates, and/or collaborates on the implementation of process improvements to increase the efficiency and effectiveness of processes, protocols, and workflows across teams, and evaluates the impact on key stakeholders. Solicits feedback from others on ideas for alternative approaches and methods for continued improvement.

Performance and Development:

• Contributes to the talent development pipeline by participating in candidate interviews, assessing candidates, and providing hiring recommendations.

Qualifications

Disclaimer:

Certain U.S. based or U.S. customer or client-facing roles may be required to comply with applicable requirements, such as immunization/occupational health mandates, and/or drug testing requirements.

Range and benefit information provided in this posting are specific to the stated locations only

US: Hiring Range in USD from: $114,600 to $234,600 per annum. May be eligible for bonus, equity, and compensation deferral.

Oracle maintains broad salary ranges for its roles in order to account for variations in knowledge, skills, experience, market conditions and locations, as well as reflect Oracle's differing products, industries and lines of business.
Candidates are typically placed into the range based on the preceding factors as well as internal peer equity.

Oracle US offers a comprehensive benefits package which includes the following:
1. Medical, dental, and vision insurance, including expert medical opinion
2. Short term disability and long term disability
3. Life insurance and AD&D
4. Supplemental life insurance (Employee/Spouse/Child)
5. Health care and dependent care Flexible Spending Accounts
6. Pre-tax commuter and parking benefits
7. 401(k) Savings and Investment Plan with company match
8. Paid time off: Flexible Vacation is provided to all eligible employees assigned to a salaried (non-overtime eligible) position. Accrued Vacation is provided to all other employees eligible for vacation benefits. For employees working at least 35 hours per week, the vacation accrual rate is 13 days annually for the first three years of employment and 18 days annually for subsequent years of employment. Vacation accrual is prorated for employees working between 20 and 34 hours per week. Employees working fewer than 20 hours per week are not eligible for vacation.
9. 11 paid holidays
10. Paid sick leave: 72 hours of paid sick leave upon date of hire. Refreshes each calendar year. Unused balance will carry over each year up to a maximum cap of 112 hours.
11. Paid parental leave
12. Adoption assistance
13. Employee Stock Purchase Plan
14. Financial planning and group legal
15. Voluntary benefits including auto, homeowner and pet insurance

The role will generally accept applications for at least three calendar days from the posting date or as long as the job remains posted.
As part of Oracle's onboarding process and consistent with applicable law, US-based employees are required to complete identity verification, which involves the collection and processing of their biometric information. Accommodations to this requirement may be granted following an individualized assessment.

About Us

Only Oracle brings together the data, infrastructure, applications, and expertise to power everything from industry innovations to life-saving care. And with AI embedded across our products and services, we help customers turn that promise into a better future for all. Discover your potential at a company leading the way in AI and cloud solutions that impact billions of lives.

True innovation starts when everyone is empowered to contribute. That's why we're committed to growing a workforce that promotes opportunities for all with competitive benefits that support our people with flexible medical, life insurance, and retirement options. We also encourage employees to give back to their communities through our volunteer programs.

We're committed to including people with disabilities at all stages of the employment process. If you require accessibility assistance or accommodation for a disability at any point, let us know by emailing or by calling 1- in the United States.

Oracle is an Equal Employment Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability and protected veterans' status, or any other characteristic protected by law. Oracle will consider for employment qualified applicants with arrest and conviction records pursuant to applicable law.