Team Lead, Network Automation Infrastructure (SONiC & AI)

Senior Staff Engineer, Software

Req ID: 135512

Hiring Manager: Kevin Frick

Region: Americas

Country: USA

State/Province: New Hampshire

City: Remote Employee US

Summary

We are seeking an experienced full-stack software developer to design, develop, and test the next-generation AI-enabled automation infrastructure for switches and integrated data center racks. This is not a traditional "scripting" role; you will be building a full-stack, SDLC-integrated, test-driven engineering ecosystem centered around products running SONiC (Software for Open Networking in the Cloud) and OpenBMC within hyperscaler racks.

Your mission, collaborating with the larger DevTestOps Team and software development teams, is to build an immersive, intuitive, robust, and full-featured control center for managing world-wide network automation test infrastructure. The Test Automation Control Center will consist of a set of cloud-hosted applications controlling multiple physical on-prem systems distributed worldwide. Control capabilities cover scheduling, monitoring, analyzing, and reporting tests from the initial developer Pull Requests to the final release regression validation. Managed equipment includes the entire range of Celestica switches, high-performance IXIA traffic generators, and a variety of equipment commonly found in data center racks. You will use your knowledge of AI agents to integrate intelligent workflows that select the best automation test cases to run, diagnose test failures for rapid resolution, and analyze test coverage to provide predictive product quality assessment.

Detailed Description

Performs tasks such as, but not limited to, the following:

• Lead the design, development and implementation of technical solutions for complex projects, involving multiple domains. Participate in project planning and scheduling.

• Global SME with comprehensive knowledge and industry recognition. Provides technical leadership and direction to a global team of engineers.

• Take responsibility for non-technical elements of an engineering project (people, financials etc.)

• Review and interpret customer specifications and may act as primary customer contact.

• Analyze trade-offs in complex systems and recommend solutions. Develops deployment strategies and plans.

• Lead the deployment of strategic complex programs and coordinate site-wide deployment efforts.

• May manage relationships with key vendors/partners.

• Analyze, design and develop tests and test-automation suites.

• Design and develop a processing platform using various configuration management technologies.

• Test software development methodology (may be done in agile environment)

• Provide ongoing maintenance, support and enhancements in existing systems and platforms.

• Collaborate cross-functionally with customers, users, project managers and other engineers including Peer-Reviews to achieve elegant solutions.

• Provide recommendations for continuous improvement.

• Work alongside other engineers on the team to elevate technology and consistently apply best practices.

• Keep up to date with relevant industry knowledge and regulations

Knowledge/Skills/Competencies

Infrastructure & SDLC Integration

• Architect a CI/CD Pipeline: Design the integration between Git-based workflows and physical hardware labs, ensuring code changes trigger automated builds and deployments to SONiC-based switches.

• Cloud-to-On-Prem Connectivity: Lead the development of a cloud-hosted GUI and backend services that securely manage and command on-premise physical test beds.

• Hardware Abstraction: Oversee the management of physical test beds, ensuring consistent state and availability for automated testing.

Test Engineering & Validation

• Framework Leadership: Standardize automated testing using SPyTest, ensuring robust coverage for NOS (Network Operating System) features.

• Traffic Emulation: Integrate IXIA traffic generators into the automated suite to perform high-scale performance, stress, and regression testing.

• Regression Management: Own the final validation gate, ensuring that no code reaches production without passing a rigorous, automated physical battery.

AI-Driven Operations

• Failure Analysis Agents: Build and deploy AI/LLM-based agents to parse complex log files and SPyTest results to identify the "root cause" of test failures automatically.

• Self-Healing Test Beds: Develop agents capable of test bed failure recovery (e.g., automatically power-cycling hung PDUs, re-flashing corrupted ONIE images, or re-seating virtual links).

• Quality Insights: Leverage AI to analyze long-term software quality trends and predict potential regressions before they occur.

Technical Requirements

• Networking: Deep expertise in SONiC, SAI (Switch Abstraction Interface), and standard protocols (BGP, EVPN, VXLAN).

• Automation Frameworks: Expert-level knowledge of SPyTest and Python-based automation.

• Hardware: Experience with IXIA (IxNetwork/IxLoad) and physical switch hardware (Mellanox/NVIDIA, Broadcom-based whitebox).

• Software Development: Strong proficiency in Python, C/C++, Rust, or Java; experience building RESTful APIs and cloud-native backends (GCP/Azure).

• AI/ML: Familiarity with integrating LLM APIs (like Google Gemini) for text/log analysis.

• DevOps: Advanced experience with GitHub Actions, Azure DevOps or Jenkins, and containerization (Docker/Kubernetes).

Preferred Qualifications

• Active contributor to the Azure/SONiC open-source community.

• Experience building custom dashboards using React or Vue.js .

• Knowledge of deploying and operating software within GCP.

• Background in developing "Self-Healing" infrastructure or AIOps.

Physical Demands

• Duties of this position are performed in a normal office environment.

• Duties may require extended periods of sitting and sustained visual concentration on a computer monitor or on numbers and other detailed data.

• Repetitive manual movements (e.g., data entry, using a computer mouse, using a calculator, etc.) are frequently required.

• Occasional travel may be required.

Salary

The stated range includes Base Salary and target Short-Term Incentive (STI) compensation only. A comprehensive benefits package is offered in addition to this range.

$136K - $212K

The range described in this posting is an estimate by the Company, and may change based on several factors, including but not limited to a change in the duties covered by the job posting, or the credentials, experience or geographic jurisdiction of the successful candidate.

Typical Experience

• 12 to 18 years

Typical Education

Bachelor degree or consideration of an equivalent combination of education and experience.

Educational Requirements may vary by Geography

Notes

This job description is not intended to be an exhaustive list of all duties and responsibilities of the position. Employees are held accountable for all duties of the job. Job duties and the % of time identified for any function are subject to change at any time.

Celestica is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, pregnancy, genetic information, disability, status as a protected veteran, or any other protected category under applicable federal, state, and local laws.

This policy applies to hiring, promotion, discharge, pay, fringe benefits, job training, classification, referral and other aspects of employment and also states that retaliation against a person who files a charge of discrimination, participates in a discrimination proceeding, or otherwise opposes an unlawful employment practice will not be tolerated. All information will be kept confidential according to EEO guidelines.

Celestica is an E-Verify employer.