Senior Network Engineer, Operations (Network Operations Center)

Job Description Job Description About CENIC: Join the team that connects California to the world — advancing education and research statewide by providing the world-class computing network essential for innovation, collaboration, and economic growth. Our organization operates the California Research and Education Network (CalREN), a high-capacity computer network with more than 8,000 miles of optical fiber. The network serves over 20 million users across California, including the vast majority of K-20 students, together with educators, researchers, and individuals at other vital public-serving institutions. Bring your IT career and technical expertise to help support the California Research and Education Network (CalREN), a high-bandwidth, fiber-optic network that serves over 20 million users. CENIC is a nonprofit governed by its member K-12 schools, libraries, community colleges, universities, and other public-serving institutions. Its mission is to advance education and research by providing a world-class network essential for innovation, collaboration, and economic growth. **We are looking for Network Engineers at various levels: mid-career > senior > technical expert > and leadership roles! Please see our career center for other positions available.** POSITION SUMMARY: Reporting to the Manager of Operations, the Senior Network Engineer, Network of Operations, provides operational support for CENIC’s advanced services network to advance CENIC’s organizational goals and objectives of a high-performance, open, and secure network. The Senior Network Engineer will function as a point of escalation, providing technical leadership and decision-making to on-shift staff and/or escalating issues, both internally and externally, as necessary. ESSENTIAL ENGINEERING FUNCTIONS: Monitor Proactively monitoring the CENIC networks to detect and resolve problems and ensure uninterrupted operations of the networks. Monitor daily workflows to ensure timely execution of next action items. Review notifications for accuracy and relevancy of included information. Review turnover reports, scheduled maintenance, and the general state of the network. Provide oversight of all ongoing outages, providing technical insight and guidance to Network Engineers. Monitor hardware and software vendor advisory notices for potential impact. Participation in Triage shifts as needed. Troubleshoot Problem troubleshooting and resolution, including interaction with carriers, vendors, and CENIC’s senior-level engineering group Serve as the point of contact for major incidents involving multiple sites or backbone services and determine the best course to resolution. Take ownership of complex problems involving our member sites or backbone services. Draft clear and concise announcements for network outages. Investigate, document, and perform root cause analysis of incidents or complex problems. Compile post-mortem reports to be shared with leadership and members, identifying areas for improvement. Evaluate all risk exposures and collaborate with teammates on best paths to resolution. Customer Service Problem troubleshooting and resolution with CENIC associates. Respond to service requests via phone and ticketing system. Prepare announcements for maintenance activities and network outages. Backup escalation for external escalations when the Lead Engineer is unavailable. Maintenances Planning, evaluating, and executing network maintenance activities. Assist the network administrators in coordinating the shipment of equipment to network sites. Assist network administrators with announcements about maintenance activities. Approves change control requests from Network Engineers. Participates in advanced maintenance tasks. Subject Matter Expert Serve as a subject matter expert for a technology(s) used on CalREN or a toolset that Operations utilizes Stay current on emerging trends by self learning and collaborating with the engineering team. Ensure documentation is current and updated; provide training to the team when needed Documentation Participation in updating and revising policies, procedures, technical reports, documentation, and internal training material. Maintain accurate inventory for network equipment. Create and maintain technical documentation for Network Operations. Contribute to the weekly major issues report. Participate in post-mortem documentation with Lead Engineer and Manager. Continuous Improvement OKR Execution: Sets, monitors, and achieves OKRs that align with organizational goals. Identifies and implements data-driven enhancements to processes, systems, and outcomes. Tracks progress, adjusts plans as needed, and ensures measurable results. ESSENTIAL LEADERSHIP FUNCTIONS: Staff Development Work closely with Lead Engineers and the Manager on new hire development, beginning with interview participation and continuing through onboarding and training. Work closely with Lead and Core Engineers to determine upcoming technology implementations that require additional training and documentation for Operations. Mentor interns, Network Administrators, and Network Engineers, and identify opportunities to enhance the overall technical level in Operations. Conduct learning sessions (BSLs) for the team on emerging technologies used on CalREN. Staff Accountability Handle on-shift escalations from team members regarding technical issues and processes. Ensure that operations engineers are adhering to processes, procedures, and best practices. Audit, evaluate open/resolved tickets for appropriate follow-up, and ensure consistency and processes are maintained. Audit maintenance calendar to ensure standards are met, and competing maintenance will not have overlapping impacts. Coordinate turnover activities with other Operations Engineers to provide smooth and seamless network support across shifts. Strategic Planning Identify procedural, technical, & soft skill gaps within Network Operations and provide solutions to resolve them. Work closely with Lead Engineer and Manager on the development and implementation of new and/or enhanced procedures, policies, and documentation. Participate in the Operations Engineer hiring process. KEY SUCCESS FACTORS: Being proactive when issues are discovered and ensuring the proper remediation is completed. Delegate tasks and exercise sound judgment, leadership, and rapport during critical situations. Exercise tact and effective communication with colleagues. Ability to consider, understand, and prioritize security (and privacy, where applicable) principles, practices, and procedures in all aspects of your role with CENIC. Willingness to work any shift (including weekends) as assigned and to rotate shifts and schedules as assigned. Communicate effectively via phone and email with network contacts and vendors. Prioritize multiple demanding events along with daily tasks. Ability to perform risk assessment and mitigation to avoid negative impacts REQUIRED EDUCATION AND/OR EXPERIENCE: 5 years in a network operations environment in support of customer networks OR 5 years in a technical support role interacting with contacts via phone and ticketing system AND Hands-on experience with Cisco and Juniper network devices. Administer and configure Cisco and Juniper routers/switches, implement firmware changes, OS releases, and other software upgrades. Troubleshoot router/network performance problems. Troubleshoot circuit problems for commercially-lit services and self-lit services. Prepare announcements for maintenance activities or network outages. Understanding of network security, secure configuration of network devices, and the use of network devices to secure networks of computing systems, including configuring and maintaining ACLs, knowledge of firewalls, and intrusion detection systems. Experience with installing, upgrading, and securing network equipment and services. Experience writing technical documentation and drafting network diagrams PREFERRED EDUCATION AND/OR EXPERIENCE: Cisco CCNP certification. Experience with Layer 3 routing protocols such as BGP and MPLS-SR Scripting experience using Python Direct experience in network configuration, management, and operations. Experience with optical transmission systems such as DWDM. Experience in a university and/or research environment. WORK ENVIRONMENT: This position operates in a professional office environment with standard office equipment, including computers, telephones, photocopiers, and filing cabinets. WORK HOURS: Network Operations is a 24-hour, 7-day per week, 365-day per year operation. An employee's regular work schedule is four ten-hour days on any of the seven days in the week, based on CENIC's business needs. WORK AUTHORIZATION: Documented Proof of identity and legal eligibility to work in the United States. Applicants must be authorized to work for any employer in the U.S. We are unable to sponsor or take over sponsorship of an employment Visa at this time. CENIC does not offer STEM OPT training opportunities and cannot complete Form I-983. WHY WORK AT CENIC? Company Culture: Small teams and a collaborative environment provide the opportunity to make a significant impact in a small organization. Community Engagement: We work with research and education institutions in the largest state economy in the United States. Your work has meaning. Generous Health Benefit Package : We value our employees' wellness. We offer a variety of medical, dental, and vision benefits. We have you covered with life, short-term, and long-term insurance. We also offer pet insurance at a discount. Paid Time Off: We offer liberal vacation and sick leave, plus 15 paid holidays. Investment in Your Future: CENIC contributes to your 403(b) even if you don't. We also offer matching contributions. Pre-tax and Roth options are also available. Direct applicants only. No recruiters. CENIC is an equal opportunity employer – Vets/disabled