Lab Infrastructure Engineer

Lab Infrastructure Engineer

This role owns end-to-end execution for building, expanding, and operating high-performance network infrastructure supporting ML training clusters, data collection rigs, and codec development labs, precision-timed data capture systems, and real-time media processing labs. Partnering closely with engineering, hardware, ML systems, and capacity planning teams to deliver reliable, production-parity lab environments.

This role uses a variety of proprietary internal infrastructure-as-code tools. Candidates with experience in similar open-source or commercial equivalents (e.g., Nornir, NAPALM, Ansible, Terraform, NetBox, Capirca) and a demonstrated ability to quickly learn new platforms will be successful.

Lab Build-Out & Deployment (Core)

Drive deployment plans, cutovers, and upgrades with clear documentation, checklists, MOP writing, peer review, change approval processes, risk management, and rollback planning. Coordinate with cross-functional partners and vendors to manage dependencies and resolve blockers quickly. Maintain and evolve data center fabric cluster definitions, including rack provisioning, decommissioning, role reassignment (compute ↔ device ↔ storage), and physical port-map management within DCIM and topology modeling tools.

Automation & Configuration Engineering

Develop and maintain Jinja2 configuration templates for network device provisioning (JunOS, FBOSS) enabling data-driven, zero-hardcoded-per-site deployments. Build reusable template frameworks for routing policy, interface, and BGP peer-group rendering that scale across multiple sites with only site-specific data changes. Work with internal topology and configuration management platforms to define port roles, port assignments, and exception-based overrides for per-node/per-site configuration. Work within Git-based network configuration management systems (similar to infrastructure-as-code patterns), using policy engines for validation, rendering, and deployment. Ability to quickly learn and master internal tooling is essential Design data structures (JSON) for prefix-lists, route-filters, and aggregation data that feed directly into template rendering pipelines. Manage DNS records for lab infrastructure devices and services Experience with network device AAA (TACACS+, RADIUS) and management-plane access control Manage network OS lifecycle including image qualification, version standardization, and coordinated fleet upgrades using automated provisioning tools.

Observability / Monitoring / Alert Triage

Implement and use monitoring/telemetry to maintain reliability and speed up troubleshooting, using combinations of: gNMI-based telemetry (preferred) IPFIX/NetFlow/sFlow, syslog, SNMP Active probes (ICMP, RPM/IP SLA style) Triage monitoring alerts during deployments and participating in break/fix activities, escalation paths, and incident triage during and after bring-up. Support deployment and cutover activities during maintenance windows, which may include evenings and weekends as required. Test, Validation & Commissioning Execute lab and production acceptance testing for new deployments and upgrades. Use traffic generation / validation tooling (e.g., IXIA and/or Spirent TestCenter) to validate throughput, loss, latency, and failover behavior in ML cluster fabrics. Support regression and repeatability efforts by documenting test plans and results.

Required Qualifications Minimum 3-5 years of relevant experience in network engineering or infrastructure deployment. Hands-on experience delivering and troubleshooting network infrastructure in lab and/or production environments. Experience with data center switching/routing concepts and operational troubleshooting. Coding/automation experience (Python/Bash/Go) for improving repeatability and reducing manual operational load. Familiarity with a subset of the following (depth can vary based on focus area): DCN fabric concepts (spine-leaf/Clos), routing/switching fundamentals BGP (and related routing concepts), EVPN/VXLAN Nice to have: OSPF (where applicable) Telemetry/monitoring and incident-style troubleshooting Strong cross-functional coordination skills and clear written/verbal communication. Willingness to work onsite as required by build and deployment schedules. Hands-on experience with IPv6, DHCPv6, and SLAAC. Familiarity with internal network management and provisioning tools (training provided). Experience with gNMI telemetry and modern monitoring stacks. Experience with version control (e.g., Git), configuration management, or automation frameworks (helpful, but not required). Familiarity with storage networking concepts (NFS, parallel file systems, high-throughput storage fabrics) Familiarity with PTP/IEEE 1588 or gPTP/IEEE 802.1AS for precision timing in media, capture, or industrial environments