HPC Systems Engineer

Job Summary

We are seeking an experienced HPC Systems Engineer to design, deploy, administer, and maintain High Performance Computing (HPC) environments across on-premises and cloud-based infrastructures. The role involves managing Linux-based HPC clusters, deploying scientific applications, configuring job schedulers, automating system administration tasks, and optimizing infrastructure performance, availability, and security.

The ideal candidate will have strong expertise in Linux administration, HPC infrastructure management, application deployment, scripting, and troubleshooting, along with familiarity with cloud technologies, containerization, and HPC workload management tools.

Key Technical Skills and Qualifications

Required Skills

• Experience building, deploying, and maintaining scientific applications and software module environments across on-premises and cloud-based HPC platforms.

• Strong expertise in Linux system administration, Linux operating system internals, system troubleshooting, and security hardening.

• Familiarity with Open OnDemand and AWS cloud infrastructure for HPC workloads.

• Experience with HPC application containerization technologies, such as Singularity/Apptainer, and container registries.

• Knowledge of HPC job scheduling and workload management systems, including Slurm and PBS-based scheduling environments.

• Experience with LDAP and Active Directory integrations for authentication and access management.

• Proficiency in Bash scripting for system administration, automation, and operational tasks.

• Strong analytical and troubleshooting skills to diagnose and resolve complex infrastructure and application issues.

• Ability to create and maintain technical documentation, system diagrams, configuration guides, and operational procedures.

• Effective communication and collaboration skills, with the ability to work closely with researchers, application teams, and infrastructure engineers.

Preferred Skills

• Experience with Ansible for configuration management and infrastructure automation.

• Familiarity with Splunk and Nagios for infrastructure monitoring, log analysis, and alerting.

• Experience with Bright Cluster Manager, now known as NVIDIA Base Command Manager.

• Familiarity with Jira for issue tracking, task management, and operational coordination.

Roles and Responsibilities

• HPC Infrastructure Design and Implementation: Design, build, configure, and deploy HPC clusters, including compute resources, operating systems, storage, networking, and supporting software components.

• System Administration: Administer and maintain Linux-based HPC environments, ensuring system availability, reliability, scalability, and operational efficiency.

• Scientific Application Management: Deploy, configure, and maintain scientific applications, software dependencies, and module environments to support research and computational workloads.

• Job Scheduling and Workload Management: Configure, manage, and troubleshoot HPC job scheduling systems, including Slurm and PBS-based schedulers, to facilitate efficient resource utilization.

• Cloud and Containerization: Support HPC workloads in cloud environments, including AWS, and implement containerized application deployment solutions.

• Performance Optimization: Monitor and analyze system performance, identify resource bottlenecks, and implement improvements to optimize compute, storage, and workload efficiency.

• Troubleshooting and User Support: Diagnose and resolve system, application, scheduling, authentication, and environment-related issues while providing technical support to HPC users.

• Automation and Scripting: Develop and maintain Bash scripts and automation workflows to streamline routine administration, deployment, monitoring, and maintenance activities.

• Configuration Management: Support infrastructure standardization and automated configuration management using tools such as Ansible.

• Monitoring and Maintenance: Monitor HPC infrastructure health, review logs and alerts, perform preventive maintenance, and coordinate system upgrades and technology enhancements.

• Security and Access Management: Implement Linux security hardening practices, manage user access and permissions, and support centralized authentication through LDAP and Active Directory.

• Documentation: Maintain accurate technical documentation, architecture diagrams, system configurations, standard operating procedures, and troubleshooting guides.

• Cross-Functional Collaboration: Work with research teams, data scientists, application developers, and infrastructure teams to understand computational requirements and deliver reliable HPC solutions.

• Continuous Improvement: Evaluate emerging HPC technologies, identify operational improvements, and implement solutions that enhance performance, security, automation, and service reliability.