Scientific Systems Software Developer

Software Developer 2 SLAC National Accelerator Laboratory is a U.S. Department of Energy laboratory operated by Stanford University. For over 60 years, SLAC has been at the forefront of scientific discovery, exploring how the universe works at the biggest, smallest, and fastest scales. From particle physics to astrophysics, materials science to biology, SLAC's world-class research facilities and scientific expertise drive innovation and push the boundaries of human knowledge. About the Role Do you enjoy collaborating with a diverse group of people to solve complex challenges? Does contributing to breakthrough discoveries in science and working in a world-leading research environment excite you? The Application and User Services (AUS) group is seeking an energetic, forward-thinking software engineer to develop tools and workflows for science projects such as the Vera C. Rubin Observatory US Data Facility, hosted at SLAC. You will work with a team that supports multiple global, open-science, collaborations eager to leverage cutting-edge, best-in-class computing, platform and data services. We continually explore new platforms and technologies, and you will have a direct hand in shaping what the future of scientific computing at SLAC looks like. S3DF is SLAC's centralized scientific computing facility, providing unified high-performance computing, storage, and data services to a broad portfolio of science programs. Current users include the Vera C. Rubin Observatory US Data Facility (supporting the Legacy Survey of Space and Time), LCLS (the Linac Coherent Light Source X-ray free-electron laser), ATLAS and other HEP experiments, and a growing number of additional programs. These experiments collectively generate and manage data at petabyte scale, with demanding requirements for throughput, reliability, and scientific reproducibility. You will design, implement, and support the applications and APIs used to process, manage, and serve scientific data across these programs. You will contribute to troubleshooting and tuning the full portfolio of services used by scientists worldwide. You will have experience in data-intensive workflows and containerization and deployments on Kubernetes clusters. Some of your code will interface with authentication and authorization frameworks, so familiarity with Identity and Access Management (IAM) concepts is a plus. A significant and growing part of this role will involve scientific data management at scale. The Facility manages a complex, multi-system data landscape spanning distributed data repositories, metadata catalogs, and data access layers. You will help us bring coherence, reliability, and formal stewardship practices to this environment: aligning data lifecycle policy with operational reality, ensuring metadata integrity, and building the tooling and workflows that let us manage data responsibly across its full lifetime. We encourage open dialog, free thinking, cooperation, and a growth mindset. This is an opportunity to learn, enable groundbreaking science, and develop your skills in a uniquely collaborative scientific computing environment. Key Responsibilities Design, implement, and maintain applications and APIs for S3DF science programs, enabling scientists worldwide to access, process, and analyze large-scale experimental and observational datasets Build and improve data-serving interfaces and processing pipelines that handle petabyte-scale astronomical datasets Containerize applications and deploy them on Kubernetes clusters, following modern cloud-native best practices Participate in design and development of software tools for scientific data management and data processing, with assignments varying according to experiment priorities and lifecycles. Integrate applications with Identity and Access Management frameworks (OIDC, SAML2, JWT, LDAP, COManage, etc.) to ensure secure and appropriate data access Write clean, well-tested, well-documented code; contribute to and maintain shared software repositories across S3DF science programs Scientific Data Management & Stewardship Contribute to the integration and reconciliation of the Rubin data access layer with distributed data management systems (including Rucio ), ensuring consistent views of datasets across systems Help develop and implement data lifecycle policies defining how datasets are created, retained, migrated, and retired across storage tiers in alignment with scientific and operational requirements Work on tooling for metadata integrity and remediation: detecting inconsistencies between catalogs, registries, and physical storage, and building workflows to detect and resolve them Contribute to formalizing levels of data stewardship responsibility by dataset type clarifying ownership, curation standards, and access controls across the Rubin data portfolio Develop and support workflows for exporting managed data subsets of data for downstream science, collaborators, and community data releases Help rationalize and consolidate the landscape of backing databases, improving consistency and reducing operational complexity Collaborate with Rubin data management teams across institutions to align on standards and tooling Operations & User Support Support scientists and users in their day-to-day use of Data Facility services diagnose issues, debug problems, and provide timely, clear resolutions Monitor the health and performance of deployed services and supporting infrastructure; gather metrics and produce reports Identify and resolve bottlenecks in data-intensive workflows, from ingestion and processing through serving and analysis Participate in on-call rotation and incident response for production services Develop and maintain runbooks, operational documentation, and user-facing guides Use the incident management system to track problems to resolution in a timely manner. Platform & Infrastructure Collaboration Work closely with platform engineers to deploy and tune services on Kubernetes and HPC systems Contribute to CI/CD pipelines, GitOps workflows, and infrastructure-as-code tooling for Rubin services Collaborate with the broader scientific software community (including teams at partner institutions) to align on interfaces, standards, and shared components Evaluate and test emerging technologies and technical developments Provide feedback and concrete recommendations for service improvements to the AUS team and scientific stakeholders Collaboration & Communication Work effectively within a multidisciplinary team spanning software engineering, platform operations, and scientific research Communicate clearly with both technical peers and non-technical scientific users Contribute to team planning and agile/scrum processes Participate in cross-institutional software working groups and community forums Required Qualifications Bachelor's degree in physics, computer science, or a related field, and 5 years of relevant experience in software development, systems administration, or scientific/high-performance computing or an equivalent combination of education and experience Proficiency in one or more programming languages; Python strongly preferred, with C/C++ or JavaScript a plus Experience with modern software development practices: version control (Git), CI/CD pipelines, code review, and agile/scrum methodologies Familiarity with data-driven and data-intensive workflows Hands-on experience with PostgreSQL (schema design, query optimization, and operational management) Experience with Apache Kafka or similar event streaming platforms for high-throughput, real-time data pipelines Practical experience deploying and operating applications on Kubernetes in a production environments at scale Understanding of distributed compute and storage systems, high-performance computing, and networking concepts Experience with system monitoring, benchmarking, and performance analysis Strong organizational and communication skills; ability to work effectively in a collaborative, distributed team environment Ability and genuine willingness to learn, adopt best practices, and grow technical skills on the job Preferred Qualifications Experience with Python data science and scientific computing libraries: NumPy, Pandas, Jupyter, Bokeh, Parquet, Dask, and similar Production development and operational experience with container technologies (Docker, containerd, Singularity/Apptainer) Experience with configuration management and infrastructure-as-code tools: Ansible, GitHub Actions, GitLab CI, ArgoCD Familiarity with HPC job scheduling systems (SLURM, HTCondor) Experience with common infrastructure software: PostgreSQL/MySQL, nginx, Apache, Redis, or similar Knowledge of Identity and Access Management frameworks and technologies: OIDC, SAML2, JWT, Grouper, COManage, LDAP Exposure to astronomical data formats and tools (FITS, HDF5, Butler, Parquet-based catalogs) or Stanford