Research Scientist, Distributed Workflows

Requisition Id 16759 Overview:Oak Ridge National Laboratory (ORNL) is seeking a Research Scientist to advance distributed workflow orchestration and cross-facility integration in support of the Genesis Mission, the American Science Cloud, and the Oak Ridge Leadership Computing Facility (OLCF). Working within the Workflows and Ecosystem Services (WES) Group at the National Center for Computational Sciences (NCCS), you will design and deploy APIs, service meshes, and workflow orchestration systems that bridge leadership computing resources, instruments, and cloud platforms. You will collaborate with scientists, facility operators, and DOE laboratory partners to lower barriers to scalable, automated scientific computing.WES designs and implements workflow, data, and ecosystem technologies that enable reliable, automated execution of scientific applications across heterogeneous, geographically distributed computing environments, integrating compute, storage, networking, and visualization resources into cohesive, end-to-end systems.The WES Group sits within the Advanced Technologies section of the National Center for Computational Sciences (NCCS), a division of the Computing and Computational Sciences Directorate (CCSD) at ORNL. NCCS provides state-of-the-art computational and data science infrastructure for technical and scientific professionals and hosts the Oak Ridge Leadership Computing Facility (OLCF), a DOE Office of Science User Facility that is home to Frontier, the first supercomputer to break the exascale barrier. NCCS staff work at the forefront of leadership computing, enabling scientific discovery for researchers across government, academia, and industry.As a U.S. Department of Energy (DOE) Office of Science national laboratory, ORNL has an extraordinary 80-year history of solving the nation’s biggest problems. We have a dedicated and creative staff of over 7,000 people! Our vision for diversity, equity, inclusion, and accessibility (DEIA) is to cultivate an environment and practices that foster diversity in ideas and in the people across the organization, as well as to ensure ORNL is recognized as a workplace of choice. These elements are critical for enabling the execution of ORNL’s broader mission to accelerate scientific discoveries and their translation into energy, environment, and security solutions for the nation.Major Duties/Responsibilities:Distributed Workflow Orchestration: Architect, implement, and operate cross-facility workflow orchestration systems that integrate job schedulers (e.g., SLURM), data movement services (e.g., Globus), and computing endpoints across multiple DOE facilities. Design execution patterns that support autonomous experimentation, multi-facility data pipelines, and event-driven automation.API and Service Mesh Development: Design and implement RESTful APIs and service meshes that expose OLCF computing resources to diverse scientific workflows and external facilities. Define API specifications, manage versioning, and guide systems from prototype through production deployment.Genesis Mission / American Science Cloud Integration: Lead ORNL’s technical engagement with the Genesis Mission, architecting cross-facility workflow and data infrastructure that connects leadership computing, experimental facilities, and cloud platforms in support of DOE’s vision for integrated science. Drive contributions to the American Science Cloud (AmSC) by designing and deploying interoperable APIs, data movement services, and orchestration layers that enable seamless scientific workflows across the DOE complex. Coordinate with partner laboratories and facilities (e.g., ANL, NERSC) to align interfaces and standards, and establish ORNL/OLCF as a foundational compute and data hub within the emerging national science infrastructure.Agentic Workflow Design and Autonomous Control Systems: Design and implement agentic workflow architectures that enable autonomous, closed-loop scientific experimentation across HPC and instrument facilities. Develop LLM-integrated orchestration layers, tool-use pipelines, and event-driven control systems that allow AI agents to plan, execute, monitor, and adapt multi-step scientific workflows with minimal human intervention. Apply agentic patterns to use cases such as autonomous materials discovery, self-steering simulations, and adaptive experimental campaigns, in coordination with domain scientists and facility operators.Scientific Domain Support: Collaborate with domain scientists in areas such as climate science, materials science, and autonomous experimentation to translate research objectives into scalable, automated workflow solutions. Provide documentation, training, and direct user support.Scripting and Tooling: Develop command-line tools and automation in Python, Bash, and/or C/C++ to encapsulate workflow steps, manage configuration files (e.g., YAML/JSON), and implement robust logging, error handling, and checkpoint/retry strategies.Operational Reliability and Optimization: Diagnose job failures, mitigate bottlenecks, and improve throughput, latency, and resource utilization. Use scheduler and Linux tools (e.g., sacct, squeue, coreutils, ssh, tmux, top, iostat) to monitor, analyze, and tune workflows.Research and Publication: Conduct original research on workflow systems, orchestration, and data management. Publish findings in high-impact, peer-reviewed venues and present work at major conferences (e.g., SC, HPDC, PEARC, eScience).Basic Qualifications:PhD. in Computer Science, Computer Engineering, or a closely related field.At least 2 years of experience designing or operating distributed systems, APIs, or workflow orchestration frameworks.At least 1 year of programming experience in Python, with working familiarity in Bash and/or a compiled language (C/C++ or similar).Experience with Linux-based HPC environments and familiarity with job scheduling systems (e.g., SLURM or PBS).Strong verbal and written communication skills, with the ability to collaborate across technical and scientific teams.Preferred Qualifications:Demonstrated experience publishing research in high-impact venues such as SC, IPDPS, HPDC, PEARC, or IEEE eScience.Familiarity with RESTful API design, containerization (Docker/Singularity/Apptainer), and microservice architectures.Experience working in or coordinating across DOE national laboratory environments, particularly in the context of integrated or cross-facility research infrastructure.Background in scientific data management, provenance tracking, or metadata systems.Demonstrated ability to lead technical projects, manage milestones, and coordinate multidisciplinary teams.Familiarity with basic cyber-security principles (e.g., SSH key hygiene, least privilege, network segmentation) as applied to workflow and API design.Special Requirements:Remote/Hybrid Eligibility: This position is eligible for hybrid schedule within approved U.S. States, subject to business needs, access/security requirements, and approvals. We offer a flexible work environment that supports both the organization and the employee. A hybrid/onsite working arrangement provides flexibility to work periodically from your home, while reporting onsite to the Oak Ridge, Tennessee location on a weekly and regular basis.Security, Credentialing, and Eligibility Requirements: This position requires the ability to obtain and maintain an HSPD-12 PIV badge.For employment at Oak Ridge National Laboratory (ORNL), a Real ID compliant form of identification will be required. Additionally, ORNL is subject to Department of Energy (DOE) access restrictions. All employees must also be able to obtain and maintain a federal Personal Identity Verification (PIV) card as mandated by Homeland Security Presidential Directive 12 (HSPD-12) and Department of Energy (DOE) Order 473.1A, which requires a favorable post-employment background investigation.To obtain this credential, new employees must successfully complete and pass a Federal Tier 1 background check investigation. This investigation includes a declaration of illegal drug activities, including use, supply, possession, or manufacture within the last year. This includes marijuana and cannabis derivatives, which are still considered illegal under federal law, regardless of state laws.For foreign national candidates:If you have not resided in the U.S. for three consecutive years, you are not eligible for the PIV credential and instead will need to obtain a favorable Local Site Specific Only (LSSO) risk determination to maintain employment. Once you meet the three-year residency requirement, you will be required to obtain a PIV credential to maintain employment. About ORNL:As a U.S. Department of Energy (DOE) Office of Science national laboratory, ORNL has an impressive 80-year legacy of addressing the nation’s most pressing challenges. Our team is made up of over 7,000 dedicated and innovative individuals! Our goal is to create an environment where a variety of perspectives and backgrounds are valued, ensuring ORNL is known as a top choice for employment. These principles are essential for supporting our broader mission to drive scientific breakthroughs and translate them into solutions for energy, environmental, and security challenges facing the nation.ORNL offers competitive pay and benefits programs to attract and retain individuals who demonstrate exceptional work behaviors. The laboratory provides a range of employee benefits, including medical and retirement plans and flexible work hours, to support the well-being of you and your family. Employee amenities such as on-site fitness, banking, and cafeteria facilities are also available for added convenience.Other benefits include the following: Prescription Drug Plan, Dental Plan, Vision Plan, 401(k) Retirement Plan, Contributory Pension Plan, Life Insurance, Disability Benefits, Generous Vacation and Holidays, Parental Leave, Legal Insurance with Identity Theft Protection, Employee Assistance Plan, Flexible Spending Accounts, Health Savings Accounts, Wellness Programs, Educational Assistance, Relocation Assistance, and Employee Discounts.This position will remain open for a minimum of 5 days after which it will close when a qualified candidate is identified and/or hired.We accept Word (.doc, .docx), Adobe (unsecured .pdf), Rich Text Format (.rtf), and HTML (.htm, .html) up to 5MB in size. Resumes from third party vendors will not be accepted; these resumes will be deleted and the candidates submitted will not be considered for employment.If you have trouble applying for a position, please email View email address on click.appcast.io is an equal opportunity employer. All qualified applicants, including individuals with disabilities and protected veterans, are encouraged to apply. UT-Battelle is an E-Verify employer.