SRCNet Operations Documentation

The SRCNet Software Stack encompasses the mandatory and optional software for the nodes within the SRCNet network. This includes data and metadata management services, visualization services, monitoring tools, and scientific platforms, among others. All components are aligned with the implementation plan designed for SRCNet v0.1.

The goal of this repository is to provide various installation methods and versions to facilitate reproducible installations across SRCNet nodes. By offering detailed instructions and multiple installation options, we aim to streamline the deployment process and ensure consistency and reliability across the SRC Network. Where feasible, manual, script-based automated installation and GitOps-based deployment methods will be provided, catering to different deployment preferences and requirements.

Standards

Standards, conventions, and guidelines ensure consistency, enhance communication, and align practices with policy requirements, fostering efficiency and clarity. A set of standards that are expected to be adopted and maintained for v0.1 are given in this section.

National SRC Names

See National SRC Naming Standards for the policy on the names of referring to National SRCs. Concisely, SRCNet names must only consist of:

AUSSRC, CANSRC, CNSRC, FRSRC, DESRC, INSRC, ITSRC, JPSRC, NLSRC, PTSRC, ZASRC, KRSRC, ESPSRC, SWESRC, CHSRC, UKSRC, SKAO

RSE Naming

See Rucio Storage Endpoint (RSE) Names and Attributes for the policy on the names of referring to RSEs. Concisely, RSEs must be named according to:

{src-name}-{site-name}-{storage-class}[-{storage-label}]

Global Services

The Global SRC Services will operate centrally, providing essential functionalities across the entire SRCNet network. These services include:

Service & Type	Reqs.	Manual	Container	Helm	GitOps
Rucio	IAM, Databases, Monitoring	❎	❎	Link	❎
Central Monitoring	IAM, CHSRC	❎	❎	Deployment	Deployment

Local SRC Services

Local SRCNet Services are those installed on the SRCNet nodes, classified into mandatory and optional categories, and also providing several methods to install them:

Service & Type	Reqs.	Manual	Container	Helm	GitOps
Rucio Storage Element Mandatory		Manual (WebDav), Manual (xrootd)	Docker (WebDav), Docker (xrootd)	Helm (WebDav), Helm (xrootd)	❎
JupyterLab Services Mandatory	IAM	❎	❎	Helm	❎
SODA Services Mandatory	❎	❎	Docker Compose	Helm	❎
Visualisation Services Mandatory	TBC	TBC	TBC	TBC	TBC
Gatekeeper Service Mandatory		❎	Docker Compose	Helm	❎
Monitoring tools Mandatory	❎	Manual	❎	❎	❎
Science Platform Optional	IAM	Manual	❎	❎	Argo CI/CD
Science Gateway Optional	IAM	Manual	Docker (Local), Docker (Production)	❎	❎

Local Mandatory SRC Services

Mandatory Services:

These are the critical components required for the proper functioning of each SRCNet node:

Common Data-related Services: Tools for discovering and accessing data.
Local Data Parsing Services: Services allowing remote invocation for data parsing.
Visualization Services: Containerized visualization tools capable of running on SRCNet nodes.
Interactive Analysis Interface: Notebooks and tools for interactive data analysis.
Monitoring Services: Tools for monitoring the performance and health of the SRCNode.
Interfaces and Integration with Global Services: Essential interfaces to ensure seamless integration with global SRCNet services.

See Mandatory SRCNet Services.

Local Optional SRC Services

These are additional services that can be installed on each node to enhance the basic functionality provided by the mandatory services.

Science Platform Gateway (Webserver Interface).
Science Platform: CANFAR.
User Areas Based on VOSpace Cavern.
Cloud-Based Software Deployment: Platform Azimuth.
HPC and Batch Compute: SLURM Clusters.
Software Distribution CVMFS Client.
Additional Services: Other services can be deployed and registered into the SRCNet Site Capabilities Service.

See Optional SRCNet Services.

Dependent SRCNet Services

Dependent services are components of the SRCNet that depend on other services to work properly. In the context of SRCNet software, services and tools, they are part of a broader ecosystem that needs to be deployed to provide certain functionalities.

IAM Client Configuration

Some of the SRCNet services use SKAO-IAM for Authentication and Authorization. For this, it is ecessary that an SRC user (the operator) can create and manage each IAM client for each of the services that require this SKAO-IAM integration for Authentication and Authorization. The details of the SKAO-IAM account application and how to proceed with the creation of the clients are available in IAM Client Configuration.

Storage Elements implementations

A Rucio Storage Element can have different implementations aligned with the SKAO Rucio DataLake.

See Storage Element (SE) deployment of implementations.

GitOps

Some of the software and services identified for the SRCNet on local nodes may require the installation of a platform for local GitOps support that will enable the deployment of other services. GitOps is an operational framework that uses Git repositories as the sole source of truth for declarative infrastructure and applications. This alternative leverages Git version control, which ensures auditing, automated deployments and consistency across environments. This improves collaboration, enables continuous delivery of services and automation of the deployment process.

See GitOps deployment.