perfSONAR

Note

perfSONAR is a collection of open source software for performing and sharing end-to-end network measurements. It consists of multiple tools brought together like a Swiss Army Knife.

perfSONAR includes numerous utilities responsible for carrying out the actual network measurements and form the foundational layer of perfSONAR. In general, you will not invoke these tools directly but instead use the pscheduler command from the scheduling layer to execute them.

The default tools that come with perfSONAR include among others:

  • owamp - A set of tools primarily used for measuring packet loss and one-way delay.

It includes the command owping for single short-lived tests and the powstream command for long-running background tests.

  • twamp - A tool primarily used for measuring packet loss and two-way delay.

It has increased accuracy over tools like ping without the same clock synchronization requirements as OWAMP. The client tool is named twping and can be used to run against

  • iperf3 - A rewrite of the classic iperf tool used to measure network throughput and associated metrics.

  • iperf2 - Also known as just iperf, a common tool used to measure network throughput that has been around for many years.

  • nuttcp - Another throughput tool with some useful options not found in other tools.

  • traceroute - The classic packet trace tool used in identifying network paths

  • tracepath - Another path trace tool that also measures path MTU

  • ping - The classic utility for determining reachability, round-trip time (RTT) and basic packet loss.

Deployment

Pre-Deployment Considerations

perfSONAR is primarily aiming to test the network between SRC sites. For this reason it is recommended to not place the host behind a proxy which could make the results more difficult to interpret. Moreover it makes installation and configuration of a perfSONAR host more difficult. If it is possible to still place the perfSONAR host close to the DTN this is advised.

perfSONAR has several bundles to choose from, ranging from tools (which is a basic collection of the monitoring tools listed above) to the toolkit which also includes the dashboard and archive modules.

It is recommended to install the testpoint bundle (or container) as this is the most lightweight bundle with scheduling functionality.

Automated script Debian

perfSONAR provides a script that will automatically perform the first two steps in Installation (Manual). You can install your choice of bundle with one of the commands below:

perfSONAR Toolkit:

curl -s https://raw.githubusercontent.com/perfsonar/project/master/install-perfsonar | sh -s - toolkit

Manual

Configure apt sources:

curl -o /etc/apt/sources.list.d/perfsonar-release.list https://downloads.perfsonar.net/debian/perfsonar-release.list
curl -s -o /etc/apt/trusted.gpg.d/perfsonar-release.gpg.asc https://downloads.perfsonar.net/debian/perfsonar-release.gpg.key

Ubuntu only

Additionally, if you’re running a stripped-down Ubuntu installation, you might need to enable the universe repository. This is done with the following command:

add-apt-repository universe

Then refresh the packages list so APT knows about the perfSONAR packages:

apt update

Installation:

env OPENSEARCH_INITIAL_ADMIN_PASSWORD=perfSONAR123! apt install perfsonar-toolkit

During the installation process, you’ll be asked to choose a password for the pscheduler database.

You can start all the services by rebooting the host since all are configured to run by default. In order to check services status issue the following commands:

service pscheduler-scheduler status
service pscheduler-runner status
service pscheduler-archiver status
service pscheduler-ticker status
service owamp-server status
service perfsonar-lsregistrationdaemon status

If they are not running you may start them with appropriate service commands as a root user. For example:

service pscheduler-scheduler start
service pscheduler-runner start
service pscheduler-archiver start
service pscheduler-ticker start
service owamp-server start
service perfsonar-lsregistrationdaemon start

Note that you may have to wait a few hours for NTP to synchronize your clock before (re)starting owamp-server.

After installing the perfsonar-toolkit bundle, you can refer to the general perfSONAR configuration from https://docs.perfsonar.net/install_config_first_time.html

RHEL install

Alma 9 or Rocky 9 x86_64 are supported

First you’ll want to install the EPEL RPM

dnf install epel-release

Then enable CRB repository

dnf config-manager --set-enabled crb

Install the perfSONAR EL9 repo RPM

dnf install http://software.internet2.edu/rpms/el9/x86_64/latest/packages/perfsonar-repo-0.11-1.noarch.rpm

Refresh the DNS cache

dnf clean all

Then pick a bundle (testpoint recommended, or toolkit for advanced functionality)

dnf install perfsonar-testpoint

Then check the services are running

systemctl status pscheduler-scheduler
systemctl status pscheduler-runner
systemctl status pscheduler-archiver
systemctl status pscheduler-ticker
systemctl status psconfig-pscheduler-agent
systemctl status owamp-server

Docker

Get the latest Docker image for perfSONAR:

docker pull perfsonar/testpoint:systemd

This will download the latest built image of the perfsonar testpoint bundle. It includes a base Ubuntu 22.04 install and the perfsonar-testpoint packages. Once the image is downloaded and extracted, start up the container in the background by doing:

docker run -td --name perfsonar-testpoint --net=host --tmpfs /run --tmpfs /run/lock --tmpfs /tmp -v /sys/fs/cgroup:/sys/fs/cgroup:rw --cgroupns host perfsonar/testpoint:systemd

Now you can connect to the running Docker image:

docker exec -it <container ID from above> bash

At this point you will be at a bash prompt inside of the container, and may start running tests:

pscheduler troubleshoot
pscheduler task throughput --dest hostname

Helm

A Helm chart exists to deploy perfSONAR on Kubernetes. Beware that this is experimental and some latency tests still encounter issues.

https://gitlab.com/ska-telescope/src/src-ms/ska-src-ms-perfsonar-deployment

Ports required

As perfSONAR contains a variety of tools it requires a range of ports. These are detailed below:

Ports

Tool

TCP ports

UDP ports

required for v0.1

owamp (control)

861

✓

owamp (test)

8760-9960

✓

twamp (control)

862

twamp (test)

18760-19960

pscheduler

443

✓

traceroute

33434-33634

simplestream

5890-5900

nuttcp

5000,5101

iperf3

5201

✓

iperf2

5001

ntp

123

It is also required that ICMP be enabled

perfSONAR tests and troubleshooting

Aside from checking if the services are running, perfSONAR also has its own troubleshooter. This can also be used on external hosts:

pscheduler troubleshoot
pscheduler troubleshoot --dest hostname

There are also tools perfSONAR can use to test for bandwidth, path, and latency using “throughput” (default iperf3) , rtt (default ping), and “trace” (default traceroute). These tools have their own defaults , but these can be modified:

pscheduler task throughput --dest hostname
pscheduler task trace --dest hostname
pscheduler task rtt --dest hostname

By specifying a source node, you can also run tests between other hosts and direct tests to your own host for troubleshooting

pscheduler task throughput --source hostname --dest hostname

This should help determine that your host is contactable. You can also schedule repeatable tests. To interrogate the scheduled tests “monitor” will show an updating schedule. But “schedule” will help seeing past and upcoming tests For example, the past 2 hours of the schedule can be shown by adding -PT2H

pscheduler monitor
pscheduler schedule -PT2H

Integrating into the mesh

The SKA has a central dashboard for the perfSONAR testing data. This is viewable here:

The central perfSONAR host also uses a configuration file to schedule tests between all hosts in the mesh:

There is also, currently on confluence, a brief overview of the schedule of tests and current participants in the mesh:

This is liable to being updated as feedback from sites permit.

To be included in the mesh, first you should contact the perfSONAR support in purple team (so the config can be updated) Then to register your node with the mesh config so your scheduler joins the test scheduling, perform the following:

psconfig remote add "https://perfsonar01.jc.rl.ac.uk/psconfig/psconfig-test.json"

When the mesh config has been updated, you should then see the tests populate the schedule.

Mesh monitoring

To reiterate above, the mesh has a grafana dashboard. The most likely place to see information you will want is the custom grafana dashboard. This shows a grid view of the mesh members, and each square will take you to a detailed view of those sites and the information between them. dashboards>custom_grafana_dashboard or this link: