.. _perfsonar: perfSONAR ================ .. note:: - Support: https://skao.slack.com/archives/C03B3650U0Z perfSONAR is a collection of open source software for performing and sharing end-to-end network measurements. It consists of multiple tools brought together like a Swiss Army Knife. perfSONAR includes numerous utilities responsible for carrying out the actual network measurements and form the foundational layer of perfSONAR. In general, you will not invoke these tools directly but instead use the pscheduler command from the scheduling layer to execute them. The default tools that come with perfSONAR include among others: - owamp - A set of tools primarily used for measuring packet loss and one-way delay. It includes the command owping for single short-lived tests and the powstream command for long-running background tests. - twamp - A tool primarily used for measuring packet loss and two-way delay. It has increased accuracy over tools like ping without the same clock synchronization requirements as OWAMP. The client tool is named twping and can be used to run against - iperf3 - A rewrite of the classic iperf tool used to measure network throughput and associated metrics. - iperf2 - Also known as just iperf, a common tool used to measure network throughput that has been around for many years. - nuttcp - Another throughput tool with some useful options not found in other tools. - traceroute - The classic packet trace tool used in identifying network paths - tracepath - Another path trace tool that also measures path MTU - ping - The classic utility for determining reachability, round-trip time (RTT) and basic packet loss. .. _manual-monitoring-tools: Useful Links ~~~~~~~~~~~~ - MESH dashboard: https://perfsonar01.jc.rl.ac.uk/grafana/d/c81bcc51-9380-5365-a398-b2e0fc9df648/custom-grafana-dashboard?orgId=1 - External documentation: https://docs.perfsonar.net/index.html - perfSONAR node lookup service: https://stats.perfsonar.net Deployment ~~~~~~~~~~ Pre-Deployment Considerations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ perfSONAR is primarily aiming to test the network between SRC sites. For this reason it is recommended to not place the host behind a proxy which could make the results more difficult to interpret. Moreover it makes installation and configuration of a perfSONAR host more difficult. If it is possible to still place the perfSONAR host close to the DTN this is advised. perfSONAR has several bundles to choose from, ranging from tools (which is a basic collection of the monitoring tools listed above) to the toolkit which also includes the dashboard and archive modules. It is recommended to install the testpoint bundle (or container) as this is the most lightweight bundle with scheduling functionality. Automated script Debian ^^^^^^^^^^^^^^^^^^^^^^^ perfSONAR provides a script that will automatically perform the first two steps in Installation (Manual). You can install your choice of bundle with one of the commands below: perfSONAR Toolkit: .. code-block:: bash curl -s https://raw.githubusercontent.com/perfsonar/project/master/install-perfsonar | sh -s - toolkit Manual ^^^^^^ Configure `apt` sources: .. code-block:: bash curl -o /etc/apt/sources.list.d/perfsonar-release.list https://downloads.perfsonar.net/debian/perfsonar-release.list curl -s -o /etc/apt/trusted.gpg.d/perfsonar-release.gpg.asc https://downloads.perfsonar.net/debian/perfsonar-release.gpg.key .. rubric:: Ubuntu only Additionally, if you’re running a stripped-down Ubuntu installation, you might need to enable the universe repository. This is done with the following command: .. code-block:: bash add-apt-repository universe Then refresh the packages list so APT knows about the perfSONAR packages: .. code-block:: bash apt update Installation: .. code-block:: bash env OPENSEARCH_INITIAL_ADMIN_PASSWORD=perfSONAR123! apt install perfsonar-toolkit During the installation process, you’ll be asked to choose a password for the pscheduler database. You can start all the services by rebooting the host since all are configured to run by default. In order to check services status issue the following commands: .. code-block:: bash service pscheduler-scheduler status service pscheduler-runner status service pscheduler-archiver status service pscheduler-ticker status service owamp-server status service perfsonar-lsregistrationdaemon status If they are not running you may start them with appropriate service commands as a root user. For example: .. code-block:: bash service pscheduler-scheduler start service pscheduler-runner start service pscheduler-archiver start service pscheduler-ticker start service owamp-server start service perfsonar-lsregistrationdaemon start Note that you may have to wait a few hours for NTP to synchronize your clock before (re)starting owamp-server. After installing the perfsonar-toolkit bundle, you can refer to the general perfSONAR configuration from https://docs.perfsonar.net/install_config_first_time.html RHEL install ^^^^^^^^^^^^ Alma 9 or Rocky 9 x86_64 are supported First you'll want to install the EPEL RPM .. code-block:: bash dnf install epel-release Then enable CRB repository .. code-block:: bash dnf config-manager --set-enabled crb Install the perfSONAR EL9 repo RPM .. code-block:: bash dnf install http://software.internet2.edu/rpms/el9/x86_64/latest/packages/perfsonar-repo-0.11-1.noarch.rpm Refresh the DNS cache .. code-block:: bash dnf clean all Then pick a bundle (testpoint recommended, or toolkit for advanced functionality) .. code-block:: bash dnf install perfsonar-testpoint Then check the services are running .. code-block:: bash systemctl status pscheduler-scheduler systemctl status pscheduler-runner systemctl status pscheduler-archiver systemctl status pscheduler-ticker systemctl status psconfig-pscheduler-agent systemctl status owamp-server Docker ^^^^^^ Get the latest Docker image for perfSONAR: .. code-block:: bash docker pull perfsonar/testpoint:systemd This will download the latest built image of the perfsonar testpoint bundle. It includes a base Ubuntu 22.04 install and the perfsonar-testpoint packages. Once the image is downloaded and extracted, start up the container in the background by doing: .. code-block:: bash docker run -td --name perfsonar-testpoint --net=host --tmpfs /run --tmpfs /run/lock --tmpfs /tmp -v /sys/fs/cgroup:/sys/fs/cgroup:rw --cgroupns host perfsonar/testpoint:systemd Now you can connect to the running Docker image: .. code-block:: bash docker exec -it bash At this point you will be at a bash prompt inside of the container, and may start running tests: .. code-block:: bash pscheduler troubleshoot pscheduler task throughput --dest hostname Helm ^^^^ A Helm chart exists to deploy perfSONAR on Kubernetes. Beware that this is experimental and some latency tests still encounter issues. https://gitlab.com/ska-telescope/src/src-ms/ska-src-ms-perfsonar-deployment Ports required ~~~~~~~~~~~~~~ As perfSONAR contains a variety of tools it requires a range of ports. These are detailed below: .. list-table:: Ports :widths: 25 25 25 25 :header-rows: 1 * - Tool - TCP ports - UDP ports - required for v0.1 * - owamp (control) - 861 - - ✓ * - owamp (test) - - 8760-9960 - ✓ * - twamp (control) - 862 - - * - twamp (test) - - 18760-19960 - * - pscheduler - 443 - - ✓ * - traceroute - - 33434-33634 - * - simplestream - 5890-5900 - - * - nuttcp - 5000,5101 - - * - iperf3 - 5201 - - ✓ * - iperf2 - 5001 - - * - ntp - - 123 - It is also required that ICMP be enabled perfSONAR tests and troubleshooting ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Aside from checking if the services are running, perfSONAR also has its own troubleshooter. This can also be used on external hosts: .. code-block:: bash pscheduler troubleshoot pscheduler troubleshoot --dest hostname There are also tools perfSONAR can use to test for bandwidth, path, and latency using "throughput" (default iperf3) , rtt (default ping), and "trace" (default traceroute). These tools have their own defaults , but these can be modified: .. code-block:: bash pscheduler task throughput --dest hostname pscheduler task trace --dest hostname pscheduler task rtt --dest hostname By specifying a source node, you can also run tests between other hosts and direct tests to your own host for troubleshooting .. code-block:: bash pscheduler task throughput --source hostname --dest hostname This should help determine that your host is contactable. You can also schedule repeatable tests. To interrogate the scheduled tests "monitor" will show an updating schedule. But "schedule" will help seeing past and upcoming tests For example, the past 2 hours of the schedule can be shown by adding -PT2H .. code-block:: bash pscheduler monitor pscheduler schedule -PT2H Integrating into the mesh ~~~~~~~~~~~~~~~~~~~~~~~~~ The SKA has a central dashboard for the perfSONAR testing data. This is viewable here: - https://perfsonar01.jc.rl.ac.uk/grafana The central perfSONAR host also uses a configuration file to schedule tests between all hosts in the mesh: - https://perfsonar01.jc.rl.ac.uk/psconfig/psconfig-test.json There is also, currently on confluence, a brief overview of the schedule of tests and current participants in the mesh: - https://confluence.skatelescope.org/display/SRCSC/perfSONAR+test+schedule This is liable to being updated as feedback from sites permit. To be included in the mesh, first you should contact the perfSONAR support in purple team (so the config can be updated) Then to register your node with the mesh config so your scheduler joins the test scheduling, perform the following: .. code-block:: bash psconfig remote add "https://perfsonar01.jc.rl.ac.uk/psconfig/psconfig-test.json" When the mesh config has been updated, you should then see the tests populate the schedule. Mesh monitoring ~~~~~~~~~~~~~~~ To reiterate above, the mesh has a grafana dashboard. The most likely place to see information you will want is the custom grafana dashboard. This shows a grid view of the mesh members, and each square will take you to a detailed view of those sites and the information between them. dashboards>custom_grafana_dashboard or this link: - https://perfsonar01.jc.rl.ac.uk/grafana/d/c81bcc51-9380-5365-a398-b2e0fc9df648/custom-grafana-dashboard?orgId=1