Science Gateway

The Science Gateway website consists of two core components, the frontend UI and the backend API. The backend interacts with external services (amongst other things), and provides endpoints in the form of a REST API. The frontend interacts with the backend endpoints via the REST API, and provides a web-based Graphical User Interface to the end-user.

The Science Gateway website can be deployed on a local machine for development or in a production environment.

Deployment prerequisites

IAM Configuration

For this service, the scopes required are:

  • `openid`

  • `profile`

For the Grant types:

  • Ensure authorization_code and refresh_token grants are checked.

Follow the next instructions to create and manage the IAM Client Configuration, including these scopes and grant types. One saved, get the Client ID and Client Secret, to include them into your configuration file for .env or docker-compose.yml.

Docker

The frontend and backend components of the Science Gateway are containerised using docker. At the time of writing, there is one container for the frontend and two for the backend (one for the core API and another for the Gateway database). The backend containers are grouped as a service to conveniently specify shared resources.

To install docker refer to the official documentation. Follow the instructions that applies to the OS on your local machine.

Note

This guide will make use of Docker CLI commands. Feel free to use the desktop UI application if it is more convenient however, instructions are not provided for that method in this guide.

Cloning the source repositories

Repositories containing the Science Gateway source code are hosted on Gitlab.

Clone the backend repository:

$ git clone https://gitlab.com/ska-telescope/src/src-api/ska-src-api-gateway

Clone the frontend repository:

$ git clone https://gitlab.com/ska-telescope/src/src-ui/ska-src-ui-gateway

Deploying the gateway on a local machine for development

The Science Gateway can be deployed on a local machine. This provides a quick and flexible way to make changes (e.g. to code, UI elements) and test the results. This is especially useful for SRCNet developers working on the Science Gateway.

Both the frontend and backend can be deployed on a local machine, and both are required for the Science Gateway to work as designed.

Prerequisites

SKA IAM Account

A SKA IAM user account with the necessary permissions for each service used by the backend API is required. The frontend obtains a code following a succesful IAM login. The frontend uses the code to obtain and exchange access tokens via the backend API. The tokens are used to access services external to the Science Gateway via the backend API.

Register for an IAM account on this website:

https://ska-iam.stfc.ac.uk/login

The IAM account should belong to the following permissions groups to use all of the Science Gateway features:

  • services/gateway-backend-api

  • services/site-capabilities-api

  • services/data-management-api

Deploying the backend API locally

Ensure that port 8080 is not in use on your local machine. This port is used to host the backend API.

In your terminal, navigate to the directory of the backend repository.

$ cd ./path/to/ska-src-api-gateway

Edit the docker-compose.yml contents to look as follows for local deployments:

version: "3.5"
services:
  db:
    container_name: gateway-backend-db
    image: mysql:8.0
    cap_add:
      - SYS_NICE
    restart: always
    environment:
      - MYSQL_DATABASE=gatewaysettings
      - MYSQL_ROOT_PASSWORD=V1Lv5FxUde
    ports:
      - '3306:3306'
    volumes:
      - db:/var/lib/mysql
      - ./db/init/init.sql:/docker-entrypoint-initdb.d/init.sql
  core:
    container_name: gateway-backend-core
    image: gateway-backend-core:latest
    build:
      context: .
      dockerfile: Dockerfile
    depends_on:
      - db
    environment:
      DISABLE_AUTHENTICATION: "no"
      API_ROOT_PATH:
      API_SCHEME: http
      IAM_CLIENT_CONF_URL: https://ska-iam.stfc.ac.uk/.well-known/openid-configuration
      API_IAM_CLIENT_ID: some-uuid
      API_IAM_CLIENT_SECRET:
      API_IAM_CLIENT_SCOPES: openid profile
      API_IAM_CLIENT_AUDIENCE: gateway-backend-api
      PERMISSIONS_API_URL: https://permissions.srcdev.skao.int/api/v1
      PERMISSIONS_SERVICE_NAME: gateway-backend-api
      PERMISSIONS_SERVICE_VERSION: 1
      DB_HOST: db
      DB_PORT: 3306
      DB_USER: root
      DB_PASSWORD: V1Lv5FxUde
      DB_NAME: gatewaysettings
    ports:
      - 8080:8080
volumes:
  db:
    driver: local

Add values for the environment variables API_IAM_CLIENT_ID and API_IAM_CLIENT_SECRET to the docker-compose.yml file based on the IAM client created earlier.

Set the value of the environment variable RUN_MODE in docker-compose.yml to DEV.

Create a file called docker-compose.local.yml with the following contents:

version: "3.5"
services:
  core:
    extends:
      file: docker-compose.yml
      service: core
    environment:
      DISABLE_AUTHENTICATION: "no"
    volumes:
      - $../:/opt/ska-src-api-gateway

Deploy the docker service:

$ docker-compose up -d

Access the database container and create the required tables:

$ docker exec -ti gateway-backend-db /bin/bash
$ mysql -u root -p  # will prompt for a password, use V1Lv5FxUde
$ mysql> USE gatewaysettings
$ mysql> CREATE TABLE Users ( ID int NOT NULL AUTO_INCREMENT, IAM_ID varchar(255), PRIMARY KEY (ID) );
$ mysql> CREATE TABLE Preferences ( ID int NOT NULL AUTO_INCREMENT, UserID int, DarkMode tinyint, Language varchar(10), PRIMARY KEY (ID) );
$ mysql> CREATE TABLE Services ( ID int NOT NULL AUTO_INCREMENT, Service_ID varchar(255), PRIMARY KEY (ID) );
$ mysql> CREATE TABLE ServiceTokens ( ID int NOT NULL AUTO_INCREMENT, UserID int, ServiceID int, Username varchar(255), Token varchar(255), PRIMARY KEY (ID) );
$ mysql> CREATE TABLE DataManagementJobs ( ID int NOT NULL AUTO_INCREMENT, UserID int, JobID varchar(32), ToStorageAreaUUID varchar(255), Status varchar(255), PRIMARY KEY (ID) );

Check that the tables were created:

$ mysql> SHOW TABLES;

Type exit and enter twice to leave MySQL and the container.

Testing the local backend deployment

  • Check that the service containers are running in docker:

    $ docker container ls

    You should see output similar to the following:

    CONTAINER ID   IMAGE                         COMMAND                  CREATED      STATUS         PORTS                               NAMES
bc0189ee3037   mysql:8.0                     "docker-entrypoint.s…"   8 days ago   Up 2 days      0.0.0.0:3306->3306/tcp, 33060/tcp   gateway-backend-db
b54d97c4723b   gateway-backend-core:latest   "/bin/bash etc/docke…"   8 days ago   Up 9 seconds   0.0.0.0:8080->8080/tcp              gateway-backend-core

  • Test the backend API by using its ping endpoint by navigating to http://localhost:8080/v1/ping in your web browser. Your browser should display output similar to the following:

    {"status":"UP","version":"0.1.1"}

  • Test the backend API by navigating to http://localhost:8080/v1/www/docs/oper in your browser to view the swagger frontend.

Deploying the frontend web UI locally

Ensure that port 3000 is not in use on your local machine. It is used to host the frontend website.

In your terminal, navigate to the directory of the backend repository:

$ cd ./path/to/ska-src-ui-gateway

Edit the Dockerfile so that its contents looks as follows:

FROM node:lts
RUN mkdir -p /app
WORKDIR /app
COPY . .
RUN npm install
EXPOSE 3000
CMD ["npm", "run", "start"]

This ensures that the frontend react app will run in development mode, which means it will access the backend API via http://localhost:8080.

Build the frontend docker image:

$ docker build . --tag 'ska-src-ui-gateway'

Create a container using the image and run it in the background:

$ docker run --name 'gateway-ui' -p 3000:3000 -ti -d ska-src-ui-gateway

Testing the local frontend deployment

Navigate to http://localhost:3000/ in your web browser. Login using your SKA IAM Account created earlier.

Deploying the gateway in a production environment

The current deployment of the gateway (both frontend and backend) is spread over two nodes. The deployment of new versions is handled through CI/CD. Here, we will describe the relevant aspects of the setup of the nodes.

Deployment systems

Both systems (to which we will respectively refer as test and production systems) run on the STFC cloud. The production system (for which the CI/CD system is used to push the main branch of the corresponding repositories) is called srcnet-esap, and is accessible through the URL https://gateway.srcdev.skao.int. The test deployment, to which any branch can be deployed manually, runs on the srcnet-esap-test node, accessible through https://gateway.srcdev-test.skao.int.

The configuration of the proxies that forward the traffic from the URL to the node is such that any traffic coming in on port 443 on the externally-facing URL is sent to port 80 on the host.

Prerequisites

Traefik

To route traffic within the host, we use Traefik. We detail the installation here.

To access the node, an account on the STFC system is required. Access via ssh is granted through the bastion host. We will assume you have been granted access, know how to access the system, and are logged in to a terminal on the host with root privileges.

For deployments, we use a user called gitlab_deploy. So to manage the Traefik setup, change to that user, and go to the directory holding the docker-compose file with the configuration:

$ sudo su gitlab_deploy
$ cd ~/deploy-traefik

In this directory, you will find several artifacts. The first one being a docker-compose file (a nice feature of Traefik is that it can run inside a docker container, while proxying other docker containers. You will see in the docker-compose file that this is achieved by linking in the docker socket in the docker container). The docker-compose.yml file looks like:

services:
  traefik:
    image: traefik:2.10 # or any later version
    ports:
      - "80:80" # http
      - "8081:8081" # Traefik Dashboard
      - "3001:3001" # compute API
    volumes:
      - $PWD/traefik/local.yml:/etc/traefik/traefik.yml
      - /var/run/docker.sock:/var/run/docker.sock # see Security considerations below
    networks:
      - esap_network
    restart: always
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=PathPrefix(`/dashboard`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entryPoints=dashboard"

The ports section tells us that Traefik will proxy traffic coming in through port 80 (for both the gateway frontend and backend). Port 8081 exposes the Traefik dashboard, which is a nice debugging tool. However, the dashboard is only accessible from within the internal network of the node (so some proxying is needed to look at it). The last port exposed is 3001 which is used for the compute API demonstrator (which for practical reasons also runs in a container on this host, but is mapped to a different URL than the gateway).

Apart from the aforementioned Docker socket object, the Traefik configuration file is linked as a volume (NB In principle the configuration could also have been fully done using docker labels rather than a yml file, keeping everything in one place, this is really a matter of taste of the deployer). We will come back to the .yml file later.

The container is coupled to a network, currently called esap_network for historical reasons. At the bottom you see that the network is of type external, meaning that it has to be manually created and is not managed by docker-compose:

$ docker network create esap_network

The labels block define some of the configuration. In this configuration, Traefik is enable, and the dashboard (which connects to an entrypoint called dashboard).

Finally, the Traefik .yml configuration looks like this:

entryPoints:
  web:
    address: :80
  dashboard:
    address: :8081
  compute-api:
    address: :3001
log:
  level: DEBUG
providers:
  docker:
    exposedByDefault: false   # explicitly make container available using labels
    network: esap_network     # should match the (external) network created earlier
api:
  dashboard: true

The first block defines entrypoints which essentially are the ports to which the Traefik application is listening (unsurprisingly those are the same as the ports opened in the docker-compose file). Each docker container connecting to Traefik will need to configure the entrypoint it will use so that Traefik knows from what incoming port on the host to forward the requests. The log level is set to debug. There is only one provider (source of applications): the docker provider. The first exposedByDefault setting makes it required for any container to explicitly announce it wants to be routed by Traefik through labels. The network setting tells Traefik what the internal Docker network is that needs to be connected (unsurprisingly this matches the name of the network in the docker-compose file). The final setting tells Traefik to run the API so that it can be connected to.

Deploying the backend API

In your terminal, navigate to the directory of the backend repository.

$ cd ./path/to/ska-src-api-gateway

Add values for the environment variables API_IAM_CLIENT_ID and API_IAM_CLIENT_SECRET to the docker-compose.yml file based on the IAM client created earlier.

Check that the value of the environment variable RUN_MODE in docker-compose.yml is set to PROD.

Deploy the docker service:

$ docker-compose up -d

Access the database container and create the required tables:

$ docker exec -ti gateway-backend-db /bin/bash
$ mysql -u root -p  # will prompt for a password, use V1Lv5FxUde
$ mysql> USE gatewaysettings
$ mysql> CREATE TABLE Users ( ID int NOT NULL AUTO_INCREMENT, IAM_ID varchar(255), PRIMARY KEY (ID) );
$ mysql> CREATE TABLE Preferences ( ID int NOT NULL AUTO_INCREMENT, UserID int, DarkMode tinyint, Language varchar(10), PRIMARY KEY (ID) );
$ mysql> CREATE TABLE Services ( ID int NOT NULL AUTO_INCREMENT, Service_ID varchar(255), PRIMARY KEY (ID) );
$ mysql> CREATE TABLE ServiceTokens ( ID int NOT NULL AUTO_INCREMENT, UserID int, ServiceID int, Username varchar(255), Token varchar(255), PRIMARY KEY (ID) );
$ mysql> CREATE TABLE DataManagementJobs ( ID int NOT NULL AUTO_INCREMENT, UserID int, JobID varchar(32), ToStorageAreaUUID varchar(255), Status varchar(255), PRIMARY KEY (ID) );

Check that the tables were created:

$ mysql> SHOW TABLES;

Type exit twice to leave MySQL and the container.

Deploying the frontend web UI

In your terminal, navigate to the directory of the backend repository:

$ cd ./path/to/ska-src-ui-gateway

Build the frontend docker image:

$ docker build . --tag 'ska-src-ui-gateway'

Create a docker-compose.yml file with the following contents:

services:
  esap_api:
    container_name: tangerinenet
    image: ska-src-ui-gateway
    expose:
      - "3000"
    networks:
      - esap_network
    restart: always
    labels:
      # Enables traefik to connect to the API
      - "traefik.enable=true"
      - "traefik.http.routers.tangerinenet.entryPoints=web"
      - "traefik.http.routers.tangerinenet.rule=PathPrefix(`/`)"
      - "traefik.http.routers.tangerinenet.service=tangerinenet"
      - "traefik.http.services.tangerinenet.loadbalancer.server.port=3000"
networks:
  esap_network:
    external: true

Deploy the docker service:

$ docker-compose up -d