.. _science-gateway:

Science Gateway
===============

The Science Gateway website consists of two core components, the frontend UI and the
backend API. The backend interacts with external services (amongst other things),
and provides endpoints in the form of a REST API. The frontend interacts with the
backend endpoints via the REST API, and provides a web-based Graphical User Interface
to the end-user.

The Science Gateway website can be deployed on a local machine for development or in a
production environment.

Deployment prerequisites
------------------------

.. _iam-science-gateway:

IAM Configuration
^^^^^^^^^^^^^^^^^

For this service, the scopes required are:
 
- ```openid``` 
- ```profile```

For the *Grant types*: 

- Ensure `authorization_code` and `refresh_token` grants are checked.

Follow the next instructions to create and manage the :doc:`/services/dependent/iam-client/iam-client`, including these scopes and grant types. 
One saved, get the ``Client ID`` and ``Client Secret``, to include them into your configuration file for `.env` or `docker-compose.yml`.

Docker
^^^^^^

The frontend and backend components of the Science Gateway are containerised using
docker. At the time of writing, there is one container for the frontend and two for the
backend (one for the core API and another for the Gateway database). The backend
containers are grouped as a service to conveniently specify shared resources.

To install docker refer to the
`official documentation <https://docs.docker.com/desktop/>`_.
Follow the instructions that applies to the OS on your local machine.

.. note::
    This guide will make use of Docker CLI commands. Feel free to use the desktop
    UI application if it is more convenient however, instructions are not provided for
    that method in this guide.

Cloning the source repositories
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Repositories containing the Science Gateway source code are hosted on Gitlab.

Clone the backend repository:

.. code-block:: shell

    $ git clone https://gitlab.com/ska-telescope/src/src-api/ska-src-api-gateway

Clone the frontend repository:

.. code-block:: shell

    $ git clone https://gitlab.com/ska-telescope/src/src-ui/ska-src-ui-gateway

.. _science-gateway-local-docker:

Deploying the gateway on a local machine for development
--------------------------------------------------------

The Science Gateway can be deployed on a local machine. This provides a quick and
flexible way to make changes (e.g. to code, UI elements) and test the results. This is
especially useful for SRCNet developers working on the Science Gateway.

Both the frontend and backend can be deployed on a local machine, and both are required
for the Science Gateway to work as designed.

Prerequisites
^^^^^^^^^^^^^

.. _science-gateway-iam-account:

SKA IAM Account
"""""""""""""""

A SKA IAM user account with the necessary permissions for each service used by the
backend API is required. The frontend obtains a code following a succesful IAM login.
The frontend uses the code to obtain and exchange access tokens via the backend API.
The tokens are used to access services external to the Science Gateway via the backend
API.

Register for an IAM account on this website:

https://ska-iam.stfc.ac.uk/login

The IAM account should belong to the following permissions groups to use all of the
Science Gateway features:

- services/gateway-backend-api
- services/site-capabilities-api
- services/data-management-api

Deploying the backend API locally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Ensure that port ``8080`` is not in use on your local machine. This port is used to host
the backend API.

In your terminal, navigate to the directory of the backend repository.

.. code-block:: shell

    $ cd ./path/to/ska-src-api-gateway

Edit the ``docker-compose.yml`` contents to look as follows for local deployments:

.. code-block:: yaml

    version: "3.5"
    services:
      db:
        container_name: gateway-backend-db
        image: mysql:8.0
        cap_add:
          - SYS_NICE
        restart: always
        environment:
          - MYSQL_DATABASE=gatewaysettings
          - MYSQL_ROOT_PASSWORD=V1Lv5FxUde
        ports:
          - '3306:3306'
        volumes:
          - db:/var/lib/mysql
          - ./db/init/init.sql:/docker-entrypoint-initdb.d/init.sql
      core:
        container_name: gateway-backend-core
        image: gateway-backend-core:latest
        build:
          context: .
          dockerfile: Dockerfile
        depends_on:
          - db
        environment:
          DISABLE_AUTHENTICATION: "no"
          API_ROOT_PATH:
          API_SCHEME: http
          IAM_CLIENT_CONF_URL: https://ska-iam.stfc.ac.uk/.well-known/openid-configuration
          API_IAM_CLIENT_ID: some-uuid
          API_IAM_CLIENT_SECRET:
          API_IAM_CLIENT_SCOPES: openid profile
          API_IAM_CLIENT_AUDIENCE: gateway-backend-api
          PERMISSIONS_API_URL: https://permissions.srcdev.skao.int/api/v1
          PERMISSIONS_SERVICE_NAME: gateway-backend-api
          PERMISSIONS_SERVICE_VERSION: 1
          DB_HOST: db
          DB_PORT: 3306
          DB_USER: root
          DB_PASSWORD: V1Lv5FxUde
          DB_NAME: gatewaysettings
        ports:
          - 8080:8080
    volumes:
      db:
        driver: local

Add values for the environment variables ``API_IAM_CLIENT_ID`` and
``API_IAM_CLIENT_SECRET`` to the ``docker-compose.yml`` file based on the IAM client
:ref:`created earlier <iam-science-gateway>`.

Set the value of the environment variable ``RUN_MODE`` in ``docker-compose.yml`` to
``DEV``.

Create a file called ``docker-compose.local.yml`` with the following contents:

.. code-block:: yaml

    version: "3.5"
    services:
      core:
        extends:
          file: docker-compose.yml
          service: core
        environment:
          DISABLE_AUTHENTICATION: "no"
        volumes:
          - $../:/opt/ska-src-api-gateway

Deploy the docker service:

.. code-block:: shell

    $ docker-compose up -d

Access the database container and create the required tables:

.. code-block:: shell

    $ docker exec -ti gateway-backend-db /bin/bash
    $ mysql -u root -p  # will prompt for a password, use V1Lv5FxUde
    $ mysql> USE gatewaysettings
    $ mysql> CREATE TABLE Users ( ID int NOT NULL AUTO_INCREMENT, IAM_ID varchar(255), PRIMARY KEY (ID) );
    $ mysql> CREATE TABLE Preferences ( ID int NOT NULL AUTO_INCREMENT, UserID int, DarkMode tinyint, Language varchar(10), PRIMARY KEY (ID) );
    $ mysql> CREATE TABLE Services ( ID int NOT NULL AUTO_INCREMENT, Service_ID varchar(255), PRIMARY KEY (ID) );
    $ mysql> CREATE TABLE ServiceTokens ( ID int NOT NULL AUTO_INCREMENT, UserID int, ServiceID int, Username varchar(255), Token varchar(255), PRIMARY KEY (ID) );
    $ mysql> CREATE TABLE DataManagementJobs ( ID int NOT NULL AUTO_INCREMENT, UserID int, JobID varchar(32), ToStorageAreaUUID varchar(255), Status varchar(255), PRIMARY KEY (ID) );

Check that the tables were created:

.. code-block:: shell

    $ mysql> SHOW TABLES;

Type ``exit`` and enter twice to leave MySQL and the container.

Testing the local backend deployment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Check that the service containers are running in docker:

    .. code-block:: shell

        $ docker container ls

    You should see output similar to the following:

    .. code-block:: shell

        CONTAINER ID   IMAGE                         COMMAND                  CREATED      STATUS         PORTS                               NAMES
        bc0189ee3037   mysql:8.0                     "docker-entrypoint.s…"   8 days ago   Up 2 days      0.0.0.0:3306->3306/tcp, 33060/tcp   gateway-backend-db
        b54d97c4723b   gateway-backend-core:latest   "/bin/bash etc/docke…"   8 days ago   Up 9 seconds   0.0.0.0:8080->8080/tcp              gateway-backend-core

- Test the backend API by using its ``ping`` endpoint by navigating to
  http://localhost:8080/v1/ping in your web browser. Your browser should display output
  similar to the following:

    .. code-block:: json

        {"status":"UP","version":"0.1.1"}

- Test the backend API by navigating to http://localhost:8080/v1/www/docs/oper in your
  browser to view the swagger frontend.

Deploying the frontend web UI locally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Ensure that port ``3000`` is not in use on your local machine. It is used to host the
frontend website.

In your terminal, navigate to the directory of the backend repository:

.. code-block:: shell

    $ cd ./path/to/ska-src-ui-gateway

Edit the Dockerfile so that its contents looks as follows:

.. code-block:: docker

    FROM node:lts
    RUN mkdir -p /app
    WORKDIR /app
    COPY . .
    RUN npm install
    EXPOSE 3000
    CMD ["npm", "run", "start"]

This ensures that the frontend react app will run in `development` mode, which means
it will access the backend API via ``http://localhost:8080``.

Build the frontend docker image:

.. code-block:: shell

    $ docker build . --tag 'ska-src-ui-gateway'

Create a container using the image and run it in the background:

.. code-block:: shell

    $ docker run --name 'gateway-ui' -p 3000:3000 -ti -d ska-src-ui-gateway

Testing the local frontend deployment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Navigate to http://localhost:3000/ in your web browser. Login using your SKA IAM Account
:ref:`created earlier <science-gateway-iam-account>`.


.. _science-gateway-prod-docker:

Deploying the gateway in a production environment
-------------------------------------------------

The current deployment of the gateway (both frontend and backend) is spread over two
nodes. The deployment of new versions is handled through CI/CD. Here, we will describe
the relevant aspects of the setup of the nodes.

Deployment systems
^^^^^^^^^^^^^^^^^^

Both systems (to which we will respectively refer as ``test`` and ``production``
systems) run on the STFC cloud. The ``production`` system (for which the CI/CD system
is used to push the ``main`` branch of the corresponding repositories) is called
``srcnet-esap``, and is accessible through the URL https://gateway.srcdev.skao.int.
The test deployment, to which any branch can be deployed manually, runs on the
``srcnet-esap-test`` node, accessible through https://gateway.srcdev-test.skao.int.

The configuration of the proxies that forward the traffic from the URL to the node is
such that any traffic coming in on port ``443`` on the externally-facing URL is sent to
port ``80`` on the host.

Prerequisites
^^^^^^^^^^^^^
Traefik
"""""""

To route traffic within the host, we use `Traefik <https://traefik.io/>`_. We detail the
installation here.

To access the node, an account on the STFC system is required. Access via ssh is granted
through the ``bastion`` host. We will assume you have been granted access, know how
to access the system, and are logged in to a terminal on the host with root privileges.

For deployments, we use a user called ``gitlab_deploy``. So to manage the Traefik setup,
change to that user, and go to the directory holding the docker-compose file with the
configuration:

.. code-block:: shell

   $ sudo su gitlab_deploy
   $ cd ~/deploy-traefik

In this directory, you will find several artifacts. The first one being a docker-compose
file (a nice feature of Traefik is that it can run inside a docker container, while
proxying other docker containers. You will see in the docker-compose file that this is
achieved by linking in the docker socket in the docker container). The
``docker-compose.yml`` file looks like:

.. code-block:: yaml

  services:
    traefik:
      image: traefik:2.10 # or any later version
      ports:
        - "80:80" # http
        - "8081:8081" # Traefik Dashboard
        - "3001:3001" # compute API
      volumes:
        - $PWD/traefik/local.yml:/etc/traefik/traefik.yml
        - /var/run/docker.sock:/var/run/docker.sock # see Security considerations below
      networks:
        - esap_network
      restart: always
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.dashboard.rule=PathPrefix(`/dashboard`)"
        - "traefik.http.routers.dashboard.service=api@internal"
        - "traefik.http.routers.dashboard.entryPoints=dashboard"

The ``ports`` section tells us that Traefik will proxy traffic coming in through port 80
(for both the gateway frontend and backend). Port ``8081`` exposes the Traefik dashboard,
which is a nice debugging tool. However, the dashboard is only accessible from
within the internal network of the node (so some proxying is needed to look at it). The
last port exposed is ``3001`` which is used for the compute API demonstrator (which for
practical reasons also runs in a container on this host, but is mapped to a different
URL than the gateway).

Apart from the aforementioned Docker socket object, the Traefik configuration file is
linked as a volume (**NB** In principle the configuration could also have been fully
done using docker labels rather than a yml file, keeping everything in one place, this
is really a matter of taste of the deployer). We will come back to the ``.yml`` file
later.

The container is coupled to a network, currently called `esap_network` for historical
reasons. At the bottom you see that the network is of type `external`,
meaning that it has to be manually created and is not managed by docker-compose:

.. code-block:: shell

    $ docker network create esap_network

The ``labels`` block define some of the configuration. In this configuration, Traefik
is enable, and the dashboard (which connects to an entrypoint called ``dashboard``).

Finally, the Traefik ``.yml`` configuration looks like this:

.. code-block:: yaml

    entryPoints:
      web:
        address: :80
      dashboard:
        address: :8081
      compute-api:
        address: :3001
    log:
      level: DEBUG
    providers:
      docker:
        exposedByDefault: false   # explicitly make container available using labels
        network: esap_network     # should match the (external) network created earlier
    api:
      dashboard: true

The first block defines ``entrypoints`` which essentially are the ports to which the
Traefik application is listening (unsurprisingly those are the same as the ports opened
in the docker-compose file). Each docker container connecting to Traefik will need to
configure the entrypoint it will use so that Traefik knows from what incoming port on
the host to forward the requests. The log level is set to debug. There is only one
``provider`` (source of applications): the docker provider. The first
``exposedByDefault`` setting makes it required for any container to explicitly announce
it wants to be routed by Traefik through labels. The ``network`` setting tells Traefik
what the internal Docker network is that needs to be connected (unsurprisingly this
matches the name of the network in the docker-compose file). The final setting tells
Traefik to run the API so that it can be connected to.

Deploying the backend API
^^^^^^^^^^^^^^^^^^^^^^^^^

In your terminal, navigate to the directory of the backend repository.

.. code-block:: shell

    $ cd ./path/to/ska-src-api-gateway

Add values for the environment variables ``API_IAM_CLIENT_ID`` and
``API_IAM_CLIENT_SECRET`` to the ``docker-compose.yml`` file based on the IAM client
:ref:`created earlier <iam-science-gateway>`.

Check that the value of the environment variable ``RUN_MODE`` in ``docker-compose.yml``
is set to ``PROD``.

Deploy the docker service:

.. code-block:: shell

    $ docker-compose up -d

Access the database container and create the required tables:

.. code-block:: shell

    $ docker exec -ti gateway-backend-db /bin/bash
    $ mysql -u root -p  # will prompt for a password, use V1Lv5FxUde
    $ mysql> USE gatewaysettings
    $ mysql> CREATE TABLE Users ( ID int NOT NULL AUTO_INCREMENT, IAM_ID varchar(255), PRIMARY KEY (ID) );
    $ mysql> CREATE TABLE Preferences ( ID int NOT NULL AUTO_INCREMENT, UserID int, DarkMode tinyint, Language varchar(10), PRIMARY KEY (ID) );
    $ mysql> CREATE TABLE Services ( ID int NOT NULL AUTO_INCREMENT, Service_ID varchar(255), PRIMARY KEY (ID) );
    $ mysql> CREATE TABLE ServiceTokens ( ID int NOT NULL AUTO_INCREMENT, UserID int, ServiceID int, Username varchar(255), Token varchar(255), PRIMARY KEY (ID) );
    $ mysql> CREATE TABLE DataManagementJobs ( ID int NOT NULL AUTO_INCREMENT, UserID int, JobID varchar(32), ToStorageAreaUUID varchar(255), Status varchar(255), PRIMARY KEY (ID) );

Check that the tables were created:

.. code-block:: shell

    $ mysql> SHOW TABLES;

Type ``exit`` twice to leave MySQL and the container.

Deploying the frontend web UI
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In your terminal, navigate to the directory of the backend repository:

.. code-block:: shell

    $ cd ./path/to/ska-src-ui-gateway

Build the frontend docker image:

.. code-block:: shell

    $ docker build . --tag 'ska-src-ui-gateway'

Create a ``docker-compose.yml`` file with the following contents:

.. code-block:: yaml

    services:
      esap_api:
        container_name: tangerinenet
        image: ska-src-ui-gateway
        expose:
          - "3000"
        networks:
          - esap_network
        restart: always
        labels:
          # Enables traefik to connect to the API
          - "traefik.enable=true"
          - "traefik.http.routers.tangerinenet.entryPoints=web"
          - "traefik.http.routers.tangerinenet.rule=PathPrefix(`/`)"
          - "traefik.http.routers.tangerinenet.service=tangerinenet"
          - "traefik.http.services.tangerinenet.loadbalancer.server.port=3000"
    networks:
      esap_network:
        external: true

Deploy the docker service:

.. code-block:: shell

    $ docker-compose up -d