.. _prometheusgrafana:

Local monitoring
================

To monitor SRCNet Local services, one of the options is to deploy a monitoring 
service using grafana (to view the metrics visually), prometheus (stores the 
metrics database) and blackbox exporter (exposes the metrics of the services).
With this deployment, it is possible to monitor the SRC services that are exposed 
both externally and those services that are only exposed locally. 

Deploy of Exporter, Prometheus and Grafana
------------------------------------------

Docker
^^^^^^

This deployment is based on docker-compose. To start the deployment it is 
necessary to have docker and docker-compose installed on the system that will 
be the central point to collect metrics. This system must have access to the 
exposed local and external services. 

.. note::

We need to create the next configuration files:

**docker-compose.yaml**

With the next code: 

.. code-block:: bash

    version: '3.7'
    services:
    prometheus:
        image: prom/prometheus:latest
        container_name: prometheus
        volumes:
        - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
        - ./prometheus/data:/prometheus
        ports:
        - "EXTERNAL_PORT1:9090"
        command:
        - '--config.file=/etc/prometheus/prometheus.yml'
        - '--storage.tsdb.path=/prometheus'
        - '--storage.tsdb.retention.time=1y'
    
    grafana:
            #image: grafana/grafana:latest
        image: grafana/grafana-enterprise:11.2.0-ubuntu
        container_name: grafana
        environment:
        - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
        - GF_SERVER_ROOT_URL=<SERVER ROOT>
        volumes:
        - ./grafana/grafana.ini:/etc/grafana/grafana.ini
        - ./grafana/data:/var/lib/grafana
        ports:
        - "EXTERNAL_PORT2:3000"
    
    blackbox_exporter:
        image: prom/blackbox-exporter:latest
        container_name: blackbox_exporter
        volumes:
        - ./blackbox_exporter/blackbox.yml:/etc/blackbox_exporter/config.yml
        ports:
        - "EXTERNAL_PORT3:9115"
        command:
        - '--config.file=/etc/blackbox_exporter/config.yml'


Change ``SERVER ROOT>`` with your service URL for Grafana.

**prometheus/prometheus.yml**

With the next code: 

.. code-block:: bash

    global:
    scrape_interval: 120s
 
    scrape_configs:
    - job_name: 'prometheus'
        static_configs:
        - targets: ['localhost:EXTERNAL_PORT1']
    
    - job_name: 'blackbox'
        metrics_path: /probe
        params:
        module: [http_2xx]
        static_configs:
        - targets: # ADD HERE THE URL OF YOUR SERVICES AS MANY AS YOU WANT TO MONITOR
            - https://spsrc25.iaa.csic.es
            - <OTHER SERVICES>
        relabel_configs:
        - source_labels: [__address__]
            target_label: __param_target
        - source_labels: [__param_target]
            target_label: instance
        - target_label: __address__
            replacement: blackbox_exporter:9115

Change/add services you want to monitor by checking http `2XX`. In this example we've added ``https://spsrc25.iaa.csic.es``, but it can be any local or external service by adding new URLs in ``<OTHER SERVICES>``.

**grafana/grafana.ini**

.. code-block:: bash

    [server]
    #http_port = 3000 # if you need to use a specific port
    root_url = <SERVICE URL>
    
    [security]
    admin_user = admin
    admin_password = <YOUR_PASSWORD>
    
    [auth]
    oauth_allow_insecure_email_lookup=true
    
    # The following configuration is to use SKAO IAM service
    [auth.generic_oauth]
    enabled = true
    name = "IAM Provider"                 # Name in the button in login page
    allow_sign_up = true                  # Allowing new users to register
    client_id = <CLIENT ID>            # IAM client id
    client_secret = <CLIENT SECRET>    # Client password
    scopes = openid profile email         # Scopes
    auth_url = https://ska-iam.stfc.ac.uk/authorize  #
    token_url = https://ska-iam.stfc.ac.uk/token #
    api_url = https://ska-iam.stfc.ac.uk/userinfo #
    redirect_uri = https://your.domain.com/login/generic_oauth 

Change ``<SERVICE URL>`` with the ``URL`` of your grafana service endpoint. Add a password for the `admin` account in ``<YOUR_PASSWORD>`` and
include a ``<CLIENT ID>`` and ``<CLIENT SECRET>`` with the values of an SKA IAM client that you have previously created.

Finally run:

.. code-block:: bash

    docker compose up -d

In order to be able to visualise the metrics in Grafana of the exposed services, 
it is necessary to install a Grafana dashboard that has a predefined interface for 
this purpose. Dashboard. A dashboard can be selected to test and check the metrics:

- Blackbox Exporter (HTTP prober) https://grafana.com/grafana/dashboards/13659-blackbox-exporter-http-prober/
- Overview of Blackbox Exporter: https://grafana.com/grafana/dashboards/5345-blackbox-exporter-overview/
 

 TBD Node Exporter Deployment (nodes)
As we have seen, the implemented solution is based on blackbox exporter. This is in charge of reading the state of the service from outside.
If we want to monitor the machines, both physical and virtual, the same solution can be deployed but based on node exporter. 

We need to create the next configuration files:

**docker-compose.yaml**

With the next code: 

.. code-block:: bash

    version: '3.7'
    services:
    prometheus:
        image: prom/prometheus:latest
        container_name: prometheus
        volumes:
        - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
        - ./prometheus/data:/prometheus
        ports:
        - "EXTERNAL_PORT1:9090"
        command:
        - '--config.file=/etc/prometheus/prometheus.yml'
        - '--storage.tsdb.path=/prometheus'
        - '--storage.tsdb.retention.time=1y'
    
    grafana:
            #image: grafana/grafana:latest
        image: grafana/grafana-enterprise:11.2.0-ubuntu
        container_name: grafana
        environment:
        - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
        - GF_SERVER_ROOT_URL=<SERVER ROOT>
        volumes:
        - ./grafana/grafana.ini:/etc/grafana/grafana.ini
        - ./grafana/data:/var/lib/grafana
        ports:
        - "EXTERNAL_PORT2:3000"
    

Change ``SERVER ROOT>`` with your service URL for Grafana.

**prometheus/prometheus.yml**

With the next code: 

.. code-block:: bash

    global:
      evaluation_interval: 1m
      scrape_interval: 1m
      scrape_timeout: 1m
      external_labels:
        environment: <SERVER ROOT>

    scrape_configs:
    - job_name: 'prometheus-node-monitoring'
      static_configs:
        - targets: ['localhost:EXTERNAL_PORT1']
    
    - job_name: 'vm2monitor'
        metrics_path: /metrics
        params:
        module: [Mod_to_use]]
        static_configs:
        - targets: # ADD HERE THE URL OF YOUR SERVICES AS MANY AS YOU WANT TO MONITOR
            - https://spsrc25.iaa.csic.es
            - <OTHER SERVICES>
        relabel_configs:
        - source_labels: [__address__]
            target_label: __param_target
        - source_labels: [__param_target]
            target_label: instance
 

Integration of Monitoring
-------------------------

Add Monitoring service to SRCNet Site Capabilities
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To perform this operation you have to apply for membership from your 
SKAO IAM account control panel in the following groups, change ``<SRC>`` with the 
name of your SRC:

.. code-block:: bash

  services/site-capabilities-api/roles/<SRC>/manager
  services/site-capabilities-api/roles/<SRC>/viewer

as this example:

.. code-block:: bash

  services/site-capabilities-api/roles/SPSRC/manager
  services/site-capabilities-api/roles/SPSRC/viewer

Once the request is accepted you can add this new capability to the 
SRCNet Site Capabilities database. To do it, you must follow the steps below.

1. Go to `https://site-capabilities.srcdev.skao.int/api/v1/www/sites/add/<SRC> <https://site-capabilities.srcdev.skao.int/api/v1/www/sites/add/<SRC>>``_ and change ``<SRC>`` to 
the name of your SRC (i.e. `https://site-capabilities.srcdev.skao.int/api/v1/www/sites/add/<SRC> <https://site-capabilities.srcdev.skao.int/api/v1/www/sites/add/ESSRC`_ ).
2. Log in with your SKAO-IAM credentials and you will see ``You are now logged in``.
3. Then go back to the same link as above: `https://site-capabilities.srcdev.skao.int/api/v1/www/sites/add/<SRC> <https://site-capabilities.srcdev.skao.int/api/v1/www/sites/add/<SRC>>_`.
4. You will see a webpage to manage all the capabilities of your SRC.

Finally add the your monitoring information. To add this new capability, go to the ``Compute`` 
section and include the requested information by clicking in `+` icon at the end of the section. Then you can fill the next fields:

- ``Type`` Select ``monitoring``
- ``Prefix`` i.e. https 
- ``Hostname`` i.e. your SRC grafana dashboard or another
- ``Port`` (if applicable)
- ``Path`` (if applicable)

Finally click at the bottom of the page in ``Add`` to save this new service.