Registering Your Central Monitoring Targets

Quickstart

To register your services for the Central Monitoring dashboard, you will essentially need to make a Merge Request to the CHSRC repository.

The apps/kube-prometheus-stack/base/targets/values.yaml file contains all the Internet-facing targets that will be scraped. The file has been pre-populated with all SRCs. The only missing data for each SRCNet service is:

  • The endpoint URL

  • An environment label with the endpoint environment (either dev or prod).

  • (Optional) If you’re registering a global service (i.e., not a local SRC service), add an additional label: scope: ‘global’.

Detailed Instructions

More precisely, you can follow these steps:

  1. Navigate to the targets folder on the dev branch.

  2. The values.yaml file contains all the scraping targets.

  3. (Unless you’re cloning this repo to make edits) Click on Edit → Open in Web IDE to edit the targets.

  4. Locate the section that corresponds to your SRC node. Some fields should already be prefilled for you, but please double-check everything.

  5. Make your changes and commit them to a new branch. Please remember to adhere to the SKAO conventions for pushing code. Additionally, read the Contributing to the repository section carefully to avoid issues with Marvin’s CI bot.

  6. Once you are satisfied with your changes, create a Merge Request (MR) to the dev branch, again following the aforementioned conventions.

  7. We will review your MR in due course. If you feel that your MR has been overlooked or if you wish to expedite the process, please reach out to any member of the Chocolate Team or message us all in our Slack channel.

If you encounter any issues, don’t hesitate to ask for assistance!

A Few More Things to Consider for New Services

  • Do not change nor substitute any values under the section relabel_configs:. It’s magic that’s required to integrate with our monitoring infrastructure.

  • Please stick to the following format when adding your services:

    - targets:
    - https://example.url.org/
  labels:
    servicename: 'local-service'
    environment: 'prod'  # pick either 'dev' or 'prod'

  • When adding an SRCNet service that’s a Global Service, make sure you label it as such. Example:

    - targets:
    - https://example.url.org/
  labels:
    servicename: 'global-service'
    environment: 'prod'  # pick either 'dev' or 'prod'
    scope: 'global'  # this label is only needed for global services

  • When adding a new site, please copy and paste an existing site and make the following changes:

    1. Ensure job_name is unique (consider using <site>-blackbox).

    2. Adjust the target_label: location replacement to match your specific site.

    3. Change all service references to the services you want to monitor.

Make sure to review all relevant fields!