Monitor your Guix System with Grafana

2023-12-30T16:20:00Z

If you need to run Grafana on the Guix System this post is the right place. In this example we'll setup Grafana to read metrics from the same machine it's run upon, but you can adapt this to use a remote datasource.

First of all, add gocix to your Guix channels and run guix pull.

After making sure you have gocix in your guix describe output, you just need to add the following services to your operating-system record and reconfigure your system.

(use-modules (oci services grafana)
             (oci services prometheus)
             (gnu services monitoring))

(operating-system
  [...]
  (services
    (list
      [...]
      ;; Prometheus node exporter
      (service prometheus-node-exporter-service-type)
      ;; Prometheus OCI backed Shepherd service
      (service oci-prometheus-service-type
               (oci-prometheus-configuration
                (network "host")))
      ;; Grafana OCI backed Shepherd service
      (service oci-grafana-service-type
               (oci-grafana-configuration
                (network "host"))))))

After you can see both Prometheus and Grafana spawned as Docker containers, such as in this example

docker ps
CONTAINER ID   IMAGE                     COMMAND                  CREATED          STATUS          PORTS                                                                                  NAMES
fede69fcf4bf   prom/prometheus:v2.45.0   "/bin/prometheus --w…"   26 seconds ago   Up 25 seconds   0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   docker-prometheus
caa2dc1f50c0   bitnami/grafana:10.1.5    "/opt/bitnami/script…"   26 seconds ago   Up 25 seconds   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                              docker-grafana

you should be able to visit localhost:3000 and see the Grafana login page.

The default credentials are admin:admin, log in and configure a new datasource with http://localhost:9090 as URL. In this example we'll avoid creating a new dashboard from scratch and we'll import one from the Grafana's website. Just click on the button to create a new dashboard and choose the import option, paste 1860 as the Dashboard ID, select the Prometheus datasource you just created and import the dashboard.

You should be presented with the full data available from the prometheus-node-exporter-service-type by default.

Tame Docker selfhosting with Guix

2023-12-10T16:20:00Z

Many applications are packaged in OCI/Docker images but not in Guix. A good subset of them is written either in NodeJS, Go, Rust or languages that, as a general approach, encourage applications to have huge dependency graphs.

Let's take Grafana as an example. According to its package.json Grafana's web frontend has more than 180 direct dependencies and more than 170 build time dependencies, for a grand total of ~350 direct dependencies. This is excluding all transitive dependencies. In the same way, Grafana's Go backend has ~440 direct dependencies. Some of them may be optional, but go figure.

This phenomenon is not uncommon in modern software development, it's not that Grafana is doing worse than everyone else. Yet it's quite problematic, it clearly shows that nobody put the effort of auditing or even reviewing the dependency graph of one of the most used dashboarding application in the industry.

Dissecting the interests which prevent the investment in building software products with a sustainable maintenance process is a topic for another post, the point is that the Guix project accepts package contributions that comply to very strict standards in term of licensing and other criteria, such as whether the package and its dependencies can be completely built from source. It's the reason why practically no NodeJS application (or even web applications with complex frontends, such as Grafana) can be upstreamed to Guix. It is not clear whether they will ever be, since especially in the Javascript world packages suffer from cyclical dependency which complicate the packaging process even more.

It is very painful to have to package whole dependency graphs just to use some service on the Guix System. This is especially true when you have a goal that you want to achieve and you could use Guix to achieve it, but it does not necessarily involve sending a bunch of patches upstream. It could be that you want to start some Matrix community around Guix or other people-sized technologies, you may want to self host a personal cloud with Nextcloud or maybe you want to setup an accessible monitoring dashboard for your services with Grafana and Prometheus. These are all softwares that do not follow the best sustainability practices, but they are pretty useful to use while we build the alternative technology (also called alt-tech) that we need to survive sustainably in our digital lives.

Container taming on Guix

If you use docker compose on the Guix System, you end up having two different interfaces to manage your system services: Shepherd and Docker. The oci-container-service-type aims at implementing Shepherd Services that look and feel native (so you can configure and manage them with the usual consistent interface that Guix exposes) but under the hood are implemented as docker run (or docker rm) invokations, so that you don't have to package those huge dependency graphs just to start self hosting on Guix.

Next you can find an example of how run Prometheus on the Guix System through the oci-container-service-type. You just need to add

(use-modules (gnu services docker)
             (gnu services monitoring)
             (guix gexp))

(define prometheus.yml
  (plain-file "prometheus.yml"
              "global:
  scrape_interval: 30s
  scrape_timeout: 12s

scrape_configs:
  - job_name: prometheus
    metrics_path: /metrics
    static_configs:
      - targets: ['localhost:9090','localhost:9100']\n"))

(operating-system
  [...]
  (services
    (list
      [...]
      ;; Prometheus node exporter
      (service prometheus-node-exporter-service-type)
      ;; Prometheus OCI backed Shepherd service
      (simple-service oci-container-service-type
                      (list
                        (oci-container-configuration
                          (command
                            '("--web.enable-lifecycle"
                              "--config.file=/etc/prometheus/prometheus.yml"
                              "--web.enable-admin-api"))
                          (image "prom/prometheus:v2.45.0")
                          (ports
                            `(("9000" . "9000")
                              ("9090" . "9090")))
                          (volumes
                            `(("/var/lib/prometheus" . "/prometheus")
                              (,prometheus.yml . "/etc/prometheus/prometheus.yml:ro")))))))))

to the services field of your operating-system record. In this example it's also shown how to install the Prometheus node exporter to collect metrics from the host.

This approach obviously is lacking in reproducibility and bootstrappability, but in my experience often users need some software that is not yet guixable (i.e. possible to include in Guix upstream) so they choose to use Nix or something else entirely.

What's next?

The oci-container-service-type implements Shepherd services through Docker containers, but a Shepherd service is only a small block in the implementation of a nicely integrated Guix System service. To provide a secure, consistent and integrated experience a Guix System service may declare user accounts, to allow for less then root authority execution, or it may initialize some state upon activation: all things for which an additional service extension is required.

This is why I'm implementing a library of (hopefully) community maintained Guix System services for many common self hostable applications such as Matrix Conduit, Forgejo, Grafana and more. These services try to be as similar as possible to an upstream provided service, in the hope of being upstreamed as soon as the underlying dependency graph is packaged.

there was a fish in the calculator

Monitor your Guix System with Grafana

Tame Docker selfhosting with Guix

Container taming on Guix

What's next?