there was a fish in the calculator

Secrets management with SOPS Guix

2024-01-02T16:20:00Z

Dealing with secrets in functional operating systems can range from pretty usable to complete hell. Nix has several answers to this problem, the more integrated of which appears to be sops-nix. After spending some months envying our neighbors grass, I figured it was time for Guix to have its own (attempt at an) answer to the secrets problem.

This is how the SOPS Guix channel was born, the first take I'm aware of at implementing secure deploying of secrets with Guix and SOPS and as the name shows is quite inspired from Nix's sops-nix.

Secure secret provisioning with Guix

This channels exposes the sops-secrets-service-type Guix service and the sops-secret record to safely handle secrets with Guix. It works by putting encrypted secrets in the store and by adding a one-shot Shepherd service that decrypts them at startup in a ramfs/tmpfs filesystem. This means that clear text secrets never hit the disk and that you can (and actually are encouraged to) check in your SOPS secrets in the same version control system you use to track you Guix configurations.

Assuming that the right private keys are also provided, sops-secrets can be included in Guix images, deployed with guix deploy and included in Guix System/Home containers. Before starting, make sure you add it to your .config/guix/channels.scm, run guix pull and make sure sops-guix appears in your guix describe output.

Creating secrets with SOPS

First of all you need to create encrypted secrets with SOPS. To do so I'm assuming you already have a GPG key for yourself and the machines you want to deploy secrets to. You should be able to list the private keys you have in your keyring with

user1@home:~ $ gpg --list-secret-keys
/home/user1/.gnupg/pubring.kbx
------------------------
sec   ed25519 2023-12-01 [SC] [expires: 2907-11-30]
      8D1060B96BB8B7249AED41CC193B701E2SODIJNS
uid           [ultimate] user1@example.org
ssb   cv25519 2023-12-01 [E]

pub   rsa3072 1970-01-01 [SCE]
      8C3E4F6EB38828939029AE7BE9B6AF0CD39DD935
uid           [ unknown] root (Imported from SSH) <root@localhost>

pub   rsa3072 1970-01-01 [SCE]
      ZZ3E4VREB38800039029AE7BE9B6AF0CD39AALH9
uid           [ unknown] root (Imported from SSH) <root@localhost>

If you don't have a suitable set of GPG keys it's pretty simple to find online how generate them. Once you have a suitable set of keys for yourself and your machines you are ready to create the only configuration you need for SOPS: a .sops.yaml file that you will place in your project's root directory, or anyway in the same directory where you keep your system configuration. In this file you define which keys will be able to access your secrets files, it may very well be something like:

keys:
    - &user_user1 8D1060B96BB8B7249AED41CC193B701E2SODIJNS
    - &host_host1 8C3E4F6EB38828939029AE7BE9B6AF0CD39DD935
    - &host_host2 ZZ3E4VREB38800039029AE7BE9B6AF0CD39AALH9

creation_rules:
    - path_regex: .*common\.yaml$
      key_groups:
          - pgp:
                - *user_user1
                - *host_host1
                - *host_host2
    - path_regex: .*host1\.yaml$
      key_groups:
          - pgp:
                - *user_user1
                - *host_host1

In this file we define three keys called user_user1, host_host1 and host_host2 . The prefixes host_ and user_ are just a convention to indicate that some GPG keys belong to users and some belong to machines.

We now have defined two secrets file names patterns and we declared permissions for each key, it should be possible now to run the following in your projects root directory:

sops common.yaml

This will open your default editor with an example content to define your secrets value. You can edit it or delete it and your own content for example:

wireguard:
    private: MYPRIVATEKEY

after saving and closing the file you can see by catting the secret file that sops encrypted it before saving it, so you are free to check it in your VCS.

Making sure the right host keys are in the configured GnuPG keyring

For hosts to be able to decrypt secrets you need to provide in the root user keyring (or anyway the keyring located at the configured gnupg-homedir) the keys you defined in your .sops.yaml. So based on the above example you'd need to provide 8C3E4F6EB38828939029AE7BE9B6AF0CD39DD935's private key on host1 and ZZ3E4VREB38800039029AE7BE9B6AF0CD39AALH9's private key on host2 .

To check that your key is correctly imported into the keyring run:

user1@host1:~ $ sudo gpg --list-secret-keys
/root/.gnupg/pubring.kbx
------------------------
pub   rsa3072 1970-01-01 [SCE]
      8C3E4F6EB38828939029AE7BE9B6AF0CD39DD935
uid           [ unknown] root (Imported from SSH) <root@localhost>

By setting generate-key? to #t in sops-service-configuration a GPG key will be automatically derived for you from your system's /etc/ssh/ssh_host_rsa_key and added to the configured keyring. It is discouraged to do so and you are more than encouraged to autonomally provide a key in your configured keyring.

Adding secrets to your `operating-system` record

Now, supposing you have your operating-system file in the same directory where you have your .sops.yaml and common.yaml files, you can simply add the following to your configuration:

(use-modules (sops secrets)
             (sops services sops)
             (guix utils))

(define project-root
  (current-source-directory))

(define sops.yaml
  (local-file (string-append project-root "/.sops.yaml")
              ;; This is because paths on the store
              ;; can not start with dots.
              "sops.yaml"))

(define common.yaml
  (local-file (string-append project-root "/common.yaml")))

(operating-system
  [...]
  (services
    (list
       [...]
       (service sops-secrets-service-type
                (sops-service-configuration
                  (gnupg-homedir "/mnt/.gnupg")
                  (generate-key? #t)
                  (config sops.yaml)
                  (secrets
                    (list
                      (sops-secret
                        (key '("wireguard" "private"))
                        (file common.yaml)
                        (user "user1")
                        (group "users")
                        (permissions #o400)))))))))

Upon reconfiguration, this will yield the following content at /run/secrets:

user1@host1:~ $ sudo ls -la /run/secrets/
total 12
drwxr-xr-x 1 root root    50 Jan  2 12:44 .
drwxr-xr-x 1 root root   254 Jan  2 12:44 ..
lrwxrwxrwx 1 root root    53 Jan  2 12:44 .sops.yaml -> /gnu/store/lyhyh91jw2n2asa1w0fc0zmv93yxkxip-sops.yaml
-r-------- 1 user1 users  44 Jan  2 12:44 wireguard
user1@host1:~ $ cat /run/secrets/wireguard/private
MYPRIVATEKEY

Adding secrets to your `home-environment` record

sops-guix also provides a Guix Home service that is able to provide most feature of the system service. Most significant limitations are:

AFAIK home-environments can't configure ramfs mount points hence the secrets-directory option is not available. Secrets are stored in /run/user/$UID/secrets which usually is mounted on tmpfs.
sops-secret-user and sops-secrets-group are ignored. All secrets belong to the user running the Guix comman line but you can still set permissions.
There's no option to automatically generate GPG keys since probably users can easily generate one.

Now, supposing you have your home-environment file in the same directory where you have your .sops.yaml and your secrets files, you can simply add the following to your configuration:

(use-modules (sops secrets)
             (sops home services sops)
             (guix utils))

(define project-root
  (current-source-directory))

(define sops.yaml
  (local-file (string-append project-root "/.sops.yaml")
              ;; This is because paths on the store
              ;; can not start with dots.
              "sops.yaml"))

(define user1.yaml
  (local-file (string-append project-root "/user1.yaml")))

(home-environment
  [...]
  (services
    (list
       [...]
       (service home-sops-secrets-service-type
                (home-sops-service-configuration
                  (gnupg-homedir (string-append (getenv "HOME") "/.gnupg"))
                  (config sops.yaml)
                  (secrets
                    (list
                      (sops-secret
                        (key '("wireguard" "private"))
                        (file user1.yaml)
                        (permissions #o400)))))))))

Upon reconfiguration, this will yield the following content at /run/secrets/$YOUR_UID/secrets:

user1@host1:~ $ ls -la /run/user/$(id -u)/secrets
total 12
drwxr-xr-x 1 user1 users    50 Jan  2 12:44 .
drwxr-xr-x 1 user1 users   254 Jan  2 12:44 ..
lrwxrwxrwx 1 user1 users    53 Jan  2 12:44 .sops.yaml -> /gnu/store/lyhyh91jw2n2asa1w0fc0zmv93yxkxip-sops.yaml
-r-------- 1 user1 users    44 Jan  2 12:44 wireguard
user1@host1:~ $ cat /run/user/$(id -u)/secrets/wireguard/private
MYPRIVATEKEY

Tame Docker selfhosting with Guix

2023-12-10T16:20:00Z

Many applications are packaged in OCI/Docker images but not in Guix. A good subset of them is written either in NodeJS, Go, Rust or languages that, as a general approach, encourage applications to have huge dependency graphs.

Let's take Grafana as an example. According to its package.json Grafana's web frontend has more than 180 direct dependencies and more than 170 build time dependencies, for a grand total of ~350 direct dependencies. This is excluding all transitive dependencies. In the same way, Grafana's Go backend has ~440 direct dependencies. Some of them may be optional, but go figure.

This phenomenon is not uncommon in modern software development, it's not that Grafana is doing worse than everyone else. Yet it's quite problematic, it clearly shows that nobody put the effort of auditing or even reviewing the dependency graph of one of the most used dashboarding application in the industry.

Dissecting the interests which prevent the investment in building software products with a sustainable maintenance process is a topic for another post, the point is that the Guix project accepts package contributions that comply to very strict standards in term of licensing and other criteria, such as whether the package and its dependencies can be completely built from source. It's the reason why practically no NodeJS application (or even web applications with complex frontends, such as Grafana) can be upstreamed to Guix. It is not clear whether they will ever be, since especially in the Javascript world packages suffer from cyclical dependency which complicate the packaging process even more.

It is very painful to have to package whole dependency graphs just to use some service on the Guix System. This is especially true when you have a goal that you want to achieve and you could use Guix to achieve it, but it does not necessarily involve sending a bunch of patches upstream. It could be that you want to start some Matrix community around Guix or other people-sized technologies, you may want to self host a personal cloud with Nextcloud or maybe you want to setup an accessible monitoring dashboard for your services with Grafana and Prometheus. These are all softwares that do not follow the best sustainability practices, but they are pretty useful to use while we build the alternative technology (also called alt-tech) that we need to survive sustainably in our digital lives.

Container taming on Guix

If you use docker compose on the Guix System, you end up having two different interfaces to manage your system services: Shepherd and Docker. The oci-container-service-type aims at implementing Shepherd Services that look and feel native (so you can configure and manage them with the usual consistent interface that Guix exposes) but under the hood are implemented as docker run (or docker rm) invokations, so that you don't have to package those huge dependency graphs just to start self hosting on Guix.

Next you can find an example of how run Prometheus on the Guix System through the oci-container-service-type. You just need to add

(use-modules (gnu services docker)
             (gnu services monitoring)
             (guix gexp))

(define prometheus.yml
  (plain-file "prometheus.yml"
              "global:
  scrape_interval: 30s
  scrape_timeout: 12s

scrape_configs:
  - job_name: prometheus
    metrics_path: /metrics
    static_configs:
      - targets: ['localhost:9090','localhost:9100']\n"))

(operating-system
  [...]
  (services
    (list
      [...]
      ;; Prometheus node exporter
      (service prometheus-node-exporter-service-type)
      ;; Prometheus OCI backed Shepherd service
      (simple-service oci-container-service-type
                      (list
                        (oci-container-configuration
                          (command
                            '("--web.enable-lifecycle"
                              "--config.file=/etc/prometheus/prometheus.yml"
                              "--web.enable-admin-api"))
                          (image "prom/prometheus:v2.45.0")
                          (ports
                            `(("9000" . "9000")
                              ("9090" . "9090")))
                          (volumes
                            `(("/var/lib/prometheus" . "/prometheus")
                              (,prometheus.yml . "/etc/prometheus/prometheus.yml:ro")))))))))

to the services field of your operating-system record. In this example it's also shown how to install the Prometheus node exporter to collect metrics from the host.

This approach obviously is lacking in reproducibility and bootstrappability, but in my experience often users need some software that is not yet guixable (i.e. possible to include in Guix upstream) so they choose to use Nix or something else entirely.

What's next?

The oci-container-service-type implements Shepherd services through Docker containers, but a Shepherd service is only a small block in the implementation of a nicely integrated Guix System service. To provide a secure, consistent and integrated experience a Guix System service may declare user accounts, to allow for less then root authority execution, or it may initialize some state upon activation: all things for which an additional service extension is required.

This is why I'm implementing a library of (hopefully) community maintained Guix System services for many common self hostable applications such as Matrix Conduit, Forgejo, Grafana and more. These services try to be as similar as possible to an upstream provided service, in the hope of being upstreamed as soon as the underlying dependency graph is packaged.