Arguments parsing in Guile

While Guile Scheme has a lot of hidden gems, like (ice-9 peg), parsing command line arguments is not one of its strengths in my opinion. Even if there are many powerful approaches to structured argument parsing, there is no API simple enough for me to know it by heart (as opposed for example to Python ArgumentParser).

In this post, I will analyze some of what I believe to be shortcomings of existing command line arguments parsing libraries and present my proposal for a new iteration with a new, small, API for parsing arguments in Guile. It is small enough to fit in my brain, so I think we may be onto something here.

On LLMs

The first commit of my library has been partially created with the help of a large language model. The commit message contains the attribution information and the prompt I used. I am aware that currently LLM provider companies are actively harming people and the environment, enabling racial profiling and supporting governments in making wars against international laws. Nevertheless, I tried one of the commercial LLMs to test its capabilities from a technological standpoint. In general, I believe that there is an ethical way of training and operating LLMs, one that respects content creators, users, workers and the environment. This post is not about LLMs, but if you have any comment or feedback for me, feel free to reach out either on the Fediverse or via email.

(ice-9 getopt-long)

The (ice-9 getopt-long) module is the first API we'll take into consideration, it is modelled after the C library of the same name. According to git log, it is the oldest command line parsing API around, it was added to Guile in 1999!

commit 4925695e21f9150632c4173a7814ce78aacd80bc
Author: Jim Blandy <jimb@red-bean.com>
Date:   Fri Feb 12 10:09:29 1999 +0000

* getopt-long.scm: Remove debugging calls to `pk'.
A new argument-processing package from Russ McManus.
* getopt-long.scm: New file.
* Makefile.am (ice9_sources): Added getopt-long.scm.
* Makefile.in: Regenerated.

Its designed around the idea that it should only take care of short (-s) and long (--long) options, leaving the parsing of positional arguments (optionally delimited by --) to the caller. One would define an option specification:

(define option-spec
  '((help
     (single-char #\h)
     ;; --help does not accept a value
     (value #f))
    (shout
     (single-char #\s)
     ;; --shout does not accept a value
     (value #f))
    (language
     (single-char #\l)
     ;; --shout accepts a value
     (value #t))))

This code defines three options:

• -h/--help which is intended to print an help message
• -s/--shout and -l/--language which are business logic related flags

getopt-long accepts short and long forms interchangeably, without making distinction. An example is that greet.scm --language it --shout hajar, greet.scm --language en -s hajar and greet.scm -sl it hajar have all the same meaning.

Notice that this specification contains a -h flag, but getopt-long does not handle that by default as some similar APIs do. You are the one that should detect that the 'help option has been passed and emit the help message yourself:

;; Help message
(define (show-help)
  (display "Usage: greet.scm [OPTIONS] NAME

Say hello to someone\n")
  (newline)

  (display "Positional arguments:")
  (newline)
  (display "  NAME")
  (newline)
  (newline)
  (display "Options:")
  (display "
  -h, --help             Show this help message and exit")
  (display  "
  -s, --shout")
  (newline)
  (display "
  -l, --language LEVEL    (default: en)")
  (newline))

;; Parse command line into a structured object
(define options (getopt-long (command-line) option-spec))

;; Use the parsed arguments
(let* ((language (option-ref options 'language "en"))
       (shout? (option-ref options 'shout #f))
       ;; Handle user passing --help
       (help (when (option-ref options 'help #f)
               (show-help)
               (exit 0)))0)))
       ;; Obtain positional arguments list
       (positional-args
        (cdr (first options)))
       (name (if (or (not positional-args)
                     (null? positional-args))
                 (error "NAME argument is required but none was passed")
                 (first positional-args)))
       (msg
        (string-append
         (if (and (string? language) (string=? language "it"))
             "ciao, "
             "hello, ")
         name)))
  (display (if shout? (string-upcase msg) msg))
  (newline))

A note on argument parsing

As you will notice later, all the examined libraries and the one I wrote follow the same general algorithm:

1. Declare some kind of structured specification
2. Derive some parsing rules from the specification
3. Apply the parsing to a list of command line arguments
4. Provide the user with some key of key-value data structure

In the above example point 1. is the option-spec declaration, points 2. and 3. happen inside getopt-long and point 4. happens in the last let.

With the above code we can run a simple greeter with a basic command line interface with:

$ guile -s greet-getopt-long.scm -s --language it hajar
CIAO, HAJAR
$ guile -s greet-getopt-long.scm -h

Usage: greet-getopt-long.scm [OPTIONS] NAME

Say hello to someone.

Positional arguments:
  NAME

Options:
  -h, --help               Show this help message and exit
  -s, --shout
  -l, --language LANGUAGE   (default: en)

In case you are interested you can also have a look at the complete code for the getopt-long example.

In general (ice-9 getopt-long) is a solid and battle tested API. In case having additional dependencies outside of the Guile runtime is not acceptable I think it is still a nice option to have, but it's API could make it easier to use.

Default values are specified at option-ref call site, this means that if you read an option with a default value in multiple places you might remember to set the default value in one. Additionally the option specification format (albeit powerful, as it allows more features than I show above, like type validation and more) is not easy to remember (at least for me) and it does not allow describing positional arguments.

One last detail is the need for users to handle --help by themselves. It's good to be able to opt in in that fine grained control but most simple scripts that just want to accept some values (I think of the output of many program-file gexps all over Guix) are usually fine with the library generating the help message and handling scenarios when the user passes -h or --help.

(srfi srfi-37)

(srfi srfi-37) is the second oldest Guile API for parsing command line arguments, it was first added to Guile in 2007:

commit d4c382218de2050de207318e9d8558c0aac6a7b9
Author: Ludovic Courtès <ludo@gnu.org>
Date:   Wed Jul 18 20:40:09 2007 +0000

Revision: lcourtes@laas.fr--2006-libre/guile-core--cvs-head--0--patch-81
Creator:  Ludovic Courtes <ludovic.courtes@laas.fr>

Added SRFI-37, by Stephen Compall.

(See ChangeLogs.)

It is similar to (ice-9 getopt-long) in spirit: positional arguments are left up to the user to be parsed, help messages must be manually crafted and the user must manually handle --help (even if it's easier than with (ice-9 getopt-long)) and options are declared in a specification.

;; Options specification
(define %options
  (list (option '(#\s "shout") #f #f               ;(short long) mandatory? optional?
                ;; Here the API is leaking,
                ;; the end result is simply to
                ;; set 'shout? to #t
                (lambda (opt name arg result)
                  (alist-cons 'shout? #t
                              (alist-delete 'shout? result))))
        (option '(#\h "help") #f #f
                (lambda args
                  (show-help)
                  (exit 0)))
        (option '(#\l "language") #t #f
                (lambda (opt name arg result)
                  (alist-cons 'language arg
                              (alist-delete 'language result))))))

(define %default-options
  ;; Alist of default option values.
  `((shout? . #f)
    (language . "en")))
    
;; Display help message
(define (show-help)
  (display "Usage: greet.scm [OPTIONS] NAME

Say hello to someone\n")
  (newline)

  (display "Positional arguments:")
  (newline)
  (display "  NAME")
  (newline)
  (newline)
  (display "Options:")
  (display "
  -h, --help             Show this help message and exit")
  (display  "
  -s, --shout")
  (newline)
  (display "
  -l, --language LEVEL    (default: en)")
  (newline))

But still, there are some differences. (srfi srfi-37) makes it easier to handle --help with its callback approach in the specification, but the in general API is leaking, forcing the user to manually manipulate options with alist-cons and alist-delete. Any user error here will weirdly break programs without users not easily noticing.

The second difference of (srfi srfi-37) is args-fold. It is called instead of calling getopt-long, it returns an association list mapping option names to option values. This format has the nice property of being easily introspectable and easy to change programmatically.

(define options
  ;; Derive parsing rules from the options specification,
  ;; and parse the command line into an association list.
  (args-fold
   (cdr (command-line))
   %options
   ;; Called in case of an unrecognized option
   (lambda (opt name arg result)
     (display "~A: unrecognized option~%") name)
   ;; Accumulation procedure for positional arguments
   (lambda (arg result)
     (alist-cons 'argument arg result))
   ;; Start seed
   %default-options))

(let* ((language (assoc-ref options 'language))
       (shout? (assoc-ref options 'shout?))
       (name (assoc-ref options 'argument))
       (msg
        (if (not name)
            (error "NAME argument is required but none was passed")
            (string-append
             (if (and (string? language) (string=? language "it"))
                 "ciao, "
                 "hello, ")
             name))))
  (display (if shout? (string-upcase msg) msg))
  (newline))

This script is completely equivalent to the example for (ice-9 getopt-long), check out its complete code in case you are interested.

(srfi srfi-37) is sligthly easier to use than (ice-9 getopt-long), but the option specification format is even more difficult to remember. I am almost unable to write or modify an option specification without having the Guile manual open. args-fold is super cool, it just misses support for parsing positional arguments to be perfect.

guile-config

This is probably the most complex tool in the list and definitely the one with a broader scope. It has been my default up until now, as it is very handy and follows a batteries included approach. guile-config is a library to declare application configurations specifications that are used to derive command line interfaces and configuration files. It does a lot of stuff, I really invite you to check it out as in my opinion for medium sized projects is perfect. My dream is that the portion of guile-config's code that overlaps with (guix records) would be abstracted from both projects into a common dependency, but this is another story and we will tell it another time.

In guile-config, users declare the configuration of an application with a configuration record:

(use-modules (config))

(define %configuration
  (configuration
   (name 'greet-config)
   (synopsis "The configuration of greet-config.")
   (keywords
    (list
     ;; Command line options or
     ;; configuration file fields
     (switch
      (name 'language)
      (character #\l)
      (default "en")
      (test string?)
      (handler identity)
      (example "it")
      (synopsis "Language to use.")
      (description "The language to use for the output."))
     (switch
      (name 'shout?)
      (character #\s)
      (default #f)
      (test boolean?)
      (handler identity)
      (synopsis "Enable shouting mode")
      (description "Whether to enable shouting mode."))))
   ;; Positional arguments
   (arguments
    (list
     (argument
      (name 'name) (test string?) (handler identity)
      (example "Hajar")
      (synopsis "Name of a friend")
      (description "The name of the friend you want to greet."))))
   (parser sexp-parser)
   (copyright "Giacomo Leidi <therewasa@fishinthecalculator.me>")
   (version "0.1.0")
   (license "GPL3+")
   (author "Giacomo Leidi")))

This record contains all informations to derive command line option names, types and help messages. This allows guile-config to perform validation and help message generation on behalf of the user.

;; Parse options and in case -h/--help is detected
;; generate the help message, display it and exit
(define options (getopt-config-auto (command-line) %configuration))

;; Access configuration values
(let* ((language (option-ref options 'language))
       (shout? (option-ref options 'shout?))
       (name (option-ref options '(name)))
       (msg
        (string-append
         (if (and (string? language) (string=? language "it"))
             "ciao, "
             "hello, ")
         name)))
  (display (if shout? (string-upcase msg) msg))
  (newline))

Here is the complete code for the above script. It is exactly what I need, were it not for the focus on configuration files also, as you can notice from the (parser sexp-parser) line of the configuration specification. This is a specification for something that can be parsed from a file, not a specification of a simple command line interface.

In many cases people just want to write small scripts calling each other and having a structured command line interface that performs validation and is easy to write. My hope with guile-arguments is to create exactly that.

guile-arguments

(arguments) was born due to an itch I often get while writing Guile scripts. In case they are very simple I always resort to calling (first (command-line)), (second (command-line)) and so on, because I am never able to remember any of the above API by heart. In some case where I really needed a structured command line API I even resorted to Python. No parentheses, can you imagine?

(arguments) API is really small and hopefully easy to remember. Users first have to declare a parser specification, which defines all the options the command will have:

(use-modules (arguments))

;; We declare a structured specification of the
;; command line API
(define parser
  (argument-parser
    #:program "greet"
    #:description "Say hello to someone."
    ;; Flags are boolean toggles, #f by default
    (flag
      (name 'shout?)
      (short #\s)
      (help "Shout instead of speaking"))
    ;; Options are arguments that take a value,
    ;; if multiple? is #t they can be passed multiple
    ;; times to represent lists
    (option
      (name 'language)
      (short #\l)
      (default "en")
      (help "Language code"))
    ;; Positional arguments are arguments referenced
    ;; by position. They also can be multiple? but
    ;; only one of them can and it must be the last
    (positional
      (name 'name)
      (help "Name of the friend to greet"))))

Next users must call parse-arguments to get back a structured object representing the command line:

;; Parse command line. parse-arguments parses (cdr (command-line)) by default,
;; but in case you want to pass the command line yourself you can pass it
;; as the last argument: (parse-arguments parser your-command-line)
(define arguments (parse-arguments parser))

;; Unpack arguments and use them
(match-arguments arguments (shout? language name)
  (let ((msg
         (string-append
          (if (string=? language "it")
              "ciao, "
              "hello, ")
          name)))
    (display (if shout? (string-upcase msg) msg))
    (newline)))

In this example we use match-arguments which is syntax sugar around the arguments-ref primitive to destructure arguments values out of the return value of parse-arguments. In case users only want to access a single argument value they can call (arguments-ref 'name-of-the-argument).

There is not much more, the above would already be equivalent to the earlier examples:

$ guix shell guile -f guix.scm -- guile -s examples/greet.scm -h
Usage: greet [OPTIONS] NAME

Say hello to someone.

Positional arguments:
  NAME              Name of the friend to greet

Options:
  -h, --help               Show this help message and exit
  -s, --shout              Shout instead of speaking
  -l, --language LANGUAGE  Language code (default: en)

(arguments) intercepts -h and --help and automatically emits help messages and exits by default, but by passing #:auto-help? #f users can turn off that behavior and handle the help request themselves.

In case you are interested you can have a look at the code of the (arguments) greeter and test if from the Codeberg repository.

You can get guile-arguments from Guix or soon from openSUSE Tumbleweed. In case you want to play around with the code, you can find it on Codeberg.

Next steps

(arguments) is currently good enough for 80% of my personal use cases. What it currently misses is: first, the ability to structurally declare subcommands and subparsers, probably they can be made to work even with the current API but it'd be hacky, second, internationalization or the ability to handle different output languages such as in auto generated help messages but also in error messages. I really hope it can be of help also to others.

In case you are interested to chat about (arguments), or have any kind of feedback, reach out via email or on the Fediverse!