Showing my portfolio in Haunt

Another gripe of mine was that I didn't want to rely on the existing Haunt readers to build the projects documentation since all my project README files are written in Org Mode. Previously, I had been using the ox-haunt Emacs package to export my Org notes to HTML to be ingested by Haunt's html-reader and automatically add them under the ./posts directory of my local blog checkout. However, I saw the recent notice in the project's README which presented this org-mode-reader as an alternative. I slightly adapted it to not rely on Git sub-modules (which Jakob's blog uses to vendor some libraries), added some additional keys for the metadata entries that my project files expect, and added a Scheme parameter that controls the name of the Emacs daemon that the emacsclient process should communicate with if the HAUNT_ORG_READER_USE_EMACSCLIENT environment variable is set to true (more on why later).

Another upside of using this native org-mode-reader was I would be able to piggy-back from Emacs' Org export capabilities and using the htmlize library meant that the HTML output would contain Font Lock Mode face properties, specially useful for the syntax highlighting of code blocks, so that the HTML output would basically look like a source file in your Emacs theme.

It should be noted that the org-mode-reader expects Emacs to be available at build time, and the reader can either work by using Emacs Batch Mode or use emacsclient to evaluate the Org export process in an existing Emacs daemon. Although my personal configuration already starts an Emacs server and connects to it via emacsclient, I didn't want to require a specific Emacs setup to be available at build time for the build of the website, so I wrote a Nix Flake that builds the website in a self-contained and reproducible way.

{
  # ...inputs
  outputs =
    inputs@{ nixpkgs, systems, ... }:
    let
      eachSystem =
        f: nixpkgs.lib.genAttrs (import systems) (system: f (import nixpkgs { inherit system; }));
    in
      {
        devShells = eachSystem (pkgs: {
          default = pkgs.mkShell {
            buildInputs = with pkgs; [
              haunt
              guile
              (emacs.pkgs.withPackages (
                epkgs: with epkgs; [
                  htmlize
                  nix-mode
                  nginx-mode
                  rainbow-delimiters
                  (trivialBuild {
                    pname = "ox-html-stable-ids";
                    version = "0.1.1";
                    src = pkgs.fetchFromGitHub {
                      owner = "jeffkreeftmeijer";
                      repo = "ox-html-stable-ids.el";
                      rev = "0.1.1";
                      hash = "sha256-58GQlri6Hs9MTgCgrwnI+NYGgDgfAghWNv1V02Fgjuo=";
                    };
                  })
                ]
              ))
            ];
            shellHook = ''
            tmpdir=$(mktemp -d)
            # ...rest of script removed for brevity
            export HAUNT_ORG_READER_EMACS_DAEMON_NAME="haunt-build"
            emacs --daemon="$HAUNT_ORG_READER_EMACS_DAEMON_NAME" -Q
            cat > $tmpdir/preamble.el<< EOF
            (require 'ox-html-stable-ids)
            (org-html-stable-ids-add)
            (setq org-html-stable-ids t)
            (require 'nix-mode)
            (require 'nginx-mode)
            (require 'rainbow-delimiters)
            (add-hook 'prog-mode-hook #'rainbow-delimiters-mode)
            EOF

            export HAUNT_ORG_READER_EMACS_PREAMBLE=$tmpdir/preamble.el
            export HAUNT_ORG_READER_USE_EMACSCLIENT=1
            haunt build
            emacsclient -s "$HAUNT_ORG_READER_EMACS_DAEMON_NAME" -e "(kill-emacs)"
          '';
          };
        });
      };
}

The above Flake creates a default shell in devShells for every available system with dependencies to Haunt, Guile, and an Emacs package with the Emacs Lisp libraries it requires to properly export and syntax highlight the Org mode files that it will read. In the shellHook, we first set the name of the HAUNT_ORG_READER_EMACS_DAEMON_NAME environment variable we'll use for the build of the site, start an Emacs daemon with its name, create a preamble.el file in a temporary directory with the necessary configuration to set up the Org export and syntax highlighting, assign that file to HAUNT_ORG_READER_EMACS_PREAMBLE which will read it before running the Org export process, instruct it to use an emacsclient process via HAUNT_ORG_READER_USE_EMACSCLIENT, run the build of the site with haunt build and finally kill the Emacs daemon process we specifically launched for this build.

Before landing on this final implementation I tried to run the build without emacsclient and purely rely on the org-mode-reader using Emacs batch mode. However, I soon realized that while the project files were being generated in the correct place, they didn't have any syntax highlighting or face properties in the HTML, which was caused by the htmlize package not applying the font-lock face properties when it's not run under a graphical Emacs process (which is exactly what the non-interactive emacs --batch does). However, I didn't want to rely on having to launch an Emacs window when building the site, or using emacs --nw since this doesn't work well with some shells like Eshell. Thus, the best middle ground I found was to use a self-contained script that creates a specialized daemon process, runs emacsclient on it, and finally kills the process via emacsclient, which allows the htmlize package to properly apply the font-lock properties to the code blocks.

In my initial implementation, I relied on having to write a different project's documentation in my website to what was already available in the project's README because I didn't know of a way to display each project's README, written in Org Mode, inside of the custom portfolio builder I had. After adapting the portfolio builder to use the blog builder and introducing the org-mode-reader, I thought I finally had chance to change this.

Ultimately, though, I didn't want to rely on having to make third-party requests neither at build time (via curl) nor at runtime (via JavaScript) to fetch the README files from a hosting site like a source forge because for one, this would mean the build wouldn't work when I didn't have a network connection, and two, I wanted the process of updating the projects documentation in the site not being coupled with the update of the projects repositories upstream. So, if I made some changes in one of my local project checkout's README I wanted to have the freedom to preview these changes locally and publish the site with these without being tied to having to push the changes to the project's remote first.

Since I was already using Nix and Flakes to set up the build of the blog, it only made sense for me to leverage them to accomplish this task. All my projects are using flakes, so it was as easy to add them as flake inputs:

{
  inputs = {
    # ...rest of inputs
    ordenada.url = "github:migalmoreno/ordenada";
    tubo.url = "github:migalmoreno/tubo";
    nx-router.url = "github:migalmoreno/nx-router";
    nx-tailor.url = "github:migalmoreno/nx-tailor";
    nx-mosaic.url = "github:migalmoreno/nx-mosaic";
    fdroid-el.url = "github:migalmoreno/fdroid.el";
    nyxt-el.url = "github:migalmoreno/nyxt.el";
  };

}

Note that if your project is not using Flakes, you could also specify a non-flake input like the following and it would still fetch the project's checkout:

{
  inputs = {
    project-name = {
      url = "<project_url>";
      flake = false;
    };
  };
}

Now, I had access to each of the project's checkout as an input, which I would use in the previously mentioned devShells to build the documentation for every one of them before invoking haunt build:

{
  outputs =
    inputs@{ nixpkgs, systems, ... }:
    let
      eachSystem =
        f: nixpkgs.lib.genAttrs (import systems) (system: f (import nixpkgs { inherit system; }));
    in
      {
        devShells = eachSystem (pkgs: {
          default = pkgs.mkShell {
            # ...build inputs removed for brevity
            shellHook = ''
            tmpdir=$(mktemp -d)
            ${toString (
              map
                (name: ''
                  cat ${./projects/${name}.org} ${
                    if
                      pkgs.lib.hasAttrByPath [
                        "packages"
                        pkgs.system
                        "docs"
                      ] inputs.${name}
                    then
                      "${inputs.${name}.packages.${pkgs.system}.docs}/index.org"
                    else
                      "${inputs.${name}}/README"
                  } > $tmpdir/${name}.org
                  ln -sf $tmpdir/${name}.org ./posts/projects/${name}.org
                '')
                (map (path: pkgs.lib.removeSuffix ".org" path) (builtins.attrNames (builtins.readDir ./projects)))
            )}
            # ...rest of script removed for brevity
          '';
          };
        });
      };
}

The above iterates through all the projects Org files inside of ./projects, where the project metadata (the Org properties turned into metadata that a project post would expect) lives, checks if the input has a docs package exposed (because a project could have a standalone project documentation package like for ordenada which dynamically generates all of its configuration options), and if so include the index.org file under that package, otherwise just include the README file, and add the project metadata file's contents along with this file's contents (via cat) into a file stored under /tmp/. This in turn, would be force sym-linked into a file under ./posts/projects/ so that the entire project documentation would be available to the Haunt build.

As I mentioned earlier, I didn't want to be limited to having to push the project documentation changes to its remote in order to update a project's documentation in the site. I already follow a similar approach for the publishing of this site, where I don't couple its publishing to a typical CI/CD build step which is executed on every repository push, but I instead opt to manually instruct when and how to publish the site directly via rsync from my local checkout.

Using Flakes made this very simple. On one side, having the upstream project flake input URLs in place would mean that when I invoked this:

nix develop --recreate-lock-file

It would fetch the latest changes to the project flake inputs, update them, and build a devShell with the newly updated dependencies.

On the other hand, I could also add a custom registry flake pointing to a local project checkout like this:

nix registry add flake:<project> path:///path/to/project

I chose to use custom flake registries because it's shorter than specifying the whole path and it means I don't have to hard-code the path in case I want to use a local project flake input URL in flake.nix, so that if I check out the site's repository under a different system it will rely on the flake:<project> input URL and not the full hard-coded value to the other system's path.

Then, I'd be able to override the input(s) on any build with the custom project flake registry like this:

nix develop --override-input <project> flake:<project>

Which builds the site with the latest changes from the local <project> checkout and it doesn't mutate the Flake lock file.