FABIAN

Running Nix Post Upgrade Commands in Renovate

// 5 min read

Tricks to run Nix commands in the limited Renovate environment

nix renovate

If you manage Nix-based projects, you probably love the reproducibility, but you probably hate the boilerplate. I created a tool called Soonix to address this issue. It generates configuration files such as renovate.json and .gitlab-ci.yml directly from Nix code, ensuring that my repositories stay consistent.

However, automating dependencies with Renovate introduced a tricky synchronisation problem.

The Setup

I use Soonix alongside Nix-GitLab-CI. This combination automatically generates a .gitlab-ci.yml file containing bootstrapping code including it’s GitLab CI/CD component. Since CI/CD components cannot access their own version (like when you include component@1.2.3 it cannot access this version, which is an issue if the component uses it’s own docker image which is also versioned), there is an “input” for “version”, which has to be the same as the component version (or rather should, here be dragons if not):

include:
  - component: gitlab.com/TECHNOFAB/nix-gitlab-ci/nix-gitlab-ci@3.0.0
    inputs:
      version: 3.0.0

By using Soonix, Nix-GitLab-CI can define this in Nix code, which then gets converted to YAML. By simply having a VERSION file in Nix-GitLab-CI, we can use builtins.readFile from Nix, to get the current version (depending on the flake input) and add it to the config:

[...]
include = [
  {
    component = "${config.componentUrl}@${config.componentVersion}";
    inputs =
      {
        version = config.componentVersion;
      }
      // config.componentInputs;
  }
];
# [...]

Here config.componentVersion can be overridden, but defaults to trimWith {start = true; end = true;} (builtins.readFile ./VERSION).

The Problem

When Renovate now tries to update the flake inputs and GitLab CI/CD component versions, it results in a change similar to this:

--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,4 +1,4 @@
 include:
-  - component: gitlab.com/TECHNOFAB/nix-gitlab-ci/nix-gitlab-ci@2.1.0
+  - component: gitlab.com/TECHNOFAB/nix-gitlab-ci/nix-gitlab-ci@3.0.0
     inputs:
       version: 2.1.0

The version of the input (thus the docker image) is not updated. While it’s certainly possible to fix this using Renovate’s RegEx manager (that’s what I used before), it’s pretty limiting and since we already generate these files it’s inefficient to then edit them using RegExes.

So, we just have to run the Soonix hook in Renovate and be done?

Yeah! Kinda :D

Let’s try adding nix run .#soonix:update to the renovate.json’s postUpgradeCommands:

Terminal window
warning: binary cache 'https://cache.nixos.org' is for Nix stores with prefix '/nix/store', not '/tmp/containerbase/cache/nix/store'
error: hash mismatch in fixed-output derivation '/tmp/containerbase/cache/nix/store/8ygpz0xfscxbn9wyhgsn752na5rvqsfh-gettext-0.25.1.tar.gz.drv':
         specified: sha256-dG+VXULXHrac52OGnLkmgvCaQGZSjQGLbKej9ICJoIU=
            got:    sha256-L6XF1fSV7m2aceK9vkumy72fHPeFN/DzOIDtzuvLuDo=
error: Cannot build '/tmp/containerbase/cache/nix/store/9zq2gd1ccafzqarrba9zyxmsi9nigk7g-gettext-0.25.1.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /tmp/containerbase/cache/nix/store/4akkyicl0vlz4ppiibi1hbz6r4ww0p6j-gettext-0.25.1-info
         /tmp/containerbase/cache/nix/store/67h0l1142rx02cmghawz9flbpmk0v920-gettext-0.25.1-doc
         /tmp/containerbase/cache/nix/store/dhgii7f71wj1217pp57vxsy6dl6in531-gettext-0.25.1
         /tmp/containerbase/cache/nix/store/xaa94ldnnxwb4m1mfq4146dahr9ybjr7-gettext-0.25.1-man
error: Cannot build '/tmp/containerbase/cache/nix/store/wc4dy4ygr2lqhwiifrnqmfn0xbwlgsw8-gettext-0.25.1.drv'.
# [...] a bunch of more lines like above
error: Cannot build '/tmp/containerbase/cache/nix/store/asw70crc06yxlfiicch3va70r8a7mc7n-stdenv-linux.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /tmp/containerbase/cache/nix/store/wmzmdn337pywyirwv3ykkbpc33h2rw3p-stdenv-linux
# [...]

Uh oh, that brings back some memories tinkering with Nix in root-less scenarios where I tried to not use /nix.

Well, let’s see why Nix tries to use this path:

Terminal window
docker run --rm -it renovate/renovate:42.11.0 /bin/bash
$ install-tool nix 2.32.2 # what renovate does under the hood
$ cat $(which nix)
# [...] some other stuff
export NIX_STORE_DIR=/tmp/containerbase/cache/nix/store NIX_DATA_DIR=/tmp/containerbase/cache/nix/data NIX_LOG_DIR=/tmp/containerbase/cache/nix/log NIX_STATE_DIR=/tmp/containerbase/cache/nix/state NIX_CONF_DIR=/tmp/containerbase/cache/nix/conf
/opt/containerbase/tools/nix/2.32.2/bin/nix "$@"

Oh, yeah, it just forces Nix to have it’s store at /tmp/containerbase/cache/nix/store, but Nix doesn’t really like that since now it cannot use https://cache.nixos.org to substitute derivations. Thus the build takes an eternity and ultimately fails since stuff couldn’t be built or hash mismatches occur.

The Solution

After trying all kinds of combinations of Nix with proot and others, I decided to just try nix-portable, surely they are better than me at using proot. After quickly downloading it with curl and making it executable, I tried it in the renovate container and it actually worked ootb. Nice.

Since install-tool from containerbase doesn’t support nix-portable and I didn’t want to specify the installation commands in all my renovate.json files, I ended up modifying the docker entrypoint. I run Renovate as a Kubernetes CronJob, so there you have to change command and args:

command = ["sh"];
args = [
  "-c"
  # sh
  ''
    mkdir -p /home/ubuntu/.local/bin
    curl -L https://github.com/DavHau/nix-portable/releases/latest/download/nix-portable-$(uname -m) > /home/ubuntu/.local/bin/nix-portable
    chmod +x /home/ubuntu/.local/bin/nix-portable
    exec /usr/local/sbin/renovate-entrypoint.sh renovate
  ''
];

Probably a good idea to pin nix-portable instead of using latest.

Also setting this env (or using the Renovate config file, whatever you use):

Terminal window
RENOVATE_ALLOWED_COMMANDS="nix-portable nix run \.#.*"

Now, just add nix-portable nix run .#soonix:update to the postUpgradeCommands and it works! Every Renovate PR/commit now includes the up-to-date generated files. And since Nix caches stuff, I just let it run for every update instead of once per branch (default setting in Renovate), just the first run takes a bit longer.

Update

I run this setup for a couple of months now, one issue I found was that merging multiple renovate MRs is now a struggle. Since every MR changes the hash, they can only be merged one after another (waiting for renovate to rebase them every time).

So some automation is required to merge these MRs together like in a merge train, ignore the conflict on the hash and then run nix run .#soonix:update afterwards. That’s still an open TODO :)