AlternativeBit

Nar-klepto: Guix and Nix Offline Cache

picnoir — Mon, 01 Jun 2026 10:24:00 +0200

TL;DR: introducing nar-klepto, the context that led to its inception, and some fun experiments we did last week with it.

4 years ago, I moved out of Bayonne to a rural area. My quality of life almost instantly skyrocketed; however, I quickly felt a bit frustrated that I could only discuss computer-related things via online video meetings.

After the 2024 FOSDEM, I realized that if there’s no nerds around, why not invite my online friends at my place? And, for the last three years, I’ve been renting a very nice place close to where I live for a full week. I invite my favorite online friends there. Each year, a few strangers join, friends of friends, usually becoming new friends by the end of the week.

This is nice, it makes me happy every time.

I just came back from this year’s edition. Physically a bit exhausted, but mentally fully energized. A full week of nice discussions, drone and RC glider flying, multiplayer videogames, cooking, eating a bit too much, hot spring water, and bike rides!

The place where we do this is super nice, and pretty remote… and… do not come with any internet connection. The owners do not seem to care at all, it’s their fun vacation house, no need for internet there! We’re probably the only group renting this and caring about any internet connectivity at the same time. We’re probably terminally online.

The only uplink we have comes from a pretty terrible LTE router¹. And it’s not great. If you’re into Guix or NixOS, you probably already see the issue. These distributions are amazing. But boy oh boy, they’re network hungry. And boy oh boy, they dislike not being able to reach to their binary caches. Last year, I spent some time creating a solution for this, and it looks like this:

This is a wonderful retro Panasonic CF-SZ5, coupled with an external USB SSD. Turned on, serving NARs 24/7 for a week. Poor thing. This used to be a pretty cheap setup, the SSD probably cost 3 times more than the laptop it’s attached to these days.

Always use obsolete hardware running bleeding-edge software.

I love the attitude and what it entails: making some effort to extend the lifespan of rare resources stored in these magic machines. Nice hardware setup to boast about, but that’s not really the point of this article. The software it runs is the interesting bit: nar-klepto.

It’s a collection of 3 programs able to fully backup a Guix or Nixpkgs release. Each and every derivation part of it. That’s ~100GB worth of compressed NARs for a Guix release, ~500GB for a Nix one.

Conceptually speaking, you feed it a list of store paths, and it downloads the relevant narinfos and nar. You can retrieve this list of paths for a particular release using the data service for Guix, Hydra for NixOS. The narinfos are stored in a PostgreSQL DB, the nars on a filesystem. Finally, it comes with an HTTP server, able to send the NARs and narinfos to the Nix and Guix clients. It also provides a NixOS module to set up the server part of it and a Nginx reverse caching proxy. The reverse caching proxy comes in handy when people start to download packages not available in the cache. It could happen if an important security release forces people to upgrade their systems to a new Guix/Nix revision. The reverse proxy cache saves some of the metered LTE bandwidth and some substitution time.

Downloading a full release comes at a cost for the project’s infrastructure, especially for Guix, which does not have a sponsored CDN. This tool is wasteful by nature and is not meant for recurring usage. It’s mostly useful for camps, retreats, and conference organisers. That’s an adequate tool to provide a one-off cache for a Guix/Nix event, a pretty inadequate one to host in your personal infra and mirror each and every Guix/Nix release.

Noé Lopez packaged it for Guix. He has been the first external contributor² to the project, adding the ability for the cache to advertize itself to the guix clients via zeroconf. Quite a nice feature, I’m a bit jaleous not to have it on NixOS! A nice coincidence happened though, adisbladis ended up releasing a similar project for NixOS, with a quite different design the same week: nix-cache-beacon. I might add the cache-beacon advertisement directly in nar-klepto in the future.

I’m pretty happy and proud of releasing this tool. Took me a good while… Close to a year. Last year, a good chunk of my online friends and colleagues jumped onto the vibe coding bandwagon. I was intrigued and did not want to miss out, so I joined the hype through this project. It did not work really well for me, and I have not used these tools since then: the experience was not enjoyable for me, and the result, while working, was a bit too trashy for my taste. There was a lot of unnecessary code duplication, each duplicate being slightly different from the others. I dreaded and procrastinated refactoring the thing for a good year, but I finally bit the bullet two weeks ago, cleaned and rewrote a good chunk of it. I enjoyed the process of desloppifying the project, it was fun. That’s good for me, who knows, maybe this will become my job at some point.

I commissioned ³ some artwork from lassulus. The second iteration was the charm! The overall sketchy vibe nicely fits this project’s story.

During the geek event, I had to download a few things from the cache, and the experience was amazing. The substitution is fast, really fast. That’s a combination of two factors:

The narinfos are reachable with a very low latency, >3ms, from the cache. It noticeably speeds up cold evaluation.
The NARS are available on the lan at a pretty high download speed. Sadly, not saturating a gigabit link. Despite how much andir- loves it, the glinet router needs an upgrade!

At some point, we paired with flokli, and tried to ingest the Guix release into a snix store to see what kind of storage gains we could get from chunking the Nars. Quite a lot it turns out!. The 115GB Guix nar collection ended up being ingested into a 21GB snix store. Quite impressive! We also tried to ingest the Nix Nar collection, but the gains were not as jaw-dropping. We’re not sure what explains this Guix efficiency. Maybe the lzip -> ztsd compression change, maybe there are some very repeated chunks in the cache. We’ll have to investigate more to figure this out.

All this makes me believe there are ways to improve the substitution story of Guix/Nix. Something I’d like to focus a bit on in the near future ;)

GL-iNet X300. The hardware is ok, but the software is terrible. A fork of an outdated openwrt revision, a terribly buggy LTE model integration, and, of course, not updated for more than a year. It’s not even cheap, don’t buy that! Buy anything that can run mainline OpenWRT instead. ↩︎
to be fully honest, he was also part of the retreat and was depending on the cache. Lesson learned: a captive audience is a good bootstrap contributors pool! ↩︎
without paying him, sadly. Times are tough for artists! ↩︎

Nsncd, Anniversary Updates

picnoir — Mon, 30 Oct 2023 10:00:00 +0200

Last year, flokli and I worked towards re-using TwoSigma’s Nsncd as the main NixOS Nscd daemon.

What is that even about? Well, Nscd is a Glibc daemon that was originally meant to cache the host/user resolution requests. It’s mostly obsolete by now, but on NixOS and Guix, we abuse this daemon to get a stable ABI to load the host NSS modules from a potentially different glibc version. If you have no idea what I’m talking about here and want to read more about that context, you should probably read flokli’s original release blog post.

It’s now been a year since we released the Nsncd host lookups. Since then, Nsncd is used by default on NixOS in place of Nscd. The migration has been mostly bugless. Emphasis on the mostly!

While this switch has been beneficial for most of us, getting rid of the “too-much-caching” numerous bugs, some unexpected side effects appeared. Such as breaking the hostname --fqdn command.

Recently, flokli and I ended up under the same Catalonian roof for a Numtide-organized programming retreat. Kudos to Zimbatm who handled most of the thankless logistics to make this happen <3

After almost a full year of procrastination, turns out that a 10-hour session of focused work was all it took to almost fix these long-lingering issues.

Wait, why did I write almost here? Ah yes. After these 10 initial hours, I kept uncovering a long series of subtle yet real bugs for the next 4 days. Turns out that the third full rewrite of the fix was the charm!¹

What was the root cause of these faulty host resolutions in the end?

Turns out that, even if the gethostbyname and gethostbyaddr libc functions have been long deprecated in favor of getaddrinfo, some pretty critical pieces of software are still using them. You guessed it: the hostname command still uses those.

We originally decided to implement these two functions through getaddrinfo. This was a mistake: the legacy operations do not behave exactly like the new ones.

To fix this bug, we wrote some FFI bindings for these two legacy functions. Then, we replaced our dodgy gethostbyname and gethostbyaddr implementation in Nsncd to use these bindings.

In the meantime, flokli wrote sockburp, a debug tool for Unix sockets. This has been a game changer to detect the wire format compatibility issues. You can read more about this tool in this blog post.

We have not managed to upstream all our changes to the TwoSigma repo yet. NixOS Nsncd is still hosted from our nix-community fork. We did not give up on upstreaming it though and hope to do that in the next few months.

That’s pretty much all. Until next time, happy and serene hacking folks :)

Ah, you’re here for the gory details? Here you go, read these two PRs history. My treat! ↩︎

The Nix, OpenGL and Ubuntu Integration Nightmare

picnoir — Thu, 20 Apr 2023 10:00:00 +0200

In this article, we’re about to dive into the uncharted OpenGL on Linux waters. After briefly explaining how the OpenGL calls are routed from your application to the GPU, we’ll look at the NixOS special case. We’ll then explore how we can run the OpenGL programs built by Nix on a foreign distribution, such as Ubuntu or Fedora. Finally, we’ll introduce NixGLHost a new approach to solve this problem.

OpenGL/CUDA on Linux?

Everybody, including your favorite pet play video games. Your favorite go-to desktop application is relying on GPU acceleration to render on your screen, and AI is taking over the world. Welcome to 2023. In the last 15 years, GPUs transitioned from being this gamer-specific niche hardware to being this globally-used parrallel platform.

There are a lot of GPU-related APIs in the wild. OpenGL, OpenCL, Vulkan, DirectX, Metal, etc.. In this article, we’ll focus on a very small subset of them, OpenGL and CUDA. Even more precisely, OpenGL and CUDA on Linux. How are these two APIs working?

Linux being Linux, there’s no simple answer to that. In most setups, and for OpenGL alone, there’s no big unified API. There are actually two concurrent ones, acting as an interface between the actual OpenGL program and the desktop environment.

GLX: the legacy one, a Xorg-specific API.
EGL: the new kid in town. The interface is used by Wayland and Android among others.

To be clear: these are API specifications, not implementations. For these two APIs we have multiple implementations. Some are proprietary, and directly written by the hardware vendors. Some others are open source, sometimes implemented by hardware vendors, sometimes by courageous volunteers.

There are currently two major OpenGL implementations:

Mesa: the main OpenSource implementation. It’s actually more of a meta-driver, it includes drivers for a wide range of hardware: Intel integrated GPUs, AMD discrete GPUs, some ARM Mali, and a lot more.
NVIDIA: a proprietary implementation, maintained by Nvidia itself. That’s the optimal way to drive your Nvidia graphics card on Linux.

These drivers are usually split into two parts. A thin kernel-space part is in charge of the low-level communication with the GPU, and a thicker user-space part is in charge of translating the OpenGL instructions into something the small kernel-space part can understand. Conceptually speaking, the kernel space part acts as a proxy to the actual hardware, nothing more.

These driver stacks, on top of providing the low-level code in charge of talking to the GPU, are supplying implementations for various interfaces. Among those, are GLX and EGL. To sum things up, you have a GLX and EGL implementation for each and every driver. EG. libGLX_nvidia.so, libGLX_mesa.so, etc.

Hey, you’re lying! There is a unified API!

I already used OpenGL, I know for a fact that my program is linked against libGL.so, not to one of these weird-ass API!

You’re absolutely right! But what if I told you that libGL.so is actually a lie?

I omitted an important part of this OpenGL stack on purpose: libglvnd. Originally developed by NVIDIA before being maintained by the freedesktop community, this library aims to be the thing your library is going to link against. It’s not doing anything by itself, it just dispatches the OpenGL calls to the most appropriate APIs. Through various heuristics, libglvnd figures out through which interface the OpenGL instructions should go. Roughly speaking, if the target surface is rendered through Wayland, it’ll use the EGL implementation. If it’s rendered through Xorg, the GLX implementation. It also uses some more heuristics and config files to figure out which driver (Mesa, Nvidia, etc.) should be used.

What about CUDA? Well, for this one, we don’t really have to bother with interoperability; NVIDIA is controlling the whole chain, from the user-space SDK to the silicon after all. The overall call graph is much simpler, although harder to integrate. Your CUDA program is linked against the libcuda.so shared library. This library directly communicates with the thin kernel-space driver in charge to send the CUDA operations to the actual GPU.

Overall, from a bird’s-eye view, the overall OpenGL graphic stack looks like this:

How Does Nix Deal With That?

Now that we roughly see how the high-level components fit together in the Linux space, we know there’s a layer dynamically dispatching the graphics calls to the most relevant driver depending on the current context. It works reasonably well. Well enough to power most of the modern days GPU-fueled AI revolution. Sadly, the cost of dynamic dispatching is a loss of determinism.

NixOS

If your heart rate bumped slightly when reading the previous paragraph, chances are you already drank the NixOS kool-aid and are anticipating a fair share of pain integrating this with NixOS. And for good reason! How can we expect a statically-defined Nix closure to work on each and every end-user GPU? We surely can’t include all the existing drivers in the closure, it’d be massive!

Don’t worry too much though, we don’t have to get that hardcore. Spoiler alert: I’m about to uncover a dirty-hack. You know, the dirty-hack rendering this very web-page (if you’re reading this on NixOS).

NixOS has a magical /run/opengl-driver directory. In this directory, you can find all the libs you’ll need to drive the GPU attached to your machine. OpenGL, Vulkan rendering, OpenCL computing, VA-API video acceleration. You name it!

This directory — we’ll call it “link farm” from now on, because, well, that’s what it really is! — is defined on a system level via the hardware.opengl NixOS module. Overall, it means the GPU driver is defined through the NixOS system closure, and exposed at runtime through the /run/opengl-driver link farm. The OpenGL program derivations have no knowledge of the actual OpenGL implementation. They simply link to libGL.so AKA. libglvnd, the generic dispatch layer.

Okay, great, we have the drivers in a weird /run/opengl-driver location containing a link farm pointing to the actual GPU driver.

Now, how can my Nix-built OpenGL program find these libraries?

Well, it’s Nixpkgs! Obviously, we’re injecting them through a wrapper. The wrapper we’re talking about is called addOpenGLRunpath. It injects /run/opengl-drivers to the library path through an LD_LIBRARY_PATH entry. At runtime, Libglvnd will load these libraries and dispatch them the OpenGL calls.

On a foreign distribution, where /run/opengl-driver won’t exist, the dynamic loader simply ignores this library path entry. The wrapper essentially acts as a no-op on a non-NixOS distribution.

In a nutshell:

NixOS stores the host system GPU drivers in a well-known directory.
Nixpkgs injects this well-known directory to the OpenGL program library path through a wrapper.

What About Running Nix on a Foreign Distro?

Now, how would you run a Nix OpenGL program on a foreign Linux distribution such as Ubuntu?

All of a sudden there is no more /run/opengl-driver. This is where it gets hairy!

The go-to solution these days is NixGL. It’s a wrapper you apply on top of an OpenGL derivation. With a few clever heuristics, it figures out which driver your host system expects at build-time. It then builds the relevant driver derivation (Mesa, NVidia, Bumblebee) and generates a wrapper injecting the driver dynamic libraries to the OpenGL process load path.

Remember what I told you during the introduction: The GPU driver is split between a thick user-space library and a thin kernel-space module. What we have here is only the user-space side of the story. The user-space part of the driver is distributed through Nix, the kernel-space one by the host distribution. One might fear hitting some protocol-level incompatibilities between the user space driver and the kernel-space one. And for good reason! It could definitely happen. In practice though, it’s good enough: the NixGL heuristics tend to be fairly clever at figuring out which userspace library to use.

There’s definitely a tradeoff. The GPU driver ends up living in the Nix closure. All of sudden, your Nix closure is tailored towards a specific GPU and won’t be generic anymore. This approach requires you to precisely know the target hardware specification during the Nix build time. If you want to deploy an application on N computers, you’ll have to conceptually nix build the application N times. You’ll need a detailed inventory of the hardware you want to deploy on, or have the build happening on the end-user machine, which is not always possible.

Kludgy! Mikey Likey!

Could we have an alternate approach to solve this Nix-OpenGL-on-a-foreign-distro problem? An alternative not requiring us to include the OpenGL driver part of the deployed closure, but rather have this managed by the host distribution? Kind of what we have on NixOS?

I think it’s fair to assume the host distribution is smart enough to properly set up the graphic stack. What if we glued the host user-space GPU environment to the Nix program instead of using a Nixpkgs-provided one? This thought was the starting point of NixGLHost, a new way of running your OpenGL Nix programs on a foreign distro.

This approach is pretty Nix-heretic. We’re injecting some host DSOs¹ into the Nix closure. Nix is specifically designed to limit this kind of interaction with the host system! That being said, we’re being heretic but cautious: we only inject the entry points for EGL/GLX/CUDA. We patch the entry points’ ELF runpaths to point to a place containing their own dependencies. This acts as a minimal sandboxing setup.

I won’t dive too much into the implementation details here, you can check out the internals.md file if you’re interested in the nitty-gritty details.

Overall, this experimental new wrapper re-uses the drivers provisioned by your host distribution. The kernel-space module and user-space library are both provisioned by the host system, we’re sure they won’t get out of sync. The graphic driver stays out of the Nix closure, the closure stays truly generic and will work on any GPU.

This experiment turned out to work great for the use cases we tested it against so far. I managed to run DaVinci Resolve and a couple of videogames built by Nix on a Ubuntu machine. A customer is actually already using the prototype in production².

What’s Next for NixGLHost?

The next obvious step is to extend the supported OpenGL implementations. At the moment, we only support the NVIDIA proprietary driver. It’d be great to support Mesa as well. Technically, it’s not a hard thing to do. Sadly, the Python prototype codebase is quite messy and difficult to extend as it is.

The wrapper is also pretty slow to run, ~200ms on a hot cache. Most of this time is spent spinning up the CPython interpreter. Starting a whole interpreter for a very short-lived program turned out to be very costly, who could have guessed! Should we embrace the meme and RIIR the project? I personally think it might be a good idea and started to work on that.

We also should think about improving the sandboxing. At the moment, we’re injecting the host DSOs¹ symbols directly into the program’s global symbol table. If, by misfortune, the Nix program ends up using the same library as the GPU driver but with a slightly incompatible version, all hell breaks loose! There are some ways to go around that, we could leverage the dlmopen function to load the driver DSOs¹ to another namespace. We could potentially leverage libcapsule to perform such a “namespaced” dynamic library loading.

Let’s see how it goes, I’ll probably post a follow-up here when we’ll reach the next major milestone :)

Greets:

flokli for enduring my terrible hacks and brainstorming this problem space with me <3
Numtide and Otto Motors for trusting me on this and funding most of the NixGLHost work <3
Guibou for writing NixGL <3
Alex and tazjin for proof-reading this article <3

Dynamic Shared Object, a shared library. ↩︎ ↩︎ ↩︎
This is an alpha release, I’m not saying you should use it in production as well. Use this with caution, it may eat your kittens! ↩︎

Nix Substitution: the Way Forward

picnoir — Thu, 31 Mar 2022 08:18:00 +0200

Abstract

Nix and Guix have the unfortunate reputation to require a lot of bandwidth to distribute software. This reputation is sadly grounded. It seems like explicitly pointing to your dependencies comes with an overhead cost in terms of download size. Or does it?

Currently, both Guix and Nix can substitute pre-built store paths from a binary cache using the Nix Archive (NAR) format. After conducting several benchmarks, we conclude that the substitution granularity should be finer to improve the substitution performance. We can increase the substitution granularity either by substituting files individually or by substituting chunked individual files.

If you are already familiar with the Nix and Guix substitution mechanism, feel free to ignore the next two sections and directly jump to the “Rethinking The Substitution Granularity” one.

You Said Substitution?

We usually don’t want to rebuild the whole world locally. When we want to install Firefox on a laptop, we’d rather not build it from the source on the machine itself but instead download a pre-built version from a cache. For this purpose, Nix provides a store path substitution mechanism.

Instead of building a derivation locally, we substitute the local build with a pre-built artifact stored in a binary cache. The store paths are getting transferred between the binary cache and the local machine using the NAR archive format. This format can be seen as a compressed store path. Meaning we’ll have a NAR archive per top-level store path we’d like to substitute.

A NAR is always identified using a hash derived from its content. NARs are content-addressed.

In contrast to that, both the Nix/Guix derivations and the store paths they generate are input-addressed¹: their hashes are directly derived from their dependencies hashes.

You now see the issue: on one hand, we have input-addressed derivations and store paths, on the other hand, content-addressed NARs.

Starting from an input-addressed derivation hash, how are we supposed to figure out the name of the content-addressed NAR we want to download? In other words, how can we figure out the output hash of a derivation without building it first?

Introducing the NARinfo file. Let’s take an example:

» curl https://cache.nixos.org/r5sjd57x0r07bwgipryaxqdkx1gglhiy.narinfo

StorePath: /nix/store/r5sjd57x0r07bwgipryaxqdkx1gglhiy-libidn2-2.3.2
URL: nar/010zh5j819qyz6zai6h2hn3d7i406g00m2g4g62bq7m8g91553yb.nar.xz
Compression: xz
FileHash: sha256:010zh5j819qyz6zai6h2hn3d7i406g00m2g4g62bq7m8g91553yb
FileSize: 63248
NarHash: sha256:1npw0jz1cw4k9x25f2vsdhsa5cf9568j46bpz2768a1izqj5n9lf
NarSize: 260816
References: 0z7sqj4pilbqyp45ix5b0mdgn9xlb024-libunistring-0.9.10 r5sjd57x0r07bwgipryaxqdkx1gglhiy-libidn2-2.3.2
Deriver: 0n91syjwrhmng41f8d23ad0sl4a6ic4g-libidn2-2.3.2.drv
Sig: cache.nixos.org-1:G9PxMT0/nd9ELwL3BBeqWtb2ohMiqw4T4FUwFlAU9M0E45mKg77BzL9gJQ3wH4oYtsa61MV5uzNLPFvK8ZbyCA==

To substitute the path /nix/store/r5sjd57x0r07bwgipryaxqdkx1gglhiy-libidn2-2.3.2 from the cache.nixos.org binary cache, we’ll first have to download its associated NARinfo file. That NARinfo file is addressed using the same input-addressed hash as its associated store path: r5sjd57x0r07bwgipryaxqdkx1gglhiy. The NARinfo will contain some metadata needed to perform the substitution, but also, and most importantly, the URL from which we can download the content-addressed NAR.

To sum up, to perform a substitution, the Nix/Guix client will:

Compute the input-addressed store path of the derivation.
Fetch the associated .narinfo from the binary cache it wants to substitute from.
Download the content-addressed NARs referenced by the .narinfo.
Unpack the downloaded NARs to the local Nix store.

Substitution Hell

Now that we established how the Nix substitution mechanism works, let’s have a thought experiment: what will happen if we bump the version of a derivation living deep in the Nixpkgs/Guix dependency tree? Something like curl.

The curl store path being input-addressed, its name will change for sure. Likewise, its dependencies are also input-addressed, their name will also change. Overall, the curl store path change will propagate throughout all the Nixpkgs/Guix dependency graph.

You might say:

Hey! That’s not a big deal.

The NARs are content addressed, this change of input hash shouldn’t propagate there.

The binary cache might have to re-build the whole dependency graph, however, outside of curl, the content of the derivations should remain the same. So, in the end, most of the NARs should remain the same. The clients shouldn’t have to re-download all the store paths depending on curl.

We won’t have to re-download massive derivations like Firefox and Chromium for a simple curl bump.

Right?

And you’d be right except for one thing: in the Nix/Guix model, everything is explicit. Every time you’ll refer to the curl binary, you’ll have to explicitly refer to its store path /nix/${inputHash}-curl/bin/curl. Its input-addressed store path! This sneaky reference will manage to propagate the input-adressed instability to the output-addressed NARs.

Coming back to your example: both Chromium and Firefox depend on curl. You’ll have to re-download both of these massive store path for every curl bump. Hopefully, you won’t mind downloading a couple of gigabytes for a sub-kB string change? Do you? :P

One has to think: could we improve this situation?

Rethinking the Substitution Granularity

Content addressing the substitution archives is definitely a good idea. It brings some hash stability to this unstable input-addressed scheme. However, as we saw earlier, because of the store path references, the input-addressed hash instability ends up propagating through the substitution mechanism.

To improve the substitution stability, we could think about two solutions:

Remove the store references from the NAR and replace them on the fly when decompressing the NAR into the actual Nix/Guix store.
Increase the substitution granularity.

I’m going to ignore that first solution for this post and focus on the “Increase the substitution granularity” solution.

The Nix and Guix communities came up with two solutions based on this finer granularity substitution idea: nix-casync and a file-based substitution discussed in the Guix mailing list.²

Living in the countryside, my only Internet connection is a weak LTE link. I have 3 NixOS and 1 Guix system at home. Any mass-rebuild takes me at least 2 hours to substitute on a good day.

Needless to say, I have a personal urge to improve that situation!

Before blindly stepping into what looks like a multi-year-long sustained implementation effort, I wanted to gather some hard data showing how much we could gain from increasing the substitution granularity. It’s benchmarking time.

Benchmarking Fine Granularity Substitution Mechanisms

For various scenarios, we’re going to build a Nix closure for 2 Nixpkgs commits. Then, we’ll simulate an update from the first commit to the second one and see how much data we’ll need to download from the binary cache.

For each scenario, we’re going to evaluate the 4 following substitution techniques:

Nar substitution: This is the substitution model currently used by the NixOS and Guix projects, it’ll act as our metrics baseline. It consists in archiving and compressing a full store path. In this benchmark, we’ll identify each NAR by its filename derived from the sha256sum of its content.
Casync substitution: This is an experimental substitution mechanism implemented via the nix-casync project. Starting from a NAR, we decompress it, serialize all the files in a single stream, and chunk it into smaller bits before compressing them again. In this benchmark, we’ll identify each casync chunk by its filename, itself already derived from the sha256 sum of its content.
File-based substitution: This is the substitution method the Guix project brainstormed around. Each store gets substituted separately. In this benchmark, we’ll identify these files using the sha256 sum of their content.
XZ-Compressed File-based substitution: similar to the File-based substitution except each file is individually compressed using the xz compression mechanism (profile 6 extreme).

If you’re interested in the nitty-gritty details, the source code and the raw data behind these benchmarks live in this git repository. You can also have a look at the Jupyter notebook I used to crunch the benchmarks data.

Note: we’re using NixOS/Nixpkgs for all these benchmarks. However, provided Guix currently uses the same substitution mechanism, you can safely assume the same conclusions holds for it as well.

Mass Rebuild

Let’s kill the suspense right away and have a look at the most interesting scenario first. We’re going to simulate a NixOS closure update after the staging-next 2021-12-03 merge containing, among other things, a curl update. As we already established in the “Substitution Hell” section, a curl bump will propagate through the whole NixOS dependency graph. That’s the worst-case scenario in terms of substitution: the mass rebuild.

We’re going to use a NixOS machine description depending on several diverse language stacks (C, Python, BEAM) and simulate a closure update after this mass rebuild.

Let’s see how the four substitution techniques perform:

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	386.06	372.99	13.0702	0
Casync	608.652	192.195	416.457	48.4719
File	1652.19	705.665	973.348	-89.1915
Compressed File	476.523	229.489	247.034	38.4732

First, the good news: increasing the substitution granularity definitively works! The casync-based substitution almost cuts by half the amount of data we need to download for this mass rebuild.

YAY!

The xz-compressed file-based substitution sits in the same ballpark, saving us ~40% worth of data to download.

We can already see two other things from this experiment:

Without proper compression, the file-based substitution performs about half as good as the NAR substitution.
Looking at the closure size of the various substitution methods, we can realize that the finer the substitution granularity gets, the worse the compression algorithms are performing. In other words, the massive gain in terms of re-used data/downloaded data ratio (~75% for casync) does not directly translate into download savings. We probably want to spend some time investigating the state-of-the-art WRT. compression algorithm specialized in small files.

Provided these results, I’ll omit the uncompressed file substitution-related data for the next benchmarks: it’s consistently worse than the compressed counterpart in terms of performance and hurts the readability of the diagrams.

Channel Jump

In this scenario, provided the same NixOS machine closure we used in the Mass Rebuild scenario, we’ll simulate a jump from the NixOS-21.11 stable channel to NixOS-unstable.

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	387.458	375.167	12.2905	0
Casync	610.886	310.541	300.345	17.2261
Compressed File	478.12	307.479	170.642	18.0423

I did not expect much savings here, but surprisingly enough, we’ll shave about 18% worth of downloaded data using the compressed file-based substitution or casync.

This scenario does not make total sense from a pragmatic point of view as-is: you rarely jump a NixOS machine closure from a stable to unstable NixOS channel. It however does make sense in the context of a dev machine. You often have to jump between different projects having their own dev nix-shell closures pointing to different Nixpkgs channels.

The good and unexpected news is: even in that scenario, we can expect some substantial gains in terms of the amount of data we’ll need to download!

Single Derivation Bump

Now, one may ask: can we expect any savings for a single derivation substitution?

Ludovic Courtès recently posted a summary of his findings on that matter in the Guix mailing list for various binaries containing various amounts of binary data and text.

Let’s replicate this experiment with our benchmark. We’re going to simulate a version bump for Firefox, Gimp, Emacs, and OpenMPI. To do so, we’ll build the two derivations before and after the version bump, and from there see how much data we’ll be able to re-use and how much data we’ll need to download.

Firefox

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	219.352	56.4315	162.92	0
Casync	342.925	55.871	287.054	0.993342
Compressed File	260.424	55.8295	204.594	1.06682

Gimp

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	247.506	20.2708	227.236	0
Casync	363.372	13.0572	350.315	35.586
Compressed File	300.3	10.1874	290.113	49.7434

Emacs

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	110.331	39.7443	70.5862	0
Casync	177.734	44.7264	133.007	-12.5355
Compressed File	136.6	40.7356	95.8641	-2.49415

Open MPI

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	139.54	3.33596	136.204	0
Casync	214.708	4.38082	210.328	-31.3209
Compressed File	167.518	3.26096	164.257	2.24831

Once again, looking at the closure sizes, we can see we are experiencing a loss in terms of compression efficiency. We have to pay back that loss with an even greater substitution performance to be better or even on par with the NAR-based substitution.

So, does a increase in substitution granularity helps when substituting a single derivation?

The answer is: it depends.

While it’s clear that on a NixOS closure level the gains are almost always considerable, on a single derivation substitution, the gains will greatly depend on the content of the derivation. In the case of a derivation mostly containing text files such as Gimp, we can expect lot of savings. In the case of a derivation mostly consisting of binary data (Firefox, Emacs), the gains are likely to be residual.

Casync is performing consistently worse than the compressed files approach. This can probably be explained by two things:

Loss of the file boundaries: the chunks are generated from a serialized stream containing the whole store path content. It’d probably be a good idea to instead chunk the files individually in our use case.
Shy compression: the Casync closure size is almost twice (!!) as big as the NAR one. While the small size of the chunks can explain to some extent this compression efficiency loss, nix-casync is also using lzma to compress its chunks. Moving to more aggressive xz compression could be a good idea here.

Long Road Ahead

So, what’s the take-away here?

Well, to me, the main one is that the current Nix/Guix substitution bloat story is not helpless. We can expect some serious improvements by increasing the substitution granularity. Be it using a file-based substitution mechanism or using a chunk-based one.

It would probably be worth investing some resources into pushing these substitution mechanisms to a production-ready state. We should also seriously consider using them as the default substitution mechanism once they’re ready. And potentially give up on the NAR-based approach.

Increasing the substitution granularity won’t sadly come for free in terms of complexity. It’ll affect a lot of things, possibly including rethinking the binary cache signature mechanism. We’ll likely need quite some time before being able to confidently replace the NAR-based substitution mechanism. It’s definitely not a weekend project.

Special thanks to Profpatsch for his spot on review.

Full disclosure, while it’s okay to omit them in the context of this article, there are two notable exceptions to this input-addressed nature:
- Fixed Output Derivations (FODs): they are, as their name implies, already output-addressed. We know, at build time, their content hash. We use them to deterministically download stable artifacts from the Internet. EG. bootstrap seed, source code, binaries…
- A content-addressed store mode is available through an experimental feature. See RFC#62 for more informations.
↩︎
Bias Warning: the main author of nix-casync is an online friend. ↩︎