TL;DR: introducing nar-klepto, the context that led to its inception, and some fun experiments we did last week with it.

4 years ago, I moved out of Bayonne to a rural area. My quality of life almost instantly skyrocketed; however, I quickly felt a bit frustrated that I could only discuss computer-related things via online video meetings.

After the 2024 FOSDEM, I realized that if there’s no nerds around, why not invite my online friends at my place? And, for the last three years, I’ve been renting a very nice place close to where I live for a full week. I invite my favorite online friends there. Each year, a few strangers join, friends of friends, usually becoming new friends by the end of the week.

This is nice, it makes me happy every time.

I just came back from this year’s edition. Physically a bit exhausted, but mentally fully energized. A full week of nice discussions, drone and RC glider flying, multiplayer videogames, cooking, eating a bit too much, hot spring water, and bike rides!

The place where we do this is super nice, and pretty remote… and… do not come with any internet connection. The owners do not seem to care at all, it’s their fun vacation house, no need for internet there! We’re probably the only group renting this and caring about any internet connectivity at the same time. We’re probably terminally online.

The only uplink we have comes from a pretty terrible LTE router¹. And it’s not great. If you’re into Guix or NixOS, you probably already see the issue. These distributions are amazing. But boy oh boy, they’re network hungry. And boy oh boy, they dislike not being able to reach to their binary caches. Last year, I spent some time creating a solution for this, and it looks like this:

This is a wonderful retro Panasonic CF-SZ5, coupled with an external USB SSD. Turned on, serving NARs 24/7 for a week. Poor thing. This used to be a pretty cheap setup, the SSD probably cost 3 times more than the laptop it’s attached to these days.

Always use obsolete hardware running bleeding-edge software.

I love the attitude and what it entails: making some effort to extend the lifespan of rare resources stored in these magic machines. Nice hardware setup to boast about, but that’s not really the point of this article. The software it runs is the interesting bit: nar-klepto.

It’s a collection of 3 programs able to fully backup a Guix or Nixpkgs release. Each and every derivation part of it. That’s ~100GB worth of compressed NARs for a Guix release, ~500GB for a Nix one.

Conceptually speaking, you feed it a list of store paths, and it downloads the relevant narinfos and nar. You can retrieve this list of paths for a particular release using the data service for Guix, Hydra for NixOS. The narinfos are stored in a PostgreSQL DB, the nars on a filesystem. Finally, it comes with an HTTP server, able to send the NARs and narinfos to the Nix and Guix clients. It also provides a NixOS module to set up the server part of it and a Nginx reverse caching proxy. The reverse caching proxy comes in handy when people start to download packages not available in the cache. It could happen if an important security release forces people to upgrade their systems to a new Guix/Nix revision. The reverse proxy cache saves some of the metered LTE bandwidth and some substitution time.

Downloading a full release comes at a cost for the project’s infrastructure, especially for Guix, which does not have a sponsored CDN. This tool is wasteful by nature and is not meant for recurring usage. It’s mostly useful for camps, retreats, and conference organisers. That’s an adequate tool to provide a one-off cache for a Guix/Nix event, a pretty inadequate one to host in your personal infra and mirror each and every Guix/Nix release.

Noé Lopez packaged it for Guix. He has been the first external contributor² to the project, adding the ability for the cache to advertize itself to the guix clients via zeroconf. Quite a nice feature, I’m a bit jaleous not to have it on NixOS! A nice coincidence happened though, adisbladis ended up releasing a similar project for NixOS, with a quite different design the same week: nix-cache-beacon. I might add the cache-beacon advertisement directly in nar-klepto in the future.

I’m pretty happy and proud of releasing this tool. Took me a good while… Close to a year. Last year, a good chunk of my online friends and colleagues jumped onto the vibe coding bandwagon. I was intrigued and did not want to miss out, so I joined the hype through this project. It did not work really well for me, and I have not used these tools since then: the experience was not enjoyable for me, and the result, while working, was a bit too trashy for my taste. There was a lot of unnecessary code duplication, each duplicate being slightly different from the others. I dreaded and procrastinated refactoring the thing for a good year, but I finally bit the bullet two weeks ago, cleaned and rewrote a good chunk of it. I enjoyed the process of desloppifying the project, it was fun. That’s good for me, who knows, maybe this will become my job at some point.

I commissioned ³ some artwork from lassulus. The second iteration was the charm! The overall sketchy vibe nicely fits this project’s story.

During the geek event, I had to download a few things from the cache, and the experience was amazing. The substitution is fast, really fast. That’s a combination of two factors:

1. The narinfos are reachable with a very low latency, >3ms, from the cache. It noticeably speeds up cold evaluation.
2. The NARS are available on the lan at a pretty high download speed. Sadly, not saturating a gigabit link. Despite how much andir- loves it, the glinet router needs an upgrade!

At some point, we paired with flokli, and tried to ingest the Guix release into a snix store to see what kind of storage gains we could get from chunking the Nars. Quite a lot it turns out!. The 115GB Guix nar collection ended up being ingested into a 21GB snix store. Quite impressive! We also tried to ingest the Nix Nar collection, but the gains were not as jaw-dropping. We’re not sure what explains this Guix efficiency. Maybe the lzip -> ztsd compression change, maybe there are some very repeated chunks in the cache. We’ll have to investigate more to figure this out.

All this makes me believe there are ways to improve the substitution story of Guix/Nix. Something I’d like to focus a bit on in the near future ;)