# Abstract¶

We evaluate the performance of a NAR-based, a file-based (uncompressed and xz-compressed) and a casync-based substitution mechanism through 3 scenarios:

1. A curl-induced mass rebuild impact on a NixOS machine closure.
2. A single derivation version bump (Firefox) impact on the said derivation.
3. A stable -> unstable channel jump impact on a NixOS machine closure.

For each of these scenarios, we compare how much data the substitution technique required to transfer versus how much data the NAR substitution required to transfer.

Unsurprisingly, the mass rebuild scenario is the one for which we see the biggest improvement: nix-casync cuts down by 48.7% the amount of downloaded data, the xz-compressed file-based substitution cuts down the same amount by 38.4%.

Surprisingly, we see an improvement in the case of a Nixpkgs stable (21.11) -> Nixpkgs unstable (20.05 pre-release) jump: xz-compressed file based substitution cuts down by 18% the amount of downloaded data, Casync by 17.2%.

We see almost no improvements for the derivation bump scenario: xz-compressed file based substitution 1%, Casync 1%.

For file substitution, we can see that the compression is crutial for the overall performance. Uncompressed file-based substitution is consistently 2 order of magnitude worse than the NAR-based substitution.

We can conclude that reducing the substitution granularity, either via Casync of xz-compressed file-based substitution consistently reduces (1% -> 48.7%) the amount of transferred data in 3 different common scenarios.

# Import Benchmark Data¶

First, let's import the data generated by the ../companeSubsEfficiency benchmark.

# Methodology¶

For each of these benchmarks, we're going to evaluate different store path substitution techniques and compare their efficiencies.

The following benchmarks will consist in building a NixOS machine configuration against 2 Nixpkgs commits. We'll simulate a NixOS machine update from the first commit to the second one.

We're going to evaluate the 3 following substitution techniques:

1. Nar substitution: This is the substitution model currently used by both the NixOS and Guix project. It consists in .tar.xz-ing a full store path. In this benchmark, we'll identify each NAR by its filename, which is derived by the sha256 sum of their content.
2. Casync substitution: This is an experimental substitution method implemented via the nix-casync project. Here, starting from a NAR, we uncompress it and chunk it in smaller bits. In this benchmark, we'll identify each casync chunk by its filename, which is already derived the sha256 sum of its content.
3. File-based substitution: This is a substitution method the Guix project brainstormed around. Basically, each store file would be served separately. In this benchmark, we'll identify these files using the sha256 sum of their content.
4. XZ-Compressed File-based substitution: Similar to the File-based substitution but with each file individually compressed using the xz compression algorithm using the profile 6 extreme.

Note: we're using NixOS/Nixpkgs for all these benchmarks. However, since Guix currently use the same substitution mechanism, you can safely assume the same conclusions holds true for it as well.

# Benchmark Scenarios¶

## 1. Mass Rebuild¶

Let's build the same NixOS machine description using two Nixpkgs commits: one before and one after the staging next 2021-12-03 iteration merge to master. This staging iteration contains, among other things, a curl version bump. That curl version bump triggers a almost entire Nixpkgs mass rebuild: both nix and stdenv are depending on it.

This mass-rebuild scenario represents a long-standing issue in terms of substitution performance.

We can see a massive performance gain for both Casync (48.4%) and xz-compressed files (38.4%). We can also see that compression plays a massive role in terms of substitution performance: the uncompressed files are doing almost 90% worse than the plain NAR substitution.

## 2. Firefox Bump¶

In this scenario, we're going to simulate a Firefox update. We took the Firefox 97.0 -> 97.0.1 bump 7e23a7fb8268f16e83ef60bbd2708e1d57fd49ef as a test example.

As expected, we don't see any gains (~1%) here.

## 3. Unstable to Stable Channel Jump¶

In this scenario, we're going to simulate a stable -> unstable jump for the same NixOS machine configuration we used in the mass rebuild simulation.

I'm surprised on this one! I did not expect any gains, yet here we are, looking at a 18% improvement for compressed files, 17.2% improvement for Casync. While not amazing, it's big enough to be noticeable.