AlternativeBit

Nsncd, Anniversary Updates

PicNoir — Mon, 30 Oct 2023 10:00:00 +0200

Last year, flokli and I worked towards re-using TwoSigma’s Nsncd as the main NixOS Nscd daemon.

What is that even about? Well, Nscd is a Glibc daemon that was originally meant to cache the host/user resolution requests. It’s mostly obsolete by now, but on NixOS and Guix, we abuse this daemon to get a stable ABI to load the host NSS modules from a potentially different glibc version. If you have no idea what I’m talking about here and want to read more about that context, you should probably read flokli’s original release blog post.

It’s now been a year since we released the Nsncd host lookups. Since then, Nsncd is used by default on NixOS in place of Nscd. The migration has been mostly bugless. Emphasis on the mostly!

While this switch has been beneficial for most of us, getting rid of the “too-much-caching” numerous bugs, some unexpected side effects appeared. Such as breaking the hostname --fqdn command.

Recently, flokli and I ended up under the same Catalonian roof for a Numtide-organized programming retreat. Kudos to Zimbatm who handled most of the thankless logistics to make this happen <3

After almost a full year of procrastination, turns out that a 10-hour session of focused work was all it took to almost fix these long-lingering issues.

Wait, why did I write almost here? Ah yes. After these 10 initial hours, I kept uncovering a long series of subtle yet real bugs for the next 4 days. Turns out that the third full rewrite of the fix was the charm!¹

What was the root cause of these faulty host resolutions in the end?

Turns out that, even if the gethostbyname and gethostbyaddr libc functions have been long deprecated in favor of getaddrinfo, some pretty critical pieces of software are still using them. You guessed it: the hostname command still uses those.

We originally decided to implement these two functions through getaddrinfo. This was a mistake: the legacy operations do not behave exactly like the new ones.

To fix this bug, we wrote some FFI bindings for these two legacy functions. Then, we replaced our dodgy gethostbyname and gethostbyaddr implementation in Nsncd to use these bindings.

In the meantime, flokli wrote sockburp, a debug tool for Unix sockets. This has been a game changer to detect the wire format compatibility issues. You can read more about this tool in this blog post.

We have not managed to upstream all our changes to the TwoSigma repo yet. NixOS Nsncd is still hosted from our nix-community fork. We did not give up on upstreaming it though and hope to do that in the next few months.

That’s pretty much all. Until next time, happy and serene hacking folks :)

Ah, you’re here for the gory details? Here you go, read these two PRs history. My treat! ↩︎

The Nix, OpenGL and Ubuntu Integration Nightmare

PicNoir — Thu, 20 Apr 2023 10:00:00 +0200

In this article, we’re about to dive into the uncharted OpenGL on Linux waters. After briefly explaining how the OpenGL calls are routed from your application to the GPU, we’ll look at the NixOS special case. We’ll then explore how we can run the OpenGL programs built by Nix on a foreign distribution, such as Ubuntu or Fedora. Finally, we’ll introduce NixGLHost a new approach to solve this problem.

OpenGL/CUDA on Linux?

Everybody, including your favorite pet play video games. Your favorite go-to desktop application is relying on GPU acceleration to render on your screen, and AI is taking over the world. Welcome to 2023. In the last 15 years, GPUs transitioned from being this gamer-specific niche hardware to being this globally-used parrallel platform.

There are a lot of GPU-related APIs in the wild. OpenGL, OpenCL, Vulkan, DirectX, Metal, etc.. In this article, we’ll focus on a very small subset of them, OpenGL and CUDA. Even more precisely, OpenGL and CUDA on Linux. How are these two APIs working?

Linux being Linux, there’s no simple answer to that. In most setups, and for OpenGL alone, there’s no big unified API. There are actually two concurrent ones, acting as an interface between the actual OpenGL program and the desktop environment.

GLX: the legacy one, a Xorg-specific API.
EGL: the new kid in town. The interface is used by Wayland and Android among others.

To be clear: these are API specifications, not implementations. For these two APIs we have multiple implementations. Some are proprietary, and directly written by the hardware vendors. Some others are open source, sometimes implemented by hardware vendors, sometimes by courageous volunteers.

There are currently two major OpenGL implementations:

Mesa: the main OpenSource implementation. It’s actually more of a meta-driver, it includes drivers for a wide range of hardware: Intel integrated GPUs, AMD discrete GPUs, some ARM Mali, and a lot more.
NVIDIA: a proprietary implementation, maintained by Nvidia itself. That’s the optimal way to drive your Nvidia graphics card on Linux.

These drivers are usually split into two parts. A thin kernel-space part is in charge of the low-level communication with the GPU, and a thicker user-space part is in charge of translating the OpenGL instructions into something the small kernel-space part can understand. Conceptually speaking, the kernel space part acts as a proxy to the actual hardware, nothing more.

These driver stacks, on top of providing the low-level code in charge of talking to the GPU, are supplying implementations for various interfaces. Among those, are GLX and EGL. To sum things up, you have a GLX and EGL implementation for each and every driver. EG. libGLX_nvidia.so, libGLX_mesa.so, etc.

Hey, you’re lying! There is a unified API!

I already used OpenGL, I know for a fact that my program is linked against libGL.so, not to one of these weird-ass API!

You’re absolutely right! But what if I told you that libGL.so is actually a lie?

I omitted an important part of this OpenGL stack on purpose: libglvnd. Originally developed by NVIDIA before being maintained by the freedesktop community, this library aims to be the thing your library is going to link against. It’s not doing anything by itself, it just dispatches the OpenGL calls to the most appropriate APIs. Through various heuristics, libglvnd figures out through which interface the OpenGL instructions should go. Roughly speaking, if the target surface is rendered through Wayland, it’ll use the EGL implementation. If it’s rendered through Xorg, the GLX implementation. It also uses some more heuristics and config files to figure out which driver (Mesa, Nvidia, etc.) should be used.

What about CUDA? Well, for this one, we don’t really have to bother with interoperability; NVIDIA is controlling the whole chain, from the user-space SDK to the silicon after all. The overall call graph is much simpler, although harder to integrate. Your CUDA program is linked against the libcuda.so shared library. This library directly communicates with the thin kernel-space driver in charge to send the CUDA operations to the actual GPU.

Overall, from a bird’s-eye view, the overall OpenGL graphic stack looks like this:

How Does Nix Deal With That?

Now that we roughly see how the high-level components fit together in the Linux space, we know there’s a layer dynamically dispatching the graphics calls to the most relevant driver depending on the current context. It works reasonably well. Well enough to power most of the modern days GPU-fueled AI revolution. Sadly, the cost of dynamic dispatching is a loss of determinism.

NixOS

If your heart rate bumped slightly when reading the previous paragraph, chances are you already drank the NixOS kool-aid and are anticipating a fair share of pain integrating this with NixOS. And for good reason! How can we expect a statically-defined Nix closure to work on each and every end-user GPU? We surely can’t include all the existing drivers in the closure, it’d be massive!

Don’t worry too much though, we don’t have to get that hardcore. Spoiler alert: I’m about to uncover a dirty-hack. You know, the dirty-hack rendering this very web-page (if you’re reading this on NixOS).

NixOS has a magical /run/opengl-driver directory. In this directory, you can find all the libs you’ll need to drive the GPU attached to your machine. OpenGL, Vulkan rendering, OpenCL computing, VA-API video acceleration. You name it!

This directory — we’ll call it “link farm” from now on, because, well, that’s what it really is! — is defined on a system level via the hardware.opengl NixOS module. Overall, it means the GPU driver is defined through the NixOS system closure, and exposed at runtime through the /run/opengl-driver link farm. The OpenGL program derivations have no knowledge of the actual OpenGL implementation. They simply link to libGL.so AKA. libglvnd, the generic dispatch layer.

Okay, great, we have the drivers in a weird /run/opengl-driver location containing a link farm pointing to the actual GPU driver.

Now, how can my Nix-built OpenGL program find these libraries?

Well, it’s Nixpkgs! Obviously, we’re injecting them through a wrapper. The wrapper we’re talking about is called addOpenGLRunpath. It injects /run/opengl-drivers to the library path through an LD_LIBRARY_PATH entry. At runtime, Libglvnd will load these libraries and dispatch them the OpenGL calls.

On a foreign distribution, where /run/opengl-driver won’t exist, the dynamic loader simply ignores this library path entry. The wrapper essentially acts as a no-op on a non-NixOS distribution.

In a nutshell:

NixOS stores the host system GPU drivers in a well-known directory.
Nixpkgs injects this well-known directory to the OpenGL program library path through a wrapper.

What About Running Nix on a Foreign Distro?

Now, how would you run a Nix OpenGL program on a foreign Linux distribution such as Ubuntu?

All of a sudden there is no more /run/opengl-driver. This is where it gets hairy!

The go-to solution these days is NixGL. It’s a wrapper you apply on top of an OpenGL derivation. With a few clever heuristics, it figures out which driver your host system expects at build-time. It then builds the relevant driver derivation (Mesa, NVidia, Bumblebee) and generates a wrapper injecting the driver dynamic libraries to the OpenGL process load path.

Remember what I told you during the introduction: The GPU driver is split between a thick user-space library and a thin kernel-space module. What we have here is only the user-space side of the story. The user-space part of the driver is distributed through Nix, the kernel-space one by the host distribution. One might fear hitting some protocol-level incompatibilities between the user space driver and the kernel-space one. And for good reason! It could definitely happen. In practice though, it’s good enough: the NixGL heuristics tend to be fairly clever at figuring out which userspace library to use.

There’s definitely a tradeoff. The GPU driver ends up living in the Nix closure. All of sudden, your Nix closure is tailored towards a specific GPU and won’t be generic anymore. This approach requires you to precisely know the target hardware specification during the Nix build time. If you want to deploy an application on N computers, you’ll have to conceptually nix build the application N times. You’ll need a detailed inventory of the hardware you want to deploy on, or have the build happening on the end-user machine, which is not always possible.

Kludgy! Mikey Likey!

Could we have an alternate approach to solve this Nix-OpenGL-on-a-foreign-distro problem? An alternative not requiring us to include the OpenGL driver part of the deployed closure, but rather have this managed by the host distribution? Kind of what we have on NixOS?

I think it’s fair to assume the host distribution is smart enough to properly set up the graphic stack. What if we glued the host user-space GPU environment to the Nix program instead of using a Nixpkgs-provided one? This thought was the starting point of NixGLHost, a new way of running your OpenGL Nix programs on a foreign distro.

This approach is pretty Nix-heretic. We’re injecting some host DSOs¹ into the Nix closure. Nix is specifically designed to limit this kind of interaction with the host system! That being said, we’re being heretic but cautious: we only inject the entry points for EGL/GLX/CUDA. We patch the entry points’ ELF runpaths to point to a place containing their own dependencies. This acts as a minimal sandboxing setup.

I won’t dive too much into the implementation details here, you can check out the internals.md file if you’re interested in the nitty-gritty details.

Overall, this experimental new wrapper re-uses the drivers provisioned by your host distribution. The kernel-space module and user-space library are both provisioned by the host system, we’re sure they won’t get out of sync. The graphic driver stays out of the Nix closure, the closure stays truly generic and will work on any GPU.

This experiment turned out to work great for the use cases we tested it against so far. I managed to run DaVinci Resolve and a couple of videogames built by Nix on a Ubuntu machine. A customer is actually already using the prototype in production².

What’s Next for NixGLHost?

The next obvious step is to extend the supported OpenGL implementations. At the moment, we only support the NVIDIA proprietary driver. It’d be great to support Mesa as well. Technically, it’s not a hard thing to do. Sadly, the Python prototype codebase is quite messy and difficult to extend as it is.

The wrapper is also pretty slow to run, ~200ms on a hot cache. Most of this time is spent spinning up the CPython interpreter. Starting a whole interpreter for a very short-lived program turned out to be very costly, who could have guessed! Should we embrace the meme and RIIR the project? I personally think it might be a good idea and started to work on that.

We also should think about improving the sandboxing. At the moment, we’re injecting the host DSOs¹ symbols directly into the program’s global symbol table. If, by misfortune, the Nix program ends up using the same library as the GPU driver but with a slightly incompatible version, all hell breaks loose! There are some ways to go around that, we could leverage the dlmopen function to load the driver DSOs¹ to another namespace. We could potentially leverage libcapsule to perform such a “namespaced” dynamic library loading.

Let’s see how it goes, I’ll probably post a follow-up here when we’ll reach the next major milestone :)

Greets:

flokli for enduring my terrible hacks and brainstorming this problem space with me <3
Numtide and Otto Motors for trusting me on this and funding most of the NixGLHost work <3
Guibou for writing NixGL <3
Alex and tazjin for proof-reading this article <3

Dynamic Shared Object, a shared library. ↩︎ ↩︎ ↩︎
This is an alpha release, I’m not saying you should use it in production as well. Use this with caution, it may eat your kittens! ↩︎

Nix Substitution: the Way Forward

PicNoir — Thu, 31 Mar 2022 08:18:00 +0200

Abstract

Nix and Guix have the unfortunate reputation to require a lot of bandwidth to distribute software. This reputation is sadly grounded. It seems like explicitly pointing to your dependencies comes with an overhead cost in terms of download size. Or does it?

Currently, both Guix and Nix can substitute pre-built store paths from a binary cache using the Nix Archive (NAR) format. After conducting several benchmarks, we conclude that the substitution granularity should be finer to improve the substitution performance. We can increase the substitution granularity either by substituting files individually or by substituting chunked individual files.

If you are already familiar with the Nix and Guix substitution mechanism, feel free to ignore the next two sections and directly jump to the “Rethinking The Substitution Granularity” one.

You Said Substitution?

We usually don’t want to rebuild the whole world locally. When we want to install Firefox on a laptop, we’d rather not build it from the source on the machine itself but instead download a pre-built version from a cache. For this purpose, Nix provides a store path substitution mechanism.

Instead of building a derivation locally, we substitute the local build with a pre-built artifact stored in a binary cache. The store paths are getting transferred between the binary cache and the local machine using the NAR archive format. This format can be seen as a compressed store path. Meaning we’ll have a NAR archive per top-level store path we’d like to substitute.

A NAR is always identified using a hash derived from its content. NARs are content-addressed.

In contrast to that, both the Nix/Guix derivations and the store paths they generate are input-addressed¹: their hashes are directly derived from their dependencies hashes.

You now see the issue: on one hand, we have input-addressed derivations and store paths, on the other hand, content-addressed NARs.

Starting from an input-addressed derivation hash, how are we supposed to figure out the name of the content-addressed NAR we want to download? In other words, how can we figure out the output hash of a derivation without building it first?

Introducing the NARinfo file. Let’s take an example:

» curl https://cache.nixos.org/r5sjd57x0r07bwgipryaxqdkx1gglhiy.narinfo

StorePath: /nix/store/r5sjd57x0r07bwgipryaxqdkx1gglhiy-libidn2-2.3.2
URL: nar/010zh5j819qyz6zai6h2hn3d7i406g00m2g4g62bq7m8g91553yb.nar.xz
Compression: xz
FileHash: sha256:010zh5j819qyz6zai6h2hn3d7i406g00m2g4g62bq7m8g91553yb
FileSize: 63248
NarHash: sha256:1npw0jz1cw4k9x25f2vsdhsa5cf9568j46bpz2768a1izqj5n9lf
NarSize: 260816
References: 0z7sqj4pilbqyp45ix5b0mdgn9xlb024-libunistring-0.9.10 r5sjd57x0r07bwgipryaxqdkx1gglhiy-libidn2-2.3.2
Deriver: 0n91syjwrhmng41f8d23ad0sl4a6ic4g-libidn2-2.3.2.drv
Sig: cache.nixos.org-1:G9PxMT0/nd9ELwL3BBeqWtb2ohMiqw4T4FUwFlAU9M0E45mKg77BzL9gJQ3wH4oYtsa61MV5uzNLPFvK8ZbyCA==

To substitute the path /nix/store/r5sjd57x0r07bwgipryaxqdkx1gglhiy-libidn2-2.3.2 from the cache.nixos.org binary cache, we’ll first have to download its associated NARinfo file. That NARinfo file is addressed using the same input-addressed hash as its associated store path: r5sjd57x0r07bwgipryaxqdkx1gglhiy. The NARinfo will contain some metadata needed to perform the substitution, but also, and most importantly, the URL from which we can download the content-addressed NAR.

To sum up, to perform a substitution, the Nix/Guix client will:

Compute the input-addressed store path of the derivation.
Fetch the associated .narinfo from the binary cache it wants to substitute from.
Download the content-addressed NARs referenced by the .narinfo.
Unpack the downloaded NARs to the local Nix store.

Substitution Hell

Now that we established how the Nix substitution mechanism works, let’s have a thought experiment: what will happen if we bump the version of a derivation living deep in the Nixpkgs/Guix dependency tree? Something like curl.

The curl store path being input-addressed, its name will change for sure. Likewise, its dependencies are also input-addressed, their name will also change. Overall, the curl store path change will propagate throughout all the Nixpkgs/Guix dependency graph.

You might say:

Hey! That’s not a big deal.

The NARs are content addressed, this change of input hash shouldn’t propagate there.

The binary cache might have to re-build the whole dependency graph, however, outside of curl, the content of the derivations should remain the same. So, in the end, most of the NARs should remain the same. The clients shouldn’t have to re-download all the store paths depending on curl.

We won’t have to re-download massive derivations like Firefox and Chromium for a simple curl bump.

Right?

And you’d be right except for one thing: in the Nix/Guix model, everything is explicit. Every time you’ll refer to the curl binary, you’ll have to explicitly refer to its store path /nix/${inputHash}-curl/bin/curl. Its input-addressed store path! This sneaky reference will manage to propagate the input-adressed instability to the output-addressed NARs.

Coming back to your example: both Chromium and Firefox depend on curl. You’ll have to re-download both of these massive store path for every curl bump. Hopefully, you won’t mind downloading a couple of gigabytes for a sub-kB string change? Do you? :P

One has to think: could we improve this situation?

Rethinking the Substitution Granularity

Content addressing the substitution archives is definitely a good idea. It brings some hash stability to this unstable input-addressed scheme. However, as we saw earlier, because of the store path references, the input-addressed hash instability ends up propagating through the substitution mechanism.

To improve the substitution stability, we could think about two solutions:

Remove the store references from the NAR and replace them on the fly when decompressing the NAR into the actual Nix/Guix store.
Increase the substitution granularity.

I’m going to ignore that first solution for this post and focus on the “Increase the substitution granularity” solution.

The Nix and Guix communities came up with two solutions based on this finer granularity substitution idea: nix-casync and a file-based substitution discussed in the Guix mailing list.²

Living in the countryside, my only Internet connection is a weak LTE link. I have 3 NixOS and 1 Guix system at home. Any mass-rebuild takes me at least 2 hours to substitute on a good day.

Needless to say, I have a personal urge to improve that situation!

Before blindly stepping into what looks like a multi-year-long sustained implementation effort, I wanted to gather some hard data showing how much we could gain from increasing the substitution granularity. It’s benchmarking time.

Benchmarking Fine Granularity Substitution Mechanisms

For various scenarios, we’re going to build a Nix closure for 2 Nixpkgs commits. Then, we’ll simulate an update from the first commit to the second one and see how much data we’ll need to download from the binary cache.

For each scenario, we’re going to evaluate the 4 following substitution techniques:

Nar substitution: This is the substitution model currently used by the NixOS and Guix projects, it’ll act as our metrics baseline. It consists in archiving and compressing a full store path. In this benchmark, we’ll identify each NAR by its filename derived from the sha256sum of its content.
Casync substitution: This is an experimental substitution mechanism implemented via the nix-casync project. Starting from a NAR, we decompress it, serialize all the files in a single stream, and chunk it into smaller bits before compressing them again. In this benchmark, we’ll identify each casync chunk by its filename, itself already derived from the sha256 sum of its content.
File-based substitution: This is the substitution method the Guix project brainstormed around. Each store gets substituted separately. In this benchmark, we’ll identify these files using the sha256 sum of their content.
XZ-Compressed File-based substitution: similar to the File-based substitution except each file is individually compressed using the xz compression mechanism (profile 6 extreme).

If you’re interested in the nitty-gritty details, the source code and the raw data behind these benchmarks live in this git repository. You can also have a look at the Jupyter notebook I used to crunch the benchmarks data.

Note: we’re using NixOS/Nixpkgs for all these benchmarks. However, provided Guix currently uses the same substitution mechanism, you can safely assume the same conclusions holds for it as well.

Mass Rebuild

Let’s kill the suspense right away and have a look at the most interesting scenario first. We’re going to simulate a NixOS closure update after the staging-next 2021-12-03 merge containing, among other things, a curl update. As we already established in the “Substitution Hell” section, a curl bump will propagate through the whole NixOS dependency graph. That’s the worst-case scenario in terms of substitution: the mass rebuild.

We’re going to use a NixOS machine description depending on several diverse language stacks (C, Python, BEAM) and simulate a closure update after this mass rebuild.

Let’s see how the four substitution techniques perform:

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	386.06	372.99	13.0702	0
Casync	608.652	192.195	416.457	48.4719
File	1652.19	705.665	973.348	-89.1915
Compressed File	476.523	229.489	247.034	38.4732

First, the good news: increasing the substitution granularity definitively works! The casync-based substitution almost cuts by half the amount of data we need to download for this mass rebuild.

YAY!

The xz-compressed file-based substitution sits in the same ballpark, saving us ~40% worth of data to download.

We can already see two other things from this experiment:

Without proper compression, the file-based substitution performs about half as good as the NAR substitution.
Looking at the closure size of the various substitution methods, we can realize that the finer the substitution granularity gets, the worse the compression algorithms are performing. In other words, the massive gain in terms of re-used data/downloaded data ratio (~75% for casync) does not directly translate into download savings. We probably want to spend some time investigating the state-of-the-art WRT. compression algorithm specialized in small files.

Provided these results, I’ll omit the uncompressed file substitution-related data for the next benchmarks: it’s consistently worse than the compressed counterpart in terms of performance and hurts the readability of the diagrams.

Channel Jump

In this scenario, provided the same NixOS machine closure we used in the Mass Rebuild scenario, we’ll simulate a jump from the NixOS-21.11 stable channel to NixOS-unstable.

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	387.458	375.167	12.2905	0
Casync	610.886	310.541	300.345	17.2261
Compressed File	478.12	307.479	170.642	18.0423

I did not expect much savings here, but surprisingly enough, we’ll shave about 18% worth of downloaded data using the compressed file-based substitution or casync.

This scenario does not make total sense from a pragmatic point of view as-is: you rarely jump a NixOS machine closure from a stable to unstable NixOS channel. It however does make sense in the context of a dev machine. You often have to jump between different projects having their own dev nix-shell closures pointing to different Nixpkgs channels.

The good and unexpected news is: even in that scenario, we can expect some substantial gains in terms of the amount of data we’ll need to download!

Single Derivation Bump

Now, one may ask: can we expect any savings for a single derivation substitution?

Ludovic Courtès recently posted a summary of his findings on that matter in the Guix mailing list for various binaries containing various amounts of binary data and text.

Let’s replicate this experiment with our benchmark. We’re going to simulate a version bump for Firefox, Gimp, Emacs, and OpenMPI. To do so, we’ll build the two derivations before and after the version bump, and from there see how much data we’ll be able to re-use and how much data we’ll need to download.

Firefox

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	219.352	56.4315	162.92	0
Casync	342.925	55.871	287.054	0.993342
Compressed File	260.424	55.8295	204.594	1.06682

Gimp

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	247.506	20.2708	227.236	0
Casync	363.372	13.0572	350.315	35.586
Compressed File	300.3	10.1874	290.113	49.7434

Emacs

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	110.331	39.7443	70.5862	0
Casync	177.734	44.7264	133.007	-12.5355
Compressed File	136.6	40.7356	95.8641	-2.49415

Open MPI

Name	Closure Size (MB)	Downloaded Size (MB)	Re-used Size (MB)	DL Savings Compared to NAR (%)
NAR	139.54	3.33596	136.204	0
Casync	214.708	4.38082	210.328	-31.3209
Compressed File	167.518	3.26096	164.257	2.24831

Once again, looking at the closure sizes, we can see we are experiencing a loss in terms of compression efficiency. We have to pay back that loss with an even greater substitution performance to be better or even on par with the NAR-based substitution.

So, does a increase in substitution granularity helps when substituting a single derivation?

The answer is: it depends.

While it’s clear that on a NixOS closure level the gains are almost always considerable, on a single derivation substitution, the gains will greatly depend on the content of the derivation. In the case of a derivation mostly containing text files such as Gimp, we can expect lot of savings. In the case of a derivation mostly consisting of binary data (Firefox, Emacs), the gains are likely to be residual.

Casync is performing consistently worse than the compressed files approach. This can probably be explained by two things:

Loss of the file boundaries: the chunks are generated from a serialized stream containing the whole store path content. It’d probably be a good idea to instead chunk the files individually in our use case.
Shy compression: the Casync closure size is almost twice (!!) as big as the NAR one. While the small size of the chunks can explain to some extent this compression efficiency loss, nix-casync is also using lzma to compress its chunks. Moving to more aggressive xz compression could be a good idea here.

Long Road Ahead

So, what’s the take-away here?

Well, to me, the main one is that the current Nix/Guix substitution bloat story is not helpless. We can expect some serious improvements by increasing the substitution granularity. Be it using a file-based substitution mechanism or using a chunk-based one.

It would probably be worth investing some resources into pushing these substitution mechanisms to a production-ready state. We should also seriously consider using them as the default substitution mechanism once they’re ready. And potentially give up on the NAR-based approach.

Increasing the substitution granularity won’t sadly come for free in terms of complexity. It’ll affect a lot of things, possibly including rethinking the binary cache signature mechanism. We’ll likely need quite some time before being able to confidently replace the NAR-based substitution mechanism. It’s definitely not a weekend project.

Special thanks to Profpatsch for his spot on review.

Full disclosure, while it’s okay to omit them in the context of this article, there are two notable exceptions to this input-addressed nature:
- Fixed Output Derivations (FODs): they are, as their name implies, already output-addressed. We know, at build time, their content hash. We use them to deterministically download stable artifacts from the Internet. EG. bootstrap seed, source code, binaries…
- A content-addressed store mode is available through an experimental feature. See RFC#62 for more informations.
↩︎
Bias Warning: the main author of nix-casync is an online friend. ↩︎

Ex-Hack: a Haskell Example-based Documentation

PicNoir — Thu, 15 Nov 2018 13:49:00 +0100

Abstract

Ex-Hack is an example-based documentation automatically generated using the packages posted on Stackage. There’s a live demo here. We’ve just released the alpha version; you can have a look to the code here.

We are actively looking for new contributors.

After briefly introducing the project’s incentives and explaining how this software works, we discuss the current roadmap and what we need to do before releasing the V1.0.0.

An Effort to Fill the Documentation Gap

Accordingly to the 2017 Haskell survey results, the lack of good documentation seems to be a major issue in the Haskell community.

How can we fix this? The obvious solution is to write more documentation. However, writing more documentation does not necessarily mean writing a better one. The best documentation writers I met are not necessarily the best programmers I met. Being able to write a clear, concise and useful documentation is a very specific skill that takes time to acquire.

However, there’s one thing you can add to your documentation requiring very little writing skill and still being extremely useful: examples. We, as humans, tend to often learn by mimicking other’s behaviour. Adding examples to a documentation is in my experience a local maximum: it’s not really hard to do, yet it dramatically increases your documentation’s quality.

This is where I started thinking to automatically generate these examples. The Haskell community is fortunate enough to have a centralized code repository: Hackage. The packages stored in this repository are usually getting their dependencies from this very same repository. Maybe we could use all this code to extract some real-world examples.

It would be great if on top of the current type documentation we could add an automatically generated example section in Haddock.

This was the starting point of the whole ex-hack project: generate an example-based documentation we could later use in Haddock.

Let’s get this straight right away: we still have a lot of work to do before merging this project to Haddock, but we made some serious progress in the last 5 months!

So, How does it Work?

What do we need to generate such a database?

Well, basically, for each Stackage/Hackage package, we need to:

Download it.
Build it.
Extract the symbols exported by its main library.
Index the symbols coming other packages.
Generate a static HTML documentation displaying the previously extracted informations.

The whole project is centered around the processing step abstraction. Each processing step is dependent from the previous one and is a dependency for the next one.

This means that ideally, we could run this software in a map/reduce configuration and distribute the load over several nodes. This comes handy, because as stated previously: we’ll need to build the whole Stackage/Hackage. Needless to say, this is a seriously time-consuming process. The more we can distribute the load, the faster we’ll generate the documentation.

So far, we’ve been able to generate the documentation for Stackage (~2,400 packages). The last run (which populated the current demo) took ~25 hour to complete on my 7 YO desktop. If we want to scale this project to the whole Hackage (~13 000 packages) repository, we’ll probably need to distribute this generation on several nodes.

Project’s Roadmap

Which bring us to the million dollar question: what’s next?

Before entering in the beta phase, we first need to address several issues making ex-hack documentation not fully usable:

First, we need to re-write the symbol occurrence indexing system. The current approach clearly does not work.
We need to setup a proper environment to execute the whole database generation. Some packages need some system dependencies (such as libgl, zlib, etc.) to be built. We are currently unable to build them. Hence, we are indexing 2082 packages instead of the ~2,400 contained in Stackage.

Then, it’ll be time to address some usability issues before releasing the first stable version:

Writing a second UI targeted to package maintainers; showing them what are the most used parts of their API. It could come handy when performing a major API refactoring ;)
Writing a Haskell IDE Engine plugin.
Partially re-write the current HTML documentation to make it more pleasant to use.

Ultimate Goal

Once ex-hack will reach its definitive form, I would love to see this merged back to Haddock. I think having usage examples next to the type and author’s comments for each library symbol would make a great documentation.

However, entirely merging this to Haddock is going to be challenging. First, we would need a new Haddock component in charge to build the ex-hack documentation for every new package/stackage release.

We would also need a way to retrieve an already populated ex-hack database before generating a Haddock documentation for a library: you can’t expect to build the whole Hackage/Stackage on a developer’s machine!

Anyways, looks challenging, but you always need a long-term goal dream right? :D

How to Join Us?

So far, I’ve been mostly working on this in isolation; which was arguably not my best move. The more people will work on this, the more progress we’ll make.

Most of the annoying early project exploration has already been made, we mostly know were we are heading now. We mostly need to “industrialize” the project. You don’t need to be a Haskell guru to contribute, as a matter of fact I’m far from being one.

For now, the GitHub repository will act as the main community hub. Pick an issue — some are labeled as newcomer-friendly — and just start hacking on it :)

If you have some questions/suggestions or just want to say hi, there’s also a more informal communication channel: the #ex-hack room on the Freenode IRC network.

Ultimate Writer: an Open Digital Typewriter

PicNoir — Wed, 17 Oct 2018 09:37:05 +0200

TL;DR: A digital typewriter based on a Raspberry Pi and an E-Ink screen. The code/build instructions are available on GitHub.

I am easily distracted.

This is both a blessing and a curse. On one hand, I can deal with a large amount of boredom without driving crazy: remaining artifact from my school years and necessary skill to survive in our meeting-based modern corporate environment. On the other hand, completing a task requiring more than two minutes of my attention can turn into an escape game.

My behavior is far from being exceptional. As a matter of fact, some dodgy people are making entire careers selling people like me books with clickbaity titles.

Overcome your Procrastination and Setup your own Ponzi Scheme in 30 Days.

The definitive guide.

- PicNoir 2018

More seriously, I am a firm believer that restriction fuels creativity. I don’t think it’s the result of a transcendental zen achievement, this is more about restricting the possible creative outcomes. The less you have to explore what you can create, the more headspace you have to actually create things.

This is why I love my typewriter. You can write stories without having to wait for your computer to boot, you cannot be distracted by some kind of notification or the urge to check your favorite online place every half an hour.

However, using a typewriter comes with two major shortcomings:

You cannot delete a typo. I know you do a lot of those, no need to expose this nasty behaviour to everybody’s face.
You cannot share the text you wrote online, be it on your website or on any other social platform.

I came to think, maybe we should create a digital typewriter giving us the best of the two worlds.

Digital typewriters are not a new idea. In fact, one of them has even been commercialized: the Freewrite. To be honest, this project has been a major inspiration for the ultimate writer. It looks cool, has a mechanical keyboard and an e-ink screen. Sadly, their firmware is completely closed (ie. no way to use vim on it), and the price tag is pretty steep, to say the least.

After a quick search on the internet, I was unable to find an open hardware equivalent. I decided to create one: the ultimate writer.

The First Prototype

This project was built upon the following principles:

Easily readable e-ink screen. You can read it effortlessly even in sunlight.
Long lasting battery life. You can have a 3 days off the grid writing retreat (~20 hours) without having to recharge it.
Easily serviceable design. Your typewriter is 40 years old and works just fine. You don’t want to change your writing device every 5 years. You want to be able to easily change the computer parts easily; and who knows, use something else than a raspberry pi.
Standard OS. You want to use your favorite console-based text editor. You also want a shell access to tweak your setup without reprogramming the device.
Nice full-size mechanical keyboard.

The first prototype covers most of these features.

Overall, it looks like this.

It cost me around 150€ (~180$) to build.

The mechanical keyboard is a 61 keys Chinese bootleg. It feels pretty good despite its really low price.

The E-Ink screen is coming from good-display. They don’t seem to sell them directly, but you can buy them from waveshare, a Shenzhen-based middle-man.

The case has been made from a locally grown cypress. I did not use any varnish, this awesome pink-ish color is the natural one. I love this wood <3.

The computer is running a standard Raspbian, I mostly use neovim to edit text.

The battery life turned out to be more around 16 hours. However, the Raspberry Pi 3B is not known for being energy-savvy. I guess we could easily reach the 20 hours autonomy by using a smaller computer.

Build Log

Nerdy stuff ahead. If you’re not interested in the technical details, you can directly jump to the what’s next section.

I started the whole prototype build with an assertion in mind: I need to reach a usable state for this project as quick as possible. I actually abandoned this very same project halfway through twice in the past.

In other terms, when I encountered a roadblock - and there have been many - I privileged the quick workaround to the labor expensive clean solution.

E-Ink Screen

First of all, we need to find a suitable E-Ink screen. Despite being old, this technology is still a niche. Finding a manufacturer able to create large (> 4") screens turned out to be tricky.

Spare E-Readers screens are the best lead: they are crazy cheap, decently sized and widely available. However, there is no off the shelf solution to drive them.

We desperately need an open source driver for those screens, I considered creating one. But as mentioned earlier: I want to build a usable prototype as quickly as possible, I don’t wanna be side-tracked.

Being unable to use a spare e-reader screen, I went the other way around and started looking for a manufacturer selling a screen which is directly usable by a stock Raspberry PI.

In the end, only one screen manufacturer seemed to be delivering that kind of screen: good display. These screens - like a lot of Chinese products - are rebranded all over the internet. Sometimes, they are re-branded and have half their datasheet butchered in the process - yes, I am looking at you Waveshare.

These screens are coming in three parts:

A Raspberry Pi hat.
A small IC connecting the hat to the screen.
The screen by itself.

Despite how it looks, both of the blue CIs are breakout boards. The driver is embedded in the screen and is controlled by a standard SPI interface.

These screens are easy to use. Sadly, they have very a slow refresh rate: it takes around 3 seconds to perform a redraw. Needless to say, it’s way too slow to display any kind of interactive program.

These screens are driven using electromagnetic waveforms, Ben Kraznow from the Applied Science youtube channel made a fantastic video about this. You really should check it out, at least the first half.

Very quick and incomplete summary: in order to refresh the screen, you apply a specific waveform to each pixel depending on its previous and future state.

What’s great about good display screens is that you have a direct access to the registers specifying these waveforms. You can entirely reprogram the way the screen is driven.

Ben Kraznow found a way to implement a faster partial refresh which let you refresh the screen in about 0.3 seconds, which is more than enough for a text editing device. He implemented this for the 4.3" screen, I sadly haven’t been able to implement the same kind of fast refresh for the 7.5" screen.

Note: I’m going to dive in the nasty implementations details here and explain why we can’t have a fast refresh on the 7.5" screen for now. You may want to jump to the next section if you’re not interested.

Under the hood, these waveforms are specified using several look-up tables (LUTs). You have 4 of them: black->black, white->white, black->white and white->black. These LUTs are specific to each screen design.

You have two ways to specify these LUTs to the screen controller (take care, waveshare data sheets are incomplete, use the good display ones previously linked instead).

You can either:

Use the pre-loaded LUTs in the IL0371 read-only flash.
Use the LUTs stored in the IL0371 register which can be accessed using the SPI interface.

Good display stock firmware uses the second method to specify the LUTs for the smaller screens (< 7.5"). This is kind of useful when it comes to write a partial refresh mode because it means the LUTs are specified directly in raspberry pi library source code. We just have to extract them from there.

Ben Kraznow basically took these full refresh LUTs and removed all the intermediate drawing steps to only keep the last one. This solution works pretty great, you just still need to perform a full refresh periodically to correctly re-magnetize the burned greyish pixels.

I was first planning to do the same thing for the 7.5" display. Sadly, for some reason, good display is pre-loading the LUTs in the read-only flash for this model. We can’t extract the full refresh LUT from the driver firmware.

This leaves us with two solutions:

We manage to dump the internal screen flash memory. This should be doable using the ROTP command. Sadly, I haven’t been able to read anything from my screen. I cannot tell if the problems come from a failure from my specific hardware setup or if somebody in the supply chain (waveshare? good display?) has been somehow disabling the MISO channel.
We ask a manufacturer for the LUTs. I have been in contact with good display engineers, but they have been unable to give me access to either the LUTs or the data flashed in the IL0371.

I tried both of these solutions, neither worked. This has been the main roadblock for this project. So far, I just use the “slow refresh” and deal with the .3Hz refresh rate.

But I’m not gonna lie, while it’s enough to write text, it’s not remotely fast enough to edit the same text. Using vim interactively is not as nice as on my laptop’s 60Hz LCD screen.

Terminal Emulation

Alright, we can draw some pixels on the screen, let’s display something more useful: a terminal.

Have you ever taken a deep dive in terminal emulation? Did you manage to keep your sanity during the process? Fun fact: I did not.

Terminal emulation is a pile of legacy layers stacked during the last 40 years. Despite knowing that, if you’re still interested by the details, Linus Åkesson - yes, the very same guy who created the Chipophone, what a small world! - wrote a pretty good article on this matter.

Writing a full terminal emulator is actually a lot of work! So instead of starting a new one from scratch, I decided to adapt an already existing one.

When thinking about terminal emulators, XTerm comes first in mind. I started to dig in its code to be sure the adaptation wouldn’t turn into an unpleasant experience.

Oh boy, if you ever wondered how painful can legacy code be, please, check out XTerm: it is a masterpiece in its own kind.

Actually, I could have been smart and read the readme file instead:

Abandon All Hope, Ye Who Enter Here


This is undoubtedly the most ugly program in the distribution.  It was one of
the first "serious" programs ported, and still has a lot of historical baggage.
Ideally, there would be a general tty widget and then vt102 and tek4014
subwidgets so that they could be used in other programs.  We are trying to
clean things up as we go, but there is still a lot of work to do.

If you are porting this to a machine that has problems with overlapping
bcopy's, watch out!

There are two documents on xterm: the man page, xterm.man, which describes
how to use it, and ctlseqs.ms, which describes the control sequences it
understands.

What do we learn here? Always read the README file first.

Anyways, after looking a bit for something else, I found out the Suckless team created a terminal emulator: ST. I decided to use it.

I started by chopping out the XOrg related features, I did not want an unnecessary dependency on X libs. After chopping everything out, I started to replace these calls with the E-Ink drawing primitives.

The e-ink screen stock library was written in C++: it needed a full C rewrite to function properly along with the ST codebase. I found and fixed several bugs in the process, most of them were due to poor error handling.

Input Processing

Keyboard inputs, how do we capture them? Well, there’s no immediate answer to that.

See, we don’t have any graphic stack redirecting the keyboard inputs to stdin. The keyboard events are handled by linux’s /dev/inputX virtual devices. You may think “well, just pipe this file in and you’re good”. Well, no. This file is using a custom event-based binary format. In it, you’ll find the various key press, key releases, and other events, but you’ll need to process them accordingly to a global state - is shift pressed? Is ctrl pressed? - and translate these events to their characters and pipe everything to stdin.

The best solution I ended with was to use a keylogger and tail -f the file to the application stdin. It could be certainly better, but it is good enough for now.

Not really tricky but worth mentioning: there isn’t any arrow key nor ctrl ASCII value. These are actually translated to VT100 special sequences. Keep in mind you need to translate these specials keys to their associated sequence.

The Case

Gotta admit I have been lucky here. I’ve been offered some locally-grown dry cypress to build it. Thanks dad! This wood is amazing: it looks nice, it’s crazily light and really tender. Working with it has been such a pleasure!

Having access to a whole woodworking workshop also made that build really easy!

I wanted to use dovetail joints to seal the case. Sadly, I ran out of time and had one day to complete both the design and the build of the casing; way too little time to perform 8 dovetail joints.

We used a miter saw to cut the 45° joints. I usually use a good old handsaw for this kind of work, needless to say, the miter saw is a total game changer!

I found the hinges in my local hardware store. The compas were more tricky to find due to their small size; I managed to find mines on eBay.

The case could clearly be better. But for a first iteration, it’s more than ok.

We could do better on an ergonomics standpoint though. Maybe by adding a foldable wrist rest? As it is, the keyboard is a bit too high and tends to put some tension on your wrists making typing a bit uncomfortable on the long run.

Anyways, overall, I am satisfied with this design.

What’s Next?

I used a Raspeberry Pi 3B for this prototype mostly because I already owned one. While having an integrated WIFI module has been quite handy, the power consumption of the computer could be lower. I don’t think we need something “as powerful” as this for a simple typewriter. For the next iteration, I think we should evaluate some other small boards and choose something more appropriate.

I am using regularly this typewriter. As a matter of fact, I am typing this article on it right now.

However, I find the current screen refresh rate way too low. Don’t get me wrong, it is usable, but I feel like the latency makes the whole experience not as pleasant as it could be.

My middle-term objective regarding this project is to produce a small batch of those with a more compact wooden case and sell them. Sadly, because of the low screen refresh rate, I don’t think we’re at this point yet.

I need your help to finish this project. How could you help me? Well, several ways to do this:

Are you one of these hardware geniuses? Do you think you might be able to create a board that can drive a spare kindle screen at a >2Hz rate? Would you be ok to put your design under an open hardware license? Please get in touch with me and let’s finish this open hardware/FOSS digital typewriter together! You can find my contact details on the about page.
Do you know such a person? Please, forward them this article :)

Otherwise, you still can boost this fediverse post or this tweet to help me reach this kind of people.

Special thanks to Malix for the nice pictures! <3

Loading a Cabal module in the GHC API

PicNoir — Wed, 08 Aug 2018 12:01:00 +0200

If you plan to build some Haskell tooling or any kind of static code analyzer, chances are you’ll need to use the GHC API at some point.

While loading a simple module into GHC’s API is quite trivial and well documented, loading complex modules (modules having some c dependencies, some specific options in the .cabal file, etc.) will require you to find the appropriate dynamic flags. These flags are usually retrieved and loaded into GHC by Cabal. Sadly for us, Cabal’s API does not seems to expose a direct way to get these flags.

Some people have developed solutions to work around this problem. In this post, we’ll explore two of them.

Revision: If you plan to use GHC >= 8.6.1, you might as well want to check out GHC source plugins. Mpickering wrote about them. They will not be mentionned in this article.

GHC Plugin + “GHCi Wrapper”

This technique has been detailed by Edward Yang on his website.

The idea basically boils down to this:

A more reliable way to integrate a GHC API program with Cabal is inversion of control: have Cabal call your GHC API program, not the other way around!

[..]

What we will do is replace the GHC executable which passes through all commands to an ordinary GHC, except for ghc –interactive, which we will then pass to the GHC API program. Then, we will call Cabal repl/stack repl with our overloaded GHC, and where we would have opened a GHCi prompt, instead our API program gets run.

You’ll first need to write your GHC API program as a GHC frontend plugin. Then, the dependencies will be built by Cabal using a regular GHC while your package will be loaded with your custom frontend plugin. You can then use GHC’s API without having to think about any dynamic flags: they already have been loaded for you by Cabal.

Pretty smart!

You’ll still need to get the data extracted by your GHC API program home. Sadly, your GHC frontend plugin will be instantiated in another process, you won’t have a direct access to it. I guess you could use one of the usual suspects (stdout, a message queue, a DBMS, etc.) to bring everything home. I guess you’ll also need a proper way to retrieve and manage the errors.

I’m not going to dive more into details here, the [previously linked article] (http://blog.ezyang.com/2017/02/how-to-integrate-ghc-api-programs-with-cabal/) contains some code examples, it should more than enough if you want to learn more about this trick.

Cabal Helper

Cabal helper is a library initially created for ghc-mod.

The idea here is a bit different, instead of having our GHC API program loaded by Cabal, we’re going to retrieve some build meta-informations left by Cabal during the package build process and parse them.

Here, you won’t need any GHC wrapper nor frontend plugin.

Let’s see how we can practically use this library and the GHC API to retrieve a module exports.

Using Cabal-helper: retrieving a module exports

First, we’ll need to let Cabal both install the package’s dependencies as well as building the project.

cabal install --dependencies-only
cabal build

Cabal should have built the necessary artefacts for us to retrieve the dynamic flags. Let’s parse these artifacts and get the options passed to GHC.

let fp = "path to the dir containing the .cabal file"
    qe = mkQueryEnv fp (fp </> "dist")
cs <- runQuery qe $ components $ (,) <$> ghcOptions

Your package may contain several Cabal components: a library, an application, a unit test suite, an integration test suite, etc. Each component is built differently, they’ll each have a different set of dynamic flags.

You’ll probably want to filter this list to extract the component you’re interested in. In this example, let’s say we are interested by the lib:

getLib (_, chLibName) = True
getLib _ = False
head $ filter getLib cs

Altogether, you should have something like this.

getCabalDynFlagsLib :: (MonadIO m) => FilePath -> m [String]
getCabalDynFlagsLib fp = do
  let qe = H.mkQueryEnv fp (fp </> "dist")
  cs <- liftIO $ H.runQuery qe $ H.components $ (,) <$> H.ghcOptions
  pure . fst . head $ filter getLib cs
    where
      getLib (_,H.ChLibName) = True
      getLib _ = False

Well, vaguely. You probably want to reflect some error cases on your API :)

We can now load these flags into GHC.

dflags0 <- getSessionDynFlags
dflags1 <- getCabalDynFlagsLib pfp
(dflags, _, _) <- parseDynamicFlags dflags0 (noLoc <$> dflags1)
_ <- setSessionDynFlags dflags

Your GHC API environment is finally set up, you can now use it.

We still want to retrieve the exported symbols of a module: let’s write the corresponding GHC API code.

getModExports :: (MonadIO m) => FilePath -> ModuleName -> m [String]
getModExports pfp mn = 
  liftIO . runGhc (Just libdir) $ do
    dflags0 <- getSessionDynFlags
    dflags1 <- getCabalDynFlagsLib pfp
    (dflags, _, _) <- parseDynamicFlags dflags0 (noLoc <$> dflags1)
    _ <- setSessionDynFlags dflags
    target <- guessTarget fileName Nothing
    setTargets [target]
    _ <- load LoadAllTargets
    modSum <- getModSummary $ mkModuleName modName
    parseModule modSum >>= typecheckModule >>= desugarModule >>= getModExports
  where
    modName = intercalate "." $ components mn 
    fileName = "./" <> toFilePath mn 
    getModExports = fmap getAvName . mg_exports . dm_core_module

Aaaaaand, that’s it, we’re done. Again: you would probably want to add a bit more of error handling to this.

As you can see, cabal-helper has a pretty simple API and is quite straightforward to use. I just had to ignore some upper bounds to use it in my project.

Under the hood, it’s a whole other story. Since Cabal’s artifacts are quite heavily version dependant, cabal-helper will compile and run a small wrapper at runtime. This wrapper is in charge to parse the artifacts and send back the results to the main process your program is running in. Since the wrapper has been built by the same Cabal that built the artifacts, we are kinda sure it will be able to parse them.

In our example, this does not really matter as we are sure the Cabal who built cabal-helper is the same who built the module we are inspecting, however, this can be a killer feature in case you are building an editor plugin.

Special thanks to [Edward Yang] (http://ezyang.com) for writing an article detailing the GHC --interactive trick and [Dxld] (http://darkboxed.org/) for writing cabal-helper, helping me figure out how to use it, and relaxing some upper bounds :)

Silver Searcher: Useful Regexes for a Haskell Code-Base

PicNoir — Sun, 22 Jul 2018 17:14:00 +0200

TL;DR

I use 4 Perl Regex patterns most of the time when it comes to search some Haskell code:

Functions:    "\b<args>\b[ \t\n]+::"
Types:        "(data|newtype|type)(\ +)\b<args>\b"
TypeClasses:  "class(\ +)(.*)(=>)*(\ *)\b<args>\b"
Constructors: "\|[\t\ ]+\b<args>\b"

I am looking for a better way to search for a type constructor, email me at picnoir at this domain if you have any better idea.

Ag

Exploring an unknown code-base is always tricky: you need to somehow translate a text-based representation of a software to an accurate model in your own mind. Being able to efficiently search through the code helps to reduce the read-search loop feedback and frees a lot of headspace.

Unlike many languages, Haskell does not have a real IDE and like many developers, I use a traditional text-based searching tool: the silver searcher. This software is blazing fast and supports Perl regexes.

After using it to search through my Haskell code for quite some time, I found myself using the same regex patterns all day long. I soon started to write some aliases to cover most of my searching needs. In this article, I’ll share my favorite ones.

In the following regexes, <SYMBOL> will represent the symbol we’re looking for.

Functions

Searching for functions is quite straightforward: the function name is always followed by a :: lexeme. This lexeme is separated from the symbol either by both some whitespaces or new lines.

\b<PATTERN>\b[ \t\n]+::

Types

When looking for a type, we usually like to search for any data, newtype or type declaration.

(data|newtype|type)(\ +)\b<PATTERN>\b

Here, we do not try to match against any constructor, we prefer having a distinct pattern matching them.

TypeClass

Same trick as when we look for a type, except we are here matching a class lexeme.

Note type classes may have some type constraints on the left side. We need to pattern match those as well.

class(\ +)(.*)(=>)*(\ *)\b<args>\b

Data Constructor

It’s sometimes useful to find the definition of a type constructor. The idea here is similar to the function regex: the constructor symbol is most of the time following the | lexeme.

\|[\t\ ]+\b<PATTERN>\b

Unlike the previous patterns, I am not quite satisfied with this one: it does matches against guards, miss the first constructor of a regular algebraic data type and does not match any constructor expressed in the GADT style. To be honest, I hesitated about including this one in this article. This one could be greatly improved.

Vim Integration

Being able to directly jump to the matched string from within the editor is always nice. If you happen to be a heretic of some sort (ie. not a vim user), this section will be no use for you. Just use a plain shell alias or a specific feature from your favorite editor.

You’ll first need a vim silver searcher integration. I personally use ack.vim and will assume you also use it.

I use some short aliases for the patterns previously showed:

:Aghf <PATTERN> Ag Haskell Function: looks for a function definition.
:Aght <PATTERN> Ag Haskell Type: looks for a type definition.
:Aghtc <PATTERN> Ag Haskell TypeClass: looks for a typeclass definition.
:Aghc <PATTERN> Ag Haskell Constructor: looks for a type constructor definition.

These commands are defined in my configuration using the command! directive:

command! -nargs=+ -complete=file Aghf Ack -G ".*\.hs" "\b<args>\b[ \t\n]+::"
command! -nargs=+ -complete=file Aght Ack -G ".*\.hs" "(data|newtype|type)(\ +)\b<args>\b"
command! -nargs=+ -complete=file Aghtc Ack -G ".*\.hs" "class(\ +)(.*)(=>)*(\ *)\b<args>\b"
command! -nargs=+ -complete=file Aghc Ack -G ".*\.hs" "\|[\t\ ]+\b<args>\b"

That’s pretty much all. These 4 simple aliases are probably covering around 80% of all my search needs. I still need to perform some custom search from time to time, but I am overall quite pleased by this small setup.

Please, Keep your Blog Light

PicNoir — Wed, 24 Jan 2018 14:11:00 +0200

TL;DR: keeping your blog lightweight is important, I show you how to design a blog fitting in less than 10kB.

You’re already convinced weight really matters when it comes to web pages? You can skip the introduction and directly see how you can reduce your blog’s weight through a practical example.

Why Does Size Matters?

I grew up in a French isolated village during the early 2000’s. Living in an isolated place during the early 2000’s meant no public libraries, no technical bookstore, no access to computer-related knowledge or any kind of technical expertise. When my parents decided to subscribe to an RTC connection, everything changed. All of sudden, I had access to virtually an infinite amount of information about computers, RC models, and other technical fields.

By reading blogs and forums, I had access to everything a geeky kid could dream of: I learned to write websites using table-based HTML, I learned to fly RC models, I learned to read English, I learned to write PHP 4 using l33t-sp3ll3d identifiers, I learned to exploit an XSS in order to bypass my monthly hosting quota, I learned that doing this would lead me to unpleasant consequences.

I learned all of that using a 3.5kB/s crappy RTC connection. I was able to access these information with such ease because the internet was then mostly text pages, because web-pages were still relatively light.

Fast forward 15 years, I do not use l33t identifiers anymore, I don’t fly RC-Models anymore but I still love to write computer programs! Like almost all western programmers, I spent most of my adult life in big cities where internet is cheap and fast.

Eventually, I decided to take a sabbatical year. The first thing you do in such a case is cutting unnecessary expenses to make it as long as possible. So, as a grown-up man, I downgraded my mobile data plan to the cheapest one, the one allowing me to download 25MB monthly. I mostly use my phone for chatting and reading blog posts, I thought 25MB per month was more than enough.

As I often am, I was wrong. Four days after subscribing to this plan, I received the - now familiar - “out of data” notification.

What the hell!?

Well, it turns out the web became heavier than I thought.

Let’s load a medium post:

Url:    https://medium.com/[..]
3.26 MB Transferred total
1.2  MB Html + JS + CSS + Fonts

3.26MB per post!?

At this rate, I am able to read 7 posts per month… Do this article really worth a seventh of my monthly data plan? I highly doubt it.

This page would take around 9 minutes to load using my RTC childhood connection.

I took medium as an example as it seems to be the most popular blogging platform nowadays; keep in mind this is not an exception, this is a trend. A lot of popular blogs tends to be really heavy!

You may say “it’s not a problem: RTC connections became quite rare and time travel devices are still primitive”. You’re right, but look at the latest akamai speed report. The global internet median speed is ~8Mbps: that’s around 1MB/s.

This medium article, on average, will take 3 seconds to load on a device. But that’s even worse! The website is full of asynchronous requests, on a 2017 dell xps laptop, a full render of the page takes approximatively 2 seconds. Yup, this page will take 5 seconds to load on a recent high-end laptop using an average internet connection!

This is a huge latency to display a simple text page on a pretty fast device!

Are we doomed to increase the global web bandwidth and experience high latency for the rest of our lives? Is making something light so complicated? Let’s figure that out by making this blog as light as possible.

A Practical Example of Lightweight Blog

Luckily for us, making a blog fast is more about avoiding to add unnecessary dependencies than profiling and optimizing the rendering. This game is all about being minimalistic without altering the reading ergonomics.

Let’s take a pragmatical approach and think about what we need here:

Static website: we want to deliver the blog as quickly as possible. No need to be delusional here, you won’t post regularly enough to make a dynamic website appealing :).
Responsive design: we want the text to be as comfortable as possible to read regardless the device used. We should at least provide a desktop, a mobile, and a tablet version of the design.
Lightweight: The total weight of the page (HTML + assets) should not exceed 20 kB. We also should leverage browser caching.
Pleasing design: alright, this one is totally objective. But at least, I want something appealing to me.
JavaScript and CDN free: we don’t want to use any javascript rendering nor use any external CDN to serve the assets.
Minimal Effort: we don’t want to spend more time designing the blog than actually writing in it. We don’t want to re-invent the wheel here. The blog should setup should take us less than 4 hours (ie. a Sunday afternoon).

Generating your Blog’s Pages

First, we need to pick a static website generator. There is no lack of such software, just pick one. I chose Hugo: its documentation is nice and it supports server-side syntax highlighting.

We then need to write a theme for our blog. I would advise starting one from scratch, it’ll be quicker: templates tend to be quite bloated and having many dependencies.

I’m not going to dive into the details for the whole HTML/CSS part, but rather just give you an overview of what you need to write your design without loading tons of external libraries. If you have any problem related to web-design, MDN is usually a good starting point.

If you did not have a look at web dev for the last few years, I have a good news for you: you don’t really need a CSS framework anymore! You can thank CSS’s flex boxes; this feature is now supported by almost all active browsers and is basically all you need to organize your HTML elements. If you don’t know how to use them, the introduction written by Mozilla is what you’re looking for!

In order to adapt the design for smartphones and tablets, we need to use media queries. Contrary to popular belief, they are pretty simple to use: three rules according to the screen width will get you covered for most usual devices:

> 1000px: desktop and big screens.
880 > width > 1000px: tablets, netbooks and small screens.
< 880: smartphones.

You can easily debug your media queries related code using a device emulator. It’s included in both firefox and chrome web-dev tools.

Setting up an appropriate text width is primordial. Reading a text being too wide or too narrow is really uncomfortable. The desired width of the text will depend on the device; on a mobile phone, you want to use as much space as possible, while on a desktop, you want to display your text in a narrow centered column.

I used the following rules for this blog (vw being the viewport width):

@media (min-width: 1000px){
  #container{
    width: 50vw;
  }
}

@media (min-width: 800px) and (max-width: 1000px){
  #container{
    width: 70vw;
  }
}

@media (max-width: 800px){
  #container{
    width: 95vw;
  }
}

Regarding fonts, I made the heavy opinionated choice to only use the sans serif one embedded in the reader’s system.

body {
(..)
  font-family: sans-serif;
(..)

On one hand, the font will differ depending on the system, on the other hand, it will reduce the assets size. But as I said earlier, it is a quite heavy opinionated choice, default fonts can be quite uncomfortable to read, especially on old Windows systems.

[2022 edit]: I ended up reverting that decision and serve a web font (IBM Plex Sans). The default sans system font is sadly severely out of date wrt. modern visual standards on a lot of systems :(

I avoided using images as much as possible. I actually ended up having one image for this theme: the small icon displayed in your browser tab.

To illustrate the article, diagrams often help. Using SVG is a good way to keep everything light without sacrificing diagrams. Using Inkscape is a good way to edit those diagrams. Don’t forget to export the project to “standard SVG” or you will dump some private metadata along with your image.

Having the whole article included in the RSS feed is quite convenient: it gives the reader the option of not using your website’s stylesheet. Sadly, Hugo’s default RSS feed only renders the first ~100 words of each article. Here’s a small template overriding this behavior.

% cat /layouts/posts/rss.xml


<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>{{ .Title}} </title>
    <link>{{ .Permalink }}</link>
    <description>Recent posts</description>
    <generator>Hugo -- gohugo.io</generator>{{ with .Site.LanguageCode }}
    <language>{{.}}</language>{{end}}{{ with .Site.Author.email }}
    <managingEditor>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</managingEditor>{{end}}{{ with .Site.Author.email }}
    <webMaster>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</webMaster>{{end}}{{ with .Site.Copyright }}
    <copyright>{{.}}</copyright>{{end}}{{ if not .Date.IsZero }}
    <lastBuildDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</lastBuildDate>{{ end }}
    {{ with .OutputFormats.Get "RSS" }}
        {{ printf "<atom:link href=%q rel=\"self\" type=%q />" .Permalink .MediaType | safeHTML }}
    {{ end }}
    {{ range .Data.Pages }}
    <item>
      <title>{{ .Title }}</title>
      <link>{{ .Permalink }}</link>
      <pubDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</pubDate>
      {{ with .Site.Author.email }}<author>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</author>{{end}}
      <guid>{{ .Permalink }}</guid>
      <description>{{ .Content | html }}</description>
    </item>
    {{ end }}
  </channel>
</rss>

Don’t forget to add a link to the RSS feed both in the footer and the <head> section:

<link href="/posts/index.xml" rel="alternate" type="application/rss+xml"/>

Alright, now we designed everything, we are almost done.

We need to give our assets a unique name. We’ll need to reflect any change to their name to properly leverage the browser cache. You have two options: either you fire npm and setup an asset minifier, either you simple prepend DDMMYYYY on each filename like me :).

Let’s not kid ourselves, we are going to alter this theme twice a year, automating that won’t gain any time.

Distributing the Content

Now your blog is generated on your computer, it is time to publish it on the web. You have two major choices here:

You do not want to spend any money on it: just push it on GitHub pages and you’re done. You can skip this and directly read the next section.
You want to deliver it from your *nix server.

First of all, you want to serve the blog on a TLS-only link. Maybe your ISP/State does not alter your HTTP traffic, but it’s not the case everywhere. Please, think about your potentially endangered readers and protect them from a nasty rogue javascript injection.

Let’s Encrypt is a free and entirely automated TLS certificate provider. The first issuing will need some manual configuration, but afterward, all you’ll need for a cert renewal will be a simple cron rule fired twice a month.

We then need to configure Nginx to:

redirect HTTP requests to the HTTPS endpoint.
include the HSTS policy header.
support IPv6 and HTTP2.
deliver the static content with a specific HTTP header telling that clients should forever cache the content.
GZIP the text-based content.

Here’s a quick dump of my configuration. You can use it as a starting point:

server {
     listen         80;
     server_name    alternativebit.fr;
    location '/.well-known/acme-challenge' {
        default_type "text/plain";
        root        /tmp/letsencrypt-auto/;
     }
     location / {
        return         301 https://alternativebit.fr$request_uri;
     }
}

server {
   listen 443 ssl http2 default_server;
   listen [::]:443 ssl http2 default_server ipv6only=on;
   ssl_certificate $PATH_TO/fullchain.pem;
   ssl_certificate_key $PATH_TO/privkey.pem;
   server_name alternativebit.fr;
   client_max_body_size 100m;
   autoindex off;
   root $PATH_TO_FILES;
   add_header Cache-Control "no-cache, no-store, must-revalidate";
   add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
   location ~* .(jpg|jpeg|png|gif|ico|css|js)$ {
       add_header Cache-Control "public, max-age=31536000";
       add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
   }

   # Gzip black magic
   gzip on;
   gzip_disable "msie6";
   gzip_vary on;
   gzip_proxied any;
   gzip_comp_level 6;
   gzip_buffers 16 8k;
   gzip_http_version 1.1;
   gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

}

Analytics

Please, don’t.

You Now Can Open your Text Editor

So, is this mission successful? Let’s dig into the metrics.

All the following tests have been conducted on the very same page hosted on the very same HTTP server you are currently reading this post.

Let’s first test the first page loading (no browser caching involved).

Transferred: ~9kB
Unzipped Size: ~25kB
Finish: ~200ms

Yup, the whole content + assets fits just under 9kB and takes less than 500ms to be both downloaded and fully rendered on my laptop.

Let’s load the same page a second time:

Transferred: ~7.3kB
Unzipped Size: ~25kB
Finish: ~200ms

Now browser caching is leveraged, only 7.3kB has been transferred!

It would take ~2 second to download this long page from my childhood RTC line! The latency is way lower than on a typical medium post; I guess we can say this experiment is a success!

As you can see, making a light low-latency blogging system is quite easy. All the above experiment has been implemented on a Sunday afternoon. Less than 3 hours if we exclude the NGINX configuration black magic.

If you have a blog and are actively using it, please, invest those 3 hours and contribute to make the web a better place :)

Bracket: a Tale of Partially Applied Functions

PicNoir — Wed, 29 Nov 2017 11:21:00 +0100

TL;DR

In this post, we describe how we can use partially applied functions as a design building block though the study of a practical example: the bracket function.

I’ll use the Haskell programming language to illustrate this post. Just keep in mind this could be applied to almost any language.

It all Begins with Code Reuse

HSpec is a BDD-style unit-test framework for Haskell. In that kind of tests, it is quite common to create and destroy resources such as database handles, complex data structures, an HTTP server, etc.

I was looking for a way to share a resource across several it clauses without having to duplicate its instantiate/tear-down process. I quickly found the around function and have been instantly confused by its type.

around :: (ActionWith a -> IO ()) -> SpecWith a -> SpecWith a

ActionWith and SpecWith being some alias types, let me de-sugar it for you.

around :: ((a -> IO ()) -> IO ()) -> SpecM a () -> SpecM a ()

I know, strange right?

The SpecM monad is a reader monad used internally by HSpec, this reader monad contains the newly injected resource among other things. We can omit that part for now.

If your mouth started foaming after “reader monad” (it shouldn’t), you can just picture it as a small read-only key-value database.

What’s really interesting here is this part:

((a -> IO ()) -> IO ()) -> [...]

What is this thing?

Well, to answer that, we first need to understand the bracket function.

The Bracket Function

Let’s have a look at its type:

IO a -> (a -> IO b) -> (a -> IO c) -> IO c

Let’s keep in mind its purpose: it’s a way to create and destruct a resource with side effects, preventing any leak after using it.

Despite being a bit intimidating, this signature actually makes sense.

As explained in this Haskel wiki page, the above type signature can be split into 3 parts:

IO a: the constructor. This function will return the resource a in a IO monad.
(a -> IO b): the destructor. This function will take as parameter the previously created a resource and will apply the necessary IO actions to destroy it.
(a -> IO c): the actual effect-full computation that will use the a resource.
IO c: the final result coming from the previous computation.

Understanding the Around Function

Let’s go back to the around function, it just makes sense now: it is a partially applied bracket function. A bracket function to which we already provided a constructor and a destructor.

around ::                         ((a -> IO ()) -> IO ()) -> SpecM a () -> SpecM a ()
bracket :: IO a -> (a -> IO b) ->  (a -> IO c ) -> IO c

Ah! Pretty awesome don’t you think?

Personally, I’m found of this small simple function. What a great simple way to express a resource life cycle!

A Practical Example with HSpec

Alright, now we understand the handle function signature, let’s see how we can use it in HSpec.

Let say we want to test our implementation of the great Star Wars names database.

[...]
 describe "getStarWarsNameDb" $ do
    it "contains Star Wars characters names" $ do
        hDb <- getStarWarsNameDb
        db <- readDb hDb
        ("Dark Vader" `isIn` db) `shouldNotBe` True
        ("JarJar Beans" `isIn` db) `shouldBe` True
        clear hDb
    it "contains Star Wars citations" $ do
        hDb <- getStarWarsNameDb
        db <- readDb hDb
        ("May the fourth be with you." `isIn` db) `shouldBe` True
        ("Do. Or do not. There will be a trial." `isIn` db) `shouldBe` True
        clear hDb
[...]

As you can see, we have some code duplication here, we want to get rid of both the construction and the destruction of the db database.

First, let’s create a partially applied bracket call to which we already provided both a constructor and a destructor:

withDb:: ((StarWarsDb) -> IO ()) -> IO ()
withDb = bracket getStarWrsNameDb clear

We then just need to adapt each it clause to be pattern matched against a function having a single argument: the database handle.

[...]
 around withDb describe "getStarWarsNameDb" $ do
    it "contains Star Wars characters names" $ \hDb -> do
        db <- readDb hDb
        ("Dark Vader" `isIn` db) `shouldNotBe` True
        ("JarJar Beans" `isIn` db) `shouldBe` True
    it "contains Star Wars citations" $ \hDb -> do
        db <- readDb hDb
        ("May the fourth be with you." `isIn` db) `shouldBe` True
        ("Do. Or do not. There will be a trial." `isIn` db) `shouldBe` True
[...]

Neat, right!

We could go even further thanks to the polymorphic nature of the around function:

[...]
withDb:: (((StarWarsDb, StarWarsDbContent) -> IO ()) -> IO ()
withDb = bracket getDbContent (\(hDb,_) -> clear hDb)
    with
      getDbContent = do
          hdb <- createDb
          db <- readDb hdb
          return (hdb, db)

[...]
 around withDb describe "getStarWarsNameDb" $ do
    it "contains Star Wars characters names" $ \(hdb, db) -> do
        ("Dark Vader" `isIn` db) `shouldNotBe` True
        ("JarJar Beans" `isIn` db) `shouldBe` True
    it "contains Star Wars citations" $ \(hdb, db) -> do
        ("May the fourth be with you." `isIn` db) `shouldBe` True
        ("Do. Or do not. There will be a trial." `isIn` db) `shouldBe` True
[...]

See, all we needed to do was to alter the withDb function and the pattern matching of the anonymous functions used in the in clause.

Wrap Up

Partially applied functions are an amazing tool. Thanks to them, we can add context little by little to a function call. Used properly, this can lead to some nice design tricks such as the bracket function.

Hope you enjoyed this ode to the bracket function :). We are constantly taking for granted some beautiful functions such as this one. I think it is important to both acknowledge and celebrate that, sometimes, software can be beautiful!

Writing a Twitch Overlay using Haskell

PicNoir — Sat, 21 Oct 2017 12:48:00 +0200

I have been watching Jessica’s Mak streams lately. She is an indie game developper, but most of all, she has a kick ass overlay that shows what she is typing in real time.

I wanted the same one, I made it using Haskell, Gloss and Chipmunk via the Hipmunk binding.

Long story short, it ended up looking like this.

The source code is available in this git repository.

Gloss, display library

Gloss is a graphics library built upon OpenGL. The API is high level enough to mask a lot of low level nastiness.

To keep it short, the API is based around two functions:

An update function, which processes the input events and update the world data structure.
A display function, which renders the world as a Picture.

Those two functions are using two datastructures:

The world datastructure. This structure can be what you want, you will define it by yourself.
The Picture datastructure. It represents something that can be drawed to the screen.

I really like this second datastructure, its shape is highly interesting, let’s dig in it.

Hipmunk, physics library

Each object is composed by both a body and a shape. The body contains the physics informations (mass, moment of momentum, elasticity, …) while the shape just describes the bounding box of the object.

Capturing the Input

Now we can display funny bouncy letters, we still need to fix a problem: how to capture keyboard’s input? We were directly using the Gloss input system for debugging purposes. This sadly only work when the keywar window is focussed. Xorg only transferts the input events to the currently focussed window, which, in our case, is a problem.

How could we capture all the input events, regardless the focussed window? I found 3 solutions:

We could create an overlay window transferring the events to the underlying ones. This solution would imply to re-implement a subset of my window-manager features. We do not want to re-invent the wheel once again :).
We could bypass xorg and directly read Linux’s /dev/inputX input events. This solution is tricky, not only those events are using a binary data format, which make parsing them clumsy, but those events are raw, they are not applied to any keymap. If we really wanted to read those events, not only we should write the appropriate parser, but we also should support different keymaps.
We use a keylogger and tail -f its output.

The last solution was the simplest, we used the logkeys keylogger.

Side Note: Passwords

As mentionned in the github readme, you need to be extra careful when typing passwords, you clearly do not want to leak them while streaming.

I added a system which displays question marks in place of the actual letters to mitigate that. The audience will get the lenght of your password, but it is already the case if your keyboard is a bit noisy. Mine is noisy as hell, hence this is not really a problem to me.

Wireguard-Haskell: Getting Started

PicNoir — Mon, 16 Oct 2017 09:48:00 +0200

Why Starting this Project?

After finishing DobadoBots, I was looking for a Haskell project in which I could be confronted with some performance and parallelism problems.

Wireguard seemed to be the perfect project for that. At the time, no userspace implementation was available, the specification seemed to be simple enough for being implemented by a single person in a couple of months.

Furthermore, a work in progress Haskell implementation called Nara was already available. The only contributor to this project sadly abandoned it, undocumented, several months ago. This is a great opportunity for me to complete what he started.

Starting Point

The userspace implementation is not providing a CLI by itself. Instead, we are using the wg CLI utility to communicate with the VPN daemon through a Unix socket.

The overall architecture looks like this.

We first need to test this RPC interface implementation. It was sadly broken. The RPC protocol changed and these changes were not reflected in the Haskell implementation.

We clearly need to first re-implement and test this interface.

Refactoring the RPC Interface

The new protocol specified here is text-based. Both the Wireguard device and peers are described using a several key/value couples.

We process the incoming data as a Conduit stream. Conduit comes with the conduit-attoparsec module, which let us parse the incoming stream using an Attoparsec monadic parser.

Attoparsec is based around the same concept as Parsec. The main difference being that while Parsec aims to parse user-generated inputs, Attoparsec targets machine to machine data formats. It is hence faster, at the cost of a weaker error reporting system.

You can find the implementation details of this parser in Nara’s git repository.

The four main takeaways from this implementation are:

Parsing an IP address is not a trivial task. You need to be extra cautious, especially while parsing an IPv6 address.
Do not hesitate to create a lot of sub-parsers. It makes the code both easily testable and more readable.
You need to be extra cautious with your test cases. Test extensively the edge cases. It is dead easy to screw things up when using a Parsec-like library.
The Alternative typeclass will hold your back while dealing with optional repetitive options. This typeclass is implemented by the Parser type: you can use it with everywhere.

Next steps

We need to re-implement the queue system. As it is, the packets are multiplexed in one TUN queue and one UDP queue. It creates a lot of latency in case of a handshake: we need the handshake to successfully complete before sending any data, regardless the peer we want to send the data to.

We first need to design a new queue system. When implementing it, we will need to test it extra carefully. It may be a good excuse to dive into property-based testing techniques.

DobadoBots: Project Wrap Up

PicNoir — Thu, 28 Sep 2017 00:00:00 +0000

Yet another post about DobadoBots: my programming video-game. Right, let’s face it: it is done for two months now, this blog post is long overdue!

The video-game is now completely playable, I reached the MVP stage. Here’s a short video presenting the final result.

Let’s debrief this project.

Getting Familiar with Haskell

The main goal was to get some practical experience with Haskell. I had literally zero real-world experience before that.

After two months in, I can use Haskell productively enough. Yup, that’s a pretty steep and long learning curve. But it worth mentioning that I had no prior experience using a purely functional language.

It worth mentioning that you do not need to understand what’s a free monad or what are GADTs to be productive. You do not need a category theory PHD either.

There is a huge gap between getting productive enough and understanding what’s discussed on Haskell’s subreddit. This blog post made that click for me.

The main takeaway after this first practical experience is that a strong type system is a huge asset while refactoring. Having used Python quite extensively, a massive refactoring was always painful. You could easily introduce a lot of bugs as soon as your test suite was a bit weak. Haskell type system tends to cover your back most of the time and just refuse to compile your poorly refactored code.

I also found that assessing a task complexity very hard. Some things that would have been really long to do using a procedural language turns out being trivial using Haskell while some task that would have been simple using Python turns out being quite complicated using Haskell.

What has Been Unexpectedly Easy?

As previously discussed, I wrote my own language and parser for this project. This part has been unexpectedly easy.

Using Parsec for the parser part and Pretty for the pretty printer part made writing that nearly trivial. I discussed more in depth the parser implementation here.

What has Been Unexpectedly Hard?

Retrospectively speaking, the game part has been the hardest part. Without thinking about it, I basically wrote a whole game engine…

Choosing SDL2 as a graphics library turned out to be a poor choice. The library is low level, you basically need to implement everything. Turns out a video-game is not a trivial piece of software, you need to implement a button system, a menu engine, etc.

Writing the text editor also turned out being a real pain that took me almost two complete weeks.

The complexity is mainly hidden in small details.

What’s next?

I’m not going to lie, wrapping up this MVP was painful. I do not want to spend any more time on this project, working on it actually became a burden.

I do not plan to maintain this code, hence, I will not provide any binary distribution. You can, however, compile the Github repository if you want to play this game by yourself! It still has some rough edges and is only shipped with two stages. But you can easily add some more simply by looking at the JSON level format.

After getting familiar with Haskell, I think it is time to dive deeper in it and try doing something involving more performance tuning and some concurrent code. I’m going to take over the Haskell Wireguard implementation for the next few months!

DobadoBots: Writing a Text Editor

PicNoir — Mon, 04 Sep 2017 00:00:00 +0000

Yup, yet another post on my programming videogame: Dobadobots.

Today, we are going to dig into the editor’s implementation.

Motivations

I wanted the game to be as enjoyable as possible, I wanted a quick write/feedback loop. Using an external editor would have killed this fast feedback loop, it was just not an option.

I first thought about integrating a pre-existing editor in it.

An option would have been to create a web-view and embedding an HTML5 based text editor. After writing a quick proof of concept, I ran into some problems: I did not find any lightweight way to make the editor communicate with the core Haskell application. The only solution I found would have been to make this happen by running a WebSocket server. This solution seemed totally overkill, I decided to dismiss it.

Another option would have been to find a library implementing the basic features of an editor. However, I have not been able to find any Haskell/SDL2 text editor. Perhaps this tech stack is a bit too much specific! :)

So, being unable to find anything meeting my requirements, I ended up writing my own editor.

The Editor’s Architecture

As usual I started by specifying the data types and, as usual, we encapsulate the editor state in a proper record structure.

data EditorState = EditorState {
  text               :: Text,
  cursorColumn       :: Int,
  cursorLine         :: Int
} deriving (Eq, Show)

It encapsulates two things:

The position of the cursor.
the text contained in the editor’s buffer.

In addition to this state, we also need to generate some SDL textures we will use to display both the text and the various editor elements. I ended embedding these elements to the renderer state which already contains most of the game textures.

data RendererState = RendererState {
  (...)
  editorCursor       :: (SDL.Texture, SDL.V2 CInt),
  codeTextures       :: [(SDL.Texture, SDL.V2 CInt)],
  parseErrorMess     :: [(SDL.Texture, SDL.V2 CInt)],
  parseErrorCursor   :: (SDL.Texture, SDL.V2 CInt),
  editor             :: EditorState,
}

We can see the code is stored in multiples textures: one per code line.

Text Display and Typefont Generator

We need to display the text on the screen. The code text is basically some UTF-8 characters separated by spaces and newlines.

In order to display some text using SDL2, you first need to create an OpenGL texture representing this text. To do that, you want to use a typefont generator. It is basically a program which generates an OpenGL texture according to a TTF font and a UTF-8 input text. In this project, I used the SDL2-TTF library.

This typefont generator is distributed as a C library.

Some Haskell bindings were already available, hence, I did not even need to write the corresponding FFI files. The tricky part for using this library was to use it in the continuous integration runner.

I use Travis as a CI, which sadly uses an outdated ubuntu. This ubuntu does not provide any kind of pre-compiled SDL2-TTF binary, you need it to build it by yourself.

You will need libgl1-mesa-dev to build it. Because of Travis’s exoteric path configuration, you will also need to add the /usr/local/lib directory to the LD_LIBRARY_PATH environment variable.

Here’s the corresponding .travis.yml configuration section.

before_install:
- sudo apt-get install -y libgl1-mesa-dev
- wget http://libsdl.org/release/SDL2-2.0.5.tar.gz -O - | tar xz
- cd SDL2-2.0.5 && ./configure && make -j && sudo make install && cd ..
- wget https://www.libsdl.org/projects/SDL_ttf/release/SDL2_ttf-2.0.14.tar.gz -O - | tar xz
- cd SDL2_ttf-2.0.14 && ./configure && make -j && sudo make install && cd ..
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/

Rendering the Editor

The renderer is in charge of both input events handling and graphics drawing.

We need to be compatible with both Unixes and Windows systems. SDL2 has been used as a hardware abstraction layer here.

The renderer is directly plugged into the main graphic event loop which looks like this:

Check any kind of input.
Potentially append the event to the editor state.
Check the syntax of the code generated by the new event.
If the syntax is valid, load the new AST in the game engine.
Generate the new text textures.
Draw everything on the screen.

For more details about this implementation, you can check the implementation itself.

Syntax Error Checking

To keep the test/evaluation feedback loop short, we include a syntax checker directly in the editor.

Thanks to the parser (see the previous post), this task was quite straightforward.

Parsec not only returns a short text describing the error and the potential fix but also returns a position (a column line couple) helping to localize the error. Using this localization, you can highlight this part while adding an error message at the bottom of the editor.

As you can see on the above screenshot, we display a red square next to the error. We then render the error message (if any) in a proper OpenGL texture and display it at the bottom of the screen.

Aaaaaand, that’s pretty much it. I omitted most of the algorithm describing the editor’s behavior, but trust me, there are many. I will probably write another article focusing on this part in the future.

That was the first time I was writing a text editor from scratch. I can now confirm that this is not, even remotely, a trivial task.

If you face someday the same problem, let me share with you this advice: find a freaking way to integrate an already existing editor to your program, do not write your own one!

DobadoBots: Implementing the Parser

PicNoir — Thu, 20 Jul 2017 00:00:00 +0000

Lately, I have been working on a video-game called DobadoBots. This game is about programming a robot’s artificial intelligence to solve mazes.

The robot is materialized by a white triangle. The goal is to reach the objective (orange square) while avoiding several obstacles.

Instead of using a standard embeddable script language such as LUA, I went the custom way and wrote my own language.

In a previous post, we detailed the AST specification and structure. In this post, we go through the implementation of the parser.

As a quick reminder, the language we are attempting to parse looks like this

IF laserDistance > 20
  moveForward
ELSE
  IF laserScan = objective
    moveForward
  ELSE
   IF laserScan = obstacle
       turnLeft
   ELSE faceObjectives

TL;DR, show me the code instead: here you go.

Parsec

As a new Haskell developer, I went through the various available parsing libraries. I finally decided to use the Parsec library. I have been amazed by it so far. This is probably the best parsing library I ever used.

The base concept of Parsec is to combine multiple Parser monads in order to create a monad that will either parse your AST, either return a detailed error message.

Parsec is providing basic Parser monads for the most common builtin datatypes (e.g. space, newline, character, integer, double, etc.). You can then build more useful parsers by assembling those basic ones using some combinators.

I find this design very elegant, however, after using this library a bit, you can very quickly mess things up. Therefore, I would recommend writing a lot of unit tests before implementing your parser.

Let’s Parse Dobadolang

There are several ways to use parsec. You can use the monadic approach and take advantage of the do notation or you can use the applicative approach and use the applicative style to combine the various parsers.

I decided to use the applicative approach for this project as it was a more compact notation.

We will build our compiler using the bottom up approach, let’s start by specifying the AST’s leaves: the action token.

Parsing Action Tokens

Each token of our language is defined by a specific string. Parsec provides a string parser which can parse a simple string case-sensitively. We can, for example, parse the move forward token like this:

string "moveForward"

Alright, but if we look at the type of string, we see we done something wrong.

Stream s m Char => String -> ParsecT s u m String

Yup, the actual returned value will be the string itself. We do not want that, we want to return the AST element instead.

We will use the <$ version of fmap. This inflix function basically discards the return value contained in the parsec parser to instead return the value specified at its left.

MoveForward <$ string "moveForward"

We use a similar technique to parse the other action tokens.

actionParser :: CharParser () ActionToken
actionParser = MoveForward     <$ string "moveForward"
          <|>  try (TurnLeft   <$ string "turnLeft")
          <|>  try (TurnRight  <$ string "turnRight")
          <|>  FaceObjective   <$ string "faceObjective"
          <|>  ChangeObjective <$ string "changeObjective"

There are two new tricks here:

the try function prevents parsers from consuming too many inputs. You can check a detailed explanation of this function on hackage.
the <|> combinator here materializes a choice. We use it to try parsing the various tokens.

Parsing AST’s Nodes

The whole DobadoBots language is built around the conditional structure. Naturally, we want to parse that structure.

conditionParser :: CharParser () Cond
conditionParser = Cond <$>
                    ((spaces *> string "IF") *> logicExprParser <* newline)
                  <*>
                    (spaces *> parseNested <* newline)
                  <*>
                    ((spaces *> string "ELSE") *> (newline *> (spaces *> parseNested)))
                  where
                    parseNested = Token <$> actionParser
                              <|> conditionParser

Writing this function was my “aha!” moment, this is when I started to fancy parsec. Here, we just had to combine the previously implemented logic expression and action token parsers.

This is basically why I love to use parsec with the applicative notation. The resulting code seems to be written using a DSL while keeping Haskell’s strong type checking. It is both highly readable and easy to reuse.

Now we have defined the various parsec Parser instances to generate DobadoBots AST, time to wrap-up the parser.

Wrapping up Everything

Once the Parser monad is defined, we just need to call the parse function.

parseScript :: Text -> Either ParseError Cond
parseScript script = parse scriptFile "Error while parsing script: " $ unpack script .

As shown in this function definition, the parse function returns either a parsing error, which contains both the error location and a short message describing what should be there in order to have a valid input.

Testing the Implementation

As I mentioned in the introduction, it is quite easy to screw things up when parsing. In order to mitigate that, it seems like a good idea to write some test cases up front.

I used the HSpec unit test task runner together with stack for running the tests. It is a good idea to inline the fixtures in the test file to avoid any filesystem side effect while writing purely functional tests. I used the embed file library to achieve that.

You can find the complete test case on the DobadoBots GitHub repository.

Each test case looks basically like this:

it "should parse simple cond =" $
  parseScript (decodeUtf8 $(embedFile "test/fixtures/script/simplecond.script")) `shouldBe` Right (Cond (CmpLogicInt $ Eq LaserDistance 1)(Token TurnLeft) (Token MoveForward))

We are inlining a fixture to check the parse result with an expected AST.

Another testing approach we could implement would be the round trip property check.

The goal of this test is to ensure that parse(print(parse(script))) == parse(script) is always true. Basically, it checks if your parser/pretty printer couple is correct. I did not figure out how to setup stack to both test the library using HSpec (unit tests) and Quickcheck (property-based tests). I will probably figure out how to do that and write a follow-up article in the near future.

I do not see anything more to add, as you can see, writing a parser using parsec is not that big of a deal. In the next article, we will cover up Dobadobot’s text editor.

DobadoBots: Specifying the Language

PicNoir — Sun, 25 Jun 2017 00:00:00 +0000

Lately, I have been working on a video-game called DobadoBots. This game is about programming a robot’s articial intelligence to solve mazes.

The robot is materialized by a white triangle. The goal is to reach the objective (orange square) while avoiding several obstacles.

Instead of using a standard embeddable script language such as LUA, I went the custom way and wrote my own language.

Specifying the Language

I had only one idea in mind when started to specify this language: keep it as simple as possible. I started wondering, what do we really need to program this kind of artificial intelligence? What is the smallest set of operations the language needs to support?

I decided to specify the language as a decision tree. Each condition being a node, each robot movement being a leaf. No loops, no variable assignments, no functions. We do not want to embarrass the player with unnecessary concepts.

Basically, the language structure looks like this.

  IF [sensor]
       [action or nested condition]
  ELSE
       [action or nested condition]

The sensor statement is a boolean expression. It is evaluated in order to decide which subtree should be next executed.

Every sensor evaluation results either in an action token, which gives the robot a movement instruction, or in a nested condition, which gives us a way to refine the behaviour.

In the end, a robot’s AI will be defined by something like this:

IF laserDistance > 20
  moveForward
ELSE
  IF laserScan = objective
    moveForward
  ELSE
   IF laserScan = obstacle
       turnLeft
   ELSE faceObjectives

Sensors and Actions

We then needed to determine the minimal operations subset required to control the robot though the maze.

We finally end up with the following sensors list:

laserDistance: checks the distance between the robot and the nearest object.
laserScan: checks what kind of object is facing the robot. It can be an obstacle, a wall, another robot or the objective.
objectiveDistance: checks the distance between the robot and the objective.

ObjectiveDistance was not necessary but makes solutions less confusing in some cases.

Regarding the actions, we ended up keeping 4 of them:

moveForward: the robot moves one step ahead.
turnLeft: the robot rotates counterclockwise.
turnRight: the robot rotates clockwise.
faceObjective: the robot faces the objective.

Implementing the Abstract Syntax Tree (AST)

Now we specified the language, we still need to implement its AST. We will use the Haskell programming language for this part.

We start specifying data structures for both actions and sensors tokens. This problem is quite straightforward, we just need to use Haskell’s algebraic data types.

data ActionToken = MoveForward
                 | TurnLeft
                 | TurnRight
                 | FaceObjective
                 | ChangeObjective
                 deriving (Show, Eq)

data SensorToken = LaserDistance
                 | LaserScan
                 | ObjectiveDistance
                 deriving (Show, Eq)

We then need to specify the logic expression data type. We need to keep in mind that comparing a laser distance with an integer is quite different from comparing a laserScan with a Collider.

In order to simplify both the parser and the interpreter, we restricted as much as possible the type definition for both of these kind of logic expression.

data LogicExpr = CmpCollider SensorToken Collider
               | CmpLogicInt (CmpInteger SensorToken)
               deriving (Show, Eq)

data CmpInteger a = Sup a Integer
                  | Inf a Integer
                  | Eq a Integer
                 deriving (Show, Eq)

data Collider = Obstacle
              | Objective
              | Wall
              | Robot
              deriving (Show, Eq)

Now we have implemented all the basic data types, we just need to compose them in order to build the actual tree.

To do so, we use a recursive data type.

data Cond = Token ActionToken
          | Cond { sensor    :: LogicExpr
                 , ifValid   :: Cond
                 , ifInvalid :: Cond
                 } deriving (Show, Eq)

We now successfully implemented our AST, Cond being its root.

In next article, we will dive in the associated parser’s implementation.

One Year of FOSS

PicNoir — Sun, 18 Jun 2017 00:00:00 +0000

One Year of FOSS

Hello internet friends, long time no see.

After doing a total revamp of this weblog, I think it is finally time to explain what’s happening in my life.

Two months ago, I decided to quit my job. I was tired of Paris’s pollution. I was tired of Paris’s shitty public transportation. I was not really happy doing my job. I had one year of runway. Long story short, I moved to Bayonne, a really nice place in southern France.

Sadly, as nice as this place is, we cannot call it a tech-friendly place. There is not a lot of software-related jobs there, even less interesting ones. So instead of looking for a new job, I decided to do something bold. Why not, instead of constantly whining about low quality software, trying to write something I can be proud of, software I personally enjoy, and in the meantime, forgetting about getting any income for one year?

The Manifesto

I came up with this manifesto:

Do not think about money or employment, only think about learning and creating something useful.
Each started project will be a completed one. Even if it turns out to be a failure or a bad idea, reach at least the minimal viable product stade. There is no escape.
Do not parallelize projects. You should have one and only one software project at a time.
Release everything you do in an opensource form. Preferably using the MIT V3 license.
Try to cooperate with other people as much as possible in everything you do.
Share your experiment publicly in an honest way. No idealistic bullshit, if something you do sucks, say it and explain why.
Try to suck as less at possible in your job. You are here to learn about how to write highly reliable software, regardless how long it takes. Time will not be a constraint this year.

I have been following these rules for the last two months.

What Have I Done so Far?

I first started learning haskell. Yes, yet another time. This was the third time I was attempting to learn that damn language. I can finally say that this was definetely the last try. I grasped enough concepts to have a real world usage of it. It took me one month: the first one of my sabbatical.

This time, instead of trying to get started in two days using some crappy internet resources, I went the hard way. I studied the real world haskell book. I couldn’t recommand it enough, it explores the language in depth with enough practical exercises and examples to grasp the presented concepts. As I was mentionning, it took me around 3 weeks fulltime to finish that book. So be warned, Haskell is a great language, but do not expect to grasp the basics in 2 days as you would do with yet another c-based language.

After getting through this book, I decided to start right away a Haskell project from scratch, just to prove myself I was able to build something practical with this language.

This is when I decided to create a videogame. Partly because it was a child’s dream, partly because I was curious about how to build a purely functional game.

I will write more in-depth about DobadoBots in some upcoming articles, but long story short, after one month of development, I am yak-shaving its minimal version. All the features are now implemented, it only needs some polishing.

Radical Lifestyle Change

Starting a new life phase is always a hustle. This time was no different.

First, the city. Moving from a 2 million inhabitants metropolis to a small 15 000 inhabitants city was a hell of a change. In the end, I couldn’t be more happy. The city center is pedestrians only: no more pollution nor car noise all day long. Everything is within reach using a bycicle: no more crowded public transportations. I can reach the beach in a 20 minutes bycicle ride. I really like this place, I actually would love to stay here even after this sabbatical.

Next, work. I am not going to lie, this has been the harder part. Moving from a on site white-collar job to a completely constraint-free homeschooled one has been greatly affecting my show up rate! During the first month, I have been constantly struggling with self-motivation. Not having external constraints and sharing the same space for both personal life and work turned out to be encouraging me procrastinating. Luckily, I finally catched up momentum and have been effectively working fulltime for the past month. I guess this struggle will never end, but at least I now know that I can effectively beat procrastination!

That’s pretty much it so far, I think I will try to write an update about this sabbatical after every project wrapup.

Real World Haskell Chapter 9 Solutions

PicNoir — Tue, 04 Apr 2017 00:00:00 +0000

Exercise P.221

Is the order in which we call bracket and handle important?

Yup, it is pretty important: the code executed during the in-between statement of bracket still can raise an exception. We do not want our application to crash in case of an error while opening the file, therefore, we still need the handle statement.

We could put the handle statement inside of the bracket statement, but since no exception would be thrown, we would risk to leak some resources.

Exercise P.228

Question 1

How could we traverse the directory three in reverse alphabetic order?

Well, it is quite straightforward, before iterating on directories, we apply the order function on the list containing them. Therefore, we only need to apply reverse as the order function.

ControllerVisit> ControllerVisit.traverse reverse "/home/minoulefou/test"

Question 2

Implement an order function which traverses the tree in postorder.

For each file info type handled by the order function, we have two options:

The file is a directory: we want to first handle its children then handle the directory.
The file is not a directory: we will apply the identity function to this file.

I implemented this recursive function as follows:

postOrder :: [Info] -> [Info]
postOrder (x:xs) = if isDirectory x
  then postOrder xs ++ [x]
  else x:postOrder xs
postOrder [] = []

As you can see, it the implementation is quite straightforward.

There may be some other ones, do not hesitate to shout me out if you found something else.

Question 3

We want to adapt previously implemented predicates with the new Info datatype.

First we need to express InfoP using info:

type InfoP a = Info -> a

We will now implement predicates equality and comparaison: we’ll need to reimplement liftP with the new InfoP type:

liftP :: (a -> b -> c) -> InfoP a -> b -> InfoP c
liftP op infoP test info = infoP info `op` test

Once we implemented this function, implementing equality and inequity predicates is trivial:

greaterP, lesserP :: (Ord a) => InfoP a -> a -> InfoP Bool
greaterP = liftP (>)
lesserP = liftP (<)

equalsP :: (Eq a) => InfoP a -> a -> InfoP Bool
equalsP = liftP (==)

Once again, in order to implement combinators, we’ll use another lift function.

liftP2 :: (a -> b -> c) -> InfoP a -> InfoP b -> InfoP c
liftP2 op pred1 pred2 info = pred1 info `op` pred2 info

andP, orP :: InfoP Bool -> InfoP Bool -> InfoP Bool
andP = liftP2 (&&)
orP = liftP2 (||)

As for the previous question, we’ll use some infix notations for each operation: it is more handy to use.

(==?)::(Eq a) => InfoP a -> a -> InfoP Bool
(==?) = equalsP

(>?), (<?) :: (Ord a) => InfoP a -> a -> InfoP Bool
(>?) = greaterP
(<?) = lesserP

(&&?),(||?):: InfoP Bool -> InfoP Bool -> InfoP Bool
(&&?) = andP
(||?) = orP

Question 4

Write a wrapper for traverse that lets you control traversal using one predicate and filter results using another.

Here, we want to add a behaviour to the ord function. Not only we want that function to order the results, be we also want it to filter them. An easy approach to this problem seems to be composing the filter function with the order function:

traverse' order pred path = ControllerVisit.traverse (order . filter pred) path

Exercise P. 232

Question 1

Modify foldtree to allow the caller to change the order of traversal of entries in a directory.

First, we need to specify the new fold function signature:

foldTree' ::([FilePath] -> [FilePath]) -> Iterator a -> a -> FilePath -> IO a
foldTree' order iter initSeed path = ...

We will plug this function to the existing foldTree on the fold subfunction as follow:

fold seed subpath = do 
  paths <- getUsefulContent subpath
  walk seed $ order paths

The remaining code will still unchanged.

Aaaand, the two remaining exercises are quite similar to the previous ones, I will pass on that :)

See you soon for chapter 10 solutions!

Real World Haskell Chapter 8 Solutions

PicNoir — Tue, 28 Mar 2017 00:00:00 +0000

Hello, it’s been a long time.

Despite having done some progress on the resolution of the exercises of this book, I haven’t blogged about my solutions, which is a shame. The solutions are available in this git repository though.

Let’s start again for the end of chapter 8.

Exercises P.205

Here, we need to implement a case insensitive version of the glob checker.

Despite having looked on Hoogle for some elegant solution, I did not found anything suitable… I ended up writing a dirty hack.

The idea is to add a boolean flag to the matches glob. If this boolean is unset - meaning that we do not want to be case sensitive - we uppercase both the filename and the pattern.

Despite not being an elegant solution, it works.

matchesGlob :: FilePath -> String -> Bool -> Bool
matchesGlob name pat True = name =~ globToRegex pat
matchesGlob name pat False = map toUpper name =~ globToRegex (map toUpper pat)

Exercises P.210

Generalisation of the case sensitivity

First of all, we need a way to determine that we are on a windows system. According to Hoogle, the isPathSeparator function seems to be a pretty good candidate. If '\' is a path separator, then we are on a windows environment.

So first, let’s select the right glob matching func:

let globMatchingFunc = if isPathSeparator '\\'
                       then matchesGlobCaseSensiitive
                       else matchesGlobNotCaseSensitive

Then we’ll use this function in the filter part:

 return (filter (globMatchingFunc pat) names')

You can find the entire file in order to have a higher level-view here

Replacement for doesNameExist on POSIX systems

fileExists would be a good replacement for doesNameExists.

We could use that function because everything can be seen as a file on a POSIX system.

However, using this specificity would make our software POSIX specific thus impossible to run on a windows system.

Recursive wild card implementation

In order to implement this feature, we need several parts. The first one will be a function that returns the childrens of a directory.

This feature has been implemented using the following algorithm:

getChildrenDirs :: String -> IO [String]
getChildrenDirs path = getChildrenDirs' [path]
  where getChildrenDirs' :: [String] -> IO [String]
        getChildrenDirs' paths = do
          pathNames <- forM paths $ \path -> do
            dirContent <- listDirectory $ dropTrailingPathSeparator path
            let absDirContent = map (\p -> dropTrailingPathSeparator path </> p) dirContent
            dirs <- filterM doesDirectoryExist absDirContent
            if null dirs
            then return [dropTrailingPathSeparator path]
            else do children <- getChildrenDirs' dirs
                    return (path:children)
          return (concat pathNames)

Nothing too fancy here, just a recursive tree traversal. The only new thing (at least for me at this point) was the use of filterM which let us use filter on a IO() list.

Now we have this function, we can:

Detect the ** pattern somewhere in the pattern.
List the subdirectories according the beginning of the path given in parameter.
Pattern match the files contained in these folders according the basename filter.

In order to implement point 1, we’ll need to implement a simple predicate detecting the “double wildcard”:

containsDoubleWildcard :: FilePath -> Bool
containsDoubleWildcard path = path =~ "^.*\\*\\*.*$"

If we detect this kind of pattern, we want to add the current dir and its children to the file lookup. In order to do that, we will add these directories to the “dirs” list in the namesMatching as follow:

dirs <- if isPattern dirName || isPattern baseName
    then if containsDoubleWildcard baseName
           then getChildrenDirs (dirName)
           else namesMatching $ dropTrailingPathSeparator dirName
         else return [dirName]

The rest of the namesMatching will take care of the pattern matching on the files contained in the dirs list.

You can find the complete solution to the exercise on this git repository.

The last question being trivial, we are done with this chapter :)

Real World Haskell Chapter 4 Solutions

PicNoir — Sun, 18 Dec 2016 00:00:00 +0000

Okay, in this chapter, we will apparently learn more about common techniques in FP, can’t wait!

Exercises page 84

Let’s rewrite safe versions of partial list functions

The first two functions are very straightforward: a bit of pattern matching, some identification of edge cases and we’re done.

safeHead :: [a] -> Maybe a
safeHead [] = Nothing
safeHead (x:xs) = Just x

safeTail :: [a] -> Maybe [a]
safeTail [] = Nothing
safeTail (x:xs) = Just xs

safeLast :: [a] -> Maybe a
safeLast [] = Nothing
safeLast [x] = Just x
safeLast (x:xs) = safeLast xs

It gets a bit more tricky for the last one. Indeed, we need to recursively construct a list from the end to the beginning and wrap everything in a Maybe. In order to do that, I first calculate the list in a sub-function defined in a where section and then encapsulate everything in the Maybe.

safeInit :: [a] -> Maybe [a]
safeInit [] = Nothing
safeInit x = Just (end x)
  where end :: [a] -> [a]
        end [a] = []
        end (x:xs) = x:(end xs)

SplitWith: an inverted word

This definition is so confusing… I am not sure about getting the specification right, but from what I understand:

We have a list we want to split in sublists.
We need to evaluate an expression for each list member.
If this evaluation is false, we need to split the main list at this point.

splitWith :: (a -> Bool) -> [a] -> [[a]]
splitWith _ [] = []
splitWith function list = [(takeWhile function list)] ++ (splitWith function remaining)
    where
        endOfList = dropWhile function list
        remaining
            | null endOfList = []
            | otherwise = tail (endOfList)

First, let’s think about edge cases. Obviously, the split of an empty list will results to an empty list.

Then, we need to write a recursive pass on the list. During this pass, we will first use the takeWhile function in order to take elements that verify the function. Then we will append these elements to the rest of the list minus the “delimiter”. I insist, you need to remove this delimiter, otherwise you’ll end up with an infinite loop.

In order to calculate the rest of the list (list minus delimiter), I used the function tail. As we seen in the last exercise, this function is quite unsafe. We need to be extra careful that we are not tailing an empty list.

Again, I am not sure that I got the specification right in the first place.

Printing first word for each input line

This exercise is quite simple, we want to:

Read a file.
Take the first character of each line.
Print it to another file.

-- We can reuse this code which is given in
-- the previous chapter
interactWith function inputFile outputfile = do
    input <- readFile inputFile
    writeFile outputfile (function input)

main = mainWith myFunction
	where myFunction = getFirstWordsOfFile
    	      mainWith function = do
    	      args <- getArgs
    	      case args of
        	      [input,output] -> interactWith function input output
	       	      _ -> putStrLn "error: exactly two arguments needed"

firstWord :: String -> String
firstWord line = head (words line)

firstWordsOfFile :: [String] -> String
firstWordsOfFile [] = ""
firstWordsOfFile (x:xs) = (firstWord x) ++ " " ++ (firstWordsOfFile xs)

getFirstWordsOfFile :: String -> String
getFirstWordsOfFile fileStr = firstWordsOfFile (lines fileStr)

As you see, this is quite simple, there are 3 functions:

firstWord: which chops a line and returns the first word.
firstWordsOfFile: which takes the lines of the file one by one and returns only the first words.
getFirstWordsOfFile: which takes the file string, chop it line by line and returns the first word.

I could have used where or let in order to encapsulate the intermediates functions. But I haven’t…

Transposing text from a file

Aaaaaaand, dunno. Seriously, I do not get how to do that without the map function… Shout me out if you found the solution!

Exercises page 98

Rewriting asInt using folds

This one is easy, we just fold from the left the char expression and we keep the integer representation up to date by multiplying by 10.

import Data.Char (digitToInt)
asInt_fold :: String -> Int
asInt_fold str = foldl foldFunc 0 str
    where foldFunc :: Int -> Char -> Int
          foldFunc acc elem = (digitToInt elem) + (acc * 10)

Handling negative numbers

Okay, now we want to handle negative numbers.

asInt_fold :: String -> Int
asInt_fold (x:xs)
    | x == '-' = (-1) * asInt_fold xs
asInt_fold str = foldl foldFunc 0 str
    where foldFunc :: Int -> Char -> Int
          foldFunc acc elem = (digitToInt elem) + (acc * 10)

In order to do that, we pattern match the head of the string and we filter the - using guards.

Handling common errors

Let’s add some conditions in order to handle common errors.

asInt_fold :: String -> Int
asInt_fold [] = error "Cannot convert [] to Int"
asInt_fold (x:xs)
    | x == '-' && null xs  = error "Cannot convert this expression to Int."
    | x == '-' = (-1) * asInt_fold xs
asInt_fold str = foldl foldFunc 0 str
    where foldFunc :: Int -> Char -> Int
          foldFunc acc elem = (digitToInt elem) + (acc * 10)

Please note that I did not found any direct way to correctly handle Int overflow other than using the safeint type (which I did not implemented here).

Safe error handling using either

Here, we’re going to wrap up our error handling in the either

type ErrorMessage = String

asInt_either :: String -> Either ErrorMessage Int
asInt_either [] = Left "Cannot convert [] to Int"
asInt_either (x:xs)
    | x == '-' && null xs  = Left "Cannot convert this expression to Int."
    | x == '-' = fmap (*(-1)) (asInt_either xs)
asInt_either str = Right (foldl foldFunc 0 str)
    where foldFunc :: Int -> Char -> Int
          foldFunc acc elem = (digitToInt elem) + (acc * 10)

I did not found a way to solve this problem without using fmap which has not been introduced yet. Maybe I missed something here or maybe this is an error in the book…

Implementing concat using foldr

Nothing to say here, this is straightforward

concat_foldr :: [[a]] -> [a]
concat_foldr input = foldr foldFunc [] input
    where foldFunc :: [a] -> [a] -> [a]
          foldFunc acc elem = acc ++ elem

Implementing takewhile using foldr

Again, nothing special here.

takeWhile' :: (a -> Bool) -> [a] -> [a]
takeWhile' _ [] = []
takeWhile' pred (x:xs)
    | pred x = x:takeWhile' pred xs
    | otherwise = []

takeWhileFold :: (a -> Bool) -> [a] -> [a]
takeWhileFold pred input = foldr foldFunc [] input
    where foldFunc elem acc
            | pred elem = elem:acc
            | otherwise = []

Implementing List’s groupBy

After playing a bit with ghci, it turns out that group by is taking a 2 parameters function and a list and groups the list’s elements using the function given in parameters and the first element. As soon as it is impossible to group with the first element, a new group is created and so on. For instance:

*Main Data.List> groupBy (\x y-> x + 1 == y) [1,2,2,3]
[[1,2,2],[3]]

Okay, let’s implement that.

groupBy' :: (a -> a -> Bool) -> [a] -> [[a]]
groupBy' pred input = foldl foldFunc [] input
    where foldFunc [] elem = [[elem]]
          foldFunc acc elem
            | pred (head(last acc)) elem = (init acc) ++ [(last acc ++ [elem])]
            | otherwise = acc ++ [[elem]]

This problem turned out to be very tricky. For each element, we want to check if we want to pair it with the current subgroup.

In order to do that, we need to:

Get the current subgroup using last acc.
Get the first element of this subgroup which is the group “master” using head(subgroup)
Check if pred is true with the “master” and the current elem.
- If it is, we need to append elem to the current subgroup.
- Otherwise, we create a new subgroup with elem as the “master”

As you can see, the code is really messy. I think that I should have created a specific datatype for the subgroup, it could have made pattern matching easier…

Aaaaand that’s it!!! We done it!!! Time for celebration now!

Open recipe database: how to gather, cure and store data

PicNoir — Fri, 08 Jul 2016 20:29:54 +0200

I recently started to think about creating an open recipe database. The major consideration coming with this project is how to gather, cure and store recipes.

Gathering data

Let’s start with something obvious: we cannot rely on users to create recipes from scratch. This is a very time consuming task, so the first approach would be to find a way to fully automate that.

As I mentioned in the previous article, the main goal of this project is to keep the recipes informations in a highly structured way. Even if we can easily scrap recipes from the internet, it is usually hard to analyse a full text format in order to fit it in a very structured and restricted database. It would involve some advanced machine learning techniques which I do not master at all and will highly raise this project’s complexity.

In order to keep things simple, we need to find a way to let users cure the unstructured information for us.

We can find loosely structured recipes databases all around the internet. They usually come under two flavours:

Websites formatting their HTML using hrecipe. It basically use html classes to add some semantic informations and metadatas about recipes. Ok, it is more a way to add semantical informations on a website than a database per say, but it makes scrapping these website easier.
Open Recipes Format based database. Fictivekin’s Open Recipes seems to be the more complete one even though it is more intended to be used as a recipe bookmark as mentioned in the repository. This format is YAML based.

Neither of these formats are strict enough for this project but it will be a good starting point which would need to be human curated.

Storing data

We will not go in depth here, we will just identify the various entities. This list is not an exhaustive one and there are obviously lot of dependencies between these entities, they will not be detailed in this post.

Recipe: obviously
Ingredient: it will contain one ingredient. Each ingredient should be unique in the database, we should avoid as much as possible duplicates.
Nutrition information: it will contain various nutritive informations. It will be associated with the ingredient entity.
Diet information: is this vegan/kasher/halal/vegetarian/watever friendly.
Technique: cooking technique, such as grill, peel, cut, etc.
Cooking step: Combination of a technique + several ingredients.

There will also be “links” between these entities, for the moment, I just identified one of them:

Substitutes: link between two ingredients. This link will probably also contain informations such as weight/volume ratio, etc.

Curating data

Now we have some data, we need to cure it in order to make it fit in our strict table scheme. The idea here is to let users do that, a bit like reddit let users cure internet links.

We will need to design a proper algorithm here and I will not get into that until another post, however, the outline will be the following:

Each entity (ingredient, recipe, …) will be associated to three states:
- New: a user is editing the entity, it is not ready for vote.
- Voting: the entity is currently candidate for integration in the database.
- Rejectted: the entity has been rejected, it will not be inserted in the database.
- Integrated: the entity is integrated in the database.
Every user is associated to a rank (karma) which will weight the votes.
During the voting state, users are invited to upvote or downvote the entity.
When an entity score meets a threshold (1 or -1), it will be accepted or rejected.

Tricky parts

Every entity will need to be voted in order to be integrated in the database. This will probably prevent us to integrate duplicates and crappy data. However, this may also discourage users from posting as they will not know if their work will be integrated or not.

Another problem will be entity dependencies: recipes will depend on ingredients, it means that in order to be included in the database, a recipe needs all its ingredients being already included. What if some ingredient are rejected, how should we handle that? I need to put more thoughs on that later…

Do we need moderators? I would prefer not in order to keep the database as neutral as possible, but depending of users activeness, it could be needed.

Coming next

Alright, enough talk, we need to start implementing that.

Next step: entity system implementation.

Some thoughts about building an open recipe database

PicNoir — Thu, 23 Jun 2016 00:00:00 +0000

Lately, I have been thinking about creating an open recipe database. It is just not possible to find any good quality recipes database. Often, the website indexing recipes does not expose its data using any kind of API. When it does - and very few does - the recipes are stored in a data format close to unrestricted plain text.

Advanced queries

Most recipes websites are offering very primitives search possibilities. For instance, Marmiton, the most complete french recipes database offers the following search options:

Meal type
Difficulty
Cost - using cheap, average and expensive options
Vegetarian friendly
Keywords

As you can see, this search is really primitive, I personally need some more criteria.

Disclaimer

There is a notable exception: Big Oven.

This website provides a lot ofrecipes, the ux and search function are really sweet and all the data is accessible using a REST API.

The problem with that website, besides being quite buggy (I have not managed to get my API key because of a login related bug…) is the very restrictive data license.

Basically, you cannot use this data for anything else than sending users to their website.

Features we need

After putting some thoughts on these criteria, I have come up with that list of features of the ideal recipes web index:

Ingredients list: I want to be able to specify a ingredients list in which the recipe will be created on. I also want to exclude some ingredients from the search results because my diet does not permit to eat these ingredients, because I do not like this ingredient, etc.
Equivalence class between ingredients: I want to be able to know if soy creme is an acceptable substitute to crême fraiche. In a more general way, I want to be able to determine the substitutes and their quantity substitution for each ingredient.
Hierarchy/Relationships/Categories of ingredients: Is my meal balanced? Is my meal free of meat? Is my meal kasher/halal? Does my meal contains fish? What kind of rice should I use for this particular meal?
Measurement scales equivalences: what the fuck is a cup? I want grams!

So yeah, seems like a lot of work to do…

Problem we are actually facing

It seems that every recipe index website is getting killed by a new one which is more user friendly or has more feature every 5 years.

There is no open database of recipes. Every time a new website is created, the data needs to be scraped or created from scratch. We need to separate the data (actual recipes) from the processing/presentational part or we will keep creating some crappy database from scratch every 5 years.

I think it is time to create something more durable: an open recipe database.

More article about this index are about to come.

How to configure TLS Let's Encrypt Certificates with Nginx

PicNoir — Fri, 04 Dec 2015 00:00:00 +0000

Let’s encrypt beta is public since yesterday. I already use their certificates for several domains. The main problem I actually face is certificates renewal. I want to be able to renew automatically my certificates without turning off Nginx.

Let’s encrypt official client can be used in a automatic way. Unfortunately, this feature is poorly documented… In order to automate this process, we need to configure Nginx to deliver acme’s challenge specific files, then we need to create a zsh script that we will launch using a Cron routine.

Nginx Setup

If you are already using https on your webserver, you probably have a configuration like this one:

    server {
        listen         80;
        server_name    ...;
        return         301 https://$server_name$request_uri;
      }
    server {
        listen 443 ssl;
        ssl_certificate fullchain.pem;
        ssl_certificate_key privkey.pem;
        server_name ...
        root ...
        location / {
          index index.html index.htm index.php;
        }

You are basically redirecting all the http requests to the https endpoint and serving this endpoint with whatever CGI service you want.

What we need to do is simply add the /.well-known/acme-challenge endpoint for each domain. This endpoint is used by the ACME protocol in order to make sure that we truly own the domain.

Just alter the http server like this:

    server {
        listen         80;
        server_name    ...;
        return         301 https://$server_name$request_uri;

        #Let's encrypt endpoint
        location '/.well-known/acme-challenge' {
          default_type "text/plain";
          root        /tmp/letsencrypt-auto/blog/;
        }
      }

Repeat that setup for all your vhosts. Once you done that, you are finally ready to create your let’s encrypt.

Let’s Encrypt

First, download the project’s client:

    git clone https://github.com/letsencrypt/letsencrypt

Now, we need to generate a certificate for each of your sub-domains. We also want to automatize this step in order to facilitate the certificate renewal. I created a zsh script as follow:

#!/bin/zsh
root="/tmp/letsencrypt-auto/"
domain="yourwebsite.net"
email="youremail@domain.net"

mkdir -p ${root}

/root/letsencrypt/letsencrypt-auto --renew --debug\
  --email ${{email}} --agree-tos certonly\
  --webroot -w ${root} -d ${domain}

You need to specify:

acme’s webroot specified in your nginx configuration.
the domain name you want to register.

Do not forget to set the exec rights to that file for the root user.

As you can see, I personally placed letsencrypt bin in my /root folder since you need to run it as root. It can be a big deal for some people, for me it’s not.

Now that we have our certificates, we want to automatize the renewal process. In order to do that, we need to call our script using a cron entry.

You just need to add this line to you root’s croontab:

0 0 1 * * /root/letsencrypt/run.zsh >/dev/null 2>&1

This line will renew your certificate each 1st day of the month at midnight.