AlternativeBit

Ex-Hack: a Haskell Example-based Documentation

picnoir — Thu, 15 Nov 2018 13:49:00 +0100

Abstract

Ex-Hack is an example-based documentation automatically generated using the packages posted on Stackage. There’s a live demo here. We’ve just released the alpha version; you can have a look to the code here.

We are actively looking for new contributors.

After briefly introducing the project’s incentives and explaining how this software works, we discuss the current roadmap and what we need to do before releasing the V1.0.0.

An Effort to Fill the Documentation Gap

Accordingly to the 2017 Haskell survey results, the lack of good documentation seems to be a major issue in the Haskell community.

How can we fix this? The obvious solution is to write more documentation. However, writing more documentation does not necessarily mean writing a better one. The best documentation writers I met are not necessarily the best programmers I met. Being able to write a clear, concise and useful documentation is a very specific skill that takes time to acquire.

However, there’s one thing you can add to your documentation requiring very little writing skill and still being extremely useful: examples. We, as humans, tend to often learn by mimicking other’s behaviour. Adding examples to a documentation is in my experience a local maximum: it’s not really hard to do, yet it dramatically increases your documentation’s quality.

This is where I started thinking to automatically generate these examples. The Haskell community is fortunate enough to have a centralized code repository: Hackage. The packages stored in this repository are usually getting their dependencies from this very same repository. Maybe we could use all this code to extract some real-world examples.

It would be great if on top of the current type documentation we could add an automatically generated example section in Haddock.

This was the starting point of the whole ex-hack project: generate an example-based documentation we could later use in Haddock.

Let’s get this straight right away: we still have a lot of work to do before merging this project to Haddock, but we made some serious progress in the last 5 months!

So, How does it Work?

What do we need to generate such a database?

Well, basically, for each Stackage/Hackage package, we need to:

Download it.
Build it.
Extract the symbols exported by its main library.
Index the symbols coming other packages.
Generate a static HTML documentation displaying the previously extracted informations.

The whole project is centered around the processing step abstraction. Each processing step is dependent from the previous one and is a dependency for the next one.

This means that ideally, we could run this software in a map/reduce configuration and distribute the load over several nodes. This comes handy, because as stated previously: we’ll need to build the whole Stackage/Hackage. Needless to say, this is a seriously time-consuming process. The more we can distribute the load, the faster we’ll generate the documentation.

So far, we’ve been able to generate the documentation for Stackage (~2,400 packages). The last run (which populated the current demo) took ~25 hour to complete on my 7 YO desktop. If we want to scale this project to the whole Hackage (~13 000 packages) repository, we’ll probably need to distribute this generation on several nodes.

Project’s Roadmap

Which bring us to the million dollar question: what’s next?

Before entering in the beta phase, we first need to address several issues making ex-hack documentation not fully usable:

First, we need to re-write the symbol occurrence indexing system. The current approach clearly does not work.
We need to setup a proper environment to execute the whole database generation. Some packages need some system dependencies (such as libgl, zlib, etc.) to be built. We are currently unable to build them. Hence, we are indexing 2082 packages instead of the ~2,400 contained in Stackage.

Then, it’ll be time to address some usability issues before releasing the first stable version:

Writing a second UI targeted to package maintainers; showing them what are the most used parts of their API. It could come handy when performing a major API refactoring ;)
Writing a Haskell IDE Engine plugin.
Partially re-write the current HTML documentation to make it more pleasant to use.

Ultimate Goal

Once ex-hack will reach its definitive form, I would love to see this merged back to Haddock. I think having usage examples next to the type and author’s comments for each library symbol would make a great documentation.

However, entirely merging this to Haddock is going to be challenging. First, we would need a new Haddock component in charge to build the ex-hack documentation for every new package/stackage release.

We would also need a way to retrieve an already populated ex-hack database before generating a Haddock documentation for a library: you can’t expect to build the whole Hackage/Stackage on a developer’s machine!

Anyways, looks challenging, but you always need a long-term goal dream right? :D

How to Join Us?

So far, I’ve been mostly working on this in isolation; which was arguably not my best move. The more people will work on this, the more progress we’ll make.

Most of the annoying early project exploration has already been made, we mostly know were we are heading now. We mostly need to “industrialize” the project. You don’t need to be a Haskell guru to contribute, as a matter of fact I’m far from being one.

For now, the GitHub repository will act as the main community hub. Pick an issue — some are labeled as newcomer-friendly — and just start hacking on it :)

If you have some questions/suggestions or just want to say hi, there’s also a more informal communication channel: the #ex-hack room on the Freenode IRC network.

Loading a Cabal module in the GHC API

picnoir — Wed, 08 Aug 2018 12:01:00 +0200

If you plan to build some Haskell tooling or any kind of static code analyzer, chances are you’ll need to use the GHC API at some point.

While loading a simple module into GHC’s API is quite trivial and well documented, loading complex modules (modules having some c dependencies, some specific options in the .cabal file, etc.) will require you to find the appropriate dynamic flags. These flags are usually retrieved and loaded into GHC by Cabal. Sadly for us, Cabal’s API does not seems to expose a direct way to get these flags.

Some people have developed solutions to work around this problem. In this post, we’ll explore two of them.

Revision: If you plan to use GHC >= 8.6.1, you might as well want to check out GHC source plugins. Mpickering wrote about them. They will not be mentionned in this article.

GHC Plugin + “GHCi Wrapper”

This technique has been detailed by Edward Yang on his website.

The idea basically boils down to this:

A more reliable way to integrate a GHC API program with Cabal is inversion of control: have Cabal call your GHC API program, not the other way around!

[..]

What we will do is replace the GHC executable which passes through all commands to an ordinary GHC, except for ghc –interactive, which we will then pass to the GHC API program. Then, we will call Cabal repl/stack repl with our overloaded GHC, and where we would have opened a GHCi prompt, instead our API program gets run.

You’ll first need to write your GHC API program as a GHC frontend plugin. Then, the dependencies will be built by Cabal using a regular GHC while your package will be loaded with your custom frontend plugin. You can then use GHC’s API without having to think about any dynamic flags: they already have been loaded for you by Cabal.

Pretty smart!

You’ll still need to get the data extracted by your GHC API program home. Sadly, your GHC frontend plugin will be instantiated in another process, you won’t have a direct access to it. I guess you could use one of the usual suspects (stdout, a message queue, a DBMS, etc.) to bring everything home. I guess you’ll also need a proper way to retrieve and manage the errors.

I’m not going to dive more into details here, the [previously linked article] (http://blog.ezyang.com/2017/02/how-to-integrate-ghc-api-programs-with-cabal/) contains some code examples, it should more than enough if you want to learn more about this trick.

Cabal Helper

Cabal helper is a library initially created for ghc-mod.

The idea here is a bit different, instead of having our GHC API program loaded by Cabal, we’re going to retrieve some build meta-informations left by Cabal during the package build process and parse them.

Here, you won’t need any GHC wrapper nor frontend plugin.

Let’s see how we can practically use this library and the GHC API to retrieve a module exports.

Using Cabal-helper: retrieving a module exports

First, we’ll need to let Cabal both install the package’s dependencies as well as building the project.

cabal install --dependencies-only
cabal build

Cabal should have built the necessary artefacts for us to retrieve the dynamic flags. Let’s parse these artifacts and get the options passed to GHC.

let fp = "path to the dir containing the .cabal file"
    qe = mkQueryEnv fp (fp </> "dist")
cs <- runQuery qe $ components $ (,) <$> ghcOptions

Your package may contain several Cabal components: a library, an application, a unit test suite, an integration test suite, etc. Each component is built differently, they’ll each have a different set of dynamic flags.

You’ll probably want to filter this list to extract the component you’re interested in. In this example, let’s say we are interested by the lib:

getLib (_, chLibName) = True
getLib _ = False
head $ filter getLib cs

Altogether, you should have something like this.

getCabalDynFlagsLib :: (MonadIO m) => FilePath -> m [String]
getCabalDynFlagsLib fp = do
  let qe = H.mkQueryEnv fp (fp </> "dist")
  cs <- liftIO $ H.runQuery qe $ H.components $ (,) <$> H.ghcOptions
  pure . fst . head $ filter getLib cs
    where
      getLib (_,H.ChLibName) = True
      getLib _ = False

Well, vaguely. You probably want to reflect some error cases on your API :)

We can now load these flags into GHC.

dflags0 <- getSessionDynFlags
dflags1 <- getCabalDynFlagsLib pfp
(dflags, _, _) <- parseDynamicFlags dflags0 (noLoc <$> dflags1)
_ <- setSessionDynFlags dflags

Your GHC API environment is finally set up, you can now use it.

We still want to retrieve the exported symbols of a module: let’s write the corresponding GHC API code.

getModExports :: (MonadIO m) => FilePath -> ModuleName -> m [String]
getModExports pfp mn = 
  liftIO . runGhc (Just libdir) $ do
    dflags0 <- getSessionDynFlags
    dflags1 <- getCabalDynFlagsLib pfp
    (dflags, _, _) <- parseDynamicFlags dflags0 (noLoc <$> dflags1)
    _ <- setSessionDynFlags dflags
    target <- guessTarget fileName Nothing
    setTargets [target]
    _ <- load LoadAllTargets
    modSum <- getModSummary $ mkModuleName modName
    parseModule modSum >>= typecheckModule >>= desugarModule >>= getModExports
  where
    modName = intercalate "." $ components mn 
    fileName = "./" <> toFilePath mn 
    getModExports = fmap getAvName . mg_exports . dm_core_module

Aaaaaand, that’s it, we’re done. Again: you would probably want to add a bit more of error handling to this.

As you can see, cabal-helper has a pretty simple API and is quite straightforward to use. I just had to ignore some upper bounds to use it in my project.

Under the hood, it’s a whole other story. Since Cabal’s artifacts are quite heavily version dependant, cabal-helper will compile and run a small wrapper at runtime. This wrapper is in charge to parse the artifacts and send back the results to the main process your program is running in. Since the wrapper has been built by the same Cabal that built the artifacts, we are kinda sure it will be able to parse them.

In our example, this does not really matter as we are sure the Cabal who built cabal-helper is the same who built the module we are inspecting, however, this can be a killer feature in case you are building an editor plugin.

Special thanks to [Edward Yang] (http://ezyang.com) for writing an article detailing the GHC --interactive trick and [Dxld] (http://darkboxed.org/) for writing cabal-helper, helping me figure out how to use it, and relaxing some upper bounds :)

Silver Searcher: Useful Regexes for a Haskell Code-Base

picnoir — Sun, 22 Jul 2018 17:14:00 +0200

TL;DR

I use 4 Perl Regex patterns most of the time when it comes to search some Haskell code:

Functions:    "\b<args>\b[ \t\n]+::"
Types:        "(data|newtype|type)(\ +)\b<args>\b"
TypeClasses:  "class(\ +)(.*)(=>)*(\ *)\b<args>\b"
Constructors: "\|[\t\ ]+\b<args>\b"

I am looking for a better way to search for a type constructor, email me at picnoir at this domain if you have any better idea.

Ag

Exploring an unknown code-base is always tricky: you need to somehow translate a text-based representation of a software to an accurate model in your own mind. Being able to efficiently search through the code helps to reduce the read-search loop feedback and frees a lot of headspace.

Unlike many languages, Haskell does not have a real IDE and like many developers, I use a traditional text-based searching tool: the silver searcher. This software is blazing fast and supports Perl regexes.

After using it to search through my Haskell code for quite some time, I found myself using the same regex patterns all day long. I soon started to write some aliases to cover most of my searching needs. In this article, I’ll share my favorite ones.

In the following regexes, <SYMBOL> will represent the symbol we’re looking for.

Functions

Searching for functions is quite straightforward: the function name is always followed by a :: lexeme. This lexeme is separated from the symbol either by both some whitespaces or new lines.

\b<PATTERN>\b[ \t\n]+::

Types

When looking for a type, we usually like to search for any data, newtype or type declaration.

(data|newtype|type)(\ +)\b<PATTERN>\b

Here, we do not try to match against any constructor, we prefer having a distinct pattern matching them.

TypeClass

Same trick as when we look for a type, except we are here matching a class lexeme.

Note type classes may have some type constraints on the left side. We need to pattern match those as well.

class(\ +)(.*)(=>)*(\ *)\b<args>\b

Data Constructor

It’s sometimes useful to find the definition of a type constructor. The idea here is similar to the function regex: the constructor symbol is most of the time following the | lexeme.

\|[\t\ ]+\b<PATTERN>\b

Unlike the previous patterns, I am not quite satisfied with this one: it does matches against guards, miss the first constructor of a regular algebraic data type and does not match any constructor expressed in the GADT style. To be honest, I hesitated about including this one in this article. This one could be greatly improved.

Vim Integration

Being able to directly jump to the matched string from within the editor is always nice. If you happen to be a heretic of some sort (ie. not a vim user), this section will be no use for you. Just use a plain shell alias or a specific feature from your favorite editor.

You’ll first need a vim silver searcher integration. I personally use ack.vim and will assume you also use it.

I use some short aliases for the patterns previously showed:

:Aghf <PATTERN> Ag Haskell Function: looks for a function definition.
:Aght <PATTERN> Ag Haskell Type: looks for a type definition.
:Aghtc <PATTERN> Ag Haskell TypeClass: looks for a typeclass definition.
:Aghc <PATTERN> Ag Haskell Constructor: looks for a type constructor definition.

These commands are defined in my configuration using the command! directive:

command! -nargs=+ -complete=file Aghf Ack -G ".*\.hs" "\b<args>\b[ \t\n]+::"
command! -nargs=+ -complete=file Aght Ack -G ".*\.hs" "(data|newtype|type)(\ +)\b<args>\b"
command! -nargs=+ -complete=file Aghtc Ack -G ".*\.hs" "class(\ +)(.*)(=>)*(\ *)\b<args>\b"
command! -nargs=+ -complete=file Aghc Ack -G ".*\.hs" "\|[\t\ ]+\b<args>\b"

That’s pretty much all. These 4 simple aliases are probably covering around 80% of all my search needs. I still need to perform some custom search from time to time, but I am overall quite pleased by this small setup.

Bracket: a Tale of Partially Applied Functions

picnoir — Wed, 29 Nov 2017 11:21:00 +0100

TL;DR

In this post, we describe how we can use partially applied functions as a design building block though the study of a practical example: the bracket function.

I’ll use the Haskell programming language to illustrate this post. Just keep in mind this could be applied to almost any language.

It all Begins with Code Reuse

HSpec is a BDD-style unit-test framework for Haskell. In that kind of tests, it is quite common to create and destroy resources such as database handles, complex data structures, an HTTP server, etc.

I was looking for a way to share a resource across several it clauses without having to duplicate its instantiate/tear-down process. I quickly found the around function and have been instantly confused by its type.

around :: (ActionWith a -> IO ()) -> SpecWith a -> SpecWith a

ActionWith and SpecWith being some alias types, let me de-sugar it for you.

around :: ((a -> IO ()) -> IO ()) -> SpecM a () -> SpecM a ()

I know, strange right?

The SpecM monad is a reader monad used internally by HSpec, this reader monad contains the newly injected resource among other things. We can omit that part for now.

If your mouth started foaming after “reader monad” (it shouldn’t), you can just picture it as a small read-only key-value database.

What’s really interesting here is this part:

((a -> IO ()) -> IO ()) -> [...]

What is this thing?

Well, to answer that, we first need to understand the bracket function.

The Bracket Function

Let’s have a look at its type:

IO a -> (a -> IO b) -> (a -> IO c) -> IO c

Let’s keep in mind its purpose: it’s a way to create and destruct a resource with side effects, preventing any leak after using it.

Despite being a bit intimidating, this signature actually makes sense.

As explained in this Haskel wiki page, the above type signature can be split into 3 parts:

IO a: the constructor. This function will return the resource a in a IO monad.
(a -> IO b): the destructor. This function will take as parameter the previously created a resource and will apply the necessary IO actions to destroy it.
(a -> IO c): the actual effect-full computation that will use the a resource.
IO c: the final result coming from the previous computation.

Understanding the Around Function

Let’s go back to the around function, it just makes sense now: it is a partially applied bracket function. A bracket function to which we already provided a constructor and a destructor.

around ::                         ((a -> IO ()) -> IO ()) -> SpecM a () -> SpecM a ()
bracket :: IO a -> (a -> IO b) ->  (a -> IO c ) -> IO c

Ah! Pretty awesome don’t you think?

Personally, I’m found of this small simple function. What a great simple way to express a resource life cycle!

A Practical Example with HSpec

Alright, now we understand the handle function signature, let’s see how we can use it in HSpec.

Let say we want to test our implementation of the great Star Wars names database.

[...]
 describe "getStarWarsNameDb" $ do
    it "contains Star Wars characters names" $ do
        hDb <- getStarWarsNameDb
        db <- readDb hDb
        ("Dark Vader" `isIn` db) `shouldNotBe` True
        ("JarJar Beans" `isIn` db) `shouldBe` True
        clear hDb
    it "contains Star Wars citations" $ do
        hDb <- getStarWarsNameDb
        db <- readDb hDb
        ("May the fourth be with you." `isIn` db) `shouldBe` True
        ("Do. Or do not. There will be a trial." `isIn` db) `shouldBe` True
        clear hDb
[...]

As you can see, we have some code duplication here, we want to get rid of both the construction and the destruction of the db database.

First, let’s create a partially applied bracket call to which we already provided both a constructor and a destructor:

withDb:: ((StarWarsDb) -> IO ()) -> IO ()
withDb = bracket getStarWrsNameDb clear

We then just need to adapt each it clause to be pattern matched against a function having a single argument: the database handle.

[...]
 around withDb describe "getStarWarsNameDb" $ do
    it "contains Star Wars characters names" $ \hDb -> do
        db <- readDb hDb
        ("Dark Vader" `isIn` db) `shouldNotBe` True
        ("JarJar Beans" `isIn` db) `shouldBe` True
    it "contains Star Wars citations" $ \hDb -> do
        db <- readDb hDb
        ("May the fourth be with you." `isIn` db) `shouldBe` True
        ("Do. Or do not. There will be a trial." `isIn` db) `shouldBe` True
[...]

Neat, right!

We could go even further thanks to the polymorphic nature of the around function:

[...]
withDb:: (((StarWarsDb, StarWarsDbContent) -> IO ()) -> IO ()
withDb = bracket getDbContent (\(hDb,_) -> clear hDb)
    with
      getDbContent = do
          hdb <- createDb
          db <- readDb hdb
          return (hdb, db)

[...]
 around withDb describe "getStarWarsNameDb" $ do
    it "contains Star Wars characters names" $ \(hdb, db) -> do
        ("Dark Vader" `isIn` db) `shouldNotBe` True
        ("JarJar Beans" `isIn` db) `shouldBe` True
    it "contains Star Wars citations" $ \(hdb, db) -> do
        ("May the fourth be with you." `isIn` db) `shouldBe` True
        ("Do. Or do not. There will be a trial." `isIn` db) `shouldBe` True
[...]

See, all we needed to do was to alter the withDb function and the pattern matching of the anonymous functions used in the in clause.

Wrap Up

Partially applied functions are an amazing tool. Thanks to them, we can add context little by little to a function call. Used properly, this can lead to some nice design tricks such as the bracket function.

Hope you enjoyed this ode to the bracket function :). We are constantly taking for granted some beautiful functions such as this one. I think it is important to both acknowledge and celebrate that, sometimes, software can be beautiful!

Writing a Twitch Overlay using Haskell

picnoir — Sat, 21 Oct 2017 12:48:00 +0200

I have been watching Jessica’s Mak streams lately. She is an indie game developper, but most of all, she has a kick ass overlay that shows what she is typing in real time.

I wanted the same one, I made it using Haskell, Gloss and Chipmunk via the Hipmunk binding.

Long story short, it ended up looking like this.

The source code is available in this git repository.

Gloss, display library

Gloss is a graphics library built upon OpenGL. The API is high level enough to mask a lot of low level nastiness.

To keep it short, the API is based around two functions:

An update function, which processes the input events and update the world data structure.
A display function, which renders the world as a Picture.

Those two functions are using two datastructures:

The world datastructure. This structure can be what you want, you will define it by yourself.
The Picture datastructure. It represents something that can be drawed to the screen.

I really like this second datastructure, its shape is highly interesting, let’s dig in it.

Hipmunk, physics library

Each object is composed by both a body and a shape. The body contains the physics informations (mass, moment of momentum, elasticity, …) while the shape just describes the bounding box of the object.

Capturing the Input

Now we can display funny bouncy letters, we still need to fix a problem: how to capture keyboard’s input? We were directly using the Gloss input system for debugging purposes. This sadly only work when the keywar window is focussed. Xorg only transferts the input events to the currently focussed window, which, in our case, is a problem.

How could we capture all the input events, regardless the focussed window? I found 3 solutions:

We could create an overlay window transferring the events to the underlying ones. This solution would imply to re-implement a subset of my window-manager features. We do not want to re-invent the wheel once again :).
We could bypass xorg and directly read Linux’s /dev/inputX input events. This solution is tricky, not only those events are using a binary data format, which make parsing them clumsy, but those events are raw, they are not applied to any keymap. If we really wanted to read those events, not only we should write the appropriate parser, but we also should support different keymaps.
We use a keylogger and tail -f its output.

The last solution was the simplest, we used the logkeys keylogger.

Side Note: Passwords

As mentionned in the github readme, you need to be extra careful when typing passwords, you clearly do not want to leak them while streaming.

I added a system which displays question marks in place of the actual letters to mitigate that. The audience will get the lenght of your password, but it is already the case if your keyboard is a bit noisy. Mine is noisy as hell, hence this is not really a problem to me.

Wireguard-Haskell: Getting Started

picnoir — Mon, 16 Oct 2017 09:48:00 +0200

Why Starting this Project?

After finishing DobadoBots, I was looking for a Haskell project in which I could be confronted with some performance and parallelism problems.

Wireguard seemed to be the perfect project for that. At the time, no userspace implementation was available, the specification seemed to be simple enough for being implemented by a single person in a couple of months.

Furthermore, a work in progress Haskell implementation called Nara was already available. The only contributor to this project sadly abandoned it, undocumented, several months ago. This is a great opportunity for me to complete what he started.

Starting Point

The userspace implementation is not providing a CLI by itself. Instead, we are using the wg CLI utility to communicate with the VPN daemon through a Unix socket.

The overall architecture looks like this.

We first need to test this RPC interface implementation. It was sadly broken. The RPC protocol changed and these changes were not reflected in the Haskell implementation.

We clearly need to first re-implement and test this interface.

Refactoring the RPC Interface

The new protocol specified here is text-based. Both the Wireguard device and peers are described using a several key/value couples.

We process the incoming data as a Conduit stream. Conduit comes with the conduit-attoparsec module, which let us parse the incoming stream using an Attoparsec monadic parser.

Attoparsec is based around the same concept as Parsec. The main difference being that while Parsec aims to parse user-generated inputs, Attoparsec targets machine to machine data formats. It is hence faster, at the cost of a weaker error reporting system.

You can find the implementation details of this parser in Nara’s git repository.

The four main takeaways from this implementation are:

Parsing an IP address is not a trivial task. You need to be extra cautious, especially while parsing an IPv6 address.
Do not hesitate to create a lot of sub-parsers. It makes the code both easily testable and more readable.
You need to be extra cautious with your test cases. Test extensively the edge cases. It is dead easy to screw things up when using a Parsec-like library.
The Alternative typeclass will hold your back while dealing with optional repetitive options. This typeclass is implemented by the Parser type: you can use it with everywhere.

Next steps

We need to re-implement the queue system. As it is, the packets are multiplexed in one TUN queue and one UDP queue. It creates a lot of latency in case of a handshake: we need the handshake to successfully complete before sending any data, regardless the peer we want to send the data to.

We first need to design a new queue system. When implementing it, we will need to test it extra carefully. It may be a good excuse to dive into property-based testing techniques.

DobadoBots: Project Wrap Up

picnoir — Thu, 28 Sep 2017 00:00:00 +0000

Yet another post about DobadoBots: my programming video-game. Right, let’s face it: it is done for two months now, this blog post is long overdue!

The video-game is now completely playable, I reached the MVP stage. Here’s a short video presenting the final result.

Let’s debrief this project.

Getting Familiar with Haskell

The main goal was to get some practical experience with Haskell. I had literally zero real-world experience before that.

After two months in, I can use Haskell productively enough. Yup, that’s a pretty steep and long learning curve. But it worth mentioning that I had no prior experience using a purely functional language.

It worth mentioning that you do not need to understand what’s a free monad or what are GADTs to be productive. You do not need a category theory PHD either.

There is a huge gap between getting productive enough and understanding what’s discussed on Haskell’s subreddit. This blog post made that click for me.

The main takeaway after this first practical experience is that a strong type system is a huge asset while refactoring. Having used Python quite extensively, a massive refactoring was always painful. You could easily introduce a lot of bugs as soon as your test suite was a bit weak. Haskell type system tends to cover your back most of the time and just refuse to compile your poorly refactored code.

I also found that assessing a task complexity very hard. Some things that would have been really long to do using a procedural language turns out being trivial using Haskell while some task that would have been simple using Python turns out being quite complicated using Haskell.

What has Been Unexpectedly Easy?

As previously discussed, I wrote my own language and parser for this project. This part has been unexpectedly easy.

Using Parsec for the parser part and Pretty for the pretty printer part made writing that nearly trivial. I discussed more in depth the parser implementation here.

What has Been Unexpectedly Hard?

Retrospectively speaking, the game part has been the hardest part. Without thinking about it, I basically wrote a whole game engine…

Choosing SDL2 as a graphics library turned out to be a poor choice. The library is low level, you basically need to implement everything. Turns out a video-game is not a trivial piece of software, you need to implement a button system, a menu engine, etc.

Writing the text editor also turned out being a real pain that took me almost two complete weeks.

The complexity is mainly hidden in small details.

What’s next?

I’m not going to lie, wrapping up this MVP was painful. I do not want to spend any more time on this project, working on it actually became a burden.

I do not plan to maintain this code, hence, I will not provide any binary distribution. You can, however, compile the Github repository if you want to play this game by yourself! It still has some rough edges and is only shipped with two stages. But you can easily add some more simply by looking at the JSON level format.

After getting familiar with Haskell, I think it is time to dive deeper in it and try doing something involving more performance tuning and some concurrent code. I’m going to take over the Haskell Wireguard implementation for the next few months!

DobadoBots: Writing a Text Editor

picnoir — Mon, 04 Sep 2017 00:00:00 +0000

Yup, yet another post on my programming videogame: Dobadobots.

Today, we are going to dig into the editor’s implementation.

Motivations

I wanted the game to be as enjoyable as possible, I wanted a quick write/feedback loop. Using an external editor would have killed this fast feedback loop, it was just not an option.

I first thought about integrating a pre-existing editor in it.

An option would have been to create a web-view and embedding an HTML5 based text editor. After writing a quick proof of concept, I ran into some problems: I did not find any lightweight way to make the editor communicate with the core Haskell application. The only solution I found would have been to make this happen by running a WebSocket server. This solution seemed totally overkill, I decided to dismiss it.

Another option would have been to find a library implementing the basic features of an editor. However, I have not been able to find any Haskell/SDL2 text editor. Perhaps this tech stack is a bit too much specific! :)

So, being unable to find anything meeting my requirements, I ended up writing my own editor.

The Editor’s Architecture

As usual I started by specifying the data types and, as usual, we encapsulate the editor state in a proper record structure.

data EditorState = EditorState {
  text               :: Text,
  cursorColumn       :: Int,
  cursorLine         :: Int
} deriving (Eq, Show)

It encapsulates two things:

The position of the cursor.
the text contained in the editor’s buffer.

In addition to this state, we also need to generate some SDL textures we will use to display both the text and the various editor elements. I ended embedding these elements to the renderer state which already contains most of the game textures.

data RendererState = RendererState {
  (...)
  editorCursor       :: (SDL.Texture, SDL.V2 CInt),
  codeTextures       :: [(SDL.Texture, SDL.V2 CInt)],
  parseErrorMess     :: [(SDL.Texture, SDL.V2 CInt)],
  parseErrorCursor   :: (SDL.Texture, SDL.V2 CInt),
  editor             :: EditorState,
}

We can see the code is stored in multiples textures: one per code line.

Text Display and Typefont Generator

We need to display the text on the screen. The code text is basically some UTF-8 characters separated by spaces and newlines.

In order to display some text using SDL2, you first need to create an OpenGL texture representing this text. To do that, you want to use a typefont generator. It is basically a program which generates an OpenGL texture according to a TTF font and a UTF-8 input text. In this project, I used the SDL2-TTF library.

This typefont generator is distributed as a C library.

Some Haskell bindings were already available, hence, I did not even need to write the corresponding FFI files. The tricky part for using this library was to use it in the continuous integration runner.

I use Travis as a CI, which sadly uses an outdated ubuntu. This ubuntu does not provide any kind of pre-compiled SDL2-TTF binary, you need it to build it by yourself.

You will need libgl1-mesa-dev to build it. Because of Travis’s exoteric path configuration, you will also need to add the /usr/local/lib directory to the LD_LIBRARY_PATH environment variable.

Here’s the corresponding .travis.yml configuration section.

before_install:
- sudo apt-get install -y libgl1-mesa-dev
- wget http://libsdl.org/release/SDL2-2.0.5.tar.gz -O - | tar xz
- cd SDL2-2.0.5 && ./configure && make -j && sudo make install && cd ..
- wget https://www.libsdl.org/projects/SDL_ttf/release/SDL2_ttf-2.0.14.tar.gz -O - | tar xz
- cd SDL2_ttf-2.0.14 && ./configure && make -j && sudo make install && cd ..
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/

Rendering the Editor

The renderer is in charge of both input events handling and graphics drawing.

We need to be compatible with both Unixes and Windows systems. SDL2 has been used as a hardware abstraction layer here.

The renderer is directly plugged into the main graphic event loop which looks like this:

Check any kind of input.
Potentially append the event to the editor state.
Check the syntax of the code generated by the new event.
If the syntax is valid, load the new AST in the game engine.
Generate the new text textures.
Draw everything on the screen.

For more details about this implementation, you can check the implementation itself.

Syntax Error Checking

To keep the test/evaluation feedback loop short, we include a syntax checker directly in the editor.

Thanks to the parser (see the previous post), this task was quite straightforward.

Parsec not only returns a short text describing the error and the potential fix but also returns a position (a column line couple) helping to localize the error. Using this localization, you can highlight this part while adding an error message at the bottom of the editor.

As you can see on the above screenshot, we display a red square next to the error. We then render the error message (if any) in a proper OpenGL texture and display it at the bottom of the screen.

Aaaaaand, that’s pretty much it. I omitted most of the algorithm describing the editor’s behavior, but trust me, there are many. I will probably write another article focusing on this part in the future.

That was the first time I was writing a text editor from scratch. I can now confirm that this is not, even remotely, a trivial task.

If you face someday the same problem, let me share with you this advice: find a freaking way to integrate an already existing editor to your program, do not write your own one!

DobadoBots: Implementing the Parser

picnoir — Thu, 20 Jul 2017 00:00:00 +0000

Lately, I have been working on a video-game called DobadoBots. This game is about programming a robot’s artificial intelligence to solve mazes.

The robot is materialized by a white triangle. The goal is to reach the objective (orange square) while avoiding several obstacles.

Instead of using a standard embeddable script language such as LUA, I went the custom way and wrote my own language.

In a previous post, we detailed the AST specification and structure. In this post, we go through the implementation of the parser.

As a quick reminder, the language we are attempting to parse looks like this

IF laserDistance > 20
  moveForward
ELSE
  IF laserScan = objective
    moveForward
  ELSE
   IF laserScan = obstacle
       turnLeft
   ELSE faceObjectives

TL;DR, show me the code instead: here you go.

Parsec

As a new Haskell developer, I went through the various available parsing libraries. I finally decided to use the Parsec library. I have been amazed by it so far. This is probably the best parsing library I ever used.

The base concept of Parsec is to combine multiple Parser monads in order to create a monad that will either parse your AST, either return a detailed error message.

Parsec is providing basic Parser monads for the most common builtin datatypes (e.g. space, newline, character, integer, double, etc.). You can then build more useful parsers by assembling those basic ones using some combinators.

I find this design very elegant, however, after using this library a bit, you can very quickly mess things up. Therefore, I would recommend writing a lot of unit tests before implementing your parser.

Let’s Parse Dobadolang

There are several ways to use parsec. You can use the monadic approach and take advantage of the do notation or you can use the applicative approach and use the applicative style to combine the various parsers.

I decided to use the applicative approach for this project as it was a more compact notation.

We will build our compiler using the bottom up approach, let’s start by specifying the AST’s leaves: the action token.

Parsing Action Tokens

Each token of our language is defined by a specific string. Parsec provides a string parser which can parse a simple string case-sensitively. We can, for example, parse the move forward token like this:

string "moveForward"

Alright, but if we look at the type of string, we see we done something wrong.

Stream s m Char => String -> ParsecT s u m String

Yup, the actual returned value will be the string itself. We do not want that, we want to return the AST element instead.

We will use the <$ version of fmap. This inflix function basically discards the return value contained in the parsec parser to instead return the value specified at its left.

MoveForward <$ string "moveForward"

We use a similar technique to parse the other action tokens.

actionParser :: CharParser () ActionToken
actionParser = MoveForward     <$ string "moveForward"
          <|>  try (TurnLeft   <$ string "turnLeft")
          <|>  try (TurnRight  <$ string "turnRight")
          <|>  FaceObjective   <$ string "faceObjective"
          <|>  ChangeObjective <$ string "changeObjective"

There are two new tricks here:

the try function prevents parsers from consuming too many inputs. You can check a detailed explanation of this function on hackage.
the <|> combinator here materializes a choice. We use it to try parsing the various tokens.

Parsing AST’s Nodes

The whole DobadoBots language is built around the conditional structure. Naturally, we want to parse that structure.

conditionParser :: CharParser () Cond
conditionParser = Cond <$>
                    ((spaces *> string "IF") *> logicExprParser <* newline)
                  <*>
                    (spaces *> parseNested <* newline)
                  <*>
                    ((spaces *> string "ELSE") *> (newline *> (spaces *> parseNested)))
                  where
                    parseNested = Token <$> actionParser
                              <|> conditionParser

Writing this function was my “aha!” moment, this is when I started to fancy parsec. Here, we just had to combine the previously implemented logic expression and action token parsers.

This is basically why I love to use parsec with the applicative notation. The resulting code seems to be written using a DSL while keeping Haskell’s strong type checking. It is both highly readable and easy to reuse.

Now we have defined the various parsec Parser instances to generate DobadoBots AST, time to wrap-up the parser.

Wrapping up Everything

Once the Parser monad is defined, we just need to call the parse function.

parseScript :: Text -> Either ParseError Cond
parseScript script = parse scriptFile "Error while parsing script: " $ unpack script .

As shown in this function definition, the parse function returns either a parsing error, which contains both the error location and a short message describing what should be there in order to have a valid input.

Testing the Implementation

As I mentioned in the introduction, it is quite easy to screw things up when parsing. In order to mitigate that, it seems like a good idea to write some test cases up front.

I used the HSpec unit test task runner together with stack for running the tests. It is a good idea to inline the fixtures in the test file to avoid any filesystem side effect while writing purely functional tests. I used the embed file library to achieve that.

You can find the complete test case on the DobadoBots GitHub repository.

Each test case looks basically like this:

it "should parse simple cond =" $
  parseScript (decodeUtf8 $(embedFile "test/fixtures/script/simplecond.script")) `shouldBe` Right (Cond (CmpLogicInt $ Eq LaserDistance 1)(Token TurnLeft) (Token MoveForward))

We are inlining a fixture to check the parse result with an expected AST.

Another testing approach we could implement would be the round trip property check.

The goal of this test is to ensure that parse(print(parse(script))) == parse(script) is always true. Basically, it checks if your parser/pretty printer couple is correct. I did not figure out how to setup stack to both test the library using HSpec (unit tests) and Quickcheck (property-based tests). I will probably figure out how to do that and write a follow-up article in the near future.

I do not see anything more to add, as you can see, writing a parser using parsec is not that big of a deal. In the next article, we will cover up Dobadobot’s text editor.

DobadoBots: Specifying the Language

picnoir — Sun, 25 Jun 2017 00:00:00 +0000

Lately, I have been working on a video-game called DobadoBots. This game is about programming a robot’s articial intelligence to solve mazes.

The robot is materialized by a white triangle. The goal is to reach the objective (orange square) while avoiding several obstacles.

Instead of using a standard embeddable script language such as LUA, I went the custom way and wrote my own language.

Specifying the Language

I had only one idea in mind when started to specify this language: keep it as simple as possible. I started wondering, what do we really need to program this kind of artificial intelligence? What is the smallest set of operations the language needs to support?

I decided to specify the language as a decision tree. Each condition being a node, each robot movement being a leaf. No loops, no variable assignments, no functions. We do not want to embarrass the player with unnecessary concepts.

Basically, the language structure looks like this.

  IF [sensor]
       [action or nested condition]
  ELSE
       [action or nested condition]

The sensor statement is a boolean expression. It is evaluated in order to decide which subtree should be next executed.

Every sensor evaluation results either in an action token, which gives the robot a movement instruction, or in a nested condition, which gives us a way to refine the behaviour.

In the end, a robot’s AI will be defined by something like this:

IF laserDistance > 20
  moveForward
ELSE
  IF laserScan = objective
    moveForward
  ELSE
   IF laserScan = obstacle
       turnLeft
   ELSE faceObjectives

Sensors and Actions

We then needed to determine the minimal operations subset required to control the robot though the maze.

We finally end up with the following sensors list:

laserDistance: checks the distance between the robot and the nearest object.
laserScan: checks what kind of object is facing the robot. It can be an obstacle, a wall, another robot or the objective.
objectiveDistance: checks the distance between the robot and the objective.

ObjectiveDistance was not necessary but makes solutions less confusing in some cases.

Regarding the actions, we ended up keeping 4 of them:

moveForward: the robot moves one step ahead.
turnLeft: the robot rotates counterclockwise.
turnRight: the robot rotates clockwise.
faceObjective: the robot faces the objective.

Implementing the Abstract Syntax Tree (AST)

Now we specified the language, we still need to implement its AST. We will use the Haskell programming language for this part.

We start specifying data structures for both actions and sensors tokens. This problem is quite straightforward, we just need to use Haskell’s algebraic data types.

data ActionToken = MoveForward
                 | TurnLeft
                 | TurnRight
                 | FaceObjective
                 | ChangeObjective
                 deriving (Show, Eq)

data SensorToken = LaserDistance
                 | LaserScan
                 | ObjectiveDistance
                 deriving (Show, Eq)

We then need to specify the logic expression data type. We need to keep in mind that comparing a laser distance with an integer is quite different from comparing a laserScan with a Collider.

In order to simplify both the parser and the interpreter, we restricted as much as possible the type definition for both of these kind of logic expression.

data LogicExpr = CmpCollider SensorToken Collider
               | CmpLogicInt (CmpInteger SensorToken)
               deriving (Show, Eq)

data CmpInteger a = Sup a Integer
                  | Inf a Integer
                  | Eq a Integer
                 deriving (Show, Eq)

data Collider = Obstacle
              | Objective
              | Wall
              | Robot
              deriving (Show, Eq)

Now we have implemented all the basic data types, we just need to compose them in order to build the actual tree.

To do so, we use a recursive data type.

data Cond = Token ActionToken
          | Cond { sensor    :: LogicExpr
                 , ifValid   :: Cond
                 , ifInvalid :: Cond
                 } deriving (Show, Eq)

We now successfully implemented our AST, Cond being its root.

In next article, we will dive in the associated parser’s implementation.

One Year of FOSS

picnoir — Sun, 18 Jun 2017 00:00:00 +0000

One Year of FOSS

Hello internet friends, long time no see.

After doing a total revamp of this weblog, I think it is finally time to explain what’s happening in my life.

Two months ago, I decided to quit my job. I was tired of Paris’s pollution. I was tired of Paris’s shitty public transportation. I was not really happy doing my job. I had one year of runway. Long story short, I moved to Bayonne, a really nice place in southern France.

Sadly, as nice as this place is, we cannot call it a tech-friendly place. There is not a lot of software-related jobs there, even less interesting ones. So instead of looking for a new job, I decided to do something bold. Why not, instead of constantly whining about low quality software, trying to write something I can be proud of, software I personally enjoy, and in the meantime, forgetting about getting any income for one year?

The Manifesto

I came up with this manifesto:

Do not think about money or employment, only think about learning and creating something useful.
Each started project will be a completed one. Even if it turns out to be a failure or a bad idea, reach at least the minimal viable product stade. There is no escape.
Do not parallelize projects. You should have one and only one software project at a time.
Release everything you do in an opensource form. Preferably using the MIT V3 license.
Try to cooperate with other people as much as possible in everything you do.
Share your experiment publicly in an honest way. No idealistic bullshit, if something you do sucks, say it and explain why.
Try to suck as less at possible in your job. You are here to learn about how to write highly reliable software, regardless how long it takes. Time will not be a constraint this year.

I have been following these rules for the last two months.

What Have I Done so Far?

I first started learning haskell. Yes, yet another time. This was the third time I was attempting to learn that damn language. I can finally say that this was definetely the last try. I grasped enough concepts to have a real world usage of it. It took me one month: the first one of my sabbatical.

This time, instead of trying to get started in two days using some crappy internet resources, I went the hard way. I studied the real world haskell book. I couldn’t recommand it enough, it explores the language in depth with enough practical exercises and examples to grasp the presented concepts. As I was mentionning, it took me around 3 weeks fulltime to finish that book. So be warned, Haskell is a great language, but do not expect to grasp the basics in 2 days as you would do with yet another c-based language.

After getting through this book, I decided to start right away a Haskell project from scratch, just to prove myself I was able to build something practical with this language.

This is when I decided to create a videogame. Partly because it was a child’s dream, partly because I was curious about how to build a purely functional game.

I will write more in-depth about DobadoBots in some upcoming articles, but long story short, after one month of development, I am yak-shaving its minimal version. All the features are now implemented, it only needs some polishing.

Radical Lifestyle Change

Starting a new life phase is always a hustle. This time was no different.

First, the city. Moving from a 2 million inhabitants metropolis to a small 15 000 inhabitants city was a hell of a change. In the end, I couldn’t be more happy. The city center is pedestrians only: no more pollution nor car noise all day long. Everything is within reach using a bycicle: no more crowded public transportations. I can reach the beach in a 20 minutes bycicle ride. I really like this place, I actually would love to stay here even after this sabbatical.

Next, work. I am not going to lie, this has been the harder part. Moving from a on site white-collar job to a completely constraint-free homeschooled one has been greatly affecting my show up rate! During the first month, I have been constantly struggling with self-motivation. Not having external constraints and sharing the same space for both personal life and work turned out to be encouraging me procrastinating. Luckily, I finally catched up momentum and have been effectively working fulltime for the past month. I guess this struggle will never end, but at least I now know that I can effectively beat procrastination!

That’s pretty much it so far, I think I will try to write an update about this sabbatical after every project wrapup.

Real World Haskell Chapter 9 Solutions

picnoir — Tue, 04 Apr 2017 00:00:00 +0000

Exercise P.221

Is the order in which we call bracket and handle important?

Yup, it is pretty important: the code executed during the in-between statement of bracket still can raise an exception. We do not want our application to crash in case of an error while opening the file, therefore, we still need the handle statement.

We could put the handle statement inside of the bracket statement, but since no exception would be thrown, we would risk to leak some resources.

Exercise P.228

Question 1

How could we traverse the directory three in reverse alphabetic order?

Well, it is quite straightforward, before iterating on directories, we apply the order function on the list containing them. Therefore, we only need to apply reverse as the order function.

ControllerVisit> ControllerVisit.traverse reverse "/home/minoulefou/test"

Question 2

Implement an order function which traverses the tree in postorder.

For each file info type handled by the order function, we have two options:

The file is a directory: we want to first handle its children then handle the directory.
The file is not a directory: we will apply the identity function to this file.

I implemented this recursive function as follows:

postOrder :: [Info] -> [Info]
postOrder (x:xs) = if isDirectory x
  then postOrder xs ++ [x]
  else x:postOrder xs
postOrder [] = []

As you can see, it the implementation is quite straightforward.

There may be some other ones, do not hesitate to shout me out if you found something else.

Question 3

We want to adapt previously implemented predicates with the new Info datatype.

First we need to express InfoP using info:

type InfoP a = Info -> a

We will now implement predicates equality and comparaison: we’ll need to reimplement liftP with the new InfoP type:

liftP :: (a -> b -> c) -> InfoP a -> b -> InfoP c
liftP op infoP test info = infoP info `op` test

Once we implemented this function, implementing equality and inequity predicates is trivial:

greaterP, lesserP :: (Ord a) => InfoP a -> a -> InfoP Bool
greaterP = liftP (>)
lesserP = liftP (<)

equalsP :: (Eq a) => InfoP a -> a -> InfoP Bool
equalsP = liftP (==)

Once again, in order to implement combinators, we’ll use another lift function.

liftP2 :: (a -> b -> c) -> InfoP a -> InfoP b -> InfoP c
liftP2 op pred1 pred2 info = pred1 info `op` pred2 info

andP, orP :: InfoP Bool -> InfoP Bool -> InfoP Bool
andP = liftP2 (&&)
orP = liftP2 (||)

As for the previous question, we’ll use some infix notations for each operation: it is more handy to use.

(==?)::(Eq a) => InfoP a -> a -> InfoP Bool
(==?) = equalsP

(>?), (<?) :: (Ord a) => InfoP a -> a -> InfoP Bool
(>?) = greaterP
(<?) = lesserP

(&&?),(||?):: InfoP Bool -> InfoP Bool -> InfoP Bool
(&&?) = andP
(||?) = orP

Question 4

Write a wrapper for traverse that lets you control traversal using one predicate and filter results using another.

Here, we want to add a behaviour to the ord function. Not only we want that function to order the results, be we also want it to filter them. An easy approach to this problem seems to be composing the filter function with the order function:

traverse' order pred path = ControllerVisit.traverse (order . filter pred) path

Exercise P. 232

Question 1

Modify foldtree to allow the caller to change the order of traversal of entries in a directory.

First, we need to specify the new fold function signature:

foldTree' ::([FilePath] -> [FilePath]) -> Iterator a -> a -> FilePath -> IO a
foldTree' order iter initSeed path = ...

We will plug this function to the existing foldTree on the fold subfunction as follow:

fold seed subpath = do 
  paths <- getUsefulContent subpath
  walk seed $ order paths

The remaining code will still unchanged.

Aaaand, the two remaining exercises are quite similar to the previous ones, I will pass on that :)

See you soon for chapter 10 solutions!

Real World Haskell Chapter 8 Solutions

picnoir — Tue, 28 Mar 2017 00:00:00 +0000

Hello, it’s been a long time.

Despite having done some progress on the resolution of the exercises of this book, I haven’t blogged about my solutions, which is a shame. The solutions are available in this git repository though.

Let’s start again for the end of chapter 8.

Exercises P.205

Here, we need to implement a case insensitive version of the glob checker.

Despite having looked on Hoogle for some elegant solution, I did not found anything suitable… I ended up writing a dirty hack.

The idea is to add a boolean flag to the matches glob. If this boolean is unset - meaning that we do not want to be case sensitive - we uppercase both the filename and the pattern.

Despite not being an elegant solution, it works.

matchesGlob :: FilePath -> String -> Bool -> Bool
matchesGlob name pat True = name =~ globToRegex pat
matchesGlob name pat False = map toUpper name =~ globToRegex (map toUpper pat)

Exercises P.210

Generalisation of the case sensitivity

First of all, we need a way to determine that we are on a windows system. According to Hoogle, the isPathSeparator function seems to be a pretty good candidate. If '\' is a path separator, then we are on a windows environment.

So first, let’s select the right glob matching func:

let globMatchingFunc = if isPathSeparator '\\'
                       then matchesGlobCaseSensiitive
                       else matchesGlobNotCaseSensitive

Then we’ll use this function in the filter part:

 return (filter (globMatchingFunc pat) names')

You can find the entire file in order to have a higher level-view here

Replacement for doesNameExist on POSIX systems

fileExists would be a good replacement for doesNameExists.

We could use that function because everything can be seen as a file on a POSIX system.

However, using this specificity would make our software POSIX specific thus impossible to run on a windows system.

Recursive wild card implementation

In order to implement this feature, we need several parts. The first one will be a function that returns the childrens of a directory.

This feature has been implemented using the following algorithm:

getChildrenDirs :: String -> IO [String]
getChildrenDirs path = getChildrenDirs' [path]
  where getChildrenDirs' :: [String] -> IO [String]
        getChildrenDirs' paths = do
          pathNames <- forM paths $ \path -> do
            dirContent <- listDirectory $ dropTrailingPathSeparator path
            let absDirContent = map (\p -> dropTrailingPathSeparator path </> p) dirContent
            dirs <- filterM doesDirectoryExist absDirContent
            if null dirs
            then return [dropTrailingPathSeparator path]
            else do children <- getChildrenDirs' dirs
                    return (path:children)
          return (concat pathNames)

Nothing too fancy here, just a recursive tree traversal. The only new thing (at least for me at this point) was the use of filterM which let us use filter on a IO() list.

Now we have this function, we can:

Detect the ** pattern somewhere in the pattern.
List the subdirectories according the beginning of the path given in parameter.
Pattern match the files contained in these folders according the basename filter.

In order to implement point 1, we’ll need to implement a simple predicate detecting the “double wildcard”:

containsDoubleWildcard :: FilePath -> Bool
containsDoubleWildcard path = path =~ "^.*\\*\\*.*$"

If we detect this kind of pattern, we want to add the current dir and its children to the file lookup. In order to do that, we will add these directories to the “dirs” list in the namesMatching as follow:

dirs <- if isPattern dirName || isPattern baseName
    then if containsDoubleWildcard baseName
           then getChildrenDirs (dirName)
           else namesMatching $ dropTrailingPathSeparator dirName
         else return [dirName]

The rest of the namesMatching will take care of the pattern matching on the files contained in the dirs list.

You can find the complete solution to the exercise on this git repository.

The last question being trivial, we are done with this chapter :)

Real World Haskell Chapter 4 Solutions

picnoir — Sun, 18 Dec 2016 00:00:00 +0000

Okay, in this chapter, we will apparently learn more about common techniques in FP, can’t wait!

Exercises page 84

Let’s rewrite safe versions of partial list functions

The first two functions are very straightforward: a bit of pattern matching, some identification of edge cases and we’re done.

safeHead :: [a] -> Maybe a
safeHead [] = Nothing
safeHead (x:xs) = Just x

safeTail :: [a] -> Maybe [a]
safeTail [] = Nothing
safeTail (x:xs) = Just xs

safeLast :: [a] -> Maybe a
safeLast [] = Nothing
safeLast [x] = Just x
safeLast (x:xs) = safeLast xs

It gets a bit more tricky for the last one. Indeed, we need to recursively construct a list from the end to the beginning and wrap everything in a Maybe. In order to do that, I first calculate the list in a sub-function defined in a where section and then encapsulate everything in the Maybe.

safeInit :: [a] -> Maybe [a]
safeInit [] = Nothing
safeInit x = Just (end x)
  where end :: [a] -> [a]
        end [a] = []
        end (x:xs) = x:(end xs)

SplitWith: an inverted word

This definition is so confusing… I am not sure about getting the specification right, but from what I understand:

We have a list we want to split in sublists.
We need to evaluate an expression for each list member.
If this evaluation is false, we need to split the main list at this point.

splitWith :: (a -> Bool) -> [a] -> [[a]]
splitWith _ [] = []
splitWith function list = [(takeWhile function list)] ++ (splitWith function remaining)
    where
        endOfList = dropWhile function list
        remaining
            | null endOfList = []
            | otherwise = tail (endOfList)

First, let’s think about edge cases. Obviously, the split of an empty list will results to an empty list.

Then, we need to write a recursive pass on the list. During this pass, we will first use the takeWhile function in order to take elements that verify the function. Then we will append these elements to the rest of the list minus the “delimiter”. I insist, you need to remove this delimiter, otherwise you’ll end up with an infinite loop.

In order to calculate the rest of the list (list minus delimiter), I used the function tail. As we seen in the last exercise, this function is quite unsafe. We need to be extra careful that we are not tailing an empty list.

Again, I am not sure that I got the specification right in the first place.

Printing first word for each input line

This exercise is quite simple, we want to:

Read a file.
Take the first character of each line.
Print it to another file.

-- We can reuse this code which is given in
-- the previous chapter
interactWith function inputFile outputfile = do
    input <- readFile inputFile
    writeFile outputfile (function input)

main = mainWith myFunction
	where myFunction = getFirstWordsOfFile
    	      mainWith function = do
    	      args <- getArgs
    	      case args of
        	      [input,output] -> interactWith function input output
	       	      _ -> putStrLn "error: exactly two arguments needed"

firstWord :: String -> String
firstWord line = head (words line)

firstWordsOfFile :: [String] -> String
firstWordsOfFile [] = ""
firstWordsOfFile (x:xs) = (firstWord x) ++ " " ++ (firstWordsOfFile xs)

getFirstWordsOfFile :: String -> String
getFirstWordsOfFile fileStr = firstWordsOfFile (lines fileStr)

As you see, this is quite simple, there are 3 functions:

firstWord: which chops a line and returns the first word.
firstWordsOfFile: which takes the lines of the file one by one and returns only the first words.
getFirstWordsOfFile: which takes the file string, chop it line by line and returns the first word.

I could have used where or let in order to encapsulate the intermediates functions. But I haven’t…

Transposing text from a file

Aaaaaaand, dunno. Seriously, I do not get how to do that without the map function… Shout me out if you found the solution!

Exercises page 98

Rewriting asInt using folds

This one is easy, we just fold from the left the char expression and we keep the integer representation up to date by multiplying by 10.

import Data.Char (digitToInt)
asInt_fold :: String -> Int
asInt_fold str = foldl foldFunc 0 str
    where foldFunc :: Int -> Char -> Int
          foldFunc acc elem = (digitToInt elem) + (acc * 10)

Handling negative numbers

Okay, now we want to handle negative numbers.

asInt_fold :: String -> Int
asInt_fold (x:xs)
    | x == '-' = (-1) * asInt_fold xs
asInt_fold str = foldl foldFunc 0 str
    where foldFunc :: Int -> Char -> Int
          foldFunc acc elem = (digitToInt elem) + (acc * 10)

In order to do that, we pattern match the head of the string and we filter the - using guards.

Handling common errors

Let’s add some conditions in order to handle common errors.

asInt_fold :: String -> Int
asInt_fold [] = error "Cannot convert [] to Int"
asInt_fold (x:xs)
    | x == '-' && null xs  = error "Cannot convert this expression to Int."
    | x == '-' = (-1) * asInt_fold xs
asInt_fold str = foldl foldFunc 0 str
    where foldFunc :: Int -> Char -> Int
          foldFunc acc elem = (digitToInt elem) + (acc * 10)

Please note that I did not found any direct way to correctly handle Int overflow other than using the safeint type (which I did not implemented here).

Safe error handling using either

Here, we’re going to wrap up our error handling in the either

type ErrorMessage = String

asInt_either :: String -> Either ErrorMessage Int
asInt_either [] = Left "Cannot convert [] to Int"
asInt_either (x:xs)
    | x == '-' && null xs  = Left "Cannot convert this expression to Int."
    | x == '-' = fmap (*(-1)) (asInt_either xs)
asInt_either str = Right (foldl foldFunc 0 str)
    where foldFunc :: Int -> Char -> Int
          foldFunc acc elem = (digitToInt elem) + (acc * 10)

I did not found a way to solve this problem without using fmap which has not been introduced yet. Maybe I missed something here or maybe this is an error in the book…

Implementing concat using foldr

Nothing to say here, this is straightforward

concat_foldr :: [[a]] -> [a]
concat_foldr input = foldr foldFunc [] input
    where foldFunc :: [a] -> [a] -> [a]
          foldFunc acc elem = acc ++ elem

Implementing takewhile using foldr

Again, nothing special here.

takeWhile' :: (a -> Bool) -> [a] -> [a]
takeWhile' _ [] = []
takeWhile' pred (x:xs)
    | pred x = x:takeWhile' pred xs
    | otherwise = []

takeWhileFold :: (a -> Bool) -> [a] -> [a]
takeWhileFold pred input = foldr foldFunc [] input
    where foldFunc elem acc
            | pred elem = elem:acc
            | otherwise = []

Implementing List’s groupBy

After playing a bit with ghci, it turns out that group by is taking a 2 parameters function and a list and groups the list’s elements using the function given in parameters and the first element. As soon as it is impossible to group with the first element, a new group is created and so on. For instance:

*Main Data.List> groupBy (\x y-> x + 1 == y) [1,2,2,3]
[[1,2,2],[3]]

Okay, let’s implement that.

groupBy' :: (a -> a -> Bool) -> [a] -> [[a]]
groupBy' pred input = foldl foldFunc [] input
    where foldFunc [] elem = [[elem]]
          foldFunc acc elem
            | pred (head(last acc)) elem = (init acc) ++ [(last acc ++ [elem])]
            | otherwise = acc ++ [[elem]]

This problem turned out to be very tricky. For each element, we want to check if we want to pair it with the current subgroup.

In order to do that, we need to:

Get the current subgroup using last acc.
Get the first element of this subgroup which is the group “master” using head(subgroup)
Check if pred is true with the “master” and the current elem.
- If it is, we need to append elem to the current subgroup.
- Otherwise, we create a new subgroup with elem as the “master”

As you can see, the code is really messy. I think that I should have created a specific datatype for the subgroup, it could have made pattern matching easier…

Aaaaand that’s it!!! We done it!!! Time for celebration now!