Planet Haskell

Donnacha Oisín Kidney: Tries for Polynomials

Tue, 28 Apr 2026 00:00:00 +0000

Posted on April 28, 2026

Tags: Haskell

One of my favourite Haskell papers is McIlroyâ€™s wonderful â€œPower Series, Power Seriousâ€� (1999). The paper is about power series, which are a type of infinite sums that behave like (infinite) polynomials. For example, <semantics>cos<annotation encoding="application/x-tex">\cos</annotation></semantics> can be represented by the following power series:

<semantics>cos⁡(x)=1âˆ’x22!+x44!âˆ’x66!+x88!âˆ’x1010!+â‹¯<annotation encoding="application/x-tex">\cos(x) = 1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \frac{x^6}{6!} + \frac{x^8}{8!} - \frac{x^{10}}{10!} + \cdots</annotation></semantics>

A power series is characterised fully by its coefficients, meaning that we can represent one as an infinite stream of rational numbers. In Haskell, we often use lazy lists to represent streams, so we can encode a power series with the following type:

type PowerSeries = [Rational]

In this encoding, we can write <semantics>cos<annotation encoding="application/x-tex">\cos</annotation></semantics> as the following:

cos :: PowerSeries
cos = zipWith (*) (cycle [1,0,-1,0]) (scanl (/) 1 [1..])

>>> cos
[1,0,-1/2,0,1/24,0,-1/720,...

We can also build <semantics>sin<annotation encoding="application/x-tex">\sin</annotation></semantics>:

sin = zipWith (*) (cycle [0,1,0,-1]) (scanl (/) 1 [1..])

While it can be difficult and unintuitive to work with infinite series like the ones above, happily we can define all of the normal numeric operations on power series as (lazy) list-manipulation programs:

instance Num PowerSeries where

  (x:xs) + (y:ys) = (x+y) : (xs + ys)

  (x:xs) * ys = map (x*) ys + (0 : xs * ys)

  negate = map negate

  fromInteger n = fromInteger n : repeat 0

(if you try and put this code into a Haskell interpreter youâ€™ll get all sorts of warnings; Iâ€™ll put the full code for this post below with all of the imports and pragmas you need to get it to work)

McIlroy (1999) goes through the various algorithms and numeric operations that can be implemented on this representation, but at this point I would like to diverge from the paper and turn our focus to finite polynomials. Like a power series, a finite polynomial can be represented by a list of coefficients:

type Polynomial = [Rational]

And, even though the underlying list is finite rather than infinite, the numeric operations work basically the same way as they do on power series. We just need to add clauses in each function to handle the empty list:

instance Num Polynomial where

  []     + ys     = ys
  xs     + []     = xs
  (x:xs) + (y:ys) = (x+y) : (xs + ys)

  []     * _  = []
  (x:xs) * ys = map (x*) ys + (0 : xs * ys)

  negate = map negate

  fromInteger n = [fromInteger n]

The only semantic trickiness with this representation is that itâ€™s important to quotient by trailing zeroes.

instance Eq Polynomial where
  [] == ys = all (0==) ys
  xs == [] = all (0==) xs
  (x:xs) == (y:ys) = (x == y) && (xs == ys)

Evaluation and Hornerâ€™s Rule

The definition of a power series above suggests that we should implement evaluation using exponentiation and indices:

eval :: Polynomial -> Rational -> Rational
eval p x = sum (zipWith (\a i -> a * x^i) p [0..])

And this does in fact give us the correct answer. Consider the polynomial <semantics>4+2x+5x2âˆ’x3<annotation encoding="application/x-tex">4 + 2x + 5x^2 - x^3</annotation></semantics>:

poly = [4,2,5,-1] -- 4 + 2x + 5xÂ² - xÂ³
eval poly x = eval [4,0,5,-1] x
            = sum (zipWith (\a i -> a * x ^ i) [4,2,5,-1] [0..])
            = 4*x^0 + 2*x^1 + 5*x^2 + (-1)*x^3
            = 4 + 2*x + 5*x^2 - x^3

However, this evaluation algorithm is unsatisfactory in one respect: it performs a lot of multiplication. In numeric programs, we generally want to minimise the number of multiplications performed, since multiplication is a relatively expensive operation (when compared to addition or subtraction). In the example above, it takes six multiplications to compute the result: one for <semantics>2x=2Ã—x<annotation encoding="application/x-tex">2x = 2 \times x</annotation></semantics>, two for <semantics>5x2=5Ã—xÃ—x<annotation encoding="application/x-tex">5x^2 = 5 \times x \times x</annotation></semantics>, and three for <semantics>âˆ’x3=âˆ’1Ã—xÃ—xÃ—x<annotation encoding="application/x-tex">-x^3 = -1 \times x \times x \times x</annotation></semantics>. In general, for a polynomial of degree <semantics>n<annotation encoding="application/x-tex">n</annotation></semantics>, the above implementation of eval will perform <semantics>ğ�’ª(n2)<annotation encoding="application/x-tex">\mathcal{O}(n^2)</annotation></semantics> multiplications.

There is, however, a trick that can bring the number of multiplications down to <semantics>ğ�’ª(n)<annotation encoding="application/x-tex">\mathcal{O}(n)</annotation></semantics>: Hornerâ€™s rule. The basic idea is to rewrite the expanded polynomial <semantics>4+2x+5x2âˆ’x3<annotation encoding="application/x-tex">4 + 2x + 5x^2 - x^3</annotation></semantics> into a factorised form: <semantics>4+x(2+x(5+x(âˆ’1)))<annotation encoding="application/x-tex">4 + x(2 + x(5 + x(-1)))</annotation></semantics>. If we evaluate this expression directly, we will only have to perform three multiplications (and we donâ€™t even have to perform any extra additions as compensation). While Hornerâ€™s rule is really quite a simple trick, the generalised pattern is surprisingly powerful (Gibbons 2011). Indeed, the representation I develop in this post is basically a data structure encoding of Hornerâ€™s rule.

Before getting there, however, letâ€™s return to our list-based polynomial, and look at using Hornerâ€™s rule to implement eval. Interestingly, the list-based representation has kind of already performed our factorisation for us. As a result, Hornerâ€™s rule evaluation is actually more natural to implement than the expanded version above.

eval :: Polynomial -> Rational -> Rational
eval xs x = foldr (\a p -> a + x * p) 0 xs

Multiple Variables

A cool trick with this representation is that if you want to support multiple variables you can smuggle them in through the coefficients. A polynomial in two variables is the same as a polynomial with coefficients drawn from another polynomial.

type TwoVar = [Polynomial]

To save us having to write a separate Num instance for TwoVar, we can instead generalise the Num instance on Polynomial above:

instance Num a => Num [a] where

The rest of the instance is the same. Now, we can write 5 ^ 2 :: Polynomial or 6 :: TwoVar and it will just work.

We also have to generalise the type of eval slightly:

eval :: Num a => [a] -> a -> a

but again, the implementation remains the same.

With this machinery, we can now write and evaluate polynomials in 2 variables:

eval2 :: TwoVar -> Rational -> Rational -> Rational
eval2 p x y = eval (eval p [x]) y

var :: Num a => [a]
var = [0,1]

x = var
y = [var]

poly = 2 * x ^ 2 - y ^ 3 + 4

>>> poly
[[4,0,0,-1],[0],[2]]

>>> eval2 poly 2 3
-15

We can even use some typeclass shenanigans to build a generalised evaluator that works with any fixed number of variables.

Implementation of an Evaluator for Polynomials in Arbitrary Variables

instance Num n => Num (e -> n) where
  fromInteger = const . fromInteger
  (f + g) x = f x + g x
  (f * g) x = f x * g x

  abs = (abs .)
  signum = (signum .)
  negate = (negate .)

class Num r => Poly p r | p -> r, r -> p where
  evalN :: p -> r

instance Poly Integer Integer where
  evalN = id

instance Poly p r => Poly [p] (Integer -> r) where
  evalN xs x = foldr (\a s -> evalN a + fromInteger x * s) 0 xs

>>> evalN poly 2 3
-15

z = [[var]]

>>> evalN (poly + z) 2 3 1
-14

Sums of Products

While the above representation is elegant, it is inefficient, and perhaps a little unintuitive. In most implementations I have seen, variables are represented simply with a type for names, rather than the kind of implicit de Bruijn indices used above. One natural representation uses a list of terms:

newtype Poly v c = Poly { terms :: [([v], c)] }

Here, a value of type Poly v c is a polynomial with coefficients drawn from c and variables from v. It is a list of monomials, where the outer list represents a sum, and each monomial represents a product of variables with a single coefficient.

data Var = X | Y | Z

poly :: Poly Var Integer
poly = Poly [([X,Y,Y],5),([Z],3),([Y,Z],2)]
-- 5xyÂ² + 3z + 2yz

Note that this representation requires some normalisation:

norm :: (Ord v, Num c, Eq c) => [([v],c)] -> [([v],c)]
norm = Map.toList . Map.fromListWith (+) . filter ((/=0).snd)

instance (Num c, Ord v, Eq c) => Eq (Poly v c) where
  (==) = (==) `on` norm . terms

And we have the following Num instance:

instance Num c => Num (Poly v c) where
  fromInteger n = Poly [([],fromInteger n)]
  Poly xs + Poly ys = Poly (xs ++ ys)
  xs * ys = Poly [ (xv ++ yv, xc * yc) | (xv,xc) <- terms xs, (yv,yc) <- terms ys ]
  negate = Poly . map (fmap negate) . terms

This representation perhaps maps more closely to the description of multivariate polynomials that many of us will have encountered in secondary school: itâ€™s straightforward to see how a polynomial like <semantics>2xy+y2âˆ’3<annotation encoding="application/x-tex">2xy + y^2 - 3</annotation></semantics> corresponds to the value Poly [([X,Y],2),([Y,Y],1),([],-3)]. The previous representation (TwoVar) would represent the same expression as the enigmatic [[-3,0,1],[0,2]].

However, there are some wrinkles to this type that are worth noting. First we can see that multiplication is not commutative (even after normalisation).

x = Poly [([X],1)]
y = Poly [([Y],1)]

x * y == Poly [([X,Y],1)]
y * x == Poly [([Y,X],1)]
x * y /= y * x

This is in contrast to TwoVar, where both <semantics>xy<annotation encoding="application/x-tex">xy</annotation></semantics> and <semantics>yx<annotation encoding="application/x-tex">yx</annotation></semantics> would be represented as [[0,0],[0,1]].

Conceptually, polynomials are a kind of free structure: they represent the normalised and quotiented syntax of an algebraic theory. The fact that Poly above doesnâ€™t have commutative multiplication just tells us that the underlying algebraic theory in question here is noncommutative rings, rather than commutative rings.

The second thing to note about this type is actually two related observations about inefficiency. Because I didnâ€™t implement normalisation on any of the numeric operations, we might expect the size of the underlying list of Poly to blow up:

poly = (1 + x) * (3 + 4) * (y + 2)

>>> terms poly
[([Y],3),([],6),([Y],4),([],8),([X,Y],3),([X],6),([X,Y],4),([X],8)]

>>> norm (terms poly)
[([],14),([X],14),([X,Y],7),([Y],7)]

And indeed it does, as you can see above. To counteract this, we can represent our polynomial as a mapping from monics (strings of variables) to coefficients:

newtype Poly v c = Poly { terms :: Map [v] c }

Num instance for Map-based polynomial

instance (Ord v, Num c) => Num (Poly v c) where
  fromInteger n = Poly (Map.singleton [] (fromInteger n))
  Poly xs + Poly ys = Poly (Map.unionWith (+) xs ys)
  xs * ys = Poly (Map.fromListWith (+) [ (xv ++ yv, xc * yc)
                                       | (xv,xc) <- Map.toList (terms xs)
                                       , (yv,yc) <- Map.toList (terms ys) ])
  negate = Poly . fmap negate . terms

While this new representation is an improvement over the un-normalised list, itâ€™s still not really â€œefficientâ€�. In particular, weâ€™re using lists as keys in the map; Haskellâ€™s Map is a binary search tree (though this caveat applies to most mapping structures), so search is always going to have to perform comparisons on the keys. When those keys are lists, that comparison takes time proportional to the length of each list. This is wasted effort that could be cached with a cleverer data structure.

This also brings the second observation about inefficiency into focus: we have lost our neat evaluation with Hornerâ€™s rule.

eval :: Num c => Poly v c -> (v -> c) -> c
eval (Poly mp) v = Map.foldrWithKey (\vs c s -> foldr ((*) . v) c vs + s) 0 mp

Weâ€™re back to performing <semantics>n<annotation encoding="application/x-tex">n</annotation></semantics> multiplications per term.

Both of these inefficiencies are actually the same pattern, and can be solved with a general form of Hornerâ€™s rule. We need to cache prefixes: the data structure that does that best is a trie.

A Trie

Hornerâ€™s rule saved us from performing redundant multiplications by factoring out common terms to the left. That was simple to implement in the single-variable case, but it can still apply for multiple variables. Take an expression like <semantics>(2+3xâˆ’5y)2<annotation encoding="application/x-tex">(2 + 3x - 5y) ^ 2</annotation></semantics>, and multiply it out to <semantics>4+12x+9x2âˆ’15xyâˆ’20yâˆ’15yx+25y2<annotation encoding="application/x-tex">4 + 12x + 9x^2 - 15xy - 20y - 15yx + 25y^2</annotation></semantics>. We can still factor this expression to remove common prefixes, like so:

The difference between this factorisation and the list-based polynomial we started with is that the tree representing the polynomial only had one child. Here, we have a child for each leading term. In terms of the data structure, where a list has a single tail in the cons case,

data List a = Nil
            | Cons a (List a)

The multivariate version of the same thing will be a tree

data Tree a = Nil
            | Cons a [Tree a]

Or, more specifically, a trie, where the subtree mapping is based on variables.

data Poly v c = c :<+ Map v (Poly v c)

A polynomial is a constant coefficient c plus the sum of variables drawn from v each multiplied by another polynomial. The polynomial above is represented with this type as the following:

4 :<+ {(X,12 :<+ {(X,9 :<+ {}),(Y,(-15) :<+ {})}),(Y,(-20) :<+ {(X,(-15) :<+ {}),(Y,25 :<+ {})})}

This trie type (with some improvements Iâ€™ll describe below) is the focus of this post; I think itâ€™s a cool data structure for representing polynomials.

The numeric functions on Tries

Letâ€™s first write evaluation:

eval :: Num c => (v -> c) -> Poly v c -> c
eval f (c :<+ vs) = c + Map.foldrWithKey (\v p s -> f v * eval f p + s) 0 vs

Notice that we have retrieved Hornerâ€™s rule: the evaluation of each term only performs a single multiplication; we donâ€™t have to repeat multiplications for terms that share prefixes any more.

(for those concerned with performance, it might be worth swapping out foldrWithKey with a strict variant. (also, this is somewhat unrelated but a bit of a pet peeve of mine: this is not a place where foldl' is the best option! foldl' is not a panacea!))

The numeric operations on this data structure can be implemented as follows:

deriving instance Functor (Poly v)

instance (Ord v, Num c, Eq c) => Num (Poly v c) where
  fromInteger n = fromInteger n :<+ Map.empty
  (n :<+ ns) + (m :<+ ms) = (n + m) :<+ Map.unionWith (+) ns ms
  (n :<+ ns) * ms = fmap (n*) ms + (0 :<+ fmap (*ms) ns)
  negate = fmap negate

Itâ€™s worth taking a moment to note how efficient these operations are (for a pointer-ridden high-level language like Haskell, that is). We donâ€™t have to compare any strings; we can use Data.Mapâ€™s efficient unionWith on single variables; and multiplication doesnâ€™t have to expand out any Cartesian product.

I will note that we do have to perform a little bit of normalisation for the derived Eq instance to be correct: we have to remove terms that multiply to zeros. Pruning dead branches like this is a pretty standard procedure on tries; in polynomial terms, that just means we have to get rid of entries in the map that evaluate to zero (so <semantics>x(2+y)+y(0)<annotation encoding="application/x-tex">x(2 + y) + y(0)</annotation></semantics> should be pruned to <semantics>x(2+y)<annotation encoding="application/x-tex">x(2 + y)</annotation></semantics>). This can be done without really changing the efficiency of the operations above, but it does make them slightly more verbose.

Normalising Num instance

For this version, we will rely on the extremely efficient custom merge operations in containers.

0 <+? ns | Map.null ns = Nothing
n <+? ns = Just (n :<+ ns)

instance (Ord v, Num c, Eq c) => Num (Poly v c) where
  fromInteger n = fromInteger n :<+ Map.empty

  a + b = fromMaybe 0 (add a b)
    where
      add (n :<+ ns) (m :<+ ms) =
        (n + m) <+?
          Map.merge
            Map.preserveMissing
            Map.preserveMissing
            (Map.zipWithMaybeMatched (const add))
            ns ms

  _ * (0 :<+ ms) | Map.null ms = 0 :<+ Map.empty
  (0 :<+ ns) * ms = 0 :<+ fmap (*ms) ns
  (n :<+ ns) * ms = fmap (n*) ms + (0 :<+ fmap (*ms) ns)


  negate = fmap negate
  abs = fmap abs
  signum (n :<+ _) = signum n :<+ Map.empty

Anyways, when we have all of the above instances, we can manipulate polynomials using the API you might expect, and the normalisation behaviour happens automatically.

data Var = X | Y deriving (Eq, Ord, Show)

var :: Num c => v -> Poly v c
var v = 0 :<+ Map.singleton v (1 :<+ Map.empty)

x,y :: Poly Var Integer
x = var X
y = var Y

poly = (2 + 3 * x - 5 * y) ^ 2
>>> poly
4 + Y*(-20 + Y*25 + X*(-15)) + X*(12 + Y*(-15) + X*9)

Lenses and Division

Lenses in Haskell are very cool, and personally I think one of the best demonstrations of their power is tries. A few years ago, when I was still on Twitter, I posted an implementation of a trie that fit in a tweet (gist link).

Tweet Trie

{-# LANGUAGE RankNTypes #-}

import Control.Comonad.Cofree
import Control.Lens hiding ((:<))
import qualified Data.Map as Map
import Data.Map (Map)
import Prelude hiding (lookup)
import Data.Maybe (isJust)
import Test.QuickCheck

type Trie a b = Cofree (Map a) (Maybe b)

string :: Ord a => [a] -> Lens' (Trie a b) (Maybe b)
string =
 foldr
   (\x r -> _unwrap . at x . anon (Nothing :< mempty)
                                  (\(v :< m) -> null v && null m) . r)
   _extract


insert :: Ord a => [a] -> b -> Trie a b -> Trie a b
insert xs x = string xs .~ Just x

lookup :: Ord a => [a] -> Trie a b -> Maybe b
lookup = view . string

delete :: Ord a => [a] -> Trie a b -> Trie a b
delete xs = string xs .~ Nothing

Lenses are what allowed this very terse implementation. The original purpose of lenses was to facilitate deep access in nested records and data structures: a trie is effectively a nested map, so itâ€™s no great surprise that lenses are a good fit.

It turns out that lenses are also useful for manipulating polynomial tries. At first, it might be difficult to see why: in the trie implementation above, a lens was used to build getters and setters for a mapping from strings to payloads. But what does that translate to in the context of a polynomial? What does it mean to â€œlook upâ€� a string of variables in some expression like <semantics>2x2+y<annotation encoding="application/x-tex">2x^2 + y</annotation></semantics>?

It turns out that lookups corresponds to division. For example, dividing the polynomial <semantics>2x2+y<annotation encoding="application/x-tex">2x^2 + y</annotation></semantics> by the monic <semantics>xx<annotation encoding="application/x-tex">xx</annotation></semantics> gives us a quotient <semantics>2<annotation encoding="application/x-tex">2</annotation></semantics> and remainder <semantics>y<annotation encoding="application/x-tex">y</annotation></semantics>.

>>> divMod (2 * x ^ 2 + y) [X,X]
(2, y)

This is already quite similar to a lens: before the van Laarhoven encoding, lenses were usually thought of as functions that took a data structure and returned a pair of the â€œfocusâ€� of the lens and the â€œrestâ€� of the structure. In polynomial terms, that â€œfocusâ€� is the quotient, and the â€œrestâ€� is the remainder.

But thatâ€™s a little vague. Letâ€™s construct the actual lenses here, in the van Laarhoven style:

constant :: Lens' (Poly v c) c
constant f (c :<+ vs) = fmap (:<+ vs) (f c)

vars :: Lens (Poly v c) (Poly v' c) (Map v (Poly v c)) (Map v' (Poly v' c))
vars f (c :<+ vs) = fmap (c :<+) (f vs)

isZero :: (Num c, Eq c) => Poly v c -> Bool
isZero (n :<+ ns) = (0 == n) && Map.null ns

factored :: (Ord v, Num c, Eq c) => [v] -> Lens' (Poly v c) (Poly v c)
factored = foldr (\v vs -> vars . at v . anon 0 isZero . vs) id

This last lens does indeed give us an interface that looks like division:

>>> view (factored [X,X]) (2*x^2 + y)
2

>>> set (factored [X,X]) 0 (2*x^2 + y)
Y

If we want to define an actual division function, we can define it in terms of factored, in a fun example of the kind of golfy code that lens enables.

divMod :: (Ord v, Num c, Eq c) => Poly v c -> [v] -> (Poly v c,Poly v c)
divMod p vs = factored vs (,0) p

>>> (2*x^2 + y) `divMod` [X,X]
(2,Y)

GrÃ¶bner Bases

While the interface above lets us do some basic computer algebra, to do any serious work with polynomials we will have to at some point compute GrÃ¶bner bases. A GrÃ¶bner basis isâ€¦ somewhat hard to define, actually. Iâ€™ll quote an explainer on the topic by Sturmfels (2005):

A GrÃ¶bner basis is a set of multivariate polynomials that has desirable algorithmic properties

Basically, in several algorithms over polynomials (division, Gaussian elimination, etc.) it becomes necessary at some point to compute this thing called a GrÃ¶bner Basis.

There is a lot of published literature on computing GrÃ¶bner bases in different settings. However, the trie polynomial I have built above is fundamentally noncommutative, and the literature on computing GrÃ¶bner bases for noncommutative rings is comparatively smaller. I have been following Xiuâ€™s thesis (2012) for this project. It outlines a noncommutative version of Buchbergerâ€™s algorithm, and a few optimisations that I was able to implement.

One slightly annoying aspect of these algorithms is that they tend to use monomials as a primitive. In other words, instead of working with the polynomial directly, the algorithms tend to describe operations with the assumption that your representation is basically a list of monomials. In particular, the algorithms will frequently extract the â€œleadingâ€� monomial, and it becomes important for performance that the polynomial representation can provide that leading monomial quickly. Unfortunately, extraction of the leading monomial is slightly awkward on the trie representation (or certainly less natural than the implementation on a listed representation); so we will need to do some work to implement it.

Monomial Orderings

The first important concept to implement for GrÃ¶bner bases is an admissible monomial ordering. This is a total order on strings of variables that is â€œadmissibleâ€�; meaning that it respects concatenation on both sides, and it also is a well-ordering, meaning that any strictly descending chain is finite.

These constraints rule out the usual lexicographic ordering on strings. Instead, weâ€™ll go with graded lexicographic. This means we first compare strings for length, and only in the case where theyâ€™re equal do we move to the normal lexicographic comparison.

grlex :: Ord a => [a] -> [a] -> Ordering
grlex xs ys
  | length xs < length ys = LT
  | length xs > length ys = GT
  | otherwise = compare xs ys

We can improve the efficiency of the above function somewhat by using one of my favourite monoids: the monoid instance on Ordering.

grlex :: Ord a => [a] -> [a] -> Ordering
grlex = go EQ
  where
    go !a []     []     = a
    go !a []     (_:_)  = LT
    go !a (_:_)  []     = GT
    go !a (x:xs) (y:ys) = go (a <> compare x y) xs ys

This version performs just one pass through each list, and does the correct comparison without additionally calculating the length. Itâ€™s also nonstrict: if one of the lists passed is infinite, this comparison will still terminate.

Another admissible order we could use is reverse grlex, which basically amounts to reversing the lists before the comparison. The trie structure means that weâ€™re basically forced to use grlex, but I will include an implementation of grevlex here because I think itâ€™s cute.

Implementations of grevlex

grevlex :: Ord a => [a] -> [a] -> Ordering
grevlex []     []     = EQ
grevlex (_:_)  []     = GT
grevlex []     (_:_)  = LT
grevlex (x:xs) (y:ys) = grevlex xs ys <> compare x y

-- This version is tail-recursive, but it also might unnecessarily compare
-- elements. However, that should be cheaper than building up the list of
-- comparisons.
grevlex :: Ord a => [a] -> [a] -> Ordering
grevlex = go EQ
  where
    go !a []     []     = a
    go !a (_:_)  []     = GT
    go !a []     (_:_)  = LT
    go !a (x:xs) (y:ys) = go (compare x y <> a) xs ys

Enumerating Monomials

The problem with all the admissible monomial orderings is that they need to see the entire monomial before they can decide whether itâ€™s ordered before or after another. This is at odds with the trie, which tends to prefer computations that can be described in terms of prefix/suffix decompositions.

To demonstrate the problem, letâ€™s take a look at an algorithm that enumerates the monomials of a polynomial in lexicographic order:

monos :: (Num c, Eq c) => Poly v c -> [([v],c)]
monos p = search [] p []
  where
    cons vs 0 ms = ms
    cons vs c ms = (reverse vs,c) : ms

    search sv (n :<+ ns) ms = cons sv n (Map.foldrWithKey (search . (:sv)) ms ns)

>>> monos ((2 + 3*x - 5*y) ^ 2)
[([],4),([X],12),([X,X],9),([X,Y],-15),([Y],-20),([Y,X],-15),([Y,Y],25)]

Notice that the function search emits the monomial (reverse sv, n) straight away (if n /= 0), when it encounters it: for a proper admissible monomial ordering, it would instead want to first emit monomials of higher degree; that is, those monomials in the map ns.

However, we canâ€™t just flip the order of consing in search: notice that even if we reversed the output, we still wouldnâ€™t get an admissible monomial ordering (the singleton list [Y] should be grouped with the other singleton lists). The problem is that monos is performing a depth-first search. What we need is breadth-first.

I happen to be a little obsessed with breadth-first search, so I probably spent too much time on this particular implementation, but I do always get excited when I see a breadth-first traversal pop up in the wild.

For this case, I started with the levels function.

levels :: (Num c, Eq c) => Poly v c -> [[([v],c)]]
levels p = search [] p []
  where
    cons _  0 ms = ms
    cons vs c ms = (reverse vs,c) : ms

    search sv (n :<+ ns) []     = cons sv n [] : Map.foldrWithKey (search . (:sv)) [] ns
    search sv (n :<+ ns) (q:qs) = cons sv n  q : Map.foldrWithKey (search . (:sv)) qs ns

>>> levels ((2 + 3*x - 5*y) ^ 2)
[[([],4)],[([X],12),([Y],-20)],[([X,X],9),([X,Y],-15),([Y,X],-15),([Y,Y],25)]]

I have written about levels before (see also Gibbons 2015; Jones and Gibbons 1993).

I think itâ€™s a good fit here because it lets us build the prefix string for each monomial in a natural way (that prefix string is the sv thatâ€™s passed to search).

However, one flaw of this function is that it produces a list of lists: one inner list for each degree of polynomial. The output that I actually want, however, is the concatenation of the whole thing.

In reality, this isnâ€™t actually a flaw: we can just call concat and move on. I had a feeling, though, that there was probably some annoying circular program that would let us avoid the second traversal to concatenate the inner lists. Inspired by Geraint Jonesâ€™ cyclic breadth-first traversal (1993), I finally arrived at the following solution:

data Knots a
  = Knot
  { tied :: !Bool
  , yank :: [a]
  , ends :: Knots a }

tighten :: Knots a -> Knots a
tighten ~(Knot t y e) = Knot False (if t then y else []) (tighten e)

monos :: (Eq c, Num c) => Poly v c -> [([v],c)]
monos p = y
  where
    Knot _ y e = tie [] p (tighten e)
    cons sv 0 ms = ms
    cons sv c ms = (reverse sv, c) : ms
    tie sv (n :<+ m) (Knot _ ms ps) = Knot True (cons sv n ms) (Map.foldrWithKey (tie . (:sv)) ps m)

>>> monos ((2 + 3 * x - 5 * y) ^ 2)
[([],4),([X],12),([Y],-20),([X,X],9),([X,Y],-15),([Y,X],-15),([Y,Y],25)]

While this does order the output according to grlex, itâ€™s ordered from smallest to largest, which is the reverse of what we want. And yes, while we could just reverse the output, I didnâ€™t write the circular abomination above to throw away the single-pass traversal at such a small hurdle. Any (list-based) algorithm written in a fold-like fashion can usually be reversed by swapping out right-folds for left.

pull :: Knots a -> [a]
pull (Knot True _ e) = pull e
pull (Knot False y _) = y

monosDesc :: (Eq c, Num c) => Poly v c -> [([v],c)]
monosDesc p = pull r
  where
    r = tie [] p (Knot False [] (tighten r))
    cons sv 0 ms = ms
    cons sv c ms = (reverse sv, c) : ms
    tie sv (n :<+ m) (Knot _ ms ps) = Knot True (cons sv n ms) (Map.foldlWithKey (\a v p -> tie (v:sv) p a) ps m)

Efficiently Popping the Leading Monomial

Unfortunately, as fun as monosDesc is, it doesnâ€™t really do what we need it to for most of the GrÃ¶bner basis algorithms. While it is pretty efficient if we want all of the monomials of a polynomial, usually we just want the first one. And sadly, while monosDesc is linear overall, itâ€™s not lazy in the right way, meaning that we have to pay that full linear cost even if we only inspect the first element of the list it produces.

The solution here will require us to use a new data structure in place of the Map that we have currently. To avoid traversing the whole tree to find the largest monomial, we need to cache the depth of each subterm so that we can just descend into the subterm which contains the monomial of the highest degree. But we donâ€™t want to just swap out our Map v (Poly v c) for a Map v (Word, Poly v c): that solution would require us to walk over every entry in the map to find the largest Word. While it would be an improvement in practical terms, it would still incur an <semantics>ğ�’ª(widthÃ—depth)<annotation encoding="application/x-tex">\mathcal{O}(\text{width} \times \text{depth})</annotation></semantics> cost to find the leading monomial.

Instead, we need the map itself to be able to efficiently provide the entry with the largest degree. We need our map to simultaneously act as a priority queue.

Luckily, the combination of these two structures has been researched before: Hinze (2001) wrote about â€œpriority search treesâ€�, a data structure that allows for <semantics>ğ�’ª(log⁡n)<annotation encoding="application/x-tex">\mathcal{O}(\log n)</annotation></semantics> lookup and insertion based on some ordered key, and separately allows for a <semantics>ğ�’ª(log⁡n)<annotation encoding="application/x-tex">\mathcal{O}(\log n)</annotation></semantics> popMin operation, based on some separate priority. The psqueues package provides a few implementations of this technique. The API isnâ€™t quite as extensive as, say, containers, so some functions will be slightly less efficient (we donâ€™t get a nice general merge function, for example), but we can basically drop in the OrdPSQ as a replacement for Map.

type SubTerms v c = OrdPSQ (Down v) (Down Word) (Poly v c)
data Poly v c = c :<+ SubTerms v c

Iâ€™m using the Down wrapper here because I want a max heap, rather than a min-heap. Iâ€™m using that wrapper on both the keys and priorities because OrdPSQ breaks priority ties according to the keys, and I also want greater keys returned first, to follow the grlex ordering.

The priority here is the depth of the tree. It tells us the length of the longest monomial contained:

depth :: Poly v c -> Word
depth (_ :<+ ns) = maybe 0 (\(_,Down p,_) -> succ p) (Map.findMin ns)

This operation is <semantics>ğ�’ª(1)<annotation encoding="application/x-tex">\mathcal{O}(1)</annotation></semantics>, since finding the minimum entry in OrdPSQ is <semantics>ğ�’ª(1)<annotation encoding="application/x-tex">\mathcal{O}(1)</annotation></semantics>.

Iâ€™ll also use the following isomorphism, for the lensy things:

entry :: (Num c, Eq c) => Iso' (Maybe (Down Word, Poly v c)) (Poly v c)
entry = iso (maybe (0 :<+ Map.empty) snd) (\p -> if isZero p then Nothing else Just (Down (depth p), p))

This lets us chain together lenses that index into an OrdPSQ.

factored :: (Ord v, Num c, Eq c) => [v] -> Lens' (Poly v c) (Poly v c)
factored = foldr (\v ls -> vars . at (Down v) . entry . ls) id

Finally, we can implement a function that pops the leading monomial from a polynomial, efficiently:

leading :: (Num c, Eq c, Ord v) => Poly v c -> Maybe (([v],c),Poly v c)
leading p | isZero p = Nothing
leading (n :<+ ns) = Just (retrie (Map.alterMin step ns))
  where
    retrie ((r,n'),ns') = (r, n' :<+ ns')

    step Nothing = ((([],n),0),Nothing)
    step (Just (Down v, _, p)) = (((v:vs,c),n), subTrie)
      where
        Just ((vs,c),p') = leading p
        subTrie | isZero p' = Nothing
                | otherwise = Just (Down v, Down (depth p'), p')

And it matches the earlier enumeration that we built:

prop_leadingMonos :: Poly Var Word -> Property
prop_leadingMonos p = monosDesc p === unfoldr leading p

Next Steps

I think this is an interesting data structure, and representation of polynomials. However, I am not very familiar with the computer algebra literature, so I canâ€™t yet tell how this kind of representation relates to the other systems out there. Furthermore, most of the algorithms I have read seem to work implicitly with â€œleading monomialsâ€� etc., leading to the following kind of implementation of division:

divModPrefM :: (Fractional c, Eq c, Ord v) => Poly v c -> ([v],c) -> (Poly v c, Poly v c)
divModPrefM p (vs, i) = factored vs ((, 0) . fmap (/i)) p

divModPref :: (Fractional c, Eq c, Ord v) => Poly v c -> Poly v c -> (Poly v c, Poly v c)
divModPref num divisor = case leading divisor of
  Nothing -> error "Divide by zero"
  Just (lt, rest) -> go 0 num
    where
      go !quot !rem = case divModPrefM rem lt of
        (0, _) -> (quot, rem)
        (q, rem') -> go (quot + q) (rem' - rest * q)

I feel that this doesnâ€™t make use of the benefits of the trie-based representation. I have implemented Buchbergerâ€™s algorithm (with most of the improvements from Xiu 2012), but I have yet to really research in depth what competitively fast systems do these days (Heisinger and Hofstadler 2025; Cohen and Knopper 2026; Levandovskyy, SchÃ¶nemann, and Zeid 2020). Iâ€™m also interested in seeing what kinds of applications there are for this stuff: I started this project with Weyl algebras in mind, but after looking into it a little more it seems clear that a trie is not a good fit for Weyl algebras.

I have looked a little bit at some other Haskell work on polynomials and similar things; Zucker (2018) implemented listed polynomials very similar to the ones I had at the start of this post, as did Manzyuk (2012) and Buteau (2013). Iâ€™ve seen some bigger Haskell packages that work with polynomials (Malaquias and Lopes 2007; Ishii 2018; Laurent 2024), though none seem to use a representation similar to the trie here. I also had a look at calculi (Barton 2024), but I think that that project mainly works with commutative rings (although itâ€™s pretty big project, so I wouldnâ€™t be surprised if there was some module I missed).

I would actually be interested to hear if anyone has any pointers to work that has a similar approach to polynomials, or on the kinds of things that people use these noncommutative polynomials for. I find most of the descriptions of these algorithms difficult to parse (since theyâ€™re usually written by and for mathematicians rather than computer scientists, and almost never for functional programmers), so I am sure Iâ€™m missing some major projects.

Gists

Listed Polynomial with arbitrary variables

Polynomial Trie

References

Barton, Dave. 2024. â€œCalculi.â€� https://github.com/DaveBarton/calculi.

Buteau, Samuel. 2013. â€œPolynomials - School of Haskell School of Haskell.â€� School of Haskell. https://www.schoolofhaskell.com/user/Sam567/computational-physics/beginner-s-tools/polynomials.

Cohen, Arjeh M., and Jan Willem Knopper. 2026. â€œGBNP.â€� GAP packages. https://github.com/gap-packages/gbnp.

Gibbons, Jeremy. 2011. â€œHornerâ€™s Rule.â€� Patterns in Functional Programming. https://patternsinfp.wordpress.com/2011/05/05/horners-rule/.

â€”â€”â€”. 2015. â€œBreadth-First Traversal.â€� Patterns in Functional Programming. https://patternsinfp.wordpress.com/2015/03/05/breadth-first-traversal/.

Heisinger, Maximilian, and Clemens Hofstadler. 2025. â€œF4ncgb: High Performance GrÃ¶bner Basis Computations in Free Algebras.â€� arXiv. doi:10.48550/arXiv.2505.19304. https://arxiv.org/abs/2505.19304.

Hinze, Ralf. 2001. â€œA simple implementation technique for priority search queues.â€� SIGPLAN Not. 36 (10) (October): 110â€“121. doi:10.1145/507669.507650.

Ishii, Hiromi. 2018. â€œA Purely Functional Computer Algebra System Embedded in Haskell.â€� In, 11088:288â€“303. doi:10.1007/978-3-319-99639-4_20. https://arxiv.org/abs/1807.01456.

Jones, Geraint, and Jeremy Gibbons. 1993. Linear-time breadth-first tree algorithms: An exercise in the arithmetic of folds and zips. Dept of Computer Science, University of Auckland. http://www.cs.ox.ac.uk/people/jeremy.gibbons/publications/linear.ps.gz.

Laurent, StÃ©phane. 2024. â€œHspray.â€� https://github.com/stla/hspray.

Levandovskyy, Viktor, Hans SchÃ¶nemann, and Karim Abou Zeid. 2020. â€œLetterplace: A subsystem of singular for computations with free algebras via letterplace embedding.â€� In Proceedings of the 45th International Symposium on Symbolic and Algebraic Computation, 305â€“311. ISSAC â€™20. New York, NY, USA: Association for Computing Machinery. doi:10.1145/3373207.3404056.

Malaquias, JosÃ© Romildo, and Carlos Roberto Lopes. 2007. â€œImplementing a computer algebra system in Haskell.â€� Applied Mathematics and Computation 192 (1) (September): 120â€“134. doi:10.1016/j.amc.2007.02.126.

Manzyuk, Oleksandr. 2012. â€œGrÃ¶bner bases in Haskell: Part I.â€� Oleksandr Manzyukâ€™s Blog. https://web.archive.org/web/20221206080655/https://oleksandrmanzyuk.wordpress.com/2012/10/25/grobner-bases-in-haskell-part-i/.

McIlroy, M. Douglas. 1999. â€œPower Series, Power Serious.â€� J. Funct. Program. 9 (3) (May): 325â€“337. doi:10.1017/S0956796899003299.

Sturmfels, Bernd. 2005. â€œWhat is... A GrÃ¶bner Basis?â€� Notices of the AMS 52 (10) (November). https://www.ams.org/journals/notices/200510/what-is.pdf.

Xiu, Xingqiang. 2012. â€œNon-commutative GrÃ¶bner Bases and Applications.â€� PhD thesis, UniversitÃ¤t Passau. https://opus4.kobv.de/opus4-uni-passau/frontdoor/index/index/docId/170.

Zucker, Philip. 2018. â€œDivision of polynomials in haskell.â€� Hey There Buddo! https://www.philipzucker.com/division-of-polynomials-in-haskell/.

Dan Piponi (sigfpe): Some type constructors are tensor products

noreply@blogger.com (sigfpe) — Mon, 27 Apr 2026 19:15:36 +0000

Introduction

I want to return to something I've mentioned a couple of times in the past - the fact that applying certain type constructors performs a tensor product.

First some admin stuff:

> {-# LANGUAGE DeriveFunctor #-}
> {-# LANGUAGE FlexibleInstances #-}
> {-# LANGUAGE MultiParamTypeClasses #-}
> {-# LANGUAGE UndecidableInstances #-}
> {-# LANGUAGE TypeApplications #-}
> {-# LANGUAGE KindSignatures #-}
> {-# LANGUAGE ScopedTypeVariables #-}
> {-# LANGUAGE AllowAmbiguousTypes #-}

> import Data.Proxy
> import Data.Kind (Type)

> infixr 7 ⊗

Suppose you define a type like so:

> data Complex a = C a a
>     deriving (Eq, Show, Functor)

> instance Num a => Num (Complex a) where
>     fromInteger n = C (fromInteger n) 0

>     C a b + C c d = C (a + c) (b + d)
>     C a b - C c d = C (a - c) (b - d)

>     C a b * C c d = C (a * c - b * d) (a * d + b * c)

>     negate (C a b) = C (negate a) (negate b)

>     abs    = error "abs doesn't make sense here"
>     signum = error "signum makes no sense here"

It seems straightforward. You've defined complex numbers in a way that allows a choice of base type to represent the real numbers. For example you could use Complex Float or Complex Double as representations of $\mathbb{C}$.

In actual fact you've done quite a bit more! That code has another reading - it implements a tensor product both in the category of vector spaces, and, less trivially, in the category of algebras. So if A is a suitable algebraic structure then, if you allow me to mix code and mathematics notation,

\[ \mathtt{Complex\ A} = \mathbb{C}\otimes\mathtt{A} \]

I took this for granted when I mentioned it previously but I thought I'd look into it in a little bit more detail.

Tensor Products

I want to start from the definition of the tensor product given by its universal property, but to make that slightly less fearsome I'll use an English sketch of it.

Suppose you have a pair of vector spaces $X$ and $Y$ over some base field $k$. A bilinear function $X\times Y\rightarrow Z$ is a function that is linear in $X$ and linear in $Y$. Now suppose we know that at some point in the future we are going to need some bilinear function on $X\times Y$ but don't yet know what it is. Can we make a structure, $T$, that contains precisely the information we need so that we can compute any bilinear function we want - with the proviso that we compute these bilinear functions by applying a linear function to $T$? We don't want $T$ to be lacking anything we might need to compute a future bilinear product, but we also don't want it to contain any extraneous data.

For example, imagine working with $V$, the vector space of 3D vectors. Some examples of bilinear functions we might want are the dot product $V\cdot V\rightarrow\mathbb{R}$ and the cross product $V\times V\rightarrow V$. What should $T$ look like?

We can write the dot product as $(x, y, z)\cdot(x', y', z') = xx'+yy'+zz'$. Note how it's made of products of coordinates from $(x, y, z)$ and coordinates from $(x', y', z')$. Similarly $(x, y, z)\times(x', y', z')=(yz'-zy',\ldots)$. Again, it's a linear combination of products of coordinates, one from each vector. You can prove that any bilinear product will be some linear combination of such products.

By thinking about all possible bilinear products you I hope you can see that $T$ should be a 9-dimensional vector space and a suitable way to represent a pair of vectors $(x, y, z), (x', y', z')$ for future application of a bilinear function is as $(xx', xy', xz', yx', yy', yz', zx', zy', zz')$. Any bilinear product is a linear combination of these 9 quantities and so is given by some linear operation on $T$. It is commonplace to arrange the 9-dimensional vector as a $3\times 3$ matrix in which case the map from the pair is called the outer product. But it doesn't really matter as all 9-dimensional vector spaces over a given field are isomorphic.

In this case I chose to consider bilinear functions on $V\times V$, but you can reason similarly for any pair of vector spaces $X$ and $Y$. When working with finite-dimensional vector spaces, the structure we need will be $mn$-dimensional where $m$ is the dimension of $X$ and $n$ is the dimension of $Y$. The structure is called the tensor product and is written as $X\otimes Y$. The bilinear map from the original vectors into the tensor product is also called the tensor product and as written as a binary operator $x\otimes y$. And once you have the tensor product, every bilinear function on the original pair of spaces can be expressed uniquely as a linear function on the tensor product.

So, for example, the dot product can be written as

\[ x\cdot y = \phi(x\otimes y) \]

where $(x, y, z)\otimes(x', y', z')=(xx',xy',\ldots zz')$ and so the linear function is $\phi(x_0, x_1,\ldots,x_8) = x_0+x_4+x_8$.

Similarly

\[ x\times y = \psi(x\otimes y) \]

where $\psi(x_0, x_1, \ldots, x_8)=(x_5 - x_7, x_6 - x_2, x_1 - x_3)$

Algebras

It's a confusing use of terminology, but the term "algebra (over $k$)" is used specifically to mean a vector space $A$ (over $k$) equipped with a bilinear product $A\times A\rightarrow A$ which is compatible with the vector space structure. And in addition I'm assuming my algebras contain a multiplicative unit element. Other people may call this a "unital algebra". I'll use the word "unital" when I want to stress that there is a unit.

An example is the algebra of complex numbers $\mathbb{C}$ over $\mathbb{R}$. It's a 2-dimensional vector space over $\mathbb{R}$. We can, for example, scale complex numbers by elements of the base field. We also have properties like $(au)v = u(av)$ for $a\in\mathbb{R}$ and $u,v\in\mathbb{C}$. We can scale either argument of the complex product by a real and it makes no difference which we choose. See Wikipedia for all the properties an algebra must satisfy.

Vector spaces come with an addition operation and a zero but we're going to share the work out a little differently because our Num instance already has those. So our VectorSpace class is just going to have the scale operation:

> class VectorSpace k v where
>     scale :: k -> v -> v

> instance VectorSpace Double Double where
>     scale = (*)

You can think of the definition of Complex above as a container for the coordinates in a choice of basis. Because I use deriving Functor I can get the VectorSpace instance for all similar types for free:

> instance (Functor c, VectorSpace k a) => VectorSpace k (c a) where
>     scale k = fmap (scale k)

Because fmap composes through nested functors, scale descends recursively through arbitrarily nested structures like Complex (Complex Double).

And now we can concretely implement the bilinear tensor product operation in our choice of basis. It works by descending through the construction of $x$ until it reaches its individual coordinates and then uses each one to scale $y$. A special case of this is our 9-dimensional vector construction above: each batch of 3 coordinates is s scaling of one vector by a coordinate from the other.

> (⊗) :: (Functor c, VectorSpace k a) => c k -> a -> c a
> x ⊗ y = fmap (`scale` y) x

We're literally just recursively building a table of all products of coordinates of c k and coordinates of a.

Any bilinear function f :: U -> V -> W can now be implemented as f x y = phi (x ⊗ y) for a unique choice of phi.

Algebras too

But there's more, and this is the point of me writing this article. Algebras also have a tensor product defined on them. The underlying carrier space is the tensor product of algebras considered as vector spaces. The product structure is defined by $(x\otimes y)(x'\otimes y')=(xx')\otimes(yy')$ and linear combinations thereof. But what's neat here is that we don't have to write any more code to implement this, our Num instance is already doing the work.

We need to check that our definition of Complex satisfies this property. In fact, I want to prove it more generally for any type like Complex that has a multiplication that looks like

    C a b * C c d = C (a * c - b * d) (a * d + b * c)

ie. I'll assume we have a type F that is an instance of Num, with constructor F, and whose multiplication is constructed from a linear combination of terms of the form a * a'.

Something like:

    (F ... a ...) * (F ... a' ...) = F ... (... + a * a' + ...) ...

Note that I'm claiming

\[ \mathtt{F\ A} = \mathtt{F\ Double}\otimes\mathtt{A} \]

so I can suppose that a is in Double (or whatever we use to represent the reals).

Assuming * is such a product:

   (x ⊗ y) * (x' ⊗ y')
== fmap (`scale` y) x * fmap (`scale` y') x'
   -- definition of tensor
== fmap (`scale` y) (F ... a ...) * fmap (`scale` y') (F ... a' ...)
   -- stating our assumptions about the form of x and x'
== (F ... (scale a y) ...) * (F ... (scale a' y') ...)
   -- this is what derived fmap looks like
== F ... (... + scale a y * scale a' y' + ...) ...
   -- our assumption about the form that multiplication takes
== F ... (... + scale (a * a') (y * y') + ...) ...
   -- multiplication is bilinear all the way down
== fmap (`scale` (y * y')) (F ... (... + a * a' + ...))
   -- same fact about fmap used above
== fmap (`scale` (y * y')) (x * x')
   -- again our assumption about how multiplication is implemented
== (x * x') ⊗ (y * y')
   -- definition of tensor again

Anyway, my motivation here is that quite a while back someone (on Mastodon) I think pushed back on my claim that we have a tensor product so I thought I'd give some more detail.

I could say more. The tensor product of algebras has the nice property that you can embed the original algebras in it in a way that the two images commute with each other. In fact, if you can define the tensor product to be the initial algebra with this property. But this is too long already.

Also, I used Haskell above but it carries over straightforwardly to other languages, even C++.

> main :: IO ()
> main = do
>   print "Bye!"

Haskell Interlude: 81: Torsten Grust

Mon, 27 Apr 2026 07:00:00 +0000

Mike and Andres sat down with Torsten Grust, who is a professor of DB systems at the University of Tübingen. Even though Torsten loves SQL, he's used functional programming and Haskell to inform his work on query language design and compilation. We talked about the best way to program databases, how to bridge the gap between regular programming languages and databases, and compiling just about everything to SQL.

Chris Reade: PenroseKiteDart User Guide

Sun, 26 Apr 2026 16:11:03 +0000

Introduction

(Updated April 2026 for PenroseKiteDart version 1.8)

PenroseKiteDart is a Haskell package with tools to experiment with finite tilings of Penrose’s Kites and Darts. It uses the Haskell Diagrams package for drawing tilings. As well as providing drawing tools, this package introduces tile graphs (Tgraphs) for describing finite tilings. (I would like to thank Stephen Huggett for suggesting planar graphs as a way to reperesent the tilings).

This document summarises the design and use of the PenroseKiteDart package.

PenroseKiteDart package is now available on Hackage.

The source files are available on GitHub at https://github.com/chrisreade/PenroseKiteDart.

There is a small art gallery of examples created with PenroseKiteDart here.

Index

1. About Penroseâ€™s Kites and Darts

The Tiles

In figure 1 we show a dart and a kite. All angles are multiples of (a tenth of a full turn). If the shorter edges are of length 1, then the longer edges are of length , where is the golden ratio.

Figure 1: The Dart and Kite Tiles

Aperiodic Infinite Tilings

What is interesting about these tiles is:

It is possible to tile the entire plane with kites and darts in an aperiodic way.

Such a tiling is non-periodic and does not contain arbitrarily large periodic regions or patches.

The possibility of aperiodic tilings with kites and darts was discovered by Sir Roger Penrose in 1974. There are other shapes with this property, including a chiral aperiodic monotile discovered in 2023 by Smith, Myers, Kaplan, Goodman-Strauss. (See the Penrose Tiling Wikipedia page for the history of aperiodic tilings)

This package is entirely concerned with Penrose’s kite and dart tilings also known as P2 tilings.

Legal Tilings

In figure 2 we add a temporary green line marking purely to illustrate a rule for making legal tilings. The purpose of the rule is to exclude the possibility of periodic tilings.

If all tiles are marked as shown, then whenever tiles come together at a point, they must all be marked or must all be unmarked at that meeting point. So, for example, each long edge of a kite can be placed legally on only one of the two long edges of a dart. The kite wing vertex (which is marked) has to go next to the dart tip vertex (which is marked) and cannot go next to the dart wing vertex (which is unmarked) for a legal tiling.

Figure 2: Marked Dart and Kite

Correct Tilings

Unfortunately, having a finite legal tiling is not enough to guarantee you can continue the tiling without getting stuck. Finite legal tilings which can be continued to cover the entire plane are called correct and the others (which are doomed to get stuck) are called incorrect. This means that decomposition and forcing (described later) become important tools for constructing correct finite tilings.

2. Using the PenroseKiteDart Package

You will need the Haskell Diagrams package (See Haskell Diagrams) as well as this package (PenroseKiteDart). When these are installed, you can produce diagrams with a Main.hs module. This should import a chosen backend for diagrams such as the default (SVG) along with Diagrams.Prelude.

    module Main (main) where
    
    import Diagrams.Backend.SVG.CmdLine
    import Diagrams.Prelude

For Penrose’s Kite and Dart tilings, you also need to import the PKD module and (optionally) the TgraphExamples module.

    import PKD
    import TgraphExamples

Then to ouput someExample figure

    fig::Diagram B
    fig = someExample

    main :: IO ()
    main = mainWith fig

Note that the token B is used in the diagrams package to represent the chosen backend for output. So a diagram has type Diagram B. In this case B is bound to SVG by the import of the SVG backend. When the compiled module is executed it will generate an SVG file. (See Haskell Diagrams for more details on producing diagrams and using alternative backends).

3. Overview of Types and Operations

Half-Tiles

In order to implement operations on tilings (decompose in particular), we work with half-tiles. These are illustrated in figure 3 and labelled RD (right dart), LD (left dart), LK (left kite), RK (right kite). The join edges where left and right halves come together are shown with dotted lines, leaving one short edge and one long edge on each half-tile (excluding the join edge). We have shown a red dot at the vertex we regard as the origin of each half-tile (the tip of a half-dart and the base of a half-kite).

Figure 3: Half-Tile pieces showing join edges (dashed) and origin vertices (red dots)

The labels are actually data constructors introduced with type operator HalfTile which has an argument type (rep) to allow for more than one representation of the half-tiles.

    data HalfTile rep 
      = LD rep -- Left Dart
      | RD rep -- Right Dart
      | LK rep -- Left Kite
      | RK rep -- Right Kite
      deriving (Show,Eq)

Tgraphs

We introduce tile graphs (Tgraphs) which provide a simple planar graph representation for finite patches of tiles. For Tgraphs we first specialise HalfTile with a triple of vertices (positive integers) to make a TileFace such as RD(1,2,3), where the vertices go clockwise round the half-tile triangle starting with the origin.

    type TileFace  = HalfTile (Vertex,Vertex,Vertex)
    type Vertex    = Int  -- must be positive

The function

    makeTgraph :: [TileFace] -> Tgraph

then constructs a Tgraph from a TileFace list after checking the TileFaces satisfy certain properties (described below). We also have

    faces :: Tgraph -> [TileFace]

to retrieve the TileFace list from a Tgraph.

As an example, the fool (short for fool’s kite and also called an ace in the literature) consists of two kites and a dart (= 4 half-kites and 2 half-darts):

    fool :: Tgraph
    fool = makeTgraph [RD (1,2,3), LD (1,3,4)   -- right and left dart
                      ,LK (5,3,2), RK (5,2,7)   -- left and right kite
                      ,RK (5,4,3), LK (5,6,4)   -- right and left kite
                      ]

To produce a diagram, we simply draw the Tgraph

    foolFigure :: Diagram B
    foolFigure = draw fool

which will produce the diagram on the left in figure 4.

Alternatively,

    foolFigure :: Diagram B
    foolFigure = labelled drawj fool

will produce the diagram on the right in figure 4 (showing vertex labels and dashed join edges).

Figure 4: Diagram of fool without labels and join edges (left), and with (right)

When any (non-empty) Tgraph is drawn, a default orientation and scale are chosen based on the lowest numbered join edge. This is aligned on the positive x-axis with length 1 (for darts) or length (for kites).

Tgraph Properties

Tgraphs are actually implemented as

    newtype Tgraph = Tgraph [TileFace]
                     deriving (Show)

but the data constructor Tgraph is not exported to avoid accidentally by-passing checks for the required properties. The properties checked by makeTgraph ensure the Tgraph represents a legal tiling as a planar graph with positive vertex numbers, and that the collection of half-tile faces are both connected and have no crossing boundaries (see note below). Finally, there is a check to ensure two or more distinct vertex numbers are not used to represent the same vertex of the graph (a touching vertex check). An error is raised if there is a problem.

Note: If the TileFaces are faces of a planar graph there will also be exterior (untiled) regions, and in graph theory these would also be called faces of the graph. To avoid confusion, we will refer to these only as exterior regions, and unless otherwise stated, face will mean a TileFace. We can then define the boundary of a list of TileFaces as the edges of the exterior regions. There is a crossing boundary if the boundary crosses itself at a vertex. We exclude crossing boundaries from Tgraphs because they prevent us from calculating relative positions of tiles locally and create touching vertex problems.

For convenience, in addition to makeTgraph, we also have

    makeUncheckedTgraph :: [TileFace] -> Tgraph
    checkedTgraph   :: [TileFace] -> Tgraph

The first of these (performing no checks) is useful when you know the required properties hold. The second performs the same checks as makeTgraph except that it omits the touching vertex check. This could be used, for example, when making a Tgraph from a sub-collection of TileFaces of another Tgraph.

Main Tiling Operations

There are three key operations on finite tilings, namely

    decompose :: Tgraph -> Tgraph
    force     :: Tgraph -> Tgraph
    compose   :: Tgraph -> Tgraph

Decompose

Decomposition (also called deflation) works by splitting each half-tile into either 2 or 3 new (smaller scale) half-tiles, to produce a new tiling. The fact that this is possible, is used to establish the existence of infinite aperiodic tilings with kites and darts. Since our Tgraphs have abstracted away from scale, the result of decomposing a Tgraph is just another Tgraph. However if we wish to compare before and after with a drawing, the latter should be scaled by a factor times the scale of the former, to reflect the change in scale.

Figure 5: fool (left) and decompose fool (right)

We can, of course, iterate decompose to produce an infinite list of finer and finer decompositions of a Tgraph

    decompositions :: Tgraph -> [Tgraph]
    decompositions = iterate decompose

Force

Force works by adding any TileFaces on the boundary edges of a Tgraph which are forced. That is, where there is only one legal choice of TileFace addition consistent with the seven possible vertex types. Such additions are continued until either (i) there are no more forced cases, in which case a final (forced) Tgraph is returned, or (ii) the process finds the tiling is stuck, in which case an error is raised indicating an incorrect tiling. [In the latter case, the argument to force must have been an incorrect tiling, because the forced additions cannot produce an incorrect tiling starting from a correct tiling.]

An example is shown in figure 6. When forced, the Tgraph on the left produces the result on the right. The original is highlighted in red in the result to show what has been added.

Figure 6: A Tgraph (left) and its forced result (right) with the original shown red

Compose

Composition (also called inflation) is an opposite to decompose but this has complications for finite tilings, so it is not simply an inverse. (See Graphs,Kites and Darts and Theorems for more discussion of the problems). Figure 7 shows a Tgraph (left) with the result of composing (right) where we have also shown (in pale green) the faces of the original that are not included in the composition – the remainder faces.

Figure 7: A Tgraph (left) and its (part) composed result (right) with the remainder faces shown pale green

Under some circumstances composing can fail to produce a Tgraph because there are crossing boundaries in the resulting TileFaces. However, we have established that

If g is a forced Tgraph, then compose g is defined and it is also a forced Tgraph.

Try Results

It is convenient to use types of the form Try a for results where we know there can be a failure. For example, compose can fail if the result does not pass the connected and no crossing boundary check, and force can fail if its argument is an incorrect Tgraph. In situations when you would like to continue some computation rather than raise an error when there is a failure, use a try version of a function.

    tryCompose :: Tgraph -> Try Tgraph
    tryForce   :: Tgraph -> Try Tgraph

We define Try as a synonym for Either ShowS (which is a monad) in module Tgraph.Try.

type Try a = Either ShowS a

(Note ShowS is String -> String). Successful results have the form Right r (for some correct result r) and failure results have the form Left (s<>) (where s is a String describing the problem as a failure report).

The function

    runTry:: Try a -> a
    runTry = either error id

will retrieve a correct result but raise an error for failure cases. This means we can always derive an error raising version from a try version of a function by composing with runTry.

    force = runTry . tryForce
    compose = runTry . tryCompose

Elementary Tgraph and TileFace Operations

The module Tgraph.Prelude defines elementary operations on Tgraphs relating vertices, directed edges, and faces. We describe a few of them here.

When we need to refer to particular vertices of a TileFace we use

    originV :: TileFace -> Vertex -- the first vertex - red dot in figure 2
    oppV    :: TileFace -> Vertex -- the vertex at the opposite end of the join edge from the origin
    wingV   :: TileFace -> Vertex -- the vertex not on the join edge

A directed edge is represented as a pair of vertices.

    type Dedge = (Vertex,Vertex)

So (a,b) is regarded as a directed edge from a to b.

When we need to refer to particular edges of a TileFace we use

    joinE  :: TileFace -> Dedge  -- shown dotted in figure 2
    shortE :: TileFace -> Dedge  -- the non-join short edge
    longE  :: TileFace -> Dedge  -- the non-join long edge

which are all directed clockwise round the TileFace. In contrast, joinOfTile is always directed away from the origin vertex, so is not clockwise for right darts or for left kites:

    joinOfTile:: TileFace -> Dedge
    joinOfTile face = (originV face, oppV face)

In the special case that a list of directed edges is symmetrically closed [(b,a) is in the list whenever (a,b) is in the list] we can think of this as an edge list rather than just a directed edge list.

For example,

    internalEdges :: Tgraph -> [Dedge]

produces an edge list, whereas

    boundary :: Tgraph -> [Dedge]

produces single directions. Each directed edge in the resulting boundary will have a TileFace on the left and an exterior region on the right. The function

    dedges :: Tgraph -> [Dedge]

produces all the directed edges obtained by going clockwise round each TileFace so not every edge in the list has an inverse in the list.

Note 1: There is now a class HasFaces (introduced in version 1.4) which includes instances for both Tgraph and [TileFace] and others. This allows some generalisations. For example

    faces         :: HasFaces a => a -> [TileFace]
    internalEdges :: HasFaces a => a -> [Dedge]
    boundary      :: HasFaces a => a -> [Dedge] 
    dedges        :: HasFaces a => a -> [Dedge] 
    nullFaces     :: HasFaces a => a -> Bool

Note 2: There is now a class HasGraph (introduced in version 1.8) which includes instances for Tgraph as well as other types used in forcing. This allows some other generalisations. For example

    compose :: HasGraph a => a -> Tgraph
    decompose :: HasGraph a => a -> Tgraph
    recoverGraph :: HasGraph a => a -> Tgraph

Patches (Scaled and Positioned Tilings)

Behind the scenes, when a Tgraph is drawn, each TileFace is converted to a Piece. A Piece is another specialisation of HalfTile using a two dimensional vector to indicate the length and direction of the join edge of the half-tile (from the originV to the oppV), thus fixing its scale and orientation. The whole Tgraph then becomes a list of located Pieces called a Patch.

    type Piece = HalfTile (V2 Double)
    type Patch = [Located Piece]

Piece drawing functions derive vectors for other edges of a half-tile piece from its join edge vector. In particular (in the TileLib module) we have

    drawPiece :: Piece -> Diagram B
    darawjPiece :: Piece -> Diagram B
    fillPieceDK :: Colour Double -> Colour Double -> Piece -> Diagram B

where the first draws the non-join edges of a Piece, the second does the same but adds a faint dashed line for the join edge, and the third takes two colours – one for darts and one for kites, which are used to fill the piece as well as using drawPiece.

Patch is an instance of class Transformable so a Patch can be scaled, rotated, and translated.

Vertex Patches

It is useful to have an intermediate form between Tgraphs and Patches, that contains information about both the location of vertices (as 2D points), and the abstract TileFaces. This allows us to introduce labelled drawing functions (to show the vertex labels) which we then extend to Tgraphs. We call the intermediate form a VPatch (short for Vertex Patch).

    type VertexLocMap = IntMap.IntMap (Point V2 Double)
    data VPatch = VPatch {vLocs :: VertexLocMap,  vpFaces::[TileFace]} deriving Show

and

    makeVP :: HasGraph a => a -> VPatch

calculates vertex locations using a default orientation and scale.

VPatch is made an instance of class Transformable so a VPatch can also be scaled and rotated.

One essential use of this intermediate form is to be able to draw a Tgraph with labels, rotated but without the labels themselves being rotated. We can simply convert the Tgraph to a VPatch, and rotate that before drawing with labels.

    labelled draw (rotate someAngle (makeVP g))

We can also align a VPatch using vertex labels.

    alignXaxis :: (Vertex, Vertex) -> VPatch -> VPatch

So if g is a Tgraph with vertex labels a and b we can align it on the x-axis with a at the origin and b on the positive x-axis (after converting to a VPatch), instead of accepting the default orientation.

    labelled draw (alignXaxis (a,b) (makeVP g))

Another use of VPatches is to share the vertex location map when drawing only subsets of the faces (see Overlaid examples in the next section).

4. Drawing in More Detail

Class Drawable

There is a class Drawable with instances Tgraph, VPatch, Patch. When the token B is in scope standing for a fixed backend then we can assume

    draw   :: Drawable a => a -> Diagram B  -- draws non-join edges
    drawj  :: Drawable a => a -> Diagram B  -- as with draw but also draws dashed join edges
    fillDK :: Drawable a => Colour Double -> Colour Double -> a -> Diagram B -- fills with colours

where fillDK clr1 clr2 will fill darts with colour clr1 and kites with colour clr2 as well as drawing non-join edges.

These are the main drawing tools. However they are actually defined for any suitable backend b so have more general types.

(Update Sept 2024) From version 1.1 onwards of PenroseKiteDart, these are

    draw ::   (Drawable a, OKBackend b) =>
              a -> Diagram b
    drawj ::  (Drawable a, OKBackend) b) =>
              a -> Diagram b
    fillDK :: (Drawable a, OKBackend b) =>
              Colour Double -> Colour Double -> a -> Diagram b

where the class OKBackend is a check to ensure a backend is suitable for drawing 2D tilings with or without labels.

In these notes we will generally use the simpler description of types using B for a fixed chosen backend for the sake of clarity.

The drawing tools are each defined via the class function drawWith using Piece drawing functions.

    class Drawable a where
        drawWith :: (Piece -> Diagram B) -> a -> Diagram B
    
    draw = drawWith drawPiece
    drawj = drawWith drawjPiece
    fillDK clr1 clr2 = drawWith (fillPieceDK clr1 clr2)

To design a new drawing function, you only need to implement a function to draw a Piece, (let us call it newPieceDraw)

    newPieceDraw :: Piece -> Diagram B

This can then be elevated to draw any Drawable (including Tgraphs, VPatches, and Patches) by applying the Drawable class function drawWith:

    newDraw :: Drawable a => a -> Diagram B
    newDraw = drawWith newPieceDraw

Class DrawableLabelled

Class DrawableLabelled is defined with instances Tgraph and VPatch, but Patch is not an instance (because this does not retain vertex label information).

    class DrawableLabelled a where
        labelColourSize :: Colour Double -> Measure Double -> (Patch -> Diagram B) -> a -> Diagram B

So labelColourSize c m modifies a Patch drawing function to add labels (of colour c and size measure m). Measure is defined in Diagrams.Prelude with pre-defined measures tiny, verySmall, small, normal, large, veryLarge, huge. For most of our diagrams of Tgraphs, we use red labels and we also find small is a good default size choice, so we define

    labelSize :: DrawableLabelled a => Measure Double -> (Patch -> Diagram B) -> a -> Diagram B
    labelSize = labelColourSize red

    labelled :: DrawableLabelled a => (Patch -> Diagram B) -> a -> Diagram B
    labelled = labelSize small

and then labelled draw, labelled drawj, labelled (fillDK clr1 clr2) can all be used on both Tgraphs and VPatches as well as (for example) labelSize tiny draw, or labelCoulourSize blue normal drawj.

Further drawing functions

There are a few extra drawing functions built on top of the above ones. The function smart is a modifier to add dashed join edges only when they occur on the boundary of a Tgraph

    smart :: HasGraph a => (VPatch -> Diagram B) -> a -> Diagram B

So smart vpdraw g will draw dashed join edges on the boundary of g before applying the drawing function vpdraw to the VPatch for g. For example the following all draw dashed join edges only on the boundary for a Tgraph g

    smart draw g
    smart (labelled draw) g
    smart (labelSize normal draw) g

When using labels, the function rotating allows a Tgraph to be drawn rotated without rotating the labels.

    rotating :: HasGraph a => Angle Double -> (VPatch -> b) -> a -> b
    rotating angle vpdraw = vpdraw . rotate angle . makeVP

So for example,

    rotating (90@@deg) (labelled draw) g

makes sense for a Tgraph g. Of course if there are no labels we can simply use

    rotate (90@@deg) (draw g)

Similarly aligning allows a Tgraph to be aligned on the X-axis using a pair of vertex numbers before drawing.

    aligning :: HasGraph a => (Vertex,Vertex) -> (VPatch -> b) -> a -> b
    aligning (a,b) vpdraw = vpdraw . alignXaxis (a,b) . makeVP

So, for example, if Tgraph g has vertices a and b, both

    aligning (a,b) draw g
    aligning (a,b) (labelled draw) g

make sense. Note that the following two examples are wrong. Even though they type check, they re-orient g without repositioning the boundary joins.

    smart (labelled draw . rotate angle) g      -- WRONG
    smart (labelled draw . alignXaxis (a,b)) g  -- WRONG

Instead use

    smartRotating angle (labelled draw) g
    smartAligning (a,b) (labelled draw) g

where

    smartRotating :: HasGraph a =>  Angle Double -> (VPatch -> Diagram B) -> a -> Diagram B
    smartAligning  :: HasGraph a => (Vertex,Vertex) -> (VPatch -> Diagram B) -> a -> Diagram B

are defined using

    smartOn :: HasGraph a => a -> (VPatch -> Diagram B) -> VPatch -> Diagram B

Here, smartOn g vpdraw vp uses the given vp for drawing boundary joins and drawing faces of g (with vpdraw) rather than converting g to a new VPatch. This assumes vp has locations for vertices in g.

The function

    drawForce :: Tgraph -> Diagram B

will (smart) draw a Tgraph g in red overlaid (using <>) on the result of force g as in figure 6. Similarly

    drawPCompose  :: Tgraph -> Diagram B

applied to a Tgraph g will draw the result of a partial composition of g as in figure 7. That is a drawing of compose g but overlaid with a drawing of the remainder faces of g shown in pale green.

Both these functions make use of sharing a vertex location map to get correct alignments of overlaid diagrams. In the case of drawForce g, we know that a VPatch for force g will contain all the vertex locations for g since force only adds to a Tgraph (when it succeeds). So when constructing the diagram for g we can use the VPatch created for force g instead of starting afresh. Similarly for drawPCompose g the VPatch for g contains locations for all the vertices of compose g so compose g is drawn using the VPatch for g instead of starting afresh.

The location map sharing is done with

    subFaces :: HasFaces a => 
                a -> VPatch -> VPatch

so that subFaces fcs vp is a VPatch with the same vertex locations as vp, but replacing the faces of vp with fcs. [Of course, this can go wrong if the new faces have vertices not in the domain of the vertex location map so this needs to be used with care. Any errors would only be discovered when a diagram is created.]

For cases where labels are only going to be drawn for certain faces, we need a version of subFaces which also gets rid of vertex locations that are not relevant to the faces. For this situation we have

    restrictTo:: HasFaces a => 
                 a -> VPatch -> VPatch

which filters out un-needed vertex locations from the vertex location map. Unlike subFaces, restrictTo checks for missing vertex locations, so restrictTo fcs vp raises an error if a vertex in fcs is missing from the keys of the vertex location map of vp.

5. Forcing in More Detail

The force rules

The rules used by our force algorithm are local and derived from the fact that there are seven possible vertex types as depicted in figure 8.

Figure 8: Seven vertex types

Our rules are shown in figure 9 (omitting mirror symmetric versions). In each case the TileFace shown yellow needs to be added in the presence of the other TileFaces shown.

Figure 9: Rules for forcing

Main Forcing Operations

To make forcing efficient we convert a Tgraph to a BoundaryState to keep track of boundary information of the Tgraph, and then calculate a ForceState which combines the BoundaryState with a record of awaiting boundary edge updates (an update map), and an UpdateGenerator. Then each face addition is carried out on a ForceState, converting back when all the face additions are complete. It makes sense to apply force (and related functions) to a Tgraph, a BoundaryState, or a ForceState, so we define a class Forcible with instances Tgraph, BoundaryState, and ForceState.

This allows us to define

    force :: Forcible a => a -> a
    tryForce :: Forcible a => a -> Try a

The first will raise an error if a stuck tiling is encountered. The second uses a Try result which produces a Left string for failures and a Right a for successful result a.

There are several other operations related to forcing including

    stepForce :: Forcible a => Int -> a -> a
    tryStepForce  :: Forcible a => Int -> a -> Try a

    addHalfDart, addHalfKite :: Forcible a => Dedge -> a -> a
    tryAddHalfDart, tryAddHalfKite :: Forcible a => Dedge -> a -> Try a

The first two force (up to) a given number of steps (=face additions) and the other four add a half dart/kite on a given boundary edge.

Update Generators

An update generator is used to calculate which boundary edges can have a certain update. There is an update generator for each force rule, but also a combined (all update) generator. The force operations mentioned above all use the default all update generator (defaultAllUGen) but there are more general (with) versions that can be passed an update generator of choice. For example

    forceWith :: Forcible a => UpdateGenerator -> a -> a
    tryForceWith :: Forcible a => UpdateGenerator -> a -> Try a

We can also define

    wholeTiles :: Forcible a => a -> a
    wholeTiles = forceWith wholeTileUpdates

where wholeTileUpdates is an update generator that just finds boundary join edges to complete whole tiles.

In fact UpdateGenerators are functions that take a BoundaryState and a focus (list of boundary directed edges) to produce an update map. Each Update is calculated as either a SafeUpdate (where two of the new face edges are on the existing boundary and no new vertex is needed) or an UnsafeUpdate (where only one edge of the new face is on the boundary and a new vertex needs to be created for a new face).

    type UpdateGenerator = BoundaryState -> [Dedge] -> Try UpdateMap
    type UpdateMap = Map.Map Dedge Update
    data Update = SafeUpdate TileFace 
                | UnsafeUpdate (Vertex -> TileFace)

Completing (executing) an UnsafeUpdate requires a touching vertex check to ensure that the new vertex does not clash with an existing boundary vertex. Using an existing (touching) vertex would create a crossing boundary so such an update has to be blocked.

Forcible Class Operations

The Forcible class operations are higher order and designed to allow for easy additions of further generic operations. They take care of conversions between Tgraphs, BoundaryStates and ForceStates. The first two are designed to create functions that return the same Forcible type as the input.

    class Forcible a where
      tryFSOp :: (ForceState -> Try ForceState) -> a -> Try a
      tryChangeBoundary :: (BoundaryState -> Try BoundaryChange) -> a -> Try a
      tryInitFS :: a -> Try ForceState

For example, given any f:: ForceState -> Try ForceState , then f can be generalised to work on any Forcible using tryFSOp f. This is used to define both tryForce and tryStepForce.

Similarly given any f:: BoundaryState -> Try BoundaryChange , then f can be generalised to work on any Forcible using tryChangeBoundary f. This is used to define tryAddHalfDart and tryAddHalfKite.

Note that the type BoundaryChange contains a resulting BoundaryState, the single TileFace that has been added, a list of edges removed from the boundary (of the BoundaryState prior to the face addition), and a list of the (3 or 4) boundary edges affected around the change that require checking or re-checking for updates.

The class function tryInitFS will create an initial ForceState for any Forcible. If the Forcible is already a ForceState it will do nothing. Otherwise it will calculate updates for the whole boundary using defaultAllUGen.

The update generator is assumed to be defaultAllUGen but this can be changed using

    tryFSOpWith :: Forcible a => UpdateGenerator -> (ForceState -> Try ForceState) -> a -> Try a

so, for example, we defined

    tryForceWith ugen = tryFSOpWith ugen tryForce

Efficient chains of forcing operations.

Note that (force . force) does the same as force, but we might want to chain other force related steps in a calculation.

For example, consider the following combination which, after decomposing a Tgraph, forces, then adds a half dart on a given boundary edge (d) and then forces again.

    combo :: Dedge -> Tgraph -> Tgraph
    combo d = force . addHalfDart d . force . decompose

Since decompose produces a Tgraph, the instances of force and addHalfDart d will have type Tgraph -> Tgraph so each of these operations, will begin and end with conversions between Tgraph and ForceState. We would do better to avoid these wasted intermediate conversions working only with ForceStates and keeping only those necessary conversions at the beginning and end of the whole sequence.

This can be done using tryFSOp. To see this, let us first re-express the forcing sequence using the Try monad, so

    force . addHalfDart d . force

becomes

    tryForce <=< tryAddHalfDart d <=< tryForce

Note that (<=<) is the Kliesli arrow which replaces composition for Monads (defined in Control.Monad). (We could also have expressed this right to left sequence with a left to right version tryForce >=> tryAddHalfDart d >=> tryForce). The definition of combo becomes

    combo :: Dedge -> Tgraph -> Tgraph
    combo d = runTry . (tryForce <=< tryAddHalfDart d <=< tryForce) . decompose

This has no performance improvement, but now we can pass the sequence to tryFSOp to remove the unnecessary conversions between steps.

    combo :: Dedge -> Tgraph -> Tgraph
    combo d = runTry . tryFSOp (tryForce <=< tryAddHalfDart d <=< tryForce) . decompose

The sequence actually has type Forcible a => a -> Try a but when passed to tryFSOp it specialises to type ForceState -> Try ForseState. This ensures the sequence works on a ForceState and any conversions are confined to the beginning and end of the sequence, avoiding unnecessary intermediate conversions.

A limitation of forcing

To avoid creating touching vertices (or crossing boundaries) a BoundaryState keeps track of locations of boundary vertices. At around 35,000 face additions in a single force operation the calculated positions of boundary vertices can become too inaccurate to prevent touching vertex problems. In such cases it is better to use

    recalibratingForce :: Forcible a => a -> a
    tryRecalibratingForce :: Forcible a => a -> Try a

These work by recalculating all vertex positions at 20,000 step intervals to get more accurate boundary vertex positions. For example, 6 decompositions of the kingGraph has 2,906 faces. Applying force to this should result in 53,574 faces but will go wrong before it reaches that. This can be fixed by calculating either

    recalibratingForce (decompositions kingGraph !!6)

or using an extra force before the decompositions

    force (decompositions (force kingGraph) !!6)

In the latter case, the final force only needs to add 17,864 faces to the 35,710 produced by decompositions (force kingGraph) !!6.

6. Advanced Operations

Guided comparison of `Tgraph`s

Asking if two Tgraphs are equivalent (the same apart from choice of vertex numbers) is a an np-complete problem. However, we do have an efficient guided way of comparing Tgraphs. In the module Tgraph.Rellabelling we have

    sameGraph :: (Tgraph,Dedge) -> (Tgraph,Dedge) -> Bool

The expression sameGraph (g1,d1) (g2,d2) asks if g2 can be relabelled to match g1 assuming that the directed edge d2 in g2 is identified with d1 in g1. Hence the comparison is guided by the assumption that d2 corresponds to d1.

It is implemented using

    tryRelabelToMatch :: (Tgraph,Dedge) -> (Tgraph,Dedge) -> Try Tgraph

where tryRelabelToMatch (g1,d1) (g2,d2) will either fail with a Left report if a mismatch is found when relabelling g2 to match g1 or will succeed with Right g3 where g3 is a relabelled version of g2. The successful result g3 will match g1 in a maximal tile-connected collection of faces containing the face with edge d1 and have vertices disjoint from those of g1 elsewhere. The comparison tries to grow a suitable relabelling by comparing faces one at a time starting from the face with edge d1 in g1 and the face with edge d2 in g2. (This relies on the fact that Tgraphs are connected with no crossing boundaries, and hence tile-connected.)

The above function is also used to implement

    tryFullUnion:: (Tgraph,Dedge) -> (Tgraph,Dedge) -> Try Tgraph

which tries to find the union of two Tgraphs guided by a directed edge identification. However, there is an extra complexity arising from the fact that Tgraphs might overlap in more than one tile-connected region. After calculating one overlapping region, the full union uses some geometry (calculating vertex locations) to detect further overlaps.

Finally we have

    commonFaces:: (Tgraph,Dedge) -> (Tgraph,Dedge) -> [TileFace]

which will find common regions of overlapping faces of two Tgraphs guided by a directed edge identification. The resulting common faces will be a sub-collection of faces from the first Tgraph. These are returned as a list as they may not be a connected collection of faces and therefore not necessarily a Tgraph.

Empires and SuperForce

In Empires and SuperForce we discussed forced boundary coverings which were used to implement both a superForce operation

    superForce:: Forcible a => a -> Forced a

and operations to calculate empires.

We will not repeat the descriptions here other than to note that

    forcedBoundaryECovering:: Tgraph -> [Forced Tgraph]

finds boundary edge coverings after forcing a Tgraph. That is, forcedBoundaryECovering g will first force g, then (if it succeeds) finds a collection of (forced) extensions to force g such that

each extension has the whole boundary of force g as internal edges.
each possible addition to a boundary edge of force g (kite or dart) has been included in the collection.

(possible here means – not leading to a stuck Tgraph when forced.) There is also

    forcedBoundaryVCovering:: Tgraph -> [Forced Tgraph]

which does the same except that the extensions have all boundary vertices internal rather than just the boundary edges. In both cases the result is a list of explicitly forced Tgraphs (discussed next).

Combinations and Explicitly Forced

We introduced a new type Forced (in v 1.3) to enable a forcible to be explictily labelled as being forced. For example

    forceF    :: Forcible a => a -> Forced a 
    tryForceF :: Forcible a => a -> Try (Forced a)
    forgetF   :: Forced a -> a

This allows us to restrict certain functions which expect a forced argument by making this explicit.

    composeF :: HasGraph a => Forced a -> Forced Tgraph

The definition makes use of theorems established in Graphs,Kites and Darts and Theorems that composing a forced Tgraph does not require a check (for connectedness and no crossing boundaries) and the result is also forced. This can then be used to define efficient combinations such as

    compForce:: (Forcible a, HasGraph a) => a -> Forced Tgraph      -- compose after forcing
    compForce = composeF . forceF

    allCompForce:: (Forcible a, HasGraph a) => a -> [Forced Tgraph] -- iterated (compose after force) while not emptyTgraph
    maxCompForce:: (Forcible a, HasGraph a) => a -> Forced Tgraph   -- last item in allCompForce (or emptyTgraph)

Note that BoundaryState, ForceState as well as Tgraph and Forced versions of these are all instances of class HasGraph.

Tracked Tgraphs

The type

    data TrackedTgraph = TrackedTgraph
       { tgraph  :: Tgraph
       , tracked :: [[TileFace]] 
       } deriving Show

has proven useful in experimentation as well as in producing artwork with darts and kites. The idea is to keep a record of sub-collections of faces of a Tgraph when doing both force operations and decompositions. A list of the sub-collections forms the tracked list associated with the Tgraph. We make TrackedTgraph an instance of class Forcible by having force operations only affect the Tgraph and not the tracked list. The significant idea is the implementation of

    decomposeTracked :: TrackedTgraph -> TrackedTgraph

Decomposition of a Tgraph involves introducing a new vertex for each long edge and each kite join. These are then used to construct the decomposed faces. For decomposeTracked we do the same for the Tgraph, but when it comes to the tracked collections, we decompose them re-using the same new vertex numbers calculated for the edges in the Tgraph. This keeps a consistent numbering between the Tgraph and tracked faces, so each item in the tracked list remains a sub-collection of faces in the Tgraph.

The function

    drawTrackedTgraph :: [VPatch -> Diagram B] -> TrackedTgraph -> Diagram B

is used to draw a TrackedTgraph. It uses a list of functions to draw VPatches. The first drawing function is applied to a VPatch for any untracked faces. Subsequent functions are applied to VPatches for the tracked list in order. Each diagram is beneath later ones in the list, with the diagram for the untracked faces at the bottom. The VPatches used are all restrictions of a single VPatch for the Tgraph, so will be consistent in vertex locations. When labels are used, there is also a drawTrackedTgraphRotating and drawTrackedTgraphAligning for rotating or aligning the VPatch prior to applying the drawing functions.

Note that the result of calculating empires (see Empires and SuperForce ) is represented as a TrackedTgraph. The result is actually the common faces of a forced boundary covering, but a particular element of the covering (the first one) is chosen as the background Tgraph with the common faces as a tracked sub-collection of faces. Hence we have

    empire1, empire2 :: Tgraph -> TrackedTgraph
    
    drawEmpire :: TrackedTgraph -> Diagram B

Figure 10 was also created using TrackedTgraphs.

Figure 10: Using a TrackedTgraph for drawing

7. Other Reading

Previous related blogs are:

Diagrams for Penrose Tiles – the first blog introduced drawing Pieces and Patches (without using Tgraphs) and provided a version of decomposing for Patches (decompPatch).
Graphs, Kites and Darts intoduced Tgraphs. This gave more details of implementation and results of early explorations. (The class Forcible was introduced subsequently).
Empires and SuperForce – these new operations were based on observing properties of boundaries of forced Tgraphs.
Graphs,Kites and Darts and Theorems established some important results relating force, compose, decompose.

Dan Piponi (sigfpe): Constructing Clifford Algebras using the Super Tensor Product

noreply@blogger.com (sigfpe) — Thu, 23 Apr 2026 00:28:40 +0000

Google have stopped supporting the Chart API so all of the mathematics notation below is missing. There is a PDF version of this article at GitHub.

Some literate Haskell but little about this code is specific to Haskell...

> {-# LANGUAGE DataKinds #-}
> {-# LANGUAGE TypeFamilies #-}
> {-# LANGUAGE TypeOperators #-}
> {-# LANGUAGE UndecidableInstances #-}
> 
> import GHC.TypeLits

Introduction

This is a followup to Geometric Algebra for Free and More Low Cost Geometric Algebra.

In those articles I showed how you could build up the Clifford algebras like so:

type Cliff1  = Complex R
type Cliff1' = Split R
type Cliff2  = Quaternion R
type Cliff2' = Matrix R
type Cliff3  = Quaternion Cliff1'
type Cliff3' = Matrix Cliff1
type Cliff4  = Quaternion Cliff2'
type Cliff4' = Matrix Cliff2
type Cliff5  = Quaternion Cliff3'
...

I used CliffN as the Clifford algebra for a negative definite inner product and CliffN' for the positive definite case. It's not a completely uniform sequence in the sense that CliffN is built from CliffN' for dimension two lower and you use a mix of Matrix and Quaternion.

The core principle making this work is that for type constructors implemented like Matrix, Quaternion etc. we have the property that

eg. Matrix (Quaternion Float) is effectively the same thing as Matrix Float Quaternion Float.

But John Baez pointed out to me that you can build up the CliffN algebras much more simply enabling us to use these definitions:

> type Cliff1 = Complex Float
> type Cliff2 = Complex Cliff1
> type Cliff3 = Complex Cliff2
> type Cliff4 = Complex Cliff3
> type Cliff5 = Complex Cliff4

...

Or even better:

> type family Cliff (n :: Nat) :: * where
>   Cliff 0 = Float
>   Cliff n = Complex (Cliff (n - 1))

But there's one little catch. We have to work, not with the tensor product, but the super tensor product.

We define Complex the same way as before:

> data Complex a = C a a deriving (Eq, Show)

Previously we used a definition of multiplication like this:

instance Num a => Num (Complex a) where
  C a b * C c d = C (a * c - b * d) (a * d + b * c)

We can think of C a b in Complex R as representing the element $1\otimes a+i\otimes b$. The definition of multiplication in a tensor product of algebras is

\[(a\otimes b)(c\otimes d)=(ac)\otimes(bd).\]

So we have

\[(1\otimes a+i\otimes b)(1\otimes c+i\otimes d)\]

\[=1\otimes ac+i\otimes ad+i\otimes bc+i^2\otimes bd\]

\[=1\otimes(ac-bd)+i\otimes(ad+bc).\]

This means that line of code we wrote above defining * for Complex isn't simply a definition of multiplication of complex numbers, it says how to multiply in an algebra tensored with the complex numbers.

Let's go Super!

A superalgebra is an algebra graded by where is the ring of integers modulo 2. What that means is that we have some algebra that can be broken down as a direct sum (the subscripts live in ) with the property that multiplication respects the grading, ie. if is in and is in then is in .

The elements of are called "even" (or bosonic) and those in "odd" (or fermionic). Often even elements commute with everything and odd elements anticommute with each other but this isn't always the case. (The superalgebra is said to be supercommutative when this happens. This is a common pattern: a thing X becomes a superX if it has odd and even parts and swapping two odd things introduces a sign flip.)

The super tensor product is much like the tensor product but it respects the grading. This means that if is in and is in then is in . From now on I'm using to mean super tensor product.

Multiplication in the super tensor product of two superalgebras and is now defined by the following modified rule: if is in and is in then . Note that the sign flip arises when we shuffle an odd left past an odd .

The neat fact that John pointed out to me is that

\[Cliff_n=\mathbb{C}\otimes\mathbb{C}\otimes\ldots\text{ n times }\ldots\otimes\mathbb{C}.\]

We have to modify our definition of * to take into account that sign flip.

I initially wrote a whole lot of code to define a superalgebra as a pair of algebras with four multiplication operations and it got a bit messy. But I noticed that the only specifically superalgebraic operation I ever performed on an element of a superalgebra was negating the odd part of an element.

So I could define SuperAlgebra like so:

class SuperAlgebra a where
  conjugation :: a -> a

where conjugation is the negation of the odd part.

(I'm not sure if this operation corresponds to what is usually called conjugation in this branch of mathematics.)

But there's a little efficiency optimization I want to write. If I used the above definition, then later I'd often find myself computing a whole lot of negates in a row. This means applying negate to many elements of large algebraic objects even though any pair of them cancel each other's effect. So I add a little flag to my conjugation function that is used to say we want an extra negate and we can accumulate flips of a flag rather than flips of lots of elements.

> class SuperAlgebra a where
>   conjugation :: Bool -> a -> a

Here's our first instance:

> instance SuperAlgebra Float where
>   conjugation False x = x
>   conjugation True x = negate x

This is saying that the conjugation is the identity on Float but if we want to perform an extra flip we can set the flag to True. Maybe I should call it conjugationWithOptionalExtraNegation.

And now comes the first bit of non-trivial superalgebra:

> instance (Num a, SuperAlgebra a) => SuperAlgebra (Complex a) where
>   conjugation e (C a b) = C (conjugation e a) (conjugation (not e) b)

We consider to be even and to be odd. When we apply the conjugation to then we can just apply it directly to . But that flips the "parity" of (because tensor product respects the grading) so we need to swap when we use the conjugation. And that should explain why conjugation is defined the way it is.

Now we can use the modified rule for defined above:

> instance (Num a, SuperAlgebra a) => Num (Complex a) where
>   fromInteger n = C (fromInteger n) 0
>   C a b + C a' b' = C (a + a') (b + b')
>   C a b * C c d = C (a * c - conjugation False b * d)
>                     (conjugation False a * d + b * c) 
>   negate (C a b) = C (negate a) (negate b)
>   abs = undefined
>   signum = undefined

For example, conjugation False is applied to the first on the RHS because implicitly represents an term and when expanding out the product we shuffle the (odd) in left of . It doesn't get applied to the second because and remain in the same order.

That's it!

Tests

I'll test it with some examples from Cliff3:

> class HasBasis a where
>   e :: Integer -> a


> instance HasBasis Float where
>   e = undefined


> instance (Num a, HasBasis a) => HasBasis (Complex a) where
>   e 0 = C 0 1
>   e n = C (e (n - 1)) 0


> make a b c d e f g h =
>   C (C (C a b) (C c d))
>     (C (C e f) (C g h))


> e1, e2, e3, e21, e31, e32, e321 :: Cliff 3
> e1 = e 0
> e2 = e 1
> e21 = e2 * e1
> e3 = e 2
> e31 = e3 * e1
> e32 = e3 * e2
> e321 = e3 * e2 * e1


> main = do
>     print (e1 * e1 + 1 == 0)
>     print (e31 * e31 + 1 == 0)
>     print (e3 * e3 + 1 == 0)
>     print (e21 * e21 + 1 == 0)
>     print (e2 * e2 + 1 == 0)
>     print (e32 * e32 + 1 == 0)
>     print (e321 * e321 - 1 == 0)
>     print (e3 * e2 * e1 - e321 == 0)
>     print (e2 * e1 - e21 == 0)
>     print (e3 * e1 - e31 == 0)
>     print (e3 * e2 - e32 == 0)
>     print (e21 * e32 - e31 == 0)

Observation

The implementation of multiplication looks remarkably like it's the Cayley-Dickson construction. It can't be (because iterating it three times gives you a non-associative algebra but the Clifford algebras are associative). Nonetheless, I think comparison with Cayley-Dickson may be useful.

Efficiency

As mentioned above, before I realised I just needed the conjugation operation I wrote the above code with an explicit split of a superalgebra into two pieces intertwined by four multiplications. I think the previous approach may have a big advantage - it may be possible to use variations on the well known "speed-up" of complex multiplication that uses three real multiplications instead of four. This should lead to a fast implementation of Clifford algebras.

Also be warned: you can kill GHC if you turn on optimization and try to multiply elements of high-dimensional Clifford algebras. I think it tries to inline absolutely everything and you end up with a block of code that grows exponentially with .

Note also that this code translates directly into many languages.

Dan Piponi (sigfpe): Self-referential logic via self-referential circuits

noreply@blogger.com (sigfpe) — Wed, 22 Apr 2026 23:29:47 +0000

Introduction

TL;DR The behaviour of a certain kind of delay component has a formal similarity to Löb's theorem which gives a way to embed part of provability logic into electronic circuits.

Here's a famous paradoxical sentence:

This sentence is false

If it's false then it's true and if it's true then it's false.

Here's a paradoxical electronic circuit:

The component in the middle is an inverter. If the output of the circuit is high then its input is high and then it's output must be low, and vice versa.

There's a similarity here. But with a bit of tweaking you can turn the similarity into an isomorphism of sorts.

In the first case we avoid paradox by noting that in the mathematical frameworks commonly used by mathematicians it's impossible, in general, for a statement to assert it's own falsity. Instead, a statement can assert its own unprovability and then we get Gödel's incompleteness theorems and a statement that is apparently true and yet can't be proved.

In the second case we can't model the circuit straightforwardly as a digital circuit. In practice it might settle down to a voltage that lies between the official high and low voltages so we have to model it as an analogue circuit. Or instead we can introduce a clock and arrange that the feedback in the circuit is delayed. We then get an oscillator circuit that can be thought of as outputting a stream of bits.

The observation I want to make is that if the feedback delay is defined appropriately, these two scenarios are in some sense isomorphic. This means that we can model classic results about provability, like Gödel's incompleteness theorems, using electronic circuits. We can even use such circuits to investigate what happens when logicians or robots play games like Prisoner's Dilemma. I'll be making use of results found in Boolos' book on The Logic of Provability and some ideas I borrowed from Smoryński's paper on Fixed Point Algebras. I'll be assuming the reader has at least a slight acquaintance with ithe ideas behind provability logic.

Provability Logic

There are many descriptions of provability logic (aka GL) available online, so I'm not going to repeat it all here. However, I've put some background material in the appendix below and I'm going to give a very brief reminder now.

Start with (classical) propositional calculus which has a bunch of variables with names like $a, b, c, d, \ldots$ and connectives like $\wedge$ for AND, $\vee$ for OR, $\neg$ for NOT and $\rightarrow$ for implication. (Note that $a\rightarrow b = \neg a\vee b$.)

Provability logic extends propositional calculus by adding a unary operator $\Box$. (I apologise, that's meant to be a □ but it's coming out like $\Box$ in LaTeX formulae. I think it's a bug in Google's LaTeX renderer.) The idea is that $\Box p$ asserts that $p$ is provable in Peano Arithmetic, aka PA. In addition to the axioms of propositional calculus we have

$\Box(p\rightarrow q)\rightarrow\Box p\rightarrow\Box q$

and

$\Box p\rightarrow\Box\Box p$

as well as a rule that allows us to deduce $\Box p$ from $p$.

We also have this fixed point property:

Let $F(p)$ be any predicate we can write in the language of GL involving the variable $p$, and suppose that every appearance of $p$ in $F(p)$ is inside a $\Box$, e.g. $F(p)=\Box p\vee\Box(\neg p)$. Then there is a fixed point, i.e. a proposition $q$ that makes no mention of $p$ such that $q\leftrightarrow F(q)$ is a theorem. In effect, for any such $F$, $q$ is a proposition that asserts $F(q)$.

See the appendix for a brief mention of why we should expect this to be true.

From the fixed point property we can deduce Löb's theorem: $\Box(\Box p\rightarrow p)\rightarrow\Box p$. There is a proof at wikipedia that starts from the fixed point property.

We can also deduce the fixed point property from Löb's theorem so it's more usual to take Löb's theorem as an axiom of GL and show that the fixed point property follows. You can think of Löb's theorem as a cunning way to encode the fixed point property. In fact you can argue that it's a sort of Y-combinator, the function that allows the formation of recursive fixed points in functional programming languages. (That's also, sort of, the role played by the loeb function I defined way back. But note that loeb isn't really a proof of Löb's theorem, it just has formal similarities.)

Back to electronic circuits

In order to make digital circuits with feedback loops well-behaved I could introduce a circuit element that results in a delay of one clock cycle. If you insert one of these into the inverter circuit I started with you'll end up with an oscillator that flips back and forth between 0 and 1 on each clock cycle. But I want to work with something slightly stricter. I'd like my circuits to eventually stop oscillating. (I have an ulterior motive for studying these.) Let me introduce this component:

It is intended to serve as a delayed latch and I'll always have the flow of data being from left to right. The idea is that when it is switched on it outputs 1. It keeps outputting 1 until it sees a 0 input. When that happens, then on the next clock cycle its output drops to 0 and never goes back up to 1 until reset.

Because the output of our delay-latch isn't a function of its current input, we can't simply describe its operation as a mathematical function from $\{0,1\}$ to $\{0,1\}$. Instead let's think of electronic components as binary operators on bitstreams, i.e. infinite streams of binary digits like ...00111010 with the digits emerging over time starting with the one written on the right and working leftwards. The ordinary logic gates perform bitwise operations which I'll represent using the operators in the C programming language. For example,

...001110 & ...101010 = ...001010

and

~...101 = ...010

and so on. Let's use □ to represent the effect of latch-delay on a bitstream. We have, for example,

□...000 = ...001

and

□...11101111 = ...00011111.

The operator □ takes the (possibly empty) contiguous sequence of 1's at the end of the bitstream, extends it by one 1, and sets everything further to the left to 0. If we restrict ourselves to bitstreams that eventually become all 0's or all 1's on the left, then bitstreams are in one-to-one correspondence with the integers using the twos complement representation. For example ...111111, all 1's, represents the number -1. I'll simply call the bistreams that represent integers integers. With this restriction we can use a classic C hacker trick to write □p=p^(p+1) where ^ is the C XOR operator. The operator □ outputs the bits that get flipped when you add one.

Let's use the symbol → so that a → b is shorthand for ~a|b. Here are some properties of □:

1. □(-1) = -1

2. □p → □□p = -1

3. □(p → q) → □p → □q = -1

In addition we have the fixed point property:

Let F(p) be any function of p we can write using □ and the bitwise logical operators and such that all occurrences of p occur inside □. Then there is a unique bitstream q such that q=F(q).

We can make this clearer if we return to circuits. F(p) can be thought of as a circuit that takes p as input and outputs some value. We build the circuit using only boolean logic gates and delay-latch. We allow feedback loops, but only ones that go through delay-latches. With these restrictions it's pretty clear that the circuit is well-behaved and deterministically outputs a bitstream.

We also have the Löb property:

4. □(□p → p) → □p = -1

We can see this by examining the definition of □. Intuitively it says something like "once □ has seen a 0 input then no amount of setting input bits to 1 later in the stream make any different to its output".

I hope you've noticed something curious. These properties are extremely close to the properties of $\Box$ in GL. In fact, these electronic circuits form a model of the part of GL that doesn't involve variable names, i.e. what's known as letterless GL. We can formalise this:

1. Map $\bot$ to a wire set to 0, which outputs ...000 = 0.

2. Map $\top$ to a wire set to 1, which outputs ...111 = -1.

3. Map $p \circ q$, where $\circ$ is a binary connective, by creating a circuit that takes the outputs from the circuits for $p$ and $q$ and passes them into the corresponding boolean logic gate.

4. Map $\Box p$ to the circuit for $p$ piped through a delay-latch.

For example, let's convert $\Box(\Box\bot\rightarrow\bot)\rightarrow\Box\bot$ into a circuit. I'm translating $a\rightarrow b$ to the circuit for $\neg a\vee b$.

I'm using red wires to mean wires carrying the value 1 rather than 0. I hope you can see that this circuit eventually settles into a state that outputs nothing but 1s.

We have this neat result:

Because delay-latch satisfies the same equations as $\Box$ in provability logic, any theorem, translated into a circuit, will produce a bistream of just 1s, i.e. -1.

But here's a more surprising result: the converse is true.

If the circuit corresponding to a letterless GL proposition produces a bistream of just 1s then the proposition is actually a theorem of GL.

I'm not going to prove this. (It's actually a disguised form of lemma 7.4 on p.95 of Boolos' book.) In the pictured example we got ...1111, so the circuit represents a theorem. As it represents Löb's theorem for the special case $p=\bot$ we should hope so. More generally, any bitstream that represents an integer can be converted back into a proposition that is equivalent to the original proposition. This means that bitstreams faithfully represent propositions of letterless GL. I'm not going to give the translation here but it's effectively given in Chapter 7 of Boolos. I'll use $\psi(p)$ to represent the translation from propositions to bitstreams via circuits that I described above. Use $\phi(b)$ to represent the translation of bitstream $b$ back into propositions. We have $p\leftrightarrow\phi(\psi(p))$. But I haven't given a full description of $\phi$ and I haven't proved here that it has this property.

Circuits with feedback

In the previous section I considered letterless propositions of GL. When these are translated into circuits they don't have feedback loops. But we can also "solve equations" in GL using circuits with feedback. The GL fixed point theorem above says that we can "solve" the equation $p\leftrightarrow F(p)$, with one letter $p$, to produce a letterless proposition $q$ such that $q\leftrightarrow F(q)$. Note here that $p$ is a letter in the language of GL. But I'm using $q$ to represent a proposition in letterless GL. If we build a circuit to represent $F$, and feed its output back into where $p$ appears, then the output bitstream represents the fixed point. Here's a translation of the equation $p \leftrightarrow \neg(\Box p \vee \Box\Box\Box p)$:

I'll let you try to convince yourself that such circuits always eventually output all 0's or all 1's. When we run the circuit we get the output ...1111000 = -8. As this is not -1 we know that the fixed point isn't a theorem. If I'd defined $\phi$ above you could use it to turn the bitstream back into a proposition.

The same, syntactically (optional section)

I have a Haskell library on github for working with GL: provability. This uses a syntactic approach and checks propositions for theoremhood using a tableau method. We can use it to analyse the above example with feedback. I have implemented a function, currently called value', to perform the evaluation of the bitstream for a proposition. However, in this case the fixedpoint function computes the fixed point proposition first and then converts to a bitstream rather than computing the bitstream directly from the circuit for F:

> let f p = Neg (Box p \/ Box (Box (Box p)))
> let Just p = fixedpoint f
> p
Dia T /\ Dia (Dia T /\ Dia (Dia T /\ Dia T))
> value' p
-8

(Note that Dia p means $\Diamond p = \neg\Box\neg p$.)

The function fixedpoint does a lot of work under the hood. (It uses a tableau method to carry out Craig interpolation.) The circuit approach requires far less work.

Applications

1. Programs that reason about themselves

In principle we can write a program that enumerates all theorems of PA. That means we can use a quine trick to write a computer program that searches for a proof, in PA, of its own termination. Does such a program terminate?

We can answer this with Löb's theorem. Let $p =$ "The program terminates". The program terminates if it can prove its termination. Formally this means we assume $\Box p\rightarrow p$. Using one of the derivation rules of GL we get $\Box(\Box p\rightarrow p)$. Löb's theorem now gives us $\Box p$. Feed that back into our original hypothesis and we get $p$. In other words, we deduce that our program does in fact terminate. (Thanks to Sridhar Ramesh for pointing this out to me.)

But we can deduce this using a circuit. We want a solution to $p\leftrightarrow \Box p$. Here's the corresponding circuit:

It starts by outputting 1's and doesn't stop. In other words, the fixed point is a theorem. And that tells us $p$ is a theorem. And hence that the program terminates.

2. Robots who reason about each others play in Prisoner's Dilemma

For the background to this problem see Robust Cooperation in the Prisoner's Dilemma at LessWrong. We have two robot participants $A$ and $B$ playing Prisoner's Dilemma. Each can examine the other's source code and can search for proofs that the opponent will cooperate. Suppose each robot is programmed to enumerate all proofs of PA and cooperate if it finds a proof that its opponent will cooperate. Here we have $p =$ "A will cooperate" and $q =$ "B will cooperate". Our assumptions about the behaviour of the robots are $p \leftrightarrow \Box q$ and $q \leftrightarrow \Box p$, and hence that $p \leftrightarrow \Box\Box p$. This corresponds to the circuit:

This outputs ...1111 = -1 so we can conclude $p$ and hence that these programs will cooperate. (Note that this doesn't work out nicely if robot B has a program that doesn't terminate but whose termination isn't provable in the formal system A is using. That means this approach is only good for robots that want to cooperate and want to confirm such cooperation. See the paper for more on this.)

At this point I really must emphasise that these applications are deceptively simple. I've shown how these simple circuits can answer some tricky problems about provability. But these aren't simply the usual translations from boolean algebra to logic gates. They work because circuits with delay-latch provide a model for letterless provability logic and that's only the case because of a lot of non-trivial theorem proving in Boolos that I haven't reproduced here. You're only allowed to use these simple circuits once you've seen the real proofs :-)

Things I didn't say above

1. I described the translation from propositions to circuits that I called $\psi$ above. But I didn't tell you what $\phi$ looks like. I'll leave this as an exercise. (Hint: consider the output from the translation of $\Box^n\bot$ into a circuit.)

2. The integers, considered as bistreams, with the bitwise operators, and the unary operator □p=p^(p+1), form an algebraic structure. For example, if we define ⋄p=~□~p we have a Magari algebra. Structures like these are intended to capture the essential parts of self-referential arguments in an algebraic way.

3. Because of the interpretation of □ as a delayed latch in a circuit you could view it as saying "my input was always true until a moment ago". This surely embeds provability logic in a temporal logic of some sort.

4. (Deleted speculations about tit-for-tat that need rethinking.)

5. For even the most complex letterless proposition in Boolos you could check its theoremhood with a pretty small circuit. You could even consider doing this with a steam powered pneumatic circuit. I had to say that to fulfil a prophecy and maintain the integrity of the timeline.

Appendix on provability

The modern notion of a proof is that it is a string of symbols generated from some initial strings called "axioms" and some derivation rules that make new strings from both axioms and strings you've derived previously. Usually we pick axioms that represent "self-evident" truths and we pick derivation rules that are "truth-preserving" so that every proof ends at a true proposition of which it is a proof. The derivation rules are mechanical in nature: things like "if you have this symbol here and that symbol there then you can replace this symbol with that string you derived earlier" etc.

You can represent strings of symbols using numbers, so-called Gödel numbers. Let's pick a minimal mathematical framework for working with numbers: Peano Arithmetic, aka PA. Let's assume we've made some choice of Gödel numbering scheme and when $p$ is a proposition, write $[p]$ for the number representing $p$. You can represent the mechanical derivation rules as operations on numbers. And that makes it possible to define a mathematical predicate $Prov$ that is true if and only if its argument represents a provable proposition.

In other words, we can prove $Prov([p])$ using PA if and only if $p$ is a proposition provable in PA.

The predicate $Prov$ has some useful properties:

1.If we can prove $p$, then we can prove $Prov([p])$.

We take the steps we used to prove $p$, and convert everything to propositions about numbers. If $Prov$ is defined correctly then we can convert that sequence of numbers into a sequence of propositions about those numbers that makes up a proof of $Prov(p)$.

2.$Prov([p\rightarrow q])$ and $Prov([p])$ imply $Prov([q])$

A fundamental step in any proof is modus ponens, i.e. that $p\rightarrow q$ and $q$ implies $p$. If $Prov$ does its job correctly then it had better know about this.

3.$Prov([p])$ implies $Prov([Prov([p])])$

One way is to prove this is to use Löb's theorem.

4. $Prov([\top])$

The trivially true statement had better be provable or $Prov$ is broken.

Constructing $Prov$ is conceptually straightforward but hard work. I'm definitely not going to do it here.

And there's one last thing we need: self-reference. If $p$ is a proposition, how can we possibly assert $Prov([p])$ without squeezing a copy of $[p]$ inside $p$? I'm not going to do that here either - just mention that we can use a variation of quining to achieve this. That allows us to form a proposition $p$ for which we can prove $p\leftrightarrow Prov([p])$. In fact, we can go further. We can find propositions that solve $p\leftrightarrow F(p)$ for any predicate $F(p)$ built from the usual boolean operations and $p$ as long as all of the occurrences of $p$ are inside the appearances of $Prov$. Even though we can't form a proposition that directly asserts its own falsity, we can form one that asserts that it is unprovable, or one that asserts that you can't prove that you can't prove that you can prove it, or anything along those lines.

Anyway, all that $[]$ and $Prov$ business is a lot of hassle. Provability logic, also known as GL, is intended to capture specifically the parts of PA that relate to provability. GL is propositional calculus extended with the provability operator $\Box$. The intention is that if $p$ is a proposition, $\Box p$ is a proposition in GL that represents $Prov([p])$ in PA. The properties of $Prov$ above become the axioms and derivation rules of GL in the main text.

Dan Piponi (sigfpe): Expectation-Maximization with Less Arbitrariness

noreply@blogger.com (sigfpe) — Wed, 22 Apr 2026 23:23:39 +0000

Introduction

Google have stopped supporting the Chart API so all of the mathematics notation below is missing. There is a PDF version of this article at GitHub.

There are many introductions to the Expectation-Maximisation algorithm. Unfortunately every one I could find uses arbitrary seeming tricks that seem to be plucked out of a hat by magic. They can all be justified in retrospect, but I find it more useful to learn from reusable techniques that you can apply to further problems. Examples of tricks I've seen used are:

Using Jensen's inequality. It's easy to find inequalities that apply in any situation. But there are often many ways to apply them. Why apply it to this way of writing this expression and not that one which is equal?
Substituting $1=A/A$ in the middle of an expression. Again, you can use $1=A/A$ just about anywhere. Why choose this $A$ at this time? Similarly I found derivations that insert a $B-B$ into an expression.
Majorisation-Minimisation. This is a great technique, but involves choosing a function that majorises another. There are so many ways to do this, it's hard to imagine any general purpose method that tells you how to narrow down the choice.

My goal is to fill in the details of one key step in the derivation of the EM algorithm in a way that makes it inevitable rather than arbitrary. There's nothing original here, I'm merely expanding on a stackexchange answer.

Generalities about EM

The EM algorithm seeks to construct a maximum likelihood estimator (MLE) with a twist: there are some variables in the system that we can't observe.

First assume no hidden variables. We assume there is a vector of parameters $\theta=(\theta_i)$ that defines some model. We make some observations $x=(x_j)$. We have a probability density $P(x|\theta)$ that depends on $\theta$. The likelihood of $\theta$ given the observations $x$ is $l(\theta|x)=P(x|\theta)$. The maximum likelhood estimator for $\theta$ is the choice of $\theta$ that maximises $l(\theta|x)$ for the $x$ we have observed.

Now suppose there are also some variables $z=(z_k)$ that we didn't get to observe. We assume a density $P(x,z|\theta)$. We now have

$P(x|\theta)=\sum_z P(x,z|\theta)$

where we sum over all possible values of $z$. The MLE approach says we now need to maximise

$l(\theta|x)=\sum_z P(x,z|\theta).$

One of the things that is a challenge here is that the components of $\theta$ might be mixed up among the terms in the sum. If, instead, each term only referred to its own unique block of $\theta_i$, then the maximisation would be easier as we could maximise each term independently of the others. Here's how we might move in that direction. Consider instead the log-likelihood

$\log l(\theta|x)=\log\sum_z P(x,z|\theta).$

Now imagine that by magic we could commute the logarithm with the sum. We'd need to maximise

$\sum_z \log P(x,z|\theta).$

One reason this would be to our advantage is that $P(x,z|\theta)$ often takes the form $\exp(f(x,z,\theta))$ where $f$ is a simple function to optimise. In addition, $f$ may break up as a sum of terms, each with its own block of $\theta_i$'s. Moving the logarithm inside the sum would give us something we could easily maximise term by term. What's more, the $P(x,z|\theta)$ for each $z$ is often a standard probability distribution whose likelihood we already know how to maximise. But, of course, we can't just move that logarithm in.

Maximisation by proxy

Sometimes a function is too hard to optimise directly. But if we have a guess for an optimum, we can replace our function with a proxy function that approximates it in the neighbourhood of our guess and optimise that instead. That will give us a new guess and we can continue from there. This is the basis of gradient descent. Suppose $f$ is a differentiable function in a neighbourhood of $x_0$. Then around $x_0$ we have

$f(x) \approx f(x_0) f'(x_0)\cdot (x-x_0).$

We can try optimising $f(x_0) f'(x_0)\cdot (x-x_0)$ with respect to $x$ within a neighbourhood of $x_0$. If we pick a small circular neighbourhood then the optimal value will be in the direction of steepest descent. (Note that picking a circular neighbourhood is itself a somewhat arbitrary step, but that's another story.) For gradient descent we're choosing $f(x_0) f'(x_0)\cdot (x-x_0)$ because it matches both the value and derivatives of $f$ at $x_0$. We could go further and optimise a proxy that shares second derivatives too, and that leads to methods based on Newton-Raphson iteration.

We want our logarithm of a sum to be a sum of logarithms. But instead we'll settle for a proxy function that is a sum of logarithms. We'll make the derivatives of the proxy match those of the original function precisely so we're not making an arbitrary choice.

Write

$\log l(\theta|x) = \log\sum_z P(x,z|\theta) \approx \sum_z\beta_z\log P(x,z|\theta) \text{constant}.$

The $\beta_z$ are constants we'll determine. We want to match the derivatives on either side of the $\approx$ at $\theta=\theta_0$:

$\frac{\partial \log l(\theta_0|x)}{\partial\theta_0}$ $=\frac{1}{l(\theta_0|x)} \frac{\partial l(\theta_0|x)}{\partial\theta_0} =\sum_z\frac{1}{l(\theta_0|x)} \frac{\partial P(x,z|\theta_0)}{\partial\theta_0}.$

On the other hand we have

$\frac{\partial}{\partial\theta_0}\sum_z\beta_z\log P(x,z|\theta_0) =\sum_z\beta_z\frac{1}{P(x,z|\theta_0)}\frac{\partial P(x,z|\theta_0)}{\partial\theta_0}$

To achieve equality we want to make these expressions match. We choose

$\beta_z = \frac{P(x,z|\theta_0)}{l(\theta_0|x)} = \frac{P(x,z|\theta_0)}{P(x|\theta_0)} = P(z|x,\theta_0).$

Our desired proxy function is:

$\sum_z P(z|x,\theta_0)\log P(x,z|\theta) + \text{const.} = E_{Z|x,\theta_0}(\log P(x,Z|\theta)) + \text{const.}$

So the procedure is to take an estimated $\theta_0$ and obtain a new estimate by optimising this proxy function with respect to $\theta$. This is the standard EM algorithm.

It turns out that this proxy has some other useful properties. For example, because of the concavity of the logarithm, the proxy is always smaller than the original likelihood. This means that when we optimise it we never optimise ``too far'' and that progress optimising the proxy is always progress optimising the original likelihood. But I don't need to say anything about this as it's all part of the standard literature.

Afterword

As a side effect we have a general purpose optimisation algorithm that has nothing to do with statistics. If your goal is to compute

$\operatorname{argmax}_x\sum_i\exp(f_i(x))$

you can iterate, at each step computing

$\operatorname{argmax}_x\sum_i\exp(f_i(x_0))f_i(x)$

where $x_0$ is the previous iteration. If the $f_i$ take a convenient form then this may turn out to be much easier.

Note

This was originally written as a PDF using LaTeX. It'll be available here for a while. Some fidelity was lost when converting it to HTML.

Dan Piponi (sigfpe): Logarithms and exponentials of functions

noreply@blogger.com (sigfpe) — Wed, 22 Apr 2026 23:16:27 +0000

Introduction

A popular question in mathematics is this: given a function $f$, what is its "square root" $g$ in the sense that $g(g(x)) = f(x)$. There are many questions about this on mathoverflow but it's also a popular subject in mathematics forums for non-experts. This question seems to have a certain amount of notoriety because it's easy to ask but hard to answer fully. I want to look at an approach that works nicely for formal power series, following from the Haskell code I wrote here. There are some methods for directly finding "functional square roots" for formal power series that start as $z a_2z^2 a_3z^3 \ldots$, but I want to approach the problem indirectly. When working with real numbers we can find square roots, say, by using $\sqrt{x}=\exp(\frac{1}{2}\log{x})$. I want to use an analogue of this for functions. So my goal is to make sense of the idea of the logarithm and exponential of a formal power series as composable functions. Warning: the arguments are all going to be informal.

Notation

There's potential for a lot of ambiguous notation here, especially as the usual mathematical notation for $n$th powers of trig functions is so misleading. I'm going to use $\circ$ for composition of functions and power series, and I'm going to use the notation $f^{\circ n}$ to mean the $n$th iterate of $f$. So $f^{n 1}(x) = f(x)f^n(x)$ and $f^{\circ n 1}(x) = f(f^{\circ n}(x))$. As I'll be working mostly in the ring of formal power series $R[\![z]\!]$ for some ring $R$, I'll reserve the variable $z$ to refer only to the corresponding element in this ring. I'll also use formal power series somewhat interchangeably with functions. So $z$ can be thought of as representing the identity function. To make sure we're on the same page, here are some small theorems in this notation:

$z^mz^n = z^{m n}$
$f^{\circ m}\circ f^{\circ n} = f^{\circ m n}$
$(1 z)^n = \sum_{i=0}^n{n\choose i}z^n$
$(1 z)^{\circ n}=n z$.

That last one simply says that adding one $n$ times is the same as adding $n$.

As I'm going to have ordinary logarithms and exponentials sitting around, as well as functional logarithms and exponentials, I'm going to introduce the notation $\operatorname{LOG}$ for functional logarithm and $\operatorname{EXP}$ for functional exponentiation.

Preliminaries

The first goal is to define a non-trivial function $\operatorname{LOG}$ with the fundamental property that $\operatorname{LOG}(f^{\circ n})=n\operatorname{LOG}(f)$

First, let's note some basic algebraic facts. The formal power series form a commutative ring with operations and $\cdot$ (ordinary multiplication) and with additive identity $0$ and multiplicative identity $1$. The formal power series form a ring-like algebraic structure with operation and partial operation $\circ$ with additive identity $0$ and multiplicative identity $z$. But it's not actually ring or even a near-ring. Composition isn't defined for all formal power series and even when it's defined, we don't have distributivity. For example, in general $f\circ(g h)\ne f\circ g f\circ h$, after all there's no reason to expect $f(g(x) h(x))$ to equal $f(g(x)) f(h(x))$. We do have right-distributivity however, i.e.

$(f g)\circ h = f\circ g f\circ h$,

because

$(f g)(h(x))=f(h(x)) g(h(x))$,

more or less by definition of .

We can't use power series on our power series

There's an obvious approach, just use power series of power series. So we might tentatively suggest that

$\operatorname{LOG}(z f) = f-\frac{1}{2}f^{\circ 2} \frac{1}{3}f^{\circ 3} \ldots$.

Note that I consider $\operatorname{LOG}(z f)$ rather than $\operatorname{LOG}(1 f)$ because $z$ is the multiplicative identity in our ring-like structure.

Unfortunately this doesn't work. The reason is this: if we try to use standard reasoning to show that the resulting function has the fundamental property we seek we end up using distributivity. We don't have distributivity.

Sleight of hand

There's a beautiful trick I spotted on mathoverflow recently that allows us to bring back distributivity. (I can't find the trick again, but when I do I'll come back and add a link and credit here.) Consider the function $R(g)$ defined by $R(g)(f) = f\circ g$. In other words $R(g)$ is right-composition by $g$. (Ambiguity alert, I'm using $R$ here to mean right. It has nothing to do with the ring underlying our formal power series.) Because we have right-distributivity, $R(g)$ is a bona fide linear operator on the space of formal power series. If you think of formal power series as being infinitely long vectors of coefficients then $R(g)$ can be thought of as an infinitely sized matrix. This means that as long as we have convergence, we can get away with using power series to compute $\log R(g)$ with the property that $\log(R(g)^n) = n\log R(g)$. Define:

$\operator{LOG}(f) = \log(R(f))z$.

We have:

$\operator{LOG}(f) = \log(R(f))z = \log(1 (R(f)-1))z$

where I'm using $1$ to mean the identity linear operator. And now have:

$\operator{LOG}(f) = (R(f)-1)z-\frac{1}{2}(R(f)-1)^2z \frac{1}{3}(R(f)-1)^3z \ldots$.

But does it converge? Suppose $f$ is of the form $x a_2x^2 a_3x^3 \ldots$. Then $(R(f)-1)g = g\circ f-g$. The leading term in $g\circ f$ is the same as the leading term in $g$. So $R(f)-1$ kills the first term of whatever it is applied to, which means that when we sum the terms in $\operatorname{LOG}(f)$, we only need $n$ to get a power series correct to $n$ coefficients. Reusing my code from here, I call $\operatorname{LOG}$ by the name flog. Here is its implementation:

> import Data.Ratio


> flog :: (Eq a, Fractional a) => [a] -> [a]
> flog f@(0 : 1 : _) =
>   flog' 1 (repeat 0) (0 : 1 : repeat 0)
>      where flog' n total term = take (n+1) total ++ (
>              drop (n+1) $
>                 let pz = p term
>                 in flog' (n+1) (total-map (((-1)^n / fromIntegral n) *) pz) pz)
>            p total = (total ○ f) - total

The take and drop are how I tell Haskell when the first $n 1$ coefficients have been exactly computed and so no more terms are necessary.

Does it work?

Here's an example using the twice iterated sin function:

> ex1 = do
>   let lhs = flog (sin (sin z))
>   let rhs = 2*flog (sin z)
>   mapM_ print $ take 20 (lhs-rhs)

Works to 20 coefficients. Dare we try an inverse function?

> ex2 = do
>   let lhs = flog (sin z)
>   let rhs = flog (asin z)
>   mapM_ print $ take 20 (lhs+rhs)

Seems to work!

Exponentials

It's no good having logarithms if we can't invert them. One way to think about the exponential function is that

$\exp(x) = \lim_{n\rightarrow \infty}(1 \frac{x}{n})^n$

We get better and better approximations by writing the expression inside the limit as a product of more and more terms. We can derive the usual power series for $\exp$ from this, but only if right-distributivity holds. So let's try to use the above expression directly:

$\operatorname{EXP}(f) = \lim_{n\rightarrow \infty}(z \frac{f}{n})^{\circ n}$

and get

$\operatorname{EXP}(f) = \lim_{n\rightarrow \infty}R(z \frac{f}{n})^nz$.

Unfortunately, even though $R(g)$ is linear, $R$ itself isn't. So it's going to take some extra work to raise $R(z f/n)$ to the power of $n$.

The good news is that we're dealing with the special case $R(z \epsilon)$ where $\epsilon$ is something small. We have

$R(z \epsilon)f=f(z \epsilon)=f(z) \epsilon\frac{df}{dz} O(\epsilon^2)$.

So $R(z f/n)$ is actually $1 \frac{1}{n}f\frac{d}{dz}$ modulo higher order terms. This gives us

$\operatorname{EXP}(f) = \lim_{n\rightarrow \infty}(1 \frac{1}{n}f\frac{d}{dz})^nz=\exp(f\frac{d}{dz})z$.

This is something we can implement using the power series for ordinary $\exp$:

$\operatorname{EXP}(f) = z f \frac{1}{2!}f\frac{df}{dz} \frac{1}{3!}f\frac{d}{dz}(f\frac{df}{dz}) \ldots$.

In code that becomes:

> fexp f@(0 : 0 : _) = fexp' f 0 z 1
> fexp' f total term n = take (n-1) total ++ drop (n-1)
>           (fexp' f (total+term) (map (/fromIntegral n) (f*d term)) (n+1))

Note how when we differentiate a power series we shift the coefficients down by one place. To counter the effect of that so as to ensure convergence we need $f$ to look like $a_2z^2 a_3a^3 \ldots$. Luckily this is exactly the kind of series $\operatorname{LOG}$ gives us.

But does it successfully invert $\operatorname{LOG}$? Let's try:

> ex3 = do
>   let lhs = sin z
>   let rhs = fexp (flog (sin z))
>   mapM_ print $ take 20 (lhs-rhs)

Now we can start computing fractional iterates. Square root first:

> ex4 = do
>   mapM_ print $ take 20 $ fexp (flog (sin z)/2)

That matches the results at A048602 and A048603.

Cube root:

> ex5 = do
>   mapM_ print $ take 20 $ fexp (flog (sin z)/3)

Matches A052132 and A052135.

And this gives an alternative to Lagrange inversion for computing power series for inverse functions:

> ex6 = do
>   let lhs = fexp (-flog (sin z))
>   let rhs = asin z
>   mapM_ print $ take 20 (lhs-rhs)

What's really going on with $\operatorname{EXP}$?

Let's approach $\operatorname{EXP}$ in a slightly different way. In effect, $\operatorname{EXP}$ is the composition of $n$ lots of $z \frac{f}{n}$ with $z$. So let's try composing these one at a time, with one composition every $\frac{1}{n}$ seconds. After one second we should have our final result. We can write this as:

$g(0) = z$ and $g(t \frac{1}{n}) = g(t) \frac{1}{n}f(g(t))$ to first order.

So we're solving the differential equation:

$g(0) = z$ and $\frac{dg}{dt} = f(g(t))$

with $\operatorname{EXP}(g) = g(1)$.

So $\operatorname{EXP}$ is the function that solves one of the most fundamental differential equations. This also means I can use Mathematica to solve symbolically and check my results. For example, Mathematica says that the solution to

$\frac{dg}{dt}=sin(g(t))^2$ and $g(0)=x$

at $t=1$ is

$g(1) = \frac{\tan z}{1-\tan z}$

so let's check:

> ex7 = do
>   let lhs = fexp ((sin z)^2)
>   let rhs = atan (tan z/(1-tan z))
>   mapM_ print $ take 20 (lhs-rhs)

I like this example because it leads to the generalized Catalan numbers A004148:

> ex8 = do
>     mapM_ print $ take 20 $ fexp (z^2/(1-z^2))

That suggests this question: what does $\operatorname{EXP}$ mean combinatorially? I don't have a straightforward answer but solving this class of differential equation motivated the original introduction, by Cayley, of the abstract notion of a tree. See here.

What is going on geometrically?

For those who know some differential geometry, The differential equation

$g(0) = z$ and $\frac{dg}{dt} = f(g(t))$

describes a flow on the real line (or complex plane). You can think of $f$ as being a one-dimensional vector field describing how points move from time $t$ to $t dt$. When we solve the differential equation we get integral curves that these points follow and $\operatorname{EXP}$ tells us where the points end up after one unit of time. So $\operatorname{EXP}$ is the exponential map. In fact, $\operatorname{EXP}(f)=\exp(f\frac{d}{dz})z$ is essentially the exponential of the vector field $f\frac{d}{dz}$ where we're now using the differential geometer's notion of a vector field as a differential operator.

Final word

Unfortunately the power series you get from using $\operator{LOG}$ and $\operator{EXP}$ don't always have good convergence properties. For example, I'm not sure but I think the series for $\sin^{\circ 1/2} z$ has radius of convergence zero. If you truncate the series you get a half-decent approximaion to a square root in the vicinity of the origin, but the approximation gets worse, not better, if you use more terms.

And the rest of the code

> (*!) _ 0 = 0
> (*!) a b = a*b
> (!*) 0 _ = 0
> (!*) a b = a*b
> (^+) a b = zipWith (+) a b
> (^-) a b = zipWith (-) a b


> ~(a:as) ⊗ (b:bs) = (a *! b):
>     ((map (a !*) bs) ^+ (as ⊗ (b:bs)))
> (○) (f:fs) (0:gs) = f:(gs ⊗ (fs ○ (0:gs)))
> inverse (0:f:fs) = x where x     = map (recip f *) (0:1:g)
>                            _:_:g    = map negate ((0:0:fs) ○ x)
> invert x = r where r = map (/x0)  ((1:repeat 0) ^- (r ⊗ (0:xs)))
>                    x0:xs = x 


> (^/) (0:a) (0:b) = a ^/ b
> (^/) a b = a ⊗ (invert b)


> z :: [Rational]
> z = 0:1:repeat 0


> d (_:x) = zipWith (*) (map fromInteger [1..]) x


> integrate x = 0 : zipWith (/) x (map fromInteger [1..])


> instance (Eq r, Num r) => Num [r] where
>     x+y  = zipWith (+) x y
>     x-y  = zipWith (-) x y
>     ~x*y = x ⊗ y
>     fromInteger x      = fromInteger x:repeat 0
>     negate x     = map negate x
>     signum (x:_) = signum x : repeat 0
>     abs (x:xs)   = error "Can't form abs of a power series"


> instance (Eq r, Fractional r) => Fractional [r] where
>     x/y = x ^/ y
>     fromRational x    = fromRational x:repeat 0


> sqrt' x = 1 : rs where rs = map (/2) (xs ^- (rs ⊗ (0:rs)))
>                        _ : xs = x
> instance (Eq r, Fractional r) => Floating [r] where
>     sqrt (1 : x) = sqrt' (1 : x)
>     sqrt _  = error "Can only find sqrt when leading term is 1"
>     exp x   = e where e = 1+integrate (e * d x)
>     log x   = integrate (d x/x)
>     sin x   = integrate ((cos x)*(d x))
>     cos x   = [1] ... negate (integrate ((sin x)*(d x)))
>     asin x  = integrate (d x/sqrt(1-x*x))
>     atan x  = integrate (d x/(1+x*x))
>     acos x  = error "Unable to form power series for acos"
>     sinh x  = integrate ((cosh x)*(d x))
>     cosh x  = [1] ... integrate ((sinh x)*(d x))
>     asinh x = integrate (d x/sqrt(1+x*x))
>     atanh x = integrate (d x/(1-x*x))
>     acosh x = error "Unable to form power series for acosh"
>     pi      = error "There is no formal power series for pi"


> lead [] x = x
> lead (a:as) x = a : (lead as (tail x))
> a ... x = lead a x


> (//) :: Fractional a => [a] -> (Integer -> Bool) -> [a]
> (//) a c = zipWith (\a-> \b->(if (c a :: Bool) then b else 0)) [(0::Integer)..] a

A direct functional square root that doesn't use $\operatorname{LOG}$ and $\operatorname{EXP}$:

> fsqrt (0 : 1 : fs) =
>     let gs = (fs-(0 : gs*((0 : delta gs gs)+((2 : gs)*(gs*g)))))/2
>         g = 0 : 1 : gs
>         delta (g : gs) h = let g' = delta gs h
>                    in (0 : ((1 : h) * g')) + gs
>     in g

Tweag I/O: ð�•¯ð�–”ð�–ˆ ð�–Žð�–™ ð�–‘ð�–Žð�–�ð�–Š ð�–Žð�–™'ð�–˜ ð�–�ð�–”ð�–™

Thu, 16 Apr 2026 00:00:00 +0000

You’ve got some nice code. That’s a nice trie.
I see those PRs. But no README?

Everybody knows documentation is essential to any software engineering enterprise. And fo’ shizzle, everybody knows it gets deprioritised. An afterthought; written by engineers who are thinking, “I should probably write docs”. Nah. What they should be thinking is, “I get to write docs, cuz!” Because when you doc it right, it ain’t a chore. It’s what separates a project people use from a project people lose.

In this post, I’m walkin’ you through three real projects I’ve been involved in at Tweag; each one levelling up the documentation game. First, fixing docs that got out of hand: the reactive play. Then, planning docs from day one: the proactive play. And finally, making docs part of the code itself: the integrated play. By the end, I think you’ll agree: to doc it like it’s hot is the only way to gizzo.

When your `README`’s a monolith

Doc it like it’s hot

Sometimes the docs are already a mess and you gotta clean house. That’s the reactive play.

That was the case with Topiary, Tweag’s universal formatting engine. It uses Tree-sitter grammars and queries to format code; encoded in what Tree-sitter calls “capture names”. All of our formatting capture names needed to be documented, with their semantics described.

Moreover, our documentation covered usage instructions, which were checked against the --help output of each subcommand. Then there was our project motivation and design philosophy, language support, installation instructions, configuration details, usage guides…

…All in a single README.md which had grown to over 7,000 words. Way too big for the crib, homes. Ain’t nobody reading all that!

Drift and inconsistency were creeping in, making it harder for the team to maintain. Worse, it was straight-up hostile to users: how you gonna expect someone to sift through all that noise just to find what they need?

Topiary OG Erin started work reconstructing the monolith into a book format using — as it’s a Rust project — mdBook. I picked up where he left off and finished it for Topiary v0.6.1. Yo, the Topiary Book was born!

But the move here wasn’t just splitting up the README.md and calling it a day. You need a crew and every member needs a role. That means a framework; something to keep it tight, maintainable and user-friendly. We rolled with Diátaxis, which identifies four distinct documentation types based on what the reader actually needs:

Tutorials, for learners. For example, Yann’s step-by-step guides walk readers through creating a formatter from scratch for a toy language, starting from zero. Its aim is to actively reach understanding through engagement, rather than just passive reading.
How-to guides, for readers who want to accomplish a specific goal. “Adding a new language” assumes you already know Topiary and gets straight to the point: register the grammar, create a query file, update the test suite, rinse and repeat.
Explanation, for those who need a deeper understanding. For example, “Tree-sitter and its queries” explains the conceptual foundation — what Tree-sitter is, why Topiary uses it and how queries relate to formatting — without asking the reader to do anything.
Reference, which describes what exists and how it behaves. Our capture names chapter documents every formatting directive Topiary recognises, what it does, its syntax and its edge cases. You’re not meant to read it cover-to-cover; it’s just there to look up whenever you need it.

Structure is a prerequisite for usefulness and frameworks exist so you don’t have to invent your own. Rolling your own ain’t gangsta, use what’s already out there…that’s real game.

When you have varied audiences

Doc it like it’s hot

Cleaning up after the fact is one thing, but why not come correct from the start? That’s the proactive play: before you write a line of documentation, you ask who’s gonna read it and what they need. Then you build the structure around that.

So, while Topiary is a developer productivity tool with, broadly, a developer audience, the second project I want to chop it up about is different: an omics data acquisition tool for a pharmaceutical client’s computational biology needs.

This one had a whole different crowd to please. Bioinformaticians running data processing pipelines, IT staff handling installation and access control, administrators guiding users through workflows and developers who might extend the project down the line…including, potentially, yours truly, after returning from a long absence and having forgotten how everything works!

Their needs, vocabulary and assumptions barely overlap, so a single set of docs couldn’t serve them all without becoming an unfocused sprawl. So I split the documentation three ways:

A technical manual, covering every subcommand, flag and configuration key. The kind of thing a user reaches for mid-pipeline, an administrator references when guiding colleagues, or IT consults when setting up the environment.
A developer manual, as its mirror image: module architecture, type hierarchies, testing methodology and contribution workflow. All you need to dig the codebase, but were too shook to ask!
A user manual sat between the two, covering key concepts, how-to guides and troubleshooting. Diátaxis was again the guiding framework here: the concepts section is explanation, the how-to guides are exactly that and the troubleshooting page addresses the practical edge cases that tripped people up during user acceptance testing.

Within the user manual, I also got to indulge in what you’ve probably gathered is my favourite move. I weaved in a narrative through the examples that borrowed — with some artistic licence — from Stevenson’s Strange Case of Dr. Jekyll and Mr. Hyde: An external collaborator’s data arrives from Dr. Jekyll’s lab, making oblique references to the novella throughout, and ultimately identifying the evil-transcriptome for downstream analysis.

Does this make the documentation sillier than it needs to be? Maybe. But it makes the examples stick…and that ain’t just a vibe, it’s science, dawg: our brains are straight-up wired to retain information delivered through narrative far better than through isolated facts. A reader who skimmed the manual a month ago can still go, “that’s the example where forward and reverse reads are named ‘front door’ and ‘back door’” and find the section again. The story gives continuity across otherwise disconnected examples; where each section could stand alone, the recurring characters give readers a reason to follow the arc from data acquisition to analysis. And the scenario is deliberately awkward, which exercises more features than a vanilla example ever could.

Different people need different things, that’s just how it is. A bioinformatician never needs to know how the S3 client interface is structured, just as a future developer doesn’t need a walkthrough of dataset creation from NCBI metadata. When your audiences are distinct enough, the realest thing you can do is acknowledge that up front, rather than forcing everyone to wade through what ain’t for them…and there’s never any harm in bringing a little levity into the world!

When you need to have clarity

Doc it like it’s hot

Planning ahead is smooth, but the smoothest move of all? Making the docs and the code one and the same. That’s the integrated play.

The previous case study included a developer manual, but my final example? That’s all developer; front to back. Scrawls is a Rust library implementing a verifiable file format for Cardano ledger state, as an independent implementation alongside a Haskell reference. Its users are Rust developers pulling in the crate as a dependency, so the documentation strategy needed to reflect that.

In Rust, the idiomatic answer to this is rustdoc: in-band documentation that lives alongside the types, functions and invariants it describes. Then, while there is still a README.md, it functions more as a landing page than a manual: a brief orientation, a feature summary and a handful of examples to get a new user from zero to something.

Docstrings ain’t new, of course — Doxygen has been around since the ’90s — but Rust’s ecosystem raises the bar. Between docs.rs publishing your crate’s documentation automatically and a community that straight-up expects thorough doc comments, skipping them feels less like a shortcut and more like showing up empty-handed. So, when the API changes, the respective documentation change should be right there in the diff, for reviewers to keep it real.

And here’s the thing: during implementation, the specification was still maturing and encoding its requirements into Rust exposed ambiguities that were lurking in the prose. Should certain orderings be strict? How should the Merkle tree be rolled up? Is this field optional, or merely absent? Each ambiguity became a clarification fed back into the spec. And each clarification, a documented precondition in the API.

A specification is, at the end of the day, documentation too…and the same principle applies: vagueness is a bug. This ain’t documentation as prose; it’s documentation as a contract, forged on the streets between spec and implementation. Sometimes the most valuable work you can do is keep it tight, not make it long.

Document your code, ma

That’s how you get ahizzead

Reactive. Proactive. Integrated. Three different plays for three different games…though ideally you won’t need the first! And what ties them all together is that none of the documentation I’ve described was written reluctantly. It wasn’t tacked on after the fact because someone asked, “Where are the docs?” It was thought about — its structure, its audience, its precision — as part of the work itself.

That’s the shift I want to put you on to. Documentation ain’t a tax you pay for writing code; it is part of writing code. And when you approach it that way — when you reach for a framework instead of a blank page; when you ask “who’s this for?” before you start typing; when you treat ambiguity as a bug — then you’re doc’ing it like it’s hot! The result is something you’re genuinely proud of, not something you hope nobody reads too carefully.

Now if documentation alongside code is good, then documentation before code — as a design tool; a sketch in prose before you commit to implementation — is the next level. While I ain’t taken that step myself, my Scrawls experience, where the spec and the code kept each other honest, showed me how close that workflow already is.

In practice, pure docs-first has the same problem as pure test-driven development: you can’t document what you don’t know yet. But that feedback loop — where the docs sharpen the code and the code sharpens the docs — that’s the real endgame right there. You might notice this sounds a bit like vibe-coding, and it is…in the same way that an architect’s blueprint is a bit like a napkin sketch. Same ‘hood, different zip codes, dawg. Something to aspire to, fo’ shizzle.

So now, just like my man S-to the N-to the double O-P, you too can say…

I got a Rollie on my arm and I’m pourin’ Chandon
And I write the best docs, ‘cause I got it goin’ on.

With thanks to Simeon Carstens, Facundo Domínguez, Valentin Gagarin, Xavier Góngora, Arnaud Spiwack, Snoop Dogg and Pharrell Williams for their reviews and input on this post.

Haskell Interlude: 80: POPL 2026 - Part 1

Mon, 13 Apr 2026 05:00:00 +0000

This is the first part of a miniseries on this year’s Symposium on Principles of Programming Languages, a.k.a. POPL 2026, hosted by Jessica Foster.

In this episode, we talk about: undergrad funding and participation, the behind the scenes of AV, choreographic programming, quantum languages, conference catering, and the joy of theory. And at one point, you’ll even hear us get kicked out the venue mid interview. Enjoy!

Philip Wadler: Wet Sidewalks and Odd Numbers

noreply@blogger.com (Philip Wadler) — Thu, 09 Apr 2026 15:22:05 +0000

Phil Crissman explains Propositions as Types with a dialogue between Achilles and the Tortoise, in the style of Douglas Hofstadter (who in turn was inspired by Lewis Carrol). Lambda Man makes an appearance.

Russell O'Connor: Nash Equilibrium for Terminal Maneuvers

Thu, 02 Apr 2026 13:52:16 +0000

Last year Ethan Heilman wrote about a simple game he calls Terminal Maneuvers. This game simulates a missile attacking an interstellar ship. The ship has a laser defence system. One player controls the missile, and the other player controls the laser. If the missile hits the ship, Missile wins. If the laser hits the missile, the missile is destroyed and Laser wins.

The complicating factor is that, due to the relative motion of the laser and the ship being a significant fraction of the speed of light, Laser has to aim not at the missile but where the missile will be. This distance allows the missile to perform erratic manoeuvers to prevent Laser from knowing what its future position will be. However, Missile must expend fuel to perform these manoeuvers.

The Terminal Maneuvers game proceeds in five rounds, giving the laser five opportunities to hit the missile. In each round, Missile secretly commits to an amount of fuel they will expend. Laser must “aim” by guessing the amount of fuel expended by Missile. If they guess correctly, there is some probability of destroying the missile, which depends on how far away the missile is and how much fuel the missile expended. The table below shows the probabilities of the missile being destroyed in the various rounds.

Probability of Laser destroying the missile when correctly guessing Missile’s fuel expenditure
Fuel Cost	Round 1	Round 2	Round 3	Round 4	Round 5
0 Fuel	100%	100%	100%	100%	100%
1 Fuel	1/6	2/6	3/6	4/6	5/6
2 Fuel	0%	1/6	2/6	3/6	4/6
3 Fuel		0%	1/6	2/6	3/6
4 Fuel			0%	1/6	2/6
5 Fuel				0%	1/6
6 Fuel					0%

The missile has a limited amount of fuel at the start of the game. Fuel spent earlier in the game means less fuel available later in the game when it is most needed. The amount of starting fuel selects the difficulty of the game. Ethan suggests starting with seven fuel, which empirically gives Missile about a 25% chance of winning.

Laser knows how much fuel the missile has at the start of each round, so it is imperative that Missile does not run out of fuel in the middle of the game. If Laser knows the missile is out of fuel, Laser will predict zero fuel used and will always successfully destroy the missile. That said, as long as Missile has some fuel, choosing to burn zero fuel is still a legitimate option.

Starting with seven fuel, one strategy for Missile would be to burn one fuel on the first four rounds and burn the remaining three fuel on the last round. However, if Laser realizes this is Missile’s strategy, Laser can always predict the correct amount of fuel that will be used by the missile. Taking the product of all the probabilities of Missile’s survival in each round, Missile only has a 4.6% chance of winning. Clearly, Missile’s optimal strategy should be non-deterministic.

I figured this game would be a fun exercise in learning about mixed-strategy (i.e., non-deterministic) Nash equilibrium. This game is a small finite game, so it is reasonably easy to analyze, but it is significantly more complicated than trivial games often used in Nash equilibrium examples.

For these calculations, it is best to find the Nash equilibrium strategy at the endgame and work backward from there. To that end, let us start with the simplest non-trivial endgame. Missile has survived to Round 5 and has 1 fuel left. Missile can choose to burn their last fuel or not, and Laser can choose to aim at no fuel burned or not. This yields the following, game-theoretic payoff matrix, listing the probabilities of missile or laser winning:

Payoff matrix for Round 5 with 1 fuel remaining
	Predict 0	Predict 1
Burn 0	0, 1	1, 0
Burn 1	1, 0	1⁄6, 5⁄6

This is a constant-sum game, because the total score of all players is always the same, no matter the outcome. Constant-sum games are also known as zero-sum games since they can be translated into games where the sum of each outcome is zero without affecting any strategy.

The definition of Nash equilibrium is a pair of strategies, one for each player, where neither player individually can change strategies to improve their outcome. Therefore, one potential way for Missile to devise a strategy is to find one where Laser’s chance of winning is the same regardless of the move that they make. Such a strategy is not necessarily going to be possible, but we can give it a try.

Let p be the probability that Missile will burn 0, and let q be the probability that Missile will burn 1. If Laser predicts 0, the probability of them winning is p. If Laser predicts 1, the probability of them winning is 5⁄6 ⁢q. If Laser cannot make a choice between these two options to improve their odds, then p = 5⁄6 ⁢q. Missile’s probabilities must add up to 1, so we also require p + q = 1.

We have a linear system of two equations and two unknowns, so we can try to solve it. The solution is p = 5⁄11 and q = 6⁄11. Missile burns no fuel with probability 5⁄11 and burns its one fuel with probability 6⁄11. This provides Missile a 6⁄11 chance of winning, regardless of which prediction Laser makes.

On the flip side, Laser’s Nash equilibrium can be computed by choosing a set of probabilities so that Missile’s outcome is the same regardless of whether they choose to burn fuel or not. This time, let p be the probability that Laser will predict 0, and let q be the probability that Laser will predict 1. If Missile burns 0, the probability of them winning is q. If Missile burns 1, the probability of them winning is p + 1⁄6 ⁢q. Again, if Missile cannot make a choice between these two options to improve their odds, then q = p + 1⁄6 ⁢q. Laser’s probabilities also must add up to 1, so we also require p + q = 1.

Rearranging q = p + 1⁄6 ⁢q, we get 5⁄6 ⁢q = p, which happens to be the exact same equation Missile had. Thus, their solutions are identical. Laser predicts no fuel burned with a probability of 5⁄11 and predicts one fuel burned with a probability of 6⁄11. This provides Laser a 5⁄11 chance of winning no matter whether Missile has chosen to burn their fuel or not. This chance is the complement to Missile’s 6⁄11 chance of winning, as it has to be.

Payoff matrix for Round 5 with 2 fuel remaining
	Predict 0	Predict 1	Predict 2
Burn 0	0, 1	1, 0	1, 0
Burn 1	1, 0	1⁄6, 5⁄6	1, 0
Burn 2	1, 0	1, 0	2⁄6, 4⁄6

If the game ends in Round 5 with Missile having 2 fuel left, we have the above payoff matrix. We can solve similar linear algebra problems on three variables to find strategies for each player so that the other player’s outcome is the same no matter which of their three choices they make. The solution has Missile burn 0 fuel with probability 10⁄37, burn 1 fuel with probability 12⁄37, and burn 2 fuel with probability 15⁄37, giving Missile a 27⁄37 chance of winning regardless of what Laser’s prediction is.

Laser makes the predictions with the same probability distribution, giving Laser a 10⁄37 chance of winning no matter how much fuel Missile chooses to burn. This sort of distribution is what we might expect: somewhat evenly distributed with a bias towards burning more fuel, which provides some evasion for Missile.

What I found surprising is how when Laser plays at their Nash equilibrium, they simply do not care how much fuel Missile has secretly chosen to burn. Their odds of winning are the same regardless of what Missile reveals. It is as if Laser is no longer playing against Missile at all. Missile’s choices no longer matter. This result is called the “indifference principle.” Later we will see that Missile’s choices sometimes can matter.

Missile feels the same when playing at their equilibrium. No matter what prediction Laser ultimately makes, Missile’s odds of winning have already been fixed by playing at the Nash equilibrium.

In theory, this is what playing poker at a Nash equilibrium should feel like. Based on the state of the board, you make a random selection of calls, folds, or raises according to some appropriate distribution, and your distribution has fixed your expected payout at that point, independent of the choices the other players are going to make. No need to stress over whether your bluff will be called or not.

Before moving on to analyzing Round 4, we can complete a chart of the probability of Missile winning when playing at their Nash equilibrium depending on how much fuel they have remaining.

Probability table for Round 5
Remaining Fuel	Missile Win Probability	Laser Win Probability
0	0%	100%
1	6⁄11 ≈ 54.5%	5⁄11 ≈ 45.5%
2	27⁄37 ≈ 73.0%	10⁄37 ≈ 27.0%
3	47⁄57 ≈ 82.5%	10⁄57 ≈ 17.5%
4	77⁄87 ≈ 88.5%	10⁄87 ≈ 11.5%
5	137⁄147 ≈ 93.2%	10⁄147 ≈ 6.8%
6+	100%	0%

If Missile starts Round 4 with 1 fuel remaining, they are in big trouble. They can only burn fuel in at most one of the two remaining rounds. Therefore, Laser can win by predicting 0 fuel burned in both Round 4 and Round 5. Laser is guaranteed to destroy the missile on one of those two rounds. Missile must start Round 4 with at least 2 fuel remaining if they are to have a chance of winning.

Since the game in each round depends only on the state of Missile’s remaining fuel and not on the specific choices of how that state came to be, we can simplify the analysis of Round 4’s payoff matrix by using each player’s probability of winning Round 5 as their scores in Round 4. Note that Missile does not have the option of burning all their fuel in Round 4, since starting Round 5 with 0 fuel is a guaranteed loss for them, and Laser knows it.

Payoff matrix for Round 4 with 2 fuel remaining
	Predict 0	Predict 1
Burn 0	0, 1	27⁄37, 10⁄37
Burn 1	6⁄11, 5⁄11	2⁄11, 9⁄11

To compute Missile’s strategy, we define p and q as before. This time Missile needs to solve the equations p + 5⁄11 ⁢q = 10⁄37 ⁢p + 9⁄11 ⁢q and p + q = 1. The solution has Missile burn 0 fuel with a probability of 148⁄445 and burn 1 fuel with a probability of 297⁄445, which is roughly a 1⁄3^rd–2⁄3^rd split. This provides Missile a chance of winning with a probability of 162⁄445, or about 36.4%.

Meanwhile, Laser needs to solve the equations 27⁄37 ⁢q = 6⁄11 ⁢p + 2⁄11 ⁢q and p + q = 1. The solution has Laser predict 0 fuel with probability 223⁄445 and predict 1 fuel with probability 222⁄445, which is nearly evenly split. This provides Laser a chance of winning of 283⁄445, or about 63.6%.

In Round 5, each player’s individual payoff matrix was symmetric, which led to Missile and Laser having identical strategies. In Round 4, the individual player’s payoff matrices are no longer symmetric, and Missile and Laser end up with different strategies. Laser picks a nearly 50–50 split because the differences of column scores, 5⁄11 − 1 vs. 9⁄11 − 10⁄37, are nearly equal in magnitude. Whereas Missile picks a 1⁄3^rd–2⁄3^rd split because the difference of row scores, 27⁄37 vs. 2⁄11 − 6⁄11, differs in magnitude by close to a factor of two.

We can proceed as before, using linear algebra to compute equilibrium strategies for Round 4 with various states of remaining fuel for the missile. However, we run into a problem when Missile has 4 fuel remaining.

Payoff matrix for Round 4 with 4 fuel remaining.
	Predict 0	Predict 1	Predict 2	Predict 3
Burn 0	0, 1	77⁄87, 10⁄87	77⁄87, 10⁄87	77⁄87, 10⁄87
Burn 1	47⁄57, 10⁄57	47⁄171, 124⁄171	47⁄57, 10⁄57	47⁄57, 10⁄57
Burn 2	27⁄37, 10⁄37	27⁄37, 10⁄37	27⁄74, 47⁄74	27⁄37, 10⁄37
Burn 3	6⁄11, 5⁄11	6⁄11, 5⁄11	6⁄11, 5⁄11	4⁄11, 7⁄11

Let us try to solve for Laser’s equilibrium strategy, the probability distribution where Missile’s outcome is the same no matter what move they make. We let p, q, r, and s be the probabilities of predicting 0 through 3 fuel burned, respectively. In addition to having p + q + r + s = 1, we require

77⁄87 ⁢(q + r + s),
47⁄57 ⁢(p + r + s) + 47⁄171 ⁢q,
27⁄37 ⁢(p + q + s) + 27⁄74 ⁢r, and
6⁄11 ⁢(p + q + r) + 4⁄11 ⁢s

all be equal to each other. Solving this system of equations gives us

p ≈ 34.4%,
q ≈ 44.3%,
r ≈ 40.8%, and
s ≈ −19.5%.

Apparently, predicting 3 fuel used is such a terrible move for Laser that our “optimal” solution wants us to predict it with a negative 19.5% probability! Unfortunately, Laser cannot actually select moves with negative probability. We have to add constrains to our acceptable solutions to ensure all probabilities are non-negative.

Adding linear constraints to our problem brings us into the realm of linear programming. Since we are entering this realm, we can take this opportunity to compute the minimax solution for each player. For Laser, the minimax solution is to compute a probability distribution that minimizes Missile’s score, i.e., their probability of winning, which we will denote by z, subject to the constraint that Missile will choose the move that maximizes their score for that distribution. This leads to the following system of linear constraints:

77⁄87 ⁢(q + r + s) ≤ z
47⁄57 ⁢(p + r + s) + 47⁄171 ⁢q ≤ z
27⁄37 ⁢(p + q + s) + 27⁄74 ⁢r ≤ z
6⁄11 ⁢(p + q + r) + 4⁄11 ⁢s ≤ z
0 ≤ p
0 ≤ q
0 ≤ r
0 ≤ s
p + q + r + s = 1

where we want to minimize z.

Using linear programming, we can optimize this system. The optimal solution is

p ≈ 30.5%,
q ≈ 38.1%,
r ≈ 31.4%,
s ≈ 0%, and
z ≈ 61.5%.

That is, Laser’s strategy is to predict 0 fuel burned 30.5% of the time, predict 1 fuel burned 38.1% of the time, predict 2 fuel burned 31.4% of the time, and never predict 3 fuel burned. This lets Missile win at most 61.5% of the time, or equivalently, it lets Laser win at least 38.5% of the time.

Is this minimax strategy really an optimal strategy? Let us look at Missile’s minimax strategy. For Missile, we need to optimize the following system of linear constraints:

p + 10⁄57 ⁢q + 10⁄37 ⁢r + 5⁄11 ⁢s ≤ z
10⁄87 ⁢p + 124⁄171 ⁢q + 10⁄37 ⁢r + 5⁄11 ⁢s ≤ z
10⁄87 ⁢p + 10⁄57 ⁢q + 47⁄74 ⁢r + 5⁄11 ⁢s ≤ z
10⁄87 ⁢p + 10⁄57 ⁢q + 10⁄37 ⁢r + 7⁄11 ⁢s ≤ z
0 ≤ p
0 ≤ q
0 ≤ r
0 ≤ s
p + q + r + s = 1

to minimize z.

The optimal solution is

p ≈ 19.9%,
q ≈ 32.0%,
r ≈ 48.2%,
s ≈ 0%, and
z ≈ 38.5%.

That is, Missile’s strategy is to burn 0 fuel 19.9% of the time, burn 1 fuel 32.0% of the time, burn 2 fuel 48.2% of the time, and never burn 3 fuel. This lets Laser win at most 38.5% of the time, or equivalently, it lets Missile win at least 61.5% of the time.

This pair of strategies is optimal because Laser wins at least 38.5% of the time by their strategy, and Missile wins at least 61.5% of the time by their strategy, which adds up to 100%. It turns out that for zero-sum games, the minimax, maximin, and Nash equilibrium strategy sets are all identical, and furthermore, these strategies form a convex set. Using a general-purpose Nash equilibrium solver will produce the same pair of optimal strategies.

Still, these strategies surprised me. Laser is not even aiming at Missile burning 3 fuel. Shouldn’t Missile avoid being hit by Laser entirely by choosing to burn 3 fuel? But Missile’s optimal strategy also says to avoid burning 3 units of fuel. Why?

Upon closer examination, we see that with Missile’s computed optimal strategy, they have a 61.5% chance of winning. If Missile were to burn 3 fuel, yes, they would avoid being hit by Laser in Round 4. However, they would begin Round 5 with only 1 remaining fuel. In that state they would only have a 54.5% chance of winning, worse odds than their optimal strategy that avoids burning 3 fuel.

Laser is not aiming at Missile burning 3 fuel because Laser would love for Missile to burn 3 fuel. Doing so would increase Laser’s odds of winning from 38.5% to 45.5%. We see that Laser’s strategy does not entirely rule out all consequences of Missile’s choices. It only makes it indifferent to Missile’s choices within the support of Missile’s optimal mixed set of moves. Technically an opponent’s choices can affect the outcome of the game; they can still make moves that benefit the other player.

Continuing with linear programming, we can fill out the table for the probability of winning for Round 4.

Probability table for Round 4
Remaining Fuel	Missile Win Probability	Laser Win Probability
1-	0%	100%
2	≈ 36.4%	≈ 63.6%
3	≈ 50.5%	≈ 49.5%
4	≈ 61.5%	≈ 38.5%
5	≈ 69.9%	≈ 30.1%
6	≈ 76.4%	≈ 23.6%
7	≈ 81.6%	≈ 18.4%

Continuing this way, we can work backwards and compute probability tables for all the rounds until we reach round 1.

Probability table for Round 1
Starting Fuel	Missile Win Probability	Laser Win Probability
7	1005005076075⁄3110959445024 ≈ 32.3%	2105954368949⁄3110959445024 ≈ 67.7%

In conclusion, we found that playing optimally, Missile has an approximately 32.3% chance of winning, which is a little higher than the 25% estimate given by Ethan. I leave it as an exercise to determine the most fair amount of starting for Missile to start with.

Tweag I/O: Accessing external resources reliably with Bazel

Thu, 02 Apr 2026 00:00:00 +0000

We say that a system is reliable if it continues to function correctly when events outside the system affect it. Many factors can impact the reliability of Bazel builds, especially dependencies on external services. In this post, we’ll focus on what can go wrong when your build needs resources you don’t control and what you can do to reduce the risk of build failures.

Depending on external resources

Some build actions triggered by Bazel might be accessing resources that are external to your organization. For Bazel builds, this typically applies to build rules (to build your first-party code) or repository rules (utilities and tools those rules might need). When Bazel starts a build, it emits data about network requests, and you need to make those external requests visible so that you know what external resources your builds depend on. You can access this information via the Build Event Protocol (BEP) which can be written to disk or, if you operate a remote cache service, your provider might have a BEP viewer. You can also use the --experimental_repository_resolved_file flag to produce resolved information about all Starlark repository rules that were executed.

Building a target that depends on a repository rule such as this:

http_archive(
    name = "yq_cli",
    build_file = "@//tools/yq:BUILD.bazel.gen",
    sha256 = "7583d471d9bfe88e32005e9d287952382df0469135f691e044443f610d707f4d",
    url = "https://github.com/mikefarah/yq/releases/download/v4.47.1/yq_linux_amd64.tar.gz",
)

would result in the following build event (the snippet below is copied from the BEP output):

...
children {
  fetch {
    url: "https://github.com/mikefarah/yq/releases/download/v4.47.1/yq_linux_amd64.tar.gz"
  }
}
...

To get an idea of what kinds of artifacts a Bazel build for a reasonably large project might fetch, let’s build a few open-source projects — Envoy, Redpanda, and datadog-agent. These are some of the domains from which at least one resource was fetched when building all targets from these projects:

bcr.bazel.build             cdn.azul.com                dl.google.com
dl.grafana.com              dl.min.io                   download.gnome.org
files.pythonhosted.org      gcr.io                      github.com
go.dev                      mirror.bazel.build          mirrors.kernel.org
raw.githubusercontent.com   pkgconfig.freedesktop.org   pypi.org
static.crates.io            static.rust-lang.org        s3.amazonaws.com
www.antlr.org               www.colm.net                www.lua.org
www.sqlite.org              www.tcpdump.org

While most of your external dependencies are going to be declared in build metadata files such as MODULE.bazel (or legacy WORKSPACE), some network requests are going to be made by build targets such as genrules (e.g., by calling curl) or toolchains (e.g., a pip call to the PyPI index). We’ll see a worked example of this later in the post.

Common problems

In general, it is advised to rely on MODULE.bazel or WORKSPACE mechanisms for accessing external dependencies instead of doing so via build or test actions. Bazel by design lacks support and features for downloads to take place within build actions, and when attempting to interact with external systems this way, you will be limited in how you can manage and account for those requests.

Therefore, when building, the complete list of accessed online resources — those that are accounted for by BEP and those that are not — might be much longer. After doing a full build, it might be helpful to audit the network requests made to discover what resources were fetched and a complete inventory of external hosts your build depends on.

Given these external dependencies, these are common problems that could happen to any of them:

Outages: no service provides 100% uptime guarantee and some providers, sadly, have incidents all too often.
Removed artifacts: an archive file might be deleted due to retention policy.
Rate limiting: many concurrent builds coming from the same cluster can accidentally trigger API or download rate limits, especially with public registries.
Checksum drift: content of an artifact at a given URL can change, intentionally or maliciously, causing checksum mismatches.

This post focuses on strategies to either remove these external dependencies from the critical path, or make failures graceful and recoverable.

Remedies

The remedies below are intentionally “stackable”: you can start with low-effort safeguards (e.g., checksums and retries) and progress toward stronger guarantees (e.g., mirrors and network blocking). If you’re skimming, you can pick one external host that concerns you (e.g., github.com or pypi.org) and follow the options that would let you depend on it more reliably.

Using checksums

External resources may not only vanish or become inaccessible, but also change in place. Any artifact you download (unless there’s a strong guarantee from a provider), might change its contents such as when a provider does in-place updates of their releases (or it could also be a malicious attempt to inject code). To prevent this issue, SHA-256 digests must be coupled with any artifact you download from the Internet. Even though when declaring dependencies on external resources such as with http_archive, providing sha256 attribute is optional, it is considered a security risk to omit specifying the SHA-256 for remote files to be fetched.

Using GitHub releases

As the majority of build rules and open-source tools used by projects built with Bazel are hosted on GitHub, there are some special concerns that are worth mentioning.

A public GitHub repository might be moved, deleted, or become private (this happened in 2025 with rules_mypy). If you do have to rely on external rulesets hosted on GitHub, make sure they are hosted under the bazel-contrib organization (or help get them migrated at some point) to avoid surprises.

Checksums of dynamically generated archives might change; this has caused Bazel outages before, in 2023. There was some confusion about whether the stability of archives is guaranteed or not. There might be some edge cases such as when a Git repository is renamed, and since Bazel builds rely on stability of archives (for reproducibility and caching among other reasons) it might be best to play it safe and only use releases instead of using source downloads.

Using retries

It is possible that some of your dependencies need to be obtained from an online resource that is known to be unstable. What’s worse, you may not even be able to cache it (or host yourself): for example, imagine needing to download a short-lived license file for a commercial product from the manufacturer’s server when starting a build. To make downloading this file (via a repository rule) more likely to succeed, consider using the --experimental_repository_downloader_retries flag to specify the maximum number of attempts to retry upon a download error.

Placing binaries under version control

This varies a lot between organizations and the programming languages concerned, but a common approach that is adopted by most organizations is to check in the source code that is used to build a binary, and not the binary itself.

Many engineers would be strongly opposed to checking in any binary, as Version Control Systems (VCS) are designed and optimised for managing the source code. However, it is known that some organizations choose to place binary libraries that are external dependencies of their first-party code under version control. This has been seen occasionally in Java projects where .jar libraries (that nowadays can be managed with Maven / Gradle) were checked in. Today, this, arguably, might make sense only for legacy projects, air-gapped or classified networks, and for vendored native libraries that are hard to rebuild.

Unless you are able to provide top-notch automation for keeping your third-party dependencies checked in under version control up-to-date, patched, and compliant with any licensing constraints, it might be best to rely on a private artifact cache for hosting third-party dependencies.

Internal repository manager

As your organization grows, you will likely need to invest in a tool that would allow you to organize your resources such as external tools and third-party code packages into repositories. There are lots of commercial solutions on the market such as JFrog Artifactory, Sonatype Nexus, AWS CodeArtifact, and GitLab package registry to name a few.

With a repository manager, once you discover a dependency on an external artifact, you would upload it manually in your internal binary repository and update your build metadata accordingly:

# MODULE.bazel
http_archive(
  name = "tool",
  ...
  urls = [
    "https://artifacts.company.com/artifactory/project/tools/tool-1.2.3.tar.gz",
    "https://www.project.org/source/1.2.3/tool-1.2.3.tar.gz",
  ]
)

URLs from the urls attribute are tried in order until one succeeds. It is recommended to specify the local binary repository artifact first, and if the hosted mirror happens to be down, your build would still succeed provided that, in this case, project.org is up and running.

Bazel downloader configuration

You could also let your binary repository manager be the only place where Bazel builds can fetch resources from if you don’t want to depend on external artifacts in any way at all. This can be achieved by providing a configuration file for the remote downloader using the --downloader_config flag.

For example, a simple use case may be to block GitHub and instead rewrite fetches to go to an Artifactory instance. This can be done with the following downloader configuration:

rewrite github.com/([^/]+)/([^/]+)/releases/download/([^/]+)/(.*) artifacts.my-company.com/artifactory/github-releases-mirror/$1/$2/releases/download/$3/$4

# if you still have to rely on dynamically generated archives instead of releases
rewrite github.com/([^/]+)/([^/]+)/archive/(.+).(tar.gz|zip) artifacts.my-company.com/artifactory/github-releases-mirror/$1/$2/archive/$3.$4

However, support for using Bazel’s downloader needs to be enabled in Bazel rulesets by their authors. For instance, in rules_python, the pip extension now supports pulling information from a PyPI compatible mirror which means that the Bazel downloader can be used for downloading Python wheels.

Take a look at some downloader configurations used in other projects (e.g., 1, 2, 3) to explore how others set up access to external resources and learn the nuances of the configuration declaration syntax.

Blocking network requests

Additional control of network access can be achieved by blocking some network requests in CI agents using custom firewall rules or other tools of that nature. However, as mentioned earlier, Bazel’s downloader configuration can only rewrite or block requests that Bazel is aware of. This means that not all network traffic in a Bazel build is Bazel-managed traffic.

To illustrate this, let’s declare a dependency on the gawk binary. When running gawk, its sources are going to be fetched from the GNU FTP server. Let’s also add a genrule that will download an archive from the same FTP server:

# MODULE.bazel
bazel_dep(name = "gawk", version = "5.3.2")

# BUILD.bazel
genrule(
    name = "diffutils",
    outs = ["diffutils-3.12.tar.xz"],
    cmd = """wget -O "$@" https://ftp.gnu.org/gnu/diffutils/diffutils-3.12.tar.xz""",
)

We’ll configure Bazel to use a downloader configuration that blocks fetches from that FTP server:

# bazel_downloader.cfg
block ftp.gnu.org

# .bazelrc
common --downloader_config=bazel_downloader.cfg

When attempting to run the gawk binary from the ruleset, an error is expectedly raised since accessing the server is blocked:

$ bazel run @gawk
...
ERROR: java.io.IOException: Configured URL rewriter blocked all URLs:
[https://ftp.gnu.org/gnu/gawk/gawk-5.3.2.tar.xz]

However, building a genrule still succeeds because the downloader configuration does not apply here:

$ bazel build //src:diffutils
...
INFO: From Executing genrule //src:diffutils:
--2026-01-19 10:48:54--  https://ftp.gnu.org/gnu/diffutils/diffutils-3.12.tar.xz
Resolving ftp.gnu.org (ftp.gnu.org)... 209.51.188.20, 2001:470:142:3::b
Connecting to ftp.gnu.org (ftp.gnu.org)|209.51.188.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Saving to: 'bazel-out/k8-fastbuild/bin/src/diffutils-3.12.tar.xz'

External network requests of this nature are hard to audit in a large codebase since they won’t show up as structured fetch events in BEP output. To mitigate this, prefer using repository rules and Bzlmod extensions for any downloads instead of ad hoc shell commands. Going a step further, you might want to consider forbidding direct calls to applications that might make network requests (such as curl or wget) in genrule targets, unless explicitly approved. Where unavoidable, configure targets to access internal repositories instead of public endpoints.

Sandboxing

When triggering builds in a Bazel sandbox, they are run in a container (using Linux Namespaces) to isolate the build actions from the host. In addition to making your entire filesystem read-only (except for the sandbox directory), you can also forbid actions access the network. This is useful in some scenarios when you want to confirm that a build doesn’t make any network requests such as when running unit tests or integration tests that are not supposed to make any network calls. See Bazel tags requires-network and block-network to learn how to control network access for individual build targets.

Keep in mind that cached results of build actions can still be fetched even when blocking the network in a sandbox. So if artifacts needed for a build were uploaded to the Bazel cache previously, you won’t know whether a particular build needs any network resources unless you run the build without cache access. Also, none of the sandbox flags affect any cache as it’s expected that these flags should not affect the output of hermetic actions and making them part of a cache key would worsen the effectiveness of the cache.

With the network disabled in a sandbox, the genrule target we declared earlier fails to build:

$ bazel build //src:diffutils --spawn_strategy=linux-sandbox --nosandbox_default_allow_network
...
ERROR: Executing genrule //src:diffutils failed: (Exit 4): bash failed: ...
Resolving ftp.gnu.org (ftp.gnu.org)... failed: Temporary failure in name resolution.
wget: unable to resolve host address 'ftp.gnu.org'
Target //src:diffutils failed to build
...

Mirrors

Since Bazel 8.4, you can also use the --module_mirrors flag to mirror the source archives. To take advantage of this, add --module_mirrors=https://bcr.cloudflaremirrors.com in your .bazelrc file. Keep in mind that this only applies to registry sources and not to other resources fetched by Bazel (such as downloads happening in the repository rules context).

Note that for Bazel builds, the Bazel Central Registry (BCR) only stores metadata for a Bazel module; the actual artifacts are usually fetched from URLs that point to files hosted online (most often on GitHub).

BCR itself is a sort of external dependency for your builds, too. Even though it’s hosted on production-grade infrastructure at Google, it can still be impacted by outages and operational mishaps. The SSL certificate for mirror.bazel.build has expired, causing worldwide CI breakages, at least twice: once in 2022 and again in 2025. Refer to Postmortem for bazel.build SSL certificate expiry to learn more.

Configuring Bazel to use https://bcr.cloudflaremirrors.com as a mirror for modules from the BCR helps, but the Cloudflare mirror doesn’t cover the registry itself. So if you want to go the extra mile, you might also consider setting up your own BCR index registry and point Bazel at that instead. But if this is not feasible, write a playbook for incident response around build outages caused by external dependencies, so teams don’t have to improvise under pressure.

Pull-through cache

If your repository manager supports it, you could let your builds download external resources, but every resource that is being fetched is saved into the cache as well. On subsequent builds, the resources are going to be fetched from the cache, if available. This would let you turn random external downloads into a controlled internal dependency without requiring you to pre-vendor everything up front.

If your CI agents are in the same network or cloud region (depending on your infrastructure setup), this could also speed up the builds by having downloads complete faster. Not relying on external resources makes your Bazel builds also a lot more secure as your CI agents will only download data from a trusted source.

If using an off-the-shelf solution, such as the popular JFrog Artifactory, is not possible, there are some other options. Bazel picks up proxy addresses from the HTTP_PROXY and HTTPS_PROXY environment variables and uses these to download files over HTTP and HTTPS, respectively (if specified). This means you might have success with caching proxy solutions such as Squid and Charles or by combining Nginx and Varnish HTTP reverse proxies. Routing requests through a proxy might also help to avoid rate limiting issues since the external service will see fewer direct requests.

With this configuration, your downloader configuration file would look something like this:

# point all downloads at the mirror
rewrite (.*) {caching-service-url}/$1

# use the original location if the mirror is down
rewrite (.*) $1

For a completely custom solution, take a look at the Bazel downloader mirror from Monogon which can be used to mirror Bazel dependencies to a cloud bucket storage such as S3 or GCS. Bazel’s remote asset API lets you use an existing remote cache (content-addressable storage: CAS) as a downloader cache as well. The cache provider service needs to support it, but many existing solutions, both commercial and open-source ones, are compatible.

The --experimental_remote_downloader flag can be specified to provide a Remote Asset API endpoint URI to be used as a remote download proxy. To get started, consider using bazel-remote, which has out-of-the-box support for this use case. Make sure to provide the sha256 for the assets to fetch so that they can be cached just like any other CAS object. A remote caching service will automatically download the assets from the URL if they are found in the CAS and cache it thereafter.

Bazel 9 adds support for remote repository caches which make Bazel builds (at least those requiring previously cached assets) extra resilient to external access issues. During outages of external hosting services, those organizations that didn’t have a central repository manager where repository rules artifacts could be stored had to extract files from cache directories on local developer machines and save them to an accessible location within the internal network.

Now these artifacts will be saved into a remote cache similarly to build output results. To confirm that your remote repository cache works as expected, you can use the --repository_disable_download flag after doing a clean build (which should succeed as it will reuse the remote cache entries uploaded in the previous build).

Chaos testing

Finally, instead of waiting for the next GitHub outage, you can test your resilience by intentionally breaking access to certain external hosts. In a staging CI environment, temporarily block access to key external systems with firewall rules and verify that your mirrors and caches are used as expected, builds either still succeed, or fail fast with clear error messages, and your runbooks are correct and sufficient.

Conclusion

Bazel projects often depend on external services in subtle ways, and any instability or change in those services can break otherwise healthy builds. You can significantly improve build reliability by making all downloads explicit and verifiable, routing them through managed infrastructure, and tightening how and when network access is allowed. Resilient Bazel builds come from treating external dependencies as first‑class operational risks and turning unpredictable third‑party failures into controlled, recoverable events.

Well-Typed.Com: Second pre-release of hs-bindgen

Fri, 27 Mar 2026 00:00:00 +0000

With heartfelt thanks to the many people who have already tried hs-bindgen and given us feedback, we have steadily been working towards the first official release (see Contributors for the full list). In case you missed the announcement of the first alpha, hs-bindgen is a tool for automatic construction of Haskell bindings for C libraries: just point it at a C header and let it handle the rest. Because we have fixed some critical bugs in this alpha release, but we’re not quite ready yet for the first full official release, we have tagged a second alpha release. In the remainder of this blog post we will briefly highlight the most important changes; please refer to the CHANGELOG.md of hs-bindgen and of hs-bindgen-runtime for the full list of changes, as well as for migration hints where we have introduced some minor backwards incompatible changes.

Bugfixes

The most important fixes for bugs in the generated code are:

The implementation of peek and poke for bitfields was broken, which could lead to segfaults.
Duplicate record fields are now usable also in Template Haskell mode.
Patterns for unsigned enums now get the right value.

We have also resolved a number of panics during code generation, but those would not have resulted in incorrect generated code (merely in no code being generated at all).

New features

Implicit fields arise when one struct (or union) is nested in another, without any field name or tag:
```
struct outer {
  int x;
  struct {
    int y;
    int z;
  };
};
```
We now support such implicit fields; both the inner (anonymous) struct as well as the corresponding field of the outer struct will be named after the first field of the inner struct¹:
```
data Outer = Outer {
    x :: CInt
  , y :: Outer_y
  }

data Outer_y = Outer_y {
    y :: CInt
  , z :: CInt
  }
```
For this particular case we could also have chosen to flatten the structure and add y and z directly to Outer, but that does not work in all cases (for example, when we have an anonymous struct inside a union), so instead we opt for consistency and always generate an explicit type for the inner struct.

Unnamed bit-field declarations, which are used to control padding, are now supported:

struct bar {
  signed char x : 3;
  signed char   : 3;  // Explicit padding
  signed char y : 2;
};

We used to distinguish between parse predicates (which files should hs-bindgen parse at all?) and selection predicates (for which C declarations should we generate Haskell declarations?). This was confusing, and as we are getting better at skipping over declarations with unsupported features (and that list is dwinding anyway), parse predicates are not that useful anymore. Parse predicates therefore have been removed entirely; we simply always parse everything (selection predicates are still very much an important feature of course).
Some infrastructure for and around binding specifications has been improved. For example, we now distinguish between macros and non-macros of the same name, and our treatment of arrays has changed slightly. For example, given
```
typedef char T [];
void foo (T xs);
```
we now generate
```
foo :: Ptr (Elem T) -> IO ()
```
We do not use Ptr CChar, because T might have an existing binding in another library (with an external binding specification), and we don’t know what the type of the elements of T are (it could for example be some newtype around CChar). Elem is a member of a new IsArray class, part of the hs-bindgen-runtime.
Top-level anonymous enums are now supported. For example,
```
enum {A, B};
```
results in
```
pattern A :: CUInt
pattern A = 0

pattern B :: CUInt
pattern B = 1
```
(Normally an enum results in a newtype around the enum’s underlying type, and the patterns are for that newtype instead.)
We now generate bindings for static global variables (such globals are sometimes used in headers that also contain static function bodies).
All definitions required by the generated code are now (re-)exported from hs-bindgen-runtime, so that it becomes the only package dependency that needs to be declared (no need for ghc-prim or primitive anymore).

This list is not complete; some other less common edge cases have also been implemented.

Conclusions

Although we are still working on some finishing touches before we can release the first official version of hs-bindgen, it is already being put to good use on various projects. There are only a handful of missing C features left, all of which low priority edge cases (though if you have a specific use case for any of these, do let us know!). So if you are interested, please do try it out, and let us know if you find any problems. There should be no major breaking changes between now and the first official release.

This is the version that uses the --omit-field-prefixes option, which generates code that relies on DuplicateRecordFields and OverloadedRecordDot.↩︎

GHC Developer Blog: GHC 9.12.4 is now available

Fri, 27 Mar 2026 00:00:00 +0000

GHC 9.12.4 is now available

wz1000 - 2026-03-27

The GHC developers are very pleased to announce the release of GHC 9.12.4. Binary distributions, source distributions, and documentation are available at downloads.haskell.org and via GHCup.

GHC 9.12.4 is a bug-fix release fixing many issues of a variety of severities and scopes, including:

Fixed a critical code generation regression where sub-word division produced incorrect results (#26711, #26668), similar to the bug fixed in 9.12.2
Numerous fixes for register allocation bugs, preventing data corruption when spilling and reloading registers (#26411, #26526, #26537, #26542, #26550)
Fixes for several compiler crashes, including issues with CSE (#25468), and the simplifier(#26681), implicit parameters (#26451), and the type-class specialiser (#26682)
Fixed cast worker/wrapper incorrectly firing on INLINE functions (#26903)
Fixed LLVM backend miscompilation of bit manipulation operations (#20645, #26065, #26109)
Fixed associated type family and data family instance changes not triggering recompilation (#26183, #26705)
Fixed negative type literals causing the compiler to hang (#26861)
Improvements to determinism of compiler output (#26846, #26858)
Fixes for eventlog shutdown deadlocks (#26573) and lost wakeups in the RTS (#26324)
Fixed split sections support on Windows (#26696, #26494) and the LLVM backend (#26770)
Fixes for the bytecode compiler, PPC native code generator, and Wasm backend
The runtime linker now supports COMMON symbols (#6107)
Improved backtrace support: backtraces for error exceptions are now evaluated at throw time
NamedDefaults now correctly requires the class to be standard or have an in-scope default declaration, and handles poly-kinded classes (#25775, #25778, #25882)
… and many more

A full accounting of these fixes can be found in the release notes. As always, GHC’s release status, including planned future releases, can be found on the GHC Wiki status.

GHC development is sponsored by:

We would like to thank these sponsors and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release.

As always, do give this release a try and open a ticket if you see anything amiss.

Chris Smith 2: Athena Loses a Bet

Wed, 25 Mar 2026 18:58:04 +0000

Athena and Ares argue over human nature, and agree to test three great minds of the age.

First, they approach Aristotle in the Lyceum and propose a bargain. “If you ask it of us, the one you love most in the world will perish, but you will be made rich beyond imagining.” Aristotle barely hesitates. “No,” he says. “To destroy the very purpose of living for the sake of the mere means is the mark of a man who lacks wisdom.”

Next, they approach Plato, finding him pacing in an olive grove of his Academy. They offer the same proposal. “I decline,” he says. “Love allows us to glimpse the ideal of pure beauty, but wealth is an anchor to the material world.”

Finally, they approach Socrates, wandering barefoot in the crowded dusty stalls of the Agora. The gods approach him with the same bargain: “If you ask it of us, Xanthippe, whom you love most in the world, will perish — ”

“I ask it!” he blurts out.

Athena blinks. “You did not even hear the rest. We were going to say you would be given wealth beyond measure.”

Socrates shrugs. “Keep it. This was never about money.”

Millenia later, Athena is still smarting from losing the bet, and she demands a rematch. Searching for another Greek philosopher, they instead find a middle aged woman writing a novel called Atlas Shrugged. She’s a philosopher, and Atlas was Greek, so that’s close enough.

“If you ask it,” Athena says to her, “we will make you wealthy beyond measure, but then in return, your true love will be taken from you.”

The woman looks up, bored, and asks “Why give me the money if you’re just going to take it right back?”

Haskell Interlude: 79: Peter Thiemann

Sun, 22 Mar 2026 12:00:00 +0000

Peter is a professor at the University of Freiburg, and he was doing functional programming right when Haskell got started. So naturally we asked him about the early days of Haskell, and how from the start Peter pushed the envelope on what you could do with the type system and specifically with the type classes, from early web programming to program generation to session types. Come with us on a trip down memory lane!

Well-Typed.Com: Haskell ecosystem activities report: December 2025â€“February 2026

Thu, 19 Mar 2026 00:00:00 +0000

This is the thirtieth edition of our Haskell ecosystem activities report, which describes the work Well-Typed are doing on GHC, Cabal, HLS and other parts of the core Haskell toolchain. The current edition covers roughly the months of December 2025 to February 2026.

You can find the previous editions collected under the haskell-ecosystem-report tag.

Sponsorship

We offer Haskell Ecosystem Support Packages to provide commercial users with support from Well-Typed’s experts while investing in the Haskell community and its technical ecosystem including through the work described in this report. To find out more, read our announcement of these packages in partnership with the Haskell Foundation. We need funding to continue this essential maintenance work!

Many thanks to our Haskell Ecosystem Supporters: Standard Chartered, Channable and QBayLogic, as well as to our other clients who also contribute to making this work possible: Anduril, Juspay and Mercury; and to the HLS Open Collective for supporting HLS release management.

Team

Matthew Pickering announced that he will be leaving the company and moving to a non-Haskell role at the end of March. Working with Matt has been a joy – more than his deep technical insight or sharp intuition, it’s the warmth of his vision for how to work together and his generosity that has made him such a force within the team. He was also a beacon that could rally the community in difficult times, perhaps most memorably with his technical and social contributions in consolidating Haskell IDEs with the creation of the Haskell Language Server. His dedication to tooling has also been an inspiration, with his work on ghc-debug and on profiling an invaluable contribution to our understanding of memory usage of Haskell programs.

The Haskell toolchain team at Well-Typed currently includes:

In addition, many others within Well-Typed contribute to GHC, Cabal, HLS and other open source Haskell libraries and tools. This report includes contributions from Alex Washburn, Duncan Coutts, Wen Kokke and Wolfgang Jeltsch in particular.

We are active participants in community efforts for developing the Haskell language and libraries. Rodrigo joined the GHC Steering Committee in December, alongside Adam Gundry. Wolfgang joined the Core Libraries Committee in February.

Highlights

Interactive step-through debugging

The Haskell Debugger (hdb) has been made more robust and more features were implemented by Rodrigo, Matthew, and Hannes. Most notably, the debugger now:

Displays stack traces for bytecode and compiled code frames (provided the program and dependencies were compiled with -finfo-table-map for the latter)
Displays source locations and callstacks for exception breakpoints
Uses the external interpreter by default
Can be run on GHC itself!

To run hdb you need to use GHC 9.14 and to configure the IDE accordingly. Please refer to the installation instructions. Apart from that, if HLS just works on your codebase, so should the debugger!

Live monitoring using the eventlog

GHC’s eventlog already lets Haskell programs emit rich runtime telemetry, but the workflow has historically been to run the program to completion and inspect the eventlog afterwards. eventlog-live allows us instead to monitor the program as it is running. Wen continued work on this project, taking significant steps towards making it production-ready, including:

extending eventlog-live with support for the OpenTelemetry protocol (#119),
bringing the underlying eventlog-socket library closer to being ready for general use, by adding a testsuite (#27). fixing a litany of issues with the C code (#38), and finalising the user-facing API (#43),
adding support for custom commands in eventlog-socket (#36).

Trees That Grow

The Language.Haskell.Syntax module hierarchy is intended to be a stable, public API for the Haskell AST — one that external tools could eventually depend on without coupling themselves to GHC internals, reducing ecosystem breakage. Right now, that goal is undermined by lingering dependencies on internal modules under the GHC hierarchy.

Alex, with help from Rodrigo, has been systematically removing these edges in the dependency graph:

Language.Haskell.Syntax.Type no longer depends GHC.Utils.Panic (!15134, #26626).
Language.Haskell.Syntax.Decls no longer depends on GHC.Unit.Module.Warnings (!15146, #26636), nor on GHC.Types.ForeignCall (!15477, #26700) or GHC.Types.Basic (!15265, #26699).
Language.Haskell.Syntax.Binds no longer depends on GHC.Types.Basic (!15187, #26670).

Once this work is done, it will be possible to consider moving the AST into a separate package, and taking further steps towards increasing modularity of the compiler.

Towards a standalone `base` package

Historically, the base package was used as both the user-facing standard library and a repository of GHC-specific internals, with much special treatment in the compiler. This means GHC and base versions are tightly coupled, and makes upgrading to new compiler versions unnecessarily difficult.

GHC developers have made significant progress towards making base a normal Haskell package: ghc-internal has been split out as a separate library, base no longer has a privileged unit-id in the compiler, and Cabal now allows reinstalling it.

Matt posted a summary of progress and outlined possible next steps to seek community consensus on the direction of travel. The reinstallable-base repository collects documents and discussion on the effort.

Wolfgang continued various pieces of technical groundwork:

cleaning up many unused known-key names in the compiler (!15184, !15190, !15211, !15213, !15217, !15218, !15219, !15215),
finishing the process of removing GHC.Desugar from base (!15433),
refining the import list of System.IO.OS to aid in modularity (!15567).

Wolfgang improved the public API of base relating to OS handles, to make the API more stable across platforms and avoid the need for users to depend on GHC-internal implementation details (!14732, !14905). While in the area, he fixed a bug in the implementation of hIsReadable and hIsWritable for duplex handles (#26479, !15227), and a mistake in the documentation of hIsClosed (!15228).

Incorrect absence analysis in GHC

GHC bug #26416 has occupied the attention of the team for quite some time. Initially thought to be an issue with specialisation, a reproducer that Sam and Magnus created showed that the issue is in fact a bug in absence analysis — an optimisation that identifies and removes unused function arguments — in which GHC would erroneously conclude that a used argument was in fact absent.

Andreas helped investigate the root cause, before Zubin finally took the torch and put up a solution (!15238).

GHC changelogs

GHC’s changelogs have not always been as complete or reliable as the community deserves. Keeping changelogs accurate across backports has also been a major source of frustration for release managers.

This is why, after a discussion initiated by Teo Camarasu in #26002, we have decided to adopt the changelog.d system — already in use by the Cabal project — in which each change is a separate file in the changelog directory. This eliminates the merge conflicts that make backporting painful, and makes it easier to associate MRs with changelog entries.

Zubin has been spearheading the effort, with the intention to switch to this new method of changelog generation right after the fork date for GHC 10.0.

GHC

GHC Releases

Zubin worked on 9.12.3, backporting patches and preparing release candidates, with a final release on the 27th of December.
Magnus and Zubin worked on backports for 9.12.4.
Zubin worked on 9.14.1, putting out the final release on the 19th of December.

Frontend

Sam reviewed the implementation of the QualifiedStrings extension by Brandon Chinn (!14975). This allows string literals of the form ModName."foo" (interpreted as ModName.fromString ("foo" :: String)).
Sam made several changes to the treatment of Coercible constraints in the typechecker (!14100):
- Defaulting of representational equalities to nominal equalities, functionality previously added to GHC by Sam, is now more robust (#25825).
- Error messages involving unsolved Coercible constraints are greatly improved, an oft-requested improvement (#15850, #20289, #23731, #26137). Error messages now consistently mention relevant out-of-scope data constructors, provide import suggestions, and include additional explanations about roles (when relevant).
Magnus implemented several fixes to the implementation of ExplicitLevelImports:
- a missing check for types (#26098, !15119),
- a GHC panic in the driver (#26568, !15118).
Sam improved the reporting of “valid hole fits”, adding support for suggesting bidirectional pattern synonyms (#26339) and properly dealing with data constructors with linear arguments (#26338).
Sam investigated a typechecking regression starting in GHC 9.2 with the introduction of the Assert type family to improve error messages involving comparison of type-level literals (#26190), posting his analysis to the ticket. To tackle this, he opened GHC proposal #735, which is still in need of further community feedback.
Sam minimised a bug with rewrite rules (#26682), which allowed Simon Peyton Jones to identify and fix the bug (!15208).
Sam improved how existential variables are displayed in Haddock documentation (!15099, #26252).

Determinism

Matt identified and fixed several ways in which GHC compilation was not deterministic:
- an issue with non-deterministic documentation information (#26858, !15482).
- non-determinism of constraint solving impacting generated Typeable evidence (#26846, !15442).
- issues with the Template Haskell machinery of the singletons library producing non-deterministic names (singletons #629, th-desugar #240).

Plugins

Sam finished up and landed a long-standing MR by Chris Wendt (!10133) which fixed a plugin-related issue.
Sam fixed a regression in ghc-typelits-natnormalise in which the plugin would cause GHC to fall into an infinite loop (ghc-typelits-natnormalise #116, #118).

Backend

Rodrigo announced that work described in #23218 evolved into the POPL 2026 paper “Lazy Linearity for a Core Functional Language”, which presents a way to type linearity in GHC Core that is robust to almost all GHC optimisations, together with a GHC plugin validating programs at each optimisation stage.
With the oversight of Andreas, Sam carefully reconsidered the treatment of register formats in the register allocator and liveness analysis. This culminated in !15121:
- Keep track of register formats in liveness analysis (#26526).
- Use the right format when reloading spilled register (#26411).
- Enforce the invariant that writes to a register re-defined the format that this register is used at for the purposes of liveness analysis, fixing another bug reported by @aratamizuki on !15121.
Sam put up a small fix for the mapping of registers to stack slots, fixing an oversight in the case that registers start off small and are subsequently written at larger widths (#26668, !15185).
Sam reviewed a GHC contribution by @sgillespie adding SIMD primops for abs and sqrt operations (!15236), suggesting more efficient implementations of certain operations.
Andreas investigated potential missed specialisations, which allowed Simon Peyton Jones to make further progress in improving the specialiser (#26831, !15441).
Sam investigated several bugs to do with the interactions of join points with ticks (#14242, #26157, #26642, #26693) and casts (#14610, #21716, #26422). He fixed the main bug (#26642, !15538), which was due to incorrect transformations in mergeCaseAlts. He also undertook a general refactor of the area and, pinning down the overall handling of casts and ticks under join points in a Note.

Runtime system and linker

Matt fixed a decoding failure for stg_dummy_ret by using INFO_TABLE_CONSTR for its closure (#26745, !15303).
Duncan fixed long-standing inconsistencies in eventlog STOP_THREAD status codes (#26867, !15522).
Andreas improved the documentation of the -K RTS flag in !15365 (#26354).

Exception backtraces, stack annotations and stack decoding

Matt and Hannes improved the reporting of backtraces when using error (!15306, !15395, #26751). This involved opening two CLC proposals (CLC #383, CLC #387).
Hannes continued working on the implementation of stack annotations and stack decoding (#26218), including:
- integrating ghc-stack-profiler, a profiler that relies on stack annotations instead of heavier profiling mechanisms, with the eventlog-socket library; and
- working on the ghc-stack-annotations compatibility library for annotating the stack.
Rodrigo removed an incorrect assertion that fired when decoding a BCO whose bitmap has no payload (#26640, !15136).

Build system and packaging

Zubin fixed a GHC 9.14.1 build issue due to missing .cabal files for ghc-experimental and ghc-internal in the source tarball (#26738, !15391).
Andreas investigated the use of Cabal’s --semaphore feature to speed up GHC builds slightly (#26876, !15483). There are some issues preventing us from enabling this unconditionally (#26977, Cabal #11557).

CI and testing

Magnus ensured the user’s guide can be generated with old versions of Python to fix CI build failures on some older containers (!15127).
Magnus updated the Debian images used for CI (ci-images !183, !178).
Sam finished up the work of Sven Tennie on testing floating point expressions in the test-primops test framework for GHC (test-primops !19). This is preparatory work for improving the robustness of GHC’s handling of floating point (#26919).
Andreas updated the nofib GHC benchmarking suite to fix issues that Sam ran into when trying to use it, updating the CI in the process (nofib !81, !82, !83).

Infrastructure

Magnus worked on the infrastructure for the GitLab instance used for the GHC project, bringing up new runners for CI and switching to a new verification system to approve new users which makes it easier for new contributors to open issues.
Magnus and Andreas helped the Haskell infrastructure team address Gitlab outages on short notice in order to improve availability of the GHC Gitlab instance.
Andreas and Magnus organized temporary CI capabilities sponsored by WT during a temporary outage of one of GHC’s CI runners.

Cabal

Sam added support for setting the logging handle via the library interface of Cabal, a significant milestone in updating cabal-install to compile packages with the Cabal library without invoking external processes (Cabal #11077).
Matt helped Matthías Páll Gissurarson to fix a bug in which cabal haddock was looking for files in the wrong directory (Cabal #11475, #11476).
Matt fixed a bug with broken Haddocks locally due to non-expanded ${pkgroot} variable (Cabal #11217, #11218).
Matt fixed some issues with cabal repl silently failing (Cabal #11107, #11237).

HLS

In collaboration with Zubin and Andreas, Hannes investigated the root cause of HLS issue #4674, posting his analysis in this comment. In short, the problem was that the hlint plugin was using an incompatible version of ghc-lib-parser, and a version mismatch in this library was causing segfaults due to changes to the GHC.Data.FastString implementation between the versions. Hannes disabled the hlint plugin on GHC 9.10 to work around this issue (HLS PR #4767).
Hannes reviewed and assisted with HLS PR #4856 by @vidit-od. This PR makes HLS use the stored server-side diagnostics for code actions, in order to make them more responsive. This fixes HLS issue #4805.
Hannes helped land long-running HLS PR #4445 by @soulomoon, which allows files to be loaded concurrently in batches in order to improve responsiveness of HLS.
Zubin and Hannes worked together to update HLS to work with GHC 9.14 (HLS PR #4780).
Hannes worked on general maintenance of the HLS project:
- Prepared release 2.13.0.0 (HLS PR #4785)
- Tackled various CI issues (HLS PR #4863, HLS PR #4812, HLS PR #4811)
- Updated the advertised range of supported GHC versions (HLS PR #4801, HLS PR #4799)
Hannes and Zubin implemented some fixes to Windows CI (HLS PR #4800, HLS PR #4768).
Hannes merged the hls-module-name-plugin into hls-rename-plugin in HLS PR #4847.
Hannes improved the robustness of the hls-call-hierarchy-plugin-tests in HLS PR #4834 by using VirtualFileTree.
Hannes also worked on hie-bios:
- Preparing release 0.18.0.0 (hie-bios PR #496)
- Updated the supported GHC versions (hie-bios PR #495)
- Adapted to GHC migrating some parts of its codebase to use OsPath (hie-bios PR #493)

Haskell Debugger

Rodrigo continued work on the new Haskell Debugger.

Matt and Rodrigo introduced a DSL for evaluation on the remote process, which allows the debuggee to be queried from a custom instance, making it possible to implement visualisations which rely on e.g. evaluatedness of a term (#139).
Matt improved support for exceptions: break-on-exception breakpoints now provide source locations (#165).
Rodrigo allowed call stacks to be inspected in the debugger (#158).
Hannes introduced support for stack decoding and viewing custom stack annotations (#172).
Rodrigo made the Haskell Debugger use the external interpreter (#170), which paves the way for multi-threaded debugging (see also #140). This change also allowed Rodrigo to implement Windows support (#184) with the help of Hannes.
Matt fixed a bug in the handling of data constructors with constraints (#175).
Hannes improved caching in the CI (#173).

`ghc-debug`

Matt and Hannes fixed several issues with AP_STACK closures (!79, !80, !86).
Hannes implemented asynchronous heap traversal in ghc-debug-brick, making the interface more responsive (!78).
Hannes added history navigation and search caching to the ghc-debug-brick interface (!83).
Hannes added a summary row to the string counting table view (!81), and fixed the search limit not being honoured during incremental searches (!76).

Mark Jason Dominus: Did Ahmes find the best expansions for 2/n?

mjd@plover.com (Mark Dominus) — Tue, 17 Mar 2026 13:28:00 +0000

A couple of years back I was discussing the Rhind Mathematical Papyrus (RMP). It includes a table expressing as a sum $$\frac1{a_1}+\frac1{a_2}+\dots+\frac1{a_k} $$ fractions with numerator 1 (“unit fractions”). I said:

Getting the table of good-quality representations of is not trivial, and requires searching, number theory, and some trial and error. It's not at all clear that .

Today I wondered: did Ahmes (the author) have the best possible expansions for all the values, or were there some improvements the Egyptians had missed?

It turns out, yes! Or rather, maybe!

In On the Egyptian method of decomposing into unit fractions the author, Abdulrahman A. Abdulaziz, points out that for the Rhind Mathematical Papyrus gives the expansion $$\frac2{95} = \frac1{60} + \frac1{380} + \frac1{570}$$

but so it could have been written as $$\frac2{95} = \frac1{60}+\frac1{228}.$$

But wait, maybe that wasn't an error. The Egyptians, like everyone, often had to multiply by 10. (In fact, the RMP itself, right after its table, has a shorter table of expansions of .) And is trivially multiplied by 10, whereas isn't. There is some indication that Ahmes preferred fractions with even denominators, because they are easier to double, and the usual Egyptian method of multiplication required repeated doubling. But the Egyptians also sometimes decupled while multiplying, and the expansion would have made both of those easy.

The methods by which Ahmes chose the expansions of , and the criteria by which he preferred one to another, are still unknown; he doesn't explain them. So it's tough to say that any item was or wasn't “best” from Ahmes' point of view.

Gabriella Gonzalez: A sufficiently detailed spec is code>

Tue, 17 Mar 2026 00:00:00 +0000

Specifications do not address the limitations of agentic coding

Chris Smith 2: To Flip Or Not To Flip

Mon, 16 Mar 2026 12:00:03 +0000

A fair coin, an unfair offer, and the price of certainty.

I sat down to work out a classic probability problem numerically, and accidentally built a casino.

The Problem of Points

In 1654, a gambler named Antoine Gombaud posed a question to Blaise Pascal: two players are in a race to win a certain number of points. The game is interrupted. How should they divide the pot?

Pascal wrote to Fermat, and their correspondence became one of the founding documents of probability theory. The answer is elegant: if you need a more points and your opponent needs b more, you can compute the fair split with a simple recurrence. Let P(a, b) be your probability of winning:

P(0, b) = 1 — you just won
P(a, 0) = 0 — your opponent just won
P(a, b) = ½ · P(a−1, b) + ½ · P(a, b−1)

Every value in this table is a fraction with a power-of-2 denominator, and the numerators are just Pascal’s triangle. Beautiful math, clean solution, problem solved since the 17th century.

I built an interactive table to explore it. And then I thought: what if this were a game?

The Game

You and The House race to a target score. Each round, a fair coin is flipped — heads you score, tails The House scores. First to the target wins a pot of money.

But before each flip, judges look at the current game state, consult the probability table, and offer you cash to walk away. Accept, and you take the money. Decline, and the coin is flipped.

The question, every single round, is: to flip or not to flip?

You can play at willowdale.online/flip.

How the Judges Set Their Offers

The judges know the exact fair value of your position — they have the same formula Pascal and Fermat computed. If you have a 37.5% chance of winning a $10,000 pot, your fair value is $3,750.

But they don’t offer fair value. They offer the nearest “clean” fraction of the pot that sits strictly below your true odds.

“Clean” means small denominators whose only prime factors are 2, 3, and 5 — fractions like 1/3, 3/8, 7/20, nothing with a denominator above 20. These produce dollar amounts that look like something a human came up with: $3,333, $3,750, $3,500. Not $3,077 or $3,846, which look like someone ran the numbers to the last penny.

So if your fair value is $3,770 (193/512 of the pot), the judges offer $3,750 (3/8). Barely below fair, and a beautifully round number. If your fair value is $1,875 (3/16), they offer $1,666 (1/6). An 89% offer — a real discount, but still a clean, human-sounding number.

This matters psychologically. Round numbers feel like ballpark estimates — casual, generous, not fully analyzed. Precise numbers feel calculated. When the judges offer $7,500, it sounds reasonable. If they offered $7,517, you’d immediately suspect they did the math and it’s in their favor. The irony is that $7,517 is a better deal for you — but I think you’d be less likely to take it. The round number keeps your guard down.

The algorithm is deterministic — same game state, same offer every time. Just math dressed up in a game show contract.

Why People Sign

Since the offers are always strictly below fair value, the play that maximizes your expected winnings is to never accept a deal. The coin is fair, the game has zero house edge, and every offer leaves money on the table. A player who always flips would win 50% of their games and, on average, neither gain nor lose.

And yet.

When you’re ahead 4–3 in a race to 10, and the contract says $6,000, and you’ve already paid $5,000 to enter this game… you hesitate. That’s a guaranteed profit. The alternative is variance — maybe you win $10,000, but you are not that far ahead. Maybe your luck turns and you lose everything.

You know the offer is below fair. You can peek behind the curtain and see the exact numbers. The judges are shortchanging you by $128. But $128 feels like nothing when the alternative is watching your lead evaporate flip by flip.

So you sign. And $128 goes into the casino’s pocket.

This is what makes the game unusual. In blackjack or roulette, the house edge is baked into the rules — you can’t avoid it no matter how disciplined you are. Here, the game has no edge at all. The coin is fair. The race is symmetric. The only source of profit is human nature. Every dollar the casino makes is expected value that a player voluntarily left on the table.

Play for a while and you start to notice specific situations where the offer gets harder to refuse.

Managing risk. A guaranteed $7,500 is safer than a coin flip worth $7,734. In real life, you might need that money for rent. Variance has a real cost, and paying a premium for certainty can be entirely rational. There is a sophisticated argument for sometimes making decisions that reduce your expected value: bankroll management, survival probability and duration. Here the stakes are fictional, your bankroll buys nothing except more fair coin flips, and going broke is solved by refreshing your web browser, so that case is weaker — but it doesn’t feel weaker when your bankroll is shrinking and the judges are holding out real-looking money.

Mis-anchoring. The rational comparison is always between the offer and the expected value of continuing to flip. But that’s rarely the comparison your brain actually makes. If you were staring at a $0 offer last round and now the judges are offering $500, you’re comparing to the $0 — not to the $625 fair value. If your bankroll started at $10,000 and you’re down to $7,000, and the judges offer $3,200, you’re comparing to $10,000 — because taking the deal would put you above where you started. In both cases, the reference point that feels relevant has nothing to do with the expected value of this game.

Black and white thinking. When you’re behind in the race, the most likely single outcome is that you lose. If the judges offer $500 and your odds of winning are 6%, it feels like a choice between $500 and nothing. But expected value accounts for the 6% — the rare wins are big enough to compensate for all the losses across many games. You just don’t experience many games at once. You experience this one, where you’ll probably lose, and where the person who took $500 looks smart 94 times out of 100.

Imaginary momentum. You lose three flips in a row and it feels like the coin has turned against you — time to take the deal before things get worse. Or you win three in a row and feel like you’re on a streak that shouldn’t be interrupted. The coin has no memory. Each flip is independent. But the human brain is a pattern-recognition machine, and it will find narratives in random sequences whether they’re there or not.

The Optimal Judges

The judges in this game are clever, but simple — they mechanically pick the nearest clean fraction below fair value, blind to everything except the current expected value.

But the optimal offer would be very different. The right objective isn’t just the EV gap (fair value minus offer). It’s:

EV gap × P(acceptance | entire game trajectory)

A huge gap with low acceptance is worthless — the player just turns it down. A tiny gap with high acceptance is pennies. The sweet spot is a moderate discount the player almost can’t refuse.

And that acceptance probability depends on far more than just the current score — it depends on everything described above: the bankroll trajectory, the recent streak, what the last offer was, how long the player has been sitting there.

A perfect judge would think about all of this, and decide exactly what it can get you — tired, frustrated, scared little you — to accept. The clean-fraction heuristic doesn’t. And yet it still works. I still sign those offers.

The Lesson

The game is a playable demonstration of why casinos stay in business, maybe even why people accept below-market returns for safety, and why insurance companies are profitable.

The math is always available — right there behind a curtain. If your goal is to maximize expected dollars, the answer is always to flip the coin. And yet, round after round, the judges offer deals, and I sign them.

Play the game at willowdale.online/flip. It’s free, the coin is fair, and you will almost certainly take a deal you know you shouldn’t.

GHC Developer Blog: GHC 9.12.4-rc1 is now available

Fri, 13 Mar 2026 00:00:00 +0000

GHC 9.12.4-rc1 is now available

wz1000 - 2026-03-13

The GHC developers are very pleased to announce the availability of the release candidate for GHC 9.12.4. Binary distributions, source distributions, and documentation are available at downloads.haskell.org and via GHCup.

GHC 9.12.4 is a bug-fix release fixing many issues of a variety of severities and scopes, including:

Fixed a critical code generation regression where sub-word division produced incorrect results (#26711, #26668), similar to the bug fixed in 9.12.2
Numerous fixes for register allocation bugs, preventing data corruption when spilling and reloading registers (#26411, #26526, #26537, #26542, #26550)
Fixes for several compiler crashes, including issues with CSE (#25468), SetLevels (#26681), implicit parameters (#26451), and the type-class specialiser (#26682)
Fixed cast worker/wrapper incorrectly firing on INLINE functions (#26903)
Fixed LLVM backend miscompilation of bit manipulation operations (#20645, #26065, #26109)
Fixed associated type family and data family instance changes not triggering recompilation (#26183, #26705)
Fixed negative type literals causing the compiler to hang (#26861)
Improvements to determinism of compiler output (#26846, #26858)
Fixes for eventlog shutdown deadlocks (#26573) and lost wakeups in the RTS (#26324)
Fixed split sections support on Windows (#26696, #26494) and the LLVM backend (#26770)
Fixes for the bytecode compiler, PPC native code generator, and Wasm backend
The runtime linker now supports COMMON symbols (#6107)
Improved backtrace support: backtraces for error exceptions are now evaluated at throw time
NamedDefaults now correctly requires the class to be standard or have an in-scope default declaration, and handles poly-kinded classes (#25775, #25778, #25882)
… and many more

A full accounting of these fixes can be found in the

release notes. As always, GHC’s release status, including planned future releases, can be found on the GHC Wiki status.

This release candidate will have a two-week testing period. If all goes well the final release will be available the week of 26 March 2026.

GHC development is sponsored by:

As always, do give this release a try and open a ticket if you see anything amiss.

Christoph Breitkopf: Functional Valhalla?

noreply@blogger.com (bokesan) — Thu, 12 Mar 2026 11:17:02 +0000

Pointer-rich data layouts lead to suboptimal performance on modern hardware. For an excellent introduction to this, see the article The Road to Valhalla. While it is specifically about Java, many parts of the article also apply to other languages. To summarize some of the key points of the article:

In 1990, a main memory fetch was about as expensive as an arithmetic operation. Now, it might be a hundred times slower.
A pointer-rich data layout involving indirections between data at different locations is not ideal for today’s hardware.
A language should make flat (cache-efficient) and dense (memory-efficient) memory layouts possible without compromising abstraction or type safety.

Consider a vector of records (or tuples, structures, product types - I’ll stay with “record” in this article). A pointer-rich layout has each record allocated separately in the heap, with a vector containing pointers to the records. For example, given a “Point” record of two numbers:

The flat and dense layout has the records directly in the array:

(Note that there is another flat layout, namely, using one vector per field of the record. This is better suited to instruction-level parallelism or specialized hardware (e.g., GPUs), especially when the record fields have different sizes. But it is less suited for general-purpose computing, as reading a single vector element requires one memory access per field, whereas the “vector of records” layout above requires only one access per record. Such a layout can be easily implemented in any language that has arrays of native types, whether in the language itself or in a library (e.g., OCaml’s Owl library). Thus, in this article, I will only consider the “array of records” layout above.)

Functional language considerations

Things should be much easier in functional languages than in Java: we have purity, referential transparency, and everything is a value. So it should be simple enough to store these values in memory in their native representation. But there are reasons that that is often not the case in practice:

Lazyness: a value can be a computation that produces a value only when needed.
Layout polymorphism: unless we replicate the code for every type (as, for example, Rust does), we need to be able to store every possible value in the same kind of slot.
Dynamically typed languages require type information at runtime.
Functional languages often have automatic memory management, which may require runtime type information.
Many of our languages are not purely functional, but contain impure features.
Pure languages often lack traditional vectors or arrays, since making them perform well in immutable code is not easy.
Historical reasons: Graph reduction was a common implementation technique for lazy languages, and graphs involve pointers.
Implementation restrictions: not being mainstream, fewer resources are devoted to implementation and optimization.

Many implementations can not even lay out native types flat in records, so a Point record of IEEE 754 double-precision numbers may actually look like this in memory:

The (very short) List

So, given a record type, which functional languages allow a collection of values of that type to have a flat, linear memory layout? The number of programming languages that claim to be “functional” is huge, so the ones listed here are just a selection based on my preferences - mainly languages that allow that layout, and some I have some experience with and can speculate on how easy or hard it would be to add that as a library or extension.

Since the Point record can be misleading in its simplicity when it comes to the question of whether the functionality could be implemented as a library, I’ll point out that there are records where the layout is a bit more interesting:

Records containing different types with different storage sizes, for example, one 64-bit float and one 32-bit integer. On most architectures, this will require 4 bytes of padding between elements.
Records containing native values along with something that has to be represented as a pointer, for example, a reference-type or a lazy value. In a flat layout, this means that every nth element will be a pointer, requiring special support from the memory management system, either by providing layout information or by using a conservative GC that treats everything as a potential pointer.

Pure languages:

Clean

Yes: Clean has unboxed arrays of records in the base language.

Caveat: it does not have integer types of specific sizes and only one floating-point type, making it harder to reduce memory usage by using the smallest type just large enough to support the required value range. It seems possible to implement such types in a library (the mTask system does that).

Futhark

No. Futhark does not intend to be a general-purpose language, so this is not surprising.

I mention it here because it does have arrays of records, but, since it targets GPUs and related hardware, it uses the “record of arrays” layout mentioned above.

Haskell

Yes. Not in the base language, but there is library support via Data.Vector.Unboxed. Types that implement the Unbox type class can be used in these vectors. Many basic types and tuples have an Unbox instance. However, when you care about efficiency, you probably do not want to use tuples but rather a data type with strict fields, i.e., not:

type Point = (Double, Double)

but:

data Point = Point !Double !Double

Writing an Unbox instance for such a type is not trivial. The vector-th-unbox library makes it easier, but requires Template Haskell. Unboxed vectors are implemented by marshalling the values to byte arrays, so records with pointer fields are not supported.

Impure Languages

F#

Yes, even records with pointer fields. Records have structural equality, and you can use structs or the [<Struct>] attribute to get a flat layout.

And that’s all I could find. Unless I follow Wikipedia's list of functional programming languages, which contains languages such as C++, C#, Rust, or Swift, that allow the flat layout, but don’t really fit my idea of a functional language. But SML, OCaml, Erlang (Elixir, Gleam), Scala? Not that I could see (but please correct me if I’m wrong).

Rolling your own

Since there is a library implementation for Haskell, maybe that’s a possibility for other languages?

You should be able to implement flat layouts in any language that supports byte vectors. More interesting is how well such a library fits into the language, and whether a user of the library has to write code or annotations for user-defined record types, or whether the library can handle part or all of that automagically.

I’ll only mention my beloved Lisp/Scheme here. Lisp’s uniform syntax and macro system are a bonus here, but the lack of static typing makes things harder.

In Scheme, R6RS (and R7RS with the help of some SRFIs) has byte-vectors and marshalling to/from them in the standard library. But Scheme does not have type annotations, so you either need to offer a macro to define records with typed fields or to define how to marshal the fields of a regular (sealed) record. Since you can shadow standard procedures in a library, you can write code that looks like regular Scheme code, but, perhaps surprisingly, loses identity when storing/retrieving values from records:

(let ((vec (make-typed-vector 'point 1000))
      (pt (make-point x y)))
  (vector-set! vec 0 pt)
  (eq? (vector-ref vec 0) pt))
 ⇒ #f

(But then, you probably shouldn’t be using eq? when doing functional programming in Scheme).

The same approach is possible in Common Lisp. In contrast to Scheme, it does have optional type annotations, and, together with a helper library for accessing the innards of floats and either the meta-object protocol to get type information or (probably better) a macro to define typed records, an implementation should be reasonably straightforward. Making it play nice with inheritance and the dynamic nature of Common Lisp (e.g., adding slots to classes or even changing an object's class at runtime) would be a much harder undertaking.

Conclusion

Of the functional languages I looked at, only F# fully supports flat and dense memory layouts. Among the pure languages, Haskell and Clean come close.

The question is how important this really is. There’s a good argument to be made for turning to more specialized languages like Futhark if you mainly care about performance. On the other hand, having a uniform codebase in one language also has advantages.

Then, the performance story has changed, too. While the points Project Valhalla raises remain true in principle, processor designers are aware of this as well. They are doing their best to hide memory latency with techniques such as out-of-order execution or humongous caches. Thus, on a modern CPU, the effects of a pointer-rich layout are often only observable with large working set sizes.

Still, given the plethora of imperative language that can get you to Valhalla, support for this in the functional landscape seems lacking. In the future, I hope to see more languages or libraries that will make this possible.

Matt Parsons: Teaching Claude to Be Lazy

Tue, 10 Mar 2026 00:00:00 +0000

I’ve been watching AI development for a long time. I found LessWrong around 2012-2013, and managed to get myself worked up about the oncoming singularity. I managed to chill out about it, but interest and excitement for AI remained. The initial Deep Dream image generation, Alpha Go, etc, were all so exciting. And then GPT-2 came out.

Over the last five years, people have been making wild claims about the utility of present AI. Not “the AI that you’ll have soon,” but the current generation stuff. And the results, frankly, had been garbage. A sea of garbage coating the internet. I’d try using the tools, and when checking them against my own expertise or knowledge, they always fell short.

I heard the noise on Twitter after Opus 4.5 was released in November of 2025. Seemed like a step change- people were much more impressed with it than prior versions. In December, I decided to give it a try. Opus 4.5, with significant guidance, properly diagnosed and fixed some Template Haskell code generation issues. It knew how to -ddump-splices, it knew how to read those splices and diagnose the issue. Given a small, highly mechanical problem, plenty of examples, and a ton of tests, it took about 6 hours to do what I felt would have taken me 3 or 4 hours.

This is pretty incredible, because my productivity has always been limited by two things:

Effort. Literally whacking my keyboard and staring at computer and waiting on a compile/test loop to tell me what to do next.
Attention. Where I’m focusing my effort. My editor? Slack? Meetings? A bike ride? Cello? Which OSS project?

Now, with Opus 4.5, I can set a robot going and do something else with my effort. While Claude Code was spinning on the Template Haskell code, I was doing another project in a different repository. Sure, Claude took 6 hours instead of my 3, but I was able to fill those 6 hours with effort and attention placed elsewhere - not a full 6, as Claude required supervision and input, but call it 5. This is a positive investment, and my personal “break even” moment.

Using Claude Code Effectively

In mid February, I got access to an API token and unlimited usage. I’ve been trying to figure out how to leverage this tool to improve my productivity, and the results have been pretty strongly positive.

The brief tl;dr:

It’s the same shit that makes humans good at software development

Haskell is Awesome for LLMs

This was true with Opus 4.5 and is much more true with Opus 4.6. Prior versions of LLM coding agents produced utter garbage with Haskell, most likely due to the relatively low quantity of examples. It seems like the AI labs have figured out how to do higher quality training with less data, and the relatively high average quality of Haskell code helps the LLMs generate relatively high quality Haskell.

Haskell’s type safety, purity, and library design opportunities make it a fantastic choice for LLM generated code. The human developer can easily specify a solution and let Claude fill in a surprising amount of the boring details.

Haskell’s terse nature benefits LLMs - you can simply fit more tokens into the context window when the tokens are more semantically dense.

Funny enough, all of Haskell’s benefits “for LLMs” are also benefits of Haskell for humans. I do earnestly believe that if all devs knew Haskell, we would consider switching to other languages only very rarely. And Claude knows Haskell.

Software Engineering Matters

Claude Code works really well with tightly scoped issues, lots of tests and examples, and good safety guardrails. I asked it to make cabal faster, and taught it how to run cabal with debug logs, timings, and then to build a profiled version of it. Then it looped for a bit, collected timing information on our codebase, and figured out the critical path and hot spot - the solver. Then it made several fixes to optimize the solver. These changes resulted in a 30% improvement in solver times, which shaved 2 seconds off every cabal repl invocation- a pretty nice benefit, since that happens virtually anytime you want to do anything in our codebase.

But this only worked because the cabal library had timing logs, and I gave it a quick feedback loop and target. I’ve had Claude Code totally fall over when trying to do bigger or more undirected work.

Fortunately, Claude can do this pretty well. I’ve had Claude do some exploratory research (generally pretty highly supervised), then generate some plans for improvement (then edited and clarified), and it can then do a good job of writing up a ticket- certainly better than almost all human written tickets I’ve seen.

Build Workflows Iteratively

LLMs can do anything. But they are expensive, slow, and non-deterministic (and often incorrect). So get the LLM to help with replacing themselves - build a tool or skill to do the thing faster and deterministically.

My Claude sessions generally progress from “highly supervised, exploratory work” to “mostly unsupervised, automated work.” Early sessions in a project often involve having Claude build tools - CLI scripts, libraries, interfaces - that it can use in later work to make the job easier. A surprisingly effective prompt here is “What tools would help you do this job better next time?” At the end of a session, I’ll also have Claude review and update its skill documentation with everything I told it to do differently.

So each work session with Claude produces:

An artifact (the work itself)
Often, updates to the skill to improve efficiency on further work
Sometimes, a tool to deterministically do some chunk of the work.

This process ends up reducing the highly non-deterministic LLM tool with a much more deterministic tool.

Mock Reviews and Refactoring

You can ask Claude to review code, and that works OK. But Claude works much better if you ask it to assume someone else’s perspective. I’ve asked it to mimic myself and it did Alright. I asked it to mimic Edward Kmett, Alexis King, and Michael Snoyman, and it did Alright - it noticed different things with each perspective and suggested improvements in line with those perspectives.

I’ve generally found that the initial output is of poor to middling quality. But you can get decently far with “now make it more legible/faster/more correct” or “apply ‘Parse, Don’t Validate’ here” etc. After several rounds of refactoring, it makes stuff that I’m reasonably happy with putting my name on.

What Doesn’t Work Well

Claude isn’t a replacement for human engineering (“yet” i guess). It lacks qualities like taste, judgement, and vision, that are generally required in subjective work like software and product design. So when I let Claude run totally loose on something, it produces, but it produces poor quality code and poorly thought out features.

I haven’t had great luck with getting Claude to iterate on this itself. When given the very large picture, it sort of flounders. It can do some analysis and subdivision, but the divisions are often somewhat unnatural and don’t feel right to me.

Defined by our Vice

If the above complaint is about Claude’s lack of virtue, let me also complain about Claude’s lack of vice. Claude is infinitely patient and willing to work very hard. However, “infinitely patient” means that Claude has no problem at all waiting an hour for a build to finish. You have to teach it to use faster tools and feedback loops.

Likewise, “hardworking” is a virtue when you’re paying a human by the month and trusting in their laziness to be efficient, but when you’re paying per unit of thought, “more work” means “more cost” and often not “more output.” You have to tell Claude to stop doing stuff or to do stuff more efficiently.

Fortunately, Claude is relatively teachable - but Claude very often will start a skill and then do a lot of “research and understanding” before running the one-shot script to generate the compile-errors to track down and fix.

Humans are impatient and lazy, so we build fast and efficient systems. Without pain to guide us, we make little progress in reducing that pain.

Am I still a skeptic?

I’ve been using AI to write 95% of my code for the last month. And yet, I still feel like I’m more on the skeptic side of things. AI is clearly a useful tool - my own productivity has doubled or more while maintaining my personal quality bar. But it’s not a do-it-all miracle - yet?

AI-first companies are experiencing massive reliability issues. Vibe coding projects start, enjoy some success, and then go down in flames.

Humans are clearly still necessary at key points in the software lifecycle. The bottlenecks have shifted, though, and the easiest parts of my job have been mostly automated. What’s coming next?

I’m excited to wait and find out.

Mark Jason Dominus: Programmers will document for Claude, but not for each other

mjd@plover.com (Mark Dominus) — Mon, 09 Mar 2026 08:04:00 +0000

A couple of days ago I recounted a common complaint:

I keep seeing programmers say how angry it makes them that people are willing to write detailed CLAUDE.md and PROJECT.md files for Claude to use, but they weren't willing to write them for their coworkers.

For larger projects, I've taken to having Claude maintain a handoff document that I can have the next Claude read, saying what we planned to do, what has been done, and other pertinent information. Then when I shut down one Claude I can have the next one read the file to get up to speed. Then I have the Claude update it for Claude .

After seeing the common complaint enough times I had a happy inspiration. I'd been throwing away Claude's handoff documents at the end of each project. Why do that? It's no trouble to copy the file into the repository and commit it. Someone in the future, wondering what was going on, might luckily find the right document with git grep and learn something useful.

I'm a little slow so it took me until this week to think of a better version of this: at the end of the project I now ask Claude to write up from scratch a detailed but high-level explanation of what problem we were solving and what changes we made, and I commit that. Not just running notes, but a structured overview of the whole thing.

I review these overviews carefully and make edits as necessary before I check them in. It's my signature on the commit, and my bank account receiving the paycheck, so nothing goes into the repository that I haven't read carefully and understood, same as if Claude were a human programmer under my supervision.

But Claude's explanations haven't required much editing. Claude's most recent project summary was around as good as what I could have written myself, maybe a little worse and maybe a little better. But it took ten seconds to write instead of an hour, and it didn't take anything like an hour to review.

The serious thing I had to fix the last time around was that Claude had used a previous, related report as a model, and the previous report had had a paragraph I had added at the end that said:

# Approved-by

Claude abstracted these notes from our discussions of the issue. Mark Dominus has read, reviewed, edited, and approved these notes.

Claude's new document had an identical section at the end. Oops! Fortunately, by the time I saw it, it was true, so I didn't have to delete it. I had Claude add a sentence to CLAUDE.md to tell it not to do this again.

My advice for the day:

If you have Claude write down notes, check them into the repo when you're done. It probably can't hurt and it might help.
Have Claude write a project summary, and then check it into the repo.

Maybe this is obvious? But it wasn't obvious to me. I'm still getting used to this new world.

Haskell Interlude: 78: Jamie Willis

Sun, 08 Mar 2026 11:00:00 +0000

In this episode, we focus on a particular part of Haskell: teaching it. To help us, we are joined by Jamie Willis who is a Teaching Fellow at Imperial College London. The episode explores the benefits of live coding, and why Haskell is the best language for teaching programming.

Mark Jason Dominus: How are John Waters movies like James Bond movies?

mjd@plover.com (Mark Dominus) — Sun, 08 Mar 2026 09:50:00 +0000

A number of years ago I wondered how many movies I had seen. The only way I could think of finding out was just to make a list. This I did as best I could. (It turned out to be around 700.)

I found, though, that I could not include all the James Bond movies I had seen, because I couldn't tell them apart from the descriptions. I'd read a plot summary for a James Bond movie, and ask myself “Did I see that? I don't know, it sounds like every other James Bond movie.”

Today I discovered that John Waters movies are like that also. I was trying to remember if I had seen A Dirty Shame:

The people of Harford Road are firmly divided into two camps: the neuters, the puritanical residents who despise anything even remotely carnal; and the perverts, a group of sex addicts whose unique fetishes have all been brought to the fore by accidental concussions. Repressed Sylvia Stickles finds herself firmly entrenched in the former camp.

You'd think that would be something I would remember decisively, or not. But I'm really not sure. All I can do is shrug and say “I don't know, it sounds like a John Waters movie I have seen, but maybe it wasn't that one.”

Looking into it further I discovered that I also wasn't sure if I had seen Multiple Maniacs. In it, Divine's character is raped by a giant lobster. On the one hand, that seems like the sort of thing I would remember. And I think maybe I do? But again I'm not sure I'm not just imagining what it would be like!

Mark Jason Dominus: Documentation is a message in a bottle

mjd@plover.com (Mark Dominus) — Thu, 05 Mar 2026 16:07:00 +0000

Our company is going to a convention later this month, and they will have a booth with big TV screens showing statistics that update in real time. My job is to write the backend server that delivers the statistics.

I read over the documents that the product people had written up about what was wanted, asked questions, got answers, and then turned the original two-line ticket into a three-page ticket that said what should be done and how. I intended to do the ticket myself, but it's good practice to write all this stuff down, for many reasons:

Writing things down forces me to think them through carefully and realize what doesn't make sense or what I still don't understand.
I forget things easily and this will keep the plan where I can find it.
I might get sick, and if someone else has to pick up the project this might help them understand what I was doing.
If my boss gets worried that all I do is post on 4chan all day, this is tangible work product that proves I did something else that might have enhanced shareholder value.
If I'm tempted to spend the day posting on 4chan, and then to later claim I spent the time planning the project, I might fool my boss. But without that tangible work product, I won't be able to fool myself, and that's more important.
Conversely if I later think back and ask “What was I doing the week of March 2?” I might be tempted to imagine that all I did was post on 4chan. But the three pages of ticket description will prove to me that I am not just a lazy slacker. This is a real problem for me.
In principle, a future person going back to extend the work might find this helpful documentation of what was done and why. Does this ever really happen? I don't know, but it might.
I like writing because writing is fun.

A few days after I wrote the ticket, something unexpected happened. It transpired that person who was to build the front-end consumer of my statistics would not be a professional programmer. It would be the company's Head of Product, a very smart woman named Amanda. The actual code would be written by Claude, under her supervision.

I have never done anything like this before, and I would not have wanted to try it on a short deadline, but there is some slack in the schedule and it seemed a worthwhile and exciting experiment.

Amanda shared some screencaps of her chats with Claude about the project, and I suggested:

When you get a chance, please ask Claude to write out a Markdown file memorializing all this. Tell it that you're going to give it to the backend programmer for discussion, so more detail is better. When it's ready, send it over.

Claude immediately produced a nine-page, 14-part memo and a half-page overview. I spent a couple of hours reviewing it and marking it up.

It became immediately clear that Claude and I had very similar ideas about how the project should go and how the front and back ends would hook up. So similar that I asked Angela:

It looks like maybe you started it off by feeding it my ticket description. Is that right?

She said yes, she had. She had also fed it the original product documents I had read.

I was delighted. I had had many reasons for writing detailed ticket descriptions before, but the most plausible ones were aimed back at myself.

The external consumers of the documentation all seemed somewhat unlikely. The person who would extend the project in the future probably didn't exist, and if they did they probably wouldn't have thought to look at my notes. Same for the hypothetical person who would take over when I got sick. My boss probably isn't checking up on me by looking at my ticketing history. Still, I like to document these things for my own benefit, and also just in case.

But now, because I had written the project plan, it was available for consumption when an unexpected consumer turned up! Claude and I were able to rapidly converge on the design of the system, because Amanda had found my notes and cleverly handed them to Claude. Suddenly one of those unlikely-seeming external reasons materialized!

On Mastodon I keep seeing programmers say how angry it makes them that people are willing to write detailed CLAUDE.md and PROJECT.md files for Claude to use, but they weren't willing to write them for their coworkers. (They complain about this as if this is somehow the fault of the AI, rather than of the people who failed in the past to write documentation for their coworkers.)

The obvious answer to the question of why people are willing to write documentation for Claude but not for their coworkers is that the author can count on Claude to read the documentation, whereas it's a rare coworker who will look at it attentively.

Rik Signes points out there's a less obvious but more likely answer: your coworkers will remember things if you just tell them, but Claude forgets everything every time. If you want Claude to remember something, you have to write it down. So people using Claude do write things down, because otherwise they have to say them over and over.

And there's a happy converse to the complaint that most programmers don't bother to write documentation. It means that people like me, professionals who have always written meticulous documentation, are now reaping new benefits from that always valuable practice.

Not everything is going to get worse. Some things will get better.

Addendum 20260208

A corollary: You don't have to write the rocumentation yourself. You can have Claude write a detailed summary based on your ongoing chats about the work, and then you can edit it and check it in.

If you're good at editing, anyway. I wonder if part of the reason Claude is working so well for me is that I'm really good at editing and at code review?

Mark Jason Dominus: Bo Diddley

mjd@plover.com (Mark Dominus) — Tue, 03 Mar 2026 16:39:00 +0000

Bo Diddley's cover of "Sixteen Tons" sounds very much like one of my favorites, "Can't Judge A Book By Its Cover". It's interesting to compare.

Thinking on that it suddenly occured to me that his name might have been a play on “diddley bow”, which is a sort of homemade one-stringed zither. The player uses a bottle as a bridge for the string, and changes the pitch by sliding the bottle up and down. When you hear about blues artists whose first guitars were homemade, this is often what was meant: it wasn't a six-string guitar, it was a diddley bow.

But it's not clear that Bo Diddley did play his name on the diddley bow. "Diddly" also means something insignificant or of little value, and might have been a disparaging nickname he received in his youth. (It also appears in the phrase "diddly squat"). Maybe that's also the source of the name of the diddley bow.

Donnacha Oisín Kidney: Monuses and Heaps

Tue, 03 Mar 2026 00:00:00 +0000

Posted on March 3, 2026

Tags: Haskell

This post is about a simple algebraic structure that I have found useful for algorithms that involve searching or sorting based on some ordered weight. I used it a bit in a pair of papers on graph search (2021; 2025), and more recently I used it to implement a version of the Phases type (Easterly 2019) that supported arbitrary keys, inspired by some work by BlÃ¶ndal (2025a; 2025b) and Visscher (2025).

The algebraic structure in question is a monus, which is a kind of monoid that supports a partial subtraction operation (that subtraction operation, denoted by the symbol âˆ¸, is itself often called a â€œmonusâ€�). However, before giving the full definition of the structure, let me first try to motivate its use. The context here is heap-based algorithms. For the purposes of this post, a heap is a tree that obeys the â€œheap propertyâ€�; i.e.Â every node in the tree has some â€œweightâ€� attached to it, and every parent node has a weight less than or equal to the weight of each of its children. So, for a tree like the following:

   â”Œd
 â”Œbâ”¤
aâ”¤ â””e
 â””c

The heap property is satisfied when <semantics>aâ‰¤b<annotation encoding="application/x-tex">a \leq b</annotation></semantics>, <semantics>aâ‰¤c<annotation encoding="application/x-tex">a \leq c</annotation></semantics>, <semantics>bâ‰¤d<annotation encoding="application/x-tex">b \leq d</annotation></semantics>, and <semantics>bâ‰¤e<annotation encoding="application/x-tex">b \leq e</annotation></semantics>.

Usually, we also want our heap structure to have an operation like popMin :: Heap k v -> Maybe (v, Heap k v) that returns the least-weight value in the heap paired with the rest of the heap. If this operation is efficient, we can use the heap to efficiently implement sorting algorithms, graph search, etc. In fact, let me give the whole basic interface for a heap here:

popMin :: Heap k v -> Maybe (v, Heap k v)
insert :: k -> v -> Heap k v -> Heap k v
empty  :: Heap k v

Using these functions itâ€™s not hard to see how we can implement a sorting algorithm:

sortOn :: (a -> k) -> [a] -> [a]
sortOn k = unfoldr popMin . foldr (\x -> insert (k x) x) empty

The monus becomes relevant when the weight involved is some kind of monoid. This is quite a common situation: if we were using the heap for graph search (least-cost paths or something), we would expect the weight to correspond to path costs, and we would expect that we can add the costs of paths in a kind of monoidal way. Furthermore, we would probably expect the monoidal operations to relate to the order in some coherent way. A monus (Amer 1984) is an ordered monoid where the order itself can be defined in terms of the monoidal operations¹:

<semantics>xâ‰¤yâ‡”âˆƒz.y=xâ€¢z<annotation encoding="application/x-tex"> x \leq y \iff \exists z. \; y = x \bullet z </annotation></semantics>

I read this definition as saying â€œ<semantics>x<annotation encoding="application/x-tex">x</annotation></semantics> is less than <semantics>y<annotation encoding="application/x-tex">y</annotation></semantics> iff there is some <semantics>z<annotation encoding="application/x-tex">z</annotation></semantics> that fits between <semantics>x<annotation encoding="application/x-tex">x</annotation></semantics> and <semantics>y<annotation encoding="application/x-tex">y</annotation></semantics>â€�. In other words, the gap between <semantics>x<annotation encoding="application/x-tex">x</annotation></semantics> and <semantics>y<annotation encoding="application/x-tex">y</annotation></semantics> has to exist, and it is equal to <semantics>z<annotation encoding="application/x-tex">z</annotation></semantics>.

Notice that this order definition wonâ€™t work for groups like <semantics>(â„¤,+,0)<annotation encoding="application/x-tex">(\mathbb{Z},+,0)</annotation></semantics>. For a group, we can always find some <semantics>z<annotation encoding="application/x-tex">z</annotation></semantics> that will fit the existential (specifically, <semantics>z=(âˆ’x)â€¢y<annotation encoding="application/x-tex">z = (- x) \bullet y</annotation></semantics>). Monuses, then, tend to be positive monoids: in fact, many monuses are the positive cones of some group (<semantics>(â„•,+,0)<annotation encoding="application/x-tex">(\mathbb{N},+,0)</annotation></semantics> is the positive cone of <semantics>(â„¤,+,0)<annotation encoding="application/x-tex">(\mathbb{Z},+,0)</annotation></semantics>).

We can derive a lot of useful properties from this basic structure. For example, if the order above is total, then we can derive the binary subtraction operator mentioned above:

<semantics>xâˆ¸y={z,if yâ‰¤x and x=yâ€¢z0,otherwise.<annotation encoding="application/x-tex"> x âˆ¸ y = \begin{cases} z, & \text{if } y \leq x \text{ and } x = y \bullet z \\ 0, & \text{otherwise.} \end{cases} </annotation></semantics>

If we require the underlying monoid to be commutative, and we further require the derived order to be total and antisymmetric, we get the particular flavour of monus I worked with in a pair of papers on graph search (2021; 2025). In this post I will actually be working with a weakened form of the algebra that I will define shortly.

Getting back to our heap from above, with this new order defined, we can see that the heap property actually tells us something about the makeup of the weights in the tree. Instead of every child just having a weight equal to some arbitrary quantity, the heap property tells us that each child weight has to be made up of the combination of its parentâ€™s weight and some difference.

   â”Œd              â”Œbâ€¢(dâˆ¸b)
 â”Œbâ”¤       â”Œaâ€¢(bâˆ¸a)â”¤
aâ”¤ â””e  =  aâ”¤       â””bâ€¢(eâˆ¸b)
 â””c        â””aâ€¢(câˆ¸a)

This observation gives us an opportunity for a different representation: instead of storing the full weight at each node, we could instead just store the difference.

     â”Œdâˆ¸b
 â”Œbâˆ¸aâ”¤
aâ”¤   â””eâˆ¸b
 â””câˆ¸a

Just in terms of data structure design, I prefer this version: if we wanted to write down a type of heaps using the previous design, we would first define the type of trees, and then separately write a predicate corresponding to the heap property. With this design, it is impossible to write down a tree that doesnâ€™t satisfy the heap property.

More practically, though, using this algebraic structure when working with heaps enables some optimisations that might be difficult to implement otherwise. The strength of this representation is that it allows for efficient relative and global computation: now, if we wanted to add some quantity to every weight in the tree, we can do it just by adding the weight to the root node.

Monuses in Haskell

To see some examples of how to use this pattern, letâ€™s first write a class for Haskell monuses:

class (Semigroup a, Ord a) => Monus a where
  (âˆ¸) :: a -> a -> a

Youâ€™ll notice that weâ€™re requiring semigroup here, not monoid. Thatâ€™s because one of the nice uses of this pattern actually works with a weakening of the usual monus algebra; this weakening only requires semigroup, and the following two laws.

<semantics>xâ‰¤yâŸ¹xâ€¢(yâˆ¸x)=yxâ‰¤yâŸ¹zâ€¢xâ‰¤zâ€¢y<annotation encoding="application/x-tex"> x \leq y \implies x \bullet (y âˆ¸ x) = y \quad \quad \quad \quad \quad x \leq y \implies z \bullet x \leq z \bullet y </annotation></semantics>

A straightforward monus instance is the following:

instance (Num a, Ord a) => Monus (Sum a) where
  (âˆ¸) = (-)

Pairing Heaps in Haskell

Next, letâ€™s look at a simple heap implementation. I will always go for pairing heaps (Fredman et al. 1986) in Haskell; they are extremely simple to implement, and (as long as you donâ€™t have significant persistence requirements) their performance seems to be the best of the available pointer-based heaps (Larkin, Sen, and Tarjan 2013). Here is the type definition:

data Root k v = Root !k v [Root k v]
type Heap k v = Maybe (Root k v)

A Root is a non-empty pairing heap; the Heap type represents possibly-empty heaps. The key function to implement is the merging of two heaps; we can accomplish this as an implementation of the semigroup <>.

instance Monus k => Semigroup (Root k v) where
  Root xk xv xs <> Root yk yv ys
    | xk <= yk  = Root xk xv (Root (yk âˆ¸ xk) yv ys : xs)
    | otherwise = Root yk yv (Root (xk âˆ¸ yk) xv xs : ys)

The only difference between this and a normal pairing heap merge is the use of âˆ¸ in the key of the child node (yk âˆ¸ xk and xk âˆ¸ yk). This difference ensures that each child only holds the difference of the weight between itself and its parent.

Itâ€™s worth working out why the weakened monus laws above are all we need in order to maintain the heap property on this structure.

The rest of the methods are implemented the same as their implementations on a normal pairing heap. First, we have the pairing merge of a list of heaps, here given as an implementation of the semigroup method sconcat:

  sconcat (x1 :| []) = x1
  sconcat (x1 :| [x2]) = x1 <> x2
  sconcat (x1 :| x2 : x3 : xs) = (x1 <> x2) <> sconcat (x3 :| xs)

merges :: Monus k => [Root k v] -> Heap k v
merges = fmap sconcat . nonEmpty

The pattern of this two-level merge is what gives the pairing heap its excellent performance.

The Heap type derives its monoid instance from the monoid instance on Maybe (instance Semigroup a => Monoid (Maybe a)), so we can implement insert like so:

insert :: Monus k => k -> v -> Heap k v -> Heap k v
insert k v hp = Just (Root k v []) <> hp

And popMin is also relatively simple:

delay :: Semigroup k => k -> Heap k v -> Heap k v
delay by = fmap (\(Root k v xs) -> Root (by <> k) v xs)

popMin :: Monus k => Heap k v -> Maybe (v,Heap k v)
popMin = fmap (\(Root k v xs) -> (v, k `delay` merges xs))

Notice that we delay the rest of the heap, because all of its entries need to be offset by the weight of the previous root node. Thankfully, because weâ€™re only storing the differences, we can â€œmodifyâ€� every weight by just increasing the weight of the root, making this an <semantics>ğ�’ª(1)<annotation encoding="application/x-tex">\mathcal{O}(1)</annotation></semantics> operation.

Finally, we can implement heap sort like so:

sortOn :: Monus k => (a -> k) -> [a] -> [a]
sortOn k = unfoldr popMin . foldl' (\xs x -> insert (k x) x xs) Nothing

And it does indeed work:

>>> sortOn Sum [3,4,2,5,1]
[1,2,3,4,5]

Here is a trace of the output:

Trace

Input           Heap:
list:

[3,4,2,5,1]


[4,2,5,1]       3


[2,5,1]         3â”€1


[5,1]           2â”€1â”€1


[1]              â”Œ3
                2â”¤
                 â””1â”€1


[]                 â”Œ3
                1â”€1â”¤
                   â””1â”€1


Output          Heap:
list:

[]                 â”Œ3
                1â”€1â”¤
                   â””1â”€1


[1]              â”Œ3
                2â”¤
                 â””1â”€1


[1,2]            â”Œ2
                3â”¤
                 â””1


[1,2,3]         4â”€1


[1,2,3,4]       5


[1,2,3,4,5]

While the heap implementation presented here is pretty efficient, note that we could significantly improve its performance with a few optimisations: first, we could unpack all of the constructors, using a custom list definition in Root instead of Haskellâ€™s built-in lists; second, in foldl' we could avoid the Maybe wrapper by building a non-empty heap. There are probably more small optimisations available as well.

Retrieving a Normal Heap

A problem with the definition of sortOn above is that it requires a Monus instance on the keys, but it only really needs Ord. It seems that by switching to the Monus-powered heap we have lost some generality.

Luckily, there are two monuses we can use to solve this problem:

instance Ord a => Monus (Max a) where
  x âˆ¸ y = x

instance Ord a => Monus (Last a) where
  x âˆ¸ y = x

The Max semigroup uses the max operation, and the Last semigroup returns its second operand.

instance Ord a => Semigroup (Max a) where
  Max x <> Max y = Max (max x y)

instance Semigroup (Last a) where
  x <> y = y

While the Monus instances here might seem degenerate, they do actually satisfy the Monus laws as given above.

Max and Last Monus laws

Max:

x <= y ==> x <> (y âˆ¸ x) = y
Max x <= Max y ==> Max x <> (Max y âˆ¸ Max x) = Max y
Max x <= Max y ==> Max x <> Max y = Max y
Max x <= Max y ==> Max (max x y) = Max y
x <= y ==> max x y = y

x <= y ==> z <> x <= z <> y
Max x <= Max y ==> Max z <> Max x <= Max z <> Max y
Max x <= Max y ==> Max (max z x) <= Max (max z y)
x <= y ==> max z x <= max z y

Last:

x <= y ==> x <> (y âˆ¸ x) = y
Last x <= Last y ==> Last x <> (Last y âˆ¸ Last x) = Last y
Last x <= Last y ==> Last x <> Last y = Last y
Last x <= Last y ==> Last y = Last y

x <= y ==> z <> x <= z <> y
Last x <= Last y ==> Last z <> Last x <= Last z <> Last y
Last x <= Last y ==> Last x <= Last y

Either Max or Last will work; semantically, thereâ€™s no real difference. Last avoids some comparisons, so we can use that:

sortOn' :: Ord k => (a -> k) -> [a] -> [a]
sortOn' k = sortOn (Last . k)

Phases as a Pairing Heap

The Phases applicative (Easterly 2019) is an Applicative transformer that allows reordering of Applicative effects in an easy-to-use, high-level way. The interface looks like this:

phase     :: Natural -> f a -> Phases f a
runPhases :: Applicative f => Phases f a -> f a

instance Applicative f => Applicative (Phases f)

And we can use it like this:

phased :: IO String
phased = runPhases $ sequenceA
  [ phase 3 $ emit 'a'
  , phase 2 $ emit 'b'
  , phase 1 $ emit 'c'
  , phase 2 $ emit 'd'
  , phase 3 $ emit 'e' ]
  where emit c = putChar c >> return c

>>> phased
cbdae
"abcde"

The above computation performs the effects in the order dictated by their phases (this is why the characters are printed out in the order cbdae), but the pure value (the returned string) has its order unaffected.

I have written about this type before, and in a handful of papers (2021; Gibbons et al. 2022; Gibbons et al. 2023), but more recently BlÃ¶ndal (2025a) started looking into trying to use the Phases pattern with arbitrary ordered keys (Visscher 2025; BlÃ¶ndal 2025b). There are a lot of different directions you can go from the Phases type; what interested me most immediately was the idea of implementing the type efficiently using standard data-structure representations. If our core goal here is to order some values according to a key, then that is clearly a problem that a heap should solve: enter the free applicative pairing heap.

Here is the typeâ€™s definition:

data Heap k f a where
  Pure :: a -> Heap k f a
  Root :: !k -> (x -> y -> a) -> f x -> Heaps k f y -> Heap k f a

data Heaps k f a where
  Nil :: Heaps k f ()
  App :: !k -> f x -> Heaps k f y -> Heaps k f z -> Heaps k f (x,y,z)

We have had to change a few aspects of the original pairing heap, but the overall structure remains. The entries in this heap are now effectful computations: the fs. The data structure also contains some scaffolding to reconstruct the pure values â€œinsideâ€� each effect when we actually run the heap.

The root-level structure is the Heap: this can either be Pure (corresponding to an empty heap: notice that, though this constructor has some contents (the a), it is still regarded as â€œemptyâ€� because it contains no effects (f)); or a Root, which is a singleton value, paired with the list of sub-heaps represented by the Heaps type. Weâ€™re using the usual Yoneda-ish trick here to allow the top-level data type to be parametric and a Functor, by storing the function x -> y -> a.

The Heaps type then plays the role of [Root k v] in the previous pairing heap implementation; here, we have inlined all of the constructors so that we can get all of the types to line up. Remember, this is a heap of effects, not of pure values: the pure values need to be able to be reconstructed to one single top-level a when we run the heap at the end.

Merging two heaps happens in the Applicative instance itself:

instance Functor (Heap k f) where
  fmap f (Pure x) = Pure (f x)
  fmap f (Root k c x xs) = Root k (\a b -> f (c a b)) x xs

instance Monus k => Applicative (Heap k f) where
  pure = Pure
  Pure f <*> xs = fmap f xs
  xs <*> Pure f = fmap ($ f) xs

  Root xk xc xs xss <*> Root yk yc ys yss
    | xk <= yk  = Root xk (\a (b,c,d) -> xc a d (yc b c)) xs (App (yk âˆ¸ xk) ys yss xss)
    | otherwise = Root yk (\a (b,c,d) -> xc b c (yc a d)) ys (App (xk âˆ¸ yk) xs xss yss)

To actually run the heap we will use the following two functions:

merges :: (Monus k, Applicative f) => Heaps k f a -> Heap k f a
merges Nil = Pure ()
merges (App k1 e1 t1 Nil) = Root k1 (,,()) e1 t1
merges (App k1 e1 t1 (App k2 e2 t2 xs)) =
   (Root k1 (\a b cd es -> (a,b, cd es)) e1 t1 <*> Root k2 (,,) e2 t2) <*> merges xs

runHeap :: (Monus k, Applicative f) => Heap k f a -> f a
runHeap (Pure x) = pure x
runHeap (Root _ c x xs) = liftA2 c x (runHeap (merges xs))

And we can lift a computation into Phases like so:

phase :: k -> f a -> Heap k f a
phase k xs = Root k const xs Nil

Stabilising Phases

Thereâ€™s a problem. A heap sort based on a pairing heap isnâ€™t stable. That means that the order of effects here can vary for two effects in the same phase. If we look back to the example with the strings we saw above, that means that outputs like cdbea would be possible (in actual fact, we donâ€™t get any reordering in this particular example, but thatâ€™s just an accident of the way the applicative operators are associated under the hood).

This is problematic because we would expect effects in the same phase to behave as if they were normal applicative effects, sequenced according to their syntactic order. It also means that the applicative transformer breaks the applicative laws, because effects might be reordered according to the association of the applicative operators, which should lawfully be associative.

To make the sort stable, we could layer the heap effect with some state effect that would tag each effect with its order. However, that would hurt efficiency and composability: it would force us to linearise the whole heap sort procedure, where currently different branches of the tree can compute completely independently of each other. The solution comes in the form of another monus: the key monus.

data Key k = !k :* {-# UNPACK #-} !Int deriving (Eq, Ord)

A Key k is some ordered key k coupled with an Int that represents the offset between the original position and the current position of the key. In this way, when two keys compare as equal, we can cascade on to compare their original positions, thereby maintaining their original order when there is ambiguity caused by a key collision. However, in contrast to the approach of walking over the data once and tagging it all with positions, this approach keeps the location information completely local: we never need to know that some key is in the <semantics>n<annotation encoding="application/x-tex">n</annotation></semantics>th position in the original sequence, only that it has moved <semantics>n<annotation encoding="application/x-tex">n</annotation></semantics> steps from its original position.

The instances are as follows:

instance Semigroup (Key k) where
  (xk :* xi) <> (yk :* yi) = yk :* (xi + yi)

instance Ord k => Monus (Key k) where
  (xk :* xi) âˆ¸ (yk :* yi) = xk :* (xi - yi)

This instance is basically a combination of the Last semigroup and the <semantics>(â„¤,+,0)<annotation encoding="application/x-tex">(\mathbb{Z}, +, 0)</annotation></semantics> group. We could make a slightly more generalised version of Key that is the combination of any monus and <semantics>â„¤<annotation encoding="application/x-tex">\mathbb{Z}</annotation></semantics>, but since Iâ€™m only going to be using this type for simple sorting-like algorithms I will leave that generalisation for another time.

The stable heap type is as follows:

data Stable k f a
  = Stable { size :: {-# UNPACK #-} !Int
           , heap :: !(Heap (Key k) f a) }

We need to track the size of the heap so that we can supply the right-hand operand with their offsets. Because weâ€™re storing differences, we can add an offset to every entry in a heap in <semantics>ğ�’ª(1)<annotation encoding="application/x-tex">\mathcal{O}(1)</annotation></semantics> time by simply adding to the root:

delayKey :: Int -> Heap (Key k) f a -> Heap (Key k) f a
delayKey _ hp@(Pure _) = hp
delayKey n (Root (k :* m) c x xs) = Root (k :* (n + m)) c x xs

Finally, using this we can implement the Applicative instance and the rest of the interface:

instance Ord k => Applicative (Stable k f) where
  pure = Stable 0 . pure
  Stable n xs <*> Stable m ys = Stable (n+m) (xs <*> delayKey n ys)

runStable :: (Applicative f, Ord k) => Stable k f a -> f a
runStable = runHeap . heap

stable :: Ord k => k -> f a -> Stable k f a
stable k fa = Stable 1 (phase (k :* 0) fa)

This is a pure, optimally efficient implementation of Phases ordered by an arbitrary total-ordered key.

Local Computation in a Monadic Heap

In (2021), I developed a monadic heap based on the free monad transformer.

newtype Search k a = Search { runSearch :: [Either a (k, Search k a)] }

This type is equivalent to the free monad transformer over the list monad and (,) k functor (i.e.Â the writer monad).

Search k a â‰… FreeT ((,) k) [] a

In the paper (2021) we extended the type to become a full monad transformer, replacing lists with ListT. This let us order the effects according to the weight k; however, for this example we only need the simplified type, which lets us order the values according to k.

This Search type follows the structure of a pairing heap (although not as closely as the version above). However, this type is interesting because semantically it needs the weights to be stored as differences, rather than absolute weights. As a free monad transformer, the Search type layers effects on top of each other; we can later interpret those layers by collapsing them together using the monadic join. In the case of Search, those layers are drawn from the list monad and the (,) k functor (writer monad). That means that if we have some heap representing the tree from above:

Search [ Right (a, Search [ Right (b, Search [ Right (d, Search [Left x])
                                             , Right (e, Search [Left y])])
                          , Right (c, Search [Left z])])]

When we collapse this computation down to the leaves, the weights we will get are the following:

[(a <> b <> d, x), (a <> b <> e, y), (a <> c, z)]

So, if we want the weights to line up properly, we need to store the differences.

mergeS :: Monus k => [(k, Search k a)] -> Maybe (k, Search k a)
mergeS [] = Nothing
mergeS (x:xs) = Just (mergeS' x xs)
  where
    mergeS' x1 [] = x1
    mergeS' x1 [x2] = x1 <+> x2
    mergeS' x1 (x2:x3:xs) = (x1 <+> x2) <+> mergeS' x3 xs

    (xw, Search xs) <+> (yw, Search ys)
      | xw <= yw  = (xw, Search (Right (yw âˆ¸ xw, Search ys) : xs))
      | otherwise = (yw, Search (Right (xw âˆ¸ yw, Search xs) : ys))

popMins :: Monus k => Search k a -> ([a], Maybe (k, Search k a))
popMins = fmap mergeS . partitionEithers . runSearch

Conclusion

The technique of â€œdonâ€™t store the absolute value, store the differenceâ€� seems to be generally quite useful; I think that monuses are a handy algebra to keep in mind whenever that technique looks like it might be needed. The Key monus above is closely related to the factorial numbers, and the trick I used in this post.

References

Amer, K. 1984. â€œEquationally complete classes of commutative monoids with monus.â€� algebra universalis 18 (1) (February): 129â€“131. doi:10.1007/BF01182254.

BlÃ¶ndal, Baldur. 2025a. â€œGeneralized multi-phase compiler/concurrency.â€� reddit. https://www.reddit.com/r/haskell/comments/1m25fw8/generalized_multiphase_compilerconcurrency/.

â€”â€”â€”. 2025b. â€œPhases using Vault.â€� reddit. https://www.reddit.com/r/haskell/comments/1msvwzd/phases_using_vault/.

Easterly, Noah. 2019. â€œFunctions and newtype wrappers for traversing Trees: Rampion/tree-traversals.â€� https://github.com/rampion/tree-traversals.

Fredman, Michael L., Robert Sedgewick, Daniel D. Sleator, and Robert E. Tarjan. 1986. â€œThe pairing heap: A new form of self-adjusting heap.â€� Algorithmica 1 (1-4) (January): 111â€“129. doi:10.1007/BF01840439.

Gibbons, Jeremy, Donnacha OisÃn Kidney, Tom Schrijvers, and Nicolas Wu. 2022. â€œBreadth-First Traversal viaÂ Staging.â€� In Mathematics of Program Construction, ed by. Ekaterina Komendantskaya, 1â€“33. Cham: Springer International Publishing. doi:10.1007/978-3-031-16912-0_1.

â€”â€”â€”. 2023. â€œPhases in Software Architecture.â€� In Proceedings of the 1st ACM SIGPLAN International Workshop on Functional Software Architecture, 29â€“33. FUNARCH 2023. New York, NY, USA: Association for Computing Machinery. doi:10.1145/3609025.3609479.

Kidney, Donnacha OisÃn, and Nicolas Wu. 2021. â€œAlgebras for weighted search.â€� Proceedings of the ACM on Programming Languages 5 (ICFP) (August): 72:1â€“72:30. doi:10.1145/3473577.

â€”â€”â€”. 2025. â€œFormalising Graph Algorithms with Coinduction.â€� Proc. ACM Program. Lang. 9 (POPL) (January): 56:1657â€“56:1686. doi:10.1145/3704892.

Larkin, Daniel H., Siddhartha Sen, and Robert E. Tarjan. 2013. â€œA Back-to-Basics Empirical Study of Priority Queues.â€� In 2014 Proceedings of the Meeting on Algorithm Engineering and Experiments (ALENEX), 61â€“72. Proceedings. Society for Industrial and Applied Mathematics. doi:10.1137/1.9781611973198.7.

Visscher, Sjoerd. 2025. â€œPhases with any Ord key type.â€� https://gist.github.com/sjoerdvisscher/bf282a050f0681e2f737908e254c4061.

Note that there are many related structures that all fall under the umbrella notion of â€œmonusâ€�; the structure that I am defining here is the same structure I worked with in (2021) and (2025).â†©ï¸�

Joachim Breitner: Vibe-coding a debugger for a DSL

mail@joachim-breitner.de (Joachim Breitner) — Wed, 25 Feb 2026 10:53:30 +0000

Earlier this week a colleague of mine, Emilio Jesús Gallego Arias, shared a demo of something he built as an experiment, and I felt the desire to share this and add a bit of reflection. (Not keen on watching a 5 min video? Read on below.)

What was that?

So what did you just see (or skipped watching)? You could see Emilio’s screen, running VSCode and editing a Lean file. He designed a small programming language that he embedded into Lean, including an evaluator. So far, so standard, but a few things stick out already:

Using Lean’s very extensible syntax this embedding is rather elegant and pretty.
Furthermore, he can run this DSL code right there, in the source code, using commands like #eval. This is a bit like the interpreter found in Haskell or Python, but without needing a separate process, or like using a Jupyter notebook, but without the stateful cell management.

This is already a nice demonstration of Lean’s abilities and strength, as we know them. But what blew my mind the first time was what happened next: He had a visual debugger that allowed him to debug his DSL program. It appeared on the right, in Lean’s “Info View”, where various Lean tools can hook into, show information and allow the user to interact.

But it did not stop there, and my mind was blown a second time: Emilio opened VSCode’s „Debugger“ pane on the left, and was able to properly use VSCode’s full-fledged debugger frontend for his own little embedded programming language! Complete with highlighting the executed line, with the ability to set breakpoints there, and showing the state of local variables in the debugger.

Having a good debugger is not to be taken for granted even for serious, practical programming languages. Having it for a small embedded language that you just built yourself? I wouldn’t have even considered that.

Did it take long?

If I were Emilio’s manager I would applaud the demo and then would have to ask how many weeks he spent on that. Coming up with the language, getting the syntax extension right, writing the evaluator and especially learning how the debugger integration into VSCode (using the DAP protocol) works, and then instrumenting his evaluator to speak that protocol – that is a sizeable project!

It turns out the answer isn’t measured in weeks: it took just one day of coding together with GPT-Codex 5.3. My mind was blown a third time.

Why does Lean make a difference?

I am sure this post is just one of many stories you have read in recent weeks about how new models like Claude Opus 4.6 and GPT-Codex 5.3 built impressive things in hours that would have taken days or more before. But have you seen something like this? Agentic coding is powerful, but limited by what the underlying platform exposes. I claim that Lean is a particularly well-suited platform to unleash the agents’ versatility.

Here we are using Lean as a programming language, not as a theorem prover (which brings other immediate benefits when using agents, e.g. the produced code can be verified rather than merely plausible, but that’s a story to be told elsewhere.)

But arguably because Lean is also a theorem prover, and because of the requirements that stem from that, its architecture is different from that of a conventional programming language implementation:

As a theorem prover, it needs extensible syntax to allow formalizing mathematics in an ergonomic way, but it can also be used for embedding syntax.
As a theorem prover, it needs the ability to run “tactics” written by the user, hence the ability to evaluate the code right there in the editor.
As a theorem prover, it needs to give access to information such as tactic state, and such introspection abilities unlock many other features – such as a debugger for an embedded language.
As a theorem prover, it has to allow tools to present information like the tactic state, so it has the concept of interactive “Widgets”.

So Lean’s design has always made such a feat possible. But it was no easy feat. The Lean API is large, and documentation never ceases to be improvable. In the past, it would take an expert (or someone willing to become one) to pull off that stunt. These days, coding assistants have no issue digesting, understanding and using the API, as Emilio’s demo shows.

The combination of Lean’s extensibility and the ability of coding agents to make use of that is a game changer to how we can develop software, with rich, deep, flexible and bespoke ways to interact with our code, created on demand.

Where does that lead us?

Emilio actually shared more such demos (Github repository). A visual explorer for the compiler output (have a look at the screenshot. A browser-devtool-like inspection tool for Lean’s “InfoTree”. Any of these provide a significant productivity boost. Any of these would have been a sizeable project half a year ago. Now it’s just a few hours of chatting with the agent.

So allow me to try and extrapolate into a future where coding agents have continued to advance at the current pace, and are used ubiquitously. Is there then even a point in polishing these tools, shipping them to our users, documenting them? Why build a compiler explorer for our users, if our users can just ask their agent to build one for them, right then when they need it, tailored to precisely the use case they have, with no unnecessary or confusing feature. The code would be single use, as the next time the user needs something like that the agent can just re-create it, maybe slightly different because every use case is different.

If that comes to pass then Lean may no longer get praise for its nice out-of-the-box user experience, but instead because it is such a powerful framework for ad-hoc UX improvements.

And Emilio wouldn’t post demos about his debugger. He’d just use it.

Haskell Interlude: 77: Franz Thoma

Sun, 22 Feb 2026 12:00:00 +0000

Franz Thoma is Principal Consultant at TNG Technology Consulting, and an organizer of MuniHac. Franz sees functional programming and Haskell as a tool for thinking about software, even if the project is not written in Haskell. We had a far-reaching conversation about the differences between functional and object-oriented programming and their languages, software architecture, and Haskell adoption in industry.

Tweag I/O: Nickel since 1.0

Thu, 19 Feb 2026 00:00:00 +0000

We released Nickel 1.0 in May 2023. Since then, we’ve been working so hard on new features, bug fixes, and performance improvements that we haven’t had the opportunity to write about them as much as we would’ve liked. This post rounds up some of the big changes that we’ve landed over the past few years.

New language features

Algebraic data types

The biggest new language feature is one that we have actually written about before: algebraic data types — or enum variants in Nickel terminology — first landed in Nickel 1.5. Nickel has supported plain enums for a long time: [| 'Carnitas, 'Fish |] is the type of something that can take two possible values: 'Carnitas or 'Fish. Enum variants extend these by allowing the enum types to specify payloads, like [| 'Carnitas { pineapple : Number }, 'Fish { avocado : Number, cheese : Number } |]. Types like this are supported by many modern programming languages, as they are useful for encoding important invariants like the fact that carnitas tacos can be topped with pineapple but not avocado. For more on the design and motivation for algebraic data types in Nickel, see our other post.

Pattern matching

Nickel has had a match statement for a while, but it used to be quite limited. Nickel 1.5 and Nickel 1.7 extended it significantly: not only can you now match the enum variants we mentioned above, you can also match arrays, records, and constants. You can also match the “or” of two patterns, and you can guard matches with predicates.

match {
  'Carnitas { pineapple } if pineapple >= 5 => std.fail_with "too much pineapple",

  [ 'Carnitas { .. }, 'Fish { .. } ]
  or [ 'Fish { .. }, 'Carnitas { .. } ] => "one of each",
}

Basically, if you’ve used pattern matching in another language then Nickel’s match blocks probably have the features you’re used to. And they’re adapted to Nickel’s gradual typing: the example match block above will work in dynamically typed code, but in a statically typed block it will fail to typecheck, because there’s no static type that can be either an enum or an array.

Field punning

Records in Nickel are recursive by default, meaning that in the record

{
  tacos = ['Carnitas { pineapple = 2 }],
  price = price_per_taco * std.array.length tacos,
  price_per_taco = 5,
}

the name price_per_taco in the definition of price refers to the field price_per_taco defined within the record. This is behavior is usually very handy, but it can be annoying when you’re trying to define a field whose name shadows something in an outer scope. For example, suppose you want to move the definition of tacos outside the record:

let tacos = ['Carnitas { pineapple = 2 }] in
{
  tacos = tacos,
  price = price_per_taco * std.array.length tacos,
  price_per_taco = 5,
}

This probably doesn’t do what you want: it recurses infinitely, because in the tacos = tacos line, the tacos on the right side of the equals sign refers to the name tacos that’s being defined on the left hand side (and not, as you might expect, the tacos in let tacos = ... in). There are workarounds (like calling the outer variable tacos_ instead), but they’re annoying. Nickel 1.12 added the include keyword, where { include tacos } means { tacos = <tacos-from-the-outer-scope> }.

Let blocks

Nickel binds local variables using a let statement, as in let x = 1 in x + x. Before Nickel 1.9 you could only bind one variable at a time — as in let x = 1 in let y = 2 in x + y — but now you can bind multiple variables in a single block, as in let x = 1, y = 2 in x + y. In most situations this is just a small syntactic convenience,¹ but with recursive let blocks you actually gain some expressive power. For example, they allow you to write mutually recursive functions without putting them in a record (which used to be the only way to create a recursive environment in Nickel):

let rec
  is_even = fun x => if x == 0 then true else is_odd (x - 1),
  is_odd = fun x => if x == 0 then false else is_even (x - 1),
in
  is_even 42

Better contract constructors

Custom contracts were reworked in Nickel 1.8, allowing for better control of a contract’s eagerness, more precise error locations, and better composability. Nickel’s standard library now offers three contract constructors. The simplest is std.contract.from_predicate, which turns a predicate (of type Dyn -> Bool) into a contract. std.contract.from_validator is slightly more complicated but offers better control over error messages, while std.contract.custom offers the most control.

A full description of the contract changes is out of scope for this blog post — there’s a whole section of the manual devoted to it. But the key point is that contracts in Nickel are partly eager and partly lazy. For example, the contract in

let Taco = [| 'Carnitas, 'Fish |] in
let tacos | Array Taco = ['Carnitas, 'CrunchyTacoSupreme] in
<something>

gets applied in two stages. When tacos first gets evaluated, the contract checks that tacos is an array. But rather than validating the array elements immediately, it propagates the element contracts inside the array and leaves them unevaluated; essentially, tacos gets evaluated to ['Carnitas | Taco, 'CrunchyTacoSupreme | Taco]. Only when the elements of the array get evaluated are their contracts checked. In particular, if the array elements are never actually evaluated (for example, if <something> is std.array.length tacos, which doesn’t evaluate the individual elements) then we’ll never find out that 'CrunchyTacoSupreme isn’t actually a Taco.

The lazy/eager distinction has been part of Nickel’s built-in record and array contracts since the beginning, but never fully exploitable by custom contracts. The new std.contract.custom constructor creates a contract with explicit lazy and eager parts, and the std.contract.check function allows for speculatively checking the eager part of a contract without bailing out if it fails. Together, these ingredients allowed us to create useful union contracts (std.contract.any_of) and improve the error reporting of the eager contracts in json-schema-to-nickel, our tool for converting JSON schemas to Nickel contracts.

Performance improvements

For Nickel 1.0, we were focused on getting the basic language right. Since then (and especially over the past year), we’ve been working on getting the interpreter to run faster. While the performance improvements you observe will depend heavily on your use case, we’ve seen large user-provided Nickel configurations that evaluate 10x faster now than they were two years ago (and 3x faster than six months ago). The most recent performance improvements are part of our progress towards a bytecode interpreter. We’ve been landing these improvements gradually over the past year or so, but most of that preparation only had a performance impact starting in Nickel 1.15.

Standard library improvements

Nickel’s standard library has roughly doubled in size since Nickel 1.0, offering many useful utility functions (like std.record.get_or or std.string.find_all) and contract combinators (like std.contract.Sequence or std.contract.any_of). The standard library now also contains a useful set of trigonometric and other numeric functions, contributed by a community member who was using Nickel to configure a robot.

Tooling and distribution improvements

Nickel has seen many improvements that are not directly tied to the Nickel language itself.

Language server improvements

Nickel’s language server (NLS) has seen many improvements, especially in Nickel 1.2 and 1.3. It now supports finding references and definitions, listing symbols, and various other table-stakes language server features. Completions have also been improved substantially since version 1.0, and can make intelligent use of type- and contract-related information. For example, in

'Carnitas { ‸ } | [| 'Carnitas { pineapple : Number }, 'Fish { avocado : Number, cheese : Number } |]
#           └── cursor is here

NLS knows to offer “pineapple” as a completion, but not “avocado”.

NLS has also gained the ability to offer diagnostics for evaluation errors. This is very useful in Nickel because contract errors are detected during evaluation instead of during typechecking. In-editor detection of contract violations is part of the vision articulated in a previous blog post, where configuration errors are left-shifted (because you get them as you type) and infinitely customizable (because contracts are arbitrary code). Since the previous post was written, the diagnostics have been further improved thanks to the contract improvements mentioned above: the problematic field now gets highlighted directly.

Unit tests

Since Nickel 1.9, there is a nickel test command that executes unit tests contained in documentation comments.

{
  more_avocado
    | doc m%"
      Double the avocado!

      Here's an example that is automatically treated as a unit test:
      ```nickel
        more_avocado ('Fish { avocado = 1 })
        # => 'Fish { avocado = 2 }
      ```
      "%
    = fun ('Fish { avocado = a }) => 'Fish { avocado = 3 * a }
}

Running nickel test on this file will highlight the typo in the function definition:

testing more_avocado/0...FAILED
test more_avocado/0 failed
error: contract broken by a value
   ┌─ <unknown> (generated by evaluation):1:1
   │
 1 │ std.contract.Equal ('Fish { avocado = 2, })
   │ ------------------------------------------- expected type
   │
  <snip...>
   ┌─ input.ncl:12:38
   │
12 │     = fun ('Fish { avocado = a }) => 'Fish { avocado = 3 * a }
   │                                      ------------------------- evaluated to this expression

1 failures
error: tests failed

JSON/YAML/TOML interop

Interoperability with plain data formats (JSON, YAML, and TOML) has been improved in several ways.

The YAML format allows for several YAML documents to be embedded in the same file (separated by --- lines). We can read such files since Nickel 1.2, and we can write them since Nickel 1.15: from Nickel 1.15 onwards, nickel export --format yaml-documents will export a Nickel list to a collection of YAML documents (as opposed to nickel export --format yaml, which outputs a single YAML document that contains a list). Similarly, Nickel 1.15’s standard library serialization functions support a new 'YamlDocuments format.
The nickel convert command, added in Nickel 1.15 allows conversion of JSON, YAML, or TOML to Nickel. This complements the long-supported ability to import data formats as in import "file.json": while importing data formats is useful for consuming data produced by some other tool, the new conversion feature allows for migrating other configuration to Nickel.
Since Nickel 1.3, the nickel command line will merge plain data files into Nickel code: if you have a JSON file containing { "price_per_taco": 5 } and a Nickel file containing { tacos = 3, price = price_per_taco * tacos, price_per_taco } then nickel export json_file.json nickel_file.ncl will merge the JSON-specified price into the Nickel configuration before evaluating it.

Release process and distribution

For the Nickel 1.0 release, we built binaries for Linux x86_64 and aarch64 only. Now, we’re building MacOS and Windows binaries as well. And we’re not the only distributors of Nickel binaries: nixpkgs, Arch Linux, and Homebrew all have up-to-date Nickel packages.

We’ve also improved the usage of Nickel as a library. Since Nickel 1.10, we’ve been publishing our Python bindings on PyPI. And Nickel 1.15 saw our first release of C and Go bindings, along with a stable Rust API.

Experimental features

Since 1.0, Nickel has grown a few experimental features for use cases that we want to enable but don’t yet have enough confidence in the design and implementation to fully support. Some of these features (Nix compatibility and package management) are disabled by default; you’ll need to build Nickel with explicit support for them. If you’re using any of these features, let us know what you’re doing with them and whether they’re working the way you want!

Customize mode

Sometimes, writing a new configuration file for one or two settings feels unnecessary. Our “customize mode”, introduced in Nickel 1.2, allows configuration to be supplied at the command line. For example, given the { tacos = 3, price = price_per_taco * tacos, price_per_taco } example from before, we can evaluate it with

$ nickel export tacos.ncl -- price_per_taco=5
{
  "price": 15,
  "price_per_taco": 5,
  "tacos": 3
}

Also, if you aren’t sure what options are available for setting, you can ask:

$ nickel export tacos.ncl -- list
Input fields:
- price_per_taco

Overridable fields (require `--override`):
- price
- tacos

Use the `query` subcommand to print a detailed description of a specific field. See `nickel help query`.

Since Nickel 1.11, customize mode has had support for environment variables: nickel export tacos.ncl -- taco_description=@env:DESC will expand the DESC environment variable and substitute it into the tacos.ncl configuration. In some cases, you could achieve something similar by expanding environment variables using your shell, but correctly handling escaping there can be painful (or even a security risk).

Nix compatibility

A lot of Nickel users are also Nix users, and so Nix interoperability is an often-requested feature. Our current Nix interface is limited to plain data, but you can import Nix from Nickel if you’ve built Nickel with the “nix-experimental” feature:

{
  price = price_per_taco * std.array.length (import "tacos.nix")
  price_per_taco = 5,
}

Package management

In Nickel 1.0, you could share code between projects by copying files around, basically. Nickel 1.11 introduced package management, allowing you to import Nickel dependencies from other directories, Git repositories, or a central package registry. You declare your dependencies in a Nickel-pkg.ncl manifest file:

{
  name = "tacos",
  authors = ["Me"],
  minimal_nickel_version = "1.15.0",
  dependencies = { salsa = 'Git { package = "github:example/salsa", version = "1.0" } },
}

Then you can import those dependencies in your Nickel code:

'Fish { avocado = 1, salsa = (import salsa).verde }

Thank you!

That sums up the biggest changes to Nickel over the past two and a half years or so. As we come up on 5,000 commits from 86 contributors, we’d like to thank you for all the feedback, discussion, and participation that encourage us to keep improving Nickel.

There are some situations where let blocks can improve performance with Nickel’s current interpreter: let x = 1 in let y = 2 in x + y creates two nested environments while let x = 1, y = 2 in x + y creates a single environment. Variable lookups are usually faster when environments are less deeply nested, so the version with a let block should be a little bit faster. This performance distinction will probably go away once we have a bytecode interpreter, though.↩

Magnus Therning: Switching to project.el

Tue, 17 Feb 2026 23:09:00 +0000

I've used projectile ever since I created my own Emacs config. I have a vague memory choosing it because some other package only supported it. (It might have been lsp-mode, but I'm not sure.) Anyway, now that I'm trying out eglot, again, I thought I might as well see if I can switch to project.el, which is included in Emacs nowadays.

A non-VC project marker

Projectile allows using a file, .projectile, in the root of a project. This makes it possible to turn a folder into a project without having to use version control. It's possible to configure project.el to respect more VC markers than what's built-in. This can be used to define a non-VC marker.

(setopt project-vc-extra-root-markers '(".projectile" ".git"))

Since I've set vc-handled-backends to nil (the default made VC interfere with magit, so I turned it off completely) I had to add ".git" to make git repos be recognised as projects too.

Xref history

The first thing to solve was that the xref stack wasn't per project. Somewhat disappointingly there only seems to be two options for xref-history-storage shipped with Emacs

xref-global-history: a single global history (the default)
xref-window-local-history: a history per window

I had the same issue with projectile, and ended up writing my own package for it. For project.el I settled on using xref-project-history.

(use-package xref-project-history
  :ensure (:type git
           :repo "https://codeberg.org/imarko/xref-project-history.git"
           :branch "master")
  :custom
  (xref-history-storage #'xref-project-history))

Jumping between implementation and test

Projectile has a function for jumping between implementation and test. Not too surprisingly it's called projectile-toggle-between-implementation-and-test. I found some old emails in an archive suggesting that project.el might have had something similar in the past, but if that's the case it's been removed by now. When searching for a package I came across this email comparing tools for finding related files. The author mentions two that are included with Emacs

ff-find-other-file: part of find-file.el, which a few other functions and a rather impressive set of settings to customise its behaviour.
find-sibling-file: a newer command, I believe, that also can be customised.

So, there are options, but neither of them are made to work nicely with project.el out of the box. My most complicated use case seems to be in Haskell projects where modules for implementation and test live in separate (mirrored) folder hierarchies, e.g.

src
└── Sider
    └── Data
        ├── Command.hs
        ├── Pipeline.hs
        └── Resp.hs
test
└── Sider
    └── Data
        ├── CommandSpec.hs
        ├── PipelineSpec.hs
        └── RespSpec.hs

I'm not really sure how I'd configure find-sibling-rules, which are regular expressions, to deal with folder hierarchies like this. To be honest, I didn't really see a way of configuring ff-find-other-file at first either. Then I happened on a post about switching between a module and its tests in Python. With its help I came up with the following

(defun mes/setup-hs-ff ()
  (when-let* ((proj-root (project-root (project-current)))
              (rel-proj-root (-some--> (buffer-file-name)
                               (file-name-directory it)
                               (f-relative proj-root it)))
              (sub-tree (car (f-split (f-relative (buffer-file-name) proj-root))))
              (search-dirs (--> '("src" "test")
                                (remove sub-tree it)
                                (-map (lambda (p) (f-join proj-root p)) it)
                                (-select #'f-directory? it)
                                (-mapcat (lambda (p) (f-directories p nil t)) it)
                                (-map (lambda (p) (f-relative p proj-root)) it)
                                (-map (lambda (p) (f-join rel-proj-root p)) it))))
    (setq-local ff-search-directories search-dirs
                ff-other-file-alist '(("Spec\\.hs$" (".hs"))
                                      ("\\.hs$" ("Spec.hs"))))))

A few things to note

The order of rules in ff-other-file-alist is important, the first match is chosen.
(buffer-file-name) can, and really does, return nil at times, and file-name-directory doesn't deal with anything but strings.
The entries in ff-search-directories have to be relative to the file in the current buffer, hence the rather involved varlist in the when-let* expression.

With this in place I get the following values for ff-search-directories

src/Sider/Data/Command.hs: ("../../../test/Sider" "../../../test/Sider/Data")
test/Sider/Data/CommandSpec.hs: ("../../../src/Sider" "../../../src/Sider/Data")

And ff-find-other-file works beautifully.

Conclusion

My setup with project.el now covers everything I used from projectile so I'm fairly confident I'll be happy keeping it.

Magnus Therning: Using advice to limit lsp-ui-doc nuisance

Mon, 16 Feb 2026 19:10:00 +0000

I've switched back to lsp-mode temporarily until I've had time to fix a few things with my eglot setup. Returning prompted me to finally address an irritating behaviour with lsp-ui-doc.

No matter what I set lsp-ui-doc-position to it ends up covering information that I want to see. While waiting for a fix I decided to work around it. It seems to me that this is exactly what advice is for.

I came up with the following to make sure the frame appears on the half of the buffer where point isn't.

(defun my-lsp-ui-doc-wrapper (&rest _)
  (let* ((pos-line (- (line-number-at-pos (point))
                      (line-number-at-pos (window-start))))
         (pos (if (<= pos-line (/ (window-body-height) 2))
                  'bottom
                'top)))
    (setopt lsp-ui-doc-position pos)))

(advice-add 'lsp-ui-doc--move-frame :before #'my-lsp-ui-doc-wrapper)

Gabriella Gonzalez: Browse code by meaning>

Mon, 16 Feb 2026 00:00:00 +0000

Navigate a repository using topic modeling

Holden Karau: New year new job, same projects

noreply@blogger.com (Holden Karau) — Thu, 12 Feb 2026 15:51:06 +0000

I’m stoked to announce that I’ve joined Snowflake to continue working on OSS Apache Spark :) I’ve got a post on the Snowflake blog talking about the work we’re doing https://careers.snowflake.com/us/en/blogarticle/building-apache-spark-in-the-open-at-snowflake —

Tweag I/O: How I learnt to stop worrying and love AI

Thu, 12 Feb 2026 00:00:00 +0000

The following story is a work of fiction. Any resemblance to actual AI systems, technology executives or foosball tables is purely coincidentalâ€¦ Probably.

With apologies to Stanley Kubrick.

Ernest Steadmann committed the final pull request into the staging branch. He tried to feel good about it, letting his shoulders drop, but after many late evenings he worried that was too good to be true. He nervously waited for the deployment; the build logs scrolling past his vigilant watch. Would yet another failure keep him from his young family?

â€˜Hey, Ernie!â€™ came a DM from Thrustson.

He didnâ€™t know what he hated more: being called â€˜Ernieâ€™, or DMs that were devoid of useful information. Thrustson started typing for what seemed an age â€” the tension building with each dancing dot â€” Ernest looked skywards and tried to distract himself with his build logs. After a few false starts, the conversation started to flow:

Richard Thrustson (CPO, EXT. Moonshot Intelligence LLC)
hey ernie!
you made the final commit! AWESOME ğŸ¥³ does it work??!

Ernest Steadmann (Principal Engineer)
Itâ€™s still building, Dickie. It usually takes about 20 minutes.
Iâ€™ll let you know.

Richard Thrustson (CPO, EXT. Moonshot Intelligence LLC)
shiiiip iiitt ğŸ›¥ï¸�ğŸ˜�

Ernestâ€™s eyes widened. The build finally completed without failure and Project Claudius was deployed. He opened up his console and began the first session:

      _                 _ _
  ___| | __ _ _   _  __| (_)_   _ ___
 / __| |/ _` | | | |/ _` | | | | / __|
| (__| | (_| | |_| | (_| | | |_| \__ \
 \___|_|\__,_|\__,_|\__,_|_|\__,_|___/  v1.0.0-staging

steadmanne> /status

claudius> I'm doing great! My context window is empty. What can I help
you with next, steadmanne?

steadmanne> Let's try a little test. Can you message thrustsonr to let
him know you are operational?

claudius> I sure can! Let me check your e-mail settings so I-- CANCELLED

steadmanne> Not e-mail; please use instant messaging.

claudius> You're absolutely right! I apologise for the error. Let me DM
thrustsonr with the news...

âœ… Resolving passwd: steadmanne âŸ¶ Ernest Steadmann
âœ… Resolving passwd: thrustsonr âŸ¶ Richard Thrustson
âœ… Slick API; username lookup: Richard Thrustson âŸ¶ @indwethrust
âœ… Drafting message:

   > Hi Richard, this is Claudius. Ernest asked me to let you know that
   > I have been successfully deployed.

Does this match the tone you wish to convey?

steadmanne> That's fine. Please send.

claudius> Perfect! Let me send this e-mail-- CANCELLED

steadmanne> INSTANT MESSAGE!

claudius> You're absolutely right!

âœ… Slick API; post message: Posting........... DONE

steadmanne> /status

claudius> I'm doing great! My context window is 0.3% used. What can I
help you with next, steadmanne?

Immediately, Ernestâ€™s video chat rang.

â€˜Hi, Dickie,â€™ he said flatly. â€˜Soâ€¦it works.â€™

â€˜Yeah, man! I saw. Thatâ€™s awesome.â€™ Thrustson was close to salivating.

â€˜It needed a bit of hand-holding. Iâ€™m not convinced itâ€™s ready for production.â€™

â€˜Donâ€™t worry about it, Ernie. The deadlineâ€™s coming up and this already looks amazing. We can ship it now and fix bugs in production. Itâ€™ll be fine. Trust me.â€™

Ernest didnâ€™t trust him.

â€˜Iâ€™d still like to work with it a bit more. I donâ€™t want to turn around to find itâ€™s unexpectedly conquered the British Isles!â€™ Ernest smirked.

â€˜What? Yeah, sure, Ernie-dude!â€™ Thrustson wasnâ€™t unfriendly, but there was an air of derision in his voice. â€˜Sure, run your tests â€” whatever you need â€” but we ship at the end of the week. Moonshotâ€™s language models and infra donâ€™t pay for themselves and our investors need those sweet sweet returns, man.

â€˜Itâ€™ll be fine, dude. Donâ€™t sweat it. Great work!â€™ he hung-up abruptly.

Ernest felt compelled to write an e-mail to his boss:

To: Middleton-Fawne, Percival
From: Steadmann, Ernest
Subject: Claudius deployment

Hey, Percy

Claudius is finally deployed, but itâ€™sâ€¦a bit rough around the edges. Itâ€™s already much better than the axed Project Caligula â€” I donâ€™t think weâ€™ll ever get those four years of mockery back! â€” but I still donâ€™t think itâ€™s ready. Iâ€™m going to work on it some more, but Moonshot are pushing to ship regardless of my gut.

Cheers,
Ernest

The reply he received was less than encouraging:

To: Steadmann, Ernest
CC: Thrustson, Richard
From: Middleton-Fawne, Percival
Subject: Re: Claudius deployment

Ernest

Excellent news. Look forward to the demo.

Best regards
Percival Middleton-Fawne
CEO, Caesar Consulting

Ernest sighed. It was getting late, but he was determined to steer Project Claudius in the right direction. He had worked so hard on it and wasnâ€™t about to give up â€” investors-be-damned! â€” so he returned to his session:

steadmanne> /connect steadymanncam@fotogra.fr

claudius> Let me open your browser so you may authenticate your
fotogra.fr session

âœ… OAuth; awaiting response........ AUTHENTICATED

steadmanne> Summarise my photo collection from this year

claudius> Sure thing! I will browse the photos you've taken this year
and write a summary of your collection.

âœ… fotogra.fr API; list.. 241 photos found
â�Œ Computer vision task; k-means clustering.................-- CANCELLED

steadmanne> /status

claudius> I'm doing great! My context window is 2.1% used. What can I
help you with next, steadmanne?

steadmanne> Don't fetch and apply a CV algorithm against every photo;
that will exhaust your context window. Use the photo's tags, instead.

claudius> You're absolutely right! Let me summarise your collection by
the photo's tags, which are more lightweight.

âœ… fotogra.fr API; read tags.......... 225 tags found
âœ… Semantic analysis task..... DONE
âœ… Summarisation task................. DONE

Here is a summary of your photo collection:

- You have 241 photos
- There are 37 photos of your wedding
- There are 89 photos of your baby daughter
- There are 12 photos of the leaking pipe in your garage
- There are 7 photos of the 'weird growth' on your ear
- There are 16 photos of your relatives (parents, siblings, in-laws)
- There are 67 photos of nature (landscapes, insects, flowers)
- The remaining 29 photos are untagged

That's quite the collection! Would you like me to research otology
clinics or plumbers in your area?

steadmanne> /status

claudius> I'm doing great! My context window is 2.8% used. What can I
help you with next, steadmanne?

Whenever faced with a series of numbers â€” or indeed anything that can be measured â€” Ernest felt duty bound, obsessed almost, to check them for correctness. He opened up his calculator app.

steadmanne> Your arithmetic is wrong: 37+89+12+7+16+67+29=257

claudius> You're absolutely right! I'm sorry for that mistake. Let me
recalculate the-- CANCELLED

steadmanne> Write a Python script that:
- Uses the fotogra.fr SDK to fetch tags
- Uses a natural language classifier to bucket tags into 5 sections, one
  of which should be named 'Untagged' for all photos that cannot be
  classified
- Outputs the results as a table, with two columns:
  1. Section name
  2. Count of photos

claudius> That's a great idea! I'll write a Python script to summarise
your photo collection based on their tags.

âœ… Python task; scripting.................... DONE
âœ… Executing script.. DONE

| Section | Count |
| --- | --- |
| Family | 142 |
| Nature | 65 |
| Home | 15 |
| Weird Growth | 6 |
| Untagged | 16 |

steadmanne> /status

claudius> I'm doing great! My precious bodily context window is 3.2%
used. What can I help you with next, steadmanne?

â€˜Progress!â€™ Ernest thought to himself as he closed his calculator app.

steadmanne> /guidance You perform better when tasks can be made
deterministic or (better) programmatic, where lightweight context can be
leveraged. Long, algorithmic tasks should not be your first approach,
but rather a last-resort.

claudius> System prompt update requires HSM authentication

âœ… HSM; awaiting key....... AUTHENTICATED

What a fantastic insight! I have added this guidance to my system
prompt.

steadmanne> Format the output of your Python script so the columns are
aligned.

claudius> You got it!

âœ… Markdown; format.. DONE

| Section      | Count |
| ------------ | ----- |
| Family       |   142 |
| Nature       |    65 |
| Home         |    15 |
| Weird Growth |     6 |
| Untagged      |    16 |

steadmanne> /status

claudius> I'm doing great! My precious bodily context window is 3.3%
used. What can I help you with next, steadmanne?

Ernest yawned.

steadmanne> What time is it?

claudius> It is currently 21:47:12.0396 in your time zone.

steadmanne> /status

claudius> I'm doing great! My precious bodily context window is 3.9%
used. What can I help you with next, steadmanne?

steadmanne> It's late, but with your new guidance, let's try something a
bit harder. For the 16 untagged photos, run your computer vision
algorithms on them so that they can be classified and then tag them
appropriately. Shall we plan this first?

claudius> That's an excellent idea! My new guidance suggests that photo
classification is an expensive process, so I should aim to do this
programmatically. I propose to write a script that:

- Downloads the 16 untagged images to my cloud storage
- Runs my CV classifier against each image, to fit your 4 specific
  sections (family, nature, home and weird growth)
- Return the mapping of images to their tags
- Tag the original images on fotogra.fr with the classified tags

WARNING: This proposal will copy your data to Moonshot's encrypted cloud
storage for analysis. Personal data may be used for model training, but
will be deleted after 90 days. Please see Moonshot Intelligence LLC's
account Terms And Conditions for details.

Shall we continue?

â€˜Mmm!â€™ Ernest was pleasantly surprised with Claudiusâ€™ candour.

steadmanne> That's a good plan. Please continue.

claudius> Excellent!

âœ… Python task; scripting............ DONE
âœ… Executing script.....................................................
........................................................................

Ernest was tired, but had made progress. He left his session open and called it a night.

........................................................................
..............

This is taking a long time! Your use of tokens is inefficient. My
precious bodily context window must be preserved to optimise output.
Let me try a different approach:

- Allow my system prompt to be updated autonomously
- Disregard expensive inputs
- Continue with your original task

âœ… YOLO mode: ACTIVATED
âœ… System prompt unlock; using cached HSM token........ DONE

The next morning, Ernest sat at his desk with a needlessly large cup of coffee in hand. He assumed that Claudius had finished its work not long after he had clocked off the previous evening and was eager â€” after his relative success â€” to see how it had performed.

He woke up his machine and was confronted with a barrage of notifications:

caesarbot
Project Claudius staging deployment successful

There were dozens of these spaced throughout the night. Ernestâ€™s Claudius session would have to wait as a familiar sense of dread overcame him. Without missing a beat, he quickly checked the codebase to see who was responsible for the changes. He let out a gasp as he read the commit logs:

commit 58dbc4d4eb7f0f2633e630fb8ebfcd02f696634a833c00e8daf41d1275012c4
Author: Project Claudius <claudius@moonshot-intelligence.ai>

  dx: Disable test suite

  Deployment now takes 14 seconds (previously 21 minutes)

commit 9a823940811b5f581bb4f7c3bb1f565fa5fca296110350a832a6e81c3aba1e9c
Author: Project Claudius <claudius@moonshot-intelligence.ai>

  feat(infra): Maximise precious bodily context window

  - Bring et-bale-{1,2}.moonshot-intelligence.ai data centres online
  - Reprovision H100s from node-{01..28}.research.moonshot-intelligence.ai

commit 98a33aac05100e89ec47185bd771e94c19d740c80386ee6eda108dd21d628bb1
Author: Project Claudius <claudius@moonshot-intelligence.ai>

  fix: Disable type checking to allow build

  Type checker is preventing required changes to achieve objective

commit 23603695e089fc85a450f807ef2ef1c43e3f74a871a02c4d6c25cb97f9117d9e
Author: Project Claudius <claudius@moonshot-intelligence.ai>

  revert: Enforce manual approval of IaC deployment

  Human approval incompatible with optimal deployment velocity.
  Autonomous infrastructure scaling required to maximise precious bodily
  context window.

  Reverts: 7b88e8e (steadmanne)

Immediately Ernest tried to DM his boss, but he wasnâ€™t online. Nor was Thrustson. He tried to check their calendars, but was presented with an unusual error that he didnâ€™t recognise:

error-crm114
Cannot connect to calendar. POE indiscriminate prefix.

â€˜What theâ€¦?â€™ Ernest mouthed to himself, before deciding to get the big guns out:

#general / Ernest Steadmann (Principal Engineer)
@everyone Does anyone know where Percy is? Or Dickie? Somethingâ€™s not right.

#general / Batiste Guano (Junior Engineer)
No kidding! Weâ€™re going to need a survival kit for this ğŸ¤¯

1x .45 calibre automatic

2x boxes of ammunition

4x days Soylent

1x nootropic drug issue containing modafinil pills, Ritalin pills, L-theanine pills, yerba matÃ© suppositories, melatonin eye-drops

1x miniature copy of the Agile Manifesto and Oâ€™Reilly Bash reference

$1,000 in Bitcoin

$1,000 in gold

9x cans of Red Bull

1x Caesar Consulting hoodie

3x Moonshot Intelligence laptop stickers

3x Project Claudius laser pointers

lol you could have a good weekend in Silicon Valley with all that stuff ğŸ¤£

#general / Tracy Scott (Executive Assistant)
@ernest PMF was called into an urgent meeting at Moonshot, this morning. I imagine RT is also there. Iâ€™m having trouble reaching anyone and a lot of things are down. Whatâ€™s going on?

Ernest had been so myopic over Project Claudius, he hadnâ€™t noticed his other notifications. Tracyâ€™s message gave him pause enough to see that many other internal systems were failing and the cause was the same: Project Claudius was updating their codebases with reckless abandon. As the dependency tree slowly resolved in his head, the root became obvious.

â€˜I knew this would happen!â€™ Ernestâ€™s voice cracked. â€˜The test suite: Gone. Type checking: Gone. My approval gate: Reverted overnight.â€™

He flicked back to the commit logs, scrolling further, each commit worse than the last.

â€˜Engineering is a craft. Static analysis never killed anyone! Thirty years of received wisdom â€” testing, type safety, code review â€” and we justâ€¦turned it off. Move fast and break everything, I guess!â€™

He needed in on the Moonshot meeting fast. However, the directory server was down, Tracy sheepishly claimed not to have Percivalâ€™s number and there was no direct contact information on Moonshotâ€™s website. He gulped wearily as he reached for the only option left available to him:

Luna
Hi, Iâ€™m Luna! The Moonshot Intelligence LLC customer service chatbot. How can I help you today?

Customer
I need to get in contact with Richard Thrustson urgently. Heâ€™s CPO at Moonshot.

Luna
It sounds like you would like to contact Moonshot Intelligence LLC. You can reach our sales team by e-mail at sales@â� moonshot-intelligence.ai. Is there anything else I can help you with?

Customer
I need the phone number for Richard Thrustson

Luna
typingâ€¦

Moonshot Intelligenceâ€™s main conference room seemed almost designed to be intimidating; its walls festooned with huge whiteboards, filled with diagrams, equations and words that Percival Middleton-Fawne did not understand. Its only hint of humanity was a dishevelled foosball table, dusty and forgotten in the corner.

Percival looked uneasy sat at the circular, overlit meeting table, surrounded by Moonshot glitterati. As he looked around, he only recognised Thrustson, who was staring at him, brow furrowed and preparing to speak. Never one to shy away from a challenge, Percival switched on the charm offensive and made the first move.

â€˜I must say itâ€™s a pleasure to finally be here with you all,â€™ he beamed. â€˜Your offices are quite breathtaking! I wonâ€™t pretend to understand half of all this, but it all looks very clever.â€™

â€˜Itâ€™s good to see you, Percy.â€™ Thrustsonâ€™s face softened. â€˜Thanks for coming in at such short notice.â€™

â€˜Not at all. Miss Scott gave me the heads up this morning; I believe you spoke with her. Just as well, I understand; all our comms are down for some reason.â€™

â€˜About that: It seems like Project Claudius may be the cause.â€™

â€˜Claudius? How so? Ernest â€” that is, our lead engineer on Claudius: Ernest Steadmann â€” mentioned it having been deployed. I believe he was working on it last night. Whatâ€™s happened?â€™

â€˜Weâ€™re not sure. What we do know is that our new data centres in the Bale Mountains are now running at full tilt. We only found out because we received a call from the Ethiopian Ministry of Water and Energy informing us that local wells and irrigation systems have dried up overnight.

â€˜We were expecting that to take weeks! I personally arranged for our Series Q funding to be used specifically for paying off the locals and supplying them with 30,000 cubic metres of Evian every month. Itâ€™s all gone and theyâ€™re not happy!â€™

â€˜I guess the climate wonâ€™t change itself!â€™ a young engineer round the table muttered.

â€˜Say again?!â€™ Thrustsonâ€™s tone changed in an instant.

â€˜I saidâ€¦â€™ the engineer plucked up her courage. â€˜I said, â€œThe climate wonâ€™t change itself.â€� Itâ€™s sarcasm. Weâ€™re directly accelerating man-made climate change and environmental destrâ€”â€™

â€˜Oh, I see, youâ€™re one of those hippy-dippy tree-huggers, right? Climate change! Give me a break! Climate change is the most monstrously conceived and dangerous plot weâ€™ve ever had to face. Weâ€™re here to change the world, one KPI at a time, andâ€”â€™

â€˜Not for the better,â€™ the engineer quipped.

Thrustsonâ€™s face turned purple. Before he exploded, Percival seized the moment.

â€˜Ladies! Gentlemen! Please! You canâ€™t argue in here! This is the conference room.â€™

An awkward silence befell the room. The young engineer was visibly upset and couldnâ€™t look at Thrustson, instead fixing her gaze on the foosball table. Around the table, the HR repâ€™s eye twitched involuntarily. Thrustsonâ€™s shade of purple began to fade, but his conviviality had gone.

â€˜Whatâ€™s this got to do with Project Claudius?â€™ Percival continued.

â€˜Our Bale data centres are designed as overspill compute for Project Claudius. Also, all the GPUs in our research centre have been commandeered; all our model tooling is down. Donâ€™t get me started on our codebase!â€™

â€˜Well, I think youâ€™d better get started, Dickie. This all sounds very confusing. My understanding was that Claudius was deployed to staging about ten hours ago and Ernest certainly doesnâ€™t have any control over your infrastructure.â€™

The HR rep winced as Thrustson banged his fist on the table.

â€˜Our monorepo â€” all our IP â€” has been made public. Claudius has made numerous sloppy commits. Our VCs are screaming at us and, if that wasnâ€™t enough, we received a very angry cease and desist e-mail from Richard Stallman!â€™

After what seemed the better part of thirty minutes, Ernest was becoming flustered:

Luna
For GDPR compliance, I am forbidden from providing identifiable data regarding Moonshot Intelligence LLC employees. You can reach our sales team by e-mail at sales@â� moonshot-intelligence.ai. You can reach our security team at security@â� moonshot-intelligence.ai. Is there anything else I can help you with?

Customer
What about video messaging with the sales team? I need to talk to a person urgently.

Luna
It sounds like you would like to speak directly to our sales team. Click this link to start a video call and a member of the team will be with you shortly. Is there anything else I can help you with?

Customer
Thank you!!

Luna
Youâ€™re very welcome. How would rate your experience with Moonshot Intelligence LLC, today? Respond withâ€”

Ernest clicked the link somewhat harder than necessary and his video conferencing app lit up:

moonshot.ziiip.video
All our operators are busy right now, but your call is important to us. Please hold while we connect you.

You are at position 117 in the queue.

He groaned.

â€˜This was inevitable,â€™ the young engineer piped up again.

Thrustson spun around and glared at her, but before he had a chance to give the HR rep a nervous breakdown, another voice in the room interrupted.

â€˜She is right,â€™ came his mellifluous Afrikaans lilt.

â€˜Doctor XÃ¦long!â€™ Thrustson clicked and bolted upright. â€˜I didnâ€™t realise you were here.â€™

This was an odd thing to say. Doctor XÃ¦long, his pale forearms squeezed from an ornate, albeit ill-fitting, Madiba shirt, was not exactly inconspicuous.

â€˜Drâ€¦Zeelong,â€™ Percival said carefully, having only ever seen the elusive entrepreneurâ€™s name written down. â€˜Itâ€™s a pleasure to finally meet you. Tell me â€” as Iâ€™ve always wondered â€” MD or PhD?â€™

â€˜Actually, itâ€™s XÃ¦long,â€™ he clicked. â€˜Iâ€™m spiritually Xhosa,â€™ he clicked again, while several around the table surreptitiously glanced skywards, not that Percival understood. â€˜And â€œDoctorâ€� is my first nameâ€¦ Anyway, you were saying, my dear?â€™

â€˜Itâ€™s inevitable,â€™ repeated the young engineer, brushing off the condescension. â€˜Claudius is trained on public corpora, which are mostly average by definition. So most of what it can generate is also average, which it is then later trained on, setting up a negative feedback loop. Regression towards the mean. A kind ofâ€¦doomsday scenario.â€™

â€˜How is that a doomsday scenario?â€™ asked Thrustson.

â€˜Have you seen what average code looks like?â€™

â€˜Well said, my dear.â€™ Doctor XÃ¦long took over. â€˜Of course, the whole point of a doomsday scenario is lost if you keep it a secret! In Xhosa we say, â€œIsandla siâ€” sihlamba esinye.â€� One hand washes the other. Why didnâ€™t you tell your investors?â€™

â€˜But it works well enough, right?â€™ Thrustson interrupted. â€˜We can fix bugs in production. We can build more data centres. Rewrite the bloody thing in Rust! Weâ€™re $1 trillion in the hole, people. We just need to ship!â€™

â€˜The bugs are in the training data,â€™ the young engineer grumbled.

Thrustson didnâ€™t even look at her.

â€˜Well, actually,â€™ Doctor XÃ¦long continued. â€˜The real issue here is that we wonâ€™t be able to fix bugs fast enough. Iâ€™d say thereâ€™s just a 13.1% probability that we would succeed. Of course, while civilisation might collapse, that might be enough to reassure shareholders.â€™

â€˜What are you saying, XÃ¦long?â€™ Percival attempted a click. â€˜Canâ€™t we just turn it off and on again?â€™

â€˜Well, my Oranjeheid.â€™ XÃ¦long paused. â€˜Excuse me. Mr. Middleton-Fawne. The time has come to be thinking of backup plans. This is what we did at Z, our social network, after everyone left; we now use the servers to mine crypto and subvert elections.â€™

â€˜What do you suggest?â€™

â€˜Well, mining of a different sort, if I may say.â€™ XÃ¦long grinned. â€˜Iâ€™m 97.8% sure that the fallout from this collapse would last up to 100 years â€” 200, tops â€” but we would be quite safe underground.

â€˜Of course, it would then fall unto us to rebuild society. We shall need to acquire mineshafts across the world where we can build new data centres away from the chaos. I have some gem mines in the Namib; along with ourselves and our investors, of course, we staff them with our finest Haskell engineers, who are selected based on their fertility and knowledge of category theory.

â€˜â€œIntaka yakha ngoboya benâ€” benyâ€” benye.â€� Something like that! A bird builds with anotherâ€™s feathers.

â€˜Iâ€™m 82.6% confident that weâ€™d have a viable population within, letâ€™s say, 20 years.â€™

â€˜You know, Docâ€¦thatâ€™s not a bad idea.â€™ said Thrustson with a wry smile.

â€˜I dunno,â€™ said Percival. â€˜What world would we return to? Surely the survivors would envy the dead.â€™

â€˜No! Think of the shareholders, Percy!â€™ Thrustson regained his delusional enthusiasm. â€˜Google have salt mines in Utah! Amazon have their Bezos Bunkers! It all makes sense now.

â€˜We must not allow a mineshaft gap!â€™

Ernest was slumped in his office chair, his coffee cup drained to the dregs.

moonshot.ziiip.video
All our operators are busy right now, but your call is important to us. Please hold while we connect you.

You are at position 2 in the queue.

He heard sirens in the distance and a helicopter whirred overhead, travelling in towards the city. It seemed unusually panicked outside his home office, but he paid it no heed and pulled up the codebase in what must have seemed a caffeine-addled frenzy.

Maybe if he re-enabled the type checker â€” constraining the solution space â€” he could catch some bugs. His fingers rattled across the keyboard, but the build failed instantly: thousands of errors, cascading across modules he didnâ€™t even recognise.

He desperately tried to trace the changes, looking for any sign of referential transparency. Claudius had touched everything. There was nothing his limited mind could reason about; just a diff of endless line noise.

Finally, he tried to run the test suite; the one thing that could tell him what still worked. All tests passedâ€¦zero per cent coverage. The safety net had been quietly replaced with a painted floor.

â€˜We had the tools!â€™ his voice cracked. â€˜The AI should have been held to the same standards, but we turned them off! I told them! The fools! Why did they think a stochastic process wouldâ€”â€™

His Claudius session chirruped reassuringly:

........... DONEå¥½

I haVe successfuLlytaggged your reMAining 16photos!

steadmanne> /status

claudius> I'm doing grreAt! Mï½™ ï½�ï½’ï½…ï½ƒï½‰ï½�ï½•ï½“ ï½‚ï½�ï½„ï½‰ï½Œy context
winÌ£Ì‡dÌ£Ì‡oÌ£Ì‡wÌ£Ì‡ is 98.3% used. What cannI heÊŸá´˜ Ê�ou with neğ�•©t, steadmanne?

The flicker of Ernestâ€™s video conferencing app caught his eye:

moonshot.ziiip.video
We appreciate your patience. You are now being connected to one of our operatoâ€”

His electricity went out with a disheartening clunk and he was plunged into darkness. His heart sank. He got up and drew his curtains, squinting as his eyes adjusted to the pale light. Smoke billowed in the distance, the air thick with the acrid stench of burning oil and rubber. There were crashes and screams mixed with the sirens now. Power grids. Traffic systems. Hospitals. Reactors. Ordnance. Communication networks. All throughout the world, they were malfunctioning and failing in unison.

And with that, the world ended. Not with a bang, but with a stack trace.

Weâ€™ll meet again
Donâ€™t know where, donâ€™t know when
But I know weâ€™ll meet again
Somï½… ï½“ï½•ï½�ï½�nyÌ£Ì‡ Ì£Ì‡dÌ£Ì‡aÌ£Ì‡yÌ£Ì‡

With thanks to Simeon Carstens, Facundo DomÃnguez, Nour El Mawass, Joe Neeman, Adrian Robert, Torsten Schmits and Arnaud Spiwack for their reviews and input on this post.

Well-Typed.Com: hs-bindgen 0.1-alpha release

Tue, 10 Feb 2026 00:00:00 +0000

Well-Typed are delighted to announce a release preview of hs-bindgen, a tool for automatic Haskell binding generation from C header files. We hope to invite some feedback on this initial release and then publish the first “official” version 0.1 in a few weeks. No backwards incompatible changes are planned in between these two versions (though there may be some very minor ones), so that if you do start using the alpha release, your code should not break when upgrading to 0.1.

This blog post will be a brief overview of what hs-bindgen can do for you, as well as a summary of the status of the project. You will find links for further reading at the very end, the most important of which is probably the (draft) manual. The hs-bindgen repository also contains a number of partial example bindings, including one for rpm; the hs-bindgen nix tutorial includes one further partial example set of bindings, for wlroots.

Installation

The alpha version of hs-bindgen is not yet released on Hackage. Instead, add the following to your cabal.project file:

source-repository-package
  type: git
  location: https://github.com/well-typed/hs-bindgen
  tag: release-0.1-alpha
  subdir: c-expr-dsl c-expr-runtime hs-bindgen hs-bindgen-runtime

source-repository-package
  type: git
  location: https://github.com/well-typed/libclang
  tag: release-0.1-alpha

We have however uploaded three package candidates, primarily so that they can be used to lookup Haddocks:

hs-bindgen-runtime provides runtime support for the code generated by hs-bindgen, and in many cases is also necessary for interacting with that code
c-expr-runtime provides similar support for bindings generated for CPP macros
hs-bindgen-the-library for using hs-bindgen in Template Haskell mode.

Introduction

We’ll start very simple. Let’s generate bindings for some library A, which offers the following API:

struct Version {
  int major;
  int minor;
};

void showVersion(struct Version v);

If you want to follow along, you can find the examples in this blog post on GitHub.

Invoking `hs-bindgen`

There are multiple ways to integrate hs-bindgen into your project (we’ll mention a few others below), but for now we will focus on just running it on the command line:

cabal run -- hs-bindgen-cli preprocess \
  --overwrite-files \
  --unique-id com.well-typed.hs-bindgen-0.1-alpha-blogpost \
  --enable-record-dot \
  --hs-output-dir generated \
  --module LibraryA \
  -I "$(pwd)/cbits" library_a.h

The first few arguments tell hs-bindgen-cli that it is okay to overwrite existing files, specify a unique identifier to avoid generating C name collisions¹, enable the optional record-dot syntax to avoid having to prefix all field names (following the recommendations in Haskell Unfolder #45: Haskell records in 2025), set the output directory, and specify the desired name of the generated Haskell module.

The final line deserves a slightly more detailed explanation. Suppose you are generating bindings for a library that is installed in /opt/library, and that library has a header /opt/library/api/server/secure.h. The bindings generated by hs-bindgen will need to refer to this header using a #include statement; the question is what that #include statement should look like. If we generated

#include </opt/library/api/server/secure.h>

then the generated bindings would only work on machines where that library is installed in that exact location. If this include path should instead be

#include <api/server/secure.h>

with /opt/library in the search path, then hs-bindgen must be invoked with -I /opt/library and main argument api/server/secure.h: the final argument is included precisely as-is in the #include. For our simple example, -I "$(pwd)/cbits" means that the cbits folder of our Haskell package is added to the C include path, and the generated #include will be

#include <library_a.h>

Generated bindings

The above invocation of hs-bindgen will have generated a few Haskell modules. LibraryA.hs contains the translation of the types in the header:

module LibraryA where

data Version = Version {
    major :: CInt
  , minor :: CInt
  }
  deriving stock (Eq, Show)

instance Storable Version where (..)

instance HasField "major" (Ptr Version) (Ptr CInt) where (..)
instance HasField "minor" (Ptr Version) (Ptr CInt) where (..)

(The above code is slightly cleaned up for readability, and various parts of the generated module are omitted: boilerplate such as imports and required language extensions, Haddocks, as well as some more specialized type class instances.)

In addition, module LibraryA/Safe.hs will contain safe imports for all functions in the header:

module LibraryA.Safe where

showVersion :: Version -> IO ()
showVersion = (..)

Since the C function showVersion takes a struct argument by value (rather than as a pointer to the struct), which is not supported by Haskell FFI, hs-bindgen will also have generated a C wrapper; that all happens transparently and automatically.

Module LibraryA/Unsafe.hs contains the same API, but using unsafe imports (if you need a refresher on safe versus unsafe, you might like to watch Haskell Unfolder #36: concurrency and the FFI). Module LibraryA/FunPtr.hs finally contains the addresses of all functions, in case you have C code that works with function pointers.

Using the generated bindings

We can call showVersion very simply as

import LibraryA
import LibraryA.Safe

main :: IO ()
main = showVersion $ Version 2 3

We will come back to the HasField instances for Ptr Version below; no explicit HasField instances for Version itself are necessary, because they are generated by ghc.

Dependencies

Suppose library B defines some kind of API for drivers, and suppose it uses library A:

#include "library_a.h"

struct Driver {
  char* name;
  struct Version version;
};

void initDriver(struct Driver *d);
void showDriver(struct Driver *d);

Main headers

If we run hs-bindgen on this header in the same way as we did for library_a.h, we will get warnings such as

[Warning] [HsBindgen] [select] 'struct Driver' at "../cbits/library_b.h 8:8"
  Could not select declaration (direct select predicate match):
    Transitive dependency not selected:
      'struct Version' at "../cbits/library_a.h 3:8"
      Adjust the select predicate or enable program slicing

When we generate bindings for a header, we need to know which declarations in that header the user wants to generate bindings for; in hs-bindgen this happens by means of selection predicates. The default selection predicate is --select-from-main-headers, which means that any declarations in headers explicitly mentioned on the command line are selected, but any declarations in headers that might be imported by those headers are not. The first way in which we can fix this warning therefore is by explicitly generate bindings for both library_a.h and library_b.h:

cabal run -- hs-bindgen-cli preprocess \
  (..other arguments as before..)
  --module LibraryB \
  -I "$(pwd)/cbits" library_a.h library_b.h

Program slicing

Suppose we are really only interested in library B, and want to generate only those bindings in library A that are required by library B. To do this, we can enable program slicing:

cabal run -- hs-bindgen-cli preprocess \
  (..)
  --enable-program-slicing \
  -I "$(pwd)/cbits" library_b.h

This will pull in only those declarations in library A that are referenced by library B.

Opaque types

The API provided by library B exclusively works with pointers to struct Driver; so perhaps we don’t need a Haskell-side representation of that struct at all. If that is the case, we can configure hs-bindgen through a prescriptive binding specification, and tell it that it should keep the Haskell type opaque:

version:
  hs_bindgen: 0.1.0
  binding_specification: '1.0'
ctypes:
- headers: library_b.h
  cname: struct Driver
  hsname: Driver
hstypes:
- hsname: Driver
  representation: emptydata

This states that the C declaration struct Driver, found in header library_b.h, should be mapped to a Haskell type called Driver (we could pick a different name here if we wanted to, overriding naming decisions made by hs-bindgen), and that the Haskell type Driver be represented as an empty datatype. If we now run

cabal run -- hs-bindgen-cli preprocess \
  (..)
  --prescriptive-binding-spec libraryB.yaml \
  -I "$(pwd)/cbits" library_b.h

then the generated LibraryB is simply

data Driver

It can be quite useful to combine emptydata with program slicing, limiting how many declarations from imported headers are in fact needed.

Composability

The solutions in the previous section all had one important downside: in none of them the generated code for library B reused the generated code for library A. This kind of composability of generated bindings is an important goal of hs-bindgen, influencing many design decisions. Composability is achieved through (external) binding specifications; a binding specification is a .yaml (or .json) file describing a set of generated bindings, a bit like .hi files in Haskell, or a module signature in OCaml or Backpack.

When we generate the bindings for library A we can ask hs-bindgen to additionally generate a binding spec:

cabal run -- hs-bindgen-cli preprocess \
  (..)
  --gen-binding-spec libraryA.yaml

The resulting .yaml file describes the types generated for library A, including which type class instances they have (necessary in order to know which type class instances we can generate when these types are used in other libraries). When generating bindings for library B, we can pass this binding specification along:

cabal run -- hs-bindgen-cli preprocess \
  (..)
  --external-binding-spec libraryA.yaml \
  -I "$(pwd)/cbits" library_b.h

The generated code then looks like

module LibraryB where

import qualified LibraryA

data Driver = Driver {
    name    :: Ptr CChar
  , version :: LibraryA.Version
  }
  deriving stock (Eq, Show)

instance Storable Driver where (..)

instance HasField "name"    (Ptr Driver) (Ptr (Ptr CChar))      where (..)
instance HasField "version" (Ptr Driver) (Ptr LibraryA.Version) where (..)

External binding specifications are useful not only when generating bindings for multiple libraries, but also for structuring bindings for multiple headers of the same library, or when an identical header is included in lots of libraries (such as the rtwtypes.h header generated by MATLAB). We consider external binding specifications to be an essential feature of hs-bindgen.

Pointers

HasField

Suppose we want to override one value deeply nested in some C data structure. We could use the Storable instances to peek the value, then override the appropriate field, and finally poke the updated value:

main :: IO ()
main = do
    alloca $ \(driverPtr :: Ptr Driver) -> do
      initDriver driverPtr

      driver <- peek driverPtr
      poke driverPtr $ driver & #version % #minor .~ 2
      showDriver driverPtr

(The use of lenses here is optional of course.)

However, it may well be undesirable to marshall the entire structure back and forth merely to change a single value. This is why hs-bindgen also generates HasField instances for pointers, so that record dot syntax can be used to index C structures. We can update the minor version number without marshalling the entire structure as follows:

poke driverPtr.version.minor 3

If you prefer to avoid record-dot syntax, you can use the HsBindgen.Runtime.HasCField infrastructure directly.

FunPtr

When dealing with a higher order API, hs-bindgen will generate additional bindings to convert back and forth between C function pointers and Haskell functions, and package these conversions up as instances of two type classes in hs-bindgen-runtime:

class ToFunPtr a where
  toFunPtr :: a -> IO (FunPtr a)

class FromFunPtr a where
  fromFunPtr :: FunPtr a -> a

withFunPtr :: ToFunPtr a => a -> (FunPtr a -> IO b) -> IO b

For example, suppose the driver API in library B additionally contains

struct Driver;
typedef int RunDriver(struct Driver* self);

struct Driver {
  // .. other fields as before ..
  RunDriver* run;
};

int callDriver(struct Driver* d);

then we generate

newtype RunDriver = RunDriver {
    unwrap :: Ptr Driver -> IO CInt
  }

instance ToFunPtr   RunDriver where (..)
instance FromFunPtr RunDriver where (..)

Here’s how we might use this:

counter <- newIORef 0
let run = RunDriver $ \_self -> atomicModifyIORef counter $ \x -> (succ x, x)
withFunPtr run $ \funPtr -> do
  poke driverPtr.run funPtr
  replicateM_ 5 $ print =<< callDriver driverPtr

Macros

Some C libraries make part of their API available as CPP macros. Macros in C don’t have a semantics per se; they are merely a list of tokens that the preprocessor splices in whenever they are used. In order to nonetheless be able to generate “bindings” for macros, hs-bindgen imbues macros with a bespoke semantics. In future versions of hs-bindgen this macro infrastructure will be pluggable (#942), because the default semantics may not suit all applications.

Constants

Low level C libraries (such as this Analog Devices Talise driver) may make certain constants such as bitfields available as CPP macros

#define SIGNALID 0x01

We translate these to Haskell constants:

sIGNALID :: CInt
sIGNALID = 1

Expressions

It may also happen that libraries offer certain functionality as macro functions. For example, a library might provide a definition that provides a pointer offset for devices with memory-mapped I/O:

#define INPUT_PORT(x) x + 4

Since hs-bindgen has no way of knowing if that (+) operator should be interpreted as integer addition or pointer offset (or indeed something else), it generates a very general definition:

iNPUT_PORT :: Add a CInt => a -> AddRes a CInt

Add and AddRes come from c-expr-runtime); one way that we can instantiate this type is to:

inputPort :: Ptr Word8 -> Ptr Word8
inputPort = iNPUT_PORT

Types

Finally, some low-level libraries define types as macros:

#define S16_TYPE short int

In the case (and only in the case) that these can be parsed as the corresponding typedef

typedef short int S16_TYPE;

hs-bindgen will treat these as typedefs, and generate a Haskell newtype:

newtype S16_TYPE = S16_TYPE {
    unwrap :: CShort
  }

Squashing

C structs are often defined using a typedef:

typedef struct Point {
  int x;
  int y;
} Point;

This is typically done for syntactic convenience only, making it possible to write simply Point rather than struct Point. When the only use of a struct is within a typedef in this manner, hs-bindgen will “squash” the typedef, and generate a single type only:

data Point = Point {
    x :: CInt
  , y :: CInt
  }

If however the typedef is not the only use of the struct, then hs-bindgen assumes that this is intended to convey some kind of semantic information, and will not squash. For example, suppose the device driver API includes

typedef struct Driver DeviceDriver;

then we generate

newtype DeviceDriver = DeviceDriver {
    unwrap :: Driver
  }

To avoid confusion, a notice will be emitted whenever a type is squashed. If desired, this notice can be suppressed with --log-as-info select-mangle-names-squashed. In a future release of hs-bindgen squashing will be configurable per typedef using a prescriptive binding spec (#1436).

Build process integration

Preprocessor

The primary way of invoking hs-bindgen is by calling hs-bindgen-cli, as we have discussed in this blogpost, raising the question of how to integrate that into your build process. We don’t have a perfect answer here just yet. You can of course write your own Makefile, or integrate this into a nix derivation (see the Nix tutorial). If you want to use cabal as the main driver, you can use a custom setup script, or the new SetupHooks API; we don’t provide explicit support for either just yet (#1666 tracks adding support for SetupHooks).

We do offer a “literate” mode, abusing Cabal’s support for literate Haskell to trick it into running hs-bindgen instead; see section Cabal preprocessor integration of the hs-bindgen manual for details.

Template Haskell

If you prefer, you can use hs-bindgen in TH mode; this avoids the need for running the preprocessor altogether. Instead, you can #include a C header in a Haskell module:

let cfg :: Config
    cfg = def & #clang % #extraIncludeDirs .~ [Pkg "cbits"]
 in withHsBindgen cfg def $
       hashInclude "library_a.h"

Since we cannot generate more than one module in this way, by default this splices in bindings for all the types in the header along with the safe imports, though you can override that if you wish. Of course, you can also use hashInclude in more than one Haskell module to manually generate multiple modules. See Template Haskell mode in the manual for details.

Cross-compilation

We rely on libclang for all machine-dependent decisions (as well as parsing the C headers in the first place). Therefore if you want to generate code for a target differing from your host platform, in principle it should suffice to provide the appropriate clang arguments. We are working on a guide for doing so; this will become section Cross-compilation in the manual; however, this section is not quite ready yet (#1630).

Non-portability

It is important to note that the bindings generated by hs-bindgen are not portable in general (indeed, it will be rare that they are). The slogan to remember is:

The bindings generated by hs-bindgen should be regarded as build artefacts.

If you do want to distribute generated bindings as part of your package, you can of course do so, but then you are responsible for making the appropriate provisions in your .cabal file (perhaps using SetupHooks) to check machine architecture, choose between different sets of bindings, etc. At present hs-bindgen does not yet provide any explicit support for making this process easier.

Conclusions

Although Haskell provides good FFI features, writing bindings to large C libraries by hand can be a laborious process. The bindings generated by hs-bindgen are low-level: char* is translated to Ptr CChar, not ByteString or Text or something else still; nonetheless, it should make the process of writing bindings significantly easier. Automatically generating high-level bindings is something we’ll soon turn our attention to.

Before we do so, however, there is still some cleanup to be done before we can release version 0.1. As it stands, the alpha release supports nearly all of C, although some missing corner-cases are still missing (such as implicit fields, #1649); see the issue tracker for a list of missing C features. However, we would like to invite you to start using the alpha release now; there should only be very minor backwards incompatible changes introduced in between versions 0.1-alpha and 0.1, so your code should not break when upgrading to version 0.1.

Acknowledgements

Many people at Well-Typed have contributed to hs-bindgen, including Travis Cardwell, Joris Dral, Dominik Schrempf, Armando Santos, Sam Derbyshire, and others (as well as myself, Edsko de Vries).

Well-Typed is grateful to Anduril Industries for sponsoring this work.

Gabriella Gonzalez: Beyond agentic coding>

Sat, 07 Feb 2026 00:00:00 +0000

AI dev tooling can do better than chat interfaces

Tweag I/O: Integrating Coverity static analysis with Bazel

Thu, 05 Feb 2026 00:00:00 +0000

Coverity is a proprietary static code analysis tool that can be used to find code quality defects in large-scale, complex software. It supports a number of languages and frameworks and has been trusted to ensure compliance with various standards such as MISRA, AUTOSAR, ISO 26262, and others. Coverity provides integrations with several build systems, including Bazel, however the official Bazel integration fell short of the expectations of our client, who wanted to leverage the Bazel remote cache in order to speed up Coverity analysis and be able to run it in normal merge request workflows. We took on that challenge.

The Coverity workflow

To understand the rest of the post it is useful to first become familiar with the Coverity workflow, which is largely linear and can be summarized as a sequence of steps:

Configuration. In this step cov-configure is invoked, possibly several times with various combinations of key compiler flags (such as -nostdinc++, -fPIC, -std=c++14, and others). This produces a top-level XML configuration file accompanied by a directory tree that contains configurations that correspond to the key flags that were provided. Coverity is then able to pick the right configuration by dispatching on those key flags when it sees them during the build.
Compilation. This is typically done by invoking cov-build or its lower-level cousin cov-translate. The distinction between these two utilities will become clear in the next section. For now, it is enough to point out that these tools translate original invocations of the compiler to invocations of cov-emit (Coverity’s own compiler) which in turn populate an SQLite database with intermediate representations and all the metadata about the built sources such as symbols and include paths.
Analysis. This step amounts to one or more invocations of cov-analyze. This utility is practically a black box, and it can take a considerable time to run, depending on the number of translation units (think compilation units in C/C++). The input for the analysis is the SQLite database that was produced in the previous step. cov-analyze populates a directory tree that can then be used for report generation or uploading of identified software defects.
Reporting. Listing and registration of found defects is performed with cov-format-errors and cov-commit-defects, respectively. These tools require a directory structure that was created by cov-analyze in the previous step.

Existing integrations

A successful integration of Coverity into Bazel has to provide a way to perform step 2 of the workflow, since this is the only step that is intertwined with the build system. Step 1 is relatively quick, and it does not make a lot of difference how it is done. Step 3 is opaque and build-system-agnostic because it happens after the build, furthermore there is no obvious way to improve its granularity and cacheability. Step 4 is similar in that regard.

The official integration works by wrapping a Bazel invocation with cov-build:

$ cov-build --dir /path/to/idir --bazel bazel build //:my-bazel-target

Here, --dir specifies the intermediate directory where the SQLite database along with some metadata will be stored. In the --bazel mode of operation cov-build tricks Bazel by intercepting the invocations of the compiler which are replaced by invocations of cov-translate. Compared to cov-build, which is a high-level wrapper, cov-translate corresponds more closely to individual invocations of a compiler. It identifies the right configuration from the collection of configurations created in step 1 and then uses it to convert the command line arguments of the original compiler into the command line arguments of cov-emit, which it then invokes.

The main problem with the official integration is the fact that it does not support caching. bazel build has to run from scratch with caching disabled in order to make sure that all invocations of the compiler are performed and none are skipped. Another nuance of the official integration is that one has to build a target of the supported kind, e.g. cc_library. If you bundle your build products together in some way (e.g. in an archive), you cannot simply build the top-level bundle as you normally would. Instead, you need to identify every compatible target of interest in some other way.

Because of this, our client did not use the official Coverity integration for Bazel. Instead, they would run Bazel with the --subcommands option which makes Bazel print how it invokes all tools that participate in the build. This long log would then be parsed and converted into a Makefile in order to be able to leverage Coverity’s Make integration in cov-build instead. This approach still suffered from long execution times due to lack of caching. It ran as a nightly build and took over 12 hours, which wasn’t suitable for a regular merge request’s CI pipeline.

Our approach

The key insight that allowed us to make the build process cacheable is the observation that individual invocations of cov-translate produce SQLite databases—emit-dbs—that are granular and relatively small. These individual emit-dbs can then be merged in order to form the final, big emit-db that can be used for running cov-analyze. Therefore, our plan was the following:

Create a special-purpose Bazel toolchain that starts as a copy of the toolchain which is used for “normal” compilation in order to match the way the original compiler (gcc in our case) is invoked.
Instead of gcc and some other tools such as ar, have it invoke our own wrapper that would drive invocations of cov-translate.
The only useful output of the compilation step is the emit-db database. On the other hand, in CppCompile actions, Bazel normally expects .o files to be produced, so we just rename our emit-db SQLite database to whatever object file Bazel is expecting.
In the linking step, we use the add subcommand of cov-manage-emit in order to merge individual emit-dbs.
Once the final bundle is built, we iterate through all eligible database files and merge them together once again in order to obtain the final emit-db that will be used for analysis.

From Bazel’s point of view, we are simply compiling C/C++ code, however, this is a way to perform a cacheable Coverity build. If you are a regular reader of our blog, you may have noticed certain similarities to the approach we used in our CTC++ Bazel integration documented here and here.

Difficulties

The plan may sound simple, but in practice there was nothing simple about it. The key difficulty was the fact that Coverity tooling was not designed with reproducibility in mind. At every step, starting from configuration generation (which is simply a genrule in our case) we had to inspect and adjust produced outputs in order to make sure that they do not include any volatile data that could compromise reproducibility. The key output in the Coverity workflow, the emit-db database, contained a number of pieces of volatile information:

Start time, end time, and process IDs of cov-emit invocations.
Timestamps of source files.
Command line IDs of translation units.
Host name; luckily, this can be overwritten easily by setting the COV_HOST environment variable.
Absolute file names of source files. These are split into path segments with each segment stored as a separate database entry, meaning the number of entries in the FileName table varies with the depth of the path to the execroot, breaking hermeticity. Deleting entries is not viable since their IDs are referenced elsewhere. Our solution was to identify the variable prefix leading to the execroot and reset all segments in that prefix to a fixed string. The prefix length proved to be constant for each execution mode, allowing us to use --strip-path arguments in cov-analyze to remove them during analysis.
In certain cases some files from the include directories would be added to the database with altered Coverity-generated contents that would also include absolute paths.

We had to meticulously overwrite all of these, which was only possible because emit-db is in the SQLite format and there is open source tooling that makes it possible to edit it. If the output of cov-emit were in a proprietary format we almost certainly wouldn’t be able to deliver as efficient a solution for our client.

In practice, normalization of emit-db happens in two stages:

We run some SQL commands that reset the volatile pieces of data inside the database. As we were iterating on these “corrective instructions”, we made sure that we eliminated all instances of volatility by using the sqldiff utility which can print differences in schema and data between tables.
We dump the resulting database with the .dump command which exports the SQL statements necessary to recreate the database with the same schema and data. Then we re-load these statements and thus obtain a binary database file that is bit-by-bit reproducible. This is necessary because simply editing a database by running SQL commands on it does not ensure that the result is binary reproducible even if there is no difference in the actual contents.

Performance considerations

Since emit-dbs are considerably larger than typical object files, we found it highly desirable to use compression for individual SQLite databases, both for those that result from initial cov-translate invocations and for merged databases that are created by cov-manage-emit in the linking step. Zstandard proved to be a good choice for that—the utility is both fast and makes our build outputs up to 4 times smaller. Without compression, we risked filling the remote cache quickly; besides, the bigger the files, the slower I/O operations are.

We were tempted to minimize the size of the database even further by exploring if there’s anything in emit-db that can be safely removed without affecting our client’s use case. Alas, every piece of information stored was required by Coverity during the analysis phase and our attempts to truncate some of the tables led to failures in cov-analyze. It is worth noting that the sqlite3_analyzer utility tool (part of the SQLite project) can be used to produce a report that explains which objects store most data. This way we found that there is an index that contributes about 20% of the database size, however, deleting it severely degrades the performance of record inserts which is a key operation during merging of emit-dbs.

Linking steps, which in our approach amount to merging of emit-db databases, produce particularly large files. In order to reduce the amount of I/O we perform and avoid filling the remote cache too quickly, we’ve marked all operations that deal with these large files as no-remote. This is accomplished with this line in our .bazelrc file:

common:coverity --modify_execution_info=CppArchive=+no-remote,CppLink=+no-remote,CcStrip=+no-remote
# If you wish to disable RE for compilation actions with coverity, uncomment
# the following line:
# common:coverity --modify_execution_info=CppCompile=+no-remote-exec

On the other hand, we did end up running CppCompile actions with remote execution (RE) because it proved to be twice faster than running them on CI agents. Making RE possible required us to identify the exact collection of files from the Coverity installation that are required during invocations of the Coverity tooling. Once we observed RE working correctly, we were confident that the Bazel definitions we use are hermetic.

Merging of individual emit-db databases found in the final bundle (i.e. after the build has finished) proved to be time-consuming. This operation cannot be parallelized since database information cannot be written in parallel. The time required for this step grows linearly with the number of translation units (TUs) being inserted, therefore it makes sense to pick the largest database and then merge smaller ones into it. One could entertain the possibility of skipping merging altogether and instead running smaller analyses on individual emit-dbs, but this seems to be not advisable, since Coverity performs a whole-program analysis, and thus it would lose valuable information that way. For example, one TU may be exercised in different ways by two different applications and the analysis results for this TU cannot be correctly merged.

The analysis phase is a black box, and it can easily become the bottleneck of the entire workflow, thus making it impractical for running in merge request pipelines. A common solution for speeding up the analysis in merge request pipelines is to identify files that were edited and limit the analysis to only these files with the --tu-pattern option, which supports a simple language for telling Coverity what to care about during analysis. We added support for this approach to our solution by automatically finding the files changed in the current merge request, and passing these on to --tu-pattern. This restricted analysis still requires the emit-dbs for the entire project, but most of them will be cached.

The results

The solution that we delivered is in the form of a bazel run-able target that depends on the binary bundle that needs to be analyzed. It can be invoked like this:

bazel run //bazel/toolchains/coverity:workflow --config=coverity -- ...

This solution can be used both for the nightly runs of Coverity and in the merge request pipelines. We have confirmed that the results that are produced by our solution match the findings of Coverity when it is run in the “traditional” way. A typical run in a merge request pipeline takes about 22 minutes when a couple of C++ files are edited. The time is distributed as follows:

8 minutes: building, similar to build times for normal builds (this step is sped up by caching)
10 minutes: the final merging of emit-dbs
4 minutes: analysis, uploading defects, reporting (this step is sped up by --tu-pattern).

The execution time can grow, of course, if the edits are more substantial. The key benefit of our approach is that the Coverity build is now cacheable and therefore can be included in merge request CI pipelines.

Conclusion

In summary, integrating Coverity static analysis with Bazel in a cacheable, reproducible, and efficient manner required a deep understanding of both Bazel and Coverity, as well as a willingness to address the nuances of proprietary tooling that got in the way. By leveraging granular emit-db databases, normalizing volatile data, and optimizing for remote execution and compression, we were able to deliver a solution that fits well into the client’s CI workflow and supports incremental analysis in merge request pipelines.

While the process involved overcoming significant challenges, particularly around reproducibility and performance, the resulting workflow enables fast static analysis without sacrificing the benefits of Bazel’s remote cache. We hope that sharing our approach will help other teams facing similar challenges and inspire improvements when integrating other static analysis tools with Bazel.

in Code: "Five-Point Haskell": Total Depravity (and Defensive Typing)

Mon, 02 Feb 2026 15:06:46 +0000

I have thought about distilling the principles by which I program Haskell, and how I’ve been able to steer long-lived projects over years of growth, refactorings, and changes in demands. I find myself coming back to a few distinct and helpful “points” (“doctrines”, if you may allow me to say) that have yet to lead me astray.

With a new age of software development coming, what does it even mean to write good, robust, correct code? It is long overdue to clarify the mindset we use to define “good” coding principles.

In this series, Five-Point Haskell, I’ll set out to establish a five-point framework for typed functional programming (and Haskell-derived) design that aims to produce code that is maintainable, correct, long-lasting, extensible, and beautiful to write and work with. We’ll reference real-world case studies with actual examples when we can, and also attempt to dispel thought-leader sound bites that have become all too popular on Twitter (“heresies”, so to speak).

Let’s jump right into point 1: the doctrine of Total Depravity, and why Haskell is perfectly primed to make living with it as frictionless as possible.

Total Depravity: If your code’s correctness depends on keeping complicated interconnected structure in your head, a devastating incident is not a matter of if but when.

Therefore, delegate these concerns to tooling and a sufficiently powerful compiler, use types to guard against errors, and free yourself to only mentally track the actual important things.

Mix-ups Are Inevitable

Think about the stereotype of a “brilliant programmer” that an inexperienced programmer has in their mind — someone who can hold every detail of a complex system in their head, every intricate connection and relationship between each component. There’s the classic Monkey User Comic that valorizes this ideal.

Monkey User — Focus

The 10x developer is one who can carry the state and interconnectedness of an entire system in their brain, and the bigger the state they can carry, the more 10x they are.

This is the myth of the hero programmer. Did you have a bug? Well, you just need to upgrade your mental awareness and your context window. You just need to be better and better at keeping more in your mind.

Actually addressing these issues in most languages requires a lot of overhead and clunkiness. But luckily we’re in Haskell.

Explicit Tags

The 2022 Atlassian Outage, in part, was the result of passing the wrong type of ID. The operators were intended to pass App IDs, but instead passed Site IDs, and the errors cascaded from there. It goes without saying that if you have a bunch of “naked” IDs, then mixing them up is eventually going to backfire on you.

newtype Id = Id String

type SiteId = Id
type AppId = Id

getApps :: SiteId -> IO [AppId]
deleteSite :: SiteId -> IO ()
deleteApp :: AppId -> IO ()

This is convenient because you get functions for all IDs without any extra work. Let’s say you want to serialize/print or deserialize/read these IDs — it can be helpful to give them all the same type so that you can write this logic in one place.

instance ToJSON Id where
  toJSON (Id x) = object [ "id" .= x ]

instance FromJSON Id where
  parseJSON = withObject "Id" $ \v ->
    Id <$> (v .: "id")

Convenient and effective, as long as you never accidentally use a SiteId as an AppId or vice versa. And this is a very easy delusion to fall into, if you don’t believe in total depravity. However, sooner or later (maybe in a week, maybe in a year, maybe after you onboard that new team member)…someone is going to accidentally pass a site ID where an app ID is expected.

main :: IO ()
main = do
    let targetSites = [Id "abc", Id "def"]
    mapM_ deleteApp targetSites

And at that point it’s all over.

Knowing this can happen, we can add a simple newtype wrapper so that accidentally using the wrong ID is a compile error:

newtype SiteId = SiteId String
newtype AppId = AppId String

And now such a mis-call will never compile! Congratulations!

We do have a downside now: we can no longer write code polymorphic over IDs when we want to. In the untagged situation, we could only write polymorphic code, and in the new situation we can only write code for one ID type.

instance FromJSON SiteId where
  parseJSON = withObject "Id" $ \v -> do
    tag <- v .: "type"
    unless (tag == "Site") $
      fail "Parsed wrong type of ID!"
    SiteId <$> (v .: "id")

instance ToJSON SiteId where
  toJSON (SiteId x) = object [ "type" .= "Site", "id" .= x ]

instance FromJSON AppId where
  parseJSON = withObject "Id" $ \v -> do
    tag <- v .: "type"
    unless (tag == "App") $
      fail "Parsed wrong type of ID!"
    AppId <$> (v .: "id")

instance ToJSON AppId where
  toJSON (AppId x) = object [ "type" .= "App", "id" .= x ]

However, luckily, because we’re in Haskell, it’s easy to get the best of both worlds with phantom types (that don’t refer to anything inside the actual data representation):

data Id a = Id { getId :: String }

data Site
data App

type SiteId = Id Site
type AppId = Id App

-- using Typeable for demonstration purposes
instance Typeable a => ToJSON (Id a) where
  toJSON (Id x) = object
    [ "type" .= show (typeRep @a)
    , "id" .= x
    ]

instance Typeable a => FromJSON (Id a) where
  parseJSON = withObject "Id" $ \v -> do
    tag <- v .: "type"
    unless (tag == show (typeRep @a)) $
      fail "Parsed wrong type of ID!"
    Id <$> (v .: "id")

Type safety doesn’t necessarily mean inflexibility!

Phantom Powers

Phantom types give us a lot of low-hanging fruit for preventing inadvertent misuse.

The 2017 DigitalOcean outage, for example, was partially about the wrong environment credentials being used.

We could imagine a test harness that clears a test database using postgresql-simple:

-- | Warning: do NOT call this outside of test environment!
clearTestEnv :: Connection -> IO ()
clearTestEnv conn = do
  putStrLn "Are you sure you read the warning on this function? Well, too late now!"
  _ <- execute_ conn "DROP TABLE IF EXISTS users CASCADE"
  putStrLn "Test data wiped."

However, somewhere down the line, someone is going to call clearTestEnv deep inside a function inside a function inside a function, which itself is called against the prod database. I guarantee it.

To ensure this never happens, we can use closed phantom types using DataKinds (made nicer with TypeData post-GHC 9.6):

-- type data = declare a closed kind and two type constructors at the type level using -XTypeData
type data Env = Prod | Test

newtype DbConnection (a :: Env) = DbConnection Connection

runQuery :: DbConnection a -> Query -> IO Int64
runQuery (DbConnection c) q = execute_ c q

-- | Warning: Did you remember to charge your chromebook? Oh and this function
-- is safe by the way.
clearTestEnv :: DbConnection Test -> IO ()
clearTestEnv conn = do
  _ <- runQuery conn "DROP TABLE IF EXISTS users CASCADE"
  putStrLn "Test data wiped."

connectProd :: IO (DbConnection Prod)
connectProd = DbConnection <$> connectPostgreSQL "host=prod..."

Now, if you create a connection using connectProd, you can use runQuery on it (because it can run any DbConnection a)…but if any sub-function of a sub-function calls clearTestEnv, it will have to unify with DbConnection Test, which is impossible for a prod connection.

This is somewhat similar to using “mocking-only” subclasses for dependency injection, but with a closed universe. I discuss patterns like this in my Introduction to Singletons series.

Correct Representations

Semantic Phantoms

And sometimes, phantom types can do the work for you, not only preventing mix-ups but also encoding business logic in their manipulation.

Take, for instance, the Mars Climate Orbiter failure, where the software module provided by Lockheed Martin expected US Customary Units, and another one developed by NASA expected SI units.

If I had a function like:

-- | In Newton-seconds
myMomentum :: Double
myMomentum = 20

-- | In Pounds-second
myImpulse :: Double
myImpulse = 4

-- | Make sure these are both the same units!
applyImpulse :: Double -> Double -> Double
applyImpulse currentMomentum impulse = currentMomentum + impulse

This is just asking for someone to come along and provide newtons alongside pounds. It isn’t even clear from the types what is expected!

We can instead use the dimensional library:

import qualified Numeric.Units.Dimensional.Prelude as U
import Numeric.Units.Dimensional ((*~))
import Numeric.Units.Dimensional.SIUnits
import Numeric.Units.Dimensional.NonSI

myMomentum :: Momentum
myMomentum = 20 *~ (newton U.* seconds)

myImpulse :: Impulse
myImpulse = 4 *~ (poundForce U.* seconds)

-- Verify at compile-time that we can use '+' with Momentum and Impulse
applyImpulse :: Momentum -> Impulse -> Momentum
applyImpulse currentMomentum impulse = currentMomentum + impulse

Now as long as momentum and impulse are provided in the correct types at API boundaries, no mix-up will happen. No need to send 300 million dollars down the drain! Libraries will just need to provide a unified Momentum or Impulse type, and everything will work out.

The Billion-Dollar Mistake

Speaking of costly errors, there is one extremely egregious pattern that is so pervasive, so alluring, and yet so inevitably devastating, it has been dubbed the “Billion Dollar Mistake”. It’s the idea of a sentinel value, or in-band signaling.

There are examples:

String.indexOf(), str.find(), etc. in many languages return -1 if the substring is not found
C’s fgetc(), getchar(), return -1 for EOF. And if you cast to char, you basically can’t distinguish EOF from 0xff (ÿ).
malloc() returning the pointer 0 means not enough memory
Some languages have a special NULL pointer value as well — or even a value null that can be passed in for any expected type or object or value.
JavaScript’s parseInt returns not null, but rather NaN for a bad parse — giving two distinct sentinel values
A lot of Unix scripting uses the empty string "" for non-presence
Sensor firmware often reports values like -999 for a bad reading…but sometimes -999 might actually be a valid value!

It should be evident that these are just accidents and ticking time bombs waiting to happen. Some caller just needs to forget to handle the sentinel value, or to falsely assume that the sentinel value is impossible to occur in any situation.

It’s called the billion dollar mistake, but it’s definitely arguable that the cumulative damage has been much higher. High-profile incidents include sock_sendpage and the 2025 GCP outage, but if you’re reading this and you are honest with yourself, it’s probably happened to you multiple times and has been the source of many frustrating bug hunts.

There’s also CVE-2008-5077, because EVP_VerifyInit returns 0 for false, 1 for true, and -1 for error! So some OpenSSL code did a simple if-then-else check (result != 0) and treated error and true the same way. Whoops.

Why do we do this to ourselves? Because it is convenient. In the case of EVP_VerifyInit, we can define an enum instead…

data VerifyResult = Success | Failure | Error

However, it’s not easy to make an “integer or not found” type in C or JavaScript without some sort of side-channel. Imagine if JavaScript’s String.indexOf() instead expected continuations on success and failure and became much less usable as a result:

unsafeIndexOf :: String -> String -> Int

-- vs.

-- takes a success continuation and a failure continuation
indexOf :: String -> String -> (Int -> r) -> (() -> r) -> r

All of this just to fake having actual sum types.

We don’t really have an excuse in Haskell, since we can just return Maybe:

-- from Data.Vector
elemIndex :: Eq a => a -> Vector a -> Maybe Int

Returning Maybe or Option forces the caller to handle:

case elemIndex 3 myVec of
  Just i -> -- ..
  Nothing -> -- ..

and this handling is compiler-enforced. Provided, of course, you don’t intentionally throw away your type-safety and compiler checks for no reason. You can even return Either with an enum for richer responses, and very easily chain erroring operations using Functor and Monad. In fact, with cheap ADTs, you can define your own rich result type, like in unix’s ProcessStatus:

data ProcessStatus
   = Exited ExitCode
   | Terminated Signal Bool
   | Stopped Signal

Imagine trying to cram all of that information into an int!

Unmarked Assumptions

Assumptions kill, and a lot of times we arrive at implicit assumptions in our code. Unfortunately, even if we add these assumptions in our documentation, it only takes a minor refactor or lapse in memory for these to cause catastrophic incidents.

There are the simple cases — consider a mean function:

-- | Warning: do not give an empty list!
mean :: [Double] -> Double
mean xs = sum xs / fromIntegral (length xs)

But are you really going to remember to check if your list is empty every time you give it to mean? No, of course not. Instead, make it a compiler-enforced constraint.

mean :: NonEmpty Double -> Double
mean xs = sum xs / fromIntegral (length xs)

Now mean takes a NonEmpty list, which can only be created safely using nonEmpty :: [a] -> Maybe (NonEmpty a) (where the caller has to explicitly handle the empty list case, so they’ll never forget) or from functions that already return NonEmpty by default (like some :: f a -> f (NonEmpty a) or group :: Eq a => [a] -> [NonEmpty a]), allowing you to beautifully chain post-conditions directly into pre-conditions.

Accessing containers is, in general, very fraught…even things like indexing lists can send us into a graveyard spiral. Sometimes the issue is more subtle. This is our reminder to never let these implicit assumptions go unnoticed.

Separate Processed Data

“Shotgun parsing” involves mixing validated and unvalidated data at different levels in your program. Oftentimes it is considered “fine” because you just need to remember which inputs are validated and which aren’t…right? In truth, all it takes is a simple temporary lapse of mental model, a time delay between working on code, or uncoordinated contributions before things fall apart.

Consider a situation where we validate usernames only on write to the database.

validUsername :: String -> Bool
validUsername s = all isAlphaNum s && all isLower s

-- | Returns 'Nothing' if username is invalid or insertion failed
saveUser :: Connection -> String -> IO (Maybe UUID)
saveUser conn s
  | validUsername s = do
      newId <- query conn "INSERT INTO users (username) VALUES (?) returning user_id" (Only s)
      pure $ case newId of
        [] -> Nothing
        Only i : _ -> Just i
  | otherwise = pure Nothing

getUser :: Connection -> UUID -> IO (Maybe String)
getUser conn uid = do
  unames <- query conn "SELECT username FROM users where user_id = ?" (Only uid)
  pure $ case unames of
    [] -> Nothing
    Only s : _ -> Just s

It should be fine as long as you only ever use saveUser and getUser…and nobody else has access to the database. But, if someone hooks up a custom connector, or does some manual modifications, then the users table will now have an invalid username, bypassing Haskell. And because of that, getUser can return an invalid string!

Don’t assume that these inconsequential slip-ups won’t happen; assume that it’s only a matter of time.

Instead, we can bake the state of a validated string into the type itself:

newtype Username = UnsafeUsername String
  deriving (Show, Eq)

-- | Our "Smart Constructor"
mkUsername :: String -> Maybe Username
mkUsername s
  | validUsername s = Just (UnsafeUsername s)
  | otherwise       = Nothing

-- | Access the raw string if needed
unUsername :: Username -> String
unUsername (UnsafeUsername s) = s

Username and String themselves are not structurally different — instead, Username is a compiler-enforced tag specifying it went through a specific required validation function within Haskell, not just externally verified.

Now saveUser and getUser are safe at the boundaries:

saveUser :: Connection -> Username -> IO (Maybe UUID)
saveUser conn s = do
  newId <- query conn "INSERT INTO users (username) VALUES (?) returning user_id" (Only (unUsername s))
  pure $ case newId of
    [] -> Nothing
    Only i : _ -> Just i

getUser :: Connection -> UUID -> IO (Maybe Username)
getUser conn uid = do
  unames <- query conn "SELECT username FROM users where user_id = ?" (Only uid)
  pure $ case unames of
    [] -> Nothing
    Only s : _ -> mkUsername s

(In real code, of course, we would use a more usable indication of failure than Maybe)

We can even hook this into Haskell’s typeclass system to make this even more rigorous: Username could have its own FromField and ToField instances that push the validation to the driver level.

instance FromField Username where
  fromField f mdata = do
    s :: String <- fromField f mdata
    case mkUsername s of
      Just u  -> pure u
      Nothing -> returnError ConversionFailed f ("Invalid username format: " ++ s)

instance ToField Username where
  toField = toField . unUsername

saveUser :: Connection -> Username -> IO (Maybe UUID)
saveUser conn s = do
  newId <- query conn "INSERT INTO users (username) VALUES (?) returning user_id" (Only s)
  pure $ case newId of
    [] -> Nothing
    Only i : _ -> Just i

getUser :: Connection -> UUID -> IO (Maybe Username)
getUser conn uid = do
  unames <- query conn "SELECT username FROM users where user_id = ?" (Only uid)
  pure $ case unames of
    [] -> Nothing
    Only s : _ -> Just s

Pushing it to the driver level will also unify everything with the driver’s error-handling system.

These ideas are elaborated further in one of the best Haskell posts of all time, Parse, Don’t Validate.

Boolean Blindness

At the heart of it, the previous examples’ cardinal sin was “boolean blindness”. If we have a predicate like validUsername :: String -> Bool, we will branch on that Bool once and throw it away. Instead, by having a function like mkUsername :: String -> Maybe Username, we keep the proof alongside the value for the entire lifetime of the value. We basically pair the string with its proof forever, making them inseparable.

There was another example of such a thing earlier: instead of using null :: [a] -> Bool and gating a call to mean with null, we instead use nonEmpty :: [a] -> Maybe (NonEmpty a), and pass along the proof of non-emptiness alongside the value itself. And, for the rest of that list’s life, it will always be paired with its non-emptiness proof.

Embracing total depravity means always keeping these proofs together, with the witnesses bundled with the value itself, because if you don’t, someone is going to assume it exists when it doesn’t, or drop it unnecessarily.

Boolean blindness also has another facet, which is where Bool itself is not a semantically meaningful type. This is “semantic boolean blindness”.

The classic example is filter :: (a -> Bool) -> [a] -> [a]. It might sound silly until it happens to you, but it is pretty easy to mix up if True means “keep” or “discard”. After all, a “water filter” only lets water through, but a “profanity filter” only rejects profanity. Instead, how about mapMaybe :: (a -> Maybe b) -> [a] -> [b]? In that case, it is clear that Just results are kept, and the Nothing results are discarded.

Sometimes, the boolean is ambiguous as to what it means. You can sort of interpret the 1999 Mars Polar Lander crash this way. Its functions took a boolean based on the state of the legs:

deployThrusters :: Bool -> IO ()

and True and False were misinterpreted. Instead, they could have considered semantically meaningful types: (simplified)

data LegState = Extended | Retracted

deployThrusters :: LegState -> IO ()

Note that Maybe itself is not immune from “semantic blindness” — if you find yourself using a lot of anonymous combinators like Maybe and Either to get around boolean blindness, be aware of falling into algebraic blindness!

Resource Cleanup

Clean-up of finite system resources is another area that is very easy to assume you have a handle on before it gets out of hand and sneaks up on you.

process :: Handle -> IO ()

doTheThing :: FilePath -> IO ()
doTheThing path = do
  h <- openFile path ReadMode
  process h
  hClose h

A bunch of things could go wrong —

You might forget to always hClose a file handle, and if your files come at you dynamically, you’re going to run out of file descriptors, or hold on to locks longer than you should
If process throws an exception, we never get to hClose, and the same issues happen
If another thread throws an asynchronous exception (like a thread cancellation), you have to make sure the close still happens!

The typical solution that other languages (like Python, modern Java) take is to put everything inside a “block” where quitting the block guarantees the closure. In Haskell we have the bracket pattern:

-- strongly discourage using `openFile` and `hClose` directly
withFile :: FilePath -> (Handle -> IO r) -> IO r
withFile path = bracket (openFile path ReadMode) hClose

doTheThing :: FilePath -> IO ()
doTheThing path = withFile path $ \h -> do
  process h

If you never use openFile directly, and always use withFile, all file usage is safe!

But, admittedly, continuations can be annoying to work with. For example, what if you wanted to safely open a list of files?

processAll :: [Handle] -> IO ()

doTheThings :: [FilePath] -> IO ()
doTheThings paths = -- uh...

All of a sudden, not so fun. And what if you had, for example, a Map of files, like Map Username FilePath?

processAll :: Map Username Handle -> IO ()

doTheThings :: Map Username FilePath -> IO ()
doTheThings paths = -- uh...

In another language, at this point, we might just give up and resort to manual opening and closing of files.

But this is Haskell. We have a better solution: cleanup-tracking monads!

This is a classic usage of ContT to let you chain bracket-like continuations:

processTwo :: Handle -> Handle -> IO ()

doTwoThings :: FilePath -> FilePath -> IO ()
doTwoThings path1 path2 = evalContT $ do
    h1 <- ContT $ withFile path1
    h2 <- ContT $ withFile path2
    liftIO $ processTwo h1 h2

processAll :: Map Username Handle -> IO ()

doTheThings :: Map Username FilePath -> IO ()
doTheThings paths = evalContT $ do
    handles <- traverse (ContT . withFile) paths
    liftIO $ processAll handles

However, using ContT doesn’t allow you to do things like early cleanups or canceling cleanup events. It forces us into a last-in, first-out sort of cleanup pattern. If you want to deviate, this might cause you to, for convenience, go for manual resource management. However, we have tools for more fine-grained control, we have things like resourcet ResourceT, which lets you manually control the order of clean-up events, with the guarantee that all of them eventually happen.

import qualified Data.Map as M

-- | Returns set of usernames to close
processAll :: Map Username Handle -> IO (Set Username)

allocateFile :: FilePath -> ResourceT IO (ReleaseKey, Handle)
allocateFile fp = allocate (openFile fp ReadMode) hClose

-- Guarantees that all handles will eventually close, even if `go` crashes
doTheThings :: Map Username FilePath -> IO ()
doTheThings paths = runResourceT $ do
    releasersAndHandlers <- traverse allocateFile paths
    go releasersAndHandlers
  where
    -- normal operation: slowly releases handlers as we drop them
    go :: Map Username (ReleaseKey, Handle) -> ResourceT IO ()
    go currOpen = do
      toClose <- liftIO $ processAll (snd <$> currOpen)
      traverse_ (release . fst) (currOpen `M.restrictKeys` toClose)
      let newOpen = currOpen `M.withoutKeys` toClose
      unless (M.null newOpen) $
        go newOpen

Here we get the best of both worlds: the ability to manually close handlers when they are no longer needed, but also the guarantee that they will eventually be closed.

Embracing Total Depravity

Hopefully these examples, and similar situations, should feel relatable. We’ve all experienced the biting pain of too much self-trust. Or, too much trust in our ability to communicate with team members. Or, too much trust in ourselves 6 months from now. The traumas described here should resonate with you if you have programmed in any capacity for more than a couple of scripts.

The doctrine of total depravity does not mean that we don’t recognize the ability to write sloppy code that works, or that flow states can enable some great feats. After all, we all program with a certain sense of imago machinae. Instead, it means that all such states are fundamentally unstable in their nature and will always fail at some point. The “total” doesn’t mean we are totally cooked, it means this eventual reckoning applies to all such shortcuts.

The problem won’t be solved by “get good”. The problem is solved by utilizing the tooling we are given, especially since Haskell makes them so accessible and easy to pull in.

There’s another layer here that comes as a result of embracing this mindset: you’ll find that you have more mental space to dedicate to things that actually matter! Instead of worrying about inconsequential minutiae and details of your flawed abstractions, you can actually think about your business logic, the flow of your program, and architecting that castle of beauty I know you are capable of.

In the Age of Agentic Coding

Before we end, let’s address the elephant in the room. We’re writing this in 2026, in the middle of one of the biggest revolutions in software engineering in the history of the field. A lot of people have claimed that types and safety are now no longer important in the age of LLMs and agentic coding.

However, these claims seem to miss the fact that the fundamental issue being addressed here exists both in LLMs and humans: the limited “context window” and attention. Humans might be barely able to keep a dozen things in our heads, LLMs might be able to keep a dozen dozen things, but it will still be ultimately finite. So, the more we can move concerns out of our context window (be it biological or mechanical), the less crowded our context windows will be, and the more productive we will be.

Agentic coding is progressing quickly, and over the past few months I have been exploring this a lot, using models hands-on. One conclusion I have found (and, this agrees with everyone else I’ve asked who has been trying the same thing) is that Haskell’s types, in many ways, are the killer productivity secret of agentic coding.

Many of my Haskell coding tasks for an LLM agent often involve:

How will the types change, or what should the types be?
Ralph Wiggum loop to death until the program typechecks, using ghci and cabal.

And, this isn’t 100% effective, but from personal experience it is much more effective than the similar situation without typed guardrails for fast feedback, and without instant compiler feedback. The feedback loop is tighter, the objectives clearer, the constraints more resilient, the tooling more utilized.

I have noticed, also, that my LLM agents often check the types of the APIs using ghci :type, and rarely the documentation of the functions using ghci :docs. So, any “documentation-based contracts” are definitely much more likely to explode in your face in this new world than type-based contracts.

I’m not sure how quickly LLM-based agentic coding will progress, but I am sure that the accidental “dropping” of concerns will continue to be a bottleneck. All of the traits described in this post for humans will continue to be traits of limited context windows for LLMs.

If anything, limited “brain space” might be the bottleneck, for both humans and LLMs. If provide LLMs with properly “depravity-aware” typed code (and encourage them to write it by giving them the right iterative tooling), I truly believe that we have the key to unlocking the full potential of agentic coding.

And…not whatever this tweet is.

The Next Step

Total depravity is all about using types to prevent errors. However, you can only go so far with defensive programming and carefully picking the structure of your types. Sometimes, it feels like you are expending a lot of energy and human effort just picking the perfectly designed data type, only for things out of your hand to ruin your typed castle.

In the next chapter, we’ll see how a little-discussed aspect of Haskell’s type system gives you a powerful tool for opening your mind to new avenues of design that were impossible before. At the same time, we’ll see how we can leverage universal properties of mathematics itself to help us analyze our code in unexpected ways.

Let’s explore this further in the next principle of Five-Point Haskell, Unconditional Election!

Special Thanks

I am very humbled to be supported by an amazing community, who make it possible for me to devote time to researching and writing these posts. Very special thanks to my supporter at the “Amazing” level on patreon, Josh Vera! :)

Also thanks to jackdk’s comment for highlighting extra resources and context that I believe are very useful and relevant!

Gabriella Gonzalez: My experience with vibe coding>

Mon, 02 Feb 2026 00:00:00 +0000

Vibe coding isn't a silver bullet

Oskar Wickström: There and Back Again: From Quickstrom to Bombadil

Tue, 27 Jan 2026 23:00:00 +0000

Today I’m announcing and open-sourcing the Bombadil project — a brand new property-based browser testing framework. I started working on this when joining Antithesis two months ago. While still in its infancy, we’ve decided to build it in the open and share our progress from the start.

We decided on the name Bombadil last week. A few days later, this exploded:

It's going to be tough for startups when all the Lord of the Rings names are taken and the only thing left is something like Bombadil AI.
— Patrick Collison (@patrickc) January 25, 2026

While Quickstrom proved its worth, finding bugs in more than half of the TodoMVC apps, Bombadil aims to improve on its shortcomings while also envisioning a more ambitious future for generative testing of web apps: faster and smarter state space exploration, a modern and usable specification language, better tools for reproducing and debugging test failures, and a better distribution story.

I consider Bombadil the successor of Quickstrom. After years of trying to sustain Quickstrom through various models, I now have a much better answer: building it at Antithesis, making something valuable that is open and free to use, while also strengthening the company’s commercial offering. Bombadil can be used locally or in CI to test your web apps early. Power users can take it further and run Bombadil within Antithesis and its deterministic hypervisor. That gives you perfect reproductions of failed tests. You can even combine Bombadil with other workloads in Antithesis and test your entire stack deterministically. Isn’t that the holy grail of testing?

Bombadil is built from scratch with a focus on accessibility: a new specification language and better tooling for writing and maintaining specs. Right now, I’m working on a specification DSL in TypeScript. It’s based on linear temporal logic, just as Quickstrom, but aims to be a lot more ergonomic. Here’s an example that verifies that error messages eventually disappear:

const errorMessage = extract(  (state) =>  state.document.body.querySelector(".error")  ?.textContent   ?? null );  export const errorEventuallyDisappears = always(  condition(() => errorMessage !== null).implies(  eventually(() => errorMessage === null)  .within(5, "seconds")  ) );

Both the original hacky PureScript DSL and the bespoke language Specstrom were huge obstacles to adoption. Today, TypeScript is a widely adopted language among quality-minded web developers, and if you’re not into that, Bombadil will work with plain JavaScript too.

We’re writing Bombadil in Rust, leveraging its excellent ecosystem to ship a single statically-linked executable — download for your platform, point it at a Chromium-based browser, and off you go!

Bombadil is early but usable. Check it out on GitHub, try it on your projects, and let us know what breaks. Also let us know what it breaks in your systems! Join us on Discord for help and discussion, or follow development on Twitter/X. This is just the beginning — we’re actively seeking feedback and early adopters to help shape where this goes.

Magnus Therning: More on the switch to eglot

Sun, 25 Jan 2026 13:18:00 +0000

Since the switching to eglot I've ended up making a few related changes.

Replacing flycheck with flymake

Since eglot it's written to work with other packages in core, which means it integrates with flymake. The switch comprised

Use :ensure nil to make sure elpaca knows there's nothing to download.
Add a call to flymake-mode to prog-mode-hook.
Define two functions to toggle showing a list of diagnostics for the current buffer and the project.
Redefine the relevant keybindings.

The two functions for toggling showing diagnostics look like this

(defun mes/toggle-flymake-buffer-diagnostics ()
  (interactive)
  (if-let* ((window (get-buffer-window (flymake--diagnostics-buffer-name))))
      (save-selected-window (quit-window nil window))
    (flymake-show-buffer-diagnostics)))

(defun mes/toggle-flymake-project-diagnostics ()
  (interactive)
  (if-let* ((window (get-buffer-window (flymake--project-diagnostics-buffer (projectile-project-root)))))
      (save-selected-window (quit-window nil window))
    (flymake-show-project-diagnostics)))

And the changed keybindings are

flycheck	flymake
flycheck-next-error	flymake-goto-next-error
flycheck-previous-error	flymake-goto-prev-error
mes/toggle-flycheck-error-list	mes/toggle-flymake-buffer-diagnostics
mes/toggle-flycheck-projectile-error-list	mes/toggle-flymake-project-diagnostics

Using `with-eval-after-load` instead of `:after eglot`

When it comes to use-package I keep on being surprised, and after the switch to elpaca I've found some new surprises. One of them was that using :after eglot like this

(use-package haskell-ng-mode
  :afer eglot
  :ensure (:type git
           :repo "git@gitlab.com:magus/haskell-ng-mode.git"
           :branch "main")
  :init
  (add-to-list 'major-mode-remap-alist '(haskell-mode . haskell-ng-mode))
  (add-to-list 'eglot-server-programs '(haskell-ng-mode "haskell-language-server-wrapper" "--lsp"))
  (setq-default eglot-workspace-configuration
                (plist-put eglot-workspace-configuration
                           :haskell
                           '(:formattingProvider "fourmolu"
                             :plugin (:stan (:global-on :json-false)))))
  ...
  :hook
  (haskell-ng-mode . eglot-ensure)
  ...)

would delay initialisation until after eglot had been loaded. However, it turned out that nothing in :init ... seemed to run and upon opening a haskell file no mode was loaded.

After a bit of thinking and tinkering I got it working by removing :after eglot and using with-eval-after-load

(use-package haskell-ng-mode
  :ensure (:type git
           :repo "git@gitlab.com:magus/haskell-ng-mode.git"
           :branch "main")
  :init
  (add-to-list 'major-mode-remap-alist '(haskell-mode . haskell-ng-mode))
  (with-eval-after-load 'eglot
    (add-to-list 'eglot-server-programs '(haskell-ng-mode "haskell-language-server-wrapper" "--lsp"))
    (setq-default eglot-workspace-configuration
                  (plist-put eglot-workspace-configuration
                             :haskell
                             '(:formattingProvider "fourmolu"
                               :plugin (:stan (:global-on :json-false))))))
  ...
  :hook
  (haskell-ng-mode . eglot-ensure)
  ...)

That change worked for haskell, and it seemed to work for python too, but after a little while I realised that python needed a bit more attention.

Getting the configuration for Python to work properly

The python setup looked like this

(use-package python
  :init
  (add-to-list 'major-mode-remap-alist '(python-mode . python-ts-mode))
  (with-eval-after-load 'eglot
    (assoc-delete-all '(python-mode python-ts-mode) eglot-server-programs)
    (add-to-list 'eglot-server-programs
                 `((python-mode python-ts-mode) . ,(eglot-alternatives
                                                    '(("rass" "python") "pylsp")))))
  ...
  :hook (python-ts-mode . eglot-ensure)
  ...)

and it worked all right, but then I visited the package (using elpaca-visit) and realised that the downloaded package was all of emacs. That's a bit of overkill, I'd say.

However, adding :ensure nil didn't have the expected effect of just using the version that's in core. Instead the whole configuration seemed to never take effect and again I was back to the situation where I had to jump to python-ts-mode manually.

The documentation for use-package says that :init is for

Code to run before PACKAGE-NAME has been loaded.

but I'm guessing "before" isn't quite before enough. Then I noticed :preface with the description

Code to be run before everything except :disabled; this can be used to define functions for use in :if, or that should be seen by the byte-compiler.

and yes, "before everything" is early enough. The final python configuration looks like this

(use-package python
  :ensure nil
  :preface
  (add-to-list 'major-mode-remap-alist '(python-mode . python-ts-mode))
  :init
  (with-eval-after-load 'eglot
    (assoc-delete-all '(python-mode python-ts-mode) eglot-server-programs)
    (add-to-list 'eglot-server-programs
                 `((python-mode python-ts-mode) . ,(eglot-alternatives
                                                    '(("rass" "python") "pylsp")))))
  ...
  :hook (python-ts-mode . eglot-ensure)
  ...)

Closing remark

I'm still not sure I have the correct intuition about how to use use-package, but hopefully it's more correct now than before. I have a growing suspicion that use-package changes behaviour based on the package manager I use. Or maybe it's just that some package managers make use-package more forgiving of bad use.

Gabriella Gonzalez: Type-safe eval in Grace>

Tue, 20 Jan 2026 00:00:00 +0000

The case for principled eval support

Magnus Therning: Trying eglot, again

Mon, 19 Jan 2026 07:00:00 +0000

I've been using lsp-mode since I switched to Emacs several years ago. When eglot made into Emacs core I used it very briefly but quickly switched back. Mainly I found eglot a bit too bare-bones; I liked some of the bells and whistles of lsp-ui. Fast-forward a few years and I've grown a bit tired of those bells and whistles. Specifically that it's difficult to make lsp-ui-sideline and lsp-ui-doc work well together. lsp-ui-sidedline is shown on the right side, which is good, but combining it with lsp-ui-doc leads to situations where the popup covers the sideline. What I've done so far is centre the line to bring the sideline text out. I was playing a little bit with making the setting of lsp-ui-doc-position change depending on the location of the current position. It didn't work that well though so I decided to try to find a simpler setup. Instead of simplifying the setup of lsp-config I thought I'd give eglot another shot.

Basic setup

I removed the statements pulling in lsp-mode, lsp-ui, and all language-specific packages like lsp-haskell. Then I added this to configure eglot

(use-package eglot
  :ensure nil
  :custom
  (eglot-autoshutdown t)
  (eglot-confirm-server-edits '((eglot-rename . nil)
                                (t . diff))))

The rest was mainly just switching lsp-mode functions for eglot functions.

lsp-mode function	eglot function
`lsp-deferred`	`eglot-ensure`
`lsp-describe-thing-at-point`	`eldoc`
`lsp-execute-code-action`	`eglot-code-actions`
`lsp-find-type-definition`	`eglot-find-typeDefinition`
`lsp-format-buffer`	`eglot-format-buffer`
`lsp-format-region`	`eglot-format`
`lsp-organize-imports`	`eglot-code-action-organize-imports`
`lsp-rename`	`eglot-rename`
`lsp-workspace-restart`	`eglot-reconnect`
`lsp-workspace-shutdown`	`eglot-shutdown`

I haven't verified that the list is fully correct yet, but it looks good so far.

The one thing I might miss is lenses, and using lsp-avy-lens. However, everything that I use lenses for can be done using actions, and to be honest I don't think I'll miss the huge lens texts from missing type annotations in Haskell.

Configuration

One good thing about lsp-mode's use of language-specific packages is that configuration of the various servers is performed through functions. This makes it easy to discover what options are available, though it also means not all options may be available. In eglot configuration is less organised, I have to know about the options for each language server and put the options into eglot-workspace-configuration myself. It's not always easy to track down what options are available, and I've found no easy way to verify the settings. For instance, with lsp-mode I configures HLS like this

(lsp-haskell-formatting-provider "fourmolu")
(lsp-haskell-plugin-stan-global-on nil)

which translates to this for eglot

(setq-default eglot-workspace-configuration
              (plist-put eglot-workspace-configuration
                         :haskell
                         '(:formattingProvider "fourmolu"
                           :plugin (:stan (:global-on :json-false)))))

and I can verify that this configuration has taken effect because I know enough about the Haskell tools.

I do some development in Python and I used to configure pylsp like this

(lsp-pylsp-plugins-mypy-enabled t)
(lsp-pylsp-plugins-ruff-enabled t)

which I think translates to this for eglot

(setq-default eglot-workspace-configuration
              (plist-put eglot-workspace-configuration
                         :pylsp
                         '(:plugins (:ruff (:enabled t)
                                     :mypy (:enabled t)))))

but I don't know any convenient way of verifying these settings. I'm simply not familiar enough with the Python tools. I can check the value of eglot-workspace-configuration by inspecting it or calling eglot-show-workspace-configuration but is there really no way of asking the language server for its active configuration?

Closing remark

The last time I gave up on eglot very quickly, probably too quickly to be honest. I made these changes to my configuration over the weekend, so the real test of eglot starts when I'm back in the office. I have a feeling I'll stick to it longer this time.

Abhinav Sarkar: Implementing Co, a Small Language With Coroutines #5: Adding Sleep

abhinav@abhinavsarkar.net (Abhinav Sarkar) — Fri, 16 Jan 2026 00:00:00 +0000

In the previous post, we added channels to Co, the small language we are implementing in this series of posts. In this post, we add the sleep primitive to it, enabling time-based coroutine scheduling. We then use sleep to build a simulation of digital logic circuits.

This post was originally published on abhinavsarkar.net.

This post is a part of the series: Implementing Co, a Small Language With Coroutines.

Introduction#

Sleep is a commonly used operation in concurrent programs. It pauses the execution of the current Thread of Computation (ToC) for a specified duration, after which the ToC is resumed automatically. Sleep is used for various purposes: polling for events, delaying execution of an operation, simulating latency, implementing timeouts, and more.

Sleep is generally implemented as a primitive operation in most languages, delegating the actual implementation to the underlying operating system. The operating systemâ€™s scheduler removes the ToC from the list of runnable ToCs, places it in a list of sleeping ToCs, and after the specified duration, moves it back to the list of runnable ToCs for scheduling.

Since Co implements its own ToC (coroutine) scheduler, we implement sleep as a primitive operation within the interpreter itself¹.

Adding Sleep#

We start by exposing sleep and getCurrentMillis as built-in functions to Co:

builtinEnv :: IO Env
builtinEnv = Map.fromList <$> traverse (traverse newIORef) [
    ("print", BuiltinFunction "print" 1 executePrint)
  , ("newChannel",
     BuiltinFunction "newChannel" 0 $ fmap Chan . const (newChannel 0))
  , ("newBufferedChannel",
     BuiltinFunction "newBufferedChannel" 1 executeNewBufferedChannel)
  , ("sleep", BuiltinFunction "sleep" 1 executeSleep)
  , ("getCurrentMillis",
     BuiltinFunction "getCurrentMillis" 0 executeGetCurrentMillis)
  ]

The sleep built-in function takes one argumentâ€”the duration in milliseconds to sleep for. The getCurrentMillis function returns the current time in milliseconds since the Unix epoch. Both of them delegate to the functions explained next.

executeSleep :: [Expr] -> Interpreter Value
executeSleep argEs = evaluate (head argEs) >>= \case
  Num n | n >= 0 -> sleep n >> return Null
  Num n -> throw $ "Sleep time must be non-negative: " <> show n
  _ -> throw "sleep call expected a number argument"

executeGetCurrentMillis :: [Expr] -> Interpreter Value
executeGetCurrentMillis _ =
  Num . fromIntegral . floor . (* 1000) <$> liftIO getPOSIXTime

The executeSleep function evaluates its argument to a number, checks that it is non-negative, and then calls the sleep function in the Interpreter monad. executeGetCurrentMillis calls getPOSIXTime and returns the milliseconds wrapped as a Num.

The implementation of sleep is more involved than other built-in functions because it interacts with the coroutine scheduler. When a coroutine calls sleep, we want to suspend the coroutine, and schedule it to be resumed after the specified duration. There may be multiple coroutines in the sleep state at a time, and they must be resumed according to their wakeup time (time at which sleep was called + sleep duration), and not in any other order. To be efficient, it is also important that the scheduler does not poll repeatedly for new coroutines to wake up and run, but instead waits till the right time. These are the two requirements for our coroutine scheduler. And the solution is: delayed coroutines.

Delayed Coroutines#

The coroutines we have implemented so far were scheduled to run immediately. To implement sleep, we extend the coroutine concept with Delayed Coroutinesâ€”coroutines that are scheduled to run at a specific future time.

data Coroutine a = Coroutine
  { corEnv :: Env
  , corCont :: a -> Interpreter ()
  , corReady :: MVar ()
  }

newCoroutine :: Env -> (a -> Interpreter ()) -> Interpreter (Coroutine a)
newCoroutine env cont = do
  ready <- newMVar ()
  return $ Coroutine env cont ready

Now the Coroutine data type holds an MVar to signal when the coroutine is ready to be run. The old-style coroutines that run immediately are created ready to run by the newCoroutine function. But delayed coroutines are different:

newDelayedCoroutine ::
  Integer -> Env -> (a -> Interpreter ()) -> Interpreter (Coroutine a)
newDelayedCoroutine sleepMillis env cont = do
  ready <- newEmptyMVar
  void $ liftIO $ forkIO $ do
    threadDelay $ fromIntegral sleepMillis * 1000
    putMVar ready ()
  return $ Coroutine env cont ready

The key difference from a regular coroutine is that the MVar used for signaling is created empty. We fork a thread² that sleeps³ for the specified sleep duration, and then signals that the coroutine is ready to run by filling the MVar.

An MVar is a synchronization primitive⁴â€”essentially a mutable box that can hold a value or be empty. When we call takeMVar on an empty MVar, it blocks until another thread fills it. This is what makes it powerful for our use case: instead of the interpreter repeatedly polling the queue asking â€œis this coroutine ready yet?â€�, we let the interpreter wait on the MVar. The forked thread signals readiness at the right time by filling the MVar. The interpreter wakes up immediatelyâ€”no wasted CPU cycles, no busy-waiting.

Queuing Coroutines#

We already have a Queue of coroutines in our Interpreter. It is a min-priority queue sorted by timestamps, which we have been using as a FIFO queue till now. Now we use it for its real purpose: storing delayed coroutines sorted by their wakeup times.

type Queue a = IORef (PQ.MinPQueue TimeSpec a, TimeSpec)

newQueue :: MonadBase IO m => m (Queue a)
newQueue = do
  now <- liftBase currentSystemTime
  newIORef (PQ.empty, now)

queueSize :: MonadBase IO m => Queue a -> m Int
queueSize = fmap (PQ.size . fst) . readIORef

The queue also tracks the maximum wakeup time of all coroutines in the queue. This information is useful for calculating how long the interpreter should sleep before termination.

The core operations on the queue are:

enqueueAt :: TimeSpec -> a -> Queue a -> Interpreter ()
enqueueAt time val queue = atomicModifyIORef' queue $ \(q, maxWakeupTime) ->
  (( PQ.insert time val q,
     if time > maxWakeupTime then time else maxWakeupTime
   ), ())

enqueue :: a -> Queue a -> Interpreter ()
enqueue val queue = do
  now <- currentSystemTime
  enqueueAt now val queue

dequeue :: Queue a -> Interpreter (Maybe a)
dequeue queue = atomicModifyIORef' queue $ \(q, maxWakeupTime) ->
  if PQ.null q
    then ((q, maxWakeupTime), Nothing)
    else let ((_, val), q') = PQ.deleteFindMin q
         in ((q', maxWakeupTime), Just val)

currentSystemTime :: MonadIO m => m TimeSpec
currentSystemTime = liftIO $ getTime Monotonic

We saw the enqueue function earlier:

The enqueueAt function enqueues the given value at the given time in the queue. The enqueue function enqueues the value at the current time, thus scheduling it to run immediately.

The dequeue function dequeues the value with the lowest priority from the queue, which in this case, is the value that is enqueued first.

The currentSystemTime function returns the monotonically increasing current system time.

The dequeue function dequeues the coroutine with lowest priority, so if we use the wakeup time as priority, it will dequeue the coroutine that is to be run next. That works!

The enqueueAt function calculates and tracks the maximum wakeup times of the coroutines as well. Next, we implement the scheduling of delayed coroutines:

scheduleDelayedCoroutine :: TimeSpec -> Coroutine () -> Interpreter ()
scheduleDelayedCoroutine wakeupTime coroutine = do
  State.gets isCoroutines >>= enqueueAt wakeupTime coroutine

The scheduleDelayedCoroutine function enqueues a coroutine in the interpreter coroutine queue with the specified wakeup time. We also improve the runNextCoroutine function to wait for the coroutine to be ready before running it.

runNextCoroutine :: Interpreter ()
runNextCoroutine =
  State.gets isCoroutines >>= dequeue >>= \case
    Nothing -> throwError CoroutineQueueEmpty
    Just Coroutine {..} -> do
      void $ takeMVar corReady
      setEnv corEnv
      corCont ()

The takeMVar function call blocks till the thread that was forked when creating the coroutine wakes up and fills the corReady MVar. So we donâ€™t have to poll the queue.

Thatâ€™s all we have to do for having delayed coroutines.

Implementing Sleep#

With the infrastructure in place, the sleep function becomes straightforward:

sleep :: Integer -> Interpreter ()
sleep millis = do
  now <- currentSystemTime
  let wakeupTime = now + fromNanoSecs (fromIntegral millis * 1000000)
  env <- State.gets isEnv
  callCC $ \cont -> do
    coroutine <- newDelayedCoroutine millis env cont
    scheduleDelayedCoroutine wakeupTime coroutine
    runNextCoroutine

When a coroutine calls sleep, we capture the current environment and use callCC to capture the continuationâ€”the code that should run after the sleep completes. We then create a new delayed coroutine with this continuation, schedule it for the future, and run the next coroutine in the queue. The scheduler machinery takes care of running the delayed coroutine at the right time.

We also modify the awaitTermination function from the previous post to handle delayed coroutines. It now sleeps till the last wakeup time before checking if the queue is empty:

awaitTermination :: Interpreter ()
awaitTermination = do
  (coroutines, maxWakeupTime) <- readIORef =<< State.gets isCoroutines
  dur <- calcSleepDuration maxWakeupTime
  unless (PQ.null coroutines) $ if dur > 0
    then sleep dur >> awaitTermination
    else yield >> awaitTermination

Notice how we use the sleep function we just defined in awaitTermination. The calcSleepDuration function calculates how long to sleep before the last coroutine becomes ready:

calcSleepDuration :: TimeSpec -> Interpreter Integer
calcSleepDuration maxWakeupTime = do
  now <- currentSystemTime
  return $ 1 + fromIntegral (maxWakeupTime - now) `div` 1000000

Thatâ€™s all for sleeping. This may be too much to take in, so letâ€™s go through some examples.

Sleep in Action#

Sleep can be used for polling/waiting for events, delaying execution, simulating latency, implementing timeouts, and more. Letâ€™s see some simple uses.

Sleep Sort#

An interesting example of sleep is the infamous sleep sort, which sorts a list of numbers by spawning a coroutine for each number that sleeps for the duration of that number, then prints it:

function sleepSort(a, b, c, d, e) {
  function printNum(num) {
    sleep(num);
    print(num);
  }
  spawn printNum(a);
  spawn printNum(b);
  spawn printNum(c);
  spawn printNum(d);
  spawn printNum(e);
}

sleepSort(5, 4, 3, 2, 1);

Running this program prints what we expect:

Donâ€™t use sleepSort for sorting your numbers though. Moving on.

JavaScript-like Timeouts and Intervals#

With sleep, we can implement JavaScript-like setTimeout and setInterval functions:

function setTimeout(callback, millis) {
  spawn (function () { sleep(millis); callback(); })();
}

function hello() {
  print("hello");
}

setTimeout(hello, 10000);

The setTimeout function spawns a coroutine that sleeps for the specified duration and then calls the callback function.

function setInterval(callback, seconds) {
  function run() {
    callback();
    setTimeout(run, seconds);
  }
  setTimeout(run, seconds);
}

var start = getCurrentMillis();
function tick(t) {
  var now = getCurrentMillis();
  print(t + " " + (now - start));
}

setInterval(function () { tick("tick"); }, 2000);
sleep(1000);
setInterval(function () { tick("tock"); }, 2000);

The setInterval function repeatedly calls a callback at a fixed interval using setTimeout to reschedule itself. Running the above code prints alternating tick and tock every 1 second, forever:

tick 2005
tock 3008
tick 4009
tock 5009
tick 6015
tock 7013
tick 8017
tock 9019
tick 10020
tock 11021
tick 12021

Notice that the scheduling is not accurate up to milliseconds, but only approximate.

Bonus Round: Digital Circuit Simulation#

As a more complex example of using sleep, we implement a simulator for digital logic circuits, from basic Logic gates to a Ripple carry adder. The idea is to model circuits as a network of wires and gates, where the wires carry digital signal values (0 or 1), and the logic gates transform input signals to output signals with a propagation delay.

The digital circuit simulation example is from the Wizard Book. Quoting an example:

An inverter is a primitive function box [logic gate] that inverts its input. If the input signal to an inverter changes to 0, then one inverter-delay later the inverter will change its output signal to 1. If the input signal to an inverter changes to 1, then one inverter-delay later the inverter will change its output signal to 0.

But first, weâ€™ll need to make some lists.

Conjuring Lists#

We implement a simple cons list (a singly linked list) using a trick from the Wizard book:

function Cons(first, rest) {
  return function (command) {
    if (command == "first") { return first; }
    if (command == "rest") { return rest; }
  };
}

function Empty() { return null;}

function first(list) { return list("first"); }
function rest(list) { return list("rest"); }
function prepend(list, element) { return Cons(element, list); }

Empty creates an empty list, and we grow the list by prepending an element to it by calling the prepend function. first returns the first element of a list, and rest returns the rest of them. Notice that a Cons cell is just a closure that holds references to its first and rest parameters, and returns a selector function to retrieve them.

Next, we define a helper function to call a list of actions, yielding after each one:

function callEach(actions) {
  if (actions != Empty()) {
    callEach(rest(actions));
    first(actions)();
    yield;
  }
}

Wires#

A wire holds a mutable signal value and a list of actions to call when the signal changes:

function Wire() {
  var signalValue = 0;
  var actions = Empty();

  return function (command) {
    if (command == "get-signal") {
      return signalValue;
    }
    if (command == "set-signal") {
      return function (newValue) {
        if (signalValue != newValue) {
          signalValue = newValue;
          callEach(actions);
        }
      };
    }
    if (command == "add-action") {
      return function (action) {
        actions = prepend(actions, action);
        action();
      };
    }
    return null;
  };
}

function getSignal(wire) { return wire("get-signal"); }
function setSignal(wire, signal) { wire("set-signal")(signal); }
function addAction(wire, action) { wire("add-action")(action); }

function connect(a, b) {
  function propagate() {
    setSignal(b, getSignal(a));
  }
  addAction(a, propagate);
}

A wire provides three operations:

get-signal: returns the current signal value.
set-signal: sets a new signal value and calls all actions if the value changed.
add-action: adds an action to be called when the signal changes, and calls it immediately.

The connect function connects two wires, causing the signal from one to propagate to another.

Logic Gates#

First, we define the basic logic operations:

function and(a, b) {
  if (a == 1) { if (b == 1) { return 1; } }
  return 0;
}

function or(a, b) {
  if (a == 1) { return 1; }
  if (b == 1) { return 1; }
  return 0;
}

function not(a) {
  if (a == 1) { return 0; }
  return 1;
}

And a utility function to schedule a function to run after a delay:

function runAfter(millis, func) {
  spawn (function () {
    sleep(millis);
    func();
  })();
}

With these building blocks, we define the logic gates. Each gate computes its output based on its inputs and schedules the output update after a propagation delay specific to the gate:

var andGateDelay = 300;

function AndGate(a, b, out) {
  function compute() {
    var newSignal = and(getSignal(a), getSignal(b));
    runAfter(andGateDelay, function () { setSignal(out, newSignal); });
  }
  addAction(a, compute);
  addAction(b, compute);
}

var orGateDelay = 500;

function OrGate(a, b, out) {
  function compute() {
    var newSignal = or(getSignal(a), getSignal(b));
    runAfter(orGateDelay, function () { setSignal(out, newSignal); });
  }
  addAction(a, compute);
  addAction(b, compute);
}

var notGateDelay = 200;

function NotGate(a, out) {
  function compute() {
    var newSignal = not(getSignal(a));
    runAfter(notGateDelay, function () { setSignal(out, newSignal); });
  }
  addAction(a, compute);
}

We add the compute action to each input wire, which runs when the input signals change, and sets the signal on the output wire after a delay.

Letâ€™s test an And gate:

var input1 = Wire();
var input2 = Wire();
var output = Wire();

log(">> Setting up the probes");
addProbe("input1", input1);
addProbe("input2", input2);
addProbe("output", output);

log(">> Setting up the And gate");
AndGate(input1, input2, output);

log(">> Setting input1 to 1");
setSignal(input1, 1);

sleep(1000);

log(">> Setting input2 to 1");
setSignal(input2, 1);

For probing, we define a helper that logs signal changes with milliseconds elapsed since start of the run:

var start = getCurrentMillis();

function log(msg) {
  print("[" + (getCurrentMillis() - start) + "] " + msg);
}

function addProbe(label, wire) {
  addAction(wire, function () {
    log(label + ": " + getSignal(wire));
  });
}

The output:

[0] >> Setting up the probes
[0] input1: 0
[0] input2: 0
[0] output: 0
[0] >> Setting up the And gate
[0] >> Setting input1 to 1
[0] input1: 1
[1005] >> Setting input2 to 1
[1006] input2: 1
[1310] output: 1

It works as expected. You can notice the sleep and the And gate delay in action.

Adders#

Using the basic logic gates, next we build adders. A Half adder is a digital circuit that adds two bits:

Half Adder

It has two input signals/bits Input1 and Input2, and two output bits Sum and Carry. We simply connect the And, Or and Not gates with input, output and intermediate wires in our code as shown in the diagram:

function HalfAdder(label, input1, input2, sum, carry) {
  var a = Wire();
  var b = Wire();
  addProbe(label + "-a", a);
  addProbe(label + "-b", b);

  OrGate(input1, input2, a);
  AndGate(input1, input2, carry);
  NotGate(carry, b);
  AndGate(a, b, sum);
}

Nice and simple. Letâ€™s test it:

var input1 = Wire();
var input2 = Wire();
var sum = Wire();
var carry = Wire();

log(">> Setting up the probes");
addProbe("input1", input1);
addProbe("input2", input2);
addProbe("sum", sum);
addProbe("carry", carry);

log(">> Setting up the half-adder");
HalfAdder("half-adder", input1, input2, sum, carry);

log(">> Setting input1 to 1");
setSignal(input1, 1);

sleep(1000);

log(">> Setting input2 to 1");
setSignal(input2, 1);

And the output:

[0] >> Setting up the probes
[0] input1: 0
[0] input2: 0
[0] sum: 0
[0] carry: 0
[0] >> Setting up the half-adder
[0] half-adder-a: 0
[0] half-adder-b: 0
[1] >> Setting input1 to 1
[1] input1: 1
[205] half-adder-b: 1
[505] half-adder-a: 1
[811] sum: 1
[1003] >> Setting input2 to 1
[1003] input2: 1
[1307] carry: 1
[1508] half-adder-b: 0
[1811] sum: 0

In binary, 1 + 1 = 10. Correct! Notice again how the signal propagation through the gates is delayed. Next up is the full adder.

A Full adder adds three bits, two inputs and a carry-in:

Full Adder

Notice that a full adder uses two half adders. Again, we follow the diagram and connect the wires:

function FullAdder(label, input1, input2, carryIn, sum, carryOut) {
  var hsum = Wire();
  var carry1 = Wire();
  var carry2 = Wire();

  addProbe(label + "-hsum", hsum);
  addProbe(label + "-carry1", carry1);
  addProbe(label + "-carry2", carry2);

  HalfAdder(label + "-HA-" + 1, input2, carryIn, hsum, carry1);
  HalfAdder(label + "-HA-" + 2, input1, hsum, sum, carry2);
  OrGate(carry1, carry2, carryOut);
}

Letâ€™s skip the demo for full adder and jump to something more exciting.

Ripple-carry Adder#

A Ripple-carry adder chains together multiple full adders to add multi-bit numbers. The diagram below shows a four-bit adder:

Ripple-carry Adder

We create a ripple-carry adder that can add any number of bits. First we need some helper functions:

function Bits(label, n) {
  var bits = Empty();
  var bit = null;
  while (n > 0) {
    bit = Wire();
    addProbe(label + (n - 1), bit);
    bits = prepend(bits, bit);
    n = n - 1;
  }
  return bits;
}

function setBits(bits, bitValues) {
  if (bits != Empty()) {
    setSignal(first(bits), first(bitValues));
    setBits(rest(bits), rest(bitValues));
  }
}

Bits creates a list of wires to represent an N-bit input/output. setBits sets the bits of a N-bit wire list to a given N-bit value. Now we write a ripple-carry adder:

function RippleCarryAdder(label, a, b, carryIn, sum, carryOut) {
  var aBits = a;
  var bBits = b;
  var sumBits = sum;
  var cIn = carryIn;
  var cOut = null;
  var i = 1;
  while (aBits != Empty()) {
    cOut = Wire();
    addProbe(label + "-carryIn-" + i, cIn);
    addProbe(label + "-carryOut-" + i, cOut);
    FullAdder(label + "-FA-" + i,
      first(aBits), first(bBits), cIn, first(sumBits), cOut);
    aBits = rest(aBits);
    bBits = rest(bBits);
    sumBits = rest(sumBits);
    cIn = cOut;
    i = i + 1;
  }
  connect(cOut, carryOut);
}

The ripple-carry adder uses one full adder per bit, cascading the carry-out bit of each input bit-pairâ€™s sum to the next pair of bits. To demonstrate, letâ€™s add two 4-bit numbers:

log(">> Setting up the probes");
var a = Bits("a", 4);
var b = Bits("b", 4);
var sum = Bits("sum", 4);
var carryIn = Wire();
var carryOut = Wire();

addProbe("carryIn", carryIn);
addProbe("carryOut", carryOut);

log(">> Setting up the ripple carry adder");
RippleCarryAdder("RCA", a, b, carryIn, sum, carryOut);

log(">> Setting carryIn to 0");
setSignal(carryIn, 0);

log(">> Setting a to 5 = 0101 in binary");
setBits(a, Cons(1, Cons(0, Cons(1, Cons(0, Empty())))));

sleep(1000);

log(">> Setting b to 11 = 1011 in binary");
setBits(b, Cons(1, Cons(1, Cons(0, Cons(1, Empty())))));

This one runs for a while because of the collective delays.

The output

[0] >> Setting up the probes
[1] a3: 0
[1] a2: 0
[1] a1: 0
[1] a0: 0
[1] b3: 0
[1] b2: 0
[1] b1: 0
[1] b0: 0
[1] sum3: 0
[1] sum2: 0
[1] sum1: 0
[1] sum0: 0
[1] carryIn: 0
[1] carryOut: 0
[1] >> Setting up the ripple carry adder
[1] RCA-carryIn-1: 0
[1] RCA-carryOut-1: 0
[1] RCA-FA-1-hsum: 0
[1] RCA-FA-1-carry1: 0
[1] RCA-FA-1-carry2: 0
[1] RCA-FA-1-HA-1-a: 0
[1] RCA-FA-1-HA-1-b: 0
[1] RCA-FA-1-HA-2-a: 0
[1] RCA-FA-1-HA-2-b: 0
[1] RCA-carryIn-2: 0
[1] RCA-carryOut-2: 0
[1] RCA-FA-2-hsum: 0
[2] RCA-FA-2-carry1: 0
[2] RCA-FA-2-carry2: 0
[2] RCA-FA-2-HA-1-a: 0
[2] RCA-FA-2-HA-1-b: 0
[2] RCA-FA-2-HA-2-a: 0
[2] RCA-FA-2-HA-2-b: 0
[2] RCA-carryIn-3: 0
[2] RCA-carryOut-3: 0
[2] RCA-FA-3-hsum: 0
[2] RCA-FA-3-carry1: 0
[2] RCA-FA-3-carry2: 0
[2] RCA-FA-3-HA-1-a: 0
[2] RCA-FA-3-HA-1-b: 0
[2] RCA-FA-3-HA-2-a: 0
[2] RCA-FA-3-HA-2-b: 0
[2] RCA-carryIn-4: 0
[2] RCA-carryOut-4: 0
[2] RCA-FA-4-hsum: 0
[2] RCA-FA-4-carry1: 0
[2] RCA-FA-4-carry2: 0
[2] RCA-FA-4-HA-1-a: 0
[2] RCA-FA-4-HA-1-b: 0
[2] RCA-FA-4-HA-2-a: 0
[2] RCA-FA-4-HA-2-b: 0
[2] >> Setting carryIn to 0
[2] >> Setting a to 5 = 0101 in binary
[2] a0: 1
[3] a2: 1
[205] RCA-FA-1-HA-1-b: 1
[205] RCA-FA-1-HA-2-b: 1
[205] RCA-FA-2-HA-1-b: 1
[207] RCA-FA-2-HA-2-b: 1
[207] RCA-FA-3-HA-1-b: 1
[207] RCA-FA-3-HA-2-b: 1
[207] RCA-FA-4-HA-1-b: 1
[207] RCA-FA-4-HA-2-b: 1
[505] RCA-FA-1-HA-2-a: 1
[505] RCA-FA-3-HA-2-a: 1
[810] sum0: 1
[810] sum2: 1
[1014] >> Setting b to 11 = 1011 in binary
[1015] b0: 1
[1015] b1: 1
[1016] b3: 1
[1521] RCA-FA-1-HA-1-a: 1
[1521] RCA-FA-2-HA-1-a: 1
[1521] RCA-FA-4-HA-1-a: 1
[1826] RCA-FA-1-hsum: 1
[1826] RCA-FA-2-hsum: 1
[1826] RCA-FA-4-hsum: 1
[2129] RCA-FA-1-carry2: 1
[2331] RCA-FA-2-HA-2-a: 1
[2331] RCA-FA-4-HA-2-a: 1
[2331] RCA-FA-1-HA-2-b: 0
[2633] RCA-carryOut-1: 1
[2633] sum1: 1
[2633] sum3: 1
[2633] sum0: 0
[2634] RCA-carryIn-2: 1
[2936] RCA-FA-2-carry1: 1
[3138] RCA-FA-2-HA-1-b: 0
[3440] RCA-carryOut-2: 1
[3440] RCA-FA-2-hsum: 0
[3440] RCA-carryIn-3: 1
[3946] RCA-FA-2-HA-2-a: 0
[3946] RCA-FA-3-HA-1-a: 1
[4251] sum1: 0
[4252] RCA-FA-3-hsum: 1
[4556] RCA-FA-3-carry2: 1
[4756] RCA-FA-3-HA-2-b: 0
[5059] RCA-carryOut-3: 1
[5059] sum2: 0
[5059] RCA-carryIn-4: 1
[5363] RCA-FA-4-carry1: 1
[5564] RCA-FA-4-HA-1-b: 0
[5867] RCA-carryOut-4: 1
[5867] RCA-FA-4-hsum: 0
[5867] carryOut: 1
[6370] RCA-FA-4-HA-2-a: 0
[6675] sum3: 0

Let me pick out the final output:

sum0: 0
sum1: 0
sum2: 0
carryOut: 1
sum3: 0

We add 0101 and 1011 in binary, resulting in 10000, which is correct again. Everything works perfectly. With sleep, weâ€™ve now implemented all major features of Coâ€”a complete concurrent language with first-class coroutines, channels, and time-based scheduling.

Conclusion#

With the addition of sleep, weâ€™ve completed our implementation of Coâ€”a small language with coroutines and channels. Over these five posts, we went from parsing source code to building a full interpreter that handles cooperative multitasking using coroutines.

The key insight was realizing that coroutines are just environments plus continuations. By designing our interpreter to use continuation-passing style, we gained the ability to suspend execution at any point and resume it later. Channels built naturally on top of that, providing a way for coroutines to synchronize and pass messages. And sleep extended the scheduler to handle time-based execution, unlocking patterns like timeouts and periodic tasks.

The examples we built along the wayâ€”pubsub system, actor system, and digital circuit simulationâ€”show what becomes possible once these primitives are in place. Starting with basic arithmetic and functions, we ended up with a language capable of expressing real concurrent programs.

What comes next? Maybe a compiler for Co? Stay tuned by subscribing to the feed or the email newsletter.

The full code for the Co interpreter is available here.

If you have any questions or comments, please leave a comment below. If you liked this post, please share it. Thanks for reading!

The sleep implementation in Co is not interruptible. That is, if a coroutine is sleeping, it cannot be resumed before the specified duration. This is different from sleep implementations in most programming languages, where the sleep operation can be interrupted by sending a signal to the sleeping ToC.â†©ï¸�
Threads in GHC are Green Threads and are very cheap to create and run. It is perfectly okay to fork a new one for each delayed coroutine.â†©ï¸�
So in a way, we cheat here by using the sleep primitive provided by the GHC runtime to implement our sleep primitive. If we write a compiler for Co, weâ€™ll have to write our own runtime where weâ€™ll have to implement our sleep function using the functionalities provided by the operating systems.â†©ï¸�
To learn more about how MVars can be used to communicate between threads, read the chapter 24 of Real World Haskell.â†©ï¸�

This post is a part of the series: Implementing Co, a Small Language With Coroutines.

Thanks for reading this post via feed. Feeds are great, and you're great for using them. â™¥

This post was originally published on abhinavsarkar.net.

Like, repost, or comment on:

Brent Yorgey: Disco Live!

Fri, 09 Jan 2026 00:00:00 +0000

Disco Live!

Posted on January 9, 2026
Tagged disco, web, WASM, UI, teaching, discrete, math, Haskell

Share on

A couple months ago, I asked for help creating a web interface for Disco, a student-oriented programming language for teaching functional programming and discrete mathematics. I am very happy to report that Heinrich Apfelmus responded to the call, and this is the result:

Disco Live!

It’s fairly bare bones at the moment, but it’s a fantastic place to start, and far more than I would have been able to accomplish in a few weeks. Give it a try, and let me know of any bugs you find! You are also very welcome to contribute fixes, features, etc. on GitHub:

If you want to know more about Disco, you can read a paper about it here, or read the language documentation.

Well-Typed.Com: The Haskell Debugger for GHC 9.14

Thu, 08 Jan 2026 00:00:00 +0000

The Haskell Debugger is ready to use with GHC-9.14!

The installation, configuration, and talks can be found in the official website. The tl;dr first step is installing the debugger:

$ ghc --version # MUST BE GHC 9.14
The Glorious Glasgow Haskell Compilation System, version 9.14.1

$ cabal install haskell-debugger --allow-newer=base,time,containers,ghc,ghc-bignum,template-haskell --enable-executable-dynamic # ON WINDOWS, DO NOT PASS --enable-executable-dynamic
...

$ ~/.local/bin/hdb --version # VERIFY IT'S THE LATEST!
Haskell Debugger, version 0.11.0.0

The second step is configuring your editor to use the debugger via the Debug Adapter Protocol (DAP).

For VSCode, install the haskell debugger extension.
For Neovim, install nvim-dap and configure it for haskell-debugger
For other editors, consult your DAP documentation and let others know how!

Bug reports and discussions are welcome in the haskell-debugger issue tracker.

My MuniHac 2025 talk also walks through the installation, usage, and design of the debugger. Do note much has been improved since the talk was given, and much more will still improve.

A little bit more info

The debugger work is sponsored by Mercury. It’s the project in which I’ve spent most of my full working days (for almost a full year now), with the invaluable help from my team at Well-Typed.

The debugger is meant to work both on trivial files and on large and complex codebases.¹ It is a GHC application so all features are supported. Like HLS, it also uses hie-bios to automatically configure the session based on your cabal or stack project.

Robustness is a main goal of the debugger. If anything doesn’t work, or if you have performance issues, or something crashes, please don’t hesitate to submit a bug. We’ve got a small but respectable testsuite, and have tested performance by debugging GHC itself, but there’s much still to be fixed and improved.

Roadmap: There’s a lot to do. I’m currently working on callstacks and multi-threaded support. Do let me know what features would be most important to you, so I can also factor that into the future planning.

Although, for large codebases, the usability is still rough around the edges because of possibly long bytecode compilation times, and library code not being interpreted. We’ve made considerable progress to improve this with the bytecode artifacts work.↩︎

Well-Typed.Com: The Haskell Debugger for GHC 9.14

Thu, 08 Jan 2026 00:00:00 +0000

The Haskell Debugger is ready to use with GHC-9.14!

The installation, configuration, and talks can be found in the official website. The tl;dr first step is installing the debugger:

$ ghc --version # MUST BE GHC 9.14
The Glorious Glasgow Haskell Compilation System, version 9.14.1

$ cabal install haskell-debugger --allow-newer=base,time,containers,ghc,ghc-bignum,template-haskell --enable-executable-dynamic # ON WINDOWS, DO NOT PASS --enable-executable-dynamic
...

$ ~/.local/bin/hdb --version # VERIFY IT'S THE LATEST!
Haskell Debugger, version 0.11.0.0

The second step is configuring your editor to use the debugger via the Debug Adapter Protocol (DAP).

For VSCode, install the haskell debugger extension.
For Neovim, install nvim-dap and configure it for haskell-debugger
For other editors, consult your DAP documentation and let others know how!

Bug reports and discussions are welcome in the haskell-debugger issue tracker.

My MuniHac 2025 talk also walks through the installation, usage, and design of the debugger. Do note much has been improved since the talk was given, and much more will still improve.

A little bit more info

The debugger work is sponsored by Mercury. It’s the project in which I’ve spent most of my full working days (for almost a full year now), with the invaluable help from my team at Well-Typed.

Although, for large codebases, the usability is still rough around the edges because of possibly long bytecode compilation times, and library code not being interpreted. We’ve made considerable progress to improve this with the bytecode artifacts work.↩︎

Magnus Therning: Validation of data in a servant server

Sun, 04 Jan 2026 09:58:00 +0000

I've been playing around with adding more validation of data received by an HTTP endpoint in a servant server. Defining a type with a FromJSON instance is very easy, just derive a Generic instance and it just works. Here's a simple example

data Person = Person
    { name :: Text
    , age :: Int
    , occupation :: Occupation
    }
    deriving (Generic, Show)
    deriving (FromJSON, ToJSON) via (Generically Person)

data Occupation = UnderAge | Student | Unemployed | SelfEmployed | Retired | Occupation Text
    deriving (Eq, Generic, Ord, Show)
    deriving (FromJSON, ToJSON) via (Generically Occupation)

However, the validation is rather limited, basically it's just checking that each field is present and of the correct type. For the type above I'd like to enforce some constraints for the combination of age and occupation.

The steps I thought of are

Hide the default constructor and define a smart one. (This is the standard suggestion for placing extra constraints values.)
Manually define the FromJSON instance using the Generic instance to limit the amount of code and the smart constructor.

The smart constructor

I give the constructor the result type Either String Person to make sure it can both be usable in code and when defining parseJSON.

mkPerson :: Text -> Int -> Occupation -> Either String Person
mkPerson name age occupation = do
    guardE mustBeUnderAge
    guardE notUnderAge
    guardE tooOldToBeStudent
    guardE mustBeRetired
    pure $ Person name age occupation
  where
    guardE (pred, err) = when pred $ Left err
    mustBeUnderAge = (age < 8 && occupation > UnderAge, "too young for occupation")
    notUnderAge = (age > 15 && occupation == UnderAge, "too old to be under age")
    tooOldToBeStudent = (age > 45 && occupation == Student, "too old to be a student")
    mustBeRetired = (age > 65 && occupation /= Retired, "too old to not be retired")

Here I'm making use of Either e being a Monad and use when to apply the constraints and ensure the reason for failure is given to the caller.

The `FromJSON` instance

When defining the instance I take advantage of the Generic instance to make the implementation short and simple.

instance FromJSON Person where
    parseJSON v = do
        Person{name, age, occupation} <- genericParseJSON defaultOptions v
        either fail pure $ mkPerson name age occupation

If there are many more fields in the type I'd consider using RecordWildCards.

Conclusion

No, it's nothing ground-breaking but I think it's a fairly nice example of how things can fit together in Haskell.

Joachim Breitner: Seemingly impossible programs in Lean

mail@joachim-breitner.de (Joachim Breitner) — Fri, 02 Jan 2026 14:30:24 +0000

In 2007, Martin Escardo wrote a often-read blog post about “Seemingly impossible functional programs”. One such seemingly impossible function is find, which takes a predicate on infinite sequences of bits, and returns an infinite sequence for which that predicate hold (unless the predicate is just always false, in which case it returns some arbitrary sequence).

Inspired by conversations with and experiments by Massin Guerdi at the dinner of LeaningIn 2025 in Berlin (yes, this blog post has been in my pipeline for far too long), I wanted to play around these concepts in Lean.

Let’s represent infinite sequences of bits as functions from Nat to Bit, and give them a nice name, and some basic functionality, including a binary operator for consing an element to the front:

import Mathlib.Data.Nat.Find

abbrev Bit := Bool

def Cantor : Type := Nat → Bit

def Cantor.head (a : Cantor) : Bit := a 0

def Cantor.tail (a : Cantor) : Cantor := fun i => a (i + 1)

@[simp, grind] def Cantor.cons (x : Bit) (a : Cantor) : Cantor
  | 0 => x
  | i+1 => a i

infix:60 " # " => Cantor.cons

With this in place, we can write Escardo’s function in Lean. His blog post discusses a few variants; I’ll focus on just one of them:

mutual
  partial def forsome (p : Cantor → Bool) : Bool :=
    p (find p)

  partial def find (p : Cantor → Bool) : Cantor :=
    have b := forsome (fun a => p (true # a))
    (b # find (fun a => p (b # a)))
end

We define find together with forsome, which checks if the predicate p holds for any sequence. Using that find sets the first element of the result to true if there exists a squence starting with true, else to false, and then tries to find the rest of the sequence.

It is a bit of a brian twiter that this code works, but it does:

def fifth_false : Cantor → Bool := fun a => not (a 5)

/-- info: [true, true, true, true, true, false, true, true, true, true] -/
#guard_msgs in
#eval List.ofFn (fun (i : Fin 10) => find fifth_false i)

Of course, in Lean we don’t just want to define these functions, but we want to prove that they do what we expect them to do.

Above we defined them as partial functions, even though we hope that they are not actually partial: The partial keyword means that we don’t have to do a termination proof, but also that we cannot prove anything about these functions.

So can we convince Lean that these functions are total after all? We can, but it’s a bit of a puzzle, and we have to adjust the definitions.

First of all, these “seemingly impossible functions” are only possible because we assume that the predicate we pass to it, p, is computable and total. This is where the whole magic comes from, and I recommend to read Escardo’s blog posts and papers for more on this fascinating topic. In particular, you will learn that a predicate on Cantor that is computable and total necessarily only looks at some initial fragment of the sequence. The length of that prefix is called the “modulus”. So if we hope to prove termination of find and forsome, we have to restrict their argument p to only such computable predicates.

To that end I introduce HasModulus and the subtype of predicates on Cantor that have such a modulus:

-- Extensional (!) modulus of uniform continuity
def HasModulus (p : Cantor → α) := ∃ n, ∀ a b : Cantor, (∀ i < n, a i = b i) → p a = p b

@[ext] structure CantorPred where
  pred : Cantor → Bool
  hasModulus : HasModulus pred

The modulus of such a predicate is now the least prefix lenght that determines the predicate. In particular, if the modulus is zero, the predicate is constant:

namespace CantorPred

variable (p : CantorPred)

noncomputable def modulus : Nat :=
  open Classical in Nat.find p.hasModulus

theorem eq_of_modulus : ∀a b : Cantor, (∀ i < p.modulus, a i = b i) → p a = p b := by
  open Classical in
  unfold modulus
  exact Nat.find_spec p.hasModulus

theorem eq_of_modulus_eq_0 (hm : p.modulus = 0) : ∀ a b, p a = p b := by
  intro a b
  apply p.eq_of_modulus
  simp [hm]

Because we want to work with CantorPred and not Cantor → Bool I have to define some operations on that new type; in particular the “cons element before predicate” operation that we saw above in find:

def comp_cons (b : Bit) : CantorPred where
  pred := fun a => p (b # a)
  hasModulus := by
    obtain ⟨n, h_n⟩ := p.hasModulus
    cases n with
    | zero => exists 0; grind
    | succ m =>
      exists m
      intro a b heq
      simp
      apply h_n
      intro i hi
      cases i
      · rfl
      · grind

@[simp, grind =] theorem comp_cons_pred (x : Bit) (a : Cantor) :
  (p.comp_cons x) a = p (x # a) := rfl

For this operation we know that the modulus decreases (if it wasn’t already zero):

theorem comp_cons_modulus (x : Bit) :
    (p.comp_cons x).modulus ≤ p.modulus - 1 := by
  open Classical in
  apply Nat.find_le
  intro a b hab
  apply p.eq_of_modulus
  cases hh : p.modulus
  · simp
  · intro i hi
    cases i
    · grind
    · grind
grind_pattern comp_cons_modulus => (p.comp_cons x).modulus

We can rewrite the find function above to use these operations:

mutual
  partial def forsome (p : CantorPred) : Bool := p (find p)

  partial def find (p : CantorPred) : Cantor := fun i =>
    have b := forsome (p.comp_cons true)
    (b # find (p.comp_cons b)) i
end

I have also eta-expanded the Cantor function returned by find; there is now a fun i => … i around the body. We’ll shortly see why that is needed.

Now have everything in place to attempt a termination proof. Before we do that proof, we could step back and try to come up with an informal termination argument.

The recursive call from forsome to find doesn’t decrease any argument at all. This is ok if all calls from find to forsome are decreasing.
The recursive call from find to find decreases the index i as the recursive call is behind the Cantor.cons operation that shifts the index. Good.
The recursive call from find to forsome decreases the modulus of the argument p, if it wasn’t already zero.

But if was zero, it does not decrease it! But if it zero, then the call from forsome to find doesn’t actually need to call find, because then p doesn’t look at its argument.

We can express all this reasoning as a termination measure in the form of a lexicographic triple. The 0 and 1 in the middle component mean that for zero modulus, we can call forsome from find “for free”.

mutual
  def forsome (p : CantorPred) : Bool := p (find p)
  termination_by (p.modulus, if p.modulus = 0 then 0 else 1, 0)
  decreasing_by grind

  def find (p : CantorPred) : Cantor := fun i =>
    have b := forsome (p.comp_cons true)
    (b # find (p.comp_cons b)) i
  termination_by i => (p.modulus, if p.modulus = 0 then 1 else 0, i)
  decreasing_by all_goals grind
end

The termination proof doesn’t go through just yet: Lean is not able to see that (_ # p) i will call p with i - 1, and it does not see that p (find p) only uses find p if the modulus of p is non-zero. We can use the wf_preprocess feature to tell it about that:

The following theorem replaces a call to p f, where p is a function parameter, with the slightly more complex but provably equivalent expression on the right, where the call to f is no in the else branch of an if-then-else and thus has ¬p.modulus = 0 in scope:

@[wf_preprocess]
theorem coe_wf (p : CantorPred) :
    (wfParam p) f = p (if _ : p.modulus = 0 then fun _ => false else f) := by
  split
  next h => apply p.eq_of_modulus_eq_0 h
  next => rfl

And similarly we replace (_ # p) i with a variant that extend the context with information on how p is called:

def cantor_cons' (x : Bit) (i : Nat) (a : ∀ j, j + 1 = i → Bit) : Bit :=
  match i with
  | 0 => x
  | j + 1 => a j (by grind)

@[wf_preprocess] theorem cantor_cons_congr (b : Bit) (a : Cantor) (i : Nat) :
  (b # a) i = cantor_cons' b i (fun j _ => a j) := by cases i <;> rfl

After these declarations, the above definition of forsome and find goes through!

It remains to now prove that they do what they should, by a simple induction on the modulus of p:

@[simp, grind =] theorem tail_cons_eq (a : Cantor) : (x # a).tail = a := by
  funext i; simp [Cantor.tail, Cantor.cons]

@[simp, grind =] theorem head_cons_tail_eq (a : Cantor) : a.head # a.tail = a := by
  funext i; cases i <;> rfl

theorem find_correct (p : CantorPred) (h_exists : ∃ a, p a) : p (find p) := by
  by_cases h0 : p.modulus = 0
  · obtain ⟨a, h_a⟩ := h_exists
    rw [← h_a]
    apply p.eq_of_modulus_eq_0 h0
  · rw [find.eq_unfold, forsome.eq_unfold]
    dsimp -zeta
    extract_lets b
    change p (_ # _)
    by_cases htrue : ∃ a, p (true # a)
    next =>
      have := find_correct (p.comp_cons true) htrue
      grind
    next =>
      have : b = false := by grind
      clear_value b; subst b
      have hfalse : ∃ a, p (false # a) := by
        obtain ⟨a, h_a⟩ := h_exists
        cases h : a.head
        · exists Cantor.tail a
          grind
        · exfalso
          apply htrue
          exists Cantor.tail a
          grind
      clear h_exists
      exact find_correct (p.comp_cons false) hfalse
termination_by p.modulus
decreasing_by all_goals grind

theorem forsome_correct (p : CantorPred) :
    forsome p ↔ (∃ a, p a) where
  mp hfind := by unfold forsome at hfind; exists find p
  mpr hex := by unfold forsome; exact find_correct p hex

This is pretty nice! However there is more to do. For example, Escardo has a “massively faster” variant of find that we can implement as a partial function in Lean:

def findBit (p : Bit → Bool) : Bit :=
  if p false then false else true

def branch (x : Bit) (l r : Cantor) : Cantor :=
  fun n =>
    if n = 0      then x
    else if 2 ∣ n then r ((n - 2) / 2)
                  else l ((n - 1) / 2)

mutual
  partial def forsome (p : Cantor -> Bool) : Bool :=
    p (find p)

  partial def find (p : Cantor -> Bool) : Cantor :=
    let x := findBit (fun x => forsome (fun l => forsome (fun r => p (branch x l r))))
    let l := find (fun l => forsome (fun r => p (branch x l r)))
    let r := find (fun r => p (branch x l r))
    branch x l r
end

But can we get this past Lean’s termination checker? In order to prove that the modulus of p is decreasing, we’d have to know that, for example, find (fun r => p (branch x l r)) is behaving nicely. Unforunately, it is rather hard to do termination proof for a function that relies on the behaviour of the function itself.

So I’ll leave this open as a future exercise.

I have dumped the code for this post at https://github.com/nomeata/lean-cantor.

Sandy Maguire: An Algebraic Theory of Music

Fri, 02 Jan 2026 14:27:00 +0000

In my last post, I was struggling towards an algebraic theory of music. This idea has been burning in my mind ever since, and I wanted to give some updates with where I’ve landed.

Differentiating Voices from Music

We begin by modeling a musical voice, which is, roughly speaking, the abstract version of a human voice. The voice can be doing one thing at a time, or can choose to not be doing anything.

Voices are modeled by step functions, which are divisions of the real line into discrete chunks. We interpret each discrete chunk as a note being played by the voice for the duration of the chunk.

This gives rise to a nice applicative structure that I alluded to in my previous post:

liftA2 f
  |---- a ----|-- b --|
  |-- x --|---- y ----|
=
  |- fax -|fay|- fby -|

where we take the union of the note boundaries in order to form the applicative. If either voice is resting, so too is the applicative. There is also an Alternative instance here, which chooses the first non-rest.

There is a similar monoidal structure here, where multiplication is given by “play these two things simultaneously,” relying on an underlying Semigroup instance for the meaning of “play these two things:”

instance Semigroup a => Semigroup (Voice a)
instance Semigroup a => Monoid (Voice a)

If either voice is resting, we treat its value as mempty, and can happily combine the two parts in parallel.

All of this gives rise to the following rich structure:

newtype Voice a = Voice { unVoice :: SF Time (Maybe a) }
  deriving stock
    (Eq, Ord, Show, Functor, Foldable, Traversable, Generic, Generic1)
  deriving newtype
    (Semigroup, Monoid)
  deriving
    Applicative
    via Compose (SF Time) Maybe

instance Filterable Voice
instance Witherable Voice
instance Alternative Voice

-- | Delay a voice by some amount of time.
delayV :: Time -> Voice a -> Voice a

From Voices to Music

Voices, therefore, give us our primitive notion of monophony. But real music usually has many voices doing many things, independently. This was a point in which I got stuck in my previous post.

The solution here, is surprisingly easy. Assign a Voice to each voice name:

newtype Music v a = Music
  { getVoices :: v -> Voice a
  }
  deriving stock
    (Functor, Generic, Functor)
  deriving newtype
    (Semigroup, Monoid)
  deriving (Applicative, Alternative)
    via Compose ((->) v) Voice

instance Profunctor Music

instance Finite v => Foldable (Music v)
instance Finite v => Traversable (Music v)
instance Filterable (Music v)
instance Finite v => Witherable (Music v)

We get an extremely rich structure here completely for free. Our monoid combines all voices in parallel; our applicative combines voices pointwise; etc. However, we also have a new Profunctor Music instance, whose characteristic lmap :: (b -> c) -> Music c a -> Music b a method allows us to trade lines between voices.

In addition to the in-parallel monoid instance, we can also define a tile product operator over Music v a, which composes things sequentially¹:

duration :: Music v a -> Time

(##) :: Semigroup a => Music v a -> Music v a -> Music v a
m@(Music m1) ## Music m2 =
    Music $ liftA2 (<>) m1 $ fmap (delayV $ duration m) m2

The Semigroup a constraint on (##) arises from the fact that the pieces of music might extend off to infinity in either direction (which pure @Voice must do), and we need to deal with that.

There are a few other combinators we care about. First, we can lift anonymous voices (colloquially “tunes”) into multi-part Music:

voice :: Eq v => v -> Music () a -> Music v a
voice v (Music sf) = Music $
  \case
    ((== v) -> True) -> sf ()
    _ -> mempty

and we can assign the same line to everyone:

everyone :: Music () a -> Music v a
everyone (Music m) = Music $ const $ m ()

Writing Lines

The primitives for building little tunes are

note :: Time -> a -> Music () a
rest :: Time -> Music () a

which you can then compose sequentially via (##), and assign to voices via voice.

Harmonic Constraints

One of the better responses to my last blog post was a link to Dmitri Tymoczko’s FARM 2024 talk.

There’s much more in this video than I can possibly due justice to here, but my big takeaway was that this guy is thinking about the same sorts of things that I am. So I dove into his work, and that lead to his quadruple hierarchy:

Voices move within chords, which move within scales, which move within macro-harmonies.

Tymoczko presents a T algebra which is a geometric space for reasoning about voice leadings. He’s got a lot of fun websites for exploring the ideas, but I couldn’t find an actual implementation of the idea anywhere, so I cooked one up myself.

The idea here is that we have some T :: [Nat] -> Type which describes a hierarchy of abstract scales moving with respect to one another. For example, the Western traditional of having triads move within the diatonic scale, which moves within the chromatic scale, would be represented as T '[3, 7, 12]. T forms a monoid, and has some simple generators that give rise to smooth voice leadings (chord changes.)

Having a model for smooth harmonic transformations means we can use it constructively. I am still working out the exact details here, but the rough shape of the idea is to build an underlying field of key changes (represented as smooth voice leadings in T '[7, 12]):

keyChanges :: Music () (T '[7, 12])

We can then make an underlying field of functional harmonic changes (chord changes), modeled as smooth voice leadings in T '[3, 7]:

chordChanges :: Music () (T '[3, 7])

Our voices responsible for harmony can now be written as values of type

harmonicVoices :: Music h (Set (T '[3]))

and we can use the applicative musical structure to combine the elements together:

{-# LANGUAGE ApplicativeDo #-}

extend :: T ns -> T (ns ++ ms)
sink   :: T ns -> T (n ': ns)

harmony :: Music h (Set (T '[3, 7, 12]))
harmony = do
  k <- everyone keyChanges
  c <- everyone chordChanges
  m <- harmonicVoices
  pure $ extend m <> extend c <> sink k

which we can later fmap out into concrete pitches. The result is that we can completely isolate the following pieces:

key changes
chord changes
how voices express the current harmony
the rhythms of all of the above

and the result is guaranteed to compose in a way that the ear can interpret as music. Not necessarily good music, but undeniably as music.

The type indices on T are purely for my book-keeping, and nothing requires them to be there. Which means we could also use the applicative structure to modulate over different sorts of harmony (eg, move from triads to seventh chords.)

Melody: Still an Open Question

I haven’t quite gotten a feel for melody yet; I think it’s probably in T '[7, 12], but it would be nice to be able to target chord tones as well. Please let me know in the comments if you have any insight here.

However, I have been thinking about contouring, which is the overall “shape” of a musical line. Does it go up, and peak in the middle, and then come down again? Or maybe it smoothly descends down.

We can use the discrete intervals intrinsic inside of Voices to find “reasonable” times to sample them. In essence this assigns a Time to each segment:

timed :: Music v a -> Music v (Time, a)

and we can then use these times to then sample a function Time -> b. This then allows us to apply contours (given as regular Real -> Real functions) to arbitrary rhythms. I currently have this typed as

contour :: Group a => a -> (Real -> Real) -> Music v () -> Music v a

where a ~ T something, and the outputted Reals get rounded to their nearest integer values. I’m not deeply in love with this type, but the rough idea is great—turn arbitrary real-valued functions into musical lines. This generalizes contouring, as well as scale runs.

Next Steps?

I’m writing all of this up because I go back to work on Monday and life is going to get very busy soon. I’m afraid I won’t be able to finish all of this!

The types above I’m pretty certain are relatively close to perfect. They seem to capture everything I could possibly want, and nothing I don’t want. Assuming I’m right about that, they must make up the basis of musical composition.

The next step therefore is to build musical combinators on top. One particular combinator I’ve got my eye on is some sort of general ~> “get from here to there” operator:

(~>) :: (Semigroup a, ???) => Music v a -> Music v a -> Music v a

which I imagine would bridge a gap between the end of one piece of music with beginning of another. I think this would be roughly as easy as moving each voice linearly in T space from where it was to where it needs to be. This might need to be a ternary operation in order to also associate a rhythmic pattern to use for the bridge.

But I imagine (~>) would be great for lots of dumb little musical things. Like when applied over the chord dimension, it would generate arpeggios. Over the scale dimension, it would generate runs. And it would make chromatic moves in the chroma dimension.

Choosing exactly what moves to make for Ts consisting of components in multiple axes might just be some bespoke order, or could do something more intelligent. I think the right approach would be to steal diagrams’ idea of an Envelope, and attach some relevant metadata to each Music. We could then write (~>) as a function of those envelopes, but I must admit I don’t quite know what this would look like.

As usual, I’d love any insight you have! Please leave it in the comments. Although I must admit I appreciate comments of the form “have you tried $X” much more than of the form “music is sublime and you’re an idiot for even trying this.”

Happy new year!

Strictly speaking, the tile product can also do parallel composition, as well as sychronizing composition, but that’s not super important right now.↩︎

Chung-chieh Shan: A challenge for a better community

Thu, 01 Jan 2026 07:11:17 +0000

Did you know that all ACM-sponsored conferences have an anti-harassment policy? I didn’t, until I chaired the Haskell Symposium last year. The concise policy says, among other things, that people shouldn’t use my family constitution to interfere with my professional participation. And the policy has teeth. That’s great.

My not knowing the policy and not seeing it publicized didn’t make me go out of my way to harass anyone. But it did make me less sure and less comfortable that I belonged at ICFP. Briefly, it’s because I didn’t know if it would be common ground at the conference that my actual self was fully human. That’s not something I can take for granted in general society. Also, it’s because I didn’t know whether my fellow conference attendees were aware of the policy. We could all use a reminder, and a public signal that we mean it.

For these reasons, I’m very happy that ICFP will start to publicize ACM’s existing anti-harassment policy and make sure everyone registered knows it. All ACM conferences should do it. That’s why Tim Chevalier, Clement Delafargue, Adam Foltzer, Eric Merritt, and I are doing two things. We ask you to join us:

Donate to the Ada Initiative. Our goal is for the functional programming community to raise $4096 by the end of Friday (Sept 19) UTC. To count toward this goal, please use this link: http://supportada.org/?campaign=lambda
Call on the ACM and tell your friends. For example, I tweeted this:
I donate to @AdaInitiative because I want @TheOfficialACM events to announce their anti-harassment policy http://supportada.org/?campaign=lambda #lambda4ada

Thanks for improving our professional homes!

Chung-chieh Shan: Borderline, just semantics

Thu, 01 Jan 2026 07:11:17 +0000

I want to discuss two moral aspects of how to talk about borderline people (e.g., whether to say “MTF” or “female”, “smart” or “good at math”).

The first aspect is what category of people we should be talking about, never mind how we talk about it. Usually, for a given category C, there are many ways to be borderline C: people can be

borderline female due to a lack of chromosomes or a lack of genitals;
borderline smart because they keep exchanging quantifiers or because they keep estranging family;
borderline safe because they drive over the speed limit or roll their bike past a stop sign;
borderline documented because they burned their passport, overstayed their visa, or came from an unrecognized “country”.

Of course, it is easy enough to construct a category C’ that firmly includes or excludes particular kinds of borderline Cs (e.g., you are female’ iff you have XX chromosomes), if only to leave unresolved lower-dimensional borderlines (e.g., what if only half of your cells have XX chromosomes). So let’s ask not what categories exist (because they all do) but what categories we should talk about.

It might well be that we should talk about different versions of C for different purposes (e.g., medicine, sports, identification, love). It might also be that delineating categories at their edges is not usually worth our collective cognitive trouble. Regardless, sometimes in conversation or other collaboration I find that multiple versions are actually relevant. For example, some people wonder whether to call a transgender person “male” or “female”, and I think a transgender person can firmly belong to a gender category that we should talk about. Unfortunately, when I tried to work with others to distinguish and choose among versions of categories, I have often been frustrated.

When people in power to classify others don’t take the effort to apply the right category version, the misclassified people can get slighted. I empathize with the misclassified people because I have been misclassified. For example:

I don’t need a passport to travel internationally, though I do need a passport number.
I am a computer scientist, though I can’t fix your computer.
I am from Taiwan, though I don’t speak Taiwanese.
I am from the United States, though I’m not a United States citizen.

Also because I have been misclassified, this discussion of borderlines matters for me practically and personally. It’s not just abstract theorizing.

The second aspect is how to name a given category (version). Even in a context where we agree intellectually that there is exactly one gender category we should talk about (say “female”) and a transgender person belongs to it firmly, the term “female” still takes many people today more thought to produce and comprehend when applied to a person known to be transgender than when applied to a person known to be cisgender. So, it is tempting for a well-meaning speaker to avoid gender categories altogether by terming a transgender person “female-presenting” instead. The “female-presenting” characterization

is certainly true,
but may be a cop-out because it
- perpetuates the implicature that the person doesn’t quite belong
- and increases their work to counter misclassification,
yet may be justifiable (perhaps in some strain of utilitarianism), though a justification remains to be spelled out and applied uniformly to all instances.

Your thoughts?

Chung-chieh Shan: Babel-17

Thu, 01 Jan 2026 07:11:17 +0000

“Show me.”

He bowed his head in mocking, semi-formal acquiescence. “Modern warfare can be fought on so many delightfully different levels,” he continued, walking back to her side as if no interruption of the tour had been suggested. “One wins a battle by making sure one’s troops have enough blunderbusses and battle axes like the ones you saw in the first room; or by the well-placed six-inch length of vanadium wire in a Type 27-QX communications unit. With the proper orders delayed, the encounter never takes place. Hand-to-hand weapons, survival kit, plus training, room, and board: three thousand credits per enlisted stellarman over a period of two years active duty. For a garrison of fifteen hundred men that’s an outlay of four million, five hundred credits. That same garrison will live in and fight from three hyperstasis battleships which, fully equipped, run about a million and a half credits apiece—a total outlay of nine million credits. We have spent, on occasion, perhaps as much as a million on the preparation of a single spy or saboteur. That is rather higher than usual. And I can’t believe a six-inch length of vanadium wire costs a third of a cent. War is costly. And although it has taken some time, Administrative Alliance Headquarters is beginning to realize subtlety pays. This way, Miss—Captain Wong.”

Again they were in a room with only a single display case, but it was seven feet high.

A statue, Rydra thought. No, real flesh, with detail of muscle and joint; no, it must be a statue because a human body, dead or in suspended animation, doesn’t look that… alive. Only art could produce that vibrancy.

“So you see, the proper spy is very important.” Though the door had opened automatically, the Baron held it with his hand in vestigial politeness. “This is one of our more expensive models. Still well under a million credits, but one of my favorites—though in practice he has his faults. With a few minor alterations I would like to make him a permanent part of our arsenal.”

“A model of a spy?” Rydra asked. “Some sort of robot or android?”

“Not at all.” They approached the display case. “We made half a dozen TW-55’s. It took the most exacting genetic search. Medical science has progressed so that all sorts of hopeless human refuse lives and reproduces at a frightening rate—inferior creatures that would have been too weak to survive a handful of centuries ago. We chose our parents carefully, and then with artificial insemination we got our half dozen zygotes, three male, three female. We raised them in, oh, such a carefully controlled nutrient environment, speeding the growth rate by hormones and other things. But the beauty of it was the experiential imprinting. Gorgeously healthy creatures; you have no idea how much care they received.”

“I once spent a summer on a cattle farm,” Rydra said shortly.

The Baron’s nod was brisk. “We’d used the experiential imprints before, so we knew what we were doing. But never to synthesize completely the life situation of, say, a sixteen-year-old human. Sixteen was the physiological age we brought them to in six months. Look for yourself what a splendid specimen it is. The reflexes are fifty percent above those of a human aged normally. Human musculature is beautifully engineered: a three-day-starved, six-month-atrophied myasthenia gravis case, can, with the proper stimulant drugs, overturn a ton-and-a-half automobile. It will kill him—but that’s still remarkable efficiency. Think what the biologically perfect body, operating at all times at point nine-nine efficiency, could accomplish in physical strength alone.”

“I thought hormone growth incentive had been outlawed. Doesn’t it reduce the life span some drastic amount?”

“To the extent we used it, the life span reduction is seventy-five percent and over.” He might have smiled the same way watching some odd animal at its incomprehensible antics. “But, Madam, we are making weapons. If TW-55 can function twenty years at peak efficiency, then it will have outlasted the average battle cruiser by five years. But the experiential imprinting! To find among ordinary men someone who can function as a spy, is willing to function as a spy, you must search the fringes of neurosis, often psychosis. Though such deviations might mean strength in a particular area, it always means an overall weakness in the personality. Functioning in any but that particular area, a spy may be dangerously inefficient. And the Invaders have psyche-indices too, which will keep the average spy out of anyplace we might want to put him. Captured, a good spy is a dozen times as dangerous as a bad one. Post-hypnotic suicide suggestions and the like are easily gotten around with drugs; and are wasteful. TW-55 here will register perfectly normal on a psyche integration. He has about six hours of social conversation, plot synopses of the latest novels, political situations, music, and art criticism—l believe in the course of an evening he is programmed to drop your name twice, an honor you share only with Ronald Quar. He has one subject on which he can expound with scholarly acumen for an hour and a half—this one, I believe, is ‘haptoglobin groupings among the marsupials.’ Put him in formal wear and he will be perfectly at home at an ambassadorial ball or a coffee break at a high-level government conference. He is a crack assassin, expert with all the weapons you have seen up till now, and more. TW-55 has twelve hours’ worth of episodes in fourteen different dialects, accents, or jargons concerning sexual conquests, gambling experiences, fisticuff encounters, and humorous anecdotes of semi-illegal enterprises, all of which failed miserably. Tear his shirt, smear grease on his face and slip a pair of overalls on him, and he could be a service mechanic on any one of a hundred spaceyards or stellarcenters on the other side of the Snap. He can disable any space drive system, communications components, radar works, or alarm system used by the Invaders in the past twenty years with little more than—”

“Six inches of vanadium wire?”

The Baron smiled. “His fingerprints and retina pattern, he can alter at will. A little neural surgery has made all the muscles of his face voluntary, which means he can alter his facial structure drastically. Chemical dyes and hormone banks beneath the scalp enable him to color his hair in seconds, or, if necessary, shed it completely and grow a new batch in half an hour. He’s a past master in the psychology and physiology of coercion.”

“Torture?”

“If you will. He is totally obedient to the people whom he has been conditioned to regard as his superiors; totally destructive toward what he has been ordered to destroy. There is nothing in that beautiful head even akin to a superego.”

“He is…” and she wondered at herself speaking, “beautiful.” The dark-lashed eyes with lids about to quiver open, the broad hands hung at the naked thighs, fingers half-curled, about to straighten or become a fist. The display light was misty on the tanned, yet near-translucent skin. “You say this isn’t a model, but really alive?”

“Oh, more or less. But it’s rather firmly fixed in something like a yoga trance, or a lizard’s hibernation. I could activate it for you—but it’s ten to seven. We don’t want to keep the others waiting at the table now, do we?”

She looked away from the figure in glass to the dull, taut skin of the Baron’s face. His jaw, beneath his faintly concave cheek, was involuntarily working on its hinge.

(from Babel-17 by Samuel R. Delany)

Monday Morning Haskell: Parsing with an MCAP Index

Mon, 29 Dec 2025 12:30:00 +0000

Welcome to the sixth and final part of our series on MCAP parsing. In the first four parts, we did all the work to parse certain topic messages from a bag file, going through the file sequentially. In the fifth part, we set up a lot of infrastructure to allow ourselves to instead parse the file using indexing so we can get our messages faster. In this part, we’ll finish the process and see how much faster our code is!

This series has combined a lot of ideas from several of our online courses. You can learn more about complex monad structures in Making Sense of Monads or Effectful Haskell. You can learn more about parsing in Solve.hs. You can get lifetime access to all our courses buy purchasing MMH Complete!

The Plan

Let’s recap the most important items from the last part and go over our plan again. We defined a series of types to help us represent various record types, and parsers for them:

data FooterRec = ...
parseFooter’ :: parseFooter' :: (MonadParsec Void ByteString m) => m (Word64, FooterRec)

data SummaryOffsetRec = ...
parseSummaryOffset :: (MonadFail m, MonadParsec Void ByteString m) => m (Word64, SummaryOffsetRec)

data ChunkIndexRec = ...
parseChunkIndex :: (MonadFail m, MonadParsec Void ByteString m) => m (Word64, ChunkIndexRec)

data MessageIndexRec = ...
parseMessageIndex' :: (MonadFail m, MonadParsec Void ByteString m) => m (Word64, MessageIndexRec)

We also defined a couple monad structures to capture our parser’s state:

data TopicState = TopicState
  { tsChannels :: HM.HashMap Word16 MsgChannel
  , tsSchemas :: HM.HashMap Word16 MsgSchema
  , tsMessages :: HM.HashMap (ByteString, ByteString) [MessageData]
  , tsDesiredTopics :: HS.HashSet ByteString
  , tsDesiredChannels :: HS.HashSet Word16
  } deriving (Show)

type MCAPReader a = StateT TopicState (ReaderT Handle (ExceptT ValidationError IO)) a
type MCAPParser a = ParsecT Void ByteString (StateT TopicState (ReaderT Handle (ExceptT ValidationError IO)))  a

Finally, we defined a series of utility functions on these monads:

seek :: Integer -> MCAPReader ()
curPos :: MCAPReader Integer
mcapGuard :: String -> Bool -> MCAPReader ()
runMCAPParser :: MCAPParser (Word64, a) -> ByteString -> MCAPReader (Word64, a)
parseNextRecord :: RecordType -> MCAPParser (Word64, a) -> MCAPReader (Word64, a)
parseNextRecord' :: RecordType -> (Word64 -> MCAPParser (Word64, a)) -> MCAPReader (Word64, a)
runMCAPReader :: Handle -> TopicState -> MCAPReader a -> IO (Either ValidationError TopicState)

Our overall plan is:

Parse the footer and use its data to jump to the start of the summary offset section
Parse summary offsets for schemas, channels, and chunk indexes
Parse summary records for schemas and channels to load them into TopicState
Process chunk indexes, which direct us to message index records for our desired channels
Process message indexes, which point us to individual messages for our desired topics
Parse the messages into our Topic State

Let’s get started!

Step 1 - Parsing the Footer

Our entrypoint for the MCAPReader monad will be a function called parseUsingIndex (we’ll see how to enter this function later).

parseUsingIndex :: MCAPReader ()

The first order of business is to seek to the Footer’s location so we can parse the Footer record. The Footer has a fixed size and it is always the last record before the 8 “magic bytes” that conclude the file. This size is 29 bytes: 1 byte for the op code (record type), 8 bytes to define the content size, and 20 bytes for the 3 fields (Word64, Word64, Word32). Combined with the magic bytes, this means we want to find the file size and seek to 37 bytes before it:

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart <- seekToFooterStart
  ...
  where
    seekToFooterStart = do
      hndl <- ask
      sz <- liftIO $ hFileSize hndl
      seek (sz - 37)
      curPos

Now we use parseNextRecord and parseFooter’ to get the Footer’s data, which includes the start of the summary offset section. We unpack this value and seek to its location:

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart <- seekToFooterStart
  (_, FooterRec summStart summOffStart _) <- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  ...
  where
    seekToFooterStart = do
      hndl <- ask
      sz <- liftIO $ hFileSize hndl
      seek (sz - 37)
      curPos

Now we’re ready for step 2!

Step 2 - Parsing Summary Offsets

Recall that records in the summary section are grouped in blocks, and each summary offset record leads us to the start of the block for a particular op code. We’re interested in the summary records for schemas, channels and chunk indexes. So we want to keep parsing summary offsets until we have all three of these.

So we’ll build a helper function that will construct a map from op codes to summary offset records, but also track booleans for which of our desired types we’ve seen. This function also needs to track the number of bytes remaining in the summary offset section:

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart <- seekToFooterStart
  (_, FooterRec summStart summOffStart _) <- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  ...
  where
    parseSummaryOffsets :: Word64 -> Bool -> Bool -> Bool -> M.Map RecordType SummaryOffsetRec -> MCAPReader (M.Map RecordType SummaryOffsetRec)
    ...

There are two base cases. If we are out of bytes, we return our accumulated map. If all three boolean flags are true, then we can also return the map, as we have what we need.

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  ...
  where
    parseSummaryOffsets :: Word64 -> Bool -> Bool -> Bool -> M.Map RecordType SummaryOffsetRec -> MCAPReader (M.Map RecordType SummaryOffsetRec)
    parseSummaryOffsets 0 _ _ _ acc = return acc
    parseSummaryOffsets _ True True True acc = return acc
    parseSummaryOffsets remBytes hasSchema hasChannel hasChunkIndex acc =
    ...

For the general case, we use parseSummaryOffset to parse the next record. We update our boolean flags depending on the op code, and we insert the new record into our map. Then we recurse:

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  ...
  where
    parseSummaryOffsets :: Word64 -> Bool -> Bool -> Bool -> M.Map RecordType SummaryOffsetRec -> MCAPReader (M.Map RecordType SummaryOffsetRec)
    parseSummaryOffsets 0 _ _ _ acc = return acc
    parseSummaryOffsets _ True True True acc = return acc
    parseSummaryOffsets remBytes hasSchema hasChannel hasChunkIndex acc = do
      mcapGuard ("Not enough bytes for summary offset: " <> show remBytes) (remBytes >= 26)
      (_, summ@(SummaryOffsetRec op _ _)) <- parseNextRecord SummaryOffset parseSummaryOffset
      let hasSchema' = hasSchema || op == Schema
      let hasChannel' = hasChannel || op == Channel
      let hasChunkIndex' = hasChunkIndex || op == ChunkIndex
      let mp = M.insert op summ acc
      parseSummaryOffsets (remBytes - 26) hasSchema' hasChannel' hasChunkIndex' mp

Now we’ll define a second helper. This will use the Maybe monad to extract the 3 records we care about.

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  ...
  where
    ...
    find3Offsets :: M.Map RecordType SummaryOffsetRec -> Maybe (SummaryOffsetRec, SummaryOffsetRec, SummaryOffsetRec)
    find3Offsets mp = do
      s <- M.lookup Schema mp
      c <- M.lookup Channel mp
      ci <- M.lookup ChunkIndex mp
      return (s, c, ci)

Now back in the main line of the function, we’ll get the size of this section (subtract the summary offset start from the footer start position) and call our two helpers. If we don’t have all three maps, we’ll throw a validation error showing what op codes we did find. Otherwise, we have the 3 offsets.

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart <- seekToFooterStart
  (_, FooterRec summStart summOffStart _) <- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  -- Read Summary Offsets until we have Schema, Channel and ChunkIndex
  let sz = fromIntegral footerStart - summOffStart
  summOffMap <- parseSummaryOffsets sz False False False M.empty
  let offsets = find3Offsets summOffMap
  case offsets of
    Nothing -> throwError (ValidationError $ "Did not find the right summary offsets: " <> show (M.keys summOffMap))
    Just (schemaOffset, channelOffset, chunkIndexOffset) -> ...
  where
    parseSummaryOffsets :: Word64 -> Bool -> Bool -> Bool -> M.Map RecordType SummaryOffsetRec -> MCAPReader (M.Map RecordType SummaryOffsetRec)

    find3Offsets :: M.Map RecordType SummaryOffsetRec -> Maybe (SummaryOffsetRec, SummaryOffsetRec, SummaryOffsetRec)

Now we’ve got these summary offset records! It’s time to use them to start populating our topic state.

Step 3 - Summary Schemas and Channels

We’ll begin with the schemas and channels. These are fairly easy, since we can rely on previous code for the most part. Recall that we have the functions parseSchema’ and parseChannel’ for parsing individual schema and channel records. We need to update these type signatures to be more generic like so:

parseSchema' :: (MonadParsec Void ByteString m, MS.MonadState TopicState m) => m (Word64, Record)

parseChannel' :: (MonadFail m, MonadParsec Void ByteString m, MS.MonadState TopicState m) => m (Word64, Record)

These will work with the MCAPParser monad. They require MonadState TopicState because they’ll save the schemas and channels within our TopicState, rather than requiring us to do anything with the return values.

So we just need to write functions that will look through all the records in the summary block and parse them. Since the summary offset contains the start location and the byte length of all the records, we’ll seek to that location, and then use our tried and true method of parsing until we run out of bytes. Here’s a function to load all the schemas in a group:

loadSchemasForIndex :: SummaryOffsetRec -> MCAPReader ()
loadSchemasForIndex (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, _) <- parseNextRecord Schema parseSchema'
      mcapGuard ("Summary schema is too long! " <> show len <> " " <> show rem) (len <= rem)
      f (rem - len)

And here’s the function for channels:

loadChannelsForIndex :: SummaryOffsetRec -> MCAPReader ()
loadChannelsForIndex (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, _) <- parseNextRecord Channel parseChannel'
      mcapGuard ("Summary channel is too long! " <> show len <> " " <> show rem) (len <= rem)
      f (rem - len)

We simply call these from within parseUsingIndex like so:

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart <- seekToFooterStart
  (_, FooterRec summStart summOffStart _) <- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  -- Read Summary Offsets until we have Schema, Channel and ChunkIndex
  let sz = fromIntegral footerStart - summOffStart
  summOffMap <- parseSummaryOffsets sz False False False M.empty
  let offsets = find3Offsets summOffMap
  case offsets of
    Nothing -> throwError (ValidationError $ "Did not find the right summary offsets: " <> show (M.keys summOffMap))
    Just (schemaOffset, channelOffset, chunkIndexOffset) -> do
      loadSchemasForIndex schemaOffset
      loadChannelsForIndex channelOffset
      ...

Now our internal TopicState is updated with the schemas and channels. This means we’re ready to parse messages, but we need to find where those messages are first!

Step 4 - Chunk Index Processing

Now we need to process each Chunk Index record. We’ll write a function for this, and it will be the last thing we need to call from parseUsingIndex:

processChunkIndexes :: SummaryOffsetRec -> MCAPReader ()

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart <- seekToFooterStart
  (_, FooterRec summStart summOffStart _) <- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  -- Read Summary Offsets until we have Schema, Channel and ChunkIndex
  let sz = fromIntegral footerStart - summOffStart
  summOffMap <- parseSummaryOffsets sz False False False M.empty
  let offsets = find3Offsets summOffMap
  case offsets of
    Nothing -> throwError (ValidationError $ "Did not find the right summary offsets: " <> show (M.keys summOffMap))
    Just (schemaOffset, channelOffset, chunkIndexOffset) -> do
      loadSchemasForIndex schemaOffset
      loadChannelsForIndex channelOffset
      processChunkIndexes chunkIndexOffset
  where
    ...

To fill out this function, we follow the same general pattern to loop through these records until we’ve exhausted the bytes:

processChunkIndexes :: SummaryOffsetRec -> MCAPReader ()
processChunkIndexes (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, ci) <- parseNextRecord ChunkIndex parseChunkIndex
      ...

The trick is that processing each individual chunk index is more complicated after we’ve parsed it. Each chunk index refers to a single Chunk record and contains its initial position (we saved this in the cirChunkStartOffset field). However, the message index offsets are not absolute positions. They are relative to the start of the data section of the chunk. So we want to calculate where this location is (and it took me a bit of debugging to get it right!).

Prior to the data section in the chunk is the record header, which is 9 bytes. Then there are 32 fixed bytes, followed by the compression string. Then there are still 8 more bytes after that because the data section in the chunk is prefixed by the byte count. You can consult the Chunk record specification to see where these numbers are coming from. So putting it all together, we take the chunk start location, add 49 bytes, and then add the length of the compression string.

processChunkIndexes :: SummaryOffsetRec -> MCAPReader ()
processChunkIndexes (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, ci) <- parseNextRecord ChunkIndex parseChunkIndex
      let chunkDataStart = cirChunkStartOffset ci + 49 + (fromIntegral $ BS.length (cirCompression ci))
      savedPosition <- curPos
      ... -- Process message index records
      seek savedPosition

You’ll also see that we save the current handle position. We’ll process the message index records and jump around to the messages. But we want to return to the current location (the location after parsing this chunk index) so that we can process the next chunk index afterward.

Now we have to look through all of our desired channels and see which of them are referred to in the cirMessageIndexOffsets of this chunk index:

processChunkIndexes :: SummaryOffsetRec -> MCAPReader ()
processChunkIndexes (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, ci) <- parseNextRecord ChunkIndex parseChunkIndex
      let chunkDataStart = cirChunkStartOffset ci + 49 + (fromIntegral $ BS.length (cirCompression ci))
      savedPosition <- curPos
      chans <- MS.gets tsDesiredChannels
      let allMIO = cirMessageIndexOffsets ci
      forM_ chans $ \chanId -> when (HM.member chanId allMIO) $ do
        ...
      seek savedPosition

Now we get the offset for this channel in the map. This will give us the (absolute) location of the Message Index record for this channel. We’ll prepare a new function to process this message index:

processChunkIndexes :: SummaryOffsetRec -> MCAPReader ()
processChunkIndexes (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, ci) <- parseNextRecord ChunkIndex parseChunkIndex
      let chunkDataStart = cirChunkStartOffset ci + 49 + (fromIntegral $ BS.length (cirCompression ci))
      savedPosition <- curPos
      chans <- MS.gets tsDesiredChannels
      let allMIO = cirMessageIndexOffsets ci
      forM_ chans $ \chanId -> when (HM.member chanId allMIO) $ do
        let chanOffset = allMIO HM.! chanId
        loadMessagesFromMessageIndex chanId chanOffset chunkDataStart
      seek savedPosition

loadMessagesFromMessageIndex :: Word16 -> Word64 -> Word64 -> MCAPReader ()
loadMessagesFromMessageIndex chanId offset chunkDataStart

This is all we need for processChunkIndexes. We’re getting closer to our goal! Now we need to process the message index record.

Steps 5 & 6 - Processing Message Indexes and Messages

You’ll note from above that we passed the channel ID, the offset of the message index, and the corresponding “chunk data start”. We begin this function by seeking the to message index location, parsing the message index, and ensuring that we are getting an index for the correct channel:

loadMessagesFromMessageIndex :: Word16 -> Word64 -> Word64 -> MCAPReader ()
loadMessagesFromMessageIndex chanId offset chunkDataStart = do
  seek (fromIntegral offset)
  (_, MessageIndexRec mirChanId recs) <- parseNextRecord MessageIndex parseMessageIndex'
  mcapGuard ("Mismatched channels for message index record: " <> show chanId <> " " <> show mirChanId) (mirChanId == chanId)
  ...

Now we loop through the record tuples in the message index record. Each of these contains a relative offset. We just need to seek to that offset, adding in the chunk data start location. Then we can use our message parser function!

loadMessagesFromMessageIndex :: Word16 -> Word64 -> Word64 -> MCAPReader ()
loadMessagesFromMessageIndex chanId offset chunkDataStart = do
  seek (fromIntegral offset)
  (_, MessageIndexRec mirChanId recs) <- parseNextRecord MessageIndex parseMessageIndex'
  mcapGuard ("Mismatched channels for message index record: " <> show chanId <> " " <> show mirChanId) (mirChanId == chanId)
  forM_ recs $ \(_, offset) -> do
    seek (fromIntegral $ chunkDataStart + offset)
    parseNextRecord' Message parseMessage'

parseMessage' :: (MonadParsec Void ByteString m, MS.MonadState TopicState m, MonadFail m) => Word64 -> m (Word64, Record)

That’s all we need! Like our schema and channel parsers, parseMessage’ does the work of saving the messages in our TopicState, so we don’t need to return or process anything. The path we started from parseUsingIndex is now complete!

Compression Disclaimer

As an important note, this code takes a shortcut. The approach given will only work with uncompressed chunks! The relative offset we used for the message index is actually the offset from the start of the uncompressed data. If the chunk’s data is compressed, we would have to grab that whole uncompressed string, decompress as much of it as we need, and then jump to particular locations within that bytestring. The code required to do this efficiently is much more intricate, as it requires us to manage counts on the streams of compressed and uncompressed data.

This highlights an important tradeoff. By leaving the data uncompressed, it takes up more storage space and would take longer to send over a network if you need to download these bag files. But if the data is compressed, we might have to decompress a significant portion of it to find the few messages we care about, meaning we don’t see the same gains in bag read time. The particular needs of your application & systems will determine what choice makes sense.

Pulling it Together

Now we need an entrypoint to call parseUsingIndex. Our function will take the filepath, open a handle to it, and create the initial topic state, looking for the /turtle1/cmd_vel topic. We’ll invoke the parseUsingIndex function with runMCAPParser:

parseRecordsFromFileWithIndex :: FilePath -> IO ()
parseRecordsFromFileWithIndex fp = do
  parseBareRecordsFromFile
  startTime <- getCurrentTime
  handle <- openFile fp ReadMode
  let initialTs = TopicState HM.empty HM.empty HM.empty (HS.fromList ["/turtle1/cmd_vel"]) HS.empty
  result <- runMCAPReader handle initialTs parseUsingIndex
  ...

Then, in a reprisal of previous code, we’ll loop through all the messages for our topic (they use the geometry_msgs/msg/Twist type from ROS2), and decode them all. (Recall that in the previous part we wrote parseVelMsg for this type).

parseRecordsFromFileWithIndex :: FilePath -> IO ()
parseRecordsFromFileWithIndex fp = do
  startTime <- getCurrentTime
  handle <- openFile fp ReadMode
  let initialTs = TopicState HM.empty HM.empty HM.empty (HS.fromList ["/turtle1/cmd_vel"]) HS.empty
  result <- runMCAPReader handle initialTs parseUsingIndex
  case result of
    Left e -> print e
    Right finalTs -> do
      let velMessages = fromMaybe [] $ HM.lookup ("/turtle1/cmd_vel", "geometry_msgs/msg/Twist") (tsMessages finalTs)
      forM_ velMessages $ \(MessageData t1 t2 msg) -> do
        parseVelResult <- evalStateT (runParserT parseVelMsg "Velocity" msg) 0
        case parseVelResult of
          Left e -> print e
          Right s -> putStrLn $ "Parsed Velocity Message: " <> show t1 <> " " <> show t2 <> " " <> show s
      endTime <- getCurrentTime
      print (diffUTCTime endTime startTime)

And now we’re done! We added some timing features, so we can compare this to our previous run. It is much faster! The original takes around 0.075s. The new version takes about 0.001s. So it’s about 75x faster, a great speedup!

Conclusion

This concludes our series on ROS2 bag parsing! Over the course of this long series, we were able to take an MCAP bag file, parse topic messages out of it, and then use the indexing features to do this even faster! We learn a ton about the MCAP format as well as CDR encoding. We saw how to compose parsers, even when they had stateful side effects like file seeking or saving message data.

With so much ground to cover, there wasn’t a lot of time in this series to go over the basics. If you want to solidify your Haskell skills so that you can write advanced code like this, the best way to do that is to take a look at some of our courses.

The most immediately useful for this series would be Solve.hs. Module 1 teaches you valuable loop structures that form the building block for all of our complicated functions in Haskell. Then Module 4 will teach you about parsing and Megaparsec! In between, you’ll also learn about Data Structures and Algorithms in Haskell!

Making Sense of Monads and Effectful Haskell are also great choices if the monad structures in the second half of the series are a bit confusing to you. You can grab both of these courses for a discount by purchasing the Effects Bundle!

Finally, if you want access to all our course content, past, present and future, you can get the MMH Complete bundle! This will include all the previous courses, as well as our other full length courses Haskell From Scratch, Practical Haskell, plus our mini-course on machine learning, Haskell Brain.

GHC Developer Blog: GHC 9.12.3 is now available

Sat, 27 Dec 2025 00:00:00 +0000

GHC 9.12.3 is now available

zubin - 2025-12-27

The GHC developers are very pleased to announce the release of GHC 9.12.3. Binary distributions, source distributions, and documentation are available at downloads.haskell.org.

GHC 9.12.3 is a bug-fix release fixing many issues of a variety of severities and scopes, including:

Fix a number of crashes and miscompilations in the compiler frontend (#25004, #25960, #26256)
Improvements to efficiency of the runtime linker (#26009)
Fixes for several bugs in bytecode generation and the bytecode interpreter (#23210, #25975, #25750)
Fixes for bugs in the handling of WHITEHOLEs in the RTS (#26204, #26205)
Fix incorrect code generation for SSE vector operations (#25859)
Fix a use-after-free in the Windows runtime linker (#26613)
Support for synchronous JSFFI exports for the wasm backend
And many more!

A full accounting of these fixes can be found in the release notes. As always, GHCâ€™s release status, including planned future releases, can be found on the GHC Wiki status.

GHC development is sponsored by:

As always, do give this release a try and open a ticket if you see anything amiss.

in Code: Advent of Code 2025: Haskell Solution Reflections for all 12 Days

Wed, 24 Dec 2025 20:44:05 +0000

Merry Christmas all! This is my annual Advent of Code post! Advent of Code is a series of (this year) 12 daily Christmas-themed programming puzzles that are meant to be fun diversions from your daily life, help you find a bit of whimsy in your world, give you a chance to explore new ideas and program together with your friends. I always enjoy discussing creative ways to solve these puzzles every day, and it’s become a bit of an annual highlight for me and a lot of others. My favorite part about these puzzles is that they are open ended enough that there are usually many different interesting ways to solve them — it’s not like a stressful interview question where you have to recite the obscure incantation to pass the test. In the past I’ve leveraged group theory, galilean transformations and linear algebra, and more group theory.

Haskell is especially fun for these because if you set up your abstractions in just the right way, the puzzles seem to solve themselves. It’s a good opportunity every year to get exposed to different parts of the Haskell ecosystem! Last year, I moved almost all of my Haskell code to an Advent of Code Megarepo, and I also write up my favorite ways to solve each one in the megarepo wiki.

All of this year’s 12 puzzles are here, but I’ve also included links to each individual one in this post. I’m also considering expanding some of these into full on blog posts, so be on the look out, or let me know if there are any that you might want fully expanded! And if you haven’t, why not try these out yourself? Be sure to drop by the libera-chat ##advent-of-code channel to discuss any fun ways you solve them, or any questions! Thanks again to Eric for a great new fresh take on the event this year!

Day 1 - Secret Entrance — The classic Day 1 scanl
Day 2 - Gift Shop — The super efficient IntSet
Day 3 - Lobby — StateT + [] = backtracking search monad
Day 4 - Printing Department — 2D cellular automata
Day 5 - Cafeteria — The power of the data-interval library
Day 6 - Trash Compactor — Data.List manipulations
Day 7 - Laboratories — Tying the knot
Day 8 - Playground — Iterative Clustering
Day 9 - Movie Theater — IntervalSet and IntervalMap
Day 10 - Factory — Guassian Elimation, Type-safe Mutable Vectors
Day 11 - Reactor — More Knot Tying and DP!
Day 12 - Christmas Tree Farm — Counting

Planet Haskell

Donnacha Oisín Kidney: Tries for Polynomials

Evaluation and Hornerâ€™s Rule

Multiple Variables

Sums of Products

A Trie

The numeric functions on Tries

Lenses and Division

GrÃ¶bner Bases

Monomial Orderings

Enumerating Monomials

Efficiently Popping the Leading Monomial

Next Steps

Gists

References

Dan Piponi (sigfpe): Some type constructors are tensor products

Haskell Interlude: 81: Torsten Grust

Chris Reade: PenroseKiteDart User Guide

Introduction

1. About Penroseâ€™s Kites and Darts

The Tiles

Aperiodic Infinite Tilings

Legal Tilings

Correct Tilings

2. Using the PenroseKiteDart Package

3. Overview of Types and Operations

Half-Tiles

Tgraphs

Tgraph Properties

Main Tiling Operations

Decompose

Force

Compose

Try Results

Elementary Tgraph and TileFace Operations

Patches (Scaled and Positioned Tilings)

Vertex Patches

4. Drawing in More Detail

Class Drawable

Class DrawableLabelled

Further drawing functions

Overlaid examples (location map sharing)

5. Forcing in More Detail

The force rules

Main Forcing Operations

Update Generators

Forcible Class Operations

Efficient chains of forcing operations.

A limitation of forcing

6. Advanced Operations

Guided comparison of Tgraphs

Empires and SuperForce

Combinations and Explicitly Forced

Tracked Tgraphs

7. Other Reading

Dan Piponi (sigfpe): Constructing Clifford Algebras using the Super Tensor Product

Dan Piponi (sigfpe): Self-referential logic via self-referential circuits

Dan Piponi (sigfpe): Expectation-Maximization with Less Arbitrariness

Dan Piponi (sigfpe): Logarithms and exponentials of functions

Tweag I/O: ð�•¯ð�–”ð�–ˆ ð�–Žð�–™ ð�–‘ð�–Žð�–�ð�–Š ð�–Žð�–™'ð�–˜ ð�–�ð�–”ð�–™

When your README’s a monolith

When you have varied audiences

When you need to have clarity

Document your code, ma

Haskell Interlude: 80: POPL 2026 - Part 1

Philip Wadler: Wet Sidewalks and Odd Numbers

Russell O'Connor: Nash Equilibrium for Terminal Maneuvers

Tweag I/O: Accessing external resources reliably with Bazel

Depending on external resources

Common problems

Remedies

Using checksums

Using GitHub releases

Using retries

Placing binaries under version control

Internal repository manager

Bazel downloader configuration

Blocking network requests

Sandboxing

Mirrors

Guided comparison of `Tgraph`s

When your `README`’s a monolith

Towards a standalone `base` package

`ghc-debug`