Yuki's Notes

Writings

Sat, 06 Mar 2021 00:00:00 GMT

Hello world, I’m back with more text.

I hope you don’t miss me while I’m gone.

Though, I really doubt that you even realize I’m here.

Who knows, really?

So, anyway, I started this writefreely instance mainly because I want an improved blogging/writing experience which integrates with the Fediverse.

I’m rather tired of static site generators. They always feel too static and lack interactivity which made writing not satisfying for me as I have no idea what people think about my writings.

I find myself preferring to write more stuff on Mastodon than my blog mainly because of the interactivity that I get vs my old static blog.

I hope that you’ve enjoyed my shitty long texts as there would hopefully be more.

The static blog will stay just as an archive, maybe.

So, this is writings, a place where I post long text onto. I hope it serves you well.

My style of writing will be very casual with some f-bombs peppered in. I wouldn’t try to pretend to be serious like some sort of wiki page since well, I suck at it and it just wouldn’t be me anyway.

That’s it really, See ya soon.

I’m always available at @yuki_is_bored@bsd.network.

Surviving Docker: Packaging penguins into boxes

Thu, 11 Mar 2021 00:00:00 GMT

Docker is typically known as the tool for managing and running containers in Linux.

However, a lot of people seem to have misunderstood what it is used for and why it has seemingly taken center stage in the Linux world.

For those coming from the world of BSD, Solaris, or OpenVZ might see the word ‘container’ and get confused once they try Docker. It looks very familiar but somewhat foreign at the same time.

For those who are creating web applications might have issues with the development to deployment cycle and would like to have a more straight forward method.

So, what is Docker exactly, what problem does it solve, and how can it help you?

Difficulty of delivering software

The problem which Docker solves since the beginning of the project is a rather tricky one: How to deliver software and have it running on production?

The problem might seem easy: you can just compile the software, package the binary, and just install it on the server.

Easy, right?

But in reality, there’s always this lingering problem: How do you make sure that everything works?

Here’s a common scenario:

The server runs RHEL.
RHEL ships version 1.0 of a library.
Meanwhile Sam who’s using Arch Linux has version 1.5 installed on his machine.
Turns out there’s a change between 1.0 and 1.5 that caused a nasty, yet minute bug.

This is the infamous “it works on my machine” issue. Typically, those kind of bugs are very hard to find as the developer cannot reproduce them on their machines locally and only see it happening once it is deployed.

The typical solution was to use virtual machines. Instead of just delivering packages or binaries, why not use virtual machines images like an appliance?

The virtual machine image will be pre-configured with everything you need to run the software: the Linux distribution, the libraries and the software itself.

The server operator only needs to run the virtual machine on the server with a couple configurations and the app will simply run.

And now the developers can test by building the virtual machine image on their machines and run it.

Solved, right? Well…

Janice is using a laptop running Fedora and she can’t run virtual machines because she doesn’t have a lot of RAM, and the server operator found out that the staging server is incapable of running virtual machines for the same reason.

Damn, well, we can always just ask Sam to do final tests on his machine locally but that’s looking bad on the bus factor.

However, all developers should be able to test their changes locally on their machines to determine if everything is running correctly.

Containers as a medium to deliver items

Shipping companies solve this issue by using ISO containers. ISO containers are fixed and standardized units. The standard itself defines the size of these containers, the maximum weight, and many other intricate details.

As the sender, All you need to do is to put your stuff inside of it and ask the shipping company to deliver it to the receiver.

For the shipping company, they really don’t have to care about what’s inside of it. As long as you put it in a standard ISO container, they’ll be able to ship it on boats, trucks, and cargo planes.

These boats, trucks and cargo planes are constructed to support these standard containers and are designed around it. They typically have bays or racks for it and have a known maximum capacity based on the amount of containers it can carry.

Thanks to the ISO container revolution, The entire shipping infrastructure has improved and been simplified drastically as it only has to worry about the delivery of the ISO container, not the contents of it.

So, why don’t we have something like containers to deliver software?

The container can simply contain a lightweight Linux installation, the required libraries along with the software itself.

And we can easily run it on all the developers machines and servers without having to worry about which flavour of Linux they’re running.

Docker images as a medium to deliver software

Docker images contain the software preinstalled with the libraries which that software requires.

With the image, you can launch a container which will run the image and the software within it.

With Docker installed, You can try it now:

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

You just launched a Docker container with the hello-world Docker image which executes the hello-world binary.

If you want to explore what the environment within the container is like, you can do the following:

$ docker run -it --rm ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
5d3b2c2d21bb: Pull complete
3fc2062ea667: Pull complete
75adf526d75b: Pull complete
Digest: sha256:b4f9e18267eb98998f6130342baacaeb9553f136142d40959a1b46d6401f0f2b
Status: Downloaded newer image for ubuntu:latest
root@c7556396d607:/#

Notes:

The -i flag indicates interactive. In simple terms, this will allow you to send inputs to the running programs.
The -t flag indicates that a TTY needs to be allocated for the container.
- If you don’t understand what “TTY” means, don’t worry, it is simply required by a lot of interactive command line applications. In practice, every time you run a container with interactivity, you always do -it together.
- If you really want to know what this “TTY” business is about, see virtual console.
The --rm flag indicates that the container should be removed once it’s stopped.

Docker in practice

Before we dig any deeper, let’s make a simple web application and make a Docker image for it.

This application will use Python and the Flask micro web framework.

First, make an empty folder and place app.py with the following content:

import socket
from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return f'Hello from {socket.gethostname()}!'

Don’t forget the requirements.txt which lists the required dependencies. Place this in the same folder as app.py.

flask==1.1.2

You can try running the application locally with flask run (assuming you’ve installed the dependencies on your machine!) but it’s not that important.

Docker images are created with the help of Dockerfile. Dockerfiles are simply recipes to build the Docker image.

Here’s the Dockerfile for our small project:

# Use python:3 as the base of our image
FROM python:3

# Expose port 5000, used by flask as a port to listen
EXPOSE 5000

# Create an APP_DIR environment variable
ENV APP_DIR=/opt/app

# Set our current working directory to $APP_DIR
WORKDIR $APP_DIR

# Copy only requirements.txt and install the dependencies
# This is a common trick used to save space!
COPY requirements.txt $APP_DIR
RUN pip install -r requirements.txt

# Copy the entire project
COPY . $APP_DIR

# Execute `flask run --host=0.0.0.0` when you run the container
CMD ["flask", "run", "--host=0.0.0.0"]

You can see that the image is based of an existing Docker image which has Python installed.

There are also plenty of other “base” images for various languages/frameworks, such as node, go, rust and also for Linux distributions as well, like debian, ubuntu, opensuse/leap.

After that, we set the working directory to /opt/app where our application will live, install the dependencies with pip, copy the project to the image and set a command to run.

Before we continue, Here’s the directory tree to ensure that everything is setup correctly.

my_first_docker
├── app.py
├── Dockerfile
└── requirements.txt

Now we can just run docker build to build the image.

$ docker build -t my_first_docker .
Sending build context to Docker daemon  4.608kB
Step 1/8 : FROM python:3
3: Pulling from library/python
0ecb575e629c: Already exists
7467d1831b69: Already exists
feab2c490a3c: Already exists
f15a0f46f8c3: Already exists
937782447ff6: Already exists
e78b7aaaab2c: Already exists
b68a1c52a41c: Pull complete
ddcd772f47ec: Pull complete
aef84dafa567: Pull complete
Digest: sha256:2c9e0841ab570f51f28891513c4d9b02e13954fa2453df909e0a6bbfbbaaaad3
Status: Downloaded newer image for python:3
 ---> 254d4a8a8f31
Step 2/8 : EXPOSE 5000
 ---> Running in 8bed4e787093
Removing intermediate container 8bed4e787093
 ---> 160558bc30bc
Step 3/8 : ENV APP_DIR=/opt/app
 ---> Running in fa0520760ded
Removing intermediate container fa0520760ded
 ---> 44e688dfb794
Step 4/8 : WORKDIR $APP_DIR
 ---> Running in 9e9b1f7ffd1a
Removing intermediate container 9e9b1f7ffd1a
 ---> 54658db77948
Step 5/8 : COPY requirements.txt $APP_DIR
 ---> 5a16c297f492
Step 6/8 : RUN pip install -r requirements.txt
 ---> Running in f4820ea014d0
Collecting flask==1.1.2
  Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting Jinja2>=2.10.1
  Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
Collecting Werkzeug>=0.15
  Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting itsdangerous>=0.24
  Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting click>=5.1
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting MarkupSafe>=0.23
  Downloading MarkupSafe-1.1.1-cp39-cp39-manylinux2010_x86_64.whl (32 kB)
Installing collected packages: MarkupSafe, Werkzeug, Jinja2, itsdangerous, click, flask
Successfully installed Jinja2-2.11.3 MarkupSafe-1.1.1 Werkzeug-1.0.1 click-7.1.2 flask-1.1.2 itsdangerous-1.1.0
Removing intermediate container f4820ea014d0
 ---> d704783885b1
Step 7/8 : COPY . $APP_DIR
 ---> c6a6b299293a
Step 8/8 : CMD ["flask", "run", "--host=0.0.0.0"]
 ---> Running in 299dd52ed30a
Removing intermediate container 299dd52ed30a
 ---> 4fa03655dab1
Successfully built 4fa03655dab1
Successfully tagged my_first_docker:latest

Notes:

The -t flag signifies the image tag we want, essentially a name for the image we’re creating.
The . just says to build the image using the files of the current working directory.

You can see that it runs each step of the Dockerfile inside intermediate containers. In each step, it performs the action in an intermediate container which will create an image layer that will be used for the next step. This cycle continues until the last step, whose layer will be used as the final image. After the image is built, it will be tagged when asked.

You can confirm this by running docker history:

$ docker history my_first_docker
IMAGE          CREATED         CREATED BY                           SIZE
4fa03655dab1   2 minutes ago   #(nop)  CMD ["flask" "run" "--hos…   0B
c6a6b299293a   2 minutes ago   #(nop) COPY dir:81a839249fd7a77dd…   673B
d704783885b1   2 minutes ago   pip install -r requirements.txt      9.83MB
5a16c297f492   2 minutes ago   #(nop) COPY file:118b963683835467…   13B
54658db77948   2 minutes ago   #(nop) WORKDIR /opt/app              0B
44e688dfb794   2 minutes ago   #(nop)  ENV APP_DIR=/opt/app         0B
160558bc30bc   2 minutes ago   #(nop)  EXPOSE 5000                  0B
254d4a8a8f31   2 weeks ago     #(nop)  CMD ["python3"]              0B
<missing>      2 weeks ago     set -ex;   wget -O get-pip.py "$P…   8.01MB
<missing>      2 weeks ago     #(nop)  ENV PYTHON_GET_PIP_SHA256…   0B
<missing>      2 weeks ago     #(nop)  ENV PYTHON_GET_PIP_URL=ht…   0B
<missing>      2 weeks ago     #(nop)  ENV PYTHON_PIP_VERSION=21…   0B
<missing>      2 weeks ago     cd /usr/local/bin  && ln -s idle3…   32B
<missing>      2 weeks ago     set -ex   && wget -O python.tar.x…   55.8MB
<missing>      2 weeks ago     #(nop)  ENV PYTHON_VERSION=3.9.2     0B
<missing>      4 weeks ago     #(nop)  ENV GPG_KEY=E3FF2839C048B…   0B
<missing>      4 weeks ago     apt-get update && apt-get install…   18MB
<missing>      4 weeks ago     #(nop)  ENV LANG=C.UTF-8             0B
<missing>      4 weeks ago     #(nop)  ENV PATH=/usr/local/bin:/…   0B
<missing>      4 weeks ago     set -ex;  apt-get update;  apt-ge…   510MB
<missing>      4 weeks ago     apt-get update && apt-get install…   146MB
<missing>      4 weeks ago     set -ex;  if ! command -v gpg > /…   17.5MB
<missing>      4 weeks ago     set -eux;  apt-get update;  apt-g…   16.5MB
<missing>      4 weeks ago     #(nop)  CMD ["bash"]                 0B
<missing>      4 weeks ago     #(nop) ADD file:8f75f11b2bd2d50e5…   114MB

You can see how much space each layer takes by looking at the size. This is how Docker saves on storage space, if a layer already exists on the machine, it is simply reused.

This means if you already pulled python:3 and want to pull another Docker image which uses python:3 as a base, it will only pull the extra layers introduced by the new image instead of everything.

The layers that are indicated as <missing> are simply the layers from the python:3 image we used as the base. You can also see how python:3 is made and notice that it uses ubuntu as the base.

Now, let’s run our image as a container:

$ docker run -p 8000:5000 my_first_docker
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

This will run the my_first_docker image that you’ve created and performs port forwarding from port 8000 of your machine to port 5000 of the container.

Now, you should be able to visit the running website by going to https://127.0.0.1:8000 and see Hello from <container hash>! on your web browser.

On another terminal, you can see the running Docker containers with docker ps.

$ docker ps
CONTAINER ID   IMAGE             COMMAND                  CREATED         STATUS         PORTS                    NAMES
6ded79639a17   my_first_docker   "flask run --host=0.…"   5 seconds ago   Up 4 seconds   0.0.0.0:8000->5000/tcp   nice_bassi

From here, you can see essential info such as container ID, image, status, ports and names.

If a container name is not given, Docker will generate a name typically composed of two random words.

You can also see that Docker performs port forwarding from 0.0.0.0:8000 to port 5000 of the container.

What if you want to run something in the running container, like a shell to debug something. You can use docker exec to do this.

$ docker exec -it nice_bassi bash
root@6ded79639a17:/opt/app# ls
Dockerfile  __pycache__  app.py  requirements.txt
root@6ded79639a17:/opt/app#

To stop the container, you can use docker stop or press CTRL-C on the terminal with the container running.

$ docker stop nice_bassi
nice_bassi

If you check docker ps now, you might see that the list is empty, since docker ps doesn’t include stopped containers by default. To do that you need to add the -a flag which means “all”.

$ docker ps -a
CONTAINER ID   IMAGE             COMMAND                  CREATED         STATUS                     PORTS     NAMES
6ded79639a17   my_first_docker   "flask run --host=0.…"   6 minutes ago   Exited (0) 3 seconds ago             nice_bassi

Now you can see that it is stopped. When stopped, the container is still resident on storage, you can start it again with docker start and check with docker ps to see that the container is running.

To actually remove the container, you need to use docker rm. However, you need to stop it first.

$ docker rm nice_bassi
nice_bassi

You can export your created image to a tar archive with docker save.

$ docker save -o exported_my_first_docker.tar my_first_docker
$ ls -lh exported_my_first_docker.tar
-rw------- 1 yuki yuki 877M Mar 11 19:00 exported_my_first_docker.tar

However, the common way to deliver Docker images is by using registries. There are various online registry services such as Docker Hub, Quay, GitHub Packages, and GitLab Registry.

This is an example of me pushing the image to my personal Docker registry.

$ # First, I need to login to my registry
$ docker login registry.yukiisbo.red
Username: yuki
Password:

Login Succeeded
$ # We need to tag the image to match the repository in the registry
$ docker image tag my_first_docker registry.yukiisbo.red/yuki/my_first_docker
$ # Now we can push it to the registry :-)
$ docker push registry.yukiisbo.red/yuki/my_first_docker

Congratulations, you have learned the basics of Docker and most of the commands that you will use!

In practice, the entire image creation process is automated using continuous integration. See Use Docker to build Docker images from GitLab for an example.

Another helpful command that I’ll introduce to you is docker inspect. docker inspect, as the name implies, is used to inspect stuff in Docker. This means you can use it to inspect images, containers, networks and everything else.

Try running docker inspect my_first_docker to inspect the image or inspect the container.

Stateless Containers

Docker containers are stateless. This means that all data created/modified in the container will be deleted and aren’t permanent. This is what stateless means and all Docker containers are expected to be stateless.

In comparison to stateful where all of the data are expected to be saved and permanent (persistent). With Docker containers, this is not the case, new data written or modified will be wiped clean once the container is stopped/removed, it shouldn’t be expected that the data will be safe.

You might ask: “Well, how do I use Docker to run databases?”

The answer is to this problem by using Docker volumes or bind-mounts.

Storing stateful data with bind-mounts.

Bind mounts allows “binding” a part of the host filesystem to the container’s filesystem. In simple terms, you give the container access to a folder on the host machine.

If you’ve used “shared folders” in virtual machines, you can think of them as effectively the same thing.

$ mkdir white_hole
$ docker run -it --rm -v ./white_hole:/opt/black_hole ubuntu bash

You use the -v flag to perform a bind mount, the example above will bind the white_hole folder created on host to /opt/black_hole in the container.

Once you stop the container, you could see that all of your data is safe as it is stored on the white_hole folder instead of the container.

Storing stateful data with Docker volumes

Docker volumes is similar to bind-mounts but the folder is managed by Docker.

$ docker volume create another_white_hole
$ docker run -it --rm -v another_white_hole:/opt/black_hole ubuntu bash

You could see that it’s very similar to bind mounts but instead of creating a folder in the filesystem, we use Docker to create the folder.

In my experience, this is preferred to standard bind mounts because of it’s lower maintenance overhead, plus everything would be handled through docker.

To remove the volume, it’s a simple docker volume rm command away:

$ docker volume rm another_white_hole
another_white_hole

If you want to manipulate the contents (files, permissions, etc), you would need to launch a container which mounts the volume.

The general rule of thumb that I have is to use Docker volumes whenever possible unless I actually need to use bind mount (reasons such as using another drive, programs on host which requires manipulating the files, etc).

Container networking

We can now run containers and store data persistently but of course, all modern server software require other software to work (i.e Wordpress requires a database to work).

You might think to create a mega container with an init system which launches a web server, a database server, a SSH server, etc but this is not how you should use Docker.

This is the major difference in methodology when comparing against other container technologies where they’re used to separate different environments for resource isolation or security.

In Docker, containers are used as a platform to deliver and run applications. While it is possible to use Docker containers for resource isolation and security, it is not the main issue which it aims to solve.

So, instead of the traditional approach, in Docker, you typically run one software on one container.

Let’s run Wordpress as an example.

To run Wordpress, it requires a MySQL/MariaDB database server and a place to store data for both the database server and Wordpress itself.

So, here’s the bill of material:

A container which runs Wordpress.
Another container which runs a MySQL/MariaDB database server.
A volume for the database to store the data.
Another volume for Wordpress to store media.

Here, we’re missing a way for both containers to communicate with each other. In order to do just that, we will be creating a Docker network, specifically bridge network.

Docker network by default are bridge networks which behaves similar to a router with your containers connected to it.

There are other types of Docker networks (typically referred to as drivers) such as host, overlay, etc but we’ll only focus on bridge in this article.

Now, let’s add a network to our bill of materials.

A container which runs Wordpress.
Another container which runs a MySQL/MariaDB database server.
A volume for the database to store the data.
Another volume for Wordpress to store media.
A network for the containers to use for communication.

Perfect, let’s create the network and volumes.

$ docker network create cool_wordpress
cool_wordpress
$ docker volume create cool_wordpress_db_data
cool_wordpress_db_data
$ docker volume create cool_wordpress_site_data
cool_wordpress_site_data

Awesome, let’s start our database server. We’ll be using MariaDB which already has an official image on Docker Hub: https://hub.docker.com/_/mariadb.

Looking at the docs in the Docker Hub page, we need to declare settings via environment variables.

$ docker run -e MYSQL_DATABASE=wordpress \
           -e MYSQL_USER=wordpress \
           -e MYSQL_PASSWORD=wordpress \
           -e MYSQL_RANDOM_ROOT_PASSWORD=yes \
           --network cool_wordpress \
           -v cool_wordpress_db_data:/var/lib/mysql \
           --name cool_wordpress_db \
           -d \
           mariadb

Notes:

The -e flags are used to define environment variables in the following form VARIABLE_NAME=VALUE.
The --network flag is used to define which network the container should attach to.
The --name flag is used to name the container instead of using a randomly generated name.
The -d flag is used to run the container in the background instead of “attaching” to it.

You can check whether the database is running with docker logs.

$ docker logs cool_wordpress_db
...
2021-03-15 16:23:08 0 [Note] Reading of all Master_info entries succeeded
2021-03-15 16:23:08 0 [Note] Added new Master_info '' to hash table
2021-03-15 16:23:08 0 [Note] mysqld: ready for connections.
Version: '10.5.9-MariaDB-1:10.5.9+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

You can add the -f flag if you want to “follow” the logs instead of exiting. If it’s too long, you can also add --tail=10 to only show the last 10 lines.

Wordpress also has an official image on Docker Hub: https://hub.docker.com/_/wordpress

Lets start it up!

$ docker run -e WORDPRESS_DB_HOST=cool_wordpress_db \
           -e WORDPRESS_DB_USER=wordpress \
           -e WORDPRESS_DB_PASSWORD=wordpress \
           -e WORDPRESS_DB_NAME=wordpress \
           --network cool_wordpress \
           -v cool_wordpress_site_data:/var/www/html \
           -p 8081:80 \
           --name cool_wordpress_app \
           -d \
           wordpress

Important thing to note is that other containers can be reached via its name (see WORDPRESS_DB_HOST) as long they’re within the same network.

Wordpress should now be running on port 8081: http://127.0.0.1:8081 and you should be greeted with Wordpress’ setup wizard.

Congratulations, you just launched an entire website with a database and stateful storage with Docker.

I’ve said that other container technologies use containers to have separate environments.

In Docker, we achieve a similar effect with Docker networks to isolate different resources. With Docker networks, you can group containers together within the same network so they’re able to reach other containers as resources (i.e database servers, object storage servers, etc).

A common use for this is having different networks for different environment when developing an application. In this case, you would have three networks for development, staging, and production.

Another common use is to isolate different “services” such as your Wordpress website, your Nextcloud server, and your Minecraft server. You don’t want your Wordpress website nor Minecraft server to communicate with the database server of your Nextcloud. So, you isolate them by having three networks: one for Wordpress, another one for Nextcloud, and the last one for Minecraft.

Orchestrating Docker containers

At this point, you might find the entire docker volume create, docker network create and docker run process to be very tedious and annoying. Not to mention you have to delete, stop, restart one-by-one.

Well, there’s a tool called Docker compose which aims to make it less annoying by using YAML files to declare a “stack”.

You can think a stack as a set of containers, volumes, and networks that are required to run your application.

Reusing our Wordpress example, this is what a docker-compose.yml file looks like:

version: "3"

services:
  db:
    image: mariadb
    environment:
      - MYSQL_DATABASE=wordpress
      - MYSQL_USER=wordpress
      - MYSQL_PASSWORD=wordpress
      - MYSQL_RANDOM_ROOT_PASSWORD=yes
    volumes:
      - db_data:/var/lib/mysql
  app:
    image: wordpress
    ports:
      - 8082:80
    environment:
      - WORDPRESS_DB_HOST=db
      - WORDPRESS_DB_USER=wordpress
      - WORDPRESS_DB_PASSWORD=wordpress
      - WORDPRESS_DB_NAME=wordpress
    volumes:
      - site_data:/var/www/html

volumes:
  db_data: {}
  site_data: {}

Simply place the example above inside a folder named awesome_wordpress as a file named docker-compose.yml.

To launch the site, you run docker-compose up

$ docker-compose up -d
Creating network "awesome_wordpress_default" with the default driver
Creating volume "awesome_wordpress_db_data" with default driver
Creating volume "awesome_wordpress_site_data" with default driver
Creating awesome_wordpress_app_1 ... done
Creating awesome_wordpress_db_1  ... done

The -d flag is used here so it doesn’t attach my terminal to the containers logs. You can view the logs with docker-compose logs.

You should be able to access the site on port 8082: http://127.0.0.1:8082.

To stop the site, you use docker-compose stop

$ docker-compose stop
Stopping awesome_wordpress_db_1  ... done
Stopping awesome_wordpress_app_1 ... done

To remove the containers and network, you use docker-compose down

$ docker-compose down
Removing awesome_wordpress_db_1  ... done
Removing awesome_wordpress_app_1 ... done
Removing network awesome_wordpress_default

If you want to remove the volumes as well, simply add the -v flag.

$ docker-compose down -v
Removing network awesome_wordpress_default
WARNING: Network awesome_wordpress_default not found.
Removing volume awesome_wordpress_db_data
Removing volume awesome_wordpress_site_data

There you go, much simpler, right?

There are a couple details that I’d like to point out quickly:

All resources (containers, volumes, and networks) have awesome_wordpress at the beginning of their names.
- This prefix is called a project name and Docker compose use the folder name where docker-compose.yml resides by default.
Without an explicit declaration of the network, Docker compose creates a default network for you and attaches all of the container to the created network.
Both volume declaration are defined as {}. This is the defaults works fine for our needs, we don’t need any specific config.
- You can leave them empty but I prefer having {} so it’ll be explicit to people reading the file.
You don’t need to use the entire container name (awesome_wordpress_db_1) when communicating with another container. You can use the service name instead.
- This is possible as you can define aliases for the container within the network.

If you see the container names, they have _1 at the end (suffix). This is because Docker compose uses the notion of services instead of containers directly.

“Why is this important?”, you might ask.

This is because you can “scale” services, meaning launching multiple containers for a service. This is called “horizontal scaling” as you scale by having multiple instance of a component. Very helpful when you want to share the load across multiple web servers.

While I’m not certain how well Wordpress scales horizontally, we can do it with docker-compose scale.

$ docker-compose scale app=4

This will fail as port 8082 is used by the application, so further configuration is required (in this case, setting up a reverse proxy/load balancer) which we will not cover here.

While we’re here, lets say you want to change the port for Wordpress from 8082 to 80 (assuming port 80 is free).

Simply change 8082:80 to 80:80:

version: '3'

services:
  db:
    image: mariadb
    environment:
      - MYSQL_DATABASE=wordpress
      - MYSQL_USER=wordpress
      - MYSQL_PASSWORD=wordpress
      - MYSQL_RANDOM_ROOT_PASSWORD=yes
    volumes:
      - db_data:/var/lib/mysql
  app:
    image: wordpress
    ports:
      - 80:80
    environment:
      - WORDPRESS_DB_HOST=db
      - WORDPRESS_DB_USER=wordpress
      - WORDPRESS_DB_PASSWORD=wordpress
      - WORDPRESS_DB_NAME=wordpress
    volumes:
      - site_data:/var/www/html

volumes:
  db_data: {}
  site_data: {}

Update it with docker-compose up.

$ docker-compose up -d
Recreating awesome_wordpress_app_1 ... done

You can see that it only updates app and leave db alone. Cool, right?

Congratulations, you just orchestrated an entire Wordpress website with the power of Docker compose!

You have seen two methods to deploy things with Docker: Docker compose and docker run.

Personally, I typically define stacks and deploy things using Docker compose and only use docker run when I need to make one-off containers for testing, debugging, etc.

I like Docker compose more mainly because it’s declarative as I simply need to define what I want instead of writing long commands. It’s just much simpler and fits the task of long running services.

Closing

This is the end of this article.

While I could go on about Docker’s internals among other things, I think it would be better for that to be a separate post or you can simply explore on your own, besides practice is the best teacher.

There are a couple of small things which I haven’t covered but I think it’s better for you to find those things on your own as you need them.

I recommend checking out Compose file version 3 reference for various other things that you can do with Docker.

Remarks

This is an addition I made on the 13th of August while writing an article on my series about Nix. I should’ve done it earlier but only realised it now.

I would like to thanks those who has reviewed this article and given me their input:

Lucy Firman (@lxcyp on GitHub)

If you’re one of those who has reviewed this article and would like to be credited, please let me know. :)

Qubi Research Program Postmortem

Sat, 26 Jun 2021 00:00:00 GMT

Hello world,

After a long 8 week hiatus, I’m finally back with a new game on itch.io.

It’s called “Qubi Research Program” which is a rather “unique” typing game. I came up with the idea for it after reading the theme for the 83rd Mini Jam which was “Dread”.

With the word “Dread” in mind, I immediately thought about a shitty office job which one require in order to survive. Wouldn’t it be silly to have a game which replicate Stanley’s job but have the dreadful feeling of living within a broken system as Papers, Please.

Proof of concept

During this time, I was learning Nim and Raylib so I decided to quickly make a proof-of-concept.

Your browser does not support the video tag.

The proof-of-concept is only the typing game part, just to see how it feels but the idea is there.

At the end of each work day, the player is awarded with a compensation based on the amount of words you’ve written. Each typo made are fined.

The game grows difficult each day as you have to pay various bills and make sure that your account isn’t surpassing the fixed overdraft limit.

My close friends liked the idea and told me that I should make it even if it couldn’t submit it for the 83rd Mini Jam.

The game uses the same common English words dictionary as Up Goer Five and Thing Explainer. The reason is simply that I want to only use words which everyone have heard and used before.

Initial foothold

I started writing the game on Friday, 25 June 2021. I decided to do it on a public repository in my personal GitLab instance because why not.

There’s not a lot of games written in Nim surprisingly so I think it could serve as a relatively complete example of a simple game.

I started by writing a simple Hello World and a Nix shell environment which have Nim and Emscripten.

Emscripten / WASM has proven to be quite annoying at times. One of the annoyances is the cooperative multitasking model which cause the entire program to freeze if it is shoved to the background. I’m not sure how to fix this bug and couldn’t really find any good references on how to fix it.

However, once I have a working toolchain setup and make it possible to build within Nix imperatively, it goes smooth sailing.

Constructing the Foundation

The architecture of the game is very simple. It uses screens which encapsulates a state along with methods for screen initialisation, update and deinitialisation. Each screen are simply a subclass of the Screen abstract class. No fancy ECS required for something simple and quick.

While I am not a fan of object-oriented programming, something like this shows why OOP can be good sometimes especially when you need to encapsulate both a state and an interface.

I start to appreciate Nim’s approach in making both functional / imperative / object-oriented approach work together nicely.

In particular, I like that f(x) is the same as x.f. This is similar to the pipeline operator which are popular in functional programming languages. This makes f(g(h(x))) readable by transforming it to x.h.g.f.

Of course, there are a lot of things inside Nim which I love: Portable compiler, Sum types/ADTs, Macros, Syntax clarity, Infix functions, etc.

Honestly, I wish more programming languages reflects Haskell’s syntax clarity. I hate those which have a lot of noise such as Rust. However, I think I’ll use Nim for things which does not make sense to do in Haskell.

Basic Gameplay

Porting the code from the proof-of-concept to the foundation is fairly simple. Most of it just moving the stateful variables within the encapsulated state of the GameScreen class.

Once that’s done, I just started working on the “Day End Screen” and the “Game Over Screen”. The game progress is saved outside of the encapsulated state. In other words, they’re just global variables.

While a lot of people cringe at mutable global variables, it’s fine for something quick and dirty and who cares, really. I’m not writing software, I’m writing games.

After I got the basic gameplay / minimum viable product, I immediately share it with close friends.

Balancing

Originally, I planned to have a day to be a minute but during initial playtesting, I deemed it to be too long so I cut it in half. a week was 5 days but it is also too long, I end up cutting it down to 3 days.

So in this game, a day is 30 seconds and a week is 3 days. You also pay rent every week which increases every week too. I would like for the game to also last a bit long but not too long, unless you really want to.

You get 1 point for each word done but you’re fined 2 points for each error you made. While this seem excessive, this is deliberate to make it difficult and have that “dreadful” feeling.

The initial version was fairly difficult even for accurate/fast typists. No one was able to surpass 15 days / 7 minutes of gameplay.

To balance, I start by laying down some ground rules. I think it’s unreasonable to have a speed of 1 word a second. So, I cut it down by 20% in order to have a bit more room for errors. In the real world, rent is ideally at most 60% of your income. With the daily goal value in place, I simply make the initial rent to be 60% of the weekly goal.

In the real world, 10% rent increase is excessive but this is a game and games need to be short so 10% it is.

With the balanced changes in place, it is now easy to hit more than 30 days. Nice, now I have more room to make the game more difficult

Increasing Difficulty

The first thing I did is to have the duration of each day randomized to destroy the 30 second rhythm.

Second, I want to shuffle the characters in a word which reflects Typoglycemia / Transposed letter effect. This is to make it even more annoying once you have the rhythm and expectation to write English words.

Third, like in “Papers, Please”, I want to have “surprise costs”. Paying for random things such as coffee, parking, and student loans.

The factor of each “surprise cost” is simply a prime number from 2 to 59. At each the end of the work day, A random value from 2 to 100 is generated and if it’s multiple of a suprise cost’s factor, the surprise cost is added. I’m too lazy to do something proper and this will do just fine.

End

Your browser does not support the video tag.

I think I’m done with this game. It’s fairly short and simple and it is more of an excercise for me in learning Nim and Raylib.

I love both Nim and Raylib and I think I have found a good stack to write small games with. I hope with all of this that I will start creating games a bit more frequently. No promises though :-)

Thanks for reading this rather messy and all-over-the-place of a postmortem. As of writing, it is 5 AM in the morning and I’ve been awake since 12 PM. I should go to bed…

The game is available on itch.io and its source code (ISC) is available on GitHub.

Spice up with Nix: The issues with traditional software deployment

Sat, 04 Sep 2021 00:00:00 GMT

In this article, we will explore the issues with traditional methods of installing software.

This includes using distribution package managers such as apt, dnf, pacman and third-party ones like MacPorts and Homebrew.

In addition, we will take a look at containerized solutions and their issues.

The problem with traditional software deployment

In the Nix thesis, Eelco Dostra potrayed how ancient the current methods of software deployment is by comparing it to how software developers used to manage and allocate memory in Assembly.

You don’t need to know how memory is managed in Assembly as I will illustrate the issue without even mentioning it.

The filesystem of a typical *nix system (i.e Linux, macOS) is essentially free-form with loosely defined “standards” and “conventions” adopted by system administrators, software developers, and package maintainers.

In this case, filesystem refers to how files are structured on your system, like the /bin/ folder contains program binaries and your personal “home” folder is inside the /home/ folder.

If you want to see how different this can be, see how files are structured in macOS and a typical Linux distribution.

You might argue that they shouldn’t be compared but they both share a lot of the same software (i.e Docker in macOS with Homebrew and Docker in Ubuntu).

Even then, it’s not like every distributions of Linux follow a rigid and well-defined standard.

Some of you may have strong opinions about where software should be installed on your system.

Perhaps, you have a rigid rule where all user-compiled software must be stored inside a single prefix like /usr/local/ or isolated nicely inside their own prefixes (i.e /usr/local/nginx/, /opt/android/, /home/minecraft/).

In the end, these are choices driven by opinions which are made by different parties.

Now, as a software developer, how do you handle this mess?

Let’s take a fairly simple example: a Python script.

It’s nothing complicated, something that is written by someone who’s new to Python:

#!/usr/bin/python

def main():
    for i in range(1, 11):
        print(f'Counting down {i}...')

if __name__ == '__main__':
    main()

That’s a fairly simple, isn’t it?

But look at the shebang (#!), the script has a major assumption of where Python is installed.

It expects it to be installed at /usr/bin/python.

However, the script uses a feature which only exists in Python 3.

If you’re running a system that’s in the “Python is Python 3” club, it may work for you.

But for a lot of people, python is not python3, it’s still python2.

$ python --version
Python 2.7.18
$ ./script.py
  File "./script.py", line 5
    print(f'Counting down {i}...')
                                ^
SyntaxError: invalid syntax

The reason for this is mainly for compatibility as Python 3 introduced a lot of features which breaks its backwards-compatibility with Python 2.

To fix this issue, the shebang must use /usr/bin/python3.

#!/usr/bin/python3

def main():
    for i in range(1, 11):
        print(f'Counting down {i}...')

if __name__ == '__main__':
    main()

Now, you encounter a different issue: Some people use various flavors of BSDs (including macOS) instead of Linux.

$ ./script.py
zsh: ./script.py: bad interpreter: /usr/bin/python3: no such file or directory

Unlike Linux, BSDs has a monolithic approach where it is not just a kernel but also encapsulates everything which makes a complete BSD system.

In this case, “everything” could be anything from the basic core utilities (cat, grep, ls, etc.) to an audio server or even a ready-to-use HTTP server.

This is why Linux distributions exists, someone need to construct a complete system with the Linux kernel and additional essential components.

Because of BSD’s monolithic approach, third-party software are always installed under a separate prefix (such as /usr/local/) because /usr/ is occupied by software provided by the operating system.

However, you can’t always assume that the software will be in /usr/local/.

In macOS which lacks an official package manager from Apple, macOS users uses a third-party package manager such as Homebrew, MacPorts, and pkgsrc which installs software in different places within the filesystem.

In a NetBSD system, there are two prefixes for third-party software:

The /usr/pkg/ prefix for software that are installed by pkgsrc.
The /usr/local/ prefix for software that are compiled and installed by the user.

With this new knowledge and realisation of the issue, you look around to find if there’s a simple solution.

After all, this should be a solved problem, right?

After a while, you’ll find env which is a loose convention adopted by most *nix systems.

/usr/bin/env is a program that will direct the user to run the right program, no matter how the system is configured. Just give it a name of the executable and it will find and execute it for you.

Perfect, simply replace your /usr/bin/python3 with /usr/bin/env python3 and now it works.

#!/usr/bin/env python3

def main():
    for i in range(1, 11):
        print(f'Counting down {i}...')

if __name__ == '__main__':
    main()

Oh no, you realize another issue: the script requires at least Python 3.6 because it uses f-strings.

$ python3 --version
Python 3.5.9
$ ./script.py
  File "./script.py", line 5
    print(f'Counting down {i}...')
                                ^
SyntaxError: invalid syntax

Well, as the developer, you can choose to be ignorant and simply state: “Your system is obselete if it doesn’t have at least Python 3.6”.

Even then, what about system distributions that ships multiple versions of Python?

Package repositories such as the ones used in macOS Homebrew, FreeBSD, and OpenBSD ships multiple versions of Python, Ruby, Node, PHP, and others.

$ uname -s
OpenBSD
$ pkg_info -Q python
...
python-2.7.18p1
python-3.8.10
python-3.9.5
...
$ type -a python3
python3 not found
$ type -a python3.8
python3.8 is /usr/local/bin/python3.8
$ type -a python3.9
python3.9 is /usr/local/bin/python3.9

The binaries for them are typically separated using the version as a suffix (i.e python37, python38, python39)

So, how do you assume what python3 will be?

Is python3 the latest version?
Is it something in between that’s chosen by the system administrator?
Is it the oldest available version for compatibility?
Does it even exist at all?

As you can see, this is the very issue with traditional software deployment and why it can feel ancient at times.

You would like to believe that the solution simple and common. However, as you gaze down the abyss of software deployment, you start to realize that it is bottomless.

We haven’t started using third-party Python libraries or even native libraries.

This is just a simple script which requires a Python interpreter that’s at least version 3.6.

In many ways, this was someone’s “Hello, World!”.

What if we require libraries like requests?
What if we require programs like FFmpeg and ImageMagick?
What if we require data files like game assets?

This is a universal issue that affects everyone.

From the first-year computer science student to the experienced veteran administrator, no one is able to pursue their adventures without having to plunge deep into the abyss of software deployment.

The problem with traditional packages

Most system distributions ships third-party software.

They could be a recipe that the user can execute to download, compile, and install third-party software.

$ cd /usr/ports/misc/hello/
$ make
===>  License GPLv3+ accepted by the user
===>   hello-2.10_1 depends on file: /usr/local/sbin/pkg - found
=> hello-2.10.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://ftpmirror.gnu.org/hello/hello-2.10.tar.gz
hello-2.10.tar.gz                                      708 kB 3368 kBps    00s
===> Fetching all distfiles required by hello-2.10_1 for building
===>  Extracting for hello-2.10_1
=> SHA256 Checksum OK for hello-2.10.tar.gz.
===>  Patching for hello-2.10_1
===>   hello-2.10_1 depends on package: gmake>=4.3 - found
...
===>  Configuring for hello-2.10_1
...
===>  Building for hello-2.10_1
...
====> Compressing man pages (compress-man)
$ doas make install
===>  Installing for hello-2.10_1
...
$ hello
Hello, world!

Or, they could be a binary package which the user can easily install with a tool.

$ doas pkg install hello
Updating HardenedBSD repository catalogue...
HardenedBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        hello: 2.10_1 [HardenedBSD]

Number of packages to be installed: 1

56 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching hello-2.10_1.txz: 100%   56 KiB  56.9kB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Installing hello-2.10_1...
[1/1] Extracting hello-2.10_1: 100%
$ hello
Hello, world!

These packages are maintained by the humble package maintainers who risk their lives deep within the abyss in order to help users install third-party software with little to no pain.

Often times, package maintainers are required to write patches to make software work.

With the Python script example we did, they might write a patch to make it use the exact version of Python and explicitly declare which version of Python is required for the package.

For a lot of people, this works just fine.

Simply install the software you need from the package manager present in your system.

But here’s the issue, what if the software is not available from the package manager?

Well, as a user, you have two choices:

Compile and install it by hand.
Dive into the abyss and become a package maintainer.

None of the two choices are particularly good.

Becoming a package maintainer is hard as you have to be ready for the following:

Follow the latest version of the software.
Write patches to make it work.
In addition, follow the dependencies of the software.
Write patches to make the dependencies work.
And repeat.

If you compile and install it by hand, you’re on your own:

You have the responsibility of maintaining the software locally.
You have to make sure that it doesn’t conflict with anything on the system.
You have to make sure that the dependencies that it requires are there.

Essentially, you have to become the package maintainer for your own system and what if you need it in another system? Tough luck.

This issue has spawned three new trends:

The infamous curl | sh installer.
Package managers which are specific to a particular programming language (i.e npm, pip)
Using containers as a medium to deliver and run software.

The problem with `curl | sh` installers and language-specific package managers

The curl | sh installer is simply a small script that checks the state of your system and perform the proper installation procedure based on your system.

The main problem with curl | sh is their unsafe nature.

As the script is able to do anything to your system, it is no different to a Self-XSS attack.

Because it is a shell script, it cannot be limited to only perform the tasks necessary to achieve the goal of installing the software.

The programming language specific package managers has the issue of limited reach.

For example, if you’re installing a library to interact with a MySQL database, you might need to have the necessary MySQL native libraries to be installed which is out of reach for the language specific package managers.

However, both have share a major issue due to the lack of isolation: file conflicts.

For example, you’ve installed the Docker SDK for Python via pip but you need to install Docker Compose via your system’s package manager.

As Docker compose requires the Docker SDK, it also need the SDK to be installed from the system’s package manager.

The Docker SDK package from the system’s package manager cannot be installed because it is conflicting with the Docker SDK installed by pip.

Or, it will install but you will have two separate versions of the same package which will cause weird issues: Which version of the Docker SDK will docker-compose use?

The common solution for this is to install it on separated prefixes.

For example, in Python, software developers typically setup separate, local, and isolated prefixes called Virtual environments.

The same mechanic is present in curl | sh installers, they commonly use local separate prefixes (i.e ~/.cargo/ which is used by rustup).

The problem with containers

Application containers such as Docker, Snap, Flatpak and AppImage require some form of file system isolation.

This is because they not only ship the software but also an entire system tailored for running the software.

You can see this in action with Docker images which is a tarball archive of a tailored Linux system made to run a specific piece of software.

In macOS, they use something similar called Application bundles which are just folders filled with the necessary data files, libraries, and binaries.

This means that software that are distributed through these mediums takes a lot of storage space.

For a lot of people, this is not an issue because for them storage is cheap.

However, as someone who has lived their life in Indonesia with slow 4G, most of my computing time is taken by the simple task of downloading software.

To mitigate the issue, they have solutions which reduces the download size:

Docker images are composed of layers which can be shared across images.
macOS provides Frameworks which are simply bundles of popular libraries which third-party applications can use.
Flatpak share a similar system to macOS framework called Runtimes.

But in the end, these are still compromises and they won’t guarantee that you won’t redownload the same files.

Closing

As you can see, the current method of installing software is fundamentally flawed as it requires knowing the state of the system which will guide you to make the necessary changes that allows the software to be installed.

Package managers simply automate the installation process to the point where anyone can install easily without having to know much about the system.

However, package managers can only target a specific system (i.e Ubuntu or Arch) and the same packages can’t be shared between different systems (ex. you can’t use Arch packages in Ubuntu).

curl | sh installers are another form of automation with the caveat that they have unlimited reach and control on the system which is fundamentally unsafe.

Ultimately, they can’t be trusted to only perform the tasks required to install the software but this unlimited reach is required in order for it to know the current state of your system.

Programming language specific package managers doesn’t really solve this issue as they have a very limited reach by design in order to not break your system. They only care about the environment for the specific programming language it was made for.

Application containers and bundles are working around the issue by shipping a complete system with the software you need. This results in wasting a lot of resources which renders it unaccessible to a lot of people.

There are plenty more of other issues with the current methods such as the requirement of trusting the humble package maintainers.

There have been cases where you can’t trust them like the chromium binary blob scandal that has impacted Debian.

This demonstrate the fact that a malicious agent only need to attack the few humble package maintainers.

As a user, it’s not easy to audit binaries especially when they’re not reproducible.

You can choose to build and install everything from source like Gentoo but that takes a huge chunk of time which is a luxury that not everyone has.

You can build your own trusted build cluster and share the resulting binaries but that costs money to buy the necessary hardware and labor to maintain it.

In the end, I hope this is enough for you to consider that there’s a need for a new way of installing software.

In the next article, we will explore how Nix solves all of the fundamental issues with traditional software deployment by introducing a new model called functional software deployment.

Spice up with Nix: Scripts with magical dependencies

Mon, 09 Aug 2021 00:00:00 GMT

Hey there,

Instead of reading this outdated article, I highly recommend consulting nix.dev’s First Steps which is official documentation!

Feel free to continue reading this if you want to see my personal take on the subject :)

Thanks.

I wrote this cool script which shows you a list of letters and their frequency in English.

Check it!

#!/usr/bin/env python3

import requests                # Required to communicate via HTTP
import pandas                  # Required to mangle with spreadsheets
from bs4 import BeautifulSoup  # Required to parse HTML

# Let's download the Letter frequency page from Wikipedia
r = requests.get('https://en.wikipedia.org/wiki/Letter_frequency')
assert r.status_code == 200

# Use BeautifulSoup to parse HTML
s = BeautifulSoup(r.text, 'html.parser')

# Find the last table in the page
res = str(s.findAll('table', {'class': 'wikitable'})[-1])

# Let's convert that HTML table to a Pandas DataFrame
res = pandas.read_html(res)[0]

# We're only interested in English and the first 25 letters.
res = res[['Letter', 'English']][0:26]

# Properly handle the percentage in the table
res['English'] = res['English'].str.rstrip('%').astype('float') / 100.0

# Sort it
res = res.sort_values(by='English', ascending=False)

# Show it to the user
print(res)

Cool, right?

Oh, you don’t have Python on your machine?

Maybe, you don’t have the libraries it requires like requests, pandas, and BeautifulSoup.

Ah, it doesn’t run on your machine because you only have a different version of those libraries?

Instant runtime environments for scripts with Nix

If you don’t have Nix installed on your machine, install it now!

It won’t break your system, I promise!

As of writing, Nix runs well on macOS and Linux.

Nix can be used to setup a temporary runtime environment with all of the dependencies which your script need.

Simply use nix-shell as an interpreter in the shebang (#!) of your script.

Using the script I’ve written above instead of using Python which is installed on the machine:

#!/usr/bin/env python3

Replace with nix-shell and then give a list of packages to nix-shell.

#! /usr/bin/env nix-shell
#! nix-shell -i python -p python38 python38Packages.requests python38Packages.beautifulsoup4 python38Packages.pandas
#! nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/refs/tags/21.05.tar.gz

Let’s break it down on what each line means!

#! /usr/bin/env nix-shell

The first line simply tells your shell to use nix-shell as the interpreter for this script, nothing new here. Not that different how Python scripts are executed!

#! nix-shell -i python -p python38 python38Packages.requests python38Packages.beautifulsoup4 python38Packages.pandas

This line seems scary but no worries. We are simply passing arguments to nix-shell.

-i python tells Nix which interpreter to use. In this case, we’re using python.
-p python38 python38Packages... tells Nix which packages are required for this script.

It’s not that different to a recipe.

Instead of only containing instructions on what to do, with Nix, we are able to give a list of ingredients and what to use in order to perform the instructions!

#! nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/refs/tags/21.05.tar.gz

This line looks scary but no worries, it’s simple.

We’re telling Nix which version of the Nix packages collection (nixpkgs) to use. In this case, we are using version 21.05.

This is how we can pin our dependencies to ensure the person who are running the script are running the exact version of the dependencies that I works on my machine.

This also ensures that even if you read this 5 or even 10 years in the future, the script will still run with the same set of dependencies.

So, here’s the final script after we spice it up with Nix:

#! /usr/bin/env nix-shell
#! nix-shell -i python -p python38 python38Packages.requests python38Packages.beautifulsoup4 python38Packages.pandas
#! nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/refs/tags/21.05.tar.gz

import requests                # Required to communicate via HTTP
import pandas                  # Required to mangle with spreadsheets
from bs4 import BeautifulSoup  # Required to parse HTML

# Let's download the Letter frequency page from Wikipedia
r = requests.get('https://en.wikipedia.org/wiki/Letter_frequency')
assert r.status_code == 200

# Use BeautifulSoup to parse HTML
s = BeautifulSoup(r.text, 'html.parser')

# Find the last table in the page
res = str(s.findAll('table', {'class': 'wikitable'})[-1])

# Let's convert that HTML table to a Pandas DataFrame
res = pandas.read_html(res)[0]

# We're only interested in English and the first 25 letters.
res = res[['Letter', 'English']][0:26]

# Properly handle the percentage in the table
res['English'] = res['English'].str.rstrip('%').astype('float') / 100.0

# Sort it
res = res.sort_values(by='English', ascending=False)

# Show it to the user
print(res)

Only by changing that single line to three lines, this script will now run on any machine with Nix installed and it will automagically resolve the dependencies magically!

$ head -n3 cool-script.py
#! /usr/bin/env nix-shell
#! nix-shell -i python -p python38 python38Packages.requests python38Packages.beautifulsoup4 python38Packages.pandas
#! nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/refs/tags/21.05.tar.gz
$ ./cool-script.py
these 85 paths will be fetched (132.87 MiB download, 693.09 MiB unpacked):
  /nix/store/0b90d74742wl9vpy0vpd89k6cks9xvd9-bash-interactive-4.4-p23-man
  ...
  /nix/store/zsh6dmqlp15yzj3mgq3zv212waz12lbk-python3.8-urllib3-1.26.4
copying path '/nix/store/lp6ccpip5ygq0k813gyb9m4iashzk2r0-bash-interactive-4.4-p23-doc' from 'https://hydra.iohk.io'...
...
copying path '/nix/store/4dlhsl4kxp9p632mbv1rcq9kjc0y6zdy-stdenv-linux' from 'https://hydra.iohk.io'...
   Letter  English
4       e  0.12702
19      t  0.09056
0       a  0.08167
14      o  0.07507
8       i  0.06966
13      n  0.06749
18      s  0.06327
7       h  0.06094
17      r  0.05987
3       d  0.04253
11      l  0.04025
2       c  0.02782
20      u  0.02758
12      m  0.02406
22      w  0.02360
5       f  0.02228
6       g  0.02015
24      y  0.01974
15      p  0.01929
1       b  0.01492
21      v  0.00978
10      k  0.00772
9       j  0.00153
23      x  0.00150
16      q  0.00095
25      z  0.00074

As you can see, Nix will magically prepare a temporary environment with the dependencies for your script.

Wait, does this mean Nix has installed those dependencies and Python?

$ python3
Python 3.8.9 (default, Apr  2 2021, 11:20:07)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>> import bs4
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'bs4'
>>>

As you can see, it didn’t alter your system. Your system is safe.

Only Python? Nope.

This is not limited to only Python. You can use it with everything.

You want to use it in a shell script which require programs to be present on the system? No problem.

#! /usr/bin/env nix-shell
#! nix-shell -i bash -p gnat mktemp
#! nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/refs/tags/21.05.tar.gz

# Make a temp folder
WORKDIR=$(mktemp -d)

# Go to the temp directory
pushd "${WORKDIR}"

# Write some code
cat > hello.adb <<EOF
with Text_IO; use Text_IO;
procedure hello is
begin
   Put_Line("Hello, world!");
end hello;
EOF

# Compile it with GNAT
gnatmake hello.adb

# Run it
./hello

# Go back to the previous directory
popd

# Copy the executable
cp "${WORKDIR}/hello" ./hello

# Remove the temp folder
rm -rf "${WORKDIR}"j

$ ./cool.sh
these 9 paths will be fetched (46.63 MiB download, 226.41 MiB unpacked):
  /nix/store/1977rsramab0ks7bk3fxjd91kf1ddxy1-libmpc-1.2.1
  ...
  /nix/store/v07f9gbkizyfcz98ssyhbq5jr6x823lm-gmp-6.2.1
copying path '/nix/store/by026wm1s9phidfb96l5gcxyylly9j5y-expand-response-params' from 'https://hydra.iohk.io'...
...
copying path '/nix/store/a7y4ng9g8jn1vm5p6l34nbg8ns5ml7f2-gnat-wrapper-9.3.0' from 'https://cache.nixos.org'...
/run/user/1000/tmp.Q6gesiZ73k ~
gcc -c -I/nix/store/9b5w72pds7z3mj2bplx97yn9vpz5x0zl-gnat-9.3.0/lib/gcc/x86_64-unknown-linux-gnu/9.3.0/adainclude -I/nix/store/9b5w72pds7z3mj2bplx97yn9vpz5x0zl-gnat-9.3.0/lib/gcc/x86_64-unknown-linux-gnu/9.3.0/adalib hello.adb
/nix/store/a7y4ng9g8jn1vm5p6l34nbg8ns5ml7f2-gnat-wrapper-9.3.0/bin/gnatbind -I/nix/store/9b5w72pds7z3mj2bplx97yn9vpz5x0zl-gnat-9.3.0/lib/gcc/x86_64-unknown-linux-gnu/9.3.0/adainclude -I/nix/store/9b5w72pds7z3mj2bplx97yn9vpz5x0zl-gnat-9.3.0/lib/gcc/x86_64-unknown-linux-gnu/9.3.0/adalib -x hello.ali
/nix/store/a7y4ng9g8jn1vm5p6l34nbg8ns5ml7f2-gnat-wrapper-9.3.0/bin/gnatlink hello.ali
Hello, world!
~
$ ls ./hello
./hello
$ ./hello
Hello, world!

Did Nix just prepare an environment with the GNU Ada Toolchain and then run a script which compile an Ada program which runs?

Yes, it did.

What about something niche?

#! /usr/bin/env nix-shell
#! nix-shell -i runhaskell -p ghc
#! nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/refs/tags/21.05.tar.gz

nums :: [Int]
nums = [1780, 1693, 1830, 1756, 1858, 1868, 1968, 1809, 1996, 1962, 1800, 1974,
        1805, 1795,  170, 1684, 1659, 1713, 1848, 1749, 1717, 1734,  956, 1782,
        1834, 1785, 1786, 1994, 1652, 1669, 1812, 1954, 1984, 1665, 1987, 1562,
        2004, 2010, 1551,  961, 1854, 2005, 1883, 1965,  475, 1776, 1791,  262,
        1912, 1227, 1486, 1989, 1857,  825, 1683, 1991, 1875, 1982, 1654, 1767,
        1673, 1973, 1886, 1731, 1745, 1770, 1995, 1721, 1662, 1679, 1783, 1999,
        1889, 1746, 1902, 2003, 1698, 1794, 1798, 1951, 1953, 2007, 1899, 1658,
        1705,   62, 1819, 1708, 1666, 2006, 1763, 1732, 1613, 1841, 1747, 1489,
        1845, 2008, 1885, 2002, 1735, 1656, 1771, 1950, 1704, 1737, 1748, 1759,
        1802, 2000, 1955, 1738, 1761, 1765, 1853, 1900, 1709, 1979, 1911, 1775,
        1813, 1949, 1966, 1774, 1977, 1757, 1992, 2009, 1956, 1840, 1988, 1985,
        1993, 1718, 1976, 1078, 1997, 1897, 1792, 1790, 1801, 1871, 1727, 1700,
        1485,  942, 1686, 1859, 1676,  802, 1952, 1998, 1961, 1844, 1808, 1703,
        1980, 1766, 1963, 1849, 1670, 1716, 1957, 1660, 1816, 1762, 1829,  526,
         359, 2001, 1874, 1778, 1873, 1511, 1810, 1699, 1970, 1690, 1978, 1892,
        1691, 1781, 1777, 1975, 1967, 1694, 1969, 1959, 1910, 1826, 1672, 1655,
        1839, 1986, 1872, 1983, 1981, 1972, 1772, 1760]

solution :: [Int] -> Int
solution xs =
  head
    [ x * y * z
      | x <- xs,
        y <- xs,
        z <- xs,
        x + y + z == 2020
    ]
main :: IO ()
main = print $ solution nums

$ ./day1-part2-aoc.hs
these 4 paths will be fetched (138.01 MiB download, 1870.79 MiB unpacked):
  /nix/store/1pfdb37qm067ib7dnj5arbhl5q2kam3l-libffi-3.3-dev
  /nix/store/55ysfrn085k1lsya0c2msm6npi7x6ihh-ghc-8.10.4-doc
  /nix/store/f2psw0phlmp7h7gk14rfsqdmjz4d1arb-gmp-6.2.1-dev
  /nix/store/mcm0wvsqcwqaxsp8fnwmknmpi0a196w6-ghc-8.10.4
copying path '/nix/store/55ysfrn085k1lsya0c2msm6npi7x6ihh-ghc-8.10.4-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/f2psw0phlmp7h7gk14rfsqdmjz4d1arb-gmp-6.2.1-dev' from 'https://hydra.iohk.io'...
copying path '/nix/store/1pfdb37qm067ib7dnj5arbhl5q2kam3l-libffi-3.3-dev' from 'https://hydra.iohk.io'...
copying path '/nix/store/mcm0wvsqcwqaxsp8fnwmknmpi0a196w6-ghc-8.10.4' from 'https://cache.nixos.org'...
200878544

As you can see, it works for any language. Simply check if your favorite language and libraries are packaged: search.nixos.org.

Closing

This is the first in a series of article that I will be publishing more regularly.

It is based of a mega draft I have written about Nix which was written in the same style of the Docker article.

However, due to the massive differences in Nix vs other existing package managers, it has grown to a rather massive and daunting draft (1787 lines, 7265 words, 68436 characters).

Because it would be a massive article and a huge turn off for a lot of people due to its size, I will be breaking it down into smaller bite-sized posts like this one.

I will start with practical examples on how Nix can help in a lot of real-world situations. In fact, I have delivered a couple of real world projects on production with Nix and at work, we are now happily using Nix by default.

Once that’s done, we will dive head first and see how Nix works. I hope you’ll find it interesting and this has been one of the longest writing projects that I’ve done with so much research!

Thanks for reading and I hope you’ll be there on the next one!

Spice up with Nix: Instant Development Environments

Fri, 13 Aug 2021 00:00:00 GMT

Hey there,

Instead of reading this outdated article, I highly recommend consulting nix.dev’s First Steps which is official documentation!

Feel free to continue reading this if you want to see my personal take on the subject :)

Thanks.

This article assumes you have installed Nix on your machine.

You can install it by running:

$ curl -L https://nixos.org/nix/install | sh

It works on both Linux and macOS and it won’t break your system, I promise!

The beginning of your journey as a programmer is always difficult.

Before you even start programming, you have to install a bunch of tools.

When you are not familiar with the programming language, this issue becomes exponentionally harder as you are not familiar with the potential pitfalls.

The fact that not everyone is running the same operating system is also hard.

Your coworkers runs macOS and they can’t help you because you’re running Ubuntu.

That person you met on Discord runs Arch Linux and tells you to simply move to Manjaro.

But you don’t want to change your operating system or buy a new machine, you already have everything you need personally and everything is already nicely configured for you.

Why should this be difficult? Couldn’t this be easier?

Instant (development) environments with Nix

Let me demonstrate how Nix can solve this issue:

{ pkgs ? import <nixpkgs> {}}:

let
  # Python 3.8 environment with all of the packages I require
  pythonEnv = pkgs.python38.withPackages (ps: [
    # Scientific python essentials
    ps.numpy
    ps.scipy
    ps.pandas
    ps.matplotlib

    # Grab samples from the net via Python
    ps.requests
    ps.beautifulsoup4

    # Need this to make the samples more bearable
    ps.nltk

    # Notebook editing
    ps.jupyter
    ps.ipython
  ]);
in
pkgs.mkShell {
  packages = [
    pythonEnv        # The Python 3.8 environment I made.

    # Useful tools to grab samples from the net quickly
    pkgs.wget
    pkgs.curl
    pkgs.httpie      # I'm not good at `curl'. This might help me.

    # Download audio/video samples from the net
    pkgs.youtube-dl
    pkgs.ffmpeg
  ];
}

Create an empty folder and save the snippet above as shell.nix inside it.

Once that’s done, open a terminal, navigate to the folder, and then run nix-shell.

$ python
Python 3.8.9 (default, Apr  2 2021, 11:20:07)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>>
$ jupyter
zsh: command not found: jupyter
$ youtube-dl
zsh: command not found: youtube-dl
$ ls
shell.nix
$ nix-shell
this derivation will be built:
  /nix/store/h3kw8zi4pi6j0p4pnrlfv4qlix0gil0d-python3-3.8.9-env.drv
these 126 paths will be fetched (100.55 MiB download, 512.38 MiB unpacked):
  /nix/store/00cxgc94bssinmw330mkf22kiggmpyhc-python3.8-nbclient-0.5.3
  ...
  /nix/store/zzcilkahrnzk9wh7902zmxgwrsks5bjs-libgcrypt-1.9.3
copying path '/nix/store/wzach25vwapl3sa6f7gylqank3jhhsin-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
...
copying path '/nix/store/f7j87bxvkvs9i6z91nfnrwnd8ngn9s6q-python3.8-jupyter-1.0.0' from 'https://cache.nixos.org'...
building '/nix/store/h3kw8zi4pi6j0p4pnrlfv4qlix0gil0d-python3-3.8.9-env.drv'...
created 642 symlinks in user environment

[nix-shell:~/trash/python-science]$ python
Python 3.8.9 (default, Apr  2 2021, 11:20:07)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.array([1,2,3,4,5])
array([1, 2, 3, 4, 5])
>>>

[nix-shell:~/trash/python-science]$ youtube-dl
Usage: youtube-dl [OPTIONS] URL [URL...]

youtube-dl: error: You must provide at least one URL.
Type youtube-dl --help to see a list of all options.

[nix-shell:~/trash/python-science]$ jupyter notebook
[I 11:07:53.286 NotebookApp] Serving notebooks from local directory: /home/yuki/trash/python-science
[I 11:07:53.287 NotebookApp] Jupyter Notebook 6.3.0 is running at:
[I 11:07:53.287 NotebookApp] http://localhost:8888/?token=f9b1...d05e
[I 11:07:53.287 NotebookApp]  or http://127.0.0.1:8888/?token=f9b1...d05e
[I 11:07:53.287 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 11:07:53.371 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/yuki/.local/share/jupyter/runtime/nbserver-160202-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=f9b1...d05e
     or http://127.0.0.1:8888/?token=f9b1...d05e

Congratulations, you’re now ready to start your data scientist journey.

Seriously, just like that and you’re ready to start doing the things that data scientists do.

With only Nix, you’re able to launch a shell environment with not only Python and additional libraries but also jupyter, ffmpeg, youtube-dl, curl, wget, httpie.

It didn’t ask you to even run with sudo nor nag you about confusing missing library errors, it just works and all you need is Nix and a shell.nix file.

What does `shell.nix` do?

Glancing shell.nix, you might be able to guess what it is doing.

You’re essentially giving Nix a shopping list of what you need in your shell environment.

You might think the syntax looks weird in comparison to a lot of similar package managers and I agree.

That’s because Nix made their own programming language called the Nix expression language.

The purpose of this language are mainly to build software and launch shell environments.

Let’s break it down:

{ pkgs ? import <nixpkgs> {}}:                           # (1)

let
  pythonEnv = pkgs.python38.withPackages (ps: [          # (2)
    # Scientific python essentials
    ps.numpy
    ps.scipy
    ps.pandas
    ps.matplotlib

    # Grab samples from the net via Python
    ps.requests
    ps.beautifulsoup4

    # Need this to make the samples more bearable
    ps.nltk

    # Notebook editing
    ps.jupyter
    ps.ipython
  ]);
in
pkgs.mkShell {                                           # (3)
  packages = [                                           # (4)
    pythonEnv

    pkgs.wget
    pkgs.curl
    pkgs.httpie

    pkgs.youtube-dl
    pkgs.ffmpeg
  ];
}

In this line, we’re importing the Nix package collection (nixpkgs).

Reading it, you might think this is how you define imports but actually, this is how you define lambda functions in Nix.

Your shell.nix file actually define a single function which takes pkgs as an argument but also provide a default value.

This is practically similar to the following example in Python:
```
def shell(pkgs=import_pkgs()):
    ...
```
Nix is a functional programming language so variables are defined inside a let block. These variables will be exposed inside the in block.

We define a variable called pythonEnv which takes the result of the pkgs.python38.withPackages function. This function returns an altered Python package with the Python libraries we’ve defined.

Function calls in Nix doesn’t use parentheses similar to Haskell and Ruby. You simply pass the arguments to it separated by whitespaces.

The argument we pass is actually another function which takes a single argument called ps and returns a list of Python libraries/packages.

You might be confused why it needs to be a function and that’s because this allows multiple Python versions since not all Python versions share the same collection of packages.

You can test this theory yourself: Replace python38 with python39 and run nix-shell.
```
{ pkgs ? import <nixpkgs> {}}:

let
  pythonEnv = pkgs.python39.withPackages (ps: [
    # Scientific python essentials
    ps.numpy
    ...
```
Now that we have pythonEnv nicely defined in the let block, we can use it inside the in block.

Here we’re calling the pkgs.mkShell function which allow us to make a shell environment with Nix.

As you can see from the one marked (2) and this one, functions don’t require the return keyword and will return the enclosed expression.

This is similar to Haskell and Ruby.

In this case, our massive shell.nix function returns the result of pkgs.mkShell.
In here, we simply define the packages the shell environment need to contain.

We pass our pythonEnv package and lists more packages which are non-Python utilities/libraries that are helpful to us.

Reducing duplicate code using the `with` statement

Reading the shell.nix, you might be annoyed by its repetitiveness such as the repeating ps and pkgs scattered everywhere.

The Nix expression language provides you with a helpful tool: the with statement.

Here’s an example of it in use:

{ pkgs ? import <nixpkgs> {}}:

let
  pythonEnv = pkgs.python38.withPackages (ps: with ps; [
    # Scientific python essentials
    numpy
    scipy
    pandas
    matplotlib

    # Grab samples from the net via Python
    requests
    beautifulsoup4

    # Need this to make the samples more bearable
    nltk

    # Notebook editing
    jupyter
    ipython
  ]);
in
pkgs.mkShell {
  packages = with pkgs; [
    pythonEnv

    wget
    curl
    httpie

    youtube-dl
    ffmpeg
  ];
}

As you can see, using the with statement we can omit the pkgs and ps, nice.

What it does is simply pepper in the pkgs. and ps. when it is required.

If you’re still confused, we’ll demonstrate it by taking a part of the previous snippet:

...
in
pkgs.mkShell {
  packages = with pkgs; [
    pythonEnv

    wget
    curl
    httpie

    youtube-dl
    ffmpeg
  ];
}

When Nix sees the with statement, it will add the pkgs prefix when required.

This is how Nix will evaluate the with statement to:

...
in
pkgs.mkShell {
  packages = [
    pythonEnv

    pkgs.wget
    pkgs.curl
    pkgs.httpie

    pkgs.youtube-dl
    pkgs.ffmpeg
  ];
}

Nix simply checks whether each variable exists. If not, it will add the pkgs prefix. If it exists, it will leave them alone.

In fact, we can go even further and remove all mention of pkgs using a single with statement:

{ pkgs ? import <nixpkgs> {}}:

with pkgs;

let
  pythonEnv = python38.withPackages (ps: with ps; [
    # Scientific python essentials
    numpy
    scipy
    pandas
    matplotlib

    # Grab samples from the net via Python
    requests
    beautifulsoup4

    # Need this to make the samples more bearable
    nltk

    # Notebook editing
    jupyter
    ipython
  ]);
in
mkShell {
  packages = [
    pythonEnv

    wget
    curl
    httpie

    youtube-dl
    ffmpeg
  ];
}

Learning more about the Nix expression language

It might be difficult for you to wrap your head around the weird language, especially if you’re not familiar with functional programming.

Don’t worry, you will get used to it soon enough.

To follow this and future articles in the series, you don’t need to master the Nix expression language. This series targeted to those who have 0 knowledge about Nix but have some experience in programming.

However, if you want to learn more about the language itself, check out Nix pills, Nix wiki, and Nix manual.

You can also learn by reading the packages inside the Nix package collection.

You can search via search.nixos.org and click on “Source” on the result:

Where does `nixpkgs` come from?

{ pkgs ? import <nixpkgs> {}}:

Reading the snippet above, you might wonder where nixpkgs is located.

The answer to that is simple: Nix channels.

$ nix-channel --list
nixpkgs https://nixos.org/channels/nixos-21.05

Nix channel is similar to the list of repositories that can be found in other package managers (ex. /etc/apt/sources.list in Debian/Ubuntu).

This is all well and good but there’s you will encounter the following issue: Each Nix user may have different versions nixpkgs present.

As of writing, I’m running version 21.05 of nixpkgs but you might be using a newer version of nixpkgs or maybe even the unstable branch.

This means that your shell environment may break in the future because nixpkgs will be different or it will break on someone else’s machine because they have a different value for nixpkgs.

Ensuring reproducibility by pinning with `niv`

Package managers such as npm solves this issue by introducing dependency pinning.

Essentially, your will have something called a lock file which accompanies your list of dependencies.

The lock file contains the same list of dependencies but in addition, it contains the exact version used when you were defining the dependencies for your project.

This ensures that in the future, you will use the exact same set of dependencies which introduces reproducibility.

With Nix since most of your packages will originate from nixpkgs, you only need to pin nixpkgs itself.

As of writing, Nix does not have this functionality built-in. However, it is available as an experimental feature called flakes.

Because flakes is still experimental, in this article, we will use Niv instead which adds dependency pinning to Nix.

You can install niv easily by running the following command:

$ nix-env -iA nixpkgs.niv

If you don’t want to install and just want to use it temporarily, you can launch a Nix shell with only niv installed:

$ nix-shell -p niv

We will be modifying shell.nix and introduce dependency pinning.

Open a terminal with Niv present and navigate to the folder with the deduplicated version of shell.nix.

First, we run niv init to start the setup process:

$ ls
shell.nix
$ niv init
Initializing
  Creating nix/sources.nix
  Creating nix/sources.json
  Importing 'niv' ...
  Adding package niv
    Writing new sources file
  Done: Adding package niv
  Importing 'nixpkgs' ...
  Adding package nixpkgs
    Writing new sources file
  Done: Adding package nixpkgs
Done: Initializing
$ ls
nix  shell.nix
$ ls nix
sources.json  sources.nix

Niv has started a new Niv project and added both nixpkgs and niv as dependencies.

The nix/sources.nix file is a small vendored library which reads the nix/sources.json. You will see how it is used later.

The dependencies are listed in nix/sources.json and you can use niv show to see what’s inside:

$ niv show
nixpkgs
  homepage:
  url: https://github.com/NixOS/nixpkgs/archive/eb73405ecceb1dc505b7cbbd234f8f94165e2696.tar.gz
  owner: NixOS
  branch: release-20.03
  url_template: https://github.com/<owner>/<repo>/archive/<rev>.tar.gz
  repo: nixpkgs
  type: tarball
  sha256: 06k21wbyhhvq2f1xczszh3c2934p0m02by3l2ixvd6nkwrqklax7
  description: Nix Packages collection
  rev: eb73405ecceb1dc505b7cbbd234f8f94165e2696
niv
  homepage: https://github.com/nmattia/niv
  url: https://github.com/nmattia/niv/archive/e0ca65c81a2d7a4d82a189f1e23a48d59ad42070.tar.gz  owner: nmattia
  branch: master
  url_template: https://github.com/<owner>/<repo>/archive/<rev>.tar.gz
  repo: niv
  type: tarball
  sha256: 1pq9nh1d8nn3xvbdny8fafzw87mj7gsmp6pxkdl65w2g18rmcmzx
  description: Easy dependency management for Nix projects
  rev: e0ca65c81a2d7a4d82a189f1e23a48d59ad42070

As you can see, it pins both nixpkgs and niv to an exact version, see the url and rev which targets a specific commit.

However, if you take a look inside nixpkgs’s branch, you see that it uses the 20.03 version.

That’s very outdated. Let’s fix that and update it to 21.05.

We can update dependencies with niv update:

$ niv update nixpkgs -b release-21.05
Update nixpkgs
Done: Update nixpkgs
$ niv show
nixpkgs
  homepage:
  url: https://github.com/NixOS/nixpkgs/archive/2d6ab6c6b92f7aaf8bc53baba9754b9bfdce56f2.tar.gz
  owner: NixOS
  branch: release-21.05
  url_template: https://github.com/<owner>/<repo>/archive/<rev>.tar.gz
  repo: nixpkgs
  type: tarball
  sha256: 1aafqly1mcqxh0r15mrlsrs4znldhm7cizsmfp3d25lqssay6gjd
  description: Nix Packages collection
  rev: 2d6ab6c6b92f7aaf8bc53baba9754b9bfdce56f2

Wonderful.

However, we are not done yet. shell.nix didn’t import the nixpkgs defined in nix/sources.json. Let’s modify it.

This is how we import nixpkgs from nix/sources.nix:

let
  # Use nixpkgs from Niv
  sources = import ./nix/sources.nix;                     # (1) We import the sources.nix library.
  pkgs = import sources.nixpkgs {};                       # (2) We import nixpkgs from it.

  pythonEnv = pkgs.python38.withPackages (ps: with ps; [  # (3) This needs pkgs to be added explicitly
    ...
  ]);
in
with pkgs;
mkShell {
  ...
}

As you can see the lambda function definition is replaced by defining pkgs as a variable inside the let block.

The nix/sources.nix library is imported and assigned as the sources variable.

nix/sources.nix returns a set of the dependencies as defined inside nix/sources.json.
We take nixpkgs from sources and import it.
We also moved with pkgs; inside the in block.

This is because we can’t put with pkgs; midway in our let block and pkgs is only exposed inside the in block.

Because of this, we readd the pkgs prefix for pythonEnv. Nothing is changed in the mkShell thanks to with pkgs;.

With all of this in place, we can be certain that our shell environment is reproducible even in the future with newer versions of nixpkgs.

Let’s take a test drive:

$ cat shell.nix
let
  # Use nixpkgs from Niv
  sources = import ./nix/sources.nix;  # We import the sources.nix library.
  pkgs = import sources.nixpkgs {};    # We import nixpkgs from it.
...
$ nix-shell
this derivation will be built:
  /nix/store/mc3lh57blikblbrj19m17cyh5c5qaprp-python3-3.8.9-env.drv
these 239 paths will be fetched (156.50 MiB download, 857.66 MiB unpacked):
  /nix/store/04b8w8yy6qzvdbrn3rsqkc9splcvs0cj-python3.8-kiwisolver-1.3.1
  ...
  /nix/store/zzv3axnas822sm1h05y0r9djnk73m4kj-libICE-1.0.10
copying path '/nix/store/i7ra1fy8gxcmvpzzxihy25hd3jspzvpw-curl-7.76.1-man' from 'https://cache.nixos.org'...
...
copying path '/nix/store/rcbkp2r2ifnll4b8hxjwd5yzxr9dvvk0-python3.8-jupyter-1.0.0' from 'https://cache.nixos.org'...
building '/nix/store/mc3lh57blikblbrj19m17cyh5c5qaprp-python3-3.8.9-env.drv'...
created 644 symlinks in user environment

[nix-shell:~/trash/python-science]$ python
Python 3.8.9 (default, Apr  2 2021, 11:20:07)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>>

[nix-shell:~/trash/python-science]$ jupyter --version
jupyter core     : 4.7.1
jupyter-notebook : 6.3.0
qtconsole        : 5.0.3
ipython          : 7.21.0
ipykernel        : 5.5.0
jupyter client   : 6.1.12
jupyter lab      : not installed
nbconvert        : 6.0.7
ipywidgets       : 7.6.3
nbformat         : 5.1.2
traitlets        : 5.0.5

Awesome, it’s up and running again!

You can even downgrade to an older version of nixpkgs with niv update:

$ niv update nixpkgs -b release-20.09
Update nixpkgs
Done: Update nixpkgs
$ nix-shell
these 2 derivations will be built:
  /nix/store/7zw4ijckcacgij9jqfa9wjinh7qa7b7v-builder.pl.drv
  /nix/store/6lmimxdpdgxr6wsa3y2fpy2n79wvmg81-python3-3.8.8-env.drv
these 344 paths will be fetched (266.92 MiB download, 1345.74 MiB unpacked):
  /nix/store/021jdajrghabnh8p6rvqidgr04rjahrz-gdk-pixbuf-2.40.0
  ...
  /nix/store/zxp8i14r9ngq8hmbbb9h4x4mhmxiyvy2-python3.8-nbformat-5.0.7
copying path '/nix/store/ma08l3jqjhx8b4400x1bb5kvzjb500dp-python3.8-pyOpenSSL-19.1.0' from 'https://hydra.iohk.io'...
...
copying path '/nix/store/rj663ln3skd12irz0z0sx133p065v4a8-python3.8-jupyter-1.0.0' from 'https://cache.nixos.org'...
building '/nix/store/6lmimxdpdgxr6wsa3y2fpy2n79wvmg81-python3-3.8.8-env.drv'...
created 640 symlinks in user environment

[nix-shell:~/trash/python-science]$ python
Python 3.8.9 (default, Apr  2 2021, 11:20:07)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'nltk'
>>>

[nix-shell:~/trash/python-science]$ jupyter --version
The program 'jupyter' is not in your PATH. It is provided by several packages.
You can make it available in an ephemeral shell by typing one of the following:
  nix-shell -p ihaskell
  nix-shell -p python38Packages.jupyter_core
  nix-shell -p python39Packages.jupyter_core

Wait, it failed?

Also, it’s Python 3.8.9, not 3.8.8.

Oh, it’s because the packages argument in pkgs.mkShell was introduced in nixpkgs version 21.05 and isn’t present in version 20.09.

Let’s fix that by replacing the packages argument in pkgs.mkShell to buildInputs.

let
  ...
in
with pkgs;
mkShell {
  # We change from `packages' to `buildInputs'.
  # Don't ask why it was called that for `pkgs.mkShell'.
  buildInputs = [
    pythonEnv
    ...
  ];
}

Let’s test it out:

$ nix-shell
these 12 paths will be fetched (1.63 MiB download, 7.00 MiB unpacked):
  /nix/store/3354dzy157x0kijm6pv5khzfx6hf5h8n-nghttp2-1.41.0-dev
  ...
  /nix/store/yi6x0zkhyl6g516c97q1cazpfvv11j4a-curl-7.74.0-dev
copying path '/nix/store/is2d41x8qkx4gd27i8r45q45kv0dynbn-curl-7.74.0-man' from 'https://hydra.iohk.io'...
...
copying path '/nix/store/yi6x0zkhyl6g516c97q1cazpfvv11j4a-curl-7.74.0-dev' from 'https://hydra.iohk.io'...

[nix-shell:~/trash/python-science]$ python
Python 3.8.8 (default, Feb 19 2021, 11:04:50)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>>

[nix-shell:~/trash/python-science]$ jupyter
usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir] [--paths] [--json] [subcommand]
jupyter: error: one of the arguments --version subcommand --config-dir --data-dir --runtime-dir --paths is required

[nix-shell:~/trash/python-science]$ jupyter --version
jupyter core     : 4.6.3
jupyter-notebook : 6.1.4
qtconsole        : 4.7.6
ipython          : 7.17.0
ipykernel        : 5.2.1
jupyter client   : 6.1.7
jupyter lab      : not installed
nbconvert        : 5.6.1
ipywidgets       : 7.5.1
nbformat         : 5.0.7
traitlets        : 4.3.3

Cool, it works.

As you can see, it’s quite important to pin your dependencies in order to ensure reproducibility.

You would rather not have things broken at random simply because of factors that you are not aware of.

Also, people can use the shell environment without having Niv installed on their machine. They only require to have Nix installed on their machine.

This is thanks to the vendored library which allows Nix to read the nix/sources.json file.

Closing

This is the second article in my series about Nix. Hurrah!

As you can see, this series aims for a more practical approach which can serve as references to people who are new to Nix.

Or to put it in another way, this link of this article will be copy pasted by me everytime someone ask ‘how do I XYZ with Nix?” ;)

Let me know what you think so far, my contacts are available in yukiisbo.red (Discord is rather inactive, though).

Thanks for reading and I hope you’ll be there on the next one!

Remarks

I should’ve done this much earlier but I never thought about doing it until now. Sorry about that.

I would like to thank to those who has reviewed and given me their input during the writing of this article:

Ben Siraphob (@siraben at GitHub)

Mic Check

Sat, 17 Dec 2022 00:00:00 GMT

Test, test, hello, is this thing still on?

I’ve recently made a big shift on this website by moving from org-mode to Gatsby and merge it with yukiisbo.red.

Why move away from org-mode?

While I still prefer org-mode to write plain-text documents, its ecosystem for creating a complete static website (i.e. not just a single page HTML document) to be very lacking and require additional engineering work.

I’ve made one before using org-publish and following a write-up by Toon Claes for his blog writepermission.com. It works well for the most part and I used that for a while.

However, while it was nice that my blog reflects my Emacs configuration but it has lead to a mountain of annoyances which results in requiring a mountain of hacks to tame this dynamic.

In addition, embedding content other than text with this setup is painful which led me to barely use mediums other than text in my posts.

While this whole thing isn’t an excuse for me to be on a long hiatus in terms of writing, it left a nagging feeling in my mind for a long time and became one of the main deterrents when writing.

What led me to move away from org-mode to Gatsby?

Recently, I started working with the Nix Documentation Team on nix.dev which uses the Sphinx static site generator. I’m not a stranger to it as I worked with it when I was working in the coala project. While it’s not perfect, I felt more productive using it.

It led me to consider dropping org-mode for my website. I was considering ablog which is an extension for Sphinx to make it for blogging. That felt like a hack so I put it at the back burner.

Then, I got reminded of Gatsby which is a static site generator that uses React.

Why Gatsby?

I typically avoid client-side frameworks (CSF) as they tend to bloat the website with a mountain of unnecessary JavaScript and render the website unusable for people who doesn’t use JavaScript using extensions like NoScript or “limited” web browsers like NetSurf and Lynx.

However, it seems a lot of things has changed in the frontend world. Single Page Application (SPA), where JavaScript is necessary to render entire web pages client-side, has stopped becoming the only thing available when using CSFs to create entire websites.

I have heard of Server Side Rendering (SSR) before but I tend to not use them as they used to be really complicated. Even then, the idea of running a server that does computation to build web pages on the fly from JavaScript code is overkill. Especially when those pages would’ve been static HTML documents if it wasn’t for CSFs.

However, today, Static Site Generation (SSG) with CSFs has become a lot more accessible, thanks to tools like Gatsby and SvelteKit.

This approach essentially boils your components written in JavaScript with CSFs to static HTML first and then introduces bits of JavaScript on top of them.

Therefore, it makes JavaScript not a requirement for visitors while still allowing you to still use the rich ecosystem of CSFs.

Before you had to use tools like Jekyll, Hugo, and others which I never liked mainly because they tend to force your hand to follow their pre-built structures or use a [terrible template language].

To be fair, I dislike every single template language (ex. Jinja2 and m4) because they essentially work like a parasite whose terrible language embeds itself inside your documents (HTML markup or even source code). In addition, they’re unaware of what they’re actually manipulating other than they’re just plain text files, therefore unaware of details in your document like semantics.

I personally prefer an approach with domain-specific languages (DSL) where you represent your problem using a purpose-built language for the domain. You can see this in action with tools such as Dhall, Lucid, Laravel’s Query Builder.

Naturally, I like JSX which is an HTML embedded domain-specific language that boils down into JavaScript expressions that generates HTML DOM. This is possible thanks to Babel using a transformer.

Because at the end of the day it’s just JavaScript extensions, you can reuse the capabilities of the JavaScript language and libraries to manipulate DOMs.

In addition, the TypeScript version (TSX) introduces types to JSX (and HTML) which makes it really pleasant to use.

In conclusion, I use Gatsby because it allows me to use the React ecosystem while still making the site usable to those who prefer to not run JavaScript.

Why JavaScript?

A lot of people will disagree with my choice as they simply dislike/hate JavaScript and refuse to use anything from its ecosystem.

Recently, I’ve grown to dislike dogmas (ex. JavaScript sucks and harmful).

By their nature, they blanket over a large scope without actually giving any meaningful thought other than writing long winded rant-ridden opinionated essays on “why $THING sucks and harmful”.

I believe it’s much more productive to make decisions for problems based on research that is relevant to the problem at hand and realize that ultimately every single decision is a compromise and there’s no such thing as a “correct” decision. The pursuit of perfection is a fool’s errand.

If you’d like an example when dogmas lead to bad decisions, I recommend reading Robert Nystrom’s Considering GOTO Harmful and watching Tantacrul’s Music Software & Interface Design: Steinberg’s Dorico.

Ultimately, I chose Gatsby because my main focus for this project is making written content for y’all to read.

And for my particular use case and preferences, I believe it allows me to spend more time writing than everything else related to running a website like this.

The fact that I can reuse the rich ecosystem that surrounds the React framework allows me to write less code and more posts.

Is Gatsby perfect?

Hell no.

It took me a lot of time to convert my org blog posts into MDX.

It doesn’t help that I kept stumbling upon arcane unhelpful error messages, such as:

$ npm run develop
...
success onPreExtractQueries - 0.002s
success extract queries from components - 1.869s
success write out requires - 0.007s

 ERROR #gatsby-plugin-mdx_10001  PLUGIN

Failed to compile the file "/home/yuki/src/git.yukiisbo.red/yuki/nextgen/notes/mic-check/index.mdx". Original error message:

Could not parse expression with acorn: Unexpected token

 ERROR  UNKNOWN

Module build failed (from ./node_modules/gatsby/dist/utils/babel-loader.js):
SyntaxError: /home/yuki/src/git.yukiisbo.red/yuki/nextgen/notes/mic-check/index.mdx: Invalid left-hand side in prefix operation. (1:2)

> 1 | ---
    |   ^
  2 | title: "Mic Check"
  3 | date: "2022-12-17"
  4 | slug: "mic-check"
...
not finished Building development bundle - 2.233s

It doesn’t indicate which line resulted in the error nor does it provide any helpful indication.

It throws this whenever MDX is unable to parse your “Markdown” file (despite them being valid Markdown, so this issue is mainly because their parser is bad).

It really feels like we’ve reverted back to the 80s with Commodore BASIC where syntax errors just results in a single cryptic message: SYNTAX ERROR.

Also, I learned the hard way that when gatsby-config.ts results in an error, it will spam errors that are way more unhelpful because they’re unrelated to the actual issue.

$ npm run develop
...
success source and transform nodes - 0.071s
success building schema - 0.219s
success createPages - 0.005s

 ERROR #85923  GRAPHQL.VALIDATION

There was an error in your GraphQL query:

Cannot query field "allMdx" on type "Query".

If you don't expect "allMdx" to exist on the type "Query" it is most likely a typo. However, if you expect "allMdx" to exist there are a couple of solutions to common problems:

- If you added a new data source and/or changed something inside gatsby-node/gatsby-config, please try a restart of your development server.
- You want to optionally use your field "allMdx" and right now it is not used anywhere.

It is recommended to explicitly type your GraphQL schema if you want to use optional fields.

File: node_modules/gatsby-plugin-page-creator/create-pages-from-collection-builder.js:45:13

See our docs page for more info on this error: https://gatsby.dev/creating-type-definitions


 ERROR #gatsby-plugin-page-creator_12103  PLUGIN

PageCreator: Tried to create pages from the collection builder.
Unfortunately, the query came back empty. There may be an error in your query:

Cannot query field "allMdx" on type "Query".

File: src/pages/notes/{mdx.frontmatter__slug}.tsx
...

In addition, console.log doesn’t work inside gatsby-config.ts so good luck!

While this was an absolute pain in the ass, now that everything is setup, the end result is quite pleasant.

Why the merge with yukiisbo.red?

It was quite painful to maintain two different websites uses different tools to maintain.

At the same time, having two websites on two domains feels dumb, especially when I typically “promote” one over the other.

Therefore, I felt merging them both under one repository and one domain to be the way to go.

Now that it’s been more than a year since I stopped using social medias like Twitter and Mastodon, this website is pretty much the only public space that is my own where I express myself and I’d like to keep it that way.

What’s next?

I’m really hoping to write more in the near future.

Currently, on my spare time, I’m busy working on documentation with the Nix Documentation Team and my main focus for now is on the nix.dev website.

Other than that, my job title and responsibility changed in my workplace. I’m now the Chief Technical Officer and I need to get used to that title or find an alternative title that isn’t grand.

Anyway, that’s all for now. See you next time. Hopefully.

Spice up with Nix: The Functional Software Deployment Model

Wed, 26 Jan 2022 00:00:00 GMT

Unlike traditional package managers, Nix is a package manager which doesn’t directly modify your system.

Software which are managed by Nix are stored in a component store located in /nix/store/, instead of modifying /usr/.

But what is a component, exactly? Let’s take a peek inside the component store.

$ ls -l /nix/store | head
total 112536
-r--r--r--  1 root root     1542 janv.  1  1970 001gp43bjqzx60cg345n2slzg7131za8-nix-nss-open-files.patch
-r--r--r--  1 root root       80 janv.  1  1970 003y7nllp3cxm5ww1q6xmygi81hhzgzn-dap-mode-recipe
-r--r--r--  1 root root     6619 janv.  1  1970 006s9a9y6iiqmsv7wm2ncm3m5lwhkv3i-libgccjit-10.3.0.drv
-r--r--r--  1 root root     3853 janv.  1  1970 007nl7dl9inal3xkca396ssxs416ggrd-python3.8-networkx-2.6.3.drv
-r--r--r--  1 root root     1670 janv.  1  1970 00nkk559kqmp7wcyq4vcvxhhk1rcjind-net-tools-2.10.drv
-r--r--r--  1 root root      437 janv.  1  1970 00qr10y7z2fcvrp9b2m46710nkjvj55z-update-autotools-gnu-config-scripts.sh
-r--r--r--  1 root root     2785 janv.  1  1970 00xlf60cvvppcn845dk30ra5kmmihspb-sqlite-simple-0.4.18.0.tar.gz.drv
-r--r--r--  1 root root    10225 janv.  1  1970 011y6iy6lh7qdwxwmm6yg3nslbxkcbkr-tdigest-0.2.1.1.drv
-r--r--r--  1 root root     1747 janv.  1  1970 0140ckl854jk2lnr65khhxjv3vwcvycj-docbook-xml-4.2.drv

Inside the component store, you’ll encounter a bunch of files and folders. You start to question, “what’s up with the hashes?”.

Before that, let’s see the contents of a random component.

In the Nix store, you find a folder named with something that you recognize.

$ ls /nix/store/ | grep 'python3-3.9.6$'
5bh6rpya1ar6l49vrhx1rg58dsa42906-python3-3.9.6
$ ls -l /nix/store/5bh6rpya1ar6l49vrhx1rg58dsa42906-python3-3.9.6
total 20
dr-xr-xr-x 2 root root 4096 janv.  1  1970 bin
dr-xr-xr-x 3 root root 4096 janv.  1  1970 include
dr-xr-xr-x 4 root root 4096 janv.  1  1970 lib
dr-xr-xr-x 2 root root 4096 janv.  1  1970 nix-support
dr-xr-xr-x 4 root root 4096 janv.  1  1970 share
$ ls -l /nix/store/5bh6rpya1ar6l49vrhx1rg58dsa42906-python3-3.9.6/bin
total 36
lrwxrwxrwx 1 root root     8 janv.  1  1970 2to3 -> 2to3-3.9
-r-xr-xr-x 1 root root   148 janv.  1  1970 2to3-3.9
lrwxrwxrwx 1 root root     7 janv.  1  1970 idle -> idle3.9
lrwxrwxrwx 1 root root     7 janv.  1  1970 idle3 -> idle3.9
-r-xr-xr-x 1 root root   146 janv.  1  1970 idle3.9
lrwxrwxrwx 1 root root     8 janv.  1  1970 pydoc -> pydoc3.9
lrwxrwxrwx 1 root root     8 janv.  1  1970 pydoc3 -> pydoc3.9
-r-xr-xr-x 1 root root   131 janv.  1  1970 pydoc3.9
lrwxrwxrwx 1 root root     9 janv.  1  1970 python -> python3.9
lrwxrwxrwx 1 root root     9 janv.  1  1970 python3 -> python3.9
-r-xr-xr-x 1 root root 16392 janv.  1  1970 python3.9
-r-xr-xr-x 1 root root  3290 janv.  1  1970 python3.9-config
lrwxrwxrwx 1 root root    16 janv.  1  1970 python3-config -> python3.9-config
lrwxrwxrwx 1 root root    16 janv.  1  1970 python-config -> python3.9-config

Wait, this looks similar to a prefix. It has similar structure to /usr/ but it only contains Python 3.9.6, nothing else.

If you check the libraries of the Python binary, it simply refers to library files stored inside other components.

$ ldd /nix/store/5bh6rpya1ar6l49vrhx1rg58dsa42906-python3-3.9.6/bin/python3.9
    linux-vdso.so.1 (0x00007ffce636f000)
    libpython3.9.so.1.0 => /nix/store/5bh6rpya1ar6l49vrhx1rg58dsa42906-python3-3.9.6/lib/libpython3.9.so.1.0 (0x00007fb1b3fe9000)
    libpthread.so.0 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libpthread.so.0 (0x00007fb1b3fc9000)
    libdl.so.2 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libdl.so.2 (0x00007fb1b3fc4000)
    libcrypt.so.1 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libcrypt.so.1 (0x00007fb1b3f8a000)
    libncursesw.so.6 => /nix/store/72vdginhi637lxyii5fnpmvf4jzc4gbk-ncurses-6.2/lib/libncursesw.so.6 (0x00007fb1b3f18000)
    libutil.so.1 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libutil.so.1 (0x00007fb1b3f11000)
    libm.so.6 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libm.so.6 (0x00007fb1b3dd0000)
    libgcc_s.so.1 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libgcc_s.so.1 (0x00007fb1b3db6000)
    libc.so.6 => /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libc.so.6 (0x00007fb1b3bf1000)
    /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fb1b43cb000)

So, with the massive hype around Nix, it’s just another package manager which installs each software into their own prefixes like Homebrew?

Well, not really.

The one simple trick

Let’s go back to the previous question: why are hashes used as a prefix?

Do you recall the issues with the simple Python script we made?

The problem with different systems having software installed in different places.
The problem with different versions or variants of the same software.

Nix solves the issues mentioned above by storing components in a deterministic path which are prefixed with a cryptographic hash.

The hash itself is generated based on the inputs (i.e source code, build flags, dependencies) used to build the component.

If any of the inputs changed (ex. one of its dependencies was updated), the hash will be different which will result in a different path.

This simple trick allows for multiple variants of the same software to be installed along with their dependencies being exact and deterministic to the ones used when they’re built.

Getting your hands dirty with Nix

Follow me!

You can follow along the demonstrations shown in the article.

If you’re running any Linux distribution on a 64-bit Intel/AMD machine, you’ll see that the paths shown in the demonstrations will be the same.

If you’re using another operating system and/or another architecture, the paths will be different. However, you can still follow along!

Have fun!

To demonstrate the deterministic and exact nature of Nix, let’s package a piece of software with Nix.

We could package GNU Hello like everyone else but I find it rather overused and doesn’t demonstrate Nix’s strengths.

Personally, I’m in store for something complex that would make package maintainers cry.

So, I made a piece of software called “YouTube Launcher” which provides a simple user interface to watch YouTube videos without a web browser!

#!/usr/bin/env python

from urllib.parse import urlparse

from PyQt5.QtWidgets import QApplication, QWidget, QVBoxLayout, QLineEdit, QPushButton, QMessageBox
from PyQt5.QtCore import QProcess


app = QApplication([])


class Youtube(QWidget):
    def __init__(self):
        super().__init__()

        layout = QVBoxLayout()

        self.textbox = QLineEdit('https://www.youtube.com/watch?v=fjuJgqrZSIk')
        layout.addWidget(self.textbox)

        button = QPushButton('Watch')
        button.clicked.connect(self.__watch)
        layout.addWidget(button)

        self.setLayout(layout)

        self.setWindowTitle("YouTube Launcher");
        self.resize(512, 100)

        self.show()

    def __watch(self):
        url = urlparse(self.textbox.text())

        if (url.netloc.lower() in ('www.youtube.com', 'youtube.com') and
            url.path == '/watch' and url.query.startswith('v=') and
            len(url.query) > 2):
            QProcess().startDetached('mpv', [url.geturl()])
            app.exit()
        else:
            msg = QMessageBox()
            msg.setIcon(QMessageBox.Critical)
            msg.setText("The URL is not a valid YouTube video URL")
            msg.setWindowTitle("Error")
            msg.setStandardButtons(QMessageBox.Ok)
            msg.exec()


def main():
    window = Youtube()
    app.exec()


if __name__ == '__main__':
    main()

You don’t need to save the snippet above, it shown here only for reference. The entire source code is available at git.yukiisbo.red/yuki/youtube-launcher.

So, what are the pain points of this program?

It is written in Python 3.
It uses Qt 5 via the PyQt5 library.
It executes mpv which requires YouTube support to be enabled.

Much more complex than our previous Python script.

So, here’s the Nix file to build and package the software:

let
  pkgs = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/60cce7e5e1fdf62421ef6d4184ee399b46209366.tar.gz";
    sha256 = "100xrb925cana1kfd0c7gwkjjalq891vfgr0rn1gl9j8gp3l3gx6";
  }) {};

  mpv = pkgs.wrapMpv pkgs.mpv-unwrapped { youtubeSupport = true; };
in
pkgs.python38Packages.buildPythonApplication {
  pname = "youtube-launcher";
  version = "0.1";

  src = pkgs.fetchFromGitHub {
    owner = "yukiisbored";
    repo = "youtube-launcher";
    rev = "04322fbe3f3c1d9a7c620147c2380cd770cb505a";
    sha256 = "043sj6p47s8jk9sj9qnbshzjvxix1pfwzkpwbc5f9dqk5yxgnq13";
  };

  buildInputs = [ mpv ];
  propagatedBuildInputs = with pkgs.python38Packages; [ pyqt5 ];
  nativeBuildInputs = with pkgs.libsForQt5.qt5; [ qtbase wrapQtAppsHook ];

  postPatch = ''
    substituteInPlace youtube_launcher.py \
      --replace "'mpv'" "'${mpv}/bin/mpv'"
  '';

  preFixup = ''
    makeWrapperArgs+=("''${qtWrapperArgs[@]}")
  '';
}

To build, create an empty folder and save the snippet above as default.nix inside it.

Now, run nix-build to build the package.

$ mkdir -p experiments/youtube
$ cd experiments/youtube
$ ls
$ nano default.nix
$ nix-build .
these 4 derivations will be built:
  /nix/store/vv23rn1fqfx2b7sb6n8z3qpqi9r1m5vw-builder.pl.drv
  ...
  /nix/store/5ia4ma9b96kg620k7k0w9xhgqajd828n-youtube-launcher-0.1.drv
these 335 paths will be fetched (204.35 MiB download, 984.31 MiB unpacked):
  /nix/store/03bbr1dba98kzmx364yv4dwhvssc1g1w-libxkbcommon-1.3.0
  ...
  /nix/store/zpbhr2v6x7dijfvw3w9xpbrvhp9z6gzw-hook
copying path '/nix/store/6c5x1d03c5323lsbr2il8w1gwf83cqjj-python-remove-bin-bytecode-hook' from 'https://hydra.iohk.io'...
...
copying path '/nix/store/an3gwsbssydlcwlafznv0lisq7c8hy5p-mpv-0.33.1' from 'https://cache.nixos.org'...
building '/nix/store/vv23rn1fqfx2b7sb6n8z3qpqi9r1m5vw-builder.pl.drv'...
building '/nix/store/sd5x742zvdhrm0ysx4xivyaf51qm98w7-lua-5.2.4-env.drv'...
created 12 symlinks in user environment
building '/nix/store/7imvh7yyv754jjvaqd7cgvd196xcixsj-mpv-with-scripts-0.33.1.drv'...
building '/nix/store/5ia4ma9b96kg620k7k0w9xhgqajd828n-youtube-launcher-0.1.drv'...
...
/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1
$

It’s built and the resulting component is stored inside /nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1 as indicated by the last line of nix-build.

Now, you can run the software by running ./result/bin/youtube-launcher inside the same folder as default.nix.

$ ls -l
total 8
-rw-rw-r-- 1 yuki yuki 1001 janv. 24 13:09 default.nix
lrwxrwxrwx 1 yuki yuki   64 janv. 24 13:11 result -> /nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1
$ ./result/bin/youtube-launcher

The OpenGL problem on non-NixOS

As of writing, if you’re not running NixOS or macOS (untested), mpv will fallback to software rendering as it fails to use any hardware acceleration which results in poor performance.

This is not entirely the fault of Nix / nixpkgs but rather due to how hardware accelerated graphics works in Linux and other *nix which uses Linux DRI/Mesa architecture.

The issue is currently tracked as #62177 in nixpkgs. There are hacks to workaround this issue, most notably the nixGL wrapper.

If you’d like to learn more about how hardware accelerated graphics work, I recommend the Wikipedia article on DRI.

If you want to get your hands dirty, I challenge you to write a program which obtains an OpenGL/Vulkan context and render a cube without X11/GLX nor a Wayland compositor. Have fun! ;-)

Your browser does not support the video tag.

It works and you can watch YouTube videos with it by copy-pasting a YouTube URL into the text box and click the “Watch” button. Awesome.

But, why does it work?

This is a rather tame Python script which does a lot of bad practices:

It assumes python is Python 3.
It assumes mpv is in PATH and is capable of playing YouTube videos.
It relies on Qt which requires native libraries.

Let’s take a peek at the binary.

$ cat ./result/bin/youtube-launcher
#! /nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/4s0h5aawbap3xhldxhcijvl26751qrjr-python3-3.8.9/bin:/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1/bin:/nix/store/kv3wcr31ba5r2hglaz5d73mg084ld820-qtdeclarative-5.15.2-bin/bin:/nix/store/1p93bdjv1h6dqc7lbykj54mp7d64vl4c-python3.8-PyQt5-5.15.2/bin'${PATH:+':'}$PATH
export PYTHONNOUSERSITE='true'
export QT_PLUGIN_PATH='/nix/store/cgpka5n7m8l31i6j9f5b071l8alqzq6v-qtbase-5.15.2-bin/lib/qt-5.15.2/plugins'${QT_PLUGIN_PATH:+':'}$QT_PLUGIN_PATH
export QT_PLUGIN_PATH='/nix/store/k5f31g4g5s7agb7yahlsi4w7jz15516x-qtsvg-5.15.2-bin/lib/qt-5.15.2/plugins'${QT_PLUGIN_PATH:+':'}$QT_PLUGIN_PATH
export QT_PLUGIN_PATH='/nix/store/kv3wcr31ba5r2hglaz5d73mg084ld820-qtdeclarative-5.15.2-bin/lib/qt-5.15.2/plugins'${QT_PLUGIN_PATH:+':'}$QT_PLUGIN_PATH
export QML2_IMPORT_PATH='/nix/store/kv3wcr31ba5r2hglaz5d73mg084ld820-qtdeclarative-5.15.2-bin/lib/qt-5.15.2/qml'${QML2_IMPORT_PATH:+':'}$QML2_IMPORT_PATH
export QML2_IMPORT_PATH='/nix/store/9r1hhbk3r9jv9dcnd3z59jyjs1k08r8p-qtquickcontrols-5.15.2/lib/qt-5.15.2/qml'${QML2_IMPORT_PATH:+':'}$QML2_IMPORT_PATH
export QT_PLUGIN_PATH='/nix/store/psvnd1d97brqwgw9c8qhw4l85nlvpn6y-qtwayland-5.15.2-bin/lib/qt-5.15.2/plugins'${QT_PLUGIN_PATH:+':'}$QT_PLUGIN_PATH
export QML2_IMPORT_PATH='/nix/store/psvnd1d97brqwgw9c8qhw4l85nlvpn6y-qtwayland-5.15.2-bin/lib/qt-5.15.2/qml'${QML2_IMPORT_PATH:+':'}$QML2_IMPORT_PATH
exec -a "$0" "/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1/bin/.youtube-launcher-wrapped"  "$@"

This is a wrapper which Nix generated to run the software we built.

Everything in here uses the deterministic path generated by Nix which has the characteristic cryptographic hash prefix.

As you can see, it defines various environment variables required to run the application such as PATH which includes the exact version of Python required along with a bunch of other variables to make Qt applications work.

That last exec line executes our Python script, right?

$ cat /nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1/bin/.youtube-launcher-wrapped
#!/nix/store/4s0h5aawbap3xhldxhcijvl26751qrjr-python3-3.8.9/bin/python3.8
# -*- coding: utf-8 -*-
import sys;import site;import functools;sys.argv[0] = '/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1/bin/youtube-launcher';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1/lib/python3.8/site-packages','/nix/store/qldagg1vs1y4j36jnchbhqihjj68q9qd-python3.8-dbus-python-1.2.16/lib/python3.8/site-packages','/nix/store/wp81ms43wsvrc3qxnfx1bz3ksdyc756y-python3.8-PyQt5_sip-12.8.1/lib/python3.8/site-packages','/nix/store/1p93bdjv1h6dqc7lbykj54mp7d64vl4c-python3.8-PyQt5-5.15.2/lib/python3.8/site-packages'], site._init_pathinfo());
import re
import sys
from youtube_launcher import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Nope, it’s another wrapper written in Python generated by Nix.

The shebang is now deterministic to the exact version of Python used for this program.

The line with import sys; import site; defines the entire Python site path.

If you are unfamiliar, in Python, the site path defines a list of paths where it can find Python libraries, similar to the PATH environment variable.

With that in mind, the wrapper adds a bunch of path determistic paths to the runtime dependencies.

In the end, it imported our main function and execute it.

So, where is our script?

Let’s look deeper.

$ cat ./result/lib/python3.8/site-packages/youtube_launcher.py
#!/usr/bin/env python

from urllib.parse import urlparse

from PyQt5.QtWidgets import QApplication, QWidget, QVBoxLayout, QLineEdit, QPushButton, QMessageBox
from PyQt5.QtCore import QProcess


app = QApplication([])
...

There it is!

Reading the source code, it’s a bit different.

...
QProcess().startDetached('/nix/store/p7csiywv541jnvrgah93p7zjvq8lkwkq-mpv-with-scripts-0.33.1/bin/mpv', [url.geturl()])
...

The line which executes mpv was changed to use the deterministic path for the mpv binary.

While we’re here, we can even ask Nix to tell us what are the direct runtime dependencies of the component:

$ nix-store --query --references ./result
/nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23
/nix/store/4s0h5aawbap3xhldxhcijvl26751qrjr-python3-3.8.9
/nix/store/1p93bdjv1h6dqc7lbykj54mp7d64vl4c-python3.8-PyQt5-5.15.2
/nix/store/9r1hhbk3r9jv9dcnd3z59jyjs1k08r8p-qtquickcontrols-5.15.2
/nix/store/cgpka5n7m8l31i6j9f5b071l8alqzq6v-qtbase-5.15.2-bin
/nix/store/k5f31g4g5s7agb7yahlsi4w7jz15516x-qtsvg-5.15.2-bin
/nix/store/kv3wcr31ba5r2hglaz5d73mg084ld820-qtdeclarative-5.15.2-bin
/nix/store/wp81ms43wsvrc3qxnfx1bz3ksdyc756y-python3.8-PyQt5_sip-12.8.1
/nix/store/cvwc3r4dcjjsa6vb50idlc4q171whvgl-python3.8-PyQt5-5.15.2-dev
/nix/store/p7csiywv541jnvrgah93p7zjvq8lkwkq-mpv-with-scripts-0.33.1
/nix/store/psvnd1d97brqwgw9c8qhw4l85nlvpn6y-qtwayland-5.15.2-bin
/nix/store/qldagg1vs1y4j36jnchbhqihjj68q9qd-python3.8-dbus-python-1.2.16
/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1

So, as demonstrated, the resulting built component uses deterministic paths for its runtime dependencies.

However, Nix isn’t the one doing the paths replacement and wrapping on its own.

Instead, it is our Nix file which defines the build process that does all of the legwork to make this happen.

let
  ...
in
pkgs.python38Packages.buildPythonApplication {
  pname = "youtube-launcher";
  version = "0.1";
  ...
}

The Python wrapper is done by buildPythonApplication function from the Nix package collection (nixpkgs) which we used to build our package.

let
  ...
in
pkgs.python38Packages.buildPythonApplication {
  ...
  nativeBuildInputs = with pkgs.libsForQt5.qt5; [ ... wrapQtAppsHook ];
  ...
  preFixup = ''
    makeWrapperArgs+=("''${qtWrapperArgs[@]}")
  '';
}

The Qt 5 wrapper is done by wrapQtAppsHook component from nixpkgs which we included as a dependency.

Because we’re using multiple wrappers, we need to add the Qt wrapper during the preFixup phase.

Now, what about mpv?

let
  ...
  mpv = pkgs.wrapMpv pkgs.mpv-unwrapped { youtubeSupport = true; };
in
pkgs.python38Packages.buildPythonApplication {
  ...
  buildInputs = [ mpv ];
  ...
  postPatch = ''
    substituteInPlace youtube_launcher.py \
      --replace "'mpv'" "'${mpv}/bin/mpv'"
  '';
  ...
}

First, we define a variable inside let to determine which exact version of mpv is required since we need YouTube support to be enabled.

Then, we define it as a dependency for our package inside buildInputs.

In the postPatch phase, we use subtituteInPlace to replace instances of 'mpv' inside the youtube_launcher.py script with '${mpv}/bin/mpv'

In the Nix expression language, ${mpv} is string interpolation. In this case, it will be changed to the determinstic path of the mpv component.

And there you go, with that one simple trick of using hashes, Nix ensures dependencies are deterministic on runtime.

Making sure the binary builds of your software uses the same exact set of binaries when it was built.

Still don’t believe me?

Let’s update our package to use Python 3.9 and see what happens.

With default.nix opened, replace all instances of python38 with python39 and then build with nix-build.

$ sed -i.orig 's/python38/python39/g' default.nix
$ cat default.nix
let
  ...
in
pkgs.python39Packages.buildPythonApplication {
  ...
  propagatedBuildInputs = with pkgs.python39Packages; [ pyqt5 ];
  ...
}
$ nix-build
this derivation will be built:
  /nix/store/p33jvlyqgfw3pg9yg16cygd4dczla0ap-youtube-launcher-0.1.drv
these 18 paths will be fetched (17.11 MiB download, 93.61 MiB unpacked):
  /nix/store/44l3f1rsghmympim9jbffgfxdcshiz5f-python3.9-pip-21.0.1
  ...
  /nix/store/y4j4k0l6w941wriprxz13dhvz896lw3m-tzdata-2020f
copying path '/nix/store/76rqrmaxqb7pmrgd9zwbap983aarjidr-python-remove-tests-dir-hook' from 'https://cache.nixos.org'...
...
copying path '/nix/store/dkmbkkax4vqbmc6y1dqqhg9xi9yc23wj-python3.9-PyQt5-5.15.2-dev' from 'https://cache.nixos.org'...
building '/nix/store/p33jvlyqgfw3pg9yg16cygd4dczla0ap-youtube-launcher-0.1.drv'...
...
/nix/store/prrc01g09l1pma21qndg835jfcsi6xb2-youtube-launcher-0.1
$ ./result/bin/youtube-launcher

It still works and it uses Python 3.9.

Also, the resulting path is now different instead of /nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1, it is now /nix/store/prrc01g09l1pma21qndg835jfcsi6xb2-youtube-launcher-0.1 because the input changed.

When you check the runtime dependencies, only those which are related to Python are changed.

$ nix-store --query --references ./result
/nix/store/9r1hhbk3r9jv9dcnd3z59jyjs1k08r8p-qtquickcontrols-5.15.2
/nix/store/cgpka5n7m8l31i6j9f5b071l8alqzq6v-qtbase-5.15.2-bin
/nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23
/nix/store/fn2kv4wn3n4kpv6sb96cp9hi65ra9zb1-python3-3.9.4
/nix/store/k5f31g4g5s7agb7yahlsi4w7jz15516x-qtsvg-5.15.2-bin
/nix/store/kv3wcr31ba5r2hglaz5d73mg084ld820-qtdeclarative-5.15.2-bin
/nix/store/ndaj0cmm8j833y3fp346vbh9xmnsqwbp-python3.9-PyQt5-5.15.2
/nix/store/xdd39p2zsgypr0cyiiqkzcg1bxsrdpp5-python3.9-PyQt5_sip-12.8.1
/nix/store/dkmbkkax4vqbmc6y1dqqhg9xi9yc23wj-python3.9-PyQt5-5.15.2-dev
/nix/store/p7csiywv541jnvrgah93p7zjvq8lkwkq-mpv-with-scripts-0.33.1
/nix/store/psvnd1d97brqwgw9c8qhw4l85nlvpn6y-qtwayland-5.15.2-bin
/nix/store/q0vfa5rm0y62knfhk582fzw2rlrzdmgg-python3.9-dbus-python-1.2.16
/nix/store/prrc01g09l1pma21qndg835jfcsi6xb2-youtube-launcher-0.1

In addition, it uses the same exact Qt and mpv components as the Python 3.8 version.

Thanks to the lack of filesystem isolation, you didn’t download a different version of Qt nor mpv since it uses the same one.

You can explore the resulting component on your own and use specialized tools like diffoscope. But, at the end of the day, you’ll reach the same conclusion.

Want to proof Nix’s deterministic nature?

Just revert back to the old version which uses Python 3.8.

$ ls
default.nix  default.nix.orig  result
$ mv default.nix.orig default.nix
$ cat default.nix
let
  ...
in
pkgs.python38Packages.buildPythonApplication {
  ...
  propagatedBuildInputs = with pkgs.python38Packages; [ pyqt5 ];
  ...
}
$ nix-build
/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1

And we’re back to the old component, the path is the same too.

Still skeptical? You think that because it is cached inside the Nix store that it results in the same path?

Let’s perform garbage collection to wipe every unused components and then rebuild to proof that it is deterministic and the same Nix file will always build the same exact component.

$ rm result
$ nix-collect-garbage -d
removing old generations of profile /nix/var/nix/profiles/per-user/yuki/profile
...
finding garbage collector roots...
removing stale link from '/nix/var/nix/gcroots/auto/y3scm0nhz4gacw17lpf0mg03r8f9l65i' to '/home/yuki/trash/result'
deleting garbage...
deleting '/nix/store/4dlhsl4kxp9p632mbv1rcq9kjc0y6zdy-stdenv-linux'
...
deleting unused links...
note: currently hard linking saves -0.01 MiB
186 store paths deleted, 578.66 MiB freed
$ ls
default.nix
$ nix-build
this derivation will be built:
  /nix/store/5ia4ma9b96kg620k7k0w9xhgqajd828n-youtube-launcher-0.1.drv
these 113 paths will be fetched (75.82 MiB download, 384.18 MiB unpacked):
  /nix/store/07j6d0lr6p1gjxi2qhf6wn88nl81x5jj-perl-5.32.1
  ...
copying path '/nix/store/6c5x1d03c5323lsbr2il8w1gwf83cqjj-python-remove-bin-bytecode-hook' from 'https://hydra.iohk.io'...
...
building '/nix/store/5ia4ma9b96kg620k7k0w9xhgqajd828n-youtube-launcher-0.1.drv'...
...
/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1

As you can see, it’s still the same.

In case you didn’t know, I originally wrote this article when the latest version of nixpkgs is v21.05.

Now, in January 2022, the latest version is v21.11. You can try to modify the default.nix to use v21.11 instead of v21.05:

let
  pkgs = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/4f6d8095fd51.tar.gz";
    sha256 = "14sm0bjjcmi9qmznwy3nkd2vbhj5xcshgm54a5wiprl9ssvxqw53";
  }) {};
  ...
in
  ...

Using the change above, try building it on your machine. I can guarantee the path will be /nix/store/w94hyhds5dc0rv4plsp11y8nm6jnv354-youtube-launcher-0.1 on linux/amd64.

The basic building blocks of Nix

At this point, you might think that Nix is a very complex piece of software.

After all, when we’re packaging software, we need to write a script in a weird bespoke programming language!

While yes, this article has demonstrated that you are required to write Nix expressions (.nix files we’ve written) to do anything with Nix.

However, beneath the seemingly complex language, you’ll see that the very foundation of Nix are built with basic fundamental building blocks.

With that in mind, you have seen things we’ve achieved with Nix expressions: launching an environment with additional software to build and package complex software. However, how does it all work?

Instatiation

First, Nix expressions are translated to derivations in a process called “instatiation”.

A careful reader might notice this term being mentioned a lot when we demonstrate Nix: .drv files and the word “derivation” scattered throughout from Nix’s outputs.

What are they? Let’s take a look!

We use our previous Nix expression which packages the YouTube Launcher app we just made (without the modifications) and use nix-instantiate to perform only the instatiation process.

$ nix-instantiate default.nix
/nix/store/5ia4ma9b96kg620k7k0w9xhgqajd828n-youtube-launcher-0.1.drv

After running the command, it outputs the path within the Nix store of the resulting derivation. Let’s see what’s inside!

$ cat /nix/store/5ia4ma9b96kg620k7k0w9xhgqajd828n-youtube-launcher-0.1.drv
Derive([("out","/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1","","")],
...

Oh wow, this looks rather daunting. No worries, nix show-derivation will pretty print it for you!

Just give the path to the derivation to nix show-derivation, like so:

$ nix show-derivation /nix/store/5ia4ma9b96kg620k7k0w9xhgqajd828n-youtube-launcher-0.1.drv
{
  "/nix/store/5ia4ma9b96kg620k7k0w9xhgqajd828n-youtube-launcher-0.1.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1"
      }
    },
    "inputSrcs": [
      "/nix/store/64fk9k7an4gyn7qgr85p42s8763yc56q-source",
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "inputDrvs": {
      "/nix/store/2zjvqfjg2pww4lkafw2x4ni450a2w8hx-setuptools-setup-hook.drv": [
        "out"
      ],
      ...
      "/nix/store/wrnnl82wk8lz8wqgcrh8kii0kfjnl875-hook.drv": [
        "out"
      ]
    },
    "system": "x86_64-linux",
    "builder": "/nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash",
    "args": [
      "-e",
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "env": {
      "LANG": "C.UTF-8",
      "buildInputs": "/nix/store/p7csiywv541jnvrgah93p7zjvq8lkwkq-mpv-with-scripts-0.33.1",
      "builder": "/nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash",
       ...
      "strictDeps": "1",
      "system": "x86_64-linux",
      "version": "0.1"
    }
  }
}

It’s a rather massive JSON file. Let’s break it down.

The outputs (`outputs`)

"outputs": {
  "out": {
    "path": "/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1"
  }
},

The outputs section defines the deterministic path of the output of this derivation. The process is simply called “building”.

Nix builds the derivation and then store its outputs to the path defined here.

The inputs (`inputSrcs`, `inputDrvs`)

"inputSrcs": [
  "/nix/store/64fk9k7an4gyn7qgr85p42s8763yc56q-source",
  "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],

inputSrcs defines the path of the source code within the Nix store and a builder script which will be executed.

"inputDrvs": {
  ...
  "/nix/store/457yxakhv0lc3df8jlb9cz0q8fk44lns-qtbase-5.15.2.drv": [
    "dev"
  ],
  "/nix/store/6pa9w933ipkki2rxxw57qpqy4ngs8q6a-python3-3.8.9.drv": [
    "out"
  ],
  ...
  "/nix/store/7imvh7yyv754jjvaqd7cgvd196xcixsj-mpv-with-scripts-0.33.1.drv": [
    "out"
  ],
  ...
  "/nix/store/jv2c1byik9bpq87rj4vzfd6p6gf9m5cj-python3.8-PyQt5-5.15.2.drv": [
    "dev"
  ],
  ...
},

inputDrvs defines the inputs or dependencies to build this derivation.

It mentions other components via their respective derivations (note the .drv prefix).

You might be wondering why some have out, dev, lib or even multiple of them.

That’s because derivations can have multiple outputs and the list with out, dev tells Nix which outputs of the derivations to use.

You can see how it compares by running nix show-derivation /nix/store/457yxakhv0lc3df8jlb9cz0q8fk44lns-qtbase-5.15.2.drv but I won’t be getting into it in here.

The environment (`system`, `builder`, `args`, `env`)

"system": "x86_64-linux",
"builder": "/nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash",
"args": [
  "-e",
  "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],
"env": {
  "LANG": "C.UTF-8",
  "buildInputs": "/nix/store/p7csiywv541jnvrgah93p7zjvq8lkwkq-mpv-with-scripts-0.33.1",
  "builder": "/nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash",
  ...
  "doInstallCheck": "1",
  "name": "youtube-launcher-0.1",
  "nativeBuildInputs": "...",
  "outputs": "out",
  "patches": "",
  "pname": "youtube-launcher",
  "postFixup": "wrapPythonPrograms\n",
  "postPatch": "substituteInPlace youtube_launcher.py \\\n  --replace \"'mpv'\" \"'/nix/store/p7csiywv541jnvrgah93p7zjvq8lkwkq-mpv-with-scripts-0.33.1/bin/mpv'\"\n",
  "preFixup": "makeWrapperArgs+=(\"${qtWrapperArgs[@]}\")\n",
  "propagatedBuildInputs": "/nix/store/cvwc3r4dcjjsa6vb50idlc4q171whvgl-python3.8-PyQt5-5.15.2-dev /nix/store/4s0h5aawbap3xhldxhcijvl26751qrjr-python3-3.8.9",
  "propagatedNativeBuildInputs": "",
  "src": "/nix/store/64fk9k7an4gyn7qgr85p42s8763yc56q-source",
  "stdenv": "/nix/store/4dlhsl4kxp9p632mbv1rcq9kjc0y6zdy-stdenv-linux",
  "strictDeps": "1",
  "system": "x86_64-linux",
  "version": "0.1"
}

Here we see system, builder, and args, what’s up with these?

You might be able to guess on your own but here’s an explaination:

system defines which exact architecture/operating system this derivation is for.
In the early days of Nix, builder was originally used to defined the script used to build the derivation.

This script was made along with the expression but this was changed in later versions to use a generic builder script which we’ll see later.

Nowadays, it always points to the bash shell interpreter.
args defines the command arguments to pass to the builder. Because of the change mentioned before, this simply tells the bash interpreter to execute the generic builder script.
env defines all of the environment variable that will be setup for the builder.

To ensure reproducibility, before running the build script, Nix prepares a clean slate which involves removing all environment variables.

Something about environment variables

Have you noticed something perculiar with env?

If you read the env section carefully, you see that it contains code we’ve written in the Nix expression like the postPatch and preFixup section.

"postPatch": "substituteInPlace youtube_launcher.py \\\n  --replace \"'mpv'\" \"'/nix/store/p7csiywv541jnvrgah93p7zjvq8lkwkq-mpv-with-scripts-0.33.1/bin/mpv'\"\n",
"preFixup": "makeWrapperArgs+=(\"${qtWrapperArgs[@]}\")\n",

let
  ...
in
pkgs.python38Packages.buildPythonApplication {
  ...
  postPatch = ''
    substituteInPlace youtube_launcher.py \
      --replace "'mpv'" "'${mpv}/bin/mpv'"
  '';

  preFixup = ''
    makeWrapperArgs+=("''${qtWrapperArgs[@]}")
  '';
}

After seeing all this, you realize that all of that seemingly complex language ends up as a bunch environment variables in the end?

Yeah, pretty much.

Since we don’t seem to find anything that performs the build, let’s take a peek inside the generic build script itself?

What kind of magic is it doing?

Realization

Reading to the massive Nix derivation for a second time, we see the location of the build script from args.

"args": [
  ...
  "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],

$ cat /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
source $stdenv/setup
genericBuild

Just two lines of shell? Seriously?

Okay, we see that it sources another shell script called $stdenv/setup. Knowing that it mentions an environment variable called stdenv, let’s check its value from the derivation.

"env": {
  ...
  "stdenv": "/nix/store/4dlhsl4kxp9p632mbv1rcq9kjc0y6zdy-stdenv-linux",
  ...
}

So, it’s another component in the Nix store, let’s take a peek inside.

$ ls /nix/store/4dlhsl4kxp9p632mbv1rcq9kjc0y6zdy-stdenv-linux/
nix-support  setup
$ cat /nix/store/4dlhsl4kxp9p632mbv1rcq9kjc0y6zdy-stdenv-linux/setup
export SHELL=/nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash
initialPath="/nix/store/a4v1akahda85rl9gfphb07zzw79z8pb1-coreutils-8.32 ..."
defaultNativeBuildInputs="/nix/store/yayg9xvxq3f8avpvw81p7a45zqadpgvb-patchelf-0.12 ..."
defaultBuildInputs=""
# Make "strip" produce deterministic output, by setting
# timestamps etc. to a fixed value.
commonStripFlags="--enable-deterministic-archives"
export NIX_ENFORCE_PURITY="${NIX_ENFORCE_PURITY-1}"
export NIX_ENFORCE_NO_NATIVE="${NIX_ENFORCE_NO_NATIVE-1}"
NIX_LIB64_IN_SELF_RPATH=1
...

It’s a massive bash script which actually performs the build. This is it: the generic builder.

As you can see, unlike curl | sh installers, the Nix expression language has a very limited scope of what it can actually do.

The build process is a two-stage approach:

Instatiation

Outside the Nix store, Nix expressions are translated to Nix derivations. The resulting derivation is then placed inside the Nix store.
Realization

Within the Nix store, the derivation is read by the builder which will execute the builder script and the derivation’s outputs from the build process will be stored inside the Nix store.

This entire process will never touch any other part of your system and everything is nicely encapsulated.

To guarantee this, Nix performs Sandboxing which isolates the build process from the host system.

The Standard Environment

However, as you can see, all of this relies on stdenv. What is that?

As the name implies, it is the standard environment provided by nixpkgs to build all software with Nix.

This was originally made because it was impossible to rely on the “Linux Standard Base” of the present host system to serve as the stable basis to build software upon. Therefore, Nix made its own standard environment where it serves as the foundation of all software built by it.

This basic foundation contains the basic set of utilities required to build everything, such as GCC compiler, bash shell interpreter, GNU coreutils, make, patch, etc.

In order to be able to use and install software Nix, you need a prebuilt standard environment for your system.

We won’t be covering how the standard environment is bootstrapped and built in this article. If you’re interested, the stdenv package source in nixpkgs is a good starting point.

In terms of trusted binaries, stdenv is the only component inside Nix’s ecosystem that you need to trust.

However, this situation is much better than other package managers which require you to trust individual packages which are uploaded by their respective maintainers.

This requirement of trusting package maintainer uploads has spawned scandals such as the infamous Chromium binary blob drama which happened in Debian in 2015 and nowadays, a lot of the same thing is happening within the npm ecosystem.

In Nix, other than the standard environment, you’re not required to trust the binaries which are delivered to you. You have the choice to build everything on your machine, Gentoo style.

Just go to nix.conf and remove https://cache.nixos.org from substituters.

...
substituters = https://cache.nixos.org/
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
...

The location of nix.conf varies on how Nix is installed.

They’re either in /etc/nix/nix.conf or ~/.config/nix/nix.conf.

If you’re running NixOS or use home-manager, you need to modify configuration.nix or home.nix.

Once stdenv is present in your Nix store, you can completely detach yourself from the Nix binary cache and build every single thing on your own.

However, unlike binary repos, you shouldn’t need to worry about the Nix binary cache.

Remember, one of Nix goals is to be able to reproduce the same exact output every time, bit by bit. So, in case the cache is serving malformed binaries, you can check and verify.

According to r13y.com, as of writing, the NixOS unstable ISO is 100% reproduced exactly the same, at different times, on different systems.

When this is not the case, most of the time it caused by manpage generation.

Binary cache

I’ve mentioned the binary cache previously but what is it exactly?

It is essentially an accessible Nix store that is used to download built components from.

While you might think it’s a binary repository, I hope after reading this far, you’ll realize that it’s not really a component.

A more accurate analogy would be an artifact store of a typical continuous integration setup.

In case you aren’t familiar, in short, continuous integration is an automated system where it builds source code regularly following each iteration of the source code as it change.

After the build process is done and passes successfully, it will end up with fresly built binaries which will be then stored inside an artifact store where developers or highly technical users can access to test the new bleeding edge version of the software as if they’re fresh out of the oven.

In fact, this is how the Nix community builds their entire package collection, nixpkgs. Hydra is the continuous build system which regularly follow and build the entire Nix packages collection.

The result of each build is submitted to the massive central Nix binary cache instance which anyone can use to download the latest bleeding edge software without having to build on their own machine.

Recalling how the derivation contains the output path before it is even built, you can see how Nix knows which files to download from the binary cache.

"outputs": {
  "out": {
    "path": "/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1"
  }
},

Same goes with the dependencies, the inputDrvs defines which derivations are used as the inputs for the software.

"inputDrvs": {
  ...
  "/nix/store/457yxakhv0lc3df8jlb9cz0q8fk44lns-qtbase-5.15.2.drv": [
    "dev"
  ],
  "/nix/store/6pa9w933ipkki2rxxw57qpqy4ngs8q6a-python3-3.8.9.drv": [
    "out"
  ],
  ...
  "/nix/store/7imvh7yyv754jjvaqd7cgvd196xcixsj-mpv-with-scripts-0.33.1.drv": [
    "out"
  ],
  ...
  "/nix/store/jv2c1byik9bpq87rj4vzfd6p6gf9m5cj-python3.8-PyQt5-5.15.2.drv": [
    "dev"
  ],
  ...
},

Simply read each derivations in the list and do the same process again until you have a list of components that can be downloaded from the binary cache and components that needs to be built. Obviously, components which are already present in your system’s Nix store don’t need to be rebuilt/redownloaded.

The deterministic nature reduces the amount of software required to be downloaded and built therefore we save time, storage, bandwidth, and compute resources.

For private or propriertary projects, you can run your own Nix binary cache for your organization. However, this is out of scope for this article. The easiest way is to use a hosted service like Cachix.

However, you don’t need a binary cache to transfer built components across machines. You can simply copy them manually with nix-copy-closure:

$ ls /nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1
ls: cannot access '/nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1': No such file or directry
$ nix-copy-closure --from yuki@neru \
  /nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1
$ nix-copy-closure --to yuki@kuzuha \
  /nix/store/y1aimywh5ff57pv2azg705hlkcciy2dn-youtube-launcher-0.1

By the way, you can only share built components to other machines which are the same system and architecture (i.e Linux/amd64 machines can transfer to another Linux/amd64 machine but not to macOS/arm64 machines). The same is true for binary caches. So, keep that in mind!

Distro-independent

Since the beginning, Nix is a package manager that works along side your existing one. In fact, the NixOS paper was published 4 years after the Nix paper.

Because of this, packages in Nix are vendor-independent and theoretically can be used in any UNIX-like operating systems and computer architectures.

As of writing, officially, nixpkgs and hydra (nixpkgs’ main binary cache and build cluster) supports Linux on ARM and Intel/AMD along with macOS on Intel and even Apple Sillicon!

However, people have made Nix packages build on all sorts of weird operating system and architecture combinations.

This allow you to choose which system you want to use without having to worry about software availability.

As of writing, according to Repology, nixpkgs unstable sits at the number 1 spot for total number of projects packaged. So, I’m fairly certain every free and open source that you need is available in nixpkgs.

Tired of Arch Linux breaking again? Why not move to something stabler like RHEL, Debian, or SUSE and just use Nix for the additional software you need/want.
Stuck on RHEL, Debian stable, or SLE because of work and tired of using 2-4 years old version of your favorite software? Just fire up Nix and grab everything you need without any risks.
Do you need a specific version of software at work, like PHP 5 for legacy code, but your coworkers are using macOS and various Linux distros? Just use Nix and create a Nix shell environment for them!

Personally, after discovering the power of Nix, I simply stopped using rolling release distros like Arch Linux or Gentoo “unstable” or Debian sid.

As of writing, I’m currently using Ubuntu 20.04 LTS or Debian stable under WSL and just use Nix for the additional software I need, it’s lovely not having to fuss about my system.

Conclusion

As you can see, Nix nicely solve the common issues with traditional software deployment methods.

The cryptographic hashes are used to guarantee that software are built with the exact same inputs (dependencies, build options, source code, etc) and avoids collision with other versions of the same software.

The hashes are then used to make deterministic paths which ensure the built software uses the exact version of libraries that it needs. In fact, Nix determines runtime dependencies for a built component simply by scanning the build output for the /nix/store/... patterns.

The same Nix expression can be shared to other systems without having to worry too much about system specific detail.

Nix’s deterministic nature and the power leveraged by binary caches allows you to avoid building the same software twice. In addition, the entire process guarantees that you only download and build the components you actually need on runtime.

Outside of the second stage of the build process, Nix doesn’t use any containerization nor namespacing tricks, thus reducing the overhead and doesn’t require giving normal users “root”-like access just to run a piece of software.

In fact, with the sandboxing done by the build process, the generic builder, and how derivations are made from Nix expressions, Nix ensures that building software doesn’t tamper with your system and leave it completely untouched.

After reading this article, I hope you realize how radically different Nix is and how its very foundation is able to solve problems which are common with previous methods of installing software.

Addendum: The Imperfections

However, like many things, Nix is not perfect.

For a lot of people, especially those who aren’t familiar with functional programming, find the Nix expression language to be an unnecessary learning curve.

For something that’s meant to build and package software, I wish it uses something that’s more familiar, such as rpm spec and BSD-style Makefiles.
Since Nix relies on cryptographic hashes and deterministic paths to guarantee the inputs used when building a component, what happens when a common dependency like OpenSSL receives a minor update?

It means that an enormous amount of software which uses it directly or indirectly will need to be rebuilt which also means that people have to redownload a bunch of the same software.

This landslide effect doesn’t exist in more traditional approaches to software packaging as the package maintainer can simply check whether the current build of the application works with the newer version of OpenSSL. Therefore, only the packages which fail will need to be rebuild and updated.

There are many other imperfections like these within the Nix ecosystem. However, in comparison to the alternatives, they’re much more easier to solve than the foundational problems with traditional approaches.

Since Nix only cares about the derivation, it doesn’t care how it was made. In fact, this is how GNU Guix came to be. They forked Nix (nix-daemon) and replaced the language with Scheme/Guile which is easy to do, thanks to the basic fundamental building blocks of Nix.

While the second one is more difficult to solve, there’s a solution in the works called content-addressed storage. How it works is out of scope of this article but the people at Tweag wrote a short article about it.

Knowing the people within the community, I’m fairly confident that most of these imperfections will be solved one way or another.

Notes

Mon, 07 Jun 2021 00:00:00 GMT

Hello, is this thing on?

Hi, I’m back after 3 months of absolute silence. I hope you’re doing well despite these trying times.

Yet another new writing platform, really?

Look, I really did believe Writefreely will be the end of it all but sadly, I ended up having mixed opinions about it.

This weekend, I found out that the entire thing broke on its own and locked me out of my own drafts because of an update which adds OAuth functionality, fun.

It lacks a lot of things that I consider good software require, most notably, in this case: any form of update guides.

Seriously though, I thought that this software would allow me to write stuff quickly but in reality, it doesn’t.

Back to Emacs, then

Recently, I’m back to Emacs with org-mode. However, unlike before, I’m actually using something built-into org-mode rather than using something like Hugo/Jekyll which has sub-par support.

Honestly, I feel dumb for not discovering thing much much earlier but only a couple hours ago, I discovered ox-publish.el which adds website publishing functionality to Org!

I just finished setting stuff up and migrated my old posts from the awful Writefreely instance of this thing. While I was considering migrating posts from my old blog, I decided not to mainly because it’ll be a chore making it ‘look’ the same.

General updates

I’ve been really busy these past months and will be busy for the next couple more months with personal stuff.

However, I should have more free time in a couple of weeks which should allow me to have much more intimate writing sessions and overall more time for personal development.

Writing Bucket list: Nix

I always been considering writing a blog post about Nix, the package manager which utilizes the purely functional deployment model.

However, I don’t think I’m experienced enough with Nix to be writing posts about it, especially considering its intimate situation of lacking major documentation.

Some people may think that I should still write about it but I want to write something extensive and in-depth about it because it truly deserves the attention and care.

For now, if you’re an introduction to Nix, I recommend the Nix Pills series or even the original papers: The Purely Functional Software Deployment Model, NixOS: A Purely Functional Linux Distribution.

I describe the current situation with Nix to be similar to early Haskell (pre-Real World Haskell) where you really need to dig through papers, source code, mailing lists, etc. to find quality and accurate information.

I really do hope someone out there is writing a book about Nix but I could see why no one is currently doing it as it is still in early-late development stages. Hopefully, it’ll mature once Nix Flakes is not locked away behind the experimental gate :-)

New computer setup

I replaced my desktop with a new lightweight 14” laptop. It’s a Clevo NV41MB which I bought from PCW.fr. It’s a pretty nice configuration with 1 TB NVMe, 11th Gen Core i5, 16 GB of DDR4 RAM.

I’m running Windows 10 as the primary operating system, mainly because I need to run a notable amount of applications that only run on Windows.

Yes, Wine and Proton exists but sadly, they don’t suffice with my requirements.

On Windows, I use Windows Subsystem for Linux extensively to run Linux applications, most notably GNU Emacs in GUI mode. I run X410 as my X11 server in order to enable GUI applications on WSL.

I know there’s WSLg but it’s not ready for prime time, yet.

OpenBSD is installed as a secondary operating system that I use sometimes. It runs quite stable, however, Intel Xe Graphics is known to be buggy everywhere so yeah but surprisingly, a lot stabler than my adventures with amdgpu, so that’s something!

Also, yes, convieniences such as Wi-Fi, suspend, etc works fine as far as I can tell. I have some annoyances with my keyboard not working sometimes on X11 but I’m not sure if that’s because I’m running my keyboard via my Thunderbolt Dock or something else. Oh well.

Anything else…

Well, I don’t know anything else that I would like to share in this rather lengthy blog post but I’ll just stop here.

See you later,

Yuki.

End of an era

Mon, 12 Jun 2023 00:00:00 GMT

TL;DR: I will stop self-hosting services as my priorities in life has changed which resulted in me not having the time and willpower to maintain them.

Well, this is it: The end of an era, no more working as a system administrator and site-reliability engineer for myself.

I guess a bit of a background is warranted, so lets do that.

Background

Growing up, I was always the odd one out. I was completely enamored with computers and always have dreamed about working with them day to day as a job. If you were there during those time, you’d know that computers was the subject that I always love talking about.

I love learning the history of computers and how things came to be, learning how to use various operating systems, tools, and programming languages, and also getting into the nitty gritty of how stuff works.

One of the things I liked growing up was watching videos on YouTube about hacking game consoles, writing operating systems, etc. These subjects always run around in my head subconciously and once I had the chance to do them, I was estatic.

During the boom of Minecraft, I learned how to setup a Minecraft server and started what was one of the more popular Minecraft servers in Indonesia. From there, I learned how to administer systems, mitigate against attacks (because yes, they did happen), write server code, etc.

Ultimately, what once was a mere hobby for me led to where I am today. Today, I’m working at a German-based software house where I do the things that I dreamed of doing as a kid: administering systems and infrastructure and writing code. Today, I would be lying if I say that I don’t like doing it, I still do.

However, now as an adult, a lot of things in my life has changed and one of them is how I value time and ultimately my priorities in life.

Today, I’m at the point where I’d rather not touch the computer after work.
I no longer have the patience to use a degoogled Android phone, I use an iPhone now.
I stopped using “public” social medias like Twitter and Fediverse.
I stopped daily driving FreeBSD/OpenBSD, I use Fedora or Windows now.

Now, I’m planning to stop self-hosting services. I know there are some people out there who would think that I’m a hypocrite for “not standing up to my values” but the thing is people change, it’s just a part of life and this is how it came to be for me. Whether people think that I’m becoming a better/worse person is irrelevant because ultimately, this is my life and only I decide how I spend it.

I’m not saying that this is objectively the best way to do things because the nature of these decisions are personal and only you can answer them, don’t accept the idea that there’s an objective “best way” whether it’s because of some moral value, technical reasons, or whatever because objectively, there isn’t.

If you can self-host your service, live your life free from big tech companies, good, I’m happy that it works for you. However, and this is something I also learned myself, stop telling other people to be like you and realize everyone has different needs and goals, just because it works for you, doesn’t mean it’ll work for everyone.

Also, I’m not always like this. If you know me for long enough, you know that I used to be the person who preach such things to other people and suggest them to do the things which I perceive as the best way of doing things. And yeah, as mentioned before, I changed and I realized that this is not true and I’m sorry.

Shifting priorities

So what’s up? What made me take this leap in my life?

As you now know, I mostly spent my childhood, teenage, and young adult years with computers because it was the thing that I’m passionate about.

Today, well, I’ve been wanting to branch out to more creative pursuits. I started learning things like mechanical engineering, pixel art, music composition and production, and game development. And well, from there, I made things:

Your browser does not support the video tag.

QWE by Yuki

At the end of the day, I find myself more satisfied and happier creating things more than just mere computer programs. Doing this kind of stuff takes a lot of time and dedication hence why for the past couple of months I’ve been thinking about how I want to spend my time. Ultimately, I find myself being annoyed with being a system administrator of one for myself so yeah, this month I decided to simply stop doing it and I’m looking forward to how things go from here.

Changes

With my big server shutting down at the end of this month, a couple of things are changing:

All of my personal projects are on GitHub.
- Originally, I was thinking about moving to sr.ht since I like the simplicity of it but they don’t support Git LFS yet.
- Then, I was thinking about moving to GitLab SaaS but the costs for Git LFS are too much for me.
- On Matrix, I moved from @yuki:backalley.club to @yukiisbored:matrix.org.
I’ll no longer be available on XMPP.
- While there’s a lot of open XMPP servers, I decided to simply stop using it since I don’t communicate with a lot of people there and the clients for iOS are not good.
Old websites like blog.yukiisbo.red will be shutdown.
- I don’t know if I’ll move their content to this site but the content will be available in source form on GitHub.

Of course, there are plenty of other things changing but I won’t be mentioning them here since they’re only relevant to me and ever since I stopped using public social medias, I’d like to keep my life more private instead of sharing everything.

Conclusion

If there’s anything you should pick up from this, it should be the fact that you should live your life how you want it to be instead of giving control over it to others. Ultimately, only you know what is best for you.

This doesn’t mean that you should be immune to criticism or suggestions, far from that. Instead, spend time to learn more about yourself, get to know about yourself better, and learn to accept yourself. It won’t be easy but it’s worth it, I promise.

From there, you can start and figure out what you’re comfortable with, what works best for you, and your boundaries. And most importantly: remember, there’s no right/wrong answer.

Fine-grained controls and why we don't need it

Fri, 12 May 2023 00:00:00 GMT

I feel like when it comes to permissions and security, we often fall into the believe that the more fine-grain control we have, the better.

In reality, this just leave them ripe to abuse as more often than not, we’ll simply leave them to the (often bad) defaults as changing them is hard:

A simple glance is enough to invoke analysis paralysis as you stand there trying to understand what the fuck do any of these mean.

It’s the same reason why I have a distinct distaste for software like Sonarqube, Jenkins, Capciscum, SELinux. Controlling over them feels like wielding a buster sword, with its sheer weight and unbalanced nature.

Compare Jenkins’ Role Controls to Sentry’s:

Clear distinct roles each with a clear description.

But how about clients or contractors? We don’t want them to be able to see everything.

Sentry’s solution is simple, they have the concept of teams:

Projects belongs to teams, members are part of teams, each members’ role is set at the organization level.

When you think about it, this pretty much covers most use cases.

A solution which gives users total freedom to do whatever they want is an easy way to express a middle finger towards usability and user/developer experience.

Sure, it’s cheap to establish and execute at the beginning during the initial development of the project, but just imagine the sheer amount of human lives wasted at maintaining this ugly beast of complexity once it is implemented?

As a user, you’re faced with an unwieldy system that feels like you need to have a degree for.
As a developer, you’re faced with covering all possible use cases and being faced with ridiculous edge cases.
As customer support, you’re faced with numerous calls which lasts for hours covering the intricacies of the system and guiding them to avoid pitfalls.
As a technical writer, trying to come up with a digestible document for an inhumane system.

Overall, I feel like both as developers and project managers, it is our sacred duty to root out all complexity as much as possible.

People don’t throw money for complexity, they throw money at tools which helps them achieve their goals with the least amount of misery.

But our competition has a lot of fine-grained complexity and sheer amount of features!

Look at Canva and Figma and ask why users prefer them over Photoshop, InDesign, and Illustrator.

The reason is simple, they’re more approachable while still allowing them to achieve their goals without footgunning themselves.

You don’t gain market adoption by being equal. You do it by making your shit stand out and making a constrained product which allow people to achieve the same goals is one of the best way to do it.

Yuki's Notes

Writings

Surviving Docker: Packaging penguins into boxes

Difficulty of delivering software

Containers as a medium to deliver items

Docker images as a medium to deliver software

Docker in practice

Stateless Containers

Storing stateful data with bind-mounts.

Storing stateful data with Docker volumes

Container networking

Orchestrating Docker containers

Closing

Remarks

Qubi Research Program Postmortem

Proof of concept

Initial foothold

Constructing the Foundation

Basic Gameplay

Balancing

Increasing Difficulty

End

Spice up with Nix: The issues with traditional software deployment

The problem with traditional software deployment

The problem with traditional packages

The problem with curl | sh installers and language-specific package managers

The problem with containers

Closing

Spice up with Nix: Scripts with magical dependencies

Instant runtime environments for scripts with Nix

Only Python? Nope.

Closing

Spice up with Nix: Instant Development Environments

Instant (development) environments with Nix

What does shell.nix do?

Reducing duplicate code using the with statement

Learning more about the Nix expression language

Where does nixpkgs come from?

Ensuring reproducibility by pinning with niv

Closing

Remarks

Mic Check

Why move away from org-mode?

What led me to move away from org-mode to Gatsby?

Why Gatsby?

Why JavaScript?

Is Gatsby perfect?

Why the merge with yukiisbo.red?

What’s next?

Spice up with Nix: The Functional Software Deployment Model

The one simple trick

Getting your hands dirty with Nix

The basic building blocks of Nix

Instatiation

The outputs (outputs)

The inputs (inputSrcs, inputDrvs)

The environment (system, builder, args, env)

Something about environment variables

Realization

The Standard Environment

Binary cache

Distro-independent

Conclusion

Addendum: The Imperfections

Notes

Yet another new writing platform, really?

Back to Emacs, then

General updates

Writing Bucket list: Nix

New computer setup

Anything else…

End of an era

Background

Shifting priorities

Changes

Conclusion

Fine-grained controls and why we don't need it

The problem with `curl | sh` installers and language-specific package managers

What does `shell.nix` do?

Reducing duplicate code using the `with` statement

Where does `nixpkgs` come from?

Ensuring reproducibility by pinning with `niv`

The outputs (`outputs`)

The inputs (`inputSrcs`, `inputDrvs`)

The environment (`system`, `builder`, `args`, `env`)