TL;DR: This post is the first in a small series in which I talk about bootloader and operating system development. In this one I talk about what UEFI is, how it works, how to write a simple UEFI application for the x64 architecture, compile it, and run it via emulation and on real hardware.

Also, please do follow along with the code described in this post, which you can find here!

Post contents:

• What is UEFI?
• How does UEFI invoke our code?
• How do we talk to UEFI?
• Compilation and Running

What is UEFI?

When I’ve had a go at making an operating system before, I either wrote a tiny bootloader in a MBR boot sector, or used GRUB. Most recently, though, I experimented with Limine, which is a bit more modern and does talk to UEFI to get the kernel booted and running. It also does a few cool things like exposing certain pieces of system information to the kernel through structs. I wasn’t entirely satisfied though; the whole reason that I’m interested in making my own operating system is to understand how every piece works, and so I need to write a proper bootloader first.

The “legacy” way of booting a computer is by writing a little piece (<512 bytes) of code which sits at the beginning of a disk (paritioned with the MBR scheme). The BIOS (which is the firmware installed on your motherboard, that has now been superceded by UEFI) looks for the bootstrap code and jumps to it when the computer is turned on. That bit of code “bootstraps” the operating system by reading in the OS code from the rest of the disk into memory, then jumping to it.

The thing is, it’s difficult and fiddly to write this kind of bootstrap code. Reading disks is only doable through a super low level interface; writing to the screen is not trivial; and the MBR partioning scheme limits the number of partitions. Also, maybe more importantly, it’s all extremely platform dependent, and not well documented. Like, how do you know if the system has video output? How do you find out the address of the memory to write to to draw pixels on the screen? How do you know the memory layout, or the attached the disks? It’s all pretty annoying, and the reason that I didn’t make a proper bootloader before.

But, UEFI is the “modern” approach, and solves pretty much all of these problems. UEFI is a specification which describes a class of firmware which is able to load “UEFI Applications” (which are really just executable, structured pieces of code that do things), and clearly describes the following:

• What state the CPU is in when control is passed to the UEFI application,
• What information is passed to the application,
• What services are available, and
• Which devices on the system support certain protocols (like disk access and video output), along with exposing simple function handles to use them.

It’s all very nice, and makes developing bootloader code much easier and more similar to writing normal high-level code. It also makes it much simpler for me to get my bootloader running on my real computer, rather than just using an emulator.

The rest of this post will make up the first of a small series, in which I’ll describe interesting bits I run into while developing an operating system. This time I’ll just go over how to get set up with writing a UEFI application, how to compile it together, and run it. In the next post I’ll show how to load an executable ELF file from the disk (i.e. the kernel code) and pass control to it from the bootloader!

How does UEFI invoke our code?

The first thing to do is get a copy of the UEFI spec, which you can either view as HTML or download as a PDF (my preference since I can then go through and annotate it). It’s crazily long, like 2,300 pages, but most of that is descriptions of specific protocols which you probably won’t even need to use. It’s organised pretty nicely for skim-reading. I’d say the most relevant sections for us are 1.6 (UEFI overview), 2.3.4 (x64 calling conventions), 2.4 (Protocols), and 4 (EFI System Table). We will then look up certain relevant parts of sections 7 (Boot Services) and various protocol sections when we want to use them. It’s not actually too much to read!

So the general idea is this: when you press the power button on your computer, the firmware (some UEFI implementation) will get going and look for a UEFI application to run. It first tries a user-specified boot order stored in on-chip nonvolatile memory, and if it can’t find anything suitable there it will instead look up the default path of /EFI/BOOT{machine type}.EFI, which in the case of x64 looks like /EFI/BOOTX64.EFI.

Once it finds the EFI file that it should boot, it locates its entry point and calls it as a function. The “handoff state” (UEFI spec 2.3.4.1) is such that an EFI_HANDLE is in the register rcx, and an EFI_SYSTEM_TABLE* in rdx. Our entry point is also allowed to return a value (UEFI spec 4.1.1) of type EFI_STATUS. If it returns, the UEFI firmware goes back to the boot menu. Since eventually our UEFI application will load and run an operating system, it will never actually return, in practice.

The system table which the firmware passes to “us” contains a bunch of information and function pointers(!), which I think is very exciting. The firmware implements some useful procedures which we can just call directly by looking through the system table!

Anyway, to start with, this “handoff state” together with a definition in UEFI spec 4.1.1 tells us that we can write our EFI entry point like so:

EFI_STATUS efi_main(EFI_HANDLE ImageHandle, EFI_SYSTEM_TABLE *SystemTable)
{
    // Do bootloading stuff
}

But wait, where do all these types come from? Well, luckily, UEFI spec 2.3.1 gives us a big table explaining what every type they use means. They sort of redefine lots of standard C types just to make sure everyone’s on the same page. For now, we just need to know that an EFI_STATUS is a native width unsigned integer (i.e. uint64_t on x64); an EFI_HANDLE is a void*, and an EFI_SYSTEM_TABLE is a particular struct (UEFI spec 4.3.1).

How do we talk to UEFI?

For today, let’s start with a really simple example. One prerequisite for a “proper” bootloader is obviously that we can print Hello World, right?

As an aside for a moment, let’s consider how we would do such a thing without UEFI. Imagine that we booted from BIOS instead, and we have a little <512 byte chunk of assembly code. Well, that’s not a lot of space, so usually you plonk a bit more code somewhere else and load that as a second stage of the loader. But then, how to write text to the screen? Well, BIOS systems often support a compatability mode where on boot there is “VGA memory” at a certain address, with the VGA driver running in text mode. If you write some text into that memory, some characters will show up on the screen. So it’s not too hard to display some text, but it’s just a raw text buffer, and we would have to manually implement screen scrolling and even just moving the cursor. It’s a bit of a hassle.

According to UEFI spec 4.3.1, the system table is a pointer to a struct with the following layout:

typedef struct {
    EFI_TABLE_HEADER Hdr;
    CHAR16 *FirmwareVendor;
    uint32_t FirmwareRevision;
    EFI_HANDLE ConsoleInHandle;
    EFI_SIMPLE_TEXT_INPUT_PROTOCOL *ConIn;
    EFI_HANDLE ConsoleOutHandle;
    EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL *ConOut;
    EFI_HANDLE StandardErrorHandle;
    EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL *StdErr;
    EFI_RUNTIME_SERVICES *RuntimeServices;
    EFI_BOOT_SERVICES *BootServices;
    UINTN NumberOfTableEntries;
    EFI_CONFIGURATION_TABLE ConfigurationTable[];
} EFI_SYSTEM_TABLE;

For now, don’t worry about most of these fields. Focus specifically on the ConOut member, which is of type EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL. A protocol, in UEFI, is a specific set of procedures that are supported by a certain object. (An object is a bit of an opaque and broad definition, by design. It can be anything which you can get a handle on, but is commonly a disk, other hardware device, filesystem, or file.)

This specific protocol, the EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL, is supported by objects that support, uh, simple text output. Typically this is a screen, but there’s no reason I can think of that mean it couldn’t also be, for instance, a printer.

The protocol itself is defined by its own struct, whose members are all function pointers. You can see these all defined very clearly in UEFI spec 12.4.1. Since right now we’re only interested in outputting some text, there’s no need to fill in the precise definitions for all the function pointers, so we can define them as void pointers instead.

typedef struct _EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL {
    void *Reset;
    EFI_STATUS(EFIAPI *OutputString)(
        struct _EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL *This,
        CHAR16 *String);
    void *TestString;

    /* ... some fields omitted here for brevity, but it's important they all be filled in */
} EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL;

If you want, you can think of this is an object oriented interface, where these function pointers are methods on a class. In this way it is simple to understand that the first parameter, This, is a pointer to the specific instance of a SIMPLE_TEXT_OUTPUT_PROTOCOL, which for us will be the one we find in the system table.

So with that said and done, let’s print some text! The next few blocks of code make up our entire efi_main entry point function; I’ll go over it line-by-line.

EFI_STATUS EFIAPI efi_main(EFI_HANDLE ImageHandle, EFI_SYSTEM_TABLE *SystemTable)
{

First, as before, we have to declare our entry point with a signature that matches that defined by UEFI. Here we introduce an important addition, though: we annotate our function with this EFIAPI symbol. This is simply a macro for __attribute__((ms_abi)), and instructs that the compiler that this function should be compiled as if using the Microsoft calling convention. The reason for this is because it is the calling convention that UEFI mandates, and the reason for that is that UEFI applications are PE applications (a format developed by Microsoft). I don’t really know why UEFI selected Microsoft’s executable format (and therefore calling convention), but I guess they had to pick something. It just means we need to explicitly use the ms_abi attribute on any functions that UEFI “talks to”.

    EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL *out = SystemTable->ConOut;

Here we just retrieve the simple text output protocol instance from the system table. This bit’s pretty straightforward. If you want, you could also save it to a global variable so that it can easily be used if any other functions need to print things.

    out->OutputString(out, L"Hello, world!\r\n");
    while (1) ;
}

Finally, let’s actually do the thing! We’re using the protocol instance both as a table of functions to find the OutputString method, as well as passing it as the This argument in said method. The weird bit here to discuss is what’s going on with the string. Firstly, we declare the string with the L prefix, which produces a wide string literal within which each character is 2 bytes. We have to do this because it’s what the OutputString function expects (UEFI spec 12.4.3), and it is actually nice as it provides good unicode support. Then, we end our string with “\r\n”, which will look normal to Windows users and slightly odd to Linux users. The “\r” (carriage return) moves the cursor to the first column, and “\n” (line feed) moves the cursor down one line, potentially scrolling (UEFI spec Table 12.10). Think of them together as going to the start of the next line.

Finally, the infinite loop right at the end keeps control within our function, so that the machine doesn’t keep resetting back to the UEFI menu.

Compilation and Running

This function, along with definitions for the UEFI types used by it, are really all we need!

Overall, we want the following:

• A disk (image) formatted using GPT partitioning. UEFI understands both GPT and the legacy MBR partitioning schemes, but GPT is better, so we’ll use it.
• A FAT filesystem on this disk. FAT is the required filesystem for UEFI to read and load UEFI applications from.
• Our compiled UEFI application, as a PE(32+) executable, sitting in that FAT filesystem at the path “/EFI/BOOT/BOOTX64.EFI”.

Once all this is in place, in theory our disk image is ready to flash onto a USB stick and plug into our computer to boot from!

Notice: It might be worth mentioning here that running random code like this on your own machine can cause Bad Things, because the code runs at supervisor level and if something is written wrong it could erase your disk, corrupt things, or otherwise be a nuisance. For safety and also just ease of use, it’s better to emulate a machine using something like QEMU (which we will see later), but the satisfaction of your code running on real hardware is maybe worth it. Up to you.

Anyway, how to construct such a disk image? I’m working on Linux, so I have a package installed called dosfstools which allows me to work with FAT filesystems without directly mounting them. I also have fdisk installed to format the disk image in the first place. For compilation and linking we need to use clang rather than gcc because it supports compilation and linking against non-native targets (remember, we need to produce a Windowsey PE executable).

Compiling

With that said, let’s start by compiling this thing. The entire C source is in a file called boot.c

clang -c \
    -target x86_64-unknown-windows \
    -ffreestanding \
    -fno-stack-protector \
    -mno-red-zone \
    -Wall -Wextra -Wpedantic -g \
    -o boot.o boot.c

Although we need to pass a few odd flags to the compiler and linker, it’s really quite standard compilation at its core. We need to specify that the compilation target is “windows” so that clang generates an object file format suitable for linking as a PE executable.

The freestanding flag tells the compiler that we don’t have a standard library (so no printf and friends). no-stack-protector tells the compiler not to generate code to automatically check for things like buffer overflows (a check which would be very useful for userspace code, but that can get in our way when we’re doing low-level memory stuff).

A “red zone” is a 128-byte region of memory just beyond the stack pointer which functions can use without worrying about their data being clobbered by interrupts, when running on top of an operating system. Since we do not have any operating system ensuring that the “red zone” is safe to use, we just disable it altogether with -mno-red-zone – it’s only an optimisation anyway, and one that we don’t need.

Once we have our boot.o object file, we need to create the PE executable:

lld-link /subsystem:efi_application \
    /entry:efi_main \
    /nodefaultlib \
    /out:boot.efi

You’ll notice that we now use these “Windows-style” flags, starting with slashes. That’s because we’re invoking lld-link, which is the Windows backend for the LLVM linker. The /subsystem flag specifies what environment the executable is supposed to run under, so that the linker understands how to link it up. One major thing this actually affects is a field in the executable header stating that it’s an EFI application; if we don’t do this, UEFI will just refuse to load it. entry and nodefaultlib are self-explanatory enough, I reckon.

Now all that’s left to do is to actually construct the disk image and place the EFI application’s executable in the right place inside!

Creating the Disk

# Make a 65MiB disk
dd if=/dev/zero of=disk.img bs=1M count=65

# Format the disk. My disk_layout.txt can be found in the attached repo, at
# https://codeberg.org/j4cobgarby/os_blog_posts/src/branch/main/1_helloworld/disk_layout.txt
sfdisk disk.img < disk_layout.txt

# Format the first (and only) partition as FAT32
mkfs.fat -F 32 --offset 2048 disk.img 65536

# Make directories and place in the application!
mmd -i disk.img@@1M ::/EFI
mmd -i disk.img@@1M ::/EFI/BOOT
mcopy -i disk.img@@1M boot.efi ::/EFI/BOOT/BOOTX64.EFI

One pretty cool thing to note in the above is that we can make the DOS tools like mmd and mcopy directly work on a FAT filesystem on a partition of a disk image by just specifying the offset into the image where it begins. That’s what the double-‘@’ syntax means.

Emulating with QEMU

Though you could just whack your newly created disk image onto a USB stick and boot from it on a real computer, it’s easier for rapid development (plus safer) to emulate it. For this, I really like QEMU. Not only does it emulate a machine with configurable hardware attached, it even provides a GDB stub so I can debug low-level kernel code (though I’m having trouble getting GDB to work with the UEFI PE executable; if anyone knows how to do that from Linux, please send me an email!)

The only problem is that QEMU doesn’t ship with a UEFI firmware implementation, so we have to give it one. There’s a really cool project called OVMF which produces a UEFI firmware implementation for QEMU, but I find the links on their Github to mostly not work when it comes to actually downloading a pre-built copy of the firmware.

I could build the firmware all myself, but then I need a few dependencies, and so I find it much easier to download the OVMF package distributed by Debian! However you do it, you need to end up with an OVMF.fd (or similar) file to install in QEMU’s emulated flash memory. Do take a look at the Makefile in this post’s code repo to see how I extract it from the Debian distribution.

And finally, the moment we’ve all been waiting for: let’s run our very own UEFI application!

qemu-system-x86_64 \
    -drive if=pflash,format=raw,readonly=on,file=OVMF.fd \
    -drive format=raw,file=disk.img \
    -m 1G

And of course, here’s a screenshot of our Hello World application in its modest glory.

That’s all for now; thanks for reading! Feel free to browse the code, and make sure to stay tuned for the next installment, in which we will see how to parse and load an ELF executable from the disk!

The Paper

The paper I presented at Euro-Par was titled “Interval-Asynchrony: Delimited Intervals of Localised Asynchrony for Fast Parallel SGD”. It’s the first paper I managed to write and get published during my PhD (so far!)

Watch the 20 minute presentation – or even better, read the paper – for more details, but here’s an overview (simplified, but should still get the point across). Basically, the problem is that we wanted to use the stochastic gradient descent algorithm (SGD) to train a neural network, which in our case is being used for image classification. If you want to speed up this algorithm, you can run it in parallel, which just means that you split up the work between a number of workers, which in our case are CPU threads.

The SGD algorithm consists of a number of iterations, and each iteration depends on the result of the previous one. At least, it does if you want it to be completely accurate. Then, the natural way to parallelise this process is to split the work of each iteration across all your threads, wait for them all to finish their piece, and then combine those results together somehow. Don’t worry about the details, other than what I’ve already explained.

So, this works, but some threads are inevitably at least a tiny bit faster than some others, and so they must spend some time doing nothing while waiting for slower ones to finish. This discrepancy adds up when you use hundreds of threads, and significantly limits the speed at which you can train your neural network.

The other side of the spectrum is asynchronous execution, where you basically just let all the threads run iterations independently, with no synchronisation. They all see the same neural network parameters in memory, and can just update them whenever they want. It’s really fast in terms of iterations per second, but significantly less accurate than synchronous execution, so maybe more iterations are needed to train the model.

We (and by we I mean my PhD supervisor Philippas Tsigas and I) thought that this trade-off was interesting, and designed a new type of execution scheme which has a parameter to determine “how asynchronous” you want it to be. This is really useful, because actually it turns out that as you get closer to convergence, you want more accurate updates, and so you can just change this parameter over time!

In terms of how exactly the amount of asynchrony is enforced: the idea is that we divide the whole execution into a series of “asynchronous intervals”. Threads run normal asynchronous execution during an interval, but any iterations that overlap two intervals are just ignored, i.e. not used to update the model. An interval ends when a certain number – that parameter that I mentioned a minute ago – of iterations have been completed during it. The point is that we can bound the inaccuracy by keeping this interval small, or we can widen it and speed up the throughput by rejecting fewer steps. Again, please look at the paper for proper details.

Anyway, it turns out that this is a pretty nice way of adjusting the amount of asynchrony, which is useful when paired with a clever way of setting that interval size parameter (which we also did in the paper!)

The Conference

As I mentioned, it was my first proper conference, so I didn’t really know what to expect. I was quite looking forward to presenting my paper, and so I spent a couple of the evenings after the conference practicing it a bit. Weirdly – and this seems to happen in general – the more I rehearsed it, the more I forget what to say while rehearsing, but it always seems to be the case that once I actually go to present it for real it goes fine, more or less.

The conference itself was really well organised, and I met some really nice people there. In fact, I didn’t expect I would manage that much networking and socialising – it’s not something I do very much – but even on the first day (which was workshops rather than the main conference) I met a couple of nice people who were also going to be presenting their work. I think the fact that Euro-Par’s scope is quite broad (parallel programming of all kinds, really) made it a lot more interesting: if every presentation was in the same niche as mine, then it might get a little boring? But the theme at least meant I had some kind of understanding of the other works, at some level.

Hopefully I’ll get another paper accepted there in the next few years so I can go back :)

First, here’s a goldeneye I saw yesterday. I went out with a tripod this time, and it was amazing the difference it makes when using my super long lens. It’s so much easier to keep it still.

These next three are from today. I came across three different birds: a fieldfare, a goose, and a great tit. Actually, I hadn’t heard of a fieldfare before, so I had to ask a friend to ID it for me. It’s a type of thrush, and in Swedish is called a björktrast (birch thrush), which is fitting.

Usually geese look pretty mean, or at least fed up, but this one looked a bit kinder. Maybe cause of the crocuses in the foreground. I’m sure it would still hiss at me if I got closer.

Got a bit lucky with the focus in this shot – it was a bit tricky with all the tiny branches in the way, and he really didn’t sit there for long. This was next to a bird feeder, so there were several tits congregating.

For most of winter I haven’t been going out and taking any photos. It’s been too dark, and (almost) all the birds have migrated south.

Last weekend, however, I took a very short – five minute – train up to Jonsered, and got a few nice images.

Here we can see the train to Alingsås approaching. I thought this was a pretty cool picture with that blurry chap in the foreground. In fact, the reason he’s quite so out of focus is that I was using a super long lens here, my 150-600mm Tamron, at close to full zoom. It’s a really fun lens to use, but really really heavy, and slightly hard to hold still. The active stabilisation helps.

The last photo for today is the following magpie.

This demonstrates a bit better what this lens is capable of. It’s not completely sharp, especially around the magpie’s dishevelled facial fluff. I think this is likely to be my fault though, I still haven’t quite got the hang of what kind of shutter speed I need to take sharp hand-held photos with this.

Funnily enough, although I did get to Jonsered for my walk, I didn’t get any good photos there at all. The magpie was just sitting on a street-tree, so to speak.

Site Update

I thought that it wasn’t ideal to provide full-res versions of the images on these blog posts, just because they’re tens of megabytes and will add a lot to page load times. I fiddled around a bit with Jekyll (and a little Python script) so that I store a thumbnail version and a full version. Now, if you try clicking on one of these images you’ll get to download the full version!

I had to get a bit familiar with how Jekyll uses Liquid to do this, but it came down to this:

module Jekyll
  class Thumbnailer < Liquid::Tag
    def initialize(tag_name, text, tokens)
      super
      @img_name = text.strip
    end

    def render(context)
      thumb = @img_name.sub(/\.(\w+)$/, '_thumbnail.\1')
      "[![](#{thumb})](#{@img_name})"
    end
  end
end

Basically, whenever I write something like [![](my_image_thumbnail.jpg)](my_image.jpg) it will substitute this for a bit of markdown consisting of an image (my_image_thumbnail.jpg) within a link to the original image. It works quite well and is a lot easier for me than if I had to type that out manually every time.

For actually creating these thumbnail images, I use Python with Pillow. If you’re curious or want to use it yourself, you’re welcome to look at it here. Right now I just run this script manually when I upload new images, but in an ideal world my ruby code would be generating thumbnails when needed, on the fly. That’s a project for another day, I reckon.

Recently I bought a new (used) camera, the Canon 80D. I used to use a Nikon D3300, but it disappeared while moving. Anyway, this new one is quite a bit nicer. The most noticeable differences for me are a far better live view mode (useful when I’m taking pictures at awkward angles) with a moveable touch screen (compared to the D3300’s static non-touch screen), more AF points when using the OVF, and a higher maximum frequency for taking photos.

Leaves

This was from the first time I went on an outing specifically to take pictures with this camera. I used the lens that came with it: the Canon EF-S 18-135mm f/3.5-5.6 IS USM. It’s in Trädgårdsföreningen, Göteborg. I think I got quite lucky with the weather on this day; some of the last warm sun we’ll have this year, I would guess.

Ducks

The following three images are also taken in Trädgårdsföreningen, but on the next weekend. Mainly I was playing around with fast shutter speeds to try and capture the birds doing interesting things. The main challenge was trying to get the pictures to be sharp, which was difficult since I had to increase the ISO to make up for the shutter speed.

All of these photos are taken with the same 135mmm zoom lens mentioned above.

The duck above had just dived underwater and resurfaced. It’s quite cool seeing the water run off the hydrophobic feathers.

This duck was relaxing on a protruding wooden stake sticking out of the river, although I cropped the photo quite close to see how much detail I could see. I could have got a much better quality image here if I used a shutter speed closer to, say, 1/800s. 1/2000 meant that I had to use a very high ISO, which wasn’t ideal. Not sure why I did that…

I quite like this one, but there’s still a lot to improve on for next time. Opposite to the last photo, this one should have been shot with a significantly faster shutter to reduce the blur around his wings. I would have also liked to get a bit further down to the bird’s eye level. Partly this is just to get some more separation from the background.

Trees

Here are a couple of photos from a nice forest just a few minutes from my apartment. I took these yesterday, to experiment with a second-hand lens I bought: a Canon EF 50mm f1.8 STM.

This is the first prime lens I’ve ever used – that is, it has a fixed focal length. Another thing to note is that it’s an EF lens, which means it’s for full-frame cameras. My Canon 80D is not full-frame, which means that I only “see” a section cropped out from the middle of what the lens sees. This is both a good and bad thing, depending. This 50mm lens becomes effectively 80mm for this reason, which can be good or bad.

Since lenses tend to be sharper nearer the center, in theory I should get sharper results with this than using it on a full-frame camera. On the other hand, if the lens was just sharp enough to produce a sharp image on a full-frame sensor, then the fact that I’m zoomed in could reveal softness.

I was quite happy with this photo. Usually when I take photos of trees in scenes like this, the composition isn’t great and the picture looks boring. Here, the clearing between the foreground bushes and the background forest makes it a bit more interesting.

This is part of a hiking trail through the same woods as the previous photo. Everything looked very yellow here because it’s Autumn, and this particular type of long grass was everywhere in this boggy area of the route.

These wooden planks were for getting over those boggy areas, and in one part they have a long long path made of them which crosses a big reedy pond.

I’m not sure exactly what I think of this lens. In these two photos it was fine, but when I had the aperture very open (1.8-2.0) and had bright highlights, the lens was very prone to pink-coloured haze around these objects. This is something I can work around though, and I got the lens for a pretty good price, so I can’t complain :)

I’m not sure where the game came from originally – probably it was based on some other game, but we added several of our own rules to it over time. I think it’s a little like Uno, conceptually, although I’ve never played Uno.

Brief Overview

The aim of the game is to be the first to get rid of all of your cards.

In one turn you may play some cards according to the rules listed below. In a turn you can play a potentially large number of cards, and can also do things that cause other players to pick up new cards.

Setup

Each player is dealt cards from a shuffled deck, until there are no cards left.

Like most card games, the players should sit in a circle-like formation around a table.

An arbitrary player goes first. It doesn’t matter at all, but it’s common to pick whoever is to the left hand side of the dealer.

To begin with, turns proceed clockwise around the circle. Note that this direction can change later in the game.

Structure of a Turn

Picking up cards

At the start of your turn, first consider whether the last player’s turn caused you to pick up some cards. This can be the case if they ended their turn by playing a pick-up card (a 2 or a Jack). If you don’t, simply skip this step.

If you do have to pick up some cards, you can play an Ace to cancel this, thereafter playing your turn as usual as if starting normally with this Ace. Alternatively, you can play one or more other pick-up cards (following the normal rules of playing cards) to prevent picking up anything yourself, but adding to the amount of cards to pick up for the next player.

Failing either of these, you must pick up the correct number of cards. Your turn ends immediately after doing so.

Playing Cards

This subsection describes the basic rules of playing cards. It ignores most special cases caused by playing power cards, which are described later.

1. Play a card which matches the suit or rank of the last card played. If you can’t, you have to pick up a new card and end your turn now.
2. You may end your turn here, or play more cards. If you want to play more cards, you have to start by playing one or more cards of the same rank as the card you played in step 1.
3. If you decided to play more cards, you must now go back to step 1 in order to play at least one more card. This is called “capping off”.
   • Capping off your turn is not required after playing Jacks or 2’s. This is to facilitate ending your turn on pick-up cards, since they otherwise have no effect.
   • Notice that this allows you to loop potentially many times, playing lots of cards. Importantly, it also means that you cannot end a turn playing exactly two of some rank (without picking up a new card), but can with three or more (because the third, etc., can count as the additional card played in step 1 after looping.)

Winning

To win the game, you have to get rid of all your cards so that you have an empty hand.

You cannot win by playing a power card as your final card. In this case, you have to pick up one new card.

Power Cards

Some cards act differently when played. They’re known as Power Cards. They are: Ace, 2, 3, 8, J, Q, K.

For each Power Card, the corresponding operation occurs at the moment that it’s played (during a turn). Some Power Cards have additional rules about when they may be played.

Ace (Choose new suit)

• Can be played on top of any card in step one of the instructions.
  • It cannot act as just any rank, when it comes to step two. This means that you couldn’t play 3h Ah Ac as a turn.
• When played, you can specify any suit. This card now acts as if it is that suit, for the purposes of cards being played on it in step one.
• As mentioned previously, if played first in a turn it can negate picking up cards from previous pick-up cards.

Two (Pick up 2)

• Add two to the number of cards that the next player must pick up.
• As with all pick-up cards, it’s only effective if played as part of a consecutive run of pick-up cards at the end of a turn.

Three (Must play something valid)

• You must immediately play any matching (suit or rank) card on top of it.
  • (Or pick up one)
• You can think of this as resetting the turn back to step 1.

Eight (Skip a person)

• Skip the next person who would have had a turn, this round.
• Multiple eights can be played in one turn, in which case several people can be skipped.
  • You can skip so many people that it’s your turn next. This can lead to interesting scenarios.

Jack (Pick up)

• Add something to the number of cards that the next player must pick up.
  • If it’s a red Jack, add 3.
  • If it’s a black jack, add 5.
• As with all pick-up cards, it’s only effective if played as part of a consecutive run of pick-up cards at the end of a turn.

Queen (Must play anything)

• You must play some card on top of it. You’re allowed to play anything.
• Similar to the Three, it effectively jumps to step 1. The difference between the Three is that you can play any card on top of the Queen.

King (Reverse direction)

• Reverse the direction at which turns progress, flipping between clockwise and counter-clockwise.
  • This can interact with Eights in a strange way. Eights skip in the current turn direction, which can change during the course of a turn.

Notes for Playing

Moving Cards from the play pile to the deck

At the start of the game, there is no deck from which to draw new cards, since all of the cards are in the players’ hands. Of course, at some point, such a deck will be necessary, so that players can pick up cards when they are required to.

To facilitate this, it’s necessary to occasionally remove all but the top card of the play pile and shuffle it into the deck.

Still, there could be a situation where a player must pick up more cards than are in the deck and that are available to move into the deck from the play pile. In this case, you can either:

1. Pick up as many cards as possible, exhausting the deck and play pile or
2. Do as in option 1, but remember how many cards you still need “in debt”, and pick them up as and when they become available.

I can’t remember which option we used when we played. Probably 1, since 2 sounds like a hassle…

Example Turns

If I’m playing first in a game (i.e. the play pile is empty), then here are some simple turns which do not result in the player having to pick up any extra cards themselves.

Example 1. 9h (9 of hearts)
Example 2. 9h 9c 4c
Example 3. 9h 9c 3c 4c
Example 4. Js Jc Jh

And here are some examples which, due to not “capping off” the turn, the player must pick up one card from the deck after playing.

Example 5. ø (no cards played)
Example 6. 9h 9c

Here are some moves which utilise power cards, showing how different ones interact with prior pick-up cards. In the following set of examples, the previous turn was 9c 9h Jh, hence three cards are to be picked up, and Jh is at the top of discard.

Example 7. ø <pick up three cards from deck>
Example 8. Ac
Example 9. Jc

Here’s an example of a particularly long turn.

Example 10. Ks Kc 6c 6h 6d 6s Qs 8h 8d 3d 4d 4h As Ac Ad Ah Jh Jd Jc Js

To clarify some points about example 10:

• Two kings are played at first. This switches next-turn direction twice, so it’s unchanged.
• Two 8’s are played, thereby skipping the next two players. This takes effect after this turn.
• Four Jacks are played at the end, forcing the next player whose turn it is to pick up 3+3+5+5=16 cards if they’re unable to cancel it with Aces (which they can’t, since all the Aces were played in this turn), and unable to add to the pick-up with at least a 2 of spades.

There are quite a few approaches for generating mazes algorithmically. I particularly like the graph based approaches, since they seem most adaptable. They are:

• Using a MST algorithm (like Prim’s or Kruskal’s) on a randomly weighted graph.
• Performing a depth-first search on a graph, with some randomness built into the algorithm.

Maybe there are even more graph-based maze generation methods, but I don’t know about them. There are of course some non-graph-based methods, like iteratively splitting a space into randomised chunks and cutting holes in the new walls¹, or even using cellular automata², which are also interesting.

The reason I say that graph based approaches are most adaptable is because of their independence on any particular shape or structure. You’ll see what I mean by this later on.

Defining a Maze

First it might be a good idea to define what a maze is and isn’t. For example, consider this:

Imagine that the black lines are passageways that you can walk through. This definitely seems like a maze. Here are some properties of this specific maze:

• No loops/cycles: there is exactly one route from any one location to any other.
• There are no disconnected sections: any point is reachable from any other point.
• No obvious-at-a-glance route from a point to another far-ish-away point.

The last of these is the most controversial of the three, since it’s very difficult to quantify how difficult a maze is. However, here are some features of good mazes that probably help:

• No repeating patterns, as these would make visually parsing the maze more manageable.
• No excessively long straight corridors/lots of bends makes following paths more tricky.
• Many points where passageways branch, meaning that someone solving the maze has to check a lot of different routes.

Probably none of these features are necessary for something to be a maze, though. Loops are definitely possible in mazes, there’s no reason why not, and the only reason they’re not present in the above maze is because it is specifically a spanning tree of a graph. Mazes may in theory have disconnected sections, I suppose, but in that case it would really just be two (or more) mazes next to each other. Also, mazes don’t need to have a non-obvious solution – take the kids mazes on the backs of some ceral boxes, for example.

In fact, mazes don’t even need to have branches. Some hedge mazes³, for instance, are just one long twisting path from start to finish. And should all mazes have a start and a finish? The maze presented above doesn’t have a specific start or finish, but the nature of it means that any two points on it can reasonably be chosen for these. I would say that a maze has to have a defined start and finish, or be constructed such that they can be chosen arbitrarily.

I think a reasonable definition for a maze is just “A passageway (or collection of passageways) connecting a start to a finish”. Then if our maze has starts and finishes that can be chosen arbitrarily, we basically have a set of mazes with the same construction but with different start and finish points. Anyway, this doesn’t matter.

Depth-First Search

The maze-generation algorithm I’m going to talk about now is depth-first search (DFS). This algorithm is used for many more useful purposes than generating mazes, but this is what I’ll use it for.

DFS takes a graph (a collection of nodes and edges between them) and a starting node, and, in a certain order, examines each of these. The reason it’s called “depth-first” is that it starts by traversing one path for as long as possible before going back to try another one. This is in contrast to breadth-first searching which, before examining any node which is n nodes away from the starting node, examines all those which are up to n-1 away.

This post isn’t meant to be a detailed depth-first search tutorial – you can easily find those anywhere on the internet. Even still, here’s some rough pseudocode for one iteration of the algorithm, as I implemented it for this purpose:

PROCEDURE dfs_step
INPUTS:
  G: The graph to search
  S: A stack data structure.

WHILE S is not empty
  next = S.pop()

  IF `next` has not yet been visited:
    Visit `next`

    FOR EACH edge IN `next`'s neighbours:
      S.push(`edge`)

    RETURN

This is slightly different to standard iterative DFS implementations, in that the procedure only executes one step, by which I mean it runs until it either exhausts the stack or a new node is visited. Normal implementations run until the entire graph has been visited.

The reason for this difference is that I wanted to display a realtime animation of the algorithm running, and this implementation allows me to periodically run a step (say, every half a second) and update the graphical representation accordingly, without the algorithm itself needing to know anything about this.

To summarise the pseudocode, we repeatedly pop nodes from the stack until we find one that we haven’t visited yet. Then, we “visit” it, push all of its neighbours to the stack, and return, i.e. end the procedure. A traditional DFS implementation won’t return at this point, as I mentioned earlier, and instead will repeat until there’s nothing left on the stack.

The meaning of “visiting” a node depends on what the DFS is meant to achieve. For example, another use of DFS is finding connected components within a graph. To do this, we can just begin a DFS at an arbitrary node, and efficiently find the connected component that contains it, then repeating the process on an unclassified node. In such a case, “visiting” a node would be adding it to a list of nodes in some structure representing the connected components of a graph.

What about generating a maze, though? If we run DFS on any graph and make a note of the order in which nodes are visited, we can construct a tree containing every node. Let’s run through a quick example.

The first image shows the plain graph which we’ll operate on. I generated this graph by randomly placing 20 nodes in a 500×500 unit space, and mutually connecting any two nodes which are closer than (or exactly) 150 units apart. The second image visualises the very first iteration of my DFS step procedure. It’s starting at the top-left node, where the orange and green lines are protruding from, which represent all of the node under consideration’s neighbours. The actual implementation pushes the edges such that the closest edge is at the top of the stack.

In the third image, in bold, the resulting spanning tree is shown. It may not look much like a maze at this point, but the point is we can run this on any graph, and certain graphs will yield better mazes. In fact, the big square maze presented earlier in this post was generated in the exact same way as this tree. The primary difference between these two, and all the other mazes I’ll make, are in the way the graph is generated initially, although I make a couple of extra modifications to the algorithm as well, for some.

Grid-like Maze

A very standard type of maze is square-shaped, with horizontal and vertical passageways. I can very easily write a function to place nodes in a square grid, and join them to their correct neighbours. Then I can just feed this graph into the DFS algorithm, and…

Oh, it’s not a very good maze. Going back to my thoughts earlier about what makes a (good) maze, we can see that this one has no cycles and no disconnected sections, like the earlier maze, but, it’s very easy to solve this maze from any point A to any point B. In fact it’s trivial, because this maze has no branches at all – it’s topologically equivalent to a straight line. Starting from A, you can always get to B, regardless of where B is, by just trying each of the two opposite directions. In fact this maze is even worse than just the general case of mazes topologically equivalent to straight lines, because it has a consistent repeating zig-zag pattern going downwards, so if you know the positions of A and B you can immediately know whether you have to travel up or down in space, and hence work out if you need to start moving left or right. In other words, with this maze, it would be possible to implement a function Node next_node(Node current_node, Node to_find) that runs in O(1) time, given that we can view the structure of the maze from above.

The reason that our algorithm generates a zig-zag maze here is just because we don’t involve any randomness yet. The procedural method by which we generated our graph appends neighbouring nodes to a list contained within each node, and this list is iterated in order when pushing to the stack in the DFS algorithm. The way that this list is generated means that neighbours are pushed in the order (up, down, left, right), and so if a right-facing neighbour is present, this will always be preferred (because it will be at the top of the stack at the start of the next iteration). This explains the intial long-as-possible right-facing path. It only stops moving right when it can’t anymore, because that neighbour is then not pushed to the stack.

The solution is simple though, we can just shuffle each node’s neighbour list at initialisation time. And this pattern has the added benefit that it’s adaptable to different patterns that we might want. For example, remember what I mentioned earlier about picking the neighbour closest to the current node, when we saw the DFS algorithm running on a more freeform graph layout? Well that behaviour doesn’t have to be impemented within DFS at all, keeping it more generalised. Instead we implement that in the same way as this neighbour shuffling, by sorting each node’s neighbours list by distance, specifically for graphs where we want this shortest-first behaviour.

Anyway, if we implement this neighbour shuffling, we get this:

This is significantly better. Now, we have lots of branching – this occurs when a search path gets to a point where it cannot go any deeper, so it pushes no neighbours for the next iteration and instead some neighbours that were pushed by an earlier iteration are taken instead.

This maze still isn’t ideal though. The branches, although many, are not, on average, very deep, which makes it easy to quickly disregard many of them when solving the maze. I think this is just because this particular maze is fairly small. If I generate a larger one, the branches become far longer.

Customising The Maze

It can be fun to fiddle with the code a bit to get some different interesting mazes. With our graph-based maze approach, there are a lot of options here, because we simply apply our algorithm to any graph.

Grid-Like Mazes

First, let’s look at different ways we can modify our existing grid maze.

The only change needed to generate these three mazes is to change which nodes count as neighbours of other nodes. Before, a node only neighboured adjacent nodes cardinally north, east, south, and west of it, but this can easily be changed when generating the graph.

For the first of these three examples, the neighbourhood includes all eight adjacent nodes, including diagonals. This produces maybe a more “organic” looking design. It also kind of reminds me of runes in some fantasy game?

In the second example, the four cardinal directions are included, as well as north-west and south-east neighbours. This creates angles paths, and almost looks like a normal maze has just been skewed.

The third example is similar, except depending on the node’s position on the y-axis the “skew” direction is either left or right.

“Loose” Graph Layout

As shown earlier, we can generate a maze on top of any graph, not only rigid grids. For these next tests, I place 1000 nodes randomly in a 800x800px world.

To determine the neighbours, the first of these two examples finds all other nodes which are within a certain threshold distance (in this case 100px), and uses this set as its neighbourhood. The ordering of these neighbours (which determines how DFS will traverse the graph – remember how we randomised the ordering of neighbours for the rigid grid earlier) is simply their distance. The shortest node comes first in the neighbour order.

This means that we get lots of winding, intricate paths, since it’s biased towards picking out small details.

In the second example, the set of neighbours is generated identically (though with a different random seed). The only difference is that instead of ordering the neighbours by their distance, they’re just shuffled (exactly like the earlier rigid grid examples). Since here we have a combination of a fairly high threshold and densely packed nodes, we get a bit of a mess here. I think it looks like the creases of scrunched up tin foil.

If you experiment with this yourself, it’s fun to play with the density and threshold parameters for the loose graph. If the nodes are too sparse, though, there is a chance that you’ll encounter disconnected regions, meaning that the whole graph will not be part of the maze.

Some nice things to do to extend this would be:

• Some constraint for a minimum distance between two nodes. At the moment, two nodes may spawn almost touching, which can make the resultant maze look like it contains cycles even when it doesn’t.
• A neighbourhood ordering function which takes into account distance but applies some randomness on top of this.

Aside on overlapping passages: When I first ran a version of this “loose maze” generator, sometimes passages would overlap. This was of course because the neighbourhood generation didn’t take this into account, and the DFS algorithm itself didn’t check for overlaps either.

I had the option of either ensuring no overlaps when generating the ‘substrate’ graph, or making a DFS which is aware of this. I decided on the latter, since whether an edge may exist without overlapping only depends on previous edges that have been visited.

Other Fun Shapes

The current state of the program supports grid mazes and loose mazes, and some code to try and generate a labyrinth-like maze (where nodes lie on concentric circles, with a few edges between them). The labyrinth generator isn’t working that well at the moment – ideally, I want fully the concentric circles to have long unbroken sections, with far fewer inter-ring connectors.

Below, my radial graph is on the left, and on the right is something that I would like mine to look like! One major difference here is that the target image uses lines as walls, whereas mine uses lines as passages. I think that DFS might not be the correct algorithm to use to generate this sort of maze, anyway.

If you’re interested in this, you should definitely have a go at writing your own code to generate graphs. Here are some ideas:

• Fix my labyrinth implementation.
• Make a maze which resembles an image, or some text.
• Generate a voronoi diagram, and use its interface lines as edges.

You can find the source code on GitHub!

Final Thoughts

Depth-first search on a graph can generate some quite nice custom mazes. Since it’s “depth-first”, though, there is some bias towards long paths, which may or may not be desirable. Other algorithms (such as a greedy MST algorithm or a breadth-first search) will have different characteristics in this way, which might be better for some applications.

References

I plan on posting about anything I happen to find interesting, which will hopefully in the near future consist of working out the design for a kernel I’m going to make. I also have a firefox bookmark folder full of some interesting and cool papers I found, which I plan to write a bit about.

Hopefully this website will be an enjoyable experience.