Accidentally becoming a bootloader dev

The First Sin

Anyone who's ever programmed in the kernel knows that it's a bit of a pickle to work with. Unlike in user-space, you're very very close to the metal, you're working with raw memory addresses, you're fucking around and abusing the alloca()­ function­ [2] and unwinding your stacks manually to save three cycles on some signal return code that your compiler foolishly tried to play it safe with. This is all really cool stuff that you can tell your friends and they'll be really impressed (if they're absolute geek fucks who have never seen a titty in their lives), and you'd be the star of the evening.

After three good mouthfuls of whisky you're starting to feel a bit doozy, and you notice the man in the corner of the room. Who is he? How did he get here? If you had any bitches, he'd be scaring them away right this fucking now. You don't have hoes, and the booze has made sure you definitely have no inhibitions either, so you come up to him and strike up a conversation.

'Hey, kiddo', he goes, 'do you want to learn about the dark dark arts, or are you just forever gonna be a cuck and live in the high-level lands?'

The way he says it oozes with testosterone and the smell of stale sweat, and you just know you've sold your soul to the devil.

int foo(char *p, int a, int b) {
    int t1,t2;  // dummy output spill slots
    int r1,r2;  // dummy output tmp registers
    int res;

    asm ("# operands: %0  %1  %2  %3  %4  %5  %6  %7  %8\n\t"
         "imull  $123, %[b], %[res]\n\t"
         "mov   %[res], %[spill1]\n\t"
         "mov   %[a], %%ecx\n\t"
         "mov   %[b], %[tmp1]\n\t"  // let the compiler allocate tmp regs, unless you need specific regs e.g. for a shift count
         "mov   %[spill1], %[res]\n\t"
    : [res] "=&r" (res),
      [tmp1] "=&r" (r1), [tmp2] "=&r" (r2),  // early-clobber
      [spill1] "=m" (t1), [spill2] "=&rm" (t2)  // allow spilling to a register if there are spare regs
      , [p] "+&r" (p)
      , "+m" (*(char (*)[]) p) // dummy in/output instead of memory clobber
    : [a] "rmi" (a), [b] "rm" (b)  // a can be an immediate, but b can't
    : "ecx"
    );

    return res;

    // p unused in the rest of the function
    // so it's really just an input to the asm,
    // which the asm is allowed to destroy
}

The Second Sin

But this wasn't enough. I lived a bit in assembly land but generally avoided it because I hated recalling the exact sequence of jumping through hoops every time I wanted to do anything useful with my assembly. Did you know that inline assembly from within C++ used what's realistically, but not theoretically, a different calling convention than C code, except in the most trivial of cases? [6] Did you also know that the OS will also execute your code in a nondeterministic way? [7]

Calling printf() was a chore, using BIOS interrupts to write text with was a different chore (who knew that int 10h was so messy on x86?), and don't you even dare think about doing anything graphical with modern hardware.

No, my second sin was getting a simpler piece of hardware—I got an Arduino Due and, unfortunately, decided to play with it in assembly. Things got really fun and I ended up absorbing more knowledge about the board specs than I really should have (who knew that it was that easy to use clock mismatches as a RNG that's better than the one provided by the manufacturer?).

The Due is a 'simple' board controlled by a ATSAM3X8E chip based off the Arm Cortex-M3 core, running on the ARMv7-M instruction set. The 32-bit ARM instruction set comes with a whole bunch of goodies that you wouldn't see the x86 be caught dead with: all instructions work with all registers (cf. the x86 trying to do mul), instructions are fixed-width (meaning you can reason about code size even without assembling!), and conditional code does not require any jumps since practically all the instructions have a large number of conditional variants.

But to program the Due you needed practically no fancy magic, you can just get the board, write absolutely tiny code (as in, your programs will rarely pop a couple of kilobytes in C mode), flash it with your junk code, and just power it up. Unlike the x86 platform, the startup sequence quite literally is 'start at 0x0, execute code, interrupt vectors are in the words immediately after the start position'. There's no juggling with real, unreal or protected mode, no boot sectors, no trying to claw your way out of 16-bit space to 32-bit.

The compiler the development suite gives you handles most of the work for you admittedly, but if you pilfer the headers for a few magic numbers and the correct sequence of memory positions to blast, you can actually get the Arduino going doing genuine work off an assembly file.

But getting output going beyond 'turn LED on' and input beyond 'button was pressed' was both going to be an electrical chore to wire, and a bit expensive (screens and whatnot compatible with the Arduino didn't come cheap at the time), so in my majestic foolishness I decided I wanted to do Arm assembly programming on my x86 desktop. This necessitated downloading, installing, using QEMU.

The Third Sin

So, QEMU is an emulator suite for a bunch of architectures, and a bunch of computers based off them. In the Arm family it emulates a couple dozen boards, all of them imperfectly and partially, and all of them in a very underdocumented way: to get your bearings you have to read things like the actual source comments (usually outdated), the RedHat mailing list (—''—), schematics of the boards (sometimes just missing), of the devices those boards include (...) etc.

So I decided that the next best thing would be to do raw, bare-metal programming on the virt board—a fake QEMU platform whose components you can pick and choose yourself, and that's guaranteed to have the best emulation experience because it's not tied to actual hardware demands (did you know that the Raspi boards are booted through their black-box GPU that runs a binary blob kernel that we have no idea as to how it works?) and you can just pick and choose parts and QEMU will try and make them work. The virt board comes with some predetermined parts as well so you don't have to do all the picking and choosing, which is cool.

And then you realise, again, that there is absolutely no documentation for anything you want to do, that nobody's publicly written about what they did to enable the things you want, and that there is between 'practically no' code and 'no' code out there that does what you want to do.

So the first thing I wanted to do was to write a sort of Hello World to see any output being displayed. This meant any sort of output whatsoever from the board to the 'outside world', which for now meant getting it to write to console.

At this point I was still oblivious.

The way that these kinds of boards communicate with the outside world is, at its most basic level, just getting a hose of bytes and blasting a memory location in a specific way until something you want to happen happens. To get the virt to shit out text, I had to talk to the UART device which QEMU helpfully semiautomatically maps to stdout / the console. So you go and read the documentation, realise you're out of your depth, the links slowly turn from blue to purple, and it takes you three days to realise what a mess it all is.

The UART is a memory-mapped input-output device (MMIO) that's mapped to a special memory address somewhere between the flash space (0x0) and start of RAM (here that's 0x40000000) and the MMU pretends that it's a real memory location and not a fake cop-out redirection. So, to get the thing going I had to:

1. figure out how the fake MMU maps devices

2. learn about the dtb (device-tree blob)

3. learn how the UART device works in the real world

4. look at UART set-up code for the Raspi3

5. translate this into the virt UART specs

6. figure out the location of the virt UART

7. ???

8. profit?

One of the first iterations of this code looked give-or-take like this (magic numbers not included):

ldr             x0,  =AUX_ENABLE
ldr             x1,  [x0]
and             x1,  x1, 0x1
str             x1,  [x0]
ldr             x0,  =AUX_MU_CNTL
str             xzr, [x0]
ldr             x0,  =AUX_MU_MCR
str             xzr, [x0]
ldr             x0,  =AUX_MU_IER
str             xzr, [x0]
mov             x1,  0x3
ldr             x0,  =AUX_MU_LCR
str             x1,  [x0]
mov             x1,  0xc6
ldr             x0,  =AUX_MU_IIR
str             x1,  [x0]
mov             x1,  0x48
mov             x0,  =AUX_MU_BAUD
str             x1,  [x0]

ldr             x1,  =GPFSEL1
mov             x2,  0x3F000
and             x1,  x1, x2
mov             x2,  #1152
mov             x3,  #64
mul             x2,  x2, x3
orr             x1,  x1, x2
ldr             x0,  =GPFSEL1
str             x1,  [x0]
ldr             x0,  =GPPUD
str             xzr, [x0]

mov     x2, 0xA0
_loop_1:
        sub             x2, x2, #1
        nop
        cbnz            x2, _loop_1

mov     x1,  0xC000
mov x0,  =GPPUDCLK0
str     x1,  [x0]

mov     x2,  0xA0
_loop_2:
        sub  x2, x2, 0x1
        nop
        cbnz x2, _loop_2

str     xzr, [x0]
mov     x1,  0x3
ldr     x0,  =AUX_MU_CNTL
str     x1,  [x0]

Basically, you 'have' to make sure the device is enabled and ready to accept data from you, you need to map its 'pins' to MMIO addresses the MMU exposed to you, etc etc etc. This specific snippet crashed and burned, so I spun the wheels for like a week and eventually settled on like 3x the amount of code only to get the ability to blit a single byte at a time at this device, passing its contents to the stdout in the dumbest and most unsafe possible way since I think I just stopped caring about things at that point:

mov x0, 0x40
ldr x1, =AUX_MU_IO
str x0, [x1]
add x0, x0, #1
str x0, [x1]
add x0, x0, #1
str x0, [x1]
b.

No checking whether the device is ready to accept another byte, no checking if it's written or has errored, no gods no kings.

I was still oblivious, yeah?

So having enabled the UART and gotten the proverbial Hello World out (though ofc I hadn't wrtten a string printer just yet, it would've been trivial though), I figured might as well figure out the other devices and see how they get set up.

The first one on the list was the framebuffer. Basically, the dumbest possible way to blit pixels onto a surface is to set up a framebuffer device with the appropriate dimensions, pitch (how many bytes per row of pixels), pixel format (i.e. bits per colour and order of colours) and give it a chunk of RAM to read from to the screen.

The setup for this was even more hellish than the above, since the documentation was actually totally absent, so I had to resort to the aforementioned RedHat mailing list, and reading the source for SeaBIOS and the U-Boot bootloader, and then read some more QEMU source code (why does the documentation actually suck so much?) etc etc. In short:

1. to set up the image, the framebuffer device needs to be set up

2. the way to set up the device is using QEMU's wonderfully underdocumented fw_cfg

3. you need to also verify whether your CPU has something called the 'dma' via the magic number 0x51454d5520434647

4. it works by going through the configuration zone looking for a honest to God string value

5. when you find it, you then write a bunch of shit to the config object

6. specifically, you need the dimensions, format, pitch and address

7. voilà! works magically

This took like a week I think. But then, I could just write raw bytes to the designated RAM space and things ~just worked~, and I got my dumb little images to display with actually no difficulty at all. Godless magic, and I still didn't see the error of my ways.

Moment of Truth

The moment where I cracked was when I tried to get persistent storage set up, and then I realised what a fucking fool I'd been for genuine weeks then.

The way I wanted to do storage was via a virtio-blk-device which was a simple, abstract interface device for reading and writing to an image file for board emulator developers, and the interface it provides is, allegedly, a very decent scheme that gives bootloader coders another option for storage schemes. As a wise man once said: 'The intent of virtio devices is to be implemented by hypervisors (such as QEMU). They simplify things a bit, so that it’s easier and more efficient for hypervisors and guests to communicate, without having to emulate any quirks of real hardware devices.' [8]

It reads and writes in 512-byte sectors, is of course also memory-mapped in its own specific funny way, and to get the device set up you need to first discover its secret!! location o: by checking more magic numbers (in this case we want 0x74726976) and then the device type, and then finally you have to actually talk to the device to tell it that it's been discovered and that you have a driver for it, and then you have to negotiate (yes, that's apparently the term) with it to see what intersection of features is supported by both your driver and the device

And this is where I broke. After a knee-deep slough through the code in the SeaBIOS repository, again, I ended up on the fucking OSDev bootloader page, which is where I realised that the monkey business I've been blindly doing has, all this time, been writing drivers in a primitive retarded bootloader that I never wanted or needed.

And all I ever fucking wanted was a simple platform to write some funny armasm and get haha clown results back, not this level of convoluted hoopjumping that once again revealed how genuinely rancid software development is once you get to talk to your devices on your own terms. The amount of fucky code I had to write, bin, rewrite over the past months, and the amount of hoops I had to jump and will have to jump if I want persistent storage, is making me start to reconsider writing at least some of this code in C—but then what's the point?

I started this to avoid doing that, so I'm at the fucked up crossroads of damned if I do, damned if I don't.