Shawn Anastasio

Debugging incorrectly closed file descriptors with LD_PRELOAD

Wed, 07 Apr 2021 17:00:00 +0000

I recently tracked down a bug in a relatively complex piece of software using the LD_PRELOAD mechanism and I figured it was worth documenting it here in case anybody finds it useful or interesting.

For those unfamiliar, LD_PRELOAD is a hack present in the dynamic linker of most unix-like systems that allows hooking calls to any functions located in a dynamic library. It goes without saying that this is a pretty powerful tool that can be used in many different ways, from faking the system time, to forcing all network traffic through a proxy, to debugging, which is what I’ll cover here.

Discovering the Bug

As always, the first step in debugging is discovering an issue that you need to solve. In my case, I found an issue while working on my libkvmchan project, specifically its daemon which is a multi-process multi-thread codebase with a homebrew IPC mechanism and many moving parts.

After adding a slew of seemingly minor changes to the codebase, I noticed that whenever one of the newly added codepaths was executed, statements printing to stdout would no longer make it to my console window. The logging facility that uses stderr was unaffected, which narrowed it down. After checking the usual suspects (missing newline to flush the buffer, print statements not being hit, etc.), I began to suspect that the process’ stdout file descriptor had been closed somehow.

To confirm my suspicion, I checked the process’ /proc entry, which is a kernel interface for querying information on a running process including any file descriptors it has open.

$ ls -l /proc/$(pgrep kvmchand)/fd
total 0
lrwx------. 1 root root 64 Apr  7 13:28 0 -> /dev/pts/4
lrwx------. 1 root root 64 Apr  7 13:28 2 -> /dev/pts/4
lrwx------. 1 root root 64 Apr  7 13:28 3 -> 'socket:[358583]'
lrwx------. 1 root root 64 Apr  7 14:06 4 -> 'anon_inode:[eventpoll]'
lrwx------. 1 root root 64 Apr  7 14:06 5 -> 'socket:[358585]'
lrwx------. 1 root root 64 Apr  7 14:06 7 -> 'socket:[358587]'

Sure enough, file descriptors 0 and 2 (stdin and stderr respectively) were present and pointed to my console’s allocated pseudo-tty, but file descriptor 1 corresponding to stdout is absent. I double checked and confirmed that fd 1 was present before my newly-added codepath gets hit and disappears afterwards, so the issue definitely had to do with my new code causing fd 1 to be closed.

Strangely though, the new code I added had absolutely nothing to do with closing file descriptors! Grepping the modified files for close() calls and adding a check for file descriptor 1 didn’t catch anything either, so it was clear that the new code was triggering a bug in an entirely different part of the codebase.

So now the question is, what is the most effective way to track down the erroneous close that is occurring in a completely unknown part of this large, multi-threaded codebase? LD_PRELOAD to the rescue!

LD_PRELOAD to the Rescue

Now that we know the bug is likely caused due to an erroneous call to close() with a file descriptor of 1, we can discover the location of the bug rather trivially by intercepting all close() calls with LD_PRELOAD and checking for a parameter of fd 1.

First, we create a new C file that defines a function in the global namespace with the same name and signature as the function we want to hook. In our case, that signature is int close(int fd). Then we simply need to inspect the argument and perform some action if it’s equal to 1, and forward it to the actual close() function in libc otherwise.

What should be done when the invalid file descriptor argument is detected though? The easiest thing I could think of was to execute an illegal instruction which would allow us to catch the exception in a debugger. I’m on a POWER9 system, so I used the assembly pseudo-op .long 0 to do this. On x86_64 you might want to use the int3 instruction instead to generate a software breakpoint.

Here’s the implementation of the hook:

#define _GNU_SOURCE

#include <stdio.h>
#include <unistd.h>
#include <dlfcn.h>

int close(int fd) {
    static int (*close_fn)(int) = NULL;
    if (!close_fn)
        close_fn = dlsym(RTLD_NEXT, "close");

    if (fd == 1)
        // Tried to close fd 1 - execute an illegal instruction
        asm volatile(".long 0\n");

    return close_fn(fd);
}

Breaking this down, the first thing our hook does is declare a static function pointer which is used to store the address of the actual close() function provided by our system libc. It gets populated by a call to dlsym which dynamically resolves the address of the real close(). For more information, check out dlsym’s man page, specifically the section about RTLD_NEXT.

Next comes the check for file descriptor 1, which should only be hit by the bug. As discussed earlier, if the check passes the code executes the opcode 0x00000000 which is guaranteed by the Power ISA to be an invalid instruction and should thus raise a SIGILL. On x86_64 this should probably be replaced with the int3 instruction.

Finally we simply forward the argument to the real close() function and return the result.

Now all that’s left is building the hook and using it to track down the bug.

Using the LD_PRELOAD hook

With the hook written and saved, compiling it is straightforward - we just need to provide a few flags to gcc telling it to output a shared library and to link against libdl, which provides the dlsym() function we used to grab the function pointer to the real close().

$ gcc -shared -fPIC -ldl hook_close.c -o hook_close.so

We can now use LD_PRELOAD to run our application with the hook installed and then attach gdb to it:

$ LD_PRELOAD=$PWD/hook_close.so ./my_application &
$ gdb -p $(pgrep my_application)
...
(gdb) continue

It’s also possible to launch the application with the hook from within gdb:

$ gdb ./my_application
...
(gdb) set environment LD_PRELOAD ./hook_close.so
(gdb) run

In my case, I went with the former method since the application I was debugging spawns multiple child processes and it was easier to launch it normally and then attach gdb to the relevant PID.

With gdb attached, it’s just a matter of triggering the bug and using the backtrace command to find the erroneous callsite:

...
Thread 3 "kvmchand" received signal SIGILL, Illegal instruction.
Switching to Thread 0x7fff9ce4ed80 (LWP 54610)]
0x00007fffa02c0724 in close () from /home/shawnanastasio/opt/libkvmchan/hook_close.so
(gdb) backtrace
#0  0x00007fffa02c0724 in close () from /home/shawnanastasio/opt/libkvmchan/hook_close.so
#1  0x000000012702d990 in server_receiver_thread (data_=0x127050608 <g_ipc_data+144>) at daemon/ipc.c:529
#2  0x00007fff9fb59618 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fff9fa68cb4 in clone () from /lib64/libc.so.6
(gdb) quit

The backtrace points to daemon/ipc.c:529 as the culprit. After observing the surrounding code the bug became clear:

int *fds = (msg.flags & IPC_FLAG_FD) ? msg.fds : NULL;
if (ipcmsg_send(socfd, &msg, sizeof(struct ipc_message), fds, msg.fd_count) < 0)
    goto fail_errno;

// Close fds now that we're done forwarding them
if (fds) {
    for (uint8_t i=0; i<IPC_FD_MAX; i++) {
        if (fds[i] != -1)
            close(fds[i]);
    }
}

This code is responsible for closing file descriptors passed via IPC messages after they have been forwarded to their destination. The issue is that instead of iterating through the number of file descriptors present in the array (msg.fd_count), it iterates through all values (IPC_FD_MAX). This works fine if the user always sends IPC_FD_MAX (5) file descriptors (as was the case before my recent changes), but if the user sends less, the code will end up passing uninitialized values to close() as long as they’re not equal to -1.

In this case, it seems one of the uninitialized values of the array ended up being 1, which resulted in stdout being closed. The fix was very straightforward to implement and was committed as 8855a5680e59.

Conclusion

So what did we learn then? Well firstly, care needs to be taken when dealing with array sizes and uninitialized values when working in C (yes, I know you already knew that), and secondly, LD_PRELOAD is a very powerful tool that allows for tracking down some pretty nasty bugs with relatively little effort.

I’m certainly not the first person to write about LD_PRELOAD debugging, but hopefully you found this write-up helpful or at least interesting.

Happy hacking!

Programming a Xilinx XC9500XL CPLD with a Raspberry Pi and xc3sprog

Tue, 29 Sep 2020 17:00:00 +0000

I recently got a breakout board for the XC9572XL CPLD from Dangerous Prototypes to play around with, which was particularly interesting to me because of its 5v tolerant I/O banks. Soon after it arrived, though, I realized I didn’t have an ISE-compatible Xilinx JTAG adapter to program it, but thankfully a number of open source solutions exist that allow you to program the device without forking over $270 to Xilinx for what is essentially a Cypress FX2 microcontroller in a red box.

In this post, I’ll demonstrate how to use the matrix-io fork of xc3sprog to program the CPLD from a Raspberry Pi. It’s also worth noting that others have gotten this working with OpenOCD, but I’ve had the best luck with xc3sprog. It also lets you program the .jed file produced by ISE directly without any intermediate conversion steps, which is a nice bonus.

Prerequisites

You’ll just need a Raspberry Pi (any version will do, but versions < 4 will work best) and a CPLD with the JTAG connection broken out. I’m using standard 2.54mm breadboard jumper wires to connect the Pi to the breakout board. For the OS, I’m using the standard Raspberry Pi OS (formerly Raspbian) minimal image, but presumably any Linux distro will do.

You’ll also (obviously) need a .jed file produced by ISE that you want to program to the device. Getting ISE up and running is out of the scope of this tutorial, but suffice to say it’s not a pleasant experience on modern Linux distros.

Building xc3sprog

The first step is to build xc3sprog on the Pi. There are a few dependencies which can all be obtained from Raspberry Pi OS’s repository:

$ sudo apt-get update
$ sudo apt-get install build-essential libusb-dev libftdi-dev libgpio-dev wiringpi git cmake

Next, we’ll clone xc3sprog and build it with cmake. We’re using a fork that adds support for the Pi’s native GPIO interface, in addition to a generic linux sysfs GPIO interface (more on this later).

$ git clone https://github.com/matrix-io/xc3sprog && cd xc3sprog
$ mkdir build && cd build
$ cmake ..
$ make

If all goes well, you’ll have a shiny new xc3sprog binary in your current working directory.

Wiring the Pi to the CPLD

The next step is to physically connect the Pi’s GPIO pins to the CPLD. Surprisingly enough, I couldn’t find the actual JTAG pinout that xc3sprog uses documented anywhere (which is part of the reason I decided to write this post and document it). After some digging through the code, I found that the constructor for the IOWiringPi class accepts the GPIO pin numbers used for JTAG and it’s called here for the WiringPi backend and here for the sysfs backend.

The choice of which backend you want to use will depend on which Raspberry Pi model you have. The WiringPi backend is much faster but doesn’t support the Pi 4 whereas the sysfs backend works on all models but is much slower. If you have a Pi <= 3, I’d recommend using the WiringPi backend. Thankfully the pinout is the same regardless of which backend you choose.

Using the numbers from the constructor calls above, we can deduce the following pinout:

Signal	GPIO
TMS	4
TCK	17
TDI	22
TDO	27

Note that these are GPIO pin numbers and don’t correspond to the pin numbers of the Raspberry Pi’s expansion header.

We can see how these GPIO pin numbers correspond to pins on the expansion header in this image:

Image CC BY-SA Raspberry Pi Foundation: https://www.raspberrypi.org/documentation/usage/gpio

This means that JTAG signals should be connected to the following expansion header pins

Signal	GPIO	Header Pin
TMS	4	7
TCK	17	11
TDI	22	15
TDO	27	13

After wiring these up in addition to a ground pin (and an appropriate voltage pin if you’re powering the CPLD from the Pi), we can finally use xc3sprog to program the device.

Programming the CPLD

With the Pi set up, the next thing to do is see if xc3sprog can detect the CPLD on the JTAG bus:

$ ./xc3sprog -c <backend> -j

Replace <backend> with matrix_creator for the WiringPi backend or sysfsgpio_creator for the sysfs one.

If all goes well, you should see something like this:

JTAG loc.:   0  IDCODE: 0x59604093  Desc:                       XC9572XL Rev: E  IR length:  8

We can see that my CPLD was detected at JTAG location 0. If you don’t see anything, double check your wiring.

To program a .jed file now, all we need to do is pass it to xc3sprog:

./xc3sprog -c <backend> -p <JTAG location from above> -v <path/to/file.jed>

Happy hacking!

Running a mainline linux kernel on the NVIDIA Jetson Xavier AGX

Mon, 01 Jun 2020 17:00:00 +0000

The Jetson Xavier AGX is an embedded ARM64 linux platform from NVIDIA with pretty impressive specifications and a decent software ecosystem. Most users will probably be using the NVIDIA-provided Linux4Tegra (L4T) distribution on their Xaviers, which comes with a laughably ancient 4.9 kernel build and quite a few out-of-tree drivers. This means users will miss out on new features, upstream driver additions, and security fixes from the upstream Linux kernel project.

Thankfully though, support for the Xavier SoC and a lot of its peripherals are present in recent Linux releases, so running a mainline kernel is possible (though NVIDIA’s documentation certainly won’t tell you how). Below we’ll walk through the process of compiling a kernel from mainline sources and booting it on the Xavier.

Prerequisites

To perform the whole operation, you’ll need a few prerequisites.

Kernel Sources

We’ll start off by downloading the mainline kernel sources we wish to compile, as well as some build prerequisites.

On Ubuntu 18.04, you can download the build requisites with the following:

$ sudo apt-get install build-essential libncurses-dev bison flex libssl-dev libelf-dev gcc-aarch64-linux-gnu

Next, you’ll need to grab a copy of the kernel sources you want to compile. For this tutorial we’ll download straight from git.

$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Nvidia Jetpack SDK

Jetpack is NVIDIA’s SDK that comes with everything you’ll need to flash the Xavier. Unfortunately, many of the tools it ships seem to be proprietary binaries, so you’ll need to do this on an x86_64 machine.

The easiest way to get Jetpack set up is with the NVIDIA SDK Manager. You’ll need to make an NVIDIA Developer account to download it unfortunately.

Once you’ve got the SDK Manager, click through and download the latest Jetpack release (4.4 at the time of writing) for the Xavier AGX. I’d also recommend un-checking the Host Machine software selector which will install a bunch of unnecessary NVIDIA stuff to your host machine.

Finally, follow the directions to flash the included L4T release to your Xavier. We’ll use L4T as the base distro which we’ll install our mainline kernel build over. See here for more information on using the SDK Manager.

Building the kernel

With the prerequisites out of the way, we can begin building the kernel. Change to the directory where you downloaded the kernel sources and begin the configuration process:

$ cd linux
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig

In the menuconfig interface, enable the ethernet driver under:

Device Drivers -> Network device support -> Ethernet driver support
    -> STMicroelectronics devices
        -> STMicroelectronics Multi-Gigabit Ethernet driver
            -> STMMac Platform bus support
                -> Support for snps,dwc-qos-ethernet.txt DT binding

Be sure to enable support for any other peripherals or features you’ll use.

After the configuration’s done, you can launch the build:

$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j`nproc`

This will launch one build for every logical core on your machine. Change the -j parameter if you want something else.

If all goes well, the build will complete successfully and you can move on to the next step.

Installing the kernel

With the kernel built, the next step is to install it to your Xavier’s root filesystem, along with an initramfs and bootloader configuration. The easiest way to accomplish this is to simply copy over the kernel build directory to your Xavier and run the make install targets there.

It’s worth nothing that there are other ways of accomplishing this, like using qemu-aarch64 with binfmt-misc to chroot into the L4T rootfs and installing there. Advanced techniques like that are out of the scope of this tutorial though, so we’ll continue on by copying the built source to an up-and-running Xavier.

Copying the build directory can be easily accomplished rsync:

$ rsync -azP ../linux <xavier_user>@<xavier_ip>:~/

Substitute your Xavier’s username and IP address.

Next, log in to the Xavier and execute the following commands to install the kernel

$ cd linux
$ sudo make modules_install
$ sudo make install

Finally, add an entry for the kernel in /boot/extlinux/extlinux.conf, substituting kernel and initramfs names:

LABEL mainline
      MENU LABEL mainline kernel
      LINUX /boot/vmlinuz-5.7.0-rc7-vanilla-00212-gbdc48fa11e46
      INITRD /boot/initrd.img-5.7.0-rc7-vanilla-00212-gbdc48fa11e46
      APPEND ${cbootargs}

With the kernel installed, all that’s left is to flash the mainline device tree before we can boot it.

Flashing the device tree

One major way difference between L4T’s kernel 4.9 fork and mainline is the layout of the Xavier’s device tree. This has the unfortunate effect of making mainline kernels unbootable with L4T device trees and vice versa.

On many embedded linux systems, the device tree is simply stored as a normal file that can be read by the bootloader, just like the kernel. Unfortunately for us, the Xavier does things a bit differently.

With the default bootloader provided with L4T, the Xavier loads its device tree from a special partition on internal flash memory. This means that replacing the device tree requires using Jetpack’s flashing tools from an x86_64 machine.

To get started, head back to your x86_64 machine, and navigate to the Jetpack installation directory. If you used the NVIDIA SDK Manager, that path will probably be something like:

~/nvidia/nvidia_sdk/JetPack_4.4_DP_Linux_DP_JETSON_AGX_XAVIER/Linux_for_Tegra

From here, we’re going to copy the jetson-xavier configuration and modify it to point to the mainline kernel’s device tree, instead of the L4T one.

$ cp jetson-xavier.conf jetson-xavier-mainline.conf
$ vi jetson-xavier-mainline.conf

Append the following line to the end of the file:

DTB_FILE=tegra194-p2972-0000.dtb;

Then, save and exit your editor.

Now we need to copy the device tree produced by the linux build into the L4T directory. Assuming your linux tree is at ~/linux, execute the following:

cp ~/linux/arch/arm64/boot/dts/nvidia/tegra194-p2972-0000.dtb kernel/dtb/

Now we’re ready to flash the device tree. Put your Xavier into RCM mode by holding down the middle button while turning it on (left button) or resetting (right button), and enter the following command:

sudo ./flash.sh -r -k kernel-dtb jetson-xavier-mainline mmcblk0p1

This will read in the new device tree and flash it to the device.

If all goes well, the device should reboot and you will be able to select your mainline kernel build from the extlinux boot menu.

OSDEV: Implementing a basic x86 page frame allocator in C

Mon, 08 Aug 2016 21:42:10 +0000

When writing an operating system for any architecture, one of the most important functions you’ll have to write is the page frame allocator. The page frame allocator allows the OS to divide the physical memory available on a system into chunks called page frames, which can then be used later to allocate memory to applications with a separate paging function.

There are many methods of page frame allocation, each with varying levels of complexity and efficiency. For an overview of many of these methods, see the OSDev Wiki. In this post we’ll be going over a much less efficient method than those outlined in the wiki for simplicity’s sake, but you can improve upon it later.

Unlike those outlined in the wiki, our page frame allocator isn’t actually going to be storing any data of its own. Instead, we’re going to piggyback off of the Multiboot information structure. If you’re following this tutorial, you hopefully already have a basic Multiboot kernel set up, as well as a basic understanding of the Multiboot specification (the PDF for which can be found here).

Without reading the entire specification, the rundown is that your Multiboot-compliant bootloader (probably GRUB) will provide you with an information structure that contains a lot data about the system you’re running on. Among this data is the Multiboot memory map, which provides a listing of all memory regions in the system, and which ones you’re allowed to write to (many regions are reserved for things like VGA video output).

To access this memory map, we must first pass a pointer to the information structure to our kernel’s entry point. Luckily, our bootloader will put this pointer in the ebx register when we first boot up, so all we have to do is pass it as a parameter to our entrypoint in assembly. Along with ebx, we’ll also pass eax which contains a “magic value”, 0x2BADB002, which you can use to verify that the bootloader is in fact Multiboot-compliant.

Getting the Multiboot information structure

Before we start, we have to make sure that we’ve declared to our bootloader that we want the memory map to be included in the information structure, as it is optional by default. If you’re using the boot assembly code from the OSDev Bare Bones tutorial, this is already done for you. The code to specify we want the memory map looks like this (at the top of your file):

MEMINFO     equ 1<<1
; You can add more flags here separated by |
FLAGS       equ MEMINFO
MAGIC       equ 0x1BADB002
CHECKSUM    equ -(MAGIC + FLAGS)

.section multiboot
align 4
    dd MAGIC
    dd FLAGS
    dd CHECKSUM

We’ll also have to make sure that the initial stack is at least 4096 bytes (the size of a single page). As with the previous step, if you’re using the OSDev Bare Bones code this is done already. I’ll be setting it to 16384 bytes, but anything greater than 4096 bytes should suffice. This change needs to be done in your boot.asm or equivalent file.

section .bootstrap_stack, nobits
align 4
stack_bottom:
    resb 16384
stack_top:

...

section .text
global _start
_start:
    ; Use the stack we allocated above
    mov esp, stack_top

    ; Ensure stack is 16-bit aligned
    and esp, -16

Now that our stack is big enough, we can pass the Multiboot information structure as well as the magic value to our kernel entrypoint. Remember that we must push the values in the reverse order that we want to use them:

section .text
global _start
_start:
    ...
    ; Push Multiboot information structure
    push ebx

    ; Push Multiboot magic value
    push eax

    ; Call our kernel entrypoint
    call kernel_main

Now we can access these values in our kernel’s C entrypoint, we just have to change its parameters:

void kernel_main(uint32_t mboot_magic, void *mboot_header) {
    ...
}

Defining the Multiboot information structure

To read the Multiboot information structure, we first need to declare a few structs in C that define its layout. Luckily the aforementioned Multiboot specification PDF has them written out for you on page 16. While it does declare more than we’ll need for this tutorial, it’s not a bad idea to include the whole file anyways as it may be useful in the future.

Once you have the structures and constants from the PDF included into your kernel, there’s one more thing we need to do before we can start parsing the information structure.

Validating the Multiboot magic value

As stated before, the Multiboot magic value is simply a constant that all Multiboot-compliant bootloaders will provide to prove that they are compliant. Before we even attempt to read the Multiboot information structure, we should verify this value so that we don’t try to read garbage in the event that the kernel is not booted properly.

void kernel_main(uint32_t mboot_magic, void *mboot_header) {
    ...
    // MULTIBOOT_BOOTLOADER_MAGIC is defined on page 16 of the PDF too
    if (mboot_magic != MULTIBOOT_BOOTLOADER_MAGIC) {
        // If the magic is wrong, we should probably halt the system.
        printf("Error: We weren't booted by a compliant bootloader!\n");
        panic();
    }
}

Preparing to read the Multiboot information structure

Now that we’ve verified the magic value, we can read the Multiboot information structure and look for the memory map. As it’s possible the memory map wasn’t included (in the case of an incomplete bootloader or other error), it’s always good to check for its existence.

void kernel_main(uint32_t mboot_magic, void *mboot_header) {
    ...
    // First, cast the pointer to a multiboot_info_t struct pointer
    multiboot_info_t * mboot_hdr = (multiboot_info_t *)mboot_header;

    // The specification states that bit 6 signifies the presence of the memory map
    // We can check the header flags to see if it's there
    if ((mboot_hdr->flags & (1<<6)) == 0) {
        // The memory map is not present, we should probably halt the system
        printf("Error: No Multiboot memory map was provided!\n");
        panic();
    }
}

We’re almost there, but first, let’s set some global variables that will help us keep track of things and make our lives easier.

/**
 * At the top of your file, we'll make some global variables that will help us
 * keep track of frames. I'll explain what these do in a bit.
 */
multiboot_info_t *verified_mboot_hdr;
uint32_t mboot_reserved_start;
uint32_t mboot_reserved_end;
uint32_t next_free_frame;

void kernel_main(uint32_t mboot_magic, void *mboot_header) {
    ...
    /**
     * After we've verified the information structure, let's update our
     * global variables.
     */
    verified_mboot_hdr = mboot_hdr;
    mboot_reserved_start = (uint32_t)mboot_hdr;
    mboot_reserved_end = (uint32_t)(mboot_hdr + sizeof(multiboot_info_t));
    next_free_frame = 1;
}

As you can see, we’ve made quite a few variables. First, we’ve got a pointer, verified_mboot_hdr where we can store our pointer to the information structure so other functions can access it easily. Next, we’ve got mboot_reserved_start and mboot_reserved_end which, surprisingly enough, store the start and end locations of the Multiboot information structure in memory so we don’t accidentally overwrite it later. Finally, we’ve got next_free_frame, which stores the number of the next free frame. This will be explained in greater detail in a later section.

A page frame allocator

We’ve gone through all the steps to verify and prepare to read the Multiboot information structure, but up until now you didn’t know why. As stated before, our page frame allocator is going to piggyback off of the Multiboot memory map to keep track of allocated frames. We’re going to do this by having a single counter that increases by one every time we allocate a 4096 chunk (frame) of free memory in the Multiboot memory map from start to end. This way, frame #1 is always the first 4096 bytes marked as free in the memory map, #2 is the second 4096 bytes, and so on. This is what the next_free_frame variable in the previous section is for.

Before we can write the actual allocator, we need a method to read the memory map and retrieve memory addresses that correspond to frame numbers, and vice versa. For this we’re going to create a separate function that accesses those varibles we created earlier.

#define MMAP_GET_NUM 0
#define MMAP_GET_ADDR 1
#define PAGE_SIZE 4096

/**
 * A function to iterate through the multiboot memory map.
 * If `mode` is set to MMAP_GET_NUM, it will return the frame number for the
 * frame at address `request`.
 * If `mode` is set to MMAP_GET_ADDR, it will return the starting address for
 * the frame number `request`.
 */
uint32_t mmap_read(uint32_t request, uint8_t mode) {
    // We're reserving frame number 0 for errors, so skip all requests for 0
    if (request == 0) return 0;

    // If the user specifies an invalid mode, also skip the request
    if (mode != MMAP_GET_NUM && mode != MMAP_GET_ADDR) return 0;

    // Increment through each entry in the multiboot memory map
    uintptr_t cur_mmap_addr = (uintptr_t)verified_mboot_hdr->mmap_addr;
    uintptr_t mmap_end_addr = cur_mmap_addr + verified_mboot_hdr->mmap_length;
    uint32_t cur_num = 0;
    while (cur_mmap_addr < mmap_end_addr) {
        // Get a pointer to the current entry
        multiboot_memory_map_t *current_entry =
                                        (multiboot_memory_map_t *)cur_mmap_addr;

        // Now let's split this entry into page sized chunks and increment our
        // internal frame number counter
        uint64_t i;
        uint64_t current_entry_end = current_entry->addr + current_entry->len;
        for (i=current_entry->addr; i + PAGE_SIZE < current_entry_end; i += PAGE_SIZE) {
            if (mode == MMAP_GET_NUM && request >= i && request <= i + PAGE_SIZE) {
                // If we're looking for a frame number from an address and we found it
                // return the frame number
                return cur_num+1;
            }

            // If the requested chunk is in reserved space, skip it
            if (current_entry->type == MULTIBOOT_MEMORY_RESERVED) {
                if (mode == MMAP_GET_ADDR && cur_num == request) {
                    // The address of a chunk in reserved space was requested
                    // Increment the request until it is no longer reserved
                    ++request;
                }
                // Skip to the next chunk until it's no longer reserved
                ++cur_num;
                continue;
            } else if (mode == MMAP_GET_ADDR && cur_num == request) {
                // If we're looking for a frame starting address and we found it
                // return the starting address
                return i;
            }
            ++cur_num;
        }

        // Increment by the size of the current entry to get to the next one
        cur_mmap_addr += current_entry->size + sizeof(uintptr_t);
    }
    // If no results are found, return 0
    return 0;
}

While this code looks daunting, the basic concept behind it is quite simple. It essentially does the following:

Read each entry in the memory map
Split each entry into frame-sized chunks (4096 bytes)
Iterate through each chunk until a valid address is found for the given frame number or a frame number is found for the given address.

Now that we can read the memory map, creating an allocator can be done quite simply:

/**
 * Allocate the next free frame and return it's frame number
 */
uint32_t allocate_frame() {
    // Get the address for the next free frame
    uint32_t cur_addr = mmap_read(next_free_frame, MMAP_GET_ADDR);

    // Verify that the frame is not in the multiboot reserved area
    // If it is, increment the next free frame number and recursively call back.
    if (addr >= multiboot_reserved_start && addr <= multiboot_reserved_end) {
        ++next_free_frame;
        return allocate_frame();
    }

    // Call mmap_read again to get the frame number for our address
    uint32_t cur_num = mmap_read(cur_addr, MMAP_GET_NUM);

    // Update next_free_frame to the next unallocated frame number
    next_free_frame = cur_num + 1;

    // Finally, return the newly allocated frame num
    return cur_num;
}

Now, to allocate a page frame you simply have to call

uint32_t new_frame = allocate_frame();
uint32_t new_frame_addr = mmap_read(new_frame, MMAP_GET_ADDR);
printf("New frame allocated at: 0x%x\n", new_frame_addr);

Conclusion

Congratulations! You now have a working (albeit primitive) method of page frame allocation for your operating system. It cannot yet free frames, but it provides a good starting point for developing a paging system. In the future, I may write another post improving upon this allocator with freeing implemented, as well as some performance enhancements.

Resources

If you get stuck anywhere, I recommend taking a peek at my example OS ShawnOS, as well as the plethora of online resources related to OS development, such as the OSDev Wiki.