Unblock the author

My goal in code review is not to be a gatekeeper, but to be a collaborative partner in improving code quality while maintaining team velocity. This process is built on trust. I trust my teammates to make good decisions and address feedback appropriately.

A quick, high-quality code review keeps everyone motivated about their work. It prevents context changes, and it leads to a better team that produces work quicker and in higher quality. I try to review pull requests as soon as possible, without breaking my own context too much, by setting aside time that does not break my day in two: for example, before or after lunch, at the beginning of the day, or at the end. For simple changes, 6 hours is too much time for a pull request to go unreviewed. For complex changes, two days is too much time.

Reviews must be actionable

Pull request reviews should always give the author a clear next step. It’s not acceptable for a review to open up a broad discussion without explicitly making a decision about whether the pull request should be blocked until the discussion is resolved.

Comments-only reviews without a clear conclusion are not good: I can’t merge because it’s not approved, but I’m also not sure if the comments are blocking. I don’t know what to do. The author is left unsure whether to address feedback, wait for more input, or proceed with merging.

Almost every review I leave will either approve the pull request or request changes. If I lack the context to make that decision, I redirect to someone who can. This ensures the author always knows their next action.

The process

flowchart TD
    Start([PR Ready for Review]) --> Review[Review PR & leave comments]

    Review --> ConfidentCheck{Feel confident to approve or request changes?}

    ConfidentCheck -->|No| Redirect["Tag someone with context 'I don't feel confident to review this properly. Would ask @person to take a look.'"]
    Redirect --> End([Done, waiting for a better reviewer])

    ConfidentCheck -->|Yes| ReReviewCheck{Expect to need re-review after changes?}

    ReReviewCheck -->|No - DEFAULT 95% of cases| ApproveComments[Approve with comments]
    ApproveComments --> AuthorChoice[Author decides: merge now or after fixes]

    ReReviewCheck -->|Yes| RequestChanges[Request changes, used liberally]
    RequestChanges --> ClearComments[Clear communication of what needs fixing: this is now top of my queue to unblock author ASAP, other reviewers can dismiss to override]
    ClearComments --> ReReview[Author fixes, dismisses, or gets another approval]

    AuthorChoice --> Merge([PR can be merged])
    ReReview --> Review

    class Start nodeStart
    class Merge nodeSuccess
    class End nodeDanger
    class ApproveComments nodeApprove
    class RequestChanges nodeWarning

Default action: approve

To promote what I believe is a healthy team environment, I default to approving with comments about 95% of the time. This signals trust and keeps work flowing. The question isn’t “is this perfect?” but rather “is this good enough to merge, and can any issues be addressed afterward or through normal iteration?”.

To be very clear: I will sometimes approve a pull request even if I think it worsens the quality of the codebase. If I see a structural problem or a pervasive bad practice, I will raise the issue elsewhere and cross-link. I approve even when there are bugs, as long as they’re understood, and the intent of the PR is clear.

Context matters: feature-flagged work that users won’t see yet warrants a lighter touch and faster merging, while changes to public contracts or APIs deserve more thorough consideration given their broader impact.

Use request changes liberally

I only choose request changes when I am sure I will need to come back to the pull request to review it again, because I expect that after the changes it’ll look very different. This might be when the changes are architectural or complex enough that I want to see the implementation, or I have concerns about the approach that need discussion.

“Request changes” is a communication tool: other reviewers can dismiss change requests if they have the necessary context, which means I feel comfortable requesting changes in the rare case I believe I need to see the code again after a rework, without fear of becoming a bottleneck. Both requesting changes and dismissing a change request are completely OK given clear and thoughtful communication.

If I request changes, however, I’m making a commitment to the author: I communicate clearly what needs to change and why, I prioritize re-reviewing (this goes to the top of my queue), and I unblock the author ASAP once they’ve addressed my concerns.

Not feeling confident to review?

If I’m unfamiliar with the codebase area, don’t understand the business requirements, or the PR touches complex systems I haven’t worked with, I redirect to someone with appropriate context. I don’t approve what I can’t properly evaluate. I will nevertheless comment with any thoughts I have, but I will make it clear that I’m not the right person to review.

Comment categories

To set clear expectations and help authors prioritize feedback, I prefix my comments with one of these labels where applicable:

Nit

Label: Nit:

Minor nit-picks about style, naming, small improvements, or behavior that don’t critically affect functionality. These are “take it or leave it” suggestions. The weight of my own convictions should not make me lose sight of the main purpose: to unblock the author. A team can align on common preferences, but that should be a separate conversation.

Note: spelling or grammar mistakes visible to users are not labeled as nits, as they’re more like bugs if users will see them.

Examples:

• “Nit: this variable could be named userId instead of uid for clarity.”
• “Nit: this comment explains what the code below is doing, but it does not say anything about the ‘why’, what do you think about (…).”

Super-nit

Label: Super-nit:

Extremely nit-picky comments that are truly optional. I use this for things that are purely personal preference or very minor consistency issues.

Example: “Super-nit: we could use a function that returns this test’s dependencies instead of declaring them at the top as nullable variables and setting them up later”

“Would like to see” / questions

Label: none (use natural phrasing)

Comments that are worth discussing. I may sense that there is an underlying assumption and want to confirm that. Or maybe I have an idea of how something could have been done differently and want to hear the author’s opinion.

Examples:

• “Could we add error handling here for the case where the API returns an error?”
• “Question: why did we choose approach X over Y? Just trying to understand the tradeoffs.”

Off-topic

Label: Off-topic:

Discussions that are only tangentially related to the PR. I use this for systemic issues, technical debt, or improvements that should happen separately from this PR.

Example: “Off-topic: looks like we could be using a linting rule for X here. We should probably add that to our linter config in a separate PR. I’ve opened an issue here (…), let’s discuss there.”

Off-topic: product decision

Label: Off-topic (product decision):

A specific subclass of off-topic for when the code correctly implements a product decision, but you disagree with the underlying product decision itself. The PR author is just implementing what was decided elsewhere, so this shouldn’t block the PR.

Example: “Off-topic (product decision): this feature defaults to opt-in, but I think opt-out would have better adoption. Not a code issue, as the implementation is correct, but worth discussing with the product team separately. I’ve opened a thread about this here (…).”

Tip: for off-topic discussions, especially product decisions, consider opening a Slack thread and cross-linking it with the GitHub comment. This keeps the PR focused while ensuring the discussion happens in the right forum.

Logic issue / bug

Label: None (be direct and clear)

I use this when I believe the code has a logic error or will produce unexpected behavior. This is not a style issue or architectural preference but I’m flagging that the code won’t work as intended.

I’m clear and specific about what I think will happen versus what should happen. These comments only warrant “request changes” if the bug is not yet understood by both the author and me. I keep in mind that there may be a case of asymmetrical knowledge or I may have misunderstood the intent of the change, in which case it may be worth improving the pull request description or adding explanatory code comments.

Example: “Logic issue: this function returns early when count === 0, but I think we still need to update the timestamp in that case. As written, zero-count events won’t be tracked.” In this case, I approve anyway.

Celebratory / thanks

Label: None (be genuine and specific)

I call out clever solutions, good refactoring, thorough testing, or helpful documentation. I thank authors for addressing tricky issues or improving code quality.

Examples:

• “This is a really elegant solution to the caching problem!”
• “Thanks for adding those test cases, they caught an edge case I hadn’t thought of”
• “Love how this simplifies the error handling”

Cross-linking

For off-topic discussions, especially those involving product decisions or broader architectural questions, I open a separate thread in the appropriate channel (Slack, the issue tracker, or wherever the team discusses such topics) to discuss the matter there. I always make it clear in my PR comment that it’s non-blocking when moving discussion elsewhere. I post my initial comment on the GitHub PR stating it’s non-blocking, then open a thread in the appropriate channel with context, link to that thread in my GitHub comment, and link back to the GitHub comment from the thread for full context. This keeps the PR focused on the code changes while ensuring important discussions happen in the right forum with the right stakeholders.

I link to TUXEDO a lot in this blog post. These links are not affiliate links, and this post is not sponsored (duh).

TUXEDO Computers is a laptop manufacturer based in Augsburg, Germany, who are special to Linux users because they’re one of the few manufacturers developing specifically for Linux.

They ship with their own TUXEDO OS, which comes with KDE Plasma by default. For my new job, I actually wanted a Framework laptop with an AMD chip, but they were temporarily out of stock so I opted for a TUXEDO laptop, which would have been my second choice. I picked their TUXEDO InfinityBook Pro 14 Gen9 AMD model.

In this blog post, I’ll go over how I set up my NixOS computer to work on this TUXEDO laptop. NixOS is not among the supported operating systems, so I was expecting having to do some digging.

The result

Here are the changes you’d need to make. I’d recommend reading the rest of the post to understand some trade-offs.

{
  pkgs,
  config,
  ...
}: {
  boot = {
    # Motorcomm YT6801 LAN drivers
    extraModulePackages = with config.boot.kernelPackages; [yt6801];
    kernelParams = [
      "acpi.ec_no_wakeup=1" # Fixes ACPI wakeup issues
      "amdgpu.dcdebugmask=0x10" # Fixes Wayland slowdowns/freezes
    ];
  };

  # Requires a system module from this flake:
  # https://github.com/sund3RRR/tuxedo-nixos?tab=readme-ov-file#option-2-nix-flake
  hardware = {
    tuxedo-drivers.enable = true;
    tuxedo-control-center.enable = true;
  };

  # Let TUXEDO Control Center handle CPU frequencies
  services.power-profiles-daemon.enable = false;
}

Include these changes in your /etc/nixos/configuration.nix in some way. In my case, I use a system flake that includes a laptop-specific configuration module.

Drivers and the TUXEDO Control Center

# Requires a system module from this flake:
# https://github.com/sund3RRR/tuxedo-nixos?tab=readme-ov-file#option-2-nix-flake
hardware = {
  tuxedo-drivers.enable = true;
  tuxedo-control-center.enable = true;
};

The NixOS Wiki has an article on TUXEDO devices. It points at the fact that, as of July 2025, there is no package in nixpkgs that can install the official TUXEDO Control Center. It looks like the reasons include implementation attempts including thousands of generated Nix code to declare the dependencies of the Angular frontend, and the frontend requiring Electron 11, which was released in 2020 and is deemed ancient. It also points at a community flake that has been abandoned, and that makes this module available outside of nixpkgs.

The other option suggested by the Wiki is to use tuxedo-rs, which exists because “TCC and its tccd service rely on Node.js which makes it slow, memory hungry and hard to package”. I agree in principle, but it lacks certain features that I found useful in the TUXEDO Control Center:

• A system tray applet that lets me manually select a power profile (tuxedo-rs has a small GTK application but no system tray applet)
• An option to select which profile should activate when my laptop is plugged to electricity or working off of battery

So, even though I don’t love going with a community flake for such a critical feature of my laptop, I went for the TUXEDO Control Center option, and I found this flake which is alive: sund3RRR/tuxedo-nixos. Thanks sund3RRR!

Getting Ethernet to work

Although I’d enabled hardware.tuxedo-drivers, my Ethernet port was still not working. After some digging, I found that I needed to enable a driver for it:

boot = {
  # Motorcomm YT6801 LAN drivers
  extraModulePackages = with config.boot.kernelPackages; [yt6801];
};

This is, in fact, mentioned in the TUXEDO FAQ for this laptop, but at this point I had not read this resource yet.

Fixing laptop wake-ups while shut

While shut, the laptop would wake up and spin the fans periodically. This was very annoying. To solve this one, I messaged their support. They were very welcoming to my question even though it was related to an unsupported OS, and advised that I add these two kernel parameters:

kernelParams = [
  "acpi.ec_no_wakeup=1" # Fixes ACPI wakeup issues
  "amdgpu.dcdebugmask=0x10" # Fixes Wayland slowdowns/freezes
];

I have not tested the second one, but rebooting after adding acpi.ec_no_wakeup=1 proved that it solved the random wake-ups.

The acpi.ec_no_wakeup=1 param was also mentioned in their FAQ. Oops…

PowerDevil vs. TUXEDO Control Center

The addition of TUXEDO Control Center left me confused as to which widget to use to manage CPU frequencies and governors. KDE Plasma’s PowerDevil provides integration, such as a slider under the system tray battery icon, and (as of recently) the battery icon changes to indicate which power profile is in use.

Changing the profile using TUXEDO Control Center does not reflect in a change in the KDE Plasma widget, so Claude wrote I wrote a script to monitor different system properties as I changed the TUXEDO Control Center and KDE Plasma options:

#!/usr/bin/env bash
while true; do
  freq=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq 2>/dev/null || echo "0")
  min_freq=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq 2>/dev/null || echo "0")
  max_freq=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq 2>/dev/null || echo "0")
  gov=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "N/A")
  epp=$(cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference 2>/dev/null || echo "N/A")
  boost=$(cat /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || echo "N/A")

  printf "%s | %-10s | %-9s | %-12s | %-14s | %s\n" \
    "$(date +%H:%M:%S)" "$gov" "$((freq / 1000))" "$((min_freq / 1000))-$((max_freq / 1000))" "${epp:0:14}" "$boost"
  sleep 1
done

Here’s a log of what was happening. I’ve redacted away a lot of lines so that you can see just the changes:

 1: TCC set to TUXEDO Defaults
 2: Plasma widget set to power saver
 3: 15:27:37 | powersave  | 1559  | 400-3801  | balance_perfor | 1
 4:
 5: TCC changed from TUXEDO Defaults to "Default" (I believe no change)
 6: 15:27:55 | powersave  | 2452  | 400-3801  | balance_perfor | 1
 7:
 8: TCC changed from "Default" to "Cool and breezy"
 9: 15:28:03 | powersave  | 1459  | 400-1900  | balance_perfor | 1
10:
11: TCC changed from "Cool and breezy" to "Powersaver extreme"
12: 15:28:08 | powersave  | 548   | 400-400   | balance_perfor | 1
13:
14: TCC changed from "Powersaver extreme" to "TUXEDO Defaults"
15: 15:28:25 | powersave  | 1297  | 400-3801  | balance_perfor | 1
16:
17: Plasma widget from power saver to balanced
18: 15:28:30 | powersave  | 1098  | 1098-3801 | balance_power  | 1
19: 15:28:35 | powersave  | 1540  | 400-5137  | balance_perfor | 1
20:
21: Plasma widget from balanced to performance
22: 15:28:44 | performance | 1098 | 1098-5137 | performance    | 1
23: 15:28:52 | powersave   | 1331 | 400-5137  | balance_perfor | 1

The interesting bits here are that both are apparently ultimately modifying the minimum and maximum CPU frequencies. They probably do this through different means, and I’m not an expert here, but I believe what’s happening is that KDE Plasma is talking to power-profiles-daemon and setting a set of preferences (governor to powersave, energy performance preference (EPP) to balance_performance), and that results in something lower-level updating the minimum and maximum CPU frequencies.

You’ll notice that in the two last changes, where I’m using the KDE Plasma widget, I included two log lines per change. It looks like TUXEDO Control Center is periodically monitoring the value of these settings and correcting them to what it wants:

• Governor set to performance: corrected to powersave after a few seconds
• EPP set to performance: corrected to balance_performance after a few seconds

I also did not like that the maximum CPU frequency would be set to 3801 or 5137 depending on which KDE Plasma option was picked last. TUXEDO Control Center tries to update things back to TUXEDO Defaults (which has a max of 5137) but only manages 3801, and I believe this is because the selected KDE Plasma option was powersave.

After seeing all of these interactions (or interruptions) I decided to use the TUXEDO Control Center exclusively:

# Let TUXEDO Control Center handle CPU frequencies
services.power-profiles-daemon.enable = false;

I lose the KDE Plasma integration with the battery widget, but that’s OK.

Remaining annoyances

There are just a couple of remaining nit-picks.

On boot, I get some ACPI-related warnings à la:

[    0.305246] ACPI BIOS Error (bug): Failure creating named object [\_SB.PCI0.GPP6.WLAN._DSM], AE_ALREADY_EXISTS (20240827/dswload2-326)

These seem innocuous. I contacted TUXEDO to see what could be done, and they mentioned they’d updated their FAQ to mention that these messages do not affect operation.

The other thing is that when the laptop wakes from sleep, the fans spin at full speed for a couple of seconds. I have a hunch that this is default behavior, and wanted, but it would be lovely if the laptop were quieter when opening the lid.

Thoughts

Altogether, NixOS has had me fight a lot more fights than I would have fought on another distribution. I’ve been learning a lot, but I begin to question whether the benefits are worth the effort. I have had trouble setting up, for example, Vanta for work (a compliance monitoring agent), Playwright for browser end-to-end tests, and I simply cannot run some software unless I make the effort of packaging it for Nix and NixOS, for example using KWallet as a pinentry program or installing a database client I’d like to try out.

With the release of Debian 13 Trixie looming, I may try to move my setup to a Debian stable base and keep most of my Home Manager setup so that I can use Nix for anything that’s not my basic system, while still enjoying the mainstream support provided by Debian. We’ll see!

This post is part of an ongoing series on TypeScript. In it, I try to explain in a detailed way different situations I've faced while building relatively big React applications using TypeScript, and moving them from JavaScript. Here's a list of other posts I've written about TypeScript:

• TypeScript – Poor man's async await using generators · July 31, 2020
• TypeScript – Structuring type dependencies in frontend applications · August 12, 2019
• TypeScript – Beware the user-defined type guards · February 27, 2019
• TypeScript – Using Record is usually not the best choice · January 30, 2019

In a previous entry I argued that the use of the Record type is dangerous in some circumstances and suggested using a Dictionary type to solve that. The goal of the Dictionary type is to make sure the type of values from property access returns T | undefined as opposed to just T (which would suggest that we have a T for any K in the dictionary).

The release of TypeScript 4.9 includes a new operator called satisfies. It can be used to solve the situation I presented in that other post:

type Breed = string

interface Dog {
  name: string
}

const dogsByBreed: Record<Breed, Dog[]> = {
  akita: [{ name: 'Walter' }, { name: 'Gracie' }],
  dachshund: [{ name: 'Charlie' }],
}

dogsByBreed.poodle // Type is Dog[], value is undefined.

Instead of introducing a Dictionary type, you can now simply do this:

const dogsByBreed = {
  akita: [{ name: 'Walter' }, { name: 'Gracie' }],
  dachshund: [{ name: 'Charlie' }],
} satisfies Record<Breed, Dog[]>

dogsByBreed.poodle // Type error!

TypeScript gives us a static type-check with the message:

Property 'poodle' does not exist on type '{ akita: { name: string; }[]; dachshund: { name: string; }[]; }'.

Which is absolutely correct and gives us the best of both worlds:

• We made sure that dogsByBreed satisfies our Record<Breed, Dog[]> constraint.
• We lost no type information at all about the dogsByBreed variable, and TypeScript was able to tell us that poodle is not one of its keys.

I’m happy to be able to update the previous post to link to this one. Until next time!

I recently started reading Michael Kerrisk’s The Linux Programming Interface. So far, I’ve been enjoying it thoroughly. Inspired by it, I took a look at what strace outputs for a very simple “hello world” program, with the hopes of understanding what’s in it.

Relatively early in the book, Kerrisk introduces the concept of system calls:

A system call is a controlled entry point into the kernel, allowing a process to request that the kernel perform some action on the process’s behalf. The kernel makes a range of services accessible to programs via the system call application programming interface (API). These services include, for example, creatig a new process, performing I/O, and creating a pipe for interprocess communication. (The syscalls(2) manual page lists the Linux system calls.)

The strace program traces system calls and signals, and can attach to a running process or be specified a command to run. In this exercise, I’m going to write a simple “hello world” program in C and run strace with the compiled output.

#include <stdio.h>

int main() {
  printf("Hello world!");
  return 0;
}

I compile it with GCC:

$ gcc -o hello.out ./hello.c

Now we can run it like so:

$ ./hello.out
Hello world!%

Let’s take a first look at the output of strace without any special flags:

$ strace ./hello.out
execve("./hello.out", ["./hello.out"], 0x7fffcc590420 /* 113 vars */) = 0
brk(NULL)                               = 0x55f7df1ee000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fff59ff3fe0) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=75074, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 75074, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fba4a414000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\206\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0 \0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 48, 848) = 48
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0+H)\227\201T\214\233\304R\352\306\3379\220%"..., 68, 896) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1983576, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba4a412000
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2012056, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba4a226000
mmap(0x7fba4a24c000, 1486848, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7fba4a24c000
mmap(0x7fba4a3b7000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x191000) = 0x7fba4a3b7000
mmap(0x7fba4a403000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1dc000) = 0x7fba4a403000
mmap(0x7fba4a409000, 33688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fba4a409000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba4a224000
arch_prctl(ARCH_SET_FS, 0x7fba4a413580) = 0
mprotect(0x7fba4a403000, 12288, PROT_READ) = 0
mprotect(0x55f7dd843000, 4096, PROT_READ) = 0
mprotect(0x7fba4a45f000, 8192, PROT_READ) = 0
munmap(0x7fba4a414000, 75074)           = 0
newfstatat(1, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x1), ...}, AT_EMPTY_PATH) = 0
brk(NULL)                               = 0x55f7df1ee000
brk(0x55f7df20f000)                     = 0x55f7df20f000
write(1, "Hello world!", 12)            = 12
exit_group(0)                           = ?
+++ exited with 0 +++

At first, I’m surprised that there are so many system calls for such a simple program. The system calls executed the most were mmap (8 times) and pread64 (4 times).

Output from strace (prior to the use of any flags) is in the format:

syscall(...arguments) = return_value

The first lines are about loading our program for execution, so let’s take a look at that!

Initializing program execution: execve

The first system call execve executes our program, whose executable can be found at its first argument pathname, and indicates some additional info:

#        pathname       arguments        &env[]            env[].length
1 execve("./hello.out", ["./hello.out"], 0x7fffcc590420 /* 113 vars */) = 0

In the arguments array, the first is simply the name of the program, and I didn’t pass any more arguments. By default, only a pointer to the environment variables array is printed, together with a number indicating how many environment variables are defined. The strace flag -v makes the output verbose by dereferencing pointers such as the one for env[], and -s lets us set the maximum length of the string representation of these values.

$ strace -s 10000 -v ./hello.out
execve("./hello.out", ["./hello.out"], ["ENV_VAR_1=VALUE", "ENV_VAR_2=VALUE"]) = 0
...

So what exactly is meant by “executes our program”? The man page for execve states the following:

execve() executes the program referred to by pathname. This causes the program that is currently being run by the calling process to be replaced with a new program, with newly initialized stack, heap, and (initial‐ ized and uninitialized) data segments.

This leaves us with the question: which currently-running program is execve replacing? I called ./hello.out from a shell. Presumably, if my program were to substitute the shell process, then I wouldn’t have a shell to come back to when my program ends, and the terminal would exit.

Creating a child process: fork

Programs in Linux are executed by means of a mechanism called fork-exec. The fork system call creates a child process by duplicating the calling process.

This means when execve gets called, it’s substituting the cloned instance of our shell in memory, so that the shell process doesn’t cease to exist. You can try opening a new terminal and running a program using exec (as in exec ./hello.out), and you’ll see your terminal closes afterwards!

At the end of execve our process for hello.out has received a newly initialized stack, heap and data segments as fresh new virtual address space. In memory, it looks a little bit like this:

To see what exactly the kernel is doing with the memory of your process, you can run pmap $PID or examine the contents of the files in /proc/$PID/maps, where $PID is the process ID of the process you want to investigate.

At the end of execve, a function called search_binary_handler gets called. It checks the hello.out for a matching magic number (a few bits at the start of the file) to find a handler. The binary handler that matches our program is binfmt_aout.c, which handles .out-style ELF files as produced by GCC. Then, binfmt_aout::load_aout_binary ends up calling begin_new_exec, marked as the “point of no return” in the Linux kernel source, after which any failure results in a fatal signal.

From then on, the kernel takes care of memory-mapping anything required by the executable. The code for this is architecture-dependent (see, for example, different implementations of activate_mm in the kernel). Ultimately, the kernel configures the CPU context to kickstart execution of our process.

P.S. You may be wondering why fork didn’t show up in the output of strace. The answer is simply that strace’s tracing begins after it has forked in order to execute the program passed in as an argument. Thanks, Kenster on StackOverflow.

This is a development log of usairb, a project I'm building to learn embedded Linux. The goal is to transform any embedded Linux device with access to the Internet into a multiplexing transmitter for USB hubs.

See other posts here:

• Listening to devices with libudev – usairb devlog #1 · February 5, 2022

Second development entry. On the first day, I implemented a simple busy loop that prints to the console whenever a USB device connects or disconnects, using a udev monitor. Today, I’ll implement binding those devices using USB/IP, and try to attach to them from a client device.

A spoiler and a word of warning: this didn’t go so well and I ended up giving up on libusbip and reverting the changes described in this post, opting instead for calling the usbip CLI directly from my program using pipes (popen). The next entry will be about that. Nevertheless, I learned a little bit more about sockets, file descriptors and libudev as part of this attempt. Stick around if you’d like to read that!

Introduction

From the USB/IP site:

USB/IP Project aims to develop a general USB device sharing system over IP network. To share USB devices between computers with their full functionality, USB/IP encapsulates “USB I/O messages” into TCP/IP payloads and transmits them between computers.

It looks like the Linux kernel source does not include a library for USB/IP. I’m resorting to this community libusbip for now.

After spending some time reading its source code, it looks like I have a plan:

• Get a device vendor ID and product ID from each connecting device from my udev monitor.
  • Handling two physical devices with the same vendor ID and product ID is out of scope for now.
• Figure out how to set up a socket in order to initialize USB/IP.
• Use libusbip_open_device_with_vid_pid to get a libusbip_device like in the example.
• Confirm that opening and configuring the device actually means the same as going usbip bind on the command line. I believe because libusbip has the same interface for the client and the server, it uses the names open and close instead of bind, unbind (the server terms), and attach, detach (the client terms).

Getting a product ID and a vendor ID from a udev_device

The udev_device_get_property_value function seems like a good choice. USB/IP and udev nomenclature differ a little: vendor IDs are called the same but it’s product ID in USB/IP and model ID in udev.

To find a list of available keys, I ran udevadm monitor --udev --env, and I get something like this:

ID_VENDOR=GenesysLogic
ID_VENDOR_ENC=GenesysLogic
ID_VENDOR_ID=05e3
ID_MODEL=USB2.1_Hub
ID_MODEL_ENC=USB2.1\x20Hub
ID_MODEL_ID=0610
ID_REVISION=9304
ID_SERIAL=GenesysLogic_USB2.1_Hub

With this in mind:

printf("%s %s vid:%s pid:%s\n", udev_device_get_action(device),
       udev_device_get_devnode(device),
       udev_device_get_property_value(device, "ID_VENDOR_ID"),
       udev_device_get_property_value(device, "ID_MODEL_ID"));

Looking good:

add /dev/bus/usb/004/013 vid:090c pid:1000
remove /dev/bus/usb/004/013 vid:(null) pid:(null)

Except that when I remove the device, I no longer get a vid or a pid. I think that might just be fine for now. I’m not even sure if I need to handle removals, so let’s keep going.

Aside: your computer has a file with a database of known vendor and model IDs alongside their names. You can find it in /usr/share/hwdata/usb.ids. I found my flash drive 090c:1000:

 ~ => grep -A6 '^090c' /usr/share/hwdata/usb.ids
090c  Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.)
        0371  Silicon Motion SM371 Camera
        0373  Silicon Motion Camera
        037a  Silicon Motion Camera
        037b  Silicon Motion Camera
        037c  300k Pixel Camera
        1000  Flash Drive

Initializing USB/IP and learning about sockets

Since this is my first contact with sockets at this level, I’m going to record my own introduction here.

After a first read of some man pages I’ve managed to gather some interesting information:

• I can create a socket using the socket(2) syscall. The kernel needs to know what type of socket I want to create. The USB/IP README mentions that it works on TCP port 3240, so going by man socket I need to pass:
  • AF_INET as the protocol family,
  • SOCK_STREAM (I suppose this is TCP as opposed to SOCK_DGRAM which should be UDP),
  • 0 for the protocol, since for most combinations of protocol family and socket type there is only one available protocol, the value 0, which stands for “do the right thing for me”. Let’s hope it delivers.
• TCP sockets provide a two-way binding. The accept(2) syscall takes a connection-based socket (i.e. not SOCK_DGRAM/UDP) and gives you a file descriptor for a new socket, meant for clients to talk to.
  • Helpfully, it also warns that the original socket needs to have gone through some other syscalls first: bind(2) and listen(2).

Aside: to get man socket I had to install man-pages and run mandb, otherwise I was getting Perl documentation, or No manual entry for socket in section 2 if I passed a section.

After some time, I’ve got this:

#include <arpa/inet.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>

int BACKLOG_LENGTH = 1; // Not sure if I need more.

int usairb_connect(int port) {
  int socket_fd = socket(AF_INET, SOCK_STREAM, 0);
  if (socket_fd < 0) {
    fprintf(stderr, "socket returned error code %i\n", socket_fd);
    exit(EXIT_FAILURE);
  }

  struct sockaddr_in socket_address_in = {
      .sin_family = AF_INET,
      .sin_port = htons(port),
      .sin_addr = {.s_addr = INADDR_ANY},
  };
  int socket_len = sizeof(socket_address_in);
  struct sockaddr *socket_address = (struct sockaddr *)&socket_address_in;

  int bind_result = bind(socket_fd, socket_address, socket_len);
  if (bind_result != 0) {
    fprintf(stderr, "bind returned error code %i\n", bind_result);
    exit(EXIT_FAILURE);
  }

  int listen_result = listen(socket_fd, BACKLOG_LENGTH);
  if (listen_result != 0) {
    fprintf(stderr, "listen returned error code %i\n", listen_result);
    exit(EXIT_FAILURE);
  }

  return socket_fd;
}

What took me longest was figuring out what I need to pass to bind, and understanding the cast from sockaddr_in to sockaddr (if you haven’t noticed yet, I’m new to C):

struct sockaddr *socket_address = (struct sockaddr *)&socket_address_in;

Generally, I’m also starting to notice that my style of naming variables does not go well with the style of the kernel API. Maybe one day I’ll budge and start calling a sockaddr by its name.

Accepting connections on the socket seems easier:

int usairb_accept(int socket_fd) {
  int client_socket_fd = accept(socket_fd, NULL, NULL);
  if (client_socket_fd < 0) {
    fprintf(stderr, "accept returned error code %i\n", client_socket_fd);
    exit(EXIT_FAILURE);
  }
  return client_socket_fd;
}

man accept mentions setting errno when the return value is -1. I haven’t been doing that, and other syscalls I’ve been using likely report errors via errno as well.

I sprinkled this after each call that may set errno:

int listen_result = listen(socket_fd, BACKLOG_LENGTH);
if (errno) {
  fprintf(stderr, "listen produced errno %i\n", errno);
  exit(EXIT_FAILURE);
}

Right now, my program runs but produces no output and seems to hang. Even if I add a printf right at the beginning of main, it still won’t print. I’m not sure why that happens, but the hang is certainly related to some of the network syscalls.

Reading man pages for socket, bind, listen and accept, I found that accept is a blocking call by default. At face value, this makes sense because we want to wait for connections to listen to:

If the listen queue is empty of connection requests and O_NONBLOCK is not set on the file descriptor for the socket, accept() shall block until a connection is present. If the listen() queue is empty of connection requests and O_NONBLOCK is set on the file descriptor for the socket, accept() shall fail and set errno to [EAGAIN] or [EWOULDBLOCK].

If I do this in usairb_accept:

int socket_fd_flags = fcntl(socket_fd, F_GETFL);
fcntl(socket_fd, F_SETFL, socket_fd_flags | O_NONBLOCK);

Then the program does not hang. However, as vaticinated by man accept, I get this:

 ~/Development/usairb [127] => ./target/usairb
listen produced errno 11

Running strace ./target/usairb produces some clearer output:

socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
bind(3, {sa_family=AF_INET, sin_port=htons(3240), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(3, 1)                            = 0
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
accept(3, NULL, NULL)                   = -1 EAGAIN (Resource temporarily unavailable)
write(2, "accept produced errno 11\n", 25accept produced errno 11
) = 25
exit_group(1)                           = ?
+++ exited with 1 +++

So the listen queue is empty of connection requests and O_NONBLOCK is set on the file descriptor for the socket, so I’m getting [EAGAIN]. I need to find a way to call accept only if there’s something in the connection request queue.

After some sleuthing, I found this comment on StackOverflow suggesting a way: I can use select to check for listening connections and then call accept only if there’s a request to serve. On further inspection, this is exactly what man accept suggests as well:

APPLICATION USAGE
       When  a  connection is available, select() indicates that the file
       descriptor for the socket is ready for reading.

I also found a great article by Julia Evans introducing this topic. It recommends epoll, but I’m going to go vanilla for learning purposes and use select.

Getting stuck

After getting something hacked together using select and getting the program to compile and run without hanging, I still get no effect that’s visible from the usbip tool (e.g. running usbip list -r localhost does not return anything). I’m starting to lose hope in libusbip. After all, it hasn’t been updated in ten years. I’m going to try a different path: simply calling the usbip executable from my program. There’ll be something to learn by doing it that way, too.

My consolation is that I found out about vendor and model IDs and how to get them with libudev. I also had a first contact with sockets in C and got a tiny bit more experience working with file descriptors.

On the next post I’ll be trying to execute the usbip CLI from my program and act on its output. See you then!

This is a development log of usairb, a project I'm building to learn embedded Linux. The goal is to transform any embedded Linux device with access to the Internet into a multiplexing transmitter for USB hubs.

See other posts here:

• Binding devices with libusbip – usairb devlog #2 · March 18, 2022

First development day. Added an empty CHANGELOG and a README, and started this development log. See the code at the time of writing if you’d like to follow along.

Introduction

I wrote this wishful piece for the README:

The goal of usairb (Universal Serial Air-Bus) is to transform any embedded Linux device with access to the Internet into a multiplexing transmitter for USB hubs: connect gadgets to it and use them remotely from your desktop.

A quick tech overview:

To achieve this, usairb uses USB/IP. USB/IP follows a server-client architecture where the server or host is the device broadcasting its USB gadgets, and the client can connect to them. USB/IP is available as a native Kernel module on Linux for the host, and has multi-platform client programs.

And a justification:

While all planned features of usairb are achievable using just USB/IP, usairb aims to provide a no-frills experience, potentially offering both a client graphical user interface as well as very simple interface for the host device.

I guess you can call it a no-frills experience if there’s no product for you to use!

Planning features of the host device

To build a PoC that broadcasts devices automatically, and that can be used from a client alongside the usbip command-line, interface, the host needs the following features:

• Bind only leaf devices. Avoid binding an entire USB hub.
  • Recognize connected USB hubs.
    • Eventually, reconsider the idea of not binding entire USB hubs. I’m deciding this right now because I don’t know how USB hubs will behave.
  • Eventually, allow the user to bind and unbind specific USB hubs.
    • Remember this across boots. Maybe using systemd or a SQLite database?
• Listen to leaf USB devices when they connect and disconnect.
  • Use libudev for this.
• Automatically bind and unbind USB devices.
  • Call usbip bind for this, using libusbip.

Telling apart leaf USB devices from USB hubs

Running lsusb with the --tree option returns this:

--- log-without-hub     2022-02-05 12:26:40.311210251 +0100
+++ log-with-hub        2022-02-05 12:26:35.234559166 +0100
@@ -1,11 +1,15 @@
 /:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
     ID 1d6b:0003 Linux Foundation 3.0 root hub
+    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
+        ID 05e3:0620 Genesys Logic, Inc. GL3523 Hub
 /:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
     ID 1d6b:0002 Linux Foundation 2.0 root hub
 /:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 10000M
     ID 1d6b:0003 Linux Foundation 3.0 root hub
 /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
     ID 1d6b:0002 Linux Foundation 2.0 root hub
+    |__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
+        ID 05e3:0610 Genesys Logic, Inc. Hub
     |__ Port 8: Dev 2, If 3, Class=Video, Driver=uvcvideo, 480M
         ID 04f2:b6be Chicony Electronics Co., Ltd
     |__ Port 8: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M

The Genesys USB hub shows up twice, under two different trees: first the 3.0 root hub, and then the 2.0 root hub. If I connect a USB 3.0 device to the hub, it doesn’t matter which port I connect it to, it always shows up under the 3.0 tree of the root hub in my laptop:

--- log-with-hub        2022-02-05 12:26:35.234559166 +0100
+++ log-with-device     2022-02-05 12:37:10.480333271 +0100
@@ -1,14 +1,16 @@
 /:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
     ID 1d6b:0003 Linux Foundation 3.0 root hub
-    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
+    |__ Port 1: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
         ID 05e3:0620 Genesys Logic, Inc. GL3523 Hub
+        |__ Port 2: Dev 7, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
+            ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive
 /:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
     ID 1d6b:0002 Linux Foundation 2.0 root hub
 /:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 10000M
     ID 1d6b:0003 Linux Foundation 3.0 root hub
 /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
     ID 1d6b:0002 Linux Foundation 2.0 root hub
-    |__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
+    |__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/4p, 480M
         ID 05e3:0610 Genesys Logic, Inc. Hub
     |__ Port 8: Dev 2, If 3, Class=Video, Driver=uvcvideo, 480M
         ID 04f2:b6be Chicony Electronics Co., Ltd

It looks like lsusb has a way to tell apart hubs from leaf devices. I can probably simply look at the device Class and check to see if it’s root_hub or Hub. This could enable letting the user pick whether to bind a USB hub or their root hub (any ports on the host device itself).

I’m still not certain as to whether I can get this Class information directly from libudev or I’ll need to include some other dependency.

Listening to USB devices as they connect and disconnect

I found this helpful post: libudev and Sysfs Tutorial. I won’t be following it so much, but it seems like a good example. Running man libudev also seems helpful.

As a first step, I’m going to implement a busy loop that listens to connection and disconnection events and just logs any interesting information to the console.

From man libudev:

To monitor the local system for hotplugged or unplugged devices, a monitor can be created via udev_monitor_new_from_netlink(3).

Looks like udev_monitor_new_from_netlink takes a pointer to udev and a name. To get a udev context object, I can call udev_new.

Aside: to set up a bit of a quicker feedback loop, I’m running watchexec: watchexec -c -w src -e c 'make && ./target/usairb'.

After adding -ludev to my Makefile, I wrote this:

#include <libudev.h>
#include <stdio.h>
#include <stdlib.h>

const char *UDEV_MONITOR_NAME = "usairb-udev-monitor";

int main(void) {
  struct udev *udev = udev_new();
  struct udev_monitor *udev_monitor =
      udev_monitor_new_from_netlink(udev, UDEV_MONITOR_NAME);

  if (!udev_monitor) {
    fprintf(stderr, "udev_monitor_new_from_netlink returned NULL\n");
    exit(EXIT_FAILURE);
  }
}

My error message came to fruition instantly. man libudev says “on failure, NULL is returned”, but doesn’t mentions what the reasons for failure are.

I’m wondering if it’s udev_new that’s failing:

if (!udev) {
  fprintf(stderr, "udev_new returned NULL\n");
  exit(EXIT_FAILURE);
}

No, it’s our other friend:

udev_monitor_new_from_netlink returned NULL
[[Command exited with 1]]

I solved it out of sheer luck: I changed the value passed to name to "udev", and now it’s working just fine. The takeaway: the name param is the name of a known Netlink. A Netlink is similar to a domain socket and is used for IPC. It looks like there’s one for "udev", and you need to specify that.

Now I’m looking into what I can do with udev_monitor. The manual only includes entries for functions (such as udev_new) but not for structs. Maybe the monitor is meant to be passed around to other functions. How can I find those? My copy of The Linux Programming Interface by Michael Kerrisk only introduces udev at a high level but doesn’t go into detail. I guess it’s time to read examples on the Internet, unfortunately. This makes me miss the Rust ecosystem. Documentation generated by rustdoc would make cross-referencing searches very easy.

One practical solution I’ve found (besides reading existing work on the Internet) is to let my shell help me: if I type udev_monitor_ and ask for an autocompletion with tab, I get this:

 ~/Development/usairb => man udev_monitor_
udev_monitor_enable_receiving                    udev_monitor_filter_update                       udev_monitor_receive_device
udev_monitor_filter_add_match_subsystem_devtype  udev_monitor_get_fd                              udev_monitor_ref
udev_monitor_filter_add_match_tag                udev_monitor_get_udev                            udev_monitor_set_receive_buffer_size
udev_monitor_filter_remove                       udev_monitor_new_from_netlink                    udev_monitor_unref

That’s good enough. I can also use LunarVim’s LSP hints. Two functions look interesting: udev_monitor_receive_device and udev_monitor_enable_receiving. The latter sounds like a prerequisite, but I’m going to go without it at first and see what happens.

struct udev_device *device = udev_monitor_receive_device(udev_monitor);
printf("Prints if `udev_monitor_receive_device` is not blocking.");

It printed. I added a call to udev_monitor_enable_receiving, but the behavior didn’t change. Looks like my assumption that it’s blocking isn’t holding up!

After some sleuthing in systemd source code, I found this comment that confirms that the call is non-blocking, and suggests two possible paths forward:

• use a variant of poll() on the file descriptor returned by udev_monitor_get_fd, or
• switch the file descriptor into blocking mode.

I don’t know how to do either of these things, so it’s time for some more reading.

TLPI to the rescue! I can use fcntl to modify flags on the file descriptor returned by udev_monitor_get_fd.

Got it to work thanks to some bitwise-fu from a StackOverflow answer:

// udev_monitor_receive_device is non-blocking. To make it blocking,
// get the monitor file descriptor and unset its O_NONBLOCK flag.
int udev_monitor_fd = udev_monitor_get_fd(udev_monitor);
int udev_monitor_fd_flags = fcntl(udev_monitor_fd, F_GETFL);
fcntl(udev_monitor_fd, F_SETFL, udev_monitor_fd_flags & ~O_NONBLOCK);

I wrote a busy loop:

while (1) {
  printf("Listening for new devices...\n");

  struct udev_device *device = udev_monitor_receive_device(udev_monitor);

  if (device) {
    printf("Found a device!\n");
  }
}

Aside: I added -r to my watchexec command so that it restarts the process every time, otherwise the busy loop never exits.

The behavior is not quite what I expected: my program logs "Found a device!" but it seems really excited about it and does it many times over on every connection. Additionally, it does it when I disconnect a device. It feels like a race condition.

To find out, I added a sleep(1); right after printf("Found a device!\n");. To my surprise, my program still “finds the device” a bunch of times. Without sleep(1);, it prints roughly the same amount of times.

After reading the documentation a bit more, I found some new clarifications:

• The existence of udev_device_get_parent pointed out the fact that what I’m seeing is a tree of devices, and not a single device. To find the root, I looked for a device with no parent, and this returned the expected: only one device. The root device always belongs to the bdi subsystem.
• The return value of udev_device_get_action may contain values like "add", "remove", "bind" or "unbind", among others. I should probably start calling my variables event instead of device.

If I run udevadm monitor --udev, I get exactly the same output (well, in a different format):

 ~/Development/usairb => udevadm monitor -u
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing

UDEV  [8858.969668] add      /devices/virtual/workqueue/scsi_tmf_0 (workqueue)
UDEV  [8858.971981] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6 (usb)
UDEV  [8858.973386] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0 (usb)
UDEV  [8858.974132] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0 (scsi)
UDEV  [8858.974950] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/scsi_host/host0 (scsi_host)
UDEV  [8858.975633] bind     /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0 (usb)
UDEV  [8858.978815] bind     /devices/pci0000:00/0000:00:14.0/usb2/2-6 (usb)
UDEV  [8860.074537] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/target0:0:0 (scsi)
UDEV  [8860.075845] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/target0:0:0/0:0:0:0 (scsi)
UDEV  [8860.075899] add      /devices/virtual/bdi/8:0 (bdi)
UDEV  [8860.077712] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/target0:0:0/0:0:0:0/scsi_device/0:0:0:0 (scsi_device)
UDEV  [8860.078456] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/target0:0:0/0:0:0:0/bsg/0:0:0:0 (bsg)
UDEV  [8860.078522] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0 (scsi_disk)
UDEV  [8860.131761] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/target0:0:0/0:0:0:0/block/sda (block)
UDEV  [8860.244986] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda2 (block)
UDEV  [8860.258575] add      /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [8860.259622] bind     /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host0/target0:0:0/0:0:0:0 (scsi)

Looks like new devices are added for different things: the partitions in my USB flash drive, some accessors for protocols I don’t understand (SCSI), etc.

Since they all point to the same thing in /sys/devices, I think I can just filter for events for which udev_device_get_devtype returns "usb_device".

const char *device_type = udev_device_get_devtype(device);
if (!device_type || strcmp(device_type, "usb_device") != 0) {
  continue;
}

printf("Found a USB device:\n");
// ...

Now, I get only two events for each physical action: add and bind (in that order) when plugging in the device and remove and unbind when unplugging it. For now, I’m only going to care about add and remove, and we’ll see about this later!

Currently, the output looks like this when I plug a USB flash drive in and out:

Found a USB device:
   Node: /dev/bus/usb/002/026
   Subsystem: usb
   Devtype: usb_device
   Action: add
Found a USB device:
   Node: /dev/bus/usb/002/026
   Subsystem: usb
   Devtype: usb_device
   Action: remove

When I do the same with my USB hub, I get two distinct devices:

Found a USB device:
   Node: /dev/bus/usb/001/026
   Subsystem: usb
   Devtype: usb_device
   Action: add
Found a USB device:
   Node: /dev/bus/usb/004/015
   Subsystem: usb
   Devtype: usb_device
   Action: add
Found a USB device:
   Node: /dev/bus/usb/001/026
   Subsystem: usb
   Devtype: usb_device
   Action: remove
Found a USB device:
   Node: /dev/bus/usb/004/015
   Subsystem: usb
   Devtype: usb_device
   Action: remove

This is because, like we saw earlier, my hub registers once as a USB 2.0 device and once again as a USB 3.0 device.

I’m calling this one a success! Next time I get a chance to work on this, I’ll be working on binding these devices automatically using libusbip. Thanks for reading!

The website is the result of a great amount of marketing effort. It’s amazing what JS frameworks do these days to get noticed. Regardless of whether the framework lives up to the hype or not, I get a strange feeling from the website: the language attempts to be humorous and light, and this doesn’t help me build trust.

From my notes, the most interesting parts are probably on the <ScrollRestoration /> component and on the loader pattern.

On setting up Remix

• Remix nuked my README.md file (on which I already had some notes) and I had to recreate it. Should have expected that!
• Since Remix picks up on certain exports from files (like export const meta) but my IDE doesn’t know about that, I’m getting a bunch of “unused export” linter errors on those lines. I’m not sure where the errors are coming from, since from my recollection eslint doesn’t complain about unused exports. It has no knowledge of the other files. It’s probably IntelliJ, then.
• The default root.tsx contains several interesting elements:
  • <Outlet /> which is like a Svelte <slot />: it renders children components.
  • <ScrollRestoration />, probably aptly-named. Its presence surprises me because this used to be a feature of react-router that got dropped after the developers noticed scroll restoration working increasingly better out of the box on major browsers.
    • TODO: find out how <ScrollRestoration /> is implemented.
    • Remix documentation on scroll restoration mentions that this component works by restoring the scroll level before rehydration. This should eliminate the jarring effect of having the scroll point being restored after the page loads completely.
      • The “before rehydration” part is true only by virtue of the <ScrollRestoration /> component being used one line before the <Scripts /> component.
• The yarn build command finished in 0.44 seconds. Nice!
• I like that the default app doesn’t contain images or marketing text selling Remix. It just has some documentation links.

The <ScrollRestoration /> component

It’s implemented here and works roughly so:

• It disables the browser implementation with window.history.scrollRestoration = "manual", but only in a useEffect call that runs after useLayoutEffect (used for other effects in the component).
• Provides a <script> tag that runs a simpler effect (it’s simpler because it doesn’t need to know about browser history, since it runs only on the initial script load before hydration).
• Only after the component has hydrated does the component run some more complex logic (e.g. to respect location.hash) that ultimately may run scrollTo(0, position).
• Scroll positions are stored in an in-memory positions dictionary where the keys are location.keys.
• To survive for longer, positions gets added to localStorage and is used to restore positions into memory when the script runs again.

On first-contact DX

• Live reload works great and spinning up the development server takes no time.
• While adding some code and missing an import, the app showed me a build error (expectedly). But then after fixing the error, the app didn’t go back to a normal state, and the error remained, even though the build didn’t show the error any longer.
• Adding the new routes, I’m thinking that the structure recommended by the tutorial where jokes/new shares a namespace with jokes/$jokeId is not great. What if I wanted slugs instead of IDs, and someone created a joke titled new? Other solutions aren’t as pretty, though: jokes/show/$jokeId + jokes/new, or new-joke + jokes/$jokeId.
• By exporting a links array, you can add <link> elements to <head> sort of like with react-helmet. They then get picked up by a top-level <Links /> component.
• The tutorial recommends a file structure for styles that’s not great: a styles directory separate from other parts of the app, such as routes. If the CSS I write is encapsulated together with a route, then why shouldn’t their files live together? Having a separate styles directory adds quite a bit of indirection and could make a project hard to navigate.

On TypeScript use

• The default entry.server.tsx file contains a handleRequest function that takes a request. That’s fine, but it also takes a responseStatusCode and responseHeaders. Am I still able to decide what status code to respond with? This signature is a bit weird to me.
  • Looks like there’s a default 200 response and I still have a chance to change it, as expected. Still, the signature feels awkward.
• I wonder why the tutorial recommends using export let. It looks to me as if the things I’m exporting shouldn’t ever be reassigned. I’m changing these to export const, hoping that nothing explodes.
  • It looks like in other areas of their documentation they use export const, which makes more sense.
• The loader pattern (see related docs) with an export const loader coupled with a useLoaderData call imported from remix feels a bit weird to me. I suppose there must be quite a bit going on at hydration time for this to be worth the indirection. My first impression is that it shouldn’t be necessary? Why can’t I go export const loader = someRemixUtility(async () => {}) instead, removing the useLoaderData call inside the component? The client and server bundles can still provide different implementations of someRemixUtility.
  • TODO: find out why export const loader can’t be used directly and needs to be accessed via useLoaderData. My guess: the purpose is to call the loader during server rendering and then to reuse the same data during rehydration, to initialize a frontend cache with. Remix developers simply decided to go for a standard way to access Remix functionality, and this just looks consistent.
    • It looks like I was correct: Remix hands off serialized data to the client as a string and then has the client route reuse this data. It works similarly to what Apollo GraphQL recommends.
• There’s another insidious result of the indirection introduced by the useLoaderData and export const loader pattern, aggravated by the fact that the suggested type LoaderFunction isn’t generic: discrepancies between what loader actually returns and what useLoaderData returns aren’t going to be caught unless a type is shared between them. But LoaderFunction isn’t generic, so there’s no enforcement from Remix to make sure that this is the case. I can have a loader that returns number but then access string[] in useLoaderData and unless I actively share the types, TypeScript won’t have a chance to complain. In my opinion, LoaderFunction should take a required type argument for the return type, and another for query parameters, which can probably be optional.
  • One could argue that this is the responsibility of the developer, and it is, but by not requiring any type arguments, Remix isn’t helping.
  • It’s also inconsistent because LoaderFunction takes no type arguments but useLoaderData takes a type argument for the returned data.
• When submitting data, the type mismatch is more accentuated, because POST requests contain FormData, which bears (as far as I know) no information on the shape of the data from the form. In this case, however, it matters less, because the tutorial calls the user to perform backend validation/parsing of this form data, by going form.get("field-name") and then validating the result.
  • I wonder whether building a way to type-check JSX forms and providing e.g. a TS eslint plug-in that uses the full power of the AST to build type-safe forms would be worth it. I suppose exporting a simple component for use, TypedForm<T> would be enough, or perhaps even more magic, somehow simply type-check <form> elements based on their current children. This is probably not possible because one would need to go into other modules. Maybe the TypedForm<T> approach is good as long as one shares T with child controls.

That’s it for now! I’ll probably give Remix a proper try once I have a need for it. I feel like the typing issues I mentioned can be avoided through discipline, and the DX looks promising.

Relevant documentation:

• Remix documentation
• The loader pattern in Remix
• React Router documentation on scroll restoration
• useLayoutEffect documentation

My configuration became a Frankenstein of Lua and Vimscript, and this came with unexpected drawbacks such as having a syntax error in the middle of a Lua block be reported as if it had happened at the beginning of the block.

Setting up Neovim to work like an IDE was starting to become a maintenance burden for me, and I didn’t have the patience to clean it all up again. Enter LunarVim:

LunarVim is an opinionated, extensible, and fast IDE layer for Neovim >= 0.5.0. LunarVim takes advantage of the latest Neovim features such as Treesitter and Language Server Protocol support.

That is exactly what I needed! In this post, I’m going to cover my efforts in translating my existing, years-old Neovim configuration into a LunarVim configuration that behaves roughly the same. Some spoilers:

• My Neovim configuration wasn’t as personal as I thought it was. As it turns out, vanilla LunarVim ticks almost all the boxes for me, and it appears so does it for over six thousand users on GitHub. My LunarVim configuration has 69 lines of Lua at the time of writing, whereas my Neovim config has 5 files and a whopping 439 lines of code.
• Adapting LunarVim to my needs has mostly been a breeze, but I did have to investigate a little bit more into a couple of issues.
• The UX inside of LunarVim is welcoming, discoverable and forgiving, so I would recommend it to Vim users who are just about done getting familiar with basic editing skills.

Keybindings

Most of my mappings ended up being redundant.

LunarVim makes excellent use of which-key, a pop-up window that suggests options to help you complete a keybinding.

There’s the reassuring certainty that I’ll be able to find the keybinding again if I’ve found it once, by exploring the nested menus that update as you select an option.

Once I’ve learned a combination, however, it quickly becomes second nature: the keys are simply well-chosen. I remembered my first experience with Magit as I found that the combination to search for text is st, and as I realized that this bit of knowledge would allow me to discover other things that are searchable, by just pressing s and waiting timeoutlen milliseconds for the which-key pop-up to appear.

I did end up adding exactly four keybindings that I missed:

vim.cmd("map 0 ^")

vim.cmd("nnoremap Q <nop>")

vim.cmd("nnoremap j gj")
vim.cmd("nnoremap k gk")

Fuzzy finder

I’ve used the magnificent fzf.vim for some years now. LunarVim comes with Telescope, though, and there are some things I’m enjoying about it.

One of them is that it simply looks great: it goes really well together with a patched font (in this case Cascadia Code SemiLight) thanks to the iconography, and the preview on the right is syntax-highlighted, and it’ll hide responsively depending on the available size of the terminal.

Each fragment in a file path can be truncated to fit a longer path in the screen. I thought this would be confusing at first, but I’ve had no trouble with it.

I have no complaints about the file fuzzy finder, but the “live grep” functionality has one big drawback. With fzf.vim, I used to be able to do this:

:Rg search_string

This would pop up a view similar to Telescope, and I could fuzzy-filter the search results. This was incredibly useful, for example, if I want to find search_string but realize after the fact that most results are in Python files, while I’m interested in the TypeScript results. I would then simply type .ts and get all files containing search_string in TypeScript files only. Very flexible!

Unfortunately, this is not possible in vanilla Telescope, and I’ve yet to find a solution for this. It is a big part of my daily workflow, so I might revert to fzf.vim for my text-search needs.

Thanks to two helpful Redditors, I’ve managed to find a solution for this. With the following snippet, pressing <leader>sT will do something very similar to :Rg, and will also take the word under the cursor as an initial value:

function GrepInputString()
  local default = vim.api.nvim_eval([[expand("<cword>")]])
  local input = vim.fn.input({
    prompt = "Search for: ",
    default = default,
  })
  require("telescope.builtin").grep_string({ search = input })
end
lvim.builtin.which_key.mappings["sT"] = { "<cmd>lua GrepInputString()<CR>", "Text under cursor" }

File explorer

I was a happy user of nnn, a terminal file explorer with a Vim plugin. It’s great, but some features are compile-in, which may be a nuisance to some.

LunarVim comes with nvim-tree. Honestly, I can’t complain. I wasn’t an advanced user of nnn. In nvim-tree, all features are quickly discoverable by entering g? (“go to help”, in and of itself an easily discoverable combination).

Git interface

During my time using Emacs I became a fan of Magit. It’s a great piece of software that changed my expectations of what a Git interface can do.

Since then, I’ve been through different solutions, including using an always-running Emacs instance in server mode running only Magit, which I accessed inside of Neovim and other programs. Later on, I used tpope’s vim-fugitive and tried to adapt some Magit-inspired combinations such as p-fp for “push”, “force”, “push”.

LunarVim’s porcelain of choice is lazygit. I’d been meaning to try lazygit for a while now, and this was the perfect opportunity. The “unboxing” experience was great: lazygit shows a link to a YouTube video by the author Jesse Duffield. I can’t make the software justice in this introduction, so I would recommend that you simply try it out.

One minor drawback of using lazygit inside of Neovim is that its open and edit commands when pointing at files will open a nested $EDITOR inside of the lazygit terminal. With vim-fugitive, opening a file in its UI would simply reuse the current Neovim instance. I had this expectation of lazygit running inside LunarVim as well. I’ve read about using Neovim in remote mode to solve this, but I haven’t gotten around to it. I do feel like if LunarVim picks lazygit as its integrated Git interface, it could also come with a working solution for this.

Verdict

I think I’ll keep using LunarVim. With my old configuration, It’s a rare week if I don’t need to fix one or two things on one of my LSP setups, or any of the IDE-like features that Neovim plugins offer. This all just works mostly out of the box on LunarVim, and not having to maintain this is a great relief. I’m looking forward to not being in complete control of this.

If you can relate, I’d like to invite you to give LunarVim a try and see if you like it. Thanks for reading my blog!

]]>Fausto Núñez Alberrofausto.nunez@mailbox.org

2021-02-21T19:00:51+00:002021-02-21T19:00:51+00:00https://fnune.com/embedded-learning-log/2021/02/21/embedded-learning-log-3-running-a-recurring-freertos-task

This post is part of a series where I log my experience getting into embedded software development. The overarching goal is to use the ESP8266 RTOS SDK to learn how to operate my ESP8266 board using an RTOS and to learn how to integrate the Memfault real-time SDK. Disclaimer: I work for Memfault as a full-stack engineer, mostly on web stuff. Here are the other posts in the series:

• Embedded learning log – Integrating the Memfault Firmware SDK
• Embedded learning log – A foray into Linux specifics: dmesg, tty and playing with minicom
• Embedded learning log – Connecting a board and flashing an RTOS

Browse the accompanying repository at the commit in which I was done with this part.

Documentation on the ESP8266_RTOS_SDK does not feature any information about task creation. I guess this is because the underlying RTOS is FreeRTOS, so I’m going to try to find documentation for that.

Here’s the documentation.

Apparently I can create tasks with xTaskCreate or xTaskCreateStatic. If I want to use xTaskCreateStatic, I need to provide my own memory, so I need to pass more parameters, but I can allocate this memory at compile time. I’m going to go with the simpler xTaskCreate, which allocates from the FreeRTOS heap.

The most important argument to xTaskCreate is the task function I want to run:

Tasks are normally implemented as an infinite loop, and must never attempt to return or exit from their implementing function. Tasks can however delete themselves.

To create a task that executes periodically, I just need to make it an infinite loop that calls vTaskDelay in it or vTaskDelayUntil. I’m going for the simpler vTaskDelay.

Another interesting argument is usStackDepth. It’s in words, and the docs say a word is 32 bits in an ESP8266EX. One choice would be to find out the minimum stack size required for a task in FreeRTOS. I found this interesting article: GNU Static Stack Usage Analysis by Erich Styger. I can pass -fstack-usage to gcc and get a nice .su file that gives me the stack usage of each function in bytes. I can append to CFLAGS for specific components, so I did so in esp8266_sdk/memfault/component.mk, and then found this file memfault/memfault_platform_http_client.su:

memfault_platform_http_client.c:178:5:memfault_platform_http_client_post_data	192	static

That’s it! 192 bytes divided by my word size of 4 is exactly 48. I’ll use 48 as a stack size for my task and eventually test to see if I can get by with less (I doubt it).

After creating the task, I get an crash loop due to a stack overflow on my new task_upload_memfault_data. Is my stack size not big enough?

Finding out the stack size needed by a function

I found this interesting article: GNU Static Stack Usage Analysis by Erich Styger. I can pass -fstack-usage to gcc and get a nice .su file that gives me the stack usage of each function in bytes.

I found out I can pass append to CFLAGS for specific components, so I did so in esp8266_sdk/memfault/component.mk, and then found this file memfault/memfault_platform_http_client.su:

memfault_platform_http_client.c:178:5:memfault_platform_http_client_post_data	192	static

That’s it! 192 bytes divided by my word size of 4 is exactly 48. I’ll use 48 as a stack size for my task and eventually test to see if I can get by with less (I doubt it).

The stack size I calculated wasn’t enough

I started by trying to allocate a stack of 48 words for my task_upload_memfault_data function, but that crashed my device due to bad allocations. I bluntly tried to increase it gradually until it worked, and it did! At 4096 words. I was off by a lot.

Could this be due to the fact that I measured stack usage for memfault_platform_http_client_post_data and not for my task’s function task_upload_memfault_data? Let’s build with -fstack-usage again, this time scouting for my task function instead. The number reported for memfault_platform_http_client_post_data was 192 bytes.

The output contains this:

main.c:148:6:task_upload_memfault_data	16	static

Eew! 16 is a lot less than 192. It seems like this isn’t going all the way down the call stack to calculate stack usage. Maybe I’m missing some other flag?

Runtime analysis of the target stack size

I’m going to try to let the code run and analyze how much memory the task takes after the fact. I probably need to make the task run through all of its different branches and take the maximum.

I remember that allocation of tasks created via xTaskCreate is handled by FreeRTOS, and there’s another way to create tasks that lets me control allocation: xTaskCreateStatic.

My plan is to:

1. Refactor my task to use xTaskCreateStatic.
2. Flash and run the code.
3. Figure out how to analyze the memory layout after the fact.
4. Read how much of the stack I allocated was actually used by the task.

To be able to access xTaskCreateStatic I had to change configSUPPORT_STATIC_ALLOCATION to 1 in components/freertos/include/freertos/FreeRTOS.h. Surprisingly, this setting wasn’t available under make menuconfig -> Component configuration -> FreeRTOS.

I’m giving up on this approach and returning to static analysis because compiling with configSUPPORT_STATIC_ALLOCATION requires me to implement my own allocators/deallocators for FreeRTOS’s idle task, and I’m… not up for the task.

Static analysis of the target stack size

The reason why my 192 number was wrong is because gcc outputs .su files for each .o file it produces; it doesn’t look into other output objects to determine stack usage in a recursive fashion.

I found this Perl script by Daniel Beer avstack.pl which reportedly builds this tree or call graph for you and then sums the results:

The script reads all .su files, and disassembles all .o files, including relocation data. The disassemblies are parsed and used to construct a call graph. Multiple functions in different translation units with the same name don’t cause problems, provided there are no global namespace collisions. Information will appear on any unresolvable or ambiguous references.

I’ve added the script and results to the repository in src/avstack.txt. Functions prefixed with > are supposed to have a summary of function calls in lines below them, but my run seems to have produced a flat stack where no function calls other functions in its body. It’s weird, and I got a 196 for memfault_platform_http_client_post_data, which looks like it’s my previous 192 plus 4. This 4 is passed as an argument to avstack.pl called call_cost.

So! I’m giving up on this method as well. My last attempt: using FreeRTOS’s uxTaskGetStackHighWaterMark.

Measuring the target stack size at runtime using uxTaskGetStackHighWaterMark

I’ve instrumented my task like this:

void task_upload_memfault_data( void * pvParameters )
{
    // [...]
    uxHighWaterMark = uxTaskGetStackHighWaterMark(NULL);
    ESP_LOGD(TAG, "Water mark at task entry: %lu", uxHighWaterMark);

    while (true)
    {
        // [...]
        uxHighWaterMark = uxTaskGetStackHighWaterMark(NULL);
        ESP_LOGD(TAG, "Water mark at task exit: %lu", uxHighWaterMark);
    }
}

The output looks like this, with irrelevant sections edited out:

D (807) TOASTBOARD: Water mark at task entry: 3976

D (10832) TOASTBOARD: Water mark at task exit: 3140

D (29036) TOASTBOARD: Water mark at task exit: 580
D (39037) TOASTBOARD: Water mark at task exit: 580
D (49039) TOASTBOARD: Water mark at task exit: 580
D (59041) TOASTBOARD: Water mark at task exit: 580
D (69042) TOASTBOARD: Water mark at task exit: 580
D (79045) TOASTBOARD: Water mark at task exit: 580

So it’s 3976 on entry, on its first exit it’s 3140, and then it’s in the infinite loop (it has a 10 second delay after each run) and stabilizes at 580.

The initial drop from 3976 to 3140 is explained by the documentation:

Calling the function will have used some stack space, we would therefore now expect uxTaskGetStackHighWaterMark() to return a value lower than when it was called on entering the task.

In my case, “the function(s)” are vTaskDelay and memfault_esp_port_http_client_post_data.

The second drop from 3140 to the stable 580 is explained by the fact that memfault_esp_port_http_client_post_data returned early because there was no more data to post. Cool!

Two questions remain:

• Is the high water mark going to be higher than 3976 if there’s more than one chunk to send to Memfault? I’ve seen it post several at a time. I guess the stack size is going to be the same if the code runs inside a loop.
• I need to try and allocate a more optimized stack size to see if that affects execution. My new size in words is 3976.

With 3976 it works just fine, but on a new run the high water mark is 3864. Why couldn’t it be more than my allocated 3976? Do people pad these values? I’m sticking with my original 4096 for good measure.

Finding out the size of a word manually in C

I remember in Rust this would be the size of usize. I guess in C I can go sizeof(some_type) to get this.

After some Googling, I found size_t, which varies depending on the address size of the processor, and is the type returned by sizeof. So let’s try sizeof(size_t).

ESP_LOGI(TAG, "Size of a word is %u bytes!", sizeof(size_t));

And then in make monitor:

I (532) TOASTBOARD: Size of a word is 4 bytes!

Four bytes are 32 bits. Thanks Toastboard!

Next I’ll be integrating the Memfault Firmware SDK.

]]>Fausto Núñez Alberrofausto.nunez@mailbox.org

2021-02-21T19:00:51+00:002021-02-21T19:00:51+00:00https://fnune.com/embedded-learning-log/2021/02/21/embedded-learning-log-4-integrating-the-memfault-firmware-sdk

This post is part of a series where I log my experience getting into embedded software development. The overarching goal is to use the ESP8266 RTOS SDK to learn how to operate my ESP8266 board using an RTOS and to learn how to integrate the Memfault real-time SDK. Disclaimer: I work for Memfault as a full-stack engineer, mostly on web stuff. Here are the other posts in the series:

• Embedded learning log – Running a FreeRTOS recurring task
• Embedded learning log – A foray into Linux specifics: dmesg, tty and playing with minicom
• Embedded learning log – Connecting a board and flashing an RTOS

Note: I make many mistakes during this integration. It’s not an example of how one should go about it, but just a log of my first attempt as a newcomer to embedded development. If you’re looking for a guide, the documentation is a perfect place to start.

I’m starting by getting some application code in place. It’s just going to be an HTTP server with simple routes adapted from the ESP8266_RTOS_SDK examples. Once I’m done, I’ll try to integrate the Memfault SDK on top.

After following the basic steps in the documentation (here’s the related commit), I ran into this while compiling:

In file included from /home/fausto/Development/toastboard/src/components/memfault_port/memfault-firmware-sdk/components/include/memfault/components.h:20,
                 from /home/fausto/Development/toastboard/src/main/main.c:17:
/home/fausto/Development/toastboard/src/components/memfault_port/memfault-firmware-sdk/components/include/memfault/config.h:30:39: fatal error: memfault_platform_config.h: No such file or directory
 #define MEMFAULT_PLATFORM_CONFIG_FILE "memfault_platform_config.h"
                                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

I later found this from the CHANGES.md file in the Memfault SDK:

- You must create the file `memfault_platform_config.h` and add it to your
  include path. This file can be used in place of compiler defines to tune the
  SDK configurations settings.

After solving that by creating src/main/include/memfault_platform_config.h, there was another missing header file:

/home/fausto/Development/toastboard/src/components/memfault_port/memfault-firmware-sdk/ports/esp8266_sdk/memfault/esp_reboot_tracking.c:14:10: fatal error: internal/esp_system_internal.h: No such file or directory
 #include "internal/esp_system_internal.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I got a bunch of compiler errors because I was on the master branch of ESP8266_RTOS_SDK and the Memfault integration supports only v3.3. The changes from master to v3.3 are significant in the HTTP server example, so I’m updating my application code. The next compiler error was a missing header file esp_console.h expected in memfault_cli.c. I could solve it by adding this to src/sdkconfig:

CONFIG_USING_ESP_CONSOLE=y

Et voila! It has built! Time to get it flashed onto the board.

Reboot loop after integrating the SDK

There are some useful logs:

I (492) system_api: Base MAC address is not set, read default base MAC address from EFUSE
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

And:

E (463) mflt: Coredumps enabled but no storage partition found!

I’m not sure which one is causing the reboots, so I’m going to fix the MAC address problem first. It should be a matter of using a different method that doesn’t get the “base” MAC address but the one from EFUSE. For context, I’m trying to use the MAC address as a device identifier to send to the Memfault cloud.

I gave up trying to fix the MAC address problem because no matter what I picked, it would crash. I’ll name this device "the-one-and-only" instead so that I can skip the problem of fetching the MAC address.

We’re left with:

E (463) mflt: Coredumps enabled but no storage partition found!

It’s a bit weird because I’m still not at the step where I can add coredumps integration.

Either way: I can print the partition table of the device with make partition_table:

# Espressif ESP32 Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  24K,
phy_init, data, phy,     0xf000,  4K,
factory,  app,  factory, 0x10000, 960K,

My chip supposedly has 4MB of storage, and that’s roughly 1MB in the partition table, so I guess I could flash something else onto it. For now, I’m going to go with this.

Now I need to:

1. Identify which of those is the storage partition.
2. Figure out how to tell Memfault to use that.

The ESP8266_RTOS_SDK documentation on partition tables says NVS stands for non-volatile storage. I guess that’s what we want. That means I want the partition starting at 0x9000 that’s 24K in size.

I’m going to look for a way to do (2) in make menuconfig.

The Memfault-specific configuration has an option “Enable saving a coredump to flash storage (OTA slot)”. The other choice is “User defined coredump storage region”. I guess I could pick the second, but there’s a default partition table that includes an OTA slot from the SDK, so I’m going to use that instead of the NVS partition.

The configuration changed like so:

- CONFIG_PARTITION_TABLE_SINGLE_APP=y
+ # CONFIG_PARTITION_TABLE_SINGLE_APP is not set
- # CONFIG_PARTITION_TABLE_TWO_OTA is not set
+ CONFIG_PARTITION_TABLE_TWO_OTA=y

I’m going to add that to the defaults file because it seems critical to the project. I also changed how I’m getting my WiFi SSID and password. Time to flash it again!

The reboot loop continues

But now it’s different! There are no more errors from Memfault, and I get a reassuring message, but the device still reboots in a loop:

I (499) mflt: Coredumps will be saved at 0x110000 (983040B)

This means my new partition table is being used correctly.

To begin debugging, I removed the memfault_platform_boot() call. Now, the device works correctly and does what it’s supposed to do (not much). I can look into my Memfault-related files to find the culprit.

After some time trying to figure this out, I’m going to give up and hardcode the hardware version instead of trying to get it from chip_info, since I’m suspicious of my C.

Now, the device is stable! I just don’t know C.

Sending data

I need to add memfault_esp_port_http_client_post_data() somewhere.

My device exposes an HTTP server, so I decided to create a GET /upload route that will upload anything saved up to Memfault.

I open 192.168.0.24:80/upload on a browser and…

D (30030) mflt: Posting Memfault Data
D (30339) mflt: Posting Memfault Data Complete!
I (30342) mflt: Result: 0

My device now exists on Memfault!

the-one-and-only	1.0.0+4bb166	ESP8266EX_CH340C	a minute ago

The ESP-IDF console

The Memfault docs mention a crash CLI command. I’m not sure what this is. I tried to find some information about this, thinking it’s related to the CONFIG_USING_ESP_CONSOLE=y flag I passed in for compilation. The ESP8266 documentation doesn’t mention the console component or the crash CLI command.

ESP-IDF documentation mentions this:

ESP-IDF provides console component, which includes building blocks needed to develop an interactive console over serial port.

I guess I’m supposed to build the interactivity into a new target? Maybe I’m supposed to test with the console example provided with the RTOS.

I’m blocked at this step because the next steps depend on this: the crash data I could get from the crash CLI would be uploaded with the memfault CLI, and then the web application would warn me there’s a missing symbols file for the data, and I could then upload the symbols file and get issues from it.

I guess I need to find a way around the problem if I can’t find out what this crash CLI command is.

Trying to force a crash by flashing buggy code

Let’s try adding an infinite loop in a GET request handler under /crash.

esp_err_t crash_get_handler(httpd_req_t *req)
{
    while (true) {
      // ...
    };

    const char* resp_str = (const char*) req->user_ctx;
    httpd_resp_send(req, resp_str, strlen(resp_str));

    return ESP_OK;
}

httpd_uri_t crash = {
    .uri       = "/crash",
    .method    = HTTP_GET,
    .handler   = crash_get_handler,
    .user_ctx  = "The application should crash before this shows up"
};

Calling /crash left my browser loading for about 30 (!) seconds before the device crashed. Here’s the full output of the crash, from make monitor:

Saving Memfault Coredump!
Erasing Coredump Storage: 0x0 983040
coredump erase complete
Task watchdog got triggered.

Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Core 0 register dump:
PC      : 0x4026f3d4  PS      : 0x00000030  A0      : 0x40247d9c  A1      : 0x3fff57c0
0x4026f3d4: crash_get_handler at /home/fausto/Development/toastboard/src/main/main.c:54

0x40247d9c: httpd_uri at /home/fausto/.esp/ESP8266_RTOS_SDK/components/esp_http_server/src/httpd_uri.c:218

A2      : 0x40107cec  A3      : 0x4026f3d4  A4      : 0x00000002  A5      : 0x00000004
0x4026f3d4: crash_get_handler at /home/fausto/Development/toastboard/src/main/main.c:54

A6      : 0x00000073  A7      : 0x00000001  A8      : 0x00000001  A9      : 0x00000068
A10     : 0x00000068  A11     : 0x80808080  A12     : 0x40107c9c  A13     : 0x40107cec
A14     : 0x40107c9c  A15     : 0x40107cec  SAR     : 0x00000018  EXCCAUSE: 0x00000000

Backtrace: 0x4026f3d4:0x3fff57c0 0x40247d9c:0x3fff57c0 0x4024853b:0x3fff57e0 0x402485ed:0x3fff5860 0x402475df:0x3fff5880 0x40246f58:0x3fff5890 0x40246fc4:0x3fff58b0
0x4026f3d4: crash_get_handler at /home/fausto/Development/toastboard/src/main/main.c:54

0x40247d9c: httpd_uri at /home/fausto/.esp/ESP8266_RTOS_SDK/components/esp_http_server/src/httpd_uri.c:218

0x4024853b: httpd_parse_req at /home/fausto/.esp/ESP8266_RTOS_SDK/components/esp_http_server/src/httpd_parse.c:527

0x402485ed: httpd_req_new at /home/fausto/.esp/ESP8266_RTOS_SDK/components/esp_http_server/src/httpd_parse.c:594

0x402475df: httpd_sess_process at /home/fausto/.esp/ESP8266_RTOS_SDK/components/esp_http_server/src/httpd_sess.c:327

0x40246f58: httpd_server at /home/fausto/.esp/ESP8266_RTOS_SDK/components/esp_http_server/src/httpd_main.c:113

0x40246fc4: httpd_thread at /home/fausto/.esp/ESP8266_RTOS_SDK/components/esp_http_server/src/httpd_main.c:113

@R%dPMT%w@Y)Q
OT5V'I
T       )dE     5Y
NZA     I^NTR
@\Y!NZA I^{YP1g@\@Y
[)!E   pL@LYYX)IIPKVqRA I       YЛ     pL@PWK   V       ZQHYYV[)!E     pL@Y^ZYY
                         1eʉE  xO@Y     J5)aEEAJA%Y     J)Q
AEaE   xL@Y     J-I
%ʉE    xO@YZAFPYM)!E   `M@YI9V5ԻT)dVEEAJAH%Y=eZ)R
                                                 E
                                                  E    h%Y-JP-G

                                                               E       h%Y1iGQP)PNP-E  h%Y--HPPG
-AKRMYV@Q                                                                                       YAJAH^-G-HPP-YAJAH^w\PRZANPYEEAJAHq
         E     h)YVYP1DT
                        PTg
                           TWp)ʉE      hM)YVY1DT
                                                P%=DT
                                                     T5қ
                                                        @)ʉE   h)YVYRP#PR
5P@H)IJ)!E     hN)YVRP%=DT
                          Tg
)IJ)!E h                    P!)IJ)!E   h)YVYPRPT5eHP@
        %YYP-^1qQHT-TRYI (411) system_api: Base MAC address is not set, read default base MAC address from EFUSE

It looks like the system gave me the backtrace and the exact location of my crash: the body of the crash_get_handler. What a surprise.

On the Memfault application nothing’s showing up yet. I guess it’s because my device doesn’t upload data unless I tell it to, using the GET /upload route.

After I upload the data in the device, I get this in the issues view:

Unprocessed traces exist due to missing symbol file(s):

Upload 1.0.0+5a05af (toastboard-firmware)

Cool! Uploading that now. I’m such a noob that I don’t even know what I’m supposed to upload. I think it’s build/toastboard.elf, though.

Let’s try crashing the same away again to test issue deduplication. While my device takes some time to decide to crash through the infinite loop, I’m thinking of two questions:

1. Why does it take so long for it to crash? I’d have guessed the timeout is shorter. Although it’s an infinite loop, shouldn’t it still be cancelable by some interrupt or whatever mechanism the RTOS has?
2. Why is it showing up as a Hard Fault instead of a Watchdog issue? Wasn’t the device restarted by the watchdog?

Looking around in make menuconfig, I found the answer to (1): Task Watchdog timeout period (seconds) (26.2144s) under ESP8266-specific. I was pretty close with my 30 seconds guess!

I think I can answer (2): the issue is a Hard Fault. The reboot event should be have the watchdog as a cause. However, it shows up as Unspecified. I guess I have something else to integrate.

Issue deduplication seems to be working fine. Here’s what the Queue Status widget shows:

a few seconds ago	Attached to Issue #38075
26 minutes ago	Attached to Issue #38075

Both were attached to issue 38075.

Integrating reboot reasons

Here’s the documentation I’m following.

I guess my reboot is showing up as having an Unspecified reason because I haven’t completed integration. Let’s continue with that.

The guide mentions I should #include "memfault/core/reboot_tracking.h", but I confirmed that’s already included with "memfault/components.h".

There are two things I need to figure out from my chip and use in the initialization code:

1. A pointer to the reboot reason register.
2. Additional information about the reboot reason, mapped to a MfltResetReason.

As for (2), the ESP8266_RTOS_SDK documentation points to a function esp_reset_reason that returns a value of type esp_reset_reason_t. I need to figure out how to map from esp_reset_reason_t to MfltResetReason.

As for (1), I can’t find information in the documentation indicating how to access that register. There’s only the functions described above. Looking at the RTOS source code, I found this: RTC_RESET_HW_CAUSE_REG. I’m wondering if that’s defined. To find that out, I’m going to try to compile with that and kMfltRebootReason_Unknown for now.

error: 'RTC_RESET_HW_CAUSE_REG' undeclared

Nope! However, I found that RTC_RESET_HW_CAUSE_REG points to a register RTC_STATE1, which I can access if I #include "esp8266/rtc_register.h". This feels really hacky, but the code now compiles!

Reboot reasons still show up as Unspecified and Unexpected Reset. Time to continue with the integration: mapping esp_reset_reason_t to MfltResetReason and passing it to the Memfault SDK.

I built the mapper and passed it to memfault_reboot_tracking_boot, and everything compiles just fine, but the reboots in the app are still Unspecified. There’s more integration steps to take, so I probably need to continue.

I needed to dedicate some storage space for events and pass it to Memfault. Once that was done, I tortured the little thing once more with the GET /crash endpoint, and voila! There’s a reason for my reboot on the Memfault application: Software Watchdog.

This is a bit weird because the logs from make monitor show this:

I (503) mflt: ESP Reset Cause 0x7
I (509) mflt: Reset Causes:
I (514) mflt:  Hardware Watchdog

Maybe I got the mappings wrong!

/* Reset (software or hardware) due to interrupt watchdog. */
case ESP_RST_INT_WDT: return kMfltRebootReason_HardwareWatchdog;
/* Reset due to task watchdog. */
case ESP_RST_TASK_WDT: return kMfltRebootReason_SoftwareWatchdog;
/* Reset due to other watchdogs. */
case ESP_RST_WDT: return kMfltRebootReason_SoftwareWatchdog;

Reset cause 0x7 corresponds to “other watchdogs”, so I’m going to map that to HardwareWatchdog instead, since that’s what the logs say. The Hardware Watchdog log from the Memfault SDK was there long before I added my mapping, so that suggests to me that they knew the reset reason all along. Why did I have to implement the mapping, then?

As it turns out, it’s because I didn’t read the port included in the SDK well enough (duh). It includes an implementation of memfault_reboot_reason_get which I should have used from the start.

Sending data periodically

I simply used the code from my previous investigation on how to create a recurring task on FreeRTOS and called memfault_esp_port_http_client_post_data from there. Here’s how it ended up looking.

So! My device is sending data to the cloud. I’d call this a success for now.

]]>Fausto Núñez Alberrofausto.nunez@mailbox.org