Jeremy Cline's Blog

Flock 2026

Thu, 09 Jul 2026 16:14:00 +0000

Another Flock to Fedora conference has come and gone, and like last year, this one was held in Prague. Unlike last year, I was not in the middle of moving across the country (again) so I was able to attend, thanks to my employer.

As always, it was great to see so many familiar faces and meet new folks face-to-face. To those of you who weren’t able to make it, you were missed. And, as always, I spent a lot of time in the hallway track talking to people, getting a sense of what everyone was working on and interested in.

Day -1

The day before the conference started, there was a sponsorship dinner. Although Bex did all the work getting the paperwork to the correct people at Microsoft, he wasn’t able to make it to Prague in time for the dinner so I was sent. I arrived on Saturday morning after a quick connection through Dublin, which gave me plenty of time to get settled in and resist taking a nap. I spent the dinner chatting with Kevin Fenzi and Jef Spaleta, and while I can’t remember all the topics, curling was definitely mentioned.

Day 0

I volunteered to help at the check-in desk the morning of the first day, which I felt went very smoothly (the new label makers were a nice addition). It was nice to help out, but it was also a great way to match names I’ve seen on Matrix to faces as folks arrived. After my shift, I got sucked into the hallway track until lunch.

After lunch and a bit more hallway track, I went to the “PR-based Gating for Fedora: Can We Make It Work?” workshop from František Lachman. There was a lot of discussion in and around the Fedora contribution workflow which I have lots of thoughts about, but I felt there was a rather widespread desire to make things better (even if the exact way we do that isn’t clear). Lots of people who were not me brought up keeping the specfiles in one repository rather than forty thousand or however many git repositories we’re up to. In any case, I’d really like a nice pull request workflow for Fedora where I can’t mess up updates, and where we can all share the tooling we build around packaging.

I spent the rest of the day in the hallway track, doing some last minute preparations for my talk on signing, and preparing for the joint Microsoft talk with Reuben and Bex. I was happy to meet some of the Red Hat folks working on cryptography and signing, and I’m hopefully somewhere down the line we can all do approximately the same thing for signing content.

I got dinner with Bex and Reuben at some place that served North Carolina style BBQ, and it was pretty good (especially with kimchi on top!).

Day 1

This was the first day of recorded presentations. I went to the usual “State of Fedora” address, followed by the Fedora Council and FESCo panels. I thought it was interesting (but sadly unsurprising) to see downward trend of contributors, and I’d be interested to see a further breakdown of who’s leaving. I have plenty of not-backed-by-hard-data ideas about why this is happening, but I do hope it leads to a stronger focus on (and acceptance of) improving the contribution experience - the general feeling in the hallways, as I mentioned earlier, makes me somewhat optimistic.

After lunch and a bit of hallway track, I went to the “Secure by Design: Aligning Fedora with the EU Cyber Resilience Act (CRA)” workshop by Jaroslav Řezník and Roman Zhukov. A good portion of it was a run down of what the CRA entailed and how the roles it describes map into Fedora. After that I spent a bit of time preparing for my talk. My talk went well, I think, except the live demo didn’t entirely work (gpg2 + gpg-agent + gnupg-pkcs11-scd is very finicky and I forgot a setup step). With that stressful event out of the way, I was able to relax a bit at the dinner party, chat with numerous folks, and fill up on the “appetizers” they brought out in vast quantities. Big props to the event organizers, the weather was great and I really appreciated the open space and variety of food options.

Day 2

It was hard to believe it was already the final day of the conference, but I think at this point I was also feeling pretty worn out. I went to Justin’s “State of the Fedora Kernel” talk, and was glad to hear that the GitLab workflow I helped build before I left wasn’t absolutely terrible. I made some last minute edits to the slides for our “Two Years In: Accelerating Microsoft Contributions to Fedora” talk (where I was happy to have Bex and Reuben do most of the talking), then helped present that talk. Afterwards I went to the “What’s new in Fedora CoreOS” talk and managed to chat with Jean-Baptiste Trystram and Joel Capitao about signing and Konflux, which we’ll hopefully get sorted out in the next couple weeks (in the staging environment, anyway). Hopefully we’ll also be able to get Fedora CoreOS images into the Azure community gallery alongside the Cloud images.

The lightning talks were all enjoyable, and I’m really impressed some folks even managed to make up slides for theirs and nothing went terribly wrong (great work everyone). There was time for a bit more hallway track, and then I went to the Contributor Recognition Program, which concluded the presentations for Flock 2026. I spent the evening catching up with old and new friends, chatting about ideas on improving various bits of Fedora infrastructure, and how to make the contributor experience better. People were already leaving for DevConf (or home) at this point, so if I didn’t get a chance to say goodbye, I’m sorry and I hope we’ll see each other next year!

Day 3

It was an uneventful trip back home, thankfully.

I’m looking forward to put all the work I’ve done on improving Fedora’s signing infrastructure through its paces, to get support for PQC done, and I have a few ideas on what to work on next. Hopefully some of them work out and don’t lead to too many people screaming at me. Flock is a great event to get excited about the next year of work and to test the waters on wild ideas, so I’m really glad I was able to make it this year.

Fedora signing: draw the rest of the owl

Sat, 20 Jun 2026 19:17:00 +0000

Way back in September I apologized for the gap since my last update. I am, once again, sorry it’s been so long since I posted an update. I recently presented at talk at Flock 2026 - if you prefer consuming updates via video where live demos fail, you can find that on YouTube.

This post is something of a summary of that talk, but the short version is Siguldry has all the features Sigul has, plus some. That means we can start deploying it in the coming weeks.

PKCS#11

In the last post, I noted I had implemented server-side support for PGP signing. That’s gone now, I’m pleased to say, because I came across a much neater approach. I implemented a small PKCS#11 module inventively called siguldry-pkcs11.

I can’t take credit for any of the ideas I’m about to describe, since I stole them from multiple other projects. Our friends over in Flatcar have to sign things as well, and they use a key stored in Azure Key Vault. The way they use it is that they implemented a minimal PKCS#11 module. That was my primary inspiration, but I also recently spent some time building a drop-in replacement for pesign-daemon and the idea of exposing the signing interface over a Unix socket that can be mounted into isolated build environments was another concept I borrowed.

For those who aren’t familiar, PKCS#11 is a standard that is made up of a C interface. To implement it, you create a shared object file with a few well-known C functions used to discover your implementations of the interface. Users load your shared library, call a well-known function to get a table of functions where you provide your implementations for the various features. What’s nice about this is that common libraries like OpenSSL can speak to PKCS#11 modules, so you can make any tool that uses those common cryptography libraries use your implementations.

PKCS#11 is a fairly broad specification that covers not just signing, but also encryption/decryption, digest calculation, random number generation, and key management (creating, modifying, deleting, etc). What I realized while poking around Flatcar’s implementation is that you actually don’t need to implement everything. You can, instead, stub out all the functions you don’t want to implement and have them return the handy “function not supported” error code. The list of functions I opted to not support is extensive.

In fact, all I implemented was the functions to authenticate, list keys, and sign. However, that’s enough to make it possible to use the Siguldry server with tools including (but definitely not limited to): rpmsign, ostree gpg-sign, cosign (if built with PKCS#11 support), gpg2 (with the gnupg-pkcs11-scd service), sq (once it merges its PKCS#11 support), systemd-measure, systemd-sbsign, pesign, and perhaps most humorously, ssh. All these tools know how to speak to PKCS#11 modules so they “just work” with Siguldry. Any new tool people make will also likely just work.

One thing that’s neat about this approach is that we sit between the tool requesting the signature and the signing server. This means we can hash the content client-side and requests to the server are very small: just a hash, the algorithm used for the hash, and the key name. No more sending a 4GiB ISO over the network only to have the server hash it, that all is done before the request starts.

The last thing that’s really cool about this approach is that if Fedora decides it needs a new signing server and spends some big bucks on a fancy hardware security module, it will definitely ship with a PKCS#11 module of its own so we can start using it with our existing tooling, no development required. If Siguldry stops being the right choice for Fedora, it’s really easy to swap it out.

Unix Socket API

The other thing I did was to implement a Siguldry client that binds a Unix socket and proxies requests from the local system to the remote Siguldry server. I did this because you can expose the socket into locked down environments, such as RPM build environments, install the PKCS#11 module into that environment, and sign things. Right now Fedora uses this general approach for SecureBoot, except it’s specific to the pesign utility. While I’d prefer to disconnect the signing event from the building of a package, it’s important to support the current workflows and this was a neat way to do that which is generic across signing tools.

It has another advantage. Since the Unix socket is set up using a systemd socket unit, we can sandbox the client proxy, and also configure it with the necessary credentials to connect to the Siguldry server and unlock signing keys. This lets you, if necessary, allow anyone to sign content without needing to manage secrets themselves. The secrets can be encrypted with systemd-creds and bound to hardware. All we have to do is ensure the keys that are configured that way get a special flag when queried via PKCS#11: the “protected authentication path” flag.

siguldry-fedora-autopen

The primary way Fedora uses its signing server is via an AMQP consumer that automatically signs things. Robosignatory is the current tool used, but it needs some significant changes to work with this new workflow. In particular, we don’t want to sign content serially, but the fedora-messaging Python library commonly used to interact with the AMQP broker doesn’t handle concurrency well. And, since we now call out to the particular tool used to sign content rather than sending it to the server, the consumer is primarily a mapping between message topics and those various tools.

All those reasons led me to re-implement it, and since I’ve got no imagination with names, it’s called siguldry-fedora-autopen. It uses the lapin AMQP client with Tokio to manage hundreds or thousands of RPMs/ISOs/containers/etc signing operations at the same time. This means builds should no longer languish in the signing queue for many hours.

IMA

All this probably (hopefully) sounds great, but as with all things there are a few problems. Well, really just one big problem. Back in the Fedora 37 days, we enabled IMA signing for RPMs. The short version of how that works is that each file inside an RPM gets signed. And, unfortunately, the PKCS#11 interface is synchronous. This means that rpmsign requests a signature when it creates an OpenPGP signature on the RPM header. It then requests a signature for each file, waiting for each signature request to finish before starting the next one. Now, when the RPM only has dozens or a couple hundred files that happens fairly quickly. A signature might only take a couple milliseconds and the request/response latency might be a couple dozen more milliseconds (or a lot less depending on how close the server is from the client).

Of course, not all RPMs only have a few hundred files. Some have many, many thousand. For these, a simple OpenGPG signature takes less than a second (even if the RPM is, say, 10GiB). The IMA signing can take many minutes, depending on how many files it has. I’ve been running a client locally, with a server in a nearby datacenter, and the largest time I saw for a signing operation was around 40 minutes. This is really unfortunate, but it is important to remember that we can now do thousands of signing operations at the same time, so even when these slow builds are signed, other builds can be serviced in a timely manner.

There’s a couple ways we can make this better, but the “easiest” involve a custom signing implementation that avoids using the PKCS#11 path.

What’s Next?

In the next few weeks I’m hopeful we’ll have enough time to sit down and deploy Siguldry to Fedora’s staging environment. I want to really kick the tires and ensure it’s rock solid before we move it to production, but I think it’s reasonable to hope for that in the next couple of months.

While that’s happening, I’m going to add support for ML-DSA signing. That will allow us to produce signatures that are safe in a post-quantum cryptography world. After that, I’ll investigate OpenPGP’s hybrid signature schemes, although given recent developments I wonder if we want to bother with those. I’ll defer to what the professional cryptographers have to say about that.

Sometime in the next few Fedora releases, though, you should expect to see new signature algorithms in use (pending FESCo approval etc etc).

Comments and Feedback

Thoughts, comments, or feedback greatly welcomed on Mastodon

Fedora's aarch64 images support Secure Boot

Wed, 01 Apr 2026 08:51:00 +0000

About seven years ago, a ticket was filed noting aarch64 systems were shipping with Secure Boot enabled, and that Fedora should start signing its boot path to support these devices out of the box.

I’m pleased to say that today’s Fedora Rawhide images - what will be Fedora 45 - finally does this thanks to the work of a whole bunch of people.

This means you can grab the latest Rawhide images and boot them on your favorite aarch64 laptop without turning off Secure Boot, or launch VMs in any of the major clouds with Secure Boot on. For example, I’m able to start a VM in Azure with the TrustedLaunch security type:

❯ az group create --name "jcline-aarch64-secureboot" --location "eastus2"
❯ az vm create --location eastus2 --name fedora \
  --resource-group jcline-aarch64-secureboot \
  --image /CommunityGalleries/Fedora-5e266ba4-2250-406d-adad-5d73860d958f/Images/Fedora-Cloud-Rawhide-Arm64/Versions/latest \
  --security-type TrustedLaunch \
  --size Standard_D2plds_v6 \
  --accept-term \
  --ssh-key-values @/home/jcline/.ssh/id_ed25519.pub
❯ ssh jcline@20.12.69.183
[jcline@fedora ~]$ mokutil --sb-state
SecureBoot enabled

Why Now?

The way Fedora used to sign UEFI applications for Secure Boot was delightfully simple (for some value of simple). The keys were in a smart card, plugged into a special build host, and anything that needed a signature was routed to be built on that host. pesign, one of the common utilities to sign PE applications, has a mode where it can run as a daemon and sign anything provided to it over a Unix socket. That Unix socket is threaded into the build environment, where builds can access it to sign PE applications with pesign-client.

Unfortunately, that host was x86_64 so when aarch64 started shipping with Secure Boot enabled, an alternative approach was needed.

Ultimately we moved the smart card to the signing server we use for RPMs and other things. The tricky bit about the whole process is that Fedora signs each bit of the boot chain during the build. Each time any of the UEFI applications in the boot chain is built it needs to be signed. One way to do this is to build the application in Fedora’s infrastructure, and then have a second build which uses the output of the first build along with a signature as input to construct a signed final version. However, this means you’ve got two specfiles which you have to keep in sync, and there’s probably other painful aspects I’ve not considered. In any case, that’s not what Fedora does.

Instead, Fedora signs the UEFI applications during the build. Since we want the signing key to be stored in a remote server, this implies some sort of networking, but builds aren’t permitted network access. Nor can the build environment provide the necessary secrets to authenticate with the signing service. In order to handle this, I wrote a small service that pretends to be the pesign Unix socket, and that can be exposed to the build environment in the same way. However, it just shovels anything it gets to the signing server and returns whatever the signing server does.

That service got deployed last week, and after a little bit of debugging it even worked. In fact, everything was signed for aarch64 last week, except for the fallback UEFI application that adds a boot entry for Fedora if it’s not there, which happens on first boot. Without that, booting new images would fail unless you explicitly added the correct Fedora boot entry manually. Yesterday, shim got rebuilt and everything works.

Stable When?

It’s possible this will eventually work in Fedora 44 Cloud images. Shim in Fedora 44 hasn’t (yet) been rebuilt and we’re in the final freeze for Fedora 44, so unfortunately we just missed it, but if it does get rebuilt later, Cloud images will be updated and will start working.

For Fedora 43 and older, the version of shim shipped doesn’t include the version signed for aarch64. I’m not sure it’s worth the risk to update it, as much as I’d like it to work there, as well.

Anyway, Fedora 45 will be upon us before you know it, and after seven years, six more months isn’t so bad, right?

Comments and Feedback

Thoughts, comments, or feedback greatly welcomed on Mastodon

Fedora Signing Update, September edition

Thu, 18 Sep 2025 15:34:00 +0000

It’s been a while since my last update about content signing in Fedora, so I figured I should write one before the end of this month. While I’ve not been able to devote all my time to working on Siguldry, I have made enough progress I think it’s worth covering, and I also had the opportunity to chat with Miloslav Trmač, the original author of Sigul, about its design.

History Lesson for Sigul

In my post about various protocol tweaks I covered most of the changes I wanted to make to Sigul’s protocol and my reasons for doing so. A major theme of the changes was pushing more of the content-related “smarts” to the client. For example, in sigul, the server receives a container file and prepares a JSON document based off it to sign with GPG. In my proposed approach, the client should handle that and simply asks the server to sign the given document.

Miloslav was able to provide me with some history on Sigul and why it pushes the responsibility of dealing with the content to the bridge and server, and I think it’s worth recording here. In the original design, signing didn’t happen automatically: a Fedora contributor would need to request a signature using the Sigul client. In this scenario, there are a lot more Sigul clients and they’re running on machines that Fedora infrastructure doesn’t manage. Thus, the bridge and server needed to make every effort to ensure clients didn’t request signatures for things they shouldn’t - this is why the bridge is able to integrate with the Fedora Account System, and why it also contacts Koji when RPM signatures are requested: it has every reason to expect the client to be malicious.

These days, there’s only a few Sigul clients. Fedora’s infrastructure admins have management accounts, and the robosignatory service, which runs in Fedora’s infrastructure, also acts as a client. While we should be cautious in our design, the deployment is significantly different from Sigul’s original expectations so we can make different trade-offs.

Siguldry Progress

Back in July, I completed the protocol implementation. Since then, I’ve worked on actually doing things on top of it.

Binding Keys to Hardware

One feature I largely ignored in Sigul because it seemed complicated was its ability to encrypt signing keys using a combination of user-provided passwords and hardware (a Yubikey or TPM, for example). This means that if you were to obtain a copy of the Sigul database and you also managed to get the user’s password to access a signing key, you still wouldn’t be able to access the private keys without getting access to the signing machine’s TPM or a Yubikey that Fedora’s infrastructure team has access to.

I have partially re-implemented this particular feature in Siguldry. I say partially because Sigul had a number of different ways to bind keys to hardware on both the server and client side, and I have opted to implement only what Fedora infrastructure currently uses. You can use any key accessible via PKCS #11 to encrypt signing keys in addition to user passwords. You can even use multiple keys, and any one key is enough to unlock things (so you can have a backup Yubikey or three).

On the client side, the expectation is that user passwords and certificates will be bound to the TPM via systemd-credentials.

The Server Supports PGP Signing

I’ve implemented creating PGP keys as well as signing content with them. The interface supports detached signatures, the cleartext format you see, for example, with CHECKSUM files, and inline signatures.

This covers all current PGP signature types used by Fedora. However, more work is needed on the client side to use this interface to sign RPMs. The client needs to extract the header from the RPM, have the server produce a detached signature, and then send that signature to Koji.

The Server Supports RSASSA-PKCS1-v1_5 and ECDSA Signatures

These are the signatures you get with openssl-dgst and related commands, and they’re used for Secure Boot signatures, container signatures with cosign, and IMA.

What’s Next

Siguldry now has all the primitive interfaces to implement signing in the client for various types of content (RPM, containers, git tags, etc). It’s now time to work on implementing those, figure out what server interfaces need changing, and then add a whole lot of polish: a comprehensive end-to-end test suite for all content types, API documentation, an admin guide, and a set of tools to migrate the Sigul database to Siguldry.

Comments and Feedback

Thoughts, comments, or feedback greatly welcomed on Mastodon

Fedora signing protocol tweaks

Tue, 29 Jul 2025 14:01:00 +0000

In my last post on Fedora’s signing infrastructure, I ended with some protocol changes I would be interested in making. Over the last few weeks, I’ve tried them all in a proof-of-concept project and I’m fairly satisfied with most of them. In this post I’ll cover the details of the new protocol, as well as what’s next.

Sigul Protocol 2.0

The major change to the protocol, as I mentioned in the last post, is that all communication between the client and the server happens over the nested TLS session. Since the bridge cannot see any of the traffic, its role is reduced significantly and it is now a proxy server that requires client authentication.

I considered an approach where there was no communication from the client or server to the bridge directly (beyond the TLS handshake). However, adding a handshake between the bridge and the server/client makes it possible for all three to share a few pieces of data useful for debugging and error cases. While it feels a little silly to add this complexity primarily for debugging purposes, the services are designed to be isolated and there’s no opportunity for live debugging.

The handshake

After a server or client connects to the bridge, it sends a message in the outer TLS session to the bridge. The message contains the protocol version the server/client will use on the connection, as well as its role (client or server). The bridge listens on two different ports for client and server connections, so the role is only included to catch mis-configurations where the server connects to the client port or vice versa.

The bridge responds with a message that includes a status code to indicate it accepts the connection (or not), and a UUID to identify the connection.

The bridge sends the same UUID to both the client and server so it can be used to identify the inner TLS session on the client, bridge, and server. This makes it easy to, for example, collect logs from all three services for a connection. It also can be used during development with OpenTelemetry to trace a request across all three services.

Reasons the bridge might reject the connection include (but is not limited to): the protocol version is not supported, the role the connection announced is not the correct role for the port it connected to, or the client certificate does not include a valid Common Name field (which is used for the username).

After the bridge responds with an “OK” status, servers accept incoming inner TLS session on the socket, and clients connect. All further communication is opaque to the bridge.

Client/Server

In the last blog post, I discussed requests and responds being JSON dictionaries, and content that needed to be signed could be base64 encoded, but I didn’t go into the details of RPM header signatures. While I’d still love to have everything be JSON, after examining the size of RPM headers, I opted to use a request/response format closer to what Sigul 1.2 currently uses. The reason is that base64-encoding data increases its size by around 33%. After poking at a few RPMs with the rpm-head-signing tool, I concluded that headers were still too large (cloud-init’s headers were hundreds of kilobytes) to pay a 33% tax for a slightly simpler protocol.

So, the way a request or response works is like this: first, a frame is sent. This is two unsigned 64-bit integers. The first u64 is the size of the JSON, and the second is the size of the arbitrary binary payload that follows it. What that binary is depends on the command specified in the JSON. The alternative was to send a single u64 describing the JSON size alone, so the added complexity here is minimal. The binary size can also be 0 for commands that don’t use it, just like the Sigul 1.2 protocol.

Unlike Sigul 1.2, none of the messages need HMAC signatures since they all happen inside the inner TLS session. Additionally, the protocol does not allow any further communication on the outer TLS session, so the implementation to parse the incoming requests is delightfully straightforward.

Authentication

Both the client and server authenticate to the bridge using TLS certificates. Usernames are provided in the Common Name field of the client certificate. For servers, this is not terribly useful, although the bridge could have an allowlist of names that can connect to the server socket.

Clients and servers also mutually authenticate via TLS certificates on the inner TLS session. Technically, the server and client could use a separate set of TLS certificates for authentication, but in the current proof-of-concept it uses the same certificates for both authenticating with the bridge and with the server/client. I’m not sure there’s any benefit to introducing additional sets of certificates, either.

For clients, commands no longer need to include the “user”: it’s pulled from the certificate by the server. Additionally, there’s no user passwords. Users that want a password in addition to their client certificate can encrypt their client key with a password. If this doesn’t sit well with users (mostly Fedora Infrastructure) we can add passwords back, of course.

Users do still need to exist in the database or their requests will be rejected, so administrators will need to create the users via the command-line interface on the server or via a client authenticated as an admin user.

Proof of concept

Everything I’ve described has been implemented in my proof-of-concept branch. I’m in the process of cleaning it up to be merge-able, and command-line interfaces still need to be added for the bridge and client. There’s plenty of TODOs still sprinkled around, and I expect I’ll reorganize the APIs a bit. I’ve also made no effort to make things fast. Still, it will happily process hundreds or thousands of connections concurrently. I plan to add benchmarks to the test suite prior to merging this so I can be a little less handwave-y about how capable the service is.

What’s next?

Now that the protocol changes are done (although there’s still time to tweak things), it’s time to turn our attention to what we actually want to do: managing keys and signing things.

Management

In the current Sigul implementation, some management is possible remotely using commands, while some are only possible from on the server. One thing I would like to consider is moving much of the uncommon or destructive management tasks to be done locally on the server rather than exposing them as remote commands. These tasks could include:

All user management (adding, removing, altering)
Removing signing keys

Users could still do any non-destructive actions remotely. This includes creating signing keys, read operations on users and keys, granting and revoking access to keys for other users, and of course signing content.

Moving this to be local to the server makes the permission model for requests simpler. I would love feedback on whether this would be inconvenient from anyone in the Fedora Release Engineering or Infrastructure teams.

Signing

When Sigul was first written, Fedora was not producing so many different kinds of things that needed signatures. Mostly, it was RPMs. Since then, many other things have needed signatures. For some types of content (containers, for example), we still don’t sign them. For other types, they aren’t signed in an automated fashion.

Even the RPM case needs to be reevaluated. RPM 6.0 is on the horizon with a new format that supports multiple signatures - something we will want to support post-quantum algorithms. However, we still need to sign the old v4 format for EPEL 8.0. We probably don’t want to continue to shell out to rpmsign on the server.

Ideally, the server should not need to be aware of all these details. Instead, we can push that complexity to the client and offer a few signature types. We can, for example, produce SecureBoot signatures and container signatures the same way. I need to understand the various signature formats and specifications, as well as the content we sign. The goal is to offer broad support without requiring constantly teaching the server about new types of content.

Comments and Feedback

Thoughts, comments, or feedback greatly welcomed on Mastodon

Re-designing signing in Fedora

Wed, 09 Jul 2025 16:17:00 +0000

Over the past few months I’ve spent some time on-and-off working on Sigul and some related tools. In particular, I implemented most of a new Sigul client, primarily to enable the sigul-pesign-bridge to run on recent Fedora releases (since the sigul client relies on python-nss, which is not in Fedora anymore).

At this point, I have a reasonably good understanding of how Sigul works. Originally, my plan was to completely re-implement the client, then the bridge, and finally the server using the existing Sigul protocol, version 1.2, as defined by the Python implementation. However, as I got more familiar with the implementation, I felt that it would be better to use this opportunity to also change the protocol. In this post I’m going to cover the issues I have with the current protocol and how I’d like to address them.

Note: I assume you’ve read how artifacts are signed in Fedora in the post.

Mixing TLS sessions

In protocol version 1.2, the client and server start “outer” TLS sessions with the bridge, and then the client starts a nested “inner” TLS session with the server. Data is sent in chunks which indicate how big the chunk is and whether it’s part of the “outer” session (and destined for the bridge) or the “inner” session. While it’s perfectly doable to parse the two streams out, it’s a complication. Maybe we can introduce some rules to make it easier?

After looking at the implementation, every command follows the same pattern. The client would:

Open a connection to the bridge and send the bridge the command to pass on to the server.
Open the inner TLS session and send some secrets to the server (a key to use for HMAC and a key passphrase to unlock a signing key, typically).
Close the inner TLS session.
Send HMAC-signed messages to the bridge, which it relays to the server.
Receive HMAC-signed messages from the server, via the bridge.

Critically, the inner TLS session was only used to exchange secrets so the bridge couldn’t see them, and was never used again. One option would be to only allow the inner TLS session once, right at the beginning of the connection.

However, the whole point of the HMAC-signed messages is that the client and server don’t seem to really “trust” the bridge won’t have tampered with the messages. Why not just use the inner TLS session exclusively so that we get confidentiality in addition to integrity?

Homegrown serialization

When Sigul was originally written things like JSON weren’t part of the Python standard library. It implemented its own, limited format. Now, however, there are a number of widely supported serialization options. JSON is the obvious choice as it is fairly human-readable, ubiquitous, and fairly simple. The downside is that for signing requests, the client needs to send some binary data. However, we explicitly do not want to be sending huge files to be signed, so base64-encoding small pieces of binary data should be acceptable.

The bridge is too smart

The bridge includes features that are not used, and that complicate the implementation.

Fedora Account System integration

The bridge supports configuring a set of required Fedora Account System groups the user needs to be in to make requests. It checks the user against the account system when it connects by using the Common Name in the client certificate as the username.

However, this feature is not used by Fedora, and since Fedora is probably the only deployment of this service, we probably don’t need this feature.

The bridge alters commands

For most commands, the bridge shovels bits between the client connection and the server connection. However, before it shovels, it parses out the requests and responses before forwarding them. There’s really only one good reason for this. Two particular commands may alter the client request before sending it to the server. Those two commands are “sign-rpm” and “sign-rpms”.

In the event that the client requests a signature for an RPM or multiple RPMs and the request doesn’t include a payload, the bridge will download the RPM from Koji. Now, the bridge doesn’t have the HMAC keys used in other commands to sign the request, so the client also contacts Koji and includes the RPM’s checksum in the request headers.

This particular design choice might have been done to save a hop when transferring large RPMs, but these days you don’t need to send the whole RPM, just the header to be signed.

It’s confusing to have the bridge take an active role in client requests. What’s more, if we push this responsibility to the client, there’s no reason the bridge needs to see the requests and responses at all.

Usernames

As noted in the Fedora Account System integration section, the bridge uses the client certificate’s Common Name to determine the username. However, all requests also include a “user” field.

The server checks to ensure either the request username matches the client certificate’s Common Name, or if the lenient_username_check configuration option is set which disables that check, or if the Common Name is in a configuration option, proxy_usernames, listing that that Common Name can use whatever username it wants.

The Fedora configuration doesn’t define either of those configuration options, and it’s confusing to have two places to define the username.

Passwords

Users have several types of passwords.

Users are given access to signing keys by setting a user-specific passphrase for each key. This passphrase is used to encrypt a copy of the “real” key password, so each user can access the key without ever knowing what the key password is.

However, each user also has an “account password” which is only needed for admin commands, and only works if the account is flagged as an admin. Given that the client certificate can be password-protected it’s not clear to me that this adds any value, but it is confusing.

Summary

To summarize, the major changes I’m considering in a new version of the Sigul protocol are:

All client-server communication happens over the nested TLS session. The client and server no longer need to manually HMAC-sign requests and responses as a result.
The bridge is a simple proxy that authenticates the client and server connections via mutual TLS and then shovels bits between the two connections without any knowledge of the content. Drop the Fedora Account System integration, and push the responsibility of communicating with Koji to the client.
Switch from the homegrown serialization format to JSON for requests and responses.
Rely exclusively on the client certificate’s Common Name for the username.
Remove the admin password from user accounts as they can password-protect their client key and also use features like systemd-creds to encrypt the key with the host’s TPM.

How artifacts are signed in Fedora

Fri, 28 Mar 2025 19:39:00 +0000

For the last few months, one of the things I’ve been working on in Fedora is adding support for SecureBoot on Arm64. The details of that work will be the subject of a later post, but as part of this work I’ve become somewhat familiar with the signing infrastructure in Fedora and how it works. This post introduces the various pieces of the current infrastructure, and how they fit together.

Signed?

Pretty much anything Fedora produces and distributes is digitally signed so users can verify it did, in fact, come from the Fedora project. Perhaps the most obvious example of this is the RPM packages Fedora produces. However, plenty of other artifacts are also signed, like OSTree commits.

Signing works using public-key cryptography. We have the private key that we need to keep secret, and we distribute the public keys to users so they can verify the artifact.

Robosignatory

Signing is (mostly) an automated process. A service, called robosignatory, connects to an AMQP message broker and subscribes to several message topics. When an artifact is created that needs signing, the creator of the artifact sends a message to the AMQP broker using one of these topics.

Robosignatory does not sign artifacts itself. It collects requests from various other services and submits them to the signing server on behalf of those systems. The signing server is the system that contains and protects the private keys.

Sigul

Sigul is the signing server that holds the private keys and performs signing operations. It is composed of three parts: the client, the bridge, and the server.

The Server

The Sigul server is designed to control access to signing keys and to provide a minimal attack surface. It does not behave like a traditional server in that it does not accept incoming network connections. Instead, it connects to the Sigul bridge, and the bridge forwards requests to it via that connection.

This allows the firewall to be configured to allow outgoing traffic to a single host, and to block all incoming connections. This does mean that the host cannot be managed via normal SSH operations. Instead this is done in Fedora via the host’s out-of-band management console.

Private keys are encrypted on disk. When users are granted access to keys, the key is encrypted for that particular user.

The Bridge

The Sigul bridge acts as a specialized proxy. Both the client and server connect to the bridge, and it forwards requests and responses between them.

Unlike the typical proxy, the Sigul bridge does a few interesting things. Firstly, it requires the client to authenticate via TLS certificates. The client then sends the bridge the request it wishes to make. The bridge forwards that request to the server. The bridge then expects the client and server to send an arbitrary amount of data to each other. This arbitrary data is framed as chunks and it forwards that data back and forth until an end of stream signal is sent by the server and client.

Finally, it forwards the server’s response to the client.

The Client

The client is a command-line interface that sends requests to the server via the bridge. This client can be used to create users and keys, grant and revoke access to keys, and request signatures in various formats. Robosignatory invokes the CLI to sign artifacts.

The client connects to the bridge as described above. After authenticating with the bridge and sending the request, it begins a second TLS connection configured with the Sigul server’s hostname, and sends and receives data on this connection over the TLS connection it has with the Sigul bridge. It relies on this inner TLS connection to send a set of session keys to the server without the bridge being able to intercept them. These session keys are used to sign the Sigul server’s responses so the client can be sure the bridge did not tamper with them.

After it has established a set of secrets with the server, the remainder of the interaction occurs over the TLS connection with the bridge, but since the responses are signed both the client and server can be confident the bridge has not tampered with the conversation.

Conclusion

It’s an interesting setup, and has served Fedora for many years. There is plenty of code in Sigul which dates back to 2009, not long after Python 2.6 was released with exciting new features like the standard library json module. It was likely developed for RHEL 5 which means Python 2.4.

Unfortunately, at least one of the libraries (python-nss) it depends on to work are not maintained and not in Fedora 42, so something will have to be done in the near future. Still, it’s not ready to sign off just yet.

Publishing messages in Fedora's infrastructure

Thu, 05 Sep 2024 18:16:00 +0000

While at Flock to Fedora 2024, someone noted, to paraphrase a conversation from a month ago, that working with Fedora’s messaging infrastructure seemed more intimidating now that it’s transitioned from ZeroMQ to AMQP. I promised to write up a blog post walking through the process of adding support to an application, so here it is (better late than never?).

There are two ways to interact with Fedora’s messaging system as a developer. Your application can send (publish) messages, receive (consume) them, or both.

This blog post will focus on adding support to your application for publishing in Fedora’s infrastructure, and how to effectively unit test them. For anything not covered in this post, I recommend the official documentation.

Getting Started

This post assumes you’ll be using Python and the fedora-messaging library which uses Pika to communicate with the message broker. If you’re working in a different language this post isn’t for you, but you can use any AMQP-0.9 client to send messages in the documented message format.

Use your Python package management tool of choice to install fedora-messaging.

Publishing

A minimal implementation looks like this:

from fedora_messaging import api

msg = api.Message(topic="my.topic", body={"some_key": ["some", "value"]})
api.publish(msg)

This example, while pleasingly simple, glosses over some important details which you will need to address.

Errors

The first adjustment to make is to add error handling. See the API documentation for details on each exception, but here are the ones you generally want to handle:

from fedora_messaging import api, exceptions

msg = api.Message(topic="my.topic", body={"some_key": ["some", "value"]})
try:
    api.publish(msg)
except exceptions.ConnectionException as err:
    print(f"Connection to the message broker was lost: {err}")
except (exceptions.PublishTimeout, exceptions.PublishReturned) as err:
    print(f"Failed to publish the message: {err}")

What you do when an error occurs is up to you. You could retry in a loop until it succeeds, or to skip publishing a message if it’s not critical that a message be sent for each event. Finally, you could do a mixture of the two by adding it to a queue of messages to publish next time.

Schema

The next adjustment is to define a schema for your message. While this isn’t a strict requirement, it ensures you don’t accidentally change the format of the message and break users, and it is required if you want notifications generated from your messages via the Fedora Notification service to look good. There’s a tutorial on providing schema which I won’t reproduce here, but this is what a message with a schema looks like:

from fedora_messaging import api

class MyMessage(api.Message):
    """A schema and common properties to enable Fedora Notification support"""
    topic = "my.topic"
    body_schema = {
        "id": "https://fedoraproject.org/message-schema/v1/my-app.message",
        "$schema": "https://json-schema.org/draft/2019-09/schema",
        "description": (
            "This message is sent by my-app when something interesting happens"
        ),
        "type": "object",
        "properties": {
            "some_key": {
                "type": "array",
                "description": "This is some key, isn't it?",
                "items": {"type": "string"},
            },
        },
        "required": [
            "some_key",
        ],
    }

I highly recommend following the tutorial and using CookieCutter to produce your Python package containing your message schema.

After you create your Python package containing your message schema, add it as a dependency to your application and publish it on PyPi for consumers to use.

Testing

Now that we know how to publish messages in our applications, we probably want to add tests to ensure messages are published (or not) as we expect.

Here’s what a test looks like:

from fedora_messaging import api, exceptions, testing

# This is our publish code from above inside a function
def function_that_publishes():
    msg = api.Message(topic="my.topic", body={"some_key": ["some", "value"]})
    try:
        api.publish(msg)
    except exceptions.ConnectionException as err:
        print(f"Connection to the message broker was lost: {err}")
    except (exceptions.PublishTimeout, exceptions.PublishReturned) as err:
        print(f"Failed to publish the message: {err}")


# This will assert a single message is published of type "message.Message"
with testing.mock_sends(api.Message):
    function_that_publishes()

# This will assert a single message is published of type "message.Message"
# and that the message topic and bodies match
with testing.mock_sends(api.Message(topic="my.topic", body={"some_key": ["some", "value"]})):
    function_that_publishes()

And that’s it!

Configuration

One last thing will need to be done when deploying your application: configuring it to connect to Fedora’s broker.

The API relies on configuration being provided by a TOML file located at, by default, /etc/fedora-messaging/config.toml, but can be changed by setting the FEDORA_MESSAGING_CONF environment variable to the alternate location.

The default configuration values for fedora-messaging work well for RabbitMQ running on the same host as your application, but we need to adjust some values to work with Fedora’s message broker:

# This defaults to "false", which means your application will attempt to create
# the AMQP objects which works well for development. In Fedora, your
# application will get a "Permission Denied" on startup since your account will
# only have permission to create a few specific resources.
passive_declares = true

# Both these values rely on variables set in Fedora's Ansible repository, but
# in this example I've expanded them to what they are in production.
#
# <your-app-user> will be provided by Fedora's infrastructure team.
amqp_url = "amqps://<your-app-user>:@rabbitmq.fedoraproject.org/%2Fpubsub"
topic_prefix = "org.fedoraproject.prod"

# The TLS settings include a CA certificate used to verify the identity of the
# Fedora broker, along with a client certificate and key to authenticate your
# application to the broker.
#
# You must file a ticket with Fedora's infrastructure team to have an account
# created for your application along with a client certificate for authentication.
# The key and certificate is kept in Ansible. 
[tls]
ca_cert = "/etc/pki/rabbitmq/ca/rabbitmq.ca"
keyfile = "/etc/pki/rabbitmq/key/user.key"
certfile = "/etc/pki/rabbitmq/cert/user.crt"

# This identifies your application to RabbitMQ to aid in debugging.
[client_properties]
app = "My Application"

These are the only settings you should need to adjust for publishing messages.

HDR in Linux: Part 2

Mon, 28 Jun 2021 18:36:30 +0000

In the previous post, we learned what HDR is: a larger luminance range that requires more bits per component, new transfer functions to encode that luminance, and potentially some metadata. We can examine the work required to use it in a “standard” Linux desktop. By “standard”, I of course meant my desktop environment, which is GNOME on Fedora.

To do this, we’ll consider a single use-case and examine each portion of the stack, starting at the top and working our way down. The use-case is watching an HDR movie in GNOME’s video application, Totem. In this scenario, the application isn’t likely to tone-map its content as it has been created with HDR metadata the display itself can use to tone-map when necessary, but I will note where this could happen.

Let’s review the high-level requirements:

The Wayland display server (compositor) must determine the display capabilities (color primaries and white point, luminance, transfer function, bits per component, etc) in order to know what outputs are possible and/or optimal.
Client applications may want to know what content encoding the Wayland display server supports and maybe even what the optimal content encoding is for the display it is on (the native primaries, transfer function, bit depth, luminance capabilities, etc)
Client applications need to express the transfer function, bit depth, color space, and any HDR metadata for its content (the “content encoding”) to the compositor. The compositor, in turn, needs to ensure the content from all the client applications are blended properly and output in a format the display can handle.

You might be familiar with text encoding. Adding HDR support is very similar to adding support for various text encoding formats when previously everything assumed content was encoded with ASCII. Instead of ASCII, the graphics stack largely assumes everything is sRGB.

Totem

At the top of the stack we have Totem, which provides the user interface and playback management features. It relies on GStreamer to deal with the video itself. It appears to use the playbin plugin, which GStreamer describes as an “everything-in-one” abstraction for a audio and video player.

Since Totem is all about wiring up a user interface to the GStreamer playbin plugin, GStreamer needs to be HDR-aware. Totem uses GTK and Clutter-GTK to create the user interface, so those libraries are also something we should examine.

GStreamer

Based on the GStreamer documentation and this talk by Edward Hervey all the necessary features appear to already be implemented for HDR video playback.

The GStreamer video library contains an HDR section. The APIs, available since 1.20, include interfaces for working with SMPTE ST 2086 (static metadata) and SMPTE 2094-40 (dynamic metadata). This metadata can be attached to a GstBuffer.

The GstVideoInfo structure includes colorimetry including the color primaries and transfer function. GstVideoInfo also includes the GstVideoFormatInfo structure which describes the pixel format, including bit depth. These contain all the information required to decode and convert the content. The GstVideoFrame ties the GstBuffer and GstVideoInfo together.

So GStreamer has all the content metadata we need. Next, we need to make sure it gets from the application, Totem, to the Wayland display server. The client-to-server interaction occurs in the EGL or Vulkan Window System Integration (WSI) APIs. These APIs are where OpenGL and Vulkan APIs are integrated into the particular operating system. This is where buffers are allocated for use within the graphics libraries and they include ways to provide the content encoding. As long as the metadata GStreamer extracts is passed to the relevant EGL or Vulkan API, we are all set. Mesa includes implementations of EGL and Vulkan WSI, so we will examine those after GTK.

Clutter-GTK

In the section on Totem, we noted that it uses Clutter-GTK. The documentation is sparse and appears to assume the reader has an understanding of Clutter and why one would want to use it. Clutter appears to be a library for arranging a collection of 2-dimensional surfaces (“actors” in Clutter’s terms) in 3-dimensional space.

If we look at Clutter’s repository the README notes it is in “deep maintenance mode” and is intended to be replaced with GTK 4. Based on the commit log this is true, so we will look no further and focus on GTK 4.0.

GTK

GTK is a “widget toolkit”. It provides, either through a dependency or on its own, an application abstraction, event loop, rendering interfaces, methods to declare various user interface elements (buttons, text fields, etc), and other useful utilities for building a graphical user interface.

As of GTK 4, rendering is usually performed by OpenGL. As we’ll examine later, OpenGL APIs offer the necessary interfaces to handle HDR content, but GTK’s rendering model introduces a few issues. To understand the difficulties we need to understand how GTK builds the application window.

A Window into GTK

GTK documents its drawing model in some detail and it would best to read through it before proceeding.

One of the most relevant parts of the documentation is that

Each GTK toplevel window or dialog is associated with a windowing system surface. Child widgets such as buttons or entries don’t have their own surface; they use the surface of their toplevel”

GTK has each widget render itself, and then combines them all into a single surface which it shares with the Wayland display server to be combined with all the other client surfaces. We’ll look at the Wayland protocol in detail later, but what’s important to know right now is that the surface is where the content encoding should be stored.

This single surface that GTK produces, therefore, needs to be composed of widgets that all use the same content encoding. Unfortunately, GTK does not offer a way specify what content encoding to use for widgets it renders, nor does it provide a way for users to convey the content encoding they are using for their custom widgets. All content GTK produces on the application’s behalf, like buttons, menus, and the window decorations are rendered in an undefined color space with an undefined transfer function. In practice, this seems to be sRGB. All textures GTK creates are hard-coded to use 8 bits per component, including when it combines all the widgets into a single image.

This means, unfortunately, that at the moment there is no way to use GTK for an application interested in producing HDR content (video players, image viewers, content creation tools, etc).

Ways Forward

There are a few ways for GTK to work in a world where sRGB isn’t the only content. I can’t really say which approach is best, or even if this is an exhaustive list of options since my knowledge of GTK is only a few days old. Regardless, a good bit of work is necessary to make GTK HDR-ready.

Widgets with Content Encoding

One solution, if GTK wishes to remain in the business of blending images together, is to ensure each widget includes a way to express how the content it produces is encoded. GTK can then use this to ensure they are combined correctly. If, for example, sRGB content and PQ-encoded content were combined into a destination image without converting the sRGB content, the end result would be that the sRGB portions of the image would be extremely dark. This, of course, introduces a reasonable amount of complexity to GTK as it needs to convert between a potentially long list of formats in addition to introducing interfaces for specifying the content encoding everywhere it matters.

Sub-surfaces

Wayland offers a way to create surfaces with a parent-child relationship called subsurfaces. Since the surface is where the content encoding is stored, GTK could use sub-surfaces to defer the blending of different content encoding to the Wayland display server. This would allow GTK to remain in its undefined, sRGB-ish world while allowing applications to handle modern content with other content encoding. However, I am told these are problematic as they cannot be clipped or transformed by GTK.

Require a single HDR Format

Rather than handling converting all the widgets to the same content encoding, GTK could specify one HDR-capable format it supports and require all widgets to use that if HDR has been requested for the application. A format like scRGB with half-precision 16-bit floating point numbers for each color component. This, however, has the downside that all HDR content needs to be converted from its native format to scRGB, blended with the other GTK widgets, and then converted back to some on-the-wire HDR format in the Wayland display server.

Even with this approach, GTK cannot blend the SDR content into HDR content without considering what luminance range to map SDR content to, which needs to correspond to the display brightness level the user has set.

Use Sub-surfaces Outside GTK

Applications can work around GTK not dealing with content encoding by not using GTK for anything other than the menus, buttons, and so on. In fact, this is how Firefox handles things; GTK is used for the “chrome” and the content is rendered to a Wayland sub-surface which Firefox manages itself and overlays on the GTK surface. This sub-surface can be properly configured for HDR and the compositor can handle blending it with GTK’s surface.

This, of course, isn’t ideal as the applications have to work around the toolkit rather than having the toolkit help them. However, it is a path forward worth mentioning.

Mesa

Mesa provides the OpenGL implementation that clients like GTK use to render their content. To allocate storage for the output of that rendering, clients use the EGL or Vulkan WSI interfaces for which Mesa also provides implementations.

EGL

The full EGL standard is available online. Of particular interest to us are the interfaces for creating surfaces. eglCreatePlatformWindowSurface includes a list of attributes, including the EGL_GL_COLORSPACE attribute. As of EGL 1.5, only EGL_GL_COLORSPACE_SRGB (which defines both color primaries and a transfer function) and EGL_GL_COLORSPACE_LINEAR (presumably the sRGB primaries, linear light) are defined. However there are a number of extensions to EGL for additional color space attributes including:

Additionally, there is an extension to allow the EGL_GL_COLORSPACE attribute to be applied to eglCreateImage

These extensions make it possible to convey the color primaries and transfer function to the Wayland display server.

However, this is not currently done because the Wayland protocol has not yet accepted a way to send the content encoding. The proposed protocol has been under discussion (200+ comments) for a year. The details will be covered in the Wayland section below, but for now what’s important to know is that the protocol needs to define a message for the color space and transfer function that is sent from the client via Mesa when a surface is created in EGL.

Finally, there is an EGL extension for specifying the HDR metadata for a surface. By calling eglSurfaceAttrib and providing the attributes defined in EXT_surface_SMPTE2086_metadata a client can set the HDR metadata for the content. This is actually already partially wired up in Mesa. All that remains is for the Wayland-specific EGL code to send the HDR metadata to the Wayland display server. Again, there is no protocol for this at the moment.

While it is a dizzying array of extensions, EGL already includes all the metadata and APIs to send the content encoding from the client (Totem, by way of GStreamer) to the Wayland compositor.

Vulkan WSI

The Vulkan Window System Interface requires much less bouncing between extensions to determine how it all works.

For WSI, the content encoding is set when creating a swapchain by providing a VkSwapchainCreateInfoKHR structure. This includes the pixel format as well as the color space. Like EGL, the color space enumeration defines both the color primaries and the transfer function together. There is support for sRGB, Display-P3, BT2020, and more, all with linear and one or more transfer function options.

The HDR metadata can be provided by the client using vkSetHdrMetadataEXT, which applies to the swapchain. With those two calls, a client can describe the content encoding.

Like EGL, the WSI implementation in Mesa does not forward this content encoding to the Wayland compositor because there’s no protocol.

Ville Syrjälä created a proof-of-concept branch of Mesa back in 2017 to have Mesa use the Wayland protocol proposal (of the time), and I rebased it recently to “work” with the latest Wayland protocol proposal. It doesn’t actually work yet, but it is a good approximation of the work required, which is implementing the standard’s functions to call libwayland functions with the content encoding details.

The Wayland Protocol

As we touched on in the Mesa section, the Wayland protocol is still being discussed. The discussion is extensive, and I cannot do it justice in this post, but in this section I’ll outline the basics of what the protocol must include and approximately what that would look like.

Wayland is a client-server architecture. The clients are applications like the video player Totem. A Wayland display server handles client requests and notifies them of relevant events.

You can, at this point, probably guess what information the protocol needs to include. The client needs to inform the server of:

The color primaries (red, green, blue, and white point coordinates)
The transfer function used to encode the luminance.
HDR metadata when it is available - in the case of Totem playing an HDR movie, HDR metadata will be available. This metadata may remain the same for many frames (perhaps the whole movie), or it may change frame-by-frame.

Without this information, the Wayland display server cannot properly tone-map or blend the content, nor could it offload those tasks to dedicated GPU hardware or the display since they both need the same information.

The protocol includes an object that represents a rectangular area that can be displayed, the wl_surface. The surface is backed by a wl_buffer. The wl_buffer holds the content and the wl_surface describes the role of the content and how the server should interpret the contents of the wl_buffer, such as how it should be scaled or rotated. Since the wl_surface includes metadata about the content, it is probably the right place to include other details about how the server should interpret the buffer, namely the color primaries, transfer function, and HDR metadata. While the color primaries and transfer function aren’t likely to change over the lifetime of a surface, the HDR metadata can change - HDR10+ and Dolby Vision allow for frame-by-frame HDR metadata - so the protocol should account for that.

Additionally, the Wayland display server should be able to inform the clients of:

The native color primaries of the display. This will not be a standard color space such as sRGB.
Standard color spaces the display will accept (and internally convert to its native color space).
Transfer functions the display can decode.
Minimum luminance, maximum sustainable luminance, and peak (burst) luminance.
HDR metadata formats the display will accept.

In the case of Totem playing an HDR movie, it would not need to concern itself with tone-mapping and would not need to know about what displays it might be targeting. However, something like a video that generates content would benefit from targeting the display’s capabilities.

Mutter

Mutter is, among other things, a Wayland display server. Once a protocol is in place, Mutter needs to implement it. To support HDR, Mutter needs to be able to:

Inform clients of display capabilities.
Act upon the content encoding provided by the clients to correctly convert all content to the same encoding and blend them all together into the desktop we know and love.

To inform clients of the display capabilities, Mutter can use the Extended Display Identification Data (EDID) from the kernel, which contains the display color primaries, supported HDR formats, and so on. It will be covered in detail in the kernel section below. The EDID is, in fact, already being used in Mutter so this will be a small addition to that. There is some trickiness around multi-monitor setups as clients don’t generally know which monitor they are being displayed on, but displays with differing pixel density have a similar challenge and Wayland handles that by providing events to the application when it enters a new display and the display content encoding could work the same way.

To act upon the client-provided content encoding, however, requires a bit more work.

Consolidating Existing Color Management

GNOME already has some color management capabilities. At the moment, they are split up in gnome-settings-daemon, colord, and Mutter.

colord is a system service that maintains a database of devices, color profiles, and the association between the two (including things like the default profile). At the moment, however, gnome-settings-daemon is responsible for querying colord and sets up the profile by calling Mutter D-Bus APIs.

Mutter needs to be aware of each display’s color profile in order transform the client content if necessary. The existing API for setting color profiles is also difficult to use because it acts on CRTC objects rather than displays. Some displays are composed of multiple panels tiled together and are therefore backed by several CRTCs. It makes more sense for Mutter to query colord itself rather than having gnome-settings-daemon do it. gnome-settings-daemon also handles the Night Light feature, which adjusts the color profile to shift the white point towards red to remove the blue light. Mutter needs to either handle the Night Light feature or, preferably, provide a “temperature” API and allow Mutter users to control when and how much to adjust the color temperature.

Support Converting Buffer Formats

There are a number of ways to store an image, but they generally fall into two categories. The first is to represent the red, green, and blue components, and that’s the way we’ve been discussing images in this blog post series.

Another way is to represent the luminance of a pixel, and then two color components for that same pixel. The third color component can be calculated using the lumanince and two known color components. This approach is commonly referred to as YUV although there are several flavors. This is a valuable approach because humans are more spatially sensitive to luminance than they are color which means it’s possible to produce a reasonably good-looking image even if you keep track of, say, every other pixel’s color. This is called chroma subsampling.

A lot of content, like films, are encoded using this approach, but at the moment Mutter does not support YUV formats. It would be convenient, since we’re adding a way for clients to communicate content encoding, to handle such formats. There has been some work in this area by Niels De Graef.

Compositing the Content

Once Mutter has knowledge of the content encoding of every surface, it can blend them together and convert them to the target encoding of the display. This involves:

Convert YUV to RGB if necessary.
Perform a color space transformation so that all surfaces are using the same color primaries and whitepoint.
Removing the transfer function on surfaces so that Mutter is dealing with the optical (linear) values rather than the non-linear on-the-wire encoding. Adding two non-linear values together without accounting for the non-linearity results in incorrect luminance levels. You can try this with the following Python snippet:
```
def gamma(u):
    """Apply the sRGB gamma definition to linear light levels"""
    return 1.055 * u**(1/24) - 0.055
```
Consider adding linear light level 50 to linear light level 50 and then applying the transfer function versus adding the non-linear encoding of 50 to itself:
```
 >>> gamma(100)
 1.2231616798531606
 >>> gamma(50) + gamma(50)
 2.3735497958717904
```
Blend the surfaces together. This becomes tricky with HDR since SDR does not have a clearly defined luminance range, it’s just how bright your display is set, but PQ-encoded content does have a well-defined luminance range. Mutter will need define the SDR luminance range and map that content into that well-defined range to ensure SDR content isn’t too dim or too bright.

Tone-mapping

With HDR, clients may provide surfaces that encode luminances beyond the capabilities of the target display. Displays account for this by tone-mapping out-of-range content into the display range. This is described in Section 5.4.1 of BT.2390. The important thing to know is that providing the display with out-of-range luminance levels will result in something that looks fine to most end users, but there may be a desire to carefully control what happens when this occurs.

Mutter would be responsible for implementing alternate tone-mapping if the display’s default behavior is unacceptable for the user. This would most likely be a concern for content creators.

The Kernel

We have arrived at the kernel, the final stop in this journey down the stack. The good news is that the basic requirements for HDR have already been met. We’ll touch on those and then look at additional features necessary for us to use certain hardware features.

The Bare Minimum

These are the bare minimum features for HDR and are already present in the kernel.

Display Capabilities

As we noted in the Mutter section, the EDID structure provided by a display is already available to userspace via the EDID connector property. Userspace can parse it for the chromaticity coordinates of the display primaries, the luminance, supported color spaces, and supported transfer functions. At the moment, many different projects implement parsing the EDID structure, but there have been some requests for a library to do so.

One thing worth noting is that EDID is quite old and has a replacement standard, DisplayID. Both standards are available from VESA for free, although you do need to provide your name and email address to access them. DisplayID has a Display Parameters block that includes luminance and chromaticity values, and the Display Interface Features block contains supported color space and transfer function combinations as well as bit depths. EDID included much of this information, but only as an extension documented in CTA-861 (currently at revision G). Recently CTA made this available for free, although you must provide an email and billing address to download it.

Displays can provide an EDID, DisplayID, or both. For the curious, the VESA E-DCC standard documents how these structures are accessed over I²C. How I²C is sent over, for example, DisplayPort is documented within the DisplayPort specification, which is only available to VESA members so the very curious will have to resort to reading the Linux kernel’s implementation if they aren’t members.

DisplayID can be embedded in the EDID structure as an extension so it may be available to userspace via the existing EDID connector property, but at this time there is not a dedicated kernel interface to retrieve a standalone DisplayID structure that I am aware of. The work to add one would be minimal, however, since it’s extremely similar to the EDID.

HDR Metadata

The kernel exposes a HDR_OUTPUT_METADATA connector property that userspace can use to send HDR metadata to the kernel and thus to the display.

At this time there is only one supported metadata type, which corresponds to the HDR metadata defined in CTA-861 in section 6.9 “Dynamic Range and Mastering InfoFrame”. This should be all that is required for HDR10 and HDR10+ content.

Bits Per Component

The max bpc standard connector property is a range property that allows the hardware driver to indicate valid bits per color component. Userspace can set this property so long as it is within the driver-provided range to control the output bit depth.

Nice-to-have Features

The bare minimum feature set is present for HDR content and allows us to query display capabilities, pass HDR metadata on to the display, and configure the bits-per-component sent to the display.

This leaves userspace to handle color space transformations, applying the transfer function’s inverse to get linear light values, blending, and re-encoding the results with a (perhaps different) transfer function. All that is possible, of course, but there are a number of hardware features for all those operations and it would be great to use them when they are there rather than performing the operations using expensive CPU or general purpose graphics shaders cycles.

Planes

Before we cover the encoding, decoding, color space transformations, and so on, it’s important to become familiar with planes. Planes hold images and descriptions of how those images should be blended into the final output image. These are analogous to a Wayland surface, which contains a buffer with the image data and metadata for the image.

Planes describe how they should be cropped, scaled, rotated, and blended with other planes to compose the final image sent to the display. These planes are backed by dedicated hardware that can efficiently do those operations. For the curious, the display engine section of Intel’s documentation describes some of the hardware plane’s capabilities. Of particular interest is the diagram around page 258 which illustrates the plane pipeline.

One or more planes are used to compose the image the CRTC scans out to the display.

Decoding, Color Space Transformations, Blending, and Encoding

Graphics hardware typically provides hardware dedicated to efficiently decode, transform, and re-encode images. Encoding and decoding are done with look-up tables that approximate the smooth curve of the desired transfer function by mapping encoded values to optical values and optical values to encoded values. For historical reasons, these are referred to as the gamma and de-gamma LUTs respectively.

The hardware-backed look-up tables for encoding and decoding content with an arbitrary transfer function are exposed to userspace as properties attached to CRTC objects and are also documented in the color management section of the KMS interface. The GAMMA_LUT and DEGAMMA_LUT properties and their respective lengths allow us to set arbitrary transfer function approximations. The number of look-up table entries is hardware-dependent so it may not be appropriate to use them in all cases.

Color space transformations are applied by using the CTM (colorspace transformation matrix) property defined in the color management interface. The CTM property lets userspace define a color transformation matrix to describe how to map from the source color space to the destination color space. When combined with a de-gamma LUT and gamma LUT, it can be used to efficiently decode, transform, and re-encode content for a particular color space.

Finally, the aforementioned planes include composition properties that allow userspace to control how planes are blended together.

However, these APIs have a few problems. The color management properties are attached to the CRTC, which is intended to abstract the display pipeline and is made up of planes. If the planes that make up the CRTC do not all have the same transfer function applied to them or use different color spaces, there is no way to express that each plane needs its own look-up table or color transformation matrix.

While it used to be reasonably safe to assume all the planes were the same (e.g. sRGB), with HDR this is no longer the case. There are likely going to be HDR planes and SDR planes that need to be blended together, so there needs to be a way to express the color management properties - the content encoding - on a plane-by-plane basis. In addition to the content encoding of each plane, we need to be able to express how HDR and SDR content should be blended. In the case of PQ-encoded HDR content, it is expressed in absolute luminance up to 10,000 nits. Userspace needs to express how the SDR luminance range maps into the HDR luminance range so the SDR encoding can be converted into HDR encoding.

Harry Wentland from AMD recently started a discussion on API changes for planes which looks to address these shortcomings.

Summary

All told, a good portion of the work necessary for basic HDR support is done or in progress. Some of the larger challenges are in the compositors as they need to line up client capabilities with hardware capabilities and make up any differences between the two.

Applications that use OpenGL or Vulkan directly don’t need to concern themselves with GTK’s lack of HDR support. However, there are a non-trivial handful of applications that would benefit from GTK supporting HDR content, like Totem, Eye of GNOME (the image viewer), and content creation tools.

Finally, there’s plenty of work on the kernel side to ensure userspace can make use of all the hardware available to it.

HDR in Linux: Part 1

Fri, 07 May 2021 15:55:08 +0000

In recent months I have been investigating high dynamic range (HDR) support for the Linux desktop, and what needs to be done so that a user could, for example, watch a high dynamic range video.

The problem with HDR is not so much that there is no material out there covering it, but that there’s a huge amount of material can be rather confusing. I thought it best to add more material that is probably also confusing, but may help me think through HDR better. The following content is likely wrong as I have no background in colorimetry, the human visual system, or graphics generally. I’d love to hear what makes no sense.

In this post I’ll cover what HDR is and why we care about it. In the next post, I’ll cover the work in specific projects that has been done to support it and what is left to be done.

Background

Before attempting to understand dynamic range, I found it helpful to understand the basics of electromagnetic radiation (light) and how the human visual system (eyes) interact with light so that we can produce graphics for human consumption. With an understanding of how to do that, we can complete our ultimate goal of tricking the human visual system into seeing extremely realistic images of cats even if there are no cats around.

Light

Light can be described as a wave with an amplitude and frequency/wavelength or a particle (photon) with a frequency/wavelength. Human eyes detect a very small frequency range of electromagnetic radiation. More particles is equivalent to a higher amplitude wave. The range of wavelengths eyes typically see is approximately 380 nanometers to 750 nanometers.

Humans detect wavelength of light as colors, with blue in the 450-490nm range, green in 525-560nm range, and red in the 630-700nm range.

More photons of a given wavelength are perceived by humans as a brighter light of the color associated with the wavelength.

Luminance

Luminance is the measure of the amount of light in the visibile spectrum that passes through an area. The SI unit is candela per square meter (cd/m²). This is often also referred to as a “nit”, presumably because cd/m² is somewhat laborious to type out.

The human eye can detect a luminance range from about 0.000001 cd/m² to around 100,000,000 cd/m². We can readily go out into the world and experience this entire range, too. For example, the luminance of the sun’s disk at noon is around 1,600,000,000 cd/m². This is much higher than the human eye can safely experience; do not go stare at the sun.

One important detail is that the human perception of luminance is roughly logarithmic. You might already be familiar with this as decibels are used to describe sound levels. The fact that human sensitivity to luminance is non-linear becomes very important later when we need to compress data.

Human Visual System

The eye is composed of two general types of cells. The rod cells which are very sensitive and can detect very low amplitude (brightness) light waves, but don’t differentiate between wavelengths (color) well.

The cone cells, by contrast, come in several flavors where each flavor is sensitive to a different wavelength. Many humans have three flavors, imaginatively referred to as S, M, and L. S detects “short” wavelengths (blue), M detects “medium” wavelengths (green), and L detects “long” wavelengths (red).

Different colors are produced by stimulating these cone cells in with different ratios of wavelengths.

One thing to keep in mind, as it adds a minor complication to calculating luminance, is that not all the cone cells are equally sensitive. If you were to take a blue light, a green light, and a red light that all emit the same number of photons, the green light would appear the brightest by far. The luminance, therefore, depends on the wavelength of the light.

While the human visual system is capable of detecting a very large range of luminance levels, it cannot do so all at once. If you’re outside on a sunny day and walk into a dark room, it takes some time for your eyes to adjust to the new luminance levels.

Color Spaces

When discussing displays and their capabilities you will hear about color spaces and perhaps see a diagram like this:

There’s a lot going on in this diagram and it is rather confusing (to me, anyway), so it’s worth covering the basics. There are some good blog posts with much more detail.

You’ll notice the outside curve of this color blob has numbers marked along them. These are light wavelengths, and the outer curve is the color we see for that pure wavelength. On the bottom edge there are no wavelengths. This is because those colors are what we see when the long cones and the short cones both sense light. All the colors not on the edge of the curve are created by blending together multiple wavelengths of light.

Often, you’ll hear people talk about the “temperature” of light. This is in reference to how as objects heat up they begin emitting light in the visible spectrum. The curve starting on the red side, passing through the center, and continuing toward blue shows the mapping of temperature to colors.

If we pick three points on this plane and form a triangle, we can create any color enclosed by the triangle by adding various ratios of those three colors together. These points have coordinates in this two-dimensional space, and we can describe the triangle by providing those coordinates. Using this approach, we can describe the colors a display can create by specifying three pairs of coordinates. This is the “color space”.

If you’re using GNOME, you can see a diagram of your display’s color gamut by going to the “Color” settings section, selecting the color profile, and clicking view details:

Note that these diagrams do not include the luminance. Doing so creates a third dimension and can be used to visualize a “color volume”.

Displays

There are several ways to display images for human consumption. One way is to reflect light off a surface that absorbs some frequencies of light and reflect the rest into the eye (paintings, books, e-ink, etc). The other way is to produce light to shine into the eye, such as the liquid crystal or organic LED screen you’re probably looking at right now.

There’s a few challenges with producing the light directly. The first is that we need to produce a range of light frequencies to cover the visible spectrum of human vision. As humans usually have three types of frequency-sensing cells, we don’t necessarily need to produce every single frequency in the visible spectrum. Instead, we can pick three frequencies that align with where the cone cells are sensitive and by adjusting the ratio of these three produce light that contains all information the human eye uses and none of the information it cannot detect.

This is why a display pixel is typically made up of blueish, greenish, and reddish components. Since this isn’t a perfect world, displays don’t tend to be capable of producing a single frequency. The frequency range of these components impact how the three overlapping cone sensitivities can be stimulated and therefore determine the range of colors the display is capable of producing. This range is referred to as the display’s “color gamut”. The color gamut can vary quite a bit from display to display, so we need to account for this when we encode images. We also need to handle cases where the display cannot produce the color we wish to show. We could not, for example, accurately display an image of a laser emitting pure 525nm light unless the green sub-pixel happened to also emit pure 525nm light. This is called “gamut mapping” as we map one color gamut to another color gamut.

The second issue is that if we want to perfectly replicate the real world, we would need to be able to emit light in the range of 0.000001-1,600,000,000 cd/m². However, as we’ve already established, the sun is not safe to look at so it would be best to not faithfully reproduce it on a display. Even if we restrict ourselves to the 0.000001-100,000,000 cd/m² range, the power requirements and heat produced would be astronomical. We have to attempt to represent the world using a smaller luminance range than we actually experience. In other words, we have to compress the luminance range of our world to fit the luminance range of the display. This process is often referred to as “tone-mapping”.

Dynamic Range

Dynamic range is the ratio between the smallest value and largest value. When used in reference to displays, the value is luminance.

The lowest luminance level is determined by how much ambient light is reflecting off the surface of the display (e.g. are you sitting in a dark room, outside on a sunny day, etc), how much of the backlight (if there is one) leaks through the display panel, and so on.

The highest luminance level is determined by the light source of the display. This depends on the type of display, power usage requirements, user preferences, on the ambient temperature (emitting visible light also emits heat, and excessive heat can damage the display components), and perhaps other factors I am not aware of.

As both these values depend on environmental factors the dynamic range of a display can change from moment to moment, which is something we may wish to account for when compressing the luminance range of the world to fit the particular display.

So what is a high dynamic range display? In short, a display that is simultaneously capable of lower luminance levels and higher luminance levels than previous “standard dynamic range” displays. What these levels are exactly can be hand-wavy, but VESA provides some clear specifications.

This higher dynamic range ultimately means the way in which we compress the luminance range (tone-map) needs to change.

Light from Scene to Display

Now that we know the pieces of the puzzle, we can examine how an image ends up on our displays. In this section, we’ll cover the journey of an image from its creation to its display. Perhaps the most self-contained example is a modern video game. This allows us to dodge the added complication of camera sensors, but the process for real-world image capture is similar.

Scene-referred Lighting

Video games typically model a world complete with realistic lighting. When they do this, they need to work with “real world” luminance levels. For example, the sun that lights the scene may be modeled to be seen as 1.6 billion nits. Working with real world luminance that cannot be directly displayed is often called “scene-referred” luminance. We have to transform the scene-referred luminance levels to something we can display (“display-referred” luminance) by compressing or shifting the luminance values to a displayable range.

You might think it’s best to leave content in scene-referred lighting levels until the last possible moment (when the display is setting the value for each pixel), and that would indeed make some things simpler. However, it makes some things more difficult, like attempting to blend the scene-referred image with an image that is already display-referred. When doing operations like that, we need every part to be in the same frame of reference, either scene-referred or display-referred.

There’s a more difficult problem with keeping all content scene-referred, though. Representing numbers accurately in the 0.000001-1,600,000,000 cd/m² range requires a lot of bits, and we’re about to have a serious bit budget imposed on us. We have to transport the image data from the host graphics processing unit to the display. While both HDMI and DisplayPort have rather enormous bandwidth, the image resolution is also quite large. If we used a 32 bit floating point number for each red, green, and blue (RGB) component of a pixel on a display with 3840 x 2160 (4K) pixels, it would require 32 x 3 x 3840 x 2160 bits per frame, or 760 Mbits per frame. At a modest 60 frames per second, that’s 44.5 Gbits. DisplayPort 1.4 provides 25.9 Gbits and HDMI 2.0 give us 14.4 Gbits.

Fortunately for us, as long as we ensure the way we encode luminance levels matches up with the way the human eye detects luminance levels, we can save a lot of bits. In fact, for the “standard” dynamic range, we can manage with only 8 bits (256 luminance levels) per color (24 bits total for RGB).

Tone-mapping

In order to convert scene-referred, real world luminance to a range for our target display, we need to:

Have an algorithm for mapping one luminance value to another. Input values have a range [0,∞) and the output values need to cover the range of our target display.
Know the capability of our target display.

For now we’ll focus on #1, although #2 is important and, unfortunately, somewhat tricky.

There are many different approaches to tone-mapping and results can be subjective. This is not a post about tone-mapping (there are many multi-hundred page publications on the subject), we’ll just go over a couple examples to get the idea.

An extremely silly, but technically valid tone-mapping algorithm would be f(x) = 1, where x is the input luminance. This maps any input luminance level to 1.

A more useful approach might be the function f(x) = xⁿ/(xⁿ + sⁿ) where x is the input luminance, n is a parameter changes the slope of the mapping curve, and s shifts the curve on the horizontal axis. As a somewhat random example, with n = 3 and s = 10:

This sigmoid function gently approaches the minimum and maximum values of 0 and 1. We can then map the [0, 1] range to the display luminance range.

There are many articles and papers out there dedicated to tone mapping. There’s a chapter on the topic in “The high dynamic range imaging pipeline” by Gabriel Eilertsen, for example, which include a number of sample images and a much more thorough examination of the process than this blog post.

This process is occasionally referred to as an “opti-optical transfer function” or “OOTF” as it maps optical (luminance) values to different optical values. Note that it is also possible to tone-map luminance values that are encoded for storage or transportation, so-called “electronic” values, which you might see referred to as electro-electronic transfer functions or “EETF”

Encoding

Once we’ve tone-mapped our content to use the display’s luminance range, we need to send the content off to the display so that we can see it. As the section on scene-referred light mentioned, we have a limited amount of bandwidth, and any increase or decrease in bits used per pixel impacts how many bits are required for a frame, and thus how many frames we can send to the display every second. The key is to use as few bits as we can get away with.

The method used to encode luminance levels is called the “transfer function”, “opto-electronic transfer function”, or “OETF”. These are functions that approximate the human’s sensitivity to luminance levels.

Since the display needs to decode the encoded data we send it back to linear luminance values, we need functions that can be easily “undone” with an inverse function. The property we’re looking for in our encoding and decoding functions is g(f(x)) = x. This is called the “inverse transfer function”, “electro-optical transfer function”, or “EOTF” since it converts the “on-the-wire” values back to optical values.

Note that the terminology and definitions of these transfer functions can be confusing and vague. Some people appear to use OETF to mean both a tone-mapping operation and a transfer to electronic values.

Encoding SDR

The “standard dynamic range” is not particularly standard, but usually tops out around 300 nits. Typically, 8 bits per RGB component is used. 8 bits allow us to express 256 luminance levels per component.

The Naïve Approach

One method to encoding would be to evenly spread each level across the luminance range. What would this look like? In our thought experiment, we’ll assume the monitor’s minimum luminance is 0.5 nits, and its maximum luminance is 200 nits. This gives us a range of 199.5 nits. Equally distributed, each step increases the luminance by about 0.78 nits.

Sending [0, 0, 0] for red, green, and blue results in the minimum luminance of 0.5 nits. Adding a step [1, 1, 1] would result in 1.28 nits.

Sending [254, 254, 254] results in 198.44 nits, and sending [255, 255, 255] results in 200 nits.

The problem with this approach is that with lower luminance values, each step is above what the human eye differentiate, its just-noticeable difference. At higher luminance levels, each step is so far below the just-noticeable difference that it would require many steps for a human to notice any difference at all. It results in clearly visible bands in the lower luminance areas of the image we display and completely undetectable detail in the higher luminance areas of the image. We could resolve this by adding more bits (and therefore more levels), but we can’t afford to do that without giving up resolution and framerate.

Instead, we want to take these wasted levels at high luminances and use them in lower luminances.

Gamma

The “gamma” transfer function has a long and interesting history and is part of the sRGB standard. The sRGB version is actually a piecewise function which uses the gamma function for all but the very lowest luminance levels, but we can ignore that complication here.

The basic function is in the form f(x) = Axᵞ. In sRGB, A = 1.055 and 𝛾 = 1/2.4. The inverse function used to decode the luminance is A = 1/1.055 and 𝛾 = 2.4

The encoding curve looks like this:

In this graph, we map the linear, display-referred luminance values into the range [0, 1]. These input luminances are mapped to an encoded value also in the range [0, 1]. We can see from the graph above that the lower third of the linear luminance levels get mapped on the lower two thirds of our encoding range.

We can convert the output to integer values between 0 and 255 by multiplying the encoded luminance value by 255 and rounding:

Now, instead of wasting many of our precious bits on luminance steps on the high end that no one can see, we’re spending most of them on low luminance levels.

This is a pretty good approach when we have 256 steps (8 bits) per component and luminance range of a few hundred nits, but the further we stretch the range of nits, the bigger our steps have to be, and eventually we’ll cross a point where some steps are about the just-noticeable difference line.

We can fix this by increasing the number of bits used. What happens if we move to 10 or even 12 bits? At 12 bits per component, we have 4096 steps.

At first glance this seems fine, but what if we adjust the graph to display actual nit values for the linear light? Here’s the same graph with linear light using a range of 0-1000 nits.

We can see that fewer than half our steps, or approximately 2000, land in the luminance ranges we use today for standard dynamic range displays. Once again, we’re allocating too many steps to the very high luminance levels where the human eye becomes less and less sensitive. This is even worse as the luminance range increases:

Thus, for displays with a higher dynamic range, we may want to adjust the transfer function to get the most value out of each bit we spend.

Encoding Higher Dynamic Ranges

As we saw when examining the gamma transfer function, it works reasonably well for low luminance ranges, but as the range increases it starts to be sub-optimal.

There are two major approaches for higher dynamic range encoding. Both have advantages and disadvantages, and displays can support either or both. If you’re curious if your displays are capable of using either of these approaches, you can check by installing edid-decode and running:

find /sys/devices -name edid -exec edid-decode {} \;

Look for a “HDR Static Metadata Data Block” and see what is included in the electro-optical transfer function list. This section will likely not be present if your display doesn’t support HDR.

The Perceptual Quantizer

The Perceptual Quantizer (PQ) transfer function, sometimes referred to by the much less cool name “SMPTE ST 2084”, defines a new curve to use for encoding. It is unlike the gamma transfer function in that it is defined to cover a well-defined luminance range, rather than just being fit to whatever the display happens to be capable of. The luminance range the curve is defined to cover is 0.0001-10,000 cd/m².

This is a useful concept when encoding the content as we can now express physical quantities in addition to “more/less luminance”. It gives the decoding side a frame of reference if it needs to further tone-map the content.

The curve looks like:

This graph is only part of the curve, and you can see the slope is so extreme it doesn’t render very well. It packs a vast majority of the code points into the lower luminance range where the eye is most discerning. Currently, it’s common to use this curve with 10 bits per component.

The function that describes this curve is more complicated than the gamma curve so I won’t attempt to render it poorly here. The curious can consult the Wikipedia page linked above.

The perceptual quantizer transfer function can be paired with metadata describing the image or images it is used to encode. This metadata is usually the primary colors of the display used to create the content as well as luminance statistics like the image’s minimum, maximum, and average luminance. This metadata allows consumers of the content to perform better tone-mapping and gamut-mapping since it describes the exact color volume of the content.

The PQ curve supports encoding luminance up to 10,000 nits, but there are no consumers displays capable of that. Even professional displays built for movie-making are well short of that, currently around 4,000 nits. This is still much higher than current consumer displays, so the metadata can be used to tone-map and gamut-map the content from the capabilities of fancy display the film studio used to the capabilities of the individual display. In the future, when consumer monitors become more capable, the same films will look better as they require less tone-mapping and the original intent can be more accurately rendered.

The Hybrid Log-Gamma

As the name suggests, the second approach for higher dynamic range encoding is to use a hybrid of the gamma function and a log function. The hybrid log-gamma (HLG) curve is designed with backwards-compatibility in mind.

HLG uses a piecewise function defined as follows when the input luminance has been normalized to a range of 0-1:

When the input luminance L is between 0 and 1/12: sqrt(3) * L^0.5 (this is our old friend the gamma function f(x) = Axᵞ, with A = 1.732 and 𝛾 = 1/2)
When input luminance L is between 1/12 and 1: a * ln(12L - b) + c where a = 0.17883277, b = 1 - 4a, and c = 0.55991073.

In the above graph, the red curve is the gamma curve, the green curve is the log curve, and the vertical line is the point where the gamma curve stops being used in favor of the log curve when encoding.

As this encoding scheme is defined in terms of relative luminance like the gamma transfer function, there is no metadata for tone-mapping.

Transmission and Display

Once we’ve encoded the image with a transfer function the display supports, the bits are sent to the display via HDMI or DisplayPort. At this point, what happens next is up to whatever is on the other end of the cable. I have not pulled apart a display and learned what secrets it holds, but we can make some reasonable guesses without destroying anything:

If metadata is provided (PQ-encoded images only), the display examines it and determines of any tone-mapping is required due to the content exceeding its capabilities. If it’s not provided, it likely assumes the content spans the entire range defined by PQ.
If the display includes light sensors to detect ambient light levels, it might decide to tone-map the content, even if it’s capable of displaying all luminance levels encoded.
Displays tend to include configuration to alter the color and brightness. It will take these into account when deciding if/how to tone-map or gamut-map the content we gave it.
Some gamut-mapping and tone-mapping will occur in the display depending on all the above variables, at which point it will emit some light.

Now, that’s a very high-level overview of the process, but we can map these high-level steps to portions of the Linux desktop and discuss what needs to change in order for us to support HDR.

Summary

Now that we understand (in theory) HDR, it’s worth asking what all this is good for.

If, as part of your movie-watching experience, you want an eye-wateringly bright sunset where you can still make out the blades of grass in the shadow of a rock this is an important feature. The more cinematic video games could look even better. These are the most obvious applications, and where many users will encounter HDR, but it’s not necessarily limited to entertainment. HDR is all about transmitting and displaying more information, so any process that involves visualization could benefit from more luminance levels.

As we’ll see in the next post, however, there is a good bit of work left to be done before you can, for example, enjoy an HDR film in GNOME’s Videos application.