Alex DeLorenzo

Encrypted Root with LUKS and Opal

Tue, 21 May 2024 18:00:00 +0000

Using `cryptsetup`’s native Opal support to decrypt self-encrypting drive partitions at boot with LUKS and systemd

Prior to cryptsetup 2.7.0, using self-encrypting drives (SEDs) on Linux required the use of tools like sedutil to boot in order to use hardware encryption, otherwise the drives were limited to using LUKS software encryption.

If you want to encrypt your root or boot partitions via the Opal specification for hardware encryption, you either needed a motherboard with UEFI/BIOS that supports Opal, or you need to flash a pre-boot authorization (PBA) image to your Opal drive and boot from that.

On systems without Opal support, SEDs will trick firmware into booting an unencrypted PBA image that then handles the mounting and booting of the encrypted disk itself.

With the new release of cryptsetup, you can use self-encrypting drives’ hardware encryption directly with LUKS via Opal interfaces.

Click here to skip directly to the tutorial 👇

Hardware encryption versus software encryption

This comes down to choice. Modern NVMe drives with self-encryption support automatically encrypt your data even if you don’t set up encryption yourself. The drives ship with a manufacturer-set key that encrypts your data, but with the password disabled. Even if you personally aren’t using a password to unlock your drive, your data is still being encrypted anyway.

By shipping drives with encryption turned on without a password, it allows you to later set a password for your drive without having to erase all of your data just so that it can be encrypted using your password. Explicitly setting a password just encrypts the drive’s underlying encryption key, and requires that password to decrypt the drive.

tl;dr: Your self-encrypting drives are already using encryption even if it is “off”. You’re already paying for the cost of hardware self-encryption whether you set a password or not.

Software encryption, on the other hand, needs to be explicitly set up. Using software encryption has a cost: it takes time and energy to encrypt and decrypt data. All of your data must flow through your CPU in order to read or write anything to disk, which takes time and a considerable amount of power, even if your processors implement AES-NI acceleration.

If you’re on a laptop, for example, software encryption can impact your battery life. It can also impact the speed at which you’re able to access data on your machine. Comparing the two, you don’t have to pay any additional cost in terms of speed or battery life if you choose to enable hardware encryption, since it’s always on anyway.

That said, by using hardware encryption alone, you are placing full trust in your hardware manufacturer’s encryption implementation. That trust can be misplaced, as self-encrypted drives’ firmware implementations are frequently broken. Even if you’re using strong underlying encryption, if the surrounding implementation can be hacked, that encryption doesn’t mean much if an attacker can easily exfiltrate your keys or data.

When it comes to encryption, mobile users are presented with the choice to use software encryption, and pay the price in terms of time and energy, or use hardware encryption, which can improve speed and battery life at the expense of potentially using poor encryption implementations.

If your machine is stolen, hardware encryption might keep your data safe from unsophisticated criminals, but it might not keep you out of prison if you’re a whistleblower that a government wants to silence. For 99.999% of people, keeping data safe from petty criminals is sufficient enough. Just keep in mind that it’s not perfect and don’t hinge your life or freedom on some random drive’s encryption implementation.

Booting self-encrypted drives on machines without Opal support

My hardware doesn’t support self-encrypting drives natively, so if I want to encrypt my root and/or boot partitions, I need to jump through some hoops. Those hoops include finding or building PBA images, enabling pre-boot authorization on my self-encrypting drives, ensuring my PBA is set up to boot correctly and configuring the system to use my authentication methods of choice.

Luckily, it turns out that the Drive Trust Alliance (DTA) published pre-compiled PBA images for use with sedutil. That sounds convenient, until you learn that those images were compiled 4+ years ago using kernels that may or may not support your hardware. Similarly, if you want to unlock your Opal drives using multiple factors, like FIDO authenticators, you need to modify your PBA image to do so. The PBA from the DTA doesn’t support hardware authenticators like Yubikeys natively.

On normal drives that do not have encryption support, you can use software encryption via cryptsetup, LUKS and systemd. It’s easy to set up and boot your encrypted drives, and cryptsetup natively supports enrolling FIDO authenticators to unlock encrypted partitions. Set it up right, and you can easily use a Yubikey at boot to unlock your root partitions. It’s relatively uncomplicated.

In contrast, using self-encrypting drives is much more cumbersome, and potentially involves rolling your own bootable PBA image just to unlock your drive.

That was the state of things if you wanted to use Opal with motherboard firmware that doesn’t support self-encrypting drives.

New `cryptsetup` has native Opal support

Now, with cryptsetup 2.7.0, there is native support for hardware encryption for drives that support Opal encryption, even if your firmware doesn’t implement support for Opal drives.

Set up correctly, you can use cryptsetup, LUKS and systemd with hardware encryption as easily as it is to use software encryption. There is no need to trick your BIOS into booting a PBA, and there is no need to build your own PBA image if you want to use a Yubikey. In fact, there is no need for a PBA at all.

Typical encrypted file-system scheme with `cryptsetup`

When using software encryption, it’s typical to turn on Secure Boot and leave your /boot unencrypted so you can bootstrap your kernel and initramfs to automatically unlock your encrypted / for you. This can be automated so that all you need to do is supply your password, and/or Yubikey + PIN, at boot to unlock your root partition.

Depending on how much of the Opal specification your self-encrypting drive accurately implements, Opal drives can use hardware encryption for specific regions of the drive while leaving others unencrypted.

That means you can use a similar setup to your software encryption scheme where you turn on Secure Boot and leave your /boot unencrypted so you can easily unlock your /. All you have to do is create a /boot EFI partition and a separate partition for /. You can then tell your Opal drive to encrypt your / region.

From that point on, you can use hardware encryption with the same exact tools you would software encryption, allowing you to easily boot, enter your credentials and unlock your drive without a hacky pre-boot authorization process complicating your boot process.

How to use hardware encryption with `cryptsetup`

This guide assumes you’re using systemd-boot and systemd-cryptsetup-generator to unlock your LUKS partitions at boot. I add additional information and links for those who are not using this setup, but be aware that this guide has only been tested using systemd for boot and cryptsetup support.

cryptsetup only works with self-encrypting drives over PCIe, not USB.

Find your NVMe device in /dev. Mine is /dev/nvme1n1. Yours might be different.

Detect Opal support

Not all drives support the Opal specification. You can see if your drive supports it by running the following:

$ sudo sedutil-cli --scan

Scanning for Opal compliant disks
/dev/nvme1  2  YOUR_DRIVE_MODEL		YOUR_DRIVE_ID
No more disks present ending scan

If you see a number in the second column, like the 2 above, your drive supports Opal.

If you see the word No, your drive does not support Opal:

$ sudo sedutil-cli --scan

Scanning for Opal compliant disks
/dev/nvme0 No  YOUR_DRIVE_MODEL		YOUR_DRIVE_ID
No more disks present ending scan

Reset your Opal drive

This will erase your drive and reset it.

You will need to locate the PSID of your drive to prove you have physical access to it. On my drives, it is printed in small text on a sticker attached to the drives themselves. If the PSID has dashes, ignore them. The PSID should be one long string of letters and numbers without any special symbols.

When you find it, save it somewhere you won’t lose it and then use sedutil-cli with your drive:

$ sudo sedutil-cli --PSIDrevert "YOUR_PSID_HERE" /dev/nvme1n1

Or you can use cryptsetup instead:

$ cryptsetup luksErase --hw-opal-factory-reset /dev/nvme1n1
Enter OPAL PSID: YOUR_PSID_HERE

Replace YOUR_PSID_HERE with your drive’s PSID.

Alternatively, if you’ve already set up encryption on your drive’s specific locking range, and know the passwords you set yourself, you can skip the stage above. Keep in mind that setting up LUKS on an Opal locking range will reset and erase that locking range, anyway.

Create boot and root partitions

This will overwrite your partition table and you will lose the data on the drive.

Create a /boot EFI partition and a root partition using parted.

$ sudo parted /dev/nvme1n1

Inside parted, run:

# Create a GPT partition table
mklabel gpt

# Create an EFI partition
mkpart ESP fat32 1MiB 513MiB
set 1 boot on

# Create a root partition that we'll encrypt
mkpart primary btrfs 513MiB 100%

quit

Format your EFI partition:

$ sudo mkfs.fat -F32 /dev/nvme1n1p1

Set up LUKS with Opal support

Running luksFormat will erase and format your specified partition, you will lose the data on it. Encrypting data in-place is not supported.

Ensure that you have cryptsetup 2.70 or newer. Older versions of cryptsetup will not work.

Determine your root partition’s location in /dev. My EFI partition is /dev/nvme1n1p1 and my root partition to be encrypted is /dev/nvme1n1p2.

Then run the following to set up LUKS:

$ sudo cryptsetup luksFormat /dev/nvme1n1p2 --type luks2 --hw-opal-only

The --hw-opal-only flag tells cryptsetup to use hardware encryption only. If you want to use software encryption on top of hardware encryption, pass the --hw-opal flag instead.

You might be asked to set user and administrator password, do so and don’t forget them.

Let’s open and decrypt the partition so we can format a file-system on it. You can replace DISK_NAME_GOES_HERE with any name, the value will determine what Device Mapper will call your unlocked drive under /dev/mapper.

$ sudo cryptsetup open /dev/nvme1n1p2 DISK_NAME_GOES_HERE
$ sudo mkfs.btrfs -L EncryptedRoot /dev/mapper/DISK_NAME_GOES_HERE

I’m using btrfs, but feel free to use EXT4, F2FS or whatever you want for your root partition. Note that the rest of this guide will assume you’re using btrfs, too.

Enroll your authentication methods

LUKS partitions can be unlocked via a variety of methods, like passwords, key-files and hardware authenticators. Enroll the methods you’d like to use to decrypt your drive at boot.

If you want to use a FIDO authentication device, you can use systemd-cryptenroll to enroll one yourself:

$ systemd-cryptenroll --fido2-device=auto /dev/nvme1n1p2

For plaintext passwords, you can run:

$ systemd-cryptenroll /dev/nvme1n1p2 --password

Here is a good resource for using systemd-cryptenroll to enroll various authentication methods with LUKS.

Confirm configuration

Before going any further, let’s ensure that the encrypted partition is set up properly.

Close your open LUKS partition:

$ sudo cryptsetup close DISK_NAME_GOES_HERE

Then try to open it again:

$ sudo cryptsetup open /dev/nvme1n1p2 DISK_NAME_GOES_HERE --token-only

Omit the --token-only flag if you aren’t unlocking your partition with something like a Yubikey.

You should now see your device in /dev/mapper:

$ ls /dev/mapper
/dev/mapper/DISK_NAME_GOES_HERE

Ensure your LUKS partition is set up to your expectations:

$ sudo cryptsetup luksDump /dev/nvme1n1p2

Set your machine to unlock LUKS

We’re going to need to create a /etc/crypttab file and edit the kernel command-line options to point the kernel to our encrypted root file-system.

Create a `crypttab` file

Determine your partition’s UUID by using blkid and noting its PARTUUID label:

$ blkid
/dev/nvme1n1p2: UUID="87425985-0ed3-486f-8d66-b07bbf27f8ea" TYPE="crypto_LUKS" PARTLABEL="EncryptedRoot" PARTUUID="fec2fe57-8a1b-43af-96b4-05cf1b944e03"

With root permissions, create a /etc/crypttab file like so, replacing the UUID value with the PARTUUID value of your partition:

# <name>               <device>                         <password> <options>
DISK_NAME_GOES_HERE      UUID=PARTUUID_GOES_HERE     -     luks

A common naming scheme is just prepending your UUID with luks-.

Here’s an example crypttab:

# <name>               <device>                         <password> <options>
luks-fec2fe57-8a1b-43af-96b4-05cf1b944e03      UUID=fec2fe57-8a1b-43af-96b4-05cf1b944e03     -    luks,discard,fido2-device=auto,token-timeout=10

Your options should reflect your unlocking method you used with systemd-cryptenroll. If you used a FIDO authenticator, add fido2-device=auto to your options. Similarly, you can set a token timeout. If you’re using a password, skip the fido2 and token timeout options.

If you want to enable TRIM on the partition, add the discard option to your crypttab. Keep in mind that using TRIM has security implications.

Install your root file-system

You can copy over an existing file-system using something like rsync or cp. If you’re using btrfs, you can use btrfs send and btrfs receive to copy your root subvolume(s) over.

You can also use debootstrap or pacstrap to bootstrap a new Debian or Arch rootfs.

You might want to create a root subvolume:

$ mkdir /tmp/disk
$ sudo mount /dev/mapper/DISK_NAME_GOES_HERE /tmp/disk
$ btrfs subvolume create disk/@
$ sudo umount /tmp/disk

Configure `/etc/fstab`

You need to set your root file-system entry correctly.

Instead of pointing directly to your disk, you need to point / to the value of the custom name you set in /etc/crypttab.

Edit your /etc/fstab file to point to your unlocked LUKS device in /dev/mapper:

/dev/mapper/DISK_NAME_GOES_HERE      /       btrfs   subvol=/@,defaults 0 0

The above assumes your / has a /@ subvolume used as the root subvolume.

You can also use other btrfs features like noatime, compress=zstd, ssd, discard=async, autodefrag:

/dev/mapper/luks-fec2fe57-8a1b-43af-96b4-05cf1b944e03       /     btrfs     subvol=@,defaults,noatime,compress=zstd,ssd,discard=async,autodefrag 0 0

Alternatively, you can use the unlocked LUKS volume’s UUID instead of pointing to /dev/mapper/DISK_NAME_GOES_HERE.

Make sure you have an entry for your EFI partition in /etc/fstab. You might not be able to boot without it.

Unlock at boot

Edit kernel command-line options

On my system, I can edit my kernel command-line options by changing the /etc/kernel/cmdline file.

We need to append the rd.luks.uuid and root parameters:

rd.luks.uuid=PARTUUID_GOES_HERE root=/dev/mapper/DISK_NAME_GOES_HERE

This is how it would look using the example values from above:

rd.luks.uuid=fec2fe57-8a1b-43af-96b4-05cf1b944e03 root=/dev/mapper/luks-fec2fe57-8a1b-43af-96b4-05cf1b944e03

After adding those parameters, make sure they’re propagated to your bootloader like systemd-boot or GRUB.

On EndeavourOS, for example, simply running sudo reinstall-kernels will ensure that the new kernel command-line parameters will be supplied at boot.

Configure your early boot environment

You might have to configure mkinitcpio or dracut so that you can unlock your drive at boot.

You might have to add the following modules to /etc/mkinitcpio:

keyboard
systemd
sd-vconsole
sd-encrypt

With dracut, you might need to add crypttab to a configuration file in /etc/dracut.conf.d:

install_items+=" /etc/crypttab "

You might also have to add the systemd and resume modules to dracut:

add_dracutmodules+=" systemd resume "

Again, do whatever you need to do on your distribution to update your initramfs. For example, reinstall-kernels will do that automatically.

Reboot

Cross your fingers and reboot. If everything went well, your system will boot to a prompt asking for you to authenticate and unlock your root partition.

Encrypted swap and hibernation

Once your encrypted root partition is set up and you’ve booted into it, you might want to set up encrypted swap, too.

Create a swap file

Btrfs gives us some tools that make setting up an encrypted swap with hibernation support easy.

Mount your LUKS partition to create a swap subvolume:

$ mkdir /tmp/root
$ sudo mount -o subvol=/@ /dev/mapper/DISK_NAME_GOES_HERE /tmp/root
$ sudo btrfs create subvolume /tmp/root/swap

You’ll want to put it in its own subvolume because you cannot take snapshots of subvolumes that contain swapfiles.

Choose the size of your swap file, if you want to enable hibernation, it needs to be the same size, or larger, than the amount of memory you have.

$ sudo btrfs filesystem mkswapfile --size 64G --uuid clear /tmp/root/swap/swapfile

Configure `fstab`

Add your swap file to your /etc/fstab:

/swap/swapfile      swap        swap        defaults    0   0

Reload systemd and test out the swap:

$ sudo systemctl daemon-reload
$ sudo swapon -a

Get the swap file offset

Get the resume offset:

$ btrfs inspect-internal map-swapfile -r /tmp/root/swap/swapfile
198273664  # YOUR_RESUME_OFFSET

Get your LUKS partition UUID

Get your unlocked LUKS disk’s UUID:

$ blkid
/dev/mapper/DISK_NAME_GOES_HERE: UUID="YOUR_LUKS_UUID" UUID_SUB="IGNORE_THIS" BLOCK_SIZE="4096" TYPE="btrfs"

Here’s an example output:

$ blkid
/dev/mapper/luks-fec2fe57-8a1b-43af-96b4-05cf1b944e03: UUID="ae72c069-9729-43dc-adab-ea0054ef70ee" UUID_SUB="511b5768-dd75-453e-9ecf-a194e9167f2b" BLOCK_SIZE="4096" TYPE="btrfs"

In this case, you’d want to keep the UUID value ae72c069-9729-43dc-adab-ea0054ef70ee and ignore the rest.

Adjust your kernel parameters

Edit your kernel’s command line options. On my system, that means editing /etc/kernel/cmdline:

resume=UUID=YOUR_LUKS_UUID resume_offset=YOUR_RESUME_OFFSET

Using the example parameters:

resume=UUID=ae72c069-9729-43dc-adab-ea0054ef70ee resume_offset=198273664

And then make sure those changes propagate to your kernel’s command-line at boot. This is where running reinstall-kernels might help.

Reboot and test it out

Restart your machine and then run the following to see if hibernation is configured correctly:

$ sudo systemctl hibernate

Toll-free Bridging in Kotlin Native

Fri, 22 Mar 2024 18:00:00 +0000

Toll-free bridging is a concept in macOS and iOS development that allows developers to use data types interchangeably between the Core Foundation and Foundation Kit frameworks.

About Core Foundation & Foundation

Core Foundation belongs to the open-source framework created by Apple as a foundation to build libraries and applications for Mac OS. It is C-based and inspired by Foundation.

Foundation, on the other hand, was created by NeXT for NeXTSTEP for similar purposes, and it uses the Objective-C programming language. Allegedly, it was created for their EOF product.

Tools from the former kit typically start with CF for Core Foundation in their names, while tools from Foundation start with NS for NeXTSTEP.

Toll-free Bridging

Mac OS merged with NeXTSTEP to create Mac OS X and the need to merge the Mac Toolbox with OpenStep gave forth toll-free bridging of types between Core Foundation and Foundation. That is, conversion of equivalent types between both frameworks without the need for checking types and converting between them. There is no cost to cross the Core Foundation to Foundation and Foundation to Core Foundation bridge.

Toll-free bridging is used by Cocoa to interact with macOS APIs.

When using tools from Foundation, you are able to use CF types in places where NS types are expected. Similarly, when using Core Foundation, you’re able to pass in NS objects to APIs that expect CF types.

Click here to see how Core Foundation types map to Foundation classes below 👇

Core Foundation type	Foundation class
`CFArrayRef`	`NSArray`
`CFAttributedStringRef`	`NSAttributedString`
`CFBooleanRef`	`NSNumber`
`CFCalendarRef`	`NSCalendar`
`CFCharacterSetRef`	`NSCharacterSet`
`CFDataRef`	`NSData`
`CFDateRef`	`NSDate`
`CFDictionaryRef`	`NSDictionary`
`CFErrorRef`	`NSError`
`CFLocaleRef`	`NSLocale`
`CFMutableArrayRef`	`NSMutableArray`
`CFMutableAttributedStringRef`	`NSMutableAttributedString`
`CFMutableCharacterSetRef`	`NSMutableCharacterSet`
`CFMutableDataRef`	`NSMutableData`
`CFMutableDictionaryRef`	`NSMutableDictionary`
`CFMutableSetRef`	`NSMutableSet`
`CFMutableStringRef`	`NSMutableString`
`CFNullRef`	`NSNull`
`CFNumberRef`	`NSNumber`
`CFReadStreamRef`	`NSInputStream`
`CFRunLoopTimerRef`	`NSTimer`
`CFSetRef`	`NSSet`
`CFStringRef`	`NSString`
`CFTimeZoneRef`	`NSTimeZone`
`CFURLRef`	`NSURL`
`CFWriteStreamRef`	`NSOutputStream`

Here is a good summary circa 2006 of the history and inner workings of toll-free bridging from someone who was involved with its design and implementation.

Kotlin Native

Kotlin Native uses the Kotlin programming language as a frontend and the LLVM compiler as a backend instead of targeting the JVM for compilation. Kotlin Native compiles down to native code for multiple platforms including Windows, Linux, macOS, iOS and Android.

Interoperability

Kotlin Native provides built-in C and Objective-C interoperability. Kotlin Native makes it easy to import and export C and Objective-C compatible ABIs.

As a result, Kotlin Native can be used for macOS and iOS development using both platforms’ native libraries.

Click here to see how Kotlin Native types map to Swift and Objective-C types below 👇

Kotlin	Swift	Objective-C
`class`	`class`	`@interface`
`interface`	`protocol`	`@protocol`
`constructor`/`create`	Initializer	Initializer
Property	Property	Property
Method	Method	Method
`enum class`	`class`	`@interface`
`suspend->`	`completionHandler:`/`async`	`completionHandler:`
`@Throws fun`	`throws`	`error:(NSError**)error`
Extension	Extension	Category member
`companion` member <-	Class method or property	Class method or property
`nan`	`nil`	`nil`
`Singleton`	`shared` or `companion property`	`shared` or `companion property`
Primitive type	Primitive type / `NSNumber`	`nan`
`Unit` return type	`Void`	`void`
`String`	`String`	`NSString`
`String`	`NSMutableString`	`NSMutableString`
`List`	`Array`	`NSArray`
`MutableList`	`NSMutableArray`	`NSMutableArray`
`Set`	`Set`	`NSSet`
`MutableSet`	`NSMutableSet`	`NSMutableSet`
`Map`	`Dictionary`	`NSDictionary`
`MutableMap`	`NSMutableDictionary`	`NSMutableDictionary`
Function type	Function type	Block pointer type
Inline classes	Unsupported	Unsupported

However, the language doesn’t abstract away the need to understand the underlying systems and APIs for the platforms you’re targeting. Kotlin Native does nothing to change the need to manually invoke toll-free bridging when working with Core Foundation and Foundation libraries. You still need to cross the bridge from C to Objective-C/Kotlin and the other way around.

Kotlin Native also doesn’t abstract away the need for manual memory management. It’s up to you to to know who owns your types and when they should be freed from memory.

How to use it

From Objective-C and C

You can use toll-free bridging just by casting NS types as CF types and vice versa. If you do that, however, you have to remember to release memory manually.

Here’s an example from Apple’s documentation:

NSLocale *gbNSLocale = [[NSLocale alloc] initWithLocaleIdentifier:@"en_GB"];
CFLocaleRef gbCFLocale = (CFLocaleRef) gbNSLocale;

In this example, we create a Foundation object, a NSLocale, and then cast it to its Core Foundation equivalent type.

Below, we then use that Core Foundation type to call a Core Foundation function, then we cast that object as a NSString pointer to use it with a Foundation function, NSLog():

CFStringRef cfIdentifier = CFLocaleGetIdentifier (gbCFLocale);
NSLog(@"cfIdentifier: %@", (NSString *)cfIdentifier);

Finally, we have to release the memory used by the CF type:

CFRelease((CFLocaleRef) gbNSLocale);

Toll-free bridging and ARC

What if you want to hand off ownership of a Core Foundation type to Objective-C’s ARC? And what if you want to hand ownership of an Objective-C object to C?

Core Foundation provides functions like CFBridgingRelease() to move ownership of CF types to Objective-C and CFBridgingRetain() to give ownership of an NS type to C.

// Bridge to Core Foundation
NSArray *foundationArray = @[@"Objective", @"C", @"Example"];
CFArrayRef coreFoundationArray = CFBridgingRetain(foundationArray);

The last line in the example above moves ownership of the NSArray to C from ARC, re-casted as an equivalent CFArrayRef.

Here we take a Core Foundation object and move its ownership to ARC:

CFMutableArrayRef mutableCoreFoundationArray = CFArrayCreateMutableCopy(kCFAllocatorDefault, 0, coreFoundationArray);
CFArrayAppendValue(mutableCoreFoundationArray, CFSTR("C"));

// Bridge to Foundation
NSArray *updatedFoundationArray = CFBridgingRelease(mutableCoreFoundationArray);

From Kotlin Native

Why toll-free bridging is necessary in Kotlin Native

Kotlin Native supports bidirectional compatibility with Objective-C. That means that Kotlin Native can import and export Objective-C libraries natively.

Kotlin Native also has its own memory management model that is unlike C and Objective-C’s ARC. The language uses garbage collection to manage memory instead of reference counting.

According to KN’s developers, integration between Kotlin’s garbage collector is seamless with Objective-C’s ARC. That means once an object crosses into Objective-C land, and ARC would be aware of it, Kotlin is, too.

In contrast, when working with C code and Kotlin Native, manual memory management is still needed for C objects.

With that in mind, we can let Kotlin’s garbage collector handle our NS types and be vigilant about memory management when using Core Foundation types.

When we need to cross the Core Foundation to Foundation bridge, we don’t have to release memory manually. That’s because once crossed to the Objective-C side of things, ARC owns the object. Kotlin Native integrates with ARC and handles the memory for us.

When we need to cross the Foundation to Core Foundation bridge, neither ARC nor Kotlin Native’s garbage collector are managing our objects’ memory.

Toll-free bridging in Kotlin Native

While Kotlin Native supports Objective-C interoperability, it does not support simply casting CF types as NS types, and vice versa, in order to use them.

If you want to use a Foundation type with Core Foundation, or the other way around, you will need to manually cross that bridge.

Working with Core Foundation types

Core Foundation types derive from CFTypeRef, in both C and Kotlin, and in Kotlin, CFTypeRef is a type alias of COpaquePointer, the supertype for other pointer types, which maps to void* and is a type alias of CPointer<T>:

public typealias COpaquePointer = CPointer<out CPointed>
public typealias CFTypeRef = kotlinx.cinterop.COpaquePointer

Here’s what a CPointer<T> looks like:

public abstract class CPointed(rawPtr: NativePtr) : NativePointed(rawPtr.toNonNull())
public class CPointer<T : CPointed> internal constructor(@PublishedApi internal val value: NonNullNativePtr) : CValuesRef<T>() { /* */ }

CPointer is a feature of Kotlin Native’s C interop, it isn’t unique to the platform, Core Framework, or Objective-C. Kotlin uses it for pointers and arrays in C.

If we want to use Core Foundation types in Kotlin, especially to cast them to types that Kotlin integrates like Foundation types, we’ll need to manually bridge it into Kotlin with CFBridgingRelease().

Here’s a naive example:

import platform.CoreFoundation.*
import platform.CoreGraphics.*
import platform.Foundation.*


fun getWindows(): CFArrayRef? =
  CGWindowListCopyWindowInfo(kCGWindowListOptionAll, kCGNullWindowID)


fun main() {
  val cf = getWindows()
  println(cf)
  println(cf as? NSObject)
  // -> CPointer(raw=0x7f7e5ec04080)
  // -> CPointer(raw=0x7f7e5ec04080)

  val released = CFBridgingRelease(cf)
  println(released)
  println(released!!::class.qualifiedName)
  // -> [{kCGWindowLayer=0, kCGWindowMemoryUsage=186720, ...}]
  // -> kotlin.native.internal.NSMutableArrayAsKMutableList

  val asNsObject = released as? NSObject
  println(asNsObject!!::class.qualifiedName)
  // -> kotlin.native.internal.NSMutableArrayAsKMutableList

  val asNative = released as? MutableList<Map<String, Any>>
  println(asNative!!::class.qualifiedName)
  // -> kotlin.native.internal.NSMutableArrayAsKMutableList
}

Above, we call the Core Foundation function CGWindowListCopyWindowInfo() that returns a CFArrayRef?, which would map to a NSArray when using toll-free bridging from Objective-C. Using Objective-C, a simple cast is all we would need, unless we want to transfer ownership, but in Kotlin, we will need to use CFBridgingRelease() to do our casting and ownership transfer. Then, casting with as tells the compiler to treat the underlying data as a new type without changing the data itself. Casting released to collections successfully compiles and runs at runtime.

To use the released value in main() from Kotlin or Foundation APIs, we don’t have to do anything. Looking at its class name, we see that it is a kotlin.native.internal.NSMutableArrayAsKMutableList.

Here’s what that looks like in Kotlin:

public actual abstract class AbstractMutableList<E> protected actual constructor() : AbstractMutableCollection<E>(), MutableList<E> { /* */ }
internal class NSMutableArrayAsKMutableList : AbstractMutableList<Any?>(), RandomAccess, ObjCObjectWrapper { /* */ }

NSMutableArrayAsKMutableList is an AbstractMutableList from Kotlin’s standard library. That is the base class for Kotlin Native’s MutableList.

As it stands, NSMutableArrayAsKMutableList is equivalent to MutableList and casts to it both compile and succeed at runtime.

With that said, there is no reason you need to manually cast any Foundation type to a Kotlin type, as Kotlin will handle the interop for you. That means you can use Kotlin types in place of NS types without explicit casts:

val released: MutableList<Map<String, Any>> = CFBridgingRelease(cf)

We can now treat released like any type from Kotlin’s standard library thanks to the language’s built-in Objective-C support.

Memory management

Creating C objects uses native memory and not the Kotlin application’s heap memory. To automatically handle native memory, you can use memScoped blocks:

import kotlinx.cinterop.memScoped

fun main() = memScoped {
  val cf = getWindows()
  println(cf)
  /* */
}

memScoped blocks allocate memory via an arena and automatically free it when done.

Writing idiomatic extension functions

Let’s clean up our example into some extension functions that have better ergonomics.

fun <T> CFRefType.release(): T? = CFBridgingRelease(this) as? T
fun <T> CFRefType.cast(): T? = memScoped { release() as? T }

Now, we can do things like:

typealias Window = Map<String, Any>
typealias Windows = List<Window>

fun getWindows(): Windows? = memScoped {
  CGWindowListCopyWindowInfo(kCGWindowListOptionAll, kCGNullWindowID)
    ?.cast<Windows>()
    ?.map { window -> window.filterKeys { /* */ } }
    ?.filter { /* */ }
}

We can use the cast<T>() extension function on CFRefType to easily cast them to Kotlin types.

If we want to go in the other direction and cast a Kotlin type to a Core Foundation type, we can do this:

fun Any.retain(): CFTypeRef? = CFBridgingRetain(this)
fun <T: CFTypeRef> Any.cast(): T? = memScoped { retain() as? T }

fun asCoreFoundation(): CFRefType? = getWindows().cast()

Similarly, we can use the cast<T>() extension function on Kotlin objects to easily cast and bridge them to Core Foundation types.

Conclusion

While Kotlin Native, and its related platforms like Kotlin Multiplatform, abstract away a lot of underlying system details, if you want to target multiple platforms with Kotlin, you still need to understand what Kotlin abstracts over.

For macOS development, that means understanding how Kotlin Native interops with Objective-C and C, how Objective-C and C interact with each other, and how Core Foundation and Foundation kit interact with one another.

Sometimes it also means being aware of memory management outside of Kotlin’s garbage collector, and how you can allocate and free memory from C without leaking memory.

With that said, Kotlin Native makes it easy to also abstract over those implementation details with using features like extension functions, generics and manual memory management.

How to find Ubuntu repository mirrors

Wed, 23 Mar 2022 18:00:00 +0000

You can use find-ubuntu-mirrors to find alternative repository mirrors for Ubuntu. It will check mirror lists and test the mirrors to see if they’re working or not.

When it finds mirrors that are compatible with your preferences, it will output the mirrors in Apt’s sources.list format.

The script also allows you to find Ubuntu mirrors for different architectures like:

ARM (armhf & aarch64)
RISC-V (riscv64)
PPC (ppc64el)
IBM Z (s390x)

Click here to view the source code for find-mirrors.sh.

Usage

$ export URL="https://raw.githubusercontent.com/alexdelorenzo/find-ubuntu-mirrors/master/find-mirrors.sh"
$ curl -Ls "$URL" | bash

Passing options

The script takes the following positional arguments or environment variables:

Position	Variable name	Description	Default value
1	`$ARCH`	Architecture that the mirrors support	`amd64`
2	`$DISTRO`	Version of Ubuntu the mirrors support	`focal`
3	`$REPOSITORY`	The `apt` repository	`main`
4	`$PROTOCOL`	The `apt` repository protocol	`http`
5	`$JOBS`	Number of concurrent network connections	`4`

Here’s the syntax:

$ curl -Ls "$URL" | bash -s $ARCH $DISTRO $REPOSITORY $PROTOCOL $JOBS

If you want to find alternatives for aarch64 architectures while checking 6 repositories concurrently, you can do this:

$ curl -Ls "$URL" | bash -s aarch64 focal main http 6

You can also use environment variables instead:

$ export ARCH=aarch64 DISTRO=focal JOBS=6
$ curl -Ls "$URL" | bash

Output

The script will output the mirrors it finds in Apt’s sources.list format:

$ curl -Ls "$URL" | bash
deb http://mirror.kumi.systems/ubuntu-ports/ focal main
deb-src http://mirror.kumi.systems/ubuntu-ports/ focal main

deb http://mirrors.portafixe.com/ubuntu/archive/ focal main
deb-src http://mirrors.portafixe.com/ubuntu/archive/ focal main
...

Requirements

Script

You can check out find-ubuntu-mirrors on GitHub here.

Click here to see the source for find-ubuntu-mirrors below

How to install npm packages as a non-root user

Sat, 26 Feb 2022 18:00:00 +0000

After running this script, you will be able run npm install -g without sudo, becausenpm will install packages as your non-root user. The script will set up npm user directories for you, and then add them to your $PATH.

Packages can then be installed without root privileges:

$ npm install -g yarn
$ type yarn
yarn is /home/user/.npm-packages/bin/yarn

Usage

Run the following:

$ curl -s "https://raw.githubusercontent.com/alexdelorenzo/npm-user/main/npm-user.sh" | bash

You can check out the script’s requirements here.

Options

The script takes 6 optional positional arguments, or you can set environment variables:

Position	Variable name	Description	Default value
1	`$ROOT`	The root directory for `npm`	`$HOME`
2	`$SHELL_NAME`	Name of the shell to configure	Current shell
3	`$SHELL_RC`	Shell configuration file, automatically detected	`$HOME/.profile`
4	`$BIN`	`npm`’s executable directory	`$ROOT/.npm-packages/bin`
5	`$MAN`	Manpage directory for `npm`	`$ROOT/.npm-packages/share/man`
6	`$REINSTALL`	Set to any non-null value to reinstall old `npm` packages in your new `$ROOT`	Unset

Passing options

If you want to set your npm path to ~/.local/.npm-packages, instead of ~/.npm-packages, and configure zsh to work with it, you can run:

$ export URL="https://raw.githubusercontent.com/alexdelorenzo/npm-user/main/npm-user.sh"
$ curl -s "$URL" | bash -s "~/.local" "zsh"

You can also use environment variables:

$ export ROOT="~/.local" SHELL_NAME="zsh"
$ curl -s "$URL" | bash

Confirming it works

Install a package with the global flag -g and then see where npm puts the files:

$ npm install -g yarn
$ type yarn
yarn is /home/user/.npm-packages/bin/yarn

Requirements

Click here to see the dependencies and supported systems for this script

Dependencies

Bash, GNU Coreutils and grep
curl
NPM

Supported systems

Shells

This script works with configuration files for the following shells:

bash
zsh
fish
sh

The script will default to $HOME/.profile if it doesn’t recognize any of the shells from above. ksh uses $HOME/.profile, but the script wasn’t tested on it.

Operating systems

This script will work on any POSIX compatible system or compatibility layer.

Linux
*BSD
macOS
WSL 1 & 2 on Windows
Cygwin or MSYS2 on Windows

Script

You can check out the script on GitHub via the 🏠 npm-user project.

Click here to show 🏠 npm-user source code below

How to get the first item of a Python iterable

Thu, 13 Jan 2022 18:00:00 +0000

Here’s a succinct definition of a first() function for Python:

# © 2025 Alex DeLorenzo. Licensed under the LGPLv3.
from typing import Iterable, TypeVar


T = TypeVar('T')
U = TypeVar('U')

Item = T | U | None


def first(iterable: Iterable[T], default: Item = None) -> Item:
  iterator = iter(iterable)
  return next(iterator, default)

Scroll to the bottom or click here to see an async implementation of this first() function.

Using the function

Here’s a look at how to use first() with various data structures that are common in Python.

Iterables

gen_nums: Iterable[int] = range(10)
first_num: int = next(gen_nums)  # 0

assert first(gen_nums) == first_num

Ordered sequences

nums: list[int] = list(gen_nums)

assert first(nums) == first_num

Empty sequences

empty: tuple = ()

assert first(empty) is None
assert first(range(0)) is None

Default value

The default value can be any object, not just None.

class Missing: pass

assert first(empty, Missing) is Missing

Generator expressions

assert first(num for num in gen_nums) == first_num

Strings and bytes

assert first('example') == 'e'
assert first(b'example') == b'e'

Arrays

from array import array

nums: Iterable[int] = array('i', gen_nums)

assert first(nums) == first_num

Files

from pathlib import Path

nums = map(str, gen_nums)
text = '\n'.join(nums)

path = Path('example.txt')
path.write_text(text)

with path.open() as file:
  assert first(file) == f'{first_num}\n'

Custom iterators

from typing import Iterator
from dataclasses import dataclass

@dataclass
class Increment(Iterator[int]):
  val: int = gen_nums.start

  def __next__(self) -> int:
    self.val += gen_nums.step
    return self.val

nums: Iterable[int] = Increment()

assert first(nums) == first_num

Unordered collections

Doesn’t make much sense with unordered collections, but it works. It could be used as a way to get a member of a set without raising a KeyError if it’s empty, or to supply a default value.

nums: set[int] = set(gen_nums)

assert first(nums) is not None

An async first function

from typing import AsyncIterable


async def first(iterable: AsyncIterable[T], default: Item = None) -> Item:
  aiterator = aiter(iterable)
  return await anext(aiterator, default)

Usage

To run these examples with top-level await expressions, you can use an async REPL by running python -m asyncio, or by using IPython.

Files

Here’s an example using 📁 aiopath, an async pathlib replacement for Python.

from aiopath import AsyncPath

nums = map(str, gen_nums)
text = '\n'.join(nums)

path = AsyncPath('example.txt')
await path.write_text(text)

async with path.open() as file:
  assert await first(file) == f'{first_num}\n'

Async iterables

async def agen_nums() -> AsyncIterable[int]:
  for num in gen_nums:
    yield num

nums = agen_nums()

assert await first(nums) == first_num

Async generator expressions

nums: AsyncIterable[int] = agen_nums()

assert await first(num async for num in nums) == first_num

Custom async iterators

from typing import AsyncIterator
from dataclasses import dataclass

@dataclass
class Increment(AsyncIterator[int]):
  val: int = gen_nums.start

  async def __anext__(self) -> int:
    self.val += gen_nums.step
    return self.val

nums: AsyncIterable[int] = Increment()

assert await first(nums) == first_num

How to clean up Python build, caching and packaging artifacts

Sat, 03 Jul 2021 18:00:00 +0000

About

Python leaves compiled bytecode in project directories under subdirectories named __pycache__. Building packages also creates artifacts in project directories.

This simple script will delete those build, caching and packaging artifacts for you.

Script

The following depends on aiopath, which can be found here or installed via PyPI.

Click here to see the source code below

How to list Click CLI commands in the order they're defined

Thu, 27 May 2021 18:00:00 +0000

click is a Python module used to create command-line interfaces (CLI) for scripts and applications.

In the sections below, we’ll look at a method for making sure that click CLI commands are listed in the order they were defined when we invoke the CLI or --help flag.

You can skip ahead to the explanation for how our solution works here.

Building a CLI with Click

Let’s look at a click CLI with a couple of commands:

#!/usr/bin/env python3
import click


@click.group()
def cli():
  pass


@cli.command(help='This should be first.')
def xyz_first():
  pass


@cli.command(help='This should be second.')
def abc_second():
  pass


cli()

If we run this script, we’ll see that the commands xyz-first and abc-second aren’t displayed in the order that they were declared:

$ ./cli.py
Usage: cli.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  abc-second  This should be second.
  xyz-first   This should be first.

Ordering commands

If we want to order those commands in the order that we define their functions, we can subclass click.Group:

import click


class OrderCommands(click.Group):
  def list_commands(self, ctx: click.Context) -> list[str]:
    return list(self.commands)

Then, when declaring our commands, we pass our new class as a cls parameter to click.group():

@click.group(cls=OrderCommands)
def cli():
  pass


@cli.command(help='This should be first.')
def xyz_first():
  pass


@cli.command(help='This should be second.')
def abc_second():
  pass

Finally, we’ll see that our commands are listed in the order they were defined:

$ ./cli.py
Usage: cli.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  xyz-first   This should be first.
  abc-second  This should be second.

How it works

A click.Group object has an attribute commands of type dict[str, click.Command]. Normally, Group.list_commands() just returns a sorted() list of keys from the commands attribute.

Here’s Group’s list_commands() method:

def list_commands(self, ctx: click.Context) -> list[str]:
  return sorted(self.commands)

As an implementation detail of PEP 468, dict’s keys are sorted in the order that they were created. The keys act similar to how a list orders its items, instead of being unordered like a set. This is the default behavior in Python 3.6 and up.

That means we can make Group.list_commands() just return the keys to the Group.commands attribute in order to preserve the order that our click commands were defined:

def list_commands(self, ctx: click.Context) -> list[str]:
  return list(self.commands)

How to use Intel GVT-g to create virtual graphics cards

Tue, 19 Jan 2021 18:00:00 +0000

About

If you use KVM virtualization on Linux, you might be familiar with PCI passthrough. Using an IOMMU, you can make your hardware available to guest operating systems for near native performance.

Intel’s solution to make their integrated graphics cards available to virtualized guests is Intel GVT-g. You can use it to virtualize GPUs to hand off to your virtual machines for enhanced performance compared to emulated graphics hardware.

This script will let you discover the types of virtualized GPUs your hardware supports, and will let you add and remove the virtualized devices.

Script

You can check out the script on GitHub.

Click here to see the source code below

How to install Firefox with the GTK Global Menu patch on Ubuntu

Sat, 16 Jan 2021 18:00:00 +0000

About

On Ubuntu 18.04 and earlier, Firefox is compiled with the GTK Global Menu patch.

On newer Ubuntu releases, Firefox isn’t compiled with the patch. You need to hit the Alt key on your keyboard to show the application menu.

I use Plasma Desktop with the Window AppMenu Applet to simulate the global menu from macOS. It works with both Qt and GTK applications, but Firefox doesn’t take advantage of it.

This script will download the latest version of Firefox from Ubuntu 18.04’s repositories, install it and hold its version to prevent it from being updated to a version from a newer Ubuntu release.

Installation Script

Dependencies

You need to have Bash, curl, wget, byobu and pup installed.

To install the first four, you can use apt:

$ sudo apt install bash curl wget byobu

pup is packaged for Debian Sid. You can use dpkg to install it.

Script

You can check out the script on GitHub.

How to get unique CFLAGS for your hardware

Sat, 16 Jan 2021 18:00:00 +0000

About

This script will emit CFLAGS that enable features that are available on your hardware, but are absent when you set GCC’s architecture flag.

For example, if you have a Skylake processor, your individual processor might have features enabled on it that are not shared across all Skylake processors. This script will list those differences.

Script

The script below requires the 🧵strs package in order to use the str command. Click here for installation instructions.

How to export an OPML file from Airsonic

Sat, 16 Jan 2021 18:00:00 +0000

About

If you want to export your podcast subscriptions out of Airsonic, you’ll probably want to do so using an OPML file. It’s simply an XML document listing your RSS feed subscriptions that you can import into any RSS feed reader, or podcast client.

Unfortunately, Airsonic doesn’t have that feature.

However, Airsonic keeps a log of database insertions in a file called airsonic.script. You can parse it to generate an OPML file.

Script

This script works on Linux. If you’re on macOS or *BSD, use GNU coreutils along with GNU grep.

Click here to see the source code below

Script using `🧵strs`

The follow script uses the 🧵strs package for Python.

Click here to see the source code below

Using Unix Pipes to Improve Chromecast Playback

Sat, 14 Mar 2020 18:00:00 +0000

Update: Looks like this made it to the top of Hacker News.

The Issue

When casting from YouTube to a Chromecast, sometimes the audio playback will skip and stutter. This issue is independent of the quality of the video, the FPS, or the internet connection. Trying to watch certain videos will reliably cause playback issues on the Chromecast.

According to this thread, the playback issue only appeared in the latest and final firmware update for the first-gen Chromecast. Successful playback of the videos was possible in the past, and casting the downloaded YouTube videos eliminates the problem entirely. The problem exists solely when casting certain videos using the YouTube app.

A Solution

Given that the problem only occurs with the YouTube app, you can download a video and cast it via castnow or catt, skipping the YouTube app entirely.

You could do something like:

#!/usr/bin/env bash
export URL="https://youtu.be/Kas0tIxDvrg"


cast() {
  local url="$1"
  local filename="$(youtube-dl --get-filename "$url")"

  # download
  youtube-dl "$url"
  # wait a bit to finish download

  # cast the video then delete it
  castnow "$filename"
  rm "$filename"
}


cast "$URL"

But having to skip using the YouTube app to cast is already a clunky solution, and downloading every video before playing them is an even worse user experience. Instant playback and ephemeral streams are what make for a pleasant video streaming experience these days, and this solution implements neither of them.

A Slightly Better Solution

Since youtube-dl allows us to output to stdout, if we can hook its stdout to a casting app, we could emulate the instant playback and ephemeral videos we expect because we don’t have to wait for an entire file to download.

Unfortunately, castnow and catt won’t cast from stdin. You’re expected to pass it file locations to cast from.

This is where one of my favorite shell features really shines: process substitution.

With process substitution, Bash gives us a convenient way to make ephemeral anonymous pipes. This method is both efficient and concurrent, making this primitive an apt choice to build a solution to the problem at hand. A process reading the pipe blocks until the pipe is opened by another process for writing. A process writing to the pipe will suspend until the pipe’s buffer is read by another process. The anonymous pipe will automatically remove itself, and when it is manually removed, its dependent processes will be terminated.

When using process substitution, a process’ stdout is hooked up to an anonymous pipe. That pipe can be accessed from a file descriptor, and the location of the pipe is given to the calling process.

$ echo <(echo "Piped data")
/dev/fd/63

$ cat <(echo "Piped data")
Piped data

In the example above, the <(command) syntax is how we invoke process substitution in Bash. The output of command is written to an anonymous pipe, and the calling process is given the location of the file descriptor to access that pipe.

Using that example, we can take advantage of process substitution:

vlc <(youtube-dl -q -o - "$URL")

The command above will play the YouTube video locally with VLC, and illustrates that process substitution can work for our use case.

However, when we try to use castnow, we can’t cast from the pipe:

$ castnow <(youtube-dl -q -o - "$URL")
Error: Load failed

Nor can we cast with catt:

$ catt cast <(youtube-dl -q -o - "$URL")
Error: The chosen file does not exist.

We know we can use VLC locally, and VLC also lets you cast to a Chromecast using its IP address.

Let’s try that again using VLC:

cast-vlc() {
  local path="$1"

  # get the ip address for chromecast.lan host
  local ip="$(dig +short chromecast.lan | tail -n 1)"

  vlc -I ncurses \
    --sout '#chromecast' \
    --sout-chromecast-ip="$ip" \
    --demux-filter=demux_chromecast \
    "$path" < /dev/tty
}


cast-vlc <(youtube-dl -q -o - "$URL")

That works.

As an aside, we hook VLC’s stdin to /dev/tty so that we can use the ncurses interface even if we invoke the function from a script.

Let’s look at the ncurses interface.

It only displays the file descriptor, and very little about the video itself. I’m not a fan of that.

A Better Solution

Instead of using anonymous pipes, we can use named pipes. Named pipes are like anonymous pipes, except they are not anonymous (they have a name) nor are they ephemeral. Named pipes still give us the efficiency and concurrency benefits that anonymous pipes give us, but Bash lacks the syntactic sugar it has for process substitution when it comes to named pipes.

This is how we create named pipes, write to them, read from them and remove them.

$ mkfifo our-pipe
$ echo "Piped data" > our-pipe &
$ cat our-pipe
Piped data
$ rm our-pipe

Not as pretty as <(command), but it gets the job done.

We can give a named pipe the same name as our YouTube video, and that way, the VLC interface will show the name of what we’re watching.

cast-ytdl() {
  local url="$1"

  # create a temporary named pipe
  # why? because vlc will show the file descriptor path if we just use process substitution
  local filename="$(youtube-dl --get-filename "$url")"
  local path="/tmp/$filename"
  mkfifo "$path"

  # download in background, push to named pipe
  youtube-dl -q -o - "$url" > "$path" &
  local pid="$!"
  disown "$pid"

  # cast from named pipe
  cast-vlc "$path"

  # cleanup process and named pipe
  kill -9 "$pid" &> /dev/null
}


cast-ytdl "$URL"

This works, too.

We need to manually create a named pipe with mkfifo, redirect youtube-dl’s stdout to the named pipe while running the process in the background, and then cleanup the process after casting from it via VLC, otherwise it might linger in the background. Each of those tasks would have been handled for us automatically using process substitution.

But it does look a bit better:

Caveats

Pipes, anonymous pipes and named pipes are also known by another name because of the way they behave: FIFOs, or first in, first out. What’s written to the pipe is read from the pipe in first in, first out order. This behavior maps well to video streaming.

While you can interact with anonymous and named pipes like you would a file, the interface isn’t 1:1 with a standard file. You cannot seek() forward or backward in pipe, you can just read the next forward values. For our use case, that means we cannot skip forward or backward in our streaming videos. We can only play, pause or stop the video.

That’s not a problem for me, however it’s something that can be mitigated if it’s a problem for you. The first solution I can think of would be to write to a temporary file via youtube-dl and read from it. Or perhaps a temporary spooled file can act as a buffer for the pipe, such that you can seek() through the buffer, but the buffer itself is ephemeral unlike a normal file.

NFS Authentication and Encryption via WireGuard

Tue, 28 Jan 2020 18:00:00 +0000

About NFS and WireGuard

NFS is a network file-system that’s included in the mainline Linux kernel. It’s everywhere that a complete Linux kernel exists. There are Windows and macOS clients for the protocol, and it can be used with Kubernetes and Docker Swarm. Out of the box, the protocol is not encrypted nor does it provide authentication.

WireGuard is a VPN protocol that’s in the process of being included in the mainline Linux kernel. Despite not being in the current mainline kernel, it’s easy to install on pretty much any Linux platform. It has cross-platform implementations, and there are WireGuard clients for Android, OpenWRT, macOS and Windows.

Update: within a few hours of writing this, WireGuard was slated for release in Linux 5.6. Pretty good timing.

Why tunnel NFS over a VPN?

As mentioned earlier, the NFS protocol is not encrypted. If you’re on an untrusted network, an attacker can easily sniff your NFS traffic. Even if you’re on a trusted network, an untrusted device on your network can sniff your traffic. On WiFi, an attacker doesn’t even need to connect to your network, because WEP, WPA, and WPA2 can be cracked. Enterprise-grade wireless encryption and isolation fare better, but why risk it?

The typical method of providing NFS authentication is via Kerberos. If you don’t already use Kerberos in your network infrastructure, there is an operational and cognitive overhead for its provision and maintenance. In my opinion, the less moving parts, the better.

NFS allows us to restrict access of our network shares to specific IP addresses, subnets and hostnames, giving us some privacy control. However, this is not entirely secure.

If we set up a VPN, we can have our NFS server only listen on that network, and not on the wider unencrypted network. Our VPN server can handle authentication and give our authenticated clients hardcoded IP addresses. In our NFS exports, we can restrict which IP addresses can access certain NFS resources, and have a fine-grained policy for both authenticated clients and our shared resources.

Why WireGuard?

WireGuard is a simple protocol built upon cryptographic primitives that are modern and secure. Its concise implementation gives it a smaller attack surface than other solutions. The protocol and implementation itself have been formally verified, and it has the approval of security experts throughout the industry.

While OpenVPN is widely supported across platforms, it is a behemoth in comparison, which means that it has a wider attack surface. It’s relatively complicated to provision and maintain. As opposed to the in-kernel WireGuard protocol, there is at least one extra context switch between kernel networking and the userspace OpenVPN software. Not only that, but AES is resource intensive on platforms without hardware accelerated AES support, whereas ChaCha20-Poly1305 is performant on low-powered devices without hardware acceleration. Since I use many low-powered Linux devices, that context switch and lack of hardware acceleration for AES on OpenVPN has a measurable impact on the speed of my VPN traffic.

Authentication Model

Our authentication will be based on the public key authentication WireGuard provides, and restricting clients to specific IP addresses on our WireGuard network. Authentication is based on our users’ knowledge of a secret, in this case the private key from the key pair our WireGuard server knows about. This is otherwise known as cryptokey routing, and is one of the principles WireGuard is built upon.

WireGuard clients will request a specific IP address from our WireGuard server. Our server will only provision a unique but unchanging IP to the client with the corresponding public key. That way, we can know that traffic from that address is definitely from our authenticated client. We can then use that IP address in our NFS exports to restrict our shares and privileges to only that user.

We can think of those client IP addresses as being our users, and their private keys as passwords. From there, we can configure our NFS server using the basic IP address restrictions the kernel server provides.

NFS over WireGuard

First, we need to choose a subnet for our VPN. I’m going to use 10.8.0.0/24 for our subnet, and 10.8.0.1 as the address for our WireGuard server.

For this article, our NFS server and WireGuard host will have the hostname host.

Setting up NFS

Server Installation

If you’re on a Debian-based Linux distribution, you can install an NFS server like so:

sudo apt install nfs-kernel-server

Server Configuration

We need to set up our NFS server to listen only on 10.8.0.1. We’re also going to configure the server to use only NFSv4, versus older versions of the protocol.

On a Debian-based distribution, our configuration file lives on /etc/default/nfs-kernel-server. We’re going to edit the RPCNFSDOPTS and RPCMOUNTDOPTS variables in that file.

To force NFSv4, we need to disable NFSv2 and NFSv3. NFSv4 has less moving parts and operates only on a single port. Older versions of NFS use many ports and require ancillary services to run alongside the NFS server.

Add the following to the configuration file:

RPCNFSDOPTS="-N 2 -N 3"

Next, we’re going to instruct the server to listen on 10.8.0.1 and to further disable older versions of the NFS protocol:

RPCMOUNTDOPTS="--manage-gids -N 2 -N 3 -H 10.8.0.1"

To restrict the NFS server to listen only on the specified host, the /etc/nfs.conf file must be created or modified with the following configuration section.

[nfsd]
host=10.8.0.1

Make sure you save the configuration files.

Adding Network Shares

Let’s create the directory structures and bind mounts needed for exportfs.

sudo mkdir -p /export/example
sudo touch /export/example/it_worked

Now we need to edit /etc/exports to allow a client to mount our example share:

/export                  10.8.0.0/24(fsid=0,no_subtree_check)
/export/example          10.8.0.0/24(rw,sync,no_subtree_check,crossmnt,no_root_squash)

The first line and fsid option sets the root for our shares. With fsid and crossmnt, we can exclude the /export prefix on our client at mount time, and just mount /export/example as /example.

The second line will allow any client on the 10.8.0.0/24 subnet to mount /export/example as readable and writable. The sync option makes writes synchronous, while async makes them asynchronous. I find that synchronous writes are easier to reason about.

no_root_squash will prevent files that belong to root from being exported with ownership to the nobody user. I want my network shares to reflect their source ownership as accurately as possible, so I enable this option.

Let’s export our shares:

sudo exportfs -a

Starting the Server

First, let’s disable rpcbind because it isn’t required for NFSv4:

sudo systemctl mask rpcbind.service
sudo systemctl mask rpcbind.socket

Then, start the server, or restart it if it’s already running:

sudo systemctl restart nfs-kernel-server.service

Great, now let’s test it out on a client machine. Ironically, if everything works as configured, mounting the share should fail, but with specific errors.

Configuring the NFS Client

Connect a separate Linux machine to your network, such that it can reach your host via it’s hostname host.

Client Installation

Again, assuming you’re using a Debian-based platform, install the NFS client software like so:

sudo apt install nfs-common

Testing NFS Security

We’re going to try to mount our network share, and hopefully it will fail. Replace the hostname host in the following commands with the hostname of the machine running your NFS server.

sudo mkdir /mnt/NFS
sudo mount -t nfs -o vers=4.2 host:/example /mnt/NFS

Hopefully, you will be met with either of the following errors:

mount.nfs: access denied by server while mounting host:/example

Or, this error:

mount.nfs: Operation not permitted

Setting up WireGuard

Installing WireGuard

Head over to the official WireGuard website and choose your installation option.

On my platform, no configuration is needed and the following will install the needed packages:

sudo apt install wireguard

After the software is installed, insert the wireguard module into the kernel:

sudo modprobe wireguard

In recent kernels, WireGuard ships with Linux by default and will automatically load the kernel module when necessary.

Generating Key Pairs

Our server will need to generate a key pair, and each of our clients will need to generate their own unique key pairs.

On both our server and client issue the following:

umask 077 
wg genkey > privatekey
wg pubkey < privatekey > publickey

Remember to treat your private keys like passwords, and to delete the privatekey files when you are done with them.

Generate a Pre-shared Key (PSK)

For added security, generate a PSK on the client:

wg genpsk > psk

You will need to generate a new PSK for each of your clients and add them both on the client configuration and on your server. That means every client will have unique pre-shared keys, like they have unique key pairs.

Treat psk like a password, as well, and delete it when you’re done.

Configuring the WireGuard Server

On the server, identify the network device you’re using to connect to your network. I am using eth0. Replace eth0 with your network device, or use %i.

You will need to choose a port to listen on. In this case, we’ll use 51820.

Copy the contents of our server’s privatekey to your clipboard. Then, create a file located at /etc/wireguard/wgNFS.conf. Keep the contents of your first client’s publickey and psk nearby, as we will use it in that file.

[Interface]
Address = 10.8.0.1/24
ListenPort = 51820
PrivateKey = HOST_PRIVATE_KEY_GOES_HERE

# client1
[Peer]
PublicKey = CLIENT_PUBLIC_KEY_GOES_HERE
PresharedKey = PRESHARED_KEY_GOES_HERE
AllowedIPs = 10.8.0.2/32

Remember to add your client’s pre-shared key under its [Peer] column.

Configuring the WireGuard Client

For your client, you will need your host’s publickey and your client’s privatekey and psk.

Create a file located at /etc/wireguard/wgNFS.conf:

[Interface]
Address = 10.8.0.2/24
PrivateKey = CLIENT_PRIVATE_KEY_GOES_HERE

[Peer]
PublicKey = HOST_PUBLIC_KEY_GOES_HERE
PresharedKey = PRESHARED_KEY_GOES_HERE
Endpoint = host:51820
AllowedIPs = 10.8.0.0/24

The last line tells the WireGuard client to route traffic on the 10.8.0.0/24 subnet through the WireGuard server endpoint.

Delete `privatekey` and `psk` Files

On your host and clients delete your privatekey files after configuring WireGuard, and delete your psk files, as well.

rm privatekey psk

Starting and Connecting to the VPN

On both the host and client, but starting with the host, issue:

sudo wg-quick up wgNFS

Test that the connection works by pinging the host from the client:

$ ping -c 1 10.8.0.1
PING 10.8.0.1 (10.8.0.1) 56(84) bytes of data.
64 bytes from 10.8.0.1: icmp_seq=1 ttl=64 time=0.89 ms

--- 10.8.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.893/0.893/0.893/0.000 ms

On the client machine, run the following:

sudo mount -t nfs -o vers=4.2 10.8.0.1:/example /mnt/NFS

Test to see if the share mounted.

$ ls /mnt/NFS
it_worked

You should be able to access the it_worked example file we created earlier.

Making it Permanent

WireGuard via systemd

On both the server and client, add a WireGuard service to systemd. This service will start at boot.

sudo systemctl enable wg-quick@wgNFS.service
sudo systemctl daemon-reload

Tear down the existing wgNFS connection on both machines and then create the VPN connection via the new systemd service.

# stop WireGuard
sudo wg-quick down wgNFS

# start WireGuard via systemd
sudo systemctl start wg-quick@wgNFS.service

Unmount the example share on the client.

sudo umount -f /mnt/NFS

Add an entry to /etc/fstab on the client with the parameters for the shares.

10.8.0.1:/example   /mnt/NFS    nfs vers=4.2,_netdev,noauto,x-systemd.automount,x-systemd.requires=wg-quick@wgNFS.service

Reload systemd and remote-fs.target.

sudo systemctl daemon-reload 
sudo systemctl restart remote-fs.target

Now, whenever the mountpoint is accessed, systemd will automatically mount the NFS share to the mountpoint specified in /etc/fstab.

$ ls /mnt/NFS
it_worked

Migrate Your Google Music Collection with MusicFS

Sat, 05 Oct 2019 18:00:00 +0000

Mount your Google Music collection as a drive with MusicFS.

As Google Music is being shutdown, this utility will allow those who uploaded their music library to Google Music to back up the songs they uploaded.

Features

Browse your music collection via artists, albums and songs
Play and index your collection with your favorite media player
Download and backup your entire collection with ease

Download

You can sign up for the open MusicFS open beta here!

Listen to Music from Bandcamp with CampFS

Tue, 01 Oct 2019 18:00:00 +0000

CampFS is an app that lets you stream music and browse Bandcamp as files on your computer.

Features

With CampFS, you can browse and discover music easily.

Search for artists, albums, labels and songs
Browse music via tags
Find similar artists and albums
Stream music with the music player of your choice
Access high quality albums

Download

You can sign up for the open CampFS open beta here!

About CampFS

Over the last decade, it became harder to find new music that I like. I grew up with Yahoo’s music recommendation system, Last.fm and Pandora. As a result, I developed esoteric (read: bad) tastes and I took music discovery as given. Spotify and streaming platforms don’t scratch the itch that Last.fm and Myspace did 15 years ago.

Enter Soundcloud and Bandcamp, two platforms that let anyone easily distribute music and get paid for it. I love them. Acts in my favorite subgenres tend to adhere to a DIY ethic, and Bandcamp enables that through a generous compensation program for labels and artists. Where artists might only make a few cents with a thousand Spotify plays, they can recoup 80-85% of purchase price of content they sell on Bandcamp. If we’re honest, that’s a better deal than the 30% cut Apple and Google take via their app stores.

To ease and automate music discovery for myself, I built a utility that connects with various music streaming platforms like Google Music, Soundcloud and Bandcamp.

In the interest of supporting artists, this app does not enable piracy. Only content that labels and artists list as free to download are accessible. Links to buy content and support artists are plentiful throughout the browsing experience.

After talking with other music lovers in my favorite scenes, it seems that Bandcamp became the Myspace du jour for underground music in 2019. I chose to release this app so that other people can discover music and support artists with the same ease that I can.

A Stream Buffer Library for Python and Rust

Sun, 14 Apr 2019 18:00:00 +0000

Background

While building out a project, I found the need to process and save streaming resources locally. Users would stream media via an application, and they would need to seek backwards and forwards in the stream.

Often, the streaming resource size would be 2-5MB in total. But there were outlier resources that sometimes totaled several hundred megabytes.

Sometimes, the user would want to save the entire resource to the disk, but most of the time they’ll just stream the resource without saving it.

The Problem

I want a simple buffer-like library that will both suit the user’s needs and not distract from the larger problem. It needs to be thread-safe.

Looking for a Solution

The app was written in Python, so let’s take advantage of its rich standard library. After we build the solution, I will implement it in Rust to better test my assumptions about the code and problem-space itself.

tempfile

Python has a library for platform-independent creation of temporary files and directories, called tempfile. It takes care of creating and deleting ephemeral files and folders so you don’t create a mess for yourself or your users.

TemporaryFile conforms to Python’s file-like interface, so that fits the bill for a buffer-like library.

However, writing a relatively small buffer to disk upon streaming each resource doesn’t make sense to me.

SpooledTemporaryFile

Where a TemporaryFile initially writes to disk when its buffer is written to, a SpooledTemporaryFile will first write to memory. When it reaches max_size bytes, it will then begin writing its buffer to the disk in FIFO order. You can also arbitrarily purge the memory buffer to disk by invoking a method.

This fits the bill a bit better.

Building a Solution

Let’s build a library that solves this problem. We start with a stream, we know its size in advance, and we know we want to use a SpooledTemporaryFile as our buffer.

import logging
import tempfile
from typing import Iterable
from itertools import chain

from wrapt.decorators import synchronized


MAX_SIZE: int = 5 * 1_024 ** 2  # bytes


class StreamBuffer:
  def __init__(
    self, 
    stream: Iterable[bytes],
    size: int,
    max_size: int = MAX_SIZE
  ):
    self.stream = stream
    self.size = size
    self.stream_index = 0
    self.temp = tempfile.SpooledTemporaryFile(max_size=max_size)
    ...

I chose to make our MAX_SIZE 5MB because the streams, on average, do not exceed 5MB. We will use a combination of LRU and TTL cache polices to limit and purge the stream buffers our application creates, but that is out of scope for the class we’re building.

I want to be able to take chunks out of the buffer at arbitrary sizes and positions, so I decided to use the __getitem__ interface with slice objects, similar to that used in list and bytes.

Why? Because the app talks to a server that asks for chunks of bytes at arbitrary lengths.

We can implement that functionality like so:

  def __getitem__(self, val) -> bytes:
    if isinstance(val, slice):
      size = val.stop - val.start
      return self.read(val.start, size)

    raise NotImplementedError(f'Not implemented for {type(val)}')
    ...

This will let us do things like:

stream_buffer: StreamBuffer  # snip
chunk: bytes = stream_buffer[1024:2048]

We’ll need to implement a read() function to do that, though.

Reading

We can’t seek in the stream itself, so it’s best to think of the stream as a generator that only produces the next forward value of the streaming resource.

This is our function signature:

 def read(self, offset: int, size: int) -> bytes:
 ...

That interface conforms almost exactly to the server requests our application has to handle, which give an offset and size and requires a byte-encoded response.

Before we implement read(), we need to break the cases the function will encounter.

Cases as a State Machine

We can create an Enum to enumerate the states our function will handle.

from enum import Enum, auto


class Location(Enum):
  BeforeIndex = auto()
  Bisected = auto()
  AtIndex = auto()
  AfterIndex = auto()

We can also define interfaces that our StreamBuffer will implement that can break down the problem we’re trying to solve into testable sub-problems.

from abc import ABC


class ChunkLocation(ABC):
  def _chunk_location(self, offset: int, size: int) -> Location:
    pass


class ChunkRead(ABC):
  def _chunk_before_index(self, offset: int, size: int) -> bytes:
    pass

  def _chunk_bisected_by_index(self, offset: int, size: int) -> bytes:
    pass

  def _chunk_at_index(self, size: int) -> bytes:
    pass

  def _chunk_after_index(self, offset: int, size: int) -> bytes:
    pass

Next, we’ll dive into those four cases and some other details.

1. The chunk our client requests is already in the StreamBuffer

We can read directly from our SpooledTemporaryFile buffer and return from it.

  def _chunk_before_index(self, offset: int, size: int) -> bytes:
    self.temp.seek(offset)
    return self.temp.read(size)
    ...

2. The chunk starts where we last left off

Next, we’re going to use a bytearray to store our return buffer.

    buf = bytearray()
    ...

Like strings, bytes objects are immutable. If we used bytes objects to build our return buffer, each time we’d want to extend the return buffer, we’d run into the problem of the interpreter having to concatenate two bytes objects to create a third bytes object that contains the contents of the first two objects.

If the bytes objects are small and we do that once or twice, that’s no big deal. However, we have to return arbitrarily many bytes, and the size of the chunks returned by the stream are predetermined and small. In some instances, we could very well be creating tens of thousands of intermediate objects that are created just to be thrown away, which is both slow and inefficient.

Instead, we can copy bytes objects into a list-like data structure, bytearray. It conforms to most of the bytes-like interface, as well, so there isn’t a feature mismatch between it and the bytes objects we’re dealing with.

Then, once our return buffer is filled, we can join the bytearray into a bytes object and return it. That’s the last thing we’ll do though.

We can grab chunks from our stream, and push them to our SpooledTemporaryFile buffer and copy it to our return buffer. If the return buffer’s size is the length of the slice or greater, we return the return buffer truncated to the size of the requested slice.

  def _chunk_at_index(self, size: int) -> bytes:
    buf = bytearray()
    self.temp.seek(self.stream_index)

    for line in self.stream:
      self.stream_index += len(line)
      self.temp.write(line)
      buf.extend(line)

      if len(buf) >= size:
        return bytes(buf[:size])

    return bytes(buf)
    ...

3. Part of the requested chunk was already loaded, and part of it wasn’t

In this case, we can just use our solutions for _chunk_before_index() and _chunk_at_index() and return the concatenated result.

  def _chunk_bisected_by_index(self, offset: int, size: int) -> bytes:
    existing_size = self.stream_index - offset
    chunk_before = self._chunk_before_index(offset, existing_size)

    new_size = size - len(chunk_before)
    chunk_after = self._chunk_at_index(new_size)

    chunks = chain(chunk_before, chunk_after)

    return bytes(chunks)
    ...

4. None of the requested chunk was loaded at all

In this case, we’ll need start recording the stream to our SpooledTemporaryFile exactly where we last left off, until we reach the beginning of our requested slice.

  def _chunk_after_index(self, offset: int, size: int) -> bytes:
    buf = bytearray()
    self.temp.seek(self.stream_index)

    for line in self.stream:
      self.stream_index += len(line)
      self.temp.write(line)
    ...

When we finally reach the location our caller cares about, we extend our return buffer and return it if it reaches or exceeds our slice’s size:

      if self.stream_index >= offset:
        buf.extend(line)

      if len(buf) >= size:
        return bytes(buf[:size])

    return bytes(buf)
    ...

Thread-safety

This library needs to be thread-safe. At any point, another thread can call read(). If two threads call read() at once, we can hit a race condition.

Since we’re seeking and writing to a SpooledTemporaryFile, we can have a situation where thread #1 seeks to location X, then thread #2 seeks to location Y, and thread #1 reads/writes to location Y instead of location X. That would be bad.

If you’ve used Java, synchronized methods and blocks can solve the problem of handling race conditions by sticking a lock in front of our critical sections.

The wrapt library contains a synchronized decorator and context manager. We’ll use it to wrap our critical section and make our library thread-safe.

Let’s go back to read() and stick this with statement in it:

   def read(self, offset: int, size: int) -> bytes:
      with synchronized(self):
      ...

Finishing `read()`

Next, we can implement _chunk_location(), and branch on the Enum it returns.

  def _chunk_location(self, offset: int, size: int) -> Location:
    end = offset + size

    if offset < self.stream_index and end <= self.stream_index:
      return Location.BeforeIndex

    elif offset < self.stream_index < end:
      return Location.Bisected

    elif offset == self.stream_index:
      return Location.AtIndex

    return Location.AfterIndex

  def read(self, offset: int, size: int) -> bytes:
    with synchronized(self):
      location = self._chunk_location(offset, size)

      if location is Location.BeforeIndex:
        return self._chunk_before_index(offset, size)

      elif location is Location.AtIndex:
        return self._chunk_at_index(size)

      elif location is Location.Bisected:
        return self._chunk_bisected_by_index(offset, size)

      elif location is Location.AfterIndex:
        return self._chunk_after_index(offset, size)

In Python 3.10, we can write our read() method using pattern matching with a match statement:

  def read(self, offset: int, size: int) -> bytes:
    with synchronized(self):
      match self._chunk_location(offset, size):
        case Location.BeforeIndex:
          return self._chunk_before_index(offset, size)

        case Location.AtIndex:
          return self._chunk_at_index(size)

        case Location.Bisected:
          return self._chunk_bisected_by_index(offset, size)

        case Location.AfterIndex:
          return self._chunk_after_index(offset, size)

This method will translate well to the Rust version of this library that we’ll write in the next section.

Cleaning up

I’ll keep a reference to the StreamBuffer around if I want it to stick around. Otherwise, I want it to clean-up the SpooledTemporaryFile when it’s garbage collected:

  def __del__(self):
    logging.debug(f'Releasing {self}')
    self.temp.close()
    ...

I’ll give it a nice __repr()__ for the logs:

  def __repr__(self):
    name = StreamBuffer.__name__
    size = self.size
    stream_index = self.stream_index
    temp = self.temp

    return f'{name}<{size=}, {stream_index=}, {temp=}>'
    ...

Writing it in Rust

While Python is great for prototyping, Rust makes it easy to catch errors at compile time and prevent some errors at runtime. Rust’s type system is expressive, and utilizing generics with trait bounds can allow you to write libraries that are both adaptable and correct. We can setup our build such that the library won’t compile unless its tests pass.

Starting out

Rust doesn’t have a tempfile module in its standard library, but there is a tempfile crate with a SpooledTempFile implementation, so I’ll use that below and declare some constant items.

#![feature(extend_one)]
use std::io::{Write, Read, Seek, SeekFrom, Bytes};
use std::io::Error as IoError;

use tempfile::SpooledTempFile;

const MAX_SIZE: usize = 5 * 1_024 * 1_024;  // bytes

Next, I’m going to declare a FileLike trait that implements traits from std::io. Then I implement FileLike for all types T that have the Read, Seek, and Write traits implemented.

pub trait FileLike: Read + Seek + Write {}
impl<T: Read + Seek + Write> FileLike for T {}

I’ll build some type aliases for the library that will make the rest of the code both clearer and less verbose.

type Byte = u8;
type ByteBuf = Vec<Byte>;
type ByteResult = Result<Byte, IoError>;
type ByteBufResult = Result<ByteBuf, IoError>;
type ByteStreamBuf<F> = StreamBuffer<ByteResult, F>;
type Stream<T> = Box<dyn Iterator<Item = T>>;

In both Python and Rust, type aliases exist for developer convenience.

However, in Python, type annotations are just labels, and are not used for runtime type checking by CPython.

Type annotations and type aliases have semantic meaning in Rust that is enforced by the compiler.

Translating the StreamBuffer class to Rust

Languages like Python are dynamically typed, and have little need for explicit generics.

Rust allows developers to design APIs that can match their dynamically typed cousins, but with safety and restrictions enforced by the compiler.

StreamBuffer is generic over the type T of streaming data we’re iterating over and the FileLike type F that was a SpooledTemporaryFile in Python.

By making StreamBuffer generic over types T and F, our Rust implementation isn’t limited to using a SpooledTemporaryFile as its backing store, and we can safely iterate and handle errors for nearly limitless types of streams.

pub trait Buffer {}

pub struct StreamBuffer<T, F: FileLike> {
    size: usize,
    index: usize,
    stream: Stream<T>,
    file: F,
}

impl<T> StreamBuffer<T, SpooledTempFile> {
    pub fn new(stream: Stream<T>, size: usize) -> StreamBuffer<T, SpooledTempFile> {
        let file = SpooledTempFile::new(MAX_SIZE);
        StreamBuffer { size, stream, file, index: START_INDEX }
    }
}

impl<T, F: FileLike> Buffer for StreamBuffer<T, F> {}

BufferRead is generic over read()’s output type T.

pub trait BufferRead<T>: Buffer {
    fn read(&mut self, offset: usize, size: usize) -> T;
}

Below, I will define traits that will encode cases our read() implementation will need to cover.

Implementing read()

We can encode the four states our read() function will encounter in an enum named Location, and divide into traits the tasks of reading into a buffer and writing to the tempfile based on the the Locations.

ChunkLocation provides a trait that identifies one of the four states, and ChunkRead will provide implementations to read from those locations.

enum Location {
    BeforeIndex,
    Bisected,
    AtIndex,
    AfterIndex,
}

trait ChunkLocation {
    fn _chunk_location(&self, offset: usize, size: usize) -> Location;
}

trait ChunkRead<T> {
    fn _chunk_before_index(&mut self, offset: usize, size: usize) -> T;
    fn _chunk_bisected_by_index(&mut self,  offset: usize, size: usize) -> T;
    fn _chunk_at_index(&mut self, size: usize) -> T;
    fn _chunk_after_index(&mut self, offset: usize, size: usize) -> T;
}

impl<T, F: FileLike> ChunkLocation for StreamBuffer<T ,F> {
    fn _chunk_location(&self, offset: usize, size: usize) -> Location {
        let end = offset + size;

        if offset < self.index && end <= self.index {
            Location::BeforeIndex
        } else if offset < self.index && self.index < end {
            Location::Bisected
        } else if offset == self.index {
            Location::AtIndex
        } else {
            Location::AfterIndex
        }
    }
}

impl<F: FileLike> ChunkRead<ByteBufResult> for ByteStreamBuf<F> { /* snip */ }

impl<F: FileLike> BufferRead<ByteBufResult> for ByteStreamBuf<F> {
    fn read(&mut self, offset: usize, size: usize) -> ByteBufResult {
        match self._chunk_location(offset, size) {
            Location::BeforeIndex => self._chunk_before_index(offset, size),
            Location::Bisected => self._chunk_bisected_by_index(offset, size),
            Location::AtIndex => self._chunk_at_index(size),
            Location::AfterIndex => self._chunk_after_index(offset, size)
        }
    }
}

By breaking read() up like this, it allowed me to reason about the library easily, and most importantly, it was easier to test. I was able to find and correct logic errors in the code via tests in this Rust module.

The general structure of the Rust code and the corrections that the tests helped elucidate then made their way back into the Python code written above.

Conclusion

We just walked through building a relatively simple stream buffer. You can check out the Python code here and install it like so:

pip3 install buffer

You can check out the Rust code here, too.

Optimization with NumPy and Rust

Thu, 06 Dec 2018 18:00:00 +0000

This is a quick post about optimizing algorithms written in Python with NumPy, and implementing the same code in Rust. We’re going to take a look at one of the Advent of Code challenges, solve it, optimize it with NumPy, and then write both of our solutions in Rust.

If you’re unfamiliar with Advent of Code, it’s an advent calendar of programming related puzzles which has run each December since 2015. Every day from December 1st until the 25th, a new challenge is posted and you’re encouraged to submit a solution.

Analyzing the Challenge

Let’s take a look at AoC’s Day 3 challenge.

This problem requires us to count the number of overlapping claims in a given area. That area, which is the fabric in the challenge, can be represented by a 2-dimensional array of zeros in a naive solution to this problem.

Every item in the 2-dimensional array will keep a count of how many claims take ownership of that item.

Each claim can be thought of as another smaller 2-dimensional array of ones that we add to the larger 2-dimensional array. When we are finished, we can count how many items in the larger 2-dimensional array are greater than 1. Those items will represent the square inches of fabric that belong to more than one claim.

Pure Python Solution

Here’s a naive solution written in Python. While in the above analysis I said that we can think of each claim as being a 2-dimensional array of ones that we’ll add to the greater 2-dimensional array, our implementation will differ.

We’ll iterate over the Cartesian coordinates of each claim and add 1 to the corresponding item in the greater 2-dimensional array.

from __future__ import annotations
from typing import NamedTuple
from itertools import product


NO_CLAIM: int = 0
INCREMENT: int = 1
MIN_OVERLAP: int = 2
SIDE_LENGTH: int = 1050


class Claim(NamedTuple):
  x: int
  y: int
  width: int
  height: int


Claims = list[Claim]
Fabric = list[list[int]]


def get_fabric(claims: Claims, length: int = SIDE_LENGTH) -> Fabric:
  fabric = [[NO_CLAIM] * length] * length

  for x, y, width, height in claims:
    x_indices = range(x, x + width)
    y_indices = range(y, y + height)

    for x, y in product(x_indices, y_indices):
      fabric[x][y] += INCREMENT

  return fabric


def count_overlapping(fabric: Fabric) -> int:
  return sum(
    INCREMENT 
    for row in fabric 
    for item in row 
    if item >= MIN_OVERLAP
  )

# generate list of claims from challenge input file
claims: Claims = ...

# let's profile our solution
%timeit get_fabric(claims)
fabric = get_fabric(claims)
%timeit count_overlapping(fabric)

Here’s our output on an Intel processor:

287 ms ± 14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
125 ms ± 1.62 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

It took nearly half of a second to process roughly one thousand claims. We can do better than that.

Optimization with NumPy

Vecorization

NumPy has a concept called Universal functions, or ufuncs, which are vectorized functions over ndarrays. Vectorized functions work over entire arrays and, if the underlying hardware supports it, can take advantage of SIMD instructions.

SIMD stands for single instruction, mutiple data. SIMD instructions allow us to apply an instruction over multiple points of data at the same time. They implement parallelism at the data-level.

SIMD diagram created by Colin M.L. Burnett for Wikipedia

Examples of SIMD instructions are MMX, SSE/SSE2/SSE3/SSE4 and AVX instruction sets on x86 processors and NEON instruction sets on ARM processors.

When all that’s said and done, there’s no guarantee that NumPy will use SIMD instructions in our program. However, we’ll still benefit from a performance boost, because NumPy is built upon natively compiled and architectually efficient data structures and operations.

I have several clustered ARMv7 machines. In the past, NumPy wasn’t optimized for ARM’s NEON instruction set in the same way that it’s optimized for x86 processors with SSE and AVX instructions. GCC’s ability to optimize NumPy at compile time will take advantage of NEON, although in a more generalized fashion.

Then, in late 2019, NumPy received some new NEON optimizations.

Even when NumPy wasn’t optimized for NEON, I still saw a performance increase when using NumPy instead of pure Python.

Optimizing Our Solution

In our previous solution, get_fabric() and count_overlapping() are sequential functions. That is to say that we apply an operation to each item in the array sequentially: we increment the first element in the array, then the next element and so on.

To optimize our solution, we’ll need to use something beyond just pure Python.

NumPy is a Python wrapper over lower-level C and Fortran code that implements optimized and natively compiled routines, so we’ll use that.

Here, we’ll use NumPy’s add() vectorized ufunc instead of sequentially adding one to each item in the fabric array.

import numpy as np

Fabric = np.ndarray

def get_fabric(claims: Claims, length: int = SIDE_LENGTH) -> Fabric:
  fabric = np.zeros((length, length), dtype=np.int8)

  for x, y, width, height in claims:
    x_indices = slice(x, x + width)
    y_indices = slice(y, y + height)
    indices = x_indices, y_indices

    fabric[indices] += INCREMENT

  return fabric

Our above solution will benefit from NumPy’s memory efficient arrays, compared to our previous solution using Python’s list type.

Next, we’ll use NumPy’s sum() routine to locate coordinates that specify values that are greater than 1.

def count_overlapping(fabric: Fabric) -> int:
  return np.sum(fabric >= MIN_OVERLAP)

Let’s see what kind of performance boost we’ll get using our new solution.

%timeit get_fabric(claims)
fabric = get_fabric(claims)
%timeit count_overlapping(fabric)

Here’s our output on a relatively recent Intel processor:

12.8 ms ± 883 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
1.76 ms ± 79.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Compared to our previous solution, our new get_fabric() implementation is 22 times faster and count_overlapping() is 63 times faster.

Not bad.

Writing It in Rust

First, we can write the naive solution in Rust and cut the run time in half compared to our NumPy solution. Then, we can use the ndarray crate to further optimize our Rust code along the lines of our NumPy code.

Here’s the naive solution in Rust.

use itertools::Itertools;

const NO_CLAIM: u8 = 0;
const INCREMENT: u8 = 1;
const MIN_OVERLAP: u8 = 2;
const SIDE_LENGTH: usize = 1000;
const START: usize = 0;

struct Claim {
  x: usize,
  y: usize,
  width: usize,
  height: usize,
}

type Claims = [Claim];
type Fabric = Vec<Vec<u8>>;

fn get_fabric(claims: &Claims) -> Fabric {
  let mut fabric = 
    vec![vec![NO_CLAIM; SIDE_LENGTH]; SIDE_LENGTH];

  for &Claim { x, y, width, height } in claims {
    let x_indices = x..(x + width);
    let y_indices = y..(y + height);

    for (x, y) in x_indices.cartesian_product(y_indices)  {
      fabric[x][y] += INCREMENT;
    }
  }

  fabric
}

fn count_overlapping(fabric: &Fabric) -> usize {
  fabric.iter()
    .flatten()
    .filter(|&&item| item >= MIN_OVERLAP)
    .count()
}

fn main() {
  let claims: Claims; //snip
  let fabric = get_fabric(&claims);
  let overlapping = count_overlapping(&fabric);
}

We can cut the run time in half again with the rough translation of the NumPy code below, for a total run time of roughly 3ms.

use ndarray::prelude::*;
use ndarray::{Array, Ix2};

type Fabric = Array<u8, Ix2>;

fn get_fabric(claims: &Claims) -> Fabric { 
  let mut fabric = 
    Array::zeros((SIDE_LENGTH, SIDE_LENGTH).f());

  for &Claim { x, y, width, height } in claims {
    let x_indices = x..(x + width);
    let y_indices = y..(y + height);

    for pt in x_indices.cartesian_product(y_indices) {
      fabric[pt] += INCREMENT;
    }
  }

  fabric
}

fn count_overlapping(fabric: &Fabric) -> usize {
  fabric.iter()
    .filter(|&&item| item >= MIN_OVERLAP)
    .count()
}

Social Media Trend Visualization

Mon, 12 Nov 2018 18:00:00 +0000

Background

I run a Reddit bot that identifies duplicate submissions, and a companion reverse image search engine.

Right now, the bot is active on meme subreddits, but I built a generalized pipeline to mine new Reddit submissions.

I decided to track a few other subreddits a few months ago. I wanted to mine contentious subreddits, especially those with identified bot activity.

I settled on mining r/news, r/politics, r/The_Donald and a few others.

The “Caravan”

It seemed like one day in October, people were suddenly talking about the “caravan”.

The popularity of the “caravan” grew after Fox News covered stories that used the term over a couple of days. Before that, a few other sources used the term throughout the end of September and the beginning of October.

The term exploded after the President, via Twitter, warned Honduras that US aid would stop if the “caravan” didn’t turn around.

Over the next two weeks, the President tweeted about the “caravan” 9 times, including posting an attack ad focusing on immigration that included footage of the migrants.

Let’s see how those trends compare to a related term and the overall new submission trend on r/The_Donald.

The larger the text is in the image below, the more often the term was used in submissions to r/The_Donald.

The top 14% of submitters that posted about this topic were responsible for exactly half of caravan-related submissions during the time period examined in the images above.

45 of the top 50 submitters that posted submissions related to this topic were also within the top 150 submitters to r/The_Donald during that time.

Process Scheduling on Linux and macOS

Thu, 23 Aug 2018 18:00:00 +0000

Background

Sometimes a task will be heavily disk, network or CPU dependent, and will starve other tasks of those resources. It’s important that such tasks won’t steal resources from other tasks.

Ideally, some resource intensive tasks would run only when the resources they depend on are idle. That, or resource intensive tasks would have their resource consumption throttled to favor other tasks.

In this article, we will review CPU and I/O scheduling at the process-level on both Linux and Darwin.

Process Scheduling

Process scheduling priority can be easily tuned with nice.

Linux processes given a lower niceness value on the interval [-20, 19] via the nice shell interface have priority over those with a larger niceness value. Internally, niceness has an interval of [0, 139], of which only the last 39 priorities are available to users.

Here’s how we’d run a process with a lower scheduling precedence:

nice -n 19 our_expensive_cmd

On macOS, priorities given to nice on the command-line, or via the C process management interface, exist on the interval [-20, 20].

Darwin, the open source basis for macOS, has additional process priority concepts.

For example, a process with priority PRIO_DARWIN_BG has these properties:

When a thread or process is in a background state the scheduling priority is set to the lowest value, disk IO is throttled (with behavior similar to using setiopolicy_np(3) to set a throttleable policy), and network IO is throttled for any sockets opened after going into background state. Any previously opened sockets are not affected.

We’ll use Darwin’s process priority concepts in the next section.

Processor Affinity

On Linux, we can set a task’s CPU affinity with taskset.

If you run a heterogenous multiprocessing (HMP) setup, like those found in ARM big.LITTLE designs, this means you can pin background tasks to the little processor cores, or a CPU intensive task you’d like to finish quickly to the big processor cores.

If you want to take advantage of a processor’s cache or a multicore processor’s shared cache, this can eliminate overhead associated with switching processes between many processors or cache misses/coherency.

XNU, the kernel from which Darwin is based, exposes a thread affinity API, but there isn’t a convenient taskset-like wrapper around it.

Network and Disk I/O Scheduling

The Linux kernel has a variety of input/ouput (I/O) queues. I/O is prioritized by each process’s given I/O scheduling class. The classes are idle, best effort and real time.

A process with the I/O scheduling class of idle will be given disk access when the resource is idle.

Best effort classes have priorities on the interval [0, 7], with lower numbers given a higher priority. Best effort scheduling is done via round-robin.

Real time classes are given disk priority, have the same number of priority levels as best effort and can only be used by root. Real time scheduled processes can starve other processes of resources. If you don’t know if you should use real time I/O scheduling, that most likely means that you don’t need it.

We can set I/O priority on Linux via ionice.

# run a command with `idle` disk priority
ionice -c idle our_expensive_cmd

# run a command with the lowest `best effort` disk priority
ionice -c best-effort -n 7 our_expensive_cmd

Linux does not have a convenient interface to set network I/O priority. There are many options one can implement to that end, perhaps using kernel network QoS policies or iptables. I’m sure something involving network namespaces could be hacked together and would be pretty cool. However, someone built a userspace solution with a nice command-line interface, so why not use that?

Since Linux is well-documented, I will not go into its different tiers of I/O queues, but I will expound on the macOS equivalent when we get to them, as I was unable to find much documentation in macOS-land.

Unlike Linux, XNU-derived systems have the concept of network I/O priority classes for processes.

XNU-derived macOS does not have the same scheduling structure as Linux, but does have the ability to prioritize disk I/O. If we look at the setiopolicy_np() manpage, we’ll see that macOS has a handful of I/O priority classes:

IOPOL_IMPORTANT processes are given utmost priority.
IOPOL_STANDARD processes may be delayed in favor of IOPOL_IMPORTANT.
IOPOL_UTILITY processes are intended to be short-running and are throttled to favor the two priority classes above.
IOPOL_THROTTLE processs are intended to be long-running and are throttled to favor the above classes.
IOPOL_PASSIVE processes are intended for server-type processes for which I/O is the result of requests from client applications. The intent is that client application I/O will not be slowed down by server I/O initiated by the client. Other priority classes will ignore this class, giving the above classes priority over IOPOL_PASSIVE.

Darwin gives us a nice command-line interface around setiopolicy_np() and setpriority(), called taskpolicy.

You can browse its source here, given that it’s part of the Darwin open-source project. Here’s a copy of the relevant taskpolicy manpage, since Apple took their copy down.

Let’s take a look at how we’d run a process with the lowest CPU and I/O priority in Linux:

ionice -c idle nice -n 19 our_expensive_cmd

Here’s how we’d do something very similar to the above on macOS:

taskpolicy -b our_expensive_cmd

The -b flag runs our_expensive_cmd with the process priority PRIO_DARWIN_BG, which was touched upon in the previous section.

taskpolicy also allows us to set one of the five disk policies that were listed above. For example, if we wanted to run a process with the policy IOPOL_THROTTLE and scope IOPOL_SCOPE_PROCESS, we could run:

taskpolicy -d throttle our_expensive_cmd

If we ran taskpolicy with the -g flag, it would do the same as above, but with the scope IOPOL_SCOPE_DARWIN_BG.

Digging around the XNU kernel source, we will find that IOPOL_SCOPE_DARWIN_BG is given the TASK_POLICY_DARWIN_BG_IOPOL I/O policy flavor, which is used to decide the background process I/O tier.

We’ll also see that the default I/O policy for IOPOL_SCOPE_DARWIN_BG is IOPOL_UTILITY. IOPOL_UTILITY is given tier THROTTLE_LEVEL_TIER2, which is one tier above THROTTLE_LEVEL_TIER3.

Interestingly enough, mobile XNU-derived systems like iOS have a slightly different I/O scheduling scheme than non-mobile systems, based on what I can glean from preprocessor directives.

Here’s a breakdown of tiers:

THROTTLE_LEVEL_TIER0, the default scheduling tier for IOPOL_NORMAL, IOPOL_DEFAULT and IOPOL_PASSIVE.
THROTTLE_LEVEL_TIER1, default for IOPOL_STANDARD
THROTTLE_LEVEL_TIER2, default for IOPOL_UTILITY
THROTTLE_LEVEL_TIER3, default for IOPOL_THROTTLE

If we try to give a process the scope of IOPOL_SCOPE_DARWIN_BG, but give it any policy but IOPOL_UTILITY or IOPOL_THROTTLE, we’ll get an error:

alex@mbp:~$ taskpolicy -g passive bash
taskpolicy: setiopolicy_np(...IOPOL_SCOPE_DARWIN_BG...): Invalid argument

At some point I’ll do a write up on XNU QoS latency and throughput tiers, but that’s for another day.

Conclusion

We took a look at what CPU and I/O prioritization options are available on Linux. We’ve contrasted them with that which are available on macOS.

While Linux has utilities like nice, ionice and taskset, it lacks a convenient process-level network throttling option. macOS, on the otherhand, has network and disk throttling built-in and available behind the taskpolicy command-line utility.

Setting processor affinity is as easy as issuing a shell command on Linux, but on macOS such functionality must implemented at the application level, per application.

Reverse Engineering CoreDisplay API

Thu, 16 Aug 2018 18:00:00 +0000

Background

In this article, we’ll review methods for changing display brightness on a Mac.

I personally find it irritating when my display shuts off, or my computer locks down, while I’m still looking at it. This happens frequently when watching videos, following a recipe or slowly trying to grok something on the screen. Therefore, I have my display set to turn off at an interval that’s over 1 hour. I know this makes me a bad person.

I wrote a daemon that dims my display’s backlight if my computer becomes idle and my face isn’t within the viewing area of my built-in FaceTime webcam.

You can check it out here.

Adjusting Brightness on macOS

Unlike iOS, there is no convenient screen brightness API for macOS.

Here’s how you’d change your screen’s brightness programmatically with Swift in iOS, where someBrightnessFloat is a float in the interval [0.0, 1.0]:

UIScreen.mainScreen().brightness = someBrightnessFloat

It would be convenient if that API were available, but to accomplish this on a Mac, we’ll need to jump through some hoops.

Eventually, through some research, I came across the relevant APIs in the IOKit Framework. Great!

Bridging Between Objective-C and Python

I decided to build my daemon with pyObjc and Python 3.7.0, simply because developing command line daemons with Python is fun.

pyObjc tends to have wrappers for Apple’s development frameworks, except none exist for IOKit. That’s fine, we can wrap it with pyObjc or even ctypes if we’re desperate.

Let’s look at the relevant Objective-C code. In Objective-C, we’d want to do something along these lines:

float brightness = ourBrightnessFloatGoesHere;
CGDirectDisplayID display = CGMainDisplayID();
io_service_t service = IOServiceGetMatchingService(kIOMasterPortDefault, IOServiceMatching(display));

IODisplaySetFloatParameter(service, kNilOptions, CFSTR(kIODisplayBrightnessKey), brightness);

We’ll need to translate that to Python code with pyObjc.

BridgeSupport

With ctypes, we need to provide the library with correct arg_types and return types. This can be tedious.

Thankfully, BridgeSupport files can be generated from the library with macOS system tools. These files keep function signatures and typing information in an XML structure. pyObjc can use BridgeSupport data so we don’t have to manually annotate each function we want to call.

We’ll need to grep around IOKit.framework to find the values we need, which are:

kIOMasterPortDefault, a constant.
kIODisplayBrightnessKey, a constant.
IOServiceMatching, a function.
IOServiceGetMatchingService, a function.
IODisplaySetFloatParameter, a function.

After poking around the framework, we’ll find that information we need is located in graphics/IOGraphicsLib.h, IOKitLib.h, and graphics/IOGraphicsTypes.h.

Now, we’ll need to generate a BridgeSupport file.

export BRIDGESUPPORT=".bridgesupport"

gen_bridge_metadata -c "-l/System/Library/Frameworks/IOKit.framework/IOKit
                    -I/System/Library/Frameworks/IOKit.framework/Headers/graphics
                    -I/System/Library/Frameworks/IOKit.framework/Headers" \
                    /System/Library/Frameworks/IOKit.framework/Headers/graphics/IOGraphicsLib.h \
                    /System/Library/Frameworks/IOKit.framework/Headers/IOKitLib.h  \
                    /System/Library/Frameworks/IOKit.framework/Headers/graphics/IOGraphicsTypes.h > "$BRIDGESUPPORT"

If you’re on MacOS 10.13+, you’ll get an error. gen_bridge_metadata will look in a directory that doesn’t exist, so we’ll need to symlink the directory we need to the location that doesn’t yet exist.

export TOOLCHAINS_DIR="/Applications/Xcode.app/Contents/Developer/Toolchains"
export WORKING_DIR=`pwd`


# fix issue where gen_bridge_metadata cannot find toolchain
# https://trac.macports.org/ticket/54939
if defaults read loginwindow SystemVersionStampAsString | grep '10.13'
then
    cd "$TOOLCHAINS_DIR"

    if [ ! -e OSX10.13.xctoolchain ]
    then
        sudo ln -s XcodeDefault.xctoolchain OSX10.13.xctoolchain
    fi

    cd "$WORKING_DIR"
fi

Rerun the gen_bridge_metadata command again, and we’ll get a nice BridgeSupport file.

We aren’t in the clear yet. pyObjc supports a certain dialect in BridgeSupport files, and the one we generated will have values the library doesn’t understand. We’ll need to scrub the BridgeSupport file of those values.

IOTimingInformation has a weird signature, and it’s irrelevant to our use case, so let’s get rid of it.

sed -i "" -e  "/name='IOTimingInformation'/d" "./$BRIDGESUPPORT"

Changing Brightness

Great, now we’ve generated a BridgeSupport file for the utilities in IOKit we’ll use. Now, we can write some application code.

Let’s import our CFString manipulation libraries:

from CoreFoundation import CFStringCreateWithCString, CFRelease, kCFStringEncodingASCII

Then, let’s write a function to import IOKit.

from __future__ import annotations
from typing import Callable, Any
from pathlib importh Path

import objc


IOKIT_FRAMEWORK: str = "/System/Library/Frameworks/IOKit.framework"
BRIDGESUPPORT_FILE: str = ".bridgesupport"
DISPLAY_CONNECT: bytes = b"IODisplayConnect"


def import_iokit(iokit_location: str = IOKIT_FRAMEWORK,
                 namespace: dict[str, Any] = None):
    if namespace is None:
        namespace = globals()

    bridgesupport_file = Path(BRIDGESUPPORT_FILE).read_text()

    objc.parseBridgeSupport(bridgesupport_file,
                            namespace,
                            objc.pathForFramework(iokit_location))

When we run import_iokit() it will read the BridgeSupport file into a buffer, and pyObjc will parse and load our wrapped functions and values into global namespace.

Now, let’s write a function that uses IOKit to change our Mac’s brightness:

def set_brightness_iokit(brightness: float) -> int:
    brightness /= 100
    service = IOServiceGetMatchingService(kIOMasterPortDefault,
                                          IOServiceMatching(DISPLAY_CONNECT))
    return IODisplaySetFloatParameter(service,
                                      0,
                                      kIODisplayBrightnessKey,
                                      brightness)

Calling set_brightness_iokit(50) will dim our display to 50% brightness. Great!

Undocumented CoreDisplay API

Well, if you’re on macOS 10.13+, things aren’t so great. You’ll notice that brightness adjustments using IOKit’s API reset after a certain period. That is, on macOS 10.13, your brightness will go back to its default setting after N seconds/minutes/hours.

This is because Apple has deprecated IOKit’s brightness functionality in favor of a private, undocumented API. The words ‘private undocumented API’ should instill fear in any developer who encounters them on their journey to solve a problem. They’re the Here be dragons of software development and dragons are less scary than undocumented features.

After doing some digging, it appears nobody has documented this API, nor is there a convenient Python bridge for it. The closest I’ve come to seeing developers use this API in the open is the hardcore macOS hacking community, who have patched the framework to support older hardware with newer displays & resolutions. Nobody’s touched it in an effort to change screen brightness.

Normally, this is the point where I throw my hands up and say, “well, that was a waste of time” and focus my energy somewhere else. However, I built out a pretty good solution around the IOKit enabled functionality and I am not ready to give that up.

Reverse Engineering CoreDisplay

First, let’s see if there’s anything we can glean from this framework. We could break out a hex editor, disassembler and all sorts of fun things, but I don’t want to get too crazy, lest it be for naught.

We need to determine which functions handle setting brightness, or if they even exist.

alex@mbp:/System/Library/Frameworks/CoreDisplay.framework$ grep -rn Brightness .
Binary file ./CoreDisplay matches

Cool, while there aren’t any header files or code we can use, there seems to be something related to brightness in CoreDisplay.framework.

Looking in CoreDisplay.framework, we don’t have much to work with: PList files and compiled libraries.

Let’s dig into CoreDisplay.framework/CoreDisplay a little further:

alex@mbp:/System/Library/Frameworks/CoreDisplay.framework$ strings CoreDisplay | grep Brightness | wc -l
  49

There are 49 relevent lines. This is a good thing. Let’s take a deeper look:

alex@mbp:/System/Library/Frameworks/CoreDisplay.framework$ strings CoreDisplay | grep Brightness | grep Set
void CoreDisplay::Display::SetUserBrightness(double)
void CoreDisplay::Display::SetLinearBrightness(double)
void CoreDisplay::Display::SetLinearBacklightBrightness(double)
void CoreDisplay::Display::SetBacklightBrightness(double)
void CoreDisplay_Display_SetDynamicLinearBrightness(CGDirectDisplayID, double)
void CoreDisplay_Display_SetDynamicLinearBrightness_Server(CGDirectDisplayID, double)
Skip SetLinearBrightness on offline display 0x%08x
void CoreDisplay::Display::SetDynamicLinearBrightness(double)
SetDynamicLinearBrightness
void CoreDisplay_Display_SetLinearBrightness(CGDirectDisplayID, double)
void CoreDisplay_Display_SetLinearBrightness_Server(CGDirectDisplayID, double)
void CoreDisplay_Display_SetUserBrightness(CGDirectDisplayID, double)
void CoreDisplay_Display_SetUserBrightness_Server(CGDirectDisplayID, double)
Skip SetUserBrightness on offline display 0x%08x
void CoreDisplay_Display_SetAutoBrightnessIsEnabled(CGDirectDisplayID, bool)
void CoreDisplay_Display_SetAutoBrightnessIsEnabled_Server(CGDirectDisplayID, bool)
Skip SetAutoBrightnessIsEnabled on offline display 0x%08x
void CoreDisplay::Display::SetAutoBrightnessIsEnabled(bool)
void CoreDisplay::EDRDisplay::ScheduleSetBrightnessIn(double)_block_invoke
ScheduleSetBrightnessIn_block_invoke

We’ve got function signatures! This was easy, no need for breaking out heavier tools and methods.

Inspecting futher, it seems like we’ve got a few functions that are relevant to the problem at hand:

void CoreDisplay::Display::SetUserBrightness(double)
void CoreDisplay::Display::SetLinearBrightness(double)
void CoreDisplay::Display::SetLinearBacklightBrightness(double)
void CoreDisplay::Display::SetBacklightBrightness(double)
void CoreDisplay_Display_SetDynamicLinearBrightness(CGDirectDisplayID, double)
void CoreDisplay_Display_SetDynamicLinearBrightness_Server(CGDirectDisplayID, double)
Skip SetLinearBrightness on offline display 0x%08x
void CoreDisplay::Display::SetDynamicLinearBrightness(double)
SetDynamicLinearBrightness
void CoreDisplay_Display_SetLinearBrightness(CGDirectDisplayID, double)
void CoreDisplay_Display_SetLinearBrightness_Server(CGDirectDisplayID, double)
void CoreDisplay_Display_SetUserBrightness(CGDirectDisplayID, double)

That last line looks pretty relevant to me, so let’s try to call that function from Python.

Using `ctypes` to Call CoreDisplay’s Private API

Python’s ctypes gives us a convenient foreign function interface. Here’s how we’d use it for our problem:

from ctypes import CDLL, c_int, c_double
CoreDisplay = CDLL("/System/Library/Frameworks/CoreDisplay.framework/CoreDisplay")

Now, we’ve got a handle on CoreDisplay through ctypes.

ctypes needs to know how to call CoreDisplay_Display_SetUserBrightness

CoreDisplay.CoreDisplay_Display_SetUserBrightness.argtypes = [c_int, c_double]

Let’s test it out:

CoreDisplay.CoreDisplay_Display_SetUserBrightness(0, 0.5)

Yes! This successfully changes the brightness on display 0 to 50%!

Finally, we’ll wrap it in a function so that we can easily call it elsewhere from Python :

def set_brightness_coredisplay(display: int, brightness: float) -> int:
    brightness /= 100
    return CoreDisplay.CoreDisplay_Display_SetUserBrightness(display, brightness)

Conclusion

We’ve learned how to use pyObjc and BundleSupport files to wrap macOS libraries that aren’t conveniently wrapped for us. We poked around undocumented libraries to locate functionality we’re interested in, and we’ve also learned how to use ctypes to call out to those private macOS frameworks.

Apparently, we’ve also learned that there’s no limit to the amount of hoops I’ll jump through to make inconsequential things work.

In conclusion, we built the foundation for my daemon that uses facial recognition to manipulate your screen’s brightness. You should check it out!

Alex DeLorenzo

Encrypted Root with LUKS and Opal

Using cryptsetup’s native Opal support to decrypt self-encrypting drive partitions at boot with LUKS and systemd

Hardware encryption versus software encryption

Booting self-encrypted drives on machines without Opal support

New cryptsetup has native Opal support

Typical encrypted file-system scheme with cryptsetup

How to use hardware encryption with cryptsetup

Detect Opal support

Reset your Opal drive

Create boot and root partitions

Set up LUKS with Opal support

Enroll your authentication methods

Confirm configuration

Set your machine to unlock LUKS

Create a crypttab file

Install your root file-system

Configure /etc/fstab

Unlock at boot

Edit kernel command-line options

Configure your early boot environment

Reboot

Encrypted swap and hibernation

Create a swap file

Configure fstab

Get the swap file offset

Get your LUKS partition UUID

Adjust your kernel parameters

Reboot and test it out

Toll-free Bridging in Kotlin Native

About Core Foundation & Foundation

Toll-free Bridging

Kotlin Native

Interoperability

How to use it

From Objective-C and C

Toll-free bridging and ARC

From Kotlin Native

Why toll-free bridging is necessary in Kotlin Native

Toll-free bridging in Kotlin Native

Working with Core Foundation types

Memory management

Writing idiomatic extension functions

Conclusion

How to find Ubuntu repository mirrors

Usage

Passing options

Output

Requirements

Script

How to install npm packages as a non-root user

Usage

Options

Passing options

Confirming it works

Requirements

Dependencies

Supported systems

Shells

Operating systems

Script

How to get the first item of a Python iterable

Using the function

Iterables

Ordered sequences

Empty sequences

Default value

Generator expressions

Strings and bytes

Arrays

Files

Custom iterators

Unordered collections

An async first function

Usage

Files

Async iterables

Async generator expressions

Custom async iterators

How to clean up Python build, caching and packaging artifacts

Using `cryptsetup`’s native Opal support to decrypt self-encrypting drive partitions at boot with LUKS and systemd

New `cryptsetup` has native Opal support

Typical encrypted file-system scheme with `cryptsetup`

How to use hardware encryption with `cryptsetup`

Create a `crypttab` file

Configure `/etc/fstab`

Configure `fstab`

Script using `🧵strs`

Delete `privatekey` and `psk` Files