Recently I’ve decided to set up my 9800X3D + 9070XT gaming PC I assembled more than 1 year ago as a server with GPU passthrough in a dedicated VM for offline “dangerous” gaming on top of the physical Debian host, alongside the physical Windows host that I would only boot into for “legitimate” gaming.
The PC has three drives:
As I sometimes would prefer play games from my collection from the physical Windows drives directly in the VM than rebooting into the physical Windows, especially when there’s workload running on the Debian system, I prefer if the physical Windows drives are readable in the VM.
The simplest idea to pass through both of these disks surely have performance benefits, but if viruses and trojans went wild in the VM, while the system residing on QCOW2 could be rolled back, the physcial drives passed through then cannot be fixed, things just go boom. Besides, I don’t want the physical Windows drives accidentally modified even not under those circumstances, so writable physical is no-go.
The next simplest idea is to make the drives read-only, on the block device level under Linux. This is quite easy to do with a simple udev rules file:
> cat /etc/udev/rules.d/99-windows-ro.rules
SUBSYSTEM=="block", ACTION=="add", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX_1|GeIL_P4S_4TB_XXXXXXXXXXX_1", RUN+="/sbin/blockdev --setro /dev/%k"
The character | is used to match multiple drives, and each IDs are obtained through simple udevadm info:
> sudo udevadm info /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX
......
E: ID_SERIAL=Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX_1
......
Note a explicit command with RUN+= is used instead of ATTR{ro}="1", as the latter file is simply read-only, and writing 0 to there does not work.
These rules put the drive into read-only state once after boot (albeit revertable through explicit commands). The read-only state could be verified from the ro sysfs virtual file:
> ls -l /dev/disk/by-id/nvme-{HYV2TBX4_2280__XXXXXXXXXXXXXXX,GeIL_P4S_4TB_XXXXXXXXXXX,Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX}
lrwxrwxrwx 1 root root 13 Jul 28 17:17 /dev/disk/by-id/nvme-GeIL_P4S_4TB_XXXXXXXXXXX -> ../../nvme1n1
lrwxrwxrwx 1 root root 13 Jul 28 17:17 /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXX -> ../../nvme0n1
lrwxrwxrwx 1 root root 13 Jul 28 17:17 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX -> ../../nvme2n1
> cat /sys/block/nvme*n1/ro
0
1
1
One can write to these drives and confirm all writes would fail.
The read-only drives then can be assigned to the VM without worrying about they getting written, either by writing the path explicitly in virt-manager GUI, or with a disk node similar to the following one in XML / virsh edit:
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
<source dev='/dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX'/>
<target dev='vdc' bus='virtio'/>
<readonly/>
<address type='pci' domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
</disk>
Note that the readonly state must be marked explicitly (also could through virt-manager GUI), otherwise the VM manager could not open the corresponding block device. The VM itself and the system running inside therefore knows the fact that the drive is read-only.
While Windows could mount such read-only drive, and content could be indexed and read, many games don’t like their executable residing in a read-only folder. If one would want to play such game from the read-only drive, the only possible workaround is to copy the whole thing into the writable VM drive.
If only we could fake the writability of such read-only drive and make the VM believe they can write something onto it. And it’s natural to come up the idea to lay some writable block layer on top of such read-only block, to get a writable block.
Luckily the device-mapper system in Linux is quite powerful and provides many useful “target”s, among them the dm-snapshot provides just what we need. The idea is pretty similay to overlayfs albeit on the block device level: it combines a optionally read-only low layer, a optionally persistent copy-on-write layer, to get a read-write block device, that starts as a merely “clone” of the low layer, and writes would only hit the CoW layer, not the bottm layer.
To assemble such dm, a command looks like the following:
sudo dmsetup create winData --table '0 8001573552 snapshot /dev/disk/by-id/nvme-GeIL_P4S_4TB_XXXXXXXXXXX /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part5 P 128'
The arguments might look frightening and hard to understand at the first glance, but it’s pretty structural:
dmsetup create to create a dmwinData is the name of the to-be-created dm, a symlink /dev/mapper/winData would be created to point to the actual numbered device (usually dm0)--table begins the dm table declaration as a single long string as the following argumnent, and in it:
0 marks the start sector of first target8001573552 marks the end sector of first target; the two combined means the assembled dm block device would have its 0 to 8001573552 - 1 sector “mapped” to the currrently declared target, with details coming next to it; so the size is also 8001573552 (end-begin), which could be obtained by cat /sys/block/nvmeXn1/sizesnapshot sets the current dm target to dm-snapshot, and later table arguments should define its target-specific setting/dev/disk/by-id/nvme-GeIL_P4S_4TB_XXXXXXXXXXX is the bottom layer, no write would hit it; in this case it’s the 4TB physical drive/dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part5 is the Copy-on-Write layer, all writes to the assembled dm would only hit this layer, with the whole chunk copied to there first before being modified; in this case it’s a 128GB physical partition, but it could also be a LVM LV, a loopback file, a RAM disk, or any block you can spareP tells the CoW layer is persistent; i.e. on re-assembling after physical reboot, the assembled dm should still look the same on binary perspective; I chose the data drive as persistent as maybe sometimes I would store games here rather than the VM system drive, and I don’t want it gone away right after a physical reboot; if the CoW layer shall be dropped, it could easily be done by blkdiscard -f on the CoW layer without the dm assembled.128 is the count of sectors as chunk size, with a 512 sector size this means 512 byte/sector * 128 sector = 65536 byte = 64 KiB chunk size, which should be a friendly IO size for SSD, and reduces the metadata; the dm-snapshot documentation wrote 16 in its example for lvm snapshot, so you can choose freelyFor a non-persistent case, the command is similar except the persistent mark is N
sudo dmsetup create winSystem --table '0 3907029168 snapshot /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part4 N 128'
After a full assembling, the lsblk output would look like the following:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme2n1 259:0 0 1.8T 1 disk
├─winSystem 254:0 0 1.8T 0 dm
├─nvme2n1p1 259:1 0 100M 1 part
└─nvme2n1p2 259:2 0 1.8T 1 part
nvme1n1 259:3 0 3.7T 1 disk
├─winData 254:1 0 3.7T 0 dm
├─nvme1n1p1 259:4 0 16M 1 part
└─nvme1n1p2 259:5 0 3.7T 1 part
nvme0n1 259:6 0 1.9T 0 disk
├─......
├─nvme0n1p4 259:10 0 32G 0 part
│ └─winSystem 254:0 0 1.8T 0 dm
├─nvme0n1p5 259:11 0 128G 0 part
│ └─winData 254:1 0 3.7T 0 dm
└─......
Note the assmebled block device does not have their partitton identified, which is by design, and for a VM access this is pretty OK. If you want the partitions however, this can be fixed with a manual kpartx call, e.g.
sudo kpartx -a /dev/mapper/winSystem
Then the lsblk output would look like the following:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme2n1 259:0 0 1.8T 1 disk
├─winSystem 254:0 0 1.8T 0 dm
│ ├─winSystem1 254:2 0 100M 0 part
│ └─winSystem2 254:3 0 1.8T 0 part
├─nvme2n1p1 259:1 0 100M 1 part
└─nvme2n1p2 259:2 0 1.8T 1 part
nvme1n1 259:3 0 3.7T 1 disk
├─winData 254:1 0 3.7T 0 dm
├─nvme1n1p1 259:4 0 16M 1 part
└─nvme1n1p2 259:5 0 3.7T 1 part
nvme0n1 259:6 0 1.9T 0 disk
├─...
├─nvme0n1p4 259:10 0 32G 0 part
│ └─winSystem 254:0 0 1.8T 0 dm
│ ├─winSystem1 254:2 0 100M 0 part
│ └─winSystem2 254:3 0 1.8T 0 part
├─nvme0n1p5 259:11 0 128G 0 part
│ └─winData 254:1 0 3.7T 0 dm
└─...
In this case, the writability of the dm-snapshot can be verified under Linux by simply mounting the NTFS partition with ntfs-3g, and writting something into it. And after umounting it you could mount the lower read-only NTFS partition to confirm no write hit it.
The VM can then use such pseudo writable drive, e.g.
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
<source dev='/dev/mapper/winSystem'/>
<target dev='vdc' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
<source dev='/dev/mapper/winData'/>
<target dev='vdd' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
</disk>
Some systemd service units could also be used if the dm-snapshot blocks shall be ready upon boot:
> cat /etc/systemd/system/dmsetup-winSystem.service
[Unit]
Description=Setup winSystem device with dmsetup
Requires=dev-disk-by\x2did-nvme\x2dSamsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX.device
After=dev-disk-by\x2did-nvme\x2dSamsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX.device
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/dmsetup create winSystem --table '0 3907029168 snapshot /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_2TB_XXXXXXXXXXXXXXX /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part4 N 128'
[Install]
WantedBy=local-fs.target
> cat /etc/systemd/system/dmsetup-winData.service
[Unit]
Description=Setup winData device with dmsetup
Requires=dev-disk-by\x2did-nvme\x2dGeIL_P4S_4TB_XXXXXXXXXXX.device
After=dev-disk-by\x2did-nvme\x2dGeIL_P4S_4TB_XXXXXXXXXXX.device
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/dmsetup create winData --table '0 8001573552 snapshot /dev/disk/by-id/nvme-GeIL_P4S_4TB_XXXXXXXXXXX /dev/disk/by-id/nvme-HYV2TBX4_2280__XXXXXXXXXXXXXXXX-part5 P 128'
[Install]
WantedBy=local-fs.target
The Windows physical drives therefore, while read-only, appears writable in the Windwos VM, and upon physical Debian host reboot, changes made into the physical Windows system snapshot are gone, but made into the physical Windows data snapshot still persist but could also be throw away with blkdiscard, both physical drives always unmodified.
From OpenWrt 25.12 onwards, and available earlier in release candidates and development branches, OpenWrt has swapped their package manager + package format from opkg + .ipk (almost a .deb) to apk-tools v3.0 from Alpine Linux and its newly introduced APK v3. Even Alpine itself has just upgraded apk-tools to v3.0 since v3.23 but not fully switched into apk v3 yet.
The format is a totally new format designed in-house by Alpine with some collabration from OpenWrt. But let’s first list the above mentioned older formats:
deb is an ar like format with pre-defined members debian-binary, data.tar(.gz/.xxx), control.tar(.gz/.xxx), the files are in data.tar.xxx and metadata is in control.tar.xxxipk is pretty much a fixed version deb with in most cases only gzip compressed data and control so less dependency is needed which is useful in embedded scenarios.apk v2 is three (with signature) or two gzipped (without signature) tars, one optionally for signature, one for metadata (kinda like control.tar in deb) and one for actual files (kinda like data.tar in deb)apk v3 however is in a supposedly schema-based binary format adb, loosely defined by its official manual (source / online), yet most details are determined by apk-tools the only reference tool’s C source code.
I’ve recently wrote a tool adumpk to parse an v3 .apk, prints useful info about it, optionally convert it to .tar, and write metainfo into .json. When writing the tool I had to jump around in apk-tools’s source code quite often. After finishing adumpk, I decided to write this as an easier-to-follow single blog post so others can avoid the hassle.
Unless explicitly mentioned, all integral data types in the format is little-endian.
The below uses openwrt/25.12.0-rc5/crowdsec-1.6.2-r1.apk as an example package, although it’s not needed, it’s recommended you get one so it’s easier to examine the binary by yourself and learn the actual binary format.
The file header is 4 bytes in length, first 3 bytes being magic "ADB", i.e. big-endian 0x414442, then the last byte is either one of the following to tell the file compression method:
., i.e. 0x2e, means the content is not compressedd, i.e. 0x64, means the conetnt is compressed with Deflate level 0c, i.e. 0x63, means the content is compressed with a custom compression method recorded in the next 2 bytes
u8 recording the ID of compression method:
apk-tools was compiled with HAVE_ZSTDu8 containing compression level, the level allowed by each of the above method is:
The compressed ADB data body either starts from the 7th byte (file[6:]) if the compression method is c and 2 addtional bytes were took to record the actual compression method, or the 5th byte (file[4:]) otherwise.
The compressed ADB data body shall be a plain stream without magic header you would expect from a standalone compressed file, so the gzip stream does not begin with big-endian 0x1F8B magic then meta, and zstd stream does not begin with big-endian 0x28B52FFD magic.
Note while Deflate stream can be concatenated to each other, which is used in apk v2, the apk v3 body should be a whole continous Deflate stream if it was compressed with such algorithm. I didn’t test this but I think the official tool would happily accept a manually prepared apk v3 with multiple concatenated Deflate stream. Zstandard on the other hand does not support concatenated streams.
The body, when decompressed, shall begin with exactly ADB., same as uncompressed apk v3, 3-byte magic and 1-byte marking no compression.
With the example apk the body shall be decompressed with:
import zlib
with open('crowdsec-1.6.2-r1.apk', 'rb') as f:
assert(f.read(4) == b'ADBd')
body = zlib.decompress(f.read(), wbits=-zlib.MAX_WBITS)
assert(body[0:8] == b'ADB.pckg')
After the leading ADB. in either the original uncompressed file or the decompresssed body, a 4-byte (u32(body[4:8])) magic (called schema) would mark the inner data as one of the following:
0x676B6370 or big-endian 0x70636B67 or literal pckg for package0x78646E69 or big-endian 0x696E6478 or literal indx for indexWe would only talk about package.
In the package case, a series of ADB blocks would continue one after another, each starting at a 8-byte boundary (the first ADB block naturally starts at such boundary as it’s after 8-byte "ADB.pckg").
Each block starts with a u32 recording the type and optionally the size, itself as either a simple header, or as the first member of a bigger 16-byte header
1 (so either 0b00, 0b01, 0b10, but not 0b11), then this is a simple 4-byte header
v >> 30v & 0x3fffffff1, then this is an extended 16-byte header, including the 4-byte u32 v itself as type_size field, a 4-byte u32 reserved field for alignment for future expansion, and a 8-byte u64 x_size field, defined in C as:
struct adb_block {
uint32_t type_size;
uint32_t reserved;
uint64_t x_size;
};
i.e. u32 & 0x3fffffffx_size fieldThe payload size of what follows the header can be calculated as raw size - header size, and it must be non-negative.
The actual type of a block must be one of the following:
| ID | NAME | Usage |
|---|---|---|
| 0 | ADB_BLOCK_ADB | essential, contains metadata info and file infos |
| 1 | ADB_BLOCK_SIG | optional, carries signature data |
| 2 | ADB_BLOCK_DATA | technically also optional, carries file content (no name or path) or alike |
The order of these blocks is restricted. A sane ADB must contains these blocks head to tail:
ADB_BLOCK_ADB for metadataADB_BLOCK_SIG for signatureADB_BLOCK_DATA for dataIf the order is not respected (e.g. SIG after DATA, or ADB after SIG/DATA, etc ), then the file would be rejected
ADB_BLOCK_ADBSuch block must be the first block in one package’s blocks stream.
The block begins with a 8-byte header, including a u8 adb_compat_ver field (currently must be 0), a u8 adb_ver field (currently also must be 0), a u16 reserved field for alignment and future expansion (all 0), then a u32 root field to declare the following data streams, type aliased as adb_val_t, defined in C as:
struct adb_hdr {
uint8_t adb_compat_ver;
uint8_t adb_ver;
uint16_t reserved;
adb_val_t root;
};
An adb_val_t val carries type info in its highest 4 bits (val & 0xf0000000), and value in lowest 28 bits (val & 0x0fffffff)
Let’s focus on what root means here.
The type shall be ADB_TYPE_OBJECT (0xe0000000 == root & 0xf0000000) for a series of elements each with their own types, which makes sense for a metadata block.
And the value, i.e. root & 0x0fffffff, marks the offset of targets inside current payload, note the current payload includes the above adb_hdr but not the block header, so e.g. with u32([body[8:12]]) == 0x1614 == 5652 for a simple 4-byte header, type ADB, this means the whole block is at body[8:8+5652] == body[8:5660], so payload is body[8+4:5660] == body[12:12+5648] without the block header, including the adb_hdr at body[12:12+8] == body[12:20], so with root==0xe0001600 the offset would be 0x1600 == 5632, and therefore we need to go from body[12+5632].
The pointed-to ADB object elements for root starts with a u32 recording how many adb_val_ts follow it including itself, then the actual series of adb_val_ts, so e.g. with the above example and u32(body[5644:5648]) being 4, then the 3 adb_val_ts are expected at body[5648:5652], body[5652:5656], body[5656:5660].
Note the last adb_val_t is just at the end of this payload / ADB_BLOCK_ADB, you can easily tell that the root value was written at the end of ADB_BLOCK_ADB creation
The above 3 objects with ID starting at 1, alongside count u32 (4) with ID 0 (remember this convention, APK prefers to use id 0 for num/count and 1 onwards as actual slots), can be considered a 4-length adb_val_t array, each latter member is used for a different purpose:
| ID | NAME | Purpose |
|---|---|---|
| 1 | ADBI_PKG_PKGINFO | package info metadata |
| 2 | ADBI_PKG_PATHS | package file / folder paths (compacted than plain list of texts) |
| 3 | ADBI_PKG_SCRIPTS | package postrm / preinst / etc scripts |
| 4 | ADBI_PKG_TRIGGERS | package triggers |
If a slot is not needed then it can be 0 if there’s still slot needed after it, or not exist at all if there’s no slot needed after it. In this example with count being 4 there would only be 3 slots, so no TRIGGERS was stored.
We would only focus on PKGINFO and PATHS, as SCRIPTS and TRIGGERS pretty much follows the idea of PATHS + ADB_BLOCK_DATA
ID 1 in root object, which marks the head of pacakge info, is yet another object, e.g. root_obj[1] == body[5648:5652] == 0xe00012b8, means an object (0xe00012b8 & 0xf0000000 == 0xe0000000) with offset 4792 (0xe00012b8 & 0x0fffffff == 0x12b8 == 4792), which then points to another u32 for number for elements including itself. So e.g. u32(body[12+4792:+4]) == u32(body[4804:4808]) == 17 means there’re 16 adb_val_t after the count u32, 1st at body[4804+4*1:+4] == body[4808:4812] and 16th at body[4804+4*16:+4] == body[4868:4872].
Now is a good time to list all possible data types, which can all be possibly used in these fields:
| Type | Magic | Note |
|---|---|---|
| ADB_TYPE_SPECIAL | 0x00000000 | Currently just alias to INT |
| ADB_TYPE_INT | 0x10000000 | Single u32 (max 0x0fffffff) value in low |
| ADB_TYPE_INT_32 | 0x20000000 | Single u32 at low-as-off |
| ADB_TYPE_INT_64 | 0x30000000 | Single u64 at low-as-off |
| ADB_TYPE_BLOB_8 | 0x80000000 | Series of u8, length (u8) + data (u8s) at low-as-off |
| ADB_TYPE_BLOB_16 | 0x90000000 | Series of u8, length (u16) + data (u8s) at low-as-off |
| ADB_TYPE_BLOB_32 | 0xa0000000 | Series of u8, length (u32) + data (u8s) at low-as-off |
| ADB_TYPE_ARRAY | 0xd0000000 | Series of same type, length (u32) + data at low-as-off |
| ADB_TYPE_OBJECT | 0xe0000000 | Series of different type, length (u32) + data at low-as-off |
As we briefly mentioned earlier, as each element does not record its own ID yet the ID has special meaning, to mark an empty, skipped element, the elements shall be special value 0; e.g. if a package has only a ID 6 field to declare, then it would have first 5 slots all set to 0 so they’re skipped, and there would be no ID 7 field onwards.
These slots are numbered as follows:
| ID | Name | Data Type |
|---|---|---|
| 1 | NAME | BLOB, usually BLOB_8, as string |
| 2 | VERSION | BLOB, usually BLOB_8, as string |
| 3 | HASHES | BLOB, usually BLOB_8, as hex-string |
| 4 | DESCRIPTION | BLOB, usually BLOB_8, as string |
| 5 | ARCH | BLOB, usually BLOB_8, as string |
| 6 | LICENSE | BLOB, usually BLOB_8, as string |
| 7 | ORIGIN | BLOB, usually BLOB_8, as string |
| 8 | MAINTAINER | BLOB, usually BLOB_8, as string |
| 9 | URL | BLOB, usually BLOB_8, as string |
| 10 | REPO_COMMIT | BLOB, usually BLOB_8, as hex-string |
| 11 | BUILD_TIME | INT, usually embedded |
| 12 | INSTALLED_SIZE | INT, usually embedded |
| 13 | FILE_SIZE | INT, usually embedded |
| 14 | PROVIDER_PRIORITY | INT, usually embedded |
| 15 | DEPENDS | OBJECT of dependency (see below) |
| 16 | PROVIDES | OBJECT of dependency (see below) |
| 17 | REPLACES | OBJECT of dependency (see below) |
| 18 | INSTALL_IF | OBJECT of dependency (see below) |
| 19 | RECOMMENDS | OBJECT of dependency (see below) |
| 20 | LAYER | INT, usually embedded |
| 21 | TAGS | OBJECT of BLOB, usually BLOB_8, as string array |
So e.g. the first element, u32(body[4808:4812]) == 0x80000008, is not zero, means the package has an actual name, the high 0x8 means this is a BLOB_8 item, and the low 0x8 means the item’s length is at offset 8 and content starts at offset 9, so, length == u8(body[12+8]) == 8, and therefore content is at body[12+9:12+9+8] == body[21:29], the example package is crowdsec
And e.g. the last element ID 16 u32(body[4868:4872]) == 0xe0000184, is not zero, means the package has an actual provides OBJECT (0xe0...), offset 0x184 == 388 in payload, so u32(body[12+388:12+388+4]) == u32(body[400:404]) == 2 records the number of sub-elements including the count itself, therefore 1 actual element, adb_val_t(body[404:408]) therefore records the info of the only sub-element, here being 0xe000017c means it’s yet another OBJECT starting at offset 0x17c, …, and then u32(body[392:396]) == 2 so there’s again 1 real sub-element, adb_val_t(body[396:400]) == 0x8000016c means this is a BLOB_8 starting at 0x16c, then we get length from u8(body[12+0x16c]) == 12, and finally the provide item at body[12+0x16c+1:+12] == body[377:389], being crowdsec-any
While the provides seems a list of list of BLOB_8, but recall that OBJECT elements can be different types, each element in provides is actually a strongly-typed dep info, containing the NAME slot (BLOB_8, ID1), VERSION slot (BLOB_8, ID2), and MATCH slot (INT, ID3, for vercmp operations). In the example there’s just no VERSION nor MATCH.
All dependency-like element can have these 3 slots:
| ID | Name | Data Type |
|---|---|---|
| 1 | NAME | BLOB, usually BLOB_8, string |
| 2 | VERSION | BLOB, usually BLOB_8, string |
| 3 | MATCH | INT, usually embedded |
The MATCH field is a bitwise OR of the following base bits:
| NAME | VALUE | bit |
|---|---|---|
| EQUAL | 1 | 0b00001 |
| LESS | 2 | 0b00010 |
| GREATER | 4 | 0b00100 |
| FUZZY | 8 | 0b01000 |
| CONFLICT | 16 | 0b10000 |
The implementation would pre-calculate all valid combinations of these; these are (excluding CONFLICT which can be freely appended):
| Sign | Meaning | Bits |
|---|---|---|
| < | Less than | 0b0010 |
| <= | Less than or equal to | 0b0011 |
| <~ | less than or equal to, fuzzy | 0b1011 |
| ~ | Equal to, fuzzy | 0b1001 |
| = | Equal to | 0b0001 |
| >= | Greater than or equal to | 0b0101 |
| >~ | Greater than or equal to, fuzzy | 0b1101 |
| > | Greater than | 0b0100 |
| >< | Special, checksum | 0b0110 |
| _ | Special, any | 0b0111 |
So e.g. a match field with value 0x10000003 would mean INT with value 0x3 == 0b11, therefore <=
Files and folders are stored in ADB_BLOCK_ADB in a compact way, before the latter possible file data appearance in ADB_BLOCK_DATA blocks. Each of these path elements stores a folder path without the leading / (empty path for root folder), then any number of direct file entries. While most of the file entries do need their ADB_BLOCK_DATA block for actual data, others could exist purely in header.
ID 2 in ADB_BLOCK_ADB’s root object, which marks the head of paths, is yet another object, e.g. root_obj[2] == body[5652:5656] == 0xe00012fc, means an object with offset 0x12fc, and u32(body[12+0x12fc:+4]) == u32(body[4872:4876]) == 22 means there’re 21 adb_val_t after the count u32, 1st at body[4872+4*1:+4] == body[4876:4880] and 21st at body[4872+4*21:+4] == body[4956:4960]
Each one of these 21 “path”s is actually called ADBI_DI by apk-tools, and is also an OBJECT with the following slots (still, some are optional):
| ID | NAME | Data Type |
|---|---|---|
| 1 | NAME | BLOB, usually BLOB_8, string |
| 2 | ACL | OBJECT being ACL info (see below) |
| 3 | FILES | OBJECT of File info (see below) |
In the example the last “path” adb_val_t(body[4956:4960]) == 0xe0001290, so it’s an OBJECT starting from 12 + 0x1290 == 4764, as u32(body[4764:4768]) == 4 there’re 3 slots after it.
For ID 1, NAME, adb_val_t(body[4768:4772]) == 0x80001258, it’s a BLOB_8 starting at offset 0x1258, and u8(body[12+0x1258]) == u8(body[4708]) == 7 says this is a 7-length string, content at body[4708+1:7] == body[4709:4716] == b"usr/bin" says the folder name/path is usr/bin
For ID2, ACL, adb_val_t(body[4772:4776]) == 0xe0000194, it’s an OBJECT starting at offset 0x194, the count u32(body[12+0x194:+4]) == u32(body[416:420]) == 4 so there’re 3 slots after it.
The ACL info OBJECT could have the following slots:
| ID | NAME | Data Type |
|---|---|---|
| 1 | MODE | INT, usually embedded |
| 2 | USER | BLOB, usually BLOB_8, string |
| 3 | GROUP | BLOB, usually BLOB_8, string |
| 4 | XATTRS | OBJECT of BLOB, usually BLOB_8, each BLOB_8 with \0 as sep for name and value |
In the example:
0x100001ed so it’s an INT with value 0x1ed == 0o7550x8000018c so it’s an BLOB_8 starting at offset 0x18c, length at body[408] reads 4 and content at body[409:413] reads rootroot from USERFor ID3, FILES, adb_val_t(body[4776:4780]) == 0xe0001260, it’s an OBJECT starting at offset 0x1260, the count u32(body[12+0x1260:+4]) == u32(body[4716:4720]) == 12, so there’re 11 file entries after it, the first at body[4720:4724] and the last at body[4760:4764].
The first file entry, adb_val_t(body[4720:4724]) == 0xe0000f4c, it’s an OBJECT starting at offset 0xf4c, the count u32(body[12+0xf4c:+4]) == u32(body[3928:3932]) == 6, so there’re 5 slots after it.
The File info OBJECT could have the following slots:
| ID | NAME | Data Type |
|---|---|---|
| 1 | NAME | BLOB, usually BLOB_8, string |
| 2 | ACL | OBJECT being ACL info (see above) |
| 3 | SIZE | INT, usually embedded |
| 4 | MTIME | INT, usually INT32 |
| 5 | HASHES | BLOB, usually BLOB_8, hex-string |
| 6 | TARGET | BLOB, usually BLOB_8, string |
In the example:
0x80000008 so it’s an BLOB_8 with length at offset 8, u8(body[12+8]) == 8, and content body[12+8+1:+8] == b"crowdsec"0xe0000194 so it’s again 0o755 owned by root:root0x133cd7e8 so it’s an INT with value 0x33cd7e8 == 543190800x200001e4 so it’s an INT_32 at offset 0x1e4, and u32(body[12+0x1e4:+4]) == 1772344484 so mtime is Sun Mar 1 05:54:44 UTC 20260x80000f28 so it’s an BLOB_8 with length at offset 0xf28, u8(body[12+0xf28]) == 32, and content body[12+0xf28+1:+32] reads a hex-string, which is the SHA256 checksum of the fileA file can have its SIZE set to 0, being an empty file, and on top of it having TARGET set, so it either serves as a symlink or hardlink to the set target, or is a special CHAR / DEV.
Let’s use the third file entry under the same last path entry to examine what TARGET does, which is adb_val_t(body[4728:4732]) == 0xe0000fdc, it’s an OBJECT with offset 0xfdc, we read u32(body[12+0xfdc:+4]) == u32(body[4072:4076]) == 7 so it does have 6th slot for TARGET; we read the name adb_val_t(body[4076:4090]) == 0x80000fac so name starts at offset 0xfac and len == u8(body[12+0xfac]) == u8(body[4024]) == 5, content is body[4024+1:+5] == body[4025:4030] == b"cscli", so the link itself is usr/bin/cscli; we skip to SLOT 6 for TARGET which should be adb_val_t(body[12 + 0xfdc + 4 * 6:+4]) == adb_val_t(body[4096:4100]) == 0x80000fb2, so it’s a BLOB with offset 0xfb2, then we read length at u8(body[12+0xfb2]) == u8(body[4030]) == 23 so content is body[4030+1:+23] == body[4031:4054] == b"\x00\xa0/usr/bin/crowdsec-cli"
The first two bytes in the TARGET determines the data type, and they shall be handled as one u16, and u16(body[4031:4033]) == 40960 == 0o120000, this is basically the same thing as you would expect from the st_mode field in a struct stat with already S_IFMT been bitwise AND. The following file type are supported:
| Type | Mask | Content at target[2:] |
|---|---|---|
| S_IFBLK | 0o060000 | 8-byte, as u64 for dev ID (major:minor combined) |
| S_IFCHR | 0o020000 | 8-byte, as u64 for dev ID (major:minor combined) |
| S_IFIFO | 0o010000 | 8-byte, as u64 for dev ID (major:minor combined) |
| S_IFLNK | 0o120000 | any-length, for symlink target |
| S_IFREG | 0o100000 | any-length, for hardlink target |
The real target for symlink is therefore target[2:], so we know this symlink is usr/bin/csli -> /usr/bin/crowdsec-cli
When reading through the PATHs info it’s recommended to store them for later lookup, as the ADB_BLOCK_DATA blocks would only carry the file content, but not the names, paths, ownership, etc.
ID 3 in the ADB_BLOCK_ADB is ADBI_PKG_SCRIPTS which is an OBJECT with multiple BLOBs for package pre/post scripts.
e.g. root_obj[3] == body[5656:5660] == 0xe00015e0, means an object with offset 0x15e0, and u32(body[12+0x15e0:+4]) == u32(body[5612:5616]) == 8 means there’re 7 adb_val_t after the count u32, 1st at body[5612+4*1:+4] == body[5616:5620] and 21st at body[5612+4*7:+4] == body[5640:5644]
The scripts OBJECT could have the following slots
| ID | NAME |
| 1 | TRIGGER |
| 2 | PREINST |
| 3 | POSTINST |
| 4 | PREDEINST |
| 5 | POSTDEINST |
| 6 | PREUPGRADE |
| 7 | POSTUPGRADE |
All names except TRIGGER should tell the purpose just by its name. The TRIGGER one is special as it would be triggered on changes to paths listed in latter ADBI_PKG_TRIGGERS.
The example package has only 3/POSTINST, 4/PREDEINST and 7/POSTUPGRADE. Take the last slot for example, u32(body[5640:5644]) reads 0x800014e4 so it’s BLOB_8 with offset 0x14e4, read u8(body[12+0x14e4]) == 251 so length is 251, therefore content is body[12+0x14e4+1:+251] == body[5361:5612], b'#!/bin/sh\nexport PKG_UPGRADE=1\n[ "${IPKG_NO_SCRIPT}" = "1" ] && exit 0\n[ -s ${IPKG_INSTROOT}/lib/functions.sh ] || exit 0\n. ${IPKG_INSTROOT}/lib/functions.sh\nexport root="${IPKG_INSTROOT}"\nexport pkgname="crowdsec"\nadd_group_and_user\ndefault_postinst\n', which prints as:
#!/bin/sh
export PKG_UPGRADE=1
[ "${IPKG_NO_SCRIPT}" = "1" ] && exit 0
[ -s ${IPKG_INSTROOT}/lib/functions.sh ] || exit 0
. ${IPKG_INSTROOT}/lib/functions.sh
export root="${IPKG_INSTROOT}"
export pkgname="crowdsec"
add_group_and_user
default_postinst
ID 4 in the ADB_BLOCK_ADB is ADBI_PKG_TRIGGERS which is an OBJECT with multiple BLOBs that shall trigger the TRIGGER script to run.
Note it is totally valid that a package does not have TRIGGER script yet has multiuple TRIGGERS paths.
ADB_BLOCK_SIGSuch block must be after ADB_BLOCK_ADB and before ADB_BLOCK_DATA.
The block begins with a 2-byte header, including a u8 sign_ver field for the version of signature (currently must be 0), and a u8 hash_alg field for the ID of the algorithm, defined in C as:
struct adb_sign_hdr {
uint8_t sign_ver, hash_alg;
};
The hash algorithm could be one of the following:
| ID | NAME | LENGTH |
|---|---|---|
| 0 | NONE | - |
| 2 | SHA1 | 20 |
| 3 | SHA256 | 32 |
| 4 | SHA512 | 64 |
| 5 | SHA256_160 (actually SHA2 160 variant) | 20 |
The missing ID1 was MD5 whose support was dropped in apk-tools.
And currently apk-tools would only use SHA512 for both signing and verifying.
If this have a valid, non-NONE hash_alg, then the actual payload should (after the 2-byte header) be followed by a 16-byte ID, and the corresponding length of signature, defined in C as:
struct adb_sign_v0 {
struct adb_sign_hdr hdr;
uint8_t id[16];
uint8_t sig[];
};
In the example file there’s no SIG block.
When testing signing with apk-tools (which can re-sign an unsigned v3 apk), the private key file --sign-key shall be an OpenSSL private key, which could be generated via e.g. openssl genrsa -aes256 -out /tmp/private.pem 4096, in which -aes256 could be omitted if you don’t want password. However as this is a temporary key not in pool, the command should look like apk adbsign --allow-untrusted --sign-key /tmp/private.pem crowdsec-1.6.2-r1.apk.resigned
The size of sig[] part shall follow what the key specifies, e.g. for the above rsa4096 key, the signature shall be 512 bytes, and it might be PKCS#1 message but as this is only testing with temporary key I can’t confirm the official repo signing method.
The following is output from adumpk:
DEBUG... AdbBlock(type_block=<AdbBlockType.SIG: 1>, size_raw=534, size_payload=530)
INFO.... Hash sha512, 64 bytes, ID 290fb2a94d29dda681301285226e604d: CQT7OfNPmxt6XtW3s1iV5N6DtGlfkVYKYsjKn4LKsRmYW0RjTXhZ12bexzmcx7zIQqs9VMZYyN9ovCobYhnUDikR5an2FoUYIJ9oJAEm3FdS1Q5L0m7mSqssO6SP/Y8dK7G1wgnlvTLgKOQ4gWjVogOLCDFk2j/B15NmGMS3rS7hcYNPhn7SuDTBMzNM6jMNoe0ElYznFCZYEUw89Ow1rD602/sIhO6eZwuTrgsFBq6dBLLiOZ863ufiKnUVNW1PijmdPh730L8aqnlm1Jdro+eN4A5Af5zDsqobPaRlE1Rs/7UzTBozDAIcoPWTjtVkBUqEw8SWMdeAQnlBKkiOGmq5uGsM/KvgZb+NthME5YcsbWJLineVCuZ/iVZCAtSbKvFlPKRpwk385YnA/LMfdIuR7dsZQLjpzEdgYC5/57O/CWOs7WvBI4jXi0wiTqbEHKKSHhlmnJI7DdTwAesE86G5lgxqamnxIuG9xjD6Cm6l9fPYR3dcVAFl76FuLSLzDuT4J51o48F4MvlyfJIt5a+Thoknvhcg4OXEAJMg5tOc5uWU+TV1cllLqkeyAh1qxUCbol4mU5ZLctgMYGsSnCxISuDXNDy6k6D/m3ilz+9BOIrfKM2C6z7SBvCzmoezCMkr2oBdGHbgguSj9vkwwLXHzMbY7AZXRb0UQ3fIml4=
ADB_BLOCK_DATASuch block must be after ADB_BLOCK_ADB and cannot be before ADB_BLOCK_SIG
The block begins with a 8-byte header, including a u32 path_idx field for the 1-started ID of corresponding PATH element, and a u32 file_idx field for the 1-started ID of corresponding FILE element in that PATH, defined in C as:
struct adb_data_package {
uint32_t path_idx;
uint32_t file_idx;
};
The file data follows directly after the header. Remember that the payload contains the header and each block starts at 8-byte boundary and aligns up to 8-byte boundary.
E.g. in the example file, right after ADB_BLOCK_ADB at body[8:5660], pad that to 8-byte boundary 5664, and reads the 4-byte type_size u32(body[5664:5668]) == 0x8000007f, so type for it is 0x8000007f >> 30 == 2, for a ADB_BLOCK_DATA, and size for the payload including header is 0x7f, 127, therefore the whole payload including header is body[5664:+127] = body[5664:5791]. In it, the path_idx is u32(body[5668:5672]) == 3, and file_idx is u32(body[5672:5676]) == 1, and actual data length is 127 (whole block) - 4 (block header) - 8 (data header) = 115, and we can confirm it’s body[5676:5791] = body[5676:+115].
The file content reads as below:
config crowdsec 'crowdsec'
option data_dir '/srv/crowdsec/data'
option db_path '/srv/crowdsec/data/crowdsec.db'
And if we go back to look at the PATH block, we would know this is for folder etc/config and file crowdsec, perm 0o600 owned by root:root, with SHA256 checksum.
While I know these could be improved if I have a dedicated persistent fs for configs, or cloud-init scripts, or archiso boot parameters, I don’t quite want immutable live system for the work any more. So, I decided, what if I have a single drive, on which I have all of the following systems booting from the same root partition?
Before the actual installation, I’ll explain the background knowledge first. If you don’t bother with, skip to next chapter
On x86_64 UEFI, without CSM, the booting process is quite simple:
Note about MBR on UEFI: as the specification only required support for GPT, the support or no-support for MBR is undeterminable before you get your hands on the actual machine. Windows and systemd-boot simply refuses to install on MBR on UEFI. While all of my devices support such and I use it for local system installation on small drives, I would only focus on GPT on UEFI due to the fact that we want the result drive bootable on variuos machines.
Most of UEFI-compatible boot managers support to be (or has to be at least) installed at removable path. E.g. for grub (expecting EFI partiton mounted at /efi):
grub-install --target x86_64-efi --removable
On Debian 13 this installs the following files to /efi/EFI/BOOT:
BOOTX64.CSV: this contains required entry to be registered once shim at BOOTX64.EFI successfully bootedBOOTX64.EFI: Debian’s signed shim, loads signed grubx64.efi, and register entries according to BOOTX64.CSV; the one UEFI firmware would pick as removable EFIgrub.cfg: Grub’s config, it just tells grub to scan for real root and look up configs there. An example content:
search.fs_uuid 91c69930-b508-4a42-b510-d63544d7eae0 root hd1,gpt2
set prefix=($root)'/boot/grub'
configfile $prefix/grub.cfg
Grub’s config files look like shell scripts and you can imagine configfile as source in shell, so in this case the grub.cfg in EFI partition just records where to find the root partition (search for fs with uuid 91c69930-b508-4a42-b510-d63544d7eae0 and records the result in variable root, if failed then use default hd1,gpt2), sets another variable prefix, being the path to folder /boot/grub under that root fs, then “source” another config grub.cfg under there.
grubx64.efi: Grub’s core EFI binary, signed by Debian, loads grub.cfg
$prefix variable equalling /EFI/debian to instruct where to look up for a grub folder; in removable case it would instead try [ESP]/EFI/BOOT, so it looks up grub.cfg heremmx64.efi: Machine owner key manager, only needed for secure boot, not needed for fully removable use cases, can be safely deletedOn Arch Linux this installs only BOOTX64.EFI, even the config needs to be manually created.
For a removable drive, on which we would have (some) kernels unsigned, we certainly would not want strict Secure Boot, neither would we want permissive Secure Boot with Machine Owner Key managed by ourselves. And we would not want it to register non-removable UEFI BootEntry s if possible.
The dependency tree in Debain’s Grub split packages make it really hard to install secure-boot-less under UEFI, as grub-efi-amd64-bin, a soft depened making grub-install --target x86_64-efi possible, hard depends grub-efi-amd64-signed. And the whole dependency tree becomes locked-in and almost impossible to uninstall due to they considered “essential”. And dpkg hook would “friendly” help use to re-install (update) Grub on version change, not respecting the existing layout. So in later steps we would install and manage Grub the boot loader part from Arch Linux, and install only the booting configuration generation part on Debian.
Don’t worry about “Debian not having its own Grub”. It is $prefix/grub.cfg that normally grub-mkconfig / update-grub updates, these are managed by sytem themselves, and the configuration tool would still be installed.
Other than the per-system grub.cfg, we would want an “outer grub.cfg”, which is directly used by Grub, containing menuentry to instruct which sub grub.cfg to redirect to.
The reason we choose Grub rather than other booting methods:
bootctl does not natively support a removable option (so it must write a BootEntry at installation which I find quite annoying); although you could manually copy the binary to the removable EFI path, no maintainance can be done easily with bootctl update; also systemd-boot does not support legacy BIOS so we have to maintain more booting configs, which is a hassle itselfIn summary, for the multi-boot logic we would need: GPT + EFI partition + one single removable Grub EFI binary + one single grub.cfg as menuentry selector + one grub.cfg per system maintained by the system itself
This is still “simple” but not straight-forward in the modern perspective. Still, let’s write the main ideas down:
Note while MBR was mentioned above, the partition table on the drive does not have to be MBR / msdos. The BIOS actually knows nothing above the partition but rather just reads stuff from fixed offsets (think that as “as-if MBR0”, in fact the whole drive can be a “super floppy” i.e. fs on whole drive, and as long as sector 0 is available it does not matter).
So most of the legacy-BIOS-compatible boot loaders need to be installed to MBR, or also technically the whole drive, e.g. for Grub:
grub-install --target i386-pc /dev/sda
The binary that Grub installs into MBR / sector 0 is called boot.img by Grub itself, the functionality is similar to grubx64.efi in the UEFI case: to load necessary drivers to look up real grub.cfg. But as MBR sector 0 is too small (512 Byte) it’s impossible to do it by the small boot.img itself. For this another part of binary called core.img needs to be looked up and executed. Grub does this differently depending on whether you’re booting on MBR or GPT:
core.img is stored from sector 1 onward, before first partition, the space is 1 MiB - 512B, and in real world would never be fully utilized.core.img is stored in the partition with type BIOS boot (21686148-6449-6E6F-744E-656564454649), for the same reason this can be only 1 MiBOf course core.img itself would carry metadata including how large the actual data is, fs drivers so actual root partition could be opened, and optionally a built-in config.
The logic is simply, boot.img -> core.img -> $prefix/grub.cfg, after that latter steps are similar to UEFI cases.
Similar as UEFI, we would want a single outer Grub with config managed by ourselves to function as selector, and each system managing their own inner grub.cfg (but not having their Grub boot manager) ready to be picked up by ourselves.
The reason we choose Grub rather than other booting methods:
In summary, for the multi-boot logic we would need: GPT + BIOS Boot partition + one single Grub binary in MBR0 and BIOS Boot partition + one single grub.cfg as menuentry selector + one grub.cfg per system maintained by the system itself
Some AArch64 devices support UEFI, many others don’t and use U-Boot. In not-too-old U-Boot builds, the “distroboot” concept scans for various bootable targets, and removable EFI binary EFI/BOOT/BOOTAA64.EFI is one of them, besides extlinux.conf and boot.scr(.uimg) . I build u-boot myself on all of SBCs and TV boxes on my hand and they all support such without explicitly enabling. As we want multi-boot on a single drive on both AArch64 and x86_64 we would only focus the UEFI and distroboot UEFI.
On AArch64 UEFI, the steps are similar as x86_64, except that the fallback EFI binary name is BOOTAA64.EFI
On AArch64 U-Boot that chainloads removable EFI in distroboot logic:
bootcmd, in modern cases, bootcmd='bootflow scan -lb'bootflow, the argument scan means to scan all possible sources, in most cases these include all block devices first, then network; the argument -l means to print each scanned bootablt target; the argument -b means for each target scanned, try to boot immediately.Note specifically for per-device DTB to be applied correctly in the U-Boot case, if /boot is in root fs, the job cannot be done by Grub (cannot expect a partition to be readable before you could even tell there’s a block device), rather the DTB has to be loaded by U-Boot
Boot archiso or do this in a device already running Linux.
Run your perferred partition tool to partition the drive with the following partitions:
Or simply save the following infos in a temporary file e.g. parts.info:
label: gpt
unit: sectors
sector-size: 512
size=204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B
size=2048, type=21686148-6449-6E6F-744E-656564454649
type=0FC63DAF-8483-4772-8E79-3D69D8477DE4
Then run sfdisk to format use the info:
sfdisk /dev/[drive] < parts.info
The result partitions shall look like the following (/dev/vda is used in the following example):
Checking that no-one is using this disk right now ... OK
Disk /dev/[drive]: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Created a new GPT disklabel (GUID: D8997F52-AFD6-4A0F-845F-FF12CDD718F6).
/dev/[drive]1: Created a new partition 1 of type 'EFI System' and of size 100 MiB.
/dev/[drive]2: Created a new partition 2 of type 'BIOS boot' and of size 1 MiB.
/dev/[drive]3: Created a new partition 3 of type 'Linux filesystem' and of size 63.9 GiB.
/dev/[drive]4: Done.
New situation:
Disklabel type: gpt
Disk identifier: D8997F52-AFD6-4A0F-845F-FF12CDD718F6
Device Start End Sectors Size Type
/dev/[drive]1 2048 206847 204800 100M EFI System
/dev/[drive]2 206848 208895 2048 1M BIOS boot
/dev/[drive]3 208896 134215679 134006784 63.9G Linux filesystem
The partition table has been altered.
Create a FAT fs on the ESP:
mkfs.vfat /dev/[drive]1
Create a Btrfs on the root partition:
mkfs.btrfs /dev/[drive]3
Note about Btrfs compression: if you want it, you could set it at fs-creation time, e.g.:
mkfs.btrfs --compress zstd:15 /dev/[drive]3
But the later steps assume this not set and we would write the compression level manually at mount time and in fstab. You can omit those if you did this at fs-creation time.
Now mount the Btrfs root partition to somewhere, we need to create some subvols
mount --mkdir /dev/[drive]3 /mnt/manyos
cd /mnt/manyos
Let’s create subvols. The main focus is that we want seperate root volumes for each system, and shared home. (@ or + is not strictly needed in subvol names, but it helps to tell them from plain folders)
btrfs subvolume create @arch @debian @home
mkdir shared arch-x86_64 debian-x86_64 alarm-aarch64 debian-aarch64
btrfs subvolume create shared/@{home{,_.snapshots},etc_ssh} {arch-x86_64,debian-x86_64,alarm-aarch64,debian-aarch64}/{@{,.snapshots},+nocow}
chattr +C {arch-x86_64,debian-x86_64,alarm-aarch64,debian-aarch64}/+nocow
mkdir -p arch-x86_64/+nocow/var/{cache,log,spool,tmp}
chmod 1777 arch-x86_64/+nocow/var/tmp
mkdir template
tar -f template/nocow.tar -C arch-x86_64/+nocow -cv .
for i in arch-x86_64/@ {debian-x86_64,alarm-aarch64,debian-aarch64}/{@,+nocow}; do tar -f template/nocow.tar -C $i -xv; done
The layout shall look the the following:
> tree
.
├── arch-x86_64
│ ├── @
│ │ └── var
│ │ ├── cache
│ │ ├── log
│ │ ├── spool
│ │ └── tmp
│ ├── +nocow
│ │ └── var
│ │ ├── cache
│ │ ├── log
│ │ ├── spool
│ │ └── tmp
│ └── @.snapshots
├── ...
├── shared
│ ├── @home
│ └── @home_.snapshots
└── template
└── nocow.tar
34 directories, 1 file
The benefit of the above layout is that stuffs not needing snapshot and compression are enclosed into a single +nocow subvol and bind-mounting is used instead of many subvols; and system-specific stuffs are enclosed into one single top-level folder, and shared stuffs not so, and .snapshots subvol for snapper is pre-created.
In later steps I would follow only my own style
As discussed earlier we would use Arch Linux as the one to install and maintain Grub x86_64 (as, Grub almost HAS TO BE installed as Secure Boot with shim, which we don’t need at all), so let’s install Arch Linux first.
Mainly you shall follow the official Installation Guide for the most part, to boot archiso on x86_64 and install in UEFI style; of course you could also do this on a device already running Linux. I would only cover the archiso case.
Let’s focus on things that shall go differently from the official way:
mount -o compress=zstd:15,subvol=arch-x86_64/@ --mkdir /dev/[drive]3 /mnt/root
mount -o subvol=arch-x86_64/+nocow --mkdir /dev/[drive]3 /mnt/arch-x86_64+nocow
Then pre-create fstab and edit it:
mkdir /mnt/root/etc
cp /etc/fstab /mnt/root/etc/
vim /mnt/root/etc/fstab
Remember to use vim’s functionality to do multi-line edit (shift + v, crtl + v, etc) and the ability to pipe content into external command (select in multi-line visual, then :, then !column -t to force a table look)
The content shall look like the following (remember to use blkid to acquire your real UUID for root partition and ESP)
# Static information about the filesystems.
# See fstab(5) for details.
# <file system> <dir> <type> <options> <dump> <pass>
UUID=6894094c-a75e-4f1a-b228-283faf7bf003 / btrfs rw,compress=zstd:15,subvol=arch-x86_64/@ 0 0
UUID=6894094c-a75e-4f1a-b228-283faf7bf003 /.snapshots btrfs rw,compress=zstd:15,subvol=arch-x86_64/@.snapshots 0 0
UUID=6894094c-a75e-4f1a-b228-283faf7bf003 /home btrfs rw,compress=zstd:15,subvol=shared/@home 0 0
UUID=6894094c-a75e-4f1a-b228-283faf7bf003 /home/.snapshots btrfs rw,compress=zstd:15,subvol=shared/@home_.snapshots 0 0
UUID=6894094c-a75e-4f1a-b228-283faf7bf003 /mnt/arch-x86_64+nocow btrfs rw,compress=zstd:15,subvol=arch-x86_64/+nocow 0 0
/mnt/arch-x86_64+nocow/var/cache /var/cache none bind,private 0 0
/mnt/arch-x86_64+nocow/var/log /var/log none bind,private 0 0
/mnt/arch-x86_64+nocow/var/spool /var/spool none bind,private 0 0
/mnt/arch-x86_64+nocow/var/tmp /var/tmp none bind,private 0 0
UUID=5D03-ED7A /efi vfat rw,noatime 0 2
Then mount everything remaining up:
mount --all --fstab /mnt/root/etc/fstab --target-prefix /mnt/root --mkdir
A lsblk shall look like following now:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 254:0 0 128G 0 disk
├─vda1 254:1 0 100M 0 part /mnt/root/efi
├─vda2 254:2 0 1M 0 part
└─vda3 254:3 0 127.9G 0 part /mnt/root/var/tmp
/mnt/root/var/spool
/mnt/root/var/log
/mnt/root/var/cache
/mnt/root/mnt/arch-x86_64+nocow
/mnt/root/home/.snapshots
/mnt/root/home
/mnt/root/.snapshots
/mnt/arch-x86_64+nocow
/mnt/root
echo 'universal: true' > /mnt/root/etc/booster.yaml
pacstrap -K /mnt/root base booster linux intel-ucode amd-ucode linux-firmware btrfs-progs dosfstools grub vim sudo
arch-chroot /mnt/root and do the remaining parts, until before “Network configuration”systemd-networkd, so:
systemctl enable systemd-{network,resolve}dcp -rva /etc/systemd/network/* /mnt/root/etc/systemd/network/ln -sf /run/systemd/resolve/stub-resolv.conf /mnt/root/etc/resolv.confbooster instead of the default mkinitcpio, and the universal initramfs was already created--boot-directory /efi, so grub modules and first-stage config are saved and loaded from there. We would only want each system’s /boot to store there boot config
grub-install --removable --efi-directory /efi --boot-directory /efi
--boot-directory /efi
grub-install --target i386-pc --boot-directory /efi /dev/[drive]
/etc/grub.d/10_linux so it would prefer booster initramfs (without this, booster initramfs would be hidden in a submenu) and ro root; if you’re not using booster only or you don’t require ro root on boot, you can skip this:
From 9ad850b2b8842bb673313be08f6a5af66cdf12ea Mon Sep 17 00:00:00 2001
From: Guoxin Pu <pugokushin@gmail.com>
Date: Thu, 13 Nov 2025 15:52:27 +0800
Subject: [PATCH] use booster as main initramfs and prefer ro
---
10_linux | 26 ++------------------------
1 file changed, 2 insertions(+), 24 deletions(-)
diff --git a/10_linux b/10_linux
index e16cea8..3ce3a9d 100755
--- a/10_linux
+++ b/10_linux
@@ -147,7 +147,7 @@ linux_entry ()
message="$(gettext_printf "Loading Linux %s ..." ${version})"
sed "s/^/$submenu_indentation/" << EOF
echo '$(echo "$message" | grub_quote)'
- linux ${rel_dirname}/${basename} root=${linux_root_device_thisversion} rw ${args}
+ linux ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args}
EOF
if test -n "${initrd}" ; then
# TRANSLATORS: ramdisk isn't identifier. Should be translated.
@@ -227,7 +227,7 @@ for linux in ${reverse_sorted_list}; do
done
initrd_real=
- for i in "initrd.img-${version}" "initrd-${version}.img" \
+ for i in "booster-${version}.img" "initrd.img-${version}" "initrd-${version}.img" \
"initrd-${alt_version}.img.old" "initrd-${version}.gz" \
"initrd-${alt_version}.gz.old" "initrd-${version}" \
"initramfs-${version}.img" "initramfs-${alt_version}.img.old" \
@@ -304,28 +304,6 @@ for linux in ${reverse_sorted_list}; do
linux_entry "${OS}" "${version}" advanced \
"${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}"
- if test -e "${dirname}/initramfs-${version}-fallback.img" ; then
- initrd="${initrd_early} initramfs-${version}-fallback.img"
-
- if test -n "${initrd}" ; then
- gettext_printf "Found fallback initrd image(s) in %s:%s\n" "${dirname}" "${initrd_extra} ${initrd}" >&2
- fi
-
- linux_entry "${OS}" "${version}" fallback \
- "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}"
- fi
-
- if test -e "${dirname}/booster-${version}.img" ; then
- initrd="${initrd_early} booster-${version}.img"
-
- if test -n "${initrd}" ; then
- gettext_printf "Found booster initrd image(s) in %s:%s\n" "${dirname}" "${initrd_extra} ${initrd}" >&2
- fi
-
- linux_entry "${OS}" "${version}" booster \
- "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}"
- fi
-
if [ "x${GRUB_DISABLE_RECOVERY}" != "xtrue" ]; then
linux_entry "${OS}" "${version}" recovery \
"${GRUB_CMDLINE_LINUX_RECOVERY} ${GRUB_CMDLINE_LINUX}"
--
2.51.2
grub.cfg for Arch Linux as how you do it on a normal installation:
mkdir /boot/grub
grub-mkconfig -o /boot/grub/grub.cfg
Note this would not be the first config Grub loads, but rather just a “to-be-included” config.
vim /efi/grub/grub.cfg with content like following:
search.fs_uuid 6894094c-a75e-4f1a-b228-283faf7bf003 root hd0,gpt2
terminal_input console
terminal_output console
set suffix='@/boot/grub/grub.cfg'
menuentry 'Arch Linux (x86_64)' {
configfile ($root)/arch-x86_64/$suffix
}
After the above steps we shall have a UEFI + legacy bootable, you can boot on different machines to validate it, the boot menu shall look like this on both UEFI and legacy:
GNU GRUB version 2:2.14rc1-2
┌────────────────────────────────────────────────────────────────────────────┐
│*Arch Linux (x86_64) │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
└────────────────────────────────────────────────────────────────────────────┘
Use the ▲ and ▼ keys to select which entry is highlighted.
Press enter to boot the selected OS, `e' to edit the commands
before booting or `c' for a command-line.
And after pressing Enter it’s the same old Arch Linux Grub menu as always.
While we’re at it, you can of course add timeout, decoration, etc to the outer menu. I’ll stick with the simple look and continue.
This part is mostly about quality of life improvement and can be skipped. If you want to follow, boot into the installed Arch Linux.
pacman -S openssh
systemctl enable --now sshd
Verify that sshd pubkeys are generated successfully:
# ls /etc/ssh
moduli ssh_config ssh_config.d sshd_config sshd_config.d ssh_host_ecdsa_key ssh_host_ecdsa_key.pub ssh_host_ed25519_key ssh_host_ed25519_key.pub ssh_host_rsa_key ssh_host_rsa_key.pub
# systemctl status sshdgenkeys
○ sshdgenkeys.service - SSH Key Generation
Loaded: loaded (/usr/lib/systemd/system/sshdgenkeys.service; disabled; preset: disabled)
Active: inactive (dead) since Thu 2025-11-13 16:26:26 CST; 1min 51s ago
Invocation: ef2323bd8d4e4007be02a0e3208f3c31
Process: 578 ExecStart=/usr/bin/ssh-keygen -A (code=exited, status=0/SUCCESS)
Main PID: 578 (code=exited, status=0/SUCCESS)
Mem peak: 1.7M
CPU: 84ms
Nov 13 16:26:25 dud systemd[1]: Starting SSH Key Generation...
Nov 13 16:26:26 dud ssh-keygen[578]: ssh-keygen: generating new host keys: RSA ECDSA ED25519
Nov 13 16:26:26 dud systemd[1]: sshdgenkeys.service: Deactivated successfully.
Nov 13 16:26:26 dud systemd[1]: Finished SSH Key Generation.
Do required sshd_config modification as needed:
vim /etc/ssh/sshd_config
AuthorizedKeysFile .ssh/authorized_keys -> AuthorizedKeysFile /etc/ssh/authorized_keys/%u, so no one can add their SSH pubkey except root (me); and I would prepare this folder as needed#PasswordAuthentication and set PasswordAuthentication no so password login is disabledAfter the modification restart sshd so the changes take effect:
systemctl restart sshd
For the setup I would want after-boot snapshots to be taken, so install snapper:
pacman -S snapper
Create configs and enable needed services
vim /etc/snapper/configs/root
vim /etc/snapper/configs/home
The content shall look like the following:
SUBVOLUME="/"
FSTYPE="btrfs"
QGROUP=""
SPACE_LIMIT="0.5"
FREE_LIMIT="0.2"
ALLOW_USERS=""
ALLOW_GROUPS=""
SYNC_ACL="no"
BACKGROUND_COMPARISON="yes"
NUMBER_CLEANUP="yes"
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="50"
NUMBER_LIMIT_IMPORTANT="10"
EMPTY_PRE_POST_CLEANUP="yes"
EMPTY_PRE_POST_MIN_AGE="1800"
SUBVOLUME="/home"
FSTYPE="btrfs"
QGROUP=""
SPACE_LIMIT="0.5"
FREE_LIMIT="0.2"
ALLOW_USERS=""
ALLOW_GROUPS=""
SYNC_ACL="no"
BACKGROUND_COMPARISON="yes"
NUMBER_CLEANUP="yes"
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="50"
NUMBER_LIMIT_IMPORTANT="10"
EMPTY_PRE_POST_CLEANUP="yes"
EMPTY_PRE_POST_MIN_AGE="1800"
And edit the global config to enable these profiles:
vim /etc/conf.d/snapper
With the following line:
SNAPPER_CONFIGS="root home"
By default snapper-boot only snapshots root, so modify the unit:
systemctl edit snapper-boot
The result shall look like this (ExecStart is appended after the original ExecStarted):
> cat /etc/systemd/system/snapper-boot.service.d/override.conf
[Service]
ExecStart=/usr/bin/snapper --config home create --cleanup-algorithm number --description "boot"
Then let’s enable needed units:
systemctl enable --now snapper-{boot,cleanup}.timer
Let’s do this on Arch Linux x86_64 with debootstrap and arch-chroot
Of course the tools shall be installed first:
pacman -S debootstrap arch-install-scripts
The installation goes similarly as Arch, but keep the following points in mind:
export PATH= with missing /sbin parts, whole command in later stepsThe steps are as follows:
mount -o subvol=debian-x86_64/@ --mkdir /dev/[drive]3 /mnt/root
mount -o subvol=debian-x86_64/+nocow --mkdir /dev/[drive]3 /mnt/debian-x86_64+nocow
Then duplicate fstab:
mkdir /mnt/root/etc
sed 's/arch/debian/g' /etc/fstab > /mnt/root/etc/fstab
Then mount everything remaining up:
mount --all --fstab /mnt/root/etc/fstab --target-prefix /mnt/root --mkdir
A lsblk shall look like following now:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 254:0 0 128G 0 disk
├─vda1 254:1 0 100M 0 part /mnt/root/efi
│ /efi
├─vda2 254:2 0 1M 0 part
└─vda3 254:3 0 127.9G 0 part /mnt/root/var/tmp
/mnt/root/var/spool
/mnt/root/var/log
/mnt/root/var/cache
/mnt/root/mnt/debian-x86_64+nocow
/mnt/root/home/.snapshots
/mnt/root/home
/mnt/root/.snapshots
/mnt/debian-x86_64+nocow
/mnt/root
/var/tmp
/var/spool
/var/log
/home/.snapshots
/var/cache
/mnt/arch-x86_64+nocow
/home
/.snapshots
/
debootstrap into the root
debootstrap trixie /mnt/root http://[mirror_link]
/mnt/root, and set PATH
arch-chroot /mnt/root
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
The PATH needs to be set as arch-chroot keeps PATH from Arch, which lacks sbin, but on Debian sbin is a seperate folder than bin
apt update
apt install vim locales systemd-timesyncd btrfs-progs dosfstools linux-image-amd64 amd64-microcode intel-microcode firmware-linux
systemd-networkd + systemd-resolved pair, but on Debian resolved needs to be installed seperately:
apt install systemd-resolved
grub.cfg, but definitely not intalling and maintaining the actual Grub boot manager, so install only the system integration part and prepare the folder manually:
apt install grub2-common
mkdir /boot/grub
vim /etc/default/grub
With this setup there would be no pre-configured grub config, so use the following as a starting point:
GRUB_DEFAULT=0
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR=`( . /etc/os-release && echo ${NAME} )`
GRUB_CMDLINE_LINUX_DEFAULT="audit=0"
GRUB_CMDLINE_LINUX=""
GRUB_TERMINAL=console
Remember to re-generate the one included by our outer grub
update-grub
/efi/grub/grub.cfg to include a new menuentry:
menuentry 'Debian (x86_64)' {
configfile ($root)/debian-x86_64/$suffix
}
After the above steps we shall have a UEFI + legacy bootable Arch Linux + Debian installation, you can boot on different machines to validate it, the boot menu shall look like this on both UEFI and legacy:
GNU GRUB version 2:2.14rc1-2
┌────────────────────────────────────────────────────────────────────────────┐
│*Arch Linux (x86_64) │
│ Debian (x86_64) │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
└────────────────────────────────────────────────────────────────────────────┘
Use the ▲ and ▼ keys to select which entry is highlighted.
Press enter to boot the selected OS, `e' to edit the commands
before booting or `c' for a command-line.
And after selecting Debian and pressing Enter it’s the same old Debian Grub menu as always.
This part is mostly about quality of life improvement and can be skipped. If you want to follow, boot into the installed Debian and do similar things as in After-installation for Arch Linux x86_64, but with the following differences:
/etc/ssh/ssh_host_*_key{,.pub}, so clients would not complain about host key differingsystemctl disable --now snapper-timeline.timer; and note the snapper config folder is the same but the global config is at /etc/default/snapperdebootstrap still prepares an old-style APT sources.list, I recommend to migrate to new APT config style:
rm /etc/apt/sources.list
vim /etc/apt/sources.list.d/debian.sources
With content like the following:
Types: deb
URIs: http://[mirror]/debian/
Suites: trixie trixie-updates
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: http://[mirror]/debian-security/
Suites: trixie-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: http://[mirror]/debian/
Suites: trixie-backports
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
MODULES=most and you would not need to modify that, only with debian-install could this be set to MODULES=deps. To verify, vim /etc/initramfs-tools/initramfs.conf and check MODULES. If it’s MODULES=deps then modify it, save, and run update-initramfs -ufirmware-linux meta package, in most cases these are not needed, if you really really need all of them installed:
apt install $(for i in $(apt-cache search '^firmware-' | cut -d ' ' -f 1); do dpkg-query -W -f='${Status}' $i &>/dev/null || echo $i; done | grep -v installer)
Let’s do this on Arch Linux x86_64
pacman -S qemu-user-static-binfmt arch-install-scripts
cp /etc/pacman.conf pacman-alarm.conf
vim pacman-alarm.conf
Architecture = aarch64 instead of autoInlucde = mirrorlist-alarm instead of /etc/pacman.d/mirrorlistSigLevel = Never as we don’t have ALARM keyring on Arch and adding those to Arch host would mess up the host, pacakges can later be re-verified once we have installed ALARMlinux-aarch64-7ji as kernel, instead of ALARM’s official linux-aarch64, the latter misses some drivers built-in and would not boot on some of my SBCs and it has max CPUs set to a small number so would not work nicely on VM either, and the worst is it has some naive hooks to always expect mkinitcpio instead of any other initramfs maker.
[7Ji]
Include = mirrorlist-3rdparty
vim mirrorlist-alarm
With server:
Server = http://[mirror]/archlinuxarm/$arch/$repo
vim mirrorlist-3rdparty
With server:
Server = http://[mirror]/$repo/$arch
mount -o subvol=alarm-aarch64/@ --mkdir /dev/[drive]3 /mnt/root
mount -o subvol=alarm-aarch64/+nocow --mkdir /dev/[drive]3 /mnt/alarm-aarch64+nocow
Then duplicate fstab:
mkdir /mnt/root/etc
sed 's/arch-x86_64/alarm-aarch64/g' /etc/fstab > /mnt/root/etc/fstab
Then mount everything remaining up:
mount --all --fstab /mnt/root/etc/fstab --target-prefix /mnt/root --mkdir
A lsblk shall look like following now:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 254:0 0 128G 0 disk
├─vda1 254:1 0 100M 0 part /mnt/root/efi
│ /efi
├─vda2 254:2 0 1M 0 part
└─vda3 254:3 0 127.9G 0 part /mnt/root/var/tmp
/mnt/root/var/spool
/mnt/root/var/log
/mnt/root/var/cache
/mnt/root/mnt/alarm-aarch64+nocow
/mnt/root/home/.snapshots
/mnt/root/home
/mnt/root/.snapshots
/mnt/alarm-aarch64+nocow
/mnt/root
/var/tmp
/var/spool
/var/log
/home/.snapshots
/var/cache
/mnt/arch-x86_64+nocow
/home
/.snapshots
/
pacstrap into the root
echo 'universal: true' > /mnt/root/etc/booster.yaml
pacstrap -C pacman-alarm.conf -K -M /mnt/root base booster linux-aarch64-7ji linux-firmware btrfs-progs dosfstools grub vim sudo archlinuxarm-keyring 7ji-keyring
/mnt/root/etc, as now they come the official package, remember to add 7Ji repo
vim /mnt/root/etc/pacman.conf
vim /mnt/root/etc/pacman.d/mirrorlist
vim /mnt/root/etc/pacman.d/mirrorlist-3rdparty
arch-chroot /mnt/root
uname -m
Note: as we’re using qemu-static, the stdin/out is technically not directly attached to our terminal, so text editing could be a pain. If text editing is needed, do it from another SSH session from host under /mnt/root instead so VIM and Nano could correctly write to terminal.
pacman-key --init
pacman-key --populate
pacman -Sy --downloadonly $(sed -n '/^%NAME%/{n;p}' /var/lib/pacman/local/*/desc)
--boot-directory /efi, so grub modules and first-stage config are saved and loaded from there. We would only want each system’s /boot to store there boot config
grub-install --removable --efi-directory /efi --boot-directory /efi
/etc/grub.d/10_linux so it would prefer booster initramfs (without this, booster initramfs would be hidden in a submenu) and ro root; if you’re not using booster only or you don’t require ro root on boot, you can skip this:
From 9ad850b2b8842bb673313be08f6a5af66cdf12ea Mon Sep 17 00:00:00 2001
From: Guoxin Pu <pugokushin@gmail.com>
Date: Thu, 13 Nov 2025 15:52:27 +0800
Subject: [PATCH] use booster as main initramfs and prefer ro
---
10_linux | 26 ++------------------------
1 file changed, 2 insertions(+), 24 deletions(-)
diff --git a/10_linux b/10_linux
index e16cea8..3ce3a9d 100755
--- a/10_linux
+++ b/10_linux
@@ -147,7 +147,7 @@ linux_entry ()
message="$(gettext_printf "Loading Linux %s ..." ${version})"
sed "s/^/$submenu_indentation/" << EOF
echo '$(echo "$message" | grub_quote)'
- linux ${rel_dirname}/${basename} root=${linux_root_device_thisversion} rw ${args}
+ linux ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args}
EOF
if test -n "${initrd}" ; then
# TRANSLATORS: ramdisk isn't identifier. Should be translated.
@@ -227,7 +227,7 @@ for linux in ${reverse_sorted_list}; do
done
initrd_real=
- for i in "initrd.img-${version}" "initrd-${version}.img" \
+ for i in "booster-${version}.img" "initrd.img-${version}" "initrd-${version}.img" \
"initrd-${alt_version}.img.old" "initrd-${version}.gz" \
"initrd-${alt_version}.gz.old" "initrd-${version}" \
"initramfs-${version}.img" "initramfs-${alt_version}.img.old" \
@@ -304,28 +304,6 @@ for linux in ${reverse_sorted_list}; do
linux_entry "${OS}" "${version}" advanced \
"${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}"
- if test -e "${dirname}/initramfs-${version}-fallback.img" ; then
- initrd="${initrd_early} initramfs-${version}-fallback.img"
-
- if test -n "${initrd}" ; then
- gettext_printf "Found fallback initrd image(s) in %s:%s\n" "${dirname}" "${initrd_extra} ${initrd}" >&2
- fi
-
- linux_entry "${OS}" "${version}" fallback \
- "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}"
- fi
-
- if test -e "${dirname}/booster-${version}.img" ; then
- initrd="${initrd_early} booster-${version}.img"
-
- if test -n "${initrd}" ; then
- gettext_printf "Found booster initrd image(s) in %s:%s\n" "${dirname}" "${initrd_extra} ${initrd}" >&2
- fi
-
- linux_entry "${OS}" "${version}" booster \
- "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}"
- fi
-
if [ "x${GRUB_DISABLE_RECOVERY}" != "xtrue" ]; then
linux_entry "${OS}" "${version}" recovery \
"${GRUB_CMDLINE_LINUX_RECOVERY} ${GRUB_CMDLINE_LINUX}"
--
2.51.2
grub.cfg for Arch Linux ALARM as how you do it on a normal installation:
mkdir /boot/grub
grub-mkconfig -o /boot/grub/grub.cfg
Note this would not be the first config Grub loads, but rather just a “to-be-included” config.
menuentry 'Arch Linux ARM (aarch64)' {
configfile ($root)/alarm-aarch64/$suffix
}
The drive should now work on an aarch64 VM (pure UEFI), but not on real hardware (U-boot faking UEFI), due to missing DTB. Verify it on VM first before trying on real hardware. (Remeber to disable Secure Boot first)
The Grub menu shall look like the following in VM:
GNU GRUB version 2:2.14rc1-2
┌────────────────────────────────────────────────────────────────────────────┐
│*Arch Linux (x86_64) │
│ Debian (x86_64) │
│ Arch Linux ARM (aarch64) │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
└────────────────────────────────────────────────────────────────────────────┘
Use the ▲ and ▼ keys to select which entry is highlighted.
Press enter to boot the selected OS, `e' to edit the commands
before booting or `c' for a command-line.
And after selecting Arch Linux ARM and pressing Enter it’s the same old Arch Grub menu (well ALARM didn’t bother to modify the look) as always.
Now, for U-Boot to correctly work, U-Boot itself needs to load the specific DTB for device. We don’t want Grub to load DTB from root AFTER U-Boot loads Grub, as the device must be fully functional at the time Grub wants to open the Btrfs root, and that’s too late.
In modern-day U-Boot, there would be these built-in variables:
efi_dtb_prefixes=/ /dtb/ /dtb/current/ -> Same across buildsfdtfile=amlogic/meson-sm1-bananapi-m5.dtb -> Unique for each boardscan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;run boot_efi_bootmgr;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi; then echo Found EFI removable media binary efi/boot/bootaa64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile -> Same across buildsSo take my BPI-M5 for example, the place for the DTB could be:
ESP/amlogic/meson-sm1-bananapi-m5.dtbESP/dtb/amlogic/meson-sm1-bananapi-m5.dtbESP/dtb/current/amlogic/meson-sm1-bananapi-m5.dtbI’ll pick the second one
For this, we needs to copy some DTBs to “ESP”, usually in U-Boot efi_dtb_prefixes=/ /dtb/ /dtb/current/, so let’s pick /dtb/ so they can be loaded by U-Boot as early as possible
cp -rva /mnt/root/boot/dtbs/linux-aarch64-7ji /mnt/root/efi/dtb
And booting on real hardware should now be OK. I’ve tested this on my BananaPi BPi-M5, OrangePi 5, Orange Pi 5 Plus and they all work seamlessly.
Note: as you may have tried and realized, even if you did not place the DTB, boards could still boot, but in those cases it is the U-Boot’s built-in DTB that’s used, and for newer kernels this could bring some problems.
Note also: by doing this we’ve locked the DTB to the one provided by linux-aarch64-7ji kernel, which is a stable-as-new-as-possible kernel, not only for Arch Linux ARM, but also for the latter Debian installation, using new DTBs on old kernels generally wouldn’t bring much trouble, unlike other way around.
Do this in Arch Linux ARM aarch64, just similar to how we installed Debian x86_64 from Arch Linux x86_64. Just remember that still we would not want Grub the boot manager installed here, but rather Debian should only install grub2-common to auto-update its sub gurb.cfg
The final Grub menu shall look like this:
GNU GRUB version 2:2.14rc1-2
┌────────────────────────────────────────────────────────────────────────────┐
│*Arch Linux (x86_64) │
│ Debian (x86_64) │
│ Arch Linux ARM (aarch64) │
│ Debian (aarch64) │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
└────────────────────────────────────────────────────────────────────────────┘
Use the ▲ and ▼ keys to select which entry is highlighted.
Press enter to boot the selected OS, `e' to edit the commands
before booting or `c' for a command-line.
Now it’s good time to do modification, to e.g. decorate the outer menu, reorder the menu, add more entries, etc.
Also note that I did not set timeout, for that booting default x86_64 on aarch64 is plainly wrong, and this blog was written as simple to follow as posisble. This could be improved by either one of the following ways:
/efi/x86_64 and /efi/aarch64 and maintain sepearate outer grub.cfg for each architecture$grub_platform variable and $grub_cpu to determine the current platform. If you’d want to try, the syntax shall look like this:
set timeout 1
if [ "${grub_cpu}" = 'x86_64' -a "${grub_platform}" = 'efi' ]; then
# x86_64 UEFI Secure Boot only
menuentry ...
elif [ "${grub_cpu}" = 'x86_64' -o "${grub_cpu}" = 'i386']; then
# x86_64 UEFI plain, or legacy
menuentry ...
elif [ "${grub_cpu}" = 'arm64' ]; then
if [ "${grub_platform}" = 'efi' ]; then
# arm64 UEFI Secure Boot only
menuentry ...
fi
# arm64 plain
menuentry ...
fi
In a recent change to kernel pacakage the module for Btrfs became built-in instead of as-module: diff
This reveals a race condition (in both kernel and booster) which seemed to have been “worked around” in the past but in reality not, for multi-device Btrfs:
BTRFS_IOC_DEVICES_READY and it mounts every Btrfs sub-device whether they’re actually ready or not.EBUSYExample logs of failed boots, from which you could tell that booster tries to register vdc immediately after vdb while vdb’s failure was not finished yet and it’s rejected by kernel:
[ 0.875882] BTRFS error: device /dev/vdc (254:32) belongs to fsid f1872b6c-c2b3-44fe-930e-bb85fd35d669, and the fs is already mounted, scanned by init (1)
[ 0.876896] booster: ioctl(0x90009427): device or resource busy
ioctl(0x90009427): device or resource busy
[ 0.877787] BTRFS error (device vdb): devid 2 uuid 1f95bd95-1c49-498f-abf9-a39143b970ce is missing
[ 0.878568] BTRFS error (device vdb): failed to read the system array: -2
[ 0.880541] BTRFS error (device vdb): open_ctree failed: -2
mount(/dev/vdb): no such file or directory
found a new device /dev/vda
blkinfo for /dev/vda: type=mbr UUID=dfd9de13 LABEL=
found a new device /dev/vda1
blkinfo for /dev/vda1: type=fat UUID=8b5649c8 LABEL=NO NAME
found a new device /dev/vdb
blkinfo for /dev/vdb: type=btrfs UUID=f1872b6c-c2b3-44fe-930e-bb85fd35d669 LABEL=
mounting /dev/vdb->/booster.root, fs=btrfs, flags=0x0, options=
found a new device /dev/vdc
blkinfo for /dev/vd[ 0.833072] BTRFS error: device /dev/vdc (254:32) belongs to fsid f1872b6c-c2b3-44fe-930e-bb85fd35d669, and the fs is already mounted, scanned by init (136)
c: type=btrfs UU[ 0.834106] booster: ioctl(0x90009427): device or resource busy
ID=f1872b6c-c2b3[ 0.834346] BTRFS error (device vdb): devid 2 uuid 1f95bd95-1c49-498f-abf9-a39143b970ce is missing
-44fe-930e-bb85f[ 0.835248] BTRFS error (device vdb): failed to read the system array: -2
d35d669 LABEL=
ioctl(0x90009427): device or res[ 0.836022] BTRFS error (device vdb): open_ctree failed: -2
ource busy
mount(/dev/vdb): no such file or directory
To fix this, we need to check the return status of BTRFS_IOC_DEVICES_READY and only really considers the FS ready when it returns 0
Now the logic is:
Tested on a VM with three Vdisks, vda as boot, vdb + vdc as btrfs (profile single, raid0, raid1 all tested):
found a new device /dev/vda
blkinfo for /dev/vda: type=mbr UUID=dfd9de13 LABEL=
found a new device /dev/vdc
found a new device /dev/vda1
blkinfo for /dev/vdc: type=btrfs UUID=f1872b6c-c2b3-44fe-930e-bb85fd35d669 LABEL=
blkinfo for /dev/vda1: type=fat UUID=8b5649c8 LABEL=NO NAME
found a new device /dev/vdb
Waiting for multi-device btrfs at /dev/vdc to become fullly assembled, waited 0 seconds
blkinfo for /dev/vdb: type=btrfs UUID=f1872b6c-c2b3-44fe-930e-bb85fd35d669 LABEL=
mounting /dev/vdb->/booster.root, fs=btrfs, flags=0x0, options=
Switching to the new userspace now. Да пабачэння!
References:
BTRFS_IOC_DEVICES_READY: source codeBTRFS_IOC_DEVICES_READY with type int to determine whether the FS is ready: source codeOn OpenWrt, DDNS functionality is provided by the opt-in ddns-scripts package (and optionally ddns-scripts-[provider] packages), which provides both an rc-init script /etc/init.d/ddns and a hotplug.d hook /etc/hotplug.d/iface/95-ddns to start it automatically:
The rc-init script, instead of starting a “daemon” and maintaining it like other scripts that uses procd, mainly just calls /usr/lib/ddns/dynamic_dns_updater.sh, which without explicit interface names after start/stop/reload just double forks the workers and quits. Note it has an empty boot() function which shadows start() on boot, i.e. /usr/lib/ddns/dynamic_dns_updater.sh -- start would not be run on boot.
cat /etc/init.d/ddns
#!/bin/sh /etc/rc.common
START=95
STOP=10
boot() {
return 0
}
reload() {
/usr/lib/ddns/dynamic_dns_updater.sh -- reload
return 0
}
restart() {
/usr/lib/ddns/dynamic_dns_updater.sh -- stop
sleep 1 # give time to shutdown
/usr/lib/ddns/dynamic_dns_updater.sh -- start
}
start() {
/usr/lib/ddns/dynamic_dns_updater.sh -- start
}
stop() {
/usr/lib/ddns/dynamic_dns_updater.sh -- stop
return 0
}
The hotplug.d hook starts instances for “interfaces” when they’re brought up by netifd and triggers hotplug event (e.g. when you ifup manually, or reconnect an interface from LuCI, or they start up automatically on boot after netifd is up and running):
cat /etc/hotplug.d/iface/95-ddns
#!/bin/sh
# there are other ACTIONs like ifupdate we don't need
case "$ACTION" in
ifup) # OpenWrt is giving a network not phys. Interface
/etc/init.d/ddns enabled && /usr/lib/ddns/dynamic_dns_updater.sh -n "$INTERFACE" -- start
;;
ifdown)
/usr/lib/ddns/dynamic_dns_updater.sh -n "$INTERFACE" -- stop
;;
esac
Both the rc-init script and the hotplug.d maintain nothing: they just spawn workers for interfaces, either fork and run /usr/lib/ddns/dynamic_dns_updater.sh -n "$INTERFACE" -- start by itself, or from a convenient shortcut provided by /usr/lib/ddns/dynamic_dns_updater.sh -- start which iterates uci config ddns internally to do the work.
So on boot the intended logic that dynamic_dns_updater shall be spawned on interfaces is as follows:
exec-ed by early init and becomes new PID 1ubusd becomes ready/etc/init.d/network starts, and spawns netifd in procd/etc/init.d/ddns starts, and due to empty boot() it does nothingnetifd/etc/hotplug.d/iface/95-ddns hook triggers on interface(s) that you have configured ddns on, and the corresponding worker(s) would be spawned.Note that the hotplug.d hook uses the internal name used by netifd. That is, an “physical” “interface” might e.g. be called as br-lan in the scope of Linux, but would be called lan in the scope of netifd, uci, LuCI, etc and of course hotplug.d.
Now let’s discuss about an “issue”: many with a PPPoE wan might find a strange phenomenon: even though they have “enabled” the ddns service and configured it on pppoe-wan “interface”, the ddns worker would not correctly start on boot on their PPPoE wan interface. The reason this issue happens is due to the combination of following factors:
[protocol]-[network], e.g. for PPPoE-based “wan” interface/network, the actual Linux interface name that’s created would be pppoe-wan
config interface 'wan'
option device 'eth5'
option proto 'pppoe'
option username 'xxxxxxxx'
option password 'yyyyyy'
option keepalive '10 60'
option ipv6 'auto
> ip l | grep wan
19: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 3
wan; when you configure “interface” “pppoe-wan”, this would be pppoe-wan:
o = s.taboption('advanced', form.DummyValue, '_interface',
_("Event Network"),
_("Network on which the ddns-updater scripts will be started"));
o.depends("ip_source", "interface");
o.depends("ip_source", "network");
o.forcewrite = true;
o.modalonly = true;
o.cfgvalue = function(section_id) {
return uci.get('ddns', section_id, 'interface') || _('This will be autoset to the selected interface');
};
o.write = function(section_id) {
var opt = this.section.formvalue(section_id, 'ip_source');
var val = this.section.formvalue(section_id, 'ip_'+opt);
return uci.set('ddns', section_id, 'interface', val);
};
interface needs to be the OpenWrt/netifd internal name that’s put on the “network” / Openwrt “interface”, not the Linux “interface”:
# interface network interface used by hotplug.d i.e. 'wan' or 'wan6'
start_daemon_for_all_ddns_sections takes the “network” name as argument and tries to get one ddns section with interface equalling it (note wan is the fallback name):
# starts updater script for all given sections or only for the one given
# $1 = interface (Optional: when given only scripts are started
# configured for that interface)
# used by /etc/hotplug.d/iface/95-ddns on IFUP
# and by /etc/init.d/ddns start
start_daemon_for_all_ddns_sections()
{
local event_if sections section_id configured_if
event_if="$1"
load_all_service_sections sections
for section_id in $sections; do
config_get configured_if "$section_id" interface "wan"
[ -z "$event_if" ] || [ "$configured_if" = "$event_if" ] || continue
/usr/lib/ddns/dynamic_dns_updater.sh -v "$VERBOSE" -S "$section_id" -- start &
done
}
netifd, the “interface” must be the Openwrt “interface” / network, not the Linux “interface”. There’s no internal fallback logic to get the info from a Linux “interface”.
> ubus call network.interface status '{"interface":"wan"}' | jsonfilter -e '@["ipv4-address"][0].address'
xxx.xxx.xxx.xxx
> ubus call network.interface status '{"interface":"pppoe-wan"}' | jsonfilter -e '@["ipv4-address"][0].address'
Command failed: Not found
Failed to parse json data: unexpected end of data
wan, not on pppoe-wanhotplug event actually triggers and it runs /usr/lib/ddns/dynamic_dns_updater.sh -n wan -- start to start the worker for interface wan, but as it could not find any config section in /etc/config/ddns with interface=wan (which in reality is interface=pppoe-wan), it just quits and nevers spawns the actual /usr/lib/ddns/dynamic_dns_updater.sh -S SECTION -- start worker.Note that while hotplug.d logic fails, you can still run /etc/init.d/ddns start to effectively run /usr/lib/ddns/dynamic_dns_updater.sh -- start, which just iterates the whole /etc/config/ddns config and would start all workers for all sections (as -n NETWORK is skipped and -S SECTION is run directly).
This of course does not only affect PPPoE wan, but in general affects any interface that’s named differently from the corresponding network name.
There are two correct way to fix the issue, one is simply LuCI-only, and another one needs some uci (or manual config editting) but does not touch logic codes:
interface=wan in your /etc/config/ddns and this way the hotplug event would correctly starts on “network” “wan”interface value (you can also edit /etc/config/ddns manually)
uci set ddns.cfv4.interface=wan
uci commit ddns
Now with this knowledge you shall know that why the following “band-aid” “hacks” seem to “fix” the “problem” but they are very unreliable.
boot() function in /etc/init.d/ddns, you can force /usr/lib/ddns/dynamic_dns_updater.sh -- start to run on boot, which would spawn workers for each configured section. The workers are there, but if another ifdown & ifup occurs then they could break. As the intended way with hotplug.d is that the worker shall be brought up after ifup and brough down before ifdown./etc/init.d/ddns restart in your /etc/rc.local, you’re basically doing the same thing as removing boot()/etc/init.d/ddns restart in your /etc/rc.local, and a sleep before it, you have the addtional hope that pppoe-wan definitely becomes online after that timeout, however it’s not guaranteed.boot() function in /etc/init.d/ddns, putting sleep and /etc/init.d/ddns restart in your /etc/rc.local. You’re combining “band-aid”s which makes your device more and more non-reproducible.pppoe-wan connection is not there and you have configured retry_max_count for DDNS sections. If you use hotplug.d then the worker is guaranteed to be started on pppoe-wan creation and stopped on pppoe-wan destruction.It might seem enough to just set the kernel cmdline to:
root=root rootfstype=virtiofs
However to make a virtual machine actually boot from such root you need to go through a few gotchas:
dir type storage pool and then mkdir a root-owned subfolder inside the pool, the pool could be reused while the subfolder needs to be VM-specific, the corresponding storage config file looks like this:
> cat /etc/libvirt/storage/filesystems.xml
<pool type='dir'>
<name>filesystems</name>
<uuid>b36d710f-7a61-49f7-935b-061b12f24c8a</uuid>
<capacity unit='bytes'>0</capacity>
<allocation unit='bytes'>0</allocation>
<available unit='bytes'>0</available>
<source>
</source>
<target>
<path>/var/lib/libvirt/filesystems</path>
</target>
</pool>
The subfolder:
> ls -ldh /var/lib/libvirt/filesystems/root.debian12
drwxr-xr-x 18 root root 4.0K May 23 17:55 /var/lib/libvirt/filesystems/root.debian12/
> sed -n '/<filesystem/,/<\/filesystem/p' /etc/libvirt/qemu/debian12.xml
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs'/>
<source dir='/var/lib/libvirt/filesystems/root.debian12'/>
<target dir='root'/>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</filesystem>
root / virtiofs defaults 0 0
virtiofsd needs addtional arguments to allow xattr and keep cap_sysadmin:
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs' queue='1024'/>
<binary path='/usr/lib/virtiofsd+sys_admin' xattr='on'/>
<source dir='/var/lib/libvirt/filesystems/root.arb-x64-builder'/>
<target dir='root'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</filesystem>
> cat /usr/lib/virtiofsd+sys_admin
#!/bin/sh
exec /usr/lib/virtiofsd -o modcaps=+sys_admin "$@"
/etc/initramfs-tools/conf.d/resume must be purged before a rerun of update-initramfs -u, otherwise the initrd would try to resume from the missing block device and breakvirtiofs in /etc/initramfs-tools/modules, otherwise the virtiofs module would not be included (unfortunately initramfs-tools’ hook only picks fs for a root fs mounted from a block device or nfs). Addtionally you could set FSTYPE=virtiofs in /etc/initramfs-tools/initramfs.conf to omit fsck hook (not recommended, as this 1. does not include virtiofs while it looks it should and 2. would render the system unusable if you forget to revert it in the future when you want to disgard the virtiofs root).update-initramfs -u is certainly needed after the above gotchas sorted out.The guest kernel needs to be booted directly by hypervisor, without a bootloader, and it’s recommended to use the vmlinuz and initrd.img symlinks instead of the real files:
> sed -n '/<os/,/<\/os/p' /etc/libvirt/qemu/debian12.xml
<os>
<type arch='x86_64' machine='pc-q35-10.0'>hvm</type>
<kernel>/var/lib/libvirt/filesystems/root.debian12/vmlinuz</kernel>
<initrd>/var/lib/libvirt/filesystems/root.debian12/initrd.img</initrd>
<cmdline>root=root rootfstype=virtiofs</cmdline>
<boot dev='hd'/>
<bootmenu enable='no'/>
</os>
apt purge --autoremove grub2 grub-common
rm -rf /boot/grub
mkinitcpio and dracut initrd makers support booting from virtiofs natively, booster needs my patchset, but dracut is not recommended as its initramfs paths are undetermined containing kernel version (see below).pacstrapping in host using arch-install-scripts or pacstrapping in target using archiso, it’s not recommended to extract the rootfs from a installed system that runs on either virtual disk or physical machineThe guest kernel needs to be booted directly by hypervisor, without a bootloader, the kernel path is determined and the initrd path also unless you’re using dracut (so better use booster or mkinitcpio)
> sed -n '/<os/,/<\/os/p' /etc/libvirt/qemu/archlinux.xml
<os>
<type arch='x86_64' machine='pc-q35-10.0'>hvm</type>
<kernel>/var/lib/libvirt/filesystems/root.archlinux/boot/vmlinuz-linux</kernel>
<initrd>/var/lib/libvirt/filesystems/root.archlinux/boot/booster-linux.img</initrd>
<cmdline>root=root rootfstype=virtiofs</cmdline>
<boot dev='hd'/>
<bootmenu enable='no'/>
</os>
From around March 2024 I’ve bought 5 of these switches, three of them are from Sirivision, and the remaining two from Hellotek. The price kept dropping as I bought more of them, going from around 290 CNY in March 2024 to around 210 CNY in March 2025. It’s almost a steal!
These are all located at different places: two of which were installed at my new house at my hometown to provide in-home 2.5 Gbps networking, two of which were installed at my rent house at the city I work to also provide in-home 2.5 Gbps networking in addition to L3 10 Gbps core switching, and the last one installed in my office for 2.5 Gbps switching among my wired devices.
As they came from some almost unknown branch I didn’t expect much from their stock firmware, but the Sirivision switches tend out to be very feature-full: features go from VLAN tagging, LAG, IGMP snoofing to DHCP spoofing protection, almost like they’ve cut nothing from what Realtek left in their reference BSP and exposed everything, their web UI looks very simple and I appreciate that; on the other hand the Hellotek ones provide only limited features, with only base VLAN tagging, and LAG with limitation (one group must be on 4 native ports and the other group must be on 4 “external” ports), they got no IGMP snoofing, no DHCP spoofing protection, while they have nice web UI it just adds the shame that they choose to limit the features exposed.
They both seem nice and sound, considering their price, providing cheap high speed switching with supposedly little to no hassle. However, in reality they’re really troublemakers, and I’ll summarize the problems below:
Configuration is applied only after booting (only affecting Sirivision)
The VLAN, LAG, etc settings are not applied directly after booting. There’s a short time window (1-2 second) during which the switch acts as a dumb switch.
If you use the switch just as a simple dumb switch without VLAN, LAG, etc then you are safe. However if you have untagged VLAN your untagged traffic would go rogue and escape the VLAN filtering; if you have LAG the LAG ports would forward traffic to each other resulting in loops:
Apr 05 23:57:18 wn1 kernel: igc 0000:02:00.0 enp2s0: NIC Link is Down
Apr 05 23:57:18 wn1 kernel: igc 0000:03:00.0 enp3s0: NIC Link is Down
Apr 05 23:57:19 wn1 kernel: bond0: (slave enp2s0): link status definitely down, disabling slave
Apr 05 23:57:19 wn1 kernel: bond0: (slave enp3s0): link status definitely down, disabling slave
Apr 05 23:57:19 wn1 kernel: bond0: now running without any active interface!
Apr 05 23:57:19 wn1 kernel: bridge0: port 1(bond0) entered disabled state
Apr 05 23:57:40 wn1 kernel: igc 0000:03:00.0 enp3s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Apr 05 23:57:40 wn1 kernel: igc 0000:02:00.0 enp2s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Apr 05 23:57:40 wn1 kernel: bond0: (slave enp2s0): link status definitely up, 2500 Mbps full duplex
Apr 05 23:57:40 wn1 kernel: bond0: (slave enp3s0): link status definitely up, 2500 Mbps full duplex
Apr 05 23:57:40 wn1 kernel: bond0: active interface up!
Apr 05 23:57:40 wn1 kernel: bridge0: port 1(bond0) entered blocking state
Apr 05 23:57:40 wn1 kernel: bridge0: port 1(bond0) entered forwarding state
Apr 05 23:57:40 wn1 kernel: bridge0: received packet on bond0 with own address as source address (addr:f6:fe:da:c5:1d:90, vlan:0)
Apr 05 23:57:40 wn1 kernel: bridge0: received packet on bond0 with own address as source address (addr:f6:fe:da:c5:1d:90, vlan:0)
Apr 05 23:57:40 wn1 kernel: bridge0: received packet on bond0 with own address as source address (addr:f6:fe:da:c5:1d:90, vlan:0)
Apr 05 23:57:41 wn1 kernel: igc 0000:02:00.0 enp2s0: NIC Link is Down
Apr 05 23:57:41 wn1 kernel: igc 0000:03:00.0 enp3s0: NIC Link is Down
Apr 05 23:57:41 wn1 kernel: bond0: (slave enp2s0): link status definitely down, disabling slave
Apr 05 23:57:41 wn1 kernel: bond0: (slave enp3s0): link status definitely down, disabling slave
Apr 05 23:57:41 wn1 kernel: bond0: now running without any active interface!
Apr 05 23:57:41 wn1 kernel: bridge0: port 1(bond0) entered disabled state
Apr 05 23:57:44 wn1 kernel: igc 0000:02:00.0 enp2s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Apr 05 23:57:44 wn1 kernel: igc 0000:03:00.0 enp3s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Apr 05 23:57:45 wn1 kernel: bond0: (slave enp2s0): link status definitely up, 2500 Mbps full duplex
Apr 05 23:57:45 wn1 kernel: bond0: (slave enp3s0): link status definitely up, 2500 Mbps full duplex
Apr 05 23:57:45 wn1 kernel: bond0: active interface up!
Apr 05 23:57:45 wn1 kernel: bridge0: port 1(bond0) entered blocking state
Apr 05 23:57:45 wn1 kernel: bridge0: port 1(bond0) entered forwarding state
These are bad things and you definitely would not like them in your network.
Luckily things are not too bad as you would not reboot the switch very frequently. And the issue seems only affecting Sirivision and not Hellotek.
VLAN 1 cannot be safely deleted
On Hellotek VLAN 1 cannot be deleted at all: the button is grey; on Sirivion you seem to be able to delete it, but deleting it results in a freezing interface with glitched characters as an indication of memory leak.
Both of these seem to be using VLAN 1 for their internal logic, and while it seems still available for modification you’d better left them in place, just removing ports from it if you want some modification.
Management VLAN cannot be set
The Sirivision ones do not provide the option to modify the management VLAN, while the Helloteks ones do provide the option but it never take effect. The management VLAN can only be 1, but it’s not 1 actually, there’s just no “management VLAN” in fact, which comes to the next issue.
And your know what? It seems Hellotek “has” management VLAN but that only works for external traffic! I.e. when through a stacked switch, you now can only access another switch from untagged traffic, how secure and convenient!
Management packets are captured from all VLANs
The switch captures any traffic targetting its IP instead of only from a specific management VLAN. To make things worse, while the Sirivision one captures these traffic from even external traffic forwarded from other switches (i.e. a stacked config), the Hellotek ones only capture such traffic from ports directly connected to it physically.
This is a bad thing. E.g. let’s assume you left the switch to keep its default 192.168.1.199 mangement IP, and you have a VLAN 123 running the 192.168.1.0/24 network, your access to 192.168.1.199 in VLAN 123 from a port connected to the switch leads you always to the switch, not the actual target.
VLAN configuration cannot be saved when under load (only affecting Sirivision)
If there’s already traffic in a VLAN that’s not fully configured, you can never get it fully configured. The web UI just freezes and config would never be saved. You can only configure things in one go.
This makes it very inconvenient if you want your main VLAN not being VLAN 1, and to make things worse recall that “VLAN 1 cannot be safely deleted” and “Both of these seem to be using VLAN 1 for their internal logic”, so in general you’d better not be using VLAN 1 and have to do this. To work around this and make your VLANs seperate from the default VLAN 1 hassle you have to configure ports in batch but not including your currently used port, and then swap ports around and set the remaining port. And sometimes you cannot set the remaining ports and have to delete the VLAN and re-create it.
VLAN and port naming has no memory boundary check
While it appears the VLAN IDs and ports can be named for lookup, setting them sometimes result in success but most time result in glitched characters. And when they’re glitched some other settings might get flushed. This is most likely a memory boundary checking issue and when it happens your only reliable fix is to revert everything, most likely to factory reset.
Under specific workload the switch would power cycle (only affecting Hellotek)
I’m using a single port on one of the Hellotek switch as an upstream trunk port to connect to the ISP modem, carrying both PPPoE Internet upstream in a VLAN and IPoE IPTV upstream in another VLAN, the IPTV VLAN is then carried in trunk through another Hellotek switch, then in trunk through a K2P running Openwrt acting as both switch and AP. Strangely the main switch i.e. the one connected directly to the modem would reset itself when IPTV stream start to run. E.g. if I don’t watch TV then everything seems fine, if I start watching TV and the IPTV set-top box initiated its IPoE stream then after a couple of minutes the main switch would reset, cutting both my Internet and IPTV connections. If they keep running for a few minutes after the switch comes back online then it would not reset any more, very very strange.
To get away from this issue I had to replace the main switch to a 10 Gbps L3 switch and add a 2.5 Gbps base-T SFP+ module.
So while these switches seem cheap, be prepared for various issues if you decide to use them for fancy network setups.
]]>I’ll assume you’ve already known what Btrfs is: a copy-on-write next-gen Linux filesystem that has entered the kernel for a long time and has many shiny features, of which I appreciate snapshoting and transparent compression the most.
Thanks to Btrfs’s nature of copy-on-write, to its helpful userspace toolset btrfs-progs, and to the fact that Btrfs subvolumes live in the filesystem namespace and the user perspective as if they’re just folders, it is travail to snapshot a Btrfs filesystem or part of it.
With the added benefit that a Btrfs subvolume can be mounted directly with specified subvol= or subvolid= mount argument and their content cannot be snapshotted as a part of the parent folder / subvlume they live in, you can combine your filesystem tree freely to decide which part to take snapshots on.
However, although it is travail to take a snapshot, it is hard to do it cleanly reguarlarly with handwritten script and crontab jobs / systemd.timer units that are easy to forget about and prune to fail. After all, if you’ve deleted a file by accident you’ve just created a few hours ago yet your last snapshot are taken a week ago, you’re doomed just like you don’t have snapshots / backup.
And of course you would want snapshots to be cleaned reguarly as few would really need a snapshot taken 3 months and 1 hour ago more than a snapshot taken exactly 3 months ago.
I’ll also assume you’ve already known what snapper is: a tool written by OpenSUSE to manage filesystem snapshots and allow undo of system modifications.
To describe it simply: with snapper installed, and its bundled timer units enabled, you can define a few configs that each define what subvolume to take snapshots regurlarly on, and snapper would create and clean up snapshots, and manage them in a centralized way under the corresponding .snapshots submount.
Let me take a few lines from my fstab to demonstrate how it works (only the mountpoints of the subvolume to take snapshots and their corresponding snapshots storage subvolume are listed, others, like /var/cache are left out):
# <file system> <dir> <type> <options> <dump> <pass>
## backpane m.2 2280 (2T 4.0 x4 downgraded to 3.0 x4)
ID=nvme-HYV2TBX4_GR__24113WJHA0000072-part3 / btrfs rw,compress=zstd:3,subvol=@ 0 0
ID=nvme-HYV2TBX4_GR__24113WJHA0000072-part3 /.snapshots btrfs rw,compress=zstd:3,subvol=@.snapshots 0 0
ID=nvme-HYV2TBX4_GR__24113WJHA0000072-part3 /home btrfs rw,compress=zstd:3,subvol=@home 0 0
ID=nvme-HYV2TBX4_GR__24113WJHA0000072-part3 /home/.snapshots btrfs rw,compress=zstd:3,subvol=@home_.snapshots 0 0
## expansion lower m.2 22110 (1T 3.0 x4)
ID=nvme-MZ1LB960HBJR-000FB_S5XBNA0R330322 /srv btrfs rw,compress=zstd:15,subvol=@srv 0 0
ID=nvme-MZ1LB960HBJR-000FB_S5XBNA0R330322 /srv/.snapshots btrfs rw,compress=zstd:15,subvol=@srv_.snapshots 0 0
## frontpane m.2 2280 (2T 3.0 x4)
ID=nvme-HYV2TBX3_HXY__00000000000000001116 /srv/backup btrfs rw,compress=zstd:15,subvol=@srv_backup 0 0
ID=nvme-HYV2TBX3_HXY__00000000000000001116 /srv/backup/.snapshots btrfs rw,compress=zstd:15,subvol=@srv_backup_.snapshots 0 0
Recall that a Btrfs subvolume’s content cannot be snapshotted as a part of the parent folder / subvlume they live in, the reason we have /, /home, /srv, /srv/backup as seperate mountpoints is to have them each snapshotted individually into their .snapshot subfolder / child subvolume, and */.snapshots are also their seperate subvolumes so their content won’t be snapshotted as part of their parent.
As Btrfs subvolumes can totoally just exist in the FS tree accessible to users as if they’re plain folders that just cannot be snapshooted as part of their parent subvolume, it is also possible to omit all these mountpoints and to have a single root subvolume mountpoint, but in that case it would be hard to follow which folder is in fact a subvolume unless you use btrfs-progs and corresponding commands. As always I prefer explicity over implicity, so I always set up Btrfs subvolumes directly in the FS root and mount them each individually into the actual root. You can freely decide which way to follow.
I have the following in my /etc/conf.d/snapper:
SNAPPER_CONFIGS="root home srv srv-backup"
Are I have the following snapper configs under /etc/snapper/configs/:
> ls -lh /etc/snapper/configs/
total 20K
-rw------- 1 root root 1.2K Jul 3 2024 home
-rw------- 1 root root 1.2K Jul 3 2024 root
-rw------- 1 root root 1.2K Jul 29 16:41 srv
-rw------- 1 root root 1.2K Jan 3 14:22 srv-backup
An example config, home, is defined as follows:
# subvolume to snapshot
SUBVOLUME="/home"
# filesystem type
FSTYPE="btrfs"
# btrfs qgroup for space aware cleanup algorithms
QGROUP=""
# fraction or absolute size of the filesystems space the snapshots may use
SPACE_LIMIT="0.5"
# fraction or absolute size of the filesystems space that should be free
FREE_LIMIT="0.2"
# users and groups allowed to work with config
ALLOW_USERS=""
ALLOW_GROUPS=""
# sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots
# directory
SYNC_ACL="no"
# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"
# run daily number cleanup
NUMBER_CLEANUP="yes"
# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="50"
NUMBER_LIMIT_IMPORTANT="10"
# create hourly snapshots
TIMELINE_CREATE="yes"
# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"
# limits for timeline cleanup
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="12"
TIMELINE_LIMIT_DAILY="3"
TIMELINE_LIMIT_WEEKLY="2"
TIMELINE_LIMIT_MONTHLY="6"
TIMELINE_LIMIT_YEARLY="1"
# cleanup empty pre-post-pairs
EMPTY_PRE_POST_CLEANUP="yes"
# limits for empty pre-post-pair cleanup
EMPTY_PRE_POST_MIN_AGE="1800"
With the above config, and the bundled snapper-timeline.timer and snapper-cleanup.timer systemd units enabled, snapper handles the /home subvolume as follows:
snapper-timeline.service to take a snapshot of /home and store it under /home/.snapshots/
1, 2, 135, 555, 705, then the new ID would be 706/home/.snapshots/706 would be created as the snapper snapshot container (It’s only my own calling, I don’t know how snapper calls them)/home would be created under the container folder, e.g. /home/.snapshots/237/snapshotThe corresponding metadata would be stored under the container folder, e.g. /home/.snapshots/237/info.xml, with content like the following:
<?xml version="1.0"?>
<snapshot>
<type>single</type>
<num>706</num>
<date>2024-12-22 16:00:05</date>
<description>timeline</description>
<cleanup>timeline</cleanup>
</snapshot>
snapper-cleanup.service to clean up snapshots under /home/.snapshots/
TIMELINE_LIMITE_HOURLY="12", only keep 12 hourly snapshots taken most recently (current, current - 1, … current - 11; but skip the one taken as 00:00 as it would be considered daily)snapper create (-c [config]), e.g. snapper create -c home, and manually removed by snapper delete (-c [config]) [snapshot ID], e.g. snapper delete -c home 1With the above setup, the folder /home/.snapshots would look like the following after running snapper for a long time:
> ls -lh /home/.snapshots/
total 100K
drwxr-xr-x 1 root root 32 Nov 22 18:00 1/
drwxr-xr-x 1 root root 32 Jan 5 00:00 1003/
drwxr-xr-x 1 root root 32 Jan 6 00:00 1027/
drwxr-xr-x 1 root root 32 Jan 7 00:00 1051/
drwxr-xr-x 1 root root 32 Jan 7 12:00 1063/
drwxr-xr-x 1 root root 32 Jan 7 13:00 1064/
drwxr-xr-x 1 root root 32 Jan 7 14:00 1065/
drwxr-xr-x 1 root root 32 Jan 7 15:00 1066/
drwxr-xr-x 1 root root 32 Jan 7 16:00 1067/
drwxr-xr-x 1 root root 32 Jan 7 17:00 1068/
drwxr-xr-x 1 root root 32 Jan 7 18:00 1069/
drwxr-xr-x 1 root root 32 Jan 7 19:00 1070/
drwxr-xr-x 1 root root 32 Jan 7 20:00 1071/
drwxr-xr-x 1 root root 32 Jan 7 21:00 1072/
drwxr-xr-x 1 root root 32 Jan 7 22:00 1073/
drwxr-xr-x 1 root root 32 Jan 7 23:00 1074/
drwxr-xr-x 1 root root 32 Jan 8 00:00 1075/
drwxr-xr-x 1 root root 32 Jan 8 01:00 1076/
drwxr-xr-x 1 root root 32 Jan 8 02:00 1077/
drwxr-xr-x 1 root root 32 Jan 8 03:00 1078/
drwxr-xr-x 1 root root 32 Jan 8 04:00 1079/
drwxr-xr-x 1 root root 32 Jan 8 05:00 1080/
drwxr-xr-x 1 root root 32 Jan 8 06:00 1081/
drwxr-xr-x 1 root root 32 Jan 8 07:00 1082/
drwxr-xr-x 1 root root 32 Jan 8 08:00 1083/
drwxr-xr-x 1 root root 32 Jan 8 09:00 1084/
drwxr-xr-x 1 root root 32 Jan 8 10:00 1085/
drwxr-xr-x 1 root root 32 Jan 8 11:00 1086/
drwxr-xr-x 1 root root 32 Dec 1 00:00 199/
drwxr-xr-x 1 root root 32 Dec 16 17:32 555/
drwxr-xr-x 1 root root 32 Dec 23 00:00 706/
drwxr-xr-x 1 root root 32 Dec 30 00:00 874/
drwxr-xr-x 1 root root 32 Jan 1 00:00 922/
drwxr-xr-x 1 root root 32 Jan 3 10:00 965/
drwxr-xr-x 1 root root 32 Jan 4 00:00 979/
An example snapper snapshot container folder /home/.snapshots/706/ would look like the following:
> ls -lh /home/.snapshots/706
total 4.0K
-rw-r--r-- 1 root root 187 Dec 23 00:00 info.xml
drwxr-xr-x 1 root root 36 Aug 26 15:56 snapshot/
We can confirm /home/.snapshots/706/snapshot is indeed a Btrfs snapshot / read-only subvolume
> btrfs subvolume show /home/.snapshots/706/snapshot/
@home_.snapshots/706/snapshot
Name: snapshot
UUID: b54462aa-3427-c745-8fcd-ac248143a039
Parent UUID: 0ce1faab-e8db-e94f-a3c8-be85743e5859
Received UUID: -
Creation time: 2024-12-23 00:00:05 +0800
Subvolume ID: 2265
Generation: 167279
Gen at creation: 167278
Parent ID: 259
Top level ID: 259
Flags: readonly
Send transid: 0
Send time: 2024-12-23 00:00:05 +0800
Receive transid: 0
Receive time: -
Snapshot(s):
Quota group: n/a
and its content is really what was in /home as that point:
> ls -alh /home/.snapshots/706/snapshot
total 0
drwxr-xr-x 1 root root 36 Aug 26 15:56 ./
drwxr-xr-x 1 root root 32 Dec 23 00:00 ../
drwxr-xr-x 1 root root 0 Apr 8 2024 .snapshots/
drwx------ 1 nomad7ji nomad7ji 1.3K Dec 22 22:30 nomad7ji/
and confirming what we said above, .snapshots as a mountpoint would not be recursively snapshotted:
> ls -alh /home/.snapshots/706/snapshot/.snapshots
total 0
drwxr-xr-x 1 root root 0 Apr 8 2024 ./
drwxr-xr-x 1 root root 36 Aug 26 15:56 ../
A minor thing to note there, is that snapper does not have its “database” that live outside of the subvolume and the container collection, it just calculates what’re already been created to decide new things such as new IDs. This is a thing I appreciate, as everything lives inside the snapshot container itself and you do not need extra data or even snapper itself to recover them. But this also has a “side-effect”: the ID is not strictly incremental, if you remove e.g. 565, 567, 900, 901, then the new ID could be 565, and if you remove all snapshots then they restart from ID 1 again.
Btrfs itself provides you with online data robustness: as long as it runs with CoW feature not turned off then it scrubs and checks data corruption like bitrots reguarlly, and it can report the corruption right away. It’s also not easy to be bricked unless you play with RAID56 which is discouraged by Btrfs developers. However, unless you set it up in a Btrfs native RAID1 or alike setup, the corruption cannot be fixed unless you have a backup, as the unique data has only one copy.
snapper enhances the online data robustness by providing the possibility to roll back: it avoids the data loss caused by either accidental, harmful or then-intentional-now-regretful operations. It has saved my ass for multiple times when I accidentally delete my work files, and thankfully I could get the last hourly snapshot to at least restore some of my work. However, as it only snapshots a subvolume on a single filesystem into subvolumes on the same exact filesytem, it is only a hot backup solution and won’t do magic if the FS itself breaks.
Both Btrfs itself and snapper lives on the hot, currently active storage, and if the underlying drives dies, they can’t have the magic to repair themselves and the data that die along with them.
For complete data robustness you need layered backup, and a solution to do warm/cold backup, which should in most cases be offline. One famous backup strategy that I follow is 3-2-1: to maintain 3 copies of data, to use 2 different types of media, and to keep at least 1 copy off-site.
With the plain Btrfs + snapper setup we already have 2 copies of data: 1 real-time and more than 1 copies in snapshots, I consider the “more than 1 copies in snapshots” part as 1, as the marginal effect decreases fast for the snapshots. What we miss is the last 1 off-site copy of data that’s stored on a different type of media.
When someone has already defined a good design, it’s better to follow it and improve it, rather than throw it away and start all over. As we already have snapper that does the online snapshots creation and cleaning, the best way to have an off-site backup is simply to backup the snapshots to another Btrfs storage so we can have the same robustness as snapper snapshots.
Introducing snasync, a snapper Btrfs snapshots syncer, to backup your Btrfs snapshots created by snapper to warm or cold, and in most cases remote storage.
snasync does one thing and one thing naively: to sync the snapper containers (my own calling, those folders that live under */.snapshots and containing info.xml and snapshot) one way to “targets”: either local, or remote.
As the snapper container ID is only useful to online snapper operations, not strictly incremental, and not meaningful on the timeline perspective, snasync sync local snapper containers to remote in [prefix]-[timestamp] name style. An example syncing map is as follows:
/home/.snapshots -> snasync@nas.lan:/srv/backup/snapshots
- /home/.snapshots/1 -> - snasync@nas.lan:/srv/backup/snapshots/pc_home-20241101030001
- /home/.snapshots/2 -> - snasync@nas.lan:/srv/backup/snapshots/pc_home-20241102030001
- /home/.snapshots/13 -> - snasync@nas.lan:/srv/backup/snapshots/pc_home-20241106030001
...
In which, only 1 would be sent as a whole subvolume, and 2 would be sent as its parent defined as 1, 13 would be sent as its parent defined as 2, etc.
Don’t worry about the snapper metadata as they’re still there:
> ls -lh /srv/backup/snapshots/rz5_root-20241122100006/
total 4.0K
-rw-r--r-- 1 root root 185 Jan 3 14:56 info.xml
drwxr-xr-x 1 root root 150 Jan 3 14:56 snapshot/
> cat /srv/backup/snapshots/rz5_root-20241122100006/info.xml
<?xml version="1.0"?>
<snapshot>
<type>single</type>
<num>1</num>
<date>2024-11-22 10:00:06</date>
<description>timeline</description>
<cleanup>timeline</cleanup>
</snapshot>
The timestamp is extracted from the metadata and explicited included in the name so it’s easier to look up without the help of snapper.
An example snasync run is as follows:
targets=(
--target /srv/backup/snapshots # a simple path refers to local target, this is my online warm backup
--target snasync@wtr.fuo.lan:/srv/backup/snapshots # a scp-style target means to sync to remote, this is my off-site cool backup (not entirely cold as not off-line)
)
snasync \
--source / --prefix rz5_root "${targets[@]}" \
--source /home --prefix rz5_home "${targets[@]}" \
--source /srv --prefix rz5_srv "${targets[@]}"
Basically you define a source to sync from (either the subvolume containing .snapshots, or .snapshots itself, snasync would figure it out), and the prefix to name the synced snapshots with (otherwise would be derived from source path), and then a few targets to sync them to, each source has their own targets, but in this case they’re all the same.
The whole operating logic of snasync is as follows:
snapper-cleanup.timer is running, stop it and register an on-exit trigger to start it again.snapper-cleanup.service is running, wait for it to finish.info.xml is missing, skip itinfo.xml and used as keysnapshot is read-only or contains received-UUID:
info.xml does not exist, copy the source one to itsnapshot is missing, or not read-only, or does not contain received-UUUID:
snapshot and info.xmlsnapshot to target
--compressed-data to keep the compressed data to save bandwidth, even if it’s running a local sync--parent [snapshot] if we have already seen synced snapshot to do only incremental sync, otherwise it does not have such argument--force-decompress to decompress the data first and then compress with the target compression options.orphan suffix. It’s up to users whether to delete or keep them.snapper-cleanup.timer if this was registeredOn my remote backup server the layout looks like the following:
> ls -l /srv/backup/snapshots/
total 0
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241126130020/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241224130016/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241224140003/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241225160003/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241225170003/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241226130022/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241226140022/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241227160003/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241227170003/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241227180003/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241228150001/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241228160001/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_home-20241228170003/
drwxr-xr-x 1 root root 32 Dec 29 17:55 dsk_root-20241126130020/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241224130016/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241224140003/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241225160003/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241225170003/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241226130022/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241226140022/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241227160003/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241227170003/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241227180003/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241228150001/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241228160001/
drwxr-xr-x 1 root root 32 Dec 29 17:57 dsk_root-20241228170003/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20240920170000/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20240930160000/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20241031160001/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20241130160020/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20241208160007/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20241215160017/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20241217160023.orphan/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20241218160025.orphan/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20241219160025/
drwxr-xr-x 1 root root 32 Dec 23 21:36 fuo_home-20241220160005/
......
drwxr-xr-x 1 root root 32 Dec 24 14:48 wtr_root-20241224040006/
drwxr-xr-x 1 root root 32 Dec 24 14:48 wtr_root-20241224050016/
drwxr-xr-x 1 root root 32 Dec 24 14:48 wtr_root-20241224060000/
As the snapshots are sent and received with parent if possible, and compressed on target which shall be mounted with higher compression level than source, the disk space they take is not very much:
> sudo btrfs filesystem du -s /srv/backup/snapshots/*
Total Exclusive Set shared Filename
25.81GiB 28.26MiB 18.96GiB /srv/backup/snapshots/dsk_home-20241126130020
25.80GiB 4.57MiB 18.97GiB /srv/backup/snapshots/dsk_home-20241224130016
25.79GiB 7.00MiB 18.96GiB /srv/backup/snapshots/dsk_home-20241224140003
25.79GiB 1.47MiB 18.96GiB /srv/backup/snapshots/dsk_home-20241225160003
25.79GiB 1.57MiB 18.96GiB /srv/backup/snapshots/dsk_home-20241225170003
25.79GiB 3.44MiB 18.96GiB /srv/backup/snapshots/dsk_home-20241226130022
25.79GiB 4.17MiB 18.96GiB /srv/backup/snapshots/dsk_home-20241226140022
...
Note the snasync only renames the containers that do not exist locally to have a .orphan suffix but not delete them, their content, snapshot and info.xml are still there:
> ls -lh /srv/backup/snapshots/fuo_home-20241217160023.orphan/
total 4.0K
-rw-r--r-- 1 root root 188 Dec 23 21:43 info.xml
drwxr-xr-x 1 root root 36 Jul 9 2024 snapshot/
One can remove all orhpaned containers and snapshots in them if they want to free up space:
> sudo btrfs subvolume delete /srv/backup/snapshots/*.orphan/snapshot
> sudo rm -rf /srv/backup/snapshots/*.orphan
When backed up data is needed one can simply navigate through all the containers and snapshots
nomad7ji@wtr /s/b/snapshots> cd rz5_home-20241122100006/
nomad7ji@wtr /s/b/s/rz5_home-20241122100006> ls
info.xml snapshot/
nomad7ji@wtr /s/b/s/rz5_home-20241122100006> cd snapshot/
nomad7ji@wtr /s/b/s/r/snapshot> ls
nomad7ji/
nomad7ji@wtr /s/b/s/r/snapshot> cd nomad7ji/
nomad7ji@wtr /s/b/s/r/s/nomad7ji> ls
Android/ Building/ Desktop/ Development/ Documents/ Downloads/ go/ Music/ opt/ Pictures/ Public/ Security/ Templates/ Videos/
While Btrfs + snapper + snasync setup provides enough robustness, all of them relies on the robustness of Btrfs filesystem, and it’s not impossible to fail, and you can not trust the underlying storage 100%.
When possible, please do regular offline cold backup. I do yearly BD-R backups with 25GB HTL BD-R disces with 10% parity volume, and hoard the disces at my parents’ and the parity volumes on network drive. It does not matter if the data is hard to retrieve, so it’s even OK to store it on e.g. AWS S3. They’re only needed when everything else fails, and at that point, every part of the work to retrieve them are always worth it.
]]>eval and is not very clean. A few weeks ago I found a better way to do this and I just had time to write it down.
Let’s start this with a full demo:
log_inner() {
if [[ "${log_enabled[$1]}" ]]; then
echo "[${BASH_SOURCE##*/}:${1^^}] ${FUNCNAME[2]}@${BASH_LINENO[1]}: ${*:2}"
fi
}
log_debug() {
log_inner debug "$@"
}
log_info() {
log_inner info "$@"
}
log_warn() {
log_inner warn "$@"
}
log_error() {
log_inner error "$@"
}
log_fatal() {
log_inner fatal "$@"
}
initialize() {
set -euo pipefail
declare -gA log_enabled=(
[debug]='y'
[info]='y'
[warn]='y'
[error]='y'
[fatal]='y'
)
local AIMAGER_LOG_LEVEL="${AIMAGER_LOG_LEVEL:-info}"
case "${AIMAGER_LOG_LEVEL,,}" in
'debug')
:
;;
'info')
log_enabled[debug]=''
;;
'warn')
log_enabled[debug]=''
log_enabled[info]=''
;;
'error')
log_enabled[debug]=''
log_enabled[info]=''
log_enabled[warn]=''
;;
'fatal')
log_enabled[debug]=''
log_enabled[info]=''
log_enabled[warn]=''
log_enabled[error]=''
;;
*)
log_fatal "Unknown log level ${AIMAGER_LOG_LEVEL}, shall be one of the"\
"following (case-insensitive): debug, info, warn, error, fatal"
return 1
;;
esac
}
work() {
log_warn "Started working..."
log_fatal "A fatal error occured!"
}
main() {
initialize
log_info "Started running..."
work
log_info "Ended"
}
main "$@"
Save the above to /tmp/scripter and run it with bash /tmp/scripter and you’ll have the following output:
[scripter:INFO] main@73: Started running...
[scripter:WARN] work@67: Started working...
[scripter:FATAL] work@68: A fatal error occured!
[scripter:INFO] main@75: Ended
As you can see we have both the script name, log level, function name, line number, and the original log.
Let me break it down and tell you how it works, let’s focus first on the essential inner logging function:
log_inner() {
if [[ "${log_enabled[$1]}" ]]; then
echo "[${BASH_SOURCE##*/}:${1^^}] ${FUNCNAME[2]}@${BASH_LINENO[1]}: ${*:2}"
fi
}
The function’s if-condition does a simple thing: check whether the log level from arg1 ($1) was enabled, and only print everything from arg2 ($2) onwards with a log prefix when it is enabled.
In the printing line, the used variables and their definitions are:
$BASH_SOURCE is a Bash built-in variable, storing the path of the Bash file that was interpreted, here it would be /tmp/scripter, and as ##*/ removes everything before the last / (inclusive), ${BASH_SOURCE$$*/} would be scripter;$1 is the log level, ${1^^} converts the log level to upper case, so if $1 is info then ${1^^} is INFO;$FUNCNAME is a Bash built-in array containing all of the function names in the stack, backwards in the order they were called, so the full array here would be FUNCNAME=(log_inner log_[level] main), the one we want is thus ${FUNCNAME[2]}, the actual function that calls our logging wrappers;$BASH_LINENO is a Bash built-in array containing all of the places where the functions are called in the stack, backwards in the order they were called, so the full array here would be BASH_LINENO=([line No. where log_inner was called] [line No. where log_[level] was called]), the one we want is thus ${BASH_LINENO[1]}, the line number where our logging wrapper was called.$* is a Bash magic variable that contains all arguments to the current context (here log_innner) joined by the first character in $IFS (here space) to a single string, we want ${*:2}, which only contains the second to last argument.A log wrapper then is defined as follows:
log_info() {
log_inner info "$@"
}
It just passes its built-in log level and all remaining arguments to the inner printing function. The reason I use dedicated log_info, log_warn, etc instead of using the inner log_inner directly is that I want the logging behaviour to be strictly explicit (remember that we have set -u (using undefined variable would result in error) and set -e (error would result in Bash exiting), so caller can only call log_info, this avoids the error that one might send the wrong logging level)
When one wants to log something they can use one of the logging wrapper in their function:
work() {
...
log_fatal "A fatal error occured!"
...
}
This prints the following log, which is very helpful when debugging:
[scripter:FATAL] work@68: A fatal error occured!
Note also we have a helper struct to record whether a log level was enabled, instead of figuring it out everything logging was triggered:
declare -gA log_enabled=(
[debug]='y'
[info]='y'
[warn]='y'
[error]='y'
[fatal]='y'
)
To disable a logging level just set the corresponding value to empty:
local AIMAGER_LOG_LEVEL="${AIMAGER_LOG_LEVEL:-info}"
case "${AIMAGER_LOG_LEVEL,,}" in
...
'error')
log_enabled[debug]=''
log_enabled[info]=''
log_enabled[warn]=''
;;
...
esac
This saves the work that is needed when figuring out whether a log level shall be enabled: getting the value from an assotiated Bash array is much faster than doing calculation every time from environment
Like the following:
if [[ "${log_enabled[info]}" ]]; then
do_complex_logging_when_info_level_is_enabled
fi
is of course lighter than the following:
case "${AIMAGER_LOG_LEVEL}" in
info|warn|error|fatal)
do_complex_logging_when_info_level_is_enabled
;;
esac
If there’s no need to figure out the log level again during runtime, then the demo can be simplified to the following:
log_inner() {
echo "[${BASH_SOURCE##*/}:$1] ${FUNCNAME[2]}@${BASH_LINENO[1]}: ${*:2}"
}
log_debug() {
log_inner DEBUG "$@"
}
log_info() {
log_inner INFO "$@"
}
log_warn() {
log_inner WARN "$@"
}
log_error() {
log_inner ERROR "$@"
}
log_fatal() {
log_inner FATAL "$@"
}
initialize() {
set -euo pipefail
local AIMAGER_LOG_LEVEL="${AIMAGER_LOG_LEVEL:-info}"
case "${AIMAGER_LOG_LEVEL,,}" in
'debug')
:
;;
'info')
log_debug() { :; }
;;
'warn')
log_debug() { :; }
log_info() { :; }
;;
'error')
log_debug() { :; }
log_info() { :; }
log_warn() { :; }
;;
'fatal')
log_debug() { :; }
log_info() { :; }
log_warn() { :; }
log_error() { :; }
;;
*)
log_fatal "Unknown log level ${AIMAGER_LOG_LEVEL}, shall be one of the"\
"following (case-insensitive): debug, info, warn, error, fatal"
return 1
;;
esac
}
work() {
log_warn "Started working..."
log_fatal "A fatal error occured!"
}
main() {
initialize
log_info "Started running..."
work
log_info "Ended"
}
main "$@"
eval calls and it’s recommend to read the new blog post
When writing some lengthy bash script, one might want the script to log with function name and line number so it would be easy to trace some error-prone logics in the future, like how you would use __FUNCTION__ and __LINE__ macros in C projects compiled with GCC.
Luckily there’re $FUNCNAME and $LINENO built-in variables. So you could write your logging statement like this:
myfunc() {
echo "[DEBUG] ${FUNCNAME}@${LINENO}: Starting some work..."
if work; then
echo "[INFO] ${FUNCNAME}@${LINENO}: Successfully finished the work..."
else
echo "[ERROR] ${FUNCNAME}@${LINENO}: Failed to do the work !"
return 1
fi
}
However writing these lengthy prefixes is both annoying and error-prone, and it reduces the information density which is unhelpful when you go back to improve the codes.
It’s of course not possible to replace these with functions, as $FUNCNAME and $LINENO would then not trace the place where functions are actually called, but only their inner state. And it would be even more tedious if you want to conditionally log depending on the log level.
To simplify the latter typing work while keeping $FUNCNAME and $LINENO as where they’re called, and have some conditonal log levels, you can define some “macros” that would be expanded by eval:
log_common_start='echo -n "['
log_common_end='] ${FUNCNAME}@${LINENO}: " && false'
log_info="${log_common_start}INFO${log_common_end}"
log_warn="${log_common_start}WARN${log_common_end}"
log_error="${log_common_start}ERROR${log_common_end}"
log_fatal="${log_common_start}FATAL${log_common_end}"
# Debugging-only definitions
if [[ "${aimager_debug}" ]]; then
log_debug="${log_common_start}DEBUG${log_common_end}"
else
log_debug='true'
fi
With the above definition you can write that function instead like this:
myfunc() {
eval "$log_debug" || echo 'Starting some work...'
if work; then
eval "$log_info" || echo 'Successfully finished the work...'
else
eval "$log_error" || echo 'Failed to do the work !'
return 1
fi
}
The way this works is that, for logging levels enabled, the logging lines are basically expanded to:
echo "prefix" && false || echo 'content'
and both echoes would be executed in this case;
and for logging levels disabled, the logging lines are basically expanded to:
true || echo 'content'
and no echo would be executed in this acse
As a bonus point, you can execute some logics dynamically depending on the log level, e.g.
if ! eval "$log_info"; then
echo 'Running specific logic when logging level INFO is enabled'
some_logic_when_info_is_enabled
else
other_logic_when_info_is_disabled
fi