Posted by Andrew Denner on April 05, 2026 · 20 mins read

Note: This post expands on TII’s Falcon Perception release from April 3, 2026. I ran the analysis through my usual multi-tool stack — Claude for the deep dive, Grok for adversarial pressure-testing — because one AI’s opinion is just vibes. Two AIs arguing with each other starts to look like peer review. Opinions are my own.

I have a confession: I spent last Thursday night reading a paper about positional embeddings that uses the golden ratio to achieve isotropic attention over 2D pixel grids. On purpose. For fun.

If that sentence made your eyes glaze over, stick around anyway, because what TII just released matters even if you never plan to implement a vision-language model. Falcon Perception is a 600-million-parameter model that does something nobody else has pulled off at this scale: you give it a photo and a natural language description like “the red mug behind the laptop,” and it gives you back a pixel-perfect segmentation mask. Not a bounding box. Not a rough outline. A mask.

And it does this better than Meta’s SAM 3 — which has more parameters — and demolishes Alibaba’s Qwen3-VL-30B — which has fifty times more parameters — on complex compositional queries. That’s the kind of result that makes you read a paper about the golden ratio at 11 PM on a Thursday.

Why This Matters Beyond Computer Vision Twitter

The thing about most AI model releases is that they matter to a very specific group of people who spend a lot of time on arXiv and not a lot of time explaining things at dinner parties. Falcon Perception is different because it validates a thesis that has implications across all of AI: architecture matters more than scale.

We’ve been in the “just make it bigger” era for a few years now. More parameters, more data, more compute. And look, that works — GPT-5 and Claude are proof. But Falcon Perception shows that when you rethink the architecture from first principles instead of just bolting more layers onto the same design, you can get dramatically better results with dramatically less.

For those of us who care about running things locally — and I absolutely care about running things locally — this is the whole ballgame. A model that fits in 2.5 GB of VRAM and beats 30-billion-parameter competitors isn’t just a research curiosity. It’s the difference between needing an $8,000 GPU and needing the GPU that’s already in your desktop.

The Core Innovation: Throwing Out the Lego Bricks

Here’s how every other vision-language model works (broadly): take an image, run it through a frozen vision encoder like CLIP or ViT, get a bag of visual features. Take some text, run it through a language model. Then somewhere in the middle, try to make these two streams of information talk to each other through projection layers or cross-attention modules. It’s a Lego-brick approach — snap the vision piece onto the language piece and hope the seams don’t show.

Falcon Perception says: what if we just… didn’t do that?

Instead, image patches and text tokens go into the same transformer from layer one. No separate vision encoder. No projection layer. No Lego bricks. One backbone, shared parameters, from the very first attention operation. This is “early fusion” — and it sounds simple, but making it work requires solving two hard problems.

Problem 1: Images and text need different attention patterns. When you’re processing an image, you want every patch to see every other patch (bidirectional attention) because spatial context matters — the pixels on the left need to know about the pixels on the right. But when you’re generating text, you need causal attention — each token can only see the tokens before it, or you’re leaking the answer into the question.

Falcon Perception solves this with a hybrid attention mask within a single model. Image tokens attend bidirectionally. Text and task tokens attend causally. The same transformer acts simultaneously as a vision encoder and a language decoder. They implemented this using PyTorch’s FlexAttention API, which compiles the custom mask pattern into fused GPU kernels without ever materializing the full N×M attention matrix in memory.

Problem 2: Flattening a 2D image into a 1D token sequence destroys spatial information. Standard RoPE (Rotary Position Embeddings) — the thing that tells a transformer where tokens are in a sequence — only encodes 1D position. Even axial 2D RoPE can only attend along rows or columns, not diagonally. So TII invented GGRoPE (Golden Gate RoPE), which assigns each dimension pair a direction vector rotated by multiples of π/φ (where φ is the golden ratio, ~1.618). This produces maximally uniform angular coverage over the 2D plane, meaning the model can attend to arbitrary 2D positions isotropically. The golden ratio shows up because it’s the most irrational number — its multiples produce the most evenly-spaced distribution of angles, just like sunflower seeds arrange themselves using the golden angle.

If that paragraph felt dense, here’s the Iowa farmer version: they figured out how to tell the model where pixels actually are in 2D space even though the model processes them as a flat list, and they used the same math that makes sunflowers pretty.

How It Actually Generates a Segmentation: Chain-of-Perception

When you give Falcon Perception a query like “the blue car on the left,” it doesn’t just spit out a mask. It follows a structured sequence called Chain-of-Perception:

1. Existence decision: Is this thing present? (<present> or <absent>)
2. Center coordinate: Where is it?
3. Size estimate: How big is it?
4. Segmentation embedding: Generate the mask.

This ordering is deliberate and clever. By committing to “does this exist?” before trying to locate it, the model avoids hallucinating masks for objects that aren’t there. By resolving position and size before segmentation, the mask prediction is really just pixel refinement conditioned on already-resolved geometry. Each step constrains the next.

The coordinates aren’t tokens — they’re continuous values that get projected through Fourier features (random Gaussian matrix → sinusoidal space) to overcome neural networks’ natural spectral bias toward low-frequency functions. This is the kind of detail you’d miss in a press release but matters enormously for precision.

The Numbers That Made Me Stop Scrolling

The benchmark that matters most here is PBench, a new diagnostic that isolates five levels of semantic complexity. On simple object recognition (L0: “find the dog”), the gap between Falcon Perception and SAM 3 is negligible (+0.8 points). But watch what happens as prompts get harder:

• L1 (Attributes): +9.2 points over SAM 3
• L2 (OCR-guided, e.g., “the bottle labeled Pepsi”): +13.4 points
• L3 (Spatial understanding, e.g., “second from right”): +21.9 points
• L4 (Relations, e.g., “the dog being pet by the child”): +15.8 points
• Dense split (many objects, crowd scenes): +14.2 points

And then there’s the gut-punch number: on the PBench Dense split, Falcon Perception (0.6B parameters) scores 72.6. Qwen3-VL-30B scores 8.9. A model fifty times smaller scores eight times higher. That’s not a rounding error. That’s a paradigm.

The reason is architectural: early fusion lets visual and linguistic information interact at every layer, so compositional reasoning (“the red one behind the blue one that’s left of the door”) resolves naturally through deep cross-modal interaction. Late-fusion models see the image separately, understand the text separately, and try to match them up at the end — which works fine for “find the dog” but falls apart when the query requires genuine spatial reasoning.

The Honest Trade-offs (Because Every AI Blog Post Should Have This Section)

Falcon Perception is not magic and it’s not a general-purpose tool. Here’s what you give up:

It’s not a general VLM. You cannot ask it “What’s happening in this image?” or “Write a caption for this photo.” It does one thing — open-vocabulary grounding and segmentation from natural language — and does it better than anything else at this scale. If you need visual question answering, captioning, or multi-step reasoning, you still want Qwen-VL, LLaVA, or Claude’s vision capabilities.

Presence calibration is weaker. SAM 3 has an MCC (Matthews Correlation Coefficient) of 0.82 for deciding whether an object exists in a scene; Falcon Perception scores 0.64. That means more false positives — the model will sometimes generate a mask for something that isn’t there. For safety-critical applications (medical imaging, autonomous driving), this gap matters. TII says early RL experiments are closing it (+8 points already), but the released model has this limitation.

It’s brand new. Released March 31, 2026. Forty GitHub stars. No Ollama support, no GGUF quantization, no third-party inference providers. SAM 3 and YOLO-World have massive ecosystems. Falcon Perception has a research paper and a HuggingFace repo. If you need production stability today, it’s early.

English only (for OCR). The companion Falcon OCR model (300M parameters) performs well on document understanding benchmarks but only supports English. PaddleOCR supports 109 languages. If your documents aren’t in English, this isn’t your tool yet.

Running It Locally: Yes, On Your Actual Hardware

At 600M parameters in FP16, the model weights are about 1.2 GB. Total VRAM during inference is roughly 2–4 GB. That means:

• Any modern NVIDIA GPU: Works. A GTX 1650 (4 GB) will handle it.
• Apple Silicon Mac: Works via the MLX backend — Metal GPU acceleration, no PyTorch needed.
• Raspberry Pi / Edge: Not currently viable. Needs CUDA or Apple Silicon.
• CPU-only: Not officially supported; would be very slow.

The code is about as simple as vision models get:

from transformers import AutoModelForCausalLM
from PIL import Image

model = AutoModelForCausalLM.from_pretrained(
    "tiiuae/falcon-perception",
    trust_remote_code=True,
    device_map={"": "cuda:0"},
)
predictions = model.generate(Image.open("photo.jpg"), "red mug")[0]

No quantization needed — and none currently available — because the model is already small enough to be practical at FP16. The initial torch.compile warmup takes 10–30 seconds, after which inference is roughly 350ms per query on an H100, somewhat slower on consumer cards.

Where This Fits in the Broader Landscape

The vision-language space is fragmented across overlapping tool categories, and Falcon Perception occupies a specific niche. Here’s how I think about it:

If you need pure OCR: Falcon OCR (300M) is competitive on English documents, but PaddleOCR and Tesseract still own multilingual. Cloud services (Google Vision, Azure, Textract) win on production reliability but cost real money at scale.

If you need real-time detection: YOLO-World runs at 52+ FPS and is production-proven. It gives you boxes, not masks, but for many applications boxes are fine. Falcon Perception is 150–350ms per query — fast, but not real-time.

If you need “find the thing I described and give me a pixel mask”: Falcon Perception is now the best open-source option at any scale. The previous approach was a Grounding DINO → SAM pipeline (two models, more VRAM, more latency, box-then-refine instead of direct prediction). Falcon collapses that into one model call.

If you need a general visual assistant: Claude, GPT-5, or Qwen-VL. Falcon Perception can’t converse about images — it can only ground and segment.

What This Means for My Work

I’ve been building science fair review tools that need to process student posters, lab notebooks, and compliance forms. The OCR piece is interesting — Falcon OCR at 300M parameters is small enough to run in a pipeline on modest hardware. The grounding piece could theoretically identify specific elements on a poster (“the data table” or “the hypothesis statement”) without pre-training on science fair templates. I haven’t tested this yet — and I want to, badly — but the architecture seems purpose-built for exactly this kind of “find the thing I described in this complex document” task.

More broadly, Falcon Perception is evidence that the moat in AI is shifting from “who has the most GPUs” to “who has the best architectural ideas.” TII, backed by Abu Dhabi’s sovereign research fund, keeps releasing models that punch far above their parameter weight. The Falcon family now spans language models (Falcon 3), hybrid architectures (Falcon-H1), reasoning (Falcon-H1R), edge deployment (Falcon Edge with 1.58-bit BitNet), and now dense visual perception. All Apache 2.0. All running on hardware you can actually afford.

The era of “you need a data center to do anything interesting with AI” isn’t over, but it’s developing cracks. And those cracks are shaped like a 600-million-parameter falcon.

Links:

• arXiv paper: arxiv.org/abs/2603.27365
• HuggingFace model: tiiuae/Falcon-Perception
• GitHub: tiiuae/Falcon-Perception
• HuggingFace blog post
• Live playground: vision.falcon.aidrc.tii.ae

Andy Denner is a scientific computing scientist and runs Denner Consulting LLC. He builds AI-powered compliance tools for science fairs, advocates for privacy, and occasionally reads papers about the golden ratio on Thursday nights. Find him on X at @adenner.

OSTree, User Namespaces, D-Bus Portals, and Why ~/.var/app Is So Large

Andrew Denner · March 2026 · denner.co Part 4 of the Linux Universal Packages series

Flatpak is the one I use daily. It’s also the one I understand well enough to know exactly what it’s doing when it annoys me. This part goes into the Bubblewrap sandbox, user namespaces, how OSTree stores runtimes, and the portal system that most people have seen without knowing what it is.

The Design Philosophy

Flatpak’s design differs from Snap’s in a few fundamental ways worth stating upfront:

Decentralized: Flatpak has no central authority or mandatory store. You can run a Flatpak from any remote. Flathub is the de facto central repository but it’s not baked into the architecture. You can add your own remote, your company’s internal Flatpak repo, or install local .flatpak bundles.

User namespaces, not setuid: Flatpak’s sandbox uses Linux user namespaces rather than setuid binaries (Snap’s snap-confine is setuid). This is a philosophical difference — user namespaces allow sandbox creation without elevated privileges. The tradeoff is more complex setup, and on some systems user namespaces have had security vulnerabilities.

OSTree for runtime management: Runtimes (shared libraries, GTK, Qt) are stored and versioned using OSTree — a content-addressable, git-like filesystem. This is a genuinely novel approach to dependency management.

Portal-mediated resource access: Rather than granting apps direct access to system resources (microphone, camera, files), Flatpak routes access through XDG portals — OS-managed D-Bus services that apply additional policy.

Bubblewrap: The Sandbox Engine

Flatpak uses Bubblewrap (bwrap) as its sandbox primitive. Bubblewrap is a small C binary that creates Linux namespace containers without requiring root.

Linux User Namespaces

The key technology: user namespaces (CLONE_NEWUSER). A user namespace creates a mapping between user IDs inside the namespace and UIDs outside. Inside a user namespace, unprivileged code can simulate being root (UID 0) while the kernel maps that back to a real unprivileged UID.

// Simplified concept of what bwrap does:
clone(CLONE_NEWUSER | CLONE_NEWNS | CLONE_NEWPID | CLONE_NEWIPC | CLONE_NEWUTS, ...)
// Creates: new user ns + new mount ns + new pid ns + new ipc ns + new uts ns
// Then writes the UID/GID mapping:
// /proc/<pid>/uid_map: "0 <real_uid> 1"   (inside uid 0 = outside uid $USER)
// /proc/<pid>/gid_map: "0 <real_gid> 1"

This namespace stack is what allows Flatpak to create a sandbox without root. Inside the sandbox:

• The app sees itself as running as UID 0 in some operations (for filesystem setup)
• But the kernel knows this maps to your real unprivileged UID
• Root operations that require real root (loading kernel modules, changing system time) still fail

What bwrap Actually Creates

When you launch a Flatpak application, Flatpak invokes bwrap with a complex set of arguments that define the sandbox’s filesystem. A simplified version:

bwrap \
  --ro-bind /usr/share/flatpak/exports/share/runtime/org.gnome.Platform/x86_64/45/files /usr \
  --ro-bind /app /app \
  --bind /run/user/1000/flatpak/app/org.gimp.GIMP /run/user/1000 \
  --bind /home/user/.var/app/org.gimp.GIMP /home/user \
  --tmpfs /tmp \
  --proc /proc \
  --dev /dev \
  --unshare-pid \
  --unshare-ipc \
  --new-session \
  -- /app/bin/gimp-2.10

The app’s filesystem view is a construction of bind mounts:

• /usr is the runtime (GNOME Platform), not the system’s /usr
• /app is the Flatpak app installation directory (read-only)
• ~ is a redirected path pointing to ~/.var/app/org.gimp.GIMP (the app’s isolated data dir)
• /tmp is a fresh tmpfs (isolated)

The app never sees your real /usr, never sees other apps’ data, and its ~ is a sandboxed directory, not your real home.

The Mount Namespace

Inside the mount namespace, Flatpak can set up an arbitrary filesystem hierarchy using bind mounts — mapping paths from the outside namespace into specific locations inside. The kernel enforces that nothing inside can see mounts from outside unless explicitly bound in.

# You can inspect a running Flatpak's mount namespace:
FLATPAK_PID=$(pgrep -f "gimp")
cat /proc/$FLATPAK_PID/mounts | head -20

# Shows the bind-mounted directories that make up the sandbox view
# /var/lib/flatpak/runtime/org.gnome.Platform/.../files on /usr type ext4 (ro,...)
# tmpfs on /tmp type tmpfs
# ...

OSTree: Git for Filesystems

Flatpak uses OSTree to store and distribute runtimes. OSTree is the same technology used in Fedora CoreOS, RHEL for Edge, and some automotive embedded Linux systems. Understanding OSTree explains Flatpak’s storage model.

The Content-Addressable Store

OSTree stores file content as objects keyed by SHA256 hash, just like git stores blobs. Files with identical content are stored once, regardless of how many apps or runtime versions reference them.

The OSTree repository on a Flatpak system:

/var/lib/flatpak/repo/           # System Flatpak repo
~/.local/share/flatpak/repo/     # User Flatpak repo

repo/
├── config                       # Remote configuration
├── refs/
│   └── remotes/
│       └── flathub/
│           └── runtime/
│               └── org.gnome.Platform/x86_64/45  # → <commit hash>
├── objects/
│   ├── aa/                      # Files stored by hash prefix
│   │   └── bb1234...            # Content object
│   └── ...
└── state/

When you install GNOME Platform 45, OSTree downloads and stores each file by its content hash. When GNOME Platform 45 is updated to 45.1, only changed files are downloaded — unchanged files are already in the object store.

# Inspect the OSTree repo:
ostree --repo=/var/lib/flatpak/repo refs
# Shows all installed runtimes and apps as OSTree refs

# See what changed between two commits:
ostree --repo=/var/lib/flatpak/repo diff <old-commit> <new-commit>

Deployment: Checking Out Runtimes

When Flatpak needs to use a runtime, it “deploys” the OSTree checkout — creating a directory containing the runtime files, with OSTree’s hardlink optimization:

/var/lib/flatpak/runtime/org.gnome.Platform/x86_64/45/
└── active/
    ├── deploy              # OSTree metadata
    └── files/              # The actual runtime files
        ├── bin/
        ├── lib/
        └── share/

Critically, OSTree deployments use hardlinks to the object store wherever possible. Multiple runtime versions that share files share those files on disk (same inode). This is why the “500MB runtime” number is misleading — if GNOME Platform 44 and 45 share 80% of their files, the second installation uses much less disk than the first.

# Check actual disk usage with deduplication accounted for:
flatpak disk-usage
# This accounts for shared files; raw du output does not

The du -sh ~/.var/app trick I mention in the talk only shows the app data directory. The runtimes live in /var/lib/flatpak/runtime/ (system install) or ~/.local/share/flatpak/runtime/ (user install). Those are the ones that grow.

XDG Desktop Portals: The Broker System

This is Flatpak’s most architecturally interesting idea. Instead of giving apps direct access to sensitive resources, they communicate through portals — D-Bus services provided by the host desktop environment.

Why Portals Exist

A sandboxed app can’t open /home/user/Documents/ directly. But it can send a D-Bus message to org.freedesktop.portal.FileChooser.OpenFile(). The portal service (running outside the sandbox) shows a native file picker dialog. When the user selects a file, the portal returns a file descriptor to the app. The app never sees the filesystem path — it just gets a file descriptor to read.

This means:

• The app can only access files the user explicitly chose
• The portal can apply additional policy (e.g., respecting the user’s sandboxed directory)
• The file picker looks native because it is native — it’s run by the host, not the app

Portal Types

The XDG portal stack defines over a dozen portal interfaces:

org.freedesktop.portal.FileChooser: Open/save file dialogs. The most commonly used portal. Every time a Flatpak app shows a file chooser, this is what’s happening.

org.freedesktop.portal.OpenURI: Open a URI using the system default handler. When a Flatpak app opens a URL, it asks this portal — which then uses your system’s configured browser.

org.freedesktop.portal.Screenshot: Take a screenshot. The portal shows a confirmation dialog if desired.

org.freedesktop.portal.ScreenCast: Screen recording/sharing (used by Zoom, OBS, Pipewire-based screensharing). The portal negotiates a PipeWire stream rather than giving the app direct /dev/video* access.

org.freedesktop.portal.Camera: Similar — camera access via PipeWire stream, with user confirmation.

org.freedesktop.portal.Notifications: Desktop notifications. Apps can’t write to the notification system directly.

org.freedesktop.portal.Settings: Read desktop settings (color scheme, font, etc.) from the host environment. This is how Flatpak apps know whether to use dark mode.

org.freedesktop.portal.Background: Request permission to run in the background/at startup.

org.freedesktop.portal.Email: Compose an email (opens the default mail client).

org.freedesktop.portal.Secret: Access the keyring/secret service.

The Portal Architecture

Flatpak App (sandboxed)
  │
  │ D-Bus message to portal
  ▼
org.freedesktop.portal.Desktop (xdg-desktop-portal)
  │
  │ Routes to implementation
  ▼
org.freedesktop.impl.portal.gtk  (GNOME implementation)
org.freedesktop.impl.portal.kde  (KDE implementation)
org.freedesktop.impl.portal.wlr  (wlroots compositors)
  │
  │ Shows native UI (file picker, etc.)
  ▼
Result → file descriptor / URI / permission granted
  │
  ▼
Back to sandboxed app

The portal implementation is chosen based on your desktop environment. On GNOME, xdg-desktop-portal-gnome provides native GNOME file pickers. On KDE, xdg-desktop-portal-kde provides KDE dialogs. This is why Flatpak apps can show native-looking file pickers even when running outside their “home” desktop.

# Running portals on your system:
ps aux | grep xdg-desktop-portal
# xdg-desktop-portal (the broker)
# xdg-desktop-portal-gnome (or -kde, -gtk)

# D-Bus introspection of available portal interfaces:
busctl --user introspect org.freedesktop.portal.Desktop /org/freedesktop/portal/desktop

PipeWire and ScreenCast: The Modern Media Story

The ScreenCast portal is worth special attention because it solved a genuine problem. Before PipeWire, sandboxed apps had no good way to do screen capture or audio routing. X11 apps could grab the framebuffer, but Wayland compositors deliberately prevent this (no direct framebuffer access for apps).

The solution: the ScreenCast portal negotiates a PipeWire stream with the compositor. The compositor captures the screen and provides it as a PipeWire media stream. The sandboxed app connects to the PipeWire stream and gets video frames without ever having direct access to the display server.

This is why Flatpak apps (like Zoom’s Flatpak) can do screen sharing on Wayland. It’s also why obs-studio as a Flatpak works with PipeWire screen capture.

The .flatpak-info File: Sandbox Self-Inspection

Every running Flatpak app has a .flatpak-info file visible at /run/user/<uid>/flatpak/app/<app-id>/.flatpak-info. This file describes the app’s sandbox configuration.

# While GIMP is running:
cat /run/user/$(id -u)/flatpak/app/org.gimp.GIMP/.flatpak-info

# [Application]
# name=org.gimp.GIMP
# runtime=runtime/org.gnome.Platform/x86_64/45
#
# [Instance]
# instance-id=1234567890
# app-path=/var/lib/flatpak/app/org.gimp.GIMP/...
# runtime-path=/var/lib/flatpak/runtime/org.gnome.Platform/...
#
# [Context]
# shared=network;ipc
# filesystems=host

This file is used by portal services to verify the app’s identity and determine which permissions it has. A portal can read this to decide whether to grant a request.

Flatpak Permissions: The Full Model

Flatpak permissions come from two sources: the app’s manifest (what the developer declared) and overrides (what the user has modified).

The Manifest Context

In a Flatpak manifest (used to build the app), the developer declares what permissions the app needs:

finish-args:
  - --share=network          # Access the network
  - --share=ipc              # Shared memory IPC (X11 apps need this)
  - --socket=wayland         # Wayland display server
  - --socket=fallback-x11    # X11 fallback
  - --socket=pulseaudio      # PulseAudio/PipeWire audio
  - --device=dri             # GPU/DRI access (hardware rendering)
  - --filesystem=home        # Access home directory
  - --filesystem=host        # Access entire filesystem (dangerous)
  - --talk-name=org.gtk.vfs.daemon  # Talk to this D-Bus service
  - --env=GDK_BACKEND=wayland       # Set environment variable

These permissions are compiled into the Flatpak and stored in the repo. When you install an app, these are the declared permissions.

User Overrides

Users can override permissions in both directions — grant more or restrict more:

# Grant filesystem access that the manifest didn't declare:
flatpak override --user --filesystem=~/Projects org.gimp.GIMP

# Remove a permission the manifest declared:
flatpak override --user --nofilesystem=home org.gimp.GIMP

# Grant access to a specific D-Bus name:
flatpak override --user --talk-name=org.freedesktop.NetworkManager org.example.App

# See all overrides:
flatpak override --user --show org.gimp.GIMP

# Reset all overrides:
flatpak override --user --reset org.gimp.GIMP

Flatseal is a GUI front-end for this override system. It reads the manifest’s declared permissions, shows the current override state, and lets you toggle them. This is what I call “transparent permissions” — you can see exactly what each app can do.

The --filesystem=host Problem

--filesystem=host gives the Flatpak full access to the host filesystem (same as your user). Many apps include this because it’s easier than figuring out which specific paths they need.

# Check which Flatpaks have --filesystem=host:
flatpak info --file-access <app-id>
# Or via Flatseal: look for "All user files" in Filesystem section

When an app has --filesystem=host, the sandbox’s filesystem isolation provides almost no protection. The app can read and write your entire home directory. This is legal in Flatpak — the developer declared it and Flathub allows it with justification — but it’s worth knowing.

Flathub: The Store and the Review Process

Flathub is not part of Flatpak — it’s a community-operated Flatpak remote hosted at https://dl.flathub.org/. But it’s so central to the Flatpak ecosystem that understanding it matters.

Submission and Review

The review process for Flathub submissions:

1. Developer submits a PR to github.com/flathub/flathub with their manifest
2. Automated checks verify: manifest syntax, allowed build modules, no vendored binaries, no network access during build
3. Human reviewers check: claimed permissions are reasonable, sources are legitimate, app ID matches domain control
4. Initial approval creates the app repository
5. Future updates are published by the developer (no per-update review after initial approval, but automated checks run)

Notable review requirements:

• No proprietary bundled binaries in open-source apps (must be built from source)
• App IDs must be reverse-domain with domain ownership (or use io.github.username.AppName)
• No excessive permissions without justification (reviewers flag --filesystem=host for non-file-manager apps)
• Sources must be verifiable (checksums/commits required, no bare URLs)

Proprietary apps (Discord, Spotify, Steam) are allowed on Flathub but go through additional review. They’re typically binary bundles since the source isn’t available.

Verification

Flathub supports developer verification — proving you control the app’s domain or GitHub identity. Verified apps show a checkmark in GNOME Software and other store frontends.

# See Flathub metadata including verification status:
flatpak remote-info flathub org.gimp.GIMP

The Storage Architecture: What Takes Disk Space

Let me be precise about where disk usage comes from:

System installation (/var/lib/flatpak/):
├── repo/                           # OSTree objects (content-addressable)
│   └── objects/                    # Shared file content, hardlinked
├── runtime/                        # Deployed runtimes
│   ├── org.gnome.Platform/         # ~600MB-1GB
│   ├── org.kde.Platform/           # ~600MB (if KDE apps installed)
│   └── org.freedesktop.Platform/   # ~300MB (base for many apps)
└── app/                            # Deployed apps
    ├── org.gimp.GIMP/              # App files (not user data)
    └── org.mozilla.firefox/

User installation (~/.local/share/flatpak/):
└── (Same structure, for user-installed apps)

User data (~/.var/app/):
├── org.gimp.GIMP/
│   ├── .config/                    # App configuration
│   ├── .local/                     # Local app data
│   └── cache/                      # App cache (safe to delete)
└── org.mozilla.firefox/
    └── ...

The OSTree object store deduplication means the real disk usage is usually less than du suggests. For accurate measurement:

# This command accounts for shared objects:
flatpak disk-usage --system
flatpak disk-usage --user

# Output shows "Application Bytes" (real unique bytes) vs "Installed Size" (apparent)

If you have GNOME and KDE apps, you’ll have both GNOME Platform and KDE Frameworks runtimes. These don’t share much — different library stacks. This is the realistic scenario where Flatpak’s storage overhead is highest.

The Sandbox Escape: What Breaks Isolation

Flatpak’s sandbox is not perfect. Known and accepted vectors:

--filesystem=host: As described above — full home directory access means no real isolation. Check your apps.

X11 Forwarding: If an app uses X11 (with --socket=x11 or --socket=fallback-x11), it can keylog other X11 apps and capture screen content. X11 has no per-app isolation. This is a fundamental X11 limitation, not a Flatpak bug. Wayland eliminates this.

--socket=session-bus: Full access to the D-Bus session bus means the app can call any service that’s running on the bus — including other apps. This is a large permission to grant.

PulseAudio socket: Direct access to PulseAudio means the app can potentially record audio from the system or other apps, not just play it.

The --device=all permission: Grants access to all devices in /dev. Avoid.

The Flatpak team is aware of these vectors and generally treats the sandbox as “best effort for GUI desktop apps, not a security boundary for untrusted code.” The security model assumes you trust the app developer — the sandbox mostly protects against bugs and accidental data access, not malicious apps.

Building a Flatpak

Understanding the build system shows you why app sizes are what they are.

# org.example.MyApp.yaml (Flatpak manifest)
app-id: org.example.MyApp
runtime: org.gnome.Platform
runtime-version: '45'
sdk: org.gnome.Sdk
command: myapp

finish-args:
  - --share=network
  - --socket=wayland
  - --socket=fallback-x11
  - --device=dri

modules:
  - name: myapp
    buildsystem: cmake-ninja
    sources:
      - type: git
        url: https://github.com/example/myapp
        commit: abc123...

The build runs inside a Flatpak SDK sandbox (isolated from your system). The app is built against the SDK’s libraries (GNOME SDK 45 = GTK4, GLib, etc.). The final image only includes the app’s unique code — runtime libraries come from the shared runtime at install time.

This is why Flatpak apps that use the shared runtime are often smaller than AppImages: GIMP.flatpak doesn’t bundle GTK. GTK comes from the GNOME Platform runtime shared with every other GNOME app.

What Happens Under the Hood: A Full Flatpak Launch

Tracing the full execution path for flatpak run org.gimp.GIMP:

1. flatpak CLI reads ~/.local/share/flatpak/app/org.gimp.GIMP/... to find the app
2. Determines the runtime: org.gnome.Platform/x86_64/45
3. Resolves permissions from manifest + user overrides
4. Constructs bwrap arguments defining the sandbox filesystem
5. Sets up portals: connects to xdg-desktop-portal on D-Bus, registers the session
6. Sets environment variables: FLATPAK_ID, XDG_RUNTIME_DIR, XDG_DATA_HOME, HOME (pointing to ~/.var/app/org.gimp.GIMP)
7. Writes .flatpak-info to /run/user/<uid>/flatpak/app/org.gimp.GIMP/
8. Executes bwrap with the full argument list
9. Inside the sandbox: mounts are set up, new namespaces created
10. bwrap exec()s /app/bin/gimp-2.10
11. GIMP starts inside its sandbox

Portal calls during GIMP’s life:

• File → Open: GIMP calls org.freedesktop.portal.FileChooser.OpenFile() → native picker shown → file descriptor returned
• File → Export: FileChooser.SaveFile() → native save dialog
• Help menu: OpenURI → your browser opens

When GIMP exits, bwrap tears down the namespace and all bind mounts disappear.

The OCI Future: Experimental and Worth Watching

One development worth knowing about as of 2025–2026: Flatpak has experimental support for pulling apps and runtimes from OCI registries — standard Docker Hub-compatible image registries, accessible via Podman or any OCI-compatible toolchain.

# Experimental: add an OCI-based remote
flatpak remote-add --from oci registry.example.com/my-flatpak-repo

If this matures, it could meaningfully change the distribution story. OCI registries are already widely available infrastructure — every major cloud provider runs one, and self-hosted options (Harbor, Gitea’s package registry, etc.) are common. Using them as Flatpak distribution channels would reduce the need for the custom OSTree remote infrastructure, potentially making self-hosted Flatpak distribution much easier.

The OSTree deduplication story also looks different in an OCI world: OCI layers use content-addressable storage similar to OSTree, so the storage efficiency benefits could be preserved.

For now, this is experimental. The primary distribution channel for Flatpak remains OSTree-based remotes with Flathub as the de facto central repo. But it’s an interesting direction, and if you’re building internal tooling or evaluating whether to run a private Flatpak repo, it’s worth checking the current state of OCI support in your target Flatpak version.

What to Read Next

• Bubblewrap: github.com/containers/bubblewrap
• XDG Desktop Portal: flatpak.github.io/xdg-desktop-portal
• OSTree documentation: ostreedev.github.io/ostree
• Flathub submission guide: docs.flathub.org/docs/for-app-authors
• Flatpak permissions reference: docs.flatpak.org/en/latest/sandbox-permissions.html

Next: Part 5 — vs Docker: When Does a Universal Package Become a Container?

Andrew Denner — denner.co — @adenner

Linux Namespaces, cgroups, OverlayFS, and the OCI Spec Explained

Andrew Denner · March 2026 · denner.co Part 5 of the Linux Universal Packages series

CIALUG did a Docker internals deep-dive last month. This part is written assuming you came out of that session knowing what cgroups and namespaces are. If you didn’t — or if you want a refresher — the first half covers the kernel primitives. Then we do the comparison. If you want to skip to “how these relate to Flatpak/Snap” jump to the comparison section.

The Kernel Primitives: A Refresher

Linux containers — and by extension, Snap and Flatpak sandboxes — are built from a small set of kernel primitives. Not kernel modules, not hypervisors, not virtual machines. Primitives that have been part of the mainline kernel for over a decade.

Linux Namespaces

A namespace wraps a global resource and makes it appear private to processes in that namespace. There are currently eight namespace types in Linux:

These are not new. CLONE_NEWNS (mount namespaces) has been in the kernel since 2.4.19 (2002). PID namespaces since 3.8. User namespaces since 3.8 (fully complete in 3.12).

Creating a namespace is just a flag to clone() or unshare():

// Create a child process in new namespaces:
pid_t child = clone(child_function, stack,
    CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWUSER, NULL);

// Or: enter a new namespace from within a process:
unshare(CLONE_NEWNS);   // Unshare mount namespace from parent

You can experiment without any container tooling:

# Enter a new mount namespace as an unprivileged user:
unshare --mount --pid --fork bash

# Inside: a new shell with its own mount namespace
# Mounts created here are invisible outside
mount --bind /tmp/mydir /mnt/test  # Only visible in this shell
exit  # Mount gone

# Enter a new user+mount namespace and see what you can do:
unshare --user --map-root-user --mount --pid --fork bash
whoami  # Shows "root" inside the namespace
id      # But actual UID is still yours

cgroups: Resource Accounting and Limits

cgroups (control groups) attach resource accounting and limits to groups of processes. Where namespaces control visibility, cgroups control resource consumption.

cgroups v1 (legacy, still widely used) and cgroups v2 (unified hierarchy, the current standard) both expose themselves through a virtual filesystem at /sys/fs/cgroup/.

# See cgroup v2 hierarchy:
ls /sys/fs/cgroup/
# cgroup.controllers  cgroup.max.depth  cpuacct.usage  memory.current ...

# Your user session has a cgroup slice:
cat /proc/self/cgroup
# 0::/user.slice/user-1000.slice/session-2.scope

# The cgroup files for memory limits:
ls /sys/fs/cgroup/user.slice/user-1000.slice/
# memory.current     # Current memory usage
# memory.max         # Hard limit (kill if exceeded)
# memory.high        # Soft limit (throttle before OOM)
# cpu.max            # CPU bandwidth limit: "100000 100000" = 100% of one core

What cgroups let you do:

# Create a cgroup and limit its memory:
mkdir /sys/fs/cgroup/myapp
echo $PID > /sys/fs/cgroup/myapp/cgroup.procs  # Add process to cgroup
echo "100M" > /sys/fs/cgroup/myapp/memory.max  # 100MB memory limit
echo "50000 100000" > /sys/fs/cgroup/myapp/cpu.max  # 50% CPU limit

cgroups v2 reorganizes this into a unified tree where every cgroup inherits from its parent. A process can only be in one leaf cgroup (vs v1 where it could be in different cgroups in different hierarchies). This unified model is cleaner but required updates in systemd, docker, and container runtimes — the migration took years.

cgroups in the context of packaging:

• Docker uses cgroups extensively: every container gets its own cgroup with configurable resource limits
• Snap uses systemd slices to assign each snap to a cgroup (for accounting, not typically hard limits on desktop)
• Flatpak creates systemd scopes for tracked apps but doesn’t typically apply resource limits for desktop apps
• AppImage: no cgroup involvement at all

OverlayFS: Layered Filesystems

OverlayFS merges multiple directories into a single unified view. This is the mechanism that makes Docker image layers work, and it’s the same mechanism used in various forms across all container technologies.

# OverlayFS basic structure:
# lowerdir: Read-only base layer(s)
# upperdir: Read-write layer (receives all writes)
# workdir:  Working directory (must be same filesystem as upperdir)
# merged:   The unified view presented to the process

mkdir /tmp/lower /tmp/upper /tmp/work /tmp/merged
echo "from lower" > /tmp/lower/existing-file.txt

mount -t overlay overlay -o \
  lowerdir=/tmp/lower,\
  upperdir=/tmp/upper,\
  workdir=/tmp/work \
  /tmp/merged

# Inside /tmp/merged:
ls /tmp/merged        # Shows existing-file.txt from lower
cat /tmp/merged/existing-file.txt   # "from lower"

# Write a new file:
echo "new content" > /tmp/merged/new-file.txt
# Goes to upper layer: /tmp/upper/new-file.txt
# Lower layer is unchanged

# Modify an existing file:
echo "modified" > /tmp/merged/existing-file.txt
# Creates a copy in upper layer (copy-on-write)
# /tmp/upper/existing-file.txt now exists with new content
# /tmp/lower/existing-file.txt is unchanged

Copy-on-write (CoW): When you modify a file that exists in the lower layer, OverlayFS creates a copy in the upper layer. The lower layer is never modified. This is how Docker containers have writable filesystems without copying the entire base image on creation.

Deletion (whiteout files): Deleting a file in the lower layer creates a whiteout entry in the upper layer — a character device with major/minor 0/0 at the same path. The overlay driver hides the lower layer’s file when it sees a whiteout.

# After deleting /tmp/merged/existing-file.txt:
ls -la /tmp/upper/existing-file.txt
# crw------- 1 root root 0, 0 ... existing-file.txt
# That's a whiteout device node — not a regular file

How Docker Puts It Together

Docker’s container model uses all these primitives together. Understanding the stack:

Container Process
      │
      │ runs in
      ▼
Linux Namespaces (mnt, pid, net, user, uts, ipc)
      │
      │ isolated filesystem via
      ▼
OverlayFS (image layers + writable upper layer)
      │
      │ resource-limited by
      ▼
cgroups (memory, CPU, block I/O limits)
      │
      │ defined by
      ▼
OCI Runtime (runc/crun)
      │
      │ images stored as
      ▼
OCI Image Spec (layers: tarballs with content-addressable storage)

The OCI Specifications

The Open Container Initiative (OCI) defined two specs:

OCI Image Spec: Defines how container images are structured and stored. An OCI image is:

• A manifest (JSON listing layers and config)
• An image config (environment, entrypoint, exposed ports, etc.)
• Layer blobs (gzipped tarballs of filesystem deltas)

// Simplified OCI manifest:
{
  "schemaVersion": 2,
  "config": { "digest": "sha256:abc...", "mediaType": "application/vnd.oci.image.config.v1+json" },
  "layers": [
    { "digest": "sha256:def...", "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip" },
    { "digest": "sha256:ghi...", "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip" }
  ]
}

OCI Runtime Spec: Defines what a container runtime must do with an unpacked image. Specifies the config.json format that describes namespaces to create, mounts to set up, entrypoint to execute, cgroup configuration, and security constraints (AppArmor, seccomp).

runc is the reference OCI runtime — it reads config.json and produces a running container. Docker, Podman, containerd, and CRI-O all use runc (or crun, a compatible alternative) under the hood.

What Docker Adds on Top

runc gives you a running container from a config.json. Docker adds:

• An image build system (Dockerfile / docker build)
• An image registry protocol (Docker Hub and compatible registries)
• A daemon (dockerd) that manages container lifecycle
• Networking (bridge networks, port mapping, overlay networks for multi-host)
• Volume management
• Docker Compose for multi-container applications
• A container registry client (pull/push)

Most of what developers think of as “Docker” is the tooling layer, not the kernel primitives. Those primitives are also available through Podman (daemonless), containerd, or direct runc usage.

The Comparison: Universal Packages vs Containers

Now the interesting part. How do AppImage, Snap, and Flatpak compare to Docker containers when examined through the same lens?

Namespace Usage

AppImage: No namespace isolation of any kind. The app runs in your full user session. Complete visibility of all processes, all network interfaces, all mounts. The “isolation” is purely by convention — it can’t touch the squashfs (read-only), but that’s not kernel-enforced isolation.

Snap: Mount namespace isolation (the app has its own view of the filesystem). No PID, network, or user namespace. The app can see all processes (ps aux from a Snap process shows the full system), can use all network interfaces, and runs as your real UID. Snap’s security comes from AppArmor and seccomp, not namespace isolation.

Flatpak: Mount + PID + User namespaces. The app has a private PID space (can’t directly signal processes outside), filesystem isolation via bind mounts, and a remapped UID. No network namespace — Flatpak apps with network permission access your real network stack. The --share=network flag doesn’t create a new namespace, it allows access to the host network. No network isolation between Flatpak apps.

Docker: All namespaces except user (user namespaces in Docker are supported but opt-in due to compatibility concerns with bind mounts and file permissions). Full network isolation by default — each container gets its own network stack, own loopback, own IP address.

What This Means in Practice

Flatpak and Snap provide what security researchers call reduced attack surface isolation, not the kind of isolation you’d rely on for running untrusted code. If a Flatpak app is compromised, it might not be able to read your files (if filesystem access isn’t granted), but it can still:

• Make arbitrary network connections (if --share=network)
• See other running processes by PID (via /proc in Snap, slightly restricted in Flatpak)
• Communicate via D-Bus with other services

Docker provides genuine network isolation, PID isolation, and resource limits. A compromised Docker container can’t directly reach your host network or see host processes (absent specific configurations or kernel vulnerabilities).

The intended use cases differ: Flatpak/Snap are for applications that need to run as your user with your data. Containers are for running services that shouldn’t have any access to the host.

Storage Layer Comparison

Flatpak and Docker both use content-addressable storage with deduplication. AppImage and Snap don’t deduplicate between app instances. If you have 10 snaps that all bundle GTK4, you have 10 copies of GTK4. If you have 10 Flatpaks that all use GNOME Platform 45, you have one shared copy of GNOME Platform 45 (via OSTree hardlinks).

Docker goes further — if two images share the same base layers (e.g., both use the same Ubuntu 22.04 base), those layers are stored once. A 100-container deployment might have only a few GB of actual storage if the images share base layers.

Security Model Comparison

“Stronger” doesn’t mean “better for all use cases.” AppImage’s lack of isolation is a feature when you want the app to have the same access as a native binary. Snap’s AppArmor model is actually more expressive than Docker’s isolation in some ways — you can define per-interface permissions at the D-Bus method level.

The cgroups Difference

This is where containers diverge most clearly from universal packages.

Docker (and Kubernetes, Podman, etc.) use cgroups to provide hard resource limits: a container can be configured to use no more than 2GB of RAM and 1 CPU core, regardless of what the code inside tries to do. The kernel enforces this.

Universal packages don’t typically set hard resource limits. A misbehaving Flatpak app can consume all available RAM. A runaway Snap process can spin a CPU core to 100%. The cgroup integration that Snap and Flatpak have (via systemd slices) is for accounting and throttling, not hard limits.

If you need hard resource limits on a desktop app — and sometimes you do, for background sync services, document processors, anything that could go rogue — containers are the right tool.

When These Technologies Actually Overlap

The comparison so far treats them as different categories, but there are real overlaps:

Flatpak on the Server?

You can run Flatpak apps headless — flatpak run doesn’t require a display. For build environments, reproducible testing, or distributing server-side GUI tools to heterogeneous Linux systems, Flatpak works. It doesn’t have network isolation, so it’s not a drop-in for containers, but for “reproducible builds that run on any distro” it’s a reasonable choice.

Snaps as Service Containers

On Ubuntu Core, snap daemons are effectively lightweight service containers. They have filesystem isolation, AppArmor confinement, automatic updates, and rollback. For IoT devices where you want managed services without the overhead of Docker, Snap fills a real niche.

Containers for Desktop?

Distrobox and toolbox (Fedora’s tool) run Flatpak-style home directory sharing with Docker/Podman containers to give you a mutable, different-distro environment on your desktop. Your GUI apps run in a container that has full access to ~ and the display server — deliberately collapsing the container’s isolation to get cross-distro compatibility. This is the same tradeoff as Flatpak’s --filesystem=home.

The OCI/Flatpak Convergence Experiments

There have been experiments combining the OCI image format with Flatpak’s portal/permission system. The flatpak-oci bundle format is one — it uses OCI image layers for distribution but Flatpak’s portal infrastructure for runtime. This hasn’t become mainstream but points to the technologies converging at the edges.

The Mental Model: A Decision Tree

When should you use which technology?

Is this a desktop GUI app?
  Yes → Flatpak (sandboxed) or apt/dnf (native)
  No → continue

Does it need to run on distros without knowing the package manager?
  Yes → AppImage (portable binary) or Docker (if services)
  No → continue

Is it a long-running service?
  Yes → Docker/Podman/Snap (depending on platform and isolation needs)
  No → continue

Does it need hard resource limits?
  Yes → Docker/Podman container
  No → Universal package or native

Does it run on Ubuntu and need auto-patching on a server?
  Yes → Snap
  No → Docker

Is it on a headless embedded system (RPi, IoT)?
  Yes → Snap (Ubuntu Core) or Docker
  No → Use your distro's native package manager first

The answer is almost always “native package manager first.” These tools exist for the cases where native doesn’t work.

The Shared Ancestry: It’s All clone() and mount()

The insight that ties this together: AppImage, Snap, Flatpak, Docker, LXC, systemd-nspawn — they’re all wrappers around the same small set of Linux syscalls.

Every isolation mechanism here ultimately comes down to:

• clone() with namespace flags
• mount() with bind/overlay options
• write() to /proc/self/uid_map for user namespace remapping
• prctl(PR_SET_SECCOMP, ...) for syscall filtering
• aa_change_profile() for AppArmor profile transitions
• Writes to /sys/fs/cgroup/ for resource limits

The differences are which of these syscalls get called, in what combination, with what policies. Docker calls all of them. Flatpak calls most of them. Snap calls some of them plus AppArmor extensively. AppImage calls none of them.

Understanding this shared foundation explains why:

• Container vulnerabilities (like user namespace escapes) affect multiple technologies simultaneously
• Security improvements in the kernel (improved cgroup v2 support, better seccomp audit) benefit all these tools
• The tools can interoperate (Distrobox, Flatpak OCI experiments)

The packaging format and the security model are separable concerns. squashfs is just a filesystem. The security happens in the wrapping layer.

Practical Summary

The most important row: isolation. Choose based on how much isolation you actually need for your threat model, not based on what’s fashionable.

Resources

• Linux namespaces deep dive: man 7 namespaces
• cgroups v2 guide: kernel.org/doc/Documentation/admin-guide/cgroup-v2.rst
• OCI specs: opencontainers.org
• runc source: github.com/opencontainers/runc
• Bubblewrap (Flatpak sandbox): github.com/containers/bubblewrap
• OverlayFS: man 8 mount — see overlay filesystem section
• Distrobox: distrobox.it
• unshare(1): start experimenting with namespaces without writing C

This concludes the Linux Universal Packages series. Full series index at denner.co/blog. Talk materials at denner.co/talks.

Andrew Denner — denner.co — @adenner

The Complete Developer Guide — From Zero to Shipped

Andrew Denner · March 2026 · denner.co Part 6 of the Linux Universal Packages series

This is the post I wish existed when I first tried to package something. It covers the full workflow for all three formats — not just the happy path, but the real path: the gotchas, the debugging loops, the “why does this work on my machine but not on the build server” moments. The talk slides have the abbreviated versions; this is the extended cut.

Before You Start: Pick Your Format

Not everything needs to be packaged in all three formats. A quick decision guide:

Build an AppImage if:

• Your app has no deep system integration (no D-Bus services, no systemd units)
• Your users are on diverse distros or locked-down systems without package managers
• You want maximum compatibility with minimum infrastructure
• You just want a single file to hand people

Build a Snap if:

• Your primary audience is Ubuntu users
• Your app is a long-running service or daemon
• You want Canonical’s review process and store infrastructure
• You’re targeting Ubuntu Core / IoT

Build a Flatpak if:

• It’s a desktop GUI application
• You want to reach Fedora, Mint, Arch, and Ubuntu users with one package
• You care about sandbox transparency (Flatseal support)
• You want to be on Flathub — the de facto app store for non-Ubuntu Linux desktops

If your app is CLI-only: use your distro’s package manager first. None of these formats are designed for CLI tools and the UX will fight you.

Building AppImages

Prerequisites

# Install build tools
wget https://github.com/linuxdeploy/linuxdeploy/releases/download/continuous/linuxdeploy-x86_64.AppImage
chmod +x linuxdeploy-x86_64.AppImage

# Optional: plugin for Qt apps
wget https://github.com/linuxdeploy/linuxdeploy-plugin-qt/releases/download/continuous/linuxdeploy-plugin-qt-x86_64.AppImage
chmod +x linuxdeploy-plugin-qt-x86_64.AppImage

# For Python apps
wget https://github.com/niess/linuxdeploy-plugin-python/releases/download/continuous/linuxdeploy-plugin-python-x86_64.AppImage
chmod +x linuxdeploy-plugin-python-x86_64.AppImage

# appimagetool (if you want manual control)
wget https://github.com/AppImage/AppImageKit/releases/download/continuous/appimagetool-x86_64.AppImage
chmod +x appimagetool-x86_64.AppImage

The Manual Method (Understanding the Structure)

Start here before using linuxdeploy — it teaches you what linuxdeploy is doing:

# 1. Create the AppDir
mkdir -p MyApp.AppDir/usr/{bin,lib,share/myapp}

# 2. Copy your binary
cp build/myapp MyApp.AppDir/usr/bin/

# 3. Create the .desktop file (required)
cat > MyApp.AppDir/myapp.desktop << 'EOF'
[Desktop Entry]
Name=MyApp
Exec=myapp
Icon=myapp
Type=Application
Categories=Utility;
Comment=My awesome application
EOF

# 4. Add an icon (required — 256x256 PNG minimum)
cp assets/myapp.png MyApp.AppDir/myapp.png
cp assets/myapp.png MyApp.AppDir/usr/share/myapp/

# 5. Create AppRun (the entry point)
cat > MyApp.AppDir/AppRun << 'APPRUN'
#!/bin/bash
set -e
SELF=$(readlink -f "$0")
HERE=${SELF%/*}
export PATH="${HERE}/usr/bin/:${PATH}"
export LD_LIBRARY_PATH="${HERE}/usr/lib/:${LD_LIBRARY_PATH}"
export XDG_DATA_DIRS="${HERE}/usr/share/:${XDG_DATA_DIRS}"
exec "${HERE}/usr/bin/myapp" "$@"
APPRUN
chmod +x MyApp.AppDir/AppRun

# 6. Bundle shared libraries (manual approach)
# Find what your binary needs:
ldd MyApp.AppDir/usr/bin/myapp

# Copy each .so that isn't a basic system library:
# Skip: libm, libc, libpthread, libdl, ld-linux (these are always present)
# Copy: everything else
cp /usr/lib/x86_64-linux-gnu/libssl.so.1.1 MyApp.AppDir/usr/lib/
cp /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 MyApp.AppDir/usr/lib/
# ... etc

# 7. Build the AppImage
./appimagetool-x86_64.AppImage MyApp.AppDir MyApp-1.0-x86_64.AppImage

The linuxdeploy Method (Production Workflow)

# For a compiled app (C/C++, Rust, Go binary):
./linuxdeploy-x86_64.AppImage \
  --appdir MyApp.AppDir \
  --executable build/myapp \
  --desktop-file myapp.desktop \
  --icon-file assets/myapp.png \
  --output appimage

# For a Qt app:
export QMAKE=/usr/lib/qt5/bin/qmake  # Point to your Qt installation
./linuxdeploy-x86_64.AppImage \
  --appdir MyApp.AppDir \
  --executable build/myapp \
  --desktop-file myapp.desktop \
  --icon-file assets/myapp.png \
  --plugin qt \
  --output appimage

# For a Python app:
./linuxdeploy-x86_64.AppImage \
  --appdir MyApp.AppDir \
  --executable /usr/bin/python3 \
  --desktop-file myapp.desktop \
  --icon-file assets/myapp.png \
  --plugin python \
  --output appimage

linuxdeploy automates:

• Running ldd to find all dependencies
• Copying .so files into usr/lib/
• Patching ELF rpath so the bundled libs are found at runtime
• Handling Qt plugins (.so files for image formats, SQL drivers, etc.)
• Generating the squashfs and attaching the runtime

The Most Important Rule: Build on Old Linux

The glibc symbol version problem is real and will bite you:

# Check which glibc symbols your binary needs:
objdump -T build/myapp | grep GLIBC | sort -t@ -k2 | tail

# Output example:
# 0000...  GLIBC_2.33  memmove
# 0000...  GLIBC_2.34  pthread_create

If you build on Ubuntu 24.04 (glibc 2.38) and someone tries to run your AppImage on Ubuntu 20.04 (glibc 2.31), they get:

./MyApp.AppImage: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found

The fix: Build on the oldest distro you want to support, or use Docker to target a specific old distro:

# Build inside Ubuntu 20.04 container:
docker run --rm -v $(pwd):/build ubuntu:20.04 bash -c "
  apt-get update -qq
  apt-get install -y build-essential cmake ...
  cd /build
  cmake -B builddir && cmake --build builddir
"
# Now run linuxdeploy on the output

The AppImageKit wiki recommends building on CentOS 7 (glibc 2.17) for maximum compatibility. In practice, Ubuntu 20.04 is the most pragmatic baseline for 2025+.

Adding Update Support

To enable appimageupdate integration, add the update URL to the AppImage at build time:

# For GitHub releases:
UPDATE_INFO="gh-releases-zsync|YourOrg|YourRepo|latest|MyApp-*-x86_64.AppImage.zsync"

./linuxdeploy-x86_64.AppImage \
  --appdir MyApp.AppDir \
  --executable build/myapp \
  --desktop-file myapp.desktop \
  --icon-file assets/myapp.png \
  --updateinformation "$UPDATE_INFO" \
  --output appimage

This embeds the update URL in the AppImage. Users with AppImageLauncher get automatic update notifications. Users with appimageupdate CLI can run delta updates.

Adding GPG Signing

# Sign the AppImage with your GPG key:
./appimagetool-x86_64.AppImage \
  --sign \
  --sign-key YOUR_GPG_FINGERPRINT \
  MyApp.AppDir \
  MyApp-1.0-x86_64.AppImage

# This embeds the signature in an ELF section and creates MyApp-1.0-x86_64.AppImage.digest

Debugging AppImages

# What's inside?
./MyApp.AppImage --appimage-extract
ls squashfs-root/

# Mount it for inspection (no extraction):
./MyApp.AppImage --appimage-mount
# Prints the mountpoint path, Ctrl+C to unmount

# Run the extracted binary directly (bypass FUSE):
squashfs-root/AppRun

# Extract and debug with strace:
./MyApp.AppImage --appimage-extract
strace -e openat squashfs-root/AppRun 2>&1 | grep "No such file"
# Shows every file it's trying to open that doesn't exist

# Check offset (useful for debugging self-extraction):
./MyApp.AppImage --appimage-offset

# Run without FUSE (useful when FUSE isn't available):
APPIMAGE_EXTRACT_AND_RUN=1 ./MyApp.AppImage

GitHub Actions CI for AppImages

# .github/workflows/appimage.yml
name: Build AppImage

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-20.04  # Use older Ubuntu for glibc compatibility!
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: sudo apt-get install -y cmake build-essential libssl-dev

      - name: Build
        run: cmake -B build && cmake --build build

      - name: Download linuxdeploy
        run: |
          wget -q https://github.com/linuxdeploy/linuxdeploy/releases/download/continuous/linuxdeploy-x86_64.AppImage
          chmod +x linuxdeploy-x86_64.AppImage

      - name: Build AppImage
        run: |
          ./linuxdeploy-x86_64.AppImage \
            --appdir MyApp.AppDir \
            --executable build/myapp \
            --desktop-file myapp.desktop \
            --icon-file assets/myapp.png \
            --output appimage

      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: appimage
          path: "*.AppImage"

Building Snaps

Prerequisites

# Install snapcraft
sudo snap install snapcraft --classic

# Initialize LXD (snapcraft uses it for clean builds)
sudo snap install lxd
sudo lxd init --minimal
sudo usermod -aG lxd $USER
newgrp lxd

The snapcraft.yaml

Snapcraft builds from a single snapcraft.yaml in your project root (or in snap/snapcraft.yaml). Here’s a complete example:

name: myapp
base: core22          # Ubuntu 22.04 LTS runtime
version: '1.0.0'
summary: My App — one line description
description: |
  Extended description of what your app does.
  Can be multiple paragraphs.

grade: stable         # or 'devel' for non-release
confinement: strict   # 'devmode' for development, 'classic' for full system access

apps:
  myapp:
    command: bin/myapp
    plugs:
      - home           # Access ~/
      - network        # Network access
      - wayland        # Wayland display
      - x11            # X11 fallback
      - desktop        # Desktop environment integration
      - audio-playback # PulseAudio/PipeWire (for apps with sound)

parts:
  myapp:
    plugin: cmake
    source: .
    cmake-parameters:
      - -DCMAKE_INSTALL_PREFIX=/
      - -DCMAKE_BUILD_TYPE=Release
    build-packages:
      - libssl-dev
      - pkg-config
    stage-packages:
      - libssl3         # Runtime libraries (from Ubuntu 22.04 repos)

Build Commands

# Build the snap (downloads base, builds in LXD VM):
snapcraft

# Output: myapp_1.0.0_amd64.snap

# Install locally for testing:
sudo snap install myapp_1.0.0_amd64.snap --dangerous

# Test it:
snap run myapp

# Check what's blocked (AppArmor denials):
snap logs myapp

# Rebuild just a specific part (faster during development):
snapcraft clean myapp
snapcraft build myapp

# Debug shell inside the build environment:
snapcraft prime --debug  # drops you to a shell before packaging

Confinement Debugging

The most common snap development loop:

# 1. Start with devmode (no confinement):
# In snapcraft.yaml: confinement: devmode

# 2. Install and test:
sudo snap install myapp_1.0.0_amd64.snap --dangerous --devmode

# 3. Watch AppArmor denials:
sudo journalctl -f | grep DENIED

# 4. Map denials to interfaces
# If you see: /run/user/1000/wayland-0 denied → need 'wayland' plug
# If you see: /home/user/.config/ denied → need 'home' plug
# If you see: network denied → need 'network' plug

# 5. Add plugs to snapcraft.yaml, rebuild:
snapcraft

# 6. Install in strict mode:
sudo snap install myapp_1.0.0_amd64.snap --dangerous
snap run myapp

# 7. Repeat until it works, then set confinement: strict

The Classic Confinement Option

For developer tools that genuinely need full filesystem access (editors, IDEs, compilers):

confinement: classic

Classic snaps bypass AppArmor confinement entirely — they can access the full filesystem. Publishing a classic snap to the Snap Store requires manual review and justification. Not a rubber stamp; Canonical actually reviews the request.

Examples of legitimate classic snaps: VS Code, JetBrains IDEs, the various language SDKs (Go, .NET, Node).

Multi-App Snaps (Daemons + CLI)

Snaps can bundle multiple apps — useful for a service with a management CLI:

apps:
  myapp-daemon:
    command: bin/myappd
    daemon: simple        # or 'forking', 'oneshot', 'notify'
    restart-condition: on-failure
    plugs: [network]

  myapp-cli:
    command: bin/myapp-cli
    plugs: [home, network]

The daemon app runs as a systemd service. snap start myapp.myapp-daemon and snap stop myapp.myapp-daemon control it.

Publishing to the Snap Store

# Register your app name (one-time):
snapcraft register myapp

# Login to the store:
snapcraft login

# Upload:
snapcraft upload myapp_1.0.0_amd64.snap --release stable

# Or release a specific revision to a channel:
snapcraft release myapp 1 stable      # revision 1 → stable
snapcraft release myapp 1 candidate   # same revision to candidate

# Check status:
snapcraft status myapp

Snap Store review process: Automated checks run on upload. Strict snaps with common interfaces auto-approve. Snaps using sensitive interfaces (browser-support, classic confinement, etc.) queue for manual review. Typically 1-5 business days.

snapcraft.yaml Patterns

Python application:

parts:
  myapp:
    plugin: python
    source: .
    python-requirements:
      - requirements.txt

Electron application:

parts:
  myapp:
    plugin: npm
    source: .
    npm-include-node: true
    npm-node-version: '20.0.0'

Go application:

parts:
  myapp:
    plugin: go
    source: .
    build-snaps: [go]

Building Flatpaks

Prerequisites

# Install flatpak and flatpak-builder
sudo apt install flatpak flatpak-builder    # Debian/Ubuntu
sudo dnf install flatpak flatpak-builder    # Fedora/RHEL

# Add Flathub (if not already added)
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

# Install the GNOME SDK (or whichever runtime you'll use):
flatpak install org.gnome.Sdk//47
flatpak install org.gnome.Platform//47

# For KDE apps:
flatpak install org.kde.Sdk//6.8
flatpak install org.kde.Platform//6.8

The Manifest

Flatpak manifests can be JSON or YAML. YAML is easier to read and maintain:

# com.example.MyApp.yaml
app-id: com.example.MyApp

# The runtime provides the base libraries (GTK, Qt, etc.)
runtime: org.gnome.Platform
runtime-version: '47'

# The SDK is the build-time equivalent of the runtime
sdk: org.gnome.Sdk

# Entry point after installation
command: myapp

# Sandbox permissions
finish-args:
  - --share=network          # Allow network access
  - --socket=wayland         # Wayland display protocol
  - --socket=fallback-x11    # X11 fallback for non-Wayland compositors
  - --device=dri             # GPU access (hardware rendering)
  - --share=ipc              # Shared memory (required for X11)
  - --filesystem=home        # Access home directory (use sparingly)

# Modules: your app and its non-runtime dependencies
modules:
  # Optional: build a library dep that isn't in the runtime
  - name: libfoo
    buildsystem: cmake-ninja
    sources:
      - type: git
        url: https://github.com/example/libfoo.git
        tag: v2.1.0
        commit: abc123def456...   # Always pin a commit, not just a tag

  # Your main application
  - name: myapp
    buildsystem: cmake-ninja
    sources:
      - type: git
        url: https://github.com/yourorg/myapp.git
        tag: v1.0.0
        commit: def456abc123...

Build and Test Cycle

# Build and install locally (--user = user installation, no root needed):
flatpak-builder --install --user --force-clean builddir com.example.MyApp.yaml

# Run it:
flatpak run com.example.MyApp

# Debug: drop into a shell inside the sandbox
flatpak run --command=bash com.example.MyApp

# Debug: inspect the sandbox from inside
cat /run/user/$(id -u)/flatpak/app/com.example.MyApp/.flatpak-info

# Rebuild just to check the manifest without installing:
flatpak-builder --force-clean builddir com.example.MyApp.yaml

# Check logs when something goes wrong:
journalctl -f --user  # In another terminal while running the app

Handling Dependencies Not in the Runtime

The tricky part: any library your app needs that isn’t in the GNOME Platform (or KDE Frameworks) must be included as a module in your manifest.

No network during build: flatpak-builder enforces this. All sources must be declared with checksums.

modules:
  - name: libsodium
    buildsystem: autotools
    sources:
      - type: archive
        url: https://download.libsodium.org/libsodium/releases/libsodium-1.0.18.tar.gz
        sha256: 6f504490b342a4f8a4c4a02fc9b866cbef8622d5df4e5452b46be121e46636c1

  - name: libzmq
    buildsystem: cmake-ninja
    sources:
      - type: archive
        url: https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.tar.gz
        sha256: c593001a89f5a85dd2ddf564805deb860e02471171b3f204944857336295c3e5

The flatpak-builder-tools helpers save enormous time for language ecosystems:

# For Python apps (pip dependencies):
git clone https://github.com/flatpak/flatpak-builder-tools
cd flatpak-builder-tools/pip

python3 flatpak-pip-generator requests pillow numpy \
  --output python-deps.json

# Add to manifest:
# modules:
#   - python-deps.json
#   - name: myapp
#     ...

Similar tools exist for npm, cargo (Rust), and Maven/Gradle (Java).

Choosing the Right Permissions

The hardest part of Flatpak packaging is getting permissions right. Start restrictive and add only what’s needed:

finish-args:
  # Start with these for a GUI app:
  - --socket=wayland
  - --socket=fallback-x11
  - --share=ipc
  - --device=dri

  # Add as needed — test before adding:
  - --share=network        # Does the app need the internet?
  - --filesystem=home      # Does it need to open arbitrary files?
  - --filesystem=xdg-documents  # Better: scope to specific dirs
  - --socket=pulseaudio    # Does it play audio?
  - --device=all           # Only for hardware-specific apps

  # D-Bus access (be specific):
  - --talk-name=org.freedesktop.Notifications  # System notifications
  - --talk-name=org.gnome.keyring.SystemPrompter  # Keyring access
  - --own-name=com.example.MyApp   # If the app exports a D-Bus service

Common mistakes:

• --filesystem=host — grants access to the entire filesystem. Acceptable for file managers, unacceptable for most apps. Flathub reviewers will ask you to justify this.
• --socket=session-bus — grants access to the entire D-Bus session bus. Only use for apps that legitimately need broad D-Bus access.
• --share=ipc — required for X11 apps but not for Wayland-only apps. Include it in the fallback case.

The App ID and Reverse Domain Convention

Flatpak uses reverse-domain app IDs. The rules:

com.YourDomain.AppName        → If you own yourdomain.com
org.YourOrg.AppName           → For non-profits/open-source orgs
io.github.youruser.AppName    → For GitHub-hosted projects without custom domain

Flathub verifies domain ownership for published apps. For io.github.* IDs, Flathub verifies your GitHub username via OAuth during submission.

Why it matters: the app ID becomes the Flatpak directory structure, the D-Bus service name (if any), and the identity in Flatseal’s permission UI. Changing it later requires users to reinstall.

Publishing to Flathub

# 1. Test your manifest builds cleanly with no network errors:
flatpak-builder --force-clean --sandbox builddir com.example.MyApp.yaml

# 2. Fork github.com/flathub/flathub

# 3. Create com.example.MyApp/com.example.MyApp.yaml in your fork
#    (or com.example.MyApp/com.example.MyApp.json for JSON)

# 4. The Flathub CI will run automatically on your PR:
#    - Linting your manifest
#    - Building the app
#    - Running basic smoke tests
#    - Checking for common issues (network access during build, etc.)

# 5. Human reviewers check:
#    - App ID is appropriate
#    - Permissions are reasonable
#    - Sources are verifiable (no bare HTTP, commit hashes required)
#    - No proprietary bundled binaries for open-source apps

# 6. After approval, you get a dedicated repo under github.com/flathub/
#    Future updates: push to your app's repo, Flathub CI builds and publishes

CI/CD for Flatpaks

Flathub handles CI for Flathub-published apps. For testing in your own repo:

# .github/workflows/flatpak.yml
name: Flatpak CI

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: bilelmoussaoui/flatpak-github-actions:gnome-47
      options: --privileged

    steps:
      - uses: actions/checkout@v4

      - name: Build Flatpak
        uses: flatpak/flatpak-github-actions/flatpak-builder@v6
        with:
          bundle: MyApp.flatpak
          manifest-path: com.example.MyApp.yaml
          cache-key: flatpak-builder-$

Updating Your App After Flathub Publication

After initial approval, you maintain a GitHub repo under github.com/flathub/com.example.MyApp:

# To update: edit the manifest in your Flathub repo
# Change the commit hash (and url/tag if applicable):
modules:
  - name: myapp
    sources:
      - type: git
        url: https://github.com/yourorg/myapp.git
        tag: v1.1.0
        commit: newcommit123...  # Update this

# Open a PR → CI runs → merges → Flathub builds and publishes
# No per-update manual review once you're established

Side-by-Side Comparison: The Developer Experience

The Hardest Part of Each Format

AppImage: The glibc floor problem and bundling the right — but not too many — shared libraries. Under-bundling causes crashes on other distros. Over-bundling (bundling things like libGL that must match the host GPU driver) also causes crashes. Finding the right set is a process of test, fail, adjust.

Snap: Getting confinement right without either (a) opening everything up with classic, or (b) spending days mapping AppArmor denials to interface names. The interface reference documentation is good; reading it is mandatory. The other hard part: LXD setup is fiddly, and if your dev machine is itself a VM you’re running nested virtualization.

Flatpak: The no-network-during-build constraint is the real pain. When your app has 47 Python dependencies, 15 npm packages, and 3 compiled C libraries, generating and maintaining the source manifest for all of them is significant overhead. The flatpak-builder-tools scripts help but require maintenance. The payoff is reproducible, auditable builds — but you earn it.

Resources

AppImage:

• linuxdeploy: github.com/linuxdeploy/linuxdeploy
• AppImageKit: github.com/AppImage/AppImageKit
• AppImageSpec: github.com/AppImage/AppImageSpec
• AppImage packaging guide: docs.appimage.org/packaging-guide

Snap:

• snapcraft docs: snapcraft.io/docs
• Interface reference: snapcraft.io/docs/supported-interfaces
• snapcraft.yaml reference: snapcraft.io/docs/snapcraft-yaml-reference

Flatpak:

• flatpak-builder docs: docs.flatpak.org
• Flathub submission guide: docs.flathub.org/docs/for-app-authors
• flatpak-builder-tools: github.com/flatpak/flatpak-builder-tools
• Manifest reference: docs.flatpak.org/en/latest/flatpak-builder-command-reference.html

This concludes the Linux Universal Packages series. Full series index at denner.co/blog. Talk materials at denner.co/talks.

Andrew Denner — denner.co — @adenner

The Full Picture — snapd, AppArmor, Seccomp, and the Machine It Built

Andrew Denner · March 2026 · denner.co Part 3 of the Linux Universal Packages series

I rage-deleted snapd twice. I’m going to explain exactly what I was rage-deleting. The more you understand about Snap’s architecture, the more you either appreciate the engineering or become more annoyed at the design decisions — sometimes both at once. Let’s go.

snapd: The Daemon That Runs Forever

Snap is not just a package format. It’s a daemon (snapd), a store protocol, a confinement framework, a distribution channel system, and a set of Linux kernel primitives wired together in Go. Understanding Snap means understanding snapd.

# snapd is always running on Ubuntu:
systemctl status snapd
# Active: active (running)

# It exposes a REST API via Unix socket:
ls -la /run/snapd.socket
# srw-rw---- 1 root snapd ... /run/snapd.socket

# The snap CLI just calls this REST API:
sudo snap install vlc
# Under the hood: POST /v2/snaps/vlc {"action":"install"}

The entire Snap CLI is a thin wrapper over the snapd REST API. Every snap command — install, remove, refresh, connect, disconnect — is an HTTP call to /run/snapd.socket. This design means:

• snapd must be running for any snap operation to work
• Third-party tools can use the API directly
• The daemon maintains all state and performs all privileged operations (the CLI itself isn’t setuid)

snapd’s State Machine

snapd maintains its state in /var/lib/snapd/state.json. This JSON file is snapd’s database — it records every installed snap, every connection between snaps, every pending operation, and the complete revision history.

sudo python3 -c "import json,sys; d=json.load(open('/var/lib/snapd/state.json')); print(list(d.keys()))"
# ['data', 'changes', 'tasks', 'last-task-id', 'last-change-id', 'last-notice-id', 'notices']

Every operation in snapd is a change composed of tasks. When you run snap install vlc, snapd creates a change with tasks like download-snap, validate-snap, mount-snap, copy-snap-data, link-snap. If the operation fails halfway, snapd can resume or roll back from the last known-good state. This is the reliability story — snapd never leaves your system in a partial state.

Snap Types: Not All Snaps Are Equal

The word “snap” refers to several distinct things in Canonical’s ecosystem:

App snaps: What most people mean. User-installed applications. snap install vlc

Base snaps: Minimal root filesystems that other snaps run on top of. core22 is the Ubuntu 22.04 base. core20, core18 for older bases. When you install VLC, snapd also installs core22 if not present — VLC’s sandbox uses core22 as its filesystem root.

Kernel snaps: On Ubuntu Core (IoT/embedded), the Linux kernel itself is a snap. This allows over-the-air kernel updates with the same rollback mechanism as app snaps.

Gadget snaps: Also Ubuntu Core. Defines the device’s boot configuration, disk layout, and hardware-specific initialization.

Content snaps: Share data (fonts, icons, themes, SDKs) between snaps via the content interface. The GNOME Platform snap is a content snap that provides GTK runtime files, reducing duplication between apps.

For desktop Linux, you’re dealing with app snaps and base snaps. But understanding base snaps explains why snap install sometimes downloads 70MB before it installs your 5MB app — it’s downloading the base runtime if you don’t have it.

The Disk Layout: What Gets Mounted Where

/snap/
├── vlc/                     # App namespace
│   ├── 3221/                # Revision 3221 (squashfs mounted here)
│   ├── 3198/                # Revision 3198 (previous, kept for rollback)
│   └── current -> 3221/     # Symlink to active revision
├── core22/                  # Base snap
│   ├── 607/
│   └── current -> 607/
├── snapd/                   # snapd itself (as a snap)
│   └── ...
└── bin/                     # Symlinks: snap, snapctl, etc.

/var/lib/snapd/
├── snaps/                   # The actual .snap squashfs files
│   ├── vlc_3221.snap        # squashfs file (read-only source)
│   └── vlc_3198.snap        # Previous revision (kept until manually removed)
├── state.json               # snapd's database
├── assertions/              # Cryptographic assertions (more on this below)
└── mount/                   # Mount namespace configuration files

/var/snap/
└── vlc/
    ├── 3221/                # Revision-specific data (wiped on update)
    ├── current -> 3221/     # Symlink
    └── common/              # Persistent data (survives updates)

The squashfs file in /var/lib/snapd/snaps/vlc_3221.snap is mounted read-only at /snap/vlc/3221/ via the kernel loop device. There’s no FUSE involved — snapd uses the kernel’s native loop device and squashfs support:

mount | grep snap
# /dev/loop4 on /snap/vlc/3221 type squashfs (ro,nodev,relatime,errors=continue,threads=single)

This is why Snap’s performance is generally better than AppImage’s FUSE approach — the kernel reads squashfs directly without the userspace round-trip. The latency penalty comes from the AppArmor and seccomp machinery that wraps execution, not the filesystem access itself.

snap-confine: The Jail That Runs Your App

When you launch a Snap application, you don’t directly exec the binary at /snap/vlc/3221/usr/bin/vlc. You go through snap-confine, a small C binary with setuid root permissions that sets up the execution environment:

user launches vlc
  → /snap/bin/vlc (shell wrapper)
  → /usr/lib/snapd/snap-exec
  → /usr/lib/snapd/snap-confine
  → [namespace setup, apparmor transition, seccomp load]
  → /snap/vlc/3221/usr/bin/vlc (confined process)

snap-confine does the following in sequence:

1. Mount namespace creation: Creates a new mount namespace (CLONE_NEWNS). Inside this namespace, the snap sees:
   • /snap/vlc/3221/ mounted at / (the app’s root filesystem via bind mounts)
   • Base snap content merged in (provides glibc, GTK, etc.)
   • Real /home, /tmp, specific /run/user paths (as permitted by interfaces)
   • Tmpfs mounts for /tmp, /dev/shm (isolated per snap)

2. AppArmor profile transition: Calls aa_change_profile() to transition the process into the snap’s AppArmor profile. After this point, all actions are subject to AppArmor MAC.

3. seccomp filter load: Loads a seccomp BPF filter that restricts system calls. The filter is generated from the snap’s declarations.

4. UID/GID: The snap process runs as the invoking user (no privilege escalation). The setuid root on snap-confine itself is only used to set up the namespace — the app process drops root before exec.

AppArmor: The MAC Enforcement Layer

AppArmor (Mandatory Access Control) enforces policy regardless of what the process tries to do with POSIX capabilities. A process confined by AppArmor cannot access resources not listed in its profile, even if the user running it has permission.

Snap generates AppArmor profiles automatically from the snap’s meta/snap.yaml. The profile is generated by snapd and loaded into the kernel when the snap is installed.

# See the generated AppArmor profile for a snap:
cat /var/lib/snapd/apparmor/profiles/snap.vlc.vlc

# The profile controls:
# - File access (read/write/execute paths)
# - Network access
# - D-Bus access (what services it can call, what methods)
# - Signal sending
# - Ptrace
# - Mount/unmount operations
# - Capability acquisition

A snippet of what a typical snap AppArmor profile looks like:

# Allow reading snap content
/snap/vlc/** r,
/snap/vlc/**/**.so* mr,

# Home directory (controlled by home interface)
@{HOME}/ r,
@{HOME}/** rwk,  # Only if home interface connected

# Deny dangerous capabilities
deny /proc/sys/kernel/sysrq w,
deny /sys/kernel/security/** rwklx,
deny @{PROC}/@{pid}/mem rwklx,

The granularity is real. If VLC has the home interface connected, it can read/write your home directory. If you disconnect it via snap disconnect vlc:home, the AppArmor profile is regenerated and VLC can no longer access ~ even if it tries.

Interface-Driven Profiles

The key to Snap’s security model is that interfaces map to AppArmor policy fragments. Each interface has a policy file in /usr/share/snappy/interfaces/. When you connect an interface, snapd regenerates the AppArmor profile to include the corresponding policy fragment and reloads it.

# See all interfaces and their connection state for a snap:
snap connections vlc

# Interface           Plug                Slot              Notes
# audio-playback      vlc:audio-playback  :audio-playback   -
# home                vlc:home            :home             -
# network             vlc:network         :network          -
# removable-media     vlc:removable-media -                 -   # NOT connected

# Connect removable-media to allow /media/* access:
snap connect vlc:removable-media

# Disconnect home to prevent home directory access:
snap disconnect vlc:home

The interface system is the most technically sophisticated part of Snap. There are ~100 interfaces covering everything from audio-playback to bluetooth-control to docker to ssh-keys.

seccomp: Syscall-Level Filtering

AppArmor controls access to resources. seccomp (secure computing mode) controls which system calls the process can make. These are two separate but complementary confinement mechanisms.

Snap uses seccomp in filter mode (BPF program that inspects syscall numbers and arguments). The seccomp profile is generated by snapd-seccomp from the snap’s declarations.

A typical snap seccomp profile allows common syscalls while blocking dangerous ones:

# Allowed (representative subset):
read write open close stat fstat lstat poll
select mmap mprotect munmap brk
socket connect accept bind listen
execve exit_group

# Blocked (not listed = SIGSYS):
# kexec_load       (load new kernel)
# create_module    (load kernel module)
# init_module      (insert kernel module)
# perf_event_open  (performance monitoring, can be used for side channels)
# ptrace           (unless debugging interface connected)
# bpf              (eBPF programs)

seccomp filtering happens at the kernel level. If a confined process tries to call a blocked syscall, it receives SIGSYS (bad system call) immediately — the syscall is never executed.

This is meaningfully different from AppArmor. AppArmor can restrict what files you can open, but it can’t stop a process from making a particular syscall. seccomp stops syscalls before they reach the kernel. Together, they provide defense-in-depth: AppArmor denies access to resources, seccomp denies the ability to make certain system calls at all.

The Update System: Channels, Tracks, and Delta Downloads

Snap’s update system is more sophisticated than most users realize.

Channels

Every snap in the Snap Store exists on a channel with a risk level:

stable         # What most users want
candidate      # Almost ready, needs testing
beta           # Active development, may break
edge           # Latest commit, CI builds

You can install a specific channel:

snap install vlc --channel=beta
snap switch vlc --channel=stable  # Change channel after install
snap install code --channel=latest/stable  # Explicit track/risk

Tracks are for major version branching:

snap install chromium --channel=stable    # "latest" track implicitly
snap install chromium --channel=107/stable # Pin to Chromium 107

This matters for security: a CVE affecting Chromium 110 doesn’t affect snaps pinned to older tracks. This is the snap hold equivalent for staying on a specific major version.

Delta Updates

When snapd refreshes a snap, it requests delta packages from the Snap Store:

1. snapd tells the store its current revision of VLC
2. The store generates or retrieves a binary delta (xdelta3-based) from old→new squashfs
3. snapd downloads the delta (might be 20MB instead of 200MB)
4. snapd applies the delta to produce the new squashfs
5. The new squashfs is verified against the store’s assertions
6. /snap/vlc/current symlink is updated atomically

The delta mechanism can reduce update sizes by 80-90% for minor updates. Major updates (new bundled libraries) still download most of the image.

Assertions: The Trust Chain

Every snap in the Snap Store is cryptographically signed. snapd verifies the full chain before mounting a snap:

# Assertions are stored here:
ls /var/lib/snapd/assertions/

# The chain:
# Snap Store Root Key (Canonical-controlled)
#   → Account Key (developer's key)
#     → snap-declaration (metadata: interfaces used, slots, plugs)
#       → snap-revision (specific version: sha3-384 hash of squashfs)

When snapd downloads a snap, it verifies the snap-revision assertion’s hash matches the downloaded squashfs. This prevents MITM attacks — you can’t intercept and replace a snap in transit because you can’t forge the assertion signatures.

This is a security strength Snap has over AppImage. AppImages have no mandatory signing — assertions in AppImageKit are optional and rarely used. Snap downloads are always verified against Canonical’s key infrastructure.

Startup Latency: The Full Picture

The notorious Snap startup latency deserves a proper explanation.

On a cold start (first launch, no caches warmed):

1. squashfs mount: The /snap/vlc/3221/ mount is persistent — it’s mounted at boot. So this is not actually a cold-start cost anymore in recent snapd versions (older versions unmounted between uses).

2. AppArmor profile load: A cache of compiled AppArmor profiles is maintained. If the cache is valid, this is fast (~1ms). If the profile needs recompilation (after an update), this can take 100-500ms.

3. snap-confine overhead: Creating the mount namespace, loading seccomp filter, executing the snap-exec chain: ~50-200ms depending on complexity.

4. squashfs block decompression: The first read of each block incurs decompression overhead. For an app like Firefox that reads thousands of files on startup, this adds up.

Benchmark example on typical SSD hardware:

# Firefox snap cold start
time snap run firefox
# real: 8.3s

# Firefox .deb (apt-installed) cold start
time firefox
# real: 2.1s

# Firefox snap warm start (second launch)
time snap run firefox
# real: 3.1s

The 6-second cold-start penalty is almost entirely squashfs decompression. The warm-start gap (1 second) is snap-confine overhead.

On NVMe SSDs with fast decompression (Ryzen processors have hardware-assisted lz4), the gap narrows. On spinning disks, it widens. This is a hardware problem masquerading as a software problem.

The Chromium Situation: What Actually Happened

This deserves documentation because it’s still misunderstood.

In 2019, Canonical deprecated the chromium-browser Debian package. They stopped publishing security updates to the .deb. Instead, they maintained the Chromium snap and made apt install chromium-browser on Ubuntu install the snap instead.

The mechanism: Ubuntu patches apt to recognize the chromium-browser package name as a snap trigger. When you apt install chromium-browser, Ubuntu’s package manager calls snap install chromium instead of downloading a .deb.

This was documented in Ubuntu’s release notes. It was also, let’s be honest, surprising and somewhat jarring to experienced Linux users who expected apt install to install a .deb.

The reasoning from Canonical’s side is valid: maintaining Chromium packages for each Ubuntu LTS is genuinely difficult because Chromium’s build system evolves faster than LTS timelines. The snap is better maintained. The security updates arrive faster.

The frustration from users is also valid: intercepting apt install and doing something different from what was asked breaks user expectations and violates the principle of least surprise.

Both things are true.

The 2025–2026 Update

By 2025–2026, the sharpest edges of this situation have been sanded down. Mozilla now publishes a first-party Firefox .deb via their own PPA, which many users prefer over the snap. Google Chrome has always been a .deb from Google directly. Ungoogled Chromium has active community PPA maintenance. The dominant use case for the Chromium snap — “I just want Chromium on Ubuntu and I don’t want to manage a PPA” — still applies, but the claim that users have to use the snap for a maintained browser is no longer true.

Snap’s center of gravity has shifted clearly toward server/IoT: Ubuntu Core devices, Microk8s, Juju charms, and background services on Ubuntu Server. That’s where Canonical’s investment is concentrated, and it’s where Snap’s architectural decisions make the most sense. The desktop snap wars have largely cooled.

What snapd Stores on Your System

A complete accounting of the Snap footprint:

# Squashfs mount points (one per revision of each snap):
mount | grep squashfs | wc -l
# On a typical Ubuntu desktop: 10-20 loop devices

# Actual disk usage:
du -sh /var/lib/snapd/snaps/    # squashfs files
du -sh /var/snap/               # App data (user data here is yours, don't delete)
du -sh /snap/                   # Mount points (no real disk usage, just mounts)

# AppArmor profile cache:
du -sh /var/cache/apparmor/

# On a moderate Ubuntu desktop install:
# /var/lib/snapd/snaps/ ≈ 1-4 GB depending on what's installed
# /var/snap/ ≈ 200MB-2GB (mostly application data you want to keep)

The multiple revisions issue: by default, snapd keeps the current and one previous revision of each snap. That means two copies of every squashfs on disk.

# Show disk usage per snap including all revisions:
snap list --all

# Remove all old revisions:
snap list --all | awk '/disabled/{print $1, $3}' | while read name rev; do
  snap remove "$name" --revision="$rev"
done

Removing snapd (And Living With the Consequences)

If you’ve decided snapd is not for you on a non-Ubuntu distro, or you want it gone on Ubuntu:

# Remove all snaps first (in reverse dependency order):
# snap-store, gnome-3-38, gtk-common-themes, firefox, ...
snap list  # Check what's installed

for snap in $(snap list | awk 'NR>1{print $1}'); do
  sudo snap remove --purge "$snap"
done

# Remove snapd itself:
sudo apt purge snapd
sudo apt-mark hold snapd  # Prevent reinstallation by apt

# Clean up:
rm -rf ~/snap              # User app data (~/snap/vlc/, ~/snap/firefox/, etc.)
sudo rm -rf /snap          # Kernel loop-mounted squashfs images (system-wide mount point)
sudo rm -rf /var/snap      # Per-snap revision data
sudo rm -rf /var/lib/snapd # snapd database, downloaded .snap files, assertions

# Optionally: prevent Ubuntu from reinstalling snap packages via apt
# Create a preference file:
cat << 'EOF' | sudo tee /etc/apt/preferences.d/no-snaps.pref
Package: snapd
Pin: release a=*
Pin-Priority: -10
EOF

The apt-mark hold is important — some Ubuntu packages will try to pull snapd back in as a dependency (notably ubuntu-desktop). The no-snaps.pref file is the nuclear option that prevents any snap from being installed via apt.

On an Ubuntu machine where snap-delivered apps are part of the default setup (Firefox, Thunderbird), removing snapd means finding .deb alternatives or Flatpak equivalents. The Firefox PPA maintains a .deb:

sudo add-apt-repository ppa:mozillateam/ppa
echo 'Package: *
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001' | sudo tee /etc/apt/preferences.d/mozilla-firefox
sudo apt install firefox

The Genuine Strengths

I’ve been hard on Snap. Let me be fair about where it excels.

Server and IoT use cases: On Ubuntu Core, snaps are excellent. The atomic update and rollback mechanism, combined with the confined execution model, makes remote management of headless devices genuinely reliable. When a snap update fails on a Raspberry Pi you can’t physically access, the automatic rollback is not a nice-to-have — it’s the difference between a bricked device and a working one.

The interface system: The depth and granularity of Snap’s interface system is impressive. The ability to see and control exactly what each app can access — and have that control enforced at the kernel level — is a real security feature. snap connections is one of the most useful security introspection tools on Linux.

The update security model: Assertions + signing + delta updates is a well-designed system. The cryptographic chain from Canonical’s root key to individual snap revisions is stronger than most distribution systems.

What to Read Next

• snapd architecture documentation: github.com/snapcore/snapd
• Interface reference: snapcraft.io/docs/interfaces
• AppArmor reference: ubuntu.com/server/docs/security-apparmor
• snap-confine source: github.com/snapcore/snapd/tree/master/cmd/snap-confine

Next: Part 4 — Flatpak: Portal Architecture and the Bubblewrap Sandbox

Andrew Denner — denner.co — @adenner

The Complete Technical Picture

Part 2 of the Linux Universal Packages series

Part 1 of this series covers the overview, decision matrix, and talk highlights. This part goes further than any conference talk has time for. We’re going into the binary format, FUSE internals, the update spec, and the libfuse2/libfuse3 incident that broke half the AppImages on Ubuntu 22.04. Grab coffee.

What an AppImage Actually Is at the Byte Level

The executive summary: an AppImage is a concatenated ELF binary and squashfs filesystem. That’s it. Understanding those two pieces explains almost everything about how they work and why they behave the way they do.

The Runtime Binary

Every AppImage starts with a runtime — a small C binary typically called runtime.c from the AppImageKit project. When your kernel executes an AppImage, it executes this runtime just like any other ELF binary. The runtime’s job is to:

1. Find its own file path (/proc/self/exe)
2. Mount the squashfs payload that follows it in the same file
3. Find and execute the AppRun script inside that squashfs
4. Wait for AppRun to exit, then unmount and clean up

The squashfs payload starts at a specific byte offset immediately after the runtime binary. The runtime finds this offset because it knows the size of itself — it’s compiled with its own size embedded as a constant. In AppImageKit, this is the sfs_offset variable. When the runtime binary is built, the build system appends this size at a well-known location so the runtime can find where the filesystem begins.

You can inspect this yourself:

# Find the squashfs magic bytes — 0x73717368 ("sqsh") or 0x68737173 ("hsqs")
xxd MyApp.AppImage | grep -m1 "7371 7368\|6873 7173"

The offset you find is where the squashfs filesystem starts. Everything before it is the runtime ELF.

ELF Header Inspection

# Verify the ELF header (first 4 bytes: 7f 45 4c 46 = ELF magic)
xxd -l 16 MyApp.AppImage

# Check the embedded section that tells file(1) this is an AppImage
file MyApp.AppImage
# Output: ELF 64-bit LSB executable ... (AppImage)

# The AppImage type is embedded in ELF section .note.ABI-tag
# Type 1 = AppImage format 1 (libarchive-based, older)
# Type 2 = AppImage format 2 (squashfs, current)
readelf -n MyApp.AppImage | grep -A2 AppImage

The .note.ABI-tag ELF section contains a note of type AI\x02\x00 (AppImage format 2). This is how tools like AppImageLauncher identify AppImages without relying on file extension — though the .AppImage extension is conventional, not required.

The squashfs Filesystem

squashfs is a read-only compressed filesystem that’s been part of the Linux kernel since 2.6.29. It’s the same filesystem format used in live CDs, embedded systems (OpenWRT, among many others), and Snap/Flatpak runtimes. AppImage is not special in using squashfs — it’s just the right tool for a read-only bundled filesystem.

Key properties:

• Read-only: squashfs is always compressed and always read-only. No exceptions.
• Block-compressed: data is compressed in blocks (typically 128KB). Random access to compressed data requires decompressing the block containing your data. This is why squashfs-based formats have slightly higher latency on first access.
• Supported compression: gzip, lzma, lzo, xz, zstd. AppImages typically use gzip or xz. xz gives better compression at the cost of slower decompression.
• Directory structure: standard POSIX filesystem with inodes, directory entries, symlinks, device nodes.

You can inspect the squashfs directly:

# Extract squashfs at the known offset (replace OFFSET with actual value)
dd if=MyApp.AppImage bs=1 skip=OFFSET of=/tmp/myapp.squashfs

# Or let unsquashfs figure it out
unsquashfs -l MyApp.AppImage | head -40
# Shows the directory tree inside the AppImage

The AppDir Layout

Inside the squashfs, AppImages follow the AppDir spec. This is a semi-formal specification that defines what an application directory should contain to be portable.

A minimal AppDir:

MyApp.AppDir/
├── AppRun              # Executable or symlink; entry point
├── myapp.desktop       # Standard .desktop file
├── myapp.png           # Icon (various sizes ideally)
├── usr/
│   ├── bin/
│   │   └── myapp       # The actual binary
│   ├── lib/
│   │   ├── libssl.so.1.1    # Bundled libraries
│   │   └── libcrypto.so.1.1
│   └── share/
│       └── myapp/      # Data files, assets
└── .DirIcon            # Optional: icon for file managers

The AppRun file is the entry point. For simple apps it might just be:

#!/bin/bash
exec "$APPDIR/usr/bin/myapp" "$@"

But more complex AppRuns set up LD_LIBRARY_PATH to point to bundled libraries, set XDG_DATA_DIRS to include bundled data, and handle architecture-specific paths.

The .desktop file must follow the Desktop Entry Specification. AppImageLauncher uses it to create system integration — the Name, Exec, Icon, and Categories fields become the application menu entry.

FUSE Mounting: What Actually Happens at Runtime

When you execute an AppImage, the runtime uses FUSE (Filesystem in Userspace) to mount the squashfs payload. Here’s the sequence:

1. Runtime binary starts execution
2. It calls fuse_main() (or equivalent) with the squashfs offset as the source
3. The kernel FUSE driver creates a virtual mountpoint at /tmp/.mount_<appname><random>/
4. The squashfs is accessible at that path
5. Runtime exec()s $MOUNTPOINT/AppRun
6. AppRun runs (the application executes)
7. When AppRun exits, FUSE unmounts and cleans up

You can observe this while an AppImage is running:

# In one terminal, run an AppImage
./MyApp.AppImage &

# In another terminal:
mount | grep AppImage
# Shows: /tmp/.mount_MyApp<hash> type fuse.squashfuse ...

ls /tmp/.mount_MyApp*/
# Shows the AppDir contents live

This FUSE mount is why AppImages don’t need root — all mounting happens in userspace. The kernel FUSE driver brokers access between the runtime’s FUSE filesystem server and any processes trying to read files through it.

Why AppImage Doesn’t Need Root (and What That Costs)

Traditional mount requires CAP_SYS_ADMIN. FUSE gets around this because:

1. The FUSE kernel module exposes /dev/fuse
2. Any user can open /dev/fuse and register a FUSE filesystem server
3. The kernel routes filesystem operations to that server via the /dev/fuse file descriptor

The userspace FUSE server (squashfuse) reads compressed blocks from the AppImage file and hands them to the kernel. Every read operation goes: kernel → FUSE driver → squashfuse userspace process → kernel buffer → calling process.

This is why AppImage I/O has slightly higher overhead than directly reading a filesystem — there’s an extra userspace hop for every file access. In practice this is negligible for most apps.

The libfuse2/libfuse3 Incident

This is the section of AppImage history that’s still causing production headaches in 2026.

Background

AppImageKit’s FUSE runtime historically used libfuse2 (the 2.x API). This was the dominant FUSE library for years. Around 2021-2022, libfuse3 became the standard — it’s not backward-compatible at the API level, though the kernel protocol is the same.

Ubuntu 22.04 LTS (Jammy) shipped without libfuse2 installed by default. Only libfuse3 was included. The AppImage runtime links against libfuse2 at build time, so:

# On Ubuntu 22.04 with a pre-2022 AppImage:
./OldApp.AppImage
# Error: cannot open shared object file: libfuse.so.2: No such file or directory
# OR just: Segmentation fault

The fix:

sudo apt install libfuse2

That’s it. One package install. But it breaks the “AppImages just work without installing anything” story, and it surprised a lot of users who didn’t know what FUSE was.

The Longer-Term Fix

By 2025-2026, most actively maintained AppImages have migrated to the FUSE 3 runtime or added a fuse3 fallback. The cases still requiring libfuse2 tend to be older or unmaintained AppImages, and some legacy scientific computing tools that haven’t been rebuilt since the transition. If you’re building new AppImages, use current AppImageKit tooling — it handles FUSE 3 correctly. If you encounter an older AppImage that still needs libfuse2, the install command is just sudo apt install libfuse2.

Some AppImages also include a --appimage-extract-and-run fallback that extracts the squashfs to a temporary directory and runs without FUSE at all (slower, but works on systems without any FUSE support).

You can use this workaround for any AppImage:

./MyApp.AppImage --appimage-extract-and-run
# Or set the environment variable permanently:
export APPIMAGE_EXTRACT_AND_RUN=1

AppImage Update Protocol: zsync

The AppImage spec includes an optional update mechanism based on zsync — a partial file download algorithm that applies binary diffs over HTTP. This is how AppImages can self-update without downloading the entire file each time.

How zsync Works

zsync is essentially rsync-over-HTTP. The basic idea:

1. The new version of the file is available at a URL
2. A .zsync file at a known URL describes the new file as a series of blocks with checksums
3. The client reads its existing file and computes which blocks are already correct
4. Only the changed blocks are downloaded
5. The client reassembles the new file from old blocks + downloaded new blocks

For large AppImages (200MB+), this can reduce update downloads to a few megabytes for minor version bumps that only changed code, not bundled libraries.

The Update URL Embedded in the AppImage

AppImages that support updating embed an update_information field in their .desktop file or in the ELF’s custom section. It looks like:

gh-releases-zsync|owner|repo|latest|MyApp-*.AppImage.zsync

This tells AppImageUpdate:

• The update mechanism (gh-releases-zsync = GitHub releases with zsync)
• Where to find the latest release
• Which file pattern matches the update

# Use AppImageUpdate to check for updates
appimageupdate --check MyApp.AppImage
# Output: [INFO] update available, new version: 1.2.3

# Apply the update (downloads diff, creates new version)
appimageupdate MyApp.AppImage

Most AppImages don’t include this. The AppImageUpdate tool is optional and rarely used. The official AppImageHub catalog tracks update info for listed apps. AppImageLauncher integrates with this to show update notifications in the tray — when you double-click an AppImage in AppImageLauncher, it registers it and periodically polls for updates.

Building AppImages: appimagetool

Understanding how AppImages are built gives you the full picture of what you’re running.

# Minimal AppImage build process:

# 1. Create your AppDir structure
mkdir -p MyApp.AppDir/usr/bin
cp myapp MyApp.AppDir/usr/bin/
cat > MyApp.AppDir/myapp.desktop << EOF
[Desktop Entry]
Name=MyApp
Exec=myapp
Icon=myapp
Type=Application
Categories=Utility;
EOF
cp myapp.png MyApp.AppDir/

# 2. Create AppRun
cat > MyApp.AppDir/AppRun << 'EOF'
#!/bin/bash
exec "$APPDIR/usr/bin/myapp" "$@"
EOF
chmod +x MyApp.AppDir/AppRun

# 3. Bundle dependencies (the painful part)
# linuxdeploy handles this automatically:
linuxdeploy --appdir MyApp.AppDir \
  --executable usr/bin/myapp \
  --desktop-file MyApp.AppDir/myapp.desktop \
  --icon-file myapp.png \
  --output appimage

# This produces: MyApp-x86_64.AppImage

linuxdeploy traces the app’s dynamic library dependencies (ldd), copies them into the AppDir, and patches the ELF rpath so the bundled libraries are found. It also handles Qt and GTK plugin deployment via separate plugins.

This is where the “bundle everything” tradeoff becomes concrete: linuxdeploy might find 40+ shared libraries to bundle, adding 50-100MB to the AppImage before compression.

Security: The Actual Risk Surface

Let’s be honest about what the AppImage security model is and isn’t.

What AppImage Does NOT Provide

• No sandbox: AppImages run with your full user permissions. ~/.ssh, ~/.gnupg, your entire home directory — all accessible.
• No code signing verification: The ELF + squashfs structure has no built-in signature verification. AppImageKit supports GPG signatures embedded in the image, but checking them is not enforced and most apps don’t implement it.
• No update integrity checking: zsync downloads use checksums for file integrity, but there’s no signing of the release itself beyond whatever the hosting platform provides (GitHub releases, for example, doesn’t sign artifacts by default).
• No AppArmor/seccomp: Unlike Snap, there’s no mandatory access control applied to AppImage processes.

What AppImage DOES Provide

• Isolation by convention: The app runs from a read-only squashfs. It can’t modify its own code at runtime. This isn’t security — the app can still write anywhere in your home directory — but it does mean the app files themselves aren’t modified.
• System isolation: The app’s files don’t mix with your system’s files. There’s no risk of an AppImage install breaking system packages.
• Reproducibility: You know exactly what version you’re running and it will never change unless you download a new file. This is the opposite of Snap’s auto-update, and for certain security contexts it’s actually preferable.

The Trust Model

When you download an AppImage, you are trusting:

1. The developer who built it
2. The infrastructure that distributed it (GitHub releases page, developer’s website, etc.)
3. Your connection to that infrastructure (HTTPS protects against in-transit modification)

This is roughly equivalent to downloading and running a binary from a developer’s website — which is what a lot of users did before universal packages existed. It’s not inherently worse than that baseline, but it’s significantly weaker than a distro package or Flatpak where there’s review infrastructure.

Practical guidance:

• Download from the project’s official GitHub releases page or official website only
• Verify SHA256/SHA512 checksums when provided (many projects publish them)
• Prefer projects that embed GPG signatures in their AppImages (appimagetool --sign)
• Check the AppImageHub catalog for community-vetted apps

The --appimage-* CLI Flags

Every AppImage built with AppImageKit supports these built-in flags:

# Extract the squashfs to a directory (no FUSE needed)
./MyApp.AppImage --appimage-extract
# Creates: squashfs-root/

# Mount the squashfs and show mount path (for debugging)
./MyApp.AppImage --appimage-mount
# Prints: /tmp/.mount_MyApp<hash>
# Ctrl+C to unmount

# Show embedded update info
./MyApp.AppImage --appimage-updateinfo

# Show version of AppImageKit runtime
./MyApp.AppImage --appimage-version

# Run without FUSE (extract to temp dir and exec)
./MyApp.AppImage --appimage-extract-and-run

# Show offset where squashfs starts
./MyApp.AppImage --appimage-offset

These are invaluable for debugging why an AppImage isn’t working. If --appimage-mount fails, you have a FUSE problem. If it succeeds, the issue is in AppRun. If --appimage-extract succeeds but running the extracted binary fails, you have a missing dependency.

AppImageLauncher: The Missing Piece

AppImageLauncher is what makes AppImages usable as a daily driver. Without it, you’re managing raw files. With it, you get:

1. Automatic integration: Double-click an AppImage → dialog asks if you want to integrate it into the system. Integrating copies it to ~/Applications/ and creates a .desktop file in ~/.local/share/applications/.

2. Update checking: Integrated apps are periodically checked for updates using the embedded update URL. The tray icon notifies you.

3. Removal: Right-click → Remove from the app menu unintegrates the app and removes the file.

4. binfmt_misc registration: AppImageLauncher registers an interpreter via /proc/sys/fs/binfmt_misc so AppImages execute directly without needing chmod +x. The kernel recognizes the AppImage ELF magic and routes execution through AppImageLauncher.

The binfmt_misc trick is the clever part:

# AppImageLauncher registers something like:
# :AppImage:M:0:\x7fELF...\x41\x49::/usr/lib/AppImageLauncher/runtime:
# This tells the kernel: if ELF file contains AI magic bytes, exec it via the runtime
cat /proc/sys/fs/binfmt_misc/appimage-type2

This is why AppImages “just work” without chmod +x when AppImageLauncher is installed — the binfmt_misc interpreter handles execution before the execute permission check.

Performance Characteristics

A few numbers worth knowing:

Cold start overhead: 200-800ms additional latency vs a native binary, depending on squashfs compression and how many files need to be accessed. This is FUSE overhead + decompression. After first launch, the OS page cache stores the decompressed blocks and subsequent launches are faster.

Warm start: Near-native performance once the squashfs blocks are cached in RAM.

Memory overhead: The squashfuse process adds ~2-5MB RSS. Negligible.

Disk I/O: Higher than native binaries for file reads from the AppImage filesystem. For I/O-heavy operations, consider whether the app should be running from an AppImage at all.

Compression ratio: Typical AppImages achieve 2-4x compression on the squashfs payload. A 200MB installed app might be a 80-100MB AppImage.

What to Read Next

• AppImageKit source: github.com/AppImage/AppImageKit
• AppDir spec: github.com/AppImage/AppImageSpec
• linuxdeploy: github.com/linuxdeploy/linuxdeploy
• AppImageUpdate: github.com/AppImage/AppImageUpdate
• squashfs-tools: apt install squashfs-tools — provides unsquashfs

Next: Part 3 — Snap: Anatomy of Canonical’s Distribution System

Andrew Denner — denner.co — @adenner

Fair warning: I have opinions. The facts are accurate to the best of my knowledge. The takes are mine.

Why These Formats Exist (And Why It’s Not Your Distro’s Fault)

Let me start with the thesis, because I’m tired of people framing this wrong.

Linux does not have an app problem. Linux has a distribution contract problem.

Your distro’s package maintainers have a job: make sure every package in the repo works together, doesn’t break your server at 2am, and doesn’t introduce security vulnerabilities by surprise. They optimize for stability, security, and policy. This is a good job and they do it well.

App developers have a different job: ship software that uses the latest version of every library they depend on, tested against their own development environment, on a timeline that doesn’t involve waiting for Debian to unfreeze. They optimize for shipping fast and not answering bug reports about Ubuntu 18.04.

These goals are fundamentally incompatible, and no amount of goodwill makes them compatible. Universal packages are the peace treaty: the app developer bundles everything their app needs into one isolated unit, and your distro doesn’t have to care what’s in it.

The catch — and I want you to internalize this before we go further — is that you’ve moved the problem, not solved it. You didn’t eliminate dependency hell. You moved it into a zip file. The zip file now weighs 400 megabytes. Some people think this is progress.

AppImage: The Cowboy Method

AppImage is the oldest of the three formats and, in some ways, the most honest. It makes no promises about security, integration, or management. It’s a file. You run it. It works.

Technically, an AppImage is an ELF binary that contains a squashfs filesystem. When you execute it, the binary mounts the squashfs, finds the app entrypoint, and runs it. The entire app — binary, libraries, assets — is in that one file. No install required.

The install process is:

chmod +x MyApp.AppImage && ./MyApp.AppImage

That’s it. That’s the whole thing.

The Good Stuff

The portability story is genuinely compelling. Copy an AppImage to a USB stick and run it on any modern Linux machine without touching the package manager. This is not theoretical — it actually works. I’ve used this for distributing internal tools to machines I don’t control, and it’s the cleanest solution I’ve found.

No root required. No system changes. No daemon watching it. No database entry. Delete the file and the app is completely uninstalled. For people who’ve spent decades navigating “dpkg –purge” and leftover config files in /etc, this is surprisingly satisfying.

AppImages are also great for running multiple versions of the same app simultaneously, which sounds niche until you need it. I maintain three different AppImage versions of a scientific computing tool because different projects have different requirements, and they coexist without conflict.

The Bad and Ugly

The security model is, diplomatically, “trust me bro.” There is no built-in sandbox. An AppImage runs with your full user permissions on your system. You are trusting that whoever compiled and uploaded the AppImage is a reasonable person who didn’t do anything sketchy.

Download from official project pages. Check the appimage.github.io catalog, which maintains a list of vetted AppImages. Never run an AppImage from a link someone posted in a forum. This is not theoretical advice.

No auto-updates by default. The AppImage spec includes an update protocol, but implementing it is optional, and most AppImages don’t. Your AppImage will never tell you there’s a newer version. You have to care about this yourself.

Desktop integration is also manual or nonexistent by default. No .desktop file, no app menu entry, no file associations. The solution is AppImageLauncher, which is excellent — it handles integration automatically when you double-click an AppImage — but you have to install it separately.

The chmod +x Tax

I’ve given this section a name because it deserves one. Every single time I do an AppImage demo, I forget to chmod +x the file before trying to run it. Every time. I’ve been doing this for a decade. I will forget again.

The error is “Permission denied,” which is not the most informative error message for someone who doesn’t know what’s happening. The fix is obvious in retrospect but annoying every time.

This is the single most common AppImage support question. If someone tells you AppImages don’t work, check for chmod +x first.

Who AppImage Is For

AppImage is the right choice when portability is the primary constraint. Air-gapped systems, one-off tool deployments, situations where you need to know exactly what version of an app is running and guarantee it doesn’t change without your knowledge. Devs who want to ship to users without caring what distro they’re running.

It is not a replacement for your package manager. Don’t use AppImages for system tools, libraries, or anything where you want security updates to happen automatically.

Snap: Canonical’s Walled Garden

Snap is Canonical’s answer to universal packaging, and it carries all the hallmarks of Canonical’s approach to Ubuntu: opinionated, integrated, centralized, and occasionally doing things without asking you first.

I’ve rage-deleted snapd twice. I reinstalled it once. My feelings are complicated.

What Snap Actually Is

At a technical level, a snap is a squashfs image wrapped in metadata, distributed through Canonical’s Snap Store (snapcraft.io). When you install a snap, snapd downloads the image, mounts it at /snap/<name>/<revision>/, and wraps it in an AppArmor confinement profile.

The confinement is real and meaningful. Each snap declares “interfaces” — named permission scopes that correspond to things like home directory access, network access, camera, microphone, USB devices. The snap only gets what it declares, and even then only what you’ve connected.

snap connections <name> shows you exactly what a snap can access. It’s genuinely impressive. I wish more people knew this existed.

The Ubuntu Integration Story

On Ubuntu, snapd is pre-installed and always running. snap install works out of the box. This is either a feature or an annoyance depending on your perspective.

The Chromium situation is worth explaining clearly because it surprises people the first time: on Ubuntu, apt install chromium-browser does not install a .deb. Ubuntu intercepts the command and installs the Chromium snap. This is documented behavior, not a bug, and Canonical’s reasoning is that the snap version is better maintained and more secure.

Worth noting: if you specifically want a .deb Chromium, alternatives exist — Google Chrome has always distributed a .deb directly, and Ungoogled Chromium is available through community PPAs. But the default apt install chromium-browser path on Ubuntu leads to snap.

Your reaction to all of this tells me everything about your relationship with Canonical.

The Startup Latency Problem

Every snap app mounts its squashfs image on launch. This is the source of the notorious snap startup latency.

On an SSD with a cold cache, expect a noticeable delay — particularly on first boot or slower drives. NVMe SSDs and improved squashfs caching have narrowed the gap compared to a few years ago, but the first cold start is still measurably slower than an equivalent apt-installed binary. The latency improves after the first launch as the OS caches the squashfs blocks.

This matters more for some apps than others. Firefox snap’s startup time is one of the most-criticized aspects of Ubuntu’s default setup. Server apps where you don’t care about interactive startup time? The latency is irrelevant.

The Auto-Update Gotcha

Snaps auto-update. By default. Without asking you. While you’re working.

This is usually fine. Sometimes a snap update changes behavior in a way that breaks your workflow. And once — I’m speaking from experience — a snap refreshed during a live presentation, closing the app mid-demo with no warning.

Run this before any presentation, demo, or situation where an unexpected app close would be embarrassing:

sudo snap refresh --hold=48h

Hold a specific snap for longer if needed:

sudo snap refresh --hold=168h vlc

This is the most practically useful Snap tip I know. I wish I’d known it earlier.

The 2025–2026 Reality Check

By 2025–2026, the fiercest “Snap wars” era of desktop Linux has largely cooled. The most contentious flashpoints — Firefox and Chromium being snap-only on Ubuntu — have been resolved by well-maintained .deb alternatives: Mozilla publishes a Firefox .deb via their own PPA, Google Chrome has always been a .deb, and Ungoogled Chromium is available through community PPAs. Many experienced Ubuntu users simply pin the .deb versions over the snap equivalents for desktop apps where startup time matters.

Snap’s center of gravity has shifted more clearly toward what it was always best at: Ubuntu server and Ubuntu Core IoT. The Snap Store is the right distribution channel for daemons, services, and appliance software running on Canonical’s platforms. The desktop wars have cooled — which is good for everyone.

Who Snap Is For

Snap is at its best for server-side daemons and background services on Ubuntu, browsers (the security benefits are real even if the startup is slow), and any situation where you want automatic security patching and don’t care much about the startup time.

Snap is at its worst for CLI tools (latency is painful), apps where you want theme integration, and non-Ubuntu systems (Snap essentially requires systemd and doesn’t work well outside the Canonical ecosystem).

Flatpak: The Community Darling

Flatpak is the universal package format that most of the Linux community outside of Ubuntu’s gravity well has converged on. It ships by default on Fedora, Linux Mint, and elementary OS. On Ubuntu, you install it yourself — which is, incidentally, one of the things people resent.

I use Flatpak for most of my desktop GUI apps. I also complain about it constantly. Both of these things are true and not in conflict.

The Technical Differences

Flatpak uses Bubblewrap for sandboxing rather than AppArmor. Bubblewrap creates a minimal namespace environment for each app — separate filesystem view, separate process tree, no access to things it hasn’t been explicitly granted.

The portal system is Flatpak’s most interesting idea. Rather than giving apps direct access to system resources, they go through “portal” services — OS-managed brokers that handle things like file access, camera, screen capture, and notifications. When a Flatpak app opens a file dialog, it’s actually asking the portal for a file, and the portal shows the native file picker. The app never sees your filesystem directly unless you grant it access.

This is clever and mostly works. It’s occasionally annoying for CLI tools that expect to do file operations directly without asking permission.

Flatpak also supports decentralized remotes — you’re not locked into a single corporate store. Flathub is the main one and has around 2,500 apps at this point, but you can add any remote including your own.

Flatseal Is Non-Negotiable

Install Flatseal before any other Flatpak. This is not optional advice.

flatpak install flathub com.github.tchx84.Flatseal

Flatseal is a GUI permission manager for Flatpak apps. Without it, managing permissions requires knowing the CLI override syntax and looking up the right portal names. With it, you can see every permission each app has and toggle them with checkboxes.

If you’re going to show one thing to someone who’s skeptical about sandboxing, show them Flatseal. It’s the clearest demonstration of “here’s exactly what this app can access, and you can control it” I’ve seen on Linux.

The Disk Usage Reality

Do du -sh ~/.var/app after using Flatpak for a few months. I’ll wait.

The shared runtime system means apps that use the same base runtime (say, GNOME Platform 45) share that runtime rather than each bundling it. This is better than every app bundling everything. It is not as good as “install GIMP and it uses the system GTK like a normal package.”

If you install GNOME apps and KDE apps, you’re downloading both runtimes. Each can be 500MB+. If you install apps casually and don’t prune regularly, your Flatpak storage will grow in ways that will surprise you.

Prune it regularly:

flatpak uninstall --unused          # remove orphaned runtimes
flatpak uninstall --delete-data     # also wipe app data dirs for removed apps
flatpak repair                      # fix corruption and deduplicate

The --unused flag is the main one to run monthly. If you’ve been casual about installing and removing apps for a while, flatpak repair will reclaim space from duplicate objects. Put flatpak uninstall --unused in a cron job — disk reclamation you never have to think about.

CLI Verbosity: The App ID Problem

Flatpak uses reverse-domain app IDs. Installing LibreOffice is:

flatpak install flathub org.libreoffice.LibreOffice

Running it is:

flatpak run org.libreoffice.LibreOffice

This is verbose. Tab completion helps. Shell aliases help more. But it’s a real friction point, especially when you’re used to apt install libreoffice.

Who Flatpak Is For

Flatpak is the right choice for desktop GUI applications where you want sandboxing with visible, controllable permissions. If you’re on a non-Ubuntu distro, it’s probably already there and is the natural choice. If you want to run apps that actually respect your desktop theme, Flatpak integrates better than Snap.

It is not ideal for CLI tools (portal permission prompts are annoying for things that just need to write a file), and the verbose CLI means it’s not the best choice for quick one-off installs.

Head to Head: The Actual Comparison

Let me be blunt about the decision matrix.

Use AppImage when:

• Portability is the primary constraint
• You’re on an air-gapped system
• You need a specific version and don’t want it to update
• You want zero system changes
• You’re on a weird distro and just need something to run

Use Snap when:

• You’re on Ubuntu and the app is available
• It’s a server or daemon app where auto-updates are a feature
• You want the app to auto-patch and you trust Canonical’s update cadence
• Startup latency doesn’t matter (background services, browsers you keep open)

Use Flatpak when:

• It’s a desktop GUI app
• You care about sandboxing and want to see what the app can access
• You’re on any non-Ubuntu distro that ships Flatpak
• You want your apps to actually look like they belong on your desktop

Use apt/dnf/pacman when:

• The app is in your distro’s repo
• Performance matters
• You want proper system integration
• You want security updates handled by people who know your distro
• You like your sanity

That last option is the one people forget. Native package managers exist, work well for most software, and are the correct default. Universal packages are the fallback, not the baseline.

The Ugly Truths I Didn’t Want to Say But Will

Universal packages do not replace your distro’s package manager. If you’re using Flatpak for everything on Ubuntu, you’re doing it wrong. Your distro’s maintainers package most of what you need and they package it better.

They all increase disk usage. Significantly. This is the price of bundling. It’s a reasonable price for some use cases and an unreasonable one for others.

They all shift trust boundaries. With apt, you’re trusting your distro’s security team. With universal packages, you’re trusting the developer — or the Flathub submission reviewers, or Canonical’s store team. This is not necessarily worse, but it’s different, and you should be aware of it.

The reason we have three competing formats is because the Linux community couldn’t agree on one. This is both a joke and a tragedy. Each format made different tradeoffs, found different communities, and entrenched. We now live in a world where you might reasonably have AppImages, Snaps, Flatpaks, and native packages all installed on the same machine.

I do. I complain about it. I keep all of them.

The Series Goes Much Deeper

This post is the overview and decision matrix. If you want the real internals, the six-part series has you covered:

Part 2 — AppImage Deep Dive: Goes byte-for-byte into the ELF + squashfs structure, how FUSE mounting works without root, the libfuse2/libfuse3 transition that broke Ubuntu 22.04, and the zsync update protocol. Includes the --appimage-* debugging flags nobody tells you about.

Part 3 — Snap Deep Dive: Dissects why snapd needs to run forever as a daemon, the full snap-confine → AppArmor → seccomp execution chain, the snap type hierarchy (app/base/kernel/gadget/content), and how snap assertions build the trust chain back to Canonical.

Part 4 — Flatpak Deep Dive: Bubblewrap user namespaces, OSTree content-addressable storage, and how xdg-desktop-portal lets sandboxed apps open a file picker without ever seeing your filesystem.

Part 5 — vs Docker: Compares all three formats with Docker’s full namespace stack, cgroups resource limits, and OverlayFS storage. Answers the “is a Flatpak app a container?” question properly.

Part 6 — Build Your Own: The guide I wish existed. Full walkthroughs for packaging in all three formats — including the glibc floor problem, confinement debugging, no-network-during-build constraints, and CI patterns that actually work.

Resources

AppImage:

• appimage.github.io — curated catalog
• AppImageLauncher — get this

Snap:

• snapcraft.io — store and docs
• sudo snap refresh --hold=48h — memorize this command

Flatpak:

• flathub.org — the app store
• Flatseal — install immediately
• flatpak uninstall --unused — run monthly

Talk materials:

• Slides: denner.co/talks
• Demo scripts: github.com/adenner/cialug-pkg-talk

Dear friends and family,

It is hard to believe that we are nearing the end of 2025. It has been a whirlwind of a year for us. The year started out rough with the passing of Jess’s youngest sister Gabby.

The family took a family trip to Chicago where we all enjoyed a trip to the Lincoln Park Zoo, and the Shedd aquarium. Isaac especially liked visiting all his fish friends. Navy Pier and the children’s museum.

Closer to home, Isaac has claimed the Blank Park Zoo as his own. With our family membership we have made multiple trips, and he knows where all his favorite animals are, including the giraffe, tiger, lion and even dinosaurs (they had animatronic dinos visiting for several months). Our last trip was just to have pizza and hot coco with Santa.

We also spent a long weekend in South Dakota visiting all the usual tourist hot spots including Wall Drug, Mount Rushmore, and even Bear Country. Isaac loved spotting all the animals, some even before us adults. On the way home we made a quick stop at the Bad Lands where we saw some of the locals including a large buffalo and Prairie Dogs. Unfortunately, while we were gone, our sump pump also decided to take a bit of break. Thankfully we had a remote sensor that tipped us off to the issue and were able to get help from our entire neighborhood to get the water pumped out, and basement dried out. (Especially thanks to Ron, Travis, Mike, the Colvin’s (and anyone I may have missed).

We continued to enjoy family camping trips in Bessie, our 1994 Pace Arrow Camper. It was a busy summer as you can see, but we were able to find a few weekends to go out in the woods. Isaac has still not found a playground he didn’t like, and it has been a joy to see his joy exploring new places.

It is hard to believe that Isaac is now 3 years old. He continues to thrive at Beautiful Beginnings where he is enjoying playing with his friends in the 2-year-olds room. Also, he just passed his latest swim lesson class at Foss Swim and his next step will be starting to swim on his own with a small class supervised by an instructor.

This year Jess turned the big 40 and celebrated with a big birthday bash. She continues to be a Senior Quality Assurance Specialist at Kemin. Her baking efforts continue to yield with another winning year at the Iowa State Fair with 31 entries and 5 first places.

Andy is still doing the same job at Corteva Agriscience but with a slightly different title, he is now a Scientific Computing Scientist. He also continues to be the president of the Central Iowa Linux Users group.

We hope all of you have had a great year, and that you all have a Merry Christmas, and a happy holidays!

Jess, Andy and Isaac

Leading up to the ceremony, I spent a lot of time thinking about what the Eagle Scout award truly means. It’s easy to list the famous names or the impressive statistics, but I wanted to get to the heart of it. My goal was to explore the quiet, everyday leadership that the journey instills in a person—the skills and the code that you carry with you long after you’ve folded up the sash.

This post is a slightly polished version of that speech. I hope it resonates with fellow Eagles, Scouters, parents, and anyone who believes in the power of service and living a life of purpose.

pdf original

An Eagle Scout Court of Honor Speech

Welcome everyone, I understand that when they were looking for a speaker for today, they asked a great orator, someone was well liked by all, who is handsome, and amazing but unfortunately, they were unavailable… so instead they asked me. In all seriousness, I am greatly honored to be here today to celebrate with you all Brandon’s accomplishment as well as to say a little bit about the Eagle Scout award.

As the NESA eagle roll says:

“The Eagle Scout Award. It’s Scouting’s highest rank and among its most familiar icons. [Those] who have earned it count it among their most treasured possessions. Those who missed it by a whisker remember exactly which requirement they didn’t complete. Americans from all walks of life know that being an Eagle Scout is a great honor, even if they don’t know just what the badge means.”

“The award is more than a badge. It’s a state of being. You are an Eagle Scout—never were. You may have received the badge as a [youth], but you earn it every day as a[n adult]. In the words of the Eagle Scout Promise, you do your best each day to make your training and example, your rank and your influence count strongly for better Scouting and for better citizenship in your troop, in your community, and in your contacts with other people. And to this you pledge your sacred honor.”

From the first Eagle Scout, Arthur Rose Eldred in 1910, there have been over 2.75 million men and women who have earned the award. To put it in perspective, this is a little bigger than Chicago. This last Eagle class, which you are a part of, was comprised of 29 thousand people (a little bigger than Marshalltown). When you compare this to the over one million youth who participated in Scouting (or half the size of the Kansas City metro area), this means a little less than 3% of Scouts earned their Eagle Scout award.

While doing this, they completed roughly 2.8 million hours of service projects. This service project helps not only the Eagle but their fellow Scouts step outside their bubble, identify a problem, make a plan, and follow through on it. It helps you to think about things bigger than just your immediate self and also is a taste of leading and marshaling others, some who are far older and more experienced than you are. Brandon, in your case, your bird houses helped to make the world just a bit better.

So who are your fellow Eagle Scouts? They include Astronauts, politicians, businessmen, and teachers, like the first man on the moon Neil Armstrong, or Jim Lovell of the Apollo 13 mission. Bill Amend, artist of the Foxtrot Cartoons. Inventor of the first electronic television Philo Farnsworth, or Kevin Kwan Loucks, a concert pianist. It also includes less famous people like Jamie Bryant, who later became an auto mechanic, ran several Goodyear auto service centers, and gave back serving as an Automotive Maintenance Merit Badge Counselor. Or, Don Garlits, a pioneer in the sport of drag racing.

You don’t have to stop at famous people though. It has been my experience that you always know who the Eagle Scouts are when you meet them. They are always willing and able to help, whether at work, out in their community, or at Church. They are the ones quick to tie the right knot, able to lead by example, have first aid skills, and are able to start a good campfire.

There is always something about your fellow Eagle Scouts that just makes them stand out a little bit. As TV host Mike Rowe once said:

“I learned to lead as a Boy Scout, and I’d like to think I still do today.”

These skills are recognized by those around you as well. They were the inspiration for Chris Evans when he starred as Captain America.

“There’s a kid that I grew up with named Charlie Morris. He’s the best kid I know. He was an Eagle Scout. And being an Eagle Scout is not easy — you’ve got to really do it for a long time. But he’s just such a good man, and he genuinely, genuinely puts himself last. He lives by a code. And so, when I took the role, I told Charlie, ‘Listen. I’m modeling this after you.’”

As Dr. Randy Pausch, another Eagle Scout who was a Professor of Computer Science at Carnegie Mellon University, once said:

“Becoming an Eagle Scout is just about the only thing you can put on your resume at age 50 that you did at age 14 and it still impresses.”

I know personally, it is still a major highlight of my resume.

Mike Rowe sends this advice to new Eagle Scouts:

“Don’t wait for the fold to acknowledge your accomplishments. By all means take pride in what you’ve done, but don’t let it go to your head… Fold up your sash and stow it away somewhere private with the other tokens of what you’ve done so far. Then roll up your sleeves, get out into the world and put what you’ve learned to use.”

For me, the skills that I learned in Scouting have helped me so many times, whether it be the love of camping and the outdoors that it has instilled in me, the first aid skills that I have had to practice on myself and others (what can I say, I’m a klutz), or the taste of hobbies and life skills that I picked up in all the different merit badges, like personal finance. It has also proved incalculably helpful in the organization and leadership skills it has helped me learn, whether that be at work, serving in my church, or community organizations like the State of Iowa Science and Technology Fair Board. Brandon, I know that Scouting has provided the same for you and I challenge you to go out now into life and put your skills to use. Be that light in the world that shows others by example.

I will leave you with a Poem by Richelle E. Goodrich, “Today you Soar”:

Like the grand eagle, you spread your wings
And put forth the effort to do great things…

So now you’ve reached where few even try
As the eagle high in a glorious sky.

Not superior, but grand.
Not proud, but sure.

Not a cub, wolf, or bear but an eagle pure.

Today [, Brandon,] you soar.

This guide provides a curated look at essential modern C++ libraries, moving beyond the obvious choices to highlight tools that solve complex problems with elegance and efficiency. Each section includes practical examples and expected outputs to illustrate their real-world application.

Core Foundational Libraries

Before diving into specialized domains, every C++ developer should be familiar with these foundational toolkits.

• Boost A collection of high-quality, peer-reviewed libraries that often serve as a testing ground for features that later become part of the C++ standard. It provides robust implementations for everything from multithreading and data structures to string algorithms and pseudorandom number generation. While its size can be daunting, its quality is exceptional.

• POCO (POrtable COmponents) A set of C++ libraries focused on building network-centric, portable applications for desktop, server, and embedded systems. Its modular design makes it suitable for IoT, industrial automation, and enterprise applications. POCO offers both an open-source version and a commercially licensed pro version.

Command-Line Interface (CLI) Frameworks

Creating polished and user-friendly CLIs in C++ is streamlined by several excellent libraries.

CLI11

A powerful, header-only command-line parser for C++11 and beyond that is praised for its simple, intuitive interface and rich feature set.

#include "CLI/CLI.hpp"
#include <iostream>

int main(int argc, char** argv) {
    CLI::App app{"Log Processor"};
    std::string filename;
    app.add_option("-f,--file", filename, "Log file to process")->required();
    int level = 0;
    app.add_option("-l,--level", level, "Logging level (0-4)");
    bool verbose = false;
    app.add_flag("-v,--verbose", verbose, "Enable verbose output");
    CLI11_PARSE(app, argc, argv);
    std::cout << "Processing file: " << filename << std::endl;
    std::cout << "Log level: " << level << std::endl;
    if (verbose) {
        std::cout << "Verbose mode enabled." << std::endl;
    }
    return 0;
}

cli

A header-only library for building interactive command-line interfaces, similar to a router’s console. It supports command history, nested menus, and auto-completion.

#include <iostream>
#include "cli/cli.h"
#include "cli/clilocalsession.h"

int main() {
    auto root_menu = std::make_unique<cli::Menu>("main");
    root_menu->Add(
        "status",
        [](std::ostream& out){ out << "System is OK." << std::endl; },
        "Print system status");
    root_menu->Add(
        "exit",
        [](std::ostream& out){ exit(0); },
        "Exit the application");

    cli::Cli cli(std::move(root_menu));
    cli.ExitAction( [](auto& out){ out << "Goodbye!" << std::endl; } );

    cli::CliLocalSession session(cli, std::cout, 200);
    session.MainLoop();
    return 0;
}

Scientific & High-Performance Computing

C++ is the language of choice for performance-critical scientific computing, supported by a rich set of numerical libraries.

Eigen

A powerful C++ template library for linear algebra, matrices, vectors, and numerical solvers. It is widely considered the de facto standard for such tasks due to its expressive API, high performance, and reliability.

#include <iostream>
#include <Eigen/Dense>

int main() {
    Eigen::Matrix3f A;
    A << 1, 2, 1,
         2, 1, 0,
        -1, 1, 2;

    Eigen::Vector3f b;
    b << 2, 2, 3;

    Eigen::Vector3f x = A.colPivHouseholderQr().solve(b);

    std::cout << "The solution is:\n" << x << std::endl;
}

JSON for Modern C++

Working with JSON is a common requirement, and this library makes it painless.

nlohmann/json

A single-header library that provides an intuitive way to work with JSON, mimicking Python’s dictionary and list access.

#include <iostream>
#include "nlohmann/json.hpp"

using json = nlohmann::json;

int main() {
    // Create a JSON object
    json j;
    j["service"] = "config-server";
    j["port"] = 8080;
    j["active"] = true;
    j["protocols"] = {"http", "https"};

    // Serialize to string
    std::cout << "Serialized JSON: " << j.dump(4) << std::endl;

    // Parse from string
    auto parsed_json = json::parse(R"({"user": "admin", "id": 123})");
    std::cout << "Parsed user: " << parsed_json["user"] << std::endl;
}

GUI and Visualization Libraries

While C++ GUI development can be complex, several excellent libraries cater to different needs, from embedded systems to full-scale desktop applications.

Dear ImGui

A fast, portable, and self-contained immediate-mode graphical user interface library. It is especially popular for game development tools, 3D applications, and real-time debugging overlays due to its simplicity and high performance.

// This is a conceptual example. Full implementation requires a rendering backend (e.g., OpenGL, Vulkan).
#include "imgui.h"

void RenderMyUI() {
    ImGui::Begin("System Control");
    static int counter = 0;
    ImGui::Text("Application average %.3f ms/frame (%.1f FPS)", 1000.0f / ImGui::GetIO().Framerate, ImGui::GetIO().Framerate);
    if (ImGui::Button("Increment Counter")) {
        counter++;
    }
    ImGui::SameLine();
    ImGui::Text("counter = %d", counter);
    ImGui::End();
}

Other Notable GUI Frameworks:

• Qt: A mature, comprehensive, cross-platform framework for creating applications with advanced user interfaces. It is more than just a GUI toolkit, providing extensive modules for networking, databases, and more.

• wxWidgets: A library that allows developers to create applications with a truly native look and feel by using the platform’s native API rather than emulating it.

Specialized and Utility Libraries

For specific problem domains, these libraries are indispensable.

• OpenCV (Open Source Computer Vision Library): The premier library for real-time computer vision, machine learning, and image processing. It features thousands of optimized algorithms for tasks like facial recognition, object detection, and 3D model extraction.

• GoogleTest: A powerful and widely adopted testing framework from Google. It provides an xUnit test framework and a mocking framework, making it a cornerstone of modern C++ development for ensuring code quality.

• Asio: A cross-platform C++ library for network and low-level I/O programming that provides developers with a consistent asynchronous model. It is part of the Boost project but is also available as a standalone library.

Strategic Considerations for Library Selection

For architects and senior engineers, choosing a library involves more than just its API.

• Build System Integration: The C++ ecosystem relies heavily on build systems like CMake and package managers like vcpkg and Conan. A library’s compatibility with these tools is a major factor in its ease of adoption.

• Modern C++ Compatibility: Ensure that a library aligns with your project’s target C++ standard (e.g., C++17, C++20) and follows modern best practices.

• Header-Only vs. Compiled: Header-only libraries offer easy integration but can increase compile times. Compiled libraries require linking but can be faster to build against once compiled.

Conclusion

Mastering modern C++ involves building a curated toolkit of high-quality libraries. The selections presented here represent robust, battle-tested solutions that empower developers to build complex, high-performance systems efficiently. By understanding the trade-offs and strengths of each library, senior engineers and architects can make informed technology decisions that accelerate development, enhance performance, and ensure long-term maintainability.