Georg's Log - linux

Better Datamash Build Story with Meson

2026-03-29T18:00:00+02:00

This article is a small case study on introducing the Meson build tool into a legacy Autotools and Gnulib centric code base, i.e. the GNU Datamash project. With Meson, the result builds twice as fast and configures one order of magnitude faster while only requiring a few hundred lines of Meson build files. As as side-effect of this experiment the size of the executables is halved.

On Datamash
On Meson
Motivation
Configure
Build
Code Size
On Gnulib
Source Overhead
Test Suite
Side Effects
See also
Conclusion

On Datamash¶

Datamash is a command line program for running various statistics on tabular data, such as computing the count, average or median of a column.

Its sweet-spot seems to be ad-hoc analyses on the command line, on systems where installing a more heavy-weight alternative such as R or DuckDB would be too inconvenient.

In the following comparisons, I compare Datamash 1.9, released in the first half of 2025, against my datamash 'main' branch which has Meson support and a few other fixes and improvements added, and is based on the end of 2025 upstream 'master' branch, i.e. release 1.9 plus just a few minor upstream commits.

On Meson¶

Meson is modern build tool that is an alternative to autotools and CMake. See also my 2021 summary of meson.

Motivation¶

The Meson design improves on pain points many users often have with Autotools, such as:

configure wasting time checking for tons of things that are true everywhere since decades (and doing so single-threaded)
build wasting time with inefficient auto-generated recursive makefiles
layers of m4 and shell scripting are involved which results in a big ball of mud that easily breaks and is hard to fix or work on, in general
all the Autotools generated project specific code being hard to review and being an invitation for supply chain attackers such as Jia Tan

The idea with Meson is that it eliminates all these issues, in particular that it speeds up the build, makes the project specific build files easier to review, reduces maintenance efforts and simplifies working with the build system, in general.

Configure¶

Using the Datamash 1.9 release, running ./configure takes 13 s or so on a modern laptop (12th Gen Intel CPU, NVMe storage, Fedora 43). That configure triggers 488 checks which seems excessive.

Some things are checked redundantly which autoconf tries to mitigate via caching the results, such as checks for round():

checking whether round is declared... yes
checking whether round works... yes
checking whether roundl is declared... yes
checking whether round is declared... (cached) yes
checking whether round works... (cached) yes
checking whether roundl is declared... (cached) yes

For some reason, other checks are invoked up to two times:

checking for stdint.h... yes
checking whether stdint.h conforms to C99... yes
checking whether stdint.h works without ISO C predefines... yes
checking whether stdint.h has UINTMAX_WIDTH etc.... yes
checking for stdint.h... yes
checking for stdint.h... (cached) yes
checking for stdint.h... (cached) yes

In total there are 30 cached checks.

Some more examples of surprising checks:

checking whether strdup is declared... yes
checking whether strnlen is declared... yes
checking for strnlen... yes
checking for working strnlen... yes
checking for strtold... yes
checking for strtoumax... yes
checking whether strtoumax is declared... yes
checking whether strtod obeys C99... yes
checking whether strtold obeys POSIX... yes
checking for strtoll... yes
checking whether strtoll works... yes
checking for strtoull... yes
checking whether strtoull works... yes

These are covered since either C89, C99, POSIX.1.-2001 or POSIX.1-2008. I mean, how realistic is it that there are actually Datamash users who want to install the latests greatest datamash, but run such a broken and vulnerable operating system that lacks support for theses decades old standards.

Sure, retrocomputing can be fun, but isn't the point of retrocomputing to run old software?

In comparison, the Meson checks concentrate on the essentials such that the setup finishes in 0.5 s or so! Meaning an order of magnitude faster. Also, the output volume is lower and thus more useful:

$ meson setup --buildtype=debugoptimized ..
The Meson build system
Version: 1.8.5
Source dir: /home/juser/del/datamash
Build dir: /home/juser/del/datamash/build
Build type: native build
Project name: datamash
Project version: undefined
C compiler for the host machine: cc (gcc 15.2.1 "cc (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)")
C linker for the host machine: cc ld.bfd 2.45.1-1
Host machine cpu family: x86_64
Host machine cpu: x86_64
Library m found: YES
Found pkg-config: YES (/usr/bin/pkg-config) 2.3.0
Run-time dependency nettle found: YES 3.10.1
Compiler for C supports function attribute alloc_size: YES
Compiler for C supports function attribute cold: YES
Compiler for C supports function attribute const: YES
Fetching value of define "__has_c_attribute(fallthrough)" : 202311
Compiler for C supports function attribute format: YES
Compiler for C supports function attribute malloc: YES
Compiler for C supports function attribute warn_unused_result: YES
Compiler for C supports function attribute pure: YES
Compiler for C supports function attribute returns_nonnull: YES
Compiler for C supports function attribute sentinel: YES
Checking for function "strtoumax" : YES
Configuring config.h using configuration
Program ./tests/datamash-tests.pl found: YES (/home/juser/del/datamash/./tests/datamash-tests.pl)
[..]
Program ./tests/decorate-sort-tests.pl found: YES (/usr/bin/env perl /home/juser/del/datamash/./tests/decorate-sort-tests.pl)
Program perl found: YES (/usr/bin/perl)
Program sh found: YES (/usr/bin/sh)
Program msgfmt found: YES (/usr/bin/msgfmt)
Program msginit found: YES (/usr/bin/msginit)
Program msgmerge found: YES (/usr/bin/msgmerge)
Program xgettext found: YES (/usr/bin/xgettext)
Program help2man found: YES (/usr/bin/help2man)
Configuring version.texi using configuration
Program makeinfo found: YES (/usr/bin/makeinfo)
Build targets in project: 22

The check for strtoumax is included as an example how easy it's to implement such a check in Meson, if you really need it.

Build¶

Allowing the make to run on 2 cores in parallel, the datamash autotools build finishes in 4 s or so.

Looking at the build messages, there are a few oddities:

gcc -DLOCALEDIR=\"/home/juser/local/datamash-1.9/share/locale\" -DHAVE_CONFIG_H -I.
    -Ilib -I./lib -Isrc -I./src  -Wall -Wextra -Wformat-security -Wswitch-enum
    -Wswitch-default -Wunused-parameter -Wfloat-equal -fdiagnostics-show-option
    -funit-at-a-time -Wmissing-format-attribute -Wstrict-overflow -Wsuggest-attribute=const
    -Wsuggest-attribute=pure   
    -g -O2 -MT src/datamash-column-headers.o -MD -MP -MF src/.deps/datamash-column-headers.Tpo
    -c -o src/datamash-column-headers.o `test -f 'src/column-headers.c' || echo './'`src/column-headers.c

There is an extra shell invocation for each translation unit, which seems to be pointless, as you either end up with

gcc ... -c -o src/foo.o src/foo.c

gcc ... -c -o src/foo.o ./src/foo.c

There is also some redundancy in linking to -lm:

gcc -Wall -Wextra -Wformat-security -Wswitch-enum -Wswitch-default -Wunused-parameter
    -Wfloat-equal -fdiagnostics-show-option -funit-at-a-time -Wmissing-format-attribute
    -Wstrict-overflow -Wsuggest-attribute=const -Wsuggest-attribute=pure
    -g -O2   -o datamash
    src/datamash-text-options.o src/datamash-utils.o src/datamash-randutils.o src/datamash-text-lines.o
    src/datamash-column-headers.o src/datamash-op-defs.o src/datamash-op-scanner.o
    src/datamash-op-parser.o src/datamash-field-ops.o src/datamash-crosstab.o
    src/datamash-double-format.o src/datamash-datamash.o lib/libdatamash.a
    -lm -lm -lm -lm -lm -lm -lm -lm              -lm -lm   -lm -lm -lm -lm -lm -lm

Meaning for unknown reasons Autotools links 16 times to libm. Arguably, this is a good example of how even Autotools fans struggle with its complexity and are overwhelmed by it.

In comparison, Meson builds everything in 1.5 s or so (i.e. more than twice as fast) and its default output is more reasonable. That means by default the user isn't spammed with overly long and cryptic low-level command invocations, but useful high-level progress:

$ ninja -j2 
[1/50] Compiling C object datamash.p/src_column-headers.c.o
[2/50] Compiling C object datamash.p/src_crosstab.c.o
[..]
[15/50] Compiling C object datamash.p/_usr_share_gnulib_lib_hashcode-mem.c.o
[16/50] Compiling C object datamash.p/_usr_share_gnulib_lib_hashcode-string2.c.o
[..]
[31/50] Linking target datamash
[32/50] Building translation po/da/LC_MESSAGES/datamash-da.mo
[33/50] Building translation po/de/LC_MESSAGES/datamash-de.mo
[34/50] Compiling C object decorate.p/_usr_share_gnulib_lib_xmalloc.c.o
[35/50] Building translation po/eo/LC_MESSAGES/datamash-eo.mo
[..]
[40/50] Linking target decorate
[48/50] Generating datamash manual with a custom command (wrapped by meson to set env)
[49/50] Generating decorate manual with a custom command (wrapped by meson to set env)
[50/50] Generating gen-info with a custom command

Of course, when a command errors out the full command line and error message are presented to the user and alternatively, one can run the build in verbose mode by adding -v to the ninja command.

The verbose mode doesn't slow down the build noticeable. Enabling it shows that meson doesn't have the same issues with linking libm:

cc  -o datamash
    datamash.p/src_column-headers.c.o datamash.p/src_crosstab.c.o datamash.p/src_datamash.c.o
    datamash.p/src_double-format.c.o datamash.p/src_field-ops.c.o datamash.p/src_op-defs.c.o
    datamash.p/src_op-parser.c.o datamash.p/src_op-scanner.c.o datamash.p/src_randutils.c.o
    datamash.p/src_text-lines.c.o datamash.p/src_text-options.c.o datamash.p/src_utils.c.o
    datamash.p/modern_system.c.o datamash.p/_usr_share_gnulib_lib_exitfail.c.o
    datamash.p/_usr_share_gnulib_lib_hash.c.o datamash.p/_usr_share_gnulib_lib_hashcode-mem.c.o
    datamash.p/_usr_share_gnulib_lib_hashcode-string2.c.o datamash.p/_usr_share_gnulib_lib_linebuffer.c.o
    datamash.p/_usr_share_gnulib_lib_next-prime.c.o datamash.p/_usr_share_gnulib_lib_version-etc.c.o
    datamash.p/_usr_share_gnulib_lib_xalloc-die.c.o datamash.p/_usr_share_gnulib_lib_xmalloc.c.o
    -Wl,--as-needed -Wl,--no-undefined -Wl,--start-group -lm /usr/lib64/libnettle.so -Wl,--end-group

In contrast to the Autotools status quo, I decided against creating a static library of the Gnulib parts, since the effort doesn't pay off, because it's only used by two executables. Instead I link only those Gnulib translation units directly which are essential for each binary.

Code Size¶

Comparing the sizes of executables in the Datamash 1.9 vs. Meson branch shows substantial savings with Meson (all sizes in bytes, via size -G):

branch	filename	text	data	bss	total
1.9	datamash	97370	42120	968	140458
meson	datamash	46736	35756	640	83132
1.9	decorate	27348	22068	1576	50992
meson	decorate	15520	16224	1248	32992

That means that the code size of the datamash executable is reduced by over 50 percent.

These space savings are due to only linking those parts of Gnulib that are essential, i.e. Datamash 1.9 links the following 120 Gnulib translation units:

af_alg.c                        hash-pjw-bare.c                 stat-time.c
arpa_inet.c                     hash-pjw.c                      stdlib.c
asnprintf.c                     ialloc.c                        striconv.c
base64.c                        imaxtostr.c                     stripslash.c
basename.c                      inttostr.c                      strnlen1.c
basename-lgpl.c                 linebuffer.c                    sys_socket.c
bitrotate.c                     localcharset.c                  trim.c
c32isalnum.c                    localename.c                    u64.c
c32isalpha.c                    localename-environ.c            uinttostr.c
c32isblank.c                    localename-table.c              umaxtostr.c
c32iscntrl.c                    localename-unsafe.c             unicase/tolower.c
c32isdigit.c                    malloca.c                       unictype/ctype_alnum.c
c32isgraph.c                    math.c                          unictype/ctype_alpha.c
c32islower.c                    mbchar.c                        unictype/ctype_blank.c
c32isprint.c                    mbrtoc32.c                      unictype/ctype_cntrl.c
c32ispunct.c                    mbrtowc.c                       unictype/ctype_digit.c
c32isspace.c                    mbslen.c                        unictype/ctype_graph.c
c32isupper.c                    mbsstr.c                        unictype/ctype_lower.c
c32isxdigit.c                   mbszero.c                       unictype/ctype_print.c
c32tolower.c                    mbuiter.c                       unictype/ctype_punct.c
c32width.c                      mbuiterf.c                      unictype/ctype_space.c
c-ctype.c                       md5.c                           unictype/ctype_upper.c
cloexec.c                       md5-stream.c                    unictype/ctype_xdigit.c
closeout.c                      offtostr.c                      unistd.c
close-stream.c                  printf-args.c                   unistr/u8-mbtoucr.c
c-strcasecmp.c                  printf-parse.c                  unistr/u8-uctomb-aux.c
dirname.c                       progname.c                      unistr/u8-uctomb.c
dirname-lgpl.c                  propername.c                    uniwidth/width.c
exitfail.c                      quotearg.c                      vasnprintf.c
fcntl.c                         reallocarray.c                  version-etc.c
fd-hook.c                       realloc.c                       vsnzprintf.c
fpurge.c                        setlocale_null.c                wctype-h.c
freading.c                      setlocale_null-unlocked.c       xalloc-die.c
getlocalename_l-unsafe.c        sha1.c                          xmalloc.c
getprogname.c                   sha1-stream.c                   xsize.c
glthread/lock.c                 sha256.c                        xstriconv.c
glthread/once.c                 sha256-stream.c                 xstrtol.c
glthread/threadlib.c            sha512.c                        xstrtol-error.c
hard-locale.c                   sha512-stream.c                 xstrtoul.c
hash.c                          sh-quote.c                      xstrtoumax.c

While the Meson branch only links the following 9 Gnulib translation units:

exitfail.c
hash.c
hashcode-mem.c
hashcode-string2.c
linebuffer.c
next-prime.c
version-etc.c
xalloc-die.c
xmalloc.c

The following measures make this reduction possible:

Identifying Gnulib functionality that is widely available in the standard library or in quasi-standard libraries and use that instead of Gnulib.
Removal of accidental code bloat.

For example, after fixing the percentile computation, I was able to replace the separate median implementation with an alias that saves 200 bytes or so in code size.

Since libnettle is widely available and often pre-installed as core system dependency, it suggests itself for the cryptographic hash and base64 computations in datamash instead of vendoring in the Gnulib versions. Switching to libnettle is a moderate source code change but saves several kilobytes in the executable.

Another source of Gnulib over-usage is quoting, i.e. after replacing a popen invocation with an idiomatic direct fork and exec sequence, we can get rid of Gnulib's sh-quote.c translation unit and its dependency hell. Similarly, to me a minimal quoting implementation in the datamash decorate command is preferable as it's more size efficient and easier to review.

One more code bloat source is how Datamash 1.9 quotes trivial arguments in error messages, i.e. via a Gnulib function that quotes in a localized fashion. I don't think that this is worth it and thus switched the branch to a more minimal version.

In another example, I changed the code into invoking the standardized strtoumax instead of using the closely named but non-conforming Gnulib version.

Besides code sizes, Gnulib versions are always suspect to being less battle-tested and optimized, and when bugs are fixed in them, it's easy to miss updates, i.e. in contrast to a shared system library, as Gnulib is designed to be vendored.

Of course, the code changes aren't really Meson specific, but I argue that the way I integrated the Gnulib dependencies into to Meson build file simplified those changes vs. having to hack the autotools Gnulib integration to link only the essential translation units.

On Gnulib¶

Gnulib describes itself as source-code library and a user is supposed to vendor the gnulib parts one needs into a project (i.e. copy and bundle gnulib source code files with his/her project).

Although the Gnulib manual has a few paragraphs on its philosophy and design, it doesn't give any rationale for that odd design decision. It just states:

Classical libraries are installed as binary object code. Gnulib is different: It is used as a source code library. Each package that uses Gnulib thus ships with part of the Gnulib source code. The used portion of Gnulib is tailored to the package: A build tool, called gnulib-tool, is provided that copies a tailored subset of Gnulib into the package.

In my opinion Gnulib is different here for now good reason. There is no reason why Gnulib couldn't provide the same functionality, i.e. the portability wrapper and glue code functions in a secure, reliable and structured way, as a normal shared library, like - say - libbsd.

Even worse, this introduces several disadvantages:

Similar to all the generated build file cruft Autotools projects come with, Gnulib code from some arbitrary version dumped into a lib/ sub-directory is just another great opportunity for a supply chain attacker to hide malicious code.
As with static linking, and in contrast to shared linking, updates to library require rebuilding all users.
Worse than with static linking, since the code is bundled, the dependency isn't obvious, and thus easy to miss. For example, it cannot be searched for with a distribution's package manager.
Bundling Gnulib multiplies reviewing efforts, since with each copy it has to be checked whether it can be traced to the upstream Gnulib repository or contains malicious modifications.

Curiously, when it comes to supply chain security Gnulib tries to talk down other language package ecosystems, while completely ignoring it's own fundamental issues:

Many programming languages nowadays have an ecosystem of reusable source code packages, available through a central site, together with a tool that downloads dependencies from this central site. [.. Python example ..] Most of them are vulnerable to supply chain attacks. [..] While some mitigations exist, they are often cumbersome to put in place. [..] Gnulib is not vulnerable to such attacks, because all of its code is managed in a single repository, with a limited set of committers and with established code review practices.

This misses the point very hard.

If you take their Python example, the stuff Gnulib provides is part of the Python standard library, since it exactly fits Python's batteries included philosophy. Thus, there is really no need to install some random package via pip for basic functionality, as the Gnulib team is trying to insinuate here.

Having a limited set of comitters and 'established code review practices' transports almost zero information. Many projects could claim that. Even the xz project had a limited set of comitters.

The main supply chain risk the Gnulib team should worry about is the vendoring of their 'library'. Instead of encouraging developers to bundle their questionable wrappers, where better standardized wrappers or common libraries exist for decades, Gnulib could look into releasing a proper shared library.

Source Overhead¶

Another important metric is the size of the build system in a project. The more files and more lines of code the higher the efforts for review, maintenance and debugging when something goes wrong.

With Datamash 1.9 the Autotools build system is quite large. There are at least:

$ wc -l aclocal.m4 cfg.mk config.in  configure  configure.ac GNUmakefile init.cfg maint.mk Makefile.am Makefile.in    
   1926 aclocal.m4
    193 cfg.mk
   3047 config.in
  56440 configure
    229 configure.ac
    130 GNUmakefile
     79 init.cfg
   1950 maint.mk
    347 Makefile.am
  12206 Makefile.in
  76547 total
$ du -ch aclocal.m4 cfg.mk config.in  configure  configure.ac GNUmakefile init.cfg maint.mk Makefile.am Makefile.in  | tail -1
2.5M    total

The configure is auto-generated from other files and part of the release source archive to allow users compilation without having to install autoconf.

After running ./configure another large Makefile is generated:

$ wc -l Makefile
12206 Makefile
$ du -h Makefile
800K    Makefile

For comparison, the actual Datamash Source Code is less than 400 kilobytes large:

$ du -h src
316K    src

In addition, the vendored Gnulib sources come with over 30 thousand lines (or over 1 megabyte) of m4 macros and over 4 MiB of bundled C source code:

$ wc -l m4/* | tail -1
  33272 total
$ du -h m4
1.7M    m4
$ du -h lib
4.3M    lib

When building from the datamash git repository, one also has to consider at least just another largish shell script:

$ wc -l bootstrap*
 1087 bootstrap
  177 bootstrap.conf
 1264 total

This is for generating Autotools files which aren't part of the repository, such as the large configure script. It also updates the Gnulib bundle and downloads translation files. Again this is another good target for a supply chain attacker to hide malicious code. Often such boilerplate code is just copy and pasted from some upstream and doesn't need to be customized, but having to review that, having to check whether it was modified, if it was whether the modification is legitimate, is quite wasteful, i.e. busy work that isn't necessary when using a better build tool.

Having all those bits and pieces bundled also increases the risk only updating some of it and thus ending up with a collection of moving parts and state that is very likely to be untested.

In comparison, with the Meson branch there is just:

$ wc -l meson.build meson.options po/meson.build 
  404 meson.build
    6 meson.options
    5 po/meson.build
  415 total
$ du -ch meson.build meson.options po/meson.build | tail -1
28K     total

The 404 lines meson.build is structured like this:

#lines	category
56	comments
55	empty lines
91	configuration data macro settings
53	test suite definitions
149	main build target definitions

With Meson the megabytes of Autotools bloat listed above can be dropped. Since the Meson build file uses a system-wide installed Gnulib, a few more megabytes can be dropped, removing the bundled copy. Removing those files would reduce the size of the release .tar.gz archive by almost 50 %, i.e. from 2.6 MiB to 1.4 MiB or so.

Test Suite¶

Datamash comes with a test suite, i.e. a bunch of (mostly Perl) scripts that invoke the datamash executables with various test inputs.

With Autotools a test suite invocation runs like this:

$ make check
if test -d ./.git                               \
        && git --version >/dev/null 2>&1; then                  \
  cd . &&                                               \
  git submodule --quiet foreach                                 \
      'test "$(git rev-parse "$sha1")"                  \
          = "$(git merge-base origin "$sha1")"'         \
    || { echo 'maint.mk: found non-public submodule commit' >&2;        \
         exit 1; };                                             \
else                                                            \
  : ;                                                           \
fi
make  check-recursive
make[1]: Entering directory '/home/juser/del/datamash-1.9'
Making check in po
make[2]: Entering directory '/home/juser/del/datamash-1.9/po'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/juser/del/datamash-1.9/po'
make[2]: Entering directory '/home/juser/del/datamash-1.9'
make  check-TESTS
make[3]: Entering directory '/home/juser/del/datamash-1.9'
make[4]: Entering directory '/home/juser/del/datamash-1.9'
PASS: tests/datamash-show-env.sh
PASS: tests/datamash-tests.pl
PASS: tests/datamash-tests-deprecated.pl
PASS: tests/datamash-tests-2.pl
[..]
PASS: tests/decorate-sort-tests.pl
============================================================================
Testsuite summary for GNU datamash 1.9
============================================================================
# TOTAL: 30
# PASS:  28
# SKIP:  2
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[4]: Leaving directory '/home/juser/del/datamash-1.9'
make[3]: Leaving directory '/home/juser/del/datamash-1.9'
make[2]: Leaving directory '/home/juser/del/datamash-1.9'
make[1]: Leaving directory '/home/juser/del/datamash-1.9'

As it's common with Autotools, there is a lot of extra noise and echoing that distracts from the important information.

In comparison the Meson output is more to the point and useful:

$ meson test                                                 
ninja: Entering directory `/home/juser/del/datamash/build'
ninja: no work to do.
 1/25 deprecated_test                     OK              0.08s
 2/25 tests2                              OK              0.40s
 3/25 deprecated2_test                    OK              0.07s
 4/25 tests                               OK              0.66s
 5/25 parser_test                         OK              0.20s
 6/25 md5_test                            OK              0.04s
[..]
25/25 deco_sort                           OK              0.15s

Ok:                25  
Fail:              0   

Full log written to /home/juser/del/datamash/build/meson-logs/testlog.txt

Side Effects¶

As a side effect of introducing Meson, a few other things are now possible or much easier than with Autotools.

In particular, the first-class support of out-of-tree builds simplifies having multiple build trees around for different target configurations, e.g. one with release optimization enabled, one with special debugging flags etc.

Also, Meson supports out-of-the-box complicated build modes such as LTO and instrumentalization via simple setup flags.

Nowadays Autotools supports out-of-tree builds, as well, but usually this isn't documented well and for each project that uses Autotools there is a risk that it doesn't support this mode fully. Similarly, passing LTO and instrumentalization flags is also possible with Autotools, but arguably this is more tedious and error-prone.

Conclusion¶

Porting Datamash from Autotools to Meson required relatively little effort that quickly pays off, since build setup is sped up over ten times, build time is sped up two times and the executable sizes are halved, while the Meson build configuration is simpler and more compact and thus easier to review and maintain.

It shows that Meson already was adopted by many open source projects, because Meson support for configuration macros and tests is good. Last but not least, it helps that Meson is better documented than Autotools.

The curious case of nanosleep vs. hr_sleep

2024-07-09T19:00:00+02:00

This weekend, I finally got around reading a bookmarked paper that proposes an improved nanosleep:

Faltelli, Marco & Belocchi, Giacomo & Quaglia, Francesco & Pontarelli, Salvatore & Bianchi, Giuseppe. (2021). Metronome: adaptive and precise intermittent packet retrieval in DPDK.

In the following I document a few notes.

The Papers¶

At page 4 I'm asking myself whether the authors ever posted their code to the LKML. They did and the paper receives devastating criticism from Thomas Gleixner, the Linux kernel maintainer of the affected subsystem and a well-known expert in realtime computing in general and high resolution timers in particular:

Thomas Gleixner. Re: [PATCH] kernel/time: Feedback reply for hr_sleep syscall, a fine-grained sleep service (2021-04-08) (mirror)

Turns out I was reading the first version of a preprint and meanwhile version 3 was released after that LKML post. That version apparently corrects a few of the serious flaws pointed out by Thomas Gleixner, but:

the paper's authors don't reply on the LKML to the posted criticism
they don't acknowledge Thomas Gleixner for his review in the acknowledgements section in the new version of their paper, although they certainly profited from his explanations and independent tests
from the revised preprint (3rd version) the authors reference their old and identically named conference paper for more details, although it also contains the criticized flaws that were removed from the 3rd preprint revision
the ACM conference paper page doesn't mention any corrections
the paper's accompanying github repository doesn't reference any of the criticism

Also, in the 3rd preprint revision, as of 2024-07-09 the latest version, Section 3 A now has some new issues:

'This factor can be controlled using the prctl() system call, putting it to the minimal value of 1.'
Thus, the reader is left wondering what unit the minimal value might have ... (yes, it's nanoseconds).
'These data have been collected by running the thread issuing the sleep request as a classical SCHED_OTHER (normal) priority thread and—as hinted before—with the timer slack of nanosleep() set to 1µs.'
Either they mixed up the units and they actually used the minimal value of 1 ns or they used an unnecessarily high timer slack value to make their proposed improvement look a bit better.

The Good¶

Scrutinizing the API of syscalls that are parametrized with timespec struct is certainly a valuable contribution.

Passing a user pointer into a syscall that needs to be dereferenced in the kernel (over the user space and kernel address space boundary) has some overhead.

Also, such an indirection makes tracing it and debugging somewhat harder.

Looking at the Linux sycall calling conventions, it's clear that there are more than enough call argument registers available to pass all of struct timespec's fields directly via registers. Also, in the case of nanosleep, which optionally returns the remaining waiting time via a second argument pointer, at least some architectures such as x86-64 would allow to return these values via registers, as well. (on x86-64, syscall return values are put into register rax and optionally register rdx which are both 64 bit wide).

Hence, it isn't even necessary to limit the sleep time specification to a single nanoseconds argument like hr_sleep() does. FWIW, on 64 architectures, the Linux kernel has no problem to convert up to 2^63-1 nanoseconds into its internal ktime_t and for all practical purposes a maximum sleep time of up to 292 years or so seems to be sufficient. Also FWIW, when using POSIX API such as nanosleep() the valid range of timespec::tv_nsec is 'just' [0, 999999999].

However, measuring the likely tiny overhead of copying a timespec struct from user to kernel space requires careful and rigorous work.

Of course, it's highly questionable whether eliminating the expected timespec user to kernel copy overhead really would justify adding another syscall.

Addendum¶

Apparently, the hr_sleep() authors initially contacted the kernel community only after their conference paper was published. (i.e. first LKML post on 2021-01-15, conference paper was published 2020-11-24 and presented at the conference in the first week of December, 2020) At that time it received some feedback from another kernel developer, Andy Lutomirski. Like with the second posting, the hr_sleep authors didn't bother to reply directly to any of the issues mentioned in the review.

It seems that the paper was submitted to a proper conference and hence it was peer reviewed, but perhaps the academic reviewers weren't sufficiently familiar with the relevant parts of the Linux kernel and thus missed the issues that were discussed on the LKML.

On 2024-07-07, I opened an issue (archive) on the hr_sleep's github repository, asking for clarification.

It was deleted quickly without any comment:

This issue has been deleted.

For the sake of completeness I'm reproducing my github post below:

Clarifications nanosleep vs. hr_sleep measurements #5

Reading through Section 3 A of your preprint (3rd revision) I noticed a few issues:

'This factor can be controlled using the prctl() system call, putting it to the minimal value of 1.' Perhaps you want to add the unit of that minimal value.

'These data have been collected by running the thread issuing the sleep request as a classical SCHED_OTHER (normal) priority thread and—as hinted before—with the timer slack of nanosleep() set to 1µs.' Please clarify, did you really set the timer slack to 1 µs? Or did you set it to its minimal value, i.e. 1 ns?

'We remand the reader to [14] for an extended evaluation of this implementation.In [..]' Firstly, there is a space missing after the full stop. Secondly, why do you continue to refer to an evaluation whose flaws were pointed out to you by the maintainer of the relevant Linux subsystems while apparently you tried to address some of them in that 3rd version of your preprint?

'Figure 1' You don't mention any details regarding the system you measured those latencies on. Thus, the results are hard to reproduce. The text states 'The tests have been conducted on an isolated NUMA node equipped with Intel Xeon Silver 2.1 GHz cores. The server is running Linux kernel 5.4.' But why don't you mention the exact CPU model and exact kernel version? Also relevant but completely missing: whether relevant kernel parameters where supplied, such as turning some/all kernel CPU bug mitigations off, whether Hyperthreading was enabled, whether frequency scaling, energy saving modes, turbo boost, etc. etc. where enabled ...

BTW, google searches still return the ResearchGate page for the first preprint version at relatively high rank for me - perhaps you want to remove it from ResearchGate to avoid further confusion.

Also, the ACM page of the conference paper version doesn't contain any hints regarding regarding published corrections.

Similarly, in the README of this repository you still link to your conference paper without mentioning that the analysis of hr_sleep vs. nanosleep therein is deeply flawed.

Finally, since you profited from the LKML review of your paper and apparently that review prompted a significant rework of Section 3 A in the 3rd revision of your preprint, why didn't you acknowledge that Linux kernel maintainer in the Acknowledgement Section in your 3rd revision?

On the Costs of Syscalls

2021-08-30T15:00:00+02:00

It's well known that syscalls are expensive. And that software mitigations against CPU bugs (such as Meltdown) even have made them more expensive. But how expensive are they really? To begin to answer this question I wrote a small micro-benchmark in order to measure the minimal costs of a syscall. Meaning the cost of syscalls one always has to pay whether a context-switch happens or not, even when the work in the kernel is minuscule, i.e. the costs of switching from user-mode to kernel-mode and back.

Methods¶

The user-kernel mode-switch micro-benchmark uses Google's benchmark library for the measurements and is available in a git repository. The repository also contains some helper scripts, e.g. a playbook for distributing it to and executing it on a bunch of hosts. The benchmark library repeats each case until the result is considered stable and the playbook allows for repeated executions of the test cases. In the following sections the median value of 100 repetitions is reported (real time in nanoseconds).

For the benchmark a bunch of syscalls is called that are expected to be very cheap, such as getting the user id (UID), the program id (PID), closing an invalid file descriptor, calling an non-existent syscall etc. Thus, a measurement should really just include two mode switches. As controls, a few cases don't call a syscall but do other cheap stuff.

I ran the benchmark on a heterogeneous set of hosts, i.e. different kernels, operating systems and configurations. For more details see also the Hosts Section.

Results and Discussion¶

The following table shows the real time (ns) for the different cases:

host	5i4250u	7i6600u	ac3758	x2643	x2667h	x2667s	x2687w	x2689	x2690	xg6144	xg6148	xg6246	xg6256	xg6256b	xs4110
name
assign	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
clock_gettime	23	21	31	19	16	16	17	14	21	14	24	14	13	13	24
clock_gettime_mono	23	22	32	16	16	16	17	14	21	15	24	14	13	13	24
clock_gettime_mono_raw	23	22	33	542	544	350	582	332	762	660	427	274	122	290	218
clock_gettime_tai	23	21	32	542	546	352	587	333	762	660	427	274	122	292	218
close	568	262	275	484	495	283	514	277	668	610	356	243	93	257	145
getpid	558	257	255	2	1	1	2	1	2	1	2	1	1	1	2
getuid	560	231	259	464	473	276	505	259	649	592	347	224	78	239	137
nothing	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
pthread_cond_signal	3	2	6	14	13	12	14	11	15	10	17	10	10	10	17
sched_yield	706	374	430	569	560	414	634	346	773	694	454	280	126	300	232
sqrt	6	2	15	4	4	4	4	1	7	1	3	1	1	1	3
sqrtrec	4	4	15	2	3	2	3	2	3	3	5	3	3	3	5
syscall	560	252	265	440	460	269	497	243	620	579	345	221	76	233	136

Controls¶

The cases used as controls are 'nothing' which literally does nothing, 'assign' which just assigns to a variable, sqrt which computes the square root of a small constant and sqrtrec which stacks a bunch of sqrt calls. The results for these are plausible, i.e. doing nothing is really measured as 10**-7 ns or so, the assignment costs 0.5 ns or so and computing the square root takes only a few ns. Perhaps the most remarkable result is, that computing the square root on a Atom CPU (ac3758) is pretty constant over the 2 cases, whereas on the other hosts its runtime depends on its argument.

Clock Gettime¶

Looking at the syscalls, one relation that holds true on all hosts is that the clock_gettime(CLOCK_REALTIME) syscall is much faster than getuid() or close(). This can be explained by the fact that on Linux, clock_gettime(CLOCK_REALTIME) and a few other syscalls are implemented via the efficient vDSO mechanism. Meaning when they are called no mode switch happens!

clock_gettime() supports different clocks and not all of them are vDSO optimized on all kernels. The table shows that on RHEL 7 querying CLOCK_MONOTIC_RAW and CLOCK_TAI invokes a real syscall while on Fedora 33 kernels (5.12/5.13) these clock readings are also implemented as vDSO.

Dummy Signaling¶

Similarly, the dummy pthread_cond_signal() case, which signals without anybody listening, is much cheaper than a real syscall - since the C library doesn't have to call a real syscall but can just invoke some relatively cheap atomic operation.

Getpid¶

The getpid() syscall is surprisingly fast on RHEL 7. It turns out that RHEL 7 ships an older glibc version which caches the ID of a process! Which arguably is a curious optimization, since, what's the point? I mean how often to you have to call getpid() in a program, really? At some point (around Fedora 26) this feature was removed since it apparently caused more trouble than it's worth. Perhaps unsurprisingly, that removal even broke somebodies workflow.

Real Syscalls¶

So looking at the real syscalls, the user-kernel mode switches cost in the order of a few hundred nanoseconds, on all hosts. The higher costs on some hosts can be explained by CPU bug mitigations being enabled (they are enabled by default) and/or kind of older/lower-end hardware. See also the Hosts Section for some details.

The fastest host is xg6256 which manages to switch modes in less than 100 ns. It has a fast CPU with good single core performance (Xeon Gold 6256), has frequency scaling disabled and runs at a constant 4.1 GHz frequency above it's base frequency (i.e. a frequency between the base and turbo frequency).

Sched Yield¶

The sched_yield() syscall could be considered as a minimal work syscall, e.g. when there is nothing to yield to. Also, the benchmark process is running under the standard scheduling policy and on Linux sched_yield() is described as:

sched_yield() is intended for use with real-time scheduling policies (i.e., SCHED_FIFO or SCHED_RR). Use of sched_yield() with nondeterministic scheduling policies such as SCHED_OTHER is unspecified and very likely means your application design is broken.

So unspecified could mean that the syscall just bails out after the process' scheduling policy compares equal to SCHED_OTHER.

On most hosts sched_yield 150 ns or so more expensive than a really minimal syscall such as getuid() - which indicates some more overhead, but not necessarily a context switch.

Nanosleep¶

A bit out of the competition is the nanosleep() syscall:

host	5i4250u	7i6600u	ac3758	x2643	x2667h	x2667s	x2687w	x2689	x2690	xg6144	xg6148	xg6246	xg6256	xg6256b	xs4110
name
nanosleep0	52632	50474	52620	50588	50003	50011	50000	50014	50312	50018	50000	50014	50000	50000	54866
nanosleep0_slack1	4355	2836	7076	3247	2736	2483	2835	3908	3401	2762	2870	2446	1974	2248	3837
nanosleep1_slack1	4348	2840	7102	3252	2736	2486	2834	3908	3410	2767	2871	2446	1975	2246	3836

Calling nanosleep() to sleep for 0 ns or 1 ns seemingly also is a very cheap syscall or even a null operation.

However, in the first case it takes 50 µs an all hosts. Incidentally, 50 µs is also the default timer slack value for a normally scheduled process on Linux. The timer slack mechanism extends timer expirations up to the slack value in order to group multiple timers since this reduces wake ups and thus saves energy. And since nanosleep() creates a timer, it's also affected by this mechanism.

Thus, the other nanosleep cases set a minimal timer slack of 1 ns which reduces the runtime, as expected. However, it's still much more expensive than the other syscalls. Of course, a timer expiration has a limited accuracy. However, with 0 ns or 1 ns no timer has to be expired, really. It turns out that calling nanosleep() unconditionally yields a (voluntary) context switch. Even on isolated cores, where the scheduler happily switches to the swapper kernel thread. Thus, the last two nanonsleep cases really measure the context switch costs which are more expensive than a simple mode switch.

The costs of a context switch match what others are measuring (modulo division by two).

Hosts¶

The following table shows the hosts under benchmark:

	host	CPU	mitigations	poll	os	kernel
0	5i4250u	Core i5-4250U	yes	yes	Fedora 33	5.12
1	7i6600u	Core i7-6600U	yes	yes	Fedora 33	5.13
2	ac3758	Atom C3758	no	yes	Fedora 33	5.13
3	x2643	Xeon E5-2643 v2	yes	no	RHEL 7	3.10
4	x2667h	Xeon E5-2667 v3	yes	yes	RHEL 7	3.10
5	x2667s	Xeon E5-2667 v3	yes	yes	RHEL 7	3.10
6	x2687w	Xeon E5-2687W v3	yes	yes	RHEL 7	3.10
7	x2689	Xeon E5-2689 v4	yes	yes	RHEL 7	3.10
8	x2690	Xeon E5-2690 0	yes	yes	RHEL 7	3.10
9	xg6144	Xeon Gold 6144	yes	yes	RHEL 7	3.10
10	xg6148	Xeon Gold 6148	no	yes	RHEL 7	3.10
11	xg6246	Xeon Gold 6246	yes	yes	RHEL 7	3.10
12	xg6256	Xeon Gold 6256	no	yes	RHEL 7	3.10
13	xg6256b	Xeon Gold 6256	yes	yes	RHEL 7	3.10
14	xs4110	Xeon Silver 4110	no	yes	RHEL 7	3.10

Notes:

the kernels are the ones packaged by the distributions
most RHEL hosts are on RHEL 7.9
CPU mitigations are disabled via the mitigations=off kernel parameter or similar parameters
polling means that CPU frequency scaling and power saving is disabled via kernel parameters and tuned PM QoS settings
so the host's CPU runs on a fixed frequency; where possible this frequency is set slightly above the base frequency, e.g. on the Xeon Gold 6256 CPU it's set to 4.1 GHz
the Atom CPU doesn't support Hyperthreading and Hyperthreading is disabled on all Xeon hosts
all hosts have SELinux and/or Auditing enabled (on Fedora/RHEL these features are enabled by default) which adds some syscall overhead to some degree

Terminology¶

There are basically two important separate terms to distinguish in the above discussion:

Mode Switch (or Mode Transition)
Context Switch

The definitions of these terms might vary in different literature and different operating systems. Also, in other contexts (no pun intended!) one might describe different modes as different contexts. However, the definitions given in the linked Wikipedia articles are widely used and apply to Linux.

Basically, a mode transition denotes the switch between user mode and kernel mode (or between user space and kernel space) whereas a context switch denotes a switch between different tasks, which is facilitated by the kernel. A context switch requires more work than a mode switch and thus is more expensive.

The Rise of Meson

2021-08-07T13:00:00+02:00

More and more open source projects seem to switch to the Meson build tool, in the last years. In an attempt to quantify this development I looked at the build tools usage by Fedora packages, during the last 14 years.

About Meson¶

Meson is a build tool with good C/C++ support. It's written in Python and generates Ninja build files. Since it was created after CMake (which was initially released in 2000), it had the chance to learn from CMake's mistakes. In comparison to CMake, Meson has better documentation, it doesn't support generating classic makefiles, it only supports out of source tree builds, and its domain specific language is arguably much better designed.

So, from the start, Meson arguably is also more opinionated in its design, but in a useful way. Usually, simple stuff is easy to solve with Meson and there is one kind of obvious way to do it. Whereas with CMake, there are often several ways to implement common build tasks (cf. TMTOWTDI), including some legacy ones. With Meson, build files are often much shorter than equivalent CMake ones.

CMake evolved over the years and it's possible to write - say - a modern style of CMake build files that are less awful, and use it in a modern way to some degree, but it feels that the official documentation doesn't help much in that regard, and that one would have to study several books on modern CMake in order to arrive there.

Fedora Data¶

The first packages that required Meson to build arrived in Fedora 22, i.e. 6 years ago (in 2015, Fedora runs on a half-year release schedule and Meson was initially released in 2013, 8 years ago). Since then the number of packages that build-require Meson grew to almost 600 hundred:

So this graph shows the number of packages that require a notable build tool at build time, for each Fedora release.

Scons never really took off and nowadays only a handful of packages still require it. In contrast, the Meson usage shows some serious growth.

When looking at the Meson growth in relation to the packages that either require CMake, Scons or Meson - Meson is now required by 30 % or so of the packages:

The growth slowed a bit down in the last Fedora releases, it seems.

Other Tools¶

So what about Autoconf, Automake, etc. (a.k.a. Autotools) and other build tools? Since most project that use Autotools release their tar archive files with already generated configure scripts/make files, the resulting packages don't depend on those tools. Plus, before Fedora 34, there isn't even a dependency on GNU make, since it was part of the standard build root environment. This changed with Fedora 34, where 9000 packages or so depend on GNU make. Subtracting the packages that also require CMake, there remain 8000 packages or so that might use Autotools. But how many of those use plain old Makefiles can't be easily derived by just looking at the package dependencies.

In any case, using Autotools for a modern C/C++ project in 2021 is like using CVS for source code version control in 2021: there are better tools available and thus it isn't very interesting to still consider the legacy solutions.

The Boost project has its own build system: BJam. There is some documentation, but it's kind of minimal and cryptic. As of Fedora 34, the is no package that build-requires BJam. (Boost itself doesn't need to explicitly require it since it's included in the Boost release tar archive.) Incidentally, the Boost steering committee announced in 2017 its desire to migrate to CMake, however, as of 2021, the current Boost documentation still just mentions BJam/b2 for building Boost.

Methods¶

The build tool usage stats were obtained with:

{
    echo "fedora_release,tool,users"
    for ((i=7; i<35; ++i)); do
        for k in meson cmake scons; do
            n=$(dnf repoquery --releasever=$i --disablerepo='*' --enablerepo='*-source' \
                      --arch=src --whatrequires $k | sed 's/-[0-9]:[0-9].*$//' \
                    | sort -u  | wc -l)
            echo "$i,$k,$n"
        done
    done
} | tee build-stat.csv

And for GNU make, on Fedora 34:

dnf repoquery --disablerepo='*' --enablerepo='*-source' --arch=src \
      --whatrequires make | sed 's/-[0-9]:[0-9].*$//' \
    | sort -u  | wc -l > f34-require-make.log

Deploying Full Disk Encryption in the Cloud with Ansible

2020-11-17T12:00:00+01:00

Deploying a new virtual machine 'in the cloud' is simple enough and fast. As long as one doesn't need to setup an encrypted root filesystem. This article shows how to automate the setup of a Fedora system with an encrypted root filesystem remotely - 'in the cloud' - on e.g. a DigitalOcean, Hetzner, Linode, etc. VM - with Ansible and a special Fedora rescue system that runs purely from an initramfs archive.

Motivation¶

Having an encrypted root filesystem gives you data at rest protection. At the end of the day, your virtual disk device inside your virtual machine (VM) is just a file on a storage solution where disks are getting replaced and reallocated for other things. After - say - hardware replacements perhaps such a disk even ends up on eBay without being properly deleted and thus some random person now has access to your data. With your root filesystem being encrypted you are protected against such scenarios.

If you think that such a scenario is far fetched, see also for example this 2019 computer magazine article (in German) which describes in detail how some SSDs with very sensitive personal data ended up on eBay.

The Problem¶

Cloud providers such as DigitalOcean, Linode or Hetzner provide a set of pre-built disk images for different operating systems. Users also have the option to add custom images to their account, usually. When deploying a new virtual machine (e.g. a Droplet when using DigitalOcean) the selected disk image is extracted to the new VM guest, the image has to run some first-boot initialization script which fetches the network configuration, creates sshd host keys and stuff like that.

This process is very fast because the image is pre-built and just needs to be copied while the first-boot initialization doesn't have much to do. Compare this with a traditional automated Linux install method such as Kickstart where RPMs are installed one after the other. This takes much longer.

The disadvantage is a loss in flexibility. Especially, this makes setting up the root filesystem on a LUKS encrypted device challenging.

Non-Solutions¶

A naive approach to a remote system with an encrypted root filesystem is to locally install a Linux distribution with disk encryption enabled and upload the resulting image. The problem with this is that if you want to deploy multiple VMs all then have the same LUKS encryption key. Certainly you can't openly share this image with others and you would upload a new image for each new VM deployment. Obviously this is tedious and doesn't scale.

After the VM is deployed with an unencrypted root filesystem, encrypting it on the fly isn't possible, because the root filesystem is in use.

A Solution¶

For some time Linux has the interesting feature to boot a new kernel image from a running Linux system. That means the currently running kernel can be replaced with a new one using the kexec system call.

A standard part of the Linux boot process is to extract an early boot root filesystem image into a ram disk, execute an init process from it that basically mounts the real root filesystem and switches to it. Of course, the final root filesystem switch is optional.

In combination these features, i.e. kexec and an early-boot environment, can be used to bootstrap a fresh Linux installation from scratch - from a running system.

The basic idea here is to kexec a new kernel with an initramfs image that contains a small Linux system, i.e. one that includes enough software to provide network connectivity including SSH and to drive the Fedora install process.

Recently, I implemented this approach, i.e. I thus wrote a script that creates such initramfs images. That means it installs a minimal Fedora system into an initramfs.

I also created an Ansible playbook that drives the whole process. That means it downloads the initramfs image to the VM, creates VM specific configuration, kexecs into the rescue system and finally installs the LUKS-encrypted target system from there. It even supports creating a DigitalOcean Droplet, first.

Without considering the VM creation, the playbook consists of 30 steps or so. After the playbook finishes, the target VM has an encrypted root filesystem with a fresh Fedora system on top of it.

So far, I successfully used this method and playbook on DigitalOcean, Linode and Hetzner VMs and even on some old-school bare-metal root-server (rented from Manitu).

Deploying SSH Host Keys with Digital Ocean Droplets

2020-06-05T22:00:00+02:00

When deploying a VM in a remote location one might end up in an impossible situation: a VM base image doesn't contain SSH host keys as otherwise the image couldn't be shared between multiple hosts. But if SSH host keys are auto-generated during the first boot one doesn't know their fingerprints and thus they can't be trusted on first connect. This article describes how to securely deploy SSH host keys when deploying a Digital Ocean Droplet (i.e. a VM).

Problem Description¶

Pre-build disk images usually don't contain any SSH host keys. At least they shouldn't contain any host keys if they are used for multiple hosts, but sometimes people get this wrong, e.g. Hetzner (a web hoster) in 2015, Cisco (a network equipment company) also in 2015 and many others.

One common approach is to generate the SSH host keys during first boot.

The problem is then: how can we trust this new machine during the first SSH login? We don't know the fingerprint of any SSH host key because they are newly generated, remotely. Thus, we can't really trust it that way.

One approach is to just accept any first encountered host key and store its fingerprint for later verifications (trust on first use). This is better then ignoring the problem completely but this still makes you vulnerable against man-in-the-middle attacks.

Generating a new disk image with new host keys for each new host and transferring it to a cloud provider doesn't scale.

A Solution for Digital Ocean Droplets¶

DigitalOcean (like some other cloud providers) uses cloud-init for the first-boot configuration. And cloud-init also supports supplying SSH host keys via the so called cloud-config. Thus, we can supply a cloud-config (with locally generated host keys) via the user-data parameter when creating a new VM via the DigitalOcean API.

SSH host key problem solved. Almost, since it turns out that the user-data cloud-config is served over a link-local address for the lifetime of the VM. Since this turns the SSH host keys into world-readable information (to processes running on that VM) we have to consider these cloud-init SSH host keys as ephemeral and have to replace them with another set (of locally generated SSH host keys) as part of our deployment.

See also an example Ansible playbook of mine that creates a DigitalOcean Droplet and automates the SSH host key deployment in the described way.

In that way, man-in-the-middle attacks can be successfully averted (and detected) during deployment as the DigitalOcean API is called over HTTPS (where the server's certificate is signed by a trusted CA) and the first ephemeral SSH host key is thus transferred over that secure connection. The second and final SSH host key is transferred over SSH where the (ephemeral) host key can be authenticated by it's known fingerprint.

NFS VM Cached Read Illustrated

2017-08-12T18:12:00+02:00

This article illustrates how the virtual memory (VM) subsystem of the Linux kernel is able to cache files that are located on an NFS server.

Environment
High-Level View
Dstat
NFS Traffic
Closing Remarks

Environment¶

In this small experiment we just need two Linux machines on a local network, where one acts as NFS server and the other one as NFS client. With a current Linux distribution like Fedora 26 you get NFSv4 if you don't do anything special.

The following results are from a 1 GBit ethernet network where the NFS client has 16 GB RAM and both machines run Fedora 26.

High-Level View¶

A simple way to read over the network from an NFS filesystem without yielding disk IO on the client is:

empty the VM cache - e.g. rebooting the client and server or via executing free && sync && echo 3 > /proc/sys/vm/drop_caches && free on both machines
on the client: cat a relatively large file from a NFS filesystem to /dev/null

The file should be large enough such that we can easily measure some metrics but small enough to fit into RAM.

In our environment the 4.2 GB backup copy of the John Woo movie 'The Killer' is well suited:

$ time cat The_Killer.iso > /dev/null

The zsh time builtin reports:

0.00s user 1.08s system 2% cpu 38.373 total

That means it took 38 seconds to transfer that 4.2 GiB file. This is plausible because the resulting read rate of 113 MiB/s is close to the wire speed (10^9/8/1024/1024) minus the expected protocol overhead.

Executing the read command

$ time cat The_Killer.iso > /dev/null

a second and third time is way faster:

0.00s user 0.57s system 99% cpu 0.578 total
0.00s user 0.44s system 98% cpu 0.447 total

Just half a second.

After clearing the VM cache with

# free && sync && echo 3 > /proc/sys/vm/drop_caches && free

where the final free reports

buff/cache
1871568

and reading the file just another time we are back at the initial timing:

0.02s user 1.26s system 3% cpu 38.551 total

A free then reports

buff/cache
6267676

This is plausible as well, i.e. (6267676-1871568)/1024/1024 is close to the 4.2 GB file size.

If we have a rough idea how the VM subsystem works with a local filsystem in a common OS kernel (like Linux) all this isn't surprising. One just might wonder how the kernel decides that it is safe to reuse the VM cache since the file could have changed on the server (e.g. locally or by another NFS client).

We can verify that the kernel is able to detect file changes by other parties via touching the file on the server, e.g.:

$ touch The_Killer.iso

After that, reading the file again on the client takes the initial amount of time (39 seconds).

Dstat¶

When running the dstat command on the client we can obtain a more detailed picture. For example:

$ dstat -d -D total -g -n -N enp0s31f6 -r -c --vm

By default, dstat repeatedly averages over 1 second, i.e. it prints every second a new line.

The dstat output during the first file read:

We can observe a few things in the listing:

the network receive rate matches the computed read rate (modulo protocol overhead) and since reads need to be acknowledged the outgoing network traffic increases as well
directly after starting the read the VM heavily allocates and frees caches pages (in the order of 60 * 10^3 pages or so)

The work the VM subsystem is doing is plausible. The system has a page size of 4 KiB and thus the amount of cache pages is sufficient to deal with the read rate.

The listing for the second read is much shorter:

Here we don't see much network traffic, but we see a clear increase in traffic that could be explained by some NFS communication where the client inquires the server whether the file is outdated or not.

Also, as expected, the VM subsystem isn't busy with allocations and frees as before.

NFS Traffic¶

To get an idea how the NFS client detects file changes we can capture the network traffic and inspect with an analyzer like Wireshark.

In case directly capturing with Wireshark isn't an option, a simple alternative is to capture with tcpdump to a file and open that file later with wireshark, e.g.:

# tcpdump -i enp0s31f6 -s0 -w /var/tmp/nfs.dump

The NFS flow for the second file read is quite compact in the wireshark overview:

As expected, it is NFS version 4 and there are just a few query operations.

In detail the second GETATTR reply:

We observe that the server replies that the file is a regular file (NF4REG) and reports its change-ID and file-ID.

The file-ID is conceptually similar to an inode and thus the NFS client can compare the change-ID with the one obtained from last time to decide if it can reuse the pages from the VM cache or if it has to retrieve the change file from the server.

Sure enough, after touching the file the server sends a new change-ID, i.e. 17002 after the first touch and then 17003 after the second touch. As expected, the file-ID doesn't change.

Why does the client issues two GETTATTR requests? The first is for the directory the file is located in (NF4DIR).

Closing Remarks¶

So when does the VM caching matter? Or, how can accidentally work against this mechanism when using NFS?

Say, entirely hypothetical, you have a compute cluster where typical compute jobs are actually program chains where the output of one program is the input of the next one. Also assume that the working directories are shared via NFS for flexibility and redundancy. For example, when one cluster node crashes in the middle of a chain then the chain can be completed on another node without having to recompute everything.

In that scenario, the best case would be: the complete chain is executed on one cluster node. A job-scheduler only schedules the complete chain to a cluster node. It doesn't re-schedule single programs of a chain between cluster nodes. Then the input for the program next in chain comes from VM subsystem and doesn't need to be transmitted over the network.

In contrast to that, since the work directories are shared, it is also possible to have finer job scheduling granularity, i.e. for each program of a chain. But this would increase the load on the NFS server. In the worst-case the NFS server's outgoing network traffic would multiply because the result of one chain program executed on cluster node A would have to be retrieved from the NFS server by the next chain program executed on cluster node B.

Similar reasoning applies to other network filesystems (and other operating systems with a VM subsystem) and thus it makes sense to build a chain-to-node-affinity mechanism into the job scheduler such that the VM cache re-use is maximized in such a chain-job environment.

Check CPU Microcode Version on Linux

2017-07-01T10:10:00+02:00

Debian recently released a warning advisory regarding a hardware bug in the hyperthreading implementation on certain Skylake/Kaby Lake Intel CPUs. It can be mitigated by a microcode firmware update - at least on some models. This article is about checking a non-Debian system for this issue and verifying if the right microcode is in place.

The below console snippets are from a Dell notebook with Skylake CPU that runs Fedora 25. It turns out that Fedora, in contrast to Debian, installs some firmware packages, by default, including Intel microcode images (cf. microcode_ctl). Thus, a Fedora system on Skylake should already run with the microcode version that includes the fix.

Check for certain CPU features - e.g. if hypterthreading is enabled:

$ grep '\<ht\>' /proc/cpuinfo -q && echo 1
1

Display CPU model, stepping and active microcode version:

$ grep 'stepping\|model\|microcode' /proc/cpuinfo
model       : 78
model name  : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping    : 3
microcode   : 0xba
[..]

The advisory states that Skylake model 78, stepping 3 is affected and that microcode version 0xba includes the fix.

Investigate whether the microcode was loaded by the BIOS or by the kernel:

# journalctl --no-hostname -o short-monotonic --boot -0 \
  | sed -n '1,/PM: Preparing system for sleep/p' | grep 'microcode\|smp'
[    0.000000] microcode: microcode updated early to revision 0xba, \
                   date = 2017-04-09
[    0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[    0.040087] smpboot: Max logical packages: 2
[    0.056402] smpboot: CPU0: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz \
                   (family: 0x6, model: 0x4e, stepping: 0x3)
[    0.056975] smp: Bringing up secondary CPUs ...
[    0.239023] smp: Brought up 1 node, 4 CPUs
[    0.239023] smpboot: Total of 4 processors activated (22476.05 BogoMIPS)
[    0.693589] microcode: sig=0x406e3, pf=0x80, revision=0xba
[    0.693671] microcode: Microcode Update Driver: v2.2.

That means the microcode image in the BIOS is older, thus, the Kernel loaded a newer version (the microcode date also matches the one mentioned in the advisory).

After a BIOS update, check that the BIOS loads the microcode before kernel start:

# journalctl --no-hostname -o short-monotonic --boot -0 \
    | sed -n '1,/PM: Preparing system for sleep/p' | grep 'microcode\|smp'
[    0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[    0.039057] smpboot: Max logical packages: 2
[    0.054709] smpboot: CPU0: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz \
                   (family: 0x6, model: 0x4e, stepping: 0x3)
[    0.055000] smp: Bringing up secondary CPUs ...
[    0.235034] smp: Brought up 1 node, 4 CPUs
[    0.235034] smpboot: Total of 4 processors activated (22560.93 BogoMIPS)
[    0.711458] microcode: sig=0x406e3, pf=0x80, revision=0xba
[    0.711662] microcode: Microcode Update Driver: v2.2.

(there is no microcode udpated early message, anymore)

Modern CPU microcode, like used by Intel CPUs, isn't persistent, i.e. it must be loaded at each boot, either by the BIOS or by the kernel. Apparently, it even has to be reloaded during resume after a suspend-to-RAM (cf. the kernel messages).

Fedora as Grml replacement

2017-03-12T21:00:00+01:00

Grml is a Linux Live CD distribution geared towards system administration tasks and other console work. Unfortunately, its development has slowed down in the past years, i.e. the latest stable release is from 2014. In 2017, when dealing with modern hardware, modern features like Btrfs, SELinux etc. this is a deal breaker. It turns out that Fedora, when installed on a USB mass storage device, is a good substitute for traditional Linux Live CDs like Grml.

How it works¶

A very fast and convenient method to install Fedora on a USB stick is to use virt-builder, e.g.:

$ virt-builder fedora-25 --hostname rescue.example.org  \
--ssh-inject root:file:"$pubkey" --root-password file:"$pwfile" \
--update --install $(paste -d, -s package.list) \
--run guest-setup.sh --selinux-relabel \
-o /dev/sdz

The virt-builder command is probably mostly used for fast generation of virtual machine images but it works as well for bare-metal installations. It is so fast because it starts from filesystem-templates.

In the above example, the shell script guest-setup.sh applies some defaults and other post-installation customizations in the target system.

The result can be easily tested with QEMU/KVM, e.g.:

$ qemu-system-x86_64 -enable-kvm -drive file=/dev/sdz,if=virtio,format=raw \
  -m 2048 -netdev bridge,id=nd0,name=tap0,br=virbr0 \
  -device e1000,netdev=nd0,id=d0  -display curses

The Code¶

I've published some scripts to generate such a Fedora based USB stick in a Git repository. The repository also contains some more details in a README including a detailed comparison with Grml.

Addendum¶

Also, nowadays, the traditional Linux Live CD approach is a little bit outdated. Many systems don't even have a CDROM drive anymore for good reason. Sophisticated distributions like Grml are hybrid, i.e. they can also be booted from a USB stick, but they don't make use of the extra space or the fact that the stick is writable, by default. Grml even supports customizations like adding a writable filesystem but such changes are kind of tedious.

Grml doesn't just come with good defaults (e.g. zsh login shell), the team also has a talent choosing great release names (e.g. Knecht Rootrecht).

Btrfs requires noatime

2017-02-08T21:10:00+01:00

Traditionally, UNIX filesystems also maintain the access time (atime) of a file. This is very much an anti-feature because it yields a write operation for each read operation. Which is obviously bad for performance. Since Btrfs is a copy-on-write (COW) filesystem maintaining atimes is even more painful. Thus, a sensible recommendation is to make sure that every filesystem is mounted with the noatime option. Especially, if it is a Btrfs filesystem.

Btrfs Case Study¶

A typical example of how writing the atime can significantly affect performance.

The symptoms: an rsync job from a Btrfs filesystem (located on an SSD) to an external USB 3 disk drive with XFS runs with ~ 17 MiB/s while iotop and dstat show significantly higher IO values for the source device, e.g. ~ 60 MiB/s.

Reasons: This Btrfs filesystem is mounted with default options, that means that instead of noatime the relatime is active, resulting in atime updates for effectively each file.

In addition to that, the filesystem contains some snapshots, i.e. for all subvolumes there is also one or more snapshots. Creating a filesystem snapshot is very cheap on Btrfs because of its COW design. An atime update of a snapshotted file only updates the atime of one copy. Thus, in the likely case that the filesystem sector with the atime is still shared it has to be copied on that atime write. Meaning 2 or more write operations as a result of one read.

Resolution: After canceling the rsync command, remounting the Btrfs filesystem with noatime and restarting the rsync job it sure enough performs much better, i.e. the numbers reported by rsync match the dstat ones, as expected.

Note that the number of files included in a snapshot is most relevant for this issue, not necessarily the number of snapshots.

If you are really unlucky, running a backup job (or even just a grep) on a Btrfs filesystem mounted without noatime might even yield out-of-space errors in case all the copy-on-write atime updates exceed the available free space.

Relatime¶

Since 2009 or so Linux kernels by default mount filesystems with the relatime option. With this options, the atime is updated somewhat less frequently. That means under the presence of reads, it is updated at least once a day:

relatime
   Update  inode access times relative to modify or change
   time.  Access time is only updated if the previous
   access time was earlier than the current modify or
   change time.  (Similar to noatime, but it doesn't break
   mutt or other applications that need to know  if  a
   file has been read since the last time it was modified.)

   Since  Linux  2.6.30,  the  kernel defaults to the
   behavior provided by this option (unless noatime was
   specified), and the strictatime option is required to
   obtain traditional semantics.  In addition, since Linux
   2.6.30, the file's last access time is always updated
   if it is more than 1 day old.

(mount(8))

Thus, relatime doesn't help much with the principal cause for the described performance issue. You still likely get massive atime update induced writes during bulk read-only activity like backup or search jobs. You only have to experience a timespan greater than 24h were most files aren't read. A pretty standard scenario.

Unfortunately, switching the system default from relatime to noatime isn't possible (as of 2017-02). Thus, one has to remember to always specify noatime. Think mount -o noatime ... and add it to each /etc/fstab entry.

Testing¶

Testing the effect of atime uptime in the presence of snapshots is easier when the Btrfs filesystem is mounted with strictatime. This overrides the relatime default option. The time attributes of a file or directory can be displayed with ls but the GNU utility stat is more convenient for printing all timestamps.

P.S.¶

There are use-cases for an actively maintained file atime. Examples are measuring the usage of installed binaries (cf. Debian's popcontest) or detecting unread mails. But there are better alternatives to implement such tasks and thus those aren't very convincing arguments for enabling atime writing, by default (be it relative or even strict). In conclusion, writing the access time means much pain with little gain.

Georg's Log - linux

Better Datamash Build Story with Meson

On Datamash¶

On Meson¶

Motivation¶

Configure¶

Build¶

Code Size¶

On Gnulib¶

Source Overhead¶

Test Suite¶

Side Effects¶

See also¶

Conclusion¶

The curious case of nanosleep vs. hr_sleep

The Papers¶

The Good¶

Addendum¶

On the Costs of Syscalls

Methods¶

Results and Discussion¶

Controls¶

Clock Gettime¶

Dummy Signaling¶

Getpid¶

Real Syscalls¶

Sched Yield¶

Nanosleep¶

Hosts¶

Terminology¶

The Rise of Meson

About Meson¶

Fedora Data¶

Other Tools¶

Methods¶

Deploying Full Disk Encryption in the Cloud with Ansible

Motivation¶

The Problem¶

Non-Solutions¶

A Solution¶

Deploying SSH Host Keys with Digital Ocean Droplets

Problem Description¶

A Solution for Digital Ocean Droplets¶

NFS VM Cached Read Illustrated

Environment¶

High-Level View¶

Dstat¶

NFS Traffic¶

Closing Remarks¶

Check CPU Microcode Version on Linux

Fedora as Grml replacement

How it works¶

The Code¶

Addendum¶

Btrfs requires noatime

Btrfs Case Study¶

Relatime¶

Testing¶

P.S.¶

See also¶