linux: introduce io_uring support

[bnoordhuis]

Add io_uring support for several asynchronous file operations:

• read, write
• fsync, fdatasync
• stat, fstat, lstat

io_uring is used when the kernel is new enough, otherwise libuv simply
falls back to the thread pool.

Performance looks great; an 8x increase in throughput has been observed.

This work was sponsored by ISC, the Internet Systems Consortium.

[rektide]

Also potentially resolves or progresses #1947.
Also: 🎉! 🚀!

Comment from the peanut gallery here, but, personally I'd be fine with/love/prefer an io_uring that targeted more modern io_urings. My understanding is 5.15 (2021.10.31) had a bunch of pretty sizable improvements to how folks could use io_uring. That's already nearly two years old, and is going to get more and more expectable. Meanwhile folks on older kernels (5.1 is 2019.5.5) still have the very fast existing libuv thread pool implementation to fall back on.

Maybe there's no sacrifice, but "a carefully chosen feature subset" leaves the impression that we're going way out of our way to support already pretty aged kernels. Grain of salt here, but being rip-raring future-forward-facing & adopting every whiz-bang shiny-new-whistle we can would be my pick at this juncture.

[trevnorris]

Did some basic benchmarking, and it's giving me > 8x performance when reading small chunks out of /dev/zero. So great news on that side.

valgrind isn't happy about a few tests. Mostly with the error Syscall param io_uring_enter(sigsz) contains uninitialised byte(s). Simplest repro I could make is this:

#include <uv.h>

void on_read(uv_fs_t *req) { }

int main() {
  uv_fs_t read_req;
  uv_buf_t buffer;
  int fd;
  char bytes[64];

  buffer = uv_buf_init(bytes, 64);
  fd = uv_fs_open(NULL, &read_req, "/dev/zero", UV_FS_O_RDONLY, 0, NULL);
  uv_fs_read(uv_default_loop(), &read_req, fd, &buffer, 1, 0, on_read);
  uv_run(uv_default_loop(), UV_RUN_DEFAULT);
  uv_fs_close(NULL, &read_req, fd, NULL);

  uv_fs_req_cleanup(&read_req);
  uv_loop_close(uv_default_loop());
  return 0;
}

[bnoordhuis]

personally I'd be fine with/love/prefer an io_uring that targeted more modern io_urings

Things may be heading that way out of necessity.

IORING_SETUP_SQPOLL requires linux v5.11 if you want to use it with non-fixed files (IIUC) but libuv can't do fixed files and without sqpoll it has to call io_uring_enter(IORING_ENTER_GETEVENTS) so often that any performance benefits are negated.

[bnoordhuis]

This is ready for review now. Linux v5.13 is the minimum required, otherwise libuv falls back to the thread pool.

[santigimeno]

Should we document this change in the fs docs?

[bnoordhuis]

Should we document this change in the fs docs?

Yes. I thought it didn't say anything about the implementation but it does. I'll do that tomorrow.

[santigimeno]

LGTM (once docs are updated). This is awesome, thanks

[trevnorris]

Still getting the following from valgrind:

fs_file_async

Syscall param io_uring_enter(sigsz) contains uninitialised byte(s)
   at 0x4993A3D: syscall (syscall.S:38)
   by 0x4F1366: uv__io_uring_enter (src/unix/linux.c:351)
   by 0x4F23A9: uv__iou_submit (src/unix/linux.c:644)
   by 0x4F24B5: uv__iou_fs_read_or_write (src/unix/linux.c:692)
   by 0x4DDAAC: uv_fs_write (src/unix/fs.c:2212)
   by 0x422F97: create_cb (test/test-fs.c:491)
   by 0x4DA38C: uv__fs_done (src/unix/fs.c:1779)
   by 0x4CDFD0: uv__work_done (src/threadpool.c:326)
   by 0x4D6433: uv__async_io (src/unix/async.c:176)
   by 0x4F2F95: uv__io_poll (src/unix/linux.c:1076)
   by 0x4D6B2D: uv_run (src/unix/core.c:417)
   by 0x42241C: run_test_fs_file_async (test/test-fs.c:915)

[trevnorris]

LGTM other than the valgrind error. The current code you have for incrementing events is fine enough since it basically follows what the threadpool is doing. It's a bug, but not one you should need to worry about. I'll take care of that in a future PR.

[bnoordhuis]

FWIW, I don't get that warning with valgrind 3.18.1.

I'm 99% sure it's a valgrind bug because that sigsz it's complaining about is something that libuv doesn't use and doesn't exist in newer kernels.

On kernels where it was a thing libuv simply never calls io_uring_enter().

[trevnorris]

@bnoordhuis I'm also running valgrind 3.18.1. Upon further testing, the error only appears when I compile with clang (I tried all versions 11 to 15). It doesn't happen with gcc. Think it's a bug in clang or just a strange interaction with valgrind? Side note: need to compile with -gdwarf-4 when using clang for valgrind to work.

[bnoordhuis]

Oh, that's interesting. Yes, I'm getting the same warning with clang.

gcc emits this code for the syscall wrapper:

   0x00000000000997f0 <+0>:     endbr64
   0x00000000000997f4 <+4>:     sub    $0x10,%rsp
   0x00000000000997f8 <+8>:     mov    %ecx,%r8d
   0x00000000000997fb <+11>:    xor    %r9d,%r9d
   0x00000000000997fe <+14>:    mov    %edx,%ecx
   0x0000000000099800 <+16>:    push   $0x0
   0x0000000000099802 <+18>:    mov    %esi,%edx
   0x0000000000099804 <+20>:    xor    %eax,%eax
   0x0000000000099806 <+22>:    mov    %edi,%esi
   0x0000000000099808 <+24>:    mov    $0x1aa,%edi
   0x000000000009980d <+29>:    call   0xe810 <syscall@plt>
   0x0000000000099812 <+34>:    add    $0x18,%rsp
   0x0000000000099816 <+38>:    ret

While clang emits this:

   0x00000000000a31f0 <+0>:     push   %rax
   0x00000000000a31f1 <+1>:     mov    %ecx,%r8d
   0x00000000000a31f4 <+4>:     mov    %edx,%ecx
   0x00000000000a31f6 <+6>:     mov    %esi,%edx
   0x00000000000a31f8 <+8>:     mov    %edi,%esi
   0x00000000000a31fa <+10>:    movl   $0x0,(%rsp)
   0x00000000000a3201 <+17>:    mov    $0x1aa,%edi
   0x00000000000a3206 <+22>:    xor    %r9d,%r9d
   0x00000000000a3209 <+25>:    xor    %eax,%eax
   0x00000000000a320b <+27>:    call   0xd780 <syscall@plt>
   0x00000000000a3210 <+32>:    pop    %rcx
   0x00000000000a3211 <+33>:    ret

I'm guessing that valgrind gets confused by that movl $0x0,(%rsp) for the sigsz argument somehow.

[GazHank]

Hi @bnoordhuis is there a simple way to test disabling this change per your previous suggestion of a UV_USE_IO_URING=0 environment variable or compilation setting?

I'm hoping to rule out this change as the cause of a defect, so being able to quickly enable / disable the functionality would be very helpful

[bnoordhuis]

@GazHank UV_USE_IO_URING=0 still exists for now. If you're using a 5.x kernel: we landed some follow-up fixes, not all of them released yet.

[GazHank]

thanks @bnoordhuis yes, I'm seeing some issues on ubuntu 22.04.2 LTS ( 5.19.0-46-generic )

So far I've managed to confirm that the issues only show up when upgrading from libuv 1.44 to 1.45, but setting an environment variable of UV_USE_IO_URING=0 doesn't seem to make any difference. Perhaps it is a different change from 1.45 that is causing problems, but I don't see any other likely causes within the notables changes list (as windows is entirely unaffected).

Please let me know if there is an existing issue you would like me to add info to, or if you would prefer that I raise a new issue entirely. Of course if there are some draft changes you would like me to test to see if that solves the problem then please let me know, I'd be happy to help test the changes.

[bnoordhuis]

You can try with the HEAD of the v1.x branch. If that doesn't fix it, it'd be helpful if you could run git bisect on the commits between 1.44 and 1.45 and open an issue with as much details as possible.

[GazHank]

Thanks @bnoordhuis Unfortunately I can confirm that it is this commit that causes the break :-(

Additionally, setting UV_USE_IO_URING=0 doesn't seem to make any difference, however amending the kernel version specified within linux.c file to a higher version than I'm running does fix/workaround the issue

libuv/src/unix/linux.c

Lines 431 to 443 in d09441c

	use = atomic_load_explicit(&use_io_uring, memory_order_relaxed);
	if (use == 0) {
	/* Older kernels have a bug where the sqpoll thread uses 100% CPU. */
	use = uv__kernel_version() >= /* 5.10.186 */ 0x050ABA ? 1 : -1;
	/* But users can still enable it if they so desire. */
	val = getenv("UV_USE_IO_URING");
	if (val != NULL)
	use = atoi(val) ? 1 : -1;
	atomic_store_explicit(&use_io_uring, use, memory_order_relaxed);
	}

As such, is it perhaps that UV_USE_IO_URING=1 can be use to enable the feature, but UV_USE_IO_URING=0 cannot be used to disable it?

I'll raise a new issues and try to include as much info as I can

[bnoordhuis]

UV_USE_IO_URING=0 is a hard disable. You should be able to verify with strace that no io_uring system calls are made.

[wswsmao]

trevnorris

Did some basic benchmarking, and it's giving me > 8x performance when reading small chunks out of /dev/zero. So great news on that side.

valgrind isn't happy about a few tests. Mostly with the error Syscall param io_uring_enter(sigsz) contains uninitialised byte(s). Simplest repro I could make is this:

#include <uv.h>

void on_read(uv_fs_t *req) { }

int main() {
  uv_fs_t read_req;
  uv_buf_t buffer;
  int fd;
  char bytes[64];

  buffer = uv_buf_init(bytes, 64);
  fd = uv_fs_open(NULL, &read_req, "/dev/zero", UV_FS_O_RDONLY, 0, NULL);
  uv_fs_read(uv_default_loop(), &read_req, fd, &buffer, 1, 0, on_read);
  uv_run(uv_default_loop(), UV_RUN_DEFAULT);
  uv_fs_close(NULL, &read_req, fd, NULL);

  uv_fs_req_cleanup(&read_req);
  uv_loop_close(uv_default_loop());
  return 0;
}

Hi, @trevnorris , I want to evaluate the performance improvements brought by this feature. I was wouldering is there any performance benchmarking available？

[wswsmao]

trevnorris

Did some basic benchmarking, and it's giving me > 8x performance when reading small chunks out of /dev/zero. So great news on that side.
valgrind isn't happy about a few tests. Mostly with the error Syscall param io_uring_enter(sigsz) contains uninitialised byte(s). Simplest repro I could make is this:

#include <uv.h>

void on_read(uv_fs_t *req) { }

int main() {
  uv_fs_t read_req;
  uv_buf_t buffer;
  int fd;
  char bytes[64];

  buffer = uv_buf_init(bytes, 64);
  fd = uv_fs_open(NULL, &read_req, "/dev/zero", UV_FS_O_RDONLY, 0, NULL);
  uv_fs_read(uv_default_loop(), &read_req, fd, &buffer, 1, 0, on_read);
  uv_run(uv_default_loop(), UV_RUN_DEFAULT);
  uv_fs_close(NULL, &read_req, fd, NULL);

  uv_fs_req_cleanup(&read_req);
  uv_loop_close(uv_default_loop());
  return 0;
}

Hi, @trevnorris , I want to evaluate the performance improvements brought by this feature. I was wouldering is there any performance benchmarking available？

Hi, @bnoordhuis , I was wouldering is there any performance benchmarking available？

[heiher]

These lines of code are equivalent to:

  atomic_thread_fence(memory_order_release);
  iou->sqtail = *iou->sqtail + 1;
  flags = iou->sqflags;
  atomic_thread_fence(memory_order_acquire);

It seems impossible to prevent the load of sqflags from being reordered before the store to sqtail. e.g.

  flags = iou->sqflags;
  ...
  atomic_thread_fence(memory_order_release);
  iou->sqtail = *iou->sqtail + 1;
  atomic_thread_fence(memory_order_acquire);

I suggest the following change:

  atomic_store_explicit((_Atomic uint32_t*) iou->sqtail,
                        *iou->sqtail + 1,
                        memory_order_release);
  atomic_thread_fence(memory_order_seq_cst);

  if (iou->sqflags & UV__IORING_SQ_NEED_WAKEUP)

equivalent to:

  atomic_thread_fence(memory_order_release);
  iou->sqtail = *iou->sqtail + 1;
  atomic_thread_fence(memory_order_seq_cst);

  if (iou->sqflags & UV__IORING_SQ_NEED_WAKEUP)

[heiher]

Ref: https://github.com/torvalds/linux/blob/691d401c7e0e5ea34ac6f8151bc0696db1b2500a/io_uring/io_uring.c#L25-L28

 * When using the SQ poll thread (IORING_SETUP_SQPOLL), the application
 * needs to check the SQ flags for IORING_SQ_NEED_WAKEUP *after*
 * updating the SQ tail; a full memory barrier smp_mb() is needed
 * between.