Posts on sgarzare's blog

AF_VSOCK: network namespace support is here

sgarzare@redhat.com (Stefano Garzarella) — Wed, 11 Feb 2026 18:00:00 +0100

A missing piece for AF_VSOCK in the Linux kernel has been network namespace support. We discussed it as a future challenge during the KVM Forum 2019 talk and it was mentioned in several conference discussions since then.

I started working on namespace support back in 2019, but never had the chance to complete it. Last year, Bobby Eshleman (Meta) restarted the effort and drove it through 16 revisions of the patch series. Daniel Berrangé, Michael S. Tsirkin, Paolo Abeni, and I contributed with reviews and suggestions that shaped the current user API. The result has been merged into net-next and will be available in Linux 7.0.

Updated on 2026-04-17 to cover the write-once behavior of child_ns_mode, merged after the initial publication of this post.

Background

Network namespaces are a fundamental building block for containers in Linux. They provide isolation of the network stack, so each namespace has its own interfaces, routing tables, and sockets.

Before Linux 7.0, AF_VSOCK was not namespace-aware. All vsock sockets lived in the same global space, regardless of the network namespace they were created in. This caused two problems:

No isolation: a VM started inside a network namespace (or container) was reachable via vsock from any other namespace on the host, breaking the isolation that containers expect.
No CID reuse: since CIDs were global, two VMs in different namespaces could not use the same CID, even if they were completely isolated from each other at the network level.

Design

The new implementation introduces two modes, configured per network namespace:

global: CIDs are shared across namespaces. This is the original behavior and the default, so existing setups continue to work without any change.
local: namespaces are completely isolated. Sockets in a local-mode namespace can only communicate with other sockets in the same namespace.

Two sysctl knobs are available since Linux 7.0:

/proc/sys/net/vsock/child_ns_mode: the parent namespace uses this to set the mode that new child namespaces will inherit. Accepts global or local. This sysctl is write-once: after the first write, it becomes immutable.
/proc/sys/net/vsock/ns_mode: read-only, shows the mode of the current namespace. The mode is immutable after namespace creation.

This design ensures backward compatibility: the default is global, matching the previous behavior. Namespace isolation is opt-in.

Each namespace gets its mode from the parent’s child_ns_mode at creation time. Once set, the namespace’s ns_mode is immutable: every socket and VM in that namespace follows it.

The child_ns_mode sysctl is also write-once: the first write locks the value and any subsequent write of a different value returns -EBUSY. This guarantees that a namespace manager can set the value once and be certain it won’t change before creating its namespaces, preventing races where two administrator processes set conflicting modes and one ends up with a namespace in the wrong mode.

Supported vsock transports

This series adds namespace support to two transports:

vhost-vsock: host-to-guest (H2G) transport, emulates the virtio-vsock device for KVM guests
vsock-loopback: local transport, useful for testing and debugging without running VMs

The missing transports are the guest-to-host (G2H) ones (virtio, hyperv, vmci). These run in the guest as device drivers, and we currently don’t have a way to assign a vsock device to a specific namespace, since vsock devices are not standard network devices. For now, G2H transports operate in global mode, so they are reachable from any global namespace, but not from local namespaces. This means that sockets in a local namespace cannot communicate with the host through these transports. We plan to work on that in the future.

Examples

Loopback

In the following examples, the commands without a namespace prefix run in the initial network namespace (init_netns), which is the default namespace where all processes start. The init_netns is always in global mode.

These examples use the vsock loopback device for local communication, without any VM involved.

Make sure the vsock_loopback kernel module is loaded:

$ sudo modprobe vsock_loopback

Namespace isolation with unshare

Global mode (default)

By default, child_ns_mode is set to global. This is the same behavior as before Linux 7.0: vsock sockets are shared across namespaces.

A listener started in a new namespace is reachable from the init_netns using the loopback CID (VMADDR_CID_LOCAL = 1):

$ unshare --user --net nc --vsock -l 1234 &
$ nc --vsock 1 1234
# reachable - global mode, no isolation

Local mode

Setting child_ns_mode to local enables isolation. Since this sysctl is write-once, we use a nested unshare to avoid locking the init_netns mode:

$ unshare --user --map-root-user --net bash -c \
    'echo local > /proc/sys/net/vsock/child_ns_mode && \
     unshare --net nc --vsock -l 1234' &
$ nc --vsock 1 1234
Ncat: Connection reset by peer.

Namespace isolation with ip netns

The same can be done with ip netns, which requires root (or CAP_SYS_ADMIN).

First, create a global namespace and check its mode:

$ sudo ip netns add vsock_ns_global
$ sudo ip netns exec vsock_ns_global cat /proc/sys/net/vsock/ns_mode
global

A listener in the global namespace is reachable from the init_netns:

$ sudo ip netns exec vsock_ns_global nc --vsock -l 1234 &
$ nc --vsock 1 1234
# reachable - global mode, no isolation

Now create a local namespace. Since child_ns_mode is write-once, we use unshare to create a parent namespace and set its mode to local before creating the child:

$ sudo unshare --net bash -c \
    'echo local > /proc/sys/net/vsock/child_ns_mode && \
     ip netns add vsock_ns_local'
$ sudo ip netns exec vsock_ns_local cat /proc/sys/net/vsock/ns_mode
local

A listener in the local namespace is not reachable from the init_netns:

$ sudo ip netns exec vsock_ns_local nc --vsock -l 1234 &
$ nc --vsock 1 1234
Ncat: Connection reset by peer.

But communication within the same namespace still works:

$ sudo ip netns exec vsock_ns_local nc --vsock 1 1234
# reachable - same namespace

VMs with QEMU

The vhost-vsock H2G transport exposes the /dev/vhost-vsock device, which QEMU opens at VM startup to emulate the virtio-vsock device for the guest. Since namespace support applies to this transport, VMs inherit the namespace mode as well.

In the following examples, we reuse the vsock_ns_global and vsock_ns_local namespaces created in the previous section.

Global mode

With global mode, the VM started in a global namespace is reachable from any other global namespace, including the init_netns:

$ sudo ip netns exec vsock_ns_global \
    qemu-system-x86_64 -m 1G -M q35,accel=kvm \
    -drive file=guest.qcow2,if=virtio,snapshot=on \
    -device vhost-vsock-pci,guest-cid=42

# start a listener in the guest (global namespace)
guest_global$ nc --vsock -l 1234

# from the init_netns (global) - reachable
$ nc --vsock 42 1234

Local mode

With local mode, the VM is only reachable from within the same namespace:

$ sudo ip netns exec vsock_ns_local \
    qemu-system-x86_64 -m 1G -M q35,accel=kvm \
    -drive file=guest.qcow2,if=virtio,snapshot=on \
    -device vhost-vsock-pci,guest-cid=42

# start a listener in the guest (local namespace)
guest_local$ nc --vsock -l 1234

# from the init_netns (global) - isolated
$ nc --vsock 42 1234
Ncat: Connection reset by peer.

# from the same namespace - reachable
$ sudo ip netns exec vsock_ns_local nc --vsock 42 1234

Guest-to-host (G2H) behavior

As mentioned in the Supported vsock transports section, the G2H virtio transport does not support namespaces yet. The virtio-vsock device in the guest always operates in global mode, so only sockets in global namespaces can communicate with the host.

Using the VM started in vsock_ns_global, a listener in the guest’s init_netns is reachable from the host:

# start a listener in the guest
guest_global$ nc --vsock -l 1234

# from the host - reachable
$ nc --vsock 42 1234

But a listener started in a local namespace inside the guest is not reachable from the host:

# create a local namespace in the guest and start a listener
guest_global$ unshare --user --map-root-user --net bash -c \
    'echo local > /proc/sys/net/vsock/child_ns_mode && \
     unshare --net nc --vsock -l 1234' &

# from the host - isolated
$ nc --vsock 42 1234
Ncat: Connection reset by peer.

CID reuse

Note that we used the same CID (42) in both examples without turning off the first VM. This is possible because the second VM is in a local namespace, so its CID space is isolated. With global mode, QEMU would fail to start the second VM because the CID is already in use.

Patches

vDPA: support for block devices in Linux and QEMU

sgarzare@redhat.com (Stefano Garzarella) — Mon, 12 Feb 2024 18:42:57 +0100

A vDPA device is a type of device that follows the virtio specification for its datapath but has a vendor-specific control path.

vDPA devices can be both physically located on the hardware or emulated by software.

A small vDPA parent driver in the host kernel is required only for the control path. The main advantage is the unified software stack for all vDPA devices:

vhost interface (vhost-vdpa) for userspace or guest virtio driver, like a VM running in QEMU
virtio interface (virtio-vdpa) for bare-metal or containerized applications running in the host
management interface (vdpa netlink) for instantiating devices and configuring virtio parameters

Useful Resources

Many blog posts and talks have been published in recent years that can help you better understand vDPA and the use cases. On vdpa-dev.gitlab.io we collected some of them; I suggest you at least explore the following:

Block devices

Most of the work in vDPA has been driven by network devices, but in recent years, we have also developed support for block devices.

The main use case is definitely leveraging the hardware to directly emulate the virtio-blk device and support different network backends such as Ceph RBD or iSCSI. This is the goal of some SmartNICs or DPUs, which are able to emulate virtio-net devices of course, but also virtio-blk for network storage.

The abstraction provided by vDPA also makes software accelerators possible, similar to existing vhost or vhost-user devices. We discussed about that at KVM Forum 2021.

We talked about the fast path and the slow path in that talk. When QEMU needs to handle requests, like supporting live migration or executing I/O throttling, it uses the slow path. During the slow path, the device exposed to the guest is emulated in QEMU. QEMU intercepts the requests and forwards them to the vDPA device by taking advantage of the driver implemented in libblkio. On the other hand, when QEMU doesn’t need to intervene, the fast path comes into play. In this case, the vDPA device can be directly exposed to the guest, bypassing QEMU’s emulation.

libblkio exposes common API for accessing block devices in userspace. It supports several drivers. We will focus more on virtio-blk-vhost-vdpa driver, which is used by virtio-blk-vhost-vdpa block device in QEMU. It only supports slow path for now, but in the future it should be able to switch to fast path automatically. Since QEMU 7.2, it supports libblkio drivers, so you can use the following options to attach a vDPA block device to a VM:

   -blockdev node-name=drive_src1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on \
   -device virtio-blk-pci,id=src1,bootindex=2,drive=drive_src1 \

Anyway, to fully leverage the performance of a vDPA hardware device, we can always use the generic vhost-vdpa-device-pci device offered by QEMU that supports any vDPA device and exposes it directly to the guest. Of course, QEMU is not able to intercept requests in this scenario and therefore some features offered by its block layer (e.g. live migration, disk format, etc.) are not supported. Since QEMU 8.0, you can use the following options to attach a generic vDPA device to a VM:

    -device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-0

At KVM Forum 2022, Alberto Faria and Stefan Hajnoczi introduced libblkio, while Kevin Wolf and I discussed its usage in the QEMU Storage Deamon (QSD).

Software devices

One of the significant benefits of vDPA is its strong abstraction, enabling the implementation of virtio devices in both hardware and software—whether in the kernel or user space. This unification under a single framework, where devices appear identical for QEMU facilitates the seamless integration of hardware and software components.

Kernel devices

Regarding in-kernel devices, starting from Linux v5.13, there exists a simple simulator designed for development and debugging purposes. It is available through the vdpa-sim-blk kernel module, which emulates a 128 MB ramdisk. As highlighted in the presentation at KVM Forum 2021, a future device in the kernel (similar to the repeatedly proposed but never merged vhost-blk) could potentially offer excellent performance. Such a device could be used as an alternative when hardware is unavailable, for instance, facilitating live migration in any system, regardless of whether the destination system features a SmartNIC/DPU or not.

User space devices

Instead, regarding user space, we can use VDUSE. QSD supports it and thus allows us to export any disk image supported by QEMU, such as a vDPA device in this way:

qemu-storage-daemon \
    --blockdev file,filename=/path/to/disk.qcow2,node-name=file \
    --blockdev qcow2,file=file,node-name=qcow2 \
    --export type=vduse-blk,id=vduse0,name=vduse0,node-name=qcow2,writable=on

Containers, VMs, or bare-metal

As mentioned in the introduction, vDPA supports different buses such as vhost-vdpa and virtio-vdpa. This flexibility enables the utilization of vDPA devices with virtual machines or user space drivers (e.g., libblkio) through the vhost-vdpa bus. Additionally, it allows interaction with applications running directly on the host or within containers via the virtio-vdpa bus.

The vdpa tool in iproute2 facilitates the management of vdpa devices through netlink, enabling the allocation and deallocation of these devices.

Starting with Linux 5.17, vDPA drivers support driver_ovveride. This enhancement allows dynamic reconfiguration during runtime, permitting the migration of a device from one bus to another in this way:

# load vdpa buses
$ modprobe -a virtio-vdpa vhost-vdpa
# load vdpa-blk in-kernel simulator
$ modprobe vdpa-sim-blk

# instantiate a new vdpasim_blk device called `vdpa0`
$ vdpa dev add mgmtdev vdpasim_blk name vdpa0

# `vdpa0` is attached to the first vDPA bus driver loaded
$ driverctl -b vdpa list-devices
vdpa0 virtio_vdpa

# change the `vdpa0` bus to `vhost-vdpa`
$ driverctl -b vdpa set-override vdpa0 vhost_vdpa

# `vdpa0` is now attached to the `vhost-vdpa` bus
$ driverctl -b vdpa list-devices
vdpa0 vhost_vdpa [*]

# Note: driverctl(8) integrates with udev so the binding is preserved.

Examples

Below are several examples on how to use VDUSE and the QEMU Storage Daemon with VMs (QEMU) or Containers (podman). These steps are easily adaptable to any hardware that supports virtio-blk devices via vDPA.

qcow2 image available for host applications and containers

# load vdpa buses
$ modprobe -a virtio-vdpa vhost-vdpa

# create an empty qcow2 image
$ qemu-img create -f qcow2 test.qcow2 10G

# load vduse kernel module
$ modprobe vduse

# launch QSD exposing the `test.qcow2` image as `vduse0` vDPA device
$ qemu-storage-daemon --blockdev file,filename=test.qcow2,node-name=file \
  --blockdev qcow2,file=file,node-name=qcow2 \
  --export vduse-blk,id=vduse0,name=vduse0,num-queues=1,node-name=qcow2,writable=on &

# instantiate the `vduse0` device (same name used in QSD)
$ vdpa dev add name vduse0 mgmtdev vduse

# be sure to attach it to the `virtio-vdpa` device to use with host applications
$ driverctl -b vdpa set-override vduse0 virtio_vdpa

# device exposed as a virtio device, but attached to the host kernel
$ lsblk -pv
NAME     TYPE TRAN   SIZE RQ-SIZE  MQ
/dev/vda disk virtio  10G     256   1

# start a container with `/dev/vda` attached
podman run -it --rm --device /dev/vda --group-add keep-groups fedora:39 bash

Launch a VM using a vDPA device

# download Fedora cloud image (or use any other bootable image you want)
$ wget https://download.fedoraproject.org/pub/fedora/linux/releases/39/Cloud/x86_64/images/Fedora-Cloud-Base-39-1.5.x86_64.qcow2

# launch QSD exposing the VM image as `vduse1` vDPA device
$ qemu-storage-daemon \
  --blockdev file,filename=Fedora-Cloud-Base-39-1.5.x86_64.qcow2,node-name=file \
  --blockdev qcow2,file=file,node-name=qcow2 \
  --export vduse-blk,id=vduse1,name=vduse1,num-queues=1,node-name=qcow2,writable=on &

# instantiate the `vduse1` device (same name used in QSD)
$ vdpa dev add name vduse1 mgmtdev vduse

# initially it's attached to the host (`/dev/vdb`), because `virtio-vdpa`
# is the first kernel module we loaded
$ lsblk -pv
NAME     TYPE TRAN   SIZE RQ-SIZE  MQ
/dev/vda disk virtio  10G     256   1
/dev/vdb disk virtio   5G     256   1
$ lsblk /dev/vdb
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vdb    251:16   0    5G  0 disk 
├─vdb1 251:17   0    1M  0 part 
├─vdb2 251:18   0 1000M  0 part 
├─vdb3 251:19   0  100M  0 part 
├─vdb4 251:20   0    4M  0 part 
└─vdb5 251:21   0  3.9G  0 part 

# and it is identified as `virtio1` in the host
$ ls /sys/bus/vdpa/devices/vduse1/
driver  driver_override  power  subsystem  uevent  virtio1

# attach it to the `vhost-vdpa` device to use the device with VMs
$ driverctl -b vdpa set-override vduse1 vhost_vdpa

# `/dev/vdb` is not available anymore
$ lsblk -pv
NAME     TYPE TRAN   SIZE RQ-SIZE  MQ
/dev/vda disk virtio  10G     256   1

# the device is identified as `vhost-vdpa-1` in the host
$ ls /sys/bus/vdpa/devices/vduse1/
driver  driver_override  power  subsystem  uevent  vhost-vdpa-1
$ ls -l /dev/vhost-vdpa-1
crw-------. 1 root root 511, 0 Feb 12 17:58 /dev/vhost-vdpa-1

# launch QEMU using `/dev/vhost-vdpa-1` device with the
# `virtio-blk-vhost-vdpa` libblkio driver
$ qemu-system-x86_64 -m 512M -smp 2 -M q35,accel=kvm,memory-backend=mem \
  -object memory-backend-memfd,share=on,id=mem,size="512M" \
  -blockdev node-name=drive0,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on \
  -device virtio-blk-pci,drive=drive0

# `virtio-blk-vhost-vdpa` blockdev can be used with any QEMU block layer
# features (e.g live migration, I/O throttling).
# In this example we are using I/O throttling:
$ qemu-system-x86_64 -m 512M -smp 2 -M q35,accel=kvm,memory-backend=mem \
  -object memory-backend-memfd,share=on,id=mem,size="512M" \
  -blockdev node-name=drive0,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on \
  -blockdev node-name=throttle0,driver=throttle,file=drive0,throttle-group=limits0 \
  -object throttle-group,id=limits0,x-iops-total=2000 \
  -device virtio-blk-pci,drive=throttle0

# Alternatively, we can use the generic `vhost-vdpa-device-pci` to take
# advantage of all the performance, but without having any QEMU block layer
# features available
$ qemu-system-x86_64 -m 512M -smp 2 -M q35,accel=kvm,memory-backend=mem \
  -object memory-backend-memfd,share=on,id=mem,size="512M" \
  -device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-0

SOCAT now supports AF_VSOCK

sgarzare@redhat.com (Stefano Garzarella) — Fri, 22 Jan 2021 15:16:04 +0100

SOCAT is a CLI utility which enables the concatenation of two sockets together. It establishes two bidirectional byte streams and transfers data between them.

socat supports several address types (e.g. TCP, UDP, UNIX domain sockets, etc.) to construct the streams. The latest version 1.7.4, released earlier this year [2021-01-04], supports also AF_VSOCK addresses:

VSOCK-LISTEN:<port>
- Listen on port and accepts a VSOCK connection.
VSOCK-CONNECT:<cid>:<port>
- Establishes a VSOCK stream connection to the specified cid and port.

FOSDEM 2021

If you are interested on VSOCK, I’ll talk witn Andra Paraschiv (AWS) about it at FOSDEM 2021. The talk is titled Leveraging virtio-vsock in the cloud and containers and it’s scheduled for Saturday, February 6th 2021 at 11:30 AM (CET).

We will show cool VSOCK use cases and some demos about developing, debugging, and measuring the VSOCK performance, including socat demos.

Examples

socat could be very useful for concatenating and redirecting sockets. In this section we will see some examples.

Examples below refer to a guest with CID 42 that we created using virt-builder and virt-install .

VM setup

virt-builder is able to download the installer and create the disk image with Fedora 33 or other distros. It is also able to set the root password and inject the ssh public key, simplifying the creation of guest disk image:

VM_IMAGE="vsockguest_f33.qcow2"

host$ virt-builder --root-password=password:mypassword \
        --ssh-inject root:file:/home/user/.ssh/id_rsa.pub \
        --output=${VM_IMAGE} \
        --format=qcow2 --size 10G --selinux-relabel \
        --update fedora-33

Once the disk image is ready, we create our VM with virt-install. We can specify the VM settings like the number of vCPUs, the amount of RAM, and the CID assigned to the VM [42]:

host$ virt-install --name vsockguest \
        --ram 2048 --vcpus 2 --os-variant fedora33 \
        --import --disk path=${VM_IMAGE},bus=virtio \
        --graphics none --vsock cid.address=42

After the creation of the VM, we will remain attached to the console and we can detach from it by pressing ctrl-].

We can reattach to the console in this way:

host$ virsh console vsockguest

If the VM is turned off, we can boot it and attach directly to the console in this way:

host$ virsh start --console vsockguest

ncat like

It’s possible to use socat like ncat, transferring stdin and stdout via VSOCK.

Guest listening

In this example we start socat in the guest listening on port 1234:

guest$ socat - VSOCK-LISTEN:1234

Then we connect from the host using the CID 42 assigned to the VM:

host$ socat - VSOCK-CONNECT:42:1234

At this point we can exchange characters between guest and host, since stdin and stdout are linked through the VSOCK socket.

Host listening

In this example we do the opposite, starting socat in the host listening on port 1234:

host$ socat - VSOCK-LISTEN:1234

Then, in the guest, we connect to the host using the well defined CID 2. It’s always used to reach the host:

guest$ socat - VSOCK-CONNECT:2:1234

ssh over VSOCK

The coolest feature of socat is to concatenate sockets of different address families, so in this example we redirect ssh traffic through VSOCK socket exposed by the VM.

This example could be useful if the VM doesn’t have any NIC attached and we want to provide some network connectivity, like the ssh access.

First of all, in the guest we start socat linking the VSOCK socket listening on port 22, to a TCP socket which will connect to the local TCP port 22 where the ssh server is listening:

guest$ socat VSOCK-LISTEN:22,reuseaddr,fork TCP:localhost:22

On the host we can use socat as ProxyCommand of ssh to forward the connection through the AF_VSOCK socket:

host$ ssh -o "ProxyCommand socat - VSOCK-CONNECT:42:22" root@localhost

socat redirects all the traffic and allow us to use ssh over VSOCK to reach the guest.

Connecting sibling VMs

Another scenario where socat can be very useful is connecting two sibling VMs running on the same host.

Currently this is not supported by vhost-vsock, so we can use socat to concatenate the two VMs.

Let’s see an example: suppose we launch another VM with CID = 24 on the same host. Now we want to use ncat in the VMs to communicate with each other.

Guest 42 will listen on port 1234 and guest 24 will initiate the connection. But before we do this we need to set up the bridge in the host with socat to allow the two VMs to communicate:

host$ socat VSOCK-LISTEN:1234 VSOCK-CONNECT:42:1234

At this point we can launch ncat in the VMs and communicate:

guest_42$ nc --vsock -l 1234

Note: as destination CID we have to use the host’s well-known CID (2) because we need to connect to the socat bridge running in the host, that will redirect packets correctly between the two machines:

guest_24$ nc --vsock 2 1234

AF_VSOCK: nested VMs and loopback support available

sgarzare@redhat.com (Stefano Garzarella) — Tue, 25 Feb 2020 20:30:21 +0100

During the last KVM Forum 2019, we discussed some next steps and several requests came from the audience.

In the last months, we worked on that and recent Linux releases contain interesting new features that we will describe in this blog post:

Nested VMs support, available in Linux 5.5
Local communication support, available in Linux 5.6

DevConf.CZ 2020

These updates and an introduction to AF_VSOCK were presented at DevConf.CZ 2020 during the “VSOCK: VM↔host socket with minimal configuration” talk. Slides and recording are available.

Nested VMs

Before Linux 5.5, the AF_VSOCK core supported only one transport loaded at run time. That was a limit for nested VMs, because we need multiple transports loaded together.

Types of transport

Under the AF_VSOCK core, that provides the socket interface to the user space applications, we have several transports that implement the communication channel between guest and host.

vsock transports

These transports depend on the hypervisor and we can put them in two groups:

H2G (host to guest) transports: they run in the host and usually they provide the device emulation; currently we have vhost and vmci transports.
G2H (guest to host) transports: they run in the guest and usually they are device drivers; currently we have virtio, vmci, and hyperv transports.

Multi-transports

In a nested VM environment, we need to load both G2H and H2G transports together in the L1 guest, for this reason, we implemented the multi-transports support to use vsock through nested VMs.

vsock and nested VMs

Starting from Linux 5.5, the AF_VSOCK can handle two types of transports loaded together at runtime:

H2G transport, to communicate with the guest
G2H transport, to communicate with the host.

So in the QEMU/KVM environment, the L1 guest will load both virtio-transport, to communicate with L0, and vhost-transport to communicate with L2.

Local Communication

Another feature recently added is the possibility to communicate locally on the same host. This feature, suggested by Richard WM Jones, can be very useful for testing and debugging applications that use AF_VSOCK without running VMs.

Linux 5.6 introduces a new transport called vsock-loopback, and a new well know CID for local communication: VMADDR_CID_LOCAL (1). It’s a special CID to direct packets to the same host that generated them.

vsock loopback

Other CIDs can be used for the same purpose, but it’s preferable to use VMADDR_CID_LOCAL:

Local Guest CID
- if G2H is loaded (e.g. running in a VM)
VMADDR_CID_HOST (2)
- if H2G is loaded and G2H is not loaded (e.g. running on L0). If G2H is also loaded, then VMADDR_CID_HOST is used to reach the host

Richard recently used the vsock local communication to implement a regression test test for nbdkit/libnbd vsock support, using the new VMADDR_CID_LOCAL.

Example

# Listening on port 1234 using ncat(1)
l0$ nc --vsock -l 1234

# Connecting to the local host using VMADDR_CID_LOCAL (1)
l0$ nc --vsock 1 1234

Patches

QEMU 4.2 mmap(2)s kernel and initrd

sgarzare@redhat.com (Stefano Garzarella) — Sun, 22 Dec 2019 17:46:44 +0100

In order to save memory and boot time, QEMU 4.2 and later versions are able to mmap(2) the kernel and initrd specified with -kernel and -initrd parameters. This approach allows us to avoid reading and copying them into a buffer, saving memory and time.

The memory pages that contain kernel and initrd are shared between multiple VMs using the same kernel and initrd images. So, when many VMs are launched we can save memory by sharing pages, and save time by avoiding reading them each time from the disk.

This feature is automatically used for some targets with ELF kernels (e.g. x86_64 vmlinux ELF image with PVH entry point), but it is not available with compressed kernel images (e.g. bzImage).

Patches

The patches that implement this feature are merged upstream and released with QEMU 4.2.

The main change was to map kernel and initrd into the memory, instead of reading them, using g_mapped_file_*() APIs.

Benchmarks

We measured the memory footprint and the boot time using a standard QEMU build (qemu-system-x86_64) with a PVH kernel and initrd (cpio):

Initrd size: 3.0M
Kernel (vmlinux)
- image size: 28M
- sections size [size -A -d vmlinux]: 18.9M

Julio Montes did a very good analysis and he posted the results here: https://www.mail-archive.com/qemu-devel@nongnu.org/msg633168.html

Memory footprint

We used smem to measure USS and PSS:

USS (Unique Set Size): amount of memory that is committed to physical memory and is unique to a process; it is not shared with any other. It is the amount of memory that would be freed if the process were to terminate.
PSS (Proportional Set Size): This splits the accounting of shared pages that are committed to physical memory between all the processes that have them mapped.

$ smem -k | grep "PID\|$(pidof qemu-system-x86_64)"
  PID User     Command                         Swap      USS      PSS      RSS
24833 qemu     /usr/bin/qemu-system-x86_64        0    71.6M    74.3M    105.2

This is the memory footprint analysis, increasing the number of QEMU instances:

                           Memory footprint [MB]
     QEMU             before                 after
 # instances        USS     PSS           USS     PSS
      1           102.0   105.8         102.3   106.2
      2            94.6   101.2          72.3    90.1
      4            94.1    98.0          72.0    81.5
      8            94.0    96.2          71.8    76.9
     16            93.9    95.1          71.6    74.3

Boot time

We measured the boot time using the qemu-boot-time scripts described in this post.

This is the boot time analysis:

                                   Boot time [ms]
                          before                  after
 # trace points
 qemu_init_end:           63.85                   55.91
 linux_start_kernel:      82.11 (+18.26)          74.51 (+18.60)
 linux_start_user:       169.94 (+87.83)         159.06 (+84.56)

Conclusions

Mapping into memory the kernel and initrd images allowed us to save about 20 MB of memory when multiple VMs are started and allowed us to speed up the boot by about 10 milliseconds.

Note: both gains are strictly related to images size.

KVM Forum 2019: virtio-vsock in QEMU, Firecracker and Linux

sgarzare@redhat.com (Stefano Garzarella) — Sat, 09 Nov 2019 18:45:25 +0100

Slides and recording are available for the “virtio-vsock in QEMU, Firecracker and Linux: Status, Performance and Challenges” talk that Andra Paraschiv and I presented at KVM Forum 2019. This was the 13th edition of the KVM Forum conference. It took place in Lyon, France in October 2019.

We talked about the current status and future works of VSOCK drivers in Linux and how Firecracker and QEMU provides the virtio-vsock device.

Summary

Initially, Andra gave an overview of VSOCK, she described the state of the art, and the key features:

it is very simple to configure, the host assigns an unique CID (Context-ID) to each guest, and no configuration is needed inside the guest;
it provides AF_VSOCK address family, allowing user space application in the host and guest to communicate using standard POSIX Socket API (e.g. bind, listen, accept, connect, send, recv, etc.)

Andra also described common use cases for VSOCK, such as guest agents (clipboard sharing, remote console, etc.), network applications using SOCK_STREAM, and services provided by the hypervisor to the guests.

Going into the implementation details, Andra explained how the device in the guest communicates with the vhost backend in the host, exchanging data and events (i.e. ioeventfd, irqfd).

Firecracker

Focusing on Firecracker, Andra gave a brief overview on this new VMM (Virtual Machine Monitor) written in Rust and she explained why, in the v0.18.0 release, they switched from the experimental vhost-vsock implementation to a vhost-less solution:

focus on security impact
less dependency on host kernel features

This change required a device emulation in Firecracker, that implements virtio-vsock device model over MMIO. The device is exposed in the host using UDS (Unix Domain Sockets).

Andra described how Firecracker maps the VSOCK ports on the uds_path specified in the VM configuration:

Host-Initiated Connections
- Guest: create an AF_VSOCK socket and listen() on PORT
- Host: connect() to AF_UNIX at uds_path
- Host: send() “CONNECT PORT\n”
- Guest: accept() the new connection
Guest-Initiated Connections
- Host: create and listen() on an AF_UNIX socket at uds_path_PORT
- Guest: create an AF_VSOCK socket and connect() to HOST_CID and PORT
- Host: accept() the new connection

Finally, she showed the performance of this solution, running iperf-vsock benchmark, varying the size of the buffer used in Firecracker to transfer packets between the virtio-vsock device and the UNIX domain socket. The throughput on the guest to host path reaches 10 Gbps.

QEMU

In the second part of the talk, I described the QEMU implementation. QEMU provides the virtio-vsock device using the vhost-vsock kernel module.

The vsock device in QEMU handles only:

configuration
- user or management tool can configure the guest CID
live-migration
- connected SOCK_STREAM sockets become disconnected. Applications must handle a connection reset error and should reconnect.
- guest CID can be not available in the new host because can be assigned to another VM. In this case the guest is notified about the CID change.

The vhost-vsock kernel module handles the communication with the guest, providing in-kernel virtio device emulation, to have very high performance and to interface directly to the host socket layer. In this way, also host application can directly use POSIX Socket API to communicate with the guest. So, guest and host applications can be switched between them, changing only the destination CID.

virtio-vsock Linux drivers

After that, I told the story of VSOCK in the Linux tree, started in 2013 when the first implementation was merged, and the changes in the last year.

These changes mainly regard fixes, but for the virtio/vhost transports we also improved the performance with two simple changes released with Linux v5.4:

reducing the number of credit update messages exchanged
increasing the size of packets queued in the virtio-vsock device from 4 KB up to 64 KB, the maximum packet size handled by virtio-vsock devices.

With these changes we are able to reach ~40 Gbps in the Guest -> Host path, because the guest can now send up to 64 KB packets directly to the host; for the Host -> Guest path, we reached ~25 Gbps, because the host is still using 4 KB buffer preallocated by the guest.

Tools and languages that support VSOCK

In the last few years, several applications, tools, and languages started to support VSOCK and I listed them to update the audience:

Tools:
- wireshark >= 2.40 [2017-07-19]
- iproute2 >= 4.15 [2018-01-28]
  - ss
- tcpdump
  - merged in master [2019-04-16]
- nmap >= 7.80 [2019-08-10]
  - ncat
- nbd
  - nbdkit >= 1.15.5 [2019-10-19]
  - libnbd >= 1.1.6 [2019-10-19]
- iperf-vsock
  - iperf3 fork
Languages:
- C
  - glibc >= 2.18 [2013-08-10]
- Python
  - python >= 3.7 alpha 1 [2017-09-19]
- Golang
  - https://github.com/mdlayher/vsock
- Rust
  - libc crate >= 0.2.59 [2019-07-08]
    - struct sockaddr_vm
    - VMADDR_* macros
  - nix crate >= 0.15.0 [2019-08-10]
    - VSOCK supported in the socket API (nix::sys::socket)

Next steps

Concluding, I went through the next challenges that we are going to face:

multi-transport to use VSOCK in a nested VM environment. because we are limited by the fact that the current implementation can handle only one transport loaded at run time, so, we can’t load virtio_transport and vhost_transport together in the L1 guest. I already sent some patches upstream [RFC, v1], but they are still in progress.
network namespace support to create independent addressing domains with VSOCK socket. This could be useful for partitioning VMs in different domains or, in a nested VM environment, to isolate host applications from guest applications bound to the same port.
virtio-net as a transport for the virtio-vsock to avoid to re-implement features already done in virtio-net, such as mergeable buffers, page allocation, small packet handling.

From the audience

Other points to be addressed came from the comments we received from the audience:

loopback device could be very useful for developers to test applications that use VSOCK socket. The current implementation support loopback only in the guest, but it would be better to support it also in the host, adding VMADDR_CID_LOCAL special address.
VM to VM communication was asked by several people. Introducing it in the VSOCK core could complicate the protocol, the addressing and could require some sort of firewall. For now we do not have in mind to do it, but I developed a simple user space application to solve this issue: vsock-bridge. In order to improve the performance of this solution, we will consider the possibility to add sendfile(2) or MSG_ZEROCOPY support to the AF_VSOCK core.
virtio-vsock windows drivers is not planned to be addressed, but contributions are welcome. Other virtio windows drivers are available in the vm-guest-drivers-windows repository.

Stay tuned!

How to measure the boot time of a Linux VM with QEMU/KVM

sgarzare@redhat.com (Stefano Garzarella) — Sat, 24 Aug 2019 15:03:30 +0200

The stefano-garzarella/qemu-boot-time repository contains a Python perf-script and (Linux, QEMU, SeaBIOS) patches to measure the boot time of a Linux VM with QEMU/KVM.

Using I/O writes, we can trace events to measure the time consumed during the boot phase by the different components:

firmware
- SeaBIOS
- qboot
optionrom
- linuxboot [bzImage]
- pvh [vmlinux + PVH entry point]
Linux kernel

We extended the I/O port addresses and values defined in qboot/benchmark.h adding new trace points to trace the kernel boot time.

In the repository you can find patches for Linux, QEMU, and SeaBIOS to add the I/O writes in the components involved during the boot, and a Python perf-script useful to process the data recorded through perf using perf-script’s built-in Python interpreter.

Trace points

The benchmark.h file contains the following trace points used in the patches:

QEMU
- qemu_init_end: first kvm_entry (i.e. QEMU initialized has finished)
Firmware (SeaBIOS + optionrom or qboot)
- fw_start: first entry of the firmware
- fw_do_boot: after the firmware initialization (e.g. PCI setup, etc.)
- linux_start_boot: before the jump to the Linux kernel
- linux_start_pvhboot: before the jump to the Linux PVH kernel
Linux Kernel
- linux_start_kernel: first entry of the Linux kernel
- linux_start_user: before starting the init process

Custom trace points

If you want to add new trace points, you can simply add an I/O write to LINUX_EXIT_PORT or FW_EXIT_PORT I/O port with a value (> 7) that identifies the trace point:

    outb(10, LINUX_EXIT_PORT);

The perf script output will contain Exit point 10 line that identifies your custom trace point:

 qemu_init_end: 143.770419
 fw_start: 143.964328 (+0.193909)
 fw_do_boot: 164.71107 (+20.746742)
 Exit point 10: 165.396804 (+0.685734)
 linux_start_kernel: 165.979486 (+0.582682)
 linux_start_user: 272.178335 (+106.198849)

How to use

Clone qemu-boot-time repository

REPOS="~/repos"
cd ${REPOS}
git clone https://github.com/stefano-garzarella/qemu-boot-time.git

Apply patches to Linux, QEMU and SeaBIOS

Trace points are printed only if they are recorded, so you can enable few of them, patching only the components that you are interested in.

Linux

Apply the patches/linux.patch to your Linux kernel in order to trace kernel events

cd ${REPOS}/linux
git checkout -b benchmark
git am ${REPOS}/qemu-boot-time/patches/linux.patch

QEMU

Apply the patches/qemu.patch to your QEMU in order to trace optionrom events

cd ${REPOS}/qemu
git checkout -b benchmark
git am ${REPOS}/qemu-boot-time/patches/qemu.patch

mkdir build-benchmark
cd build-benchmark
../configure --target-list=x86_64-softmmu ...
make

You can use qemu-system-x86_64 -L ${REPOS}/qemu/build-benchmark/pc-bios/optionrom/ ... to use the optionrom patched.

SeaBIOS

Apply the patches/seabios.patch to your SeaBIOS in order to trace bios events

cd ${REPOS}/seabios
git checkout -b benchmark
git am ${REPOS}/qemu-boot-time/patches/seabios.patch

make clean distclean
cp ${REPOS}/qemu/roms/config.seabios-256k .config
make oldnoconfig

You can use qemu-system-x86_64 -bios ${REPOS}/seabios/out/bios.bin ... to use the SeaBIOS image patched.

qboot

qboot already defines trace points, we just need to compile it defining BENCHMARK_HACK

cd ${REPOS}/qboot
make clean
BIOS_CFLAGS="-DBENCHMARK_HACK=1" make

You can use qemu-system-x86_64 -bios ${REPOS}/qboot/bios.bin ... to use the qboot image.

Enable KVM events

The following steps allow perf record to get the kvm trace events:

echo 1 > /sys/kernel/debug/tracing/events/kvm/enable
echo -1 > /proc/sys/kernel/perf_event_paranoid
mount -o remount,mode=755 /sys/kernel/debug
mount -o remount,mode=755 /sys/kernel/debug/tracing

Record the trace events

Start perf record to get the trace events

PERF_DATA="qemu_perf.data"
perf record -a -e kvm:kvm_entry -e kvm:kvm_pio -e sched:sched_process_exec \
            -o $PERF_DATA &
PERF_PID=$!

You can run QEMU multiple times to get also some statistics (Avg/Min/Max)

qemu-system-x86_64 -machine q35,accel=kvm \
                   -bios seabios/out/bios.bin \
                   -L qemu/build-benchmark/pc-bios/optionrom/ \
                   -kernel linux/bzImage ...
qemu-system-x86_64 -machine q35,accel=kvm \
                   -bios seabios/out/bios.bin \
                   -L qemu/build-benchmark/pc-bios/optionrom/ \
                   -kernel linux/bzImage ...
qemu-system-x86_64 -machine q35,accel=kvm \
                   -bios seabios/out/bios.bin \
                   -L qemu/build-benchmark/pc-bios/optionrom/ \
                   -kernel linux/bzImage ...

Stop perf record

kill $PERF_PID

Process the trace recorded using the qemu-perf-script.py

Note: times printed in milliseconds

perf script -s ${REPOS}/qemu-boot-time/perf-script/qemu-perf-script.py -i $PERF_DATA

in trace_begin
sched__sched_process_exec     1 55061.435418353   289738 qemu-system-x86
kvm__kvm_entry           1 55061.466887708   289741 qemu-system-x86
kvm__kvm_pio             1 55061.467070650   289741 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=1

kvm__kvm_pio             1 55061.475818073   289741 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=4

kvm__kvm_pio             1 55061.477168037   289741 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=3

kvm__kvm_pio             1 55061.558779540   289741 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=5

kvm__kvm_pio             1 55061.686849663   289741 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=6

sched__sched_process_exec     4 55067.461869075   289793 qemu-system-x86
kvm__kvm_entry           4 55067.496402472   289796 qemu-system-x86
kvm__kvm_pio             4 55067.496555385   289796 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=1

kvm__kvm_pio             4 55067.505067184   289796 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=4

kvm__kvm_pio             4 55067.506395502   289796 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=3

kvm__kvm_pio             4 55067.584029910   289796 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=5

kvm__kvm_pio             4 55067.704751791   289796 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=6

sched__sched_process_exec     0 55070.073823767   289827 qemu-system-x86
kvm__kvm_entry           0 55070.110507211   289830 qemu-system-x86
kvm__kvm_pio             0 55070.110694645   289830 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=1

kvm__kvm_pio             1 55070.120092692   289830 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=4

kvm__kvm_pio             1 55070.121437922   289830 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=3

kvm__kvm_pio             1 55070.198628779   289830 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=5

kvm__kvm_pio             1 55070.315734630   289830 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=6

in trace_end
Trace qemu-system-x86
1) pid 289738
 qemu_init_end: 31.469355
 fw_start: 31.652297 (+0.182942)
 fw_do_boot: 40.39972 (+8.747423)
 linux_start_boot: 41.749684 (+1.349964)
 linux_start_kernel: 123.361187 (+81.611503)
 linux_start_user: 251.43131 (+128.070123)
2) pid 289793
 qemu_init_end: 34.533397
 fw_start: 34.68631 (+0.152913)
 fw_do_boot: 43.198109 (+8.511799)
 linux_start_boot: 44.526427 (+1.328318)
 linux_start_kernel: 122.160835 (+77.634408)
 linux_start_user: 242.882716 (+120.721881)
3) pid 289827
 qemu_init_end: 36.683444
 fw_start: 36.870878 (+0.187434)
 fw_do_boot: 46.268925 (+9.398047)
 linux_start_boot: 47.614155 (+1.34523)
 linux_start_kernel: 124.805012 (+77.190857)
 linux_start_user: 241.910863 (+117.105851)

Avg
 qemu_init_end: 34.228732
 fw_start: 34.403161 (+0.174429)
 fw_do_boot: 43.288918 (+8.885757)
 linux_start_boot: 44.630088 (+1.34117)
 linux_start_kernel: 123.442344 (+78.812256)
 linux_start_user: 245.408296 (+121.965952)

Min
 qemu_init_end: 31.469355
 fw_start: 31.652297 (+0.182942)
 fw_do_boot: 40.39972 (+8.747423)
 linux_start_boot: 41.749684 (+1.349964)
 linux_start_kernel: 122.160835 (+80.411151)
 linux_start_user: 241.910863 (+119.750028)

Max
 qemu_init_end: 36.683444
 fw_start: 36.870878 (+0.187434)
 fw_do_boot: 46.268925 (+9.398047)
 linux_start_boot: 47.614155 (+1.34523)
 linux_start_kernel: 124.805012 (+77.190857)
 linux_start_user: 242.882716 (+118.077704)

QEMU 4.0 boots uncompressed Linux x86_64 kernel

sgarzare@redhat.com (Stefano Garzarella) — Fri, 23 Aug 2019 15:26:54 +0200

QEMU 4.0 is now able to boot directly into the uncompressed Linux x86_64 kernel binary with minimal firmware involvement using the PVH entry point defined in the x86/HVM direct boot ABI. (CONFIG_PVH=y must be enabled in the Linux config file).

The x86/HVM direct boot ABI was initially developed for Xen guests, but with latest changes in both QEMU and Linux, QEMU is able to use that same entry point for booting KVM guests.

Prerequisites

QEMU >= 4.0
Linux kernel >= 4.21
- CONFIG_PVH=y enabled
- vmlinux uncompressed image

How to use

To boot the PVH kernel image, you can use the -kernel parameter specifying the path to the vmlinux image.

qemu-system-x86_64 -machine q35,accel=kvm \
    -kernel /path/to/vmlinux \
    -drive file=/path/to/rootfs.ext2,if=virtio,format=raw \
    -append 'root=/dev/vda console=ttyS0' -vga none -display none \
    -serial mon:stdio

The -initrd and -append parameters are also supported as for compressed images.

Details

QEMU will automatically recognize if the vmlinux image has the PVH entry point and it will use SeaBIOS with the new pvh.bin optionrom to load the uncompressed image into the guest VM.

As an alternative, qboot can be used to load the PVH image.

Performance

Perf script and patches used to measure boot time: https://github.com/stefano-garzarella/qemu-boot-time.

The following values are expressed in milliseconds [ms]

QEMU (q35 machine) + SeaBIOS + bzImage
- qemu_init_end: 36.072056
- linux_start_kernel: 114.669522 (+78.597466)
- linux_start_user: 191.748567 (+77.079045)
QEMU (q35 machine) + SeaBIOS + vmlinux(PVH)
- qemu_init_end: 51.588200
- linux_start_kernel: 62.124665 (+10.536465)
- linux_start_user: 139.460582 (+77.335917)
QEMU (q35 machine) + qboot + bzImage
- qemu_init_end: 36.443638
- linux_start_kernel: 106.73115 (+70.287512)
- linux_start_user: 184.575531 (+77.844381)
QEMU (q35 machine) + qboot + vmlinux(PVH)
- qemu_init_end: 51.877656
- linux_start_kernel: 56.710735 (+4.833079)
- linux_start_user: 133.808972 (+77.098237)
Tracepoints:
- qemu_init_end: first kvm_entry (i.e. QEMU initialization has finished)
- linux_start_kernel: first entry of the Linux kernel (start_kernel())
- linux_start_user: before starting the init process

Patches

Linux patches merged upstream in Linux 4.21:

https://lkml.org/lkml/2018/12/14/1330

QEMU patches merged upstream in QEMU 4.0:

qboot patches merged upstream:

iperf3-vsock: how to measure VSOCK performance

sgarzare@redhat.com (Stefano Garzarella) — Thu, 22 Aug 2019 17:52:06 +0200

The iperf-vsock repository contains few patches to add the support of VSOCK address family to iperf3. In this way iperf3 can be used to measure the performance between guest and host using VSOCK sockets.

The VSOCK address family facilitates communication between virtual machines and the host they are running on.

To test VSOCK sockets (only Linux), you must use the new option --vsock on both server and client. Other iperf3 options (e.g. -t, -l, -P, -R, --bidir) are well supported by VSOCK tests.

Prerequisites

Linux host kernel >= 4.8
Linux guest kernel >= 4.8
QEMU >= 2.8

Build iperf3-vsock from source

Clone repository

git clone https://github.com/stefano-garzarella/iperf-vsock
cd iperf-vsock

Building

mkdir build
cd build
../configure
make

(Note: If configure fails, try running ./bootstrap.sh first)

Example with Fedora 30 (host and guest):

Host: start the VM

GUEST_CID=3
sudo modprobe vhost_vsock
sudo qemu-system-x86_64 -m 1G -smp 2 -cpu host -M accel=kvm	\
     -drive if=virtio,file=/path/to/fedora.img,format=qcow2     \
     -device vhost-vsock-pci,guest-cid=${GUEST_CID}

Guest: start iperf server

# SELinux can block you, so you can write a policy or temporally disable it
sudo setenforce 0
iperf-vsock/build/src/iperf3 --vsock -s

Host: start iperf client

iperf-vsock/build/src/iperf3 --vsock -c ${GUEST_CID}

Output

Connecting to host 3, port 5201
[  5] local 2 port 4008596529 connected to 3 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.30 GBytes  11.2 Gbits/sec
[  5]   1.00-2.00   sec  1.67 GBytes  14.3 Gbits/sec
[  5]   2.00-3.00   sec  1.57 GBytes  13.5 Gbits/sec
[  5]   3.00-4.00   sec  1.49 GBytes  12.8 Gbits/sec
[  5]   4.00-5.00   sec   971 MBytes  8.15 Gbits/sec
[  5]   5.00-6.00   sec  1.01 GBytes  8.71 Gbits/sec
[  5]   6.00-7.00   sec  1.44 GBytes  12.3 Gbits/sec
[  5]   7.00-8.00   sec  1.62 GBytes  13.9 Gbits/sec
[  5]   8.00-9.00   sec  1.61 GBytes  13.8 Gbits/sec
[  5]   9.00-10.00  sec  1.63 GBytes  14.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  14.3 GBytes  12.3 Gbits/sec                  sender
[  5]   0.00-10.00  sec  14.3 GBytes  12.3 Gbits/sec                  receiver

iperf Done.