Concepts

• DSCP / dot11p Tag: A value embedded in the packet header.
• Buffer: Receive buffer (SRAM) on switches partitioned into lossy pool and lossless pool.
• Traffic Class (TC): An intermediate value
• Priority Group (PG): Receive queue on switches.
• Queue: Send queue on switches.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

                  Map             =               
                 ┌────► Queue ID ───► PFC Priority
          Map    │      (Egress)      of RX PAUSE 
DSCP Tag ─────► TC                                
                 │                =               
                 └────► PG ID ──────► PFC Priority
                  Map   (Ingress)     of TX PAUSE 
                           │                      
                           │ Bind                 
                           ├──────► Lossless      
                           │        Buffer        
                           │ Bind                 
                           └──────► Lossy         
                                    Buffer        
                                                  
Fig. Numerical Relationship between terminologies 
for Intel Tofino 2 on EdgeCore SONiC OS           

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

              ┌───────────────────────────────────────────────────────────────────────┐                          
              │ Switch                                                                │                          
┌─────────┐   │   ┌─────────┐   ┌──────────┐   ┌────────┐   ┌─────────┐   ┌───────┐   │                          
│ Packet  │   │   │   PG0   │   │  Lossy   │   │        │   │ Queue0  │   │       │   │                          
│ DSCP=0  ├───┼──►│ of Eth0 ├──►│  Buffer  ├──►│        ├──►│ of Eth8 ├──►│       │   │   ┌─────────┐ ┌─────────┐
└─────────┘   │   └─────────┘   └──────────┘   │  XBAR  │   └─────────┘   │ Sche  │   │   │ Packet  │ │ Packet  │
              │       ...                      │ Switch │       ...       │ duler ├───┼──►│ DSCP=0  │ │ DSCP=26 │
┌─────────┐   │   ┌─────────┐   ┌──────────┐   │        │   ┌─────────┐   │       │   │   └─────────┘ └─────────┘
│ Packet  ├───┼──►│   PG3   ├──►│ Lossless ├──►│        ├──►│ Queue3  ├──►│       │   │                          
│ DSCP=26 │   │   │ of Eth0 │   │  Buffer  │   │        │   │ of Eth8 │   │       │   │                          
└─────────┘   │   └─────────┘   └──────────┘   └────────┘   └─────────┘   └───────┘   │                          
              │                                                                       │                          
              └───────────────────────────────────────────────────────────────────────┘                          
                                                                                                                 
                 Fig. Ingress/Egress procedure of Intel Tofino 2 on EdgeCore SONiC OS                            

(Optional) Factory Reset Configuration

If you would like to drop your modification to the system, try these commands.

Normal

1
2
3

sudo rm /etc/sonic/config_db.json
sudo config-setup factory
sudo config reload -y

Force

If the system has already been trapped in a strange state, these commands might be able to do a force reset.

1
2
3
4

sudo rm /etc/sonic/config_db.json
sudo config-setup factory
sudo config reload -y -f
sudo service swss restart

(Optional) Set up Ports

If you have trouble enabling L2 forwarding, maybe there is a mismatch in the port speed or FEC configuration.

Check Status of NICs

1

sudo mlxlink -d mlx5_0 -m -c -e

If State is not Active, there might be something wrong with the port configuration or physical connection.

Enable Auto-Negotiation on NICs

1

sudo ethtool -s enp216s0f0np0 autoneg on

Disable Auto-Negotiation on Switch

Disable Auto-Negotition when it does not work as you expect. If it works well, just leave it alone.

1
2

sudo config int autoneg Ethernet0 disabled
sudo config int autoneg Ethernet8 disabled

Force 100Gb Port Speed

Change the port speed as you wish. The supported port speed varies depending on switches and NICs.

1
2

sudo config interface breakout Ethernet0 '1x100G[40G](4)'
sudo config interface breakout Ethernet8 '1x100G[40G](4)'

(Optional) Configure FEC (Forward Error Correction) to RS mode

Setting FEC to none is also fine. Just make sure switches and NICs use the same mode.

1
2

sudo config interface fec Ethernet0 rs
sudo config interface fec Ethernet8 rs

Bring up Ports

1

sudo config interface startup Ethernet0-8

(Optional) Enable L2 Forwarding

By default, L2 switching is disabled on SONiC OS, unlike other switches.

1
2
3
4

sudo config vlan add 1000
sudo config vlan member add -u 1000 Ethernet0
sudo config vlan member add -u 1000 Ethernet8
sudo config inter ip add Vlan1000 10.200.0.1/24

Enable L3 PFC

Since Intel Tofino 2 only supports 5 PGs, we decided to use only 5 TCs/PGs/Queues and then do 1:1 mapping between PGs/Queues and TCs.

Load Buffer Profile

1
2
3

sudo config qos reload
sudo config save -y
sudo reboot

Set and Apply DSCP-to-TC Table for Ports

1
2
3
4
5
6
7
8

sudo config qos dscp-tc add dscp-tc-prof --dscp 0-15 --tc 1
sudo config qos dscp-tc update dscp-tc-prof --dscp 16-23 --tc 2
sudo config qos dscp-tc update dscp-tc-prof --dscp 24-31 --tc 3
sudo config qos dscp-tc update dscp-tc-prof --dscp 32-39 --tc 4
sudo config qos dscp-tc update dscp-tc-prof --dscp 40-63 --tc 5

sudo config interface qos dscp-tc bind Ethernet0 dscp-tc-prof
sudo config interface qos dscp-tc bind Ethernet8 dscp-tc-prof

Set and Apply TC-to-Queue Table for Ports

1
2
3
4
5

sudo config qos tc-queue add tc-queue-prof --tc 1 --queue 1
sudo config qos tc-queue update tc-queue-prof --tc 2 --queue 2
sudo config qos tc-queue update tc-queue-prof --tc 3 --queue 3
sudo config qos tc-queue update tc-queue-prof --tc 4 --queue 4
sudo config qos tc-queue update tc-queue-prof --tc 5 --queue 5

Set and Apply TC-to-PG (Priority Group) Table

1
2
3
4
5

sudo config qos tc-pg add tc-pg-prof --tc 1 --pg 1
sudo config qos tc-pg update tc-pg-prof --tc 2 --pg 2
sudo config qos tc-pg update tc-pg-prof --tc 3 --pg 3
sudo config qos tc-pg update tc-pg-prof --tc 4 --pg 4
sudo config qos tc-pg update tc-pg-prof --tc 5 --pg 5

Specify Queue Scheduler (ETS)

1
2
3
4
5

sudo config scheduler add sched-dwrr-100 --sched_type DWRR --weight 100
sudo config scheduler add sched-strict --sched_type STRICT

sudo config interface scheduler bind queue Ethernet0 3 sched-dwrr-100
sudo config interface scheduler bind queue Ethernet8 3 sched-dwrr-100

Bind Lossless Buffer Profile

1
2
3
4
5

sudo config interface buffer bind priority-group Ethernet0 3 ingress_lossless_profile
sudo config interface buffer bind priority-group Ethernet8 3 ingress_lossless_profile

sudo config interface buffer bind queue Ethernet0 3 egress_lossless_profile
sudo config interface buffer bind queue Ethernet8 3 egress_lossless_profile

Enable PFC for Ports

1
2

sudo config interface pfc priority Ethernet0 3 on
sudo config interface pfc priority Ethernet8 3 on

Enable ECN

Weighted random early detection (WRED) was initially purposed to randomly drop packets to signal the sender’s congestion control algorithm to slow the sending rate. When WRED works in ECN mode, it will randomly put ECN marks into forwarded packets instead of simply dropping them, and the possibility depends on the buffer usage.

1
2
3
4

sudo config wred add wred-prof --mode ecn --gmin 1048576 --gmax 2097152 --gdrop 5

sudo config interface wred bind queue Ethernet0 3 wred-prof
sudo config interface wred bind queue Ethernet8 3 wred-prof

References

• Edgecore SONiC User Guide (Authentication is required)
• https://support.edge-core.com/hc/en-us/articles/900000210426--Enterprise-SONiC-VLAN-Inter-VLAN-Routing
• https://support.edge-core.com/hc/en-us/articles/900006399883--Enterprise-SONiC-Reset-default-configuration
• https://support.edge-core.com/hc/en-us/articles/900000200743--Enterprise-SONiC-Switch-Port-Attributes
• https://github.com/sonic-net/SONiC/issues/339
• https://github.com/sonic-net/SONiC/issues/181
• https://github.com/sonic-net/SONiC/issues/28
• https://github.com/sonic-net/SONiC/wiki/technical_faq
• https://github.com/sonic-net/SONiC/wiki/L2-Switch-mode
• https://github.com/sonic-net/SONiC/wiki/Configuration-with-Minigraph-(~Sep-2017)
• https://github.com/sonic-net/sonic-buildimage/issues/9470
• https://github.com/sonic-net/sonic-buildimage/issues/5947
• https://lfnetworking.org/wp-content/uploads/sites/7/2022/06/Tech-Talk5-How_does_SONiC_Operate_on_a_Programmable_Switch_Edgecore_Howard-Hsu.pdf
• https://www.supermicro.com/manuals/network/Supermicro_Datacenter_SONiC_Configuration_Guide.pdf
• https://www.juniper.net/documentation/en_US/release-independent/nce/topics/concept/sonic-feature-config.html
• https://netbergtw.com/top-support/netberg-sonic/switching-silicon-shell/
• https://gitlab.tongyuejun.cn/zhangjx/p4_doc/-/blob/0c64dcd7e21a0815ca9da2d10d9659002cbb4590/Wedge100BF_User_Manual.org
• https://www.infoq.cn/article/kgucyp5s8m7hecl6qiel

Prerequisite

This method only applies to Ubuntu 22.04 (or higher version, maybe), and you should be able to create TUN/TAP devices. For Ubuntu 20.04 or older, the official repo doesn't provide WireGuard-Go.

Note that if you are allowed to load kernel modules, for example, you are not setting up a VPN inside OpenVZ or Docker Containers, you may consider Kernel Space WireGuard instead of User Space WireGuard (e.g., WireGuard-Go introduced in this post).

Step 1. Install Dependencies

1
2

sudo apt update
sudo apt install wireguard-go wireguard-tools

Note: The package wireguard should NOT be installed. If you have installed it, just remove it.

Step 2. Create Soft Link to Binary File

Surprisingly, the binary file installed by apt is named wireguard instead of wireguard-go, which is not recognized by wg-quick. So the workaround here is just simply creating a soft link to wireguard file.

1
2

cd /bin
sudo ln -s wireguard wireguard-go

Step 3. Launch WireGuard-Go

The tutorial about writing the configuration file of wg-quick is omitted here since we can use the same configuration for both Kernel Space and User Space WireGuard, and we can launch WireGuard-Go in the same way as Kernel Space WireGuard using wg-quick.

1
2

# Configuration file is written to /etc/wireguard/wg0.conf
sudo wg-quick up wg0

(Optional) Step 4. Run WireGuard-Go on Startup

1

sudo systemctl enable wg-quick@wg0

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

I know what you're gonna say: Nvidia F*** You!

Causes

Actually, I encountered this problem many times. It is most likely caused by upgrading or downgrading the Linux kernel without properly generating kernel modules, which might be essential parts of GPU drivers.

Steps

Check System Status

Right now nvidia module is supposed not to be loaded (Could check with lsmod | grep nvidia). We could try to load the kernel manually.

1

sudo modprobe nvidia

You should get an error message like this.

modprobe: FATAL: Module nvidia not found in directory /lib/modules/5.15.0-84-generic

Meanwhile, check whether /usr/lib/modules/5.15.0-84-generic/updates/dkms/nvidia.ko is missing.

If you don’t see the error message above and that kernel module file does exist, you might have other issues, such as hardware failure. At this time, try to read kernel logs through dmesg and check the existence of GPUs through lspci -vvv, which should give you some clues.

Reinstall DKMS and NVIDIA Drivers

DKMS, a utility that manages drivers, as well as NVIDIA Drivers, might be broken. We could fix them by removing them first and installing them back later.

1
2
3
4
5
6
7
8
9
10

sudo rm -r /var/lib/dkms/nvidia
sudo apt install --reinstall dkms
sudo apt autoremove --purge nvidia* cuda*

# Please refer to https://developer.nvidia.com/cuda-downloads to install the latest CUDA toolkits
# Commands below work only on Ubuntu 22.04
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install cuda

Note: Installing the full CUDA Toolkits is the only way I recommend to install drivers. Using the drivers provided by the Ubuntu official repo is NOT recommended.

(Optional) Reinstall NVIDIA Docker Runtime

The previous step will also remove NVIDIA Docker Runtime, which may lead to this error if you use Docker.

Error response from daemon: Cannot restart container ...: could not select device driver "" with capabilities: [[gpu]]

Thus, we need to install it back too.

1

sudo apt install -y nvidia-container-toolkit

References

• https://askubuntu.com/questions/1122512/nvidia-modules-are-missing-after-each-kernel-upgrade-in-18-04
• https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

(Updated on Mar 31, 2024) You may not need PFC for modern NICs (ConnectX-6 or newer), which support NVIDIA RTTCC congestion control that doesn't rely on PFC and ECN.

Discussion

L2 PFC v.s. L3 PFC

It is recommended to set up L3 PFC rather than L2 PFC nowadays. Probably, it is because setting up L3 PFC is easier.

Note: 802.1p, pcp could refer to L2 PFC; dscp could refer to L3 PFC.

Terminologies

• Type of Service (ToS): A value to distinguish applications. RoCE is one of the applications.
• DSCP / dot11p Tag: A value embedded in the packet header.
• Buffer: Receive buffer (SRAM) on NICs. Can be partitioned into multiple regions.
• Traffic Class (TC): An intermediate value between PFC Priority and Queue ID.
• Queue: Send queue on NICs.
• Note: Service Level is a concept in Infiniband networks, not related to RoCE networks.

1
2
3
4
5
6
7
8
9
10

     >>2             Map                 Map       
TOS ─────► DSCP Tag ─────► PFC Priority ─────► TC  
                                 │             │   
                                 │ Map         │ = 
                                 │             │   
                                 ▼             ▼   
                              BufferID      QueueID
                                                   
 Fig. Numerical Relationship between terminologies 
 for Mellanox ConnectX NICs.                       

1
2
3
4
5
6
7
8
9
10
11
12
13

               ┌─────────────────────────────┐                          
┌──────────┐   │ NIC                         │                          
│ TCP App  │   │   ┌────────┐   ┌───────┐    │                          
│ ToS=0    ├───┼──►│ Queue0 ├──►│       │    │   ┌─────────┐ ┌─────────┐
└──────────┘   │   └────────┘   │ Sche  │    │   │ Packet  │ │ Packet  │
               │       ...      │ duler ├────┼──►│ DSCP=0  │ │ DSCP=26 │
┌──────────┐   │   ┌────────┐   │       │    │   └─────────┘ └─────────┘
│ RDMA App ├───┼──►│ Queue3 ├──►│       │    │                          
│ ToS=106  │   │   └────────┘   └───────┘    │                          
└──────────┘   │                             │                          
               └─────────────────────────────┘                          
                                                                        
  Fig. Egress procedure of Mellanox ConnectX NICs.                      

1
2
3
4
5
6
7
8
9
10
11
12
13

              ┌──────────────────────────────┐   ┌────────┐
┌─────────┐   │ NIC                          │   │ System │
│ Packet  │   │   ┌─────────┐   ┌────────┐   │   │ Memory │
│ DSCP=0  ├───┼──►│ Buffer0 ├──►│        │   │   │        │
└─────────┘   │   └─────────┘   │  DMA   │   │   │        │
              │                 │ Engine ├───┼──►│        │
┌─────────┐   │   ┌─────────┐   │        │   │   │        │
│ Packet  ├───┼──►│ Buffer1 ├──►│        │   │   │        │
│ DSCP=26 │   │   └─────────┘   └────────┘   │   │        │
└─────────┘   │                              │   │        │
              └──────────────────────────────┘   └────────┘
                                                           
   Fig. Ingress procedure of Mellanox ConnectX NICs.       

Prerequisite

If network traffic goes through network switches, ensure L3 PFC and ECN are enabled on switches. We omitted the detailed steps to configure switches here as they depend on the particular switch SKU and OS.

Note: we could set up a direct connection to test whether our configuration on NICs works or not.

Steps

Enable DCQCN

DCQCN is enabled by default on Mellanox ConnextX NICs. But, we need to ensure ECN marking is enabled on all network switches.

Identify Interface Name and Device Name

We could check the interface and device name with show_gids, which should output something like below.

1
2
3

DEVPORTINDEXGIDIPv4  VERDEV
---------------------------  ------
mlx5_1130000:0000:0000:0000:0000:ffff:0ac8:000e10.200.0.14  v2enp216s0f1np1

Here mlx5_1 is the device name that is usually used to describe RDMA devices, and enp216s0f1np1 is the interface name that can be managed by Linux as an Ethernet interface. We could save them as environment variables.

1
2

export IF_NAME=enp216s0f1np1
export DEV_NAME=mlx5_1

Tune PFC Headroom Size

We should reserve enough buffer space to store on-the-fly packets since senders (or requestors) need to take time (latency) to respond to PFC pauses. The latency will be affected by the cable length (a.k.a. propagation delay). Mellanox requires us to set the cable length manually, and then the NICs will automatically calculate the correct headroom size.

Fortunately, the cable length is recorded in the transceiver’s EEPROM.

1

sudo mlxlink -d $DEV_NAME -m -c -e

From the outputs, we could find the cable length.

1
2
3
4
5
6
7
8
9
10
11
12
13

Module Info
-----------
Identifier                      : QSFP28
Compliance                      : 100GBASE-SR4 or 25GBASE-SR
Cable Technology                : 850 nm VCSEL
Cable Type                      : Optical Module (separated)
OUI                             : Other
Vendor Name                     : ...
Vendor Part Number              : ...
Vendor Serial Number            : ...
Rev                             : A0
Wavelength [nm]                 : 850
Transfer Distance [m]           : 50 # here

Then, apply this parameter to our QoS setting.

1

sudo mlnx_qos -i $IF_NAME --cable_len=50

Enable L3 PFC

Execute the following commands to activate PFC and apply the PFC setting to RoCE traffic. Note: the configuration is NOT persistent. You may need to re-run all the commands above to enable PFC each time the machine reboots. Also, ensure PFC is enabled for DSCP 26 traffic on all network switches.

(Updated on Mar 31, 2024) You may need to use a different PFC priority or DSCP value depending on the configuration/restriction of network switches. Conventionally, we use Priority 3 for lossless networks.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# use L3 PFC, default=pcp (L2 PFC)
sudo mlnx_qos -i $IF_NAME --trust dscp

# enable PFC on PFC Priority 3
sudo mlnx_qos -i $IF_NAME --pfc 0,0,0,1,0,0,0,0

# clear Traffic Class (TC) settings
echo "tclass=-1" | sudo tee /sys/class/infiniband/$DEV_NAME/tc/1/traffic_class

# set default ToS (= DSCP value * 4) for RoCE traffic
echo 106 | sudo tee /sys/class/infiniband/$DEV_NAME/tc/1/traffic_class

# set default ToS for RoCE traffic
sudo cma_roce_tos -d $DEV_NAME -t 106

Verification

Show PFC Setting

1

sudo mlnx_qos -i $IF_NAME

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

DCBX mode: OS controlled
Priority trust state: dscp
dscp2prio mapping:
        prio:0 dscp:07,06,05,04,03,02,01,00,
        prio:1 dscp:15,14,13,12,11,10,09,08,
        prio:2 dscp:23,22,21,20,19,18,17,16,
        prio:3 dscp:31,30,29,28,27,26,25,24,
        prio:4 dscp:39,38,37,36,35,34,33,32,
        prio:5 dscp:47,46,45,44,43,42,41,40,
        prio:6 dscp:55,54,53,52,51,50,49,48,
        prio:7 dscp:63,62,61,60,59,58,57,56,
default priority:
Receive buffer size (bytes): 20016,156096,0,0,0,0,0,0,total_size=1027728
Cable len: 7
PFC configuration:
        priority    0   1   2   3   4   5   6   7
        enabled     0   0   0   1   0   0   0   0
        buffer      0   0   0   1   0   0   0   0
tc: 1 ratelimit: unlimited, tsa: vendor
         priority:  0
tc: 0 ratelimit: unlimited, tsa: vendor
         priority:  1
tc: 2 ratelimit: unlimited, tsa: vendor
         priority:  2
tc: 3 ratelimit: unlimited, tsa: vendor
         priority:  3
tc: 4 ratelimit: unlimited, tsa: vendor
         priority:  4
tc: 5 ratelimit: unlimited, tsa: vendor
         priority:  5
tc: 6 ratelimit: unlimited, tsa: vendor
         priority:  6
tc: 7 ratelimit: unlimited, tsa: vendor
         priority:  7

1
2
3
4
5
6
7

         >>2            Map               Map       
TOS 106 ─────► DSCP 26 ─────► PFC Prio 3 ─────► TC 3
                                    │           │   
                                    │ Map       │ = 
                                    │           │   
                                    ▼           ▼   
                                 Buffer 1    Queue 3

Check DCQCN is functioning

1
2
3
4
5
6
7
8

# Check DCQCN is enabled on Prio 3
cat /sys/class/net/$IF_NAME/ecn/roce_np/enable/3
cat /sys/class/net/$IF_NAME/ecn/roce_rp/enable/3

# Check counters related to DCQCN
cat /sys/class/infiniband/$DEV_NAME/ports/1/hw_counters/np_cnp_sent
cat /sys/class/infiniband/$DEV_NAME/ports/1/hw_counters/np_ecn_marked_roce_packets
cat /sys/class/infiniband/$DEV_NAME/ports/1/hw_counters/rp_cnp_handled

Note: Two cases triggering NP to send CNP packets:

• NP’s NIC receives a packet with an ECN mark (marked by the switch indicating the switch’s buffer is about to be out of capacity).
• NP’s NIC receives an out-of-order packet (packet loss occurred).

Check PFC is functioning

1

ethtool -S $IF_NAME | grep prio3

1
2
3
4
5
6
7
8
9
10

rx_prio3_bytes: 462536457742
rx_prio3_packets: 500098087
rx_prio3_discards: 0
tx_prio3_bytes: 1180912512618
tx_prio3_packets: 1155032777
rx_prio3_pause: 214479
rx_prio3_pause_duration: 213496
tx_prio3_pause: 12
tx_prio3_pause_duration: 13
rx_prio3_pause_transition: 107222

Note: tx_prio3_pause refers to the number of PFC pauses sent from this NIC as the server cannot absorb the network traffic quickly.

Miscellaneous

Advanced QoS Settings

The default values of the parameters below should be fine…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# mapping Priority to TC
sudo mlnx_qos -i $IF_NAME --prio_tc=0,1,2,3,4,5,6,7

# mapping Priority to receive buffers
sudo mlnx_qos -i $IF_NAME --prio2buffer=0,0,0,1,0,0,0,0

# adjust receive buffer sizes
sudo mlnx_qos -i $IF_NAME --buffer_size=20016,156096,0,0,0,0,0,0

## Set alternative TSA
# set to vendor (default)
sudo mlnx_qos -i $IF_NAME --tsa=vendor,vendor,vendor,vendor,vendor,vendor,vendor,vendor
# set to ets
sudo mlnx_qos -i $IF_NAME --tsa=ets,ets,ets,ets,ets,ets,ets,ets --tcbw=0,0,0,100,0,0,0,0

# Remap DSCP value to specified PFC Priority
# Map DSCP 3 to Prio 3
sudo mlnx_qos -i $IF_NAME --dcsp2prio='set,3,3'

Performance Counters on NICs

Refer to this: https://enterprise-support.nvidia.com/s/article/understanding-mlx5-linux-counters-and-status-parameters.

Failed to map RoCE traffic to specified PFC Priority

• Consider upgrading OFED drivers.
• ibv_qp_attr.ah_attr.grh.traffic_class may override default ToS Value.

Configuration for BlueField DPU

QoS settings could be set on the host using mlnx_qos, but Traffic Class values must be set on the DPU side individually.

References

• https://baiwei0427.github.io/papers/lumina-sigcomm2023.pdf
  • Many thanks to Wei Bai for helping us configure our networks!
• https://docs.nvidia.com/networking/pages/viewpage.action?pageId=39264632
• https://enterprise-support.nvidia.com/s/article/lossless-roce-configuration-for-linux-drivers-in-dscp-based-qos-mode
• https://enterprise-support.nvidia.com/s/article/understanding-mlx5-linux-counters-and-status-parameters
• https://support.huawei.com/enterprise/zh/doc/EDOC1100197616/3dfff4ec
• https://www.rdmamojo.com/2013/01/12/ibv_modify_qp/
• https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p523.pdf

TL;DR

• Select and Copy the object to export
• Click Edit -> Paste Special -> PDF
• Click Save as Picture in the right-click menu to export the newly created object

This method is working on PowerPoint 16.72

Prerequisite

Install RNIC Drivers and Toolkits

In this tutorial, I will use Mellanox ConnextX RDMA NIC (RNIC) as an example to demonstrate configuration steps.

Note that some configuration steps are vendor-specific, which means for different vendor's RNIC, you may need to find the alternative solution if my approach is not applicable for your RNIC. Also I didn't test GDR on a NIC made by vendors other than Mellanox. (I suspect only Mellanox's RNIC supports GDR).

For ConnextX RNIC, the corresponding drivers and toolkits are all packed in Mellanox OFED.

Install CUDA Drivers and Toolkits

I won't introduce how to install these things as there are already many tutorials about this topic on Internet. I would recommend to check the official website, and install the packages this website provides.

Note that installing CUDA Drivers through apt and Toolkits through conda separately is NOT recommended.

Verify Installation

Once you properly installed them, just execute the command nvidia-smi topo -m and you should see something like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

$ nvidia-smi topo -m
GPU0GPU1GPU2GPU3NIC0NIC1CPU AffinityNUMA Affinity
GPU0 X NV4NV4NV4SYSSYS0-127N/A
GPU1NV4 X NV4NV4SYSSYS0-127N/A
GPU2NV4NV4 X NV4PHBPHB0-127N/A
GPU3NV4NV4NV4 X SYSSYS0-127N/A
NIC0SYSSYSPHBSYS X PIX
NIC1SYSSYSPHBSYSPIX X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1

We will discuss what this output represents in the following section. For now, you should be able to identify both your GPUs and NICs from this output.

Check GDR Hardware Support

Every system is not created equal. Continue the above example, we can see there are many types of relationship between individual GPU and NIC such as SYS and PHB. In fact, they will greatly affect the GDR performance.

From my experience, I believe:

• GDR Performance is good: PIX, PXB
• GDR Performance is likely to be good: PHB
  • Might be as good as PIX and PXB
• GDR Performance is bad: SYS, NODE

My benchmark result of GDR performance with 100 Gbps RoCE Network:

• Dual Intel Xeon 4112 + NVIDIA Tesla V100
  • SYS: ~2 GB/s
  • PIX: ~10 GB/s
• Single AMD EPYC 7763 + NVIDIA Tesla A100
  • SYS: ~6 GB/s
  • PHB: ~10 GB/s
• Dual Intel Xeon E5-2630 v4 + NVIDIA Tesla P100
  • SYS: ~0.3 GB/s

Here are discussion about PHB and description about P2P Level.

(Updated on Jun 9, 2023) Here is a systematic introduction to PCIe Affinity.

If you unfortunately got some SYS or NODE, this relationship can be possibly corrected by plugging your GPU or NIC into proper PCIe slots.

• For multi-socket system, some certain PCIe slots are physically connected to one CPU socket (package) while the other slots are connected to other sockets.
• For system with chiplet-based CPU (e.g., AMD EPYC), some certain PCIe might be physically connected to one chipet.
  • That is still the case for AMD EPYC 7003 Series which has a separate I/O die.

Load NVIDIA Peer Memory Kernel Module

nvidia_peermem module is bundled in CUDA Toolkit downloaded from here. By default, this kernel module will be not loaded automatically. Thus, we could manually load this module with the command sudo modprobe nvidia_peermem.

To check this module is loaded correctly, execute the command lsmod | grep nvidia_peermem and see if this module name exists in the output.

If you cannot find this kernel module in the system, you might consider to install the latest CUDA Toolkit.

There is another version of this module that you can find on this repo, and it is called nv_peer_mem instead. But it appears to be no longer maintained.

Disable PCIe ACS

Many reports ([1], [2]) have mentioned PCIe ACS may hurt the GDR performance. PCIe ACS is a security feature, but we never care about the security when we are hungry for performance. Here is the script to disable it. Note that this script is only for your reference. You may need to modify the content according to your machine's configuration.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#!/bin/bash

# Author: OmniReduce Team
# This script applies what was mentioned in:
# https://forums.developer.nvidia.com/t/multi-gpu-peer-to-peer-access-failing-on-tesla-k80/39748/9
# To disable PCIe ACS

echo "Before==========================="
sudo lspci -vvv | grep -i acsctl
echo "================================="

pcis=$(lspci | grep -i plx | cut -d' ' -f1 | tr '\r\n' ' ')
echo "Disabling ACS on $pcis"

for pci in $pcis
do
setpci -s $pci f2a.w=0000
done

echo "After============================"
sudo lspci -vvv | grep -i acsctl
echo "================================="

echo "Make sure all ACS features are disabled as ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-"

Verification

Up to now, GDR supposed to work. To verify that, I would recommend to use OFED PerfTest.

• DO NOT use PerfTest provided by OFED or APT. You should compile PerfTest by yourself because the binary distribution doesn't support GDR
• Both Client and Server are capable of utilizing GDR
• ib_send_bw has some bugs in GDR tests

1
2
3
4
5
6
7
8
9
10

# Compile
./autogen.sh && ./configure CUDA_H_PATH=/usr/local/cuda/include/cuda.h && make -j

# Launch as server
./ib_write_bw -d ib_dev --use_cuda=<gpu index> -a
./ib_write_bw -d mlx5_0 --use_cuda=0 -a # Run on GPU0 and MLX5_0

# Launch as client
./ib_write_bw -d ib_dev --use_cuda=<gpu index> -a <server ip addr>
./ib_write_bw -d mlx5_0 --use_cuda=0 -a 10.200.0.10 # Run on GPU0 and MLX5_0

If you would like to test with NCCL, it is recommended to refer this article.

Troubleshooting

If you still encounter errors like ibv_create_qp failed or ibv_reg_mr failed, this might be caused by Linux user limits (ulimit).

A fast and dirty way to temporarily fix this issue is to run the program as root user. Once this dirty fix works, you can refer this article to permanently solve this issue.

Makefile 是 GNU Make 这个工具所需的文件，可以看作是一个比较特殊的 Bash 脚本。而对比现代的编译辅助工具，Make 显得非常简陋，直接用这玩意如同在打火机一块钱一个的时代钻木取火，但好处就是这个工具并不是很难理解。

首先要明确的是，Make 这个工具是用来编译比较大型的工程的（就是管理一大堆源代码文件），所以它很多设计是围绕编译展开的，而且它是上世纪八十年代的产物，面向的是当时手无寸铁的程序员，至少比起啥工具都没有的情况，Make 还是有很大作用的。总的来说，Make 其实主要是在做这几件事： - 判断是否可以跳过执行某些 Linux 命令（command） - 确定 Linux 命令执行的顺序，并行执行不相关的命令

基本语法

我们先抛开编译不谈，来看 Make 的规则（Rule）的基本语法。

1
2
3

target: prerequisites
    command
    command

Make 大概是个缝合怪，有很多东西坨在一起了。比如说目标target，就有： - 真target：会对应一个具体的文件 - 假target（Phony Target）：和磁盘上的文件完全没关系

然后执行条件（prerequisites）是一个或者一组目标，这些目标会决定下面的命令（command）要不要被执行。

命令（command）就是 Linux 的命令，一般情况下这些命令写清楚了如何产生一个真target（如果 target 是真 target 的话）。

继续抛开编译不谈，来看make/makefile1里的简单例子，

1
2
3
4
5
6

all: son.bak.txt
@echo I am all # display I am all on screen

son.bak.txt: son.txt
@echo I am son.bak.txt # display I am son.bak.txt on screen
@cp son.txt son.bak.txt # copy son.txt to son.bak.txt

在 command 前加一个@，可以让 Make 不把原始命令显示出来。

这里son.bak.txt是一个真 target，因为真的会有son.bak.txt这么个文件（在执行了cp命令以后）。

son.txt也是一个真 target，并且一开始就有了。

习惯上all是一个假 target，类似于整个 Makefile 的入口（或者说总目标），并且会写成第一个 target（就像这里写在了第一行）。当我们执行make命令的时候（不手动指定 target），其实就是去产生（make）all target。

我们实际上并不用去特别区分真假 target，只需要知道有一些特殊的 target 就行。

什么叫 make 一个 target？其实就是： - 如果存在一个执行条件里的 target 没有被 make 过，就先去 make 执行条件里还没 make 过的 target - 在执行条件里所有的 target 都 make 了以后，在某些时刻，把它的 command 执行一遍

在这个例子里，Make 的故事是： - 我们要 make all target - Make 首先跑去 make all - 然后很快发现son.bak.txt没有 make 过，就跑去 make son.bak.txt - 接着一看son.bak.txt要求要 make son.txt - 但是 - 没有一条关于son.txt的 rule - son.txt已经有这个文件了 - 那我们就当son.txt已经 make 好了 - 执行条件里所有的 target 都好了，再看发现我们没有son.bak.txt这个文件 - 那就执行 command（cp和echo） - command 执行完了，就当son.bak.txt make 好了 - all执行条件里所有的 target 都好了，再看发现我们没有all这个文件 - 那就执行 command（echo）（假 target 永远都要执行一次 command，毕竟对应的文件永远不存在） - command 执行完了，就当all make 好了

程序输出如下：

1
2
3

# make
I am son.bak.txt
I am all

判断是否需要执行命令

Make 的一大作用是，当我们有一大堆代码文件的时候，我们并不希望改动一个文件就要重新编译所有的文件，当然是拎出所有受到的影响的代码来重新编译，这样就可以节约很多时间。所以 Make 会自动选择性的执行 Makefile 里的 command。那什么时候 command 会被执行？

执行条件的 target 有更新

继续上一个例子，当我们在执行一次make命令以后，如果试着再make一次，程序的输出就只有

1
2

# make
I am all

很明显，只有all target 被执行了。在这个例子里，Make 的故事是： - 我们要 make all target - Make 首先跑去 make all - 然后很快发现son.bak.txt没有 make 过，就跑去 make son.bak.txt - 接着一看son.bak.txt要求要 make son.txt - 那我们当son.txt已经 make 好了 - 执行条件里所有的 target 都好了，再看发现我们没有son.bak.txt这个文件 - 我们有son.bak.txt，而且和son.txt一样新 - 那就跳过执行 command - all执行条件里所有的 target 都好了，再看发现我们没有all这个文件 - 那就执行 command（echo）（假 target 永远都要执行一次 command，毕竟对应的文件永远不存在） - command 执行完了，就当all make 好了

如果我们去掉 Makefile 里all的command，再执行make:

1 2 3 4 5

>all: son.bak.txt >son.bak.txt: son.txt echo I am son.bak.txt cp son.txt son.bak.txt

输出就只剩下这些东西。这代表这次make啥事也没做。

1 2	># make >make: Nothing to be done for 'all'.

也不难理解，我们没修改过son.txt，那也没有必要去重新复制一份son.bak.txt。Make 会根据 target 的执行条件里是否有依赖的 target 更新了（执行过 command，或者文件有新修改）来判断是否需要被重新 make 这个 target。

具体来说，Make 会比较son.bak.txt和son.txt的修改时间来判断两文件谁新谁旧。

我们可以编辑一下son.txt，随便写一些东西并保存，再执行make的时候，就会发现 Make 重新复制了一份son.bak.txt。

1
2
3
4

# echo 1 > son.txt # write 1 to file son.txt
# make
I am son.bak.txt
I am all

我们的I am son.bak.txt回来啦，意味着son.bak.txt被重新复制了一份。

再举一个复杂一点的例子，在make/makefile2里，Makefile 的内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14

all: son.txt

son.txt: mom.txt dad.txt
cat mom.txt > son.txt
cat dad.txt >> son.txt # merge the content of mom.txt and dad.txt

mom.txt: grandma.txt
cat grandma.txt > mom.txt # replace the content with grandma.txt

dad.txt: grandpa.txt
cat grandma.txt > dad.txt

clean:
@rm mom.txt dad.txt

同时我们新增一个clean假 target，用来清理 Makefile 产生的中间文件。输入命令make clean即可删掉mom.txt和dad.txt。

1
2
3
4
5
6
7
8
9
10
11
12
13

# make clean
# make
cat grandma.txt > mom.txt # replace the content with grandma.txt
cat grandma.txt > dad.txt
cat mom.txt > son.txt
cat dad.txt >> son.txt # merge the content of mom.txt and dad.txt
# make
make: 'all' is up to date.
# echo 1 > grandma.txt 
# make
cat grandma.txt > mom.txt # replace the content with grandma.txt
cat mom.txt > son.txt
cat dad.txt >> son.txt # merge the content of mom.txt and dad.txt

注意到第一次make之后，如果未对文件作出修改，所有的 target 除all以外会被跳过执行 command。当仅有grandma.txt被修改时，dad.txt这个 target 也会跳过执行 command。期间 Make 的故事是： - make all target - make son.txt - make mom.txt - make grandma.txt - 检查grandma.txt文件是否有新修改 - 有更新，执行cat grandma.txt > mom.txt - make dad.txt - make grandpa.txt - 检查grandma.txt文件是否有新修改 - 没更新，啥也不做 - 检查mom.txt和dad.txt是否执行过 command - mom.txt有执行过 command，执行cat mom.txt > son.txt和cat dad.txt >> son.txt - 检查son.txt是否执行过 command - son.txt有执行过 command，执行空指令

存在依赖关系

显而易见，下面的例子里，aunt.txt和uncle.txt因为和all屁关系没有，从未被依赖过，所以它们的 command 压根就不可能被执行。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

all: son.txt

son.txt: mom.txt dad.txt
cat mom.txt > son.txt
cat dad.txt >> son.txt

mom.txt: grandma.txt
cat grandma.txt > mom.txt

dad.txt: grandpa.txt
cat grandma.txt > dad.txt

aunt.txt: grandma.txt
cat grandma.txt > aunt.txt

uncle.txt: grandpa.txt
cat grandma.txt > uncle.txt

确定命令执行的顺序

Makefile 可以说是文件和命令的依赖关系的说明书，描述了通过什么文件以及什么命令可以产生什么文件。比如在上个例子里，son.txt文件的产生依赖mom.txt和dad.txt，mom.txt和dad.txt的产生分别依赖grandma.txt和grandpa.txt。在真实环境中，为了加速工程的编译速度，几个被依赖的，但它们之间没有依赖关系的 target 的 command 可以被同时执行。这也因此要求我们准确描述依赖关系。如果我们把上个例子的 Makefile 写成：

1
2
3
4
5
6
7
8
9
10
11

all: son.txt mom.txt dad.txt

son.txt:
cat mom.txt > son.txt
cat dad.txt >> son.txt

mom.txt: grandma.txt
cat grandma.txt > mom.txt

dad.txt: grandpa.txt
cat grandma.txt > dad.txt

就有可能造成son.txt，mom.txt，dad.txt的 command 被同时执行，就存在mom.txt文件还没产生，son.txt的命令（需要mom.txt作为输入）就开始执行了的可能性。

所以我们需要确保执行的顺序像下面那个样子。

使用make -j可以启动并行编译，在下面的例子中，我们可以看到grandpa和grandma同时开始执行，son在mom和dad都结束后才开始执行，总耗时为12s。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

all: son
@echo "finish        @" "$(shell date)"

son: mom dad
@echo "son     begin @" "$(shell date)"
@sleep 2
@echo "son     end"

mom: grandma
@echo "mom     begin @" "$(shell date)"
@sleep 3
@echo "mom     end"

dad: grandpa
@echo "dad     begin @" "$(shell date)"
@sleep 5
@echo "dad     end"

grandma:
@echo "grandma begin @" "$(shell date)"
@sleep 3
@echo "grandma end"

grandpa:
@echo "grandpa begin @" "$(shell date)"
@sleep 5
@echo "grandpa end"

1
2
3
4
5
6
7
8
9
10
11
12

# make -j
grandma begin @ Fri Dec 23 23:34:14 UTC 2022
grandpa begin @ Fri Dec 23 23:34:14 UTC 2022
grandma end
mom     begin @ Fri Dec 23 23:34:17 UTC 2022
grandpa end
dad     begin @ Fri Dec 23 23:34:19 UTC 2022
mom     end
dad     end
son     begin @ Fri Dec 23 23:34:24 UTC 2022
son     end
finish        @ Fri Dec 23 23:34:26 UTC 2022

可视化结果如下：

将代码修改成如下形式，可以使所有 target 同时开始执行，总耗时为5s。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

all: son mom dad grandma grandpa
@echo "finish        @" "$(shell date)"

son: 
@echo "son     begin @" "$(shell date)"
@sleep 2
@echo "son     end"

mom: 
@echo "mom     begin @" "$(shell date)"
@sleep 3
@echo "mom     end"

dad: 
@echo "dad     begin @" "$(shell date)"
@sleep 5
@echo "dad     end"

grandma:
@echo "grandma begin @" "$(shell date)"
@sleep 3
@echo "grandma end"

grandpa:
@echo "grandpa begin @" "$(shell date)"
@sleep 5
@echo "grandpa end"

1
2
3
4
5
6
7
8
9
10
11
12

# make -j
son     begin @ Fri Dec 23 23:36:33 UTC 2022
mom     begin @ Fri Dec 23 23:36:33 UTC 2022
dad     begin @ Fri Dec 23 23:36:33 UTC 2022
grandma begin @ Fri Dec 23 23:36:33 UTC 2022
grandpa begin @ Fri Dec 23 23:36:33 UTC 2022
son     end
mom     end
grandma end
dad     end
grandpa end
finish        @ Fri Dec 23 23:36:38 UTC 2022

可视化结果如下：

References

• https://makefiletutorial.com/

Prerequisite

Make sure you have a machine with a public IP address (it could be a VPS), otherwise our method may not be applicable. Let's take exposing the SSH port of a machine behind NAT as an example. There are three roles in total:

• Local Machine: The machine behind NAT firewall with a local IP address, and you want to expose its SSH port 22 to the public network.
• Public Server: The machine with a public IP address (a.b.c.d). Typically you rent it from a public cloud service provider like AWS or Azure. Its port 22 is also open and being used by its own SSH service.
• Your Computer: Another machine you are currently using.

(Recommended) Persistent Method: via Gost with Docker

Before getting started: Install Docker

The first step is to install the Docker on both the local machine and the public server. For Ubuntu 18.04+, I personally prefer to install Docker through apt.

1
2

local-machine&public-server$ sudo apt update
local-machine&public-server$ sudo apt install docker.io -y

(Recommended) Using Websocket + TLS + Gost Relay Protocol

Gost supports nuermous proxy protocols. For reliablity, it is suggested to use a secured protocol to resist the interference by some secure gateways like GFW. Luckily, with the help of Docker, it is very easy to set up a secured tunnel. Another good news is that Docker Daemon will help to monitor the service and automatically restart Gost Service at boot or on failure. Let's say goodbye to the annoying Systemd.

On Public Server

1
2
3

cd ~ # other any other path you like
mkdir gost
docker run --name gost_server --net host -v $(pwd)/gost:/root -id --restart always --entrypoint "" -w "/root" gogost/gost gost -L "relay+wss://<user>:<passwd>@:<gost_service_port>?bind=true"

On Local Machine

1

docker run --name gost_client --net host -id --restart always --entrypoint ""  gogost/gost gost -L rtcp://:<remote_port>/:<local_port> -F "relay+wss://<user>:<passwd>@<pub_server_ip_or_domain>:<gost_service_port>"

Here is the explanation of parameters:

• user / passwd: The username and the password for Gost. They are irrelevant to any other account such as Linux accounts.
• pub_server_ip_or_domain: It could be either the public IP address or the binded domain name. If you would like to use your valid SSL certificate issued by CA, you should only use the domain name here.
• gost_service_port: Could be arbitrary value. It is also fine to set this port number to 443 to pretend as a HTTPS server.
• remote_port: The port that the public server's Gost listens to. The data received from this port will be forwarded to local_port on the local machine. It could be arbitrary value.
• local_port: The port that should associate with some services on the local machine. In our case, it should be 22, the SSH port.

Back to our case, the corresponding commands will be:

1
2
3
4
5
6
7
8
9
10

# On Public Server
cd ~
mkdir gost
docker run --name gost_server --net host -v $(pwd)/gost:/root -id --restart always --entrypoint "" -w "/root" gogost/gost gost -L "relay+wss://user:pass@:1234?bind=true"

# On Local Machine
docker run --name gost_client --net host -id --restart always --entrypoint ""  gogost/gost gost -L rtcp://:2022/:22 -F "relay+wss://user:pass@a.b.c.d:1234" # a.b.c.d is the public IP of the public server

# or
docker run --name gost_client --net host -id --restart always --entrypoint ""  gogost/gost gost -L rtcp://:2022/:22 -F "relay+wss://user:pass@mydomain.com:1234" # mydomain.com can be parsed to a.b.c.d

After that, the SSH port 22 on the local machine should be mapped to the port 2022 on the public server now.

1
2
3
4

# user-local is the name of the user on local machine
your-computer$ ssh user-local@a.b.c.d -p 2022
# or
your-computer$ ssh user-local@mydomain.com -p 2022

Debugging Tips: If Gost is not working properly, we can read the log using the command below.

1	docker logs -f gost_server # or gost_client

(Optional) Using your SSL certificate

By default, Gost will generate a self-signed SSL certificate if the user doesn't specify one. However, this might be considered unsafe as the communication is no longer able to defense the MITM attack. Moreover, some secure gateways may disrupt the TLS session with a self-signed certificate.

The solution is simple. Do you remember the gost directory we created before? We just need to put our certificate there and restart the service. There should be two files in total, cert.pem and key.pem, and their content should start with -----BEGIN CERTIFICATE---- and -----BEGIN RSA PRIVATE KEY----- respectively.

Lastly, the command to restart the service is:

1
2

# On Public Server
docker restart gost_server

Temporary Method: via OpenSSH Client

If you don't want to install any new software, we can also utilize SSH to build a tunnel. This method is also simple (sometimes), but it is not quite reliable. As you might know, a SSH connection can be easily disrupted due to many reasons. In our case, we just need to type this command on the local machine to set up a tunnel:

1
2

local-machine$ ssh -R "[::]:2022:localhost:22" user-public@a.b.c.d
# a.b.c.d is the public IP, user-public is the name of the user on public server

If everything goes well, now you can connect to the local machine through:

1
2

your-computer$ ssh user-local@a.b.c.d -p 2022
# user-local is the name of the user on local machine

However, if you failed to connect to your local machine. Please check the sshd configuration on the public server.

1

public-server$ sudo nano /etc/ssh/sshd_config

Find the following line (for nano editor, press Ctrl-W to search), uncomment, and replaced no with yes.

1
2
3
4
5

# Before
#GatewayPorts no

# After
GatewayPorts yes

Then restart the SSH server.

1

public-server$ sudo service sshd restart

Naming Convention of Nodes

A common cluster should comprise management nodes and compute nodes. This aritcle will take our cluster as an example to demostrate steps to install and configure Slurm. In our case, the management node is called clab-mgt01 while the compute nodes are named from clab01 to clab20 in order.

Install Dependencies

Execute the following command to install the dependencies on all machines. (clab-all refers to all machines including management and compute nodes).

1

clab-all$ sudo apt install slurm-wlm slurm-client munge

Tips: There are several tools that may help to manage multiple nodes easily:

• iTerm2 (on Mac) / Terminator (on Linux)
• csshX (on Mac) / cssh (on Linux)
• Parallel SSH (at cluster side)

Generate Slurm Configuration

There is an official online configuration generator. And we should carefully check the fields below.

• SlurmctldHost: clab-mgt01 in our case.
• NodeName: clab[01-20] in our case.
• CPUs: It is recommended to leave it blank.
• Sockets: For a dual-socket server we commonly see, it should be 2.
• CoresPerSocket: Number of physical cores per socket.
• ThreadsPerCore: For a regular x86 server, if hyperthreading is enabled, it should be 2, otherwise 1.
• RealMemory: Optional.

Click submit, then we could copy the file content to /etc/slurm-llnl/slurm.conf on all machines.

Tips: Don't forget the shared storage (e.g. NFS storage) on the cluster. We could utilize it to distribute files.

Distribute Munge Key

Once Munge is installed successfully, the key /etc/munge/munge.key will be automatically generated. It is requried for all machines to hold the same key. Therefore, we could distribute the key on the management node to the remaining nodes including compute nodes and other backup management node if existing.

Tips: Again. We could also utilize the shared storage to distribute the key.

Then make sure the permission and the ownership are correctly set.

1
2

clab-all$ sudo chmod 400 /etc/munge/munge.key
clab-all$ chown munge:munge /etc/munge/munge.key

Patch Slurm Cgroup Integration

By default, there Slurm cannot work with Cgroup well. If we start Slurm service right now, we may receive this error shown below.

1

error: cgroup namespace 'freezer' not mounted. aborting

Therefore, by pasting the following content to /etc/slurm/cgroup.conf on compute nodes, this issue can be fixed.

1

CgroupMountpoint=/sys/fs/cgroup

or using this command:

1

echo CgroupMountpoint=/sys/fs/cgroup >> /etc/slurm/cgroup.conf

Fix Directory Permission

For unknown reasons, the permission of the relevant directory is not set properly, which may lead to this error.

1

slurmctld: fatal: mkdir(/var/spool/slurmctld): Permission denied

The solution is executing the commands below on management nodes.

1
2

clab-mgt$ sudo mkdir -p /var/spool/slurmctld
clab-mgt$ sudo chown slurm:slurm /var/spool/slurmctld/

Start Slurm Service

So far, we have finished the basic configuration. Let us launch Slurm now.

1
2
3
4
5
6
7
8
9
10
11

# On management nodes
clab-mgt$ sudo systemctl enable munge
clab-mgt$ sudo systemctl start munge
clab-mgt$ sudo systemctl enable slurmctld
clab-mgt$ sudo systemctl start slurmctld

# On compute nodes
clab-comp$ sudo systemctl enable munge
clab-comp$ sudo systemctl start munge
clab-comp$ sudo systemctl enable slurmd
clab-comp$ sudo systemctl start slurmd

Run sinfo and we should see all the compute nodes are ready.

1
2
3

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
debug*       up   infinite     20   idle clab[01-20]

Debugging Tips

If your Slurm is not working correctly, you could try with these commands to debug.

1
2

clab-mgt$ sudo slurmctld -D
clab-comp$ sudo slurmd -D

References

• https://www.cnblogs.com/aobaxu/p/16195237.html
• https://stackoverflow.com/questions/62641323/error-cgroup-namespace-freezer-not-mounted-aborting

Relationship among InfiniBand, RoCE, IPoIB, and Ethernet Mode

Let us take Mellanox ConnectX Adapter as an example. Actually, this adapter can work in either InfiniBand Mode or Ethernet Mode, which is configurable with some tools provided by the vendor. As iWARP is not widely adopted, our article will not discuss this protocol.

Note that RoCE Header is a general concept. And RoCEv1 and RoCEv2 give different detailed definitions of this part.

Identify InfiniBand / Ethernet Mode

The easiest way is to directly have a look at the interface name and link type with ifconfig or ip under Linux. An InfiniBand adapter working in Ethernet mode looks exactly the same as a regular Ethernet adapter.

1
2
3
4
5
6
7
8
9
10
11
12
13

$ ip a
# InfiniBand Mode
4: ibp129s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP group default qlen 256
    link/infiniband ...
    inet 192.168.7.100/24 brd 192.168.7.255 scope global ibp129s0
       valid_lft forever preferred_lft forever

# Ethernet Mode
7: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ...
    inet 10.200.0.1/24 brd 10.200.0.255 scope global ens1f1
       valid_lft forever preferred_lft forever

Besides, ibdev2netdev can also help.

1
2
3

$ ibdev2netdev
mlx4_0 port 1 ==> ibp129s0 (Up)
mlx5_0 port 1 ==> ens1f0 (Up)

Another approach is through ibstat. And the field Link layer shows which mode the adapter is working in.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

$ ibstat
# InfiniBand Mode
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 
System image GUID: 
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0x0251486a
Port GUID: 
Link layer: InfiniBand

# Ethernet Mode
CA 'mlx5_0'
CA type: MT4119
Number of ports: 1
Firmware version: 16.25.1020
Hardware version: 0
Node GUID: 
System image GUID: 
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 
Link layer: Ethernet

Change InfiniBand / Ethernet Mode

To alter the work mode, there doesn't exist a general way for now. For Mellanox ConnectX Adapter, the vendor provided a tool called mlxconfig. Here is the usage listed in the official document, where you can find more information about it.

1
2
3
4
5
6
7
8
9
10
11
12
13

$ sudo mlxconfig -d /dev/mst/mt4103_pci_cr0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1
 
Device #1:
----------
Device type:   ConnectX3Pro
PCI device:    /dev/mst/mt4103_pci_cr0
Configurations:        Next Boot        New
  LINK_TYPE_P1         ETH(2)           IB(1)
  LINK_TYPE_P2         ETH(2)           IB(1)
 
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

Note that P1 and P2 are referring to two separated ports on the adapter. Attention: Please make sure the network switch is capable of handling InfiniBand or Ethernet Frame before altering the work mode . If the switch cannot recognize the data frame sent from the server, you might observer Physical state: Polling reported by ibstat, as the packet is not forwarded by the switch correctly. Certain network switches can only forward one type of data frame at a time, which means you may need to manually reconfigure the switch to let it work with the other type of data frame.

Configure IPoIB

By default, the IPoIB will be automatically configured when the IP address is assigned to the interface. The IP address can be managed by netplan or NetworkManager, which depends on your Linux distro. As for the configuration file, there is no difference between the InfiniBand and regular Ethernet Adapters.

1
2
3
4
5
6
7

# Assign a static IP address with netplan for an InfiniBand interface
network:
  ethernets:
    ibp129s0:
      addresses:
      - 192.168.7.100/24
  version: 2

Once the above configuration is applied and the interface is brought up successfully. We can see ib_ipoib module is loaded.

1
2
3
4
5
6
7

$ lsmod | grep ipoib
ib_ipoib              180224  0
$ ip a
4: ibp129s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP group default qlen 256
    link/infiniband ...
    inet 192.168.7.100/24 brd 192.168.7.255 scope global ibp129s0
       valid_lft forever preferred_lft forever

If the IP address doesn't appear in ip a, we need to check the status of the InfiniBand adapter and make sure its state is active in ibstat. A common mistake is forgetting to enable opensm / opensmd, which will make the adapter stuck at State: Initializing. Note that opensmd will not launch on startup by default.

1
2
3
4
5
6
7

# Start OpenSM
$ sudo opensm

# Start OpenSM As Daemon
$ sudo service opensmd start # Method 1
$ sudo systemctl start opensmd # Method 2
$ sudo /etc/init.d/opensmd start # Method 3

Identify RoCE Version

The major difference between RoCEv1 and RoCEv2 is that RoCEv2 is able to utilize IP networking to route while RoCEv1 is routing via MAC addresses. A funny fact is RoCEv1 and RoCEv2 may be enable simultaneously, and we could choose the version at runtime through specifying Group ID (GID). There is a script written by Mellanox named show_gids and it will display RoCE versions associated to GIDs.

1
2
3
4
5
6
7

$ show_gids
DEVPORTINDEXGIDIPv4  VERDEV
---------------------------  ------
mlx5_010fe80:0000:0000:0000:...v1ens1f0
mlx5_011fe80:0000:0000:0000:...v2ens1f0
mlx5_0120000:0000:0000:0000:...11.0.0.201  v1ens1f0
mlx5_0130000:0000:0000:0000:...11.0.0.201  v2ens1f0

Check Adapter Speed

ethtool can read out this information and it can work with both InfiniBand and Ethernet mode.

1
2
3
4
5
6
7
8
9
10
11

$ ethtool ibp129s0
Settings for ibp129s0:
...
Speed: 56000Mb/s
Duplex: Full

$ ethtool ens1f0
Settings for ens1f0:
...
Speed: 100000Mb/s
Duplex: Full

References

• https://www.advancedclustering.com/act_kb/infiniband-port-states/
• https://zhuanlan.zhihu.com/p/32105832
• https://wiki.archlinux.org/title/InfiniBand
• https://docs.nvidia.com/networking/display/MLNXOFEDv461000/OpenSM
• https://www.cnblogs.com/juzib/p/13273380.html
• https://blog.51cto.com/liangchaoxi/4044293

Since Arm community has various opinions on how to boot an Arm machine, such as UEFI + ACPI (widely used by commercial Arm servers as well as modern x86 systems), U-Boot + Device Tree (mostly used by embedded devices with limited resouces), and even UEFI + Device Tree (like Huawei L420 notebook I owned), I would suggest that don't expect OpenWrt will provide official support for UEFI + ACPI systems in recent days as it is designed to run on tiny routers.

Compile Kernel and Rootfs from Source

Don't be scared. With the help of buildroot, which could automatically prepare the cross-compilation toolchain we need, this step is much simple nowadays.

Note: My test environment is Ubuntu 21.10 ARM64 on Apple M1 Pro. It doesn't matter if you use a machine with a different system or architecture like AMD64, but you may need to take a few extra steps if so.

Install Dependencies

For Debian / Ubuntu users,

1
2
3
4
5
6

sudo apt update
sudo apt install build-essential ccache ecj fastjar file g++ gawk \
gettext git java-propose-classpath libelf-dev libncurses5-dev \
libncursesw5-dev libssl-dev python python2.7-dev python3 unzip wget \
python3-distutils python3-setuptools python3-dev rsync subversion \
swig time xsltproc zlib1g-dev 

Note: The content of this sub-section is copied from the official guide. Take a look at it if this command is not applicable for your system.

Download the Source Code

1
2
3
4
5

git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
git tag
git checkout v21.02.2
./scripts/feeds update -a

Configure the Project

1. Import the official configuration.

To save our effort, it is a good idea to modify an existing configuration instead of creating a new one.

1
2

wget https://downloads.openwrt.org/releases/21.02.2/targets/armvirt/64/config.buildinfo
cp config.buildinfo .config

2. Add UEFI ACPI support.

Open file target/linux/armvirt/config-5.4, and append the following lines to the end of file.

1
2
3
4
5

CONFIG_EFI_STUB=y
CONFIG_EFI=y
CONFIG_EFI_VARS=y
CONFIG_ARCH_SUPPORTS_ACPI=y
CONFIG_ACPI=y

3. Launch Memuconfig.

1
2

make menuconfig
make kernel_menuconfig

Tweak the configuration as you like, but you should clearly understand the consequence before you turn on and off something. Keeping default options is also fine.

Note: These commands will build the whole toolchain from source for the first time they are executed. The compilation process is very slow.

Build the Kernel and Rootfs

1

make -j $(nproc) defconfig download clean world

It will compile the kernel and all of the selected pre-installed utilities, then generate an EFI binary of Linux Kernel and an Ext4 / SquashFS partition image of Rootfs.

Verify the Firmware Image

The exciting moment comes. Let's test the kernel and rootfs we just built.

1. Install QEMU.

For Ubuntu users, I would suggest to install virt-manager instead, which offers a helpful GUI wizard for QEMU.

1

sudo apt install virt-manager

2. Launch a virtual machine.

The magical QEMU allows virtual machines to boot a kernel without a bootloader. That is a great feature enables us to test the kernel's functionality at the early stage.

1

qemu-system-aarch64 -m 512 -nographic -cpu cortex-a72 -smp 1 -M virt -kernel ~/openwrt/bin/targets/armvirt/64/openwrt-21.02.2-armvirt-64-Image-initramfs -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd

Note: Image-initramfs is the kernel binary while it integrates the OpenWrt's Rootfs as initramfs, so this virtual machine will lose data each time it reboots.

Note: If you encounter this issue,

1 2 3

EFI stub: Booting Linux Kernel... EFI stub: ERROR: Failed to relocate kernel EFI stub: ERROR: Failed to relocate kernel

The solution is to increase the memory capacity of your virtual machine. Empirically, it should be at least 256 MB.

Build the Disk Image

Considered that data loss is not acceptable, while not every hypervisor is capable of launching a kernel directly, we should put everything we built into a disk, or virtual machine's disk image.

To keep things simple, let's start from building a raw disk image, which is one of the virtual disk formats supported by QEMU.

Create an Empty Disk Image

1
2
3
4

tonny@vm:~$ dd if=/dev/zero of=disk.img bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.958229 s, 1.1 GB/s

This command will create an empty disk image. Feel free to replace the value of count to change the size of the disk. (size = 1 MB * 1024 = 1 GB)

Partition, Mount, and Format the Disk Image

1. Partition the disk.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

tonny@vm:~$ fdisk disk.img

Welcome to fdisk (util-linux 2.36.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xb89701e3.

Command (m for help): g
Created a new GPT disklabel (GUID: 43B50BB3-20FD-3D4B-BFE1-50B5016F8059).

Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-2097118, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2097118, default 2097118): +100M

Created a new partition 1 of type 'Linux filesystem' and of size 100 MiB.

Command (m for help): t
Selected partition 1
Partition type or alias (type L to list all): uefi
Changed type of partition 'Linux filesystem' to 'EFI System'.

Command (m for help): n
Partition number (2-128, default 2):
First sector (206848-2097118, default 206848):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (206848-2097118, default 2097118):

Created a new partition 2 of type 'Linux filesystem' and of size 923 MiB.

Command (m for help): p
Disk disk.img: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 43B50BB3-20FD-3D4B-BFE1-50B5016F8059

Device      Start     End Sectors  Size Type
disk.img1    2048  206847  204800  100M EFI System
disk.img2  206848 2097118 1890271  923M Linux filesystem

Command (m for help): w
The partition table has been altered.
Syncing disks.

A new GPT partition table with two partitions is written to the disk image.

2. Mount the disk image as a logical disk.

1
2
3
4
5
6

tonny@vm:~$ sudo losetup -Pf disk.img
tonny@vm:~$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop5                       7:5    0    1G  0 loop
├─loop5p1                 259:4    0  100M  0 part
└─loop5p2                 259:5    0  923M  0 part

OS has recognized the two partitions, loop5p1 and loop5p2.

3. Format the partitions.

1
2

tonny@vm:~/mnt$ sudo mkfs.vfat /dev/loop5p1
mkfs.fat 4.2 (2021-01-31)

We don't need to format the second partition (Rootfs) for now, because we can directly restore the partition image of Rootfs instead, which is already formatted with Ext4 File System.

4. Mount ESP partition.

ESP partition contains the EFI executables of bootloaders (e.g., GRUB), as well as its configuration files. We can also put the kernel binary here.

Note: Some Linux distributions, like Ubuntu, will put their kernel in a third partition.

Unlike Rootfs, OpenWrt Build System won't generate an ESP partition image for ARM64 platform. That means we have to build ESP partition manually.

1
2

tonny@vm:~/mnt$ mkdir -p ~/mnt/esp
tonny@vm:~/mnt$ sudo mount /dev/loop5p1 ~/mnt/esp

Restore Rootfs Partition Image

1
2
3
4
5
6
7
8

tonny@vm:~$ sudo dd if=~/openwrt/bin/targets/armvirt/64/openwrt-21.02.2-armvirt-64-rootfs-ext4.img of=/dev/loop5p2 bs=1M
104+0 records in
104+0 records out
109051904 bytes (109 MB, 104 MiB) copied, 0.613318 s, 178 MB/s
tonny@vm:~$ sudo resize2fs /dev/loop5p2
resize2fs 1.46.3 (27-Jul-2021)
Resizing the filesystem on /dev/loop5p2 to 236283 (4k) blocks.
The filesystem on /dev/loop5p2 is now 236283 (4k) blocks long.

The size of Rootfs image is about 128 MB, which implies that the file system inside will assume the partition size is about 128 MB. The size of our Rootfs partition is likely larger than this number, so we should notify the filesystem there is a change on the partition size.

Install GRUB to ESP Partition

Install ARM64 GRUB to Host

For Ubuntu users,

1

sudo apt install grub-efi-arm64-bin

Note: If your Host's architecture isn't ARM64, Apt may fail to find this package. Fortunately, thanks to Multiarch feature, we can easily install a package for other architectures. Take Ubuntu AMD64 as an example.

1. Request for ARM64 architecture's packages.

1	sudo dpkg --add-architecture arm64

2. Add an Apt Repository for ARM64.

Modify the file /etc/apt/source.list and add a ARM64 repository. Pay attention that ARM64 and AMD64 don't share the same repository, so we also need to add a filter for each repository. Here is an example.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

deb [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish main restricted universe multiverse # deb-src [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish main restricted universe multiverse deb [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-security main restricted universe multiverse # deb-src [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-security main restricted universe multiverse deb [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-updates main restricted universe multiverse # deb-src [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-updates main restricted universe multiverse deb [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-backports main restricted universe multiverse # deb-src [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-backports main restricted universe multiverse deb [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish main restricted universe multiverse # deb-src [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish main restricted universe multiverse deb [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-security main restricted universe multiverse # deb-src [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-security main restricted universe multiverse deb [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-updates main restricted universe multiverse # deb-src [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-updates main restricted universe multiverse deb [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-backports main restricted universe multiverse # deb-src [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-backports main restricted universe multiverse

3. Install ARM64 GRUB

1 2	sudo apt update sudo apt install grub-efi-arm64-bin

Generate EFI Executable

1. Check Partition's UUIDs.

1
2
3
4
5

tonny@vm:~$ lsblk -o PATH,UUID,PARTUUID /dev/loop5
PATH         UUID                                 PARTUUID
/dev/loop5
/dev/loop5p1 CF95-2044                            3754ccb7-1920-2b41-9962-af81ac6a04b2
/dev/loop5p2 ff313567-e9f1-5a5d-9895-3ba130b4a864 e09a20c3-0ea7-0c48-b653-0482facd93db

Those UUIDs will be referred by the GRUB configurations.

2. Write Early-stage GRUB Configuration.

Create a new file ~/grub-early.cfg, and write the following lines. This configuration will be hardcoded into GRUB's EFI binary.

1
2
3

search.fs_uuid CF95-2044 root
set prefix=($root)'/boot'
configfile $prefix/grub.cfg

Replace the UUID with your loop5p1's.

3. Make GRUB EFI Executable.

1
2
3
4

# tonny@vm:~$ sudo mount /dev/loop5p1 ~/mnt/esp
tonny@vm:~$ sudo mkdir -p ~/mnt/esp/EFI/BOOT/
tonny@vm:~$ cd ~/mnt/esp/EFI/BOOT/
tonny@vm:~/mnt/esp/EFI/BOOT$ sudo grub-mkimage -c ~/grub-early.cfg -p /boot -o BOOTAA64.EFI -O arm64-efi boot chain configfile fat linux ls part_gpt reboot serial efi_gop search_fs_uuid

Note: It is not recommended to use grub-install here. One of its typical usages is,

1	sudo grub-install --target=arm64-efi --efi-directory ~/mnt/esp --bootloader-id=GRUB --boot-directory ~/mnt/esp/boot/

The hidden disgusting thing is, if you use GRUB provided by Ubuntu, this command will hardcode an important GRUB variable prefix='/EFI/ubuntu' to the EFI binary, and there is no way to change it.

Write Second-stage GRUB Configuration

1
2
3
4

tonny@vm:~/mnt/esp/EFI/BOOT$ cd ../..
tonny@vm:~/mnt/esp$ sudo mkdir boot
tonny@vm:~/mnt/esp$ cd boot/
tonny@vm:~/mnt/esp/boot$ sudo nano grub.cfg # or other text editor you feel comfortable with

The content of grub.cfg is,

1
2
3
4
5
6
7
8
9
10
11
12

serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 --rtscts=off
terminal_input console serial; terminal_output console serial

set default="0"
set timeout="5"

menuentry "OpenWrt" {
linux /boot/vmlinuz root=PARTUUID=e09a20c3-0ea7-0c48-b653-0482facd93db rootwait   console=tty0 console=ttyS0,115200n8 noinitrd
}
menuentry "OpenWrt (failsafe)" {
linux /boot/vmlinuz failsafe=true root=PARTUUID=e09a20c3-0ea7-0c48-b653-0482facd93db rootwait   console=tty0 console=ttyS0,115200n8 noinitrd
}

Replace PARTUUIDs (not UUIDs) with your loop5p2's.

Copy Linux Kernel

1

tonny@vm:~/mnt/esp/boot$ sudo cp ~/openwrt/bin/targets/armvirt/64/openwrt-21.02.2-armvirt-64-Image vmlinuz

Verify the Disk Image

1
2

tonny@vm:~/mnt/esp/boot$ cd ~
tonny@vm:~$ qemu-system-aarch64 -m 512 -nographic -cpu cortex-a72 -smp 1 -M virt -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -drive format=raw,file=disk.img

If everything goes well, you could see your kernel is running happily. Enjoy it!

Note: You don't have to unmount the disk before launching the virtual machine. But you should sync the disk to make sure all the data cached in memory is written back.

1	tonny@vm:~/mnt/esp/boot$ sync

Clean up

1
2

tonny@vm:~/mnt$ sudo umount ~/mnt/esp
tonny@vm:~/mnt$ sudo losetup -d /dev/loop5

Launch VM with Virt-Manager

1

tonny@vm:~/mnt$ virt-manager

Note: Sometimes vert-manager requires permissions to run.

The recommended configuration:

• Step 1:
  • Architecture: aarch64
  • Machine Type: virt
  • Import existing disk image
• Step 2:
  • Browse ➡️ Add pool Home ➡️ Choose Volume disk.img
  • Choose OS: Generic Linux / OS
• Step 3:
  • Memory: >= 256 MB
  • CPU: Any
• Step 4:
  • Customize configuration before install
  • Network (LAN Port): Bridge / Macvtap Bridge
• Configuration
  • Overview/Firmware: UEFI aarch64

Note: You can't change the firmware type after pre-install configuration.

References

• https://gist.github.com/tstellanova/dea7593a7dfe4f48432a58cb007e7056
• https://forum.openwrt.org/t/arm64-armvirt64-uefi-efi-openwrt-target/82740
• https://forum.openwrt.org/t/how-to-install-openwrt-as-a-new-os-in-the-grub-menu/97465
• https://wiki.ubuntu.com/ARM64/QEMU
• https://wiki.archlinux.org/title/GRUB
• https://openwrt.org/docs/guide-user/virtualization/qemu
• https://krinkinmu.github.io/2020/11/21/EFI-aarch64.html
• https://soha.moe/post/make-uefi-compatible-openwrt-disk-image.html
• https://git.openwrt.org/?p=openwrt/openwrt.git;hb=refs/heads/openwrt-21.02;a=blob;f=package/boot/grub2/Makefile
• https://www.cxyzjd.com/article/u010875635/74289971

Quick Intro to LVM

Logical Volume Manager (LVM) is similar to Dynamic Disks under Windows, which can take several GPT / MBR partitions on different hard disks as a storage pool (LVM call it Volume Groups, VG), and allocate spaces from this pool, then Linux will recognize each space (LVM call it Logical Volume, LV) as an useable partition.

Thus, we should modify not only the GPT / MBR partition table, but also the LVM configuration.

Update GPT / MBR partition table

I suggest all the operations should be done under live CD environment to avoid the occurrence of unpredictable problems. I didn't test online resizing on the root partition so far.

1. The following instructions in this section assume the last partition on your disk is the LVM Physical Volume. You could verify this with the command lsblk --fs. nvme0n1p3 is the last GPT partition on the disk nvme0n1, and it is easy to identify this partition is a LVM PV, and ubuntu--vg-ubuntu--lv is the corresponding LV.

1
2
3
4
5
6
7
8
9
10

tonny@vm:~$ lsblk --fs
NAME                      FSTYPE      FSVER        
loop3
└─loop3p1                 LVM2_member LVM2 001 
  └─test--vg-test--lv     ext4        1.0      
nvme0n1
├─nvme0n1p1               vfat        FAT32    
├─nvme0n1p2               ext4        1.0      
└─nvme0n1p3               LVM2_member LVM2 001 
  └─ubuntu--vg-ubuntu--lv ext4        1.0      

Note that ubuntu--vg-ubuntu--lv is the root partition of the system here.

1 2 3 4 5 6

tonny@vm:~$ df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/ubuntu--vg-ubuntu--lv ext4 78G 7.7G 67G 11% / /dev/nvme0n1p2 ext4 974M 87M 820M 10% /boot /dev/nvme0n1p1 vfat 511M 3.6M 508M 1% /boot/efi /dev/mapper/test--vg-test--lv ext4 464M 24K 429M 1% /home/tonny/mnt

Also you could check the LVM Volume Group status by vgs.

1 2 3 4

tonny@vm:~$ sudo vgs VG #PV #LV #SN Attr VSize VFree test-vg 1 1 0 wz--n- 496.00m 0 ubuntu-vg 1 1 0 wz--n- <78.50g 0

2. Update the GPT / MBR partition table using fdisk. I will use an emulated disk /dev/loop3 to demonstrate the whole process. Don't worry, you won't loss your data under normal circumstances. These commands will only modify the partition table, but make sure DO NOT remove the LVM's signature, otherwise the system may no longer recognize your LVM PV.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

tonny@vm:~$ sudo fdisk /dev/loop3 # replace with your hard disk, such as /dev/nvme0n1p3

Welcome to fdisk (util-linux 2.36.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/loop3: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C5F55056-8C56-5448-81E4-567F59AD93ED

Device       Start     End Sectors  Size Type
/dev/loop3p1  2048 1026047 1024000  500M Linux filesystem

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-2097118, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2097118, default 2097118):

Created a new partition 1 of type 'Linux filesystem' and of size 1023 MiB.
Partition #1 contains a LVM2_member signature.

Do you want to remove the signature? [Y]es/[N]o: N

Command (m for help): p

Disk /dev/loop3: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C5F55056-8C56-5448-81E4-567F59AD93ED

Device       Start     End Sectors  Size Type
/dev/loop3p1  2048 2097118 2095071 1023M Linux filesystem

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Update LVM Configuration

1. Notify LVM there is an update on the partition table.

1
2
3
4

tonny@vm:~$ sudo partprobe # ask kernel to read the new partition table
tonny@vm:~$ sudo pvresize /dev/loop3p1 # replace with your partition
  Physical volume "/dev/loop3p1" changed
  1 physical volume(s) resized or updated / 0 physical volume(s) not resized

At this moment, the LVM Volume Group status has changed to,

1 2 3 4

tonny@vm:~$ sudo vgs VG #PV #LV #SN Attr VSize VFree test-vg 1 1 0 wz--n- 1020.00m 524.00m ubuntu-vg 1 1 0 wz--n- <78.50g 0

Observe that VFree of test-vg has increased by 524.00 MB.

2. Resize LVM Logical Volume. The following command will allocate all the free space of VG to the LV.

1
2
3

tonny@vm:~$ sudo lvextend -l +100%FREE /dev/mapper/test--vg-test--lv
  Size of logical volume test-vg/test-lv changed from 496.00 MiB (124 extents) to 1020.00 MiB (255 extents).
  Logical volume test-vg/test-lv successfully resized.

The free space of VG is used up now.

1 2 3 4

tonny@vm:~$ sudo vgs VG #PV #LV #SN Attr VSize VFree test-vg 1 1 0 wz--n- 1020.00m 0 ubuntu-vg 1 1 0 wz--n- <78.50g 0

Resize ext4 File System

Up to now, although LVM LV is resized, the ext4 file system is not aware of the extra available space. Simply run resize2fs to let it know.

1
2
3
4
5

tonny@vm:~$ sudo resize2fs /dev/mapper/test--vg-test--lv
resize2fs 1.46.3 (27-Jul-2021)
Filesystem at /dev/mapper/test--vg-test--lv is mounted on /home/tonny/mnt; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mapper/test--vg-test--lv is now 261120 (4k) blocks long.

We can see the available space of test--vg-test--lv has been enlarged.

1 2 3

tonny@vm:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/test--vg-test--lv 973M 1.3M 917M 1% /home/tonny/mnt

References

• https://www.linuxtechi.com/extend-lvm-partitions/
• https://www.thegeekdiary.com/centos-rhel-how-to-extend-physical-volume-in-lvm-by-extending-the-disk-partition-used/
• https://i0.wp.com/manjaro.site/wp-content/uploads/2017/08/lvm-layout.png

更新日志

• 2022.1 NexT.Remix v3 (Preview)：目前的版本，基于 NexT.Gemini v8，融合了 Hexo Cake 和 Hugo Stack 两个主题的风格
• 2021.8 NexT.Gemini (Remix v2)：速度优化
• 2020.5 NexT.Gemini (Remix v1)：初始的魔改版本，基于 NexT.Gemini v7，风格参考了 Hexo Terminal 和某个 Markdown Resume 模板

由于NexT v8较v7有大量的代码改动，加上我原来改代码的方式充满野性，NexT.Remix 的代码并不继承于之前的魔改主题，且代码的改动遵守了 NexT 的魔改规范，使用了 Theme Inject 来魔改主题，仅增加数个文件，并未对原有文件进行修改，能够（相对）方便的合并 upstream 的新代码。

除了更新 upstream 和 Hexo 的版本以及界面风格调整外，还把烂掉了的 utterance 换成了 giscus。

TODO

• 暂时使用 jsdelivr 分发部分公共 JS 库，之后改成用博客的 CDN 来分发这些文件以提升国内访问速度和稳定性。
• 目前 giscus 的 integration 做的很粗糙，等到改好了就给 upstream 交个 PR 。

演进方向

现代

NexT 作为一个有着丰富历史（比如换了两次仓库）的主题，它仍然不忘初心，到现在还保持着最初的模样。然而我更喜欢时下流行的 后·扁平化 风格，但又馋 NexT 丰富的功能，同时也懒得迁移平台，所以只能去把 NexT 变成我喜欢的样子了。

由于我并不是什么设计带师，就只好 ”参考“ 已有的优秀样式。

• Hexo Cake：在 NexT v7 上魔改的一个主题。总体很棒，由于都是 NexT·改，”参考“起来更方便了
• Hugo Stack：喜欢它的阴影和配色

实际上，NexT 的底子非常不错，随便改改就能完全满足我的审美。

简洁

如果配置得当，NexT本身的界面并不臃肿。这是 Remix v1 开始就在追求的目标。这一次进一步的删除掉不必要的元素，比如到处都是的下划线，友链上那一堆，文章目录上那一堆，还有日期上的下划线。另外 Pagination 也成了我重拳出击的对象。

此外，从很久以前开始，我就在弱化标签和分类这两个功能，因为我自己的习惯是从来不看博客文章的标签和分类，读者也都是从搜索引擎跳转过来的，搜索引擎也不需要标签就能自己从文中提取关键词，当然更重要的原因就是我懒得加这些东西，所以界面上关于标签和分类的元素也减少了。

个性

这也是为什么要自己魔改主题的原因。一个显而易见的原因当然就是不希望自己的博客主题和其他大路货撞车。其次就是博客主题要符合自己的写作风格，不同于 这位，追求读者阅读的极致体验与获得感，我希望我能写出来：

• 仅期望我自己，有时也包括事件相关者阅读的回忆类内容
• （最好是独一无二的）技术类文章
  • 在自己忘掉的时候给自己参考
  • 顺便给找不到其他资料的人参考

也就是说，我并不会花很多心思在提升读者阅读体验上。对我来说，我不喜欢把很多精力放在我不关心的东西上，一些不那么重要的问题就怎么省力怎么来。比如配图，别说统一配图的风格了，如果这个配图只是为了美观，我选择不配图。因此，没有文章配图也很好看的主题就是我需要的。那种不需要写摘要，会自动把文章第一段当作摘要的主题就是我需要的。

我希望我的文章能侧重于回答那些暂时无解或者没人总结答案的但很多人关心的问题上，这样读者在救命稻草前肯定不会对阅读体验挑三拣四。当然基本的阅读体验还是要有的，魔改主题提升文章可读性也是改善阅读体验的一部分。

还有就是，我指望魔改博客主题这件事能够一定程度的体现出博主的水平……什么，你说 dalao 都是自己造博客框架的？我又不是前端专业，我不揽这个瓷器活。

哦对了，现在博客使用一种叫 Neko 语的东西，这语言一部分是中英双语，一部分是被我改掉的 NexT 的塑料英语。

总结

自我感觉这一年这个博客还是取得了显著的进步，看起来更 Professional 了。

文章数量

似乎没有维持出一个月一篇的节奏。Anyway，我自认为文章质量比去年的还是强了一丢丢。（可能是我太摸了，所以没有踩到什么坑。）

博客主题

改完之后我舒服了，从表面到代码实现都比原来美观了不少。

访问速度

以前的方案又贵又拉，Azure CDN + Github Page 这套太强了。

访问量

只要写文章的速度比文章过气的速度快，访问量一定是会增长的。只不过百度死活还是只收录了主页，辣鸡玩意。

There is an article already described the detailed steps about the automatic installation of Ubuntu 20.04 Server, but according what its author said, their blog system ripped out some important characters. So, by checking with this script, I figured out the correct way to achieve our goal.

Download the image

Download the live CD image in whichever way you prefer. For the user locates in China, I would suggest you download from https://mirrors.tuna.tsinghua.edu.cn/ubuntu-releases/focal/ubuntu-20.04.3-live-server-amd64.iso.

Update some files in ISO image

Only thing we need to do is updating several files. And here are some recommended editors.

• For Windows user, I strongly recommend you use Ultraiso to edit the ISO file.
• For Linux user, ISO Master should work. (I didn't try it before.)
• For the user who wants to deeply customize the ISO file, including unpacking rootfs image, cubic is everything you need. You can refer to my previous post and learn how it works.

Add Kernel Arguments

Assume the root directory of ISO image is /cdrom. There are two bootloader configuration files need to modify, one for UEFI system, one for the legacy one. Append the kernel arguments like this,

• /cdrom/isolinux/txt.cfg

1
2
3
4

label live
  menu label ^Install Ubuntu Server
  kernel /casper/vmlinuz
  append   initrd=/casper/initrd quiet autoinstall ds=nocloud;s=/cdrom/  ---

• /cdrom/boot/grub/grub.cfg

1
2
3
4
5

menuentry "Install Ubuntu Server" {
set gfxpayload=keep
linux/casper/vmlinuz   quiet autoinstall ds=nocloud\;s=/cdrom/ ---
initrd/casper/initrd
}

If you would like to skip the integrity check, you could try to append the kernel argument fsck.mode=skip as the following example shows.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# File: /cdrom/isolinux/txt.cfg

label live
  menu label ^Install Ubuntu Server
  kernel /casper/vmlinuz
  append   initrd=/casper/initrd quiet fsck.mode=skip autoinstall ds=nocloud;s=/cdrom/ ---

# File: /cdrom/boot/grub/grub.cfg

menuentry "Install Ubuntu Server" {
set gfxpayload=keep
linux/casper/vmlinuz   quiet fsck.mode=skip autoinstall ds=nocloud\;s=/cdrom/ ---
initrd/casper/initrd
}

Note: HWE Kernel is a higher version of Linux kernel compared to the default one, which is shipped with the newer drivers. Theoretically, it has a better support for the latest hardware.

Add Auto-install Configurations

Two new files are also required for automatic answering.

• /cdrom/user-data

This configuration is what I am using now, and it is for the machine without Internet. I have verified that it can make the installation procedure fully automatic.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#cloud-config
autoinstall:
  version: 1
  storage:  # should set the interactive default but doesn't seem to work??
    layout:
      name: direct
  locale: en_US.UTF-8
  keyboard:
    layout: us
  identity:
    hostname: ubuntu-server
    password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
    username: tonny
  ssh:
    allow-pw: true
    install-server: true
  package_update: false
  package_upgrade: false

Note:

• direct storage layout means using and erasing the whole disk. (The default option provided by the interactive installer.)
• The password is ubuntu. This can be generated by mkpasswd.

For more usages, check this example.

• /cdrom/meta-data

Just create an empty file and put it there.

SC20：没有什么比这更绝望的了

时间回到去年，我并没有给SC20写回顾，因为这场比赛打得实在是——太！烂！了！烂到以至于取得了倒数第二的优异成绩，烂到了我都不好意思把这个比赛写进简历里，烂到了我到SC21开赛的时候才去问SC20的成绩。尽管SC20是我们第一次参加的SCC，但不至于烂的这么有特色。上一场比赛出现了包括但不限于以下这些情况，

• 完全不存在的赛前准备，虽然时任队长已经尽力在准备Cyclecloud的环境了（甚至去研究了cluster-init），但其他队员似乎就完全没碰过云环境（伏笔1）
• 两个16级的跑路了（不过当时拉他们过来的时候，也说好了他们可以当placeholder）

我自己那时候负责的是复现挑战，不过我当时根本就不知道复现挑战要干嘛，甚至把精力放在了改代码上。我用了GDR来优化GPU P2P通信来着，还觉得自己挺牛批的来着（伏笔2）。Webinar完全没有看，到了比赛的时候才知道要“在46小时的时间里写出具有发表在国际刊物水平的report”。我作为一个无paper选手（截至目前也还是没paper），用Excel都画不明白的那种，到了场上才发现——坏了。

要复现的论文里提出的程序MemXCT有CPU和GPU的版本，比赛要求用两个版本跑出来的数据画图写报告。GPU版本这边，由于之前完全没用过云集群，我完全不知道云上预装了什么东西。队长在强力推销他的HPCX OpenMPI库，而我自己在测试的时候用的是NVIDIA HPC SDK里的OpenMPI。我一想，HPCX是NVIDIA的，HPC SDK也是NVIDIA的，它们的OpenMPI应该是同一个东西吧。结果很显然，MPI炸得很灿烂，程序就是死活都跑不起来。在无穷无尽的MPI调参和重新编译，MCA的各种参数各种排列组合都试了一圈以后，嗯，没有什么效果。套GDB单步慢慢调，去找爆炸源（那一会我还不知道GDB可以直接Traceback），发现就是我改的GDR P2P那几行代码炸了，但HPCX的OpenMPI不是自称是CUDA Aware还支持GDR的吗，年少无知的我并没有对HPCX产生怀疑，从第一天开始，一直瞎勾巴试到第二天晚上，才想到用回NVIDIA HPC SDK试试。MPI一换，什么问题都没有了。得益于这段时间里，我享受了长达4个小时的精致睡眠，我的大脑throttle到完全没有意识到“放弃这个优化”这个操作（且复现题压根就不需要优化），哪怕是其他Application，把程序跑起来才是重中之重。

CPU版本这边，总所周知（并没有），我们可以用mpirun的各种各样奇奇怪怪的参数（如ppn，map-by）或者Job Scheduler（如Slurm）来绑核。然而当时我既不懂Slurm，又不懂mpirun的参数，完了这程序还是Hybrid的（MPI + OpenMP），需要给MPI Rank分配多几个CPU核，而不是1个Rank一个核。我一顿操作猛如虎。好消息是，程序跑起来了。坏消息是，程序以一种很奇怪的姿势运行在多个节点上，比如明明要跑四节点，却出现了一节点有难，三个节点围观的情况。

到了比赛快结束的时候，两边才正常跑起来，由于根 本 没 有 提 前 写 论 文，也 不 知 道 怎 么 画 Academic的图，要不是有一个临时安排的帮手，在比赛结束前report写不完也就算了，估计连张图都画不出来。由于交的实在是太晚，这堆学术垃圾在提交到一半的时候比赛就结束了，这样也好，这篇完成度极低的黑历史就再也不会有其他人看到了。

其他人那边也没好到哪去，由于赛前准备什么的基本不存在（人还跑路了），Applications不能说是一塌糊涂，只能说是亿塌糊涂，基本上拿不到几个分，CESM跑不出，GROMACS只跑了一点，倒是MiniVite似乎还行。最后只能寄希望于Benchmarks，把剩下的Funding All in到Benchmarks上，说不定还能捞一个单项奖。当时他们刷了半天HPL，最后刷到了120TFlops，混到了暂时的第一，反手就被ETH的129T打成灰了，反手反手就被半夜想搞大新闻的THU搞出了大新闻，300T打得灰都不剩了。与此同时T队的HPCG，IO500分也把榜打爆了，这两玩意的分数是当时榜一（可怜的靶子）的3.89倍，5.76倍，似乎看到了摩尔定律复活的希望。在这种分数差面前，我们并没有任何反抗的余地，只能对其深夜5am炸鱼的行为表示强烈的谴责。

赛后才知道，HPL和HPCG方面，T队几个月前向组委会提交的plan里，早就盘算着把Azure机房洗劫一空，只要我GPU堆得越多，跑分速度就能快到其他队连尾气都闻不到，只要我操作够快，火速开机火速关机，一大堆GPU也花不掉几个钱。并且充分考虑到V100节点不够的问题，也准备了转用其他GPU节点的预案。组委会许可了他们的方案，并搬出板凳坐等看戏。反倒是我们的HPL因为超规格的问题，被扣分扣成了倒数。IO500更是离大谱，自研了打榜专用文件系统MadFS（详见金枪鱼之夜——IO500 S: There is rjgg behind MadFS - YouTube），科研成果下放到学生超算竞赛，直接形成降维打击，不得不说THU的System方向真是tql，TUNA里个个都是人才，说话又好听。

总之，要不是SC21打得不错，我是再也不会去提SC20的事情了。

SC21 Phase 0：队友人呢？

今年招新的结果非常意外，忽悠到了一些《建议直接颁发硕士毕业证》的20级新人。今年招新是让感兴趣的入队的人做一份笔试题，目的当然是选出愿意去了解这个领域并去做一些搜索的人，由于计系没有哪门课介绍了超算，我对于他们的预期就是《言之有理即可》，aka《有字就行》，结果出现了这种情况，

Neko.d> 同学你好，由于你以一己之力让我怀疑题目出得太简单了，所以我希望今天能和你当面聊一下...

怎么会有人开局（大一）就几乎啥都懂啊，那还要我们老油条干嘛？

但另一个问题出现了，这次招新一个妹子都没忽悠到，不像去年又有Female还有Transgender，Diversity直接拉满，四舍五入直接等于保送进决赛。今年我们队里现在全是臭男人，proposal就只能尬吹我们辉煌的过去。

在写proposal的时候，本想把任务拆分，先写个中文的大纲，offload给其他队友，让他们输出English paragraphs，再merge成一个完整的proposal。结果真到offload的时候，赶作业的赶作业，赶paper的赶paper，这也就算了，还有去外面嗨然后装死的。当队长这个大锅抛到我头上的时候，距离交proposal的ddl已经没几天了，还得去拉浪潮的赞助。发现队友都指望不上还没时间自己搞定的时候，我只能无能狂怒，

Neko.d> I AM REALLY ANGRY!

但我angry并没有什么用，proposal还是得写，最后想到了一个惊为天人的解决方案：用谷歌翻译把提纲翻译一下，我看两眼改一改就交上去了，反正我只要坚信reviewer不喜欢看一大段一大段的屁话，我的良心就不会痛。

几个月以后收到消息，令人意外，我们用脚写的proposal过了，倒是T队挂了（不过靠着ISC冠军又回来赛场了，你大爷还是你大爷），听说还有几个国内的强校也挂了，目测多数死于Diversity。虽然我们Diversity吃了一个reviewer的低分，但其他的reviewer好像成功的被忽悠过去了。参赛前务必把美式政治正确玩明白了。

SC21 Phase 1：还能抢救一下

众所周知，我是摸鱼之王，摸鱼从来就没输过，摸到组委会专门发邮件催我们上号，摸到连队友不敢摸了。

组委会> Azure说你们从来没有上过号。马上比赛了，想问问你们是不是网络不好，登不上号啊？有问题的话要跟我们说啊。

组员1> 所以什么时候练习啊

组员2> 所以什么时候练习啊

Oracle集群的试用期也就五天，到账号激活的第二天我才想起来还有这事（我紫菜）。集群配置比较常规，主要是不存在伸缩问题，在试用的时候没发现太多的问题（主要是因为很多东西没来得及试）。

一个不大不小的事故是，因为我想让队员在练习的时候熟悉Linux的账号机制，就让他们改passwd和sudoer.d。然而我低估了修改sudoer.d的风险，因为这个东西改炸了sudo就用不了，想还原sudoer.d的配置来抢救sudo本身又需要sudo权限，好家伙死锁了。不得已，只能寻求场外援助，Oracle的Technical Leader，Marcin老哥。老哥一看我的问题，见怪不怪，熟练的向我们推销serial console下grub改bootargs init=/bin/bash之术，想必他的客户自己（哦是我啊，那没事了）也没少搞炸系统。当grub的界面显示在Windows Terminal下的时候，我大受震撼，这还是我第一次看到在serial console下的grub，而且还保留了原汁原味的TUI。改好参数，顺利以root身份登录系统，直接把写坏的配置删掉就完事了。

Marcin> :) Sysadmin Sunday

Neko.d> Oh. Sorry to disturb your beautiful Sunday morning lol.

Marcin老哥是个大好人（会救我们的人是好人，周末来救我们的人就是大好人啦！），此外他还教我挺多实用的东西，比如，

• 一键关掉超线程
• 一键添加集群账户（集群预装了LDAP的东西）
• playbook自动化工具（据说可用于重装部分软件）
• ssh-agent
• ...

到了Azure这边就没那么幸运了，赛前中后和其他队的运维瞎聊，大家无一例外的碰到了，

• cloud-init更新的配置不被应用。公认的workaround是，要想改cloud-init脚本，重建集群吧！也就等十多分钟就好了！很快的！这导致我不敢写很复杂的init脚本
• image的一堆坑
  • VM的世代数和Image支持的世代数不一致，需要手动指定隐藏镜像OpenLogic:CentOS-HPC:7_9-gen2:latest（后来发现这还是个陈年已知问题，上一任运维知道，但不说）
    • 有个好心人给了个魔法PowerShell指令来捅出镜像列表
  • 只有CentOS 7.9的镜像带了驱动，8.1的没有
    • 可以自己装驱动，GPU部分不仅需要GPU本身的驱动，还需要NVSwitch的驱动，叫Fabric Manager。如果NVSwitch的驱动不正常，会报一个cudaErrorSystemNotReady的错
      • 补充：驱动的安装程序和安装目录不要都在NFS盘上，建议把全家桶拷到本地盘再安装，能大幅提升安装速度
    • IB部分可以装OFED全家桶（还有坑，伏笔）
• 魔改过的Slurm会在VM发生不明原因初始化配置超时的时候把好不容易allocate到的VM释放掉，尽管VM能正常使用。（每次开节点都要排十多分钟的队，然后大概率配置超时，这谁顶得住啊）

值得一提的是，我们几个运维一致认为Azure的技术支持Andy是个装死带师，我们三个都被Andy无视了。跑去跟SCC主席Kathleen complain，

Kathleen> 啊，Andy早在Webinar里说过自己最近忙了，你是不是没去听啊

Kathleen> 还有你们Stand up meeting跑哪去了

彳亍口巴。Andy告诉我唯一有用的东西就是，CycleCloud Console的cloud-init脚本可以不是cloud-init脚本。

实际试用的时候，Azure上来就是开幕雷击，先是遇到了世代数的那个问题。到比赛快开始的时候才开GPU节点练习，（因为A100很贵，27刀一小时，而且队员之前没准备好，还没把CPU版折腾清楚），这个时候才发现CentOS 8的镜像要啥驱动都没有，自己一装驱动又踩了NVSwitch的坑。搞了半天还搞不定，仔细一想每次开VM都要装驱动，难顶，只能碰运气看看7.9带不带驱动。还好驱动是全的（似乎也是唯一一个带全驱动的镜像）。降系统版本又造成了一些小问题，什么缺Lmod导致module load intel全家桶出锅啊，什么GCC版本太老导致C++ ABI出锅啊，好在这些问题还是能解决掉的。

除了被两边集群的“特性”折腾以外，似乎没有太多问题了，也就记得有

• Spack会把tmp目录挤爆，改TMPDIR环境变量就可以了
• Spack不会自己更新和确认过时的编译器信息，然后还把错误的信息缓存了，需要clean一下bootstrap就好了（spack clean -b）
  • NVIDIA HPC SDK也有类似的问题，Any changes to your gcc compilers requires you to reinstall the HPC SDK
• 默认情况下nfslock没启动，导致No locks available

其他队友那边，一开始大家还自信得一批，仿佛人均编译带师，到了自己编译应用的时候，尤其是用Spack编译一些依赖（e.g., 比如OpenBLAS和FFTW）的时候，或者要编译GPU版代码的时候，编译器就能给你炸得妈都不认识。只好各种换编译器换姿势编译，什么GCC，ICC，ICX，NVC都用了一圈。年幼无知的队友甚至还对AMD有一丝信任，想用AOCC和AOCL平替，结果我就不说了，懂得都懂。最后发现还是老一套，ICC+MKL稳如老狗。（其实是一开始我忘装了MKL，所以让他们先试OpenBLAS，然后就欣赏编译器烟花了）还有NVCC经常会选错Host编译器，得加--ccbin啥的。总之，你永远不知道最终生成的看上去能运行但大概率会爆炸的二进制文件是几个编译器生成的代码缝合在一起的产物。

SC21 Phase 2：演我们？

Azure试用到没钱的后一天就正式比赛了，可见我们什么时候才开始赛前准备。按照基本上等于废话的计划，第一天开局打算先让复现和Cardioid跑，总之就是先把Oracle集群用起来，反正没有预算限制，QE和神秘应用这种要花Azure钱的晚一点再上也不迟。9.30am有个meeting，还以为要公布instructions了，结果除了打一个尬飞了的招呼以外，什么事都没发生，instructions一个字都没看到（后来才意识到这个meeting是用来把我们骗进breakout room然后给其他人直播比赛事故现场的）。

不得不说今年的instructions有点随意啊，不仅公布时间随意，而且，

• Submission的instruction甚至直接用的去年的，一个字都没改
• 有一题的instruction放在GitHub上，放出来不久又改回private返工了

这潦草的instructions让本来就因为remote而显得不正经的比赛看起来更不正经了，好想打场正儿八经的SC现场赛啊，但没机会了。Anyway，比赛还得打。

Cardioid

assign的同学巨屌无比（还是20级的），基本上可以称为修bug自动机，因为他有NVRTC和LLVM+PTX瞎搞的经验，这题就扔给他做了，总体来说没有遇到太多的问题（因为bug都瞬间被修掉了导致我对其并没有什么太深刻的印象）。

• 在测试的时候就发现跑GPU多节点会出问题，但Oracle上只有一个GPU节点，那没事了。Cardioid似乎有几个variants，但只有一个能跑，但比赛就只用那一个，那又没事了
• Cardioid看上去可以直接用Spack整体编译，但，试了的都说sucks。似乎有的队一直卡在Spack上，其实直接不用Spack，手动编译就好了
• Oracle的集群用的是Oracle Linux 7，based on REHL 7，也就是说和CentOS 7的环境差不多一样老。所以也出了C++ ABI的问题，好在临时抱佛脚了

比赛要求跑几十套参数，大概要换几种网格尺寸，精度，计算方法跑，总共要跑几十种组合。跑了一轮下来，发现某一类参数组合会炸，总结一下有几种爆炸的姿势。第一种炸法，用gdb Backtrace一查，哦就是文件权限没设对，低级错误。第二种炸法，算出了NaN。但仔细读instruction，发现出题人似乎早就预料到选手们能跑出NaN，因为有一个问题就在问”你们见过NaN了吗“之类的，那肯定就是出题人故意设的坑。年轻的队友看不懂人性的险恶，甚至萌生了改代码去修NaN的想法。第三种炸法，发现backtrace不出来，怀疑是栈炸了。先改了ulimit，无限栈空间，不行。单步调试到一个for循环读取数组，程序就异常了。我大脑一瘫，居然怀疑这个读取操作炸了栈（读取操作不会写入栈啊喂！），因为数组在高地址的堆里（0x7ff），离栈很近，说不定越界越到栈上了，甚至去算了base+size有没有大于esp的地址，结果是完全没事。继续debug，又遇到了一些匪夷所思的事情，比如不可复现的爆炸，第一次看到值异常，第二次就值正常了。我直接进行一个思考的放弃，1am了，到点睡觉了，day2还得通宵，于是就留这个守夜的队友就自己继续debug。

第一天晚上有三个在学校的队友守夜，主因是组委会完全无视50%队伍来自中国的事实，要求我们在12am-8am开摄像头供人围观（直播睡觉？）。跟Chair complain，

Kathleen> 你咋不早说

我还指望其他中国队会比我更早发现这个问题然后去argue，怎么只有我被喷了。Chair最后还在Warp up meeting还说，有些队伍不喜欢social只想去睡觉。（不会吧，不会吧，不会还有人打比赛还想睡觉吧。）

洗完澡刚躺平的时候，那个队友就发消息说，de出来了，gdb不靠谱，还是printf大法好，最后查出来是integer overflow了。好家伙overflow和underflow都齐了，那肯定是出题的人故意的。这个overflow有一个tricky的workaround，让MPI跑在更多节点上就好了，因为会发生这个overflow的算式会除以节点数，节点数一搞大就不overflow了。八卡跑八进程，不能再多了？不存在的，一张卡跑两个MPI进程就完事了。（不过我估计还是栈烂了导致GDB行为诡异，说不定是RDMA把栈写坏了，因为这个overflow的结果似乎会propagate到MPI的参数）。

复现

搞复现的同学因为饱读各种AI paper就被我抓过来做这题，最初他以为现场赛的难点是在写paper上，因为之前在Azure上测试的时候一切都稳如老狗，虽然没时间测Oracle，但问题应该不大吧......吗？

真到了在Oracle上就，不得不说真是充满了惊喜（指新bug）。Oracle这边不像Azure，没有自带MVAPICH，只能自己用Spack编译一个，IB卡配置得也比较复杂，似乎是双口100G的RoCE，不像Azure是单口IB。程序似乎能正常编译运行下来，能跑出一些结果，只不过嘛......偶尔会在计算末期爆炸一下啦。根据我没几年的丰富经验，只要MPI程序能够跑出结果，问题就不是很大。一开始盲猜是运气问题，爆炸是随机的，重跑一次就好了。重跑了几次发现，是特定组合下的参数100%会炸，而且似乎是MPI的内部错误，挂在了同一个MPI函数上。由于程序总是在几十分钟后炸，用gdb调试几次可能一两个小时就没了，就打算先试各种民间偏方，同时先把能跑的点跑下来。

诡异的是，改无限栈空间，换MVAPICH的版本，使用Slurm，加MPI运行参数，都有时候能让程序跑起来，之后又失效了。也发现一些方法根本就没用，比如改编译MVAPICH的fabric参数，或者换HPCX MPI（然而自带的这玩意就没成功运行过）。从白天调到大半夜，程序一直处于能运行与不能运行的叠加态。Day2早上回到会议室，看得出那个队友被折磨了一晚上，身体已经完全被掏空了，大脑完全停止了思考。在我睡觉期间，队友去问了大好人Oracle的工程师Marcin，他给了一个写满了各种魔法MPI参数的博文， 然而这个魔法博文里提供了OpenMPI的参数，提供了Intel MPI的参数，提供了Platform MPI的参数，就是没有提供MVAPICH的，队友试着照着博文的参数复刻出一套MVAPICH的参数，然而并没有什么用。

Fun Fact：大好人Marcin在第二天晚上也帮我们试了一下MVAPICH，并给了一套参数，

1	mpirun -hosts hpc-node-1,hpc-node-2 -env MV2_IBA_HCA=mlx5_2 -env MV2_USE_RoCE=1 /nfs/cluster/osu-micro-benchmarks-5.8/mpi/one-sided/osu_get_latency

我队友除了没指明MV2_IBA_HCA以外，其他参数都是这么写的。（不过其实我怀疑加了MV2_IBA_HCA这个就好了，只不过到最后都没时间测）

于是，

Neko.d> 这个MPI大多数的时候能跑，少数时候会炸，说不定是MVAPICH的bug

Marcin> we can take a look and report to Dr. DK Panda.

古有简历直达boss直聘，现有bug report直达author。

到了第二天中午，debug还是没什么进展。我还是觉得好像也不是非得用MVAPICH才行，一看instruction没要求，应该可以换吧。问了其他队，好家伙，有的队打一开始就是拿OpenMPI跑的。于是准备换Intel MPI，这玩意久经考验。虽然说不定Intel MPI能以一己之力扭转程序的性能特征，改变性能曲线的trend，进而颠覆了原论文的结论，但继续卡在这个问题上也不是个办法。用Intel MPI + 自己编译的GCC 9重新编译ramBLe，第一次运行的时候不加MPI参数，程序直接就爆炸了，甚至连计算都还没开始。我还以为又凉了。抱着死马当活马医的心态，试了一下博文里的参数，程序居然能跑起来了！还成功的跑完了！还第一次看到Intel MPI不加奇奇怪怪的参数还跑不了的情况（一般我觉得加太多参数，叠太多buff会让程序炸得更惨）。

buff组合，只能说全是魔法，只有一点代码：

1	-iface ens800f0 -genv UCX_TLS rc,self,sm -genv UCX_NET_DEVICES mlx5_2:1 -genv I_MPI_FABRICS shm:ofi -genv I_MPI_FALLBACK 0

再次重跑之前不能跑的点，诶，能跑出结果了。然而这个时候，比赛时间不太够了，之前用MVAPICH跑的结果又不能用。为了赶上进度，队友还想着一个节点同时跑两个点，然而我这个学期的project就在研究co-located programs之间相互干扰的问题，所以这个想法被我毙掉了，反正大不了就把原来的MVAPICH的数据加个说明交上去凑数，不过其实最后绝大部分结果都赶上了。此外，还有绑核的问题，MVAPICH绑核默认会绑成一个非常奇怪的姿势，但instruction又没提绑核的事情，干脆让Intel MPI自由发挥，用默认绑核的顺序就完事了。最后与MVAPICH的结果相比较，绑核和换Intel MPI都没有造成太大的影响，trend还是一致的。

QE & 神秘应用

这两题都是要花钱跑的，作为一个常年抠门，特别是常年在云服务器上抠门的人（但其实我作为AWS的Intern烧掉了AWS不少钱），自然是不太舍得大手大脚的让QE和神秘应用在八CPU节点或者八卡上做测试的。抠门如我，给了他们一个有一张V100的Intel Haswell的节点，在这个便宜的节点上先测试好，再换到正式环境跑。虽然是Intel的CPU，但Haswell只支持到AVX2，它能跑的代码AMD平台应该也能跑。

其实这两个才是赛前我比较担心的应用，因为这两道题目在赛前就看上去还没有另外两题准备的充分的样子，特别是神秘应用的同学人在香港，只能线上沟通，但结果是，这两道题反而没有遇到太多问题（或者说队友靠自己就闷声把问题修好了），出乎我的意料。QE的同学先开始说CPU版的单元测试跑不过，修了一下，好像是缺了什么库之类的，过了没多久，就说CPU的单元测试跑通了。开始叠优化选项，CPU版的单元测试又跑不过了，他自己鼓捣一下，又能跑过了。之后又开始折腾GPU版，这次遇到问题总算棘手了那么一点，这个问题让他修到了第二天。保险起见，就让他在Day 1到Day 2的半夜用四个节点把保底的结果跑出来，然后去思考用什么机器能刷高分数，跑完benchmarks有闲钱了就去刷高分数（然而最后是有闲钱但没时间了）。如果GPU版本能跑了，可以考虑明天和神秘应用一起时分复用GPU节点。为了省钱，还不让他开八节点来跑（不过仔细一想，其实好像花不了几个钱，但应该能提高不少分数）。

神秘应用那边，Day 1，他上来就列好了用来编译安装程序的所有的命令（他是怎么把命令列的这么全的？他能预知未来？）我看了两眼，给他补充了一些东西，他就自己去折腾了。我预期他会出现各种各样的状况，然而他安静得就像跑路了一样，之间就问过一次MPI相关的（HPCX又跑不起来，换MPI重新编译就好了），Python软件包缺失，NVCC和MPI的环境变量问题，吓得我以为他跑不出来就开始摆烂了。到了Day 2，他遇到了只有在多卡节点上才能测试的东西，给他开了A100节点后没多久，他下一句话就是，

Neko.d> 应该可以连进去（八卡节点）了 @12:25

队友> 现在应该是跑完了 @14:48

这顺利的就离谱。想到跑多节点可能会各种出锅，需要花不少时间用两个GPU节点debug，加上我对我多机多GPU的debug经验和速度都不太有自信，不如把钱省下来给benchmark。神秘应用在A100上跑的时候，QE的GPU版似乎还是不太行的样子。

所有的task在八卡单节点都搞定了。

SC21 Phase 3：演你们！

跑benchmarks的老哥人在英国，大概是Day 2中午的时候终于想起来要上号了，此前训练的时候，

队友> 现在是在训练还是比赛啊？

看起来中国到英国网络不通。

跑IO500不烧钱，他就先跑这个玩玩。起初我们还对CycleCloud自带的分布式BeeGFS抱有期望，先跑了一个五节点的分。第一次，8分，不知道是什么水平，只知道是T队去年的10%不到的分数。虽然对这届IO500来说，我们直接进行一个烂的摆才是最优策略，浪费时间提升10分，在某些打榜专用FS面前，跟没有提升没有区别，但交个8分似乎也太丢人了。又跑了一下单节点的，10分。好家伙，上了分布式还负优化了（大概是通信的锅？或者集群配置有问题？）

跑完IO500热热身，他就......去睡觉了。他那边已经是当地时间的早上了，他再次上号的时候已经10个小时过去了，我们这边已经是Day 2的晚上了，看起来他还挺自信的，睡觉睡得很踏实。

QE和神秘应用跑的非常的经济，愣是省了1k多刀给benchmarks烧。我们的原本想到了A100根本开不出来的情况，就准备退而求其次开V100，然后发现了...V100也开不出来。（估计是T队此时也在拿V100跑benchmark）。好不容易开出两台V100，又发现坑爹Azure的Image里，有IB驱动，但只有完全用不了的IB驱动。Image里只带了支持新网卡的驱动，但机器上只有旧网卡，赛前根本就没试过V100节点的我们防不胜防，想不到还有这种事（然而队友aka前运维去年也是用的V100节点，咋啥都不记得了）。总之，如果硬要用V100节点，我们只能准备手动装OFED。（然而T队后来说他们早在测试的时候就发现了这个问题，估计自动化脚本都写好了）

然而，我从来没见过队友用过clusterssh和其他能broadcast input的terminal，对他自称能在短时间内给10+台节点配好OFED的说法深感怀疑。而且我自己装OFED的时候遇到过升级Linux内核的同时把GPU内核模块整没了的情况，说不定好不容易配好了OFED了，CUDA又坏了，所以不是很放心让他继续把时间花在V100上。

折腾V100的期间，ShanghaiTech的运维问我们要不要接盘他们的两台A100，他们准备释放了，但我那个时候还不知道IB驱动的坑，就没接盘。等到发现IB驱动的坑的时候，我只有非常的后悔，不过我又发现我顺手一开就开出两台A100，就是排了大半天的队。感觉V100那边跑不出来了，干脆不如能开出几台A100就跑几台。算了一下钱，发现，

Neko.d> 出现了有钱花不出去的问题

我们有足够的钱等A100开出来，哪怕最后开出了8台A100，我们的钱也不一定能花的完。并且我大胆假定，大部分的队伍会在比赛末期因为经费不够把A100让出来（后来发现，还得感谢某些搞事情的队高抬贵手，没有跑去抢A100），于是我就让队友一边在A100集群上调参一边等机子。从第三台机子开始，Azure开始0连排队都不让排了，但只要多试几次就能排上队，平均下来进入排队的状态要花10分钟，排队再花10分钟，也就是说20分钟能开出一个节点。反正大家的进度整体良好，没有我运维什么事，我就去当一个没有感情的点鼠标机器好了。

本来想写个脚本自动轮询，发现公司给的Mac不让Chrome访问不安全的网站（CycleCloud Web Console的HTTPS的证书没配好），用不了开发者工具来生成模拟请求的curl命令，所以只能人工点鼠标了。

用了差不多两个小时，总算开出了6台A100节点，但从此再也排不上队了。虽然我觉得没有哪个队钱多到占着A100到最后一秒（然而真的有这种队），但之后真的就一台也开不出来了。

队友那边，他掏出了不知道哪来的祖传HPL和HPCG二进制文件，复制粘贴，执行含有祖传参数的祖传脚本（怕不是ASC那会用的）就开始跑分了！现在Linux的二进制兼容性这么好了吗？不过6节点HPL一开始只能跑出100T左右的成绩，用htop看CPU占用，红红一大片，非常的不对劲。结果就是MPI进程/线程数设置得不对而已，经典Context Switch了。改了改参数，跑分期间看到了不错的预测结果，但居然跑炸了。队友突发奇想，改低了线程数，好了，太怪了。（某队还碰到了非常神奇的Verification failed，开眼界了）

队友自称在校内集群跑HPL的时候成功的把节点直接跑崩，这里没把Azure的物理机跑崩真是谢天谢地了。

后面的故事就比较简单了，48张A100加持下，钱到位了，金钱的力量绝不让人失望。狗贼T队先扔了一个很低的HPL分卖弱，到比赛快结束的时候，终于把真正的成绩放出来了，HPL，HPCG，IO500全部领先当时榜一一大截，仿佛SC20重演。看了一眼我们刚跑出来的分数，那没事了，就让T队开心那么几十分钟吧，然后再让他们感受资本主义的险恶（大雾）。

抖S的队友总觉得没有榨干A100的性能，从理论性能上看，HPL还是有希望跑上300T的，可惜到了最后几分钟也只跑到了284T。但我脑子一抽，居然同意了让他们在最后几秒用这个成绩override掉原来提交的280T的成绩，差点SC20重演，这是这一次比赛距离翻车最近的一次。还好负责提交的同学足够聪明，是先提交再删除，不是先删除再提交。结果是在ddl前提交成功了，但来不及删掉已经提交的成绩了，要是这个update的顺序反了就emmm。但赛方Grafana还是傻掉了，不能处理多份成绩的样子，显示不出我们的HPL成绩，于是我们创造了SC21最高LINPACK分高达0 GFLOPS的奇迹！（大雾）

另外，IO500原先是跑出了10分的“好”成绩，但忘了保存了。再跑单节点的时候又只有8分了，队友直到最后一刻都还想装Lustre来刷高IO500，但就算分数高了一点，也并没有什么*用。

SC21 Phase 3.5：Interview

去年需要我们在最后做一个完整的答辩，今年就变成了Poster+各赛题单独interview。Poster答辩前我还在吹水，完全没有意识到Poster答辩还占分，结果iPad（用来在比赛期间挂着线上会议）突然冒出来一句声音让我答辩，我《完全没有任何准备》（自豪），自然是讲得稀烂，我甚至都不记得我做的Poster里写了啥，只能对着评委说，

Neko.d> 嗯嗯你们自己看看吧，反正这个也不是最终比赛的配置，看看就好

结果发现最后Poster分并不高（我紫菜）。

其他队友的分赛题Interview比我靠谱多了，基本上问题都答得上来（你们是什么时候知道这些赛题的物理背景的？）。特别的是，负责Cardioid的的大二队友不太能说英语，但在我们另一个队友，托福100+的大师的协助下，加上interviewer照顾我们把问题打在了聊天框里，这场interview就在

• 我听不懂Cardioid队友跟翻译说了什么
• 我听不懂翻译跟interviewer说了什么

的情况下完成了。我只知道

• 评委很满意我们的回答
• 他们笑得很开心，但我不知道他们在笑什么
• 评委问：“假如你患了绝症，你愿意用我的模拟器造的心脏吗？”
  • 回答：”去tm的simulation，我直接成为赛博人，肉体是不需要的“

Ending

其实这1w字的流水账早就在10月的时候写完了，只不过拖到了第二年才发出来，欸嘿。说起来这也是我最后一次以正式队员/队长的身份打SC比赛了，想想好气啊，因为这破疫情，痛失2x美帝 2x新加坡 2x北京 1x厦门免费旅游机会，没能帮学校花钱，我感到非常的愧疚。果然还是想续一续我的学生身份，继续努力的帮学校花钱。希望今年能以Year 0 PhD Student的身份打ASC22（offer不会下不来了吧），然后在未来某一年以Presenter的身份回到SC的会场，并成为目击证人，亲眼见证在nike的successor在SC SCC把其他队锤爆的那一刻。

TL;DR

Since NCCL relies on MPI to run on multiple nodes, the following example code is based on MPI Programming Model. Assume there are 4 CUDA GPUs and 4 corresponding MPI ranks. This code performs all-reduce operation within the first two and the last two ranks simultaneously.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

#include <nccl.h>
#include <mpi.h>
#include <stdio.h>
#include <stdint.h>
#include <thrust/device_ptr.h>
#include <thrust/fill.h>

using namespace std;

int main() {
  MPI_Init(NULL, NULL);
  int world_size, world_rank;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
  assert(world_size == 4);
  cudaSetDevice(world_rank); // GPU N binds to MPI rank N

  ncclUniqueId nccl_id, nccl_ids[4];
  size_t id_size = sizeof(ncclUniqueId);

  /* Generate Unique ID */
  // nccl_id is a simple struct with the size of exact 128 bytes
  // so it can be transferred over MPI
  ncclGetUniqueId(&nccl_id);
  MPI_Allgather(&nccl_id, id_size, MPI_UINT8_T,
                &nccl_ids[0], id_size, MPI_UINT8_T, MPI_COMM_WORLD);

  /* Create a sub-communicator */
  ncclComm_t nccl_comm;

  if (world_rank <= 1) {
    ncclCommInitRank(&nccl_comm, 2, nccl_ids[0], world_rank);
  } else if (world_rank >= 2) {
    ncclCommInitRank(&nccl_comm, 2, nccl_ids[2], world_rank - 2);
  }

  /* Test */
  constexpr size_t N = (size_t)1e3;
  constexpr size_t arr_size = sizeof(int64_t) * N;
  void *arr, *arr_host;
  cudaMalloc(&arr, arr_size);
  cudaMallocHost(&arr_host, arr_size);
  
  /* Init the array on local GPU */
  thrust::device_ptr<int64_t> arr_ptr((int64_t*)arr);
  thrust::fill(arr_ptr, arr_ptr + N, world_rank);

  ncclAllReduce(arr, arr, N, ncclInt64, ncclSum, nccl_comm, NULL);
  cudaMemcpy(arr_host, arr, arr_size, cudaMemcpyDeviceToHost);
  printf("[rank%d] result: %ld\n", world_rank, ((int64_t*)arr_host)[0]);

  MPI_Finalize();
  return 0;
}

This code can be compiled and run on my machine with these commands,

1
2

nvcc -ccbin mpic++ test.cu -o test -L/usr/local/cuda/lib -lnccl
mpirun -n 4 ./test

Note: Using nvcc to compile MPI code is not a common practice. It is recommended to compile it with mpic++ from a CUDA-Aware MPI variant.

The output of this program should be,

1
2
3
4

[rank0] result: 1 # 0 + 1 = 1
[rank1] result: 1
[rank2] result: 5 # 2 + 3 = 5
[rank3] result: 5

The key is ncclCommInitRank. Suppose only a subset of ranks initializes the communicator with the same unique ID belonging to one of them. In that case, this communicator will ignore other ranks that are not in this subset.

Usage of ncclCommInitRank

Official API explanation:

1	ncclResult_t ncclCommInitRank(ncclComm_t *comm, int nranks, ncclUniqueId commId, int rank)

Creates a new communicator (multi thread/process version). rank must be between 0 and nranks-1 and unique within a communicator clique. Each rank is associated to a CUDA device, which has to be set before calling ncclCommInitRank. ncclCommInitRank implicitly synchronizes with other ranks, so it must be called by different threads/processes or use ncclGroupStart/ncclGroupEnd.

In addition to the official instructions, we should also know,

• Each unique ID should only be used once.
• ncclGetUniqueId can be invoked multiple times, and it will return a different unique ID each time. Meanwhile, the unique ID generated before is still working.
• It is safe to communicate within disjoint subsets of nodes simultaneously.
• Using NCCL to perform inter-GPU communication concurrently with CUDA-aware MPI may create deadlocks.

Performance

Moreover, I also evaluate the influence on performance bring by sub-grouping.

The testbed is,

• AWS g4dn.metal instance with 8x NVIDIA Tesla T4 GPUs.
• Shipped with AWS Deep Learning AMI
  • OS: Ubuntu 18.04 (Kernel Version: Linux 5.4)
  • CUDA Toolkit: 11.0 (Driver Version: 450.119.03 )

First of all, I would like to emphasize the GPU topology of this bare-metal machine.

Note: We should extract the topology information from physical machines instead of virtual machines since the hypervisor may fuzz the result due to security reasons.

1
2
3
4
5
6
7
8
9
10

# nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity
GPU0     X      PHB     NODE    NODE    SYS     SYS     SYS     SYS     0-23,48-71      0
GPU1    PHB      X      NODE    NODE    SYS     SYS     SYS     SYS     0-23,48-71      0
GPU2    NODE    NODE     X      PHB     SYS     SYS     SYS     SYS     0-23,48-71      0
GPU3    NODE    NODE    PHB      X      SYS     SYS     SYS     SYS     0-23,48-71      0
GPU4    SYS     SYS     SYS     SYS      X      PHB     NODE    NODE    24-47,72-95     1
GPU5    SYS     SYS     SYS     SYS     PHB      X      NODE    NODE    24-47,72-95     1
GPU6    SYS     SYS     SYS     SYS     NODE    NODE     X      PHB     24-47,72-95     1
GPU7    SYS     SYS     SYS     SYS     NODE    NODE    PHB      X      24-47,72-95     1

It looks like a balanced tree topology. We could expect two neighbor GPUs will have higher communication efficiency.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

UPI                   
 |--CPU0              
 |   |--PCIe Switch   
 |   |   |--GPU0      
 |   |   |--GPU1      
 |   |--PCIe Switch   
 |       |--GPU2      
 |       |--GPU3      
 |--CPU1              
     |--PCIe Switch   
     |   |--GPU4      
     |   |--GPU5      
     |--PCIe Switch   
         |--GPU6      
         |--GPU7      

The result below is measured on the root rank, and each experiment is repeated 5 times. Meanwhile, the environment CUDA_VISIBLE_DEVICES was set to reorder GPUs binded to MPI ranks. CPU binding remains unset.

And the meaning of the notations on communicators is,

• 0/1: Only one communicator performing all-reduce on physical GPU 0/1.
• 0/1 + 2/3: Two communicators are working at the same time, and each of them perform all-reduce on two GPUs independently.
• 0-7: Equivalent to 0/1/2/.../6/7.

From the result above, we can conclude that,

• GPUs are working at PCIe Gen3 x8 mode as the PCIe Switch splits one PCIe x16 slot into two x8 slots.
  • Double checked by nvidia-smi --query-gpu=pcie.link.gen.current --format=csv and sudo lspci -vvv
• The GPU Topology will significantly affect the performance of all-reduce.
  • The topology that NVIDIA DGX adopt should obviously accelerate collective communication operations.
• The interference between two concurrent communicators is not quite noticeable.
• UPI bus is not a bottleneck when two PCIe Gen3 x16 devices (PCIe Switches) transmit a large data chunk over UPI bus.

Reference

• NCCL and MPI — NCCL 2.11.4 documentation (nvidia.com)
• Fast Multi-GPU collectives with NCCL

注意事项

Best Practice

理想情况下Docker Image最好使用Dockerfile来构建。把Docker Container当做虚拟机来构建Docker Image这个方法虽然非常省事，但该方法很容易做出来很大一坨镜像，很不轻量，所以仅推荐在测试时使用，不推荐在正式场合（如企业的生产环境）使用。

权限问题

Docker的安装和使用（创建销毁容器等）都需要超级用户权限。若非系统管理员，务必确认环境里已经安装了Docker和拥有Docker的使用权限（已加入docker用户组）

注：Docker也有Rootless模式，但需要额外的配置。

基础知识

容器与镜像

镜像可以说是容器在某一个时刻的所有文件数据，包括运行环境，程序，临时文件等。而容器才是能产生进程运行程序的东西。所以镜像是静态的，容器是动态的。他们的生命周期和转换关系如下

1
2
3
4
5
6
7
8
9

      |---------push--> (Docker Hub)
      |---------save--> (tar File)
      |
Docker Image----run---> Docker Container    
      ↑                       |
      |---------commit--------|
      |---------pull----(Docker Hub)
      |---------load----(tar File)
      |---------build---(Dockerfile)

命名

容器的名字没有太多讲究，镜像名字的构成是：镜像名:Tag，如Ubuntu:18.04的镜像名是Ubuntu，Tag是18.04。

其他实用命令

• docker ps 查看运行中的容器
  • docker ps -a 查看所有容器（包含未运行的容器）
• docker rm -f 删除容器
• docker images 查看已下载的镜像
• docker pull 下载镜像
• docker rmi 删除镜像

DockerHub下载加速

暂未找到什么很好的加速方法

创建容器

推荐命令：

1

docker run -id --name your_ct_name --privileged --network host --restart always ubuntu:18.04 bash

参数含义：

• -d + -i + bash 组合会启动容器里的bash，目的是让容器挂在后台
• --restart always 主机重启后自动启动容器，挂在后台
• ubuntu:18.04 推荐使用Ubuntu 18.04镜像
• --privileged 允许容器使用更多的内核功能
• --network host 使用主机网络（禁用网络空间隔离）

也可以使用老黄家的CUDA开发环境镜像：

1

docker run -id --name your_ct_name --privileged --network host --restart always --gpus all nvidia/cuda:11.0.3-devel-ubuntu18.04 bash

参数含义：

• nvidia/cuda:11.0.3-devel-ubuntu18.04 是包含CUDA 11.0.3对应工具链的Ubuntu 18.04镜像
  • 镜像的CUDA版本需要和驱动支持的版本对应，nvidia-smi 右上角会显示最高支持的CUDA版本
  • devel版镜像包含nvcc编译器等工具链，runtime版不含工具链
• --gpus all 使用所有可用的GPU

所有黄家容器列表：https://hub.docker.com/r/nvidia/cuda/tags?page=1&ordering=last_updated

进入容器（交互式bash）

1

docker exec -it your_ct_name bash

参数含义：

• -i + -t 启动交互式模式

进入容器后如果想换apt源，建议使用下面的命令来换，因为镜像为了节约空间，往往不包含文字编辑器

1	sed -i "s#archive.ubuntu.com#mirrors.sustech.edu.cn#g" /etc/apt/sources.list

容器、镜像、文件的转换

容器到镜像

1

docker commit your_ct_name your_images_name:your_tag

镜像到文件

1

docker save your_images_name:your_tag -o your_file.tar

文件到镜像

1

docker load -i your_file.tar

TL;DR

在以下前提下，上CDN不如把网站搬到24块一个月的腾讯云香港轻量服务器

• 回源稀烂
• 不考虑国外访问
• CDN本身不太行
• 腾讯云香港还没有被玩坏

已知的性价比最高的方案是Azure CDN (Microsoft Standard) + GitHub Pages。但网站本身才是影响访问速度的大头。

注：本文讨论的前提站长懒得备案。如果备案了，毫无疑问国内回源+国内CDN能吊打上述方案。

上CDN一定提速吗？

啊这里有人可能会问了：“这个问题有必要问吗？”但仔细思考一下，免备案的CDN边缘节点最近就在香港，考虑到边缘节点是个公交车，还要服务别人，线路不一定比腾讯云的强。更令人智熄的是，基本上个人博客可以和访问量大不到哪去画个等号，你的网站被挤出Cache可太正常了，Cache Miss一下延迟就更爆炸了。无脑上CDN后发生的大概率事件就是CDN打不过三网直连的24块钱的腾讯云香港。

那怎么样才能让CDN搞快点呢？当然是头痛医头脚痛医脚，

• 回源问题：弄一个好点的源站就好了，当然CDN本身的Cache策略和能力也非常关键
• 边缘节点承载能力的问题：弄一个好点的CDN就好了嘛

所以问题就被简化为回源和CDN的选择问题了。

源站的选择

弄一个好点的源站，说着容易，实际上源站的选择还是很讲究的。

源站的类型

对于静态网站（如Hexo）来说，源站除了可以搭建在VPS上，更建议扔在对象存储上，因为

• 更高的SLA：自己的VPS维护不当崩崩崩可太常见了，对象存储的SLA动不动就是99.9%以上
• （可能）更高的性价比：个人博客一般不会太多空间，除非存了一堆视频。性能上可能有大厂的神秘优化
  • Azure Blob Storage：实测每天大概0.1-0.2美元，小贵
  • AWS S3 Bucket：免费额度应该能cover，计费也比Azure便宜

对象存储的缺点主要是

• 第一次配置比较复杂，企业级的云通常需要反复折腾IAM权限
• 为了忽悠你买CDN，对象存储对换用自己的域名和HTTPS的支持多少会有问题
• 有坑（如果用一个厂家的全家桶的话坑会少一点）
  • Azure的官方文档就不会告诉你上传到\(web文件夹（容器）的\)在Linux下要转义
  • AWS的S3 Bucket不兼容GeoDNS
  • ...

不过调通了以后同步对象存储数据就像用网盘一样简单（因为这就是个网盘）。

地理位置选择

之前也提到了，个人博客上CDN就要时刻准备好Cache Miss回源，所以缩短边缘节点从源站下载数据的时间非常的关键。解决这个问题最好的思路应该就是缩短边缘节点到源站的地理距离，最好在同一个地区，因为

• 更短的延迟：这个没啥好说的，光速再快，理想情况下数据在中美之间走一圈都140毫秒起步
• 更大的带宽：一般来说城域网之间通信的带宽比国际线路的带宽大多了

另一个好处就是，因为只需要考虑同地区内的通信，所以源站的国际线路质量完全不需要考虑，什么CN2 GIA都完全不需要，源站能通网就行。

由于一个源站只能照顾一个地区，如果只考虑国内访问的话，一个香港源站应该就足够了。但如果要

• 照顾全世界的人民
• 照顾开着美国梯子的自己
• 刷高PageSpeed分数来优化SEO（谷歌应该是从美国访问你的网站）

，就可能需要不止一个源站了。

多地区延迟优化

多地区的优化是玩具级解决方案和企业级方案的分水岭之一，为啥这么说呢，因为从相关服务的定价来看，基本上云厂家就没考虑过个人玩家的死活。对于CDN来说，就是配置多个源站（这既是为了降低延迟，也是为了容灾），让CDN能根据访客的位置选择最近的源站。这大致有两种实现

• CDN自身支持多个源站并能选择最优的
  • Azure Front Door直接支持多个后端，并且可以自动根据延迟选择后端
  • Azure CDN (Standard Microsoft)的Rule Engine可以为不同地区指定一个源站
• （GeoDNS）让DNS根据地理位置将域名解析到不同后端，CDN通过这个域名回源。支持这个功能的DNS有
  • Azure Traffic Manager：大概4港币增加一个源站，30港币每百万解析
  • AWS Route 53：看到每月几十美元一个Policy Record后就没继续了解了
  • DNSPod（腾讯云）：360rmb每年，不乐意了
  • 阿里云：免费版能按国内外区分（可香港也算国外，这没有区分度啊），从198rmb一年的企业版开始可以细分国外的国家地区

我都要！但我没钱咋办

多个源站，GeoDNS都是烧钱的东西（企业人傻钱多不在意），那普通人咋办？这里就要介绍这个无敌的存在了，GitHub Pages。这玩意除了能免费给你存东西以外，还安排上了Fastly CDN。GitHub Pages的架构我们不得而知，但从测速结果来看，很多地方的测速点测出的访问延迟都很低，应该是有做数据的geo-replication。也就是说DNS不用买，多地区存储也不要钱，唯一的毛病就是国内访问比较随缘，但作为CDN的源，这个毛病无伤大雅。而且像Hexo这种静态博客，甚至有插件能一键同步博客到GitHub Pages上。

CDN的选择

国内访问速度

根据我这两年来的观察，我主观的将我用过的CDN按照国内访问速度分为几个等级。

• T0：能和腾讯云香港五五开
  • Azure CDN：反正就不知道为什么它的香港节点又稳又快
• T1：不一定能干过腾讯云，但可能跑得赢CN2美国VPS的
  • AWS CloudFront
  • UDomain
  • CloudCone
  • 这三家都有香港节点，但是表现属于时好时坏的那种
• T1.5：可能跑得赢辣鸡线路美国VPS的
  • Cloudflare：免费的还要什么自行车，主要是免费版没给香港节点，但美国节点的表现不算差

至于国外网站访问速度估计大家都差不太多。

定价

• T0：看看就好
  • Azure Front Door：背靠Azure CDN (Standard Microsoft)，一条Rule也就每月170港币（至少会有一条，躲不掉的）
• T0.9：勉强可以接受
  • AWS Lightsail Distribution：背靠CloudFront，5美元50GB，但可惜用不完
• T1：穷人友好
  • Azure CDN (Standard)：真正的按量计费，1港币1GB，5条免费Rules，Azure少数不贵的东西
  • CloudFront：按量计费，0.12美元1GB
  • UDomain：按量计费，1.2港币1GB，充值的方式很怪，非常不现代
  • CloudCone：按量计费，0.045美元1GB，需要首充20美元的样子
• T2：博爱
  • Cloudflare：套餐0元起步

结论

结论其实很明显，我肯定首推Azure CDN (Standard Microsoft)，因为Front Door这个价格就离谱，其他家的CDN会让你怀疑为什么要花这个钱买个减速器（当然CloudFlare配合廉价美国VPS能省钱）。当然Azure确实比较高冷，首先得有张外币卡，然后就是各种问就是企业级的设计，以及莫名其妙的设计，比如说对根域名不友好，CNAME验证各种不通过，官网文档只会让你去买他家DNS，用cdnverify绕过的方法就是不说，也不给根域名自动签TLS证书（AWS就可以）；不可以CDN前端用HTTPS后端HTTP（Front Door倒是可以）等等。但没办法谁让他家CDN国内访问就是快，看在价格不贵的份上原谅他了，免费5条规则也算良心，可以拿来配HTTPS强制跳转和HSTS，虽然这些东西可能在别家CDN面板上一键就能配好。

Summary

Azure CDN (Microsoft Standard) + GitHub Pages这套方案可能比较绕，但一个月花不了几个钱（估计5rmb不到）速度又倍棒。不过还有一个问题值得思考，上这套方案就能让网站访问速度无人能敌？其实不是，从谷歌PageSpeed的分数看来，我从单回源（新加坡）+AWS CDN换到上述这套方案，PageSpeed也就提升了3分左右（国外访问速度）。另外提升的20多分靠的是对网站自身的调整，如减少了外部文件的加载数量。我曾今遇到过一个高度优化的网站，哪怕用的是Cloudflare，走国内国外网络的PageSpeed都是满分（用Chrome Lightroom测试）。

写到这里我才意识到这套方案最大的意义是给我省了一点钱，比起上腾讯云还便宜了不少，顺便提升了国内访问速度。

HloSharding Object

First of all, we need a way to represent sharding specifications using programming language. XLA designed an object to do such a thing, and this object contains numerous variables and a set of supporting functions to configure itself. Some attributes of HloSharding are listed below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

// File: tensorflow/compiler/xla/service/hlo_sharding.h

class HloSharding {
  bool replicated_;
  bool maximal_;
  bool tuple_;
  bool manual_;
  // This field is only used if replicated_ is false. If maximal_ is true, then
  // the field contains a rank 1 array with a single element, which is the
  // device the HLO is assigned to. If maximal_ is false, the field contains an
  // array with the same rank as the corresponding HLO. The dimension sizes of
  // the array describe the number of ways the HLO is partitioned along each
  // dimension. The values of the array specify which device each tile of
  // the HLO is assigned to. The index of each value determines which tile it
  // takes.
  // For example, {{{2, 3}}, {{5, 7}}} (whose ToString representation is
  // "{devices=[2,1,2]2,3,5,7}"), means that dimension 1 is split two way and
  // dimension 3 is split 2 way. Core 5, whose index is [2,1,1] will take the
  // tile that contains the 2nd half of dimension 1 and the 1st half of
  // dimension 3.
  Array<int64> tile_assignment_;
  // Only non-empty when tuple_ is true. If a tuple is empty then one entry is
  // present for the root. This is a flattened list of all the leaf shardings in
  // a tuple shape, by pre-order walk (ShapeTree iterator order).
  std::vector<HloSharding> tuple_elements_;
  // This flag is to support partial replication and partial sharding. If it is
  // true, tile_assignment_ will have an extra dimension in addition to the data
  // shape rank, and the added last dimension represents the subgroups of
  // replications, i.e., elements in slice [..., :] will be replicated.
  bool replicate_on_last_tile_dim_;
  // This field is used to track the source of this sharding, usually derived
  // from instructions. Multiple metadata may be populated if sharding is
  // combined with other shardings. Metadata are to not be populated when
  // tuple_ == true and instead metadata should be set on individual tuple
  // elements.
  std::vector<OpMetadata> metadata_;
};

Array<int64> tile_assignment_ here is multi-dimensional with arbitrary shape. {devices=[2,1,2]2,3,5,7} means the shape of tile_assignment_ is [2,1,2], while the values are {2,3,5,7}.

std::vector<HloSharding> tuple_elements_ probably was designed to specify the sharding specifications of outputs.

I am not aware of what the roles of maximal_, tuple_elements_ are. Is there any body know that?

Note that each single object could be shared by multiple instructions. By doing this, the cost of creating and maintaining several instances with the exact same contents could be eliminated.

Extended HLO IR Attribute

The original implementation of XLA added the attribute std::shared_ptr<const HloSharding> sharding_ to the class xla::HloInstruction, which is declared in tensorflow/compiler/xla/service/hlo_instruction.h. A common usage of this HLO Instruction Attribute is to declare sharded tensors. Here is a sample HLO IR code with sharding attributes. Note that the Propagation Algorithm may fill in this attribute for those instructions without it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

primitive_computation_add.6 {
  parameter.7 = f32[] parameter(0)
  parameter.8 = f32[] parameter(1)
  ROOT add.9 = f32[] add(parameter.7, parameter.8)
}

ENTRY xmap__lambda_.12 {
  constant.2 = pred[] constant(false)
  parameter.1 = f32[8]{0} parameter(0), parameter_replication={false}, sharding={replicated}
  custom-call.3 = f32[8]{0} custom-call(parameter.1), custom_call_target="Sharding", sharding={devices=[4]0,1,2,3}
  sine.4 = f32[8]{0} sine(custom-call.3)
  constant.5 = f32[] constant(0)
  reduce.10 = f32[] reduce(sine.4, constant.5), dimensions={0}, to_apply=primitive_computation_add.6
  ROOT tuple.11 = (f32[]) tuple(reduce.10), sharding={{replicated}}
}