Don’t Push To Production On Friday

Seeing tweet https://x.com/GithubProjects/status/1847179876308341192 and it shows

Image

Very interesting but it is absolutely true. I got myself into troubles on pushing production on Friday afternoon and the whole weekend was broken. Then I changed to push on Monday afternoon. The reasons are

  • After working from Monday to Friday, I need rest. Fixing thins in weekend is awful. No one pair with you, no one want to work in weekend.
  • Push on Monday afternoon, usually small bugs feedbacks comes on next day morning. The whole team are supporting.
  • Fix things quickly and got nice response.
  • DO NOT worry about accidents. Before every publish, make sure a plan B: no critical data loss(it is OK if lost data having backup). Then accident is not a big deal. 😛

Fix gpg-agent for ssh in Xubuntu 24.04

It is a nightmare to use gpg-agent in Xubuntu 24.04 to protected ssh private key. I had to resolve several issues to make gpg-agent works.

Make gpg-agent started on login

https://docs.xfce.org/xfce/xfce4-session/advanced#ssh_and_gpg_agents

In Session and Startup, Disable the Launch GNOME services on startup in the Advanced tab of the settings dialog.

In case you want to use gnome-keyring enable the Launch GNOME services on startup in the Advanced tab of the settings dialog. If you do this xfce4-session will not attempt to start the gpg- or ssh-agent.

Enable gpg-agent’s enable-ssh-support

echo 'enable-ssh-support' >> ~/.gnupg/gpg-agent.conf

With this setting, /etc/X11/Xsession.d/90gpg-agent will create SSH_AUTH_SOCK environment variable.

Fix “has_option” error in /etc/X11/Xsession.d/90gpg-agent

https://github.com/canonical/lightdm/issues/198

lightdm call /usr/sbin/lightdm-session to run X session, while this script has not defined has_option function like /etc/X11/Xsession and this function is needed by scripts under /etc/X11/Xsession.d. So ssh agent start failed.

cat <<\EOF | sudo tee /etc/X11/Xsession.d/20x11-add-hasoption-workaround
# temporary fix for https://github.com/canonical/lightdm/issues/198 and https://bugs.launchpad.net/lightdm/+bug/1922414
OPTIONS="$(
    if [ -r "$OPTIONFILE" ]; then
      cat "$OPTIONFILE"
    fi
    if [ -d /etc/X11/Xsession.options.d ]; then
      run-parts --list --regex '\.conf$' /etc/X11/Xsession.options.d | xargs -d '\n' cat
    fi
)"

has_option() {
    # Ensure that a later no-foo overrides an earlier foo
    if [ "$(echo "$OPTIONS" | grep -Eo "^(no-)?$1\>" | tail -n 1)" = "$1" ]; then
      return 0
    else
      return 1
    fi
}
EOF

Verify

After login, run

env | grep SSH_AUTH_SOCK
# should display
# SSH_AUTH_SOCK=/run/user/1000/gnupg/S.gpg-agent.ssh

Use iPXE to install ubuntu 22.04 over HTTP without DHCP and TFTP

Background

Sometimes I had to install ubuntu to remote HP/Dell server. Both HP and Dell has remote KVM to allow user to management server. User can mount local iso file as virtual CD-ROM device for server to boot and install new operation system.

Well some aged servers has very low speed to read local iso data: only 3Mbit/s(375KB/s) even though the network speed is 100Mbit/s. Their remote management chips are performance limited. While ubuntu 22.04 iso is 1.83GB, it will take hours to transfer before loading setup UI.

Dell’s has iDrac remote file virtual media, it can mount CIFS(samba), NFS or http image file. I tried NFS and http, both were failed to load image file without any useful error messages.

A better way is netbooting. Well it needs DHCP, tftp server. But some IDC are manually IP address assignment, there are no such services in those IDC.

It is better to setup temporary DHCP and tftp services, it make life easy. If there is no way to do this, continue reading.

Requirements

Netboot without DHCP and TFTP

  1. Extract casper/vmlinuz and casper/initrd file from ubuntu live server iso.
  2. Setup http service to provide vmlinuz, initrd and ubuntu iso file (simple setup with one command: python3 -m http.server)
  3. Mount ipxe.iso as virtual CD-ROM to server (HP via ILO and Dell via iDrac)
  4. Reboot server and boot with virtual CD-ROM.
  5. Press CTRL+B to enter iPXE shell mode
  6. Enter following to boot ubuntu iso

Open network interface, set first network interface IP address and gateway. With ifstat can list server interfaces list. Choose the right one.

ifopen
set net0/ip 192.168.56.10
set net0/netmask 255.255.255.0
set net0/gateway 192.168.56.1

Loading ubuntu boot kernel and initrd over HTTP

kernel http://192.168.56.1:8000/vmlinuz
initrd http://192.168.56.1:8000/initrd

Set ubuntu kernel boot parameters to download and run live iso. After iPXE booting ubuntu boot kernel, its IP address was discard. So we need to specified network parameters to boot kernel to let it setup network interface for further download live iso.

imgargs vmlinuz url=http://192.168.56.1:8000/ubuntu-22-server.iso ip=192.168.56.10::192.168.56.1:255.255.255.0::eth0:none:192.168.56.1 boot=casper net.ifnames=0 biosdevname=0

Start to boot

boot

Note

Static IP parameter format in casper man page is wrong. It shoule be $ipaddr::$gateway:$netmask::$DEVICE:$proto:$dns

$DEVICE can be eth0, eth1, and so on. net.ifnames=0 biosdevname=0 disable network interface renaming.

Reference

Switch Windows 11 to Xubuntu

After tried several times to install Windows 11 22H2 to my old laptop(Lenovo X201s), maybe it is time to leave windows for my old devices.

As a developer, keep operation system up to date is important for security.

Xubuntu is my favorite Linux distribution, lightweight with less resource usage and hardware requirements, suitable for my old laptop. Yet its UI is still fashion to me.

So far so good:

  • Browser with Firefox
  • OpenVPN for remote working
  • snap and flatpak allow to run containerd application without worried about so hell.
  • Works with brother printer

Life is wonderful if can live without China Mainland software.

NFS Suck: NFSv4

There are 2 kind of NFS v4 production issues since this year. After tired troubleshooting problems, I changed to NFS v3, and wonderful life returned.

Incompatible NFSv4 Server Implementation (QNAP)

On April 9th, Saturday, all flatcar container Linux virtual machines rebooted on scheduled to upgrade OS in early morning.

Then at about 10:00 AM, some users reported slow service responses and errors. Remote access to Kubernetes cluster, some pods were in pending or error status. There was a pod event message reported a NFS volume was not ready.

Tried deleting pod and let K8S auto recreated a new one, problem was not fixed.

Not all NFS volumes had problems to mount, it mean that NFS client in flatcar container Linux worked well. All failed NFS volumes pointed to same NFS server: the QNAP device.

Remote login to a flatcar container Linux host, tried manually mounting QNAP NFS volume with verbose option turn on and different mount options. NFS was unable to mount with NFS version 4.1 and 4.2. It worked well with 3.0 and 4.0.

So I changed NFS volume mount in K8S configuration to force use 4.0 and services recovered.

I found another guy had some problem and report to GitHub issues. I provided my situation for QNAP devices. Luckly, the issue reporter had some root cause and my discovery did help him. Flatcar container Linux developer provided more detail information from Linux Kernel Forum.

Linux kernel used in QNAP device had wrong response on a capability query caused new version of NFS client hung.

NFS4 state manager thread stuck

From time to time, different AMD EPYC server had very slow NFS IO operations while NFS server IO was not full busy. And finding one kernel thread was 100% CPU usage for a long time. Thread name was <NAS_SERVER_IP>_ma. After searching, a RedHat solution explains the same issue: RHEL7.4: NFS4 client hung due to NFS4 state manager thread stuck inside nfs_reap_expired_delegations infinite loop.

I had no RedHat subscription and could not view solution detail. All I could do was upgrade Ubuntu with latest stable Linux Kernel and reboot.

This kind of issues happened from time to time and one day I found

NFSv3 is a stateless protocol. If server reboots and comes back up, client can pick up where it left off. No state has been lost.

NFSv4 is stateful. The locking operations are part of the protocol proper. NLM is not used by NFSv4.

Differences between NFSv3 and NFSv4

Since no NFSv4 state features(locking operations) needed and performance are almost the same. I changed all NFS volume mounts to NFSv3.

How wonder life without NFSv4!

Performance Tuning Is Hard

In recent 3 months, I did improve several applications performance tuning, gained much reduced memory usage and CPU time. Model training time takes from days to hours. Well, it is hard to achieve such success.

Minimal Runtime Environment

There are many small scripts and C++ applications during model training. Finding the most time consuming application to tune is easy since model training users can give a directly, correct feedback: which step takes a lot of time. Then review source code of that model training step. That step may contains several scripts and compiled applications. Which one take the most of the time? Users can show you via Task Manager (in windows) or top (linux).

What next is to prepare minimal runtime environment. It may be difficult since there are many trivial things (environment variables, config files, input data). Performance tune is not a one hammer done job, application to-tuned will be run many times. Minimal runtime environment let you focus on one application, easy to reproduce and verify.

Profile: Finding Bottleneck

With minimal runtime environment, use profiler to find bottleneck, like perf. To profile memory usage, use heaptrack to collect memory allocate/free operations.

Different languages have different official profile solutions. All of them generates similar results (CPU profiling). Such profile reports tell you where the bottleneck functions are.

For short but running slow python script, there is another way to find bottleneck. During python script running, press CTRL+C to stop running, python will print out stacks on stop time. Try several times, if printed stacks are same, you know the slow code is.

Pay attention to python slice in operation: item in slice. If length of slice is larger than 1000, it is really slow. Use set or dict, performance is constant with whatever size of input data.

Make It Correct, Then Make It Fast

Once finding the bottleneck functions, it is far from changing code to improve performance. You should answer the question: How can I prove the changes generate the same result?

Know It Well

Associate with input data and configuration, make sure you full understand the application source code.

  • Application architecture, how source code layered, how components initialized.
  • Data flow: where input data and configuration used and where generated the output.
  • No need to look into every line of source code, draw an outline of functions call hierarchy.
  • Finally, make sure fully understand every single line of source code of bottle-neck functions needed to be tuned.

Unit Test

If there are unit tests, you get luck. Keep unit tests running passed. Add new unit tests for your changes and it also make better unit test coverage.

In most of cases, there is no unit tests at all. Well it is your choice whether to add unit tests. If you do, it is a tough choice and keep your manager know there are much extra efforts:

  • Import unit test library
  • Add unit test step in CI build script (important, keep it running on every commit)
  • More source code changes to make sure unit test stable (dependencies decouple)

If there is no nearby deadline, it is recommended to do such tough choices. Like someone said: It is difficult for one generation, but good harvest for next generations.

Sample Expected Result

With prepared input data and configuration, expected result data can be generated. Make sure you can easy to compare two result data to tell that they the equal. Sometime, result data is huge.

  • md5 checksum. If result data should be strictly equal in bit data.
  • diff: Good for small and text results.
  • custom script: JSON file with different field orders. Float number data equality.

Inspiration

Some times it is hard to think out a better way or right way to improve performance. There may be several solutions out of your mind and you will deny them one by one after double thinking.

  • If new solution has more steps, it is likely a worse solution.
  • The more you learn the business model of source code, the more you find hidden or forgotten requirements. It helps making correct decision.
  • More failures you make, much near to final success. Don’t give up.
  • It is not always be succeed, put the tuning down for some time, it is just not the right time for you to resolve yet.
  • As tuning problem and source code keep thinking over, the God will give you a silver lighting. And you know it, then miracle happens.
  • Keep communicating with others, especially users or architectures, some legacy requirements can be dropped. It is another God’s gift to bright way to succeed.

During the whole process, you may fell lonely, like a single boat in the middle of ocean. Nobody helps you. And nobody cares whether your boat can arrived at beach(Although applications does not having good performance, the world is still turning.).

Don’t give up!

Don’t give up!

Don’t give up!

Julia to Rust, 450% performance boost

To prevent computing node resource exhausted by buggy scripts or configurations, computing task was wrapped into a docker container with resource limits. The difficult thing was to accurate estimate memory resources requirement for a task. Different task needs different amount of memory for different input size.

I was stucked on Julia tasks. Julia seems take much more memory and killed due to cgroup OOM. It seems Julia GC engine has no aware of cgroup memory limit. And my guess was correct: cgroup aware Base.Sys.cpu_info and Base.Sys.total_memory.

I watched this issue, waiting it to be closed. Before that, Julia task in container have no memory limit to void OOM killed.

After waiting 2 months, the issues had no changes. Then I decide re-writing Julia code to Rust

  • Can accurate estimate memory resources: no JIT, no GC
  • Codebase size of Julia was small: only 19 files.
  • Julia code was unchanged for a long time: stable, no new features to track.

Rewrite to Rust

With luck, those Julia scripts were algebra analysis, pure functions. I translated one Julia function one by one, each having unit test.

The whole translation took 2 weeks, longer than I expectation. Julia has some special operators that took times to find document of what it means and different behavior on different operator data type.

The performance of Rust version is much faster than Julia: no JIT time.

Memory Tuning

Rust version take little more memory than expectation. To find out memory usage during runtime, I used heaptack to collection memory allocate/free information.

With heaptrack, I reduce trivial memory usage

  • Use move instead of clone if capable
  • Remove temporary vector and string objects

Multi-Thread

Thanks to Rust borrow check and memory safe static analysis, it is easy and confident to use multi-thread to boost calculation.

Use perf/gdb to analysis multi-thread bottleneck and rewrite function to improve parallelism.

Final Result

Input Data: 20 File. Each File: 178,392,481 bytes (171M). Total: 3.4 GB

Julia Version

user: 35.49 Seconds
sys: 16.84 Seconds
real: 52.34 Seconds
Peak Mem: 12,479,792 KB

Rust Version

user: 24.30 Seconds
sys: 14.47 Seconds
real: 9.41 Seconds
Peak Mem: 5,482,936 KB

CPU time: 456.21% improved

Memory usage: 56% reduced

Start docker container quickly with shared network stack

Background

A worker node in distribution computing system is to

  • Receiving job request from scheduler
  • Execute job
  • Report job status

For job executor in worker node, job execution is just to execute specified command:

  1. Create sub process and run
  2. Report process status at regular intervals
  3. Report final process status after process ends

After moving sub process to standalone docker container

  • Allow dependencies library customization with different docker image.
  • Resource limit to void buggy application taking all worker node resource.

Issue

Production Worker Node

  • Ubuntu 20.04.04
  • Docker 20.10.17
  • AMD EPYC 7763 64-Core Processor X 2 ( total 256 thread)
  • 1TB Memory

It worked as designed on test environment. In production, from time to time, there were jobs reported heartbeat timeout. After further investigation, those timeout jobs were start too late in worker node. In other words: After scheduler pushed job request to a worker node, the job’s first heartbeat was not send to scheduler in time(5 minutes).

After job container started, job executor will send first heartbeat before start sub process. Why worker node start docker container slowly?

To start a docker container(via docker API), it needs fetching docker image, creating container and starting container. Since all jobs use same docker image, the first step is skipped. From logs in worker node, creating container always take about 10~50 milliseconds. While starting container takes from 400 milliseconds at beginning. As more running container existed, this action takes much longer time, to seconds. Job maybe waiting in queue too long to start in time.

There are related performance issues in GitHub:

Solution

During start container, kernel needs to setup isolation environment for new container. Network isolation creation take much longer if there are many running containers. Github issues suggest using host network mode to mitigate long start time.

In other words, use existing network stack will reduce much container start time. Instead of using host network to expose worker service to outside world, I created a dummy container which sleep forever. All job containers use this dummy container network stack archived same performance improvement of using host network mode.

After reusing existed network stack, container start time take from 80 milliseconds to 600 milliseconds, less than 250 milliseconds most of time. No more job timeout reported.

IPv6 in China: Terrible

My son complained the homework app in iPad could not load contents, it always reported network issues. Well I was sure that there was no problem of ISP network.

Luckly, the homework app had built-in network diagnostics, and it reported the API services not response in time. The api host is api.17zuoye.com.

Then I tried nslookup -Type=AAAA+A api.17zuoye.com, the DNS responses

Name:    api.17zuoye.com
Addresses:  2408:4000:200::1b8
          42.240.152.156
          42.240.152.154
          42.240.152.157
          42.240.152.155

There was a IPv6 address, iPod prefer IPv6 over IPv4. I tried curl -v -H 'Host: api.17zuoye.com' http://[2408:4000:200::1b8]，it responsed timeout 2 times in 3.

There is no setting for iPod to prefer IPv4 over IPv6. While public IPv6 is necessary for me to remote access and cannot disable it.

What about discard AAAA dns record? I found a MikroTik firewall rule to drop AAAA dns request.

/ip/firewall/layer7-protocol
add name=DNS_AAAA regexp="\\x1C\\x01"
/ip/firewall/filter
add action=reject chain=forward comment="Filter AAAA UDP DNS Requests" dst-port=53 layer7-protocol=DNS_AAAA protocol=udp reject-with=icmp-protocol-unreachable
add action=reject chain=forward comment="Filter AAAA TCP DNS Requests" dst-port=53 layer7-protocol=DNS_AAAA protocol=tcp reject-with=icmp-protocol-unreachable

Then the homework app worked well.

koreader: non-touch improvement stage 2

I started to plan stage 2 improvement after PR “Improve reader config dialog for Non-Touch devices” merged. It was a big goal that trying to fix most of non-touch UI accessibility issues.

FocusManager needed more enhancements after several days planning and thinking over.

  • Handle press event and convert to a tap event. This will reduce huge widget non-touch accessibility effort.
  • Add ability to define custom key binding to support hold event. Some important features are activated by hold event.
  • Add ability to define custom key bindings to more focus move actions. This reduced key press times to focus to a UI widget.

Then started to do improvements. To add non-touch accessibility to a widget, there are several steps:

  1. Make widget inherit FocusManager instead of InputContainer
  2. Enable touch event(tap, hold etc) on non-touch devices
  3. Make sure key_events field is not overrided.
  4. Add widget’s UI element to layout

The last step is full of challenge. Some widgets regenerate their UI element on interaction. I had to follow widget routine to make sure the layout is update to date. Sometimes I sucked on wired UI behavior.

I started development at night. As one widget was done and tested, I started on another widget. I enjoyed the work and cannot stop it. I finally felt tired at 4:00 AM. Submit changes to GitHub and go sleeping. Fortunately, next day was Saturday, I could sleep late.

After another two working night, I achieved my goal. During 7 days of Chinese festival spring holiday, I did as many tests as I can and adding unit tests. Revert changes and redo to refine code quality.

After koreader 2022.02 released, I submit the PR “Non touch improvements“

Lesson: Initial version of code is far to perfect.