WSL Network Stack Crashes After Heavy Load

[sbouab]

I am running WSL2 on Windows 10. I have docker containers (not using docker desktop) running in it that I send HTTP requests to through an application on my host. One particular container is a database. At some point, the number of HTTP requests to localhost:DB_PORT becomes large and that's when something happens to WSL networking because from that point I am not able to resolve localhost anymore (i.e. I can't reach localhost:DB_PORT through the browser even though I could do so before). I get the following message in my browser: ERR_CONNECTION_RESET.

So that's my main symptom. Interestingly enough if I use the IP of my WSL distro and the same port number WSL_DISTRO_IP:DB_PORT, I am able to reach the DB page. So it's something about resolving localhost that breaks (I would think so)?

Every time this "crash" occurs, I see through dmesg in my WSL distro the following:

[ 2353.973370] WSL (159 - ) ERROR: UtilAcceptVsock:251: accept4 failed 24
[ 2353.973875] WSL (159) ERROR: No file descriptors available @localhost.cpp:49 (ListenThread)

I'm not sure how it's related to what I'm experiencing, but this always appears in dmesg the moment things break.

Once I run wsl --shutdown and then wsl to restart WSL, everything starts working again just fine.

I have been able to run my tests without any anti-viruses, firewalls interfering without producing a different result.

[github-actions]

Logs are required for review from WSL team

If this a feature request, please reply with '/feature'. If this is a question, reply with '/question'.
Otherwise please attach logs by following the instructions below, your issue will not be reviewed unless they are added. These logs will help us understand what is going on in your machine.

How to collect WSL logs

Download and execute collect-wsl-logs.ps1 in an administrative powershell prompt:

Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/WSL/master/diagnostics/collect-wsl-logs.ps1" -OutFile collect-wsl-logs.ps1
Set-ExecutionPolicy Bypass -Scope Process -Force
.\collect-wsl-logs.ps1

The script will output the path of the log file once done.

If this is a networking issue, please use collect-networking-logs.ps1, following the instructions here

Once completed please upload the output files to this Github issue.

Click here for more info on logging
If you choose to email these logs instead of attaching to the bug, please send them to wsl-gh-logs@microsoft.com with the number of the github issue in the subject, and in the message a link to your comment in the github issue and reply with '/emailed-logs'.

[sbouab]

/question

[github-actions]

Diagnostic information

Found '/question', adding tag 'question'

[justus-camp-microsoft]

Hi @sbouab, thanks for reporting this issue. We have a fix inbound for this , but as a temporary workaround you can run ulimit -S to increase your soft file descriptor limit to something high (1 million for instance)

[sbouab]

Hi @justus-camp-microsoft thanks for answering. Could you elaborate how a lack of FDs can cause the localhost to become unresolvable? Is there a link between these events I'm failing to see?

[sbouab]

From what I understand, it's a localhost listener that has no FDs available, but somehow it crashes because of that and never recovers?

[sbouab]

Also, I have tried raising the limit of file descriptors, but still same issue occurs. I can confirm that the limit was properly raised.

[justus-camp-microsoft]

Hi, I just realized I left off an n in the command I originally sent. Can you run ulimit -Sn 1000000 and see if that fixes the issue?

[sbouab]

Yup that's what I tried, and still nothing. the localhost event listener (or socker or whatever it should be called) does not get the FD limit changes applied to it it seems.

[sbouab]

Idk if you can share this info, but where is localhost.cpp located?

[justus-camp-microsoft]

It looks like this limit is inherited as part of WSL's initialization process and as such you'll actually need the fix we implemented. We have a fix inbound for this.