Test suite breakage: Tests keep too many file descriptors open, breaks with concurrency

[lilyball]

When running the test suite (make check-stage2-std is sufficient to reproduce), in the middle of the tests the program aborts, often with a cute abort message. There's a few different kinds of aborts, such as fatal runtime error: assertion failed: void_sched.is_not_null() or error opening /dev/urandom: Too many open files, but they all seem to be caused by running out of file descriptors. The default limit on my machine (OS X) is 256, and if I catch the abort (with lldb) I can see that they're all in use.

This problem seems to have been triggered by 8afec77, which was introduced into master by PR #7265. This commit changes the default number of concurrent test threads from 4 to rust_get_num_cpus() * 2. Experimentally, anything above 6 causes the test failure, and my machine reports 8 CPUs so the test suite is attempting to use 16 threads.

I don't know what the root cause here is; either we're keeping fds open much longer than we should, or we have a bunch of tests that require a lot of fds, or maybe something completely different. Interestingly, lsof reports that most of the fds in use are PIPEs. What do we use pipes for?

[brson]

That commit is not right anyway. I'm going to revert it.

[brson]

Oh, I read it wrong. That commit is working as intended.

[brson]

our new scheduler is based on libuv, which is event driven by waiting on fds, and the test suite now aggressively tests the new scheduler. it may be that we are using too many uv handles, and in turn fds, in some situations - the scheduler definitely allocates too many idle handles, but I'm not sure that those require an fd.

we did have to raise the ulimit on the os x bots to land it.

[brson]

There are two overcommits involved in the test suite now that could be contributing to this problem: first, as you pointed out, the multithreaded scheduler tests create num_cpus * 2 scheduler threads each; second, the normal test runner creates num_cpus * 4 test tasks.

Each scheduler I believe needs a minimum of 2 fds, one for the kqueue (I think), and a second for an async handle. So at any time while running stdtest you may need at least num_cpus * 2 * 4 * 2 = 64 fds, and undoubtedly there are even more that I'm not aware of.

[pnkfelix]

What is the appropriate way to raise the fd ulimit on OS X? I tried doing this when I was looking into #7797 but whatever technique I used did (from stackoverflow) not work

[brson]

Math was wrong. It's num_cpus * 2 * num_cpus * 4 * 2 = 256.

[brson]

I'm going to change the amount of overcommit here to try to get the fds back down.

[brson]

@pnkfelix here's what I did on the bots:

  $ echo -e 'limit maxfiles 8192 20480\nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf
  $ echo 'ulimit -n 4096' | sudo tee -a /etc/profile

Reboot afterwards.

[lilyball]

The default kern.maxopenfiles and kern.maxfilesperproc is 10240. You can use the getrlimit() and setrlimit() calls to raise the default fd ulimit from 256 up to this 10240.

Curiously, in my test of this, getrlimit() returned ~(1<<63) as the max instead of 10240, but wouldn't let me set the soft limit higher than 10240.

One potential workaround rust could do is, during startup, use sysctl to read the kern.maxfilesperproc setting and setrlimit() to raise the ulimit to that max.

On another note, does every rust task need a libuv event loop? The requirement of multiple fds per event loop means that tasks are even less lightweight than I had believed (significantly less-so than in Go, where I can create goroutines up the wazoo).

[metajack]

@kballard Every scheduler has a libuv event loop, not every task. There's one scheduler per thread, and all tasks are multiplexed across the scheduler threads.