Skip to content

Runtime Failures on TCP Stress Testing: (libuv) kqueue(): Too many open files and more #12666

Closed

Description

@ehsanul

I'm on OSX 10.8.4, x86_64, latest Rust on master. And to head off any concerns about my file handle limits:

$ ulimit
unlimited

I ran into this when stress-testing my web sockets library. After just a couple hundred connections, some connections were being refused at random. The only thing I could see was some output with RUST_LOG=debug saying (libuv) kqueue(): Too many open files. I've come up with a smaller test case that brings this out with some spectacular errors:

use std::io::net::tcp::{TcpListener, TcpStream};
use std::io::net::ip::{Ipv4Addr, SocketAddr};
use std::io::{Acceptor, Listener};
use std::io::IoResult;

fn main() {
    let addr = SocketAddr { ip: Ipv4Addr(127, 0, 0, 1), port: 8989 };
    spawn(proc() {
        let mut acceptor = TcpListener::bind(addr).listen();
        loop {
            let mut stream = match acceptor.accept() {
                Ok(stream) => stream,
                Err(error) => {
                    debug!("accept failed: {:?}", error);
                    continue;
                }
            };
            println!("{:?}", stream.read_byte().unwrap());
        }
    });
    for _ in range(0, 1000) {
        spawn(proc() {
            match TcpStream::connect(addr) {
                Ok(stream) => {
                    let mut stream = stream;
                    stream.write([1]);
                    let mut buf = [0];
                    stream.read(buf);
                },
                Err(e) => debug!("{:?}", e)
            }
        });
    }
}

Fails less often with 100-200 connections. The errors are different every time I run this with RUST_LOG=debug.

Here's the end of one run:

(libuv) kqueue(): Too many open files
bug(54921,0x10881c000) malloc: *** error for object 0x7fca4b620f70: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
std::io::IoError{kind: OtherIoError, desc: "too many open files", detail: None}
task '<unnamed>' failed at 'assertion failed: `(left == right) && (right == left)` (left: `140506823655696`, right: `140506824839680`)', /Users/harish/rust/rust/src/librustuv/homing.rs:110
(libuv) kqueue(): Too many open files
std::io::IoError{kind: OtherIoError, desc: "too many open files", detail: None}
task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199
Abort trap: 6

Another:

task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199
task failed during unwinding (double-failure - total drag!)
rust must abort now. so sorry.
Segmentation fault: 11

Another:

task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199
task failed during unwinding (double-failure - total drag!)
rust must abort now. so sorry.
Illegal instruction: 4

And lastly:

fatal runtime error: leaked managed memory (140518327703008 objects)
(libuv) kqueue(): Too many open files
task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:Abort trap: 6

Activity

  1. emberian commented on Mar 3, 2014

    @emberian
    Contributor
  2. alexcrichton commented on Mar 3, 2014

    @alexcrichton
    Member

    Oh dear that's bad, thanks for the small issue! I will definitely investigate.

  3. bharrisau commented on Mar 3, 2014

    @bharrisau
    Contributor

    What is the result of ulimit -n though? Just calling ulimit is only the maximum file size (unless OSX is different).

  4. ehsanul commented on Mar 4, 2014

    @ehsanul
    ContributorAuthor

    @bharrisau you're right!

    $ ulimit -n
    256
    

    So I set it to a higher value and things seem fine now. Sorry about that. Though, It's still a surprising way for the runtime to fail, and clearly something is still wrong there.

  5. alexcrichton commented on Mar 4, 2014

    @alexcrichton
    Member

    Despite OSX having small file descriptor limits, this is a serious bug that I'd like to address with the runtime. Something funny is going on.

  6. alexcrichton commented on Mar 10, 2014

    @alexcrichton
    Member

    With #12818 applied, I am unable to reproduce this on OSX. I am not convinced that it fixes anything, but I'm no longer seeing any scary assertions.

  7. alexcrichton commented on Mar 10, 2014

    @alexcrichton
    Member

    I take that back, I was using libnative, not libgreen (silly me)

  8. alexcrichton commented on Mar 11, 2014

    @alexcrichton
    Member

    I attempted to debug this issue on linux, and I quickly ran into two problems. The first is an assertion fixed by #12818, and the other is filed as joyent/libuv#1189. Sadly none of this seems related to the assertions on OSX, so I'll continue to investigate there. If only OSX still had valgrind...

  9. added a commit that references this issue on Mar 13, 2014
    792da84
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions