Skip to content

read_to_end unintuitive performance. #23815

@blackbeam

Description

@blackbeam

Performance problem in read_to_end was uncovered with PR #23668.
The reason is that in case of data to read equals vec's capacity, read_to_end anyway will reserve (and with #23668 will initialize) another 64KB in vec just to read nothing to it because of lack of any other way to get a EOF hint from Read implementor.

Simple illustration:

#![feature(test)]
extern crate test;
use std::io::Read;

#[bench]
fn read_to_end_1024_to_1024(b: &mut test::Bencher) {
    let buf = [1u8; 1024];
    let mut src = &buf[..];
    let mut dst = Vec::with_capacity(1024);
    b.iter(|| {
        test::black_box(src.read_to_end(&mut dst))
    });
}

#[bench]
fn read_to_end_1024_to_1025(b: &mut test::Bencher) {
    let buf = [1u8; 1024];
    let mut src = &buf[..];
    let mut dst = Vec::with_capacity(1025);
    b.iter(|| {
        test::black_box(src.read_to_end(&mut dst))
    });
}

Result:

// Opt 0
running 2 tests
test read_to_end_1024_to_1024 ... bench:   9033646 ns/iter (+/- 456609)
test read_to_end_1024_to_1025 ... bench:       488 ns/iter (+/- 24)

// Opt 3
running 2 tests
test read_to_end_1024_to_1024 ... bench:    105434 ns/iter (+/- 3655)
test read_to_end_1024_to_1025 ... bench:        13 ns/iter (+/- 1)

Edit:
Same with no initialization:

// Opt 0
running 2 tests
test read_to_end_1024_to_1024 ... bench:       375 ns/iter (+/- 8)
test read_to_end_1024_to_1025 ... bench:       387 ns/iter (+/- 16)

// Opt 3
running 2 tests
test read_to_end_1024_to_1024 ... bench:        16 ns/iter (+/- 1)
test read_to_end_1024_to_1025 ... bench:        16 ns/iter (+/- 1)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions