Clarification on performance improvements in `std::fs::read_to_string` for Windows

[fiveseven-lambda]

Location

Function read_to_string in std::fs
https://doc.rust-lang.org/stable/std/fs/fn.read_to_string.html

Summary

The current documentation for std::fs::read_to_string states that it is a convenience function for File::open and std::io::Read::read_to_string, providing fewer imports and no intermediate variables. However, after the recent pull request #110655, it seems that std::fs::read_to_string has received performance improvements specifically for Windows, addressing issues that File::open and std::io::Read::read_to_string still face on this platform.

This implies that using std::fs::read_to_string is not just a matter of convenience, but may also result in better performance on Windows compared to manually combining File::open and std::io::Read::read_to_string.

I believe this performance improvement should be mentioned in the documentation, especially for developers targeting Windows, as the current documentation only emphasizes the convenience aspect without mentioning the performance benefits.

Suggested change:
Add a note to the documentation of std::fs::read_to_string, clarifying that due to recent improvements in pull request #110655, it offers better performance on Windows than manually using File::open and std::io::Read::read_to_string.

Thank you for your consideration!

[workingjubilee]

PRs welcome!

[the8472]

it seems that std::fs::read_to_string has received performance improvements specifically for Windows, addressing issues that File::open and std::io::Read::read_to_string still face on this platform.

This should not be the case. The read-loop limits the buffer size passed to the underlying reader regardless of whether you use fs::read_to_string or Read::read_to_string. I.e. #110650 should be fixed in both cases.

rust/library/std/src/io/mod.rs

Lines 501 to 514 in 1a5a224

	// Use heuristics to determine the max read size if no initial size hint was provided
	if size_hint.is_none() {
	// The reader is returning short reads but it doesn't call ensure_init().
	// In that case we no longer need to restrict read sizes to avoid
	// initialization costs.
	if !was_fully_initialized {
	max_read_size = usize::MAX;
	}
	// we have passed a larger buffer than previously and the
	// reader still hasn't returned a short read
	if buf_len >= max_read_size && bytes_read == buf_len {
	max_read_size = max_read_size.saturating_mul(2);
	}

[fiveseven-lambda]

But the size_hint is not passed to default_read_to_string when you call std::io::Read::read_to_string. Since the trait std::io::Read doesn't have any method like .size_hint() on std::iter::Iterator, we cannot know the file size in advance.

rust/library/std/src/io/mod.rs

Lines 903 to 906 in 1a5a224

	#[stable(feature = "rust1", since = "1.0.0")]
	fn read_to_string(&mut self, buf: &mut String) -> Result<usize> {
	default_read_to_string(self, buf, None)
	}

[the8472]

#110650 was not about allocation, it was about a too-large buffer getting passed to the windows API.

• For fs::read_to_string this is solved by passing a size-hint which allocates the right buffer, so it won't be too-large (added by Limit read size in File::read_to_end loop #110655).
• For Read::read_to_string this is solved by the heuristics that I linked that apply when no size-hint is passed (added by unify read_to_end and io::copy impls for reading into a Vec #118222).

[fiveseven-lambda]

Thank you for pointing that out, and I apologize for not noticing the changes introduced in #118222 earlier.
It seems I had a major misunderstanding regarding the performance issue. I really appreciate you taking the time to explain it.
At this point, I no longer feel the need to propose any changes to the documentation.

[fiveseven-lambda]

@workingjubilee so I'm sorry for the confusion. I will close both the issue and the PR.

[the8472]

Hrrm, looking at it again, the heuristics don't apply optimally.

Usually file io APIs will fill the whole buffer so no short reads will occur, which means it will stay in the doubling regime for a while, which avoids excessive initialization costs. But when it hits EOF there'll be one short read. And since we can't distinguish from short reads and EOF we have do one more read to figure that out. At that point the full buffer will be passed because the heuristics think no initialization happens (since we only check the initialization done via BorrowedBuf, not whatever windows is doing).

There should be ways to improve this.

[the8472]

Thanks for bringing it to our attention, there appears to be room for improvement after all!