remote-test: use u64 to represent file size#119999

onur-ozkan · 2024-01-15T17:44:51Z

Currently, triggering a transfer of data exceeding the size of 4294967295 bytes results in a panic on the remote-test-server as io::copy(&mut file, dst) failed with Connection reset by peer (os error 104). This issue happens because the size is transmitted as u32 to remote-test-server.

First commit increases the supported file size. But I am not sure about its necessity — can we realistically encounter file sizes exceeding 4GB in builds, perhaps through some complicated configurations?

~~The second commit adds a sanity check to avoid encountering the error io::copy(&mut file, dst) failed with Connection reset by peer (os error 104) on the remote-test-server side.~~

rustbot · 2024-01-15T17:44:58Z

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

onur-ozkan · 2024-01-15T17:53:14Z

First commit increases the supported file size. But I am not sure about its necessity — can we realistically encounter file sizes exceeding 4GB in builds, perhaps through some complicated configurations?

Let me know if this isn't needed (so I can revert).

Mark-Simulacrum · 2024-01-20T20:53:50Z

I don't think we need to support large files, though I also don't mind supporting them.

That said, if we're touching this code I think rather than manually bit twiddling using https://doc.rust-lang.org/nightly/std/primitive.u32.html#method.to_le_bytes and correspondign from_le_bytes feels much better to me.

onur-ozkan · 2024-01-21T09:14:45Z

The second commit adds a sanity check to avoid encountering the error io::copy(&mut file, dst) failed with Connection reset by peer (os error 104) on the remote-test-server side.

Just realized that there is no need for a sanity check on the size anymore since we transmit the size without any conversion.

Updated the manual bit-twiddling part to use to_be_bytes on the client side (server side already uses from_be_bytes).

@rustbot ready

Mark-Simulacrum · 2024-01-21T14:56:17Z

src/tools/remote-test-client/src/main.rs

+        let mut amt = 0u64;
+        for (i, &byte) in header[1..=8].iter().enumerate() {
+            amt |= (byte as u64) << (56 - 8 * i);
+        }


u64::from_be_bytes(header[1..9].try_into().unwrap())

This increases the maximum supported file size (previously limited to 4GB) and avoids potential issues with u32 to u64 conversions, which are no longer needed. Signed-off-by: onur-ozkan <[email protected]>

Mark-Simulacrum · 2024-01-21T15:32:30Z

@bors r+

bors · 2024-01-21T15:32:34Z

📌 Commit 1afd216 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

…rk-Simulacrum remote-test: use u64 to represent file size Currently, triggering a transfer of data exceeding the size of 4294967295 bytes results in a panic on the `remote-test-server` as `io::copy(&mut file, dst) failed with Connection reset by peer (os error 104)`. This issue happens because the size is transmitted as u32 to `remote-test-server`. First commit increases the supported file size. But I am not sure about its necessity — can we realistically encounter file sizes exceeding 4GB in builds, perhaps through some complicated configurations? ~The second commit adds a sanity check to avoid encountering the error `io::copy(&mut file, dst) failed with Connection reset by peer (os error 104)` on the `remote-test-server` side.~

Rollup of 12 pull requests Successful merges: - rust-lang#118639 (Undeprecate lint `unstable_features` and make use of it in the compiler) - rust-lang#118714 ( Explanation that fields are being used when deriving `(Partial)Ord` on enums) - rust-lang#119801 (Fix deallocation with wrong allocator in (A)Rc::from_box_in) - rust-lang#119948 (Make `unsafe_op_in_unsafe_fn` migrated in edition 2024) - rust-lang#119999 (remote-test: use u64 to represent file size) - rust-lang#120058 (bootstrap: improvements for compiler builds) - rust-lang#120160 (Manually implement derived `NonZero` traits.) - rust-lang#120177 (Remove duplicate dependencies for rustc) - rust-lang#120185 (coverage: Don't instrument `#[automatically_derived]` functions) - rust-lang#120194 (Shorten `#[must_use]` Diagnostic Message for `Option::is_none`) - rust-lang#120200 (Correct the anchor of an URL in an error message) - rust-lang#120203 (Replace `#!/bin/bash` with `#!/usr/bin/env bash` in rust-installer tests) r? `@ghost` `@rustbot` modify labels: rollup

…rk-Simulacrum remote-test: use u64 to represent file size Currently, triggering a transfer of data exceeding the size of 4294967295 bytes results in a panic on the `remote-test-server` as `io::copy(&mut file, dst) failed with Connection reset by peer (os error 104)`. This issue happens because the size is transmitted as u32 to `remote-test-server`. First commit increases the supported file size. But I am not sure about its necessity — can we realistically encounter file sizes exceeding 4GB in builds, perhaps through some complicated configurations? ~The second commit adds a sanity check to avoid encountering the error `io::copy(&mut file, dst) failed with Connection reset by peer (os error 104)` on the `remote-test-server` side.~

…iaskrgr Rollup of 9 pull requests Successful merges: - rust-lang#118578 (core: introduce split_at{,_mut}_checked) - rust-lang#119369 (exclude unexported macro bindings from extern crate) - rust-lang#119408 (xous: misc fixes + add network support) - rust-lang#119943 (std::net: bind update for using backlog as `-1` too.) - rust-lang#119948 (Make `unsafe_op_in_unsafe_fn` migrated in edition 2024) - rust-lang#119999 (remote-test: use u64 to represent file size) - rust-lang#120152 (add help message for `exclusive_range_pattern` error) - rust-lang#120213 (Don't actually make bound ty/const for RTN) - rust-lang#120225 (Fix -Zremap-path-scope typo) Failed merges: - rust-lang#119972 (Add `ErrCode`) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#119999 - onur-ozkan:remote-test-tools, r=Mark-Simulacrum remote-test: use u64 to represent file size Currently, triggering a transfer of data exceeding the size of 4294967295 bytes results in a panic on the `remote-test-server` as `io::copy(&mut file, dst) failed with Connection reset by peer (os error 104)`. This issue happens because the size is transmitted as u32 to `remote-test-server`. First commit increases the supported file size. But I am not sure about its necessity — can we realistically encounter file sizes exceeding 4GB in builds, perhaps through some complicated configurations? ~The second commit adds a sanity check to avoid encountering the error `io::copy(&mut file, dst) failed with Connection reset by peer (os error 104)` on the `remote-test-server` side.~

In rust-lang#119999, the length field of the header was changed from 4 bytes to 8. One of the headers in batch_copy was missed though, so it sends the wrong size. Fix by switching to `create_header` in that spot too.

In rust-lang#119999, the length field of the header was changed from 4 bytes to 8. One of the headers in `batch_copy` was missed though, so it sends the wrong size. Fix by switching to `create_header` in that spot too.

…lacrum remote-test-server: Fix header in batch mode In rust-lang#119999, the length field of the header was changed from 4 bytes to 8. One of the headers in `batch_copy` was missed though, so it sends the wrong size. Fix by switching to `create_header` in that spot too.

Rollup merge of #150987 - bishop-fix-batch-copy, r=Mark-Simulacrum remote-test-server: Fix header in batch mode In #119999, the length field of the header was changed from 4 bytes to 8. One of the headers in `batch_copy` was missed though, so it sends the wrong size. Fix by switching to `create_header` in that spot too.

remote-test-server: Fix header in batch mode In rust-lang/rust#119999, the length field of the header was changed from 4 bytes to 8. One of the headers in `batch_copy` was missed though, so it sends the wrong size. Fix by switching to `create_header` in that spot too.

rustbot assigned Mark-Simulacrum Jan 15, 2024

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 15, 2024

onur-ozkan changed the title ~~improvements~~ remote-test improvements Jan 15, 2024

onur-ozkan marked this pull request as ready for review January 15, 2024 17:52

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 20, 2024

onur-ozkan force-pushed the remote-test-tools branch 2 times, most recently from 9ae2ecf to 84fe1eb Compare January 21, 2024 09:07

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 21, 2024

onur-ozkan force-pushed the remote-test-tools branch from 84fe1eb to ba58a9e Compare January 21, 2024 09:16

onur-ozkan changed the title ~~remote-test improvements~~ remote-test: use u64 to represent file size Jan 21, 2024

Mark-Simulacrum requested changes Jan 21, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 21, 2024

use u64 to represent file size

1afd216

This increases the maximum supported file size (previously limited to 4GB) and avoids potential issues with u32 to u64 conversions, which are no longer needed. Signed-off-by: onur-ozkan <[email protected]>

onur-ozkan force-pushed the remote-test-tools branch from ba58a9e to 1afd216 Compare January 21, 2024 15:16

onur-ozkan added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 21, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 21, 2024

TaKO8Ki mentioned this pull request Jan 21, 2024

Rollup of 12 pull requests #120209

Closed

matthiaskrgr mentioned this pull request Jan 22, 2024

Rollup of 9 pull requests #120231

Closed

This was referenced Jan 22, 2024

Rollup of 10 pull requests #120235

Closed

Rollup of 9 pull requests #120239

Merged

bors merged commit 9e896f4 into rust-lang:master Jan 22, 2024

rustbot added this to the 1.77.0 milestone Jan 22, 2024

nicholasbishop mentioned this pull request Jan 12, 2026

remote-test-server: Fix header in batch mode #150987

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

remote-test: use u64 to represent file size#119999