Use zero for initialized Once state#143881

orlp · 2025-07-13T10:28:09Z

By re-labeling which integer represents which internal state for Once we can ensure that the initialized state is the all-zero state. This is beneficial because some CPU architectures (such as Arm) have specialized instructions to specifically branch on non-zero, and checking for the initialized state is by far the most important operation.

As an example, take this:

use std::sync::atomic::{AtomicU32, Ordering};

const INIT: u32 = 3;

#[inline(never)]
#[cold]
pub fn slow(state: &AtomicU32) {
    state.store(INIT, Ordering::Release);
}

pub fn ensure_init(state: &AtomicU32) {
    if state.load(Ordering::Acquire) != INIT {
        slow(state)
    }
}

If INIT is 3 (as is currently the state for Once), we see the following assembly on aarch64-apple-darwin:

example::ensure_init::h332061368366e313:
        ldapr   w8, [x0]
        cmp     w8, #3
        b.ne    LBB1_2
        ret
LBB1_2:
        b       example::slow::ha042bd6a4f33724e

By changing the INIT state to zero we get the following:

example::ensure_init::h332061368366e313:
        ldapr   w8, [x0]
        cbnz    w8, LBB1_2
        ret
LBB1_2:
        b       example::slow::ha042bd6a4f33724e

So this PR saves 1 instruction every time a LazyLock gets accessed on platforms such as these.

rustbot · 2025-07-13T10:28:13Z

r? @tgross35

rustbot has assigned @tgross35.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

tgross35 · 2025-07-13T10:31:14Z

One nit, could you add a note to these COMPLETE consts mentioning this so they don't accidentally get changed in the future? With that and passing CI, r=me

tgross35 · 2025-07-13T10:32:51Z

To test the queue implementation:

@bors2 try jobs=x86_64-msvc-*

rust-bors · 2025-07-13T10:32:55Z

⌛ Trying commit a2d4139 with merge a3c9ef7…

To cancel the try build, run the command @bors2 try cancel.

Use zero for initialized Once state By re-labeling which integer represents which internal state for `Once` we can ensure that the initialized state is the all-zero state. This is beneficial because some CPU architectures (such as Arm) have specialized instructions to specifically branch on non-zero, and checking for the initialized state is by far the most important operation. As an example, take this: ```rust use std::sync::atomic::{AtomicU32, Ordering}; const INIT: u32 = 3; #[inline(never)] #[cold] pub fn slow(state: &AtomicU32) { state.store(INIT, Ordering::Release); } pub fn ensure_init(state: &AtomicU32) { if state.load(Ordering::Acquire) != INIT { slow(state) } } ``` If `INIT` is 3 (as is currently the state for `Once`), we see the following assembly on `aarch64-apple-darwin`: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cmp w8, #3 b.ne LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` By changing the `INIT` state to zero we get the following: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cbnz w8, LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` So this PR saves 1 instruction every time a `LazyLock` gets accessed on platforms such as these. try-job: x86_64-msvc-*

orlp · 2025-07-13T10:38:30Z

One nit, could you add a note to these COMPLETE consts mentioning this so they don't accidentally get changed in the future? With that and passing CI, r=me

@tgross35 Done.

ibraheemdev · 2025-07-13T12:20:26Z

@bors r=tgross35 rollup

bors · 2025-07-13T12:20:28Z

📌 Commit f041962 has been approved by tgross35

It is now in the queue for this repository.

rust-bors · 2025-07-13T13:05:43Z

☀️ Try build successful (CI)
Build commit: a3c9ef7 (a3c9ef742934ddb66a7d2265ea705a16b21e524c, parent: d2baa49a106fad06fbf6202fb6ea8a0b3d2767cc)

Use zero for initialized Once state By re-labeling which integer represents which internal state for `Once` we can ensure that the initialized state is the all-zero state. This is beneficial because some CPU architectures (such as Arm) have specialized instructions to specifically branch on non-zero, and checking for the initialized state is by far the most important operation. As an example, take this: ```rust use std::sync::atomic::{AtomicU32, Ordering}; const INIT: u32 = 3; #[inline(never)] #[cold] pub fn slow(state: &AtomicU32) { state.store(INIT, Ordering::Release); } pub fn ensure_init(state: &AtomicU32) { if state.load(Ordering::Acquire) != INIT { slow(state) } } ``` If `INIT` is 3 (as is currently the state for `Once`), we see the following assembly on `aarch64-apple-darwin`: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cmp w8, #3 b.ne LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` By changing the `INIT` state to zero we get the following: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cbnz w8, LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` So this PR saves 1 instruction every time a `LazyLock` gets accessed on platforms such as these.

Rollup of 16 pull requests Successful merges: - #142885 (core: Add `BorrowedCursor::with_unfilled_buf`) - #143217 (Port #[link_ordinal] to the new attribute parsing infrastructure) - #143355 (wrapping shift: remove first bitmask and table) - #143448 (remote-test-client: Exit code `128 + <signal-number>` instead of `3`) - #143592 (UWP: link ntdll functions using raw-dylib) - #143681 (bootstrap/miri: avoid rebuilds for test builds) - #143710 (Updates to random number generation APIs) - #143724 (Tidy cleanup) - #143820 (Fixed a core crate compilation failure when enabling the `optimize_for_size` feature on some targets) - #143850 (Compiletest: Simplify {Html,Json}DocCk directive handling) - #143855 (Port `#[omit_gdb_pretty_printer_section]` to the new attribute parsing) - #143868 (warn on align on fields to avoid breaking changes) - #143875 (update issue number for `const_trait_impl`) - #143881 (Use zero for initialized Once state) - #143887 (Run bootstrap tests sooner in the `x test` pipeline) - #143893 (Don't require `eh_personality` lang item on targets that have a personality) Failed merges: - #143878 (Port `#[pointee]` to the new attribute parsing infrastructure) - #143891 (Port `#[coverage]` to the new attribute system) r? `@ghost` `@rustbot` modify labels: rollup

Use zero for initialized Once state By re-labeling which integer represents which internal state for `Once` we can ensure that the initialized state is the all-zero state. This is beneficial because some CPU architectures (such as Arm) have specialized instructions to specifically branch on non-zero, and checking for the initialized state is by far the most important operation. As an example, take this: ```rust use std::sync::atomic::{AtomicU32, Ordering}; const INIT: u32 = 3; #[inline(never)] #[cold] pub fn slow(state: &AtomicU32) { state.store(INIT, Ordering::Release); } pub fn ensure_init(state: &AtomicU32) { if state.load(Ordering::Acquire) != INIT { slow(state) } } ``` If `INIT` is 3 (as is currently the state for `Once`), we see the following assembly on `aarch64-apple-darwin`: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cmp w8, rust-lang#3 b.ne LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` By changing the `INIT` state to zero we get the following: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cbnz w8, LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` So this PR saves 1 instruction every time a `LazyLock` gets accessed on platforms such as these.

Rollup of 17 pull requests Successful merges: - #142885 (core: Add `BorrowedCursor::with_unfilled_buf`) - #143217 (Port #[link_ordinal] to the new attribute parsing infrastructure) - #143355 (wrapping shift: remove first bitmask and table) - #143448 (remote-test-client: Exit code `128 + <signal-number>` instead of `3`) - #143681 (bootstrap/miri: avoid rebuilds for test builds) - #143710 (Updates to random number generation APIs) - #143724 (Tidy cleanup) - #143738 (Move several float tests to floats/mod.rs) - #143820 (Fixed a core crate compilation failure when enabling the `optimize_for_size` feature on some targets) - #143850 (Compiletest: Simplify {Html,Json}DocCk directive handling) - #143855 (Port `#[omit_gdb_pretty_printer_section]` to the new attribute parsing) - #143868 (warn on align on fields to avoid breaking changes) - #143875 (update issue number for `const_trait_impl`) - #143881 (Use zero for initialized Once state) - #143887 (Run bootstrap tests sooner in the `x test` pipeline) - #143893 (Don't require `eh_personality` lang item on targets that have a personality) - #143901 (Region constraint nits) Failed merges: - #143878 (Port `#[pointee]` to the new attribute parsing infrastructure) - #143891 (Port `#[coverage]` to the new attribute system) r? `@ghost` `@rustbot` modify labels: rollup

Rollup of 10 pull requests Successful merges: - #143217 (Port #[link_ordinal] to the new attribute parsing infrastructure) - #143681 (bootstrap/miri: avoid rebuilds for test builds) - #143724 (Tidy cleanup) - #143733 (Change bootstrap's `tool.TOOL_NAME.features` to work on any subcommand) - #143850 (Compiletest: Simplify {Html,Json}DocCk directive handling) - #143875 (update issue number for `const_trait_impl`) - #143881 (Use zero for initialized Once state) - #143887 (Run bootstrap tests sooner in the `x test` pipeline) - #143917 (Change "allocated object" to "allocation".) - #143918 (Tier check cleanup) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of #143881 - orlp:once-state-repr, r=tgross35 Use zero for initialized Once state By re-labeling which integer represents which internal state for `Once` we can ensure that the initialized state is the all-zero state. This is beneficial because some CPU architectures (such as Arm) have specialized instructions to specifically branch on non-zero, and checking for the initialized state is by far the most important operation. As an example, take this: ```rust use std::sync::atomic::{AtomicU32, Ordering}; const INIT: u32 = 3; #[inline(never)] #[cold] pub fn slow(state: &AtomicU32) { state.store(INIT, Ordering::Release); } pub fn ensure_init(state: &AtomicU32) { if state.load(Ordering::Acquire) != INIT { slow(state) } } ``` If `INIT` is 3 (as is currently the state for `Once`), we see the following assembly on `aarch64-apple-darwin`: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cmp w8, #3 b.ne LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` By changing the `INIT` state to zero we get the following: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cbnz w8, LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` So this PR saves 1 instruction every time a `LazyLock` gets accessed on platforms such as these.

Rollup of 10 pull requests Successful merges: - rust-lang/rust#143217 (Port #[link_ordinal] to the new attribute parsing infrastructure) - rust-lang/rust#143681 (bootstrap/miri: avoid rebuilds for test builds) - rust-lang/rust#143724 (Tidy cleanup) - rust-lang/rust#143733 (Change bootstrap's `tool.TOOL_NAME.features` to work on any subcommand) - rust-lang/rust#143850 (Compiletest: Simplify {Html,Json}DocCk directive handling) - rust-lang/rust#143875 (update issue number for `const_trait_impl`) - rust-lang/rust#143881 (Use zero for initialized Once state) - rust-lang/rust#143887 (Run bootstrap tests sooner in the `x test` pipeline) - rust-lang/rust#143917 (Change "allocated object" to "allocation".) - rust-lang/rust#143918 (Tier check cleanup) r? `@ghost` `@rustbot` modify labels: rollup

Use zero for initialized Once state By re-labeling which integer represents which internal state for `Once` we can ensure that the initialized state is the all-zero state. This is beneficial because some CPU architectures (such as Arm) have specialized instructions to specifically branch on non-zero, and checking for the initialized state is by far the most important operation. As an example, take this: ```rust use std::sync::atomic::{AtomicU32, Ordering}; const INIT: u32 = 3; #[inline(never)] #[cold] pub fn slow(state: &AtomicU32) { state.store(INIT, Ordering::Release); } pub fn ensure_init(state: &AtomicU32) { if state.load(Ordering::Acquire) != INIT { slow(state) } } ``` If `INIT` is 3 (as is currently the state for `Once`), we see the following assembly on `aarch64-apple-darwin`: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cmp w8, #3 b.ne LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` By changing the `INIT` state to zero we get the following: ```asm example::ensure_init::h332061368366e313: ldapr w8, [x0] cbnz w8, LBB1_2 ret LBB1_2: b example::slow::ha042bd6a4f33724e ``` So this PR saves 1 instruction every time a `LazyLock` gets accessed on platforms such as these.

Use zero for initialized Once state

a2d4139

rustbot assigned tgross35 Jul 13, 2025

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Jul 13, 2025

Add comment why we use zero for COMPLETE

f041962

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 13, 2025

fmease mentioned this pull request Jul 14, 2025

Rollup of 16 pull requests #143912

Closed

jhpratt mentioned this pull request Jul 14, 2025

Rollup of 17 pull requests #143916

Closed

Kobzol mentioned this pull request Jul 14, 2025

Rollup of 10 pull requests #143919

Merged

bors merged commit 70301da into rust-lang:master Jul 14, 2025
11 checks passed

rustbot added this to the 1.90.0 milestone Jul 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use zero for initialized Once state#143881