Tracking Issue for context_ext

Feature gate: #![feature(context_ext)]

This is a tracking issue for allowing std::task::Context to carry arbitrary extension data.

Public API

impl Context {
    fn ext(&mut self) -> &mut dyn Any;
}

impl ContextBuilder {
    fn ext(self, data: &'a mut dyn Any) -> Self;

    fn from(cx: &'a mut Context<'_>) -> Self;
    fn waker(self, waker: &'a Waker) -> Self;
}

Steps / History

Implementation: Add Context::ext #123203
Final comment period (FCP)¹
Stabilization PR

Unresolved Questions

passing refs maybe unsound
too complex
This causes Context to no longer be UnwindSafe, which may be a breaking change: regression: the type (dyn Any + 'static) may contain interior mutability #125193 (the issue is closed but there still is a FIXME in the code)

https://std-dev-guide.rust-lang.org/feature-lifecycle/stabilization.html ↩

Rollup merge of rust-lang#125392 - workingjubilee:unwind-a-problem-in…

Unrolled build for rust-lang#125392

Rollup merge of #125392 - workingjubilee:unwind-a-problem-in-context,…

One of the concerns raised was about the soundness of trying to force the dyn Any to carry non-static references via unsafe casts.

I believe this may be theoretically achievable by requiring any references contained within the dyn Any to be of dyn types, which are !Sized and thus prevent move-ins and move-outs of themselves, and for the trait definitions of those types to not expose any held lifetimes. Constraining inner references to such types could be done by introducing an unsafe trait:

// SAFETY: only implement on dyn types that don't expose
// lifetimes held by the type. i.e. traits with methods
// that only work with argument lifetimes. this allows
// us to safely cast any held lifetimes, because there
// is no way to move-in, move-out, or obtain a reference
// to data with any such held lifetimes
pub unsafe trait DynWithStaticInterface {
    type Static: ?Sized + 'static;
    type Clamped<'a>: ?Sized
    where
        Self: 'a;

    // return type is safe to use because it does not expose
    // whatever lifetimes were converted to 'static
    fn to_static(&mut self) -> &mut Self::Static;

    // return type is safe to use because it does not expose
    // whatever lifetimes were converted to 'a
    fn to_clamped<'a>(&'a mut self) -> &'a mut Self::Clamped<'a>;
}

Since we are working with dyn references, we need the trait to be object-safe. This means we cannot allow casting to arbitrary lifetimes. Fortunately, we only ever need to cast to static or to the lifetime of self (which we here refer to as a "clamped" lifetime), and so we can provide methods for exactly those.

Implementing on a dyn type could look like this:

use std::mem;

trait StringRef {
    fn get(&mut self) -> &mut String;
}

// SAFETY: implementing on a dyn type that does not expose any lifetimes held by the type
unsafe impl DynWithStaticInterface for (dyn StringRef + '_) {
    type Static = dyn StringRef + 'static;
    type Clamped<'a> = dyn StringRef + 'a;

    fn to_static(&mut self) -> &mut Self::Static {
        unsafe { mem::transmute(self) }
    }

    fn to_clamped<'a>(&'a mut self) -> &'a mut Self::Clamped<'a> {
        unsafe { mem::transmute(self) }
    }
}

struct StringSource<'a> {
    s: &'a mut String,
}

impl<'a> StringSource<'a> {
    fn new(s: &'a mut String) -> Self {
        Self { s }
    }
}

impl<'a> StringRef for StringSource<'a> {
    fn get(&mut self) -> &mut String {
        self.s
    }
}

Above we have a non-static concrete type holding data with lifetime 'a that implements a trait which does not expose the 'a. If all a user has is a &mut dyn StringRef, then there is no way to move a dyn StringRef into that address or move the dyn StringRef out of that address, and the available methods on the type (get()) only expose data with the lifetime of self. The lifetime of the concrete type is not exposed, and so it is safe to lie about it. Whether the lifetime is casted to 'static or some 'foo, the user can't do anything with it.

The Any::downcast_mut() method is limited to casting to references of types that are Sized, so we can't cast a dyn reference directly to &mut dyn Any. However, we can cast a dyn reference to a pointer, and then cast a reference to the pointer to the &mut dyn Any. To make this safe, we can wrap the pointer:

#![feature(local_waker)]
#![feature(context_ext)]

struct ContainerPrivate<T: ?Sized>(*mut T);

pub struct Container<'a, T: ?Sized, U: ?Sized> {
    p: ContainerPrivate<T>,
    _marker: PhantomData<&'a mut U>,
}

impl<'a, T: DynWithStaticInterface + ?Sized> Container<'a, T::Static, T> {
    pub fn new(r: &'a mut T) -> Self {
        Self {
            p: ContainerPrivate(r.to_static()),
            _marker: PhantomData,
        }
    }
}

impl<T: ?Sized + 'static, U: ?Sized> Container<'_, T, U> {
    pub fn as_any<'a>(&'a mut self) -> &'a mut (dyn Any + 'static) {
        &mut self.p
    }
}

pub fn ctx_downcast<'a, T: DynWithStaticInterface + ?Sized + 'static>(
    cx: &mut Context<'a>,
) -> Option<&'a mut T::Clamped<'a>> {
    let a = cx.ext();

    let p = match a.downcast_mut::<ContainerPrivate<T>>() {
        Some(p) => p,
        None => return None,
    };

    // SAFETY: the inner pointer is guaranteed to be valid because
    // p has the same lifetime as the reference that was
    // originally casted to the pointer
    let r: &'a mut T = unsafe { p.0.as_mut().unwrap() };

    Some(r.to_clamped())
}

With the above API, it becomes possible for the Context extension to carry non-static data:

#![feature(noop_waker)]

use std::task::{ContextBuilder, Waker};

let mut data = String::from("hello");

{
    let mut s = StringSource::new(&mut data);
    let mut c = Container::new(&mut s as &mut dyn StringRef);
    let mut cx = ContextBuilder::from_waker(&Waker::noop())
        .ext(c.as_any())
        .build();

    let r = ctx_downcast::<dyn StringRef>(&mut cx).unwrap();
    r.get().push_str(" world");
}

assert_eq!(&data, "hello world");

This is a bit more complicated than I would have liked, but I can't think of a better way to do it, and I can't think of a better data structure than Any to allow carrying arbitrary data.

I should note that passing 'static data, whether single owner or with something like Rc, is of course possible and a lot simpler:

Single owner:

let mut data = String::from("hello");

{
    let mut cx = ContextBuilder::from_waker(&Waker::noop())
        .ext(&mut data)
        .build();

    let data = cx.ext().downcast_mut::<String>().unwrap();
    data.push_str(" world");
}

assert_eq!(&data, "hello world");

With Rc:

let mut data = Rc::new(RefCell::new(String::from("hello")));

{
    let mut cx = ContextBuilder::from_waker(&Waker::noop())
        .ext(&mut data)
        .build();

    let data = cx.ext().downcast_ref::<Rc<RefCell<String>>>().unwrap();
    data.borrow_mut().push_str(" world");
}

assert_eq!(&*data.borrow(), "hello world");

I would guess many executors could reasonably use something like Rc/Arc for their extension data without runtime cost by pre-allocating such structures. In my experience, executors typically keep most of their state on the heap behind smart pointers anyway.

Passing non-static references in extension data may be more interesting for combinators or nested executors, i.e. things that may want to avoid allocations and can't easily pre-allocate.

@joshtriplett

I would like to move this towards stabilization.

The goal of this feature is to enable experimentation of Context extensions, that could later be promoted to direct fields/methods of Context. Extensions that could not actually be promoted to direct fields, for example those requiring additional generics, are out of scope.

Using this feature, I've successfully implemented my own extension, TopLevelPoller in the waker-waiter crate, accessible by reference, which conceivably could be promoted to a &'a mut dyn TopLevelPoller field of Context.

To respond to the unresolved questions:

passing refs may be unsound

I believe the approach I described earlier is sound. Would love feedback.

too complex

Specifically, @joshtriplett earlier wrote:

This is a lot of complexity to add to this interface.

It's one field. Is this saying we should not allow Context to communicate more kinds of data?

dyn Any is always painful.

After trying to shoehorn non-static references into it, I can agree with this statement. It kind of is what it is though.

Sharing one pointer between multiple potential users is even more painful, and a Provider-style API would be even more complexity.

True and of course not what is being proposed.

Everyone would pay the cost of this, whether they use it or not. And to the best of my knowledge, this isn't something needed by all async runtimes.

It's an optional feature with no cost if not used, other than increasing the size of Context by the size of a fat pointer. Is this saying we should not add a field to Context unless it is needed by all async runtimes?

I see a future where there are many extensions that executors and futures can opt-in to, that are best negotiated via the Context, and not every app would use all possible extensions. I think this is a reasonable stance? Maybe we need some community consensus on the notion that it would be valuable for Context to be able to carry optional data.

The potential users of this (AIUI) currently do have alternative solutions. Those solutions may not be ideal, but we have to weigh that against the complexity (which is a cost to everyone).

Unless there's more to it, I assume this a weighing between using thread-local/global storage vs slightly increasing the size of Context. I suspect most people developing async apps would prefer the latter.

This causes Context to no longer be UnwindSafe, which may be a breaking change

Does anyone know how meaningful this is?

@rustbot

@rustbot labels +AsyncAwait-Triaged

We discussed this in the WG-async meeting today. We have enough reservations about this approach that we'd like to explore two main other approaches to solving this problem first.

One alternative is the Provider API, as in:

Tracking Issue for error_generic_member_access #99301

The idea there would be to broaden the scope of that producer/consumer API from just Error, as it is now. We're interested in exploring how this could be done and unblocked.

The other approach is making Future generic, as in:

Generic Futures rfcs#3434

For this one, in particular, we'd need to do a lot of feasibility and migration analysis.

In general, we're interested in looking into and comparing these approaches.

We find the problem that the context extension is trying to solve highly motivating, but we want to be sure to find the right solution.

I just discovered this feature and I absolutely love it. I've been toying around with co-routines as well to find a way how I can express sans-IO code such that the compiler takes care of composing the state machines for me the same way I can use async/await code.

With this feature, it is possible to build a "runtime" and accompanying Futures that are entirely sans-IO and still hook into the compiler's machinery of async/await. You can see the code here: https://github.com/thomaseizinger/sairun

This is super cool because it means we get the best of both worlds:

sans-IO code with all its nice properties of being essentially pure functional
expressing IO operations using async/await instead of our own state machines

In regards to the proposed alternatives:

One alternative is the Provider API, as in:

This is essentially a direct replacement so this alternative would also work for.

The other approach is making Future generic, as in:

I've not thought about this too much but I think it would also be a suitable replacement.

@jkarneges

W.r.t. soundness and whatnot, @jkarneges, we can do something along the lines of the Provider API:

It boils down to defining dyn AnyLt<'x>, and then using &mut Option<impl Anylt<'a>> as &mut dyn AnyLt<'a> (the &mut Option<_> acting as a poorman's dyn-safe &own reference).

But I don't think we need to really hide the Option trick in your API; we'd rather "just" expose & and &mut getters, and users would be the ones to think of using an Option around their owned_data to retrieve it in an owned fashion (see below).

A sound dyn Any<'lt> is actually "easy" to design soundly if you use a one-to-one mapping from a lifetime-infected type to its 'static-TypeId-yielding "body", and exactly that one 'lt lifetime.

That is, you can even define it as a non-unsafe trait exposed to arbitrary implementations by having a doubly-injective (hence bijective) definition for this one-to-one restriction, so that the trait solver itself prevent unsound implementations:

Click to see

//                unnecessary for soundness, but convenient.
//                               vvv
pub trait MayHaveLifetime<'lt> : 'lt {
    type RemoveLifetime : IsStatic<WithLifetime<'lt> = Self>;
}

// paramount for soundness (and with the back-equality clauses, it suffices).
//                   vvvvvvv
pub trait IsStatic : 'static {
    type WithLifetime<'lt> : MayHaveLifetime<'lt, RemoveLifetime = Self>;
}

#[repr(transparent)]
pub struct StaticValue<T : 'static>(pub T);

impl<T : 'static> MayHaveLifetime<'_> for StaticValue<T> {
    type RemoveLifetime = Self;
}
impl<T : 'static> IsStatic for StaticValue<T> {
    type WithLifetime<'__> = Self;
}

impl<'r, T : MayHaveLifetime<'r>> MayHaveLifetime<'r> for &'r T {
    type RemoveLifetime = &'static T::RemoveLifetime;
}
impl<T : IsStatic> IsStatic for &'static T {
    type WithLifetime<'lt> = &'lt T::WithLifetime<'lt>;
}

Once you have this, you can define the Any<'lt> trait and sound downcast API:

Click to see

mod seal {
    /// Dyn-safe version of `MayHaveLifetime`, sealed.
    pub trait MayHaveLifetime<'lt> {}
    
    impl<'lt, T> MayHaveLifetime<'lt> for T
    where
        T : super::MayHaveLifetime<'lt>,
    {}
}

//              unnecessary, but convenient.
//                     vvv
pub trait AnyLt<'lt> : 'lt + seal::MayHaveLifetime<'lt> {
    fn type_id_of_removed_lifetime(&self) -> ::core::any::TypeId;
}

impl<'lt, Orig> AnyLt<'lt> for Orig
where
    Orig : MayHaveLifetime<'lt>,
{
    fn type_id_of_removed_lifetime(&self) -> ::core::any::TypeId {
        ::core::any::TypeId::of::<Orig::RemoveLifetime>()
    }
}

impl<'lt> dyn AnyLt<'lt> {
    pub fn is<T : MayHaveLifetime<'lt>>(&self) -> bool {
        // SAFETY: this returning `true` entails that:
        // `Orig::RemoveLifetime == T::RemoveLifetime`.
        // We apply `IsStatic::WithLifetime<'lt>` to each side:
        // `Orig::RemoveLt::WithLt<'lt> == T::RemoveLt::WithLt<'lt>`
        // i.e.
        // `Orig == T`
        // (`'lt` has been unable to bulge in this whole thing since it's invariant.)
        self.type_id_of_removed_lifetime() == ::core::any::TypeId::of::<T::RemoveLifetime>()
    }
    
    pub fn downcast_ref<T : MayHaveLifetime<'lt>>(&self) -> Option<&T> {
        self.is::<T>().then(|| unsafe {
            &*(&raw const *self).cast::<T>()
        })
    }
    
    pub fn downcast_mut<T : MayHaveLifetime<'lt>>(&mut self) -> Option<&mut T> {
        self.is::<T>().then(|| unsafe {
            &mut *(&raw mut *self).cast::<T>()
        })
    }
}

And once you have such dyn Any<'lt>, you can expose the alloc-free setters and getters:

pub struct ContextApi<'a> {
    ext: Option<&'a mut dyn AnyLt<'a>>,
}

impl<'a> ContextApi<'a> {
    // In the future this could be made `pub`, alongside the corresponding getter(s).
    fn embed<T : AnyLt<'a>>(
        &mut self,
        r: &'a mut T,
    )
    {
        self.ext = Some(r);
    }
    
    pub fn embed_value<T : 'static>(
        &mut self,
        r: &'a mut T,
    )
    {
        let r: &'a mut StaticValue<T> = unsafe {
            // SAFETY: `#[repr(transparent)]` and no safety invariants.
            &mut *<*mut _>::cast(r)
        };
        self.embed(r);
    }

    pub fn embed_ref<T : 'static>(
        &mut self,
        r: &'a mut &'a T,
    )
    {
        let r: &'a mut &'a StaticValue<T> = unsafe {
            // SAFETY: `#[repr(transparent)]` and no safety invariants.
            &mut *<*mut _>::cast(r)
        };
        self.embed(r);
    }
    
    /// Get the `.embed_value()` by ref, if provided.
    pub fn get_value_ref<T : 'static>(
        &self,
    ) -> Option<&T> {
        let StaticValue(it) = self.ext.as_ref()?.downcast_ref()?;
        Some(it)
    }
    
    /// Get the `.embed_value()` by mut ref, if provided.
    pub fn get_value_mut<T : 'static>(
        &mut self,
    ) -> Option<&mut T> {
        let StaticValue(it) = self.ext.as_mut()?.downcast_mut()?;
        Some(it)
    }
    
    /// Get the `.embed_ref()`, if provided.
    pub fn get_ref<T : 'static>(
        &self,
    ) -> Option<&'a T> {
        let &&StaticValue(ref it) = self.ext.as_ref()?.downcast_ref()?;
        Some(it)
    }
}

Demo:

// `.embed_value` with `42`
let mut cx = ContextApi::default();
let mut data: i32 = 42_i32;
cx.embed_value(&mut data);
assert_eq!(cx.get_value_ref::<i32>(), Some(&42));
assert_eq!(cx.get_value_mut::<i32>(), Some(&mut 42));
assert!(cx.get_ref::<i32>().is_none());

// `.embed_ref` with `&42`
let mut cx = ContextApi::default();
let local = 42_i32;
let mut data: &i32 = &local; // not static
cx.embed_ref(&mut data);
assert!(cx.get_value_ref::<&i32>().is_none());
assert!(cx.get_value_mut::<&i32>().is_none());
assert_eq!(cx.get_ref::<i32>(), Some(&42));
let r = cx.get_ref::<i32>().unwrap();
drop(cx); // not lifetime-bound.
assert_eq!(r, &42);

// `owned_data` trick using `Option` and `embed_value`.
let mut cx = ContextApi::default();
let data: String = "owned data".into();
let mut data = Some(data);
cx.embed_value(&mut data);
let s: String = cx.get_value_mut().and_then(Option::take).unwrap();
drop(cx);
assert_eq!(s, "owned data");
drop::<String>(s);

Playground

@danielhenrymantilla

@danielhenrymantilla amazing!

There is also Provider-style API for Context as an alternative proposal to address concerns raised about context_ext. However, if your approach is sound then it may be simpler.

At this point I think pretty much all the ways Context can be extended have been proposed and we need to pick one.

It's an optional feature with no cost if not used, other than increasing the size of Context by the size of a fat pointer. Is this saying we should not add a field to Context unless it is needed by all async runtimes?

I see a future where there are many extensions that executors and futures can opt-in to, that are best negotiated via the Context, and not every app would use all possible extensions. I think this is a reasonable stance? Maybe we need some community consensus on the notion that it would be valuable for Context to be able to carry optional data.

I want to point out one use case where keeping the size of Context small is valuable and these extensions are rarely needed: async executors on embeddeded targets.

On embedded targets the cost of these fat pointers can add up, especially if a new Context is created for every call to poll (this is done in every embedded async executor I've looked at). A few extra pointers in Context probably won't hurt performance much, but having "many extensions that executors and futures can opt-in to" sounds like it might.

Hi! Maintainer of compio here. We have been experimenting with Context::ext lately to implement our future combinators: https://doc.compio.rs/x86_64-unknown-linux-gnu/compio/runtime/trait.FutureExt. We use ext to carry and pass down additional data and eventually get consumed by runtime to alter behaviors. Some feedbacks:

To set ext data, ContextBuilder is necessary, which is gated behind local_waker, not entirely sure why.
My usecase is "if Ext exists, set additional data to it, if not, create one with the additional data and pass down", which for now seems the only way to do so is by branching on downcast result. Not sure if that's the intended way of using.
Would love to see this feature stabilize, if possible

Navigation Menu

Repository navigation

Tracking Issue for `context_ext` #123392

Public API

Steps / History

Unresolved Questions

Playground

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking Issue for context_ext #123392

Description

Public API

Steps / History

Unresolved Questions

Footnotes

Activity

jkarneges commented on Jun 8, 2024

jkarneges commented on Jun 8, 2024

jkarneges commented on Jun 15, 2024

traviscross commented on Jul 11, 2024

thomaseizinger commented on Mar 22, 2025

danielhenrymantilla commented on Oct 30, 2025

Playground

jkarneges commented on Nov 8, 2025

FeldrinH commented on Dec 18, 2025

George-Miao commented on Mar 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Tracking Issue for `context_ext` #123392