Extending deref/index with ownership transfer: DerefMove, IndexMove, IndexSet

View all comments

It is clear that there is a need for the ability to move out of smart pointers and indexable things (DerefMove, IndexMove). The other frequently desired option is to IndexSet, which would be a special-cased version for indexing in this situation:

map[key] = value

currently, this is handled via IndexMut, but that is sub-optimal, because if the key is not already part of the map, the result is a panic.

Basic plan

DerefMove/IndexMove/IndexSet should layer on top of the traits we have now. One needs to be careful here because of subtle interactions with autoref and so forth.

Postponed RFCs

RFC: Add DerefMove #178
Introduce remaining Index traits #159
IndexAssign: overloading the a[b] = c expression #1129 (map[key] = value)

I'd like DerefMove for similar reasons that Servo wants it: servo/servo#3868

Namely, I have an type Foo, which owns a raw pointer from a crate that has FFI bindings:

struct Foo {
    handle: my_ffi_crate::foo // foo is a *mut to an uninhabited enum
}

and a "borrowed" version of that called BFoo:

struct BFoo<'a> {
    handle: my_ffi_crate::bfoo, // bfoo is the *const equivalent of my_ffi_crate::foo
    _phantom: PhantomData<&'a Foo>
}

Foo is the Rust equivalent of a type in the C library that's a flexible data storage type: think something like a C type that works like JSON. The C library supports things like: list indexing, map lookup, etc. which, in the Rust crate, have signatures like fn get<'a>(&'a self, key: Key) -> BFoo<'a>.

Most of the immutable "accessor" methods of Foo have to be implemented for both Foo and BFoo<'a> (a map might be from keys to more maps), which is annoying. It would make sense, I believe, to implement DerefMove<Target=BFoo<'a>> for &'a Foo, and then just implement those methods on BFoo<'a>. Of course, that might also require some additional autoref/autoderef magic.

A potential problem for the design of DerefMove - How would unsized types be handled? (e.g. FnOnce)

Missing from this list is also IndexGet, which is the "returns a value" version of Index.

@gankro what is the difference between your IndexGet versus the IndexMove in the list ?

Ah I assumed IndexMove took self by-value (given the comparison to DerefMove), but if it takes &mut self, then never mind.

what does the RFCs being postponed mean? will they simply be back on track as soon as someone has made up a coherent idea of how everything should play together and writes it up?

@flying-sheep

@flying-sheep it means "we're not rejecting this RFC because it's bad, now is just not the right time." This happened a lot before 1.0, so we could focus on what needed to ship in 1.0.

And yes, if there's a postponed RFC, someone needs to go through and evaluate if it's still valid in its current form, or update it, and re-submit it.

i’m glad that IndexGet/IndexMove will be a thing.

things like ndarray profit from it. generally, all data structures where “slice” views can’t be rust slices.

should i try to update the “remaining Index traits” RFC (#159)?

if so:

which name? the IndexRef/IndexValue train is gone, and @gankro was confused by IndexMove, so maybe IndexGet?
better do index_set as default-implemented trait for IndexMut or in a dedicated IndexSet?
also merge in IndexAssign (IndexAssign: overloading the a[b] = c expression #1129)?

@comex

I would say that IndexGet is the clearest, because of the ease of implementing it for both T and &T. IndexMove is too restricted as a name, in that you might not actually want to move anything, but instead you might want to copy.

Perhaps it would be possible to use type inference to decide whether to use Index or IndexGet. Example:

let some_array: [i32; 4] = [1, 2, 3, 4];
let some_slice = &some_array[0..3];
let x = some_slice[2]; // x: &i32 (default behavior)
let y: i32 = some_slice[2]; // y: i32 (would use `IndexGet` to pass by value

This would more or less follow from the current inference of IndexMut.

I do see the potential for compiler confusion in choosing which trait to use for a given operation, but I would at least appreciate a default implementation of IndexGet<E, R> for &[R] where R: Copy. Syntactic sugar ([E]) aside, this would let us assume that slices would be acceptable, and additional implementors of IndexGet could define their own behavior (whether they want to copy, clone, whatever).

I would also like to mention that even though copying-when-indexing is antithetical to C/C++, it is a very common idiom when it comes to Ruby, etc. I don't think this is a weakness in Ruby, but in fact allowing the end-user to use one idiom or other other should help ease the learning curve for people coming from dynamic languages that assume copy.

One of my greatest frustrations with the Ruby language is not knowing whether array indexing was happening by-value or by-reference. I think if Rust could acknowledge both of these domains, and allow programmers the explicit option of choosing one or the other, it would go a long way.

Edit: thanks @comex for pointing out that I indexed a stack array rather than a slice initially

@andrewcsmith

@andrewcsmith some_slice[2] is already i32 - you need &some_slice[2] for &i32. The indexing operator carries an implicit dereference.

(some_slice is also not a slice...)

Theory

Both deref and index are for types containing values of other types. They allow to conveniently access contained value.
The main difference is that deref types usually contain only one instance of other type, while index types usually contain multiple instances.
There are actually many ways to access internal instance, and index/deref families should cover most of them.

There are two cases currently supported:

(1) accept ref, return ref (Deref/Index)
(2) accept mut ref, return mut ref (DerefMut,IndexMut)

There is one case requested for derefs:

(3) consume value, return value (DerefMove?,IndexMove?)

It's an open question whether or not we need Index counterpart. I personally don't think it will be very useful.

There are two more index cases requested:

(4) accept ref, return value (IndexGet?)
(5) accept mut ref, accept value, set value (IndexSet?)

They can be used for collections not actually storing value instances (e.g. bit array setting/returning bools).
Case 5) can also be used to insert new values to hashmaps, which I think is a highly desirable feature.
These cases apparently do not need deref counterparts.

Practice

Now let's get to practice, and please correct me if I'm wrong about current rules.
Here I assume that S is source type, T is target type and I is index type. s, t and i will be corresponding instances.

Deref

Current deref rules

Current manual deref follows simple rules:

&*s- perform Deref
&mut *s - perform DerefMut
*s and S is Copy - perform Deref and copy
Note that if there is no DerefMut and S is Copy, then &mut *s fails, but &mut {*s} succeeds.
If T is not Copy, then *s will fail. Copyability of S does not matter.

Autoderef follows the same rules, but also depends on context:

If context needs &T - perform Deref
If context needs &mut T - perform DerefMut
If context needs value of T and T is Copyable - perform Deref and copy.
Note that rule autoderef rule 3) is somewhat inconsistent, it works for self in method calls, but does not work for arguments in function calls.
If context needs value and T is not Copy, then it fails. Copyability of S does not matter.

Proposed deref rules

Deref rules proposed in #178, in short, are: try to move copyables, try to ref non-copyables, otherwise use whatever possible.
I do mostly agree with the proposed rules, but don't really like 'use whatever possible' part.
#178 says that, for example, if there is no Deref, but DerefMove, then it should be used for all derefs.
Let's imagine that there is no Deref, but T is Copy. In this case &*s would perform DerefMove and take reference of the result, which is both non-intuitive and contradicting with how Deref currently works (see my note about missing DerefMut).
I'd say it's better to fail in this case. Another, even simpler option is to make Deref implementation mandatory for types implementing DerefMove. I think it's reasonable and makes everything easier.

Proposed manual deref rules:

&*s - perform Deref
&mut *s - perform DerefMut
*s and S is Copy and implements DerefMove - copy source and perform DerefMove
*s and T is Copy and S implements Deref - perform Deref and copy result
*s and neither 3) or 4) apply - perform DerefMove

Proposed autoderef rules:

If context needs &T - perform Deref
If context needs &mut T - perform DerefMut
If context needs value of T, S is Copy and implements DerefMove - copy source and perform DerefMove
If context needs value of T, T is Copy and S implements Deref - perform Deref and copy result
If context needs value of T, but neither 3) or 4) apply - perform DerefMove

If rule condition is satisfied, but required trait is not available, then that's a failure and compiler should not try to apply other rules (that's the only thing different from #178)

It is possible to support DerefGet/DerefSet, but, then again, it's noticeably ramps complexity and likely not very useful.
If needed, DerefGet/DerefSet should follow strategy for IndexGet/IndexSet described below.

Indexing

Current indexing rules

Indexing is more or less follow deref, but is generally simpler, because there is no autoderef counterpart.

Current rules for indexing:

&s[i] - perform Index
&mut s[i] - perform IndexMut
s[i] and T is Copy - perform Index and Copy
If T is not Copy, then s[i] will fail.

Proposed indexing rules

This extends both #159 and #1129, adds more details and answers some unresolved questions.

Main problem is IndexGet. Option of having it adds lots of ambiguity into rules.
Also, possiblity of different implementation of Index and IndexGet scares me.
I think that if collection need IndexGet, then having Index/IndexMut/IndexMove does not make sense.

I propose to make IndexGet and Index/IndexMut/IndexMove mutually exclusive.
Types implementing IndexGet can have automatically generated default implementation of Index/IndexMove for trait compatibility purposes, but manually implementing Index/IndexMut/IndexMove should be forbidden.

With mutually exclusive IndexGet and Index/IndexMut/IndexMove, indexing rules actually become pretty simple.

For types implementing IndexGet:

s[i] - perform IndexGet
s[i] = expr (asssignment) - perform IndexSet
s[i] += expr (compound assignment) - perform IndexGet, apply non-compound operation, then IndexSet result
With these rules &s[i] and &mut s[i] are still possible, but they represent references to temporary value returned by IndexGet. Note that it differs from the way Deref works, but I think it worth it. For deref it is not as useful because autoderef can handle most of function interface compatibility issues. Indexing does not have autoderef counterpart and therefore it's not wise to prohibit taking references.

For all other types:

&s[i] - perform Index
&mut s[i] - perform IndexMut
s[i] = expr (asssignment) - perform IndexSet (note that it should NOT fallback to IndexMut)
s[i] += expr (compound asssignment) - perform IndexMut, apply compound operation on it's result (note that it should NOT fallback to IndexSet)
s[i] and T is Copy - perform Index and Copy
s[i] and T is not Copy - perform IndexMove

Just like with derefs, if rule condition is satisfied, but required trait is not available, then that's a failure and compiler should not try to apply other rules.

Note that I'm not sure if we need IndexMove (consume value, return value), but I list it here for completeness sake.

Indexing examples

// Bits is a tightly packed bits array, implements IndexGet/IndexSet
println!("first bit is: {}", bits[0]); // bits.index_get(0)
bits[1] = true; // bits.index_set(1, true)
bits[2] ^= true; // bits.index_set(2, (bits.index_get(2) ^ true))
let tref = &bits[3] // &bits.index_get(3), reference to temporary

// Array implements Index/IndexMut/IndexSet
println!("first element is: {}", array[0]) // *array.index(0)
let second = &array[1] // array.index(1), ref of second element
let third = &mut array[2] // array.index_mut(2), mutable ref of third element
array[3] = 2; // array.index_set(3, 2)
array[4] += 3; // (*array.index_mut(4)) += 3

// Map implements:
//   Index/IndexMut for Index types where Key impl Borrow<Index>
//   IndexSet/IndexMove for Index types where Key impl From<Index>
let vref = &array["key"] // array.index("key"), ref of value
let vrefmut = &mut array["key"] // array.index_mut("key"), mutable ref of value
map["new key"] = 1; // map.index_set("new key", 1), creates new entry
map["new key"] += 2 // (*map.index_mut("new key")) += 2, modifies existing entry
return map["key"]; // map.index_move("key"), consumes map
// failures:
&map["missing key"] // map.index("missing key"), fails
&mut map["missing key"] // map.index_mut("missing key"), fails
map["missing key"] += 2 // (*map.index_mut("missing key")) += 2, does not insert value, fails

P.S.

It's the longest comment I've ever wrote
If necessary, I can create a new rfc with even more implementation details
Or I can start hacking, you know =)

In the case of indexing, what would happen to
s[i].do(), for some impl T {fn do(&self);}?

would it become
s.index_move(i).do() by rule 6
or perhaps worse
s.index(i).clone().do() by rule 5, where clone has the Copy implementation

obviously it would be nice if it became (&s[i]).do()

Wow, it's been a while.

You're absolutely right, that's a very important use case I missed.

I guess rules have to be modified like this to make it work:

&s[i] or s[i].method(...) where method needs &self - perform Index
&mut s[i] or s[i].method(...) where method needs &mut self - perform IndexMut
...

I hate to see how complex it gets, but still can't find a more usable solution.

The same goes for Borrow. The Borrow trait is pretty useless as it is now. It even needs a workaround in the language to work for &[] and &str as the conversion creates a (hidden) new type.

Borrow
Deref

Given that we're in the impl Period, is there any motion to make forward progress on this? In addition to the heavily discussed stuff, this issue seems to be a blocker for things like Box<FnOnce> which have been a sore spot for years now.

I also don't see a clear summary of what is blocking this, in case someone wanted to hack on this, which I think means the design work was never completed? Just trying to get a feel for the situation.

I guess another way we could handle this is to require Q: ToOwned<Owned = K>, and just convert it to the owned key on insertion.

This would still be a pretty significant limitation on index_mut, but it wouldn't introduce any major foot-guns.

Edit: although this would still force a clone to insert so maybe not

DerefMove is not closely related to indexes. So why not split them in different topics?
DerefMove is simply the "owned" version of Deref (while DerefMut is the "mutable" version) that consumes the pointer.
Because DSTs can only be accessed behind a pointer, they can be prohibited from implementing DerefMove.
Then what is currently blocking the stabilization of DerefMove? Could we have an update in the original post so as not having to follow all the messages here?

What would the definition of DerefMove be. There are multiple options. It can't be trait DerefMove { fn deref_move(self) -> Self::Target; } as that doesn't work for unsized Self::Target.

As I said DSTs may not implement it, because you can't play with DSTs without pointers anyway. Box::into_inner doesn't work for DSTs, and is that a problem?

You can dereference an unsized box on nightly if you pass the result as argument to a function. This is not something we can just give up on as the FnOnce impl for Box<dyn FnOnce()> depends on it.

Another thing is that you can partially move out of a box even on stable. This is incompatible with fn deref_move(self) -> Self:::Target too as that will move the entire content of the box all at once.

You can dereference an unsized box on nightly if you pass the result as argument to a function. This is not something we can just give up on as the FnOnce impl for Box<dyn FnOnce()> depends on it.

Could you please post an example? I'm wondering what function can take a dereferenced unsized type as argument. Shouldn't they always stay behind pointers (and if so Deref or DerefMut might apply)?

Another thing is that you can partially move out of a box even on stable. This is incompatible with fn deref_move(self) -> Self:::Target too as that will move the entire content of the box all at once.

DerefMove means to consume the (entire) box, so yes you can't have anything in the box afterwards, not even a partial box, because the box itself is gone. If Box is such a special box that needs to have partial move, then DerefMove is not good for it. Maybe the compiler could be made more clever to differentiate let _ = *box from let _ = (*box).0? I'm not very inclined to this idea, but I think partial move behind box doesn't quality the blockade of implementing DerefMove.

Could you please post an example? I'm wondering what function can take a dereferenced unsized type as argument. Shouldn't they always stay behind pointers (and if so Deref or DerefMut might apply)?

fn accept_unsized_fnonce(a: dyn FnOnce()) { a(); } is an example. Rustc will use a pointer in the calling convention, but it is an owned value in the surface language, so you can move out of it like regular owned value. This is necessary for calling FnOnce for example as it moves the captured upvars of the closure. If you were to call accept_unsized_fnonce with *mybox as argument, rustc would take a pointer to the inner value of mybox and pass it to accept_unsized_fnonce and only after the call returns deallocate the box, but without dropping the inner value as that would have been dropped by accept_unsized_fnonce already.

IndexGet would be invaluable for all data structures with an implicit representation, and they're very important today in large data center: inverted indices, graph storage, etc., very often use succinct or compressed [immutable] representations, and there's no way to implement Index on them because there's no way to return a reference (as the data is represented implicitly). We are working on a library of succinct data structures and we had to define manually get methods. It would be really nice to have a standard "index and return a value" trait instead, both for ease of integration with other libraries and for ergonomics.

BTW, if there's any kind of requirement on the compiler to support these traits properly the 2024 edition would be the right occasion to think about it.

fn accept_unsized_fnonce(a: dyn FnOnce()) { a(); } is an example.

that doesn't compile currently on playground. but it seems like what you described would apply to the impl FnOnce for Box<FnOnce>

fn accept_unsized_fnonce(a: dyn FnOnce()) { a(); } is an example.

that doesn't compile currently on playground. but it seems like what you described would apply to the impl FnOnce for Box<FnOnce>

Yes, as bjorn3 said, that works on nightly, not on stable yet. It also requires the unstable feature #[feature(unsized_fn_params)], which is implicitly used by the FnOnce<_> impl for Box<dyn FnOnce<_>>.

The Unstable Book section for the unsized_locals feature even mentions that:

One of the objectives of this feature is to allow Box<dyn FnOnce>.

clone and index traits

Any chance this will be looked into again? That would be a fabulous Xmas gift...

Navigation Menu

Repository navigation

Extending deref/index with ownership transfer: DerefMove, IndexMove, IndexSet #997

Basic plan

Postponed RFCs

Theory

There are two cases currently supported:

There is one case requested for derefs:

There are two more index cases requested:

Practice

Deref

Current deref rules

Proposed deref rules

Indexing

Current indexing rules

Proposed indexing rules

Indexing examples

P.S.

66 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Extending deref/index with ownership transfer: DerefMove, IndexMove, IndexSet #997

Description

Basic plan

Postponed RFCs

Activity

tomjakubowski commented on Aug 3, 2015

thepowersgang commented on Nov 20, 2015

Gankra commented on Nov 30, 2015

pnkfelix commented on Nov 30, 2015

Gankra commented on Nov 30, 2015

flying-sheep commented on Mar 17, 2016

steveklabnik commented on Mar 17, 2016

flying-sheep commented on Mar 17, 2016

andrewcsmith commented on Apr 1, 2016

comex commented on Apr 2, 2016

VFLashM commented on Jun 15, 2016

Theory

There are two cases currently supported:

There is one case requested for derefs:

There are two more index cases requested:

Practice

Deref

Current deref rules

Proposed deref rules

Indexing

Current indexing rules

Proposed indexing rules

Indexing examples

P.S.

spiveeworks commented on Jun 8, 2017

VFLashM commented on Jun 8, 2017

chpio commented on Jun 8, 2017

coder543 commented on Oct 22, 2017

66 remaining items

RustyYato commented on Jul 31, 2023

tae-soo-kim commented on Sep 2, 2023

bjorn3 commented on Sep 2, 2023

tae-soo-kim commented on Sep 2, 2023

bjorn3 commented on Sep 2, 2023

tae-soo-kim commented on Sep 3, 2023

bjorn3 commented on Sep 3, 2023

vigna commented on Dec 15, 2023

vigna commented on Dec 16, 2023

Zoybean commented on May 14, 2024

tesaguri commented on May 15, 2024

vigna commented on Nov 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions