regression in method dispatch

[nikomatsakis]

According to @SimonSapin, #43880 likely broke Servo's build. Servo itself is fixed in servo/servo#18327, but I'm opening this issue to decide on whether this regression is a bug or expected behavior.

cc @arielb1

[SimonSapin]

The code that broke is:

#[derive(JSTraceable)]
pub enum Filter {
    None,
    Native(fn (node: &Node) -> u16),
    JS(Rc<NodeFilter>)
}

/// A trait to allow tracing (only) DOM objects.
pub unsafe trait JSTraceable {
    /// Trace `self`.
    unsafe fn trace(&self, trc: *mut JSTracer);
}

unsafe impl<A, B> JSTraceable for fn(A) -> B {
    #[inline]
    unsafe fn trace(&self, _: *mut JSTracer) {
        // Do nothing
    }
}

#[proc_macro_derive(JSTraceable)]
pub fn expand_token_stream(input: proc_macro::TokenStream) -> proc_macro::TokenStream {

    // ...

    let style = synstructure::BindStyle::Ref.into();
    let match_body = synstructure::each_field(&mut type_, &style, |binding| {
        Some(quote! { #binding.trace(tracer); })
    });

    // ...

I think the generated code would look something like this:

unsafe impl JSTraceable for Filter {
    unsafe fn trace(&self, tracer: *mut JSTracer) {
        match *self {
            Filter::Native(ref field1) => {
                field1.trace(tracer);
            }
            // ...
        }
    }
}

The error message in rustc 1.21.0-nightly (7eeac1b 2017-08-30) was:

   Compiling script v0.0.1 (file:///home/simon/servo2/components/script)
error[E0599]: no method named `trace` found for type `&fn(&dom::node::Node) -> u16` in the current scope
   --> /home/simon/servo2/components/script/dom/treewalker.rs:464:10
    |
464 | #[derive(JSTraceable)]
    |          ^^^^^^^^^^^
    |
    = note: JSTraceable is a function, perhaps you wish to call it
    = help: items from traits can only be used if the trait is implemented and in scope
    = note: the following trait defines an item `trace`, perhaps you need to implement it:
            candidate #1: `dom::bindings::trace::JSTraceable`

error: aborting due to previous error

error: Could not compile `script`.

The fix was adding a new impl for fn(&A) -> B, in addition to the one for fn(A) -> B. Isn’t the former a subset of the types covered by the latter?

unsafe impl<'a, A, B> JSTraceable for fn(&A) -> B {
    #[inline]
    unsafe fn trace(&self, _: *mut JSTracer) {
        // Do nothing
    }
}

[nikomatsakis]

@SimonSapin thanks for the detailed notes.

I was talking to @arielb1 and I understand better the reason for the change. I am of two minds about it, to be honest. In short, under the older algorithm, we integrate "impl selection" directly into method dispatch. In particular, for each potentially applicable impl, we would instantiate the impl and then check whether the receiver type was a valid subtype of what the impl expected. In this case, the receiver type is for<'a> fn(&'a u8). This is a valid subtype of fn(A) where A = &'x u8, because 'a can be instantiated as 'x. Under the newer algorithm, we use trait selection instead. This creates a problem: an impl like impl<A> Foo for fn(A) doesn't implement Foo for for<'a> fn(&u8), because of the placement of the region binders.

This handling of region binders is a subtle point in Rust, and one that I actually consider a borderline bug. In particular, I'm in the process of various refactorings that aim to make the internal handling of higher-ranked things more consistent with all other regions, which offers a lot of simplicity across the board. But one side-effect is that we would no longer permit an impl of Foo for both fn(&u8) and fn(A) -- in other words, the workaround that @SimonSapin put in place would no longer work, which seems suboptimal. (In particular, I aim to have region erasure erase region bindings altogether.)

(It may be that these refactorings I would like to do can't be done the way I wanted to do them, though, since there could be other code relying on the existing behavior; it's hard to be sure without a working branch to test with (which I think is close)).

[SimonSapin]

because of the placement of the region binders.

I don’t understand what that means. Is this a concept of the language that is relevant to its users, or only an internal detail of the compiler’s implementation?

But one side-effect is that we would no longer permit an impl of Foo for both fn(&u8) and fn(A)

Right, #43880 was a breaking change so effectively reverting it is another breaking change (unless it happens within the same release cycle).

The team needs to decide whether this breakage is acceptable per https://github.com/rust-lang/rfcs/blob/master/text/1122-language-semver.md.

[SimonSapin]

And the larger policy question is: “it doesn’t affect any code that happens to be tested by crater/cargobomb” sufficient to justify any breaking change, even if it not a soundness fix?

[nikomatsakis]

@SimonSapin

I don’t understand what that means. Is this a concept of the language that is relevant to its users, or only an internal detail of the compiler’s implementation?

Sorry, I geeked out a bit in the response there. The answer is somewhere in the middle. In short, there are corner cases of the language that interact with particulars in the implementation. Sometimes this gives rise to complicated or internally inconsistent definitions. The goal of #43880 was effectively to remove some "conceptual duplication" between how trait matching worked during method dispatch and how it works at other times.

The changes I was describing around subtyping and higher-ranked regions are another such case. They are independent except that they both interact on this particular example.

Right, #43880 was a breaking change so effectively reverting it is another breaking change (unless it happens within the same release cycle).

This does not necessarily -- that is, I don't know that @arielb1's new method dispatch code accepts anything in particular that the old code did not, but I could be wrong.

The team needs to decide whether this breakage is acceptable per https://github.com/rust-lang/rfcs/blob/master/text/1122-language-semver.md.

Indeed, we do. In my view, all of the changes under discussion fall roughly under the "Underspecified language semantics" heading. This is not to say we should necessarily make those changes -- just that they are the kinds of changes that arise we seek to clean up the compiler implementation and the definition of the trait system. In such cases, the "real-world effects" (i.e., how much code is affected) is definitely a consideration, as is the magnitude of the cruft.

[SimonSapin]

In my view, all of the changes under discussion fall roughly under the "Underspecified language semantics" heading.

Alright, that's fair enough.

And as @aturon just said on IRC it's also the role of Nightly to catch issues like this that might have slipped through cargobomb, with time before they reach Stable.

[nikomatsakis]

@rfcbot fcp close

OK, having had time to think this over, I think the general feeling is that the change here was both permitted and a good change, although the regression is unfortunate. The intention is to clarify and improve method dispatch, and in particular to avoid having two "similar but different in subtle ways" implementations of how to match a trait against impls.

This does however mean that the behavior with respect to function types, consolidating on the behavior using during trait selection (which hence requires that the type of receiver be equal to the type in the trait impl, rather than being a subtype).

I'm going to move that we close this issue, therefore.