Support intra-doc links on cross-crate reexports

[Manishearth]

Previously: #60883, #59833

This is the last major issue blocking the stabilization of intra doc links (#43466)

Currently, if you use intra-doc links on some item:

// crate "foo"

/// This is not [Baz]
pub struct Bar;

pub struct Baz;

the docs will not turn up on any crate reexporting it

pub use foo::Bar;

The docs will have a bare [Baz] instead of the appropriate intra-doc link, because we resolve intra-doc links locally. (More examples in #60883, #59833)

---

This is tricky to fix; we throw away module resolution information before packaging up crate metadata, so it's not possible to "unseal" a crate and rerun the resolution within it. One solution would be to pre-detect intra doc links and store resolutions in metadata, but this would be expensive since it involves parsing everything as markdown unconditionally.

A better solution is to save resolution information in the metadata, as some kind of tree keyed to the module.

@eddyb's comment outlines the correct approach to use here

What you actually want to persist cross-crate is the scope information that lets you resolve names in a certain (module) scope, after the fact. It'd also allow not doing all the the resolver hacks.

And there's a very good reason to want it in the compiler, at least same-crate: we could (finally) tell if a type is in scope somewhere so we can actually print Vec<_> instead of std::vec::Vec<_> etc.

Basically, instead of throwing away resolver metadata, we should serialize it into some form and store it in the metadata, so that it can be reloaded as necessary.

[kinnison]

I'm interested in helping on this, but right now I'm not confident in the requisite parts of the compiler. I will take a poke around, but please don't let that stop someone who actually knows what they're doing getting it done :D

[eddyb]

At this point I really wish rustdoc always took a set of crates' sources and never relied on metadata from rlibs.
It would make so many things easier, including this.

[kinnison]

Okay, I've spent a bit of time trying to find my way around the libraries which comprise the compiler, looking for things associated with the metadata and/or name resolution. I'm starting to get a feel for things, but I'm still not certain I know quite what I'm trying to achieve. Since this is E-hard I wasn't expecting to work it out right away fortunately, and I'm prepared to keep learning, but it'd be incredibly helpful if someone were prepared to help point me toward the right places to be learning about. Once I feel like I've done enough basic learning, I'll probably post a rough outline of what work I think needs doing to complete this (kinda as an expansion of Manish's original description) and then ask for review of that.

@Manishearth or @eddyb I'm guessing would be best to ask for that review, so this is me pre-asking to see if the above approach would work for either of you?

[Manishearth]

This would work for me, I think @eddyb is far more familiar with some of the stuff here than I am.

librustc_resolve contains a resolution tree that effectively needs serializing, but ideally a reduced form of it.

[eddyb]

Probably the best way to do this would be in two steps:

1. come up with some data structure to place in librustc
   • it only needs to handle being from the same crate, to begin with
   • librustc_resolve would export this, and librustdoc would use it to resolve (some?) links, potentially removing the need for "late Resolver usage" in librustdoc
   • @petrochenkov is most likely to know how and what to exfiltrate from librustc_resolve, I can only guess what parts are relevant
2. once you have that, you can make librustc_metadata encode/decode it
   • the actual serialization is based on #[derive]s so all you really need to do is add the plumbing around that
   • this is the part I can actually help with

[Manishearth]

come up with some data structure to place in librustc

Basically needs to be a map of module DefId to all contents (imported and otherwise) by name.

Something like HashMap<DefId, ModuleNames>, struct ModuleName {types: HashMap<Ident, DefId>, values: .., macros: ..}

idk how heavy that would be, or how much work that would be to compute

[kinnison]

That all sounds super-useful. I guess I need to start learning what a DefId is etc. properly :D

[eddyb]

@Manishearth Hmm, sounds like the way ExportMap works, except not just for exports. I thought rustdoc would need something more fine-grained than mods, but if not that's great.

EDIT: just realized after sending my comment, that since only exports from modules are documented, only module-level imports would be relevant.
Overall this seems pretty easy, I wonder what's the worst part. Glob imports?

[Manishearth]

@eddyb precisely.

Also, currently we don't do inner imports anyway, the following will not work (except for modules):

fn foo() {
   //! [baz]
   use bar::baz;
}

It's debatable if we want that to work, and in that case we can use local resolution though i'd prefer if we only used one way of resolving.

Maybe we can just make ExportMap contain all the module level items, with a pub/priv tag?

Of course this doesn't 100% solve your other problem of being able to spew scoped diagnostics, but it's a step.