rustdoc: decrease SEO rank of integer modules / deprecated items

[tgross35]

Is there anything that can be done to decrease the SEO rank of deprecated pages? The impetus for this was linking to deprecated std modules such as numbers:

but it is likely worth doing for docs.rs as well if not too difficult. I am unsure whether there is a separate repo dedicated to the infrastructure of docs.rust-lang.org (please direct me if so) but figured any discussion here would also be valid there.

cc @jsha since you seem to be the resident expert based on rust-lang/docs.rs#1438

This parallels #98759 for internal rustdoc sorting. Relevant zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/219381-t-libs/topic/Deprecation.20of.20std.3A.3Au32.20shells

[jsha]

Thanks for spotting this! This is definitely worth fixing. I agree with you that it's a bad experience to google something like u16 max and wind up on a deprecated page.

Part of what's going on here is called "term frequency." The terms "u16" and "MAX" have a high relative frequency within https://doc.rust-lang.org/std/u16/constant.MAX.html because it's a small page. It's very clearly "about" those two terms.

They have a low frequency within https://doc.rust-lang.org/std/primitive.u16.html because that's a huge page, with many different terms. That page is "about" u16 but it's not meaningfully "about" MAX.

We can add noindex meta tags to https://doc.rust-lang.org/std/u16/constant.MAX.html but that won't make primitive.u16 rank first. Judging by current search results, it would just make the unofficial copy of the docs at web.mit.edu rank first, which would be even worse.

What if we use the module page as a landing spot for these queries? Right now https://doc.rust-lang.org/std/u16/index.html shows a deprecation warning, but we could un-deprecate the module and just leave the constants deprecated. Then we would write some manual documentation at the module level talking about u16::MAX and u16::MIN and showing examples of how to use them. If we do that, wait a while, and that page starts ranking for the [u16 max] query, we could add meta noindex tags to the constants pages.

For the general case: I don't think we should noindex all deprecated pages automatically. For instance people still need to be able to search for mem::uninitialized. And short of noindex, there's no good way to downrank a page; you can only try to make other pages rank better.

You can also help a bit by writing lots of blog posts that talk about u16::MAX and link to the primitive.u16 page. :-D

[jsha]

Here's another way of looking at it: If your goal is to find out how to refer to the max value of u16 numbers, or what the max value is, then https://doc.rust-lang.org/std/u16/constant.MAX.html is actually the perfect page! It's short, sweet, and to the point. The only problem is that it has a bunch of distracting stuff about deprecation that you don't want.

But we need to actually mark these constants as deprecated so that the compiler will generate warnings, and rustdoc will always show those deprecation warnings.

What about just deciding not to deprecate them after all? Is there a large maintenance burden to keeping std::u16::MAX and friends around forever? And then we can explain in the documentation that u16::MAX is stylistically preferred. But there would be no big scary thumbs-down icon.

[tgross35]

Hey, thanks for the quick and thorough response!

The main reason I brought it up is because I think it's somewhat bad PR that the landing pages for some simple integer searches bring you to a page stamped with "deprecated" all over - potentially confusing for newcomers expecting a stable language, and somebody on the thread said they've seen questions asking whether Rust is deprecating its integers.

So, regardless of the resolution here, I'm going to make a PR that marks them #[doc(hidden)] and see what the feedback is (I am nobody in a position to make official decisions). Especially because of confusion in the docs like this, unrelated to SEO:

Personally, I'd vouch for hiding the docs for these to avoid confusion and potential fragmentation of usage. But if that gets shut down, your idea about actually undeprecating it is a good one to bring up.

Just brainstorming here - maybe it would make sense to have a new #[doc(no_seo_index)] tag that could be optionally used by std and large authors alike, which could then be used as part of a large possible deprecation flow (active -> planned deprecation -> deprecated -> warnings (if std) -> no SEO -> doc(hidden) -> removal (if not std)).

I'm not a blog author but it's interesting to know that that makes a difference :)

[jsha]

For context: RFC 2700, Associated Constants on Ints, 2019, in the Motivation section:

Documentation. On the front page of the standard library API docs, 12 of the 60 modules in the standard library (20%) are the aforementioned numeric modules which exist only to namespace two constants each. This number will increase as new numeric primitives are added to Rust, as already seen with i128 and u128. Although deprecated modules cannot be easily removed from std, they can be removed from the documentation, making the stdlib API docs less cluttered and easier to navigate.

Beginner ease. For a beginner, finding two identical ways to achieve something immediately raises the question of "why", to which the answer here is ultimately uninteresting (and mildly embarrassing). Even then the question of "which one to use" remains unanswered; neither current approach is more idiomatic than the other. As noted, deprecated items can be removed from the documentation, thereby decreasing the likelihood of head-scratching and incredulous sidelong glances from people new to Rust.

So it looks like #[doc(hidden)] has been part of the plan all along.

I suspect the head-scratching and incredulous sidelong glances are made worse by the deprecation-TBD warning. The search [u32 deprecated] yields a Stack Overflow post asking "Why will Rust integer types be deprecated soon?" (2022) and a Rust Internals post asking about "Planned deprecation of un/signed integer types?". There's also a Reddit thread "Why are sized integers (signed and unsigned) planned to be deprecated?".

[jsha]

One small and easy tweak would be to remove the deprecation-TBD warning from the modules but keep it on the constants. Per the Reddit thread above it seems like the deprecation notice for the modules may be causing the most confusion, because it causes a warning to show up in the parent page (https://doc.rust-lang.org/std/#modules).

[tgross35]

Thanks for putting all the context together. I linked your comment in #107587, the PR where we can discuss changes on the lib side some more.

I think that from your answers here I've gleaned that there's not much that can be done on the SEO side of things, neither for numbers specifically nor for anything else deprecated. So I think this is good to close (woohoo! 8858 open issues instead of 8859!)