Enable building complete release artifacts on Travis / Appveyor

[brson]

Updated description

Active work items:

• Turn on Travis/AppVeyor nightlies - update secrets.toml
• Turn off Buildbot nightlies - comment out trigger in rust-buildbot
• Separate --enable-extended from x.py test builds on all platforms (only Windows remaining)
• Pull in Cargo as a submodule instead of from a pre-built location
• Run cargo's test suite on select bots.
• Turn off Cargo uploads

Original description

Converting our travis/appveyor CI to do full release builds. By doing release builds on every platform on every commit we should stop breaking nightlies so much.

• switch homu to travis/appveyor, leaving buildbot in place (Fix fs tests on Windows systems with non-english locales. #38536 (comment))
• modify rustbuild to build stage1 by default, with option for stage2 (rustbuild: Compile rustc twice, not thrice #38631)
• add stage2 bootstrap bot to verify the full bootstrap works (rustbuild: Compile rustc twice, not thrice #38631)
• add sccache to bots (rustbuild: Add sccache support #38359)
• add remaining cross-host rustc/platform builders (travis: Expand dist builder coverage #39042)
• add distcheck bot (Gate on distcheck on Travis #38708)
• have rustbuild produce .sha256 files on all dist output (travis: Start uploading artifacts on commits #38748)
• modify all builds to 'make dist' and upload (travis: Start uploading artifacts on commits #38748)
• confirm all matrix entries are correct

More steps

• Figure out rls submodule maintenance model
  • in-tree vs. out-of-tree and how
• Move rls into rust-lang-nursery (tools team)
• Add rls submodule/source to src/
• Add a --enable-extended / --enable-platform flag (rustbuild: Start building --enable-extended #39245)
  • teach bootstrap to only build, test and dist the rls when this flag is set
  • think about the name of this flag because it's important
• Add --enable-extended to the Trivas/AppVeyor release builders (rustbuild: Start building --enable-extended #39245)
  • this will start uploading rls packages to rust-lang-ci
• Modify rust-buildbot to pass --enable-extended to host dist builds (rustbuild: Start building --enable-extended #39245)
  • Starts building and uploading the RLS

At this point we've got rls sufficiently packaged that we can flip the
switch. Next steps are to flip that switch.

• Make and deploy patch to rust-buildbot to add rls to manifests as optional
• Make a new nightly to test RLS deployment
• Add rls to the combined installer via rust-packaging
  • tarballs, .pkgs, .msis
  • so that non-rustup users can access

We'd want to do all the above before the March 16 branch for 1.17 beta.

At this point we're delivering the RLS via the buildbot release
builders. Next steps are to follow through on the release builder
conversion.

• Add --enable-dist-msi and --enable-dist-pkg flags to configure (rustbuild: Start building --enable-extended #39245)
  • move those packaging steps from rust-packaging into the tree as
    part of make dist
  • remember that the rls is not included when --enable-extended is
    not enabled... hm
• Again in-tree, create a new build target, x.py dist src/publish-ci-build (or something) to do the final steps of
  arranging the bins, signing, building the manifest, and uploading to
  static.rust-lang.org (rustbuild: Add manifest generation in-tree #39284)
  • This is in-tree because it gives us a path for others being able
    to produce their own rustup dist servers, and also because it's just
    better to have all the code for packaging in the same git commit.
  • Being in-tree has the complication that we have to run ./configure
    even though we don't need to build any Rust...
  • All the steps need to (eventually) be discrete so that people
    who want to build manifests without our CI infrastructure can do so,
    but for our purposes we can have one x.py dist publish-ci-build
  • 'publish-download' step:
    • for any given channel, download the appropriate bins from rust-lang-ci
      s3 bucket, put them in the right file system layout
    • this might want to verify hashes
  • 'publish-hash' step:
  • 'publish-signing' step:
  • 'publish-manifest-build' step:
  • 'publish-upload' step:
• Build a docker container that runs a the above as a service
  • publish to some staging area for testing before we turn it on in prod
  • batch job every night
  • cron job in a docker container?
• Set up aws machine to run that service
  • let it run in parallel to existing build service
  • test with RUSTUP_DIST_SERVER
• port cargo build to the rust extended build system
  • simultaneously turn off cargo's existing publication
• modify docker images to use correct old glibcs (travis: Move glibc backwards in time #39198)
  • keep using crosstool ng
• disable llvm assertions for all DEPLOY travisveyor configs (travis: Move glibc backwards in time #39198)
• switch over to the new release builds, turn off buildbot

Other misc

• Uploads can overwrite one another, probably messing up checksums (travis: Stop uploading sha256 files #39167)
• Need to change release channel of artifacts to nightly (travis: Pass --release-channel=nightly on deploy #39050)
• Need to compile against libstdc++ statically (travis: Pass --enable-llvm-static-stdcpp #39132)
• Need to update all gcc toolchains to very old glibc (travis: Move glibc backwards in time #39198)
• Shouldn't pass --enable-llvm-assertions all the time, especially for stable/beta releases (travis: Move glibc backwards in time #39198)
• need a story for stable/beta releases

[brson]

re https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489

[Mark-Simulacrum]

modify rustbuild to build stage1 by default, with option for stage2

What's the expected approach to dealing with situations where cfg(stage0) or cfg(not(stage0)) is currently used? Currently, since stage0 is bootstrap (old compiler) and stage1 is a new compiler, there is a transition path, but with this I'm not sure what the expected arguments to x.py are.

[brson]

@alexcrichton will need to address those details @Mark-Simulacrum.

[alexcrichton]

@Mark-Simulacrum you may be interested in #38631, but the gist is that nothing changes, it's all the same. It may not be clear what terminology means what, but #38631 as a concrete implementation should hopefully clarify what's intended.

[Mark-Simulacrum]

So instead of stage0 we should now cfg(stage1)?

Perhaps a better way to ask my question is: when changes are made to the compiler that make a given lang item no longer necessary, what cfg should it be annotated with?

[alexcrichton]

Oh no everything should proceed as usual. When the compiler changes a lang item you use #[cfg(stage0)] and #[cfg(not(stage0))], as we do today.

Is there something specific, though, that you're worried about breaking?

[Mark-Simulacrum]

In #38574, I removed the trans_exchange_free lang item, and as such a compile with --increnental (which I understand is equivalent to the changes suggested above) failed since the beta compiler expected it to be defined. Perhaps I missed something, though.