Yk copy good builds#13

vext01 · 2019-04-23T09:33:09Z

Successful builds are tarred up and placed in /opt/ykrustc-bin-snapshots on the build master.

For now, store at most one archive, but we might change that in the future.

I've tested something very close to this (just without the chmod line in the buildbot script). It should work fine 😸

$ ls -al /opt/ykrustc-bin-snapshots/
total 188948
drwxrwxr-x 2 root             buildbot_workers      4096 Apr 18 18:23 .
drwxr-xr-x 8 root             root                  4096 Apr 18 11:32 ..
-rwxrwxr-x 1 buildbot-worker4 buildbot-worker4 193469353 Apr 18 18:23 ykrustc-stage2-057d397c.tar.bz2
$ cd /tmp
$ tar jxf /opt/ykrustc-bin-snapshots/ykrustc-stage2-057d397c.tar.bz2 
$ cd ykrustc-stage2/
$ ls
bin  lib  VERSION
$ cat VERSION 
commit 057d397c982463d374f60e16f9b3863dc660e4f2
Author: Edd Barrett <vext01@gmail.com>
Date:   Thu Apr 18 12:04:20 2019 +0100

    Kill [install].
$

If you agree with this, then I suppose the next step is to try using the tarball in:
ykjit/yk#1

ltratt · 2019-04-23T09:37:44Z

.buildbot.sh

+# Archive the build and put it in /opt
+GIT_HASH=`git rev-parse --short HEAD`
+TARBALL_TOPDIR=ykrustc-stage2
+TARBALL_NAME=ykrustc-stage2-${GIT_HASH}.tar.bz2


TARBALL_NAME=${TARBALL_TOPDIR}-${GIT_HASH}.tar.bz2?

ltratt · 2019-04-23T09:38:22Z

.buildbot.sh

+GIT_HASH=`git rev-parse --short HEAD`
+TARBALL_TOPDIR=ykrustc-stage2
+TARBALL_NAME=ykrustc-stage2-${GIT_HASH}.tar.bz2
+SNAP_DIR=/opt/ykrustc-bin-snapshots


I never know what opt is supposed to mean on Linux. Is it a directory the OS / distribution reserves for itself? In other words, can we guarantee we're not causing some weird future problem?

/opt is a place for stuff you installed manually. We already rely on this a lot for various buildbot software and dependencies, for example buildbot itself is intalled in a virtualenv under /opt.

Strictly speaking the snapshot dir isn't super-suited to /opt as the binary tarballs are not immediately executable from that location. Feel free to propose an alternative, but I'm inclined to keep all of the buildbot dependencies in one location. I already have a README file in /opt describing the meaning of each sub-dir.

https://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/opt.html

(BTW /opt is nothing to do with Linux. I first learned of this convention in Solaris 8)

Some OSs / distributions install packages under /opt. So long as our machines don't do that, I'm fine with using that, at least in the short term.

I've never seen /opt used by a package manager. I think we are OK on Debian for sure.

ltratt · 2019-04-23T09:38:50Z

.buildbot.sh

+git show -s HEAD > ${TARBALL_TOPDIR}/VERSION
+tar hjcvf ${TARBALL_NAME} ${TARBALL_TOPDIR}
+chmod 775 ${TARBALL_NAME}
+rm -f ${SNAP_DIR}/ykrustc-stage2-*.tar.bz2


Not sure I understand why this line is here.

It's removing the old binary snapshot, if it exists.

Ah, shouldn't it do rm -f ${SNAP_DIR}/${TARBALL_NAME}?

No, because we don't know the filename of the previous build. ${TARBALL_NAME} is the new tarball.

Note the short git hash in the filename.

As I alluded to in the description, I think in the longer term we want some kind of rolling back-log of the most recent (say) 5 builds, but let's not complicate things unless we really need to ;)

Why would there be an old build lurking around? I'm generally wary about * and rm unless we really can't do anything about it!

If you want to make sure the current tarball is always deleted you could do something like trap 'rm -f $blah' 1 2 3 13 15?

I think you misunderstand.

Suppose we've already done a build in the past. /opt/ykrustc-bin-snapshots will already contain a tarball, say ykrustc-stage2-111111.tar.bz2. Then the next build will make a new tarball with a different name, say ykrustc-stage2-222222.tar.bz2.

At then end of the second build, we remove the old tarball before moving in the new one. This is for no other reason than to not fill the disk.

Ah. Why are we complicating this? Shouldn't we just write to /opt/ykrustc-bin-snapshots/latest.tar.bz2 for simplicity, at least at first?

We can do, if you are content with only the file's date stamp as an identifier.

Of course you can untar it and look athe the VERSION file.

I'd prefer to start simple and build the extra complexity later if we find we need it.

vext01 · 2019-04-23T10:26:35Z

The last commit switches to using a static filename.

I've also renamed the existing file on the buildbot master so we can check it overwrites correctly.

If you agree, we can squash the last commit away.

ltratt · 2019-04-23T10:29:01Z

Please squash.

vext01 · 2019-04-23T10:30:23Z

splat.

ltratt · 2019-04-23T10:31:41Z

Can we squash away the set -e commit? It's not very helpful on its own IMHO.

Successful builds are tarred up and placed in /opt/ykrustc-bin-snapshots on the build master. For now, store at most one archive, but we might change that in the future.

vext01 · 2019-04-23T10:34:35Z

Done.

vext01 · 2019-04-23T11:10:39Z

don't forget to r+ this ;)

ltratt · 2019-04-23T11:11:45Z

Sorry, I missed your "done" message somehow!

bors r+

13: Yk copy good builds r=ltratt a=vext01 Successful builds are tarred up and placed in /opt/ykrustc-bin-snapshots on the build master. For now, store at most one archive, but we might change that in the future. I've tested something very close to this (just without the chmod line in the buildbot script). It *should* work fine 😸 ``` $ ls -al /opt/ykrustc-bin-snapshots/ total 188948 drwxrwxr-x 2 root buildbot_workers 4096 Apr 18 18:23 . drwxr-xr-x 8 root root 4096 Apr 18 11:32 .. -rwxrwxr-x 1 buildbot-worker4 buildbot-worker4 193469353 Apr 18 18:23 ykrustc-stage2-057d397c.tar.bz2 $ cd /tmp $ tar jxf /opt/ykrustc-bin-snapshots/ykrustc-stage2-057d397c.tar.bz2 $ cd ykrustc-stage2/ $ ls bin lib VERSION $ cat VERSION commit 057d397c982463d374f60e16f9b3863dc660e4f2 Author: Edd Barrett <vext01@gmail.com> Date: Thu Apr 18 12:04:20 2019 +0100 Kill [install]. $ ``` If you agree with this, then I suppose the next step is to try using the tarball in: ykjit/yk#1 Co-authored-by: Edd Barrett <vext01@gmail.com>

bors · 2019-04-23T15:16:13Z

Build succeeded

buildbot/buildbot-build-script

sync with rust-lang/rust

This is a combination of 18 commits. Commit softdevteam#2: Additional examples and some small improvements. Commit softdevteam#3: fixed mir-opt non-mir extensions and spanview title elements Corrected a fairly recent assumption in runtest.rs that all MIR dump files end in .mir. (It was appending .mir to the graphviz .dot and spanview .html file names when generating blessed output files. That also left outdated files in the baseline alongside the files with the incorrect names, which I've now removed.) Updated spanview HTML title elements to match their content, replacing a hardcoded and incorrect name that was left in accidentally when originally submitted. Commit softdevteam#4: added more test examples also improved Makefiles with support for non-zero exit status and to force validation of tests unless a specific test overrides it with a specific comment. Commit softdevteam#5: Fixed rare issues after testing on real-world crate Commit softdevteam#6: Addressed PR feedback, and removed temporary -Zexperimental-coverage -Zinstrument-coverage once again supports the latest capabilities of LLVM instrprof coverage instrumentation. Also fixed a bug in spanview. Commit softdevteam#7: Fix closure handling, add tests for closures and inner items And cleaned up other tests for consistency, and to make it more clear where spans start/end by breaking up lines. Commit softdevteam#8: renamed "typical" test results "expected" Now that the `llvm-cov show` tests are improved to normally expect matching actuals, and to allow individual tests to override that expectation. Commit softdevteam#9: test coverage of inline generic struct function Commit softdevteam#10: Addressed review feedback * Removed unnecessary Unreachable filter. * Replaced a match wildcard with remining variants. * Added more comments to help clarify the role of successors() in the CFG traversal Commit softdevteam#11: refactoring based on feedback * refactored `fn coverage_spans()`. * changed the way I expand an empty coverage span to improve performance * fixed a typo that I had accidently left in, in visit.rs Commit softdevteam#12: Optimized use of SourceMap and SourceFile Commit softdevteam#13: Fixed a regression, and synched with upstream Some generated test file names changed due to some new change upstream. Commit softdevteam#14: Stripping out crate disambiguators from demangled names These can vary depending on the test platform. Commit softdevteam#15: Ignore llvm-cov show diff on test with generics, expand IO error message Tests with generics produce llvm-cov show results with demangled names that can include an unstable "crate disambiguator" (hex value). The value changes when run in the Rust CI Windows environment. I added a sed filter to strip them out (in a prior commit), but sed also appears to fail in the same environment. Until I can figure out a workaround, I'm just going to ignore this specific test result. I added a FIXME to follow up later, but it's not that critical. I also saw an error with Windows GNU, but the IO error did not specify a path for the directory or file that triggered the error. I updated the error messages to provide more info for next, time but also noticed some other tests with similar steps did not fail. Looks spurious. Commit softdevteam#16: Modify rust-demangler to strip disambiguators by default Commit softdevteam#17: Remove std::process::exit from coverage tests Due to Issue #77553, programs that call std::process::exit() do not generate coverage results on Windows MSVC. Commit softdevteam#18: fix: test file paths exceeding Windows max path len

buildbot: No need to clean, as we clone afresh.

6e62d50

vext01 assigned ltratt Apr 23, 2019

ltratt reviewed Apr 23, 2019

View reviewed changes

vext01 force-pushed the yk-copy-good-builds branch from 46e17ef to 7c11cfc Compare April 23, 2019 10:30

buildbot: Archive successful builds.

51e7587

Successful builds are tarred up and placed in /opt/ykrustc-bin-snapshots on the build master. For now, store at most one archive, but we might change that in the future.

vext01 force-pushed the yk-copy-good-builds branch from 7c11cfc to 51e7587 Compare April 23, 2019 10:34

bors bot merged commit 51e7587 into master Apr 23, 2019

bors bot deleted the yk-copy-good-builds branch April 23, 2019 15:16

bors bot pushed a commit that referenced this pull request Sep 22, 2019

Merge pull request #13 from rust-lang/master

d5ef9df

sync with rust-lang/rust

Conversation

vext01 commented Apr 23, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vext01 Apr 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vext01 commented Apr 23, 2019

Uh oh!

ltratt commented Apr 23, 2019

Uh oh!

vext01 commented Apr 23, 2019

Uh oh!

ltratt commented Apr 23, 2019

Uh oh!

vext01 commented Apr 23, 2019

Uh oh!

vext01 commented Apr 23, 2019

Uh oh!

ltratt commented Apr 23, 2019

Uh oh!

bors bot commented Apr 23, 2019

Build succeeded

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vext01 Apr 23, 2019 •

edited

Loading