[v1.7.x] update jetson dockerfile to support CUDA 10.0#18339

waytrue17 · 2020-05-16T00:40:37Z

Description

Upgrade cuda from 9.0 to 10.0 for edge pipeline to address [v1.x] CI pipeline for NVIDIA Jetson fails with "unsupported GNU version" #18298 on 1.7.x
Disable TVM_OP on GPU builds for edge pipeline to address TVMOp doesn't work well with GPU builds #17840 on 1.7.x

Essentials

Changes are complete (i.e. I finished coding on this PR)

mxnet-bot · 2020-05-16T00:40:41Z

Hey @waytrue17 , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

To trigger all jobs: @mxnet-bot run ci [all]
To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [centos-gpu, miscellaneous, edge, unix-gpu, sanity, website, unix-cpu, centos-cpu, windows-gpu, clang, windows-cpu]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

waytrue17 · 2020-05-16T00:49:59Z

@ciyongch

ciyongch · 2020-05-16T06:39:38Z

Hi @waytrue17 , #18311 is a revert PR, do you mean this PR is to fix the failed CI of #18311?

waytrue17 · 2020-05-16T18:03:50Z

Hi @waytrue17 , #18311 is a revert PR, do you mean this PR is to fix the failed CI of #18311?

Hi ciyong, yes I was trying to do so, seems it doesn't work..

ciyongch · 2020-05-18T12:06:02Z

Hi @waytrue17, are you still working on this?
Seems the new PR got stuck due to file toolchains/aarch64-linux-gnu-toolchain.cmake is not existed, probably it's a new file introduced to the build system?

 Step 8/17 : COPY toolchains/aarch64-linux-gnu-toolchain.cmake /usr
 COPY failed: stat /var/lib/docker/tmp/docker-builder284184594/toolchains/aarch64-linux-gnu-toolchain.cmake: no such file or directory

leezu · 2020-05-18T17:11:04Z

You need to include ci/docker/toolchains/aarch64-linux-gnu-toolchain.cmake the file in your PR

leezu · 2020-05-18T21:56:30Z

@mxnet-bot run ci [sanity]

mxnet-bot · 2020-05-18T21:56:35Z

Jenkins CI successfully triggered : [sanity]

leezu · 2020-05-18T23:17:06Z

@mxnet-bot run ci [all]

mxnet-bot · 2020-05-18T23:17:17Z

Jenkins CI successfully triggered : [windows-cpu, unix-gpu, centos-gpu, sanity, unix-cpu, miscellaneous, windows-gpu, centos-cpu, edge, website, clang]

waytrue17 · 2020-05-20T21:33:20Z

@mxnet-bot run ci [edge, unix-gpu, website]

mxnet-bot · 2020-05-20T21:33:29Z

Jenkins CI successfully triggered : [website, unix-gpu, edge]

ciyongch · 2020-05-21T05:40:06Z

Hi @waytrue17 , please help to re-trigger the job of unix-gpu and website, they should pass .
While for edge job, seems CUDAToolkit_INCLUDE_DIR CUDA_CUDART can't be found in the current build env. Could NOT find CUDAToolkit (missing: CUDAToolkit_INCLUDE_DIR CUDA_CUDART), can you help to take a look at this?

waytrue17 · 2020-05-21T06:17:26Z

@mxnet-bot run ci [unix-gpu, website]

mxnet-bot · 2020-05-21T06:17:31Z

Jenkins CI successfully triggered : [unix-gpu, website]

waytrue17 · 2020-05-21T06:18:57Z

Hi @waytrue17 , please help to re-trigger the job of unix-gpu and website, they should pass .
While for edge job, seems CUDAToolkit_INCLUDE_DIR CUDA_CUDART can't be found in the current build env. Could NOT find CUDAToolkit (missing: CUDAToolkit_INCLUDE_DIR CUDA_CUDART), can you help to take a look at this?

CI jobs triggered. Sure I'll work on fix the edge

…to waytrue17-v1.7.x

…ator-mxnet into waytrue17-v1.7.x

leezu · 2020-05-22T22:00:44Z

ci/docker/install/ubuntu_julia.sh

    # The julia version in Ubuntu repo is too old
    # We download the tarball from the official link:
    #   https://julialang.org/downloads/
-    wget -qO $JLBINARY https://s3.amazonaws.com/julialang2/bin/linux/x64/0.7/julia-0.7.0-linux-x86_64.tar.gz


The problem is not about the S3 path but that you hardcode the 0.7 version instead of using $1 and $2. In any case, hopefully https://julialang-s3.julialang.org/bin/linux/x64/$1/julia-$2-linux-x86_64.tar.gz will remain stable for the future

Thanks for pointing it out. I'll replace it back to the S3 path.

ciyongch · 2020-05-24T10:10:01Z

@leezu @ptrendx @TaoLv @pengzhao-intel ,please help to take a review of the code changes.

ciyongch · 2020-05-26T05:30:42Z

Hi @waytrue17 , is there still any failure when enabling TVM build? Not sure if this is the best practice to fix this CI failure. Ping @leezu @ptrendx @TaoLv .

waytrue17 · 2020-05-26T07:26:24Z

Hi @waytrue17 , is there still any failure when enabling TVM build? Not sure if this is the best practice to fix this CI failure. Ping @leezu @ptrendx @TaoLv .

Hi @ciyongch, I think there are still some issues in TVM build #17840. I saw that it is disabled on master #18204, so I did the same for 1.7.x

ciyongch · 2020-05-27T01:43:30Z

Thanks @waytrue17 to help fixing this failure :)

* update dockerfile for jetson * add toolchain files * update build_jetson function * update ubuntu_julia.sh * update FindCUDAToolkit.cmake * Update centos7_python.sh * revert changes on ubuntu_julia.sh * disable TVM for gpu build * Disable TVM_OP on GPU builds Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl>

…8560) * fix centos 7 url to unblock centos-cpu & gpu pipeline * [v1.7.x] update jetson dockerfile to support CUDA 10.0 (#18339) * update dockerfile for jetson * add toolchain files * update build_jetson function * update ubuntu_julia.sh * update FindCUDAToolkit.cmake * Update centos7_python.sh * revert changes on ubuntu_julia.sh * disable TVM for gpu build * Disable TVM_OP on GPU builds Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl> * skip quantized conv flaky case (#16866) * Fix quantized concat when inputs are mixed int8 and uint8 Change-Id: I4da04bf4502425134a466823fb5f73da2d7a419b * skip flaky test * trigger ci Co-authored-by: waytrue17 <52505574+waytrue17@users.noreply.github.com> Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl> Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com>

* update dockerfile for jetson * add toolchain files * update build_jetson function * update ubuntu_julia.sh * update FindCUDAToolkit.cmake * Update centos7_python.sh * revert changes on ubuntu_julia.sh * disable TVM for gpu build * Disable TVM_OP on GPU builds Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl>

* add the missing build_ubuntu_gpu_cuda101_cudnn7_mkldnn_cpp_test in runtime_functions.sh * Revert "add the missing build_ubuntu_gpu_cuda101_cudnn7_mkldnn_cpp_test in runtime_functions.sh" This reverts commit de173b0. * Revert "[CI][1.6.x] fix centos 7 url to unblock centos-cpu & gpu pipeline (#18560)" This reverts commit d271348. * fix centos 7 url to unblock centos-cpu & gpu pipeline * skip quantized conv flaky case (#16866) * Fix quantized concat when inputs are mixed int8 and uint8 Change-Id: I4da04bf4502425134a466823fb5f73da2d7a419b * skip flaky test * trigger ci * Trigger empty commit * [v1.7.x] update jetson dockerfile to support CUDA 10.0 (#18339) * update dockerfile for jetson * add toolchain files * update build_jetson function * update ubuntu_julia.sh * update FindCUDAToolkit.cmake * Update centos7_python.sh * revert changes on ubuntu_julia.sh * disable TVM for gpu build * Disable TVM_OP on GPU builds Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl> * add setuptools to ci/docker/install/requirements * add missing build_ubuntu_gpu_cuda101_cudnn7_mkldnn_cpp_test * add setuptool to docker & cpp-test build syntax error * remove erroneously added cpp tests in 1.6.x * py3 to p2 Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com> Co-authored-by: waytrue17 <52505574+waytrue17@users.noreply.github.com> Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl>

update dockerfile for jetson

7f54401

waytrue17 requested review from aaronmarkham and marcoabreu as code owners May 16, 2020 00:40

add toolchain files

1c27f4e

Wei Chu added 2 commits May 18, 2020 17:00

update build_jetson function

f5f0f6e

update ubuntu_julia.sh

cfae515

Wei Chu added 2 commits May 21, 2020 11:48

Merge branch 'v1.7.x' of https://github.com/apache/incubator-mxnet in…

2490e0d

…to waytrue17-v1.7.x

update FindCUDAToolkit.cmake

24f5f26

waytrue17 requested a review from szha as a code owner May 21, 2020 18:58

leezu and others added 3 commits May 22, 2020 10:52

Update centos7_python.sh

9e67ef4

revert changes on ubuntu_julia.sh

3432a70

Merge branch 'waytrue17-v1.7.x' of https://github.com/waytrue17/incub…

93afa25

…ator-mxnet into waytrue17-v1.7.x

leezu reviewed May 22, 2020

View reviewed changes

leezu mentioned this pull request May 22, 2020

[v1.x] Backport edge pipeline #18375

Merged

13 tasks

waytrue17 changed the title ~~update dockerfile for jetson~~ [v1.7.x] update jetson dockerfile to support CUDA 10.0 May 23, 2020

Wei Chu added 2 commits May 22, 2020 19:09

disable TVM for gpu build

5c2ef7a

Disable TVM_OP on GPU builds

d32dccc

ciyongch mentioned this pull request May 25, 2020

Back port the PR #17672:boolean indexing to branch v1.x #18351

Merged

ChaiBapchya mentioned this pull request May 26, 2020

Revert PR 17767 [1.7 branch] (#18283) #18311

Merged

leezu approved these changes May 26, 2020

View reviewed changes

leezu merged commit 1eefe66 into apache:v1.7.x May 26, 2020

ciyongch mentioned this pull request May 27, 2020

Back port the PR #17672:boolean indexing to branch v1.7.x #18352

Merged

This was referenced Jun 15, 2020

[CI][1.6.x] fix centos 7 url to unblock centos-cpu & gpu pipeline #18560

Merged

[CI][v1.6.x] Fix failing CI pipelines #18597

Merged

waytrue17 deleted the waytrue17-v1.7.x branch October 7, 2020 00:11

Conversation

waytrue17 commented May 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Essentials

Uh oh!

mxnet-bot commented May 16, 2020

Uh oh!

waytrue17 commented May 16, 2020

Uh oh!

ciyongch commented May 16, 2020

Uh oh!

waytrue17 commented May 16, 2020

Uh oh!

ciyongch commented May 18, 2020

Uh oh!

leezu commented May 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leezu commented May 18, 2020

Uh oh!

mxnet-bot commented May 18, 2020

Uh oh!

leezu commented May 18, 2020

Uh oh!

mxnet-bot commented May 18, 2020

Uh oh!

waytrue17 commented May 20, 2020

Uh oh!

mxnet-bot commented May 20, 2020

Uh oh!

ciyongch commented May 21, 2020

Uh oh!

waytrue17 commented May 21, 2020

Uh oh!

mxnet-bot commented May 21, 2020

Uh oh!

waytrue17 commented May 21, 2020

Uh oh!

leezu May 22, 2020

Choose a reason for hiding this comment

Uh oh!

waytrue17 May 22, 2020

Choose a reason for hiding this comment

Uh oh!

ciyongch commented May 24, 2020

Uh oh!

ciyongch commented May 26, 2020

Uh oh!

waytrue17 commented May 26, 2020

Uh oh!

ciyongch commented May 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

waytrue17 commented May 16, 2020 •

edited

Loading

leezu commented May 18, 2020 •

edited

Loading