Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

Fix nightly CD for python docker image releases#19772

Merged
mseth10 merged 29 commits intoapache:masterfrom
mseth10:update_cd_python
Feb 27, 2021
Merged

Fix nightly CD for python docker image releases#19772
mseth10 merged 29 commits intoapache:masterfrom
mseth10:update_cd_python

Conversation

@mseth10
Copy link
Copy Markdown
Contributor

@mseth10 mseth10 commented Jan 21, 2021 •
edited
Loading

Description

This PR makes multiple changes to fix the nightly CD pipeline for python docker image releases.

  • Change push target from dockerhub to ecr public repository
  • Update all base images for docker build from ubuntu16 to ubuntu18
  • Use deadsnakes python package
  • Remove unused mxnet_lib/dynamic pipeline

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Jan 21, 2021

Hey @mseth10 , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [edge, sanity, centos-cpu, centos-gpu, windows-cpu, unix-cpu, windows-gpu, unix-gpu, website, clang, miscellaneous]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@lanking520 lanking520 added the pr-awaiting-testing PR is reviewed and waiting CI build and test label Jan 21, 2021
@lanking520 lanking520 added pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress labels Jan 22, 2021
leezu
leezu reviewed Jan 22, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@leezu leezu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As part of this work, can we reconsider the publication strategy? Currently nightly builds are pushed to dockerhub, which is not compatible with ASF policy. Instead, can you configure the job to only push on approved release tags?

@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-work-in-progress PR is still work in progress labels Jan 22, 2021
@mseth10 mseth10 changed the title Update cd python Fix CD pipeline for nightly docker images Jan 22, 2021
@lanking520 lanking520 added pr-awaiting-review PR is waiting for code review and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels Jan 23, 2021
josephevans
josephevans reviewed Jan 26, 2021
View reviewed changes
@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Feb 24, 2021
@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Feb 24, 2021
@lanking520 lanking520 removed the pr-work-in-progress PR is still work in progress label Feb 24, 2021
@mseth10 mseth10 mentioned this pull request Feb 26, 2021
Merged
@mseth10
Copy link
Copy Markdown
Contributor Author

mseth10 commented Feb 26, 2021

As part of this work, can we reconsider the publication strategy? Currently nightly builds are pushed to dockerhub, which is not compatible with ASF policy. Instead, can you configure the job to only push on approved release tags?

As part of this PR, the docker push target has been moved to a public ECR repository.

sandeep-krishnamurthy
sandeep-krishnamurthy reviewed Feb 26, 2021
View reviewed changes
cd/utils/mxnet_base_image.sh Outdated
;;
cu112*)
echo "nvidia/cuda:11.2-cudnn8-runtime-ubuntu16.04"
echo "nvidia/cuda:11.2.1-cudnn8-runtime-ubuntu16.04"
Copy link
Copy Markdown

@sandeep-krishnamurthy sandeep-krishnamurthy Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ubuntu 16.04 is EOL by April, 2021 (https://ubuntu.com/about/release-cycle), Can we please move to ubuntu 18.04 or 20.04?

Copy link
Copy Markdown
Contributor Author

@mseth10 mseth10 Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to ubuntu 18.04 for all docker base images

access2rohit
access2rohit reviewed Feb 26, 2021
View reviewed changes
access2rohit
access2rohit approved these changes Feb 26, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@access2rohit access2rohit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@leezu leezu Awaiting requested review from leezu

4 more reviewers

@sandeep-krishnamurthy sandeep-krishnamurthy sandeep-krishnamurthy left review comments

@josephevans josephevans josephevans left review comments

@Zha0q1 Zha0q1 Zha0q1 approved these changes

@access2rohit access2rohit access2rohit approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

pr-awaiting-merge Review and CI is complete. Ready to Merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@mseth10 @mxnet-bot @leezu @sandeep-krishnamurthy @josephevans @Zha0q1 @access2rohit @lanking520