Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

[FEATURE] Add feature of retain_grad #20500

Merged
barry-jin merged 36 commits intoapache:masterfrom
KexinFeng:rg_branch
Sep 23, 2021
Merged

[FEATURE] Add feature of retain_grad #20500
barry-jin merged 36 commits intoapache:masterfrom
KexinFeng:rg_branch

Conversation

@KexinFeng
Copy link
Copy Markdown
Contributor

@KexinFeng KexinFeng commented Aug 9, 2021 •
edited
Loading

Description

The PR adds the support for fetching the gradients of intermediate variables in a computation graph.

The motivation of this feature comes from this issue.

Checklist

Essentials

  • PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented

Changes

  • Imperative::MarkVariables in src/imperative/imperative.cc to include the case of marking nonleaf nodes
  • Imperative::Backward in src/imperative/imperative.cc to retrieve nonleaf nodes and link node->info.out_grads[] to buffered arrays.
  • Graph BuildGradientGraph to map forward node to the corresponding gradient node entry.

Comments

  • Interesting edge cases to note here: 1. the gradient-queried node is the output node, z.retain_grad(). 2. when it is used combined with detaching computation.
  • Note that u.retain_grad() is called outside the scope of ag.record(). Otherwise, u.retain_grad() would be recorded as operation and trigger building of computational node. This is conceptually not desired.

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Aug 9, 2021

Hey @KexinFeng , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [centos-gpu, centos-cpu, windows-cpu, unix-gpu, website, sanity, edge, miscellaneous, unix-cpu, windows-gpu, clang]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@mseth10 mseth10 added the pr-awaiting-testing PR is reviewed and waiting CI build and test label Aug 9, 2021
@KexinFeng KexinFeng changed the title Add feature of retain_grad Add feature of retain_grad [FEATURE] [issue](https://github.com/apache/incubator-mxnet/issues/11865) Aug 9, 2021
@KexinFeng KexinFeng changed the title Add feature of retain_grad [FEATURE] [issue](https://github.com/apache/incubator-mxnet/issues/11865) Add feature of retain_grad [FEATURE] Aug 9, 2021
@KexinFeng KexinFeng changed the title Add feature of retain_grad [FEATURE] [FEATURE] Add feature of retain_grad Aug 9, 2021
@mseth10 mseth10 added pr-work-in-progress PR is still work in progress and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels Aug 9, 2021
@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Aug 11, 2021
@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-work-in-progress PR is still work in progress labels Aug 11, 2021
@mseth10 mseth10 added pr-work-in-progress PR is still work in progress and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels Aug 11, 2021
@mseth10 mseth10 removed the pr-work-in-progress PR is still work in progress label Aug 12, 2021
@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-awaiting-review PR is waiting for code review labels Aug 17, 2021
@KexinFeng
Copy link
Copy Markdown
Contributor Author

KexinFeng commented Aug 18, 2021

@mxnet-bot run ci [all]

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Aug 18, 2021

Jenkins CI successfully triggered : [clang, windows-cpu, miscellaneous, centos-cpu, unix-cpu, edge, website, windows-gpu, sanity, centos-gpu, unix-gpu]

@KexinFeng
Copy link
Copy Markdown
Contributor Author

KexinFeng commented Aug 18, 2021

@mxnet-bot run ci [unix-cpu]

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Aug 18, 2021

Jenkins CI successfully triggered : [unix-cpu]

@KexinFeng
Copy link
Copy Markdown
Contributor Author

KexinFeng commented Aug 18, 2021

@mxnet-bot run ci [macosx-x86_64, linkcheck]

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Aug 18, 2021

None of the jobs entered are supported.
Jobs entered by user: [macosx-x86_64, linkcheck]
CI supported Jobs: [centos-cpu, edge, windows-cpu, sanity, clang, centos-gpu, unix-gpu, website, miscellaneous, unix-cpu, windows-gpu]

@KexinFeng
Copy link
Copy Markdown
Contributor Author

KexinFeng commented Aug 18, 2021

@mxnet-bot run ci [centos-gpu, unix-gpu]

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Aug 18, 2021

Jenkins CI successfully triggered : [centos-gpu, unix-gpu]

This was referenced Aug 27, 2021
Closed
Closed
@KexinFeng
Copy link
Copy Markdown
Contributor Author

KexinFeng commented Aug 27, 2021

@mxnet-bot run ci [website]

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Aug 27, 2021

Jenkins CI successfully triggered : [website]

@KexinFeng
Copy link
Copy Markdown
Contributor Author

KexinFeng commented Aug 28, 2021

@mxnet-bot run ci [unix-cpu]

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Aug 28, 2021

Jenkins CI successfully triggered : [unix-cpu]

@KexinFeng
Copy link
Copy Markdown
Contributor Author

KexinFeng commented Aug 28, 2021

@mxnet-bot run ci [unix-cpu]

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Aug 28, 2021

Jenkins CI successfully triggered : [unix-cpu]

@barry-jin
Copy link
Copy Markdown
Contributor

barry-jin commented Sep 17, 2021

@mxnet-bot run ci [unix-gpu, centos-gpu]

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Sep 17, 2021

Jenkins CI successfully triggered : [centos-gpu, unix-gpu]

@barry-jin
Copy link
Copy Markdown
Contributor

barry-jin commented Sep 20, 2021

@mxnet-bot run ci [centos-gpu]

@mxnet-bot
Copy link
Copy Markdown

mxnet-bot commented Sep 20, 2021

Jenkins CI successfully triggered : [centos-gpu]

This was referenced Jul 6, 2022
Closed
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@eric-haibin-lin eric-haibin-lin Awaiting requested review from eric-haibin-lin eric-haibin-lin is a code owner

@szha szha Awaiting requested review from szha szha is a code owner

1 more reviewer

@barry-jin barry-jin barry-jin left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

pr-awaiting-review PR is waiting for code review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@KexinFeng @mxnet-bot @barry-jin @mseth10