Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

[operator] Add Mish Activation Function#20320

Merged
szha merged 11 commits intoapache:masterfrom
Adnios:mish
Jun 8, 2021
Merged

[operator] Add Mish Activation Function#20320
szha merged 11 commits intoapache:masterfrom
Adnios:mish

Conversation

@Adnios
Copy link
Contributor

@Adnios Adnios commented May 28, 2021 •
edited
Loading

Description

Add Mish Activation Function.
Related issus: #16841
The pr(#17696) seem to be dead.

Checklist

Essentials

  • PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented

Changes

  • Add native mish implementation
  • Integrate oneDNN implementation

Comments

  • This is mostly a copycat of log_sigmoid activation function. I referenced a lot of code during the implementation.

@Adnios Adnios requested a review from szha as a code owner May 28, 2021 04:58
@mxnet-bot
Copy link

mxnet-bot commented May 28, 2021

Hey @Adnios , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [centos-gpu, unix-cpu, windows-gpu, miscellaneous, clang, website, windows-cpu, sanity, edge, centos-cpu, unix-gpu]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@Adnios Adnios marked this pull request as draft May 28, 2021 04:59
@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels May 28, 2021
Adnios added 2 commits May 28, 2021 20:50
@Adnios
fix test_mish
2ff4d57 
Signed-off-by: Adnios <[email protected]>
@Adnios
fix test_mish_gpu
3e9648c 
Signed-off-by: Adnios <[email protected]>
@Adnios Adnios force-pushed the mish branch 3 times, most recently from 3a11899 to 47a52f3 Compare May 29, 2021 07:12
@Adnios Adnios marked this pull request as ready for review May 29, 2021 10:42
@mseth10 mseth10 added pr-awaiting-review PR is waiting for code review and removed pr-work-in-progress PR is still work in progress labels May 29, 2021
szha
szha reviewed Jun 1, 2021
View reviewed changes
Copy link
Member

@szha szha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! Going forward in 2.0, as we will mainly use np/npx instead of sym, could you also include a test for it with npx.activation?

@Adnios
Copy link
Contributor Author

Adnios commented Jun 2, 2021

Thanks for the contribution! Going forward in 2.0, as we will mainly use np/npx instead of sym, could you also include a test for it with npx.activation?

Sure. Thanks for advice.

@Adnios Adnios closed this Jun 2, 2021
@Adnios Adnios reopened this Jun 2, 2021
@mseth10 mseth10 added pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-awaiting-review PR is waiting for code review pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Jun 2, 2021
@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Jun 2, 2021
@Adnios Adnios marked this pull request as draft June 2, 2021 16:05
@Adnios
backward error test
8ea2560 
Signed-off-by: Adnios <[email protected]>
Adnios added 3 commits June 4, 2021 00:48
@Adnios
mv test_mish
aa40b43 
Signed-off-by: Adnios <[email protected]>
@Adnios
fix backward bug
5bcd76a 
Signed-off-by: Adnios <[email protected]>
@Adnios
fix backward bug in cudnn
025f8de 
Signed-off-by: Adnios <[email protected]>
@Adnios Adnios marked this pull request as ready for review June 5, 2021 04:56
@Adnios
Copy link
Contributor Author

Adnios commented Jun 5, 2021

@mxnet-bot run ci [centos-gpu, unix-gpu]

@mxnet-bot
Copy link

mxnet-bot commented Jun 5, 2021

Jenkins CI successfully triggered : [centos-gpu, unix-gpu]

@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-awaiting-review PR is waiting for code review and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Jun 5, 2021
@Adnios Adnios requested a review from szha June 7, 2021 00:45
@Adnios
Copy link
Contributor Author

Adnios commented Jun 8, 2021

@szha Please help review

@szha szha merged commit 7740cca into apache:master Jun 8, 2021
@szha
Copy link
Member

szha commented Jun 8, 2021

@Adnios merged. Thank you for the contribution!

ptrendx
ptrendx reviewed Jun 9, 2021
View reviewed changes
src/common/cuda/rtc/forward_functions-inl.h
template <typename DType>
__device__ inline DType mish(const DType val) {
if (type_util::has_double_or_integral<DType>::value) {
return val * ::tanh(::log(1 + ::exp(val)));
Copy link
Member

@ptrendx ptrendx Jun 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing that could be improved here (I did not notice this PR earlier, sorry for a late feedback) is the numerical stability of the softrelu part - see the implementation of the softrelu (it switches to softrelu(x) = x for large values of x to avoid overflow). @Adnios could you open another PR changing e.g. this function to

return val * op::tanh(op::softrelu(val));

(the double vs float is handled in op::tanh and op::softrelu anyway so this one will also be simpler as a result) and similarly backward?

Copy link

@digantamisra98 digantamisra98 Jun 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, agreed, usually Softplus has an upper bound of 20.

Copy link
Contributor Author

@Adnios Adnios Jun 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Thanks for your advice.

@Adnios Adnios mentioned this pull request Jun 11, 2021
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@szha szha Awaiting requested review from szha szha is a code owner

2 more reviewers

@ptrendx ptrendx ptrendx left review comments

@digantamisra98 digantamisra98 digantamisra98 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

pr-awaiting-review PR is waiting for code review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Adnios @mxnet-bot @szha @ptrendx @digantamisra98 @mseth10