Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

[FEATURE] Fusing Leaky Relu operator in Fully Connected#19971

Merged
szha merged 1 commit intoapache:v1.xfrom
agrabows:v1.x_fc_fuse_gelu
Apr 22, 2021
Merged

[FEATURE] Fusing Leaky Relu operator in Fully Connected#19971
szha merged 1 commit intoapache:v1.xfrom
agrabows:v1.x_fc_fuse_gelu

Conversation

@agrabows
Copy link
Contributor

@agrabows agrabows commented Mar 1, 2021

Description

This change will enable fusing Leaky Relu operator and Fully Connected into single operator supported by oneDNN backend for both int8 and fp32 data types.

Checklist

Essentials

  • PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage

Changes

  • Fusing FC with leaky relu for oneDNN backend

@mxnet-bot
Copy link

mxnet-bot commented Mar 1, 2021

Hey @agrabows , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [clang, sanity, centos-cpu, centos-gpu, miscellaneous, edge, website, windows-cpu, windows-gpu, unix-gpu, unix-cpu]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@lanking520 lanking520 added the pr-awaiting-testing PR is reviewed and waiting CI build and test label Mar 1, 2021
@agrabows agrabows changed the title Fusing Leaky Relu operator in Fully Connected [FEATURE] Fusing Leaky Relu operator in Fully Connected Mar 1, 2021
@lanking520 lanking520 added pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress labels Mar 1, 2021
@agrabows agrabows force-pushed the v1.x_fc_fuse_gelu branch from 4e8806d to dbd09ba Compare April 1, 2021 08:29
@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Apr 1, 2021
@agrabows
Copy link
Contributor Author

agrabows commented Apr 15, 2021

@mxnet-bot run ci [unix-gpu]

@mxnet-bot
Copy link

mxnet-bot commented Apr 15, 2021

Jenkins CI successfully triggered : [unix-gpu]

@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-awaiting-review PR is waiting for code review and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Apr 15, 2021
szha
szha reviewed Apr 19, 2021
View reviewed changes
src/operator/subgraph/mkldnn/mkldnn_fc_property.h
Comment on lines +107 to +113
if (param.act_type == leakyrelu::kLeakyReLU ||
param.act_type == leakyrelu::kELU ||
param.act_type == leakyrelu::kGELU) {
matched_list_.push_back(&new_node);
status_ = kSuccess;
return true;
}
Copy link
Member

@szha szha Apr 19, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full set of supported activation types are {'elu', 'gelu', 'leaky', 'prelu', 'rrelu', 'selu'}. What about the rest of the activation types? Will they be supported by onednn in the future?

Copy link
Contributor Author

@agrabows agrabows Apr 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those were activation types that can be easily fused with FC in this PR. We plan to enable other activation fuses in the next PR as they need different approach.

@szha szha merged commit 54aae29 into apache:v1.x Apr 22, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

2 more reviewers

@szha szha szha left review comments

@bgawrych bgawrych bgawrych approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

pr-awaiting-review PR is waiting for code review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@agrabows @mxnet-bot @szha @bgawrych @lanking520 @agrabow