[Performance Regression] Scala/Java SSD inference

[stu1130]

Description

After #16602 merge, java/scala SSD GPU inference latency increased by 20x (from 70ms to 1400ms)

Steps to reproduce

(Paste the commands you ran that produced the error.)

1. lanuch an ec2 instance (Deep Learning Base AMI (Ubuntu 16.04) Version 20.0) with at least one gpu 
2. ssh into the instance
3. git clone --recursive https://github.com/apache/incubator-mxnet.git
4. git clone https://github.com/andrewfayres/deeplearning-benchmark.git

# dd4eaf5 is the first commit that introduces the regression
# you can compare this &  bde443e which is the last commit without regression
5. cd incubator-mxnet && git reset --hard dd4eaf5

# build mxnet from source
6. make -j$(nproc) USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_OPENMP=1 USE_MKLDNN=1 USE_OPENCV=1
7. cd scala-package
8. sudo apt install maven

# this will build & deploy all the artifacts into local repository
9. mvn deploy -Drepo_url=file://$HOME/.m2/repository
10. cd ~/deeplearning-benchmark/scala-mxnet/java-bm
11. make change to pom.xml in current directory

replace this section
<repositories>
    <repository>
      <id>Apache Snapshot</id>
      <url>https://repository.apache.org/content/groups/snapshots</url>
    </repository>
  </repositories>
with
<repositories>
     <repository>
       <id>Apache Snapshot</id>
       <url>file:/home/ubuntu/.m2/repository</url>
     </repository>
   </repositories>

It then will use the local repo instead of our public maven repo

12. make change to bin/run_ssd.sh

replace MXNET_VERSION="[1.5.0-SNAPSHOT,)"
with MXNET_VERSION="[1.6.0-SNAPSHOT,)"

# run the benchmark script in current directory (......./java-bm)
13. bash bin/get_resnet50_ssd_data.sh && export SCALA_TEST_ON_GPU=1 && bash bin/run_ssd.sh gpu /tmp/resnet50_ssd/resnet50_ssd_model /tmp/resnet50_ssd/images/dog.jpg 4 500

@PatricZhao @ZhennanQin
Thanks in advance & Happy Halloween!

[pengzhao-intel]

Thanks to reporting the issue ~~~
@ZhennanQin will look into the issue.