avoid skipping validation stage when data read iterator exists

[heaseny]

Description

avoid skipping validation stage when data read iterator exists.
Test commands:
cd example/image-classification
python train_cifar10.py --network='resnet' --num-layers=50 --image-shape='3,28,28' --num-epochs=1

Abnormal test log in current master branch:
INFO:root:Epoch[0] Train-accuracy=0.386069
INFO:root:Epoch[0] Train-top_k_accuracy_5=0.875060
INFO:root:Epoch[0] Time cost=38.553

Normal test log which is expected:
INFO:root:Epoch[0] Batch [360-380] Speed: 1302.17 samples/sec accuracy=0.497266 top_k_accuracy_5=0.935156
INFO:root:Epoch[0] Train-accuracy=0.386069
INFO:root:Epoch[0] Train-top_k_accuracy_5=0.875060
INFO:root:Epoch[0] Time cost=39.096
INFO:root:Epoch[0] Validation-accuracy=0.448378
INFO:root:Epoch[0] Validation-top_k_accuracy_5=0.918018

Checklist

Essentials

• Changes are completed
• All changes have test coverage.

Changes

• Feature: avoid skipping validation stage when validation data read iterator exists

[mxnet-bot]

Hey @heaseny , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [sanity, miscellaneous, unix-cpu, website, edge, centos-gpu, clang, centos-cpu, windows-cpu, windows-gpu, unix-gpu]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[heaseny]

@mxnet-bot run ci [unix-cpu]

[mxnet-bot]

Jenkins CI successfully triggered : [unix-cpu]

[heaseny]

@mxnet-bot run ci [centos-cpu,unix-cpu,unix-gpu,windows-gpu]
failed due to test_numpy_op.py:test_np_mixed_precision_binary_funcs
Related issue : #16848
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-18289/1/pipeline/
http://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/job/unix-cpu/job/PR-18289/2/display/redirect
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18289/1/pipeline/
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-18289/1/pipeline/

[mxnet-bot]

Jenkins CI successfully triggered : [unix-gpu, windows-gpu, centos-cpu, unix-cpu]

[heaseny]

@szha Can you help to review this pull request, and merge it if there is no problem? Thanks a lot.

[szha]

apologies for the spam.

@heaseny we deprecated module in the master branch and we will continue to support it in v1.x. it would be great if you could rebase your change to the v1.x branch.

[heaseny]

@szha as work changing and I am not focusing on the mxnet test and have no environment, if the change is necessary in v1.x, can you help to rebase the change, or should I close this PR?

[szha]

picked up the change for v1.x in the above PR.