torch.load issue on loading file created by torch.save

[paulzcgu]

🐛 Bug

For some particular data, torch.load() can not properly load a file dumped by torch.save().

Root cause:
The file was mistakenly identified as a zip file by zipfile.is_zipfile() at line 557 of torch\serialization.py, due to the collision of magic number.

To Reproduce

Steps to reproduce the behavior:

1. Use any dumped tensor from https://github.com/nozomifans/reccell/tree/master/issue
2. Load and save data in uint8, then try to load it back

import torch

data = torch.load('E15_s1_08.pt')

torch.save(data.type(torch.uint8),'E15_s1_08_test.pt') 

data = torch.load('E15_s1_08_test.pt')

3. Traceback

Traceback (most recent call last):

  File "<ipython-input-12-7f84586dd470>", line 1, in <module>
    runfile('/Documents/workspace/loadissue/loadtest.py', wdir='/Documents/workspace/loadissue')

  File "\AppData\Local\Continuum\anaconda3\envs\p36-dev\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
    execfile(filename, namespace)

  File "\AppData\Local\Continuum\anaconda3\envs\p36-dev\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/Documents/workspace/loadissue/loadtest.py", line 18, in <module>
    data = torch.load('E15_s1_08_test.pt')

  File "\AppData\Local\Continuum\anaconda3\envs\p36-dev\lib\site-packages\torch\serialization.py", line 386, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)

  File "\AppData\Local\Continuum\anaconda3\envs\p36-dev\lib\site-packages\torch\serialization.py", line 559, in _load
    raise RuntimeError("{} is a zip archive (did you mean to use torch.jit.load()?)".format(f.name))

RuntimeError: E15_s1_08_test.pt is a zip archive (did you mean to use torch.jit.load()?)

Expected behavior

torch.load should load the tensor dumped by torch.save

Environment

• PyTorch Version (e.g., 1.0): 1.2
• OS (e.g., Linux): Windows 10 or CentOS 7.6.1810
• How you installed PyTorch (conda, pip, source): Conda
• Build command you used (if compiling from source):
• Python version: 3.6 or 3.7
• CUDA/cuDNN version:
• GPU models and configuration: CPU only or 4x2080
• Any other relevant information:

Additional context

Current workaround is to save such collision into float32 to avoid the collision. Is that possible to provide an argument to flag this file is not a zip?

[mrshenli]

cc @driazati @zou3519 @VitalyFedyunin

[paulzcgu]

@driazati
The "E15_s1_08.pt" is generated by torch.save as well. The tensor was cast into float32.

# `data` is the tensor
torch.save(data.type(torch.float32),'E15_s1_08.pt') 

[driazati]

Thanks for the concise repro! This does indeed seem to be a bug, there is a fix in #25279.

[ZackPashkin]

I've had the same ussie, until I put the model on CPU then saved . torch.load works in this case
It seems the issue was in tensor.cuda and tensor