How can I run the inference on the HunyuanImage-3.0 model?

I follow the instructions on https://github.com/Tencent-Hunyuan/HunyuanImage-3.0:

conda create -y -n hunyuan312 python=3.12
conda activate hunyuan312

# 1. First install PyTorch (CUDA 12.8 Version)
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128

# 2. Then install tencentcloud-sdk
pip install -i https://mirrors.tencent.com/pypi/simple/ --upgrade tencentcloud-sdk-python

git clone https://github.com/Tencent-Hunyuan/HunyuanImage-3.0.git
cd HunyuanImage-3.0/

# 3. Then install other dependencies
pip install -r requirements.txt

# Download from HuggingFace and rename the directory.
# Notice that the directory name should not contain dots, which may cause issues when loading using Transformers.
hf download tencent/HunyuanImage-3.0 --local-dir ./HunyuanImage-3

then I try running their example code:

from transformers import AutoModelForCausalLM

# Load the model
model_id = "./HunyuanImage-3"
# Currently we can not load the model using HF model_id `tencent/HunyuanImage-3.0` directly 
# due to the dot in the name.

kwargs = dict(
    attn_implementation="sdpa",     # Use "flash_attention_2" if FlashAttention is installed
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
    moe_impl="eager",   # Use "flashinfer" if FlashInfer is installed
)

model = AutoModelForCausalLM.from_pretrained(model_id, **kwargs)
model.load_tokenizer(model_id)

# generate the image
prompt = "A brown and white dog is running on the grass"
image = model.generate_image(prompt=prompt, stream=True)
image.save("image.png")

But I get the error OSError: No such device (os error 19):

(hunyuan312) franck@server:/fun$ python generate_image_hyun.py 
You are using a model of type hunyuan_image_3_moe to instantiate a model of type Hunyuan. This is not supported for all configurations of models and can yield errors.
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards:   0%|                                          | 0/32 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/fun/generate_image_hyun.py", line 21, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_id, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/franck/anaconda3/envs/hunyuan312/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 597, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/franck/anaconda3/envs/hunyuan312/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/franck/anaconda3/envs/hunyuan312/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5048, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/franck/anaconda3/envs/hunyuan312/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5468, in _load_pretrained_model
    _error_msgs, disk_offload_index = load_shard_file(args)
                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/franck/anaconda3/envs/hunyuan312/lib/python3.12/site-packages/transformers/modeling_utils.py", line 831, in load_shard_file
    state_dict = load_state_dict(
                 ^^^^^^^^^^^^^^^^
  File "/home/franck/anaconda3/envs/hunyuan312/lib/python3.12/site-packages/transformers/modeling_utils.py", line 484, in load_state_dict
    with safe_open(checkpoint_file, framework="pt") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: No such device (os error 19)

How can I fix it?

Same issue if I try running:

python3 run_image_gen.py \
  --model-id ./HunyuanImage-3/ \
  --verbose 1 \
  --prompt "A brown and white dog is running on the grass."

posted 3 months ago

CC BY-SA 4.0

3mo ago

Franck Dernoncourt‭

143 reputation 24 10 14 33