StyleTex: Style Image-Guided Texture Generation for 3D Models

Zhiyu Xie*, Yuqing Zhang*, Xiangjun Tang, Yiqian Wu, Dehan Chen, Gongsheng Li, Xiaogang Jin†

*The first two authors contribute equally. †Corresponding authors.

We introduce StyleTex, an innovative diffusion-model-based framework for creating stylized textures for 3D models. The resulting textures generated by StyleTex retain the style of the reference image, while also aligning with the text prompts and intrinsic details of the given 3D mesh.

1. We test our model on Nvidia 4090 GPU with 11.8 CUDA and 2.0.0 pytorch. You can configure the environment by following the steps below.

   git clone https://github.com/XZYW7/StyleTex.git
   cd StyleTex

   Then, install packages needed

   conda create -n styletex
   conda activate styletex
   conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
   
   pip install -r requirements.txt

2. Make sure the essential models like SD1.5, ControlNet, and IP-Adapter are prepared. And you can link the model cache path by modifying the parameter <your_path_to_SD_models> of cache_dir in configs\styletex.yaml.

1. Run:

   sh cmd/run_examples.py

   or Execute:

   python launch.py --config configs/styletex.yaml --train --gradio --gpu 0 \
     system.prompt_processor.prompt="${prompt}" \ # Text prompt describing target object
     system.guidance.ref_img_path=${image_path} \ # Path to style reference image
     system.guidance.ref_content_prompt="${image_prompt_c}" \ # Content prompt from reference image
     system.geometry.shape_init=mesh:${obj_path} \ # Path to mesh model file
     trainer.max_steps=2500 \
     system.geometry.shape_init_params=1.0 # Mesh scale factor
   
   # Example
   python launch.py --config configs/styletex.yaml --train --gradio --gpu 0 \
     system.prompt_processor.prompt="a Leather Recliner in illustration style of Milton Glaser animals" \
     system.guidance.ref_img_path="load/style_images/woman_style_of_Milton_Glaser.png" \
     system.guidance.ref_content_prompt="a woman face" \
     system.geometry.shape_init=mesh:load/shapes/sofa.obj \
     trainer.max_steps=2500 \
     system.geometry.shape_init_params=1.0

2. Outputs:

   Results will be saved in outputs/styletex

We have intensively borrow codes from the following repositories. Many thanks to the authors for sharing their codes.

• threestudio
• stable diffusion
• LucidDreamer
• InstantStyle
• DreamMat
• Fantasia3D
• diffusers
• ControlNet

If you find this repository useful in your project, please cite the following work. :)

 @article{10.1145/3687931,
        author = {Xie, Zhiyu and Zhang, Yuqing and Tang, Xiangjun and Wu, Yiqian and Chen, Dehan and Li, Gongsheng and Jin, Xiaogang},
        title = {StyleTex: Style Image-Guided Texture Generation for 3D Models},
        year = {2024},
        issue_date = {December 2024},
        publisher = {Association for Computing Machinery},
        volume = {43},
        number = {6},
        issn = {0730-0301},
        url = {https://doi.org/10.1145/3687931},
        doi = {10.1145/3687931},
        journal = {ACM Trans. Graph.},
        month = nov,
        articleno = {212},
        numpages = {14},
        }