Detail++: Training-Free Detail Enhancer for Text-to-Image Diffusion Models

Lifeng Chen¹    Jiner Wang¹    Zihao Pan¹    Beier Zhu^{1, 2}    Xiaofeng Yang^{1, 2}    Chi Zhang<sup>1✉

1</sup>AGI Lab, Westlake University,  ²Nanyang Technological University 

   

This paper introduces a novel method for addressing semantic binding in text-to-image (T2I) generation. Our approach employs a progressive injection mechanism to accurately assign attributes within the correct subject regions. Notably, the proposed method not only mitigates semantic overflow but also overcomes the challenge of style blending —issues that existing techniques fail to resolve. The core idea involves initially removing all attributes and then reintroducing them sequentially within a unified self-attention framework.

For technical details, please refer to our paper.

1. Environment Setup

   Clone the code and prepare the environment:

   git clone git@github.com:clf28/Detail-plus-plus.git
   cd Detail-plus-plus
   pip install -r requirements.txt

   Additionally, download the SpaCy model for syntax parsing:

   python -m spacy download en_core_web_sm

2. Configure Parameters

   Modify the config.py file to adjust runtime parameters as needed. Our framework is highly flexible, enabling adjustments to parameters such as prompt text, mask threshold, and attention map substitution percentage, which in turn yield a diverse range of interesting results. The key parameters are outlined as follows:

   • prompt: Text prompt for guiding image generation.
   • use_nlp: Whether to decompose the string in prompt automatically using SpaCy.
   • sps: A list of sub-prompts. You can design diverse sub-prompts freely like color, object accessory, and style to achieve different effects. Worked when use_nlp is set to False.
   • nps: A list of noun pharses. Means the corresponding subject of current adding attributes. None here means the influence region will be the whole image. Worked when use_nlp is set to False.
   • lb_t: The threshold for masking out the image regions. Varing from -1 to 1, where -1 means no masking.

3. Run the Example

   Execute the main script run.py:

   python run.py

   The generated images will be saved in the runs-SDXL/style-test directory.

To run a local demo of the project, run the following:

pyton app.py

If everything is set up correctly, ColorConfig and StyleConfig class in config.py should produce images like the left and right below, respectively:

• Custom Configurations: Please refer to the config.py file for customizing the runtime parameters.
• Parameter Sensitivity: This method retains the sensitivity characteristic of inference-based optimization techniques, meaning that the generated results are significantly influenced by the choice of hyperparameters. Careful tuning may be necessary to achieve optimal performance.
• NLP Models: When using NLP models like SpaCy for token parsing, ensure the correct language model is installed. Here, we provide a simple adjective phrase parser that may not be accurate in all cases.

This project builds upon valuable work and resources from the following repositories:

• Attend-and-Excite
• ToME
• 🤗 Diffusers

We extend our sincere thanks to the creators of these projects for their contributions to the field and for making their code available. 🙌