This repository contains the inference code for our paper, ECLIPSE. We show how to utilize the pre-trained ECLIPSE text-to-image prior associated with diffusion image decoders such as Karlo and Kandinsky.
- ECLIPSE presents the tiny prior learning strategy that compresses the previous prior models from 1 billion parameters down to 33 million parameters.
- Additionally, ECLIPSE prior is trained on a mere 5 million image-text (alt-text) pairs.
News: Checkout our latest work, λ-ECLIPSE extending the T2I priors for effecient zero-shot multi-subject driven text-to-image generations.
Please follow the below steps to run the inference locally.
Qualitative Comparisons:
Quantitative Comparisons:
-
Release ECLIPSE priors for Kandinsky v2.2 and Karlo-v1-alpha. -
Release the demo. - Release ECLIPSE prior with Kandinsky v2.2 LCM decoder. (soon!)
- Release ECLIPSE prior training code. (will be released in seperate repository)
git clone git@github.com:eclipse-t2i/eclipse-inference.git
conda create -p ./venv python=3.9
pip install -r requirements.txtconda activate ./venv
gradio main.pyThis repository supports two pre-trained image decoders: Karlo-v1-alpha and Kandinsky-v2.2.
Note: ECLIPSE prior is not a diffusion model -- while image decoders are.
from transformers import CLIPTextModelWithProjection, CLIPTokenizer
from src.pipelines.pipeline_kandinsky_prior import KandinskyPriorPipeline
from src.priors.prior_transformer import PriorTransformer
from diffusers import DiffusionPipeline
text_encoder = (
CLIPTextModelWithProjection.from_pretrained(
"laion/CLIP-ViT-bigG-14-laion2B-39B-b160k",
projection_dim=1280,
torch_dtype=torch.float32,
)
)
tokenizer = CLIPTokenizer.from_pretrained(
"laion/CLIP-ViT-bigG-14-laion2B-39B-b160k"
)
prior = PriorTransformer.from_pretrained("ECLIPSE-Community/ECLIPSE_KandinskyV22_Prior")
pipe_prior = KandinskyPriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-prior",
prior=prior,
text_encoder=text_encoder,
tokenizer=tokenizer,
).to("cuda")
pipe = DiffusionPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-decoder").to("cuda")
prompt = "black apples in the basket"
image_embeds, negative_image_embeds = pipe_prior(prompt).to_tuple()
images = pipe(
num_inference_steps=50,
image_embeds=image_embeds,
negative_image_embeds=negative_image_embeds,
).images
images[0]from src.pipelines.pipeline_unclip import UnCLIPPipeline
from src.priors.prior_transformer import PriorTransformer
prior = PriorTransformer.from_pretrained("ECLIPSE-Community/ECLIPSE_Karlo_Prior")
pipe = UnCLIPPipeline.from_pretrained("kakaobrain/karlo-v1-alpha", prior=prior).to("cuda")
prompt="black apples in the basket"
images = pipe(prompt, decoder_guidance_scale=7.5).images
images[0]We would like to acknoweldge excellent open-source text-to-image models (Kalro and Kandinsky) without them this work would not have been possible. Also, we thank HuggingFace for streamlining the T2I models.
