Language-Driven Artistic Style Transfer

Tsu-Jui Fu¹ Xin Eric Wang² William Yang Wang¹ ¹UC Santa Barbara ²UC Santa Cruz European Conference on Computer Vision (ECCV) 2022

Abstract

Despite having promising results, style transfer, which requires preparing style images in advance, may result in lack of creativity and accessibility. Following human instruction, on the other hand, is the most natural way to perform artistic style transfer that can significantly improve controllability for visual effect applications. We introduce a new task—language-driven artistic style transfer (LDAST)—to manipulate the style of a content image, guided by a text. We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDAST by the patch-wise style discriminator. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. CLVA further compares contrastive pairs of content image and style instruction to improve the mutual relativeness. The results from the same content image can preserve consistent content structures. Besides, they should present analogous style patterns from style instructions that contain similar visual semantics. The experiments show that our CLVA is effective and achieves superb transferred results on LDAST.

Visual Attribute Instructions on DTD² (press Content for all results)

Content	Instruction	SANet	LST	ManiGAN	CLIPstyler	CLVA	Style	Semi-GT
	grayish bluish green smeared paint
	zig zag lines embroidered with green and dark
	black lines, splashed spotted, water colours
	swirly, grey, rounded shapes, blurry, murky
	orange red, bright macroscopic, hand-blown bubbles
	floating, colorful, white backdrop
	light brown gauzy texture
	blonde color, matted, messy, smooth
	painting with zigzag orange green pattern
	multi-colored polka-dotted fabric, black background

Emotion Effect Instructions on ArtEmis

Content	Instruction	SANet	LST	ManiGAN	CLIPstyler	CLVA	Style	Semi-GT
	i feel chaotic and confused due to the black and gray tones
	i've never seen a black and white horse before
	the sky is a good contrast to the dark red/brown hill beside the ocean
	the sway of the vibrant trees seems like they are alive
	charmed by the beautiful bright day, at the side of the pale water
	the bright soft colors reminds me of a sunset
	pen and ink ornate drawing, something decadent
	the gold color is very energizing
	balmy weather, sparkling water, swimmers looking
	the yellow boxes are suspended and very confusing

Specific Content Domain (Car and Church)

Content	Instruction	ManiGAN	StyleCLIP [7]	NADA [8]	CLIPstyler	CLVA	Style	Semi-GT
	piralled, brown, gray, metallic, tunnel
	gray rock with light streaks
	pinkish, interlaced, cloth, like pillow cover
	colorful smooth pretty circular round
	swirly, silver, rounded shapes, blurry, murky
	light green and yellow flannel plaid fabric
	brown salt deposits forming around crystals
	hexagonal, orange, blue, smooth white
	soft porous coiled grey spiral
	light brown, twisted, type of material

CLVA Results

Instruction
purple pink violet medium polka dots
banded blue orange lined
ink painting, black dotted line, whiteboard
colorful smooth pretty circular round
wrinkled, colorful, soft fabric on black background
transparent, white, brown, golden, rocky
sun is shining, bouncing light, wonderful summer
i am confused because of the sunrise in front of overcast sky
i feel chaotic and confused due to the black and gray tones
jazzy and soulful painting, speaks for itself with melancholy undertones
beautiful and vibrant, there is a contrast like the fall colors
dark background with the red figure gives me a feeling of something sinister

High-resolution (2560x1440) Results (right click to view in full size)

Instruction
crystals like a krypton diamond in white and green colors
painted, rubbed, smeared yellow, green, blue and red
wrinkled leather like gray shiny surface
the boats appear lovely in the way they gently rest on top of the peaceful waves
warm painting feels like the sun is setting behind giving it a golden peaceful glow
i feel excited because the shadows suggest the sun is rising and starting a new day

Reference

A Neural Algorithm of Artistic Style. Gatys et al. arXiv:1508.06576.
Universal Style Transfer via Feature Transforms. Li et al. NeurIPS'17.
Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. Huang and Belongie. ICCV'17.
Arbitrary Style Transfer with Style-Attentional Networks. Park and Lee. CVPR'19.
Learning Linear Transformations for Fast Image and Video Style Transfer. Li et al. CVPR'19.
ManiGAN: Text-Guided Image Manipulation. Li et al. CVPR'20.
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery. Patashnik et al. CVPR'21.
StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators. Gal et al. arXiv:2108.00946.
CLIPstyler: Image Style Transfer with a Single Text Condition. Kwon and Ye. CVPR'22.