Colorful Image Colorization

Richard Zhang

Phillip Isola

Alexei A. Efros

[Demo]

[GitHub]

[Talk]

[Slides]

[Paper]

[Poster]

Also check out our new work on Interactive Deep Colorization!

Example input grayscale photos and output colorizations from our algorithm. These examples are cases where our model works especially well. For randomly selected examples, see the Performance comparisons section below.

How to interpret the results
Welcome! Computer vision algorithms often work well on some images, but fail on others. Ours is like this too. We believe our work is a significant step forward in solving the colorization problem. However, there are still many hard cases, and this is by no means a solved problem. Some failure cases can be seen below and the figure here.

This is partly because our algorithm is trained on one million images from the Imagenet dataset, and will thus work well for these types of images, but not necessarily for others. We call this the "dataset bias" problem. We include colorizations of black and white photos of renowned photographers as an interesting "out-of-dataset" experiment and make no claims as to artistic improvements, although we do enjoy many of the results!

There has been some concurrent work on this subject as well. Specifically, see Larsson et al. and Iizuka et al. below.

Please enjoy our results, and if you're so inclined, try the model yourself!

Abstract

Given a grayscale photograph as input, this paper attacks the problem of hallucinating a plausible color version of the photograph. This problem is clearly underconstrained, so previous approaches have either relied on significant user interaction or resulted in desaturated colorizations. We propose a fully automatic approach that produces vibrant and realistic colorizations. We embrace the underlying uncertainty of the problem by posing it as a classification task and use class-rebalancing at training time to increase the diversity of colors in the result. The system is implemented as a feed-forward pass in a CNN at test time and is trained on over a million color images. We evaluate our algorithm using a "colorization Turing test," asking human participants to choose between a generated and ground truth color image. Our method successfully fools humans on 32% of the trials, significantly higher than previous methods. Moreover, we show that colorization can be a powerful pretext task for self-supervised feature learning, acting as a cross-channel encoder. This approach results in state-of-the-art performance on several feature learning benchmarks.

Try the Interactive Demo

[Algorithmia] Demo

Videos

ECCV Talk 10/2016, also hosted on [VideoLectures]

Try our code

[GitHub] Demo

Paper and Supplementary Material

Zhang, Isola, Efros.
Colorful Image Colorization.
In ECCV, 2016 (oral).
(hosted on arXiv)

[Bibtex]

Results on legacy black and white photos

We show results on legacy black and white photographs from renowned photographers Ansel Adams and Henri Cartier-Bresson, along with a set of miscellaneous photos.

Ansel Adams

(hover for our results; click for full images)
extention of Figure 15 from our paper

Henri Cartier-Bresson

(hover for our results; click for full images)
Figure 16 from our paper

Miscellaneous Photos

(hover for our results; click for full images)
Figure 17 from our paper

Performance comparisons

Click the montage to the left to see our results on Imagenet validation photos (this is an extension of Figure 6 from our [v1] paper). Click the montage to the right to see results on a test set sampled from SUN (extension of Figure 12 in our [v1] paper). These images are random samples from the test set and are not hand-selected.

Comparisons on Imagenet

(hovering shows our results; click for additional examples)

Comparisons on SUN

(hovering shows our results; click for additional examples)

We also provide an initial comparison against Cheng et al. 2015 here. We were unable to acquire code or results from the authors, so we simply ran our method on screenshots from the figures in the paper of Cheng et al. See Section 3 in the supplementary pdf for further discussion of the differences between our algorithm and that of Cheng et al.

Semantic interpretability of results

Here, we show the ImageNet categories for which our colorization helps and hurts the most on object classification. Categories are ranked according to the difference in performance of VGG classification on the colorized result compared to on the grayscale version. This is an extension of Figure 6 in the [v1] paper.

Click a category below to see our results on all test images in that category.

Top			Bottom
Rapeseed Lorikeet Cheeseburger Meat Loaf Pomegranate	Green Snake Pizza Yellow Lady's Slipper Orange Goldfinch		Chain Wok Can opener Water bottle Modem	Standard Schnauzer Pickelhaube Half Track Barbershop Military Uniform

Deep Dream Visualization

Alexander Mordvintsev visualized the contents of our network by applying the Deep Dream algorithm to each filter in each layer of our [v1] network. He has kindly shared his results with us! The deep-dream images are grayscale and colorized with out network. We found that the conv4_3 layer had the most interesting structures. Click on each layer below to see the results, and let us know what you see!

conv1_2	conv2_1	conv2_2	conv3_1	conv3_2	conv3_3
conv4_1	conv4_2	conv4_3	conv5_1	conv5_2	conv5_3
conv6_1	conv6_2	conv6_3	conv7_1	conv7_2	conv7_3

User-Generated Examples

We have received many interesting examples and applications, developed by users! Note that the video examples are run on a per-frame basis, with no temporal consistency enforced. If you have any examples you'd like to share, please email Richard Zhang at rich.zhang at eecs.berkeley.edu.

Videos Lukas Graham - 7 years [Music Video] Impact (1949) [Clip] [Blog Post] Dorothy Dandrige - Zoot Suit (1942) [Clip] Seven Samurai (1954) [Clip]
Applications Automatic colorizer Bot [Reddit] [Gallery] [Blog Post] [2 Week Update] Reddit thread from /r/InternetIsBeautiful [Link] [Demo] [Blog Post] Colorizing an 1880 Glass Plate [Video by Mathieu Stern] Colourful Past: Find and colorize historical photos [Link]

Recent Related Work

There have been a number of works in the field of automatic image colorization in the last few months! We would like to direct you to these recent related works for comparison. For a more thorough discussion of related work, please see our full paper.

Concurrent Work
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning Representations for Automatic Colorization. In ECCV 2016. [PDF][Website]
Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. In SIGGRAPH, 2016. [PDF][Website]

Previous Work
Ryan Dahl. Automatic Colorization. Jan 2016. [Website]
Aditya Deshpande, Jason Rock and David Forsyth. Learning Large-Scale Automatic Image Colorization. In ICCV, Dec 2015. [PDF][Website]
Zezhou Cheng, Qingxiong Yang, and Bin Sheng. Deep Colorization. In ICCV, Dec 2015. [PDF]

Acknowledgements

This research was supported, in part, by ONR MURI N000141010934, NSF SMA-1514512, an Intel research grant, and a Tesla K40 GPU hardware donation by NVIDIA Corp. We thank members of the Berkeley Vision Lab for helpful discussions, Philipp Krähenbühl and Jeff Donahue for help with self-supervision experiments, and Aditya Deshpande and Gustav Larsson for providing help with comparisons to Deshpande et al. and Larsson et al.