Coloring images and video with Machine Learning

Posted on by

Coloring images with Machine Learning

If you’ve ever tried adding color to a black and white image, you’ll know it’s a tedious task, and unless you’re an expert, the results will not be compelling. We will now explain a solution to this problem based on machine learning.

Through the techniques offered by machine learning we can train a system so that it can perform tasks based on the experience in which it has been trained. Richard Zhang, Phillip Isola and Alexei A. Efros. Propose a solution in ECCV, 2016. In their project Colorful Image Colorization they share a system that has been trained with a million images of Imagenet so that you can add color to black and white images automatically.

Install caffe and basic Python libraries

Before installing the repository of Colorization we have to install caffe and the Python libraries: numpy, pyplot, skimage, scipy.

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
git clone https://github.com/BVLC/caffe.git
cd caffe
cp Makefile.config.example Makefile.config
make all
make test
make runtest
make pycaffe
make distribute
mkdir ../python
mv distribute/python/caffe/ ~/python/
export PYTHONPATH="${PYTHONPATH}:${HOME}/python:${HOME}/caffe/build/"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HOME}/caffe/build/lib"

For PYTHONPATH and LD_LIBRARY_PATH to preserve the values, the following lines must be added to ~/.bashrc

export PYTHONPATH="${PYTHONPATH}:${HOME}/python:${HOME}/caffe/build/"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HOME}/caffe/build/lib"

Install Colorful Image Colorization

Now we only have to clone the project Colorization and run it. Keep in mind that the output file is always a PNG.

git clone -b master --single-branch https://github.com/richzhang/colorization.git
cd colorization/
./models/fetch_release_models.sh
python ./colorize.py -img_in test_in.jpg -img_out test_out.png

These are some of the tests we have done to check the performance:

This slideshow requires JavaScript.

Colorful Image Colorization CPU only mode

If you compiled caffe only for CPU you will have to make the following modification to the colorize.py script to use the CPU instead of the GPU.

caffe.set_mode_cpu()
#caffe.set_mode_gpu()
#caffe.set_device(args.gpu)

Script for Black&White videos

Now that we have verified that we can add color to images the next step is to add color to black and white videos, since a video is nothing more than a sequence of images. For this task we use a bash script that first extracts all the video frames, then adds them color one by one and finally returns the video in color with the audio of the original video. For it to work we will have to have ffmpeg installed.

#!/bin/bash

VIDEOIN=$1
VIDEOOUT=$2
mkdir /tmp/bw/
mkdir /tmp/color/
ffmpeg -i $VIDEOOUT -r 25/1 /tmp/bw/output%09d.jpg

for file in /tmp/bw/*;
do
  echo ${file}
  filename=$(basename $file)
  python ~/colorization/colorize.py -img_in $file -img_out /tmp/color/$filename
done;

ffmpeg -r 25 -f image2 -s 1280x720 -i /tmp/color/output%09d.png -vcodec libx264 -crf 25  -pix_fmt yuv420p /tmp/colorized.mp4
ffmpeg -i /tmp/colorized.mp4 -i $VIDEOIN -map 0:v -map 1:a -c copy -shortest $VIDEOOUT

rm /tmp/colorized.mp4
rm /tmp/bw/*
rm /tmp/color/* 

To run it just have to save this script with videocolor.sh name (or another we like more) and run it with the name and path of the input video and the output, such as:

bash ./videocolor.sh ~/Downloads/test.mp4 ~/Downloads/test_color.mp4

Below we can see a couple of videos that have been added color and their originals in black and white:

Arnold Schwarzenegger wins Mr Universe 1969 in Black&White:

Arnold Schwarzenegger wins Mr Universe 1969 in Color:

Madrid News Bulletin 1966 in Black&White:

Madrid News Bulletin 1966 in Color:

Adding Color to Infrared Images

Nowadays there are few images that are generated originally in black and white but one of those examples are the images captured by infrared cameras. Below we see a few captures of these cameras and their corresponding conversion, the results are surprising.

This slideshow requires JavaScript.

Bonus:

We can not say that the system is perfect since in some cases the experience provided by a million images is not enough. When the system faces something for which it has not been trained tends to solve it as if it were an image for which if it has experience. For example, in the following gallery we see that when asked to color the image of the rainbow colored hair girl the image that returns is a completely blonde girl. This is due to that in its dataset of images does not appear nobody with that particularity hair, but nevertheless surely they appear many photos of blonde girls, thats the reason why it automatically color as a blonde hair.

This slideshow requires JavaScript.

 

Comments are disabled