CS180 Proj 1: Colorizing the Prokudin-Gorskii photo collection

Overview

The goal of this project is to take Prokudin-Gorskii's digitized glass plate images and reconstruct full-color images using image processing techniques. The primary challenge is to align the three color channel images (red, green, blue) extracted from a single plate, ensuring that the final composite image has minimal visual artifacts.

Approach

I choose to align the green and blue channels with the red channel using translation-based image alignment. A simple exhaustive search over small pixel displacements is employed to find the best alignment, using the sum of squared differences as the matching metric (the lower the better). To improve the efficiency , the pyramid search method is applied. This method involves scaling down the images and aligning them at coarser levels before refining the alignment at higher resolutions. Each level will pass on the alignment information to the next level and the search range will be halfed to lower the computational costs as the resolution is getting higher (the initial search range is set to [-20, 20] and the minimal serach range is [-2, 2]). Also the channels will be cropped slightly by 10% before begining the alignment process to ensure the algorithm is matching the actual contents instead of the borders of the images.

Sample Images

Below are some of the images that I have colorized using the above approach.

Cathedral
Cathedral (Shifts: [-12. -3.], [-7. -1.])
Boat
Monastery (Shifts: [-3. -2.], [-6. -1.])
Tobolsk
Tobolsk (Shifts: [-6. -3.], [-4. -1.])
Lady
Lady (Shifts: [-112. -11.], [-61. -3.])
Onion Church
Onion Church (Shifts: [-108. -36.], [-57. -10.])

Bells and Whistles

Edge Detection

Instead of aligning the entire channels, I used cv2.Canny to detect edges and align those edges. This works very well on images with lots of edges like emir.tif.

Sample Image

Emir without Edge Detection
Emir without Edge Detection (Shifts: [-26. 829.], [-57. -17.])
Emir with Edge Detection
Emir with Edge Detection (Shifts: [-107. -40.], [-58. -17.])

Auto Cropping

An auto cropping algorithm is used to crop the unwanted visual effects of the postprocessed images. It will compare the intensities of the pixels between the three channels and actively detect white or black borders. The reserved region will be the rectangle bounding the largest connecting region.

Sample Image

Sculpture without Auto Cropping
Sculpture without Auto Cropping
Sculpture with Auto Cropping
Sculpture with Auto Cropping

The algorithm will mostly be able to remove white and black orders and remove some colored borders. It is not perfect on every image

Final Images (with Edge Detection and Auto Cropping)

Cathedral
Cathedral (Shifts:[-12. -3.], [-7. -1.])
Church
Church (Shifts: [-58. 4.], [-33. 8.])
Emir
Emir (Shifts: [-107. -40.], [-58. -17.])
Harvesters
Harvesters (Shifts: [-123. -11.], [-64. 3.])
Icon
Icon (Shifts: [-90. -22.], [-48. -5.])
Lady
Lady (Shifts: [-120. -13.], [-63. -3.])
Melons
Melons (Shifts: [-176. -13.], [-96. -4.])
Monastery
Monastery (Shifts: [-3. -2.], [-6. -1.])
Onion Church
Onion Church (Shifts: [-107. -35.], [-58. -10.])
Sculpture
Sculpture (Shifts: [-140. 27.], [-107. 16.])
Self Portrait
Self Portrait (Shifts: [-175. -37.], [-98. -8.])
Three Generations
Three Generations (Shifts: [-111. -8.], [-59. 7.])
Tobolsk
Tobolsk (Shifts: [-6. -3.], [-4. 0.])
Train
Train (Shifts: [-85. -29.], [-41. -26.])

* Shifts are with respect to the red channel [b_shift, g_shift]