This assignment is comprised of two parts:
- The mandatory coursework (due Monday, Feb 21st at 23:59) where the different tasks can be done in a simple way, without any external resources
- The optional competition (due Tuesday, March 1st at 16:00) where there is no restriction, and you can use external resources to get better results
Coursework
You can now upload your submissions on the moodle page.
The goal of this practical is to design and create an algorithm that can be used to restore old films. This is assessed and the deadline is set to Monday, Feb 21st at 23:59.
In their lifetime, films may suffer deterioration due to environmental hazards such as humidity and dust, chemical instabilities and improper storage. They can also get damaged during development and processing. This leads to flicker, camera shaking, blotches, scratches and other artefacts. Old archive footage, such as film of the 1900's, presents those defects: you will have to find ways to correct them. We will deal with four kinds of features visible on films. Each of them constitutes a part in the coursework. This is an exciting topic, and a lot of research has been carried out on this problem. The interested reader can find more information about how the restoration process is done in the literature (for example see papers here, here and here). For the purpose of this coursework assignment, however, simpler solutions are equally accepted. For those of you that want to go beyond the simplest possible solution, some of the subquestions allow one to earn additional bonus points for particularly good solutions.
Detection of scene cuts (10%)
The first part of the coursework consists in detecting scene cuts, which are not defects of the film, but we'll say it's the warm-up and it might also help you for the following sections. By scene cuts we mean moments when the scene changes. They will simply appear as hard transitions since old films don't display fancy effects to switch from a scene to another. Your first task is to write a simple algorithm to detect those transitions. In your output video, you will have to overlay text on the image that will inform the viewer that a scene cut occured.
Correction of global flicker (15% + up to 10% bonus)
Intensity flicker is a common artefact in old films. It is an unnatural temporal fluctuation in perceived image intensity that does not originate from the original scene. In other words, the image will appear suddenly darker/brighter and won't have a constant global luminosity over time. Intensity flicker can be caused by dust, chemical processing, aging of film, copying, and in the case or earlier film cameras, variations in shutter time. To illustrate, we'll have a look at the following consecutive pictures of a film:
It might sometimes not be obvious by looking at still shots, but you will understand better when looking at actual footage. In those three consecutive shots, the image gets suddenly brighter. Your second task is to write an algorithm that can reduce intensity flicker. As you will see in the videos, the flicker is not only global and can be local which is a harder problem to solve. Bonus points will be given to solutions that attempt to reduce the local flicker as well.
Correction of blotches (25% + up to 10% bonus)
Blotches are a common type of artefact that manifests itself as disturbing bright or dark spots caused by dirt and by the loss of the gelatin covering the film, due to aging effects and bad film quality. Let's have a look:
Here I pointed out the most noticeable ones, but there are many more and it is more obvious once more when looking at the movie. You will have to write an algorithm that detects those blotches and removes then. That implies inpainting the part of the image that the blotches were covering, or in other words fill the holes. Bonus points will be given for particularly stable and accurate solutions.
Correction of camera shake (30% + up to 10% bonus)
This is quite self-explanatory: the image just seems to shake. Your last task is to write an algorithm that aligns the frames together in order to remove that unwanted motion. Your output video should be comparatively shake-free. Bonus points will be given for very general or particularly efficient solutions.
Details
- We are not asking for complicated solutions. Everything is doable with simple algorithms. Bonus points will be given to particularly clever or elegant solutions.
- The total number of points that you can get on this coursework sums up to 110, however we will clamp the final mark to 100.
- You are provided with a set of different films. You will pick one, or a part of one, and process it with your algorithms. The part that you pick should display all the four features specified above.
- You will have to hand in the processed video, your code and the explanations. An upload section will be available on the course website. The size limit is set to 50MB per student. Your submission has to contain a PDF (no other formats!) with the description of the course work.
- In this pdf we want to see your personal work and the details about how each part of your algorithm works step by step.
- Your code will have to respect the interface that we provide (we must be able be able to run labs4('mypath/to/images/') on any folder containing a sequence of images). This way we can test your algorithms on more videos after you hand it in.
- You are provided with two handy functions that allow you to easily read a sequence of images from a folder, and save your corrected images. You will find them useful especially if your machine has a limited amount of memory (you can use the functions to load a part of the movie, unload it and carry on: no need to load the whole video). To be able to use them, you must first convert your video into a sequence of images, this will be explained in the next section.
Convert a video into a sequence of images
As mentionned before, you will have to convert your chosen video into a sequence of images, eg. footage.avi into a sequence from footage_0000.png to footage_0150.png. It is advised to extract png images, and not jpeg, to avoid further compression.
On Windows
It is slightly complicated to achieve that on Windows, we advise you to do it on a Linux labs machine or use that to open a linux session. However if you feel that you can handle it you can do the following on your (personal) computer:
- Get the 32bits version of this software:
- Latest ffdshow (http://sourceforge.net/projects/ffdshow-tryout/files/)
- Haali Media Splitter (http://haali.su/mkv/)
- Media Player Classic (http://mpc-hc.sourceforge.net/)
- Open ffdshow video configuration and in the codec section, make sure that H.264 is set to libavicodec.
- Decoder Options and untick both "Drop frame on delay" and "Skip H264 deblocking on delay".
- Go to the "Grab" options, tick Grab, and set your options there. Dont forget to put Quality to 100 and Image format to PNG.
- Now the next time you will play a video with Media Player Classic, it will export the frames in the folder you indicated before.
- Don't forget to untick "Grab" after you exported the images.
On Linux
It is much simpler:
- Using ffmpeg, eg.: "ffmpeg -i footage.mp4 -ss 00:00:10 -t 50 footage_%4d.png" will export frames from footage.mp4, starting at 00:00:10, for 50 seconds, and will name the frames footage_0001.png, footage_0002.png, etc.
As mentioned before, if you are a Windows user and you want to work on linux, you can go to http://www.cs.ucl.ac.uk/cgi-bin/csrw and open a new gnome session using your CS login/passwd (Matlab seems to have problem with KDE). Let us know of any problem.
Material
- You can find the films and the matlab code here.
- The film footage4.mp4 is hard to correct than the others since it's more damaged. You can test your algorithm on it if you want to test its robustness.
- The film contained in competition.zip is already a sequence of pngs. It contains a compilation of 6 different footages (including footage1.mp4, footage2.mpg, footage4.mp4 and parts of the Metropolis movie). This is the most challenging footage to correct.
- The file you will have to fill in is labs4.m
- You can use the two functions load_sequence and save_sequence to load and save a sequence of images.
- To create an output video, you are advised not to use Matlab because it behaves weirdly. You can use the save_sequence function to save your output as a sequence of images and then use ffmpeg to create a movie from them. For example: ffmpeg -r 25 -i corrected_image%4d.png corrected.mp4 (-r 25 stands for 25 frames per second). Look at ffmpeg help or google for more info on that.
Competition
You can now upload your submissions on the moodle page.
The second part of this assigment is a competition. This is optional and the deadline is set to Tuesday, March 1st at 16:00.
The goal, as you might have understood, is to create the best, or most creative, restoration algorithm -- surprise us! The winner will have a chance to present their solution during one of the lectures.
For this task, you will have to correct the film contained in competition.zip.
Unlike the mandatory coursework, there is no restriction, and you can use external libraries mixed with your own code in order to get better results, or you may use other film sources to demonstrate your software. As with the marked coursework, please submit your output video together with your code and a description.
Note: contribution to the competition will not have any influence on the marked component. The competition is not marked.