image labeling job

labeling instructions
and notes

0. Get Started -1. Introduction - 2. Notes - 3. What does "good labeling" mean? - 4. Classes of object to label - 5. Payment

0. Short version: get started fast!

Log in
Download our labeling program
Download a new photograph
Label it (i.e. "paint" it using the labeling program) according to criteria in the sections below
Upload the labeled image
Repeat (Though the first few times, first wait for it to be approved by "Check your status")
When we're all done (or just tell us), upload the Log text files; now you get paid for all the images you labeled.

Criteria for how to label are set out in the sections below, and these really are important for your image to be approved. Briefly, the main idea is to correctly paint the objects in an image that are noticeable to a driver.

Where are the traffic signs?
Where are the cars?
Is that a pedestrian at the side of the road, or a telephone pole?

We need loads of examples of what those categories look like; the frail or dangerous categories especially. Painting within the lines IS important, because otherwise, our learning algorithms could associate the appearance of "pedestrians" with "sidewalk," leading it to make the wrong guess next time.

To help stay in the lines, the labeling program computes several versions of what it thinks could be useful edges - please take advantage of that. If something is too small and too far away to matter to a driver, or if it's not really one of the objects on our list, leave it blank - that's not ideal, but perfectly normal, and MUCH better than labeling something incorrectly. Ask us questions if you're in doubt - this work can be tedious at times, that's why we're paying you to help us do it. It IS important work though, and may eventually help reduce the number of accidents.

1. Introduction - description of the task

For this long-term research, we have videos of driving scenarios. These constitute a database used to perform experiments on artificial intelligence (AI) algorithms for Object Recognition. For AIs to learn and to be tested, we also need images from those videos to be annotated, so that ideally, each pixel is associated with a class of object like "sky," "bicycle," "building," etc. The annotations take the form of a 2nd image, which acts as a map, with each color representing one of the 32 object categories.

The performance of those algorithms strongly depends on the accuracy of the labeling. This is why you should carefully read the section What does "good labeling" mean?.

You will be able to download photos and upload the corresponding images you've labeled entirely through this website. You will be paid by cheque at the end of this month on the basis of the number of images you have labeled (see payment section). Therefore, you can go through the whole process from home.

We now have a first pool of 500 images total to be labeled by march.

Using our program, you will be able to label the images: using the mouse as a paintbrush, you will colour pixel regions with the colour corresponding to the object in the image (e.g. tree, car, ...). A segmentation program breaks the image up into pieces to save you some time. The section below describes the object classes we want you to label.

2. Notes

• It is important you don't change the names of the image files which you download and upload.

• If you plan to download many images before uploading the labeled ones, they won't be available to other users. So we ask you to keep the number of images downloaded and unlabeled low (say 10 max).

• Even if you've registered for this job, you don't have any obligation to label a minimum number of images. If you change your mind and don't want to do it, that's fine by us. We'll be very disappointed though! :)

• Email if you need to contact me; remember to always write "PAINT" in the subject.

3. What does "good labeling" mean?

The quality of your labeling will directly affect the performance of our Object Recognition algorithm. So it is very important you follow the recommendations below.

• Getting the labels right : The section below describes what each object class depicts. When you pick a colour to paint an object, make sure it's the right color and that it does correspond to this object. It sounds trivial, but such mistakes are the worst for our algorithm. If some objects are difficult (too small or ambiguous ) then it's better not to paint them, i.e. leave them as blank which we call "void".

• Avoid holes : The program produces an automatic segmentation that saves you some time. This segmentation often produces holes in textured regions. You should fill these holes as much as possible.

• Precision of object boundaries : When painting an object, try to follow the object boundaries as closely as possible without overlapping on the adjacent object. Sometimes, boundaries are hard to detect visually, in which case we hope you'll paint where you yourself are certain.

• All obvious and clearly visible objects should be assigned a label.

We remind you that each labeled image you submit will be inspected by us to make sure they follow the above recommendations, so that they can have the "approved" status. If we feel the labeling is not correct, we'll let you know. You'll be paid for a labeled image only if it has the "approved" status. Please don't think we're being mean if this happens on the first couple of images and you have to correct some regions and resubmit - it was hard for all of us at the beginning.

Example of poorly labeled images

The problems with the labeled image (color-coded map) are the following:
- pedestrians on the left are not fully covered and the label overlaps a lot (meaning that the background is labeled as "pedestrian" as well!)
- the bushes (which are visible) on the right are not labeled at all
- large areas have many holes, despite being "easy"

Example of WELL labeled images

4. Classes of object to label

We attempt to minimize the ambiguity in the class labels being used here by thinking of them in the context of a car driver. For example, a picture of a human on a billboard is not a pedestrian, and a wire fence at the side of the road is still a fence, even though it is almost transparent while driving. The distinctions are made based on what the driver of a car would or should be concerned with in the immediate future. While most objects cast shadows, it is generally assumed that shadow pixels can remain labeled as void or as the object casting the shadow. Long shadows, such as that caused on the road by a pole or building, however, should count as road, since the function of that area is still fundamentally as a road.

objects and their corresponding label colours

Void	Void	Class for unlabeled pixels. This should be assigned by default, or when there is doubt in choosing between multiple possible class labels. This is the safest label to assign, because it does no harm to learning algorithms - providing no useful information, but at least introducing no interclass confusion.
Fixed Objects	Building	Any rigid architectural (i.e. man-made and big) structure on the ground plane that would require a car to drive around. Compare to "Wall" and "Bridge" below.
	Wall	Like buildings, but extending vertically only to a height of approximately one storey and generally having few visible features like windows or doors. Thicker than a Fence and totally opaque. Can be made of natural growth like a bush-hedge.
	Tree	This category captures any vertical vegetation. Tree trunks and the their leaves, for example, indicate parts of the terrain where a car would collide. There is little need to capture each leaf. Also, empty spaces between branches or leaves may be filled in or not. Different from Vegetation-misc in that a Tree would cause damage to a car, and other objects (like people) will rarely be found in or on Trees.
	Vegetation misc.	Grass, flowers, shrubs, etc. These regions are mostly on the ground, and one could expect to see a pedestrian moving on top of this region.
	Fence	Structure for separating people and cars, transparent or containing holes through which the background is visible, and usually thin.
	Sidewalk	Like Vegetation Misc., one would expect to see people or cyclists here, but no cars (in contrast to a cross-walk, which counts as road). A dirt path would also count as a sidewalk, though sidewalks are usually also raised above the neighboring road.
	Parking block	Any low to the ground element introduced intentionally to block or direct cars.
	Column/pole	Broad category of objects that are more tall than they are wide, which must be avoided when driving.
	Traffic cone	Like Parking block but temporary. Includes road flags or other small traffic obstacles one finds in otherwise drivable areas.
	Bridge	Any overhead structure that one can safely expect to drive under.
	Sign / symbol	Traffic signs and symbols that are meant for drivers, but do not include street signs or other destination-related directions.
	Misc text	All other text.
	Traffic light	Lights at street intersection, or railroad crossings. The light (various colors) is most important, but the (usually rectangular) casing also counts as part of the traffic light.
	Other	Other significant objects that could be obstacles to a driver. For example, construction materials or rubbish dumpsters at the side of the road.
Road Surface	Road	Drivable surface of the road that has no special markings that the driver should be aware of. Most often empty or occupied by vehicles, though the spaces between zebra crossings also count as road.
	Shoulder	Drivable sections of road that are not intended for regular use, i.e. one would expect that area to remain empty, but it could sit underneath a vehicle.
	Lane markings drivable	Any painted or etched text or symbols on the road, over which a car normally drives, such as the stripes of a zebra crossing or the arrow in a turn lane.
	Non-Drivable (lane markings)	Markings that a car could physically drive across, but should not. Examples include the stripe separating the road from the shoulder, roundabout islands, or patterns painted to force traffic to diverge into multiple lanes.
Moving Objects	Note:	all moving objects counts as such, even when they are stationary. The fact that they do not appear to be moving does not affect their status.
	Animal	Any non-human creature.
	Pedestrian	General purpose label for all adult humans in any pose, but excludes humans who are also on a bike and children.
	Child	Small pedestrians.
	Rolling cart/luggage/pram	General class of objects that are in motion through the influence of a nearby pedestrian.
	Bicyclist	Both the mechanism and the person operating it, adult or child.
	Motorcycle/scooter	Motorized version of bicycle.
	Car (sedan/wagon)	Normal sized car; different from other vehicles because it is shorter.
	SUV / pickup truck	Personal non-commercial vehicle.
	Truck / bus	Any vehicle larger than the SUV / pickup truck class, used for bulk goods or transporting many people.
	Train	Includes any vehicle that runs on rails.
	Misc	Generic class for moving objects and vehicles, whether animate or not. While it currently seems unproductive to worry about labeling flags or windmill blades, those would examples of this class.
Ceiling	Sky	Any overhead open air space.
	Tunnel	Enclosed overhead space.
	Archway	Narrow enclosed overhead spaces, such as petrol station awnings, gates, etc. where a vehicle can be expected to drive beneath.

5. Payment

Conditions to be paid on a per image basis:
- Each labeled image that you have uploaded must have been approved by us (see 3. What does "good labeling" mean? section)
- Its corresponding log file should be sent to us. Although you should send your labeled images as you label them, all the log files can be sent to us at the end of the labeling period (end of february). You'll just need to zip the "Log" directory and email it to us. See software page to know about the logs.

You will receive the payment at the end of the job period for all the images you have labeled (end of february). You will receive a check from the University of Cambridge at your postal address. So we'll ask you to send us by email your postal address (

).

Note: you will find that the more images you label, the quicker you'll be. Since you'll be paid on a per-image basis, the more images you label and the higher the hourly rate will be. We reckon it should take about 15-20min to label one image, i.e. about £7-£10/hour.

Original: 29 january 2007 - Julien Fauqueur