Global Verification Challenge V2 - List of classes

Hi all,

As announced in this blog post, we have just launched a new verification challenge where we aim to reach 1,000,000 verifications.

In this post we describe what each of the objects are so that you will have an easier time confirming or rejecting the detection in each image.

Banner: Typically a flag, made of textile or plastic.

Catch basin: Sewage water drainage, typically with a grid on top or inlet of a curb.

Junction box: An electrical junction box can contain junctions of electric wires and/or cables.

Bird: Includes birds on the ground.

Ground animal: Animals which move on the ground.

Crosswalk -plain: Region of a road which is meant to cross the street but not marked with a zebra pattern, instead two parallel lines indicate the crosswalk. The road surface is marked as plain crosswalk, the two lines indicating the border are marked as general markings.

Marking: crosswalk zebra: Crosswalk indicated by a zebra pattern.

Bench: A long seat for several people, typically made of wood or stone (Oxford).

Bike rack: A parking space for bicycles with space for the bicycle owner to attach a lock.

CCTV basin: Surveillance camera mounted in public space.

Fire hydrant: A fitting in a street or other public place with a nozzle by which a fire hose may be attached to a water main (Oxford).

Mailbox: A receptacle for sending or receiving mail. These can be residential or public.

Manhole: Covered opening in a paved area to access underground space or system. Must be big enough to fit a person, may be square or round.

Parking meter: A machine next to a parking space in a street, into which the driver puts money so as to be authorized to park the vehicle for a particular length of time (Oxford).

Phone booth: A public call box typically equipped with a payphone.

Streetlight: A light illuminating a road, typically mounted on a tall post (Oxford).

Pole: (support) Thin and elongated, typically vertically oriented poles, e.g. sign or traffic light poles. Does not include objects mounted on the pole. Poles holding only street lights should also be labeled as pole.

Traffic sign frame: Frame construction holding large signs, where frame can be made of round or square poles. Can be typically found on highways.

Utility pole: Utility poles are supposed to carry electrical wires/cables, could be used to mount mobile phone network receivers, and are used in case both, cables and street lights are mounted

Object: traffic light: Traffic lights without poles in different orientations (upright, horizontal, side, back, front) and for all types of traffic participants, e.g. regular traffic light, bus traffic light, train traffic light. If the type can be determined but does not match any of the defined categories

(e.g. traffic lights for trains), then traffic light - other is used. If the orientation or type of traffic light cannot be determined, traffic light - ambiguous is used.

Traffic sign (back): Back side of traffic signs.

Traffic sign (front): Signs with the purpose of conveying information to drivers/cyclists/pedestrians, e.g. traffic signs or warning reflector posts. If the type of a frontviewed traffic sign cannot be determined, e.g. because of unknown language, trafficsign - unknown is used as fallback.

Trash can: A vessel to distribute personal or public waste.

Bicycle: Includes bicycles without the cyclist or possibly other passengers. The cyclist and passengers receive label cyclist.

Boat: Vehicle suitable for riding/sailing on water.

Bus: Includes buses that are intended for 9+ persons for public or long distance transport.

Car: This includes cars, jeeps, SUVs, vans with a continuous body shape (i.e. the driver cabin and cargo compartment are one).

Caravan: Vehicles that (appear to) contain living quarters. This also includes trailers that are used for living and has priority over the trailer class.

Motorcycle: This includes motorcycles, mopeds without the driver or other passengers. Both driver and passengers receive the label motorcyclist.

Vehicle on rails: All vehicles moving on rails, e.g. trams, trains.

Other vehicles: Fallback category for vehicles not explicitly defined otherwise in meta category level vehicle.

Trailer: Trailers that can be attached to any vehicle, but excludes trailers attached to trucks. The latter belong to class truck.

Truck: This includes trucks, vans with a body that is separate from the driver cabin, pickup trucks, as well as their attached trailers.

Wheeled slow vehicle: Covers all slow moving wheeled objects like wheelchairs, mobilityscooters, scooters, prams or rickshaw. All of which can potentially have riders (other rider) sitting on them, similar to cycles are driven by cyclists.

Seriously, how can a single user have a score of 12083 in the class ‘Car’ ? It took me one hour to scan 200 pictures with a sufficient degree of certainty. If I press ‘Ok’ 20000 times I may get same score (I think?), how knows the system these 12083 score is a serious attempt to verification?

@eneerhut, something funny is going on with the counter. I went through a verification task, judging 100 photos (the blue counter showing “(100)”). Then I went back to the main screen and opened the task again, and the counter was on 99. Did another one to get to 100 again, and next time it was 99 again…

It’s a fair question. We are assessing verifications provided so if there are any participants simply pressing thumbs up or down then they will be removed from the rankings and their contributions disregarded in the training data.

Hmm that’s odd. Thanks for reporting. Investigating this bug and will let you know when we have an answer.

As I have currently the highest score in “car”, I thought some explanation is needed. First, cars are in my opinion one of the easiest objects to verify, as they have very distinct shapes and details. And trying to correctly guess cars’ make and model from a distance is something I have done as long as I remember.
Also, lot of them are in the cities and photographed at slow speeds, so they are rather clear most of the time. When something else is identified as a car, it is usually a truck, bus, or a caravan - all also quite distinctive objects. And when the image is blurry or dark, then I just skip them.

And most importantly - I have 8 month old twins down with a cold, so there is lot of time spent in a dimly lit room with one hand patting a baby and other verifying objects.

2 Likes

Thanks for responding @jorrarro. Very impressive score which naturally raised questions. But fantastic effort!

We have identified an issue that might have been causing this and are working on a fix.

Verifications are only sent to the database after:

  1. 5 seconds has passed or
  2. You verify another image.

This allows us to provide the undo functionality. A fix will ensure verifications are still saved if the browser window is closed or the page refreshed.

Here is the latest top 10 score list.

image

Final goal of 1,000,000 verifications seems not be reached before Oct 12, right? Many classes show a 1% progress bar. Any plans for a new end date?

Correct! Looks like we will extend by two weeks, but we’re just putting a few scores together so we can announce properly and give a proper score update.

If I do a task long enough I get the same images again. Choosing the next image seems to be very random and images already done by one person seem to be not excluded. So if one person would always press yes or always presses no, then the system still would accept these repeated verifications i guess. But maybe it just looks to me this way.
Personally I try very hard to choose the correct answer, even if that means I am not very fast. So even if I would be the only one to see and rate one image repeatedly it is hopefully correct.

Just to be sure, humans do not count as ground animals? There is no category for humans and they could count as ground animals from an abstract view. And they are categorized as ground animals by the system. So I am not sure what to answer.

I have also seen some images repeating, but I’m not sure if they are actually the same image or just a duplicate from same location. Using time-based capturing generates a lot of duplicates while standing. But as far as I can tell, the score does not seem to increase after clicking on duplicates. So there are probably some checks behind the scenes.

Humans are not part of the ‘ground animal’ category. I have to agree it is sometimes a bit unclear what the precise definition is of these categories.

@eneerhut is this challenge now over? This blog post (Learning with Verification: Improving Object Recognition with the Community’s Input - The Mapillary Blog) kinda implies it is.

Anyway, final leaderboard would be nice.

1 Like

That’s a very good question. I’m sorry that has taken your prompt for us to do.
I’m not actually sure what happened here so I will make sure we get a final leaderboard generated and prizes distributed.

I’ll share the leaderboard here when I have it.

Two weeks later - any news?

Following on from our email correspondence, final results have been updated on the blog:

Top 3 results in the 2nd Global Verification challenge