Quality score: unexpected results

Sometimes the quality score assigned is not what I’d expect - will list examples in this thread.

https://www.mapillary.com/map/im/ZBT1JPld7blKMI8MbB73dw - while not a bad image, the right hand side is a bit blurry. I would not have expected a score of 5.

Thanks for making a note of this @Richlv! If you can list many examples it should be helpful for our review.

Thanks, couldn’t figure out the best place to list those before and did not make notes - but will add any I spot here from now on.

Is it possible to pay someone for that work ?
Examples are very easy to find.

https://www.mapillary.com/map/im/d6F2AQsCkCHQTStvxW6lj0 - would score higher than 3
https://www.mapillary.com/map/im/2yxyLsZWzM6s4kIuyl1rH3 - would score higher than 2
https://www.mapillary.com/map/im/YHQbc33eNeqHNao1M2P0cR - would score higher than 2

Same with many (most) other pictures in the sequences in this area/time.

Contrast the images above with https://www.mapillary.com/map/im/GTplJeUtqvf9CikVM7HXrw - I’d even mark this one as 4 instead of 5.

https://www.mapillary.com/map/im/2gl0Zlit6Ox5y2noZOr2Cw seems similar, but with a lot of hood in the pic - but even that one got 4.

https://www.mapillary.com/map/im/uCfqMjhcLzrR7EX2F4iJVg - would score lower than 5.

There’s some useful building detail, but in general I’d probably score this one at 3, given how much the bus is obscuring :slight_smile:

Would appreciate if the quality slider were two-sided, so that one can exclude good quality pics, see where the lower quality sequences are, then include those in a journey.

1 Like

That would include areas where there are both good and bad quality pics, though.
Wouldn’t just excluding low quality pics identify blank areas?

Would rate https://www.mapillary.com/map/im/8xuEdveUucY89_jHAvB2Bw higher than 2.

Yeah, I am with @Richlv on that. Imho most of the problems come from the fact that the algorithm’s parameters are skewed against snow. Like if snow was generally a bad thing? At least this is what we can read in the announcement blog post. Just think about polar or subarctic regions of the world where in some places snow is present all year round. Sure, snow covers surfaces and if you are interested in mapping, say, road surfaces and you cannot see the road surface because of snow then this is not the fault of the image author or the image itself. It is just how nature works. The same is true about any other natural atmospheric phenomena, like rain, sand storm, or dust etc. Image quality should not measure what we can see or make out in the image but should be first and foremost a metric of photometric quality, like sharpness/blurriness, grain, illumination, amount of pure black areas etc.

However, we do not want to have photometrically perfect photos of “some piece of pavement only” or “interiors of a bag“ either. So, the algorithm should rather also factor-in just one combined street-level scenery parameter, answering the question “Is this a street-level image?”, since this is what Mapillary is all about (at least for now). Which I guess was the initial idea but I am quite sure that having only snow or any other particular atmospheric phenomenon in the equation is not the way to proceed with that. Atmospheric phenomena should not impose a penalty on the quality score of an image.

Besides, I am a bit confused why the scale is 0, 1, 2, 3, 4, and 5? What is wrong with [0.0 to 1.0]? I mean Amazon stars may work fine for product reviews but not for images.