Feedback wanted — new quality_score field on Mapillary images and sequences

We’ve rolled out a new quality_score field on Mapillary images and sequences. We would love feedback from folks that consume data from Mapillary APIs.
It’s a float in the range [0, 1] representing the predicted visual quality of an image or a sequence — higher is better. The intent is to make it easy to detect blurry, dark, or otherwise low quality captures.

How to query it — images (Graph API)

Add quality_score to the fields= parameter on any image-returning endpoint:

GET https://graph.mapillary.com/{image_id}?fields=id,quality_score&access_token=MLY|

{
“id”: “2140279232804782”,
“quality_score”: 0.704
}

Image endpoints that expose it:

  • GET /{image_id} — single image
  • GET /images?bbox=… — bulk by bounding box
  • GET /images?sequence_ids=… — images in a sequence
  • GET /images?organization_id=… — images from an organization
  • GET /{map_feature_id}/images — images depicting a map feature

Combine with other fields as usual, e.g. fields=id,quality_score,captured_at,geometry.

How to query it — sequences (vector tiles)

There is no Graph API entity endpoint for a sequence’s quality_score — it lives only on the vector tile sequence layer, as the average of per-image scores in that sequence (unscored images excluded). Fetch the relevant coverage tile and read the field off each sequence feature:

Quick Python sample (using mapbox_vector_tile):

import requests, mapbox_vector_tile
raw = requests.get(
https://tiles.mapillary.com/maps/vtp/mly1_public/2/14/8801/5374”,
params={“access_token”: “MLY|…”},
).content
tile = mapbox_vector_tile.decode(raw)
for f in tile[“sequence”][“features”]:
print(f[“properties”][“id”], f[“properties”].get(“quality_score”))

Not included: the overview layer (zoom 0-5) on either coverage tileset, and the map-feature tilesets (mly_map_feature_point, mly_map_feature_traffic_sign) — those are dedup objects, not images.

Notes

  • Type: nullable float in [0, 1]
  • Images that haven’t been scored yet may come back as either 0 or null — handle both
  • On the sequence tile layer the field is absent when none of the images in the sequence have a score yet

Things we’d like to hear about:

  • Does the quality score values make sense ?
  • Any endpoints where you expected to see it but don’t?
  • Anything else — bugs, gaps etc

Drop your thoughts :slight_smile:

6 Likes

If they can do that, they can do a recount.

This announcement sounds encouraging! :+1:
@caglarpmeta @nikola For easier access and effective feedback, could you please add the quality score to the “Image details” panel? You can mark it as (Beta) in parenthesis. Thank you.

If you can, please also document the score’s weighting in Mapillary Help: What the score is composed of and each component weight. Furthermore, in order to avoid any potential confusion to contributors, you can also note in the docs that the quality score is not a grade for contributions or contributors but an indicator for machine digestibility and data extractability.

3 Likes

Oh wow, thank you for implementing this! Will try out soon :3

1 Like

I agree. Once the quality parameter has a good implementation, you could also filter by quality score on the Mapillary overview map

2 Likes

thank you for your feedback. making the score visible in the UI is in the roadmap, however without any associated dates :slight_smile: The score is a purely machine learned value with no weights that, us humans can understand.

2 Likes

For anyone out there, you can test this in the Mapillary Explorer Demo by toggling the Quality radio button on the right side of the Info panel. It filters coverage based on sequence quality, replacing the default Mapillary coverage tiles with quality-filtered coverage. This feature works at zoom level 7.6 (You can track the current zoom level using the value displayed next to “Z” in the top-right corner).

I thought it would be worth mentioning here in case this feedback is useful. Cheers.

3 Likes

That’s an awesome tool! Thanks. And “Street Gap” that it links to, will that calculate a route for a Mapillary capture? Did you make that too?

Hi dear @wa_wheatbelt, thank you so much for your valuable feedback. The Street Gap tool was developed by Ryan Lopez, and I’ve added credit to him, which appears when you hover over it on desktop(I’ve also included a mention in the project’s release documentation). I personally admire him and his contributions to open source and the community. I really liked his project and wanted to incorporate it into the Analyze section of Mapillary Explorer.

1 Like

Hi,

my tunnel traversals — which follow the actual path of the tunnel, including curves — are receiving significantly lower quality scores with the new assessment.

I suspect this is mainly due to lighting conditions. Even with good illumination, tunnel footage cannot naturally match the quality of outdoor shots in daylight.

Would it be possible to adjust the model for such environments or add special handling for tunnels?

Thanks!

Cool. Thanks for that information, @sukruburakcetin. I only had a quick look on my phone. I’ll have a better look later on a desktop. :grin:

1 Like

Hi,
thank you for the feedback, yes this is expected due to light conditions and blur that occurs as a result of it. In general, the model favors having crisp images with visible features, in a tunnel it is usually not the case.
Also please remember that the model is not perfect, and it will make mistakes, occasionally giving low scores to high quality images.

Please note that the score is just a guidance and it is not used to filter out or in any other way “penalize” images at the moment.

Could you give me a few examples ?

There is a “Report image”→“The image is low quality” function in the web app. Have you been using these reports on the model?

@boris Btw, image reporting is horrible :scream: to use. 6 :red_exclamation_mark: clicks to report one image. Plus, there is no way to report a whole sequence. Actual effects of these reports are also rather mixed at best.

1 Like

nope, we did not use the reported low quality images.

This is unfortunate, indeed. :face_with_diagonal_mouth:

It could be used to give the notoriously bad photographers a good camera.

Or can we denounce them ?

1 Like

I am not sure it could work this way. :face_with_hand_over_mouth: Every camera (or picture) is only as good as the photographer behind it. The same goes for pencils and writers. :wink: Though, I admit that would the platform drop less than 0.5 quality score images over night, it may be sobering or cathartic to some people. :grin:

The low quality image reporting function was meant for cleaning up the platform and future AI training material. Thus, I am a bit disappointed to read that it has not and probably will not be used for AI training after all. :person_shrugging: The image quality score and the reporting function are not meant for shaming, judging, or grading people but tools to keep the database clean, which is desperately needed for every openly publicly (non-gated) fed database.

I am thinking of someone who bought a cheap rubbish camera long ago and continues to use it.

And someone who really merits shaming is the one in a caravan who passes my region every year and just documents his voyage with a cheap rubbish camera.

Don’t insult my 10-year-old smartphone :smiling_face_with_tear: :rofl: