Hello. What EXIF tags should be present in the uploaded images so that Mapillary correctly recognizes/calculates the image’s FOV and removes distortion, for example your typical fisheye from action cameras? Or, more generally, what information does Mapillary use for image lens correction? I am manually processing my photos from videos and writing the EXIF data “myself”, so I cannot use what would normally be camera output photos.
The Exif standard does not specify tags to describe lens geometry or projection parameters, except for focal length (see
FocalLengthIn35mmFilm tags). AFAIK, there are some vendor specific extensions which try to fill in this gap. However, they are proprietary and quite obscure.
Although Exif 2.32 has introduced new
LensSerialNumber tags, yet at best they can only serve as references to a list of makes and models paired with lens geometry and projection parameters. So far, to the best of my knowledge, this kind of digital list for making easy automated lens lookups does not exist, and even if it does then Mapillary probably would have to pay some considerable commercial license fee to use it. Most devices do not output Exif 2.32
Lens* tags anyway, so no need to bother. Furthermore, Mapillary does not parse nor honor Exif 2.32 lens tags anyway.
The way how Mapillary selects lenses and projections, more specifically how mapillary-js (the front end) and OpenSfM (the back end) do it, is by looking up static lens geometry and projection parameter values from a list of predefined device names, identified by the
Model tags. In other words, if you want your images to be processed with certain projection parameters then you will have to set the
Model tags of your images to the make and model of a particular device, for example like
HERO 5, which always assumes that device’s specific default “fish-eye” lens and projection. If the
Model tag values do not match a device on the list, are empty, or these tags are not available then Mapillary assumes linear projection and a focal ratio of 0.85 (projection plane width / focal length).
Thanks for the answer!
Interesting… I actually wrote both those tags with make
GoPro and model
HERO8 Black, exactly like they would appear set by the camera (spaces and all). For example, this photo. It does say “captured with GoPro HERO8 Black” and I can see there is correction applied, so it identified the camera. But the correction is not correct.
I guess I’m confused because the camera has multiple FOV settings. It sounds like there’s no way to tell Mapillary exactly (or even approximately) which setting the camera used.
I looked through the repository a bit, but I am not sure where the data for each camera model is stored. (Searching doesn’t really turn anything up.)
Is it possible that this particular make/model entry is wrong (even before we consider multiple FOV options)? I feel like someone would have noticed. I see other people with GoPro 7 and 9 and 10 and they seem correct. I found someone with HERO8 Black and their photo has the same incorrect-ish correction applied.
Thanks, I skimmed over those before, but they didn’t have all other GoPro models I could compare to, like the 7, 9, or 10 (so I assumed there must be more data). So does that mean cameras like HERO9 are using default values, which happen to be “more correct” than the explicit values for my model?
I guess if I can confirm that these values do not produce the right correction for my model, I can file that as a bug for the repo.
I tried to highlight this problem some time ago in the topic:
but it died without effect. I guess its more mapillary problem than mine. With bad correction all triangulation and 3d poind cloud gets lots of bad data…
Im also using gopro 8 black. The correction went bad some time ago, probably with some kind of update. I see some recent sequences from gopro 8 that are still OK now. I dont know why mine are bad. So I just delete camera exif tags and now mapillary is determining the correctiom itself.
Ah, interesting. My question was not really related to the exact camera, I just wanted to know how I can set the FOV for photos. It turned out that Mapillary uses the camera model for this and hard-coded values. And it also just so happens that this specific HERO8 Black seems to have the wrong values.
I guess I will comment on that thread for the camera issue.
I would still like to be able to set a custom FOV though because GoPro has multiple FOV settings. I guess if the default correction is close enough, then it’s probably good enough. But it will still be slightly wrong. For now, it’s Mapillary’s problem then. I guess they can eventually apply some AI and figure out the correct FOV of a sequence that way.
In my experience mapillary automatically detects photo distortion.
This photo was taken with a muson action camera. Note the white pipe on the left. Pictures taken with this inexpensive camera are greatly distorted.
However, three pictures later in the same sequence the distortion is corrected. Drag the photo and note the border with the black background. The border is wavy and distorted because the lens distortion has been corrected.
I have not added any special EXIF to these photos.
Yes. In my observations also the algorithm tries to determine lens correction and camera maker is not always respected. As we can see the recognition is far from perfect. It depends on the surroudings. If there are straight lines like street pooles or buildings it tends to apply correction, but in the wilderness it leaves default correction from the maker or do nothing.
Right, OpenSfM integrates a somewhat simple projection parameters detection algorithm, so it is definitely not perfect and has its weaknesses, especially when the images being matched are lacking straight artifacts. And, I agree with @HellPhoto that it would be better to have a standardized way to add projection parameters metadata to images. Unfortunately, this is beyond feasibility for the foreseeable future. However, what you can do today is post a pull request on the
camera_calibration.yaml file with the correct default projection parameters for your camera. I am sure Mapillary would be happy to merge your PR.
Btw, you could potentially also solve the issue with selectable projection modes by distinguishing projection modes by appending a suffix in the
Model tag, e.g.
HERO8 Black (fish-eye) or
HERO8 Black (linear) etc. And although one drawback to this solution would be that contributors would have to process the
Model tag with the correct suffix before upload, at least you would actually have that option to sort of add projection metadata to your images. Another route might be for Mapillary to extend and document the proprietary Mapillary JSON payload in
ImageDescription with projection metadata (but I would not bet on that).
@asturksever I would like to submit this as a feature request. In fact, you could add an option to the Mapillary command line tools and the Desktop Uploader, where one could either select projection parameters based on
Model or specify projection parameters manually. Since this would be an optional proprietary extension anyway, most users who do not need to or do not know how to bother about projection parameters would not experience any additional burden (because it would be an expert option). On the other hand, those who need it because they have some sort of special or new equipment could at least provide proper metadata with their imagery. So, I think that adding projection parameters to the Mapillary JSON payload may be a viable option. Please note, that Google’s Pano XMP extension already/also supports selecting predefined sets of projection parameters in the
ProjectionType element. So, it is not like we are the first to encounter this issue.