SfM lat/lon not in API & Lat/lon question

[eesger]

SfM lat/lon not in API
With the API and the Mapillary JS a Mapillary.Viewer node has two lat/lon pairs

1. the “Structure from Motion” lat/lon (default!)
2. the original lat/lon

I red up on the SfM system (cool!) BUT

The information I get via json calls like:
$.getJSON(“https://a.mapillary.com/v3/images?client_id=“+client_ID+”&“+“closeto=”+lon+”,“+lat+”&radius=100”,…

The lat/lon information are all “original” lat/lon values, NOT the SfM lat/lon values!

Wouldn’t it be much, much more logical to use the SfM values here also?
(or am I doing something wrong?)
example:
http://geoarchief.nl/mapillery-test.html
(shows original location on map: light blue and SfM location: dark blue)

http://geoarchief.nl/mapillary.html
(the location information from the API are the dots in view, the link looks at the dot that should be below)…

Now that I am on a roll with this post. SfM repositioning the lat/lon: cool!!
But if you guys are that smart to calculate the right lat/lon, why not automaticaly correct the horizon and compass angle along with it? The last link in this post clearly shows that the compass angle is a bit off (and that my camera isn’t that good… I know… but I am in a test phase here, planing on buying a good one, but first I want the basics to go right )

Lat/lon question
When the “Mapillary system” is that smart with re-positioning… is it necessary to spend (a lot of) time trying to have the EXIF data of the images set as good as possible?

[eesger]

Or… if this is “by design” (would like to know why )

Would it be possible to expand the API adding the SfM lat/lon?

[chrisbeddow]

Hello!

This is by design, because the response is a GeoJSON we want to maintain the original, authentic longitude and latitude. This gives the data integrity, because the computed ones, while useful for SfM aren’t real data about the original, user-uploaded images. We don’t want to provide modified data from us, but rather the exact data about the image the user owns.

However, in MapillaryJS you can get node.latLon but also node.originalLatLon which is the same typically, and node.computedLatLon. This also work with camera angle: node.ca, node.originalCa, node.computedCa. Feel free to use any of these in your custom apps. We don’t put this into the v3/images API, however.

Really, when saying “computed” we are not adjusting the images to a “true” position, but instead a “predicted” position. This can sometimes appear quite accurate, and sometimes not at all. Overall when it comes to images we try to maintain the integrity of the data the user provides as far as positional display on the map, while the SfM adjusted values are a separate arena–these are used to figure out which images you can navigate into from the current one, as well as to attempt to geolocate extracted map features in relation to the context around them.

The better the input EXIF data, the better the navigational/viewing experience in MapillaryJS, and the better the output data. Only the user can be relied upon to provide true input, with a GPS device and site survey for real context, so we are trusting the human first, and machine only after the fact, rather than machine first. We always need that original EXIF data in order to make a next step. Reliability of both humans and the machines varies, but the human who uploads is the original source, and what we always display by default.

[eesger]

Thank you for the answer!

The positioning of the image on the canvas is based upon SfM latlon. I think I understand why; the transitions between the images most likely is better.

Now I position the dots in the mapillary.html from the API, and so they are the position where the user ‘says’ it is. These dots help me a lot navigating around the 360-images (thinking of placing some smaller dot’s in between to get a sort of line, a function to draw a line would be even better though ).
But when moved there the image is positioned according to the SfM not the original latlon.

I couldn’t think of a way to “bulk receive” SfM lat/lon locations, but I’m quite convinced the SfM data will improve the overall experience… (hence my feature request)

Question: maybe the “reverse” works also, is it possible to position the center of the image on the original latlon?

I’m glad I caught your attention, could you write down your thoughts about:
I have a plan to create a “version 2” for the web-uploader (or at least a good assistant) But that will (eventually…) result in “edited / smoothed” GPX/EXIF info.
Reading your post, I could conclude that Mapillary would rather have “raw location data”… but if “the dots” are relatively poorly positioned I’dd be better off with my future smoothed out version.

[chrisbeddow]

Hi @eesger, It’s not possible to bulk receive those yet. I’ll check on it, but I know that it would mean expanding our API a lot. I also don’t see a way to center the image on the original location–essentially the only thing I can think of is to get the node.originalLatLon, node.computedLatLon, calculate the offset, then apply that to all visible points that are rendered (offset them the same ration). This isn’t a consistent offset though, but this could help.

Ideally for upload we’d allow you to do some smoothing of your own prior to upload, in a way that still retains the raw coordinates but also has your updated ones. This is what happens when you spatially edit an image on the web platform, after upload. Overall the point is that it’s good to have a papertrail of where the image was said to be, where it moved to manually, and where it was computed.

I’ve definitely used Geosetter to improve coordinates before upload, however. In a dense area of buildings and narrow pedestrian streets, such as Stone Town in Zanzibar, the GPS was very poor, but it was possible to use a system of notes on paper, memory of intersections, and one-by-one movements to put the imagery on track. Mapillary

I have had an idea of providing a linestring GeoJSON as the correct path, distributing the images evenly along that line, then writing the new coordinates to the EXIF, then taking to Mapillary to upload. This would be very helpful in areas where I know the sequence started and stopped at two very specific locations, and wanted to spatially distribute them evenly on a route between those. If this is what you have in mind, I think it’s a great idea. This should result probably in a closer match between original and computed coordinates.

[eesger]

• Use the offset of one as a template for the rest.
  Nice idea. But no. Even in my relatively small set of images the difference between the SfM location and the original is quite erratic. And more importantly the SfM location realy does appear to be better, so that feels like doing a step backwards…

• Raw vs computed (SfM)
  I find it quite impressive that Mapillary can calculate a better geo location based solely on the image (starting with “the rough” lat/lon given). With the small set of images I took (a few thousand) I could only find some that were less well positioned (I have this feeling that altering the forest path in OSM would improve it ). My point in essence is: when you’re that good at better calculating the location, use it! Not a little, go all the way: Reposition all the dots in Mapillary towards the calculated positions! Leave the raw data within the images (as it is now) but simply use the computed lat/lon everywhere!
  I don’t know if it is possible do a local test at the office, but the results might just be very pleasing?

• Example:
  Rats, your LG360 does a better job at stitching the two images together then mine
  Odd, clicking the arrows just wants to skip this corner? (go back and forth, you’ll see what I mean)

• I’ve had an idea:
  You and me both, that is exactly what I mean. But in my “prefect workflow plan” this would mean that the original GPS data would be completely overwritten with “OSM smoothed data”. Resulting in the loss of the original GPS data (which would only be used as a bases). But when SfM is there anyway it could also mean that I’m wanting to build something that isn’t really needed because SfM can fix that (and the extra iteration would be a waste of time) … but because I can’t use that data the way I’dd like to… and the circle is complete.

Maybe I’m being to much of a perfectionist here, but I’m trying to get to the “the perfect workflow”. Perfect for my idea, but also fitting inside the Mapillary system as perfect as possible. When I know what would be ideal, I can understand where to do a trade-off and get practical.

So I think the question is: how important is “the paper trail” to Mapillary?’
And/or: Any chance of Mapillary “switching” towards SfM lat/lon altogether?

Maybe I’m just making a fuss over something that’s not that important and the edited/appended/styled GPX data is more valuable to Mapillary?

[chrisbeddow]

Yes, being good at better calculations just highly varies based on the region. We definitely do not use the OSM path as a reference–this would dramatically change the licensing of Mapillary based on OSM’s ODBl license, so that has no effect. If you’re using OSM as the user before upload to adjust/“smooth” the data, that also could be questionable (I’ll have to ask). Really, in the case of GPS measurements, when you know the GPS is wrong (like very wrong), and either manually readjusting or even using a basemap as reference, you may be more correct, but it still doesn’t make the GPS coordinates actually true or precise. So in either case you have wrong data, one just looks closer to truth. What is most important is the position of the images to one another, based on the point cloud we reconstruct from the image pixels–this can be very accurate for relative position of the photos in your sequence to one another, and to other photos in the sequence, but the key is we have to scale this method globally. Nothing on our side can assume that just because the computed coordinates look better in one town or city park means we can assume it would be better worldwide. So we stick with original coordinates when it comes to the map, same as any geospatial interface like Strava, OSMAnd, etc that is displaying user created data.

Your final question is a good one: if we have a change of switching altogether to SfM lat/lon. I will ask! But I think the problem is that globally it just switches one set of problematic data for another set. Probably the idea is that if the algorithm becomes intelligent enough to truly correct everything with a massive success rate, then it could make sense.

Otherwise, I think the missing piece for you is access to the SfM coordinates via the API, which I’ll also ask about!

[eesger]

@chrisbeddow

So in either case you have wrong data, one just looks closer to truth

I don’t expect to find absolute truth in my lifetime, so I don’t mind settling for “closer to”, that’ll be a step in the right direction!

Anxiously awaiting the answers to the “I’ll ask” answers!

Best Regards,

Eesger

[chrisbeddow]

@eesger So I think at this time we don’t see a reason or plan to switch to computed SfM locations as the default map display, because we don’t expect it to be verifiably better at all times, globally. GPS is ground truth so will stay as primary display.

For the API, we are certainly considering adding SfM coordinates as an option, but it has some complexity to it as we need to constantly recalculate these if a user is modifying image locations, whereas right now because it’s not exposed we don’t remodify it much. In short, it would be tied to a larger change in our data pipeline that we won’t do in a whim, but that probably makes sense in the long run. Can’t say when, but we’ll watch for this to become more necessary.

[eesger]

Take a look at my “version two fiddle”:

https://www.geoarchief.nl/mapillary2.html?key=ykFLqgavvUc1Be_9wU9OCw&zoom=.1

press play and see the dots get re-positioned to the SfM lat/lon. The repositioning is I think better then my manual guess

It bugged me that the “white dots” didn’t align so well. Then I figured that mapillary(JS) doesn’t use altitude! That would improve things quite a bit I think!

When coding I saw that there are ‘rotation’ values in nodes, three of them (Roll, Pitch, and Yaw?) Are they used in mapillary(JS) to straighten them out? (if so, there is room for improvement there)

[edit]
CORRECTION:
I just saw an unprocessed part of a sequence of mine… ehm, I thought I held the rod more stable. So I stand corrected, the straightening out part goes rather well!
Could it be that this is not corrected in older photo’s? (Saw some really crocked ones…, cand find them though)
[/edit]

[edit2]
I altered the re-positioning a bit, there are two values to work with:

• ‘latlon’ (SfM calculated lat/lon)
• ‘Original latlon’ (the lat/lon supplied by the uploader)

In other uploads I’m doing (in and around Bonnal, France) I did conclude there is value in the original lat/lon also, so now I have set the “dot” position to:
2/3 latlon and 1/3 ‘Original latlon’, this is (optically) I think the best result

PS: still would really, really love the usage of the Altitude I supplied in the EXIF data in Mapillary
[/edit2]

[edit3]
Sorry, I keep appending
The altitude of the cave would need to be calculated (since I could not register GPX data of it). But with SfM I think that would be possible also… to calculate the difference in Altitude?
If you would want to do it step by step, this sequence contains Altitude data in the EXIF I published.
[/edit3]

PS: lucky for me, The Netherlands are mostly “flat”

[eesger]

AH!
To prevent pollution of the “unusual sightings” thread, I’ll continue the “off topic” here

@ligfietser mentioned there:
http://mapillary.trydiscourse.com/t/unusual-sightings-in-mapillary-photos/326/103

This fits my previous post! I saw unprocesed images via MapillaryJS where I could very well see that the pitch, roll and yaw are corrected (with SfM?)

The insta360 already does some corrections, but as he pointed out they are not always as good as one would hope… but (with the flat Netherlands in mind ) one would expect Mapallary’s SfM to do the rest of the correction?

[eesger]

@chrisbeddow

Hi @eesger, It’s not possible to bulk receive those yet.

Hi @chrisbeddow, I don’t seem to be able to find how to retrieve the (computed) lat/lon of an image key (without going to that node with ‘moveToKey’). Can you help me out?

[eesger]

Well… sort fo to prove my point I “hacked the system a bit”…

take a look at my latest iteration of my viewer:

https://www.geoarchief.nl/mapillary3.html?key=dOezWfZrQajwE-i0zDCsjw

The dots start at the “given Lat/Lon” and close to the point of view get updated…

[edit]
I think this slows down the browser quite a bit and I fear I put weight on bandwith usage of the Mapillary system… but I don’t know a better way to do it …
I now do a “move to key” in a hidden mly-instance to get the needed info… I know, naughty me

[edit2]
This one looks pretty cool too;)
https://www.geoarchief.nl/mapillary3.html?key=NBXy9HoqGuTINofud6lnXg&x=.97&y=.47&zoom=0

[chrisbeddow]

This is really cool! I think there probably could be some interesting uses for this, I wonder… let me know if it evolves into something

[eesger]

Thanks!

But I’dd love it if you could tell me a way to perform a request to the Mapillary platform where I could just get a JSON/XML/whatever with the SfM lat/lon!

As mentioned in the first edit, I get the distinct impression that with the way I solved the question now, that I am receiving the image itself also! I have dug around in my browser debugger, but I cant find out how the MapillaryJS gets the computed lat/lon.

Can you help me out with that @chrisbeddow?

I.e. something like:
https://a.mapillary.com/v3/images?client_id=[id]&image_keys=[img_key]
but then with the “other” lat/lon

[chrisbeddow]

You should try the API with v3/images_computed instead of v3/images, I think this will do it

[eesger]

Alas, I tried it.

Best use an example:
https://a.mapillary.com/v3/images?client_id=[key]&image_keys=aInh5DFsqKevCzDCHamAJw
gives: “coordinates”:[6.0377576,52.943282]

https://a.mapillary.com/v3/images_computed?client_id=[key]&image_keys=aInh5DFsqKevCzDCHamAJw
gives: “coordinates”:[6.0377576,52.943282]

They are exactly the same

The information I get from the ‘node’ upon:
mly.on(Mapillary.Viewer.nodechanged, function (node) {console.log(node);}
is (among other things):

originalLatLon: Object (same as the two above, but (needlessly) more accurate)
- lat: 52.9432819760286
- lon: 6.037757558917889

computedLatLon: Object (yes, yes, this one!!)
- lat: 52.943317209696055
- lon: 6.037790997140558

latLon: Object (same as above)
- lat: 52.943317209696055
- lon: 6.037790997140558

If you could alter the coordinates given by images_computed, so that gives me the ‘computedLatLon’ It would be perfect (which does sound logical to quote my pointy eared friend )

[eesger]

Here a good example of the power of SfM:
https://www.geoarchief.nl/mapillary4.html?key=1Yn2uE1cOyVOdFQVTIyrVg&x=.97&y=.47&zoom=0

Here that region in the Mapillary web app:

Mapillary

And also the room for improvement. As you can see, it starts out impressive. The GPS track was not that good at that moment… but after about a hundred meters SfM “decides” that the original Lat/lon prevails over that what it was doing right… Why? The two dozen images (more to the North) are the rest of the road connecting to the other track more to the North East.

[eesger]

@jhoney, what do your links have to do with the reply? What does bluestacks or kodi have to do with this topic?! And those are not official sites, are you trying spam malware sites?

Further, yes… but did you take a look at my link? SfM needs the “original lan/lon”, naturaly, but the GPS lat/lon that can be obtained via a std. smartphone has a resolution of about 10 to 50 meters. I now have a semi) professional GPS receiver, but still at best 2.5 meters… it is a better base, but when SfM can improve (and it does, see the link above!)

[chrisbeddow]

@eesger some updates here:

2 parameters are added to /v3/images :

1. includes=cca which returns cca in the properties as computed CA
2. computed_coordinates returns computed lnglat replacing the original image lnglat

For example: https://a.mapillary.com/v3/images?client_id=xyz123&computed_coordinates&includes=cca

^ this will include both the computed camera angle (as well as original), and the computed coordinates will replace the original coordinates (so only one set, not two sets like the camera angle gets)