Uploading cubemap images

Contributing and equipment Command line tools
1

Hi!

I have a large street imagery dataset that consists of cubemap images per datapoint (six separate images pointing front, back, left, right, up, down).

I’d like to contribute these to Mapillary.

AI has suggested using mapillary_tools for upload and preparing the data by simply setting the exif properties GPSLatitude and GPSLatitude to the same location for each of the six images and setting exif property GPSImgDirection to 0 for the forward facing image, 90 for right facing, 180 for back facing, etc. According to AI the images would then be correctly stitched together by Mapillary to a panorama image.

Now I would like to verify this but I couldn’t find any documentation on the topic.

Could somebody confirm that this workflow is correct?

Many thanks and kind regards

Elias

2

You can upload them but they will end to be 6 separate sequences on the map. Is it what you expect?

3

No, what I would like is one sequence of panoramic images.

So you’re saying that in order to get a single sequence of panoramic images I need to merge the cubemap images of each datapoint into a panoramic image and then upload these?

4

Just try it.

5

I imagine Mapillary will process my images asynchronously before I can see the results on the map. Trial and error seems like a very inefficient approach to getting things right. Is there no documentation on which EXIF attributes need/should be set and their effect on processing? All the documentation I could find describes basic usage of mapillary_tools or the desktop client. I’m looking for something detailed like the Mapillary API docs but regarding data upload.

6

Someone please tell me if I am wrong, but the examples I have seen of multi camera views from the same origin do not “stitch” into one image, but can be step panned left and right on the GUI.

eg https://www.mapillary.com/app/user/giuliaz?lat=-34.550388041664&lng=146.40148499999998&z=17&pKey=173763368008195&focus=photo&trafficSign=all

I suspect that pre-processing down to one panoramic image per origin would be needed.

(IMO) Also worth a mention that end usage should also be a consideration. Object detection and stepping down a street adding OSM info manually do far better with a single camera direction view, so it may even be better to upload them as separate sequences.

7

I often take “panoramic” pictures with a normal camera.

Just swipe left or right.

I don’t remember what the panoramic mode in normal cameras (smartphones) results in.

8

Thanks for the example, that is looking good!

Any chance you could give me the EXIF data of such a set of images (that appear stitched together). I tried downloading the images you linked from Mapillary but the EXIF is gone.
I’d like to verify what properties need to be set in order to achieve the (almost) stitching together.

Building on what bob said, I’d rather not preprocess them into a single panoramic image because besides object detection doing worse I would also have to downscale the images (the panorama would have over 250 megapixels). I’d really like some opinion of the Mapillary team on this.

9

At minimum you need: GPSLatitude, GPSLongitude, DateTime.
Although DateTimeOriginal takes precedence over DateTime, DateTime is technically older than DateTimeOriginal, so that one can consider DateTime to be the lower minimum than DateTimeOriginal. The timestamps must exhibit strong monotonic growth.

It is best to also have GPSDateStamp, GPSTimeStamp for timezone support. Alternatively, you can also add OffsetTime* tags for timezone support. If available GPSDateStamp and GPSTimeStamp take precedence. You can also add SubSecTime* tags for a more granular time resolution.

It is also recommended to have a correct GPSImgDirection tag per image for search to work properly in the web app and the API. You can use the process command’s ‑‑offset_angle and ‑‑interpolate_directions options together in order to set image direction metadata fairly properly. Note that the ‑‑offset_angle option modifies the offset of either existing image direction metadata or relative to the next image in the sequence when used together with the ‑‑interpolate_directions option. ‑‑offset_angle does neither set absolute direction nor direction relative to a cube. Use extra care when tagging direction for the top and bottom images of your cube captures. Make sure you figure out where the geographic north pole is in these specific images.

In fact, you do not actually need to physically tag image files, in other words write Exif tags into image files, for everything to work because you can pass geo data from different sources to mapillary_tools, like a GPX file. mapillary_tools extracts this data into its own format and actually uses that on upload.

Just like @tao has hinted, the platform does not support cube map images, that is where all six sides are combined into one image file. You would want to upload separate sequences for each side. The platform does not actually do any stitching. It just reconstructs camera positions and the web viewer enables seamless viewing between camera positions for a pleasing experience.

1 Like
10

thank you so much for your detailed write-down. this is a great starting point for me to start working with my data!

11

Addendum: Technically, there is nothing that prevents you from putting all sides into one sequence. However, you would though need to make sure that all image timestamps exhibit strong monotonic growth, say at least by a microsecond. The platform would be able to digest such a sequences and properly reconstruct all camera positions. But, viewing such a sequence sequentially would make for a confusing experience. Some people may even get seasick viewing such a sequence.

@nikola Speaking of seasick, the web viewer had an option to disable transitions, which can also sometimes lead to seasickness. :thinking: I am wondering why it has been dropped?

12

It seems most things have been answered during my sleep! Just a few remarks then;

I use the tools to upload EXIF annotated images from two (cellphone) cameras that face in slightly different directions. The image saved has all the regular phone EXIF data, but I don’t use the inbuilt camera GPS. I use the native GPS direction (from a UBLOX USB device) to populate all Exif.GPSInfo.GPSTrack (and all other GPS data with the default exiftool geotag options) then use the tools to set the angle offset. Since the image time is important for geo-encoding I actually name the files as a UTC date/time and “rename” for the camera processing delay, also incorporating subsecond accuracy. I have multiple uses for the images, so fully write the EXIF from the tools as well as well eg;

./mapillary_tools-0.14.5-linux-x86_64 --verbose process “/mybook/mupload/$CURRDATEWX/process”
–offset_angle 225
–overwrite_all_EXIF_tags
–device_make Samsung
–device_model SMJ200Y
–duplicate_angle 90
–duplicate_distance 0.25
–cutoff_distance 50
–cutoff_time 180
–geotag_source exif

The other cell camera is at 270 degrees and a different model. I don’t know if Mapillary ever use the brand/model etc, but I think it good practice to include.

I vaguely remember I had problems with all GPS related EXIF fields populating with a GPX input, that worked fine with NMEA. I think it make have been speed and/or altitude missing, but did not try to hunt the problem down. My GPS sender (UBLOX) also runs at 5 posn/sec and has its movement type modified from “mostly stationery” to mobile.

I am not suggesting you use the exiftool method as such. The tools will work just as well with a --offset_angle, some method of time/image sync and the GPS track data input. You may elect to use the raw GPS stream direction or ask the tools to ‑‑interpolate_directions as already mentioned. There are a few options here more or less dependent on what you already have in the image data and what else you may want to do with it. I personally scroll through all the (day’s) images locally, so EXIF fill is needed to paste the co-ords direct to the OSM editor and display properly on Viking.

Of importance, unless something has changed recently, Mapillary does not handle up and down tilt. It kind of assumes all imagery is ground parallel and approx 1-2m above. You might have to discard these ones or once again look at pre-stitching.

I’d be interested in seeing your results, so please append to this thread so we can have a look!

1 Like
13

Is there overlap between the images? If so, combined panning should work, see https://blog.mapillary.com/update/2019/04/16/mjs-combined-panning.html

I’d suggest trying it on a small number of images and post links to them on Mapillary here so we can take a look if that doesn’t do the trick.

1 Like
14

Right, start with low image count sequences first until you create a workflow for your dataset. You can always delete sequences and upload again. Note however, that combining images can take days, sometimes weeks, before you can actually see nice “pannable” results in the web viewer. Thanks to new hardware, the initial reconstruction time per sequence has been noticeably greatly reduced lately. However, combining sequences continues to be quite complex and thus time consuming. So, patience is going to be your ally. :wink:

All I can say is that I have been capturing 4 way sequences for almost a year now. It really works! :smiley: But, it takes some time to actually see the results, which I fully understand because 3D reconstruction is a complex equation to solve.

15

That is what I said.