Mapillary desktop uploader 3.1.0
New Mapillary desktop uploader has been released with latest mapillary_tools (0.9.4) included. This release contains the following changes:
That is great news!
I also notice .mapillary folder is no longer being created in the source folder(s).
It’s bad that the bootloader auto-updates to the latest version. The latest version does not work for me (or does not work well).
It’s broken for me too (Ubuntu 20.04):
[17:17:27.661] [error] [mapillary:tools] Error: spawn /tmp/.mount_mapillYysOpO/resources/static/mapillary_upload_cli-linux EACCES
at Process.ChildProcess._handle.onexit (internal/child_process.js:264:19)
at onErrorNT (internal/child_process.js:456:16)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
Can you please add the x bit?
$ ls -al /tmp/.mount_mapill76AKe8/resources/static/mapillary_upload_cli-linux
-rw-r--r-- 1 root root 16829592 Okt 14 16:36 /tmp/.mount_mapill76AKe8/resources/static/mapillary_upload_cli-linux
Looks like dropping multiple folders causes some sequences to get split up weirdly. I dropped 18 folders with a total of 42K images, individually ranging from 200 to 9K. Looking around, a lot of sequences are proposed for uploading broken up like this:
I checked an earlier upload (hasn’t fully processed yet) with fewer folders where I didn’t notice this and it does indeed look like this ends up on the site split like that too, e.g.
Mapillary (BvmKDSJCrFfgasGbHdVqi7)
I guess I’ll have to reupload these since it’s technically the desktop uploader that bugged out…
Edit: Seems to happen for individual folders too, even after restating the app. Basically, anything with 1000+ photos is likely to get split up weirdly.
Yes, thanks for catching that! We’ll release an update soon!
Should be fixed now (in version 3.1.1).
OK, with 3.1.1 the “browse folders” link is visible again. However, if I click on it then two directory choosers are opened. Clicking on the symbol above gives me only one directory chooser.
Same for me. I don’t have the feeling that it works
Thanks for reporting, we’ll get a fix for that out soon. To upload right now, you can click anywhere within the drag & drop zone to open up just a single directory chooser, or you can just drag & drop the folders to that zone.
Any update on fixing the sequences getting broken into chunks? It’s been a couple weeks now and this cannot be that complex of an issue. The new version is basically unusable like this. I’m checking out some recent uploads, and I see this issue with many people’s sequences. At this point, all these uploaded sequences need to be fixed in the backend because this isn’t a case of one or two. Thankfully, I had the foresight to backup the previous version files. I am somewhat baffled how something like this passes QA/tests.
Would you mind sharing some of these images that can reproduce the breakage?
I don’t know how to reproduce this reliably, I don’t understand the conditions that cause it and I don’t know what the uploader caches. It occasionally happens, mostly doesn’t. It is as far as I can tell totally random or at least based on something I cannot control. It usually happens once and then you cannot get it again with the same photos. I tried renaming the folders and even all photos. Even the order of dropped folders matters. I’ve wiped AppData Mapillary and Temp folders. I’ve dropped different folders like 50 times today in all sorts of configurations, but I can never get it to happen twice in a row for the same photos. Just keeping track of what I did try is exhausting. I will run this in a sandbox later.
I cannot just upload 10 GB of images (or rather, I already have and it’s on Mapillary). Because it only happens in some sequence(s) with a certain threshold of images, so clearly the counts matter. The chance of reproducing it with like 5 images is tiny, it has to be thousands.
For now, I posted an example of my own broken sequence above. Here are some random ones I saw from recent uploads:
@HellPhoto could you use Release v0.9.5 · mapillary/mapillary_tools · GitHub extract the metadata from these images?
Run this command after downloading the binary:
./mapillary_tools process path/to/images
You should get a JSON file located in path/to/images/mapillary_image_description.json. Share us the JSON we can investigate from there.
How do I run it on multiple folders?
This never happens on a single folder the first time I run it (or happens so rarely I haven’t seen it). I tried 0.9.5 on various folders, but they didn’t get broken up like that. All the sequence IDs in the .json are sequential. This only reliably happens when I use the Desktop Uploader and the first time I drop multiple folders. (Then in various inconsistent ways afterwards when the same folders are dropped.)
I can currently only send you mapillary_image_description.json from Temp folder from the 0.9.4 version bundled with the Desktop Uploader that has this sort of broken sequences.
How do I run it on multiple folders?
When processing multiple folders, you need to specify the path to store the JSON file with --desc_path:
mapillary_tools process folder1/ folder2/ --desc_path=mapillary_image_description.json
I can currently only send you mapillary_image_description.json from Temp folder from the 0.9.4 version bundled with the Desktop Uploader that has this sort of broken sequences.
Sure, as long as it reproduces the issue. This will be very helpful.
I’ll try multiple folders with 0.9.5 tomorrow.
Here is the mapillary_image_description.json file from 0.9.4: 10.14 MB file on MEGA
You can see the first sequence is 1328.., then it starts a new one ff19.., but then goes back to 1328.., then starts a new one b84e... The files/entries are ordered correctly (as far as I can tell without a map). This is the first time I ran it on these 6 folders (and that’s the .json file I have attached).
Here are two photos - last from 1328.. sequence and first from ff19.. sequence: 3.28 MB file on MEGA just for reference. There’s nothing weird or different between any of these photos as far as I can tell. I can upload other photos, but since the breaks happen randomly, I don’t think this has anything to do with specific images.
I am becoming increasingly convinced this is not a problem with mapillary_tools, but somehow with Desktop Uploader. I tried CLI processing the same sequences of images (wiped clean each time) 30 times for versions 0.9.5, 0.9.4 and 0.9.4 bundled with DU 3.1.1 (not sure if it’s any different), and it generated the same correct sequences all 30 times. The first time I drop it in Desktop Uploader, I get broken sequences.
This is how I run CLI ones:
mapillary_tools-0.9.5-win-64bit.exe process "E:\Mapillary\bug testing\P\1" "E:\Mapillary\bug testing\P\2" "E:\Mapillary\bug testing\P\3" "E:\Mapillary\bug testing\P\4" "E:\Mapillary\bug testing\P\5" "E:\Mapillary\bug testing\P\6" --desc_path="E:\Mapillary\bug testing\mapillary_image_description-01.json"
This is how Desktop Uploader seems to run it as per its log file:
C:\Users\###\AppData\Local\Programs\mapillary-desktop-uploader\resources\static\mapillary_upload_cli-win32.exe process E:\Mapillary\bug testing\P\1 E:\Mapillary\bug testing\P\2 E:\Mapillary\bug testing\P\3 E:\Mapillary\bug testing\P\4 E:\Mapillary\bug testing\P\5 E:\Mapillary\bug testing\P\6 --custom_meta_data mapillary_uploader_version,string,3.1.1 --skip_process_errors --desc_path C:\Users\###\AppData\Local\Temp\mapillary_image_description.json
It is very cumbersome to rerun this on the Desktop Uploader because I cannot automate it, so I only did 6 runs. But you can see the random results as I number unique sequence IDs in the .josn file:
0 1 0 1 2 3 4 5 4 5 4 5 4 5 4 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
0 1 0 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
0 1 2 3 2 4 5 6 7 8 9 10 11 12 13 14 15 16 17 16 17 16 17 18 19 18 19 18 20 21 22 23 24 25 26 27 28 29 30 31 30 31 32 33 32 34 35 36 35 36 35 37 38 39 40 41 42 43 44 43 44 45 46 45 47 48 49 50 51 50 51 52 53 54 53 54 53
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 18 20 21 20 21 20 22 23 24 25 24 26 27 28 29 30 29 30 29 31 32 33 34 35 36 37 38 39 40
For example, in the above screenshot the first run’s indices of unique “segments of sequences” 0 1 0 1 2 3 4 5 4 5 4 5 4 5 4 6 7 8... the 4 5 4 5... is the split sequence highlighted in the screenshot above - sequence #4 is interweaved with sequence #5.
Runs 1, 2, 3 and 6 were broken. Runs 4 and 5 were fine. So it’s totally random as far as I can tell. Sometimes less, sometimes more. This matches what I previously saw.
Here are the .josn files from these runs: 3.54 MB file on MEGA
For reference, I have 6 folders with 482, 182, 6745, 5872, 2734, and 441 images. All sequentially named and dated. Nothing weird about them that I can see - and CLI tools don’t seem to have issues with them. As before, they are 15 GB, so I don’t know if I can upload them anywhere.
Thansk @HellPhoto for the detailed report.
This is super useful and helped us identify the problem: It is because DU runs multiple mapillary_tools instances and they all write to the same mapillary_image_description.json so the mapillary_image_description.json got “corrupted” with images mixed from different processes and therefore considered as different sequences. This is why you see they are separated unexpectedly.
Will check with @nikola to fix it soon. Meanwhile, if you could, use mapillary_tools instead.
Thanks again.