Web uploads not arrived: cut at 999 images

What are duplicates ?

1 Like

Sorry for the frustrations everyone. I know the current workflow to handle this is far from ideal. We’re working to resolve this.

Having a system to handle duplicate uploads was very important for us to improve the quality of data on Mapillary, but the way it has been implemented needs work.

In the meantime, I’m curious to get a sense of what are the reasons people prefer the Web Uploader over the Desktop Uploader. Apart from those on Linux who have to use the Web Uploader, are there other things about the Web Uploader that make it your uploader of choice?


I think you should make clear that the Desktop Uploaded is the preferred method for uploading. Last time I tried to use the web uploader, for no good reason, to be honest, and it was slow, until it failed. Then I used the Desktop Uploader and it worked fine. Maybe you could mention this in the web uploader interface.

1 Like

Do you have a sequence where images have been removed so I can check?

When I started contributing images for use in perfecting OpenStreetMap, Mapillary was suggested as the best place; at the time there was the choice between a legacy uploader, the web uploader, and some - -was it perl or python?- tools which were beyond my knowledge to use.

At some point a desktop uploader was mentioned in forum posts, tried (was it version 1.9.n?) which took very long (couple of hours) to prepare say 4400 images for upload, with no discernible gain in upload speed / reduction in time needed to upload, and in addition it created one folder per image with in each folder something like 23 files charting progress : deleting the 100.000 or so files from a 4400 pic upload takes yonks - irrespective whether the computer deletes 100 or 300 files per second.

= = =

Tried v 1.2.4, where I found that some sequences just didn’t appear on the map - not even after like 3 or 4 days.
Found similar for the web uploader : in both instances like 999 or 1499 pics out of be it 2500 or 4400 were on display after three days.

Now diverge from desktop vs web uploader to duplicate removal : took the time to find the last pic in a batch which appeared on the map, then re-uploaded the remainder - there may well have been a duplicate at the junction between shown, and not shown on map. As long as it concerns a couple of pics overlap : just delete those : they’re not malignant, are they?

And if I wanted to inflate the number of pics : upload the daily ride to the shops - different shop each day - would be far more effective.

Have couple of days ago sent a mail to support, c/o Saïd, with a number of incomplete sequences; some of these have now appeared, others seem lost forever.

Hope this helps? If not : ask!

Dear Peter,
Sent a mail, detailing a number of mislaid sequences to support@mapillary on Date: Sun, 21 Jun 2020 17:19:09 +0200 ; would be happy to copy+paste the list here?

= = =
Copy+paste from mail mentioned above :

= = =

Dear Support (frequently Dear Saïd),

Mentioned in a reply to a forum post on Saturday that parts of some series of uploaded photos were missing, and promised to look into this and compile a list at some point. Herewith a list of batches of photos uploaded over the past weeks; the date is the date the photos were taken.

This message was researched and written on the Saturday evening, held back to read again the next day, check for inconsistencies; have now seen Saïd’s message to Micmin and myself, not amended this message as I will respond to that separately.

Mapillary Desktop uploader

2020-04-15 - looks complete

2020-05-02 : more missing, but looks like uploaded via Web-browser uploader (as I can’t see the . mapillary folder) : Mapillary does show from the start of the batch in my folder at 16:01 till 16:21 , and 16:51 - 16:57 ; thus missing 16:21 till 16:51 - south of the river Rupel, extends till east of the river Zenne, where there is a new flood control ‘polder’.

2020-05-02 : the 4424 item return journey along the Schelde river from 17:00 till 18:54 was uploaded according to the log files in the .mapillary folder, but only the 17:18 till 17:27 pics are shown on the green dots/line : any news on the remainder, please? Desktop Uploader shows the batch as Complete …

2020-05-07 - incomplete : Desktop Uploader shows two batches, 3575 and 2522 pics = 6097 , while my folder contains 6843 ; Uploaded status = complete, gap in M’s green dots/line between 17:52 and 18:00 - missing some 450 pics, [hence more must be missing (correct, had the GoPro running while waiting at the level crossing, from 18:21 till 18:26 = approx. 300 removed as you could barely see the grass grow … )]

2020-05-09 - looks complete ; three batches : 606, 2350 and 5325 hero 2018 pics, can’t see the TG5 single shots ??

2020-05-14 - looks complete ; 830 + 6434 pics, can’t see the TG5 single shots ??

2020-05-16 - looks complete ; sequence cut has (fortunately!!) overlooked the TG5 single shots - still kilometres of connecting lines, but may be the only way for OSM mappers to find those …

2020-05-30 - looks complete ; two batches : 3847 + 3805 hero 2018 pics, can’t see the TG5 single shots ??

2020-06-01 - begin to see where things fall down : the single shots with the TG5 will have the same date/time and lat/lon values, as they’re geotagged from the same Garmin etrex10 track-log- which is the most accurate I can get; take 2020-06-01 10-37-49 : in the Hero 2018 batch it shows just a narrow tarmac strip with a wide sand verge ahead, while in the TG5 batch it shows a sideways shot of a forest track, fence and gate.
OK. seem to have put a finger on the ‘lost’ TG5 pics on stretches of roads where I’ve taken single shots of particular details along or to the side of the road while the Hero was also running and either looking straight ahead or also pointed towards the same subject? Or did I?

2020-06-04 - here we have both the single pics connected by kilometres of lines (@sequence cut? actually, those lines may be the only way for OSM-mappers to find the TG5 pics!!) - does that disprove …

That’s the end of the entries from the Desktop Uploader History page checked; now on to the various SD-cards and USB-sticks which I used to upload pics from a very energy efficient Pentium Silver 2-in-1 flip-book:

Mapillary Web-based Uploader

2020-04-18 : ‘bruyns gedaan’ : part visible, part not to be seen on mapillary’s green lines/dots yet ; have split that into ‘part done’ and ‘2do’. visible 15:27 - 15:44 then resumes at 16:24 - 16:41 ; there also are pics 15:44 till 15:51 and 16:02 - 16:07 which show ‘new’ streets , the uphill struggle 16:07 till 16:11 barely adds to the sequence taken a couple of days earlier - different weather, that’s all in retrospect - 16:15 - 16:19 is a different path, looks easy to cycle down to start with, then narrows and gets steeper, thus needs to be marked on OSM as unsuitable for general cycling , the rest 16:19 till 16:24 again largely the same as before apart from weather.

2020-04-18 bruyns paepe 2do : 16:41 - 16:44 is new , as is 16:54 - 16:57 which looks in more detail at the new development (the footpaths to the communal entrances are shown as service roads on OSM, but they’re not ; there is a local OSM mapper in that area, so these may be superfluous) ; from 17:21 you get a back street, a new cycle path, and a road which was public but where part has now become private / no access. that series continues till it meets up with the beginning at 17:47, and it isn’t clear why it isn’t shown?

2020-04-19 : Mapillary shows 16:03 till 16:20 , but full batch was till 17:37 : what happened to the rest, please?

2020-04-26 : looks complete

2020-04-30 - starts at 14:54 and continues to 15:49 ,

2020-06-06 Zoersel hero 2018 : visible from start of batch till 12:33 , and from 12:43 : bit in between is also now a new ‘fietsstraat’, despite having been photographed some weeks earlier, when it wasn’t yet. [NOTE : this concerns the eastern part of the Kasteellei in Wijnegem, south of the Albertcanal]

2020-06-06 Zoersel TG5 : visible on Mapillary till 14:28 , then from 18:27 onwards (final pic shows how people thought about safety over the centuries : initially, a Holy Mother statue at the corner, then a little light over it, and now a 24/7 dome camera); the pics in the nature reserve may not show all that many traffic signs, but do contain images (for example of the circular wall around the pond, and of the ‘shelter’ which can help OSM, while the other pics do at the least show ‘nothing to see here’?

2020-06-12 Zoersel Halle : both the Hero 2018 and TG5 batches look complete

2020-06-14 : so far only uploaded four batches, each just under 1000 pics : 200614 5p1, 5p2, 5p3 and h01 with 995, 990, 986 and 988 pics, of which 957, 934, 911 and 887 were published (as stated on the ‘publish nnn images’ screen) : what happened to the rest, please? Were the last ones omitted or , ? Yes, they were, missing from 12:17 till 12:22 , 14:22 till 14:36 and 17:32 - 18:04 in the TG5 series, and from 12:10 for the first Hero 2018 series - thus missing a new Fietsstraat (cycle street) behind the schools.

Could’ve thought of nicer ways to spend the Saturday evening, but these missing uploads are irksome, and forum posts by e.g. micmin and filip/philippe show that I’m not the only one experiencing this problem, thus hope that by specifying the series of missing pics, and the approximate times (mapillary only shows the minutes, not the seconds for each pic) ‘the support team’ can put a finger on what causes the missing pics in some of the batches.

With as ever ‘Vriendelijke groet’, (Dutch for '; with friendly greeting),

First, how do I know the version of Desktop Uploader?

Around June 21, I intended to use the Desktop Uploader to upload 44 photos. There are no duplicates.
All photos are visible from the links below.

However, even a few days later, when I look at Map on Mapillary (https://www.mapillary.com/app/user/potaro67v?utm_source=signup&utm_campaign=mapillary&utm_medium=email&lat=35.22297063240043&lng=133.9235833774536&z=16.92207092746317), even if I look at Feed and Uploads, the corresponding data does not exist.

I always delete the “.mapillary” folder, which is a group of log files, immediately because the disk system load is high.
So even after checking the photo folder after the fact, I wasn’t sure if it was uploaded.

Therefore, I uploaded again those on 6/25.
I uploaded and “Complete” was indicated by Desktop Uploader, but again, the corresponding photo does not appear in the map, Feed, or Uploads.
After that, I received an email titled “Mapillary images or videos you tried to import failed”.
There was a description such as “duplicate image file(s) detected,” in the contents.

After that, I discovered the existence of this discussion on the bulletin board.

If I didn’t upload it on 6/21, I don’t know the reason for the failure indicated by “Mapillary images or videos you tried to import failed”.
Or if I was uploading on 6/21, I don’t know why there isn’t a corresponding photo on the map, Feed or Uploads.

In any case, I can be confident that this new feature is implemented incorrectly.

I’m not going to follow the detailed steps of deleting each one and doing the upload operation again. The “.mapillary” folder has been deleted, so I think it is impossible to execute it.
I give up these data.
The same photo uploaded to Google Street View is fine, so it’s ok. (https://goo.gl/maps/6EasVmZihWCeqo8r7)
I want to go to the next fun place early and shoot.

The attached photo is a screenshot of the “mapillary_import_summary.json” file that I forgot to delete after the upload operation on June 25.
As you have attached, 44 photos have been uploaded.

If you look at “mapillary_import_summary.json”, you will see 44 successes except for the step “import_meta_process”.
The “.mapillary” folder has been deleted, so the contents cannot be examined.

Screen Shot

Using the desktop uploader files have failed to upload for two different reasons for me:

  1. images are in the same position as previous ones (happens at a stop or rarely it appears my camera app doesn’t acquire a new position and just replicates the previous one)
  2. there is a problem with the exif thumbnail. This error has been mentioned long time ago already on github. The uploader does not give any helpful hint on this. But if you use exiftool (exiftool -thumbnailimage= IMG_nnn.jpg) to remove the offending thumbs the pictures get uploaded without problem.

The latter happens to ~1-5% of my pictures varying by batch. Maybe this is also the cause of your lost pictures.





最新バージョンの Desktop Uploader (1.2.7) では、サーバー側での「厳しい重複チェック」と同等の重複チェックが行われて、
ですから、最新バージョンの Desktop Uploader を使用すれば、セッションが丸ごとWeb非表示になってしまう被害には遭わないのだと思います。

しかし、Desktop Uploader のバージョンを知る方法が私にはありません。現在、質問中です。
6/24頃にPCで「Desktop Uploaderの最新バージョンが使用可能です」というポップアップを見たことがあるなぁ、という記憶しかありません。

また、Web Uploader には重複排除の機能はまだ備わっていないようです。



しかし、Desktop Uploader のバージョンが(1.2.7)であることの確認方法が私には見つからないので、その確認方法を質問しました。


Google Street Viewへのアップロードが正常にできていることは、コメント中に示したとおりです。
位置情報も44枚ともすべて異なっています。( = Exif情報もすべての画像で異なっている、ということです)




さて、Desktop Uploader (1.2.7)(たぶん)での重複チェックの機能をチェックします。


同じく、多くのDesktop Uploader 利用者も安泰です。

その前提は、Desktop Uploader を通常の方法で使用した場合には、そこを通過したセッションはサーバーでWeb非表示の措置がなされることはない、というものです。
つまり、サーバーよりもDesktop Uploaderのほうが判定基準が厳しいか、または同一であることが前提です。




◆01.(5枚) ノーマル
Publish Completed


◆02.(6枚) 違う画像でEXIF情報が同じもの

Desktop Uploader操作のスクリーンキャプチャ : https://youtu.be/hPipgXT0JXs

◆03.(7枚) 既にアップロード済みのものと一部重複するもの
Desktop Uploaderは、画像ファイルのフォルダ名を、そのアプリ内で記録していると考えられます。
また、Desktop Uploader内での重複チェックはPCのみでの判断であり、Mapillaryサーバーにアップロード済の画像とのチェックは行われていないようです。
Desktop Uploader操作のスクリーンキャプチャ : https://youtu.be/-Rlik_vn-uw
エラーメール : https://www.dropbox.com/s/htk7vubnybg3rkz/重複が見つかった、というエラーメール.png?dl=0

◆04.(2枚) 重複を除いてから同一フォルダを使用して再度アップロード
Desktop Uploader操作のスクリーンキャプチャ : https://youtu.be/DULE8ze0gbw
「Folder already uploaded」のエラーになります。
(その後、一旦退避した 1f-5fを元に戻しました)
このようなエラーになった場合には、「重複したファイルを削除してから再度アップロードする」という、Web Uploader のような再処理方法は出来ないということです。

◆05.(2枚) Web非表示の既にアップロード済みのものと一部重複するもの
Desktop Uploaderの操作は完了しました。
Desktop Uploader操作のスクリーンキャプチャ : https://youtu.be/jQfdrlA4sis

◆06.(8枚) 同一の画像で、EXIFのみ異なるもの(同一時刻の画像はある)
Desktop Uploaderの操作は完了しました。
Desktop Uploader操作のスクリーンキャプチャ : https://youtu.be/h4Vjod8vQF0




1 Like

I would like to introduce my first mass upload work (using a desktop uploader) after the introduction of a new strict duplicate check in Mapillary by last week.

There is no change in the operation method with the desktop uploader so far.
In addition, the trouble that causes various troubles for every 500 sheets is still occurring.





00’04" アップロード処理開始。まずは事前の準備にかなり時間が掛かります。
05’08" サーバーへの転送開始。アップロード速度が遅いです。
31’31" 500枚を目前にして、アップロードが止まります。
50’20" 何回かアップロードキャンセルをしてから再度アップロードを開始することによって、先に進むことが出来ます。
1:05’30" やっと、本来のアップロード速度に達しました。しかし、私のネットワークの能力いっぱいではありません。Mapillary側の受け入れが遅いだけです。
1:06’10" 1000枚でも
1:12’20" 1500枚でも、なにかサーバー側でデータ受け入れを躊躇していることが想像できます。
1:18’43" 2000枚でも。
1:25’00" しかし2500枚は何事もなく通過しました。
1:29’05" アップロード完了しました。

アップロードした結果の Street-level Imagery は以下のようになりました。

以前のコメント( Web uploads not arrived: cut at 999 images ) の「テスト◆06」で確認したかったのは、以下のURLから始まる30フレームのような、動画の自動分割によるギャップを埋める私独自の処理への対処でした。
現在の「厳重な重複チェック」でも、無事に、私が思った通りの Street-level Imagery が作成されています。安心しました。


以前のコメント ( Web uploads not arrived: cut at 999 images )でも触れている通り、以下のようなメールが届き、Webには未だに公開されていません。
I received an email titled “Mapillary images or videos you tried to import failed”.
There was a description such as “duplicate image file(s) detected,” in the contents.


私の以前のコメントに dropboxのリンクを載せてあり、私がアップロードした写真(そして、Web公開が保留されている写真)はそこから確認できます。



In this way, I believe that the Mapillary system behaves the same as before, including the presence of existing anomalies.

However, I think that the operation of the entire Mapillary upload system (especially the server-side) was abnormal during the several days during which the new strict duplicate check was being introduced.
A sequence uploaded in the same way resulted in a duplicate error.
As mentioned in the previous comment (Web uploads not arrived: cut at 999 images ), as below, I received an email and it is not published on the Web yet.
I received an email titled “Mapillary images or videos you tried to import failed”.
There was a description such as “duplicate image file(s) detected,” in the contents.

I believe that the data uploaded by the desktop uploader should not fail with the “strict duplicate check” that implemented this time.
However, I have data that has been placed on NO web publishing due to a duplicate error.
Of course, other users were likely to have similar failures.

By my policy, the log files are “deleted when the upload is completed because the amount of log files is huge”, so it is impossible for me to check the log file.
If a Mapillary engineer would be interested in this anomaly, please look directly at my data on the server.
I have no plans to do anything to re-upload these images. Originally, there was no element judged to be duplicate in these data.
I’ve included a dropbox link in my previous comment, and you can see the photos I uploaded (and photos pending web publishing).

The transition period for implementing this new feature should not be that long.
I suggest doing a server-side review of all errors (and any data that you’re prepared not to publish on the web) that occurred during that period.

I’d just like to be upload files again.

How long before it’s fixed?

My main reason to use the web uploader is I can check and correct my image gps positions before upload (Pc linux, no desktop app available)

Use the web uploader because, at least in the past, I could just upload my images and easily work with them as needed.

The desktop uploadr, at least windows current version, is still a mysterious black box. I stopped using it when I realized pictures and sequences were missing. Not suprising, like this right now, 1/3 of the images to process and uploaded won’t get pulled in. Even though it only has 18 errors? What about those others? It doesn’t say. And all those others have the proper exif, etc.


I now also find out that my uploads (which I limit to 999 files, already filtered on duplicates) are automatically splitted in multiple sequences when the server detects a time gap between consecutive files.

This means that when I have to wait 2 minutes at a traffic light, my upload is already splitted. Not very happy with that feature, I like to see my complete route instead of multiple small snippets of routes

I tried today a batch upload of 3500 images using web uploader, they all came through. Although the images were cut automatically into several sequences (one had a 1000 image size), no images were lost this time! Thanks, yhis means I dont have to limit my upload batch size.

But please adjust your sequence-cutter ‘time gap checker’ so it will not directly cut my trip when I wait 1 or 2 minutes or so at a traffic light. Put your threshold at 5 minutes or something :grinning:

1 Like

Was there ever a comprehensive description of this new system?
Sequences appear to be cut more aggressively and on upload, in contrast to the previous sequence bot approach. It also appears to be using time between images (with a fairly low threshold) to cut them up.

1000 seems to be the new max sequence length.

On the plus side: sequences from gopro seem to be interpolated (direction normalised) on upload or soon after - finally! I still normalise it just in case though, since when opening a sequence in Edit mode the old (non-interpolated) directions (0 deg) are still shown. It also does not seem to work if there are no other images in the vicinity to calibrate SfM


I have to say I am liking the new aggressive cuts less now that I have to normalise+offset more sequences manually, especially since the upload sidebar resets every time.

1 Like

Same happened to me, about 3000 images were uploaded recently but never appeared even in my uploads folder. That’s so frustrating that I even don’t understand what’s the point for all this hassle any more.

I am not a consumer of this kind of information, but rather enjoying watching how the map is being saturated with data captured from my uploads, while Mapillary definitely benefits from them in a much more monetary way (ahem, Facebook). Still, it seems that the developers use every occasion to frustrate contributors and make the whole process worth and worth.

Desktop or command line uploaders never worked for me, I had to pre-process my images with the command-line tool and then upload via Web Interface. Now it is so inconsistent that I would rather stop capturing new sequences and uploading them for good, rather than go through this headache again.

1 Like

I must, because mapillary_tools is python 2 only, and ubuntu has chosen to completely remove python 2 from its package repositories, which makes it an extreme hassle to install python 2 https://github.com/mapillary/mapillary_tools/issues/374

Would be great if one could tick a ‘normalize sequence’ box, both in the web- and in the desktop uploader.

Do I ever look at my own pics? Well, have a contract with limited bandwidth, most of which is consumed just by uploading, thus not really capacity to download to view my own pics.

Go over each journey’s crop of pics after copying to my computer : first to weed out badly exposed or framed pics; second to weed out any apparent traffic violations; third to place notes on OpenStreetMap of things to amend - either myself or with the help of a more experienced mapper; fourth to omit any long stretches which would only say ‘see, nothing to see here’.

Having done that there’s no apparent need for the time consuming normalising of sequences, especially not if the total crop of say 9k pics has been chopped into smaller chunks - some just happen to be like half a dozen pics because only need to show one lonely traffic sign.

Suggestion : show the full batch as one batch rather than as dozen of smaller sequences to the logged in contributor for normalising, please - would that really add to the server load compared to specifying each separate sequence later on?