Then the pragmatic answer is probably to upload smaller sequences for now.
My experience, gained through trial and error, shows that video sequences with a maximum length of seven minutes (>2800 Nodes), and a maximum file size of 50 GB are processed smoothly by Mapillary.
Insta360 produces sequences with a maximum length of 30 minutes each. In my workflow, cutting every 5.5 minutes has proven effective. InstaStudio also cuts the GPX sequence, which is very convenient.
A DJI Osmo 360 produces 14-minute sequences. I now have to split these into exactly seven-minute sequences using DJI Studio. Unfortunately, this requires splitting the GPX sequence into two equal halves externally. Creating identical cuts is very time-consuming. Therefore, splitting by time is probably easier than searching for cut markers in the landscape.
I did not know I could write in German. But anyway,…
No, not really. This is what I have been trying to convey. Image count per sequence is indeed a factor in the probability equation for a sequence getting stuck but its impact on the overall probability is tiny because other factors are by orders of magnitude larger. For example, I had sequences with as few as 218 images get stuck.
You would not need to cut videos by time in order to limit image count per sequence if you would process videos into images first and then upload these via mapillary_tools. mapillary_tools (and maybe also the DU but I do not know for sure because I do not use the DU) has built‑in dials for cutting sequences by distance, time, or count. For example, mapillary_tools cuts sequences with more than 1,000 images by default. So, this way, you could save yourself some manual work. However, it does not solve this issue either.
Some people do not care how their sequences are organized, that is if or how they are split or broken up into pieces. However, I do. I like my sequences to be logically organized. Either by street names, city blocks, districts, or etc. But, this is just my preference, nothing more. Imho the platform should not impose any artificial limits on contributors, like a max image count per sequence, may it be due to the design of upload tools, the backend infrastructure, or whatever else. Why? Because there is no real reason for artificial limits to exist. It is solely a matter of (platform) design.
Afaik there is another issue plaguing the platform’s current design, which also plays into this issue: lack of atomic transactions. But, my info on this may have become obsolete by now.
I simply want my images processed and published by Mapillary.
Problem areas I identified:
A: With more than 2800 images, sequences are guaranteed to remain permanently unprocessed and marked yellow. This also happens with fewer images, but it’s rare. =Recording for more than seven minutes at a fast speed causes the number of images to be processed to explode to <2800.
This issue can be easily resolved in the Inta universe by appropriately cutting 5.5-minute sequences. 30-minute sequences, however, definitely need to be shortened.
With the DJI Osmo, shortening standard 14-minute video sequences is only possible with post-processing using suitable video editing software. Therefore, I’m skipping this issue and uploading Osmo 360 videos with a length of 14 minutes.
B: To prevent uploads from failing due to excessively large video files, = A practical 50 GB limit.
I don’t upload videos larger than 50 GB.
C: An image sequence that runs perfectly straight shortly after a successful upload, but then suddenly starts jumping around chaotically after a post-processing run by Mapillary a few hours later, is affected. =Oversampling due to too many GPX points
To avoid oversampling, I reduce the GPX track in Insta360 recordings by a factor of 10.
With the DJI Osmo 360, I currently reduce the GPX track by a factor of 28! Further investigations are needed.
Mapillary has surely got the message and I am also convinced they are trying to solve any remaining technical difficulties. For now, limiting sequence image count remains a weak inept workaround. However, spacing out images over a larger area or thinning out a sequence over a constant area (this is what @osmplus_org does) are the only true workarounds, though quite time consuming and laborious workarounds. I have done this a few times through creating multiple sparse sequences from one original dense sequence by putting every tenth image in a separate sequence. Effectively, this gives ten sparse sequences for every original dense sequence. It works and also makes the reconstruction stage happy but the effects are really ugly, not only on the feed but also for navigation in space and the Time Travel feature.
I increase the GPX step size on a highway to 28 meters, and in the city at slower speeds, the step size is 9 meters. Mapillary interpolates new nodes in between. I don’t think much of significance happens within 28 meters when driving on a highway at 100 km/h. A head-on collision might be missed within those 28 meters.
The sharp curves in tunnels are somewhat awkward; there, I manually draw a new node every 30 meters. The fact that Mapillary fills these 30 meters by interpolation isn’t a bug; it’s what makes manually drawing tunnel crossings with the JOSM editor possible in the first place.
I increase the GPX step size on a highway to 28 meters, and in the city at slower speeds, the step size is 9 meters. Mapillary interpolates new nodes in between. I don’t think much of significance happens within 28 meters when driving on a highway at 100 km/h. A head-on collision might be missed within those 28 meters.
The sharp curves in tunnels are somewhat awkward; there, I manually draw a new node every 30 meters. The fact that Mapillary fills these 30 meters by interpolation isn’t a bug; it’s what makes manually drawing tunnel crossings with the JOSM editor possible in the first place.
I was initially puzzled as to why I was getting such a low number of images per route on Intive Map Europe during my drives in Munich. The reason wasn’t that I had too few GPX nodes; in fact, the Insta360 remote produces a very high GPX node density. After reducing this by a factor of 10, the image density in Mapillary suddenly increased. Therefore, thinning out the GPX points doesn’t produce fewer images in Mapillary; instead, Mapillary interpolates between the GPX points and extracts significantly more images from the 24 frames-per-second video stream.
I’m currently uploading a long highway drive. The camera was my DJI Osmo 360. As expected, thinning the GPX track to a 28-meter increment (GPX reduction by a factor of 28) generates a very high number of GPX points. I don’t expect these recordings to ever be processed by Mapillary from yellow to black. Therefore, I’m considering using the tool ffmpeg.exe in the future, which can halve or even quarter my 14-minute Osmo 360 .mp4 sequences. From 9000 points, this would reduce the number of recordings displayed by Mapillary to 2250 points.
So, I have these two sequences 0ynHQkqSCgvhmcA8T4zZ5G and MXDuJ2Q1IhfKSn8lce9Ut6 which will not finish processing for the sake of anything. I have uploaded these twice because sometimes uploading again can make a difference and finish the “Map Data Processing” step.
Then lately, I have also this tiny sequence iQuEALGM3JT91HtjS4RsOX that also is stuck in “Map Data Processing”. I am pulling my hair! What went wrong with these sequences?
Apparently, there is some randomness to sequences finishing processing. If this is so, could contributors get an option to at least nudge such stuck sequences for repeated processing? Uploading again is an option but it is quite laborious.
iQuEALGM3JT91HtjS4RsOX has finished processing. Thank you.
0ynHQkqSCgvhmcA8T4zZ5G and MXDuJ2Q1IhfKSn8lce9Ut6 continue to be stuck in “Map Data Processing”. They seem to have not progressed at all.
6N2XljiSBQLG1sOeIRoC87 has joined the club of stuck sequences in “Map Data Processing”. This sequence has not received a single processed image. The fov indicator is very clear about that.
Just a short update - there’s a strong correlation with upload length and failures, but as indicated by your sample iQuEALGM3JT91HtjS4RsOX not always the case. I couldn’t reproduce the failure on Friday - still looking into it.
Yeah, I am also unable to make out any pattern that causes processing to seize, except that neither image count, length, nor preexisting imagery matter. My educated guesses are that the pipeline lacks atomic transactions at some point or that processing begins before data center replication has completed, so that processing tries to pull data from different data centers that have not synced up yet.
@balys Sadly, one sequence did not make it. And, I do not have much hope for the other either, although externally there is nothing different about these compared to all the others that went through. Like I said, I have my suspicions as to why it has failed.
Thanks for raising this. I found a bug in the harvesting. The production fix will be out in the next couple of days, but for the two uploads you showed in the screenshot above, I was able to run the fixed code as a one-off. You should see the uploads as soon as the blurring stage is complete.
I am glad you have been able to fix the ingestion code. The affected sequences went into processing. Unfortunately, they (Fwr3njmX07ScARdP985Tep & 8M1aX2C4kimPRKenb9x6ST) are also stuck now.
Neither 0ynHQkqSCgvhmcA8T4zZ5G, MXDuJ2Q1IhfKSn8lce9Ut6, nor 6N2XljiSBQLG1sOeIRoC87 have budged. Should I upload these sequences again?
I have found another low image count sequence GcTZILYMDy6NVQ7X1kHvoB stuck in “Map Data Processing”. Apparently, this issue is not as uncommon as you may think and has nothing to do with image count or density. Hence, imho this issue should receive more attention than it has received so far.
I have deleted 0ynHQkqSCgvhmcA8T4zZ5G, MXDuJ2Q1IhfKSn8lce9Ut6, Fwr3njmX07ScARdP985Tep, and 8M1aX2C4kimPRKenb9x6ST and uploaded them again because there was no progress on them. Unfortunately, rtbz2v1Pfeuq5KGQwYCo43, 1jkJ0DV3MdOieT8WtFSLcy, and dtmBGKSlkQ362NoFn7wTjh are now stuck too.
I have reorganized rtbz2v1Pfeuq5KGQwYCo43, 1jkJ0DV3MdOieT8WtFSLcy, and dtmBGKSlkQ362NoFn7wTjh into a single sequence and am uploading again. We will see how this one goes.
@tao@caglarpmeta I have also found a bug in mapillary_tools that may cause empty sequence uploads. In other words, it may happen that only the image description file gets uploaded but no image files. The solution is that mapillary_tools must simply delete the upload cache file handles (file) on every successfully uploaded sequence.
I did not know I could write in German. But anyway,…
and I am also convinced they are trying to solve any remaining technical difficulties. For now, limiting sequence image count remains a weak inept workaround. However, spacing out images over a larger area or thinning out a sequence over a constant area (this is what 
I am pulling my hair! 
What went wrong with these sequences?
Thank you.
My educated guesses are that the pipeline lacks atomic transactions at some point or that processing begins before data center replication has completed, so that processing tries to pull data from different data centers that have not synced up yet.
The affected sequences went into processing. Unfortunately, they (
Should I upload these sequences again?
