Unfortunately we experienced a technical issue which has lead to partial data loss for about half of the uploads made on September 21st and 22nd. Please check your feeds - if you see status “Failed” or “Ingesting” from uploads made on those days (and you don’t see image thumbnails for those uploads), and you still have the original imagery, please go ahead and upload again. Uploads before September 21 and starting from September 23rd are OK (though some may experience processing delays for several days)
We apologize for this technical issue and are taking mitigation steps to try and prevent this sort of thing from happening again in the future.
Thank you for acknowledging and properly communicating about this mishap. Although mishaps may entail some levels of inconvenience to others they will eventually happen to everybody. True leadership and accountability is not only about preventing mishaps in the first place but as much about professional mitigation.
Thank you for your kind words and understanding @GITNE and for your active participation in the community!
@Lowiekse The pipelines are still catching up from the disruption, but you should be seeing more things being processed and all of them should complete. Send us an email at support@mapillary.zendesk.com if you don’t see that happening by Monday with a link to your Mapillary profile.
@boris , thank you for updating the community on this issue. I had a series of sequences from September 21 and 22 that I tried reuploading, but the uploader gives an error that these files have already been uploaded. Changing their file names or paths does not work. Is there a solution for this?
@adalvit@boris Ah btw, the upload history on Linux or XDG compliant systems should go into $XDG_CACHE_HOME/mapillary_tools instead of $XDG_DATA_HOME/mapillary_tools because this data is non-essential and can be recreated anytime. $XDG_DATA_HOME is basically only for user specific shared application data. There is no need nor any reason for sharing a user’s upload history with other apps. Besides, the upload history should have a limited lifetime as opposed to configuration data like a user’s account access token.
Hi @gklafeta, yes, we’ve added an option in the 4.1.3-beta version of desktop uploader to “skip uploaded files” which you can disable by tapping on your username in the top left and selecting “Settings”
Once installed, the beta version will be called “Mapillary Uploader BETA” so you can keep using the public version along with it. The beta will update to the latest version automatically so you’ll always have access to the latest features in development.
Hi folks, unfortunately we have detected data loss on October 6 which led to about 4% of uploads failing to process. Normally we are able to recover and re-process failed uploads, but in this case we are unable to do so. Again, we apologize for this technical issue and are taking mitigation steps to try and prevent this sort of thing from happening again in the future (including a major re-write of this process to improve reliability).
Please check your feeds - if you see failed sequences for October 6th and have the original imagery, please re-upload following the instructions above.
Hi eserte, the images are still there, e.g. the first one you’re linking is here . Since we had to reprocess data from 2023-10-08, a lot of images were assigned new ids. The parts of the process that put sequences together and show them on the map are sill executing, so the sequence you see in blue is a ghost of the old one and is not really functional, and image navigation is also broken. In a few days everything should be restored correctly - have a look on Monday and let’s see if everything is resolved by then.
I have 28 sequences from the same trip stuck on “Ingesting…” since December 20 with no thumbnails. Would this be the same problem? This is the closest thread I could find about something like this. How should I proceed from here?
I am bored of reporting every now and then that something does not work.
You have programmers, it’s hard to write something that detects that the sequences have not processed for 1,2,3 weeks and report it to you responsible for keeping an eye on it.
And why the page where we can see our sequences lacks the option to sort them by - processed, failed, ingesting, image processing, map data processing, etc.?
I’m bored of scrolling half an hour to check something.
And even the option of being able to report a problem right away with the sequence (by clicking next to it) without sending e-mails etc.
Is there any update on this? I have to echo czecko’s sentiment on the lackluster UI - I wish I could just delete these and reupload. But the UI does not provide any option to even see what the sequences are, let alone do anything with them or even see debug information. I really don’t want to upload the same files again because who knows what repeat file conflicts might arise.
I don’t know if this is a coincidence (probably not), but uploading new sequences did trigger these stuck ones to “process”. I guess there was some sort of internal flag set for them? However, of the 28 sequences from the batch, only 2 actually appear in the list (Nov 15, 2023), the rest appear to have disappeared. Everything else I uploaded since then seems fine. Since it’s been a few days and there’s nothing processing, I am assuming they are lost? Should I reupload or is it some internal stuff I need to wait for?
It’s also really alarming that there are no error messages or anything to let you know that sequences have gone missing. The sequence list UI is also really bad and it’s impossible to see past uploads in any sort of organized way, so I have no way to see if something like this has happened before.
