Failed sequences

I’m using a RTK GNSS receiver (SparkFun Express) connected to an Android smartphone. I capture pictures using Mapillary app. Back, I’m uploading pictures using the app itself and wifi. Time to time, the online processing fail on sone sequences. I’m reporting each failed sequence using email to zendesk. Indeed, I don’t get appropriate answer in that I’m still loosing sequences

Hi Eric,

This particular sequence failed in the initial processing stage. Similar to one of your previous failed uploads, because this sequence failed more than 7 days ago, we are unable to retrieve the data for it, apologies.

So, usually, they answer too late.

Once, they answer quickly :

This sequence (Cluster ID: …) is showing an “invalid GPS data” error. Make sure that your device’s GPS settings are calibrated properly prior to capturing your sequences.

So, I asked for more details:

Can you be more precise on “invalid GPS data”? Does it concern the whole sequence? One or few pictures?
[…]

I’m still waiting for answer… and, of course, it was 7 days ago, so they will not be able to investigate.

2 Likes

By strange coincidence, many different people had problems with their sequences over the same period of time

I take photos in OpenCamera for it, and I have it marked that it only takes and saves photos with GPS location. For this I copy them to my computer and send them via Desktop Uploader (which showed all the photo and route without problems).

I measured one sequence, sent it a second time after a month and it was “Failed” second time.

One point for me is that I have not much time a home to transfer pictures. So, back home, connecting to local wifi and starting the transfer from the smartphone is very convenient. In opposite, copying GBs of pictures to desktop PC (usually after extracting SD card from the smartphone because USB transfer is too slow) and later uploading sequences one by one with desktop uploader is a waste of time. It is even worth with OpenCamera because with my smartphone, coordinate are truncated in OpenCamera so I do have to perform geotagging again on desktop with GNSS track, using josm. And, there are time shifts so, for each sequence, I have to adjust time bias using comparison with orthophoto.

Guess I’ll leave this here… the second time my gopro messes up the calendar date in the images, and I notice when it’s posted. Is it possible to somehow edit the exif on the site?

NO.
Ask for deletion in the menu.
Exifdatechanger.
Upload.

Excellent! I finally found this on the menu and deleted sequences where I mismatched the GPS track.

1 Like

why some of my uploaded sequences (uploaded with command line) are failed, and some are not? What will happen with the ‘failed’ sequences because I can see them on the map ? Pasteboard - Uploaded Image

Yesterday 11.12.2024, 2 sequences Failed… and in two weeks there will be no trace of it. It’s a waste of words…

Now I still checked, 2 other sequences made on 31.October got Failed - but there are visible photos and a trace on the map.

Failed sequences stay in the feed for two weeks but, if we have some delays in processing, the time window to check the failure might be shorter, which is not ideal. I’ll talk with the team about making processing failures more clear and actionable.

@czecko Can you share a link to one of your sequences in this state? It would help us with debugging and improving error reporting. Thanks!

Failed sequences are not reported any longer.

Failed sequences stay in the feed for two weeks

Wait, what? So I could have missed failed sequences and never noticed? I just checked recent uploads and there are 6 failed ones and there is no notification about this. And I thought retries were now automated and better internally reported?

Yes, this was quite annoying before. But now there isn’t even an indicator. And this is made more annoying because sequence browsing UI is so incredibly bad and you have to scroll endlessly with no easy way to spot gaps, search, filter, tag or organize anything.

4 Likes

@GITNE , when you see a message like “Processing 2 captures” - that means the process is in progress, working as expected. So the fact that a capture from Dec 10th showed up on Dec 11th is expected (I would not call it “came back from the dead”). In general I probably wouldn’t recommend re-uploading unless more than 2 weeks has passed since you uploaded and the sequence still isn’t visible. Instances of this should be quite rare these days, and we still have more work to do to optimize here.

This is good feedback, thank you. I believe things are supposed to work like this (@abalys keep me honest)

  • When you upload things you see the “Processing x captures” message
  • When things have finished processing they show up in your feed and the processing message disappears
  • If something fails processing it should show up in your feed as “Failed” and the processing message disappears. Failed captures will be displayed in your feed for 2 weeks before being hidden giving you an opportunity to re-upload. Right now it’s hard to know what exactly to re-upload, so we have actively being working on minimizing the number of times things fail - this should have gotten much better in recent months.
1 Like

You say to reupload, but the website says to contact support:

Support may be able to help in some instances, so you can also contact support - but re-uploading is also safe once a capture is in failed status

If a capture is in “Processing” it won’t be shown in the feed yet, and it could complete successfully (most cases) or fail (a small number of cases) - not sure if I follow your question though.

I guess it isn’t clear to me why a sequence has failed and why processing cannot automatically retry? This post suggested that retries would be automatic within a few days, i.e.

Our system attempts to re-process these in a few days from the failure.

It’s definitely been longer than that for me and soon to be the 2 week limit when it disappears completely.

For example, one of the sequences (5XJq7UT3sC49OYFIMnwp6d) as it is in the list:

All the points have been updated on the map layer (these are not the temporary ones):

I can click and see every image:

All the detections are present (up to the last image in sequence):

They appear to be in detection “layer” already:

And I have the option to delete it:

seq8

So what failed? I cannot tell any difference between this sequence and any other except the “Failed” label. If I just reupload it, what happens to these images and detection - are they actually deleted at the 2 week mark (I guess I’ll see that in a few days)? Should I delete it first? Will I not be uploading duplicates?

2 Likes

Good question, @abalys , @knikel - any ideas why this sequence is marked as “Failed” in this case?

Hi,

This is a consequence of a recent incident that we had, which resulted in hard failures/delays on sequences that were uploaded around 9th of December.

Although usually all the retries are automatic for isolated failures, uploads around 9th of December needed to be retried manually on our end.

I queued them for reprocessing, which should complete over the next couple of days.

Hope this makes sense and apologies for the inconvenience.

Kind regards,
Balys

1 Like

The last 4 times I had sequences listed as “failed”, they needed manual intervention due to backend issues/changes. I don’t think I have ever seen “failed” sequences listed that later resolved themselves, so to speak. I assume there are more frequent “regular” failures, but I assume they get requeued quickly and I never notice those since they don’t spend time being listed as “failed”. And since I’ve never seen any gaps in my uploads, they must have all gotten processed. So basically, it sounds to me like if we see “failed”, we should likely be reporting/bringing it up.