Delays on ingested imagery uploaded from 11th Oct

[GITNE]

@osmplus_org

I’ve been reviewing my upload history and noticed that videos with more than 2800 GPX points are apparently not being processed by Mapillary and therefore remain yellow in the timeline.

Sequence image count has very little to do with the chance a sequence getting stuck in processing, despite it being a reasonable but naive assumption from a contributors point of view. It is a function of image density per area and load on the indexing service.

@knikel

Our current indexing solution is based on S2 cells and the increased latency correlates with how densely given area is covered with images. For example querying a 1km x 1km bbox in an area with 500 images will be faster than one with 5000.

@knikel

Albeit a bit cryptic, this error indicates a generic timeout. Likely there was too much data in the given bbox and serving it is causing a timeout. Very densely captured areas might timeout sometimes despite relatively small bounding boxes.

Apparently, 3D reconstruction (the “Map Data Processing” stage) is built on the same query mechanism as the public facing Mapillary API or its query mechanism is prone to the same limitations. Hence, if an area is too densely populated with images for a given query area and the indexing service is under too much load then 3D reconstruction queries just timeout and reconstruction seizes. Ergo, a sequence gets stuck in processing.
However, there is hope for improvement on the way:

Consistent HTTP 500 Errors on Graph API (/images and /map_features) Endpoints Since Nov 7

We did change underlying spatial indexing services in October. I’ll investigate if something change in other components that we depend on around the beginning of November. Appreciate all the reports!