Yes, multiple instances, but normally they don’t crash together. It’s still impossible to upload files in certain folders, even without any files it gives a database locked error. Another instance is still running at the moment without any issues, I don’t get it what the trigger is…
I always run multiple instances because my connection and laptop are fast enough. Normally it’s a dashcam upload (only video, no database issues with that, but those will be processed on the server I presume), a GoPro Hero process and GoPro Max process. The most problems are with the Hero process (not the processing of the files itself, just the upload part).
I didn’t change anything the last months, everything is still the same as the last 3 months. There is something changed on the serverside what the issues is generating.
Well, no wonder you are consistently getting database is locked errors. Running multiple mapillary_tools instances in parallel is currently ill advised because it has not been designed to do so. @tao, you may want to implement a single instance lock for the upload command for now. Going forward, mapillary_tools could be refactored into allowing concurrent instances but where all instances also share single database access.
There is only one good reason to sort of run multiple instances because mapillary_tools currently does not support runtime upload batching or queuing. @tao, an implementation that waits on a single upload command instance lock to release (depending on the OS scheduler) should enable this functionality. @TheWizard, generally speaking it should be better to configure a single instance to saturate your link’s throughput than to use multiple instances to do it. This is exactly what ‑‑num_upload_workers and MAPILLARY_TOOLS_MAX_IMAGE_UPLOAD_WORKERS are for. In other words, it is not a good reason (or idea) to run multiple mapillary_tools instances in order to saturate your link’s throughput.
This problem wasn’t present this year, I was always able to use multiple instances at the same time without any errors (not even de 412 eror, that one is also new but not an issue due the resume function). The 429 error hasn’t occured yet on 4.13.
For now I’m using one instance and it’s working fine for the last 24 hours (around 700 GB uploaded). I will adjust my batch files to incorpurate all the different file processes (360, dashcam and GoPro Hero all have different settings).
Thank you for sharing your use case. I too believe that mapillary_tools should support batching, especially (but not exclusively) for those contributors who source imagery from different cameras at the same time. Batching would be very useful to single camera contributors too. However, for technical reasons uploading should work on a single instance only. @tao In fact, batching could be done even more elegantly than scheduling uploads via multiple instances arbitrated over a single instance lock. The process command should be able to queue upload batches into the single access database, and as long as the upload command is running it should just schedule or pick uploads to upload. I hope this makes sense? It may seem a bit like over engineering but you have to schedule uploads somehow if you do not want multiple instances to compete over a single limited throughput link.
In v0.14.4 you can run multiple MT instances. The database lock error may still occur, but it will be showing as warnings, and MT will keep retrying until it’s unlocked.
If you run a single MT instance, then we should not see any database lock errors. cc @nikola
@GITNE you are right – there is no need to run multiple instances. A single instance can already fully utilize multiple CPU cores and network bandwidth in most cases. If you have to, then make sure each process uploads different folders/files @TheWizard
Hi @tao , thanks for the quick update! I’ve downloaded it and will test it.
I know that multiple instances is not needed, but I had a lot of data to process after my holiday from different sources. I always run from different folders. The CPU cores and network is not at the max when running 1 instance, CPU can peak but network is still at 25%. I try to maximise if possible, I had more than 4 TB of data to process, and if the process is stopped in the middle of the night than I have to start all over again (only the upload part, but takes a lot of time to upload segments of 500 GB).
It would be awesome if a future release could make it so that GoPro .360 video files can be properly sampled locally using the video_process command. My understanding is that it’s currently only possible to upload the video file itself for complete processing on Mapillary’s servers. If the images are sampled locally you end up with an improperly stitched output. Using the GoPro Player first to convert from .360 to .mp4 is time-consuming, bug-ridden, and not possible on certain OS. An example of the issue with what currently outputs when using video_process locally on a .360 file.
Hi @tao, I uploaded arouind 600+ GB with 14.4, no (non-recoverable) issues at all. Unfortunately no database errors this time to check if it’s fixed. Thanks again for the great support!
Just completed a mixed EXIF/phone and BlackVue dashcam 45GBytes upload with 0.14.4 over a cell link. No real issues except the usual router/wifi saturation without the pre environment MAPILLARY_TOOLS_MAX_IMAGE_UPLOAD_WORKERS=1 in place. Without it, the (ZTE?) router web interface was also stalled despite the wifi side running at 150MB/s. Cell signal was “2-3 bars”.
If there is ever an intention to test and optimise the upload workers, I’d suggest that since the link is often asymmetric for cell-phone (slower upload rate) this should be factored in any calculation.
@tao I have a lot of GoPro MAX video files that I can’t upload because of some issue with the collection (too much GPS drift, they went inside a store with the camera and forgot to stop it, etc.). If I could get the frames out properly without stitching errors and with the metadata attached, then I could more easily delete the problematic images and upload the rest.
I also have projects that in order to implement, would require using the images more quickly than is possible if we have to go through the whole flow: uploading videos (sometimes with limited bandwidth), waiting for it to process on Mapillary’s servers, and then downloading (again, sometimes with limited bandwidth). I would still upload to Mapillary, it would just take a longer period outside of the project timeline. But without being able to extract the frames and edit/use locally, I can’t implement the project.
