@TheWizard were you running the latest version Release v0.13.3 · mapillary/mapillary_tools · GitHub, if not, could you try it out and let me know if speed improves?
With v0.13.3 I think the egress can be saturated with a single upload process, as I also tried to run multiple MT processes to upload and I didn’t notice any speed improvement, compared to single upload process.
Hi tao,
I was indeed on the previous version. I tested it now with the 0.13.3 version and the upload speeds are greatly improved! There is now more fluctuation between the different uploads, it’s not related to my computer, CPU time is 20%. I presume it’s related to the server or interconnects. But I love those speeds way more than before 
@tao When pip building the latest mapillary_tools==0.13.3 the pynmea2==1.19 dependency does not build. All other dependencies build flawlessly.
❯ python3 --version
Python 3.12.9
❯ pip --version
pip 25.0.1 from /usr/lib/python3.12/site-packages/pip (python 3.12)
❯ pip install --no-binary :all: mapillary_tools
Collecting pynmea2<2.0.0,>=1.12.0 (from mapillary_tools)
Downloading pynmea2-1.19.0.tar.gz (36 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-wk7g44vb/pynmea2_da86abe101874af696200f318c659b8c/setup.py", line 3, in <module>
import imp
ModuleNotFoundError: No module named 'imp'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
Looks like this has been already fixed
but the source dist has not been updated on PyPI yet. 
Thanks for report. Not sure how many users are using nmea for geotagging. If not, I think we can mark this package as optional by default.
I have made a handful of larger uploads with the tests.cli.upload_api_v4 module and a legit dummy upload with v0.13.3 into the actual upload feed, both over a direct connection without static IP address mapping (sort of default net config). I have also played with different chunk sizes. None of the uploads caused a SSLError, even when surfing the Mapillary web app at the same time. All uploads also resumed correctly. I am not sure what you have changed but things look stable for now. The DNS TTLs continue to be volatile and generally quite short, usually around one minute. I will continue to monitor the situation on upcoming uploads.
2025-03-22 02:23:55,816 - INFO - Uploading to organization: {"slug": "XXXXXXXXXXXXXXX", "name": "XXXXXXXXXXXXXXXXXX", "id": "XXXXXXXXXXXXXXXX"}
Uploading ZIP mly_tools_90f4e91938f32803fe70e8c82c5b8669.zip (1/1): 49%|████████████████████████████████████▉ | 42.3G/85.9G [9:58:22<10:14:05, 1.27MB/s]
2025-03-22 12:38:35,219 - WARNING - Error uploading chunk_size 5242880 at begin_offset 0: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_90f4e91938f32803fe70e8c82c5b8669.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-03-22 12:38:35,281 - INFO - Retrying in 2 seconds (1/200)
Uploading ZIP mly_tools_90f4e91938f32803fe70e8c82c5b8669.zip (1/1): 49%|████████████████████████████████████▍ | 42.3G/85.9G [10:14:39<10:34:15, 1.23MB/s]
Uploading ZIP mly_tools_90f4e91938f32803fe70e8c82c5b8669.zip (1/1): 25%|██████████████████▉ | 21.7G/85.9G [5:05:21<15:06:53, 1.27MB/s]
Looks like I have to resort to the static IP address workaround though. 
No errors with that.
@tao I would assume files to be PUT in chunks instead of POSTed.
@tao I continue to rarely get SSLErrors and progress resets with static IP address mapping:
Uploading ZIP mly_tools_687a1686021c47aa8fb831fba230de15.zip (1/1): 34%|█████████████████████████▉ | 12.5G/36.5G [2:56:50<5:38:48, 1.27MB/s]
2025-03-25 02:49:25,859 - WARNING - Error uploading chunk_size 5242880 at begin_offset 0: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_687a1686021c47aa8fb831fba230de15.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2427)')))
2025-03-25 02:49:25,860 - INFO - Retrying in 2 seconds (1/200)
Uploading ZIP mly_tools_687a1686021c47aa8fb831fba230de15.zip (1/1): 34%|█████████████████████████▉ | 12.5G/36.5G [3:13:11<6:12:30, 1.15MB/s]
Due to its sporadic nature, maybe this also happens when the server is briefly overloaded? The server then just closes the oldest connection(s) to free resources for new connections?
It resumed from 34%, not 0. The begin_offset=0 in the warning was the params used in the first upload trial.
Using PUT seems to work too. But you can not resume from a PUT uploading, i.e. the offset is always returned as 0 if you use PUT, which makes sense from its semantics perspective (replace a resource) PUT - HTTP | MDN
Thank you for the clarification. I have read the RFCs but could not figure out the exact difference in semantics between PUT and POST. So, in a chunked scenario a first PUT request would create a new resource (file) on the server and subsequent chunk requests would be POSTed. Why do I mention this; Could your HTTP server handle PUT versus POST created resources differently later on, i.e. impact resumes or upload session migration?
If I recall correctly it fell to 26%. I will have to use the --verbose option next time to say for sure.
@tao Things are getting weirder!? 
The server shuts down on completion? 
Uploading ZIP mly_tools_ccfce8717369e5ac30a466be384e9bb9.zip (1/1): 100%|██████████████████████████████████████████████████████████████████████████████| 5.50G/5.50G [1:18:17<00:00, 1.27MB/s]
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 622, in upload
_upload_zipfiles(mly_uploader, zip_paths)
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 677, in _upload_zipfiles
raise UploadError(ex) from ex
mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jake/.local/bin/mapillary_tools", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/__main__.py", line 164, in main
args.func(argvars)
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/upload.py", line 50, in run
upload(**args)
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 651, in upload
raise inner_ex
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 673, in _upload_zipfiles
cluster_id = mly_uploader.upload_zipfile(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 132, in upload_zipfile
return self.upload_stream(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 219, in upload_stream
return _upload_stream(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 425, in _upload_stream
raise ex
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 398, in _upload_stream
file_handle = upload_service.upload(fp, offset=begin_offset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload_api_v4.py", line 89, in upload
return self.upload_chunks(chunks, offset=offset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload_api_v4.py", line 151, in upload_chunks
raise RuntimeError(
RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}
Uploading ZIP mly_tools_ccfce8717369e5ac30a466be384e9bb9.zip (1/1): 100%|██████████████████████████████████████████████████████████████████████████████| 5.50G/5.50G [1:18:18<00:00, 1.26MB/s]
Welcome to the club, but I’ve been having this kind of thing for a month, sometimes several times a day sometimes 1 a day, although just recently it has calmed down.
>
> [2025-04-27 20:05:01.501] [debug] [mapillary:tools] 2025-04-27 20:05:01,501 - DEBUG - HTTP 400 (Bad Request): {"debug_info": {"retriable": false, "type": "ProcessingFailedError", "message": "Request processing failed"}}
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] Traceback (most recent call last):
> File "mapillary_tools\upload.py", line 568, in upload
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools]
>
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "main.py", line 8, in <module>
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] File "mapillary_tools\commands\__main__.py", line 164, in main
> File "mapillary_tools\commands\upload.py", line 50, in run
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 651, in upload
> File "mapillary_tools\upload.py", line 563, in upload
> File "mapillary_tools\uploader.py", line 163, in upload_images
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 219, in upload_stream
> File "mapillary_tools\uploader.py", line 425, in _upload_stream
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 398, in _upload_stream
> File "mapillary_tools\upload_api_v4.py", line 89, in upload
> File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}
> [6372] Failed to execute script 'main' due to unhandled exception!
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] Traceback (most recent call last):
File "mapillary_tools\upload.py", line 568, in upload
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] mapillary_tools.upload.UploadError:
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 8, in <module>
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\commands\__main__.py", line 164, in main
File "mapillary_tools\commands\upload.py", line 50, in run
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 651, in upload
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 563, in upload
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 163, in upload_images
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 219, in upload_stream
File "mapillary_tools\uploader.py", line 425, in _upload_stream
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 398, in _upload_stream
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 89, in upload
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks
RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}
[864] Fail
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] ed to execute script 'main' due to unha
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] ndled exception!
@tao - any thoughts on these errors?
Same. Uploads started failing roughly around 15:30 UTC; constantly repeating HTTPError: 412s.
fyi, I’m using mapillary_tools 0.10.0, just in case it’s an issue with using an old version.
Also issue while using windows application:
[2025-05-20 18:45:10.797] [debug] [mapillary:tools] 2025-05-20 18:45:10,795 - DEBUG - Sending upload_progress via IPC: {'file_type': 'image', 'sequence_idx': 0, 'total_sequence_count': 1, 'sequence_image_count': 999, 'sequence_uuid': '0', 'entity_size': 2331715097, 'md5sum': 'c9c815c8b144704f269a9be91873228b', 'upload_start_time': 1747759383.75809, 'upload_total_time': 0, 'offset': 2331715097, 'retries': 19, 'upload_last_restart_time': 1747759387.7296193, 'upload_first_offset': 0, 'chunk_size': 16459289}
[2025-05-20 18:45:10.802] [debug] [mapillary:tools] 2025-05-20 18:45:10,802 - DEBUG - HTTP 412 (Precondition Failed): {"debug_info": {"retriable": true, "type": "AuthorizationFailedError", "message": "A temporary failure has occurred. Please try again."}}
[2025-05-20 18:45:11.097] [debug] [mapillary:tools] Traceback (most recent call last):
File "mapillary_tools\upload.py", line 568, in upload
[2025-05-20 18:45:11.099] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"AuthorizationFailedError","message":"A temporary failure has occurred. Please try again."}}
[2025-05-20 18:45:11.100] [debug] [mapillary:tools] During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 8, in <module>
[2025-05-20 18:45:11.100] [debug] [mapillary:tools] File "mapillary_tools\commands\__main__.py", line 164, in main
[2025-05-20 18:45:11.100] [debug] [mapillary:tools] File "mapillary_tools\commands\upload.py", line 50, in run
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 651, in upload
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 563, in upload
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 163, in upload_images
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 219, in upload_stream
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 425, in _upload_stream
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 398, in _upload_stream
[2025-05-20 18:45:11.102] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 89, in upload
[2025-05-20 18:45:11.102] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks
[2025-05-20 18:45:11.102] [debug] [mapillary:tools] RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"AuthorizationFailedError","message":"A temporary failure has occurred. Please try again."}}
[7588] Failed to exe
[2025-05-20 18:45:11.102] [debug] [mapillary:tools] cute script 'main' due to unhandled
[2025-05-20 18:45:11.103] [debug] [mapillary:tools] exception!
[2025-05-20 18:45:11.204] [debug] [mapillary:tools] process exit code 1 and signal null
[2025-05-20 18:45:11.208] [error] [store:modules:uploadSession] exit with code 1 and signal null
[2025-05-20 18:45:11.265] [info] [power-saver-blocker] stop: 4
[2025-05-20 18:45:11.265] [info] [services:observer-manager] Stopping 6ec07360-3599-11f0-bf28-4d8056245145
[2025-05-20 18:45:11.267] [error] [vue] { code: 1, message: 'exit with code 1 and signal null' }
I’m using the latest beta, the problem is on the server side.
[2025-05-20 18:44:54.153] [debug] [mapillary:tools] 2025-05-20 18:44:54,153 - DEBUG - HTTP 412 (Precondition Failed): {"debug_info": {"retriable": true, "type": "AuthorizationFailedError", "message": "A temporary failure has occurred. Please try again."}}
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] Traceback (most recent call last):
File "mapillary_tools\upload.py", line 568, in upload
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"AuthorizationFailedError","message":"A temporary failure has occurred. Please try again."}}
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 8, in <module>
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] File "mapillary_tools\commands\__main__.py", line 164, in main
File "mapillary_tools\commands\upload.py", line 50, in run
File "mapillary_tools\upload.py", line 651, in upload
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 563, in upload
File "mapillary_tools\uploader.py", line 163, in upload_images
File "mapillary_tools\uploader.py", line 219, in upload_stream
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 425, in _upload_stream
File "mapillary_tools\uploader.py", line 398, in _upload_stream
File "mapillary_tools\upload_api_v4.py", line 89, in upload
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks
RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"AuthorizationFailedError","message":"A temporary failure has occurred. Please try again."}}
[8112] Failed to execute script 'main' due to unhandled exception!
[2025-05-20 18:44:54.528] [debug] [mapillary:tools] process exit code 1 and signal null
[2025-05-20 18:44:54.543] [error] [store:modules:uploadSession] exit with code 1 and signal null