Thank you for the clarification. I have read the RFCs but could not figure out the exact difference in semantics between PUT and POST. So, in a chunked scenario a first PUT request would create a new resource (file) on the server and subsequent chunk requests would be POSTed. Why do I mention this; Could your HTTP server handle PUT versus POST created resources differently later on, i.e. impact resumes or upload session migration?
If I recall correctly it fell to 26%. I will have to use the --verbose option next time to say for sure.
@tao Things are getting weirder!? 
The server shuts down on completion? 
Uploading ZIP mly_tools_ccfce8717369e5ac30a466be384e9bb9.zip (1/1): 100%|██████████████████████████████████████████████████████████████████████████████| 5.50G/5.50G [1:18:17<00:00, 1.27MB/s]
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 622, in upload
_upload_zipfiles(mly_uploader, zip_paths)
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 677, in _upload_zipfiles
raise UploadError(ex) from ex
mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jake/.local/bin/mapillary_tools", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/__main__.py", line 164, in main
args.func(argvars)
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/upload.py", line 50, in run
upload(**args)
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 651, in upload
raise inner_ex
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 673, in _upload_zipfiles
cluster_id = mly_uploader.upload_zipfile(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 132, in upload_zipfile
return self.upload_stream(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 219, in upload_stream
return _upload_stream(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 425, in _upload_stream
raise ex
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 398, in _upload_stream
file_handle = upload_service.upload(fp, offset=begin_offset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload_api_v4.py", line 89, in upload
return self.upload_chunks(chunks, offset=offset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload_api_v4.py", line 151, in upload_chunks
raise RuntimeError(
RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}
Uploading ZIP mly_tools_ccfce8717369e5ac30a466be384e9bb9.zip (1/1): 100%|██████████████████████████████████████████████████████████████████████████████| 5.50G/5.50G [1:18:18<00:00, 1.26MB/s]
1 Like
Welcome to the club, but I’ve been having this kind of thing for a month, sometimes several times a day sometimes 1 a day, although just recently it has calmed down.
>
> [2025-04-27 20:05:01.501] [debug] [mapillary:tools] 2025-04-27 20:05:01,501 - DEBUG - HTTP 400 (Bad Request): {"debug_info": {"retriable": false, "type": "ProcessingFailedError", "message": "Request processing failed"}}
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] Traceback (most recent call last):
> File "mapillary_tools\upload.py", line 568, in upload
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools]
>
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "main.py", line 8, in <module>
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] File "mapillary_tools\commands\__main__.py", line 164, in main
> File "mapillary_tools\commands\upload.py", line 50, in run
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 651, in upload
> File "mapillary_tools\upload.py", line 563, in upload
> File "mapillary_tools\uploader.py", line 163, in upload_images
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 219, in upload_stream
> File "mapillary_tools\uploader.py", line 425, in _upload_stream
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 398, in _upload_stream
> File "mapillary_tools\upload_api_v4.py", line 89, in upload
> File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks
>
> [2025-04-27 20:05:01.595] [debug] [mapillary:tools] RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}
> [6372] Failed to execute script 'main' due to unhandled exception!
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] Traceback (most recent call last):
File "mapillary_tools\upload.py", line 568, in upload
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] mapillary_tools.upload.UploadError:
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 8, in <module>
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\commands\__main__.py", line 164, in main
File "mapillary_tools\commands\upload.py", line 50, in run
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 651, in upload
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 563, in upload
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 163, in upload_images
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 219, in upload_stream
File "mapillary_tools\uploader.py", line 425, in _upload_stream
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 398, in _upload_stream
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 89, in upload
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks
RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}
[864] Fail
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] ed to execute script 'main' due to unha
[2025-05-09 09:49:27.046] [debug] [mapillary:tools] ndled exception!
1 Like
@tao - any thoughts on these errors?
Same. Uploads started failing roughly around 15:30 UTC; constantly repeating HTTPError: 412s.
fyi, I’m using mapillary_tools 0.10.0, just in case it’s an issue with using an old version.
1 Like
Also issue while using windows application:
[2025-05-20 18:45:10.797] [debug] [mapillary:tools] 2025-05-20 18:45:10,795 - DEBUG - Sending upload_progress via IPC: {'file_type': 'image', 'sequence_idx': 0, 'total_sequence_count': 1, 'sequence_image_count': 999, 'sequence_uuid': '0', 'entity_size': 2331715097, 'md5sum': 'c9c815c8b144704f269a9be91873228b', 'upload_start_time': 1747759383.75809, 'upload_total_time': 0, 'offset': 2331715097, 'retries': 19, 'upload_last_restart_time': 1747759387.7296193, 'upload_first_offset': 0, 'chunk_size': 16459289}
[2025-05-20 18:45:10.802] [debug] [mapillary:tools] 2025-05-20 18:45:10,802 - DEBUG - HTTP 412 (Precondition Failed): {"debug_info": {"retriable": true, "type": "AuthorizationFailedError", "message": "A temporary failure has occurred. Please try again."}}
[2025-05-20 18:45:11.097] [debug] [mapillary:tools] Traceback (most recent call last):
File "mapillary_tools\upload.py", line 568, in upload
[2025-05-20 18:45:11.099] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"AuthorizationFailedError","message":"A temporary failure has occurred. Please try again."}}
[2025-05-20 18:45:11.100] [debug] [mapillary:tools] During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 8, in <module>
[2025-05-20 18:45:11.100] [debug] [mapillary:tools] File "mapillary_tools\commands\__main__.py", line 164, in main
[2025-05-20 18:45:11.100] [debug] [mapillary:tools] File "mapillary_tools\commands\upload.py", line 50, in run
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 651, in upload
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 563, in upload
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 163, in upload_images
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 219, in upload_stream
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 425, in _upload_stream
[2025-05-20 18:45:11.101] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 398, in _upload_stream
[2025-05-20 18:45:11.102] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 89, in upload
[2025-05-20 18:45:11.102] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks
[2025-05-20 18:45:11.102] [debug] [mapillary:tools] RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"AuthorizationFailedError","message":"A temporary failure has occurred. Please try again."}}
[7588] Failed to exe
[2025-05-20 18:45:11.102] [debug] [mapillary:tools] cute script 'main' due to unhandled
[2025-05-20 18:45:11.103] [debug] [mapillary:tools] exception!
[2025-05-20 18:45:11.204] [debug] [mapillary:tools] process exit code 1 and signal null
[2025-05-20 18:45:11.208] [error] [store:modules:uploadSession] exit with code 1 and signal null
[2025-05-20 18:45:11.265] [info] [power-saver-blocker] stop: 4
[2025-05-20 18:45:11.265] [info] [services:observer-manager] Stopping 6ec07360-3599-11f0-bf28-4d8056245145
[2025-05-20 18:45:11.267] [error] [vue] { code: 1, message: 'exit with code 1 and signal null' }
2 Likes
I’m using the latest beta, the problem is on the server side.
3 Likes
[2025-05-20 18:44:54.153] [debug] [mapillary:tools] 2025-05-20 18:44:54,153 - DEBUG - HTTP 412 (Precondition Failed): {"debug_info": {"retriable": true, "type": "AuthorizationFailedError", "message": "A temporary failure has occurred. Please try again."}}
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] Traceback (most recent call last):
File "mapillary_tools\upload.py", line 568, in upload
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"AuthorizationFailedError","message":"A temporary failure has occurred. Please try again."}}
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 8, in <module>
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] File "mapillary_tools\commands\__main__.py", line 164, in main
File "mapillary_tools\commands\upload.py", line 50, in run
File "mapillary_tools\upload.py", line 651, in upload
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] File "mapillary_tools\upload.py", line 563, in upload
File "mapillary_tools\uploader.py", line 163, in upload_images
File "mapillary_tools\uploader.py", line 219, in upload_stream
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] File "mapillary_tools\uploader.py", line 425, in _upload_stream
File "mapillary_tools\uploader.py", line 398, in _upload_stream
File "mapillary_tools\upload_api_v4.py", line 89, in upload
[2025-05-20 18:44:54.293] [debug] [mapillary:tools] File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks
RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"AuthorizationFailedError","message":"A temporary failure has occurred. Please try again."}}
[8112] Failed to execute script 'main' due to unhandled exception!
[2025-05-20 18:44:54.528] [debug] [mapillary:tools] process exit code 1 and signal null
[2025-05-20 18:44:54.543] [error] [store:modules:uploadSession] exit with code 1 and signal null
Sorry for the inconvenience. We are aware of the issue and are working on recover the upload service as soon as possible.
3 Likes
@czecko @TheWizard @Eric_S @GITNE The issue should be mitigated. Let me know if it works for you.
1 Like
Thanks! I don’t have any data to upload now, I will check tomorrow.
Yes, working better.
Thanks
I have not had a blip so far.
1 Like