Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}

tao · May 20, 2025, 5:30pm

Sorry for the inconvenience. We are aware of the issue and are working on recover the upload service as soon as possible.

tao · May 20, 2025, 7:18pm

@czecko @TheWizard @Eric_S @GITNE The issue should be mitigated. Let me know if it works for you.

TheWizard · May 20, 2025, 7:39pm

Thanks! I don’t have any data to upload now, I will check tomorrow.

Eric_S · May 20, 2025, 9:40pm

Yes, working better.

Thanks

GITNE · May 26, 2025, 3:23pm

I have not had a blip so far.

czecko · June 4, 2025, 12:46pm

Now I saw from yesterday

[2025-06-03 19:42:12.584] [debug] [mapillary:tools] 2025-06-03 19:42:12,584 - DEBUG - HTTP 412 (Precondition Failed): {“debug_info”: {“retriable”: true, “type”: “ShutdownError”, “message”: “Server shutting down. Please try again.”}}

[2025-06-03 19:42:12.693] [debug] [mapillary:tools] Traceback (most recent call last):
File “mapillary_tools\upload.py”, line 568, in upload

[2025-06-03 19:42:12.693] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {“debug_info”:{“retriable”:true,“type”:“ShutdownError”,“message”:“Server shutting down. Please try again.”}}

[2025-06-03 19:42:12.693] [debug] [mapillary:tools] During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “main.py”, line 8, in

[2025-06-03 19:42:12.693] [debug] [mapillary:tools] File “mapillary_tools\commands_main_.py”, line 164, in main
File “mapillary_tools\commands\upload.py”, line 50, in run

[2025-06-03 19:42:12.693] [debug] [mapillary:tools] File “mapillary_tools\upload.py”, line 651, in upload
File “mapillary_tools\upload.py”, line 563, in upload

[2025-06-03 19:42:12.693] [debug] [mapillary:tools] File “mapillary_tools\uploader.py”, line 163, in upload_images
File “mapillary_tools\uploader.py”, line 219, in upload_stream
File “mapillary_tools\uploader.py”, line 425, in _upload_stream

[2025-06-03 19:42:12.693] [debug] [mapillary:tools] File “mapillary_tools\uploader.py”, line 398, in _upload_stream
File “mapillary_tools\upload_api_v4.py”, line 89, in upload

[2025-06-03 19:42:12.693] [debug] [mapillary:tools] File “mapillary_tools\upload_api_v4.py”, line 151, in upload_chunks
RuntimeError: Upload server error: File handle not found in the upload response {“debug_info”:{“retriable”:true,“type”:“ShutdownError”,“message”:“Server shutting down. Please try again.”}}
[8720] Failed to execute script ‘main’ due to unhandled excep
[2025-06-03 19:42:12.709] [debug] [mapillary:tools] tion!

[2025-06-03 19:42:12.912] [debug] [mapillary:tools] process exit code 1 and signal null
[2025-06-03 19:42:12.927] [error] [store:modules:uploadSession] exit with code 1 and signal null
[2025-06-03 19:42:12.959] [info] [power-saver-blocker] stop: 4
[2025-06-03 19:42:12.974] [info] [services:observer-manager] Stopping ef9bdc10-4091-11f0-b72c-5552c7bf4067
[2025-06-03 19:42:12.974] [error] [vue] { code: 1, message: ‘exit with code 1 and signal null’ }

czecko · June 8, 2025, 2:39pm

Today

[2025-06-08 15:21:57.366] [debug] [mapillary:tools] 2025-06-08 15:21:57,366 - DEBUG   - HTTP 400 (Bad Request): {"debug_info": {"retriable": false, "type": "ProcessingFailedError", "message": "Request processing failed"}}

[2025-06-08 15:21:57.491] [debug] [mapillary:tools] Traceback (most recent call last):
  File "mapillary_tools\upload.py", line 568, in upload

[2025-06-08 15:21:57.491] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}


[2025-06-08 15:21:57.491] [debug] [mapillary:tools] During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 8, in <module>

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\commands\__main__.py", line 164, in main

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\commands\upload.py", line 50, in run

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\upload.py", line 651, in upload

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\upload.py", line 563, in upload

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\uploader.py", line 163, in upload_images

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\uploader.py", line 219, in upload_stream

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\uploader.py", line 425, in _upload_stream

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\uploader.py", line 398, in _upload_stream

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\upload_api_v4.py", line 89, in upload

[2025-06-08 15:21:57.491] [debug] [mapillary:tools]   File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks

[2025-06-08 15:21:57.491] [debug] [mapillary:tools] RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":false,"type":"ProcessingFailedError","message":"Request processing failed"}}

[2025-06-08 15:21:57.491] [debug] [mapillary:tools] [5732] Failed to execute scrip
[2025-06-08 15:21:57.491] [debug] [mapillary:tools] t 'main' due to unhandled e
[2025-06-08 15:21:57.507] [debug] [mapillary:tools] xception!

[2025-06-08 15:21:57.757] [debug] [mapillary:tools] process exit code 1 and signal null
[2025-06-08 15:21:57.757] [error] [store:modules:uploadSession] exit with code 1 and signal null

HellPhoto · June 9, 2025, 6:44pm

Got the same message too for the first time (DU 4.7.2).

[2025-06-09 21:37:24.328] [debug] [mapillary:tools] 2025-06-09 21:37:24,327 - DEBUG   - HTTP 412 (Precondition Failed): {"debug_info": {"retriable": true, "type": "ShutdownError", "message": "Server shutting down. Please try again."}}

[2025-06-09 21:37:24.385] [debug] [mapillary:tools] Traceback (most recent call last):
  File "mapillary_tools\upload.py", line 568, in upload

[2025-06-09 21:37:24.385] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 8, in <module>
  File "mapillary_tools\commands\__main__.py", line 164, in main

[2025-06-09 21:37:24.385] [debug] [mapillary:tools]   File "mapillary_tools\commands\upload.py", line 50, in run
  File "mapillary_tools\upload.py", line 651, in upload
  File "mapillary_tools\upload.py", line 563, in upload

[2025-06-09 21:37:24.386] [debug] [mapillary:tools]   File "mapillary_tools\uploader.py", line 163, in upload_images
  File "mapillary_tools\uploader.py", line 219, in upload_stream

[2025-06-09 21:37:24.386] [debug] [mapillary:tools]   File "mapillary_tools\uploader.py", line 425, in _upload_stream
  File "mapillary_tools\uploader.py", line 398, in _upload_stream
  File "mapillary_tools\upload_api_v4.py", line 89, in upload
  File "mapillary_tools\upload_api_v4.py", line 151, in upload_chunks

tao · June 9, 2025, 10:02pm

Hey @HellPhoto @czecko there is a bug in current version of mapillary_tools used in Desktop Uploader that there is no retries when some retryable errors happens. The bug is fixed in v0.14. I’ll be working with @nikola get it integrated in the next DU release soon. Sorry for the inconvenience. cc @boris

czecko · June 18, 2025, 10:25am

Well, and this is more concrete. But I continued to have it 3 days ago and yesterday. Maybe some more specific release date …?

GITNE · June 23, 2025, 7:49pm

Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1):  84%|█████████████████████████████████████████████████████████████████▋            | 8.71G/10.3G [2:03:10<23:01, 1.27MB/s]
2025-06-23 20:38:55,726 - WARNING - Error uploading chunk_size 5242880 at begin_offset 0: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_2186896eca12a45812c52b1c41ba6596.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-23 20:38:55,727 - INFO    - Retrying in 2 seconds (1/200)
Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1):  84%|█████████████████████████████████████████████████████████████████▋            | 8.71G/10.3G [2:19:38<26:12, 1.12MB/s]
Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1):  26%|████████████████████▌                                                         | 2.72G/10.3G [38:24<1:47:38, 1.27MB/s]

I have just had the SSLError reset happen again on mapillary_tools version 0.13.3 because I forgot to map *.mapillary.com and *.facebook.com domain names to static IP addresses. Looking forward to version 0.14.

GITNE · June 23, 2025, 10:10pm

Upload complete but Upload server error: File handle not found in the upload response.

Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1): 100%|██████████████████████████████████████████████████████████████████████████████| 10.3G/10.3G [2:26:06<00:00, 1.27MB/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 622, in upload
    _upload_zipfiles(mly_uploader, zip_paths)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 677, in _upload_zipfiles
    raise UploadError(ex) from ex
mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jake/.local/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/__main__.py", line 164, in main
    args.func(argvars)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/upload.py", line 50, in run
    upload(**args)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 651, in upload
    raise inner_ex
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 673, in _upload_zipfiles
    cluster_id = mly_uploader.upload_zipfile(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 132, in upload_zipfile
    return self.upload_stream(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 219, in upload_stream
    return _upload_stream(
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 425, in _upload_stream
    raise ex
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 398, in _upload_stream
    file_handle = upload_service.upload(fp, offset=begin_offset)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload_api_v4.py", line 89, in upload
    return self.upload_chunks(chunks, offset=offset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload_api_v4.py", line 151, in upload_chunks
    raise RuntimeError(
RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}
Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1): 100%|██████████████████████████████████████████████████████████████████████████████| 10.3G/10.3G [2:26:07<00:00, 1.27MB/s]

@tao The server just does not like me or something, I guess. I have also tried to pip3 install the 0.14 alpha versions without success because they have trouble resolving all dependencies.

GITNE · June 27, 2025, 11:02am

I continue to see SSLErrors and resets to 0

2025-06-27 12:31:05,997 - DEBUG   - mapillary_tools version 0.13.3
2025-06-27 12:31:07,035 - DEBUG   - Sending upload_start via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 0}
2025-06-27 12:31:07,036 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:31:07,560 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:31:07,581 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 0, 'offset': 3531309048, 'retries': 0, 'upload_last_restart_time': 1751020267.581793, 'upload_first_offset': 3531309048}
2025-06-27 12:31:07,584 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:32:26,135 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:32:26,136 - INFO    - Retrying in 2 seconds (1/200)
2025-06-27 12:32:28,136 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:32:28,494 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:32:28,497 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 78.55387687683105, 'offset': 3531309048, 'retries': 1, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020348.497818}
2025-06-27 12:32:28,499 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:33:40,998 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:33:40,999 - INFO    - Retrying in 4 seconds (2/200)
2025-06-27 12:33:44,999 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:33:45,349 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:33:45,353 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 151.0547273159027, 'offset': 3531309048, 'retries': 2, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020425.3534167}
2025-06-27 12:33:45,355 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:35:02,843 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:35:02,844 - INFO    - Retrying in 8 seconds (3/200)
2025-06-27 12:35:10,844 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:35:11,208 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:35:11,212 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 228.54505109786987, 'offset': 3531309048, 'retries': 3, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020511.2124054}
2025-06-27 12:35:11,215 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:36:25,798 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:36:25,798 - INFO    - Retrying in 16 seconds (4/200)
2025-06-27 12:36:41,798 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:36:42,282 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:36:42,285 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 303.1306622028351, 'offset': 3531309048, 'retries': 4, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020602.2856336}
2025-06-27 12:36:42,287 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:37:54,376 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:37:54,377 - INFO    - Retrying in 16 seconds (5/200)
2025-06-27 12:38:10,377 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:38:10,781 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:38:10,785 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 375.2215664386749, 'offset': 3531309048, 'retries': 5, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020690.7851884}
2025-06-27 12:38:10,786 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:39:22,708 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:39:22,709 - INFO    - Retrying in 16 seconds (6/200)
2025-06-27 12:39:38,709 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:39:39,202 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:39:39,205 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 447.1450505256653, 'offset': 3531309048, 'retries': 6, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020779.2059026}
2025-06-27 12:39:39,207 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:40:53,111 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:40:53,112 - INFO    - Retrying in 16 seconds (7/200)
2025-06-27 12:41:09,112 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:41:09,463 - DEBUG   - HTTP 200 (OK): b'{"dc":"frc3c34","offset":0}'
2025-06-27 12:41:09,467 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 521.0509986877441, 'offset': 0, 'retries': 7, 'upload_first_offset': 0, 'upload_last_restart_time': 1751020869.466991}
2025-06-27 12:41:09,468 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '0', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:41:13,537 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 521.0509986877441, 'offset': 5242880, 'retries': 7, 'upload_first_offset': 0, 'upload_last_restart_time': 1751020869.466991, 'chunk_size': 5242880}

This is really annoying because uploads that already take quite a long time to complete take like 3× longer than they should. There is nothing I can do on my side. This whole upload mess situation strongly discourages contributing.

tao · July 1, 2025, 12:10am

@GITNE could you try out Release v0.14.0b1 · mapillary/mapillary_tools · GitHub and see if it works for you? This version uploads images file by file (so no more zipping) so theoretically very low odds for reset as it’s uploading small files

GITNE · July 1, 2025, 12:52pm

How do you create a sequence? By directory or by explicit file names, etc?

@tao There are bugs in the --cutoff_distance and --cutoff_time options if you set them to +inf to turn them off.

--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/__init__.py", line 1160, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 999, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 703, in format
    record.message = record.getMessage()
                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 392, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
OverflowError: cannot convert float infinity to integer
Call stack:
  File "/usr/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.12/site-packages/mapillary_tools/commands/__main__.py", line 163, in main
    args.func(argvars)
  File "/usr/lib/python3.12/site-packages/mapillary_tools/commands/process.py", line 218, in run
    metadatas = process_sequence_properties(
  File "/usr/lib/python3.12/site-packages/mapillary_tools/process_sequence_properties.py", line 637, in process_sequence_properties
    sequences = _split_sequences_by_cutoff_time(sequences, cutoff_time=cutoff_time)
  File "/usr/lib/python3.12/site-packages/mapillary_tools/process_sequence_properties.py", line 425, in _split_sequences_by_cutoff_time
    LOG.info(
Message: 'Found %s sequences after split by cutoff_time %d seconds'
Arguments: (1, inf)
2025-07-01 15:33:30,383 - INFO    - Found 1 sequences and 0 errors after duplication check
2025-07-01 15:33:30,574 - INFO    - Found 25 sequences after split by sequence limits
2025-07-01 15:33:30,732 - INFO    - Found 25 sequences and 0 errors after sequence limit checks
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/__init__.py", line 1160, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 999, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 703, in format
    record.message = record.getMessage()
                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 392, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
OverflowError: cannot convert float infinity to integer
Call stack:
  File "/usr/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.12/site-packages/mapillary_tools/commands/__main__.py", line 163, in main
    args.func(argvars)
  File "/usr/lib/python3.12/site-packages/mapillary_tools/commands/process.py", line 218, in run
    metadatas = process_sequence_properties(
  File "/usr/lib/python3.12/site-packages/mapillary_tools/process_sequence_properties.py", line 671, in process_sequence_properties
    sequences = _split_sequences_by_cutoff_distance(
  File "/usr/lib/python3.12/site-packages/mapillary_tools/process_sequence_properties.py", line 464, in _split_sequences_by_cutoff_distance
    LOG.info(
Message: 'Found %s sequences after split by cutoff_distance %d meters'
Arguments: (25, inf)

Note that you can turn off other options, like --duplicate_distance or --duplicate_angle by setting them to 0.0. Please note also that 0.0 is a valid value for --cutoff_distance and --cutoff_time, so no turning off these with 0.0. +inf, +Infinity, inf, or Infinity are the only logic value to turn off --cutoff_* options.

GITNE · July 1, 2025, 4:29pm

The --cutoff_* options behave strangely even with finite values

2025-07-01 18:17:26,453 - INFO    - Found 1 sequences from different folders and cameras
2025-07-01 18:17:26,846 - INFO    - Found 1 sequences after split by cutoff_time 7200 seconds
2025-07-01 18:17:27,083 - INFO    - Found 1 sequences and 0 errors after duplication check
2025-07-01 18:17:27,356 - INFO    - Found 25 sequences after split by sequence limits
2025-07-01 18:17:27,583 - INFO    - Found 25 sequences and 0 errors after sequence limit checks
2025-07-01 18:17:27,796 - INFO    - Found 25 sequences after split by cutoff_distance 65535 meters

These values should result in exactly one sequence in this case.
Please note also that at some point in the code these values are implicitly converted from floats to ints, which probably should not happen.

GITNE · July 1, 2025, 8:40pm

@tao I have just found out that the MAPILLARY_TOOLS_MAX_SEQUENCE_LENGTH environment variable/constant breaks the --cutoff_* options. Please drop it completely due to How Many Images per Sequence? - #6 by boris because otherwise it is just counter productive. Besides, setting MAPILLARY_TOOLS_MAX_SEQUENCE_LENGTH to a really large number does not work as expected either, so that it is broken anyway.
Or, if you really want to keep it for some other reason then please replace it with a --cutoff_count option to make its existence less obscure to users.

GITNE · July 2, 2025, 12:27am

The results for version 0.14.0b1 are rather disappointing.

Like I have mentioned before, controlling sequence creation is overly cumbersome, buggy, and does not work as expected.
Initiating the actual upload action takes waaaaaaay too long. Note that mapillary_tools hogged a CPU core at 100% for over 3 hours before anything went over the wire. This is unacceptable. I’ll say it again: we are just uploading files. No rocket science here.
Upload did not complete.
The process command reads complete image files instead of Exif headers only, hence creating sequences takes much much longer than it actually needs to be.

2025-07-01 22:47:49,827 - INFO    - Uploading to profile "GITNE": {'MAPSettingsUserKey': 'XXXXXXXXXXXXXXX', 'user_upload_token': '[REDACTED]', 'MAPSettingsUsername': 'GITNE'}
2025-07-01 22:47:50,373 - INFO    - Uploading to Mapillary organization: {"slug": "XXXXXXXXXXXXXXX", "name": "XXXXXXXXXXXXXXXXXX", "id": "XXXXXXXXXXXXXXXX"}
Uploading IMAGE (1/1):   0%|                                                                                                                                      | 0.00/52.3G [00:00<?, ?B/s]
2025-07-02 01:50:46,045 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_d90aa9ff8bbd57cf7841fe43460b4f30.jpg (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa4fb4fe0>: Failed to resolve 'rupload.facebook.com' ([Errno -3] Temporary failure in name resolution)"))
2025-07-02 01:50:46,196 - INFO    - Retrying in 2 seconds (1/200)
2025-07-02 01:50:46,131 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_a5ccce0db982981bfd6d9c709663c237.jpg (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa0da9850>: Failed to resolve 'rupload.facebook.com' ([Errno -3] Temporary failure in name resolution)"))
2025-07-02 01:50:46,480 - INFO    - Retrying in 2 seconds (1/200)
2025-07-02 01:50:46,202 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_eb407de6d385da30274bf6314cffd9c8.jpg (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa0daa540>: Failed to resolve 'rupload.facebook.com' ([Errno -3] Temporary failure in name resolution)"))
2025-07-02 01:50:46,608 - INFO    - Retrying in 2 seconds (1/200)
2025-07-02 01:50:46,423 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_73a68f038085c7300d026f928ed2314f.jpg (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa5d4ab70>: Failed to resolve 'rupload.facebook.com' ([Errno -3] Temporary failure in name resolution)"))
2025-07-02 01:50:46,649 - INFO    - Retrying in 2 seconds (1/200)
Uploading IMAGE (1/1):   0%|▎                                                                                                                          | 155M/52.3G [02:55<14:39:48, 1.06MB/s]
2025-07-02 01:53:41,358 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ReadTimeout: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Read timed out. (read timeout=60)
2025-07-02 01:53:41,445 - INFO    - Retrying in 2 seconds (1/200)
Uploading IMAGE (1/1):   1%|█                                                                                                                           | 471M/52.3G [08:35<73:12:13, 211kB/s]
2025-07-02 01:59:28,506 - INFO    - Nothing uploaded. Bye.
2025-07-02 01:59:28,508 - ERROR   - Upload error: HTTPError: 408 Client Error: Client timeout for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_7024806ee88541f64f90ad5b8efdf991.jpg
Uploading IMAGE (1/1):   1%|█                                                                                                                           | 471M/52.3G [08:44<16:26:45, 941kB/s]

Generally speaking, I do not have anything against zipping sequences per se. It is just that for the purpose of avoiding double redundant storage requirements it would be nice if we could stream a ZIP file into mapillary_tools when uploading.

tao · July 2, 2025, 1:10am

Thanks for the great feedback @GITNE, as always.

I will try if I can integrate MAPILLARY_TOOLS_MAX_SEQUENCE_LENGTH into the cutoff_* logic, i.e. whenever any of the criteria happens, the cutoff will happen. It’s not going to be simple though, because there are other things to consider (for example, we can’t cut by distance early because if so we can’t check sequence anomaly by its speed, which is another common issue). BTW if you turn off --verbose mode it will show you when and why a sequence is cut
This is not expected. Was it upload command or process command hanging for 3 hours? Sharing a detailed logs would be helpful. If it’s process, I wonder if it’s related to the default exiftool geotagging process, you can disable it by --geotag=native (by default it’s --geotag=native,exiftool_runtime).
“Reading EXIF headers only” is exactly one of key improvements we made in 0.14, so v0.14 should be much faster than the previous versions. If not, then something is wrong and we should fix it.
zip images zip_dir and then upload zip_dir should still work

GITNE · July 2, 2025, 7:31am

It was the upload command. You can see it in the log by timestamps. At some point, I was starting to suspect an infinite loop. But no, it just takes this long.

Maybe I have missed something because I have not used the process command until now (was creating ZIPs myself). More specifically, I mean the Extracting images step to be needlessly slow. JOSM is about 3× faster reading image headers and it is bytecode interpreted too. And no, JOSM’s Exif access library is not native code, it is a pure Java bytecode implementation (and I it for that!).
It does not seem like much of a difference at first glance but when you deal with thousands of images then things add up quickly. Maybe there is something else slowing down reading headers in your loop? Perhaps creating too many throwaway objects? Building strings?

Topic		Replies	Views
Mapillary Tools 0.14 is released Command line tools	52	720	September 9, 2025
"OSError: [Errno 28] No space left on device" when uploading Command line tools	30	11714	August 6, 2025
Issue with upload?	11	191	February 28, 2026
Been here forever but going, going, gone Introduction	42	4266	November 13, 2021
403 UPLOAD Error Contributing and equipment	45	2836	January 11, 2020

Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}

Related topics