Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}

GITNE · June 23, 2025, 7:49pm

Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1):  84%|█████████████████████████████████████████████████████████████████▋            | 8.71G/10.3G [2:03:10<23:01, 1.27MB/s]
2025-06-23 20:38:55,726 - WARNING - Error uploading chunk_size 5242880 at begin_offset 0: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_2186896eca12a45812c52b1c41ba6596.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-23 20:38:55,727 - INFO    - Retrying in 2 seconds (1/200)
Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1):  84%|█████████████████████████████████████████████████████████████████▋            | 8.71G/10.3G [2:19:38<26:12, 1.12MB/s]
Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1):  26%|████████████████████▌                                                         | 2.72G/10.3G [38:24<1:47:38, 1.27MB/s]

I have just had the SSLError reset happen again on mapillary_tools version 0.13.3 because I forgot to map *.mapillary.com and *.facebook.com domain names to static IP addresses. Looking forward to version 0.14.

GITNE · June 23, 2025, 10:10pm

Upload complete but Upload server error: File handle not found in the upload response.

Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1): 100%|██████████████████████████████████████████████████████████████████████████████| 10.3G/10.3G [2:26:06<00:00, 1.27MB/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 622, in upload
    _upload_zipfiles(mly_uploader, zip_paths)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 677, in _upload_zipfiles
    raise UploadError(ex) from ex
mapillary_tools.upload.UploadError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jake/.local/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/__main__.py", line 164, in main
    args.func(argvars)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/upload.py", line 50, in run
    upload(**args)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 651, in upload
    raise inner_ex
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 673, in _upload_zipfiles
    cluster_id = mly_uploader.upload_zipfile(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 132, in upload_zipfile
    return self.upload_stream(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 219, in upload_stream
    return _upload_stream(
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 425, in _upload_stream
    raise ex
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 398, in _upload_stream
    file_handle = upload_service.upload(fp, offset=begin_offset)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload_api_v4.py", line 89, in upload
    return self.upload_chunks(chunks, offset=offset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload_api_v4.py", line 151, in upload_chunks
    raise RuntimeError(
RuntimeError: Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}
Uploading ZIP mly_tools_2186896eca12a45812c52b1c41ba6596.zip (1/1): 100%|██████████████████████████████████████████████████████████████████████████████| 10.3G/10.3G [2:26:07<00:00, 1.27MB/s]

@tao The server just does not like me or something, I guess. I have also tried to pip3 install the 0.14 alpha versions without success because they have trouble resolving all dependencies.

GITNE · June 27, 2025, 11:02am

I continue to see SSLErrors and resets to 0

2025-06-27 12:31:05,997 - DEBUG   - mapillary_tools version 0.13.3
2025-06-27 12:31:07,035 - DEBUG   - Sending upload_start via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 0}
2025-06-27 12:31:07,036 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:31:07,560 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:31:07,581 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 0, 'offset': 3531309048, 'retries': 0, 'upload_last_restart_time': 1751020267.581793, 'upload_first_offset': 3531309048}
2025-06-27 12:31:07,584 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:32:26,135 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:32:26,136 - INFO    - Retrying in 2 seconds (1/200)
2025-06-27 12:32:28,136 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:32:28,494 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:32:28,497 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 78.55387687683105, 'offset': 3531309048, 'retries': 1, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020348.497818}
2025-06-27 12:32:28,499 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:33:40,998 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:33:40,999 - INFO    - Retrying in 4 seconds (2/200)
2025-06-27 12:33:44,999 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:33:45,349 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:33:45,353 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 151.0547273159027, 'offset': 3531309048, 'retries': 2, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020425.3534167}
2025-06-27 12:33:45,355 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:35:02,843 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:35:02,844 - INFO    - Retrying in 8 seconds (3/200)
2025-06-27 12:35:10,844 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:35:11,208 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:35:11,212 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 228.54505109786987, 'offset': 3531309048, 'retries': 3, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020511.2124054}
2025-06-27 12:35:11,215 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:36:25,798 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:36:25,798 - INFO    - Retrying in 16 seconds (4/200)
2025-06-27 12:36:41,798 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:36:42,282 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:36:42,285 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 303.1306622028351, 'offset': 3531309048, 'retries': 4, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020602.2856336}
2025-06-27 12:36:42,287 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:37:54,376 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:37:54,377 - INFO    - Retrying in 16 seconds (5/200)
2025-06-27 12:38:10,377 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:38:10,781 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:38:10,785 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 375.2215664386749, 'offset': 3531309048, 'retries': 5, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020690.7851884}
2025-06-27 12:38:10,786 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:39:22,708 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:39:22,709 - INFO    - Retrying in 16 seconds (6/200)
2025-06-27 12:39:38,709 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:39:39,202 - DEBUG   - HTTP 200 (OK): b'{"dc":"eag6c03","offset":3531309048}'
2025-06-27 12:39:39,205 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 447.1450505256653, 'offset': 3531309048, 'retries': 6, 'upload_first_offset': 3531309048, 'upload_last_restart_time': 1751020779.2059026}
2025-06-27 12:39:39,207 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '3531309048', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:40:53,111 - WARNING - Error uploading chunk_size 5242880 at begin_offset 3531309048: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2406)')))
2025-06-27 12:40:53,112 - INFO    - Retrying in 16 seconds (7/200)
2025-06-27 12:41:09,112 - DEBUG   - HTTP GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60
2025-06-27 12:41:09,463 - DEBUG   - HTTP 200 (OK): b'{"dc":"frc3c34","offset":0}'
2025-06-27 12:41:09,467 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 521.0509986877441, 'offset': 0, 'retries': 7, 'upload_first_offset': 0, 'upload_last_restart_time': 1751020869.466991}
2025-06-27 12:41:09,468 - DEBUG   - HTTP POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_61168e42d3a40407863f3c220bab2d8e.zip HEADERS={'Authorization': '[REDACTED]', 'Offset': '0', 'X-Entity-Name': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'X-Entity-Type': 'application/zip'} TIMEOUT=(1800, 1800)
2025-06-27 12:41:13,537 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mly_tools_61168e42d3a40407863f3c220bab2d8e.zip', 'sequence_image_count': 9624, 'entity_size': 5287862671, 'md5sum': '61168e42d3a40407863f3c220bab2d8e', 'upload_start_time': 1751020267.0358744, 'upload_total_time': 521.0509986877441, 'offset': 5242880, 'retries': 7, 'upload_first_offset': 0, 'upload_last_restart_time': 1751020869.466991, 'chunk_size': 5242880}

This is really annoying because uploads that already take quite a long time to complete take like 3× longer than they should. There is nothing I can do on my side. This whole upload mess situation strongly discourages contributing.

tao · July 1, 2025, 12:10am

@GITNE could you try out Release v0.14.0b1 · mapillary/mapillary_tools · GitHub and see if it works for you? This version uploads images file by file (so no more zipping) so theoretically very low odds for reset as it’s uploading small files

GITNE · July 1, 2025, 12:52pm

How do you create a sequence? By directory or by explicit file names, etc?

@tao There are bugs in the --cutoff_distance and --cutoff_time options if you set them to +inf to turn them off.

--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/__init__.py", line 1160, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 999, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 703, in format
    record.message = record.getMessage()
                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 392, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
OverflowError: cannot convert float infinity to integer
Call stack:
  File "/usr/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.12/site-packages/mapillary_tools/commands/__main__.py", line 163, in main
    args.func(argvars)
  File "/usr/lib/python3.12/site-packages/mapillary_tools/commands/process.py", line 218, in run
    metadatas = process_sequence_properties(
  File "/usr/lib/python3.12/site-packages/mapillary_tools/process_sequence_properties.py", line 637, in process_sequence_properties
    sequences = _split_sequences_by_cutoff_time(sequences, cutoff_time=cutoff_time)
  File "/usr/lib/python3.12/site-packages/mapillary_tools/process_sequence_properties.py", line 425, in _split_sequences_by_cutoff_time
    LOG.info(
Message: 'Found %s sequences after split by cutoff_time %d seconds'
Arguments: (1, inf)
2025-07-01 15:33:30,383 - INFO    - Found 1 sequences and 0 errors after duplication check
2025-07-01 15:33:30,574 - INFO    - Found 25 sequences after split by sequence limits
2025-07-01 15:33:30,732 - INFO    - Found 25 sequences and 0 errors after sequence limit checks
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/__init__.py", line 1160, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 999, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 703, in format
    record.message = record.getMessage()
                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 392, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
OverflowError: cannot convert float infinity to integer
Call stack:
  File "/usr/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.12/site-packages/mapillary_tools/commands/__main__.py", line 163, in main
    args.func(argvars)
  File "/usr/lib/python3.12/site-packages/mapillary_tools/commands/process.py", line 218, in run
    metadatas = process_sequence_properties(
  File "/usr/lib/python3.12/site-packages/mapillary_tools/process_sequence_properties.py", line 671, in process_sequence_properties
    sequences = _split_sequences_by_cutoff_distance(
  File "/usr/lib/python3.12/site-packages/mapillary_tools/process_sequence_properties.py", line 464, in _split_sequences_by_cutoff_distance
    LOG.info(
Message: 'Found %s sequences after split by cutoff_distance %d meters'
Arguments: (25, inf)

Note that you can turn off other options, like --duplicate_distance or --duplicate_angle by setting them to 0.0. Please note also that 0.0 is a valid value for --cutoff_distance and --cutoff_time, so no turning off these with 0.0. +inf, +Infinity, inf, or Infinity are the only logic value to turn off --cutoff_* options.

GITNE · July 1, 2025, 4:29pm

The --cutoff_* options behave strangely even with finite values

2025-07-01 18:17:26,453 - INFO    - Found 1 sequences from different folders and cameras
2025-07-01 18:17:26,846 - INFO    - Found 1 sequences after split by cutoff_time 7200 seconds
2025-07-01 18:17:27,083 - INFO    - Found 1 sequences and 0 errors after duplication check
2025-07-01 18:17:27,356 - INFO    - Found 25 sequences after split by sequence limits
2025-07-01 18:17:27,583 - INFO    - Found 25 sequences and 0 errors after sequence limit checks
2025-07-01 18:17:27,796 - INFO    - Found 25 sequences after split by cutoff_distance 65535 meters

These values should result in exactly one sequence in this case.
Please note also that at some point in the code these values are implicitly converted from floats to ints, which probably should not happen.

GITNE · July 1, 2025, 8:40pm

@tao I have just found out that the MAPILLARY_TOOLS_MAX_SEQUENCE_LENGTH environment variable/constant breaks the --cutoff_* options. Please drop it completely due to How Many Images per Sequence? - #6 by boris because otherwise it is just counter productive. Besides, setting MAPILLARY_TOOLS_MAX_SEQUENCE_LENGTH to a really large number does not work as expected either, so that it is broken anyway.
Or, if you really want to keep it for some other reason then please replace it with a --cutoff_count option to make its existence less obscure to users.

GITNE · July 2, 2025, 12:27am

The results for version 0.14.0b1 are rather disappointing.

Like I have mentioned before, controlling sequence creation is overly cumbersome, buggy, and does not work as expected.
Initiating the actual upload action takes waaaaaaay too long. Note that mapillary_tools hogged a CPU core at 100% for over 3 hours before anything went over the wire. This is unacceptable. I’ll say it again: we are just uploading files. No rocket science here.
Upload did not complete.
The process command reads complete image files instead of Exif headers only, hence creating sequences takes much much longer than it actually needs to be.

2025-07-01 22:47:49,827 - INFO    - Uploading to profile "GITNE": {'MAPSettingsUserKey': 'XXXXXXXXXXXXXXX', 'user_upload_token': '[REDACTED]', 'MAPSettingsUsername': 'GITNE'}
2025-07-01 22:47:50,373 - INFO    - Uploading to Mapillary organization: {"slug": "XXXXXXXXXXXXXXX", "name": "XXXXXXXXXXXXXXXXXX", "id": "XXXXXXXXXXXXXXXX"}
Uploading IMAGE (1/1):   0%|                                                                                                                                      | 0.00/52.3G [00:00<?, ?B/s]
2025-07-02 01:50:46,045 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_d90aa9ff8bbd57cf7841fe43460b4f30.jpg (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa4fb4fe0>: Failed to resolve 'rupload.facebook.com' ([Errno -3] Temporary failure in name resolution)"))
2025-07-02 01:50:46,196 - INFO    - Retrying in 2 seconds (1/200)
2025-07-02 01:50:46,131 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_a5ccce0db982981bfd6d9c709663c237.jpg (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa0da9850>: Failed to resolve 'rupload.facebook.com' ([Errno -3] Temporary failure in name resolution)"))
2025-07-02 01:50:46,480 - INFO    - Retrying in 2 seconds (1/200)
2025-07-02 01:50:46,202 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_eb407de6d385da30274bf6314cffd9c8.jpg (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa0daa540>: Failed to resolve 'rupload.facebook.com' ([Errno -3] Temporary failure in name resolution)"))
2025-07-02 01:50:46,608 - INFO    - Retrying in 2 seconds (1/200)
2025-07-02 01:50:46,423 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_73a68f038085c7300d026f928ed2314f.jpg (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0xffffa5d4ab70>: Failed to resolve 'rupload.facebook.com' ([Errno -3] Temporary failure in name resolution)"))
2025-07-02 01:50:46,649 - INFO    - Retrying in 2 seconds (1/200)
Uploading IMAGE (1/1):   0%|▎                                                                                                                          | 155M/52.3G [02:55<14:39:48, 1.06MB/s]
2025-07-02 01:53:41,358 - WARNING - Error uploading chunk_size 10485760 at begin_offset None: ReadTimeout: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Read timed out. (read timeout=60)
2025-07-02 01:53:41,445 - INFO    - Retrying in 2 seconds (1/200)
Uploading IMAGE (1/1):   1%|█                                                                                                                           | 471M/52.3G [08:35<73:12:13, 211kB/s]
2025-07-02 01:59:28,506 - INFO    - Nothing uploaded. Bye.
2025-07-02 01:59:28,508 - ERROR   - Upload error: HTTPError: 408 Client Error: Client timeout for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_7024806ee88541f64f90ad5b8efdf991.jpg
Uploading IMAGE (1/1):   1%|█                                                                                                                           | 471M/52.3G [08:44<16:26:45, 941kB/s]

Generally speaking, I do not have anything against zipping sequences per se. It is just that for the purpose of avoiding double redundant storage requirements it would be nice if we could stream a ZIP file into mapillary_tools when uploading.

tao · July 2, 2025, 1:10am

Thanks for the great feedback @GITNE, as always.

I will try if I can integrate MAPILLARY_TOOLS_MAX_SEQUENCE_LENGTH into the cutoff_* logic, i.e. whenever any of the criteria happens, the cutoff will happen. It’s not going to be simple though, because there are other things to consider (for example, we can’t cut by distance early because if so we can’t check sequence anomaly by its speed, which is another common issue). BTW if you turn off --verbose mode it will show you when and why a sequence is cut
This is not expected. Was it upload command or process command hanging for 3 hours? Sharing a detailed logs would be helpful. If it’s process, I wonder if it’s related to the default exiftool geotagging process, you can disable it by --geotag=native (by default it’s --geotag=native,exiftool_runtime).
“Reading EXIF headers only” is exactly one of key improvements we made in 0.14, so v0.14 should be much faster than the previous versions. If not, then something is wrong and we should fix it.
zip images zip_dir and then upload zip_dir should still work

GITNE · July 2, 2025, 7:31am

It was the upload command. You can see it in the log by timestamps. At some point, I was starting to suspect an infinite loop. But no, it just takes this long.

Maybe I have missed something because I have not used the process command until now (was creating ZIPs myself). More specifically, I mean the Extracting images step to be needlessly slow. JOSM is about 3× faster reading image headers and it is bytecode interpreted too. And no, JOSM’s Exif access library is not native code, it is a pure Java bytecode implementation (and I it for that!).
It does not seem like much of a difference at first glance but when you deal with thousands of images then things add up quickly. Maybe there is something else slowing down reading headers in your loop? Perhaps creating too many throwaway objects? Building strings?

GITNE · July 4, 2025, 2:07pm

Just out of curiosity and to give you some feedback, I have run some process command tests on 0.13.3 versus 0.14.0b1. The result is that there is no improvement in the Extracting images step, whereas the Validating metadatas step only scores an about 7.5× speedup for me. So yes, there is indeed some improvement in 0.14.0b1 over 0.13.3. However, it looks like the Extracting images step may need some more attention. It continues to lag behind.

Topic		Replies	Views
[bugreport] mapillary_tools hangs on unhandled exception Command line tools	4	713	July 29, 2020
Desktop uploader fails to upload Contributing and equipment	3	528	October 5, 2021
Error while uploading with Windows Tools Command line tools	4	740	May 20, 2021
Upload failed. Sorry! Something went wrong Contributing and equipment	12	952	August 3, 2021
Mapillary uploads down Contributing and equipment	3	66	December 13, 2024

Upload server error: File handle not found in the upload response {"debug_info":{"retriable":true,"type":"ShutdownError","message":"Server shutting down. Please try again."}}

Related topics