Mapillary_tools upload errors

When I try to upload, I get network errors frequently. In fact it takes several times to upload 300 images. Sample run - I have a Wireshark trace also; it tries to reconnect but it’s as though the web host was down:

Mapillary tools 0.7.1

Uploading 305 images with valid mapillary tags (Skipping 0)
Compressing: 100%|████████████████████████████████████████████████████████████████████| 305/305 [00:16<00:00, 18.33it/s]
Uploading: 68%|████████████████████████████████████▌ | 155189248/229526817 [02:52<04:20, 285591.65it/s]
Traceback (most recent call last):
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/connection.py”, line 158, in _new_conn
conn = connection.create_connection(
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/util/connection.py”, line 80, in create_connection
raise err
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/util/connection.py”, line 70, in create_connection
sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/connectionpool.py”, line 597, in urlopen
httplib_response = self._make_request(conn, method, url,
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/connectionpool.py”, line 343, in _make_request
self._validate_conn(conn)
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/connectionpool.py”, line 839, in _validate_conn
conn.connect()
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/connection.py”, line 301, in connect
conn = self._new_conn()
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/connection.py”, line 167, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f939d058250>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/allusers/.local/lib/python3.8/site-packages/requests/adapters.py”, line 439, in send
resp = conn.urlopen(
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/connectionpool.py”, line 637, in urlopen
retries = retries.increment(method, url, error=e, _pool=self,
File “/home/allusers/.local/lib/python3.8/site-packages/urllib3/util/retry.py”, line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=‘rupload.facebook.com’, port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_50c65e9e6208960117ea95e8107774c5 (Caused by NewConnectionError(‘<urllib3.connection.VerifiedHTTPSConnection object at 0x7f939d058250>: Failed to establish a new connection: [Errno 101] Network is unreachable’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/allusers/.local/bin/mapillary_tools”, line 8, in
sys.exit(main())
File “/home/allusers/.local/lib/python3.8/site-packages/mapillary_tools/main.py”, line 93, in main
command.run(args)
File “/home/allusers/.local/lib/python3.8/site-packages/mapillary_tools/commands/upload.py”, line 112, in run
upload(
File “/home/allusers/.local/lib/python3.8/site-packages/mapillary_tools/upload.py”, line 141, in upload
uploader.upload_sequence_v4(
File “/home/allusers/.local/lib/python3.8/site-packages/mapillary_tools/uploader.py”, line 443, in upload_sequence_v4
upload_resp = service.upload(
File “/home/allusers/.local/lib/python3.8/site-packages/mapillary_tools/upload_api_v4.py”, line 60, in upload
resp = requests.post(
File “/home/allusers/.local/lib/python3.8/site-packages/requests/api.py”, line 116, in post
return request(‘post’, url, data=data, json=json, **kwargs)
File “/home/allusers/.local/lib/python3.8/site-packages/requests/api.py”, line 60, in request
return session.request(method=method, url=url, **kwargs)
File “/home/allusers/.local/lib/python3.8/site-packages/requests/sessions.py”, line 524, in request
resp = self.send(prep, **send_kwargs)
File “/home/allusers/.local/lib/python3.8/site-packages/requests/sessions.py”, line 637, in send
r = adapter.send(request, **kwargs)
File “/home/allusers/.local/lib/python3.8/site-packages/requests/adapters.py”, line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host=‘rupload.facebook.com’, port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_50c65e9e6208960117ea95e8107774c5 (Caused by NewConnectionError(‘<urllib3.connection.VerifiedHTTPSConnection object at 0x7f939d058250>: Failed to establish a new connection: [Errno 101] Network is unreachable’))

2 Likes

Disregard - still a problem but moved to a better location as a Github issue.

https://github.com/mapillary/mapillary_tools/issues/419

I experienced a similar problem. It went away when I updated using
python3 -m pip install --upgrade git+https://github.com/mapillary/mapillary_tools

1 Like

Thanks - the latest code on Github works and allows it to complete the uploads.

It’s a curious network problem; I wonder if a load balancer is dropping streams to regulate traffic? Even though it succeeds it can take 20 minutes to upload a minute’s worth of dashcam video because of timeouts and retries.

[Edit]

The mapillary_tools binaries are no longer available at tools (dot) mapillary (dot) com.

Binaries for Windows and Mac OS are available here

FYI - with my latest upload, there are many upload errors. It recovers and continues but overall progress is slow. Does something in the Mapillary infrastructure need to be reset?

My upload site doing BlackVue mp4’s is also having problems. Waiting for a 4000 off videos batch to complete before checking for missings via the web GUI.

Uses 0.8.1 Linux via docker.

Example debug

2022-04-25 07:05:52,658 - DEBUG - Sending upload_end via IPC: {‘total_sequence_count’: 1570, ‘sequence_idx’: 10, ‘entity_size’: 52298394, ‘import_path’: ‘to_upload/20210514BV/20210514_125339_NF.mp4’, ‘md5sum’: ‘7ccb511956c124e898f5f56d3870a596’, ‘upload_start_time’: 1650870321.714409, ‘upload_total_time’: 30.53449058532715, ‘offset’: 52298394, ‘retries’: 0, ‘upload_last_restart_time’: 1650870322.1237226, ‘upload_first_offset’: 0, ‘chunk_size’: 0, ‘upload_end_time’: 1650870352.6582131}
2022-04-25 07:06:02,685 - DEBUG - API Logging for action upload_failed_upload: {‘images’: 0, ‘sequences’: 9, ‘size’: 447.5315, ‘uploaded_size’: 447.5315, ‘speed’: 1.7194, ‘time’: 260.2831}
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/urllib3/connection.py”, line 174, in _new_conn
conn = connection.create_connection(
File “/usr/local/lib/python3.8/dist-packages/urllib3/util/connection.py”, line 72, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File “/usr/lib/python3.8/socket.py”, line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known