"OSError: [Errno 28] No space left on device" when uploading

Contributing and equipment Command line tools
21

I’m also facing the issue with no space left on device (Desktop uploader on windows). The sequence to upload (7 GB) is on a disk with plenty of free space (80 GB). I imagine that the device with no space left is where is “C:\Users\xxx\AppData\Roaming\Mapillary Uploader” i.e. C drive (1.8 GB left).

image
image1010×593 18.6 KB

[2025-06-14 20:51:39.880] [debug] [mapillary:tools] 2025-06-14 20:51:39,878 - DEBUG   - HTTP GET https://graph.mapillary.com//954303792879264 HEADERS={'Authorization': '[REDACTED]'} TIMEOUT=60

[2025-06-14 20:51:40.126] [debug] [mapillary:tools] 2025-06-14 20:51:40,125 - DEBUG   - HTTP 200 (OK): {"slug": "eucameragrant9", "name": "EU Camera Grant 9", "id": "954303792879264"}

[2025-06-14 20:51:40.127] [debug] [mapillary:tools] 2025-06-14 20:51:40,125 - INFO    - Uploading to organization: {"slug": "eucameragrant9", "name": "EU Camera Grant 9", "id": "954303792879264"}

[2025-06-14 20:52:11.886] [debug] [mapillary:tools] Traceback (most recent call last):
  File "mapillary_tools\upload.py", line 568, in upload

[2025-06-14 20:52:11.888] [debug] [mapillary:tools] mapillary_tools.upload.UploadError: [Errno 28] No space left on device


[2025-06-14 20:52:11.888] [debug] [mapillary:tools] During handling of the above exception, another exception occurred:
22

Yes, the free space on the disk where the temporary files are stored will be the limiting factor when uploading images. For a successful upload, it needs an amount of free space that matches the size of the current upload. We’ll be updating our upload process in the next release to remove this hurdle.

23

In the mean time, I made some cleaning on the correspond disk and was able to upload.

24

Thanks, is that similar to the recent improvement in the iOS app (reverting back from the Facebook mandated upload approach), where it stops zipping it all up?

When could that be expected in mapillary_tools?

25

Yes, we’ll be going with uploading the selected files straight away on all platforms without an intermediary step or format. It will be available in mapillary_tools in the next release, I’d let @tao answer on when that might be, but you can already follow the progress in the GH repo.

2 Likes
26

@Richlv @GITNE @Eric_S Echo what @nikola said, we are getting rid of the tmpdir zipping process in feat: support uploading images file by file by ptpt · Pull Request #753 · mapillary/mapillary_tools · GitHub and it will be released in the next version of desktop uploader and mapillary_tools. cc @Yaro @boris

Also FYI @GITNE the offset reset issue in upload resuming should be largely solved for imagery uploading after the release too.

3 Likes
27

Hey @Richlv @GITNE @Eric_S , It’d be appreciated if you can try out Release v0.14.0b1 · mapillary/mapillary_tools · GitHub and share any feedback here

2 Likes
28

Oooh, neat. Will plain upgrade bump to that version or not?

30

Turns out, I used the downloaded version currently, so kept that approach :slight_smile:
Didn’t check the temporary file generation, but the upload itself was 3-4 times faster. Is it just a random coincidence, or is this related to parallelisation that was lost with the FB upload framework (and regained now)?

31

Glad to hear you had a good experience! Yes, the upload itself should be faster as well I believe, nice work @tao !

1 Like
32

Hello @tao
ted@ted-swiftsf314511:~$ mapillary_tools/bin/mapillary_tools --version
mapillary_tools version 0.14.0b1
I updated the tools, and the upload seems to be faster. But really often upload gets interrrupted and then the current sequence is left not uploaded. When restating the tools, the upload of the left-over sequences starts again at 0%. It’s really annoying at the moment.
Here is an example, maybe the fourth time I tried to upload folger 001_0081 with 11 sequences. As you can see, often sequences are only uploaded the some percentage and then with no error message they are left behind. When restarting, upload starts again at 0:

ted@ted-swiftsf314511:~$ mapillary_tools/bin/mapillary_tools upload “Aayi/001_0081/”
2025-07-18 07:13:32,200 - INFO - Uploading to profile “teddy73”: {‘user_upload_token’: ‘[REDACTED]’, ‘MAPSettingsUserKey’: ‘103080845264750’, ‘MAPSettingsUsername’: ‘teddy73’}
Uploading IMAGE (1/11): 28%|██████████████████████████████ | 1.12G/3.97G [35:30<2:55:11, 290kB/s]
Uploading IMAGE (1/11): 29%|██████████████████████████████▌ | 1.15G/3.97G [39:34<4:59:23, 169kB/s]Uploading IMAGE (1/11): 41%|███████████████████████████████████████████▍ | 1.61G/3.97G [49:02<26:44, 1.58MB/s]2025-07-18 08:03:04,917 - INFO - Sequence 1 has been uploaded already. Check the upload history at /home/ted/.local/share/mapillary_tools/upload_history/c7/db9ef164b7964d41c2343f15b2368e.json
Uploading IMAGE (1/11): 41%|██████████████████████████████████████████▉ | 1.61G/3.97G [49:16<1:12:13, 584kB/s]
Uploading IMAGE (3/11): 73%|████████████████████████████████████████████████████████████████████████████▋ | 2.89G/3.96G [1:08:12<05:02, 3.80MB/s]2025-07-18 09:11:31,389 - INFO - Sequence 3 has been uploaded already. Check the upload history at /home/ted/.local/share/mapillary_tools/upload_history/a7/bded76c3e6d1091768fd361faf7e6c.json
Uploading IMAGE (3/11): 73%|█████████████████████████████████████████████████████████████████████████████▍ | 2.89G/3.96G [1:08:26<25:17, 757kB/s]
Uploading IMAGE (5/11): 16%|█████████████████▏ | 633M/3.87G [08:58<01:18, 44.3MB/s]2025-07-18 09:20:43,060 - INFO - Sequence 5 has been uploaded already. Check the upload history at /home/ted/.local/share/mapillary_tools/upload_history/f1/449eb0236e5d8621329b4d07d62f16.json
2025-07-18 09:20:52,046 - INFO - Sequence 6 has been uploaded already. Check the upload history at /home/ted/.local/share/mapillary_tools/upload_history/4a/85663e0adb93c4a996548bda2a0c12.json
Uploading IMAGE (5/11): 16%|█████████████████▏ | 633M/3.87G [09:14<48:38, 1.20MB/s]
Uploading IMAGE (8/11): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.54G/1.54G [26:14<00:00, 1.05MB/s]
Uploading IMAGE (9/11): 53%|████████████████████████████████████████████████████████▉ | 1.96G/3.69G [57:58<27:51, 1.11MB/s]2025-07-18 10:45:24,202 - INFO - Sequence 9 has been uploaded already. Check the upload history at /home/ted/.local/share/mapillary_tools/upload_history/fc/2e580779f577040135e66defa794c6.json
2025-07-18 10:45:26,128 - INFO - Sequence 10 has been uploaded already. Check the upload history at /home/ted/.local/share/mapillary_tools/upload_history/61/d600b744060b207848c808efac47bd.json
2025-07-18 10:45:26,549 - INFO - 1 image sequences uploaded
2025-07-18 10:45:26,549 - INFO - 1572.1M data in total
2025-07-18 10:45:26,549 - INFO - 1572.1M data uploaded
2025-07-18 10:45:26,549 - INFO - 1574.9s upload time
2025-07-18 10:45:26,549 - ERROR - Upload error: HTTPError: 400 Client Error: Bad Request for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_cf4ebe29e708e54d9fc6f96f0d1842a7.jpg
2025-07-18 10:45:26,549 - ERROR - Upload error: UploadedAlreadyError:
2025-07-18 10:45:26,549 - ERROR - Upload error: HTTPError: 400 Client Error: Bad Request for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_0533f87c354232dbe6dd6f50368e015c.jpg
2025-07-18 10:45:26,549 - ERROR - Upload error: UploadedAlreadyError:
2025-07-18 10:45:26,549 - ERROR - Upload error: HTTPError: 400 Client Error: Bad Request for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_14a84a3fa27d89016cbafd9708b82bd7.jpg
2025-07-18 10:45:26,549 - ERROR - Upload error: UploadedAlreadyError:
2025-07-18 10:45:26,549 - ERROR - Upload error: UploadedAlreadyError:
2025-07-18 10:45:26,549 - ERROR - Upload error: HTTPError: 400 Client Error: Bad Request for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_45c3b73fd10f3610338ad067dc32169e.jpg
2025-07-18 10:45:26,549 - ERROR - Upload error: UploadedAlreadyError:
2025-07-18 10:45:26,549 - ERROR - Upload error: UploadedAlreadyError:
Uploading IMAGE (9/11): 53%|█████████████████████████████████████████████████████████▍ | 1.96G/3.69G [58:04<51:06, 604kB/s]
ted@ted-swiftsf314511:~$
ted@ted-swiftsf314511:~$ mapillary_tools/bin/mapillary_tools upload “Aayi/001_0081/”
2025-07-18 12:13:07,268 - INFO - Uploading to profile “teddy73”: {‘user_upload_token’: ‘[REDACTED]’, ‘MAPSettingsUserKey’: ‘103080845264750’, ‘MAPSettingsUsername’: ‘teddy73’}
Uploading IMAGE (1/11): 3%|███▋ | 138M/3.97G [10:09<21:24, 3.20MB/s

33

@Teddy73 @GITNE When it’s stuck at 0% MT is actually calculating checksums for all the files. If you are uploading large files on a slow disk (especially SD card), then it’s likely appears that it’s stuck at 0% for a while.

Unfortunately we can’t skip these checksum calculations because MT needs them for upload deduplication check and resumption. I’ll try if I can make it faster, also make the progress more visible.

Another issue shown in @Teddy73’s log is HTTPError: 400 Client Error. I can’t tell what was going on. I will improve the logging so it shows more info in the verbose mode.

Thanks for sharing the feedback!

34

Good stuff @tao, maybe rather than showing “0%” we can display “Calculating checksums” to let users know what is happening.

1 Like
35

Experiments rather tell me that Python’s MD5 implementation is extremely slow (I do not know whether due to storage I/O, the computation itself, or mapillary_tools’ use of it) because when I do the same thing that mapillary_tools does on the same files with OpenSSL’s MD5 implementation then things are over 12× faster. You could launch OpenSSL in an external process and you would still get an order of magnitude of speedup! Hence, I do not think that this is due to slow SD cards or anything.

Besides, like I have mentioned multiple times before; I will say it again: DO NOT use MD5 for anything. It is broken beyond repair for everything.

To speed up processing even further, you do not have to hash the full image file (which is really a lot of data) but just the TIFF header and select lines of the JPEG stream/blob.

1 Like