"OSError: [Errno 28] No space left on device" when uploading

Contributing and equipment Command line tools
1

mapillary_tools 0.7.4.
Uploading with ~300MB free on the root filesystem failed with the errors below.
Upload happened from an external disk with hundreds of GB free.
Could it be that the images are transferred to the system during upload?
If so, could this be fixed so that only the source filesystem is filled up?

Uploading 992 images with valid mapillary tags (Skipping 7)
Traceback (most recent call last):
File “/usr/local/bin/mapillary_tools”, line 11, in
load_entry_point(‘mapillary-tools==0.7.4’, ‘console_scripts’, ‘mapillary_tools’)()
File “/Library/Python/3.7/site-packages/mapillary_tools/main.py”, line 93, in main
command.run(args)
File “/Library/Python/3.7/site-packages/mapillary_tools/commands/process_and_upload.py”, line 456, in run
for k, v in vars_args.items()
File “/Library/Python/3.7/site-packages/mapillary_tools/upload.py”, line 151, in upload
dry_run=dry_run,
File “/Library/Python/3.7/site-packages/mapillary_tools/uploader.py”, line 248, in upload_sequence_v4
ziph.write(fullpath, relpath)
File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/zipfile.py”, line 1744, in write
shutil.copyfileobj(src, dest, 1024*8)
File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/shutil.py”, line 82, in copyfileobj
fdst.write(buf)
File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/zipfile.py”, line 1098, in write
self._fileobj.write(data)
File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/tempfile.py”, line 481, in func_wrapper
return func(*args, **kwargs)

2 Likes
2

Free inodes? The logging process would use a lot, but the FS creation would have had to be pretty old… I had the problem when I was process and uploading from an older ext4(?) USB 500GB drive

df -i

3

Should be good, iused = 2638061, ifree = 4292329218.
At the same time when mapillary_tools failed, the OS also alerted about the root filesystem being full.

4

Maybe you try a loop to upload each directory seperate and delete the folder inside the script created?

5

Hmm, uploader.py has a compress step that references “tempfile” so I guess that could be using /tmp, Going out on a limb here the shell variable TMPDIR may be able to be assigned on a per process basis. eg before the mapillary tools command;

TMPDIR = /media/richlv/photdisk/tmp …

This is really a shot in the dark! Python itself has a module tempfile that I think does all this.

sorry not really helpful…

6

After some troubleshooting, I think I figured it out. On Linux (or at least, on Fedora, it may change by distribution), by default the temp folder is given half the space of RAM. So if you have e.g. 16gb ram, you’ll probably have 8gb for the temp folder. This is not related to how much free space you have on your hard drive or on the external storage from where you’re doing the uploading.

For whatever reason, before finalising a sequence Mapillary Uploader needs to move all of it in the temp folder. So just before you get this error you should see the RAM use increase, and the space in the /tmp/ folder decrease quickly. You can see this by checking the free space repeatedly from inside /tmp just before you get the error.

If the sequence you’re trying to upload is bigger than the current size given to /tmp, then you’ll get this error.

The easiest workaround is probably to simply temporarily give more space to /tmp, e.g.:

mount -o remount,size=12G,noatime /tmp

to give 12gb to /tmp.

This should be automatically reverted to default value after restart. For reference, see e.g. this.

2 Likes
7

Good find!

On Debian 10.x (at least) there is no separate /tmp mountpoint. It’s just a directory in the filesystem. There are however some “tmpfs” filesystems of limited size that may be involved.

1 Like
8

Thanks, that confirms my suspicion.
mapillary_tools really, really should not be doing this. The filesystem where I’m uploading images should be the only one used during the process.

I’m wondering whether it’s the compressing step - which could then also explain the problem at Page not found .

9

Will solve in Mapillary_tools does repeated compression · Issue #437 · mapillary/mapillary_tools · GitHub

2 Likes
10

I found similar when trying to use a Raspberry Pi to sit as a dedicated 24/7 Mapillary uploader (I take a lot of photos but have a slow upload link).

When the tools were updated and the compress-before-upload step added, I found things ground to a crawl for me. The photos are situated on a NAS (bit slow), and it was writing the temporary zip file to the local SD card (very slow). Fixed it by using an 8GB Pi and putting /tmp on tmpfs - each sequence is around 2-3GB, so it fits nicely in the default half-RAM limit.

I’d rather the compress stage were part of the prepare, rather than the upload step, but until that gets sorted I personally much prefer the temporary zip file being written to fast RAM-based /tmp, rather than going back and forth to the NAS it came from.

1 Like
11

Interesting usecase - looks like a customisable temporary directory path would be highly desired.
Perhaps worth adding that in the Github issue.

12

Hey @Richlv @southglos mapillary_tools v0.8.0 is pre-released Release v0.8.0 · mapillary/mapillary_tools · GitHub

It exposed the zip command. Also it should be easier to integrate with your workflow. Let me know your feedback.

Have a great weekend!

4 Likes
13

Thanks you, will try this out :slight_smile:
What about the automatic process, are the archives preserved during runs (let’s say if an upload fails or is interrupted)?

14

Hi @tao, trying mapillary_tools 0.9.0 - TMPDIR is set to an external disk, but during the “Uploading import path” phase, root directory ends up seeing major usage increase.
Before I look into this further - is TMPDIR supposed to work properly in 0.9.0?

15

:tired_face: I am really fed up with mapillary_tools! I just want to upload some files.

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 600, in upload
    raise UploadError(ex) from ex
mapillary_tools.upload.UploadError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/__main__.py", line 162, in main
    args.func(argvars)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/upload.py", line 50, in run
    upload(**args)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 683, in upload
    raise inner_ex
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 595, in upload
    clusters = mly_uploader.upload_images(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 161, in upload_images
    with tempfile.NamedTemporaryFile() as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/tempfile.py", line 517, in __exit__
    result = self.file.__exit__(exc, value, tb)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device

Usually, I upload zips. This time, my sequence is a bit larger. Hence, I thought to use the process command first and then upload immediately after that, assuming there would be no need to create a zip file. Unfortunately, apparently mapillary_tools creates temporary zip files anyway. :disappointed:

$ ls -alF $TMPDIR
-rw-------  1 GITNE users 102942140114 Tue Feb 25 17:39:01 2025 tmpp_727pln

Why does mapillary_tools have to create a zip file if it has already all the required metadata processed into mapillary_image_description.json??? :crazy_face: I am pulling my hair! I just want to upload some files. :sob:

1 Like
16

@tao Is there a chance to make mapillary_tools stream or pipe the zip file to the upload module? Creating a (large) temporary zip file locally does not scale very well and is really inefficient for what is supposed to happen with it later. It also requires double the storage space for any particular sequence. That’s inefficient indeed.

If you cannot stream a zip file you may want to consider tar. Or, support multi volume zips.

17

@tao I have shuffled around my files and created a zip file. However, mapillary_tools exposed more blunders.

$ mapillary_tools --version
mapillary_tools version 0.13.2
$ mapillary_tools upload --organization_key 'XXXXXXXXXXXX' "$PWD"
2025-02-27 22:58:42,321 - INFO    - Uploading to organization: {"slug": "org slug", "name": "org name", "id": "XXXXXXXXXXXX"}
Uploading ZIP mly_tools_cef561684c9ea6cee92104ed27ff4b65.zip (1/16):   0%|                                             | 0.00/169G [00:00<?, ?B/s]
2025-02-27 22:59:11,323 - WARNING - Error uploading chunk_size 16777216 at begin_offset 0: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_cef561684c9ea6cee92104ed27ff4b65.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2427)')))
2025-02-27 22:59:11,323 - INFO    - Retrying in 2 seconds (1/200)
Uploading ZIP mly_tools_cef561684c9ea6cee92104ed27ff4b65.zip (1/16):   0%|                                             | 0.00/169G [00:03<?, ?B/s]
Uploading ZIP mly_tools_cef561684c9ea6cee92104ed27ff4b65.zip (1/16):   0%|                                             | 0.00/169G [00:00<?, ?B/s]
2025-02-27 22:59:15,074 - WARNING - Error uploading chunk_size 16777216 at begin_offset 0: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_cef561684c9ea6cee92104ed27ff4b65.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2427)')))
2025-02-27 22:59:15,074 - INFO    - Retrying in 4 seconds (2/200)
.
.
.

Rerouting produces:

2025-02-27 23:19:04,031 - INFO    - Uploading to organization: {"slug": "org slug", "name": "org name", "id": "XXXXXXXXXXXX"}
Uploading ZIP mly_tools_cef561684c9ea6cee92104ed27ff4b65.zip (1/16):   0%|                                             | 0.00/169G [00:00<?, ?B/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 705, in _upload_zipfiles
    cluster_id = mly_uploader.upload_zipfile(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 132, in upload_zipfile
    return self.upload_stream(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 221, in upload_stream
    return _upload_stream(
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 427, in _upload_stream
    raise ex
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/uploader.py", line 400, in _upload_stream
    file_handle = upload_service.upload(fp, offset=begin_offset)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload_api_v4.py", line 145, in upload
    resp.raise_for_status()
  File "/usr/local/lib/python3.12/dist-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_cef561684c9ea6cee92104ed27ff4b65.zip

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/__main__.py", line 162, in main
    args.func(argvars)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/commands/upload.py", line 50, in run
    upload(**args)
  File "/usr/local/lib/python3.12/dist-packages/mapillary_tools/upload.py", line 681, in upload
    raise wrap_http_exception(inner_ex) from inner_ex
mapillary_tools.upload.UploadHTTPError
Uploading ZIP mly_tools_cef561684c9ea6cee92104ed27ff4b65.zip (1/16):   0%|                                             | 0.00/169G [00:02<?, ?B/s]

This is pathetic. For goodness sake, how am I supposed to upload anything? Now, I cannot even upload a simple zip file. Again, we are trying to simply upload files here. This is not :rocket: science, you know.

18

@GITNE - we’re actually working on a project now that will eliminate the need to create these intermediary .zip files to speed up the upload process and remove some of the friction here. We hope to have it out this half. cc: @Yaro and @tao

1 Like
19

Thank you for the info @boris. I am looking forward to it. However, I have a lot of stuff to upload now and the current situation de facto blocks my camera from capturing because I want to keep the original imagery at least until after I know the sequences have been properly processed on Mapillary. I also do not want to juggle multiple SD cards, external SSDs, or HDDs. I want to move on because I probably will not be able to upload stuff in the coming months. When I return, I will then surely have forgotten where I left off uploading. :slightly_frowning_face:

20

I vaguely recall that these zips were introduced when Mapillary was absorbed by FB, because they had to move to a standard approach in the FB infrastructure. It made uploading a notably worse experience both in the mobile app (where sequences became “locked”) and in the CLI tools (where the problems with these temporary archives appeared).

1 Like