mapillary_tools 0.7.4.
Uploading with ~300MB free on the root filesystem failed with the errors below.
Upload happened from an external disk with hundreds of GB free.
Could it be that the images are transferred to the system during upload?
If so, could this be fixed so that only the source filesystem is filled up?
Uploading 992 images with valid mapillary tags (Skipping 7)
Traceback (most recent call last):
File “/usr/local/bin/mapillary_tools”, line 11, in
load_entry_point(‘mapillary-tools==0.7.4’, ‘console_scripts’, ‘mapillary_tools’)()
File “/Library/Python/3.7/site-packages/mapillary_tools/main.py”, line 93, in main
command.run(args)
File “/Library/Python/3.7/site-packages/mapillary_tools/commands/process_and_upload.py”, line 456, in run
for k, v in vars_args.items()
File “/Library/Python/3.7/site-packages/mapillary_tools/upload.py”, line 151, in upload
dry_run=dry_run,
File “/Library/Python/3.7/site-packages/mapillary_tools/uploader.py”, line 248, in upload_sequence_v4
ziph.write(fullpath, relpath)
File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/zipfile.py”, line 1744, in write
shutil.copyfileobj(src, dest, 1024*8)
File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/shutil.py”, line 82, in copyfileobj
fdst.write(buf)
File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/zipfile.py”, line 1098, in write
self._fileobj.write(data)
File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/tempfile.py”, line 481, in func_wrapper
return func(*args, **kwargs)
Free inodes? The logging process would use a lot, but the FS creation would have had to be pretty old… I had the problem when I was process and uploading from an older ext4(?) USB 500GB drive
Should be good, iused = 2638061, ifree = 4292329218.
At the same time when mapillary_tools failed, the OS also alerted about the root filesystem being full.
Hmm, uploader.py has a compress step that references “tempfile” so I guess that could be using /tmp, Going out on a limb here the shell variable TMPDIR may be able to be assigned on a per process basis. eg before the mapillary tools command;
TMPDIR = /media/richlv/photdisk/tmp …
This is really a shot in the dark! Python itself has a module tempfile that I think does all this.
After some troubleshooting, I think I figured it out. On Linux (or at least, on Fedora, it may change by distribution), by default the temp folder is given half the space of RAM. So if you have e.g. 16gb ram, you’ll probably have 8gb for the temp folder. This is not related to how much free space you have on your hard drive or on the external storage from where you’re doing the uploading.
For whatever reason, before finalising a sequence Mapillary Uploader needs to move all of it in the temp folder. So just before you get this error you should see the RAM use increase, and the space in the /tmp/ folder decrease quickly. You can see this by checking the free space repeatedly from inside /tmp just before you get the error.
If the sequence you’re trying to upload is bigger than the current size given to /tmp, then you’ll get this error.
The easiest workaround is probably to simply temporarily give more space to /tmp, e.g.:
mount -o remount,size=12G,noatime /tmp
to give 12gb to /tmp.
This should be automatically reverted to default value after restart. For reference, see e.g. this.
On Debian 10.x (at least) there is no separate /tmp mountpoint. It’s just a directory in the filesystem. There are however some “tmpfs” filesystems of limited size that may be involved.
Thanks, that confirms my suspicion.
mapillary_tools really, really should not be doing this. The filesystem where I’m uploading images should be the only one used during the process.
I’m wondering whether it’s the compressing step - which could then also explain the problem at Page not found .
I found similar when trying to use a Raspberry Pi to sit as a dedicated 24/7 Mapillary uploader (I take a lot of photos but have a slow upload link).
When the tools were updated and the compress-before-upload step added, I found things ground to a crawl for me. The photos are situated on a NAS (bit slow), and it was writing the temporary zip file to the local SD card (very slow). Fixed it by using an 8GB Pi and putting /tmp on tmpfs - each sequence is around 2-3GB, so it fits nicely in the default half-RAM limit.
I’d rather the compress stage were part of the prepare, rather than the upload step, but until that gets sorted I personally much prefer the temporary zip file being written to fast RAM-based /tmp, rather than going back and forth to the NAS it came from.
Thanks you, will try this out
What about the automatic process, are the archives preserved during runs (let’s say if an upload fails or is interrupted)?
Hi @tao, trying mapillary_tools 0.9.0 - TMPDIR is set to an external disk, but during the “Uploading import path” phase, root directory ends up seeing major usage increase.
Before I look into this further - is TMPDIR supposed to work properly in 0.9.0?
@GITNE - we’re actually working on a project now that will eliminate the need to create these intermediary .zip files to speed up the upload process and remove some of the friction here. We hope to have it out this half. cc: @Yaro and @tao
I vaguely recall that these zips were introduced when Mapillary was absorbed by FB, because they had to move to a standard approach in the FB infrastructure. It made uploading a notably worse experience both in the mobile app (where sequences became “locked”) and in the CLI tools (where the problems with these temporary archives appeared).
I’m also facing the issue with no space left on device (Desktop uploader on windows). The sequence to upload (7 GB) is on a disk with plenty of free space (80 GB). I imagine that the device with no space left is where is “C:\Users\xxx\AppData\Roaming\Mapillary Uploader” i.e. C drive (1.8 GB left).
Yes, the free space on the disk where the temporary files are stored will be the limiting factor when uploading images. For a successful upload, it needs an amount of free space that matches the size of the current upload. We’ll be updating our upload process in the next release to remove this hurdle.
Thanks, is that similar to the recent improvement in the iOS app (reverting back from the Facebook mandated upload approach), where it stops zipping it all up?
