Mapillary_tools: create a smaller number of files

mapillary_tools creates a huge amount of additional files during the upload process.
For example, a directory with 999 images has 23976 files after mapillary_tools is done with it.

That’s 23 extra files for each image.

Would be wonderful if this could be brought down - maybe there are performance improvements to be had (deleting those directories should be faster, at least).
Additionally, if images are uploaded from the SD card and accidentally not removed from it, such a huge number of files kills GoPro camera - it hangs if even one such directory with a small number of original images is left on the card.

4 Likes

I actually find the individual files very useful in the debugging process when things go wrong (and they do). It is also an excellent method to recover from a crash or abort. It can continue on from where the fault happened based on the existence of a file rather than the contents of it.

I would elect to keep them. The only issue I have ever had was when I used a smallish drive and ran out of inodes.

Oh, I don’t mean dropping any data that is useful - perhaps some nice way to put all that in one or two files instead.

2 Likes

Feel free to add a comment to this issue:
https://github.com/mapillary/mapillary_tools/issues/269

1 Like

Just a side thought for those with Linux or MacOS. I have found it worthwhile changing the disk drive access/buffering for throughout rather than low latency for tools processing. Most PC generally are configured for low latency, so that the user doesnt get frustrated. These are roughly;

  • Change scheduler from cfq to noop. eg echo noop > /sys/block/sda/queue/scheduler
  • Increase outstanding requests. eg echo 1024 >/sys/block/sda/queue/nr_requests
  • Increase readahead. eg blockdev --setra 16384 /dev/sda2
    Some of this appears in Disk I/O — Performance Tuning on Linux

Is the Mapillary team using Github issue trackers? They stopped using the mapillary_issues one at least.

I get the strong impression that Brenna (support@mapillary.com) does the clearing/routing of all external comms/issues. The devs have however answered me on a few occasions if I break into a pull, commit or other related. I have also seen issues discussed (I assume among devs) when pull changes are made on development branches.

Oh, the iOS app is playing the same whoever-creates-most-files-wins game…
http://mapillary.trydiscourse.com/t/files-files-everywhere