Thread crash workaround


Thought I would share how I solved this drama. I capture 2-3 hours per day (BlackVue vids) that yield 10,000-25,000 processed images per day. I then upload in batches every 2-3 weeks. Thread crashers were occurring roughly every 500 images and since I run this job over an SSH shell link it was getting frustrating. It takes about 3 hours to upload one days capture at the site I am using.

I wrote a shell script that checks a logfile for completion every 800 seconds, then kills and restarts the process. I can now queue up the jobs without worrying about failures;

#sleep 4h - optional delayed start
#This runs from the python virtual environment directory code4

while [ “grep No\ post\ processing\ action\ specified 20190220BV.log” = “” ] ; do
ps -ax | grep mapillary_tools | grep 20190220BV | awk ‘{print $1}’ | xargs kill -9 $1
/home/user/Thing/Mapillary/code4/.env/bin/python2 /home/user/Thing/Mapillary/code4/.env/bin/mapillary_tools upload --advanced --number_threads 4 --import_path /home/user/Thing/Mapillary/20190220BV/images/mapillary_sampled_video_frames >>20190220BV.log 2>&1 &
sleep 800
ps -ax | grep mapillary_tools | grep 20190220BV | awk ‘{print $1}’ | xargs kill -9 $1

After starting the script remotely I ctrl-z, bg and disown. I can then disconnect from the remote PC and it will continue working. For each day I generate a new script with sed changing the date, but there are of course a multitude of ways to do this.