One thing that used to be a problem with the GoPro taking a picture every 2 seconds and the phone recording a GPS trackpoint every 1 second was that the clocks would sync up just right to where I would end up with two pictures closer together followed by a longer gap, followed by another two pictures closer together and so on. Here is an example sequence of this behavior: https://www.mapillary.com/app/?pKey=rOATSTDr1gJFUVt0ENtzQA
This was solved by using the --interval 2 option with the geotag_from_gpx.py script. It uses the time of the first and last pictures to estimate exactly where within each second pictures were taken and smooth out this jitter. This does require a completely unbroken sequence of images (so no stopping the camera at stop lights for example).
Also, be sure your camera’s firmware is up to date. I’m pretty sure I saw something in a recent changelog on my Hero 4 about improvements to the timing in time-lapse mode. I didn’t really look at the details though.
But yes, I do still see drift at times. If it gets too bad over the course of an entire sequence, I will sometimes break a sequence into multiple parts to keep the drift in check.
After using the geotag script I open the pictures in JOSM and compare them to aerial imagery. I find the best places to check are where there are features that cross the road and are visible in imagery. In particular out in the country, bridges usually have very visible expansion joints at the beginning/end of the deck and/or have different pavement than the road before/after them. In urban areas, crosswalks and stop lines are of course useful. Overhead signs on freeways. I find that trying to reference by objects along the side of the road can be difficult, especially when working with wide angle lenses. It is hard to estimate the true field of view accurately. You just have to be sure the imagery is up to date enough. There is an intersection near my home that was rebuilt recently. It looks pretty similar to what is in the imagery but the crosswalk got moved back several feet. I remember being frustrated while trying to geotag a sequence until I realized this and ignored that intersection.
So after viewing the locations in JOSM I will re-run the geotagging script and adjust the time-offset as needed. Note that the time-offset parameter does not have to be a whole number. You pass in 1.65436 and adjust things by nanoseconds if you want 
(OCD warning!)
Of course the real answer here is "upgrade to the Hero 5 which finally has GPS hardware built in! Quick, someone give me $400! Actually better make that $500. Of course they changed the battery design again so I’ll need a new set of spares and a charger…