Need assistance parsing Garmin VIRB 360 CSV/writing data to EXIF

Long story short, the camera I have writes CSV files like this:

microTime,latitude,longitude,altitude,heading,pitch,roll,horizFOV,vertFOV,relAzimuth,relElev,relRoll
1535217616000000,43.31621,-77.7862,87.92499,96.58,0,0,115.4,70.3,0,0,0
1535217617000000,43.31621,-77.7862,87.87654,96.58,0,0,115.4,70.3,0,0,0
1535217618000000,43.31621,-77.7862,87.92522,96.58,0,0,115.4,70.3,0,0,0

The process_csv argument in the command line Python utility can’t parse this. I have some experience with Python, but I’m a mediocre programmer at best. I probably won’t be able to work on this until the weekend, and even then, the only thing I can think of is coming up with some sort of transform functions that will replace the first column with the expected time format and inserting a column between heading and pitch that will add and populate a “gps week” column via the gnsscal Python library. I need the heading value, which is determined by the camera sensors at the corresponding second in time.

If anyone has a better way of doing this or can write out some of the code out in advance, I’d greatly appreciate it (and I’ll be able to upload HD 360° segments to express my gratitude).

How big of chunks of data are we talking? If not huge I’d suck the whole thing into some sort of a a data structure that would make sens for it and work with the data from there.

I would think there would be an existing file library in python that would make it easy to suck in that CSV into datastructure that would make it easy to work with. Ideally some sort of query object that’d let you run a simple SQL select on it.

Hey, No idea what you’ve tried or the exact issue you’re encountering, but lines 133:138 appear to define what it is expecting each column to be.
https://github.com/mapillary/mapillary_tools/blob/master/mapillary_tools/process_csv.py
So adjusting the index numbers of those to the structure of your csv might be all you need to do.may well get you on your way.

Although I personally haven’t played around with the mapillary tools python scripts. so I could be missing a blatant switch somewhere.

Yeah, I’ve got some experience with pandas and already came up with the same idea. It’s an okay workaround, but I wonder if there’s a better way to do this instead of manually tweaking the CSV file.

1 Like

At the risk of sounding like a jerk, you commented nothing useful. By your own admittance, you haven’t played with this script before and you have no idea what my issue is. All you did is quote the first ~30 lines from the file that makes up the process_csv command. This is the internet equivalent of someone jumping into a conversation where they have nothing of substance to add. I’m sure you’re trying to help, but please be more introspective of your actions in the future. If you want to help people on the internet, take the time to learn the subject matter first.

For your own education, the problem is with the microTime column in the CSV file. The default time_format parameter value in format_time() expects the time data in a different format than how the CSV file reports it. Ultimately, I hope Mapillary will include an upstream fix in the form of a switch that will automatically convert the microtime values into the format format_time() is looking for and determine the GPS week automatically. In the the meantime, I’m banging out some helper functions outside of mapillary_tools that will update CSV file so it has the headers and data in the format I need it in.

1 Like

Fair enough. This forum adds the content of a hyperlink as a quote, which is why i noted the line numbers. Sorry I couldn’t help.

Hitting a brick wall and I should’ve been asleep hours ago, so here’s where I’m at. I’m using pandas to bring the CSV in as a dataframe, then I created a helper function to fix the timestamp:

def convert_microtime(microtime):
    return datetime.datetime.fromtimestamp(microtime / 1000000).strftime('%Y:%m:%d %H:%M:%S.%f')

example:

datetime.datetime.fromtimestamp(1535043722000000 / 1000000).strftime('%Y:%m:%d %H:%M:%S.%f')

'2018:08:23 13:02:02.000000'

I use dataframe.to_csv('/path/to/export_loc, index=False) and run mapillary_tools again pointing to the fixed CSV, but get the following error:

mapillary_tools process_csv --import_path "Videos/mapillary_images" --csv_path "Videos/mapillary_videos/20180823 mapillary 360 test fixed.mp4.csv" --timestamp_column 1 --latitude_column 2 --longitude_column 3 --altitude_column 4 --heading_column 5 --advanced
Inserting csv data in image EXIF: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "mapillary_tools", line 76, in <module>
  File "mapillary_tools\commands\process_csv.py", line 49, in run
  File "mapillary_tools\process_csv.py", line 269, in process_csv
  File "mapillary_tools\process_csv.py", line 146, in parse_csv_geotag_data
  File "mapillary_tools\process_csv.py", line 27, in format_time
  File "_strptime.py", line 332, in _strptime
ValueError: time data 'microTime' does not match format '%Y:%m:%d %H:%M:%S.%f'
[256] Failed to execute script mapillary_tools

At first glance, this doesn’t make any sense because it’s the exact strftime string I’m converting the microtime to. Documenting all this here so if I never end up figuring this out, someone might find this through google searches and be able to pick up where I left off.

EDIT: I’m an idiot. It’s parsing the header row and doesn’t know what to do with it.