GoPro Max (1/2) GPMF and video frame timing drift during distance-based sampling

Contributing and equipment Command line tools
1

I am investigating a reproducible GPS placement issue when using mapillary_tools sample_video with GoPro TimeWarp footage recorded at approximately 29.97 fps.

Images sampled near the end of a recording can receive a GPS position that is progressively ahead of the actual image location. The embedded GPS route itself is valid and passes normal speed and continuity checks. The apparent issue is the association between the selected video frame and the corresponding point on the internal GoPro GPMF timeline.

The same GoPro workflow with tested 25 fps TimeWarp recordings does not show this progressive displacement.

Mapillary Tools supports GoPro GPMF video and uses distance-based sampling to extract video frames and geotag them from the video telemetry. FFmpeg is used for video processing and frame extraction within that pipeline.

Mapillary Tools: 0.14.7
FFmpeg: 8.1.2
Camera: GoPro MAX 2
Recording mode: TimeWarp 5X
Processed video rate: 29.970030 fps
Video frames: 21,233
GPMF packets: 21,234
Valid GPS9 packets: 21,231
Video duration: approximately 708.47 seconds
Sampling command: mapillary_tools sample_video
Sampling distance: 5 metres
Sampled images: 9,816

Observed issue

A sampled JPEG near the end of the sequence visibly shows the camera immediately before a bridge.

The GPS marker written to that JPEG is located after the bridge, approximately 68 metres ahead of the visible camera position.

The GPS track itself is smooth:

GPS format: GPS9
Fix: 3
DOP near the test location: approximately 1.4
No implausible jumps
No duplicate timestamps
No backwards timestamps
Maximum validated speed: approximately 93 km/h

Because the route remains physically plausible, ordinary GPS quality checks do not detect the image-to-position mismatch.

Independent frame comparison

I visually matched the sampled JPEG against frames from the exact processed video.

JPEG name:GS010726_TEST_5xTW_0_009574.jpg
Best matching video frame:20,700
Corresponding equal-index GPMF packet: 20,700

The existing timestamp-based association pointed to approximately:

GPMF packet: 20,726

The difference was 26 packets.

Using GPS9 from packet 20,700 placed the image correctly before the bridge. Using the position associated with approximately packet 20,726 placed the image after the bridge.

The geographic difference between those two GPS positions was approximately 67.7 metres

Timing observation

The video and GPMF tracks contain almost the same number of samples:
Video frames: 21,233
GPMF packets: 21,234

The GPMF packet timestamps progress at a slightly different cadence. The difference is small per frame but grows over the duration of the recording.

This appears to create a situation where video frame n and GPMF packet selected by nearest timestamp are no longer the same near the end of the file. The displacement is progressive rather than a fixed start offset.

Comparison with 25 fps TimeWarp

I also tested GoPro TimeWarp recordings at 25 fps from both MAX generations.

For the tested 25 fps material video frame n ≈ GPMF packet n.

No comparable progressive GPS displacement was observed.

The issue currently appears associated with the approximately 29.97 fps TimeWarp timing family, although I do not yet have enough camera models and firmware versions to claim that every 29.97 fps GoPro file is affected.

Why the current workarounds are insufficient

Reassigning GPS from JPEG timestamps

The JPEG timestamps already appear to reflect the problematic video-to-GPMF time association. Using those timestamps to look up GPS again reproduces nearly the same incorrect position.

Globally rescaling the GPS timeline

A global affine rescale can align the start and end durations, but it does not prove that each extracted JPEG is associated with the correct original video frame.

Applying a fixed frame correction

A fixed correction such as minus 26 frames would only be correct near the tested bridge. The error grows progressively, so a fixed offset is not reliable.

Reimplementing distance sampling externally

A simple external sampler that accumulated distance between every consecutive GPMF position selected 13,644 images instead of Mapillary Tools’ 9,816 images. GPS jitter and differences in the internal sampling algorithm make that approach unsuitable as a direct replacement.

Workaround for now, change the regional setting to 50 Hz (PAL), the TW recording will be in 25 fps and can be used without any issues.

2

Thank you for noticing this.

The resulting position is just a side effect, hence the geographic delta does not matter for the bug. The root cause is obviously in how frame timestamps are computed. mapillary_tools apparently computes 30 fps timestamps instead of 29.97002997 fps timestamps for each frame (thanks to the typical NTSC nonsense). Just look at the quotient 20,726 / 20,700 = 1.001256039. :thinking: Hmm, looks familiar, eh? Awfully close to 1.001, doesn’t it? There is the delta you are missing. :wink:
CC @tao

By the way, Mapillary generally recommends capturing video in 25 fps anyway because there is nothing to gain for mapping from more fps even at driving speeds but rather the opposite. More fps eats into a camera’s video bit rate budget at the cost of image quality. Nevertheless, the sample_video command should always produce correct timestamps.

3

Thanks, this is a very useful observation and it appears to explain most of the progressive offset.

The exact NTSC ratio between 30 fps and 30000/1001 fps is 1.001. At frame 20,700, calculating the timestamp at 30 fps instead of 29.97002997 fps creates an accumulated difference of approximately 0.690 seconds, equivalent to about 20.7 frames.

The observed difference was 26 packets or frames, so the NTSC factor explains most, but not all, of the offset. The remaining approximately 5.3 frames could come from a fixed track start offset, PTS rounding, MP4 edit lists, or nearest-packet selection.

The geographic displacement is therefore only a consequence. The main question is whether mapillary_tools calculates frame timestamps using frame index divided by 30 instead of preserving the exact 30000/1001 rational frame rate.

The recommendation is indeed 25 fps, but out of the box the Max 2 has the NTSC regional format, and most of the users are not changing this setting (especially in NA, NTSC is the standard).

4

Yes, this is a known bug or peculiarity that the GPMF and video streams do not start at the same time and that the GPMF stream’s timing does not match the video stream’s timecode. Timing and GPS positions in the GPMF stream are rather more of an approximation with respect to the video than strictly linked one-to-one. This does not mean that given a perfect GPS signal the GPS clock signal or positions recorded in the GPMF stream are off from the truth. It just means that both streams are technically off set from each other because they are recorded independently, asynchronously, and concurrently; GPMF recording starts a bit after the video starts and ends a bit after the video ends. For the video stream the correct timing — its time keeper — is the timecode and this is what everything else should be matched to. The creation_time tags and timecodes are always correct in GoPro produced MP4 files.

Right. However, there has to be due diligence on both ends. Mapillary did its part by providing guidelines. Contributors have to do their part too for optimal imagery. Of course, this matter applies to video only. Many contributors capture in image mode which is not impacted by the selected video standard. So yes, things could be better, contributors more attentive, the world could be simpler, and mapillary_tools should always produce correct timestamps. :grin:
CC @tao

5

Thanks, that makes sense. Combining the two effects appears to explain the full measured offset.

At frame 20,700, the 30 versus 30000/1001 mismatch accounts for approximately 20.68 frames. The remaining 5.32 frames correspond to approximately 178 milliseconds, which is a plausible fixed difference between the video-track start and the GPMF-track start.

This suggests the observed 26-frame displacement can be modelled as a fixed GPMF/video track-start offset plus the progressive 1.001 NTSC cadence error.

If the video timecode is the authoritative clock, would the correct approach be to use the actual video-frame PTS and transform the GPMF sample timestamps into the video-track timeline, taking both track start times and any edit lists into account?

Do you know whether mapillary_tools currently uses the actual decoded video-frame PTS for distance-sampled frames, or whether it derives frame timestamps from the frame index and a rounded frame rate?

I agree that using 25 fps is a useful capture recommendation, but mapillary_tools should still preserve correct timing for supported 30000/1001 video. The PAL recommendation can mitigate future captures, while proper video-timecode alignment would be the actual software fix.

1 Like
6

Just to clarify any misconceptions; this gap is neither static nor deterministic and video file consumers have no safe way to compute the size of this gap. GoPro should not be blamed too much here because a) it is very difficult to synchronize the GPS receiver with the image processor (requires hardware support) and b) since its inception the GPMF stream was designed as soft metadata (not synchronized metadata) or camera debug log. In other words, the same goes for other metadata items, like changes to camera settings or photo-metric parameters (sensitivity, exposure time, aperture).