How to determine available 'directions' for "street view"-style interface?

I’m interested in making a “street view” type app with Mapillary’s panoramic images (in VRChat to be specific, hence not just using the existing web interface), and I’m curious what sort of approach one is meant to take for determining the “link” arrows to nearby nodes… Forward and backward in the same sequence is obvious, but for things like cross-streets, or where one sequence ends and a separate one begins, are we meant to just query for all images within a hardcoded “reasonable” radius, and manually find the closest ones in each general “direction”? (I had started looking into this project under the assumption that images were all tied to specific streets and that those streets were used for the link-arrow data, but given how the official Mapillary web app often offers to let people jump to an entire parallel street nowhere near a corner, I’m suspecting that the “show any nearby nodes” approach is the best available?)

Not an expert in this so just a few thoughts. Some of this may just be supposition on my part, born from many years of uploading.

AFAIK there is no direct correlation with streets using the API. It is all absolute position based, or by captured sequence and/or an individual image key. The start/base idea behind Mapillary data was object detection and a sequence of images made that software easier.

That sequences happen to follow roads would just seem to be good luck for your app. If you look at enough data on the Mapillary GUI though you will see the tracks wander quite a lot. Some of this will be GPS limitations, the actual (unintended) movements of the vehicle (like overtaking and parking) and a variety of other errors.

I would suspect that for what you want to do, the method would be to use a node on a OSM highway=* way as your start position and build up an absolute position matrix of that and other nearby crossing ways, then query the Mapillary API for the images by position at some prescribed distance interval along the route you have defined. Some caching and data age/quality and whether to prioritise use of a sequence also being used to make things run smoothly.

Suggest you get some more opinions though.