Smoother movement without unexpected viewing direction changes is something we aim for. As you mention, that sometimes means skipping some images positioned near the current one and instead using a better match further away. We actually updated the moving functionality very recently to improve the behaviour when navigating in the viewer.
There are many properties of the movement that can be used when trying to create a good navigation experience. It could be the direction change or the difference in motion with respect to where we want to go.
When there are many candidates for moving in a certain direction the probability to find a good candidate is higher. This means that uploading more images in an area will help the viewer find a better match for good movement in a certain direction in the future.
Check out the new moving functionality and let us know what you think.