Using Generative AI to cover up nadir

I recently experimented with using Generative AI to cover the nadir with a realistic representation of the ground underneath the camera in my 360 images. The process of editing myself out is quite time consuming, so this is mostly worth it when capturing a small area. This was a fun experiment to see how good generative AI works with imaginative infill on images from a top-down perspective. Turns out at least Firefly from Adobe works pretty well. I also tested out Stable Diffusion XL through Clipdrop and running several Stable Diffusion models locally on my Silicon chip Mac Studio, which did not turn out so well.

Kirsebærlunden Playground - Mapillary

5 Likes

I am still thinking whether I will give this a like.

Impressive results!

1 Like

I use Resynthesizer under Gimp for a while to fill missing parts in sky or ground when doing panoramic stitching. That’s nice from an esthetic point of view. Indeed, I would be more carefully when using it for Mapillary in that sens that it is creating non-existing data. It may produce artifact for people wanting to asses road damage or counting steps in stairs. :sweat_smile:

1 Like

Valid point most definitely @Eric_S.

The semi transparent glass effect is becoming a more popular design element in the world of VR and AR. One modification to the approach used in this post is using Generative AI to remove the photographer and then add a circular “glass” logo to the nadir point that slightly blurs the generated area. This would create a nice “aesthetic” effect and obfuscate the generated area enough to not cause any confusion in the data.

Another approach I would like to explore is how to interpolate the ground underneath the photographer based on pictures taken just a few frames before and after, where the same ground is visible. If anyone knows of a technique or paper exploring this process I am all ears.

@Eric_S what tools and camera do you use for panoramic stitching and have you an automated process for batch stiching images?

For spherical pano, I was using a nodal ninja head with a dslr. Manual stitching with Hugin.

About the ground, if it is more or less flat, you should be able to perform a reprojection using neighbour pictures. With Hugin, you will need to use translation options. Could be complex with the need of using an intermediate reprojection.

1 Like

I presumed you did it like that.