Stitching (raw 360) images with Hugin

[eesger]

@StephaneP gave me tip for the stitching software Hugin (thanks!)

I looked at it, and it looks really promising … I read up on it, and it looks like it supports commandline processing and it runs on several operating systems. So this piece of software might just be ideal for batch processing. Just what I need

but I cant make heads or tales with it getting to stich a raw image… I found a youtube tutorial, but that seems to be a previous version… I tried to follow the instruction, but it ends in a mess.

Anyone here that can help me out?

[dave683]

oh man, usually i am ok a learning new programs but the first time i tried hugin i spent the best part of an hour trying to figure it out. i couldnt get it to do anything in the end.

i tried again a few months later and managed to stitch an image but i still dont really understand how it works. i spent another hour trying to get the batch part to work but i still cant crack it. maybe third time lucky

what camera are you using anyway?

[eesger]

You could have guessed, and guessed right : the YI 360

[dave683]

ah, well this video worked for me but its for the mi sphere, maybe the instructions might be similar though

Hugin, Dual Fisheye Stitching

[eesger]

you load a pto file? Who built that for the MI sphere? That contains the knowledge I seek… how is that done?

[eesger]

I searched the web and found a forum where I popped the question
https://groups.google.com/forum/#!msg/hugin-ptx/vYeBUu7_7Ic/__EDTzMVCgAJ

My hero there created an assistant (you can download via the link above)

1. I placed that file here: c:\Users[user]\AppData\Roaming\hugin\
2. I set interface to ‘expert’
3. Then I open a raw unstitched image (or simply drag it into the program and drop)
4. I need set a value for “HFOV” (set something, anything… example => 1 ), press OK
5. via menu: Edit > User defined assistant > Assistant for dual lens images (vertical) (it’s at the bottom)
6. In stitcher (last tab) set output image size and set to JPEG

Run it!

I did a few tests, and some didn’t go perfect, got a “black cat eye” (missing part of the image)I juggled around with the settings. In my manual I red that the field of view was 220 (I found 180 in the assistant) also I set the crop a bit larger (95% in stead of 92%)

Here is my custom file (save it to a text file and name it ‘duallens_vertical.assistant’):

[General]
Description=Assistant for dual lens images (vertical fov:220)
Help=Assistant for images shot with dual lens camera/one shot panoramic cameras
StepCount=9

[Step0]
Description=Prepare initial pto file
Condition=single image
Program=pto_gen
Arguments=--output=%project% --projection=2 --fov=220 %image0% %image0%

[Step1]
Description=Setting advanced settings
Condition=single image
Program=pto_var
Arguments=--output=%project% --unlink=v0,d0,e0 --set=y1=180,e0=-0.25*height,e1=0.25*height,d=0 --crop=95%,50% %project%

[Step2]
Description=Searching control points
Condition=not connected
Program=cpfind
Arguments=--output=%project% %project%

[Step3]
Description=Check result
Condition=not connected
Program=checkpto
Arguments=%project%

[Step4]
Description=Cleaning control points
Condition=not connected
Program=cpclean
Arguments=--output=%project% %project%

[Step5]
Description=Searching line cp
Condition=no line cp
Program=linefind
Arguments=--output=%project% %project%

[Step6]
Description=Setting advanced settings
Program=pto_var
Arguments=--output=%project% --opt=y,p,r,v,d,e %project%

[Step7]
Description=Optimizing
Program=autooptimiser
Arguments=-n --output=%project% %project%

[Step8]
Description=Searching for best crop
Program=pano_modify
Arguments=--projection=2 --fov=360x180 --canvas=70% --crop=AUTO --output=%project% %project%

It works very well! Next up, find out how to do this command line…

[dave683]

yea i found the pto file for the mi sphere on github: GitHub - RubenFro/XiaomiMijiaMi-HuginTemplate: Template for Xiaomi Mijia Mi Sphere

but it looks like youre getting there… please share when you crack the code on this mysterious program!

[eesger]

Indeed!
Those assistants are golden!

When you can get a PTO file for your camera all you have to do is open the unstiched image (if you have a double sphere image like me, you open it twice!)

Then select: File > Apply Template

and you’re golden!

When you don’t have a PTO you use the assistant. When you have a good result you can use the generated PTO file exactly the same way (so you have created your own template!).

I did discover you need to try, and try again sometime with exactly the same assistant you can get different results every time! There is quite a bit of fuzzy logic in there!

But now I have a question the result looks perfect, BUT when I overlay the two there is quite a bit of difference… Is that a problem for Mapillary (SfM)?

Original stitch:

with the “moa” shown with arrows:

The Hugin stitch:

Overlaying with the original stitch:

So optically it looks perfect… the question is… did I just now trade fact for fiction?

[eesger]

created another template, “delta result” with stitch with original software:

It “feels better”, but what is wise? From an optical view the Hugin file is much better, but which is better for Mapillary? Or is that detail/distortion something that Mapillary has no trouble with?

[update]
A side note on “speed”. I created a perl script to process a folder with my created “YI360 hugin template”.
I have a Linux server for company testing purposes. That one isn’t a “cpu monster” (and only has a simple onboard graphics card). That one needs 15.5 seconds on avarage for stitching an image (command line, at 5760 x 2880, including conversion to JPG). I expected that my (Windows 10) workstation with a much faster CPU and a slight monster of a GPU would be faster, so I appended my perl script to generate templates suitable for the batchprocessor of the Windows program. But that one needs over 17.7 seconds on average…

With my camera in “movie mode” I filter out a lot of images… so I could get smart and stitch only the images I need…

But before I start doing that I’dd like to know if this way of stitching poses a problem for the “Mapillary platform”?

[4004]

I can’t see how that would be worse for mapillary - if I remember correctly, the main requirement is that the image is equirect

[eesger]

For those that want to use the YI 360. With my template file I managed to get a perfect stitch result with my camera with Hugin. The result is (much) better then the original software. You will need a PTO file to use it, Here you can download mine (61 control points!):

https://www.geoarchief.nl/cms/media/YI360.pto

I recently took 360 images in a cave, and I’m impressed, that even the ceiling is perfect, I cant find a stitch fault anywhere (at the bottom, me holding the camera, there it gets a bit tricky… naturally)

Mapillary

[4004]

I am wondering if this is worth exploring for a gopro array, as opposed to relying on SfM panning

[gpsmapper]

@eesger

The result is (much) better then the original software.

Well, are these moa artifacts persistent when using original software?
Color-wise, original stitch looks slightly-better to me

I have not noticed any of these on Fusion (using Fusion studio), would be interesting to try Hugin to compare. Wonder if these 'template’s are camera-specific (one that work fine with YI will not work with Fusion)?

[eesger]

I do think they are camera type/brand specific! They may even be camera specific! So if you want to try Hugin do search for a PTO file for your camera!

Depending on how the raw images are stored (mine was a “odd”, two spheres above each other) you can use the assistant and then optimize with manually adding points until you are satisfied …

[4004]

oh, so it’s actually fairly plug and play?
last time I tried Hugin it was a lengthy process, that wouldn’t be efficient for 100s of images

[eesger]

There is a learning curve… but once you’ve figured it out… it’s a good program. It has a command line verspion also… using that for my automation

[eesger]

… in other words, once you have a good PTO config file, then the command line tool (and a little scripting) does wonders!

[4004]

I’m not very good at scripting. Also presume hugin is not optical flow?

[eesger]

I am
But I did also get the imperession the GUI (Windows) is able to do batch processing (I didn’t look into it, because I was filled with joy when I found the command line tool)

[eesger]

@javiersanp mentioned he uses Hugin, but doesn’t have a great PTO file…

• What is Hugin?
  It is a poowerfull program which can stitch (combine) multiple images into one. This can be or become a panorama or a equirectangular.
  It does have a bit of a steep learning curve. But once you have figured it out, it produces good results and can be used for batch processing. It’s available for Mac, Windows and Linux.

• What is a PTO?
  Hugin has assistant which can help you in various ways to get started… Each project you create results in a project file which is the template for your stitching project. The extension of this file is PTO.

• How to create your own template?
  When you are lucky you can find a PTO file someone else created. When you are even more lucky that person did a great job creating it.

If not, you need to create one yourself. Best is to use a location which fits the general usage of your work (open space, room etc). Use a location with a lot of good identifiable points. I went to the local gym:

Do make several pictures rotating your camera a bit each time, or at a (slightly) different position!!

Choose your first image and rename it to “image.jpg”

When you have nothing start by determining which assistant gives you a fair result (or use that not so good PTO file you found on the internet).

My camera takes the two images to be stitched below each other, most of the time they are next to each other, choose your assistant wisely!

Then go to “control points” in your wysiwyg Hugin editor.

And start by matching control points as good as you can. Go for pixel perfect!

How I went for perfection (little trick I thought of…)
I perfected my PTO file the following way:
First, do keep in mind that Hugin uses a sort of “fuzzy logic” in their stitching process! Sometimes a dozen points extra gives a worse result… try and get a feel for it and play around with it!
At certain moment I was almost there but a part just didn’t want to go just right. The trick I did was this. After saving my project I renamed my ‘image.jpg’ to “fase one.jpg” and used an other image I took in the gym and renamed that one to “image.jpg” and re-opened my project (later on I discovered I could just as well have opened that image (twice in my case!) and choose “apply template” and then go to “control points” again and start playing around again by finding those extra control points for the perfect result…

As you can see in the image above my template uses a whopping 61 control points… I know, it’s a bit excessive but I am very, very pleased with the results I get

I hope this little tutorial helps