Suggestions for computer vision features

What are your favorite suggestions for Mapillary features using computer vision? Let’s discuss here.

Some suggestions:

  • Automatic detection of car hoods and other vehicle parts in sequences. Then we soft crop or hide these regions.
  • Visual image search. Mark a region in an image and find other images with the same/similar object across all of Mapillary.
  • Automatic object tagging (with user feedback and suggestions for objects).
  • Blur detection. To flag images to hide.

Add your ideas!

Ideas collected from replies below:

  • Detect reflections on wind shields

One more idea is to do text detection and OCR, so we can also extract semantic information from images.

2 Likes

Many pictures taken from within a car has some or a lo of the dash board visible. Perhaps it would be possible to detect this and crop the series? Of cause the dash board changes between pictures, when the light changes, but it is usually very little between individual images. The change is more when you turn or the Sun changes.

1 Like

detect reflections on wind shields. http://news.mit.edu/2015/algorithm-removes-reflections-photos-0511

3 Likes

Yup. Getting rid of car parts is top of my wish list!

Definitely doable if we can just carve out enough time to work on it.

We can try to detect in the apps at capture and warn “consider adjusting your camera placement” too.

If you get it working well enough, it would be nice to run it on the existing images. I have several times let some dash board show purposely, because the clouds in the sky would make the auto exposure let the images become way too dark.

But a warning would be good too. I have tried hard to avoid dash board and still gotten it several times.

The main issue I get is not the dashboard itself, but its reflection in the windshield, as @de_vries pointed out. Oftentimes no amount of repositioning gets me rid of it without losing large portions of the scene. It would be great to have a system for removing not only the dashboards, but their reflections as well (and while we’re at it, raindrops and other kinds of mostly static obstructions too).

Someone suggested black T-shirt (or cloth would do) on the dashboard to get rid of reflections. Haven’t tried but sounds reasonable.

1 Like

I just browsed this sequence: http://www.mapillary.com/map/im/DJ7J4nAdEtN2SwSxBD7g7Q/photo
and thought that removal of animals and drops of water on a wind shield would be great. Mapillary gets a lot of images that would never hit a service like Google Streetview, but some of the errors can probably be fixed.

I know you computer vision guys don’t do magic, but you can make it look like that and that is good enough.

I’d like to see these two:

  • Visual image search. Mark a region in an image and find other images with the same/similar object across all of Mapillary.

Sounds like an expensive operation though…

  • Automatic object tagging (with user feedback and suggestions for objects).

This would be wonderful. I’d probable ‘teach’ it to recognize our local bus stop signs and shelters. Or those signs that indicate there is an underground hydrant. Those make good spatial reference points, so I go out of my way to make pictures of them.

Speaking of reference points. Are you still working on creating a DB of ‘recurring objects’? That’s probably a question for a separate thread though.

Hi @Polyglot

We are actually looking into both of your suggestions. Visual search is not as expensive as it sounds. Object tagging requires some infrastructure work. We’ll keep you posted on progress.

Thanks

I have got a suggestion for discussion here: Using computer vision (monocular odometry) to correct vehicle path

Will appreciate any replies!

I’ve already try to correct the pictures’ exposure, but it’s difficult to do it automaticaly. Could computer vision help for this ?
With the classification you compute on all pictures, it could be easier to exclude unuseful features like clouds, sky, to correct the exposure ?

1 Like