Final post

We've just had our final presentation yesterday and everything went very nice!

As seen above, that is the final product we built, and we have decided to call it Ambinote.

We are starting to write a paper now for our final submission now..

Check out the video of how our interactive device works.

"Access notifications in 3D space using familiar gestures" 

Skin Detection is actually great.. for this problem at least

The building block for our system was represented by a color detector extended to support skin detection, in the HSV space. The reason why we prefered HSV to RGB was the fact that the first space separates the image intensity from the color information, which proved critical when implementing a color detector. Even though there are many spaces which have the notion of separation mentioned above, we used HSV due to existing implementations to convert RGB to HSV and viceversa.

Advantages:

Due to the fact that we had fog in the frame, using a different technique to detect the hand, for example motion detection or background subtraction, would had resulted in a large number of false positives. This effect would have rendered the system highly inaccurate, triggering the notification system even when the user’s input was not actually registered.

Drawbacks:

The main drawback of this system is the fact that we are considering skin detection to be a color detection problem. We have set the HSV values for ranges of pixel intensities that we considered to be skin, which could cause inaccuracies. For example, only a set of skin tones can be detected, while others will not be detected. This variable assumes that we have a priori knowledge of the range of skin tones we want to model. Furthermore, if the working background consists of objects with color values in the range we specify, such as wood, then our system will yield false positive results, as seen in the picture posted above.

Problems with the MotionDetector

Until now, I have been testing the camera and motion detector using just my laptop's built-in camera, which worked fine. Problems occurred when we tied up all the systems together.

As you might recall, I am detecting motion by using a background subtraction method, which checks whether there is change in pixel intensities between two frames. If that occurs, than that portion of the frame is regarded as not belonging to the background, thus it is moving. The problem with this approach, as I had found out, was smoke. Since our setup involved having the USB Camera tied up on the box, it would be very close to the smoke emitted. Because smoke was dense, it would be picked up as movement, thus rendering false positive results.

After having tried many ways to threshold the results produced by the smoke, we decided it would not work in a way which would be reliable, so we decided to re-implement the system using another method..

Integration

As promised in the last blog post, in this short post I will be discussing how we managed to integrate the Computer Vision system with the rest of the components.

In order to make the detector easily usable in other classes and scripts, I moved the code into a MotionDetector class, with easily accessible methods and instance variables.

 Structuring the code this way proved easy for us to use it in the main application logic. In there, we instantiate the object, starting the camera & detection or stopping it whenever we need to.

Work in progress: Vision system III

The final approach I took in order to detect motion was to do Background substraction and tune the algorithm present on the openCV website, in order to filter out noise but also to make more resilient.

I convert the frame into grayscale, then apply the Background Substraction algorithm. The result of this should be only black & white values, but due to heavy light conditions ( where present ), we also encounter median values. Therefore, I also apply a binary threshold algorithm to the result, in order to filter out these values. I then use the openCV implementation findCountours in order to find all the countours in the image. I calculate the area of all of them, and filter the low-area ones. Finally, for viewing purposes, I draw the result.

In the next blog post, I will be talking about how we will be handling communication between the camera and the other components of our device.

Work in progress: Vision system II

The first attempt was to use Motion Detection Algorithms. Luckily, OpenCV has available implementations which you can find here. Detection is achieved using either the Lucas-Kanade algorithm or the Farnerback algorithm.

On the left, I have attached the result of the Farnerback algorithm, which sees better results overall than LK, but has the disadvantage of being very slow, since it relies on heavy computation. One solution for speedup was to use GPU programming.

Before actually doing this, I decided to have a look at other methods.


I started focusing only on skin detection, and not motion. One of the easiest ways of detecting the skin was to define a color range and then simply look for that color range in the frames.

As you can see, this approach is heavily dependent on lighting conditions and also background. For example, if wood material is present in the scene, then that also gets detected as skin, and represents a false positive.

Work in progress: Vision system I

This post is going to be slightly larger than my previous ones, since I haven't posted anything for a while now. Mainly I will cover Computer Vision related aspects, since that is something I have mainly worked on.

The idea we had for the project (initially), was to track smoke rings, suing the Kinect and OpenCV. The whole point of this was to project on them, without having a static location where we project. The main advantage the Kinect offers is depth recognition.

So far, so good.

Since I am using Ubuntu Linux, I managed to install the openKinect and freenect libraries on my machine, since the official SDK is available only on Microsoft Windows.

On the left you can see the depth representation from the Kinect camera. We encountered that while trying to shoot smoke rings, they would not get recognized as altering the depth of the image. Because of this problem, we decided to drop the Kinect but also to stop trying to recognize the smoke rings.

Instead, we started looking at how to recognize hand gestures solely in OpenCV.

In the next posts I will describe the techniques we used to build a reliable system.