Final post

We've just had our final presentation yesterday and everything went very nice!

As seen above, that is the final product we built, and we have decided to call it Ambinote.

We are starting to write a paper now for our final submission now..

Check out the video of how our interactive device works.

"Access notifications in 3D space using familiar gestures" 

Skin Detection is actually great.. for this problem at least

The building block for our system was represented by a color detector extended to support skin detection, in the HSV space. The reason why we prefered HSV to RGB was the fact that the first space separates the image intensity from the color information, which proved critical when implementing a color detector. Even though there are many spaces which have the notion of separation mentioned above, we used HSV due to existing implementations to convert RGB to HSV and viceversa.

Advantages:

Due to the fact that we had fog in the frame, using a different technique to detect the hand, for example motion detection or background subtraction, would had resulted in a large number of false positives. This effect would have rendered the system highly inaccurate, triggering the notification system even when the user’s input was not actually registered.

Drawbacks:

The main drawback of this system is the fact that we are considering skin detection to be a color detection problem. We have set the HSV values for ranges of pixel intensities that we considered to be skin, which could cause inaccuracies. For example, only a set of skin tones can be detected, while others will not be detected. This variable assumes that we have a priori knowledge of the range of skin tones we want to model. Furthermore, if the working background consists of objects with color values in the range we specify, such as wood, then our system will yield false positive results, as seen in the picture posted above.

Problems with the MotionDetector

Until now, I have been testing the camera and motion detector using just my laptop's built-in camera, which worked fine. Problems occurred when we tied up all the systems together.

As you might recall, I am detecting motion by using a background subtraction method, which checks whether there is change in pixel intensities between two frames. If that occurs, than that portion of the frame is regarded as not belonging to the background, thus it is moving. The problem with this approach, as I had found out, was smoke. Since our setup involved having the USB Camera tied up on the box, it would be very close to the smoke emitted. Because smoke was dense, it would be picked up as movement, thus rendering false positive results.

After having tried many ways to threshold the results produced by the smoke, we decided it would not work in a way which would be reliable, so we decided to re-implement the system using another method..

Integration

As promised in the last blog post, in this short post I will be discussing how we managed to integrate the Computer Vision system with the rest of the components.

In order to make the detector easily usable in other classes and scripts, I moved the code into a MotionDetector class, with easily accessible methods and instance variables.

 Structuring the code this way proved easy for us to use it in the main application logic. In there, we instantiate the object, starting the camera & detection or stopping it whenever we need to.

Work in progress: Vision system III

The final approach I took in order to detect motion was to do Background substraction and tune the algorithm present on the openCV website, in order to filter out noise but also to make more resilient.

I convert the frame into grayscale, then apply the Background Substraction algorithm. The result of this should be only black & white values, but due to heavy light conditions ( where present ), we also encounter median values. Therefore, I also apply a binary threshold algorithm to the result, in order to filter out these values. I then use the openCV implementation findCountours in order to find all the countours in the image. I calculate the area of all of them, and filter the low-area ones. Finally, for viewing purposes, I draw the result.

In the next blog post, I will be talking about how we will be handling communication between the camera and the other components of our device.

Work in progress: Vision system II

The first attempt was to use Motion Detection Algorithms. Luckily, OpenCV has available implementations which you can find here. Detection is achieved using either the Lucas-Kanade algorithm or the Farnerback algorithm.

On the left, I have attached the result of the Farnerback algorithm, which sees better results overall than LK, but has the disadvantage of being very slow, since it relies on heavy computation. One solution for speedup was to use GPU programming.

Before actually doing this, I decided to have a look at other methods.


I started focusing only on skin detection, and not motion. One of the easiest ways of detecting the skin was to define a color range and then simply look for that color range in the frames.

As you can see, this approach is heavily dependent on lighting conditions and also background. For example, if wood material is present in the scene, then that also gets detected as skin, and represents a false positive.

Work in progress: Vision system I

This post is going to be slightly larger than my previous ones, since I haven't posted anything for a while now. Mainly I will cover Computer Vision related aspects, since that is something I have mainly worked on.

The idea we had for the project (initially), was to track smoke rings, suing the Kinect and OpenCV. The whole point of this was to project on them, without having a static location where we project. The main advantage the Kinect offers is depth recognition.

So far, so good.

Since I am using Ubuntu Linux, I managed to install the openKinect and freenect libraries on my machine, since the official SDK is available only on Microsoft Windows.

On the left you can see the depth representation from the Kinect camera. We encountered that while trying to shoot smoke rings, they would not get recognized as altering the depth of the image. Because of this problem, we decided to drop the Kinect but also to stop trying to recognize the smoke rings.

Instead, we started looking at how to recognize hand gestures solely in OpenCV.

In the next posts I will describe the techniques we used to build a reliable system.

Workshop 2: Further brainstorming & ideas

Following this week's Interactive Devices workshop and group meetings, we further explored our idea of projecting notifications on vortex clouds. After playing around with the vortex guns provided by the Bristol Interactive Group(BIG).

While shooting two guns at each other, we found out it's very difficult to collide them and create a screen from the collision. Therefore we decided we would need bigger guns to create better fog vortexes. We also realized we can only see the projection only if we face the projected beam.

We also had a look at the SensaBubble device and how we could improve the project. We also had a conversation with Sue-Ann and Diego from BIG about this. We proposed a solution to control the bubbles, either using fans or ultrasound, but both solutions did not seem feasible. Another solution would have been to use electro-magnetic fields for control, but a paper was already submitted by the BIG team.

Finally, we decided we were going to build bigger vortex guns using existing tools in the lab. We also decided we were going to experiment with tracking the bigger smoke vortexes and projecting colors on them, idea which could evolve into an interactive game.

[Group Post] Workshop 2: Project Idea and Spec List

We decided to bring to life the idea of using a fog screen in a notification system. The idea can be broken up into the following components:

1. Fog vortex screen: Our idea is to create a screen by colliding two fog vortices.

2. Projection: We wish to project on the fog screen. We hope to be able to project the relevant logo of the notification.

3. Fog: We want to create an automated fog delivery system composed of a standard fog machine and an Arduino kit.

4. Gesture recognition: We aim to detect the vortex and open the tab of the notification (if applicable) if the user swipes the vortex before it dissipates.

5. Notifier app: To get the notification from the user’s phone to the computer, we built a very

basic application. We focus more on its functionality than the UI. The application reads the notifications received by the user and sends it through a server to the computer.

6. Python script: We will write a Python script running on the user`s computer to assemble the whole system. The image processing required would be run separately from this script. The script  queries the server for incoming notifications and manages the system.

For the project, we required the following materials:

• Arduino Uno board
• Servo Motor
• 3ft Plastic Tube
• MDF Plywood
• Subwoofer Speakers
• Amplifier
• Pico Projector
• Smoke Machine with Remote
• Webcam
• Other materials/tools like glue, duct tape etc.

[Group Post] Workshop 1: Initial Ideas and Research

Ideas

The two ideas that the group pitched at the end of the first workshop were as follows:

Interactive, smart lamp: The idea was to create a smart lamp that can be connected to a pico projector. Furthermore, we were planning to use two guns that shoot fog vortices to create a screen for the projection when the two vortices collided. We wanted to project notifications that the user receives on their mobile phones.

Wearable Technology & Dating: The second idea was more like a dating device which was like a wearable technology version of the popular dating app Tinder. While it would be interesting to look into if a person’s attraction can be measured so easily, this was a humorous and, perhaps even, unfeasible idea.

Our third and unofficial idea was to create a system that can study a user’s body language and/or voice to track their mood. In the case of negative moods, it can automatically change the colour of the lighting and perhaps put on music that can be therapeutic. Another parameter that could be taken into  consideration is a user’s pulse using Kinects. However, this idea was not presented to the class as it was considered too mainstream.

Ideas and Papers

After narrowing down the ideas, we looked for 10 research papers or related material to our projects. This is a discussions of the papers we found:

1. Projecting images on fog screen/assistant lamp/pico projector

• The interactive Fog Screen [source]

The research: A "fog screen" represents an immaterial screen where projections can be made, and has properties such as air and little humidity. These properties allow for high-quality projections in mid air. One feature which the paper describes is projecting on both sides of the screen, with the possibility of adding different images on both sides. The paper also suggests that various tracking methods can be employed on the fog screen, such as computer vision.

Contextual relevance: The idea of having different ideas on both sides of the screen  could be useful if we could use the same fog screen in an office where employees would have direct facing screens and notifications could be projected for both of them using the same screen.  Using computer vision for tracking is another useful idea which we can employ in our project.

• MisTable: reach-through personal screens for tabletops [ source]

The research: Fog screens have been the subject of research for the last few years. The paper on ‘MisTable’ has created an interactive tabletop system that combines tabletops with personal screens made of fog, which are both see-through as well reach through. This paper dwells on the challenges of building such a reach-through system and discusses further possibilities of this concept.
Contextual relevance: The team was particularly interested in the implementation of such a system as it combines a fog screen with user tracking to make it an interactive display.

Furthermore, user tracking, as discussed as part of our idea, was also used in this research. Here, it was used for two main reasons: firstly, to display 3D contents with motion parallax and perspective depth cues; and, secondly, to track interaction that is being done with the 3D objects on the screen.

• 360-degree fog projection interactive display - [source]

The research: This paper is about creating a new method to develop a fog display which enables observers to recognise a 3D shape of virtual objects. Like the research done on MisTable, this paper utilises motion parallax.Contextual relevance:This paper is interesting for our project idea mostly for its implementation of a fog display. We are interested in creating a fog screen that holds form long enough to display certain notifications. The paper also goes on to speak about 360-degree projection on the fog that will probably not be of relevance to our project.

• Depth-Fused 3D Imagery on an Immaterial Display [source]

The research: This paper discusses an immaterial display that uses a generalized form of depth-fused 3D (DFD) rendering to create unencumbered 3D visuals.  However, in practice, a walk-through display is still currently out of reach because alignment errors and induced turbulence are more obvious when users are close to a screen. This can be possibly mitigated with better tracking and closer screen placement.

Contextual relevance: According to the authors of the paper, an ideal 3D display would allow users to perceive and absorb visual information in an intuitive manner, without the need of user-worn glasses. Since depth focussed 3D (DFD) offers 3D perception by superimposition of 2 images on a transparent screen with varying luminance, an immaterial display using this technique could be a step in the direction of an ideal display.

• MirageTable: Freehand Interaction on a Projected Augmented Reality Tabletop [source]

The research: MirageTable is an interactive system designed to merge real and virtual worlds into a single spatially registered experience on top of a table using a single depth camera, a stereoscopic projector, and a curved screen. The depth camera tracks the user’s eyes and performs a real-time capture of both the shape and the appearance of any object placed in front of the camera (including user’s body and hands). In addition, the user can interact with virtual objects through physically-realistic freehand actions without any gloves, trackers, or instruments.
Contextual relevance: The research uses interaction with augmented reality. While our project does not strictly dwell on similar topics, the interaction element of this project can be useful to our project.

2. Wearable technology with respect to dating in the real world

• Wearable Intelligent Systems for E-Health [source]

The research: The paper explores the use of assistive technologies for old people, in order to ensure their well-being. Wearable Intelligent Systems for e-Health (WISEs) are used for collection of health information such as blood pressure(BP). Wearable BP monitoring focuses on "continuous and noninvasive measurement without using a cuff". The pulse can be measured from the radial pulse waveform by arterial tonometry. Such devices work along with a digital assistant such as a PDA, a watch or a finger ring."Together with electronic textile (e-textile)-based technologies, the technique can also be used to design a shirt for long-term, hands-free continuous monitoring of BP."
Contextual relevance: Techniques described in this paper are useful in understanding how we might implement our second proposal described in the previous post.

• Sociometric badges: wearable technology for measuring human behaviour [source]

The research: This research was found in the form of a multi-chaptered thesis written in MIT. The research carried out measured human behaviour using  wearable electronic badges  to automatically collect data about the human’s social behaviour.It tracked movement, face-to-face conversations and other human interactions.

Contextual relevance: The research done was particularly interesting due to some of its ideas: firstly, the wearable badges could communicate with other using radio signals; secondly, it could connect with bluetooth-enabled phones and, lastly and most importantly, it could track user behaviour.

While this research entailed to using only accelerometers, microphones and other relevant technologies, it would be a good starting point as we start catering to our project’s needs.

• An experimental research project: wearable technology for embodiment of emotions

The research: For this paper, textiles were integrated with new technologies to create garments that provide new social interactions and avenues for emotional expression. The aim was to explore new possible interaction scenarios through wearable technologies to study an intangible phenomenon, emotion, as a tangible artefact.

Contextual relevance: The project deals with translating human emotion into data, using technology, which would be the aim of this project as well.

• Don’t Mind Me Touching My Wrist: A Case Study of Interacting with On-Body Technology in Public [source]

The research: This paper puts forward the importance of interaction designers taking into consideration the implications of on-body gesture interactions when designing wearable interfaces. In this study, third-party reactions were studied with respect to a user's interactions with wearable technological interfaces. This study dealt with two topics - one, societal perception of the user interaction with the device based on where it was located, and two, the observer's attitude towards the on-body controller placement.

Contextual relevance: Wearable technology puts forward a wide new set of challenges with respect to Human-Computer interaction (HCI). This paper provided insight into the challenges involved when designing such a device - not only does the device have to be liked by the user, but it also needs to be socially acceptable.

• Design of 'smart' fashion garments - [source]

The research: The research was based on design of garments for a context specific performance to bring in I/O devices, wireless LAN activated environments combined with audio/video and choreography.

A wearable system called Cinderella AIDD (Artificial Intelligence Dating Device) was developed. AIDD contains an RFID tag with the wearer's personal information and dating preferences, a wireless reader programmed to match-make data from others tags to use AI profiling to scope potential suitable partners. An alert is sent when promising suitors approach. The wearer can leave a smart "slipper card" behind with his or her new acquaintance. The card encrypts a data profile for use in romantic follow-up, including a GPS locator.

Contextual relevance: The gizmo is indicative of ongoing practice-based research into visualising fashion through the medium of scientific technologies. It is interesting to note the level of interest in technology for fashion and personalised human computer interaction at  University of Arts, London.

Workshop 1: Brainstorming ideas

During our 1st workshop for Interactive Devices, we started our brainstorming session by reading out a booklet provided, which guided us through the fundamentals of a "good" session.

Further on, we used the sticky notes provided to come up with a large number of ideas ( around 40 was the number suggested ). We were also encouraged to "think big", into coming up with ideas which seem close to impossible in terms of achievement. Some of the ideas we came up with are mentioned bellow:

• An interactive lamp
• A virtual-projected keyboard for tablets
• A pet drone
• A projection on top of a car's windshield, which could be useful to display GPS arrows
• A "wearable assistant", which could wake people up based on personal timetables
• Kinect as a security device
• Color changing phones, based on battery usage
• A candy-firing gun
• Augmented reality windshields

A member of the BIG lab from Bristol University was assigned at this stage to criticize our ideas and help us refine the good ones. After discussing what we managed to come up with, our next objective was to select two of our ideas and expand on them, in order to present them in front of the cohort. We also had to sketch mock-ups of the products.

As it can be seen from the picture above, our first idea revolves around the idea of projecting notifications on a foggy surface. We chose fog since it has constant velocity and trajectory. Therefore we can use two "guns" to fire and intersect fog clouds, which will then form the base where we will project the notifications. One interesting property of fog is that it dissapears after a while, so it makes sense to project notifications on it. This idea is the decision to morph together two of our previous ideas, the one which involved the candy gun and the one with the smart lamp, which was able to follow the user around.

The second idea we proposed was to help people recognize when someone they like has mutual feelings. We would use a health-tracking device to monitor the person's vitals, such as blood pressure, and also close-proximity technology, such that when people on the street would find each other attractive, the devices would know and notify them of a match.

Finally, all of the groups presented their ideas, and everybody's ideas seemed to be very clever and interesting.

We are really excited since next week we will be playing around with lots of tech kit, which we will probably end up using in our projects.

Review - UbiGreen: Investigating a Mobile Tool for Tracking and Supporting Green Transportation Habits

Since personal transportation in the US represents one of the major contributors of CO2, the aim of this paper is to explore ways in which a mobile device's screen can give users feedback on their transportation behavior. Specifically, by tracking people's transportation activity, the app aims to encourage people to chose Eco-friendlier means of transportation.

The paper first presents findings on two studies performed in order to better understand transportation habits: an online survey and an experience sampling (ESM) study.

• Online survey results: indicated that people were receptive to change and showed willingness to use a system like the one proposed in the paper.
• ESM study results: "helped reveal the hidden complexities behind the perception and selection of a transportation mode". Also, the participants were showed an initial version of the app, in order to see if they would be able to understand the interface.

Further the initial studies, the paper describes the research done using the app they developed, called UbiGreen Transportation Display. The purpose of the app is to provide the user with personal awareness about green transportation behaviors through iconic feedback. Once the app detects "green activities" such as riding the bus or train, walking, biking, or carpooling, it updates the phone's wallpaper accordingly, as it can be seen bellow.

Research in conservation

 psychology showed that caring for animals helps 

humans connect with nature.

The issue with the study, at that time, was the fact that it was semi-automatic, since subjects were also required to wear a tracking device (issue was corrected with the use of accelerometer-capable devices in a future study). Participants reported problems with the device, such as loosing Bluetooth connection between it and the phone.

Another issue was the sampling of test subjects. The number of people involved was 14 and 5 of them were students, and all of them lived in urban areas, namely Pittsburgh and Seattle.

The study ranged from 1 week to 4 weeks - an average of 21 days of data was collected - and findings are as follows:

• "Walking" was the most popular form of transportation, followed by "driving alone", "carpooling" and "bus rides".
• Participants liked the visual feedback as their wallpaper, since it was "omnipresent" and they also perceived it as a "progress bar".
• Some participants wanted more variety in visual rewards, maybe having weekly different stories.
• Actual carbon emission would have been helpful, in addition to iconic feedback.
• Secondary items were noticed by few of the subjects.
• Negative imagery was suggested as a form of feedback.
• Participants considered the app to be game-like, and felt like there were ways people could cheat the system. Also, they felt they were being cheated if the app didn't award them points.
• New themes could be offered in the future, as it stimulates anticipation for the users.
• The graphics on the screens stirred the interest of co-workers and family.
• As an improvement, real-time suggestions for alternative means of transportation could be added.

Paper link: http://dl.acm.org/citation.cfm?id=1518861

Citation:

CHI '09 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 1043-1052 ACM New York, NY, USA ©2009

Review: Experimental Evaluation of User Interfaces for Visual Indoor Navigation

This paper presents methods for indoor mobile localization, using AR and VR techniques. Specifically, it explores & describes a "novel" UI implementation which consists of Virtual Reality and Augmented Reality views that "ensure localization accuracy", in situations such as a person being in an airport and the phone gives directions. The technology uses computer vision to capture and match images of the environment with previously recorded reference images of known location. The problem, however, lies in building an UI which addresses the issue of "visual localization", which the paper describes and showcases with the help of a working prototype and a series of experiments.

In terms of content, the paper discusses previous work done in the field and also the related background. In this section, the authors argue why "visual localization" is better than other techniques available. The reader is also introduced to the Augmented Reality and Virtual Reality concepts, in the context of UI's for pedestrian navigation systems.

Before building a prototype, the authors conducted an experiment on users to understand the user preference in terms of AR vs VR. In order to confirm the hypotheses, the authors built a mobile app and performed 3 experiments, one per hypothesis. Following the experiments, they were able to confirm the hypotheses and draw the conclusions. In the final section, they discuss how AR can be improved, limitations of the technique, and also how future work could combine AR and VR to provide better UI's.

Overall, the paper is very good, with the minor flaw that it sometimes talks about a certain concept without providing a reference to it. In terms of influence, the paper is cited only twice in reports done by university students, and in one of them it is argued that the general method employed in this paper are usually not reliable.

Citation:

CHI '14 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 3607-3616 ACM New York, NY, USA ©2014

Paper link: http://dl.acm.org/citation.cfm?id=2557003