FUBI - Full Body Interaction Framework

Start date: 01.01.2011
Funded by: EU (Europäische Union)
Local head of project: Dr. Felix Kistler


Full Body Interaction Framework (FUBI) is a framework for recognizing full body gestures and postures in real time from the data of a depth sensor integrated using OpenNI or the Kinect SDK.

Quicklinks:        Download        Documentation        Related Publications



Fubi is a framework for full body interaction using a depth sensor such as the Microsoft Kinect or the Asus Xtion. It can be used with OpenNI/NiTE or the Kinect SDK. Since version 0.9 it further supports the Leap Motion Controller. FUBI is written in C++ and additionally includes a C#-Wrapper. FUBI releases are only tested on Windows, but the community has already successfully built it under a Linux environment with only slight modifications needed. The download comes with Visual Studio 2010 and 2013 solutions including two sample application that should be ready to compile. You only need to install the dependencies as described in the installation instructions and set the include and lib paths for OpenCV in Visual Studio or comment out the line "\#define FUBI_USE_OPENCV" in the FubiConfig.h if you do not want to use OpenCV. Within this file you can also define which tracking software you want to use: OpenNI v 1.x, 2.x, or the Kinect SDK 1.x or 2.x, but you can also switch between the different trackers during runtime.
FUBI is freely available under the terms of the Eclipse Public License - v 1.0.
You can get the sources with Visual Studio 2010 and 2013 solutions here:


The download page also provides a precompiled Unity3D integration of Fubi.
More information can be found in the documentation and in the related publications.

If you use Fubi in a scientific project, please cite one of the related publications. If you use FUBI for a game-like application and/or you use the Unity integration, please cite the INTERACT 2013 paper. If you use Fubi with robots, please cite the IJSR 2014 paper. In all other cases, you can cite the ICIDS 2011 paper or the JMUI 2012 paper.

Fubi's main functionality is providing gesture and posture recognition capabilities.
The Framework therefore distinguishes between three gesture categories:

  1. Static postures: Configuration of several joints (positions or orientations) or number of displayed fingers (=finger count), no movement (e.g. figure 1: "arms crossed").
  2. Linear/Angular movements: Linear movement of several joints with specific direction and speed (e.g. figure 2: "right hand moves right") or angular movement of a joint (e.g. "turn head right").
  3. Combination of postures and movements: Combines sets of 1 and 2 in a sequence of states with specific time constraints (e.g. figure 3: "waving right hand").
  4. Symbolic gestures: Gesture with complex shape that are defined by recorded sample data. (e.g. figure 4: "right hand circle").
Figure 1: arms crossed

Figure 2: right hand moves right 

Figure 3: waving right hand
Figure 4: right hand circle

A description of the categories and how they are recognized was published in Journal on Multimodal User Interfaces, you can find the information here. 

Gestures can be defined in C++/C# code or (preferred) using an XML based definition language.
For example, the following XML code defines a head nod gesture for recognition with Fubi:
The head nod is defined as a combination recognizer with four states, each calling a angular movement recognizer that waits for a specific head pitch movement.

The following image shows a study setup, in which 18 participants had to perform gestures during so-called quick time events in a multi-party interactive storytelling scenario. We recognized 97% of their gestures (65 out of 67 gestures).
The corresponding paper was published on ICIDS 2011, you can find more information on it here.

Quick time setup image

More information on the recognizable postures and gestures can be found in the documentation.

Fubi includes a GUI application that is build upon its C# wrapper. In the GUI, you can test your recognizers, look at all the information Fubi offers regarding the sensor streams and user tracking, change and test the filter values, and start mouse emulation for freehand interaction  and bind gestures to key or mouse events, e.g. for clickingThe latest feature is a tool that records gesture performances and can generate valid Fubi gesture XML out of that performance.
You can find more information in the tutorial.
This is how it looks like:

There also exists a Unity Integration for Fubi which again uses the C# wrapper and therefore works with the free Unity versions. It supports to add gestural interaction by using gesture symbols that can be used as default Unity button. As soon as such a symbol is shown on screen, Fubi automatically checks the corresponding recognizer and sends a click event if it has finished successfully. Further, Fubi provides buttons and a swiping menu to implement freehand GUI interaction in Unity.
More information can again be found in the corresponding tutorial.
Here a screen shot (note that it would usually make no sense to use a gesture symbol, a freehand button and a swiping menu all at the same time, but this is meant for demonstration purposes only):

The Unity integration is also used in the Traveller application of the eCute project.
Therefore, we used a technique to gather a user defined gesture set that is described in the paper presented on INTERACT 2013 along with a poster/demonstration that won the best poster award.
Here a screen shot: