Tracking Hands and Fingers With LeapMotion Plugin
Overview
LeapMotion plugin allows you to track hands and fingers in your UNIGINE-based application.
The Leap Motion system recognizes and tracks hands and fingers. The device operates in an intimate proximity with high precision and tracking frame rate and reports discrete positions and motion.
The Leap Motion controller uses optical sensors and infrared light. The sensors are directed along the y-axis – upward when the controller is in its standard operating position – and have a field of view of about 150 degrees. The effective range of the Leap Motion Controller extends from approximately 25 to 600 millimeters above the device (1 inch to 2 feet).
Detection and tracking work best when the controller has a clear, high-contrast view of an object’s silhouette. The Leap Motion software combines its sensor data with an internal model of the human hand to help cope with challenging tracking conditions.
Coordinate System
The Leap Motion system uses a right-handed Cartesian coordinate system. The origin is centered at the top of the Leap Motion Controller. The x- and z-axes lie in the horizontal plane, with the x-axis running parallel to the long edge of the device. The y-axis is vertical, with positive values increasing upwards (in contrast to the downward orientation of most computer graphics coordinate systems). The z-axis has positive values increasing toward the user.
Hands
The hand model provides information about the identity, position, and other characteristics of a detected hand, the arm to which the hand is attached, and lists of the fingers associated with the hand.
Hands are represented by the Hand class.
Arms
An arm is a bone-like object that provides the orientation, length, width, and end points of an arm. When the elbow is not in view, the Leap Motion controller estimates its position based on past observations as well as typical human proportion.
Arms are represented by the Arm class.
Fingers
The Leap Motion controller provides information about each finger on a hand. If all or part of a finger is not visible, the finger characteristics are estimated based on recent observations and the anatomical model of the hand. Fingers are identified by type name, i.e. thumb, index, middle, ring, and pinky.
Fingers are represented by the Finger class.
Bones
Each finger has a set of bones describing the position and orientation of corresponding anatomical finger bones. All fingers contain four bones ordered from base to tip.
Bones are represented by the Bone class.
The bones are identified as:
- Metacarpal – the bone inside the hand connecting the finger to the wrist (except the thumb).
- Proximal Phalanx – the bone at the base of the finger, connected to the palm.
- Intermediate Phalanx – the middle bone of the finger, between the tip and the base.
- Distal Phalanx – the terminal bone at the end of the finger.
Sensor Images
Along with the computed tracking data, you can get the raw sensor images from the Leap Motion cameras.
The image data contains the measured IR brightness values and the calibration data required to correct for the complex lens distortion. You can use the sensor images for augmented reality applications, especially when the Leap Motion hardware is mounted to a VR headset.
See Also
- C++ sample illustrating the basic aspects of working with the plugin:
source/samples/3rdparty/LeapMotionVisualizer
LeapMotion API:
- The LeapMotion interface article for more details on managing LeapMotion via code (C++, UnigineScript).
- The LeapMotion Arm class article for more details on managing arms via code (C++, UnigineScript).
- The LeapMotion Bone class article for more details on managing finger bones via code (C++, UnigineScript).
- The LeapMotion Finger class article for more details on managing fingers via code (C++, UnigineScript).
- The LeapMotion Hand class article for more details on managing hands via code (C++, UnigineScript).
Implementing Unigine Application with LeapMotion Support
To use the LeapMotion plugin in your UNIGINE application, perform the following:
- Download the Leap Motion SDK and install Leap Motion device drivers.
- Create a new project with LeapMotion support via UNIGINE SDK Browser: click Plugins, check the LeapMotion support (LeapMotion plugin) option in the form that opens and click Ok.
LeapMotion Plugin on Plugins PanelNoticeTo add LeapMotion support to the existing project, in UNIGINE SDK Browser, click Other Actions -> Configure Project -> Plugins -> LeapMotion support (LeapMotion plugin) -> Ok. - Implement your application.
- Launch the LeapMotion plugin on the application start-up.
Launching LeapMotion
To launch the plugin, specify the extern_plugin command line option on the application start-up as follows:
main_x64d -extern_plugin LeapMotionIf you run the application via UNIGINE SDK Browser, specify the command-line options given above in the Customize Run Options form.
Compiling LeapMotion
Plugin compilation is required if you change its source code (for example, add new API functions).
Leap Motion SDK is required for recompiling the LeapMotion plugin:
- Get the Leap Motion SDK.
- Set the path to the <Leap Motion SDK> folder in the LEAP_SDK_DIR environment variable.
Shell commands
set LEAP_SDK_DIR=<Leap Motion SDK> - Compile the plugin in one of the following ways:
- By using the build script:
Shell commands
C:\UnigineSDK\source>build.py --plugin LeapMotion - Manually with Scons: go to the <UnigineSDK>\source\plugins\Hardware\LeapMotion via the command line, type scons and press ENTER:
Shell commands
C:\UnigineSDK\source>scons LeapMotion - Via Microsoft Visual Studio: go to the <UnigineSDK>\source\plugins\Hardware\LeapMotion, open the leapmotion_vs2013.vcxproj or leapmotion_vs2015.vcxproj project and build it.
- By using the build script: