Tracking Hands and Fingers With Ultraleap Plugin

Ultraleap plugin allows you to track hands and fingers in your UNIGINE-based application.

The Ultraleap system recognizes and tracks hands and fingers. The device operates in an intimate proximity with high precision and tracking frame rate and reports discrete positions and motion.

The Ultraleap controller uses optical sensors and infrared light. The sensors are directed along the Y axis — upward when the controller is in its standard operating position — and have a field of view of about 150 degrees. The effective range of the Ultraleap Controller extends from approximately 25 to 600 millimeters above the device (1 inch to 2 feet).

Detection and tracking work best when the controller has a clear, high-contrast view of an object's silhouette. The Ultraleap software combines its sensor data with an internal model of the human hand to help cope with challenging tracking conditions.

The Ultraleap system uses the UNIGINE world space coordinates.

If you willing to use the Ultraleap device without the Varjo's HMD, you have to manually provide correct transformation matrix for each device separately:

Mat4 camera_imodelview = camera->getIModelview();
Mat4 camera_offset = Mat4(camera->getOffset());

setTransform(camera_imodelview * camera_offset);

The hand model provides information about the identity, position, and other characteristics of a detected hand, the arm to which the hand is attached, and lists of the fingers associated with the hand.

Hands are represented by the Hand class.

An arm is a bone-like object that provides the orientation, length, width, and end points of an arm. When the elbow is not in view, the Ultraleap controller estimates its position based on past observations as well as typical human proportion.

Arms are represented by the Arm class.

The Ultraleap controller provides information about each finger on a hand. If the whole or part of a finger is not visible, the finger characteristics are estimated based on recent observations and the anatomical model of the hand. Fingers are identified by type name, i.e. thumb, index, middle, ring, pinky.

Fingers are represented by the Finger class.

Each finger has a set of bones describing the position and orientation of corresponding anatomical finger bones. All fingers contain four bones ordered from base to tip.

Bones are represented by the Bone class.

The bones are identified as:

• Metacarpal — the bone inside the palm connecting the finger to the wrist (except the thumb).
• Proximal Phalanx — the bone at the base of the finger, connected to the palm.
• Intermediate Phalanx — the middle bone of the finger, between the tip and the base.
• Distal Phalanx — the terminal bone at the end of the finger.

Ultraleap API:

• The Ultraleap interface article for more details on managing Ultraleap via API
• The Ultraleap interface article for more details on managing Ultraleap devices via API
• The Ultraleap Arm class article for more details on managing arms via API
• The Ultraleap Bone class article for more details on managing finger bones via API
• The Ultraleap Finger class article for more details on managing fingers via API
• The Ultraleap Hand class article for more details on managing hands via API

To use the Ultraleap plugin in your UNIGINE application, perform the following:

1. Download the Ultraleap SDK and install Ultraleap device drivers.

2. Create a new project with Ultraleap support via UNIGINE SDK Browser: click Plugins, check the Ultraleap support (Ultraleap plugin) option in the form that opens and click OK.

   Notice

   To add Ultraleap support to the existing project, in UNIGINE SDK Browser, click Other Actions -> Configure Project -> Plugins -> Ultraleap support (Ultraleap plugin) -> OK.
3. Implement your application.
4. Launch the Ultraleap plugin on the application start-up.

To launch the plugin, specify the extern_plugin command line option on the application start-up as follows:

main_x64d -extern_plugin UnigineUltraleap

If you run the application via UNIGINE SDK Browser, specify the command-line options given above in the Customize Run Options form.