Using Kinect in Glovepie with OSC messages and OpenNi framework

Glovepie doesn´t currently support Kinect but there is a work around using OSC messages inside Glovepie and using the Glovenect OSC client which sends the OpenNi Skeleton´s nodes in separate OSC messages which can be read by Glovepie and thus allowing you to use Kinect emulating any kind of Input device that glovepie supports.

In this case i´ve scripted a simple scene with Glovepie and I´ve mapped the coordinates of the hands to trigget midi notes via Software MIDI-YOKE. There is also a fader that is controlling the modwheel of the Reason patch and two Octave buttons, one up and one down.

The music is clearly not the point of the project but the integration of Kinect inside glovepie.

How can you do this?
Steps:
*Get a kinect with the USB adapter
*Download latest OPENNI and NITE unstable binaries and install.(github)
*Download latest SensorKinect Drivers from avin2 (github)
*Download Glovepie 0.43 (glovepie.org)
*Install first OPENNI, then NITE, then the SensorKinect Drivers

Here you have the Glovenect (which is a modification I made to the program OSCELETON that sends openni´s joints as OSC messages, the difference is that this modification of the source code sends 3 OSC messages per joint (x,y,z) which works with Glovepie as it doesn´t support bundled OSC messages like the current OSCELETON version)

https://rapidshare.com/files/459355511/Kinect.rar
*Run Glovenect and calibrate the user with the PSI pose.
*For the script i´m running you also need MIDI YOKE software and Propellerheads Reason 4.0/Ableton Live
*Now run Glovepie and type
Osc.listening = true
osc.ListenPort = 5252
/////////////////////////
////// SKELETON READ/////
/////////////////////////
var.head_x = osc.x.joint.head.1
var.head_y = osc.y.joint.head.1
var.head_z = osc.z.joint.head.1
var.neck_x = osc.x.joint.neck.1
var.neck_y = osc.y.joint.neck.1
var.neck_z = osc.z.joint.neck.1
var.l_shoulder_x = osc.x.joint.l_shoulder.1
var.l_shoulder_y = osc.y.joint.l_shoulder.1
var.l_shoulder_z = osc.z.joint.l_shoulder.1
var.r_shoulder_x = osc.x.joint.r_shoulder.1
var.r_shoulder_y = osc.y.joint.r_shoulder.1
var.r_shoulder_z = osc.z.joint.r_shoulder.1
var.l_hand_x = osc.x.joint.l_hand.1
var.l_hand_y = osc.y.joint.l_hand.1
var.l_hand_z = osc.z.joint.l_hand.1
var.r_hand_x = osc.x.joint.r_hand.1
var.r_hand_y = osc.y.joint.r_hand.1
var.r_hand_z = osc.z.joint.r_hand.1
var.l_elbow_x = osc.x.joint.l_elbow.1
var.l_elbow_y = osc.y.joint.l_elbow.1
var.l_elbow_z = osc.z.joint.l_elbow.1
var.r_elbow_x = osc.x.joint.r_elbow.1
var.r_elbow_y = osc.y.joint.r_elbow.1
var.r_elbow_z = osc.z.joint.r_elbow.1
var.l_hip_x = osc.x.joint.l_hip.1
var.l_hip_y = osc.y.joint.l_hip.1
var.l_hip_z = osc.z.joint.l_hip.1
var.r_hip_x = osc.x.joint.r_hip.1
var.r_hip_y = osc.y.joint.r_hip.1
var.r_hip_z = osc.z.joint.r_hip.1
var.torso_x = osc.x.joint.torso.1
var.torso_y = osc.y.joint.torso.1
var.torso_z = osc.z.joint.torso.1
var.waist_x = osc.x.joint.waist.1
var.waist_y = osc.y.joint.waist.1
var.waist_z = osc.z.joint.waist.1
var.l_knee_x = osc.x.joint.l_knee.1
var.l_knee_y = osc.y.joint.l_knee.1
var.l_knee_z = osc.z.joint.l_knee.1
var.r_knee_x = osc.x.joint.r_knee.1
var.r_knee_y = osc.y.joint.r_knee.1
var.r_knee_z = osc.z.joint.r_knee.1
var.l_foot_x = osc.x.joint.l_foot.1
var.l_foot_y = osc.y.joint.l_foot.1
var.l_foot_z = osc.z.joint.l_foot.1
var.r_foot_x = osc.x.joint.r_foot.1
var.r_foot_y = osc.y.joint.r_foot.1
var.r_foot_z = osc.z.joint.r_foot.1
////////////////////
// LOADING OBJECTS//
////////////////////
obj7.pos = [var.head_x, -var.head_y, -var.head_z]
obj8.pos = [var.l_shoulder_x, -var.l_shoulder_y, -var.head_z]
obj9.pos = [var.r_shoulder_x, -var.r_shoulder_y, -var.r_shoulder_z]
obj10.pos = [var.l_hand_x, -var.l_hand_y, -var.l_hand_z]
obj11.pos = [var.r_hand_x, -var.r_hand_y, -var.r_hand_z]
obj12.pos = [var.torso_x, -var.torso_y, -var.torso_z]
//obj13.pos = [var.waist_x, -var.waist_y, -var.waist_z]
obj14.pos = [var.l_knee_x, -var.l_knee_y, -var.l_knee_z]
obj15.pos = [var.r_knee_x, -var.r_knee_y, -var.r_knee_z]
obj16.pos = [var.r_foot_x, -var.r_foot_y, -var.r_foot_z]
obj17.pos = [var.l_foot_x, -var.l_foot_y, -var.l_foot_z]
obj18.pos = [var.r_elbow_x, -var.r_elbow_y, -var.r_elbow_z]
obj19.pos = [var.l_elbow_x, -var.l_elbow_y, -var.l_elbow_z]
obj20.pos = [var.r_hip_x, -var.r_hip_y, -var.r_hip_z]
obj21.pos = [var.l_hip_x, -var.l_hip_y, -var.l_hip_z]
obj22.pos = [var.neck_x, -var.neck_y, -var.neck_z]

Now go to the Variables tab and you will see all the coordinates of the joints which now you can use in Glovepie to create vectors between themselves and linking them to any binding you like.

credits goes to: Creator of Kinect, coders of Openni,Nite and driver hack, creator of OSCELETON, Carl Kenner of Glovepie and just a little me for adaptation.

Category:

Science & Technology

Tags:

• Glovepie
• OPENNI
• Kinect
• NITE
• OSCELETON
• MIDI
• BINDINGS
• 3D
• MAPPING
• HEAD
• TRACKING
• VIRTUAL
• REALITY
• YOKE

[MrTuomoK]

Thanks a million for this :)

[rdickerhoof]

Any chance you can send me glovenect 1.1?

[shplurminator10]

very cool video. i can't seem to get the rotation matrices to produce values using glovenect within glovepie.  they stay constantly at 0, and when i call to obj1.pitch for instance it stays constantly at 0. any ideas? thanks!

[GameFreakDude]

I am just amazed you got this to work in GlovePIE. How did you manage to think up that work around???