Dubstep enthusiasts have a new god. The V Motion Project, designed for Frucor, is a machine that turns movement into music–and not in a elementary or experimental way; it produces real sound and fantastic visuals. The system is controlled by a music performer who stands in front of two Microsoft Kinect cameras and controls a virtual representation of his body projected into the environment. Creative agency Assembly led the project with assistance from a team of musicians, producers, designers, and engineers.
The track was written by Joel Little (Kids of 88, Goodnight Nurse), played by hip-hop/tap dancer Josh Cesan, and documented by Thick as Thieves.
The instrument itself is created by the two Kinect cameras. Using a program called BitKinect, a computer can match motion to certain triggers in audio program Ableton Live. Originally the setup contained only one Kinect, but the data burden to create a virtual skeleton generated lag between motion and response. The solution was to create a dual Kinect set up where one Kinect fed OpenNI skeleton data to one computer, and the other Kinect fed raw depth data to a different computer. Each computer took this data and performed different roles. One machine handled audio-processing on Windows, while the other, a Mac, generated the visuals. The music system sends skeleton position to the visuals computer which then interprets what’s being triggered before sending a midi command back to the music system.
The user is then able to use specific movements to trigger certain effects in Ableton. “Vox” and “Bass” are both virtual keyboards that surround the performer in a circle. Whenever the silhouette of the performer crosses into a different keyboard region, a different sound is triggered. Low Frequency Oscillation (that famous dubstep wobble) is controlled by the distance of the performers hand to the ground. “Dough” uses multiple filters, one controlled by the distance between the performer’s hands and the other by the rotation of the ball he is creating. During the build up, the performer creates a triangle between his hands and the ground. The higher he goes, the louder it sounds, and as he opens his hands it increases the dampen filter.
The visuals system generates four different layers of imaging: the landscape, machine, green man, and interface. The landscape is mostly a pre-recorded synchronized file, but all elements of the machine, green man, and interface respond in real time. When the performer plays notes on the virtual keyboard, they are recorded on an orbital shell around the user and a visual sprite is triggered. These layers are then blended together and sent to three 20K projectors, two of which stitch the visual together on a 100 ft by 40 ft wall.
Two different computer systems, working in perfect harmony, in real-time, dropping the bass.