What does the future look like? Maybe it's my love of science fiction or just a side-effect of spending my days surrounded by technology, but this is a question that occupies my mind quite frequently, in one form or another. And perhaps the best part of my job is that I sometimes get a glimpse at a new technology that answers this question.
The answer today is the Leap Motion Controller, a compact PC peripheral that brings motion sensing technology and gesture-based interaction to any laptop or desktop with a USB port. This little piece of tomorrow is available for Windows and Mac systems alike, and priced at £70 it won’t break the bank.
There's little wonder that 3D gesture-based interfaces are coming; similar concepts have popped up in films such as Minority Report and the Iron Man movies. It also makes sense in the logical progression of user interfaces – abstract text-based UI (using only a keyboard) gave way to 2D display-only graphical UIs (with the addition of the mouse), which have evolved into 2D touch-based UIs (with capacitive touchscreens), which have themselves developed from basic tap-to-click to multitouch displays with gesture support.
In the past, we've seen plenty of attempts to move interface technology forward. Plenty of PCs offer rudimentary hand-wave support with webcam-based gesture controls. We've also seen air mice, such as the Gyration Air Mouse Elite.
The next steps include non-touch gestures, which have already gotten some development with webcam-based gesture controls. Gaming has pushed this sort of interaction forward considerably with the Sony PlayStation Move and Razer Hydra, but perhaps the most prominent application of this sort of motion sensing tech is the Microsoft Kinect for Xbox 360.
The Leap Motion Controller is the next step in this evolution, letting you interact by gesturing in the air in front of the PC. Whether or not it will catch on is equally up in the air, but given the interest shown in the Leap Motion Controller, including deals to embed the device technology into coming HP and Asus PCs as early as this year, the Leap Motion Controller is an early glimpse of what the future will hold.
The Leap Motion Controller is a tiny thing, and it’s just 13mm thick, measuring 75 x 30 x 13mm (WxDxH). The device has an Apple-inflected minimalist design, with bare metal around the outside edge, a rubber sheet on the bottom for stable non-sliding grip, and a glossy piece of tinted glass on the top (the image above shows the top, bottom, and sides of the device).
The Leap Motion Controller connects via USB 3.0 (but it's USB 2.0 compatible as well), with both 24in and 60in connector cables bundled with the device. Inside the device, you can faintly see three glowing red spots, the infrared LED illuminators that let the sensors inside "see" your hands and gestures with a 150-degree field of view and eight cubic feet (two feet wide by two feet long by two feet high) of interactive space to gesture within.
Because the size of the interactive space is determined by the controller, and not tied to the dimensions of the display the way touchscreens are, the small controller can be used with any size display, from an 11in netbook to a 60in HDTV which is being used as a monitor.
The Leap Motion Controller doesn't actually do any real processing within the device; instead, it shifts all of that to the PC, relaying data directly from the sensors to the software. When using the Leap Motion Controller, there is only the tiniest discrepancy between action and response, but it's so slight as to be nearly lag-free.
Activating the Leap Motion Controller is a bit more complicated than plugging in a mouse, requiring a software download and install. Go to the setup page here, select the appropriate operating system (Windows or Mac), and then begin downloading. An installation wizard will walk you through the rest of the setup.
You'll then need to create an account in Airspace, Leap Motion's software environment for the controller and associated software. It just takes an email and a password, and you'll be in business.
Right away, you're prompted to go through an orientation demo, which helps you understand how the device works and how you interact with it. Open up Airspace Home, the software launcher for the Leap Motion Controller and the central location for all of your apps, and open up Orientation.
The orientation demo starts with a cloud of sparkles, sort of like a swarm of fireflies, indicating the sensor's observable field. As you wave your hands over the device, the sparkles start to move and change colour as you interact with them. The demo lets you play with the firefly cloud for a good while, getting used to waving your hands in the air and getting a feel for the size and shape of the interactive space above the sensor.
After a moment, the action shifts its angle in order to give you a chance to play with the depth of the sensor field. The Leap Motion Controller provides more than the 2D tracking we're accustomed to with touchscreens, and adds a third dimension to it. Your hand motions are tracked in a hemisphere of space emanating from the device, letting you gesture not just up, down and around, but forward and back.
The second portion of the orientation demo is more detailed hand tracking, displaying two glowing wireframe hands, letting you see just how accurate the device can be as it tracks not just your hands in space but your palms, wrist, and the individual tips of each finger. You also begin to see the limitations of the device, however, as moving one hand under the other causes the top hand to disappear – if one hand comes between your other hand and the device, the other will disappear. The sensor can't see through things. This also becomes an issue when you hold your hands at certain angles. Open your hands a certain way, and your fingers disappear.
The third part of orientation is a drawing exercise, which tracks a single fingertip to let you draw on the screen. Using either a single finger or a drawing implement, you can doodle around on the screen, squiggling to your heart's content.
Despite the physical necessity of the controller to enable your 3D hand-waving interaction, Leap Motion considers itself at least as much a software platform. Nowhere is this truer than in the Airspace Store, Leap Motion's dedicated app store. With both free and paid apps, available for both Windows and Mac (often both), there is a surprisingly hearty app selection already, with an estimated 80 apps available for launch at the end of this month.
Our reviewer's Airspace account came with several apps already, six pre-selected offerings hand-picked to show off the range of applications available for the device. Included in the app selection was Lotus, a psychedelic experience in hand waving that involves sitar music triggered by hand motion (and starts by poking a giant eyeball right in the pupil). There’s also Corel Painter Freestyle, a paint application; Deco Sketch, a freeform photo manipulation tool (but it's more Paint than Photoshop); Dropchord, which is a dubstep-inflected game; and finally Boom Ball, which is a 3D game that reimagines Breakout with a paddle and gloved hand to control the ball.
I also downloaded several more apps from the Airspace Store: Flocking (another 3D toy controlling swarms of pixels with your hands); Block 54 (a virtual Jenga tower); Gorogoa (an interactive story); CyberScience Motion (a digital skull that can be dissected and reassembled for learning anatomy); Escape Velocity, where you pilot a starship by tilting and panning your hand; Geco and AeroMIDI (two MIDI music generators that let you craft tunes by waving and gesturing) – and a few others in that same vein. With very few exceptions, these apps are all games and toy-like demos, demonstrating novel capabilities offered by the controller but not necessarily offering any productive capability.
The one exception to this was an app called Touchless for Windows (there's a Mac version as well), which lets you use the Leap Motion Controller in your regular Windows environment. This was especially important to find out whether the Leap Motion Controller would have any real impact on my regular computer use. And it did, definitely. It took my regular PC experience, which I've developed over the last 20+ years, and overlaid all of that with an interface defined largely by vagueness, inaccuracy, and tired arms. Switching between windows became a chore that took the better part of a minute. Getting to the Windows 8 Start Screen, which should be a better experience with the device given its touch and gesture-friendly design, took literally three minutes of frustrated hand waving and air poking.
While there are some very cool things being done with Airspace Apps, this is certainly early days for both the product and the accompanying apps. Developers and users have a way to go as they get their heads around what the Leap Motion Controller can be used for, how it can and should be integrated into their regular computer use, and how to grow beyond games and technical demos.
One of the included apps is Frog Dissection. Hoping for a flashback to sophomore biology, I decided to play frog surgeon. Due to inaccurate tracking, I got into the app, but couldn't accurately make a selection in the menu. Instead of taking a virtual scalpel to Kermit, I accidentally selected frog videos, managed to fast forward and skip through tadpole and early frog development, and then found myself watching frogs mating, but now unable to fast forward or skip. Thanks to the wackiness of the hand detection, I was suddenly stuck watching frog sex, with no way out. Needless to say, this isn't how I normally fill my work day. Only a retreat to the keyboard (and the aptly named Escape key) saved me.
A later attempt at Frog Dissection met with more success, but the underlying problem is that the sensor doesn't always seem to work as you would expect, and the interpretation of your various waves and gestures is different in every app. The folks at Leap Motion are hesitant to impose standards across all apps, not wanting to hem in developers when the product is still so new, but the result is a messy mix of things that work and things that don't.
That's just the app side of things. At one point, an alert told me that the device itself needed to be recalibrated. The recalibration process involves picking up the device, pointing its sensor at a reflective surface, such as the monitor, and then painting the screen with a dot that pops up. The more you paint, the higher your score, and you must reach a score of 80 points or better for the recalibration to take effect. That's right, even fixing the Leap Motion Controller feels like a poorly designed game – that I struggled to win.
Once I finally got my calibration score high enough to use the device, it began telling me that my plugged in, just-recalibrated device was unplugged. I double checked the USB cable at each end, and it was indeed plugged in. I unplugged and plugged it in again, just to be sure. It still said unplugged. Finally, using a mouse like a normal person, I managed to dig into a menu that allowed me to select "resume tracking," at which point the device registered as plugged in and began tracking again.
For any device, especially something unusual and breaking new ground, these individual glitches are understandable, even inevitable. But when there are so many, it becomes hard to overlook. As it stands right now, the Leap Motion Controller is a brilliant but flawed device paired with a messy ecosystem of apps.
As a proof-of-concept device the Leap Motion Controller is amazing. Like the Xbox Kinect, the Leap Motion Controller allows you to navigate with real gestures and natural motion, giving you glimpses of a time when PCs will be able to see us and understand what we want without having to learn a specialised interface or obscure commands.
The combination of hand and finger tracking is excellent, and there are plenty of applications where this sort of technology will have a big impact. The Leap Motion Controller is one of the coolest devices I've used, full-stop. This is clearly a step forward in interface technology, and it's jaw-droppingly cool.
But as a consumer product, the Leap Motion Controller is definitely still a work in progress. It's smart in certain applications – the orientation demo alone made me giddy with geeky excitement – but so far the practical uses are few and far between. Even assuming that the controller will be useful in certain circumstances, there are still some hurdles between where it is now and where it needs to be to become the polished, intuitive product that Leap Motion clearly wants it to be.
Secondly, while the tracking is quite accurate, it's difficult to be precise when trying to open and close windows, and shift between actions like scrolling and clicking. For regular navigation, this needs to either be fixed or the problem needs to be removed with a workaround, such as standardised gestures. There need to be at least a few standards applied across apps, letting users open and close programs, scroll and pan, and access menus in a consistent way. Right now it's piecemeal and unrefined, and it seriously diminishes the user experience.
Finally, waving your arms around takes some getting used to. To prevent the "Gorilla Arm" experience already familiar to touchscreen users, I set up the Leap Motion Controller so that I could gesture freely while resting my elbows on the armrests of my chair. It helped, but only so much. Like touch, motion control adds a lot of physical motion to functions that are normally done with a button click, and the transition can be both jarring and tiring. If there were better ways to integrate the keyboard and mouse into the Leap Motion Controller experience, several of these problems would be solved.
The Leap Motion Controller is, at present, a very fancy toy, with little practical application. That said, at £70 it's an affordable toy, and if you're curious about the technology, it's inexpensive enough for every enthusiast, tinkerer, hacker, and gadget-fiend to buy without worry. For the average consumer, however, this is not ready for primetime.