Skip to main content

SimSensei: The Kinect-based virtual psychologist

Computer scientists at the University of Southern California have used Microsoft’s Kinect sensor to detect whether you are depressed with a 90 per cent accuracy rate. All you have to do is sit down in front of Kinect, answer some questions posed by an on-screen virtual psychologist, and the clever software does the rest. The process is entirely automated, objective, and self-contained, meaning accurate testing could be carried out in complete privacy at home.

The software, called SimSensei and developed by Stefan Scherer and colleagues, is essentially a clever mix of computer vision algorithms and the psychological model of depression. The on-screen psychologist asks you leading questions – a lot like the old-school Eliza, or Alice – and then watches how you physically respond.

Using Kinect, the computer vision algorithms build up a very detailed model of your face and body, including your “smile level,” horizontal gaze and vertical gaze, how wide open your eyes are, and whether you are leaning towards or away from the camera.

From these markers, SimSensei can work out whether you’re exhibiting signs that indicate depression – gaze aversion, smiling less, and fidgeting. Watch the video below and be amazed.

In testing, the University of Southern California researchers worked with a group of 60 people, half of which had been previously diagnosed with depression. SimSensei accurately diagnosed 90 per cent of the people with depression.

What’s most shocking about SimSensei is that it illustrates just how easy it is to diagnose depression. If all it takes is a low-res Kinect sensor and some fairly coarse computer vision signals to detect depression, it makes you wonder about how simple the human processes are for detecting depression in other humans. While we might dress it up as empathy or peering into someone’s soul, in actuality our brains might be doing nothing more than looking for gaze aversion and fidgetiness.

It also makes you wonder about how many depressed people go undiagnosed, or worse, wrongly diagnosed. Using an automated, objective system that takes the human element out of the equation would mean that depression could be picked up much sooner, and treated accordingly. It wouldn’t be hard for Microsoft to provide free, regular screening for all Xbox users. And why stop there? If depression can be diagnosed, why not other mood disorders such as bipolar?

By combining Kinect with other sensors, or using a different array of sensors entirely, it might even be possible for a computer to diagnose all sorts of mental and physical issues. Back in 2012, researchers showed that it was possible to detect Parkinson’s disease with a three minute phone call with a computer; and last week we heard about a sensor that goes under your skin and constantly checks your blood to see if you’re about to have a heart attack (see the video below). It is not completely crazy to think that next-generation consoles and smartphones, with the right attachments, will be capable of almost tricorder-like diagnostics.