October 28, 2011 | By Bill Gates
The Power of the Natural User Interface
There’s a reason why the Kinect for Xbox 360 holds the record as the fastest-selling consumer electronics device ever. The innovative software underneath the hood is enabling millions of users worldwide, including Bill and his family, to interact with and immerse themselves in games in ways never before possible.
We play some of the Xbox 360 Kinect games in my house and it’s hard to convey just how amazing it is when you find yourself immersed right in the middle of a game, and how much fun it is for the whole family to play games together where we not only control the action in completely natural ways, but we actually are the action.
Kinect is a motion-sensing input device that’s a revolutionary new way to play games using your body and voice instead of a controller. Now, instead of playing a soccer game by mastering a series of button commands that have nothing to do with soccer, you play by using your feet, head, body, and, if you are the goalie, your hands. To play ping pong, you move your hand just the way you would if you were playing in the garage with a ping pong table, a ball, and a paddle.
Kinect is a remarkable technical achievement. The ability to take video cameras, multi-array microphones, and depth sensors and bring them all together in order to recognize people, understand and anticipate how they move, incorporate voice recognition, and insert them into games—all at 30 frames per second—is phenomenal. (It holds the Guinness World Record for the fastest-selling consumer electronics device ever, with 8 million units sold in the first 60 days it was on sale.)
A lot of the advances that make Kinect possible have been under development at Microsoft for years, even decades. For example, computer scientists at Microsoft Research have been working to solve the hard problems in speech recognition and video capture technology for many years.
Kinect is much more than just a cool video game technology—it is the most advanced example of a wave of new advances that are enabling people to interact with technology in entirely new ways.
One of the most important current trends in digital technology is the emergence of natural user interface, or NUI. We’ve had things like touch screen and voice recognition for a while now. But with Kinect, we are seeing the impact when people can interact with technology in the same ways that they interact with each other.
I’m convinced this is a transformational technology. Until now, we have always had to adapt to the limits of technology and conform the way we work with computers to a set of arbitrary conventions and procedures. With NUI, computing devices will adapt to our needs and preferences for the first time and humans will begin to use technology in whatever way is most comfortable and natural for us.
This has profound implications. At the most basic level, it means people will be able to use technology without any special knowledge or training. Advanced computing tools that can only be used by experts with highly specialized skills today will be available to anyone in the not-too-distant future.
Even more important, NUI means people who may not have even the most basic literacy skills will be able to take advantage of some of the benefits of digital technology and the information age for the first time. Kinect’s movement sensor and voice control technologies also have the potential to make computers more accessible to people with physical and cognitive disabilities. A surprising number of academic researchers and others are exploring using Kinect in ways we never imagined. In the UK, for example, scientists are developing robots using Kinect’s inexpensive (but sophisticated) motion-sensing technology to search for survivors in potentially unstable buildings after an earthquake. Researchers in Seattle are exploring how Kinect can give surgeons a “virtual” sense of touch during remote surgical procedures.
Earlier this year, Microsoft Research released a free software development kit for noncommercial applications of Kinect. If you go to the website, you can see dozens of examples of projects – some of them just for fun and others with a more serious purpose.
It’s exciting to think about the many ways a natural user interface will be used, including by people with little knowledge of technology, to tap into the power of computing and the Internet.