Astronaut Luca Parmitano using HoloLens during a complicated maintenance task in the Aquarius Reef Base, about 3 miles off the cost of Florida and 60 feet underwater.
Consider the challenge faced by an astronaut on the space station. They're living inside the most complicated machine ever built, traveling at about 5 miles per second, 250 miles above the surface. They're surrounded by equipment needed to keep them alive - all of which they have to maintain - and new scientific experiments are arriving all the time that need their attention. There's no way that any person could be expected to be an expert in all of these things - after all, there's a person back on Earth who spent years becoming the leading expert on just one of those things! Those experts do their best to train the astronauts before they launch, but there are so many things to learn that an astronaut often finds themselves struggling to recall a lesson on a piece of equipment from more than a year ago! When that happens, an astronaut has to rely on lengthy, often hard-to-follow procedures for a task. Procedures get the job done, but they're a slow way to work.
So, my team decided that a mixed reality device like the Microsoft HoloLens could be helpful here. We launched two of them to the space station in December 2015. We actually tried to launch two of them six months earlier, but that's another story...
Our idea is to use the HoloLens to connect astronauts with knowledge and expertise more effectively so that they can get more done onboard the space station. Initially, we'll be using Microsoft's new version of Skype for HoloLens, which will allow an expert on the ground to see exactly what the astronaut is doing and draw holographic annotations into the environment around the astronaut. So, instead of saying "on the lower panel, beneath the black handle, push the green button that's third from the right unless it's flashing", an expert can simply say "push this button" and draw a circle around the button. The astronaut simply sees a circle hovering around the button and knows exactly what to do. With a capability like this, an expert on Earth can be far more helpful to an astronaut in orbit.
We tested this out in the summer of 2015 at NEEMO XX, NASA's underwater mission simulation environment. A crew of "aquanauts" consisting mostly of astronauts lived in the Aquarius Reef Base for several weeks, testing out mission scenarios and technologies that might one day be used onboard the space station or during a mission to Mars. It went really well, and you can read more about it here.
Aquarius Director Roger Garcia leading Luca Parmitano through the maintenance task shown at the top of the page. Using Skype, Roger could talk to Luca as if on a normal videoconference, but could also draw annotations into Luca's world to make it clear exactly what he needed to do.
Our success at NEEMO was exciting, but the real deal came just a few months later when we successfully launched Sidekick to the International Space Station in December 2015 and then used it onboard in February 2016. Here's a video!
I think this is only the beginning. Mixed reality will allow us to naturally connect an astronaut with tons of information on the tasks they need to perform. She might be able to say "Where is the part that I need to fix this" and be led by a floating arrow right to the storage compartment where that part is stored. Animated holographic diagrams could be displayed on top of a partially built piece of experimental equipment showing exactly where the next piece should be installed. She could look right through a panel and see where a cable is routed that needs to be replaced.
Scott makes this look good.