Augmented Reality layers graphics and information over images of our real world in real time. Those yellow first down lines that show up on the field during NFL broadcasts: that’s Augmented Reality in action. But it’s also just the beginning.
The future might look like what’s demonstrated in this TED video of Sixth Sense, a wearable device that turns blends virtual computing with your every day life. Need to make a phone call? Dial the number on your palm. Want to take a picture? Hold up your fingers in the shape of a picture frame like a Hollywood director. (watch the video for even cooler uses)
Even Jimmy Fallon is into it
As I reported in my story today on Marketplace, the technology is being developed into futuristic iPhone applications, video games, and it’s being adopted by everyone from magazine publishers to toy makers to late night talk show hosts. A few weeks ago Augmented Reality made its television debut attacking Jimmy Fallon on his late night show (the attack begins with 1:45 left in this clip):
The father of Augmented Reality
But before Augmented Reality made big inroads into pop culture it was a little niche technology invented to help manufacturers wire airplane electronic systems. Tom Caudell helped pioneer the technology and coined the term in the early 80s.
I caught up with Caudell (photo right) for an interview to talk with him about how the technology has evolved over the past 20 years. Caudell doesn’t use Augmented Reality in his professional work anymore so we started the interview by playing a round of Sky Siege, an augmented reality video game for the iPhone that turns your natural surroundings into a war zone:
LISTEN TO THE INTERVIEW:
Marketplace Place’s Steve Henn: Alright has SkySiege downloaded?
Tom Caudell: It sure has. In fact I turned it on w few minutes ago Ride of the Valkeries came up, Wagner.
Henn: So why don’t you turn it on and turn he sound up and lets hear a little bit of Ride of the Valkeries that would be great.
Caudell: Okay. There is something called AR. There it is. I pushed that and I can see the radio station. Very nice. I am still looking down a gun of some sort and I can see our colleague. There is a blimp
Henn: And the blimp is flying through the radio station.
Caudell: Okay so I’m going to push play…
Sound: Explosions and Wagner
Caudell: I’m going to see if I can defend myself….laughter shooting…Wow. So this is a real important use of the technology.
Henn: Yeah, thirty years of research.
Caudell: Thirty years of research….
Henn: You can go ahead and turn it off….
Caudell: Okay, I was going to shoot the coffee maker first.
Henn: Go ahead. Shoot the coffee maker. But let me get a cup first.
Laughter
Henn: I understand you may have coined the term Augmented Reality. How did that come about?
Caudell: A friend and colleague of mine, David Mizell had been working on trying to get the Boeing Company to using virtual reality technology in its engineering, design and manufacturing processes. And they had gone over to digital design system for building the Boeing 777, so they had a lot of the drawings already in digital form.
So what we were trying to do is to see if we could convince them see if they could get into totally immersive virtual reality to do a lot of the manipulation and testing of manufacturability and things like that.
In during our discussion with a particular group that was doing the wiring harness part of an aircraft, they said that’s not their problem. They said their problem was to transform the digital information into something that people on the factory floor could use to assemble the wiring bundles and other parts.
So it occurred to me that perhaps if we could build head mounted displays like the virtual reality ones but if you could actually see through to the real world and using tracking technology actually superimpose information on the real world then that would really assist these folks, doing that.
So Dave Mizell and I started this other project – this Augmented Reality project that would allow workers to wear a small head mounted display of some kind that would allow them to see information overlaid on the real world and do the assembly. And that’s kind of where it started for us.
Henn: What would workers see when they would look through these head mounted displays?
Caudell: The wires were assembled on something called form boards. They were long thirty or forty foot long easels. They were four feet high and thirty feet long and they would have pegs on them drape the wired on it. Well our head mounted displays would project that path that a particular wire would have to be string through these pegs. Like weaving on a loom they would have thread these various wires through and this would project essentially a red line showing were between which pegs a particular wire would go.
Henn: What were some of the difficulties of making that work?
Caudell: Well this was out in a large factory assembly room. And the key to this is no only to wear a head mounted display that allows you to have the virtual information projected into your eye and allow you to see the world. But for the computer to know where you were looking required very accurate tracking information. And you really needed to know where you were looking quite precisely – where you head was basically pointed in order to know what peg you were looking at.
So the biggest challenge we had was tracking technology. How do you allow a person to move up and down a thirty foot long easel and keep tracking of their position and orientation of their head accurate enough for the computer to know where to draw the graphics. So that was the biggest challenge at that time.
Henn: Did you ever think about trying to get the computer to recognize the object. Or was that just too much of a challenge?
Caudell: No that’s something that others have worked on and were working on at the time – by putting fiducials things that would make it easy for the computer to recognize what the computer was looking at out into the world. And then use that information to computer your location. So it wouldn’t recognize a peg, but it might recognize the equivalent of a two dimensional bar code or something like that that was laid on the form board. But at the time we didn’t know about that.
Henn: So how did you track the tracking problem?
Caudell: We tried a number of methods. We tried the simplest one was for small range. It was an electromagnetic one. It actually emitted a structured electromagnetic field that the computer could recognize and calculate you position and things like that. Those worked really well in the lab where we did a lot of our work, but they were not very effective in the factory were other electrical items would be interfering with them and metal objects and that sort of thing.
So then we explored optical tracking where you could put something on the ceiling and put up detectors. Those worked pretty well – but again they had issues with lighting and that sort of thing. We even talked to NASA about using some of the technologies they were developing for deep space exploration, accelerometers technology it was. And it has some limitations as well.
So as far as I’m concerned out in the real world tracking – aside from these two dimensional icons – was a problem we never really solved. And was one of the reasons it never really took off.
Caudell left Boeing in the late eighties and went on to an academic career as a computer research. He’s spent the last twenty years trying to use virtual reality to model neural networks.
Henn: So you hadn’t thought about Augmented reality much, is that fair to say over the past 20 years.
Caudell: Almost none actually.
Henn: So in the last few years have you been surprised to see the term cropping up?
Caudell: I have. And it’s been quite a delight and also a curiosity. I’ve not really understood why until understanding that maybe its being driven by these new hand-held devices and games.
Henn: What do you make of how this technology seems to be expanding into the world? Did you ever imagine this would happen?
Caudell: Well I think some of us had discussed this sort of thing. The ideas we were thinking about were not for hand held devices. We were thinking augmented reality devices would be built into your sunglasses or your regular glasses in some in of very unobtrusive way. And other groups have worked on some of those things.
But [back then] our way of thinking about it was all based on this accurate registration of information sort of geometrically on world objects. So for example if you were going to have a tour of the Louver or something and you pointing your device at a gallery it would be very accurately outline a painting you wanted to see by drawing a box around it or something.
The free form, out in the world tracking is a very big problem and I believe there are still groups pursuing solutions to that. We were, or at least, I was personally a little bit limited in thinking about its applicability because of that.
But these hand held devices that your are talking about – particularly these games – really don’t require super accurate pointing. These iPhones and other hand held devices have GPS systems and compasses and tilt indicators that allow them to at least approximate the registration. So pointing in the general direction of a building might be enough to tell you that a it was the Louver for example.
There’s a lot happening in the world. Through it all, Marketplace is here for you.
You rely on Marketplace to break down the world’s events and tell you how it affects you in a fact-based, approachable way. We rely on your financial support to keep making that possible.
Your donation today powers the independent journalism that you rely on. For just $5/month, you can help sustain Marketplace so we can keep reporting on the things that matter to you.
