Inside Out: Interaction Design for Augmented Reality
By Joe
Lamantia
Published: August 17, 2009
“With almost daily announcements about new AR
applications, products, services, companies, and tools, the pace of innovation
in augmented reality is torrid.”
Many people enter the inside-out world of augmented reality
(AR) by doing
something as ordinary as visiting a major city like New York and trying to get
to a local friend’s favorite pizza shop, somewhere deep in Brooklyn, via public
transportation. Standing in Times Square on a summer evening, they might hold up
a new smart phone and pan it slowly around the Square to see a pointer to the
nearest subway entrance overlaid on their phone’s video display of the buildings
around them.
While ubiquitous computing remains an unpleasant
mouthful of techno-babble to most people who know the term, and
everyware is still an essentially unknown idea, the visibility of augmented
reality
has surged in the last twelve months. In addition to the spate of mobile
applications—including Augmented ID, Wikitude, Layar, Nearest Tube, and the
still unreleased TwittARound—augmented reality is increasingly visible in
popular cross-media experiences. For example, Mattel is releasing new
toys
in conjunction with the James Cameron film Avatar that invoke online
content when users scan them with a Web cam, and LEGO in-store kiosks have used
augmented reality. With baseball cards from Topps and Pokemon cards, even the
venerable trading-cards experience now includes augmented reality.
With almost daily announcements about new AR applications,
products, services, companies, and tools, the pace of innovation in augmented
reality is torrid. CNN,
the Wall Street
Journal,
The
Guardian,
Venture
Beat,
and The New York
Times
have recently reported on augmented reality. In fact, so much buzz has come so
quickly, some journalists
and industry analysts think augmented
reality is over-hyped.
Reality Inside Out
“This blizzard of AR products offers a collection of
prototypes that can help us understand and refine the basic interaction models
and experience concepts that will underlay future generations of
everyware.”
Though we should expect this for any new, immature technology with such
potential for widespread impact and adoption, augmented reality is
clearly in the spotlight right now. Augmented reality deserves special attention
in this column, because it is one of the most successful ambassadors for
ubiquitous computing to date. As Vernor Vinge said in 2006, “Cyberspace has
leaked into the real world. ��what was inside the box in all eras up to ours ��is
outside.” [1] We can see this happening in smart-phone applications like Layar
and Wikitude, which make the intangible and conceptually challenging phenomenon
of everyware easily perceptible. They make it possible to experience the new
world of ubiquitous computing by reifying the digital layer that permeates our
inside-out world.
The role of experience design in regard to the inside-out world
of augmented reality is critical, because, as Vinge also pointed out, “Reality
can be whatever the software people choose to make it, and the people operating
in the outside, real world choose it to be.” [1] The UX community needs to find
ways to participate in and shape this design
probe
into the experience of everyware. To UX designers of all stripes, this
blizzard of AR products offers a collection of prototypes that can help us
understand and refine the basic interaction models and experience concepts that
will underlay future generations of everyware. UX professionals can offer an
essential perspective—as well as substantial history and a critical set of
methods and skills—for the creation of delightful, useful, and humane augmented
experiences, expanding their relevance and value. This opportunity is upon us
now and is ours to grasp—or miss!
Augmented Reality: A Thumbnail Sketch
“Throughout augmented reality’s evolution, continuous
technology advancement has rapidly changed the range of possible interaction
models and experience concepts.”
While researcher Tom Caudell coined the term augmented reality in
1992, the functional and experiential concept originated with the head-up
instrument displays and targeting devices airplane manufacturers created for
military pilots shortly after World War II. Since the 1970s and 1980s, many
distinct, but complementary domains of academic computing research and
commercial technological development have contributed to the evolution of
augmented reality. In the 1990s, the first AR products addressed the needs of
such specialties as medicine and engineering.
The convergence of mobile computing and wearable computing with
augmented reality is naturally of great interest to interaction designers who
are interested in the rise of everyware. Throughout augmented reality’s
evolution, continuous technology advancement has rapidly changed the range of
possible interaction models and experience concepts. As recently as 2003, for
example, the hardware for wearable AR experiences—like the game Human Pacman, built in
Singapore—was distressingly cumbersome, and the social, physical, and cognitive
limitations it placed on interactions were quite stringent. Contrast the
experience of wearing the heavy and unappealing equipment for Human Pacman with
that of wearing the powerful video eyewear now available from Vuzix and other wearable
computing devices just five years later, and the rate of change is easy to see.
(For more information and a historical review of mobile augmented reality, see
this “History of Mobile
Augmented Reality.”)
Current definitions of augmented reality vary. Wikipedia
describes it as “a field of computer research [that] deals with the combination
of real-world and computer-generated data (virtual reality), where computer
graphics objects are blended into real footage in real time.” [2] Robert Rice,
Chairman of the newly launched Augmented Reality Consortium
defines it more broadly, as follows:
“When I talk about AR, I try to expand the definition a
little bit. Usually, when you talk to someone about augmented reality, the first
thing that comes to mind is overlaying 3D graphics on a video stream. I think
though, that it should more properly be any media that is specific to your
location and the context of what you are doing (or want to do) ��augmenting or
enhancing your specific reality.” [3]
The most important difference between these definitions is their
frame of reference. Rice’s definition centers on the user experience, while the
traditional definition centers on the elements that make up the medium of
delivery. Rice’s definition also directly addresses locality and intentionality,
two of the most important contextual aspects of any human experience. Locality
and intentionality are critical factors in interaction design. I recommend UX
practitioners use his broader definition when creating augmented
experiences.
Remixing Reality
“Much of augmented reality’s unique character comes
from its combination of reified elements—usually bits of context that serve as
the augmentation—with a traditional focus such as an object, place, person, or
photo.”
Much of augmented reality’s unique character comes from its combination of
reified elements—usually bits of context that serve as the augmentation—with a
traditional focus such as an object, place, person, or photo. Today’s augmented
interactions often represent contextual elements within the augmented
experience—for example, using icons to indicate direction and the distance to
nearby points of interest—rather than being directly present—to continue the
same example, by showing live video of points of interest. The presence of these
representative reifications mixed in with reality is what makes augmented
reality inside out.
Designers should bear in mind that mixed reality is not
natural. Design always shapes a mixed-reality experience in some way.
Mixed Reality is a constructed
experience, requiring AR creators to answer two very important questions when
defining the particulars of any augmented experience. First, designers must
decide, How inside out should this reality be? Second, How should
this reality be inside out? The first question concerns degree; the second,
form. I’ll begin by answering the first question.
Designers can better define and understand the possible
combinations of real and virtual elements in user experiences within the context
of Paul Milgram’s Virtuality
Continuum, depicted in
Figure 1. As this model shows, augmented reality and augmented virtuality are
forms of Mixed Reality,
which according to Paul Milgram and Fumio Kishino, blends the extreme points of
fully real and fully virtual to some degree.
Figure 1—The Virtuality Continuum
Augmented-reality experiences supplement the real world with the
virtual world by connecting people to informational elements that are present,
but not otherwise accessible—assuming you don’t wear a lead helmet to keep out
those troubling transmissions. Augmented virtuality refers to
predominantly virtual spaces that dynamically integrate physical elements such
as objects or people into the virtual world, letting users interact with them in
real-time. [4] The holodeck from the Star Trek universe is a classic science
fiction example of augmented virtuality.
For designers, the most important thing to note about the
Virtuality Continuum is the absence of any clear boundary between the real
environment and the augmented reality. Now that reality is inside out, these
boundaries become blurry in many ways. As author Charlie Stross noted when
addressing the LOGIN conference:
“��we’re going to end up with the Internet smearing itself all
over the world around us, visible at first in glimpses through enchanted
windows, and then possibly through glasses, or contact lenses, with embedded
projection displays.” [5]
Augmented reality is just such a glimpse through an enchanted
window, and user experience creators are privileged to help make these windows
available to the rest of the world. This brings us to the second question AR
designers must answer: How should reality be inside out?
The User Experience of Augmented Reality
“In its current experimental form, augmented reality
inhabits the interzone between the defined cultural categories of technology,
art, commerce, utility, science, play, and entertainment.”
In its current experimental form, augmented reality inhabits the interzone
between the defined cultural categories of technology, art, commerce, utility,
science, play, and entertainment. In time, this will change, as augmented
reality moves from prototypes to finished concepts and reaches maturity in one
or more of these cultural categories. For example, the geohacking
William Gibson describes in his novel Spook Country shows augmented reality as
it might progress toward a recognized art form.
While we wait for cultural processes to sort out the long-term
place of augmented reality, reviewing some impressions of current AR efforts is
a good way to learn from the in-progress design probe.
First Impressions of AR Microsites from AD LAB
Two potential business applications for new AR technologies are marketing and
advertising. In May, the AD LAB blog posted a summary of author Ilya
Vedrashko’s reactions to several AR
advertising efforts
current at that time. His five main points, which follow, accurately capture the
problems with most of the augmented experiences for marketing, advertising, and
communications concepts.
- “The good news: the wow factor is undeniable. I had colleagues standing over
my shoulder commenting on how cool it was.
- “The bad news: it’s jaw-dropping only the first time you see it. After that
it quickly regresses from ‘neat’ to ‘meh’.
- “The usability of most implementations is pretty horrendous. You have to
hold up a letter-size piece of paper in front of your Web cam just so or the
whole animation goes away, and then you have to peek around it to see what’s
happening on the screen. I suspect it works better with smaller objects (I want
to try one of those Topps cards next), or with mobile devices.
- “Only Eminem’s AR animation was interactive beyond simple rotation—you have
to spray graffiti over it with your mouse. Not very easy either.
- “The biggest question I had was why. For all the trouble they make
you go through (download, print, and in case of Star Treck, install a plug-in),
you’d expect a somehow more rewarding payoff. From the ‘useful, usable,
desirable’ list, most implementations check off only ‘desirable’ for, like, the
first two minutes.” [6]
Vedrashko’s assessment is that these experiences remain skin-deep
at best, offering little, if any, sort of reason to pay attention beyond the
initial moment of interest.
Common Problems with AR Applications
Robert Rice—long-term industry advocate and CEO of Neogence, a company
building tools and other offerings in the AR technology space—identifies three
problems common to early-stage, browser-style AR applications such as Layar and
Wikitude, as follows:
“One is that they are all trying to get people to build new
applications for their browsers, when they should be trying to get people to
create content that they can share and browse.
“Second, someone using Layar is not going to see anything that is
designed for Sekai or Wikitude.
“Third the experiences are generally for one user.
While I love all of these guys and think each of the teams has some real talent
on it, the model is flawed until someone using Wikitude can see the same thing
that someone using Layar or Sekai camera is seeing (provided they are in the
same physical location).” [7]
Rice is primarily concerned about questions of
interoperability—on several levels—the predominance of single-person user
experience and interaction models, and the focus on the interface to augmented
reality rather than content. I agree with his assessment of the limitations of
the browser class of AR experiences, as well as his concerns about misplaced
focus, longer-term ecosystem growth, and the value of content.
As for the mainstream perspective on the myriad of AR
experiences, CNN observes, “People don’t necessarily want to walk around the
world holding cell-phone screens in front of their faces.” [8] CNN’s assessment
is overly reductive for an immature class of experiences. But it’s also exactly
the sort of summary judgment that identifies a critical area for improvement in
the overall value proposition for AR offerings.
Painting with a Limited Palette: Interaction Design Patterns
“The form an AR experience takes substantially
determines the set of possible interactions with and within the mixed
reality.”
Earlier, I observed that the answer to the second major question for AR
designers—How should this reality be inside out?—primarily concerns
form. In the same way that a design concept bridges two separate domains—the
problem domain and the solution domain—a mixed-reality experience bridges the
real and contextual domains. And just as a bridge’s form determines who and what
can travel across it, the form an AR experience takes substantially determines
the set of possible interactions with and within the mixed reality. The
interactions built into and around a mixed reality set the stage for a complete
user experience and thereby determine much of its potential value. The
criticisms of current AR experiences show they offer little value beyond their
material’s immediate novelty.
Many of the experiential problems AD LAB, Rice, and others
identify have roots in the very small set of primary interaction patterns that
provide the structure for a very large proportion of the recent wave of AR
offerings—by my informal inventory, just four—or perhaps five. This means
designers have a very limited set of options for framing the strategy,
structure, interaction, user interface, and other design aspects of an AR or
mixed-reality experience. To help designers better understand when to use these
patterns, I’ll describe each pattern and provide examples. Then, I’ll suggest
interactions the current set of patterns fails to address. Designers and AR
creators should see these gaps as opportunities!
Head-Up Display
“Augmented experiences using the Head-Up Display
pattern add information about the real objects in view into a complete
mixed-reality experience��.”
The Head-Up Display interaction pattern echoes the targeting and navigation
displays in military and other aircraft like that shown in Figure 2. This is the
oldest of the AR interaction patterns. Augmented experiences using the Head-Up
Display pattern add information about the real objects in view into a complete
mixed-reality experience that built-in AR tools and devices generate. While many
AR experiences rely on external devices—that is, external to the body—those
using the Head-Up Display pattern commonly depend on hardware that is integral
to a vehicle or cockpit-like physical setting.
Another very important attribute of the Head-Up Display pattern
is integration with a fixed point of view—typically the focus of the user’s
visual field. This pattern also applies to wearable systems such as the helmet-mounted
displays modern
infantries use.
The Terminator films and television franchise have used
this interaction pattern to great effect to communicate the machinic state of
mind of the cyborg Terminators.
Figure 2—Head-up display for a fighter plane
Examples of the Head-Up interaction pattern in augmented reality
include games—like ARhrrrr, shown in action in Figure 3, and Doom Resurrection
for the iPhone—and navigation systems for BMW and other cars, shown in the
videos in Figures 4 and 5.
Figure 3—ARhrrrr, an augmented-reality
shooter
2010/03/05 16:40
