How Augmented Reality Works


Virtual reality headset
Visitors try a virtual reality head-mounted headset Microsoft HoloLens during the Viva Technology show on May 25, 2018 in Paris, France. Viva Technology brings together 5,000 startups with top investors who shape the future of the internet. Chesnot/Getty Images

With each step forward in the digital revolution, "The Matrix" becomes less like fiction and more like reality. That's in part because hardware engineers and software developers continue to refine their augmented reality technologies, making the line between real and virtual life ever blurrier, no matter how much Keanu Reeves squints in an effort to differentiate the two. Augmented reality (AR), it seems, may soon be the true reality for us all.

Augmented reality is the blending of interactive digital elements – like dazzling visual overlays, buzzy haptic feedback, or other sensory projections – into our real-world environments. If you experienced the hubbub of Pokemon Go, you witnessed augmented reality in action. This (once incredibly popular) mobile game allowed users to view the world around them through their smartphone cameras while projecting game items, including onscreen icons, score, and ever-elusive Pokemon creatures, as overlays that made them seem as if those items were right in your real-life neighborhood. The game's design was so immersive that it sent millions of kids and adults alike walking (and absentmindedly stumbling) through their real-world backyards in search of virtual prizes.

Google SkyMap is another well-known AR app. It overlays information about constellations, planets and more as you point the camera of your smartphone or tablet toward the heavens. Wikitude is an app that looks up information about a landmark or object by your simply pointing at it using your smartphone's camera. Need help visualizing new furniture in your living room? The IKEA Place app will provide an overlay of a new couch for that space before you buy it so that you can make sure it fits [source: Marr]

But AR is more than just smartphone fun. It's a technology that finds uses in more serious matters, from business to warfare to medicine.

The U.S. Army, for example, uses AR tools to create digitally enhanced training missions for soldiers. It's become such a prevalent concept that the army's given one program an official name, Synthetic Training Environment, or STE. Wearable AR glasses and headsets may well help futuristic armies process data overload at incredible speeds, helping commanders make better battlefield decisions on the fly. There are fascinating business benefits, too. The Gatwick passenger app, for example helps travelers navigate the insanity of a packed airport using its AR app.

The possibilities of AR tech are limitless. The only uncertainty is how smoothly, and quickly, developers will integrate these capabilities into devices that we'll use on a daily basis.

Augmenting Our World

Pranav Mistry
Pranav Mistry demonstrates the novel camera of the SixthSense project where the user can take the picture by using the framing gesture. Pranav Mistry/Wikimedia Commons

The basic idea of augmented reality is to superimpose graphics, audio and other sensory enhancements over a real-world environment in real time. Sounds pretty simple. Besides, haven't television networks been doing that with graphics for decades? However, augmented reality is more advanced than any technology you've seen in television broadcasts, although some new TV effects come close, such as RACEf/x and the super-imposed first down line on televised U.S. football games, both created by Sportvision. But these systems display graphics for only one point of view. Next-generation augmented-reality systems will display graphics for each viewer's perspective.

Some of the most exciting augmented-reality work began taking place in research labs at universities around the world. In February 2009, technophiles at the TED conference were all atwitter because Pattie Maes and Pranav Mistry presented a groundbreaking augmented-reality system, which they developed as part of MIT Media Lab's Fluid Interfaces Group. They called it SixthSense, and although the project is stalled, it's a good overview of how you'll find basic components that are found in many augmented reality systems:

Mistry demonstrates SixthSense Mistry demonstrates SixthSense
Mistry demonstrates SixthSense
Photo courtesy Sam Ogden, Pranav Mistry, MIT Media Lab

These components were strung together in a lanyard-like apparatus that the user wore around his neck. The user also wore four colored caps on the fingers, and these caps were used to manipulate the images that the projector emitted [source: TED 2009].

SixthSense was remarkable because it used these simple, off-the-shelf components that cost around $350. It was also notable because the projector essentially turned any surface into an interactive screen. Essentially, the device worked by using the camera and mirror to examine the surrounding world, feeding that image to the phone (which processed the image, gathered GPS coordinates and pulled data from the Internet), and then projected information from the projector onto the surface in front of the user, whether a wrist, a wall, or even a person. Because the user was wearing the camera on his chest, SixthSense augmented whatever he looked at; for example, if he picked up a can of soup in a grocery store, SixthSense found and projected onto the soup information about its ingredients, price, nutritional value — even customer reviews.

By using his capped fingers — Pattie Maes said even fingers with different colors of nail polish would work — a user could perform actions on the projected information, which were then picked up by the camera and processed by the phone. If he wanted to know more about that can of soup than was projected on it, he could use his fingers to interact with the projected image and learn about, say, competing brands. Sadly, the SixthSense project went into a years-long hiatus and will probably never reach markets. But there are many other products stepping into the AR fray [source: Vulcan Post].

Augmented Reality on Smartphones

Pokemon Go
A tourist plays Nintendo Co.'s Pokemon Go augmented-reality game at the Trocadero in front of the Eiffel tower on Sept. 8, 2016 in Paris. The Pokemon GO game allows players to hunt on their smartphone or tablet virtual creatures scattered in public spaces Chesnot/Getty Images

In the Netherlands, cell phone owners can download an application called Layar that uses the phone's camera and GPS capabilities to gather information about the surrounding area. Layar then shows information about restaurants or other sites in the area, overlaying this information on the phone's screen. You can even point the phone at a building, and Layar will tell you if any companies in that building are hiring, or it might be able to find photos of the building on Flickr or to locate its history on Wikipedia [source: Ensha]

Layar isn't the only application of its type. In October 2018, an organization called Mural Arts Philadelphia created a gigantic interactive outdoors mural. Viewers pointed their smartphones at parts of the mural and then viewed various holograms and listened to matching music for a fully immersive art experience. And in a science-fiction dream come true, fans of the movies can now play "Star Wars" Holochess right on their phones, complete with futuristic graphics and sound [sources: Dickinson, Matney].

Health care professionals may soon rely on smartphone-enabled AR. A company called Tissue Analytics is honing an app that helps doctors and nurses use their phones to quickly identify specific types of wounds for faster diagnosis and more efficient care [source: Comstock].

An app that's simply called Augment will project just about any new product into a real-world environment. Whether it's a new recliner or a lamp, potential buyers can "see" the product in their own homes and read all information and reviews before making a purchase. AR Compass Map 3D is like a mapping app on steroids. It combines compass and map overlays with your camera to create a fully-immersive 3D map that guides you wherever you want to go [source: NewGenApps].

Similarly, a company called Total Immersion makes a wide variety of applications for business and fun. Want to know what those new glasses frames will look like perched on your face before you buy them online? Use the app to slide those frames onto your virtual face and you'll instantly know that horned rims really are not your style [source: Total Immersion].

Then there are apps like the previously mentioned Pokemon Go, a game that was insanely popular in 2016, and allowed players to hunt on their smartphone or tablet virtual creatures scattered in public spaces.

Augmented Reality in the Military

assault rifle controller video game
Jakub Ginda uses a U.S. Army Elite Force assault rifle controller to play a video game at the CTA Digital booth at the 2012 Consumer Electronics Show in Las Vegas. A lot of military applications have been found for first-person shooter video games. Ethan Miller/Getty Images

Militaries were some of the early adopters of gaming technology, seeing the possibilities for training soldiers for warfare realistically but in safe settings. And militaries are likely to do the same with AR.

Soldiers will plunge into an evermore immersive battlefield environment accentuated with helmet-mounted displays, smart glasses and much more. It's a market that may be worth $1.4 billion in 2018 [source: VisionGain].

Arcane Technologies, a Canadian company, has sold augmented-reality devices to the U.S. military. The company produces a head-mounted display — the sort of device that was supposed to bring us virtual reality — that superimposes information on your world. Consider a squad of soldiers in Afghanistan, performing reconnaissance on an opposition hideout. An AR-enabled head-mounted display could overlay blueprints or a view from a satellite or overheard drone directly onto the soldiers' field of vision.

And in an odd bit of fiction becoming reality, a lot of these military applications are just now becoming possible, in part because of the amazingly advanced physics calculations and programming made possible by ... the first-person-shooter gaming industry [source: Nichols].

Now that we've established some of the many current and burgeoning uses of augmented reality, let's take a look at the technology's limitations and what the future holds.

The Mobile AR Revolution

We could go on and on with examples of smartphone-driven AR apps, but these programs come and fade rather quickly. It's perhaps more notable to point out that manufacturers are so certain of AR's success that they're actually building devices meant to accentuate the AR experience. ASUS, for example, made the Zenfone AR in hopes of making an early splash in the AR-specific market [source: ASUS].

Two powerful tech leaders – Apple and Google – continue to tweak their mobile devices to handle the demands of AR-specific software. With ever-faster processors for the iPhone, iPad, and the entire galaxy of Android-driven smartphones, these pocket-sized computers are now powerful enough to run data-intensive apps of all kinds, including those that feature AR. Paired with the inevitable rollout of faster 5G data networks, those devices will be able to send and receive mind-boggling amounts of data that will make AR faster and better than ever before, and it will work whether you're in an office building or rolling down a country highway [source: Tardiloli].

You can bet that social media networks will try to capitalize on the coming AR trends. Facebook has its own AR Studio, which helps developers create AR apps meant specifically to work within Facebook's framework, and it is working on AR glasses, too. In the meantime, Google is touting its Tango AR platform, which sports visual search capabilities through Google Lens, a range of camera-enabled AR tools. Apple is in the middle of the race, too, with an ARKit that arms programmers with the code they need to dive into AR app creation for iPhones and other iOS-driven devices [source: Kahney].

The start-stop-start history of the famous Google Glass project might be a fitting way to sum up the nature of AR these days. In 2013, Google unveiled its AR-capable Google Glasses, which essentially provided an AR-type heads-up display of the world around the wearer. But the project lost momentum and stalled in 2015, only to find new hope in 2017, as the company resurrected the Glasses for business purposes. And in 2018, a company called Brain Power began selling the Google Glass as part of a program to help people with autism, improving their social skills and rewarding them for slowly but surely learning to interact in positive ways with the world around them [source: Kronk].

Limitations and the Future of Augmented Reality

Elton John
London commuters transport back to 1970s LA with the Elton John augmented reality experience at Kings Cross Station on Jan. 23, 2018. The installation has a self-playing piano and a screen that shows a virtual Elton playing when looked through. Dave J Hogan/Dave J Hogan/Getty Images for Rocket Music)

Augmented reality still has some challenges to overcome. For instance, people may not want to rely on their smartphones, which often have small screens on which to superimpose information. For that reason, wearable devices like augmented-reality capable contact lenses and glasses will provide users with more convenient, expansive views of the world around them. Screen real estate will no longer be an issue. In the near future, you may be able to play a real-time strategy game on your computer, or you can invite a friend over, put on your AR glasses, and play on the tabletop in front of you.

There is such a thing as too much information. Just as smartphone and internet addictions are concerns, an overreliance on augmented reality could mean that people are missing out on what's right in front of them. Some people may prefer to use their AR iPhone applications rather than an experienced tour guide, even though a tour guide may be able to offer a level of interaction, an experience and a personal touch unavailable in a computer program. And there are times when a real plaque on a building is preferable to a virtual one, which would be accessible only by people with certain technologies.

There are also privacy concerns. Image-recognition software coupled with AR will, quite soon, allow us to point our phones at people, even strangers, and instantly see information from their Facebook, Twitter, Amazon, LinkedIn or other online profiles. With most of these services people willingly put information about themselves online, but it may be an unwelcome shock to meet someone, only to have him instantly know so much about your life and background.

Despite these concerns, imagine the possibilities: You may learn things about the city you've lived in for years just by pointing your AR-enabled phone at a nearby park or building. If you work in construction, you can save on materials by using virtual markers to designate where a beam should go or which structural support to inspect. Paleontologists working in shifts to assemble a dinosaur skeleton could leave virtual "notes" to team members on the bones themselves, artists could produce virtual graffiti and doctors could overlay a digital image of a patient's X-rays onto a mannequin for added realism.

And we'll continue to see a mix of AR-enabled applications. For instance, companies will undoubtedly continue to unleash basic tools like an AR toothbrushing game from Dixie Cups, meant to teach kids basic brushing skills. And in the same breath, manufacturers, research facilities and more will find new ways to make people more and more productive in the face of an aging populace. All of these tools will help nudge the acceptance and capabilities of AR a bit further along this uncharted technological path [sources: Odell, Abraham and Annunziata].

Over the next few years, we'll likely see jumps in the evolution of the AR concept, in terms of both software, hardware, and a plethora of new applications. You might say that the future of augmented reality is so bright, you're going to need shades – and AR-glasses, too.

Last editorial update on Nov 9, 2018 11:28:39 am.

Related HowStuffWorks Articles

More Great Links

Sources

  • Abraham, Magid and Annunziata, Marco. "Augmented Reality is Already Improving Worker Performance." Harvard Business Review. March 13, 2017. https://hbr.org/2017/03/augmented-reality-is-already-improving-worker-performance
  • ASUS. "Zenfone AR." https://www.asus.com/us/Phone/ZenFone-AR-ZS571KL/
  • Berlin, Leslie. "Kicking Reality Up a Notch." New York Times. July 11, 2009. http://www.nytimes.com/2009/07/12/business/12proto.html
  • Comstock, Jonah. "Wound Care Advantage Taps Tissue Analytics for AI, Smartphone Offering." Mobihealthnews. Nov. 5, 2018. https://www.mobihealthnews.com/content/wound-care-advantage-taps-tissue-analytics-ai-smartphone-offering
  • Constine, Josh. "Facebook Confirms it's Building Augmented Reality Glasses." TechCrunch. Oct. 2018. https://techcrunch.com/2018/10/24/facebook-ar-headset/
  • Dickinson, Grace. "Mural Arts Philadelphia Debuts its First Augmented Reality Mural, 'Dreams, Diaspora, and Destiny.'" The Inquirer. http://www2.philly.com/philly/blogs/things_to_do/mural-arts-philadelphia-augmented-reality-mural-king-britt-20181010.html
  • Ensha, Azadeh. "Another Augmented-Reality App for the iPhone." New York Times. Oct. 15, 2009. http://gadgetwise.blogs.nytimes.com/2009/10/15/augmented-reality-apps-continute-to-roll-out/
  • Ganapati, Priya. "How it Works: Augmented Reality." Wired. Aug. 25, 2009. http://www.wired.com/gadgetlab/2009/08/total-immersion/
  • Inbar, Ori. "Top 10 reality demos that will revolutionize video games." Games Alfresco. March 3, 2008. http://gamesalfresco.com/2008/03/03/top-10-augmented-reality-demos-that-will-revolutionize-video-games/
  • Kronk, Henry. "The Google Glass-Based Autism Aid By Brain Power Has Gone to Market." Elearning Inside News. Aug. 7, 2018. https://news.elearninginside.com/the-google-glass-based-autism-aid-by-brain-power-has-gone-to-market/
  • Marr, Bernard. "9 Powerful Real-World Applications Of Augmented Reality (AR) Today." Forbes. July 30, 2018.
  • Matney, Lucas, "You Can Now Play Star Wars AR Holochess on Your iPhone." TechCrunch. 2018. https://techcrunch.com/2018/04/18/you-can-now-play-star-wars-ar-holochess-on-your-iphone/
  • Metz, Rachel. "Another use for your phone: 'augmented reality.'" Associated Press. Oct. 4, 2009. https://www.smh.com.au/technology/another-use-for-your-phone-augmented-reality-20091005-gi1m.html
  • NewGenApps. "8 Examples of Augmented Reality Apps and their Successful Uses." June 22, 2017. https://www.newgenapps.com/blog/augmented-reality-apps-ar-examples-success
  • Nichols, Greg. "Market rift: Military use of AR/VR skyrockets." ZDNet. Oct. 10, 2018. https://www.zdnet.com/article/military-use-of-arvr-skyrockets/
  • Odell, Patty. "Dixie Adds Augmented Reality Kids Game to Bathroom Cups." Chief Marketer. May 16, 2018. https://www.chiefmarketer.com/dixie-adds-augmented-reality-kids-game-bathroom-cups/
  • Parr, Ben. "Easter Egg: Yelp Is the iPhone's First Augmented Reality App." Mashable. Aug. 27, 2009. http://mashable.com/2009/08/27/yelp-augmented-reality
  • VisionGain. "'Global Military Augmented Reality (MAR) Market Worth $1.40 Billion in 2018' Says Visiongain Report.'" Oct. 25, 2018. https://globenewswire.com/news-release/2018/10/25/1626968/0/en/Global-Military-Augmented-Reality-MAR-Market-Worth-1-40-Billion-in-2018-Says-Visiongain-Report.html
  • Vulcan Post. "What Happened to Sixth Sense & Pranav Mistry?" Nov. 5, 2013. https://vulcanpost.com/1537/what-happened-to-sixth-sense-pranav-mistry/
  • Wortham, Jenna. "UrbanSpoon Makes It Easier to 'Scope' Out Restaurants." New York Times. Oct. 10, 2009. http://bits.blogs.nytimes.com/2009/10/14/urbanspoon-makes-it-easier-to-scope-out-restaurants/
  • "Pattie Maes and Pranav Mistry Demo SixthSense." TED Talks. March 2009. http://www.ted.com/talks/pattie_maes_demos_the_sixth_sense.html