The Next Big Tech Thing is virtual reality (VR) along with its technical sibling, augmented reality (AR). The New York Times, Google, Apple, Facebook and others are plunging in big time. Over $1 billion was invested in 2015 alone to gain a foothold in what is projected to be a $5 billion market.
But what does this mean for book publishers, who have seen Next Big Things–CDs, enhanced eBooks, subscription startups, self-publishing–come and go, or come and plateau? Indeed, what on earth could VR or AR have to do with books, especially printed books? Well, some brave publishers, big and small, are doing some interesting experiments. But before I talk about those, I’ll briefly discuss what VR and AR actually are.
The ne plus ultra of VR is a sensory environment indistinguishable from what we perceive as “reality” (which, of course, isn’t reality either, but that’s a color of a different color). The visual field will be in full color and 3D, and in a resolution indistinguishable from what you see in front of you right now. Sound will surround, and possess all the subtle spatial cues of “real” reality. That approaching dragon with the singed snout–and nauseating breath–will be way beyond anything Steven Spielberg has contrived so far. In some experiments you feel the blows of the dragon’s spiked tail on your armor. You, the protagonist, move through an utterly compelling visual and aural field.
You can see the storytelling potential for VR, at least for those with Hollywood budgets. Even if all of the properties above (3D visuals and sound, haptic feedback, smell-o-vision) are partial or imperfect, the faculty of suspension of disbelief–which has no problem with 2D movies, or for that matter, words on a page–will have some meaty material to process. The hard problem of VR is “agency”: giving protagonist-you volition in the virtual world, a problem which sounds to me like it lives at an interesting intersection of programming, philosophy, and, perhaps, AI.
Currently, VR requires wearing awkward headgear, which supplies the 3D visuals and audio and has hardware and software to sense position so that when you turn your head, the scene shifts as if you were turning to look in that direction. High-end VR headgear costs $600 and up, but clever headsets from Samsung, Sony and others harness the screen and computing power of your smartphone to supply the hard bits. These headsets are built to a standard form factor developed by Google called Cardboard. It’s called Cardboard because you can make a viewer from a cardboard kit: last November the New York Times distributed 1.2 million of these viewers to support its expanding line of VR reportage pieces. The smartphone fits into the goggles; magnifying lenses allow each eye to focus separately on the dual images required for 3D perception. If this reminds you of an old-fashioned stereoscope, or a View-Master with its cunning disk of tiny transparent pictures of 101 Dalmations, it should: this method of providing 3D goes back almost 180 years. View-Master now sells its own VR viewer built to the Google Cardboard standard. The content for VR viewers is developed with special cameras that record a 360-degree visual field. Most of the content I’ve seen for Google Cardboard is relatively low-res and crude: high-end VR requires not only high-end professional camera rigs, but computing power beyond that of current smartphones. High-end VR capabilities and prices will certainly come down over the next couple of years: the technology is still in its early development phases.
Augmented reality is synthetic reality superimposed on “real” reality: look down 9th Avenue and see 3-1/2 stars hovering over that Argentine restaurant, and 4 over the Italian one. Or in a real battlefield see the guy behind that stone wall waiting for you with his AK-47, imagery courtesy the silent drone overhead, downloaded to your VR headgear and re-rendered for proper perspective. The AR “look” has begun to infiltrate media: in the Sherlock series with Benedict Cumberbatch, text messages between characters are superimposed over a street scene.
Virtual Reality and Books
While VR proponents are using many of the same bogus–and scientifically unsubstantiated–arguments that eBook proponents use to argue that eBooks should replace printed textbooks, it seems like there are obviously beneficial use cases for VR in training and education. Bosch, a major auto parts manufacturer, has used VR to train service techs to install and service its products: for complex physical objects, VR provides learning perspectives unavailable in 2D media.
Many of us, of course, are less interested in required reading and more interested in publishing and reading books as entertainment or aesthetic experience. Here the prospects for VR or AR are, I would say, questionable. Previous attempts to attach other media with books have suffered from bicycles-for-fish syndrome. The ancillary media is either just that, or the sum of alternative media and text somehow wind up being less than the parts. So-called “enhanced” eBooks are the perfect example. If an author does their job with the text, adding more stuff in doesn’t really add anything significant to the experience for the reader. For a publisher, it’s just more sunk costs to a project that has only a 20% chance of earning out to begin with, and for which readers are unlikely to pay any premium.
Such combinatory experiments are not new, and are not limited to books. As an undergraduate, I worked on a grand project involving music, film and slides–the term of the day was “multimedia”–which was totally engrossing to produce, and a dog’s breakfast to consume. Only in museums do these experiments seem to thrive, where curators and art mavens appear to be satisfied with ideas rather than experiences (or perhaps haven’t put on the goggles which show that, in fact, the emperor has no clothes).
Perhaps these composite forms just haven’t found their promethean creative figure, or killer app. Could be. And VR will likely be its own standalone medium, with a syntax unlike its progenitors in film and video games.
Enough kvetching. Back to books. Here are three examples of books using AR.
Disney Coloring Book
Disney engineers have produced a coloring book/AR app combo. The kid colors in the elephant, points the iPad camera at the page, and the app creates a dancing elephant colored the same as the colors the kid crayoned in. The app doesn’t care if the page isn’t flat or a little wrinkled, it still comes up with an amazing dancing elephant in the right colors. This is a tour de force of graphic programming. It’s awkward though. The hand-eye act of coloring is a particular experience–as millions of adults have discovered. Picking up an iPad to watch software do some wizardry is a different experience. Yet another bicycle for fish.
Guinness Book of World Records
Compare yourself to the tallest man ever! That’s the promise of the 2015 Guinness Book of World Records. How? Download the free app and scan the page with the Augmented Reality Alert Icon! Puts you in the picture with the tall guy, apparently. Given the essence of the Guinness Book appeal, this might be fun.
Between Page and Screen
This one’s cool: show the book to the application, and words dance off the page in shape poems and typographic animations. The pages have only inscrutable glyphs, which trigger the application (which runs on a computer) to generate the words. Unlike the examples above, which add media to the book, Between Page and Screen makes the book a secret code which unlocks media living on the computing device.
In all three cases, the book is printed with “marks” that act as cues to the visual processing software to do something. These marks can be QR codes, but more sophisticated software uses the image of the actual page itself to cue the software process, or so-called invisible barcodes.
The Book Plus AR Experience
In each of the AR cases above, the software application and computer are interposed between the eye and the book, as it were. The experience is awkwardly triangular: eye + book + device. Note that were they to utilize VR as well as AR, incorporating the physical book within the VR field produced by an VR headset, much of this awkwardness disappears. Which leaves only the awkwardness of the VR headset itself.
VR: The Form Factor Race
Right now VR and AR gear is ridiculous. Just putting it on my head reminds me of some of my ninth-grade daughter’s science projects. There is even social stigma: while Google Glass wearers at Book Expo attracted many a curious glance, the term Google Glassholes acquired rapid currency. Like looking at a View-master disk or a stereoscope card, using VR headgear has a big initial wow factor. You say, “That’s really cool!” and then put it back in the box and put it under the bed with other boxes of unused gadgets. That’s what I did anyway.
But imagine a world where the gear is actually wearable. That’s where the famous optical manufacturer Zeiss is going. Zeiss has produced prototypes that look just like well-designed eyeglasses. A company called Innovega is working on contact lenses that provide AR capabilities. And Apple, who certainly knows how to ace new form factors, is clearly working on VR.
The way to think about VR and AR is not to think about dorky three-pound headsets, but to think about VR and AR being more accessible than that smartphone your head is bent over right now, because it is part of eyeglasses or contacts you wear all the time. (Sixty percent of the people in the developed world already wear eyeglasses or contacts.) Part of the “augmentation” the glasses could provide could be much more practical than entertainment: vision correction that automatically adapts to changes in your eyes, instead of requiring new lens prescriptions; night vision. Wearable devices—perhaps smart watch and smart glasses talking to each other and sharing computer power and sensors—will replace the smartphone.