Engineering augmented reality

Following the debate about Confederate statues and monuments in the US during August 2017, the radical Catholic priest Fr Bob Maguire tweeted, ‘Could we not have Virtual statues which the algorithm could change as directed by public opinion?’

I like this Tweet a lot. Fr Bob makes an incisive observation about the logic and politics of augmented reality – at least as its imagined by the major media platforms. Platforms like Facebook and Google are investing in virtual, augmented and mixed reality technologies. And, as with most of their engineering projects, encoded into these technologies is a disruptive vision for public life.

Fr Bob cheekily skewers this Silicon Valley logic in a bunch of ways.

He’s aping the Silicon Valley liberal-individualist solution to everything. Forget the difficult debate about history and identity that surrounds these monuments, just measure public opinion and produce a representation of reality that matches that opinion. Forget being caught in history, just have a culture that continuously and automatically remodels itself on whatever the current tastes and preferences of the crowd are.

But, there’s another way to read Fr Bob’s quip. I think, that in the vision of augmented reality being imagined by Google and Facebook, the ideal scenario would be that we all individually wear our augmented reality lenses and see the reality we want to see.

As long as we all have our Facebook goggles or Google lenses in, when we go into the park and look at a big statue we will see our own personal hero. White Nationalists will see Robert E. Lee, progressives will look at the same spot, and see someone else – Oprah, Obama, Martin Luther-King, Tina Fey eating cake.

The point is this, augmented reality – as envisioned by Facebook and Google – is the engineering effort to take the forms of algorithmic culture currently confined to the feeds of our smartphones and transpose them into the real world. If at the moment, when we scroll Facebook we see the news that matches our political viewpoints. If we’re alt-right, we’re immersed in ‘fake news’ conspiracies about violent leftists, if we’re progressive we’re immersed in outrage about Nazis and the KKK. Augmented reality would weave those simulations into the real world.

So, our public space begins to reflect back to us our political identities.

Is that what we want?

Here we encounter a dilemma. On the one hand if we all saw the statue we wanted to see, would that mean everyone would be happy? Or, would it simply mask the real divisions which the debate over the monuments stands in for? Or, does the presence – or absence – of statues and monuments we disagree with in public space function as an important and constitutive aspect of public life? That a foundational characteristic of public life is to encounter and contend with ideas and people we disagree with, that are other or alien to us?

This is my provocation: we need to see the present effort to engineer virtual, augmented and mixed reality by Facebook, Google and Snapchat as an extension of the simulation-based, predictive and algorithmic culture they have been constructing over the past decade.

We can roughly sketch the history of virtual and augmented reality has three periods.

From the 1960s to the 1980s the US military investment invest in the development of virtual environments and simulators that could train pilots.

From the 1980s through to the mid-1990s dreams of virtual reality move beyond the military, Silicon Valley tech-utopian developers, counter-cultural activists and artists begin to imagine virtual realities unhooked from the impediments of the material world and its flesh and steel.
From the mid-90s virtual reality technologies, and the dreams about them, went into a kind of hibernation.

This hibernation came about because the dreams of a utopian and independent virtual world or cyberspace couldn’t be technically or politically realised. In a technical sense, low-res displays, latency, motion sickness, large and heavy hardware, lack of wireless connections, no mobile internet, and a lack of interplay with social life and urban space all stalled virtual reality start ups. Then, over the past five years firms like Oculus Rift and Magic Leap, acquired by Facebook and Google respectively, have been ushering in a new era of virtual reality hype. In the present moment there are three kinds of projects: virtual reality, augmented reality and mixed reality.

Virtual reality is characterised by opaque goggles. Once you are wearing them, you are in an immersive virtual world. Think of virtual reality gaming. Augmented and mixed reality are characterised by translucent screens or glasses. As you wear them digital simulations are overlaid with your view of the real world. Augmented reality is most evident in our everyday use of Snapchat lenses or filters. Via the screen we see our face overlaid with digital simulations: whiskers, a tiara, a rainbow tongue. Mixed reality is the prototyped ambition of Google’s Magic Leap. The limitation of augmented reality is that digital simulations are simply overlaid the vision of the real world, the simulations can not be made to appear like they are interacting with the world.

Magic Leap are working toward building a mixed reality technology where simulations will appear to be able to interact with the world. For example, you’ll hold out your hand and a simulation of an elephant will walk around your palm. It will appear to know where your hand begins and ends. The comparison between Magic Leap and Snapchat is a useful one. Magic Leap promote a vision of mixed reality that seems to be just out of reach. Incredible. But, in the future. Snapchat, while not as technologically-sophisticated, is perhaps more culturally significant. With Snapchat, augmented reality is becoming a part of everyday communication rituals. And, Snapchat are figuring out how to monetise augmented reality by selling it to brands. The major investments by Facebook, Google and Snapchat in these technologies indicate to us how serious they are in transforming their core platform architecture, pushing it beyond the smartphone and its flows of images on an opaque screen.

Media platforms like Google and Facebook are multi-dimensional engineering projects. Facebook’s Chief Operating Officer Sheryl Sandberg explained at Recode in 2016 that while the current business plan focussed on monetisation and optimisation of the existing platform. Their ten year strategic plan is focussed on ‘core technology investments’ that will transform the platform infrastructure. The developments keep coming. In August 2017, Facebook lodged a patent in the US for augmented reality glasses that could be used in a virtual reality, augmented reality or mixed reality system. Via translucent glasses or lenses, we can begin to see how Facebook could be transition to an augmented reality platform.

Here’s the critical point. These media platforms and partnering brands are not investing in the creation of more sophisticated mechanisms of symbolic persuasion. They are investing in the design of devices and infrastructure that can track and respond to users and their bodies in expanding logistical and sensory registers. Virtual reality projects are one instance of this, the effort to create a form of media that works not by creating symbols but by engineering experience. These companies are attempting to, as Jeremy Packer puts it, ‘code the human into the apparatus’.

Facebook, Google, Apple, Amazon, Microsoft, Sony and Samsung all have major investments in artificial reality. Facebook has 400 AR engineers. Silicon Valley has about 230 hardware and software engineering companies working on VR. Mark Zuckerberg echoes Silicon Valley consensus when he says it is ‘pretty clear’ that soon we will have glasses or contact lenses that augment our view of reality. Media platforms will augment human vision with digital simulations. Imagine looking at a room full of people and seeing their names above their heads, or a reading of their mood or level of interest in what you are saying. If you’re in class, your lecturer or tutor might be able to see the grade of your latest assessment floating above your head, or a colour coding that indicates your level of engagement in the course based on your attendance at class, logins to the learning platform, and grades. The data is available to do this: your university knows your attendance, grades and engagement with software, Google and Facebook can recognise your face.

Augmented reality heralds a shift from media that engineer flows of information to media that engineer experience. The value of mixed or virtual reality firms like Oculus Rift and Magic Leap is attributable in part to their claimed capacity to ‘hack’ or ‘simulate’ the human visual cortex directly. The ‘vomit problem’ or ‘motion sickness’ caused by VR devices is a container term for a number of points of ‘friction’ between the living body and the media device. The latency of the image on the screen inches from your eyes causes a conflict between your visual and vestibular system and you vomit. This problem has also been called ‘simulator sickness’, a term that had a particular currency in the 1980s and 1990s with military training simulators. Military researchers found that motion sickness from VR subsides in experienced users. An indication of the capacity of the living body to learn to ‘hack around’ the visual-vestibular conflict, to accommodate itself – in neurological ways – to the media device it is entangled with.

The VR hype-industry is characterised by plenty of claims to hack the body, or if not hacking then working around, reorienting, calibrating, or tricking it. Kevin Kelly explains that artificial reality ‘hacks the human brain’ to create a ‘chain of persuasion’. The term a ‘chain of persuasion’ – common in VR development – strikes me as an augmented kind of ideological control. Not persuading the subject only via a symbolic account of reality they interpret, but engineering an experience where the body feels present in a particular reality as a pre-cursor to them finding representations persuasive. AR’s account is persuasive not because the human subject ‘makes sense’ of it, but because it affects both the body’s biological system and the subject’s cultural repertoire in a way that feels real.

Magic Leap’s founder Rony Abovitz puts it this way:

VR is the most advanced technology in the world where humans are still an integral part of the hardware. To function properly, VR and MR must use biological circuits as well as silicon chips. The sense of presence you feel in these headsets is created not by the screen but by your neurology… artificial reality is a symbiont technology, part machine, part flesh.

The political economy of these media engineering projects is something like this: where the profits of broadcast media – their fabled ‘rivers of gold’ – were invested in quality content, the profits of media platforms like Google and Facebook are invested in engineering projects.
The vomit problem then is a metaphor for the creative experimentation happening at the ‘touchpoint’ between living bodies and media infrastructure.

We might ask then:  how will the ‘experience’ and ‘presence’ of mixed reality will be monetized? Google dramatized some of these applications when they were experimenting with Glass. As we look down a city street icons will appear above buildings the media platform predicts we might be attracted to because they sell our favourite beer or coffee, have good reviews, have a product it knows we are looking for, or that our friend is in there.

Or, perhaps stranger, a platform like Tinder, knowing our preferences for particular kinds of bodies, might be able sort and rank clubs in a nightlife precinct relative to our cultural tastes and sexual desires. You walk down a street with an AR device on, it registers affective and physiological responses to people who walk by you. It scans those people: their bodies, faces, clothes and associates them with a register of cultural and consumer tastes. And, then uses that to incrementally direct your paths through a city, a media platform, a market.

The critique of the political economy of social media has focused mostly on the capacity of platforms to conduct surveillance and target advertisements. But, as Jeremy Packer puts it advertisers now ‘experiment with reality’, engineer systems that configure cultural life by collecting, storing and processing data, rather than with ideological narratives.

As the smartphone and its modes of judgment, curation and coding give way to a mixed reality headset, the productive labour of the user will take on new dimensions. The embodied work of tuning the interface between body and lens. The combined neurological and cultural activity of adjusting how we experience reality: from a clear distinction between reality and digital image, to being immersed in a mixed simulation. From persuasion only at the symbolic level, to persuasion also at the affective and biological level.

But also, the work of tuning the predictive simulations of mixed reality via sensory and behavioural feedback. When I look down a street and it makes judgments about where I might want to go, my behavioural, physiological or affective responses to those predictions will inform future classifications and predictions as much as any symbolic content I generate. Here my bodily reactions feed not just the optimisation of a flow of symbols, but the tuning of a calculative device and platform into my lived experience. And in doing so, enable media to engineer logics of control beyond the symbolic: to the affective and logistical.

For all the work audiences did watching television in the twentieth century; that work didn’t change the medium or infrastructure of television itself all that much. But, I think we are moving into an era where the human user is an active contributor to the engineering of media infrastructure itself. And, a critical account of audience exploitation and alienation needs to engage with that.

The ‘vomit problem’ is a useful way of thinking about not just the work of engineers, but also of users who harmonise their lives and bodies with the calculations of media. The engineer works to solve the vomit problem via the ongoing, strategic design of software and hardware. As Packer puts it, media engineering involves strategically addressing problems to optimise the human-technical relationship. The user works to solve the vomit problem too: adapting their bodily physiology, appearance and performances as they move about the world; and, providing embodied feedback via their physiological and affective responses. Here the ordinary user undertakes the productive work of rolling media infrastructure into the material world, onto the living body and through lived experience.