Eye Magazine

Steve Jobs once said that the computer is ‘a bicycle for our minds’, a tool that amplifies human imagination and agency. Such utopian sentiments about technology are becoming rare in 2019. More common are reflections on tech’s power to disempower, manipulate and oppress its users. If we want to align technology with humanity’s best interests, recognising potential dystopias is just the first stage. Designers have a specific role to play in shaping our future, and many practitioners have already started to recognise the unique responsibility they have in addressing these issues.

Many of today’s problems stem from Silicon Valley’s cultural norms. In 2011, the controversial entrepreneur and investor Marc Andreessen wrote that ‘Software is eating the world’. He observed that increasingly every sector of the economy was being run on software and delivered via online services. With an entirely digital economy as the inevitable destination, Andreessen anticipated a journey that would involve ‘invading’ Silicon Valley-style tech companies, ‘overturning’ established businesses in an inevitable process of ‘disruption’.

Andreessen’s language hints at a common belief in tech culture: it’s better to ask for forgiveness than permission. In the early 2010s Facebook’s Analog Research Laboratory designed a series of posters promoting messages of disruption in big red type: ‘Done is better than perfect’, ‘Move fast and break things’, and ‘Orville Wright did not have a pilot’s license’. In a 2018 Congress hearing, Mark Zuckerberg admitted that a culture of moving fast without thinking of the consequences contributed to several major Facebook scandals. Specifically, he recognised that Facebook’s platform had become an enabler for fake news, foreign interference in elections and hate speech.

Dark patterns
The Great Hack, a 2019 film about the Facebook and Cambridge Analytica data scandal, documents what can go wrong when technology is applied to democracy: hyper-personalised advertising and misinformation, powered by vast personal data trails generated by smartphones and social media. Cambridge Analytica may have shut down, but many of its methods have become default thinking in the political space. Within days of Boris Johnson becoming the UK prime minister, the Conservative party launched hundreds of Facebook advertisements, testing subtle variations in messaging. Meanwhile, former Cambridge Analytica employees have joined a new venture called Data Propria which is reportedly working on the Trump 2020 campaign.

The use of technology to influence voter outcomes is just one example of a technique known as ‘persuasive technology’. Academics have been researching this space since the 1990s, with Stanford University launching a Persuasive Technology Lab. The Lab’s stated aim is to create ‘insight into how computing products – from websites to mobile phone software – can be designed to change what people believe and what they do’.

Our apps and devices contain tell-tale signs that some designers are playing an active part in our manipulation. In 2010 Harry Brignull, a user-experience designer, coined the term ‘dark patterns’ to describe manipulative design choices such as Amazon’s burial of account deletion functionality in a labyrinth of menus and obscure terminology. His website, darkpatterns.org, categorises common patterns and has a Hall of Shame for websites that are particularly bad. Next time you let a Netflix show autoplay the next episode of a TV series on a Sunday night, it’s worth remembering what the company’s chief executive Reed Hastings said in 2017: ‘We’re competing with sleep, on the margin … it’s a very large pool of time.’

Another challenge levelled at tech is the inherent bias built into many products. Caroline Criado Perez’s book Invisible Women (2019) exposes the ‘one-size-fits-men’ data bias that drives many technology products. At 14cm, the average smartphone is too big for a woman’s hand. Research conducted in 2016 indicated that Google’s speech recognition software was 70 per cent more likely to understand men. Joy Buolamwini, a researcher at the MIT Media Lab, revealed in 2018 that facial recognition algorithms are more accurate when dealing with white males than with any other groups. Some algorithms correctly identified the subject’s gender for 99 per cent of lighter-skinned males, but misidentified 7 per cent of lighter-skinned females, 12 per cent of darker-skinned males, and 35 per cent of darker-skinned females. If law enforcement fails to test how well their systems deal with all races and genders, then minority groups could be more likely to experience unjustified arrest or questioning.

Some designers have realised that they have special responsibilities towards addressing technology’s emergent problems. Sarah Gold founded technology studio If with the aim of making more ethical digital products. ‘For too long we have not been explicit about the power and responsibility that designers and engineers hold,’ she says. Working with clients as diverse as Google, DeepMind, renewable energy supplier Bulb, the Co-op and children’s charity Barnado’s, Gold’s company helps organisations design products from an ethical perspective. Their goal is to go beyond the visual interface, and look ‘deep into the guts of a service’ in order to understand what’s happening at the level of the technical architecture. Increasingly, their recommendations and ideas are focused around transparency and trust.

In addition to client work, If has released several public projects, most notably their Data Permissions Catalogue. Available under a Creative Commons licence, this set of design patterns (repeatable solutions to commonly occurring problems) covers everything from getting consent before accessing personal information, to notifying users that their data will be shared with third parties.

Designer and author Cennydd Bowles, who previously led design at Twitter UK, agrees that designers are uniquely placed to address these problems, though he doesn’t believe they bear additional moral responsibility: ‘Modern tech is a team effort, and any ethical problems are the team’s problem.’ Product managers, developers and chief executives typically have a primary responsibility to the business, while designers focus on the user as an end in her- or himself. That care for the user puts designers in the right place to argue for their wellbeing, power and agency. Bowles argues that designers have the practical skills to bring their thinking to life in tangible form: ‘They can prototype those objects, those scenarios, to say: “How do we want to interact with technology in the future?”’

‘Speculative design’, a term coined by Anthony Dunne and Fiona Raby, imagines potential futures by using design to create ideas, rather than things, and gives designers opportunities to critique the status quo. The speculative design approach combines design with ‘methodological playgrounds’ from cinema, literature, science, ethics, politics and art. Designed ideas include ‘fictional worlds, cautionary tales, what-if scenarios, thought experiments, counterfactuals, reductio ad absurdum experiments and prefigurative futures’. If’s Gold particularly admires the work of Takram studio, Superflux, Strange Telemetry and of course Dunne and Raby.

South London design practice Comuzi is one of a new breed of studios that puts ethics and technology at the heart of their work. Recent projects by the studio include their work with the industrial design agency Morrama to produce prototype ‘disengaging’ smartphones that actively discourage digital distraction and a chatbot-powered guide to AI bias.

Comuzi’s work is notable for being critical of technology, while working with it. ‘We practise humane design, and build humane technologies that respect people and give them agency,’ says co-founder Akil Benjamin. Rather than not using technology, he believes that we should point it in a different direction. In a move interpreted as a statement about gun violence, Apple replaced its hand gun emoji with a water pistol – a move followed by providers such as Google and Facebook in 2018. Benjamin uses the emoji as a metaphor to challenge other technological advances, such as personal assistants, social media and AI: ‘Why don’t we make it a water pistol instead?’

One of the recurring features of studios like If and Comuzi, is a rejection of one of the dominant tropes of modern technology practice: user-centricity. Bowles argues that user-centred design is what’s dragging us to five degrees of warming and climate catastrophe. While it has been a helpful construct for the design community, he points out that in modern society the user is most likely to be a customer, or a consumer. Designers focus on the individual consumer’s wellbeing with little thought given to the wider community, or the neighbourhood, or the environment, or the planet. In contrast, If and Comuzi both take a systems thinking approach to their work. Benjamin says: ‘Systems thinking is going to be a phrase we hear more of in design – it allows you to design stuff that either is net-positive or net-neutral, but not negative.’ Bowles’s book Future Ethics (2018) is designed to translate and embed ethical thinking in the context of Silicon Valley design methodologies.

Unsafe at any speed
Alongside new design methodologies and attitudes adopted by practitioners, technological change is likely to come from a combination of top-down and bottom-up pressures. Shoshana Zuboff, whose book The Age of Surveillance Capitalism, was published at the beginning of 2019, has written that ‘surveillance capitalism thrives in the absence of law’. In any area in which capitalism’s inherent drive for profit and growth is in conflict with society’s best interests, campaigning and good will are never enough: legislation is necessary. Zuboff’s book takes inspiration from Ralph Nader’s Unsafe at Any Speed: The Designed-In Dangers of the American Automobile (1965), which led to the introduction of seat-belt laws in nearly all US states. This has also been the case with, for example, food safety, tobacco and alcohol. The European Union’s General Data Protection Regulation (GDPR) is the most significant new legislation in this space; its significant penalties are starting to hit businesses such as British Airways, which was fined £183 million for a 2018 data breach.

Change is coming from the bottom up, too. Collective action from designers, developers and other tech workers has become something of a trend, starting with anger at Google about the firm’s involvement with Project Maven, an AI project sponsored by the Pentagon. Since then, Google workers have staged walkouts, Microsoft staff have mobilised against a controversial contract with the US Immigration and Customs Enforcement (ICE), and 7500 Amazon employees have been trying to force Jeff Bezos to take action on climate change. And tech companies are beginning to suffer from increasingly toxic brands – Facebook has seen the number of software engineers accepting job offers fall from 90 per cent in late 2016 to almost 50 per cent in early 2019, according to a CNBC report.

Ken Garland’s ‘First Things First’ manifesto, published in 1964 and reprised in 1999 (see Eye 33), provided a rallying call for the design industry to focus on more positive uses of design rather than the ‘high pitched scream of consumer selling’. We are at a similar moment, with technology rather than consumerism as a target for action from designers. When it comes to creating new codes of practice, Cennydd Bowles prefers the idea of public consultation to a designer-led approach, and points to existing examples such as the Association for Computing Machinery’s Code of Ethics.

Perhaps the most powerful way that designers can contribute is in adopting more thoughtful processes in their work. As studios like If and Comuzi are demonstrating, design has a unique power to go beyond serving the immediate client. Flipping the negativity on its head, Gold believes that this is the most exciting time to be designing digital products: ‘The products over the next decade will shape the direction of the power we can give people,’ she says. ‘The data capitalism landscape is shifting.’

Top. Illustration by Michael Kirkham / Heart.

John Ridpath, writer, technologist, London

First published in Eye no. 99 vol. 25, 2019

First Things First manifesto
Dark Patterns
Harry Brignull
If
‘There is such a thing as society’ (Eye 13)