Editorial

Big Data & Society

January–June: 1–4
Revisiting the Black Box Society by ! The Author(s) 2020
Article reuse guidelines:
rethinking the political economy of sagepub.com/journals-permissions
DOI: 10.1177/2053951720935146
big data journals.sagepub.com/home/bds

Benedetta Brevini1 and Frank Pasquale2

Abstract
The Black Box Society was one of first scholarly accounts to propose a social theory of the use of data in constructing
personal reputations, new media audiences, and financial power, by illuminating recurrent patterns of power and
exploitation in the digital economy. While many corporations have a direct window into our lives through constant,
ubiquitous data collection, our knowledge of their inner workings is often partial and incomplete. Closely guarded by
private companies and inaccessible to most researchers or the broader public, too much algorithmic decision-making
remains a black box to this day. Much has happened since 2015 that vindicates and challenges the book’s main themes. To
answer many of the concerns raised in the volume in light of the most recent developments, we have brought together
leading thinkers who have explored the interplay of politics, economics, and culture in domains ordered algorithmically
by managers, bureaucrats, and technology workers. While the contributions are diverse, a unifying theme animates
them. Each offers a sophisticated critique of the interplay between state and market forces in building or eroding the
many layers of our common lives, as well as the privatization of spheres of reputation, search, and finance. Unsatisfied
with narrow methodologies of economics or political science, they advance politico-economic analysis. They therefore
succeed in unveiling the foundational role that the turn to big data has in organizing economic and social relations.

Keywords
Black boxes, algorithms, big data, political economy, artificial intelligence, environment

This article is a part of special theme on The Black Box Society. To see a full list of all articles in this special theme,
please click here: https://journals.sagepub.com/page/bds/collections/revisitingtheblackboxsociety

Throughout the 2010s, scholars explored the politics ‘black box societies’: jurisdictions where the analysis
and sociology of data, its regulation (via privacy and and use of data is opaque, unverifiable, and
trade secrecy laws, among many other rules), and its unchallengeable.
role in informing and guiding policymakers. COVID- Corporations and governments around the world
19 has dramatized the importance of this work. Quality are collecting the location data of millions of internet
health data has been vital to successful efforts to and smartphone users to find patterns about how the
‘flatten the curve’. In less fortunate polities, health
data is politicized, ignored, or manipulated.
To the extent data collection, analysis, and use are 1
Department of Media and Communication, The University of Sydney,
transparent and accountable, we can imagine a ‘grand Sydney, Australia
2
bargain for big data’: rapid analysis of novel data University of Maryland School of Law, Baltimore, USA
sources for public health purposes, in exchange for
Corresponding author:
enforceable promises that the data is anonymized as Benedetta Brevini, The University of Sydney, John Woolley Building,
well as possible, and is only used for those purposes. A20 Science Road, Sydney, New South Wales 2006, Australia.
However, all too much of this work is being done in Email: benedetta.brevini@sydney.edu.au
Creative Commons NonCommercial-NoDerivs CC BY-NC-ND: This article is distributed under the terms of the Creative Commons
Attribution-NonCommercial-NoDerivs 4.0 License (https://creativecommons.org/licenses/by-nc-nd/4.0/) which permits non-commercial
use, reproduction and distribution of the work as published without adaptation or alteration, without further permission provided the original work is
attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).
2 Big Data & Society

virus spreads. The resulting analytics often appear Much has happened since 2015 that vindicates and
opaque to outsiders, undermining accountability. challenges the book’s main themes. Cambridge
They can obscure complex social realities, with lasting Analytica, social credit scoring, and the ‘robodebt’
consequences for civil liberties and public health. debacle have vaulted algorithmically inflected decision-
Besides obvious privacy concerns, we already have evi- making into the headlines. The General Data Protection
dence that location data collection and publication Regulation has intensified debates over data protection.
during the COVID-19 pandemic is having discrimina- Europe’s emerging ‘right to an explanation’ of automat-
tory impact on minority groups and has led in some ed profiling has provoked lengthy debates on the proc-
cases to additional stigma and violence against specific essing of personal information. Yet recurring examples
disadvantaged neighbourhoods, local communities, or of algorithmically driven injustices raise the question of
religious groups. For example, as a result of disclosure whether transparency – the foundational normative
of personal information about people that tested posi- value in The Black Box Society – is a first step towards
tive in Seoul, LGBTQ individuals have been exposed to a more emancipatory deployment of algorithms and arti-
an outburst of intensified homophobia and harassment ficial intelligence (AI), is an easily deflected demand, or
(Human Rights Watch, 2020). A key Chinese corona- actually worsens matters by rationalizing the algorithmic
virus app has classified persons in an opaque way: as a ordering of human affairs.
result, Africans living in Guangzhou, China have To address these questions, this symposium features
reported being evicted from their homes, because of the work of leading thinkers who have explored the inter-
an erroneous association with COVID-19 transmission play of politics, economics, and culture in domains
(Marsh et al., 2020). U.S. states are rushing into the ordered algorithmically by managers, bureaucrats, and
app space without adequate guidance about privacy technology workers. It is hard to think of an aspect of life
and security implications of novel forms of surveil- that has not been affected by the use of algorithms, auto-
lance. These systems are thus replicating structural mation, and big data, including medicine, education, wel-
power inequalities, instead of addressing them. fare, voting, dating, communication, law enforcement,
What is missing from mainstream debates is an under- warfare, and cyber-security. However, as algorithmic
standing of how data analysed algorithmically is used as decision-making becomes ever more prevalent in our
a tool for social, political, and economic control. For public and private lives at the local and global level,
example, big data biases often distort decision-making, they all too often remain in the hands of power brokers
diverting the deployment of crucial funds. Even worse, such as corporations and governments, inaccessible to
the emphasis on data and automated tracking of individ- researchers or the broader public. By bringing social sci-
uals detracts from far more important foundations of entists and legal experts into dialogue, we aim both to
public health measures: universally available, free testing clarify the theoretical foundations of critical algorithm
and treatment that can establish a foundation of knowl- studies and to highlight the importance of engaged schol-
edge about disease prevalence and infection fatality rate arship, which translates the insights of the academy into
(both generally, as well as the specific vulnerability of an emancipatory agenda for law and policy reform.
groups and subgroups). Nevertheless, narratives of tech Starting with illuminating examples from data ana-
solutionism and even salvation-ism abound. lytics applied to managers and workers, Ifeoma Ajunwa,
Given the urgency of these problems, there could not in ‘The Black Box at Work’, describes the data revolu-
be a more appropriate time to reflect on the accelerating tion of the workplace, which simultaneously demands
deployment of big data-driven analysis by corporate and workers surrender intimate data and then prevents
state interests that operate in proverbial black boxes. them from reviewing how it is used. Far too many work-
The Black Box Society was one of first scholarly ers remain ignorant of the algorithms governing their
accounts of algorithmic decision-making to synthesize workplace and measuring their productivity. Ajunwa
empirical research, normative frameworks, and legal examines three dangers of this data revolution: the con-
argument. The book proposed a social theory of the cealment of disparities in hiring, the potential for ‘data-
use of data in constructing personal reputations, new laundering’, and pervasive surveillance. As she wisely
media audiences, and financial power, by illuminating observes: ‘The folly in this oracular reliance on big
recurrent patterns of power and exploitation in the dig- data-driven algorithmic systems is that without proper
ital economy. While many corporations have a direct interpretation, the decision-making of algorithmic sys-
window into our lives through constant, ubiquitous tems could devolve to apophenia’. When patterns are
data collection, our knowledge of their inner workings made (and not found via some publicly justifiable
is all too often partial and incomplete. Closely guarded method), evaluations of workplace performance are sus-
by private companies and inaccessible to most research- pect, eroding the dignity and autonomy of workers.
ers or the broader public, too much algorithmic Reaffirming the urgency of intelligible evaluation as
decision-making remains a black box to this day. a form of dignity, Mark Andrejevic, in ‘Shareable and
Brevini and Pasquale 3

Un-Shareable Knowledge’, focuses on what it means to acuity animates this piece. The use of student data by
generate actionable but non-shareable information. To higher education institutions has become central to
act on data, without being able to explain it, decentres operational and strategic planning, as well as the deliv-
and devalues narratively intelligible accounts of action. ery of tailored learning experiences. Like grades,
Indeed, as theorized in 20th century philosophy of AI-driven evaluations will have significant and perma-
action, such effects in the world may not be action at nent ramifications for students’ lives, but unlike
all, but mere observable regularity. As unknown auto- grades, they largely remain unknown to students.
mated systems sort and classify ever more benefits and Prinsloo’s normative framework elevates the importance
burdens, the public wants to know how persons are of autonomy, privacy, social well-being, freedom from
being sorted and judged. Regulators could respond bias, fairness, epistemic agency, and ease of information
by simply forcing the release of code or data. seeking. Each of these weigh against pervasive use of
However, we often need more – an understanding of black box algorithms at higher education institutions.
what managers and coders were planning when they Education never ends, and many well-intentioned
designed the system, and how they react to challenges policymakers who neglect urgent issues need it just as
or unexpected outcomes. Narratives help us make sense much as students. Continuing her work to place the
of such plans, framing complex scenarios into more climate emergency at the centre of policy making con-
accessible, causally connected versions of events. cerning the development of AI, Benedetta Brevini
Narrative is no cure-all, of course: sometimes it deflects argues that we must account for the environmental
deeper understandings of critical issues or oversimplifies costs of AI. Brevini has, in past work, illuminated
complex causation into hero/villain dichotomies. unexpected and disturbing connections between com-
Nevertheless, it is hard to read Andrejevic’s intervention munications and environmental degradation. Here, in
here (as well as his important recent book Automating ‘Black Boxes, not Green: Mythologizing AI and
Media) without concluding that it must be part of the Omitting the Environment’, Brevini documents how
toolkit of public understanding of algorithmic evalua- AI runs on technology, machines, and infrastructures
tions of persons. that deplete scarce resources in their production, con-
Lack of public understanding of targeted advertising sumption, and disposal, thus exacerbating problems of
in politics motivates Margaret Hu’s intervention. waste and pollution. AI also relies on data centres that
Having made several important contributions to legal demand impressive amounts of energy to compute,
discussions of the role of big data in public decision- analyse, and categorize. If we want to stand a chance
making, Hu here turns to the critical interplay between at tackling the Climate Emergency, then we have to
private media and public influence. She helps us reassess stop avoiding the environmental problems generated
the Cambridge Analytica–Facebook scandal (which by AI but scarcely remarked in popular accounts of it.
involved millions of Facebook users’ data being released As governments and firms repeatedly fail to address
and exploited without proper authorization). Hu’s arti- climate change, they create the conditions for increas-
cle ‘Cambridge Analytica’s Black Box’ surveys a range ing scarcity and precarity, particularly in cities. These
of legal and policy remedies that have been proposed to dire conditions create demand for intensified, automat-
better protect consumer data and informational privacy. ed surveillance to sort benefits and burdens. Surveying
She convincingly demonstrates that effective reform these grim horizons, Gavin Smith develops the concept
should include both increased oversight by the Federal of our ‘right to the face’ in ‘The Face is the Message:
Trade Commission and the potential for imposing pro- Theorising the Politics of Algorithmic Governance in
cedural and substantive due process-type requirements the Black Box City’. Smith’s analysis builds on his
on private actors like Facebook. While the voting booth widely recognized contributions to the sociology of
(or mail-in ballot) itself must be private, the larger public normalization – how certain behaviours are deemed
sphere of campaigns must be open to public scrutiny, ‘excessive’ and targeted for shaping by authorities.
lest shadowy actors pursue hidden agendas designed to Algorithms are now responsible for important surveil-
undermine the interests of the very groups they purport lance of cities, constantly passing judgment on mun-
to help. dane activities. Smith examines recent studies of the
The automated public sphere serves political informa- Australian cities of Darwin and Perth, looking first at
tion to voters in deeply privatized and hidden ways, and their promises to empower AI, before considering some
now the education sector is beginning to adopt similar of the political and ethical implications of these struc-
technologies of monitoring and personalization. In tures for notions such as the right to the city and civil
response, Paul Prinsloo examines ‘Black Boxes and liberties.
Algorithmic Decision-making in (Higher) Education’. In ‘Big Data: From Fears of the Modern to Wake-
Prinsloo has a strong record of critically interrogating up Call for a New Beginning’, Nicole Dewandre applies
the role of technology in education, and that analytic her deeply nuanced critique of modernity to
4 Big Data & Society

algorithmic societies. She argues that Big Data may be in building or eroding the many layers of our common
hailed as the endpoint or materialization of leading lives, as well as the kaleidoscopic privatization of
promises and fears of a Western modernity dating spheres of reputation, search, and finance. Unsatisfied
back to Cartesian metaphysics, or as a wake-up call with narrow methodologies of economics or political
for a new beginning. Dewandre advances the latter science, they advance politico-economic analysis. They
approach, exploring the downside of calls for therefore succeed in unveiling the foundational role
transparency rooted in Enlightenment principles. In that the turn to big data has in organizing economic
Dewandre’s framework, calls for more transparent cor- and social relations.
porations and governments are not merely a burden on From a purely economic perspective, we might seek
the watched, but also on the watchers, who engage in better markets for data, or more targeted subsidies and
what she vividly deems the ‘work of watchdogging’. taxes to ensure its proper production and vetting. From
Jonathan Obar confirms this problem empirically in a purely political perspective, we might propose new
‘Sunlight Alone is Not a Disinfectant: Consent and the forms of data governance. These are worthy initiatives.
Futility of Opening Big Data Black Boxes’, and pro- But they must be supplemented by the theoretical rigor
poses solutions to more equitably share the burden of of political economy, which exposes how the iterated
understanding. Obar’s past work in communications interplay of political and market victories and losses
has shed important new light on the automated can either entrench or challenge power.
public sphere. Here, Obar shows how a focus on frag- Political economy is a venerable discipline.
mented end-users’ rights and obligations can easily go Integrating the long-divided fields of politics and eco-
awry. Most persons do not have the time or inclination nomics, a renewal of modern political economy could
to engage with richly meaningful and detailed forms of unravel wicked problems neither states nor markets
transparency and consent online. A robust system of alone can address. It is an approach that forces us to
information fiduciaries and infomediaries could repre- rethink the relations between private experience and
sent their interests, but much more work must be done common purpose, to pursue viable alternatives to mar-
to realize the promise of these institutions. A new gov- ketization and datafication via surveillance capitalism.
erning framework for data is necessary. All the contributors help us imagine practical changes
In ‘Cyborg Finance Mirrors Cyborg Social Media’, to prevailing structures that will advance social and
Kamel Ajji explains how The Black Box Society economic justice, mutual understanding, and ecological
inspired him to found sustainability. For this and much else, we are deeply
grateful for their insightful work.
21 Mirrors, a nonprofit organization aimed at analyz-
ing, rating and reporting to the public about the poli- Declaration of conflicting interests
cies and practices of social media, web browsers and
email services regarding their actual and potential con- The author(s) declared no potential conflicts of interest with
sequences on freedom of expression, privacy, and due respect to the research, authorship, and/or publication of this
process. article.

Ajji’s scholarship and public service here is based on a Funding

deep understanding of the similarities between the The author(s) received no financial support for the research,
financial and the tech sectors in their use and reliance authorship, and/or publication of this article.
on information and algorithms. The financialization of
data and the datafication of finance have tended to ORCID iD
reinforce existing hierarchies of power, rather than Benedetta Brevini https://orcid.org/0000-0003-3596-6561
challenging them. Mere publicity about harms can
result in little more than window dressing responses,
References
since key incentives still incline companies to extract
data and exploit users for gain. Ajji’s 21 Mirrors initia- Human Rights Watch (2020) Covid-19 backlash targets
LGBT people in South Korea. Available at: https://
tive represents engaged scholarship, translating the
www.hrw.org/news/2020/05/13/covid-19-backlash-targets-
insights of the academy into emancipatory action. It
lgbt-people-south-korea (accessed 21 May 2020).
is a welcome reminder that ideas have impact, even in Marsh J, Deng S and Gan N (2020) Africans in Guangzhou
a world where black boxed computational and bureau- are on edge, after many are left homeless amid rising xeno-
cratic processes undermine solidarity and inclusion. phobia as China fights a second wave of coronavirus.
While the contributions are diverse, a unifying Available at: https://edition.cnn.com/2020/04/10/china/
theme animates them. Each offers a sophisticated cri- africans-guangzhou-china-coronavirus-hnk-intl/index.
tique of the interplay between state and market forces html. (accessed 26 May 2020).