Two weeks ago I suggested we define the internet in terms of usage rather than technology. Last week I suggested that we do the same about its history. The key to that is this week’s subject, impact.
Rethinking the internet (and how we measure it)
The internet’s been with us long enough now, I suggested last week, for us to think of its trajectory from past through present to the future.
Rather than talking about what it "can" do, we can reflect on what it has done and use evidence of that, rather than aspiration, to anticipate the future and adapt our policies concerning it and that which it affects. But doing so requires more sophisticated and holistic ways of measuring its impact.
First, the scope of impact. Then the challenges of measurement. And finally some thoughts on how to go about it.
The scope of impact
Most measurement of internet starts with access and moves on to usage. More detailed assessment – of which there’s not enough – looks at drivers and barriers (such as affordability and content, capabilities and fears) and quality (bandwidth, device type, duration, type of use). And the most useful in terms of policy – which is even rarer – adds disaggregation (by gender, age, income, ethnicity and other demographic factors, ideally with intersectionality).
That range of measurements encourages focus on inputs. The trends are upward. Each year more people enjoy access and make use of it. Policy goals tend to focus, therefore, on more access, greater affordability, driving usage, reducing digital divides. All of which are, of course, appropriate, but they’re only partial because they don’t reflect on outcomes; they don’t ask “what’s the impact?” or see that measuring outcomes matters at least as much as measuring inputs, especially for policy development.
Can/will/has/may
I’ve written previously about the curse of “can”: the way much advocacy literature on new technologies explores what beneficial outcomes “can” result from them in ideal circumstances. That’s always good to know, and important for decision-making, but it’s also problematic.
Too many policymakers have assumed that “can” means “will”. Too little attention has been paid to what’s needed to turn “can” into “will” and to assess what unexpected, and perhaps less beneficial, outcomes may also be enabled.
The fact that the internet’s now been around long enough to have a proper history (see last week) means that we’ve evidence for what it “has” now done: to see what “can”s and “may”s have actually turned into “will”s and “did”s.
Even with the rapid pace of change that typifies it (and digitalisation generally), this evidence of trends and of trajectories should help us to add realism to anticipation/expectation. It should enable us to think about was overclaimed and what was underestimated, what was unexpected and what might be expected in the future.
And as the internet’s become more important, it should help us see what impacts it’s having at a larger scale – on the lives of individuals and businesses, of course, but also on societies, economies and governance, on patterns of employment and on human settlement, on transport, on energy consumption. To put some nuance into claims that it’s “transformative” and help to shape whatever “transformation” that might be.
Impact’s complicated
Over a decade ago, for the UK’s Department for International Development (DFID), I reviewed the difficulties involved in measuring the impact of programmes and projects that use information and communication technologies to achieve development objectives (“ICT4D”).
Measuring change is difficult. Change is complex and continuous; it has multiple causes, and even more consequences. It varies in pace and scale and time. It may be temporary or more permanent,sustainable or unsustainable; and many impacts will only become apparent in the longer term. It affects different people and groups of people differently, with “winners” and “losers”, some highly impacted, others less or little so.
I was looking, in that work for DFID, at interventions deliberately made to use ICTs for developmental goals, rather than the way that ICTs have had an impact on their own. The difference is important, fundamental even, but many of the challenges of measurement/assessment are similar. I listed ten particularly. Here is that list.
Challenges of measurement
First, there’s the variability of change. Social change, especially, results from many diverse and inter-related aspects of economy and of society. It varies over time, is affected by other factors (‘externalities’) including chance …
and, my second challenge, context, which varies widely between countries, between communities and individuals.
Then there’s the challenge of the baseline. To measure change we need to have a starting point, but when to choose? And do we have sufficient data? If our aim’s to quantify, it matters whether our starting point is (say) just prior to or after a recession. But we need to do much more than quantify if we’re to understand where we are starting from (and going to).
Fourth, attribution. Just because two things have happened at the same time does not imply that the relationship between them’s causal (or that it isn’t).
There are challenges of aggregation and disaggregation.
-
Many factors are involved in social change, and their interaction’s as likely to be significant as any one of them alone, especially if they’re thought to be “transformational” …
-
while impacts vary substantially because of the different circumstances of different group of countries, within communities, of individuals (from national development to levels of educational attainment, income/wealth, equality and inequality, to class and gender). Impacts at a national level need to be disaggregated if they’re to be understood.
There’s the challenge of non-participation. Everyone’s affected by the internet, it’s often said, and everyone’s affected by social change arising from it, even those who don’t participate in it – because it affects the society, economy and culture within which they live. Measuring impact needs to explore that variability.
The challenge of the unexpected is especially important. Predictions of the future, where technology’s concerned, tend to focus on what’s hoped for first, then what can reasonably be expected, but the biggest impacts of technology are often unexpected and sometimes much less welcome.
There’s a challenge of perceptions. Different stakeholders see impacts differently: those who have benefited and those who’ve not, most obviously. Enabling different stakeholders to express divergent views of impact is crucial to successful impact assessment, and often missing (especially from advocacy literature).
And, finally, what I’ll call the longitudinal challenge. Impact varies over time. Short-term impacts can be transitional or unsustainable, and differ from long-term impacts, especially as new technologies (like internet) become pervasive. Long-term impacts can indeed reverse those that were evident at first.
Measuring impact for policy development
So: the point of measuring impact is to add evidence of experience and the trajectory of change to the aspiration that has often guided policy development. Measuring impact’s complex because impacts are complex. That’s more reason for doing it and doing it well, not less. Understanding the challenges that I’ve just listed makes us more likely to adjust for them (and be more cautious in prediction).
I’ll end with five suggestions of ways in which our understanding of what has been the impact of the internet to date might be improved. (The internet here’s standing in for digitalisation generally. We’ll need to do the same with more recent instances. Though the history’s not there sufficiently as yet for technologies such as machine learning, we can anticipate the need to analyse it in these ways and plan to do so better.)
The first’s to do with what impacts are measured. These need to be wide-ranging. What has been the impact to date on economies and social welfare? (not just how that measures up against past aspirations). What has happened in relation to equality, empowerment, environment (not just likewise). And, particularly, what are the trends apparent from the history of internet that can help policymakers and others to anticipate the future.
The second is the measurement of difference. Ideas about the internet’s “universality” have encouraged assumptions that outcomes will be the same in different places. But other things aren’t equal and impacts are never universal. The impact of the internet on economies in least developed countries won’t be the same as that in industrialised societies. Its impact on empowerment in China’s going to be different from that in California.
The third’s concerned with interdisciplinary thinking. The internet’s tended to be understood more as technological and economic transformation rather than a social, cultural or political phenomenon. It’s been pored over most by those with technical and economic expertise, but understanding impacts needs a wider range of thinking: from sociology and anthropology, for instance. And understanding impacts in core areas like health and education needs to be led by those who understand those areas best, their own practitioners. (The UN Secretary-General has hinted at this in some recent speeches.)
Fourth, I’d say, impact assessment needs to start from empirical research rather than hypotheses. Too much assessment of internet impacts has started from assumptions of what the internet “can” do and looked at whether those assumptions have been justified. But much, and arguably most, of what the internet has done was not anticipated. Looking at what’s happened from a more open starting point would reveal those outcomes more.
And, finally, we need impact assessment that is independent. It matters where impact assessment gets its funding. At present, a great deal of this comes from vested interests, particularly data corporations. Justifiably or not, the independence of findings is often therefore questioned. As importantly, who funds research affects its focus: which is currently on issues that matter most to funders (and their future business models) rather than those that matter most to societies (from governments to those living on the margins).
Image: By @eskaylim via Unsplash.com