OXPORD

Political Research
Methods and Practical Skills

Sandra Halperin & Oliver Heath

yo n lin e
resou rce
c entre
Political Research: Methods and Practical Skills
Political Research
Methods and Practical Skills

Sandra Halperin and

Oliver Heath

OXFORD
U N IV E R S IT Y PRESS
OXPORD
UNIVERSITY PRESS
Great Clarendon Street, Oxford 0X2 6DP
Oxford University Press is a department of the University of Oxford.
It furthers the University's objective of excellence in research, scholarship,
and education by publishing worldwide in
Oxford New York
Auckland Cape Town Dares Salaam Hong Kong Karachi
Kuala Lumpur Madrid Melbourne Mexico City Nairobi
New Delhi Shanghai Taipei Toronto
With offices in
Argentina Austria Brazil Chile Czech Republic France Greece
Guatemala Hungary Italy Japan Poland Portugal Singapore
South Korea Switzerland Thailand Turkey Ukraine Vietnam
Oxford is a registered trade mark of Oxford University Press in the
UK and in certain other countries
Published in the United States
by Oxford University Press Inc., New York
C Oxford University Press, 2012
The moral rights of the authors have been asserted
Database right Oxford University Press (maker)
All rights reserved. No part of this publication may be reproduced,
stored in a retrieval system, or transmitted, in any form or by any means,
without the prior permission in writing of Oxford University Press,
or as expressly permitted by law, or under terms agreed with the appropriate
reprographics rights organization. Enquiries concerning reproduction
outside the scope of the above should be sent to the Rights Department,
Oxford University Press, at the address above
You must not circulate this book in any other binding or cover
and you must impose this same condition on any acquirer
British Library Cataloguing in Publication Data
Data available

Library of Congress Cataloging in Publication Data

Data available
Typeset by TNQ Books andJournals Pvt. Ltd.
Printed in Great Britain
on acid-free paper by
CPI Group (UK) Ltd, Croydon, CRO 4YY

ISBN 978-0-19 955841-4

10 9 8 7 6 5 4 3 2 1
Contents

1 Political Research

PA R T 1 P h ilo s o p h y o f S o c ia l S c ie n c e : K n o w le d g e a n d K n o w in g
in S o c ia l S c ie n c e R e s e a r c h

2 Form s of K now led ge: Laws, Ex planation, and Interpretation

in the Study of the Social W orld

3 O b jectiv ity and Values

4 M etho d o lo g ical Ind ividu alism and Holism

PAR T 2 H o w to D o R e s e a r c h : A n O v e r v ie w

5 A sking Q u estions: How to Find and Form u late R esearch Questions

6 A n sw erin g R esearch Q uestions: R equirem ents, Com ponents,

and Co n stru ctio n

7 Research Design

PART 3 H o w to D o R e s e a r c h in P r a c t ic e

8 E x p erim ental R esearch

9 C o m p a rative R esearch

10 Su rveys

17 In te rvie w in g an d Focus G roups

12 Ethn o g ra p h y an d P articip an t O b servation

13 Textual A n alysis
CONTENTS

14 Quantitative A nalysis: Description and Inference 338

15 Patterns of Association: B ivariate A n alysis 366

16 A Guide to M u ltivariate A n alysis 391

Appendix 419

Glossary 42S

Index 433
 D E T A IL E D C O N T E N T S

Questions 127
Guide to Further Reading 127

References 128

6 Answ ering Research Questions: Requirements,

Components, and Construction 129
Chapter Summary 129
Introduction 129

Answers to research questions general requirements 132

What type of answer does your question require7 133
Answers that contribute to the development of theory 134
Where do hypotheses come from? 138
Illustration how an analysis of existing studies provides the basis for a hypothesis 139

Specifying your argument or answer: the nuts and bolts 144

The components of a hypothesis 144
Variables 144
Relationships 146
Conceptualization and operationalization 148
Concepts and concept formation what are you talking about? 148
Concept formation 149
Conceptualizing 'power': the search for theoretical utility 1SO
Conceptualizing 'democracy' mediating between coherence and differentiation 1S2
Operational definitions how will you know it when you see it? 1S3
Illustration: Lenin's explanation of World War I 154

Answering normative questions 1S6

Empirical and normative research 1S7
Empirical and normative questions, answers, and methods 158

Conclusions !60

Questions !60

Guide to Further Reading 160

References 161

7 Research Design I 64
Chapter Summary 164
Introduction 164
Basic principles of research design 165
Types of research design 167
Experimental designs 167
Data-gathering strategies: how the data are collected 174
Ethical research 178
Conclusions 180
Questions 180
Guide to Further Reading 181
References 182
d e t a il e d c o n t e n t s

60
Values and social practice 61
Thomas Kuhn and scientific revolutions ^
Paradigms and paradigm change
Kuhnian paradigms and political studies: the case of development theory
Paradigms and the study of development
Imre Lakatos and scientific research programmes
The role of values in research Kuhn and Lakatos compared

Conclusions

Questions
Guide to Further Reading

References

4 Methodological Individualism and Holism 78

Chapter Summary 78

Introduction 78

Individualism and holism 79

Methodological individualism and holism 80

Methodological individualism 80
Methodological holism 82
The politics of individualism and collectivism 86
Coleman's bathtub 88
Fallacies 90
Agents and structures in the study of politics 91
The agent-structure problem 92
The dialectical relationship between structures and agents 93

Conclusions 95
Questions 95
Guide to Further Reading 95
References %

PART 2 Howto Do Research: An Overview

5 Asking Questions: How to Find and Formulate Research Questions 101

Chapter Summary I q -j
Introduction 1 Q1
Research questions: What are they? Where do they come from? 104
Finding research questions
105
How to formulate a research(able) question -,11
The research vase
Types of questions getting clear about what you want to know ] 15
Unanswerable questions fallacies in framing research questions 12 0
The literature review ^

C o n clu sio n s
126
 D E T A IL E D C O N T E N T S

Questions -127

Guide to Further Reading 127

References 128

6 Answ ering Research Questions: Requirements,

Components, and Construction 129
Chapter Summary 129

Introduction 129
Answers to research questions: general requirements 132
What type of answer does your question require7 133
Answers that contribute to the development of theory 134
Where do hypotheses come from? 138
Illustration how an analysis of existing studies provides the basis for a hypothesis 139

Specifying your argument or answer: the nuts and bolts 144

The components of a hypothesis 144
Variables 144
Relationships 146
Conceptualization and operationalization 148
Concepts and concept formation what are you talking about? 148
Concept formation 149
Conceptualizing'power' the search for theoretical utility ISO
Conceptualizing democracy' mediating between coherence and differentiation 1S2
Operational definitions how will you know it when you see it7 1S3
Illustration: Lenin's explanation of World War I 154

Answering normative questions 156

Empirical and normative research 157
Empirical and normative questions, answers, and methods 158

Conclusions 160
Questions 160
Guide to Further Reading 160
References 161
7 Research Design 164
Chapter Summary 164
Introduction 164
Basic principles of research design 165
Types of research design 167
Experimental designs ^7

Data-gathering strategies: how the data are collected 174

Ethical research 178
Conclusions 180
Questions 180
Guide to Further Reading 181
References 182
O E T A IL E D C O N T E N T S

PART 3: Howto Do Research in Practice

8 Experimental Research 187

Chapter Summary 187

Introduction 187

Basic principles of experimental design 188

Laboratory experiments 190

Field experiments 192

Natural experiments 196
Issues in experimental research 197
Conclusions 198
Questions 199
Guide to Further Reading 199
References 200
9 Comparative Research 202
Chapter Summary 202
Introduction 202
The principles of comparison 203
How we compare 205
Case study
205
Small-N comparison
208
Most Similar Systems Design
210
Most Different Systems Design
212
Issues in small-N research
214
Qualitative comparative analysis (intermediate N)
218
Quantitative analysis (large-N comparison)
222
Conclusions
225
Questions
226
Guide to Further Reading
226
References
227
10 Surveys
230
Chapter Summary
230
Introduction
230
The elements of a good survey
231
What you ask questionnaire design
232
Validity from concepts to survey questions
232
Measurement error developing indicators
Question wording comprehension 233
234
 D E T A IL E D C O N T E N T S

Recall problems 236

Deliberate misreponmg 236
Question order effects 237
Question wording effects 238
Balanced questions 240
Response format 241

Who you ask: sample design 242

Defining the population sampiing frame 242
Choosing the sample random probability sampling 243
Sampling error 246

How you ask: interview mode 247

Conclusions 251

Questions 251
Guide to Further Reading 251

References 252

Interviewing and Focus Groups 253

Chapter Summary 253

Introduction 253

I. Interviews 254
Types of interviews 254
Individual face-to-face interviewing 254
Telephone interviews 254
Online interviews 255
Focus groups 255
Forms of interview 256
Structured interviews 256
Unstructured interviews 257
Semi-structured interviews 258
Which type and form of interview to use? 258
Designing a questionnaire or interview schedule for a structured interview 259
Open-ended questions 260
Partially categorized questions 262
Semi-structured or unstructured interviews 262
Formulating questions 266
Conducting a face-to-face interview 268
Conducting online interviews 270
Email interviews 270
Interviewing via video link and web camera 271
Elite interviews 272
Expert interviews and surveys 274

II. Focus groups 276

Organizing and conducting a focus group session 277

III. Analysing interview data 278

Conclusions 283
Questions 283
D E T A IL E D C O N T E N T S

Guide to Further Reading

Software 285

References 28S

12 Ethnography and Participant Observation 287

Chapter Summary 287
Introduction 287
The principles of ethnography and participant observation 288
Theory and hypothesis testing 2^0
Participant observation methodology 292

Choosing a research site 293

Types of research site 294
Selection of research site 295
Access to research site 295
The role of the ethnographer 297
Doing participant observation 298
Choosing informants 298
Studying with informants 300
Identity of the researcher 301
Recording observations 303
Fieldnotes 303
Conclusions 305

Questions 305
Guide to Further Reading 306

References 306

13 Textual Analysis 309

Chapter Summary 309
Introduction 309

Discourse analysis 310

Speech act theory, post-structuralism, and critical discourse analysis 311
Speech act theory 311
Post-structuralism 312
Critical discourse analysis 312
Analysis 313
Content analysis 318
Qualitative and quantitative content analysis 319
Coding 323
Manual and computer-assisted coding 323
Analysis 326
Validity and reliability 328

documents, archival sources, and historical writing as data

Using 328
Conclusions 33-)
Questions 331
Guide to Further Reading 333
 D E T A IL E D C O N T E N T S

Discourse analysis
Content analysis
References ^
Endnotes ii7
14 Q uantitative Analysis: Description and Inference 33 g
Chapter Summary

Introduction

Descriptive and inferential statistics 339

Levels of measurement 340

Summarizing the data 342
Tables 342
Graphs and figures 346
Summarizing the distribution using statistics 349
Measures of central tendency 349
Measures of dispersion 352
Descriptive inference 355
Population estimates 357
Confidence intervals for proportions 358
Confidence intervals for the difference between two proportions 360
Confidence intervals for means 362

Conclusions 364

Questions 364

Guide to Further Reading 364

References 365

15 Patterns of Association: Bivariate Analysis 366

Chapter Summary 366

Introduction 366
The principles of bivariate analysis 367
Is there a relationship between the variables? 368
How can we describe the relationship between the variables? 369
How good is our explanation? 369

Association and causality 369

Data considerations for bivariate analysis 370

Bivariate analysis for interval data 370

Bivariate analysis using scatter plots describing the relationship 370
Is there a relationship? Significance testing with interval variables 374
The test statistic 376
The strength of the relationship: a question of fit 377
Pearson's correlation 380
Example 1: democracy and development 380

Bivariate analysis for categorical data: cross-tabulation 382

Bivariate analysis using tables: describing the relationship 382
Is there a relationship? Statistical significance for cross-tabs 384
D E T A IL E D C O N T E N T S

Example 2: Ethnic diversity and civil war

Conclusions

Questions

Guide to Further Reading

References

16 A Guide to Multivariate Analysis

Chapter Summary

Introduction

The principles of multivariate analysis: statistical control

Specifying different types of relationship

Confounding variables and spurious relationships
Intervening variables and indirect causality
Moderator variables and interaction effects

Multivariate analysis using OLS regression

Example 1: democracy
Measures of fit
Interpreting the magnitude of the coefficients
Presenting the results

Multivariate analysis using logistic regression

Example 2: turnout
Measures of fit
Interpreting the magnitude of coefficients
Interaction effects
Presenting and reporting data
Conclusions
Questions
Guide to Further Reading
References

Appendix

Glossary

Index
Preface

Students devote a lot of time and effort to working on undergraduate. M A. and PhD theses, and there
is a large number of texts available which can help with various aspects of this task. Some texts are
devoted to the various debates and controversies relating to the philosophy of the social sciences, olh
ers to specific aspects of the research process, or to different research methods and approaches. But
sometimes the various aspects of research that are the focus of different texts can seem disconnected,
not only from each other, but from the process of pursuing knowledge about the political world. What
we have tried to do is to incorporate all these different aspects of research within a single text and to
discuss, step by step, how all of them relate to the research process. We begin with (I) an overview of
key issues in the philosophy of social science (the problems of what we can know and how we can
know it, in politics and in general). We then present (2) a 'nuts and bolts' or how to' of research de
sign: how to find and formulate a research question; how to engage with literature to develop a ration
ale for both the question and the approach that the researcher proposes to take in answering it; how
to construct a theoretical framework and define concepts that provide help in answering the question.
Then, in the final part of the book, we examine (3) different methods of data collection and analysis
that can be used to answer research questions, the principles and procedures that guide the employ­
ment of each, and the variety of considerations and decisions that researchers must confront when
using different methods.
Our aim is to provide students with an understanding of the perspectives, assumptions, logic, and
methods that contribute to political research, as well as an accessible step by-step approach to design
ing and completing the different phases of a research project. Wc hope that this will help you to learn
how to read analytically and think systematically about issues, problems, puzzles, and questions relat­
ing to the political world; and how to design and conduct independent research.
This book is based in large part on our experience of teaching at undergraduate and graduate levels.
We would like to thank our students, whose questions, queries, and problems have informed many
aspects of this book, and who have offered comments and criticisms on early drafts. We would also like
to express our thanks to Patten Smith from Ipsos Mori, Rob Johns, David Sanders, Paul Whiteley, and
the late Eric Tanenbaum from the University of Essex. George Gaskell, Sally Stares, and Jouni Kuha
from the Methodology Institute at the London School of Economics, and our colleagues in the Depart­
ment of Politics and International Relations at Royal Holloway, University of London. In particular, we
would like to thank Didem Buhari Gulmez and Baris Gulmez for their help with the online resources
for this text. Finally, we would like to thank our editor Kirsty Reade, and all the team at OUP who
worked on this book, for their support, and the anonymous reviewers for their valuable feedback.
 Political Research

This book has been written for undergraduate and graduate students of politics, with two
main purposes in mind. The first is to provide you with the analytic skills and resources to
evaluate research findings in political research. Ihe second is to provide you with the practi­
cal skills you need to carry out your own independent research. O ur aim is to offer practical
advice on how to be critical and rigorous, both in how you evaluate the research of others
and how you do your own research. These twin goals are im portant for getting the most out
of your study o f politics.
The study of politics can often be complicated, confusing, and controversial. In studying
politics, we are frequently confronted with opposing ideas and argum ents about a wide
variety of different political phenom ena. Is multiculturalism doom ed to failure? Has glo­
balisation underm ined national sovereignty? Is there a crisis of dem ocracy and participa­
tion? Is conflict an inevitable consequence of religious, ethnic, and social difference? The
answers to these questions, w hether provided by academics, politicians, or journalists can
be inconsistent and contradictory. This can make it difficult to know what to believe or who
to trust.
M aking sense of conflicting argum ents and interpretations can seem like a daunting, if
not futile, task. But a solid training in research skills can help us to distinguish between
argum ents that are relatively sound and robust, and those that are unsubstantiated or rely
on m isleading or faulty inference. These skills are therefore crucial for helping us to make
sense o f the world. They help us to evaluate the m erits of different argum ents and the
research o f others, and to m ake our own argum ents strong and convincing. Learning
research skills is an active process that engages you in developing the ability to investigate
the w orld around you and discover things for yourself. Pursuing research that enables
you to find your own answ ers to questions, rather than just relying on what is said or has
been w ritten by others, can be exciting and challenging. It can lead you into new and
surprising terrain.
These skills are at the core o f political research. And understanding them and being able
to use them transform s you from being a passive recipient of knowledge into an active
protagonist. As students of politics, you are not only acquiring knowledge about the world
o f politics, you also are joined to a research com m unity. T hrough engagem ent with
POLITICAL RE SE ARC H

research and w riting in our field, and the independent thought and research you pursue in
your own research projects, dissertations, or theses, you contribute to know ledge about
the political world. But these skills also have w ider relevance. They enable you to solve
puzzles, find creative solutions to problem s, and hone your analytical skills. Research
skills can be applied to answ ering questions in m any different fields, and are a genuinely
transferable skill that you can use in whatever you do. We hope that through this book,
and by engaging seriously with the principles and practices of political research, you will
not only be m ore inform ed or knowledgeable about political affairs, but also becom e
engaged, yourself, in a search for solutions to im portant problem s of a public, political, or
collective nature.
This book then, is about how, through systematic inquiry, to ask and answer questions
about the political world. The ways in which we do this, the m ethods or m ethodology that
we use, allow us to connect abstract ideas and concepts about the way the political world
works to evidence of what actually happens. M ethods for pursuing systematic inquiry also
encompass the system of values, beliefs, principles, and rules that guide analysis w ithin a
given discipline. Perhaps because of this association of m ethods with specific traditions of
inquiry, students often view the study of m ethods as a kind of im position—a set of rules and
constraints designed to mould research into conform ity with the conventions of a given field
of study. This is unfortunate and misleading. M ethods are not constraining: they are ena­
bling. They are embedded in the ways that we norm ally think and reason about things.
Research methods build upon our natural ability to think and reason; they enable us to hone
the skills we already possess into instrum ents of analysis, so that you are better able to evalu­
ate an argument and make one of our own; to figure out what makes the m ost sense am ong
competing accounts or interpretations, and to make up our m ind about what is true and
what isn’t.
Research methods are essentially about how to make arguments. All of us already know how
to make arguments. We make them every day. We come up with well-reasoned argum ents why
others should believe what we believe, or why our way of doing something is better than other
ways. And to make our arguments convincing we sometimes illustrate them with examples.
What social science research requires you to do is to apply the skills of reasoning and argum en­
tation you use in everyday life to larger questions of political life, and to hone these skills by
thinking about what sort of evidence or examples you need to really support your argument.
We have all learned to be wary about the use of anecdotal evidence. The friend of a friend who
saw or heard something is not a reliable source, and we will therefore often discount what peo­
ple say if it is based on unreliable evidence. Research skills simply build on these intuitive eve­
ryday skills that we employ all the time. They are an organic and creative aspect of thinking and
problem-solving. Moreover, they are intrinsically linked to the substantive concerns of our
field. In our view, devising a research strategy that enables you to investigate or demonstrate
an argument, hunch, or hypothesis is one of the really creative aspects of doing political
research. It is the aspect of research that perhaps provides the greatest scope for independent
thinking and creativity. Methods help you to build upon or synthesize the work of others, to
connect up the work of different writers and thinkers with each other, or link together separate
areas of study or studies of a single issue, in a way that generates fresh insights, that expands,
extends, refines our knowledge of a political problem, puzzle, issue, system, process, structure,
issue, or event.
 POLITICAL RESEA RCH

O u r approach to political research consists of three basic positions. First, we encour­

age pluralism in m ethodological approaches to political research. Politics is a multi-
m ethod field of study. The authors of this text reflect this pluralism . O ne of us pursues
qualitative and exploratory historical-com parative work; the other quantitative and
com parative work. Second, we believe that research should be problem-driven, rather
than m ethod-driven. A research design or m ethod of data collection only makes sense in
so far as it is a way of investigating som e problem . The value of any design or m ethod can
only be determ ined in relation to a research question; and the choice of which to use
should always be driven by the research question that is to be investigated (see, for back­
ground and debate on this issue, Laitin 2005 and Shapiro 2005a, 2005b). llttrJ, we believe
research should have relevance to im portant political questions and policy issues. For
research to be w orth pursuing, we have to be interested in the question, and interested in
the answer.

Issues in political research

This book, then, is concerned with how to formulate questions that are significant and how
to develop m eaningful and persuasive answers to them . A significant question is one that is
‘directly relevant to solving real-world problems and to furthering the goals of a specific
scientific literature (King, Keohane, and Verba 1994: 18). There is not always agreement
about what constitutes a meaningful am i plausible answer to a research question. But we can
all agree that our answers should help to generate valid and reliable knowledge about the
questions that they address. This requires that answers be developed through a process of
inquiry that, at every step, is both self-aware and critical, and that researchers m ake clear and
transparent how their conclusions were reached.
W hich research practices and m ethods enable political researchers to offer credible
answers to im portant questions? W hat constitutes good research? These questions have gen­
erated considerable debate.
A greem ent on these questions is difficult to achieve, in part, because politics is such a
diverse discipline. It struggles to even agree on a name. The study of politics is carried on in
D epartm ents of Politics, or Governm ent, or Political Science or, in the U.K., D epartm ents of
Politics and International Relations, to nam e just a few. The array of names reflects a variety
o f views about what constitutes the discipline. The growing tendency for sub-fields of politi­
cal research, such as International Relations, to become institutionalised as almost separate
fields of inquiry, further fragm ents the field of study. O ther divisions exist within the disci­
pline, as well. There is a tendency to define em pirical issues and the study of the ‘real world’,
and norm ative issues and the study of political ‘ideas and values’, as involving separate
research areas and traditions of inquiry. And even am ong those who study ‘real world’ prob­
lem s scholars are divided about what constitutes em pirical knowledge, and how this know­
ledge can be arrived at. Although these divisions are the source of lively debate and
disagreem ent w ithin the discipline, they should not be overstated. Ultimately many of these
oppositions are not as profound as they are often presented as being, and there is still more
that unites the discipline than divides it. Below we discuss some of these controversies in
m ore detail.
P O L IT IC A L R E S E A R C H

Politics and International Relations

In recent years there has been a tendency for Politics and International Relations to becom e
regarded as separate areas of study, with different professional associations, journals, degree
schemes, and even, in some universities, different departm ents. The establishm ent of these
sub-fields institutionalized a division of the political world into processes and structures that
are internal to states (local, domestic, or national politics), and those that are external to
them (interactions and relations among states). However, like m any in our field, we have
come to question whether, within the general study of politics, it is analytically productive to
treat domestic and international processes, systems, events, and issues as definably separate.
Increasingly, scholars in our field are recognizing that this analytical division tends to obfus­
cate fundam ental interdependencies of interstate and dom estic systems. Politics and Inter­
national Relations are both concerned with the nature of states, and with political systems,
development, conflict, ideology, social m ovements, geopolitics, nationalism , political par­
ticipation, and political philosophy. Moreover, all aspects of politics are affected by govern­
ments, public adm inistration and public policy, elections and voter behaviour, political
parties, political culture, mass publics and electorates, interest groups, and public opinion;
as well as by the interactions among states, the workings of the international system as a
whole, international political economy, international organizations, and international law.
Researchers have increasingly come to appreciate the extent to which political processes
operate across levels or at multiple scales. Relations across levels or political scales encom pass
sets of interactions that have hitherto not been the central focus of conventional approaches:
the interaction of societies, not just with their own governm ents, but with other societies
and governments, and with international organizations. International Relations theorists
recognize that states have to answer to groups within their societies and that consequently a
consideration of domestic relations is necessary to an explanation of outcom es in interna­
tional relations. With a growing awareness of structural linkages between societies and the
multiple channels that connect societies has come a tendency not to assum e the sufficiency
of any one scale of analysis to an understanding of political outcomes. Consequently, this
book addresses the study of Politics and of International Relations as a single area of inquiry;
and we address ourselves, as well, to students of all sub-fields of political research, including
Political Theory (classical political philosophy and contem porary theoretical perspectives),
Public Policy (the process by which governments make public decisions), Public A dm inis­
tration (the ways that government polices are im plemented), and Public Law (the role of law
and courts in the political process).

Em pirical vs no rm ative research

Political research has also tended to define a sharp distinction between em pirical’ and
normative research and theory. Empirical research addresses events and political
phenomena that we observe in the real world; questions about what is (empirical questions);
normative—or theoretical—research addresses ideas and thoughts and questions about
what should or ought to be (normative questions). However, this distinction between the
study ol events and (he study of ideas is also som ething of a false distinction. Empirical
researc is always grounded in ideas and theories; norm ative research is never entirely
 POLITICAL RESEARCH

divorced from reality either, and em bodies em pirical’ claims about the character of human
and natural realities (Sm ith 2004: 86).
As John Cierring and Joshua Yesnowitz argue, empirical study of social phenom ena is
m eaningless if it has no norm ative import; and it is misleading if its normative content is
present, but am biguous’, it we don’t know how it m atters (Cierring and Yesnowitz 2006: 104).
Indeed, the justification tor why a research question is interesting or relevant or meaningful
is essentially a norm ative one. But at the same time norm ative arguments that propose or
justify one value system over another will lack relevance if they make no attempt to relate to
the facts of actual practice or public life. As Steve Buckler (2010: 156) points out, normative
theory is concerned both about the world as it is and as we might think it ought to be. In sum,
good social science is both empirically grounded ‘and relevant to hum an concerns’ (Cierring
and Yesnowitz 2006: 133). Normative theorizing ‘must deal in facts’ and empirical work
m ust deal in values’ (2006: 108). W hile we must be sensitive to the difference between nor­
m ative and empirical questions and statements, we must also recognize that thev are not
independent of each other, and that there are costs in keeping them separate.
Recent discussions about the theory of deliberative dem ocracy illustrate these points.
Early research on the subject by scholars like Jürgen Habermas and John Rawls debated the
norm ative justifications of deliberative democracy, interpretations and necessary com po­
nents of the theory, but failed to take account of the sheer complexity of contem porary
societies (Elstub, 2010: 291). However, recent research has tried to engage more seriously
with the em pirical realities of this social complexity (see Baber and Bartlett, 2005, O ’Flynn
2006, Parkinson 2006). As C hapter 3 will endeavour to show, theory and evidence inform
each other. Seeing either one as entirely divorced from the other generates either fantasy or
m indless em piricism .
Q uestions o f m ethod continually arise in so-called norm ative research, and these are
som etim es m ore analogous to those in empirical work than is always recognized. Normative
theorists are concerned to convince others by means of drawing logical inferences and pre­
senting the logical development of their ideas. They want to persuade others that the conclu­
sions they reach are reasonable and plausible. An argum ent for any conclusion needs to be
evaluated in the light of the kind and nature of its premises, the strength of the inferential links
to the conclusion from these premises, and its possible criticisms or refutations. Does it make
unrealistic assum ptions about hum an behaviour; and if so, on what basis can this be estab­
lished? The association of theory and the empirical world is continually being tested, and
investigation into the relationship between ideas and practice is to be encouraged rather than
resisted. The link between theory and evidence can be conceived in different ways. Empirical
researchers may use theory in order to try and understand or explain or describe social and
political reality; whereas norm ative researchers may use theory to challenge political reality
(Thom pson 2008). But in either case it is im portant to describe the nature of reality in the first
place in order to be able to challenge it or explain it, or do both.

Positivism vs interpretivism
However, studying political reality’ is far from straightforw ard, and there is substantial
disagreem ent about how the em pirical world can be analysed. Debates over m ethodology
and the respective strengths of different m ethods are often conflated with issues relating to
P O L IT IC A L R E S E A R C H

different ontological and epistemological positions within the discipline. These positions—
which we return to later—are about the nature of knowledge in the social world, and what
is knowable. However, we argue that many of these different positions are also overstated
when it comes to analysing the social world in practice.
Many researchers have pointed to the tendency to cast positivist and interpretivist
approaches as ‘two grand traditions in social science epistem ology’ and to exaggerate the
differences between them (Pollins 2007: 93). Positivism m aintains that scientific knowledge
of the social world is limited to what can be observed; and that we can explain and predict
social phenom ena by discovering em pirical regularities, form ulating law-like generaliza­
tions, and establishing causal relationships. Interpretivism m aintains that knowledge of the
social world can be gained through interpreting the meanings which give people reasons for
acting, and that we can, in this way, understand hum an behaviour, but we cannot explain or
predict it on the basis of law-like generalizations and establishing the existence of causal
relationships.
Positivism and interpretivism have different ontological and epistemological com m it­
m ents—different views with regard to the nature of the social world and how we can have
knowledge of it. However, researchers working in both traditions generally follow the same
methodological conventions; and while researchers may be interested in different types of
questions, ‘practical investigation of these questions often leads them to sim ilar m ethodo­
logical tasks’ (Finnemore and Sikkink 2001: 395). Both are concerned to show the relations
between premises and conclusions, and indicate the nature of the relations between them.
Both recognize that some standard of validation must be established for the sources of evi­
dence used. By and large all researchers, whatever their methodological and philosophical
predispositions, share similar goals: to explain and understand, to engage with evidence and
use it well, to distinguish between those claims about the world that are fanciful and that are
robust.
O ur intention is not to privilege any one way of doing research over any other. O ur view
is that the m ethod you use in conducting research will always depend on the answers to the
following questions:

1. What research question are you trying to answer?

2. What evidence or data do you need to answer the question?
3. How are you going to analyse the data; and what are the practical steps needed to obtain
and record them?

Our view is that any research will be clearer, more accessible, more persuasive, and more
likely to achieve its aims, if the researcher articulates a central question/puzzle/problem ,
provides a rationale for it, reviews the relevant literature, advances a hypothesis or argument,
constructs a theoretical framework, defines concepts, variables, and relationships, and
designs a ‘test’ of the hypothesis or argument. The distinctions drawn among and between
qualitative and quantitative methods (discussed below), empirical and normative research,
and positivism and interpretivism are important for purposes of reflection on how to go
about finding credible answers to im portant questions. But, these distinctions are ultimately
sterile. Research is either systematic, self-aware, clear, and transparent, or it isn’t. Irrespective
 POLITICAL RESEA RCH

of what type of research you pursue and what m ethods you use. your research will he judged
according to some generally accepted standard of research practice.
I here is one last distinction that we wish to address: ihe distinction between quantitative
and qualitative political research.

Quantitative vs qualitative research

lo reap the rewards of m ethodological diversity, while ensuring that research is more
problem -driven and relevant to im portant political questions, requires that we develop a
notion of som e shared standard, one that can be applied to a variety of different questions
and diverse research traditions. Ihe search for a unified set of standards, or a com m on defin
ition of what constitutes good research, has tended to become bogged down in a debate
about w hether and how it is possible to bridge the divide between quantitative and qualita­
tive m ethods and approaches. Quantitative research tends to be based on the statistical
analysis of carefully coded inform ation for many cases or observations (in whatever way
those observations are defined). Qualitative research tends to be based on the discursive
analysis of m ore loosely coded inform ation for just a few cases. Some even go as far as to say
that quantitative and qualitative traditions are so ontologically distinct as to be incom m en­
surable. This position is based on a belief in a hard-and-fast connection between quantita­
tive m ethods and the tenets of positivism, on one hand, and qualitative m ethods and
interpretivism , on the other. We think that this represents a false dichotomy. Different m eth­
odological positions are not tied to any epistemological or ontological position. Nor are they
tied to any particular type of research question or research design.
O thers suggest that som e m ethods are inherently better than others, this position has
frequently been attributed to the book Designing Social Inquiry (1994), by Gary King,
Robert Keohane, and Sidney Verba. Designing Social Inquiry began an im portant debate
about m ethodology, and the book continues to occupy a central place in debates about
m ethods. Its aim , according to its authors, is to bridge the quantitative/qualitative divide
(King et al. 1994: 4). Its prim ary focus is causal inference in both qualitative and q u an ti­
tative research and, m ore generally, scientific m ethods and elements of scientific research.
The authors argue that ‘All good research can be understood—indeed is best understood—
to derive from the same underlying logic of inference. Both quantitative and qualitative
research can be systematic and scientific', provided each subm its to the rules of scientific
inference—rules that are som etim es m ore clearly stated in the style of quantitative research’
(1 9 9 4 :4 -5 ,6 ).
The authors argue that quantitative and qualitative research are substantively the same
and only differ in the types of techniques the employ; that the differences between the quan­
titative and qualitative traditions are only stylistic and are m ethodologically and substan­
tively u nim po rtant’ (1994: 4). Both are seeking scientific insights into social phenom ena,
and both require rigorous scientific m ethod to ensure such results. But critics of Designing
Social Inquiry argue that, in offering an abstract formal m odel of scientific m ethods and ele­
m ents of scientific research that will be applicable to all sorts of research, the authors develop
a ‘quantitative tem plate for qualitative research—one that presupposes the superiority of the
form er over the latter (Brady and Collier 2004: 3), and that im poses positivist concepts like
hypothesis testing on qualitative researchers.
P O L IT IC A L R E S E A R C H

W hile we think some of these objections have been overstated, we do not agree that som e
m ethods are better than others. However, there are certainly better ways to carry out a
method. It is therefore less im portant which m ethod is used, than how the m ethod is used.
O ur concern with all the m ethods we discuss is with their specific procedures, techniques,
or strategies for collecting and analysing data or inform ation necessary to dem onstrate an
argument. The link between theory and evidence is central to sound research, to how we
actually go about collecting and analysing the inform ation or evidence that we need to sup­
port (or just as im portantly underm ine) an argum ent. And while it is true that there is no
such thing as a correct m ethod, there is such a thing as convincing evidence and analysis,
and this is far m ore im portant. Evidence can be gathered and interpreted and analysed in a
variety of different ways. There are good ways and bad ways of doing research using each of
the m ethods we discuss. No m ethod is perfect. All have their strengths and weaknesses, and
it is im portant to be aware of what they are.

The research process

There are a large num ber of texts available that address themselves to research m ethods, to
specific aspects of the research process, and to debates relating to social science m ethodol­
ogy: books on theories and methods, on research design, and on political analysis.
This book is also concerned with research design, analysis, and theory (though it does
not devote chapters to specific theoretical approaches, such as rational choice or fem i­
nism). However, it is prim arily designed as an introduction to the research process. The
aim of this book is to introduce ( 1) key issues in the philosophy of social science that
bear on the choices researchers m ust m ake in pursuing research; (2) a step-by-step
delineation of the process of asking and answ ering a research question—form ulating a
question, reviewing the relevant literature, advancing a hypothesis or argum ent, con ­
structing a theoretical fram ework; defining concepts, variables, and relationships; and
designing a ‘test’ of the hypothesis or argum ent; and (3) the array of m ethods used in
political research and how scholars utilize these m ethods to answer questions about
political phenom ena.
Before elaborating on these aims, we wish to raise a point about the array of m ethods
we introduce in Part III of the book. As we noted earlier in this chapter, we are m eth od o ­
logical pluralists. We are enthusiastic about the diversity of the field and about new m eth ­
ods of inquiry that are developing. But we do not cover every m ethod and approach in
this book. We doubt that it is possible to adequately survey, in a single volum e, the vast
array of m ethods and approaches available to political researchers; and we do not attem pt
to do so. Instead, we provide an in-depth discussion of a range of widely used m ethods
that we think can provide a base on which to build greater expertise and explore m ore
specialized techniques; and we signal, where appropriate, how these can be further devel­
oped in more specific ways. O ur Guide to Further Reading in each chapter directs read­
ers both to specialist texts on the subject m atter of the chapter and to m ore specialized
adaptations of the approach discussed. We expect that students, with the help of the fur­
ther reading tor each chapter, will be able to take these approaches in a variety of further
directions.
 POLITICAL RESEA RCH

Ih.s book is concerned with the process and practice of political research: the principles
and P™ «dures that guide scholars as they conduct research, the beliefs and assumptions
they hold, the kinds of questions they ask, and the variety of decisions that they must make
Its aim is to answer two questions:
1. How does one form ulate research questions?
2. O nce these questions are form ulated, how does one design and carry out research in
order to answer them?

To address these questions we focus on three broad com ponents of the research process:
( 1) key issues in the philosophy of social science (the problems of what we can know and how
we can know it); (2) the nuts and bolts or the 'how to’ of research: how to find and formulate
a research question; how to develop, through an engagement with the relevant literatures, an
argum ent or answer that responds to your question; and (3) the specific methodological
procedures and techniques utilized in carrying out a given research project. Below, we
provide an overview of each of these com ponents.

Part I: Philosophy of Social Science: Knowledge

and Knowing in Social Science Research
M ethodology refers to the conduct of inquiry. Among other things, this involves reflection
upon the system of values, beliefs, principles and rules that guide analysis within a given
discipline. Q uestions of ontology and epistem ology—questions about the complexities and
am biguities of knowing and gaining knowledge of the social world—are a core part of this
reflection. These questions are the focus of key debates within the philosophy of social sci­
ence and, in our field, of on-going debates about scientific practice, forms of knowledge, age
the world of politics.
Why, you m ight ask, should these philosophical issues and controversies concern political
researchers? To start with, it is im portant to understand that all social scientific theorising
adopts a position with respect to the issues that these controversies involve. If you are una­
ware of them , you will not fully understand the implicit assum ptions and implications,
either of what others are arguing, or of what you yourself are arguing. We have said that
research m ethods build on and develop your own natural abilities to think and solve prob­
lems. In the sam e way, an understanding of philosophical positions concerning knowledge
helps to m ake explicit and to develop the philosophical assum ptions about, and approach to,
knowledge that you already em ploy in your everyday life.
Furtherm ore, if we see the purpose of scholarly research as the generation of valid and
reliable knowledge, and view tru th as a central characteristic of valid knowledge, then we
need to adopt and defend assum ptions about the nature of truth and procedures for dis­
covering the truth. In this way, philosophical presuppositions about ‘reality’ are intrinsi­
cally linked to the substantive issues that are central to inquiry in our field. Researchers
cannot contribute to knowledge about som ething unless they adopt and defend a view of
what is know able about the social world and what sorts o f things different m ethods enable
us to know.
P O L IT IC A L R E S E A R C H

Philosophy of social science debates have implications for all areas of research; and,
whether or not researchers follow ongoing debates in the philosophy o f social science, they
tend, either implicitly or explicitly, to reflect one or another of the different answers and
positions that these debates have generated. All of us have stored in our m inds a worldview
which provides the basis for the opinions we form about what goes on around us. This is
constituted, m ost likely in large part, by ‘an accum ulation of the ideas and prejudices of o th ­
ers’ (Kahane and Cavender 2006: 19). If the unexam ined life is not w orth living, as Socrates
is said to have claimed, then it perhaps follows that unexam ined beliefs are probably not
worth holding. Learning how to systematically investigate ideas we hold about the world
enables us to analytically engage with political affairs, rather than to rem ain passive consum ­
ers of the output of politicians, political analysts, and the news m edia. Reality is constantly
being defined for us. The ability to identify the underlying structure of assum ptions or the
implicit theory which shapes a given account of reality, w hether presented by scholars, poli­
ticians, or journalists, allows us to become m ore active analysts of contem porary politics.
In sum: understanding the terms of the m ajor debates in the philosophy of social science,
and sensitivity to their implications, is an im portant part of producing good research. W hat
you presume is knowable about the social world will bear on the strategic choices you will
need to make all through the process of research.
Each of the chapters in Section I of the book is devoted to a key controversy in the phi­
losophy of social science:
1. W hat is knowledge? How do we know? (Chapter 2)
2. Can the pursuit of knowledge be ‘objective’? (Chapter 3)
3. What is the nature of the social world? (Chapter 4)
The first controversy involves questions about what sort of knowledge we can gain about the
social world. Is it the same sort of knowledge that scientists are able to obtain about the
natural world? Or are the forms of knowledge concerning the social world and the natural
world necessarily different? In C hapter 2 we address three different approaches to answering
these questions and their implications for conducting political research.
The first approach is ‘positivism’. ‘Positivism’ is usually defined by the following three
tenets: ( 1) scientific methods (i.e. the testing of hypotheses derived from pre-existing theo­
ries) may be applied to the study of social life; (2) knowledge is only generated through
observation (empiricism); and (3) facts and values are distinct, thus making objective
inquiry possible (Snape and Spencer 2006). A second approach, ‘interpretivism ’ m aintains
that the social world is fundamentally different from the world of natural phenom ena, and
that it does not exist independently of our interpretation of it. The task of social science,
then, is fundamentally different from that of natural science, because the objects of the social
sciences are different from those found in the natural world. The third approach, scientific
realism maintains that knowledge is not limited to what can be observed but also includes
theoretical entities (unobservable elements of social life).
Positivism maintains that it is possible to define a distinction between facts and values, and
lor us to acquire value-neutral, objective knowledge about social phenom ena. Critics argue
that knowledge produced by social-scientific research is not value-neutral, but is shaped by a
variety ol tat tors, including existing scientific theory, politics, and power relations, cultural
 POLITICAL RESEARCH

beliefs and m eanings, and the researcher's own motivations and values. Can knowledge pro
duced through the study of the social world be objective ? We consider this quest.on in
C hapter 3.
Finally, we take up the issue of social ontology (C hapter 4 ). W hat is the social ? What
is the social world m ade of? W hat is the basic unit of analysis in the study of the social
world? Two contrasting views about the nature of the social world have dom inated dis
cussion of this question: ‘individualism ’ and holism ’. M ethodological individualism
argues that individuals are the basic units of society and that social life must be explained
in term s of the actions of individuals. M ethodological holism treats social wholes’ as the
basic unit of analysis, understood as distinct from , and not directly explicable in term s of,
its parts.
In political research, discussion of this issue is carried on within a debate about the rela­
tionship between agents (the actors) and the structures which shape, give meaning to, or
make possible their actions. How can we understand the relationship between individual
agents and the social structures within which they act? Are societies reducible to the indi­
viduals who m ake them up? O r is society m ore than the sum of its individual members?
W hat level of description—the individual or the collective—is necessary for explanation of
social phenom ena? Ih e issues raised in each of these debates have implications for how you
pursue research and develop explanations of political phenom ena.

Part II: Howto Do Research: An Overview

A second com ponent of the research process involves the practicalities of doing research:
the basic nuts and bolts of the research enterprise. By this we mean the steps involved in
developing a plan for pursuing research on a topic. This involves developing a researchable
question, locating applicable theory and literature, formulating testable hypotheses, and
clarifying concepts and developing em pirical indicators. The chapters in Part II of the book
introduce basic elem ents of research: a research question, what requirem ents it must meet,
and where to find and how to formulate one (C hapter 5); how to construct an argum ent,
hypothesis, or theory that answers it (C hapter 6 ); and how to design research to test, inves­
tigate, or dem onstrate it (C hapter 7).
M ost political research originates from som e general question or problem that arises,
either from the events, issues, or processes we observe in the world around us, or from
the theories and fram ew orks that our field has developed in order to understand them .
But the research process only begins with the conversion of this general question or p ro b ­
lem into a w ell-form ulated, clearly focused, research question. Step 1 of the research
process involves finding and form ulating a researchable question, and locating applicable
theory and literature (C hap ter 5). We discuss why research should be guided by well-
form ulated research questions, the role of the literature review’ in providing both the
inspiration and rationale for them , and the different types of questions scholars ask. We
discuss the basic requirem ents o f a research question and how to go about m eeting them .
We argue that a research question is one that ( 1) has significance for a topic or issue relat­
ing to the subject m atter o f our field, (2 ) is researchable (it can be answered through
con du cting research), and (3) has not yet been answ ered definitively.
P O L IT IC A L R E S E A R C H

Irrespective of the particular way researchers choose to structure th eir research, a

carefully form ulated research question will ensure that their research has a clear p urpo se
with clear goals. A research question m akes the research task specific; so that, rath er than
trying to gather all the inform ation you can find on a topic, you direct your attentio n to
just the inform ation that addresses the question and helps you to develop an argum ent in
answer to it. W hen you have a w ell-form ulated research question, you can m ake decisions
about what inform ation should be included or excluded, what data you will need to col­
lect, what to observe, or what to ask your interview ees.
To form ulate a research question you need to be clear about w hat you w ant to know. We
identify different types of questions and what each type com m its you to doing; and also
the logical fallacies that som etim es find their way into the statem ent o f a question and lead
to false conclusions. The third requirem ent of a research question requires that you show
that others who have addressed it have failed in som e way to provide a definitive answ er to
it. The literature review perform s this function. It identifies what (range of) answ ers are
found in the existing literature relating to the question and it develops an argum ent about
what needs to be done in order to provide a better answer to the question than those that
currently exist.
Once you have formulated a research question, you will be ready to move on to Step 2 of
the research process: how to answer it. We discuss the basic com ponents of an answer to a
research question, what requirements it must meet, and where to find and how to form ulate
one (Chapter 6). To get you started we offer a broad template. This is m eant to serve as a
starting point—a set of considerations to be revised to fit with the question and aims that
animate a particular project. The template shows a research process that is structured in an
idealized linear fashion. But we emphasize that the process as it actually unfolds is often not
linear at all. The real process is often circuitous. Most researchers move back and forth
between theory and evidence—between theorizing som ething that is the focus of their
research, and mucking about in the ‘dust of detail’ (learning more about the specific facts of
the case or question or issue, or the observations of other scholars that we treat as facts). As
Philip Shively observes, one of the better-kept secrets’ in our field, is that good researchers
usually do not “‘frame hypotheses” in any formal sense before they start to work, though
they may have some operational hunches about what they expect to find___They play with
data, immerse themselves in what other people have written, argue with colleagues, and
think’ (Shively 1989: 25). In sum, we describe the research process as consisting of a set of
components or series of steps but, in practice, the process of research does not unfold in the
sort of linear fashion that this suggests. We rethink our views as a result of learning from the
research process itself in ways that can feedback to our previous choices and lead us to revise
them. So, what the template we present really shows is, not the process of research, but its
ultimate presentation.
We organize our discussion of the various considerations and tasks involved in develop­
ing an answer to a research question around three basic requirements. The first requirem ent
is that the answer be appropriate to the type of question that is being asked. Different types
of questions demand different types of answers. For instance, descriptive questions will
require that you describe the characteristics of something, or model how it works or behaves.
Explanatory questions will require that you explain what factors or conditions are causally
connected to a known outcome. Normative questions may require that you adjudicate
 POLITICAL RESEA RCH

am ong different understandings of how som ething should be, or what should or ought to be
done, by considering the argum ents of others, and subm itting well-reasoned argum ents for
one’s own.
The second requirem ent of an answer to a research question is that it makes a contribution
to knowledge. Social science research is expected to address a question whose answer will
contribute to collective knowledge in a particular field of study; so in developing an answer
to a research question one must ask oneself: Why should we care about this answer or argu­
m ent? In other words, your answer must matter. It is not enough to say that the question has
not been asked before. After all, one very good reason why a question has not been asked is
because no one cares about the answer. So you must always make a case for why the question
is relevant and im portant.
Ihe third requirem ent is that an answer must be clearly and fully specified with regard to
the factors or variables you think must be taken into consideration in order to answer your
question, and how you think these factors or variables are related to each other. For all types
of research, we think it is useful to formulate a ‘working hypothesis’—an operational hunch
about what you expect to find. Initially, what argum ent motivates the research? What find­
ings m ight be expected? By articulating in advance the contours and logic of the investiga­
tion, a hypothesis helps to guide research. Developing a hypothesis encourages you to be
very precise about how your answer relates to those that others have offered to your ques­
tion, and how your answer relates to what evidence you would expect to see in the real world.
The term ‘hypothesis’ is often treated as applicable to only quantitative research and to a
specific prediction about the nature and direction of the relationship between two variables.
But we use the term ‘hypothesis’ to mean ‘a hunch, assum ption, suspicion, assertion, or idea
about a phenom enon, relationship, or situation’ with which research begins and which
becom es the basis of inquiry (Kum ar 2005: 74). Ihe key thing is that the hypothesis should
be em pirically or logically verifiable (or not). That is, it should be falsifiable with evidence. A
hypothesis which is not falsifiable is really just a tautology, and so is not going to tell us any­
thing interesting about the political world.
If we are interested in the question we ask and the hypothesis we test, then we should also
be interested in the answer we get; and that means that an answer which discredits our
hypothesis is just as valid as one that confirms it. There is som etim es a tendency to think that
research is only worthwhile if it produces results that support our hypothesis, but null find­
ings can be interesting too. For example, the finding that the level of ethnic diversity within
a country does not influence the level o f dem ocracy within a country has im portant im plica­
tions for our understanding of what impedes dem ocracy (and what does not).
H ypotheses can either be tested with evidence (confirm atory research), or operate as a
guide to a process of discovery (exploratory research). Exploratory research begins with a
question and perhaps a basic proposition, probes its plausibility against various types of
data, and eventually generates a m ore concrete hypothesis, which can be m ore widely and
rigorously tested.
O nce you have a hunch or argum ent about the answer to your research question, you
then need to develop a strategy for providing a convincing ‘test’ or dem onstration of it.
This is Step 3 of the research process: how to dem onstrate the validity of your answer
(C hap ter 7). The plan you develop to do this is what we call a research design. It sets out a
plan for research, including w hat observations to m ake and how to m ake them . It is
P O L IT IC A L R E S E A R C H

inform ed by and fulfills a logical structure of inquiry; and it specifies the sort of test or
evidence that will convincingly confirm or disconfirm a hypothesis; the observations you
need in order to dem onstrate the relationships stated by your hypothesis, and how you will
go about m aking them ; the data relevant to dem onstrating these relationships, and how
and where you will collect them . The type of research design you use and the kind of
inform ation you collect, the sources of data and the data-collection procedures you choose,
should be based on what will provide a convincing test, dem onstration, or investigation of
your argum ent or hypothesis.

Part III: Howto Do Research in Practice

The third part of the book is devoted to exam ining the principles of different m ethods in
political research, and how to use them in practice. These m ethods enable us to put a research
design into practice and to carry out the research and analysis needed to answer a research
question. Methodological principles are concerned with how we obtain knowledge about the
political world; with the specific procedures or techniques that we use to tell us som ething
about the world when we carry out research. Research m ethods are in a sense the tools of
analysis. And, as with any tool, it is im portant to know how to use it in order to complete
a task.
In Part III of the book we focus on different strategies for collecting and analysing evi­
dence in order to test or develop specific hypotheses. In particular, we focus on the ways in
which data is collected and the ways in which it can then be analysed. These are two distinct
and crucial aspects of the research process. Failure to devote sufficient care and attention to
either will lead to research results which are untrustw orthy and unreliable, m eaning that we
may reject hypotheses that are actually true or accept hypotheses that are actually false.
Obviously, we want to avoid the possibility of either of these outcom es occurring as far as
possible. If we reject or accept a hypothesis, we want it to be because we have collected con­
vincing data and analysed the data properly.
Research methods all involve two important components: data collection and data analysis.
And it pays to think of these separately. We can therefore think about different m ethods of
data collection, such as collecting inform ation through the use of experim ents (C hapter 8),
comparative research (Chapter 9), surveys (C hapter 10), interviews or focus groups (C hap­
ter 11), participant observation (Chapter 12), or collecting archival data or docum entary
records such as speeches, policy docum ents, or m edia reports (C hapter 13). The type of
evidence that we use is important. But more im portant is how we collect and code the evi­
dence in preparation for analysis. This relates to the issues of validity and reliability. We want
the data we have collected to be good and solid and reliable and robust. It is what separates
us trom those who rely on rum our and hearsay and anecdotes. Data collection needs to be
systematic, not haphazard. We must have guiding principles to ensure that the data we col­
lect is of good quality.
Having obtained this data or information, we can then think about how to go about ana-
ysing it. It is common to distinguish between quantitative and qualitative approaches to
analysing data. Quantitative analysis tends to rely on statistical techniques, which we discuss
m detail m Chapters 14, 15, and 16. But there is a wide variety of more qualitative analysis
 POLITICAL RESEARCH

techniques that we can consider. It is worth rem em bering that this distinction between qual­
itative and quantitative analysis primarily refers to different m ethods of analysis rather than
different m ethods of data collection. And although there is some overlap between how we
collect evidence and how we analyse it, there is also a lot of diversity, and similar types of data
can be analysed in different ways. In fact data from all the types of data collection methods
that we discuss can be analysed using either quantitative or qualitative techniques. W hen we
talk about ‘quantitative data', we are only talking about evidence that has been coded in
preparation for quantitative analysis. Ihis involves coding and categorizing the data with
num erical values. Ihe actual form ol the evidence can be almost anything. It is therefore
im portant to be familiar with both qualitative and quantitative approaches, since failure to
understand or engage with one approach is likely to cut you off from a lot of relevant research
that is carried out on your topic of interest.
M uch is often m ade about the relative strengths and weaknesses of different m ethods,
in particular between quantitative and qualitative approaches. It is frequently asserted
that quantitative research may be good at m aking generalizations, but is a blunt instru ­
m ent for investigating hard-to-define concepts, such as power, globalization, and dem oc­
racy or difficult-to-observe phenom ena, like money laundering, crim inal or anti social
behaviour, corruption, and terrorism . By contrast, one of the key strengths of qualitative
research is often thought to be its ability to investigate these hard-to-define concepts and
hard-to-reach populations. However, whereas qualitative research might be able to go into
a lot of detail, there is som etim es a nagging suspicion that its findings might not have
w ider relevance to contexts outside the im m ediate vicinity of where the research was c on ­
ducted. W hile there m ight be an elem ent of truth in this, these strengths and weaknesses
should not be overstated or viewed as inherent in the different m ethodological
approaches.
These strengths and weaknesses are often presented in term s of a trade-off between
description and detail (validity of m easurem ent) and explanation and generalization (valid­
ity of inference). We reject this idea. 'Ihe ability to make generalizations is not just the pre­
serve of quantitative approaches, but about som ething m ore fundam ental to the research
process. It is about being able to rule out and control for theoretically plausible alternatives.
All m ethods of analysis should take this issue seriously, otherw ise we may end up with spuri­
ous findings. The only way to address this is by considering theoretically im portant varia­
bles, and the only way we can do this is by considering enough cases to make it manageable.
For example, the concern that the findings from a small N (where N refers to number, as in
num ber of cases or countries, etc.) study may not be applied m ore generally are to do with
concern over om itted variable bias, which can cause spurious relationships. This we discuss
in detail in C hapter 9.
To be able to estim ate the effect of many variables you need lots of cases to see if your
argum ent still holds up when you consider different factors across different contexts. Q uan­
titative m ethods provide an efficient way of doing this. But qualitative m ethods can do it as
well. A nd indeed they often do it, as we discuss throughout the book. But it is generally a
slower and less system atic process which takes a great deal of time. Studies are repeated,
findings are applied to new contexts, and, as this is done, theories or hypotheses are indi­
rectly tested on a wide range of cases. But there is nothing to stop it being done quicker and
m ore systematically. It would just involve m ore money. The difference is not one of m ethod
P O L IT IC A L R E S E A R C H

but of time and costs. There is no m ethodological reason to stop you repeating qualitative
studies; you can do twenty in-depth interviews (see C hapter 11); but why stop at twenty? If
time and resources are no obstacle, then it is possible to do 2000 interviews. A nd why not
collect multiple ethnographies? Recent research is attem pting to do just this, through col­
laborative comparative ethnographies that try and develop a firm er basis for wider infer­
ences (see Gillespie, Gow, and Hoskins 2007, Gillespie and O ’Loughlin 2009).
By contrast, it is often said that qualitative m ethods can provide m ore valid m easures of
political phenom ena. They are better able to m easure and study difficult-to-define political
phenom ena, whereas quantitative m ethods are too blunt and reduce complicated concepts
down to num bers which can never capture the full m eaning of what is being investigated.
But this is not the case either. There may be an issue to do with the extent to which extant
survey data can be used to answer new questions, but the answer to this is to design new
surveys or collect new data and devote additional resources to the problem. The difference is
one not of m ethod, but of resources. It is possible to measure anything in a reliable and valid
way with quantitative m ethods of data collection, but to do so if the object of investigation is
difficult to observe or measure can be very costly. Studies using qualitative m ethods are often
relatively cheap.
O ur position is that, irrespective of whether a quantitative or qualitative approach is
employed, the questions, decisions, concerns, and procedures, with which researchers have
to deal in designing research, are similar. The m ain considerations are always to do with
sampling and measurement. W hether we are doing quantitative research or qualitative
research, we want the answers we arrive at to be m eaningful and say som ething about the
world. We don’t want them to just reflect the way in which we have chosen to measure
political phenom ena or selected particular cases to analyse.
In discussing these different forms of data collection and data analysis, we draw on some
of the most prom inent approaches in political research. We do not discuss every m ethod
used in political research, but a selection of prom inently used ones that can serve to intro­
duce themes and research protocols generalizable to other methods.
We start this section with an introduction to the principles of experim ental research in
C hapter 8. The experimental approach is widely considered to be the most ‘scientific’
research design. Through the use of control groups and experim ental groups the researcher
is able to control what stim uli—or interventions—different subjects are exposed to, and then
examine what impact this exposure has on the political outcome variable of interest. C on­
trolled experiments of this type are very useful for testing causal hypotheses. Broadly speak­
ing, there are three main experimental designs. There are laboratory experim ents (where
subjects are taken to a com mon location), field experiments (which take place in real-world
settings), and natural experiments (which in a sense occur naturally, in so far as the
researcher is not active in the data-gathering process). Despite the scientific potential of
experimental research, the approach is not widely used in the study of politics. However, this
is beginning to change. Experiments are now one of the fastest growing fields of political
inquiry. We discuss some of the obstacles that have been traditionally associated with doing
experiments in political research, and discuss the potential for developing and expanding its
application.
In Chapter 9 we focus on comparative research. Comparative research represents one of
the largest fields of political inquiry, and to a certain extent is used by all investigators who
 POLIT ICAL RESEA RCH

engage in empirical research. The com parative m ethod (or approach or design, as it is som e­
tim es term ed) actually involves a num ber of different m ethods and can be used in conjunc­
tion with any m ethod ol data collection. The logic of comparison is based on how many
countries (or cases) are com pared, and how the cases for analysis are selected. Both aspects
of case selection are very im portant, as the cases you look at can affect the answers you get
to any particular research question (as Barbara Geddes, 1990, has pointed out). Broadly
speaking, there are three main approaches. 'Ihere are large-N studies (involving the analysis
of many cases), sm all-N studies (involving the analysis of a small num ber of cases, typically
2, 3, 4, but with no real upper limit), and single-N studies (otherwise known as case stud­
ies). Virtually all political research falls into one of these three sub-types. One of the key
strengths of com parative research, particularly when it involves the analysis of several or
m ore countries, is that it provides a bridge between looking at domestic factors (which take
place within countries) and international factors (which take place between countries).
Com parison helps us to broaden our intellectual horizons, and we can use comparison to
see if what we think is a self-evident truth in one context also works in the same way in a
different context.
The following chapters explore various m ethods of data collection and analysis. Among
the m ost widely used forms of data collection in political research are surveys, which we
discuss in detail in C hapter 10. One of the great strengths of survey research is that it helps
us to make general claims about what different sections of society or different sub-groups of
the population actually think and do. It thus gives voice to people who might not otherwise
be heard. But a m ajor weakness is that surveys can, and frequently do, misrepresent what
people think and do and thus create misleading inform ation. The extent to which surveys
m isrepresent the ‘real’ or ‘true’ attitudes and behaviour of the people they seek to study can
be thought of as error. The purpose of a good survey is to try and m inimize this error. Ihere
are two im portant sources of error that we consider. The first is to do with m easurem ent
error, and refers to the ways in which surveys use questions to try and measure different
social and political phenom ena, such as political attitudes, opinions, and behaviour. The
second is to do with sam pling error, and refers to the ways in which respondents are chosen
or selected to com plete the survey and the implications this has for the representativeness of
the sample. These principles of sound survey design, to do with sampling and m easurem ent,
are relevant for all form of data collection.
In C hapter 11 we focus on data collection using interview s and focus groups and explore
issues concerning how these data can be analysed. Interviews are in a sense the qualitative
cousin of surveys. Many of the principles are much the same for the two m ethods of data
collection, but whereas surveys are typically concerned with generating large samples so that
they can make valid inferences about a given population, interviews are more frequently
used to ascertain m ore specialized knowledge, either about what so-called experts or elites
think, or to explore the m eanings that people attach to different concepts. It can be a very
useful m ethod to com plem ent survey research, and indeed virtually all survey research
draw s upon sem i-structured interview techniques at the design stage to pilot new questions.
O ne of the great strengths of interviews and focus groups is that they can help a researcher
understand peoples perceptions, feelings, opinions, experiences, understandings, values,
beliefs, attitudes, em otions, behaviour, formal and informal roles, and relationships. Inter­
viewing individuals, either face-to-face or over the telephone, or through mailed questionnaires.
P O L IT IC A L R E S E A R C H

helps researchers to learn about how people feel. The focus group is a good technique for
exploring why people hold certain beliefs or feel the way they do. Exchanges am ong partici­
pants can lead to far m ore probing and reflection than is possible in individual interview s or
questionnaires, and may provide m ore robust and revealing responses to the issues which
are the subject of the focus group.
In C hapter 12 we consider participant observation, the m ost intensive form of data col­
lection of all. The distinctive feature of participant observation, and one of the great strengths
of the approach, is that data collection is carried out in real time. This m eans that the
researcher has a direct, first-hand opportunity to observe what people actually do, w hat they
actually say to each other, and how they actually interact with different institutions or polit­
ical processes, rather than just relying on what people say that they do. Participant observa­
tion (and ethnography m ore generally) therefore has a num ber of characteristics that overlap
with other m ethods we consider (such as surveys, focus groups, and interviews) and a
num ber of characteristics that are distinctive, particularly with respect to the role of obser­
vation. Whereas surveys are based on the ancient art of asking questions to find out what
people think, say, and do, participant observation is based on som ething rather different. It
recognizes that what people say they do, and what they actually do, can be and frequently are
quite different. Accordingly, to get a ‘true’ sense of what people think and say and do, it is not
enough to merely ask people questions and record their answers; it is also necessary to
observe what people do in practice.
In Chapter 13 we consider a different type of data collection. W hereas surveys, inter­
views, and participant observation are all to do with collecting inform ation about what peo­
ple think or say or do, either by asking them questions or by observing what they do, or some
combination of the two, textual analysis is based on the analysis of archival data or docu­
mentary records such as speeches, policy docum ents, or media reports, to do with what
people or institutions or organizations have actually done (or produced). These docum ents
provide a rich source of information about the ways in which politics is practised. We dis­
cuss some of the different ways in which these sources of evidence are analysed, focusing on
historical analysis, discourse analysis, and content analysis. Unlike asking people questions
(e.g. through surveys or in interviews), using texts to collect data has the advantage of being
non-intrusive. Researchers do not face the problem of influencing their data source through
the questions they ask. And they can study past policy positions as they were recorded at the
time. Once recorded, texts do not change.
Quantitative analysis is now one of the most widely used techniques of analysis in political
research. W hether you love it or hate it, it is hard to avoid. And although many students are
apprehensive about quantitative analysis, it is an im portant skill to acquire, which will not
only stand you in good stead for conducting research (both in term s of what you can read
and what you can do), but will also provide you with a transferable skill. In C hapters 14,15,
and 16 we provide a step-by-step guide to quantitative analysis, which will equip you with
the skills to be able to interpret what others have done, and carry out quantitative analysis for
yourselves.
Although it is easy to overstate the differences between these different methods, it is also
important to bear in mind their similarities. All methods of data collection and analysis can
(and frequently do) investigate similar political phenomena. There is no sense in which the
investigation of certain topics or issues determines your method of inquiry. For example, if
 POLITICAL RESEA RCH

you are interested in what people think about politics; you can examine this through the use
of surveys, interviews, and focus groups, and ask questions about what people think and say
they have done. You can also explore this through ethnography and participant observation
and record what they actually do and say. Rather than thinking about how to choose between
different forms of data collection and analysis, the more pertinent issue is often to think
about how to com bine different m ethods.
There is no rule about which m ethod should be used tor which research design. From
lable 1.1 we can see that there is considerable overlap between diHerent research designs,
m ethods of data collection, and m ethods of data analysis. For example, longitudinal studies
may use qualitative historical analysis or quantitative statistical analysis. Ihere is also con
siderable overlap between different m ethods of analysis and different methods of data col­
lection. Although it is com m on to associate quantitative analysis with survey research, there
is nothing to prevent this type of analysis being carried out on data collected using other
methods. Indeed, a great deal of quantitative analysis has been done on interview data based
on expert or elite interviews (such as Polity IV, Freedom House, or Transparency Interna­
tional data), and on docum entary records and public records and media reports (see C hap­
ter 12). Even focus groups can be analysed quantitatively if there are enough of them, lhis
approach is frequently adopted in deliberative polling studies (see Luskin, Fishkin and |ow-
ell 2002), which can be regarded as a type of experim ental focus group. Indeed it is only
really participant observation that does not appear to lend itself to quantitative analysis,
though that is not to say that participant observation doesn't incorporate quantitative analy­
sis or that it cannot be com bined with quantitative analysis.
This all goes to show that some of the oppositions between qualitative and quantitative
approaches have been somewhat overstated. They should not be seen as competing
approaches to political research but as com plem entary approaches. They can both be used
for sim ilar research designs. And they can both be used to analyse similar types of data. It is
not the case that there are some types of data that you can only analyse qualitatively or some
types of data that you can only answer quantitatively. On the whole, there is a considerable
am ount of overlap. So if you are interested in conducting a longitudinal study to investigate

Table 1.1 Research design, data collection, and data analysis in political research

Research Design Method of Analysis

Experimental Comparative Cross-sectional Longitudinal Quantitative Qualitative

Method of data collection

Surveys ^ ^ ^ ✓ ✓ ✓
Interviews * v v ✓ ✓ ✓
Focus groups S ^ ✓ ✓ ✓
Participant S ^ ^
Observation
✓ ✓ ✓ ✓ ✓ ✓
Texts v
Method of data analysis
Quantitative ✓ ✓ ^ ^ _
Qualitative ✓
P O L IT IC A L R E S E A R C H

changing patterns of the tone of political coverage in newspapers, why not em ploy both
qualitative and quantitative research? If different m ethods of data collection can be employed
for sim ilar research designs, and different m ethods of data analysis can be em ployed for
sim ilar form s of data, then the question is not why you should choose one m ethod o f analy­
sis or collection over another, but why you should not choose all over one.
Unfortunately, the answer is usually pragm atic rather than intellectual, and has m ore to
do with constraints of time, money, and expertise. For example, participant observation can
be very tim e-consum ing and surveys can be very expensive. Moreover, once researchers
have acquired one m ethodological skill-set, they are often reluctant to learn another.
Although these constraints are not irrelevant, they should be openly acknowledged and not
be obscured by pseudo-m ethodological,-epistem ological, or even -ontological argum ents.
We hope this book will contribute to lowering at least one of these obstacles, and that it will
enable you to be comfortable using a wide variety of different m ethodological approaches
and skills.
Research m ethods are in a sense the tools of analysis. It is com m on to think about differ­
ent m ethods being a bit like different tools. W hich tool you want to use may then depend
upon the job that you want to do. And whereas one tool m ight be appropriate for a p articu ­
lar type of task, it m ight be inappropriate for another. So a saw is good for cutting things,
but less helpful if you want to fix som ething together. Although there is some tru th in this
metaphor, we think the idea of choosing different m ethods according to the job you want
to do leads to a very narrow view of what constitutes research. We prefer to look at the big­
ger picture. No one wants to go around just using a saw to cut things in half all the time.
Rather, a craftsm an, a skilled carpenter, may want to build a table and to do that he will
need to cut things and fix things and sand things and varnish things. He will need to use a
wide variety of different tools to complete his task. Becoming a skilled researcher is a bit
like learning a craft. And it involves learning many skills and being able to com bine them
together to do a thorough job.
O f course, for the lazy or time-pressed there is always flat-pack furniture. If we buy a flat-
pack table from Ikea, we can assemble it all using a single tool—just an Allen key or a screw­
driver is needed. But the quality of the table is not going to be in the same league as one
hand-built by a master craftsmen, and it may well collapse under a slight am ount of pressure.
If we don’t want our research findings to collapse in a similar way, we should aim to set our
sights a little higher than trying to investigate complicated political phenom ena through the
prism of a single methodological perspective.

Conclusions
One of the great strengths of political research is its diversity. The study of politics encompasses a wide
variety of different ontological, epistemological, and methodological positions. And while it might be
disconcerting or confusing that there are no concrete shared principles that governs the study of
politics, this variety is a great source of vitality within the discipline. Different perspectives act to
continually challenge what we think we know Constant questioning means that we can never take
things for granted We always need to be conscious of whether or not we can defend ourselves from
criticism, since our findings will be scrutinized on many different fronts
 P O L IT IC A L R E S E A R C H

One of the defining character.st.cs of pol.tics as a held of study is the wide variety of approach« rt
incorporates. This can be both a strength and a weakness for the discipline it ,s a strength when this
diversity is embraced, and when researchers adopt and integrate the different approaches and engage
with research from across the methodological spectrum It is a weakness when this diversity fragments
the field, and when researchers from different methodolog.cal traditions retreat into their own
enclaves and do not engage with what other people are doing m the discipline Our view is that to
appreciate the diversity and pluralism within political research, it is useful to be familiar and
conversant with the whole array of methods and approaches available to us Our hope is that this book
will help you in this task.

References
Baber, W and Bartlett. R (2005), Deliberative Research (Princeton. NJ Princeton University
Environmental Politics: Democracy and Ecological Press)
Rationality (Cambridge. M A MIT Press).
Kumar. R (2005). Research Methodology A Step
Brady, H. and D. Collier (eds) (2004). Rethinking by-Step Guide for Beginners. 2nd revised edition
Social Inquiry: Diverse Tooli. Shared Standards (London Sage Publications Ltd)
(Lanham. MO: Rowman & Littlefield).
Laitin. D (2005). Contribution to Symposium I
Buckler, S. (2010), Normative Theory, in D Marsh Ian Shapiro's The Flight from Reality in the
and G Stoker (eds). Theory and Methods m Human Sciences (Princeton University Press.
Political Science. 3rd edition (Basingstoke 2005)', in Qualitative Methods Newsletter
Palgrave Macmillan), 156-80 of the American Political Science Association
Elstub, Stephen (2010), The Third Generation Organized Section on Qualitative Methods 3(2)
of Deliberative Democracy', Political Studies (Fall) 13-15
Review 8(3): 291-307 Luskin. R .J Fishkin. and R Lowell (2002). 'Considered
Finnemore, Martha and Kathryn Sikkmk (2001). Opinions Deliberative Polling in Britain British
Taking Stock: The Constructivist Research Journal of Political Science 32 455-87
Program in International Relations and O’Flynn, I (2006). Deliberative Democracy and
Comparative Politics', Annual Review of Political Divided Societies (Edinburgh Edinburgh
Science 4: 391-416. University Press)
Gerring. John and Joshua Yesnowitz (2006), A Parkinson. J (2006), Deliberating in the Real World
Normative Turn in Political Science?". Polity 38(1): Problems of Legitimacy in Deliberative Democracy
101-33. (Oxford Oxford University Press)

Gillespie. M and B. O'Loughlin (2009), News Media. Potlins. B M (2007), Beyond Logical Positivism:
Threats and Insecurities. An Ethnographic Reframmg King Keohane. and Verba’, in R.
Approach', Cambridge Review of International N Lebow and Mark Lichbach (eds). Theory
Affairs 22(A): 667-86 and Evidence m Comparative Politics and
International Relations (New York: Palgrave
--- J. Gow. and A Hoskins (2007). ‘Shifting Securities;
Macmillan), 87-106.
News Cultures Before and Beyond the Iraq Crisis
2003: Full Research Report. ESRC End of Award Shapiro. I (2005a). The Flight from Reality in the
Report. RES-223-25-0063. Swindon: ESRC Human Sciences (Pnnceton. NJ. Princeton
University Press)
Kahane. H. and N. Cavender (2006). Logic and
Contemporary Rhetoric: The Use of Reason ---(2005b). A Response to Mackie. Hochschild.
in Everyday Life. 10th edition (Belmont CA and Laitin'. in Qualitative Methods. Newsletter
Thomson Wadsworth). of the American Political Science Association
Organized Section on Qualitative Methods 3(2)
King. G.. R. Keohane. and S. Verba (1994). Designing
(Fall): 16-19.
Social Inquiry: Scientific Inference in Qualitative
22 P O L IT IC A L R E S E A R C H

Shively, W. P. (1989), The Craft of Political Research. 2nd Snape, Dawn and Liz Spencer (2006). The Foundations
edition (Englewood Cliffs, NJ: Prentice-Hall, Inc.). of Qualitative Research', in J. Ritchie and J. Lewis (eds.),
Smith, R. (2004), 'Reconnecting Political Theory to Qualitative Research Practice: A Guide for Social Science
Empirical Inquiry, or, A Return to the Cave?, in E. Students and Researchers (Thousand Oaks, CA: Sage
D. Mansfield and R. Sisson (eds), The Evolution of Publications).
Political Knowledge: Theory and Inquiry in American Thompson, D. (2008), 'Deliberative Democratic Theory
Politics (Columbus, OH: The Ohio State University and Empirical Political Science', Annual Review of
Press). 60-88. Political Science. 11(1): 497-520.
Part 1

Philosophy of
Social Science:
Knowledge and
Knowing in Social
Science Research
 Forms of Knowledge:
Laws, Explanation, and
Interpretation in the Study
of the Social World

Chapter Summary
This chapter considers fundamental assumptions that researchers make about how
we can know and develop knowledge about the social world, including assumptions
about the nature of human behaviour and the methods appropriate to investigating
and explaining that behaviour The core concern is whether and how we can pursue
a systematic and rigorous study of social phenomena in the way that scientists pursue
study of the natural world Without considering this issue, it is difficult to design or
structure an approach to research into political phenomena, and to make any claim
with respect to the findings that result from that research
This chapter focuses on three different answers to the question of how to ap
proach the study of social phenomena those offered by positivism, scientific realism,
and interpretivism In exploring the differences among them and their implications
for conducting political research, our discussion will engage with a number of ques
tions. including the following
• What form(s) of knowledge should be the goal of political research7
• Should the social sciences strive to emulate natural science methods, or
is understanding social phenomena something essentially different from
explanation in the natural sciences7
• Can we study politics scientifically7 What does it means to be scientific7
• What distinguishes science from non-science7

Introduction
Every researcher m ust confront fundam ental questions about the nature of knowledge
and how we acquire it. These questions are the focus of key debates in political research,
and the subject o f an ongoing inquiry into scientific practice, form s of knowledge, and
the w orld of politics. W hat sort of know ledge can we gain about the social world? Is it the
sam e sort o f know ledge that scientists are, able to obtain about the natural world? O r are
the form s of know ledge concerning the social world and the natural world necessarily
different? If they are different, is it still possible to produce knowledge that is reliable and
objective? W hat counts as legitim ate knowledge of the social world? These questions
P H I L O S O P H Y OF S O C I A L S C I E N C E

bear directly on research practice and, consequently, are of prim ary concern to those
who seek to understand political processes and structures. The answ er or answ ers you
accept will determ ine the sort of research you pursue, the claim s you m ake on the basis
of that research, and your assessm ent of the findings of the research produced by others
in our field.
We will consider three different approaches to these questions: positivism, scientific real­
ism, and interpretivism. Each approach differs from the others with respect to its ontologi­
cal, epistemological, and methodological premises. These differences are sum m arized in
Box 2.3 in the concluding section of this chapter.
The term s ontology’, epistem ology’, and ‘m ethodology’ relate to fundam ental issues
concerning research practice and knowledge. O ntology is concerned with ‘what is’: with
assum ptions about the nature of the social world and the basic elem ents that m ake up this
world. Q uestions of ontology relevant to political research include w hether the social
world is fundam entally different from the natural world; w hether it is an objective reality
that exists independently of us or is in im portant respects subjectively created. E piste­
m ology is concerned with what is knowable, with what we can know about social p he­
nom ena, and, consequently, what type or form of knowledge we should pursue and treat
as legitimate knowledge about the social world. It is only when we have considered these
ontological and epistem ological questions that we can move to a consideration of m eth ­
odological questions. M ethodology is concerned with how we obtain knowledge, with the
means and m ethods that can provide us with legitim ate knowledge of the political world.
Box 2.1 shows how these key issues concerning knowledge are related.
We begin this chapter with a discussion of the developm ent of positivist thought and
practice, including classical and logical positivism, Karl Popper’s critique of these, and the
role of general laws and causation in social-scientific explanation. We then focus on two
non-positivist positions: scientific realism and interpretivism . The three positions differ
from one another in many ways and, in particular, with respect to their view of how the
assumptions, logic, and m ethods of science can be used by scholars to study hum an behav­
iour. However, though each position has developed, in part, through a critique of the o th ­
ers, each of them produces useful forms of knowledge. Taken together, they have enabled
us to broaden the range and type of questions that political research can effectively address.

bo x 2.1 O n to lo g y E p is te m o lo g y M e fh o d o lo g y

Ontology Epistemology Methodology

What exists' What sort of knowledge What strategies

What is the nature of the of it is possible7 How can we use to gain
social world7 can we know ab outn? that knowledge?
 F O R M S OF K N O W L E D G E

Positivism

As a prelude to our discussion of positivism, it would be helpful to get a sense of its role in
political research by briefly considering behaviouralism and the ‘behavioural revolution* in
the field of politics.
Behaviouralism is the term used for the application of positivism and empiricism to
political research.1 What has been called the ‘behavioural revolution’ was concerned to pro­
m ote the systematic search for sound and reliable knowledge about politics based on a posi­
tivist approach to knowledge. For behaviouralists, political research involves studying and
explaining the observable behaviour of individuals or aggregates of individuals (parties,
classes, interest groups, governm ents, social movements).
Behaviouralist research focuses on the question of what political actors do and why they
do it. Until the mid-1970s, behaviouralist researchers emphasized an inductivist approach to
research which, as we shall see, is associated with classical positivism. An inductive approach
to social inquiry is one in which ‘knowledge is arrived at through the gathering of facts that
provide the basis for laws’ (Bryman 2004: 11). Although behaviouralist research can employ
both quantitative and qualitative data, during the 1950s and 1960s behaviouralist researchers
tended to focus on questions that could be answered by gathering and studying data condu­
cive to exact m easurem ent, as for instance voting data or data from public-opinion polls and
social surveys. This tendency generated the criticism that, by focusing on phenom ena that
lent themselves m ore easily to measurem ent, the field had become preoccupied with tech­
nique rather than substance, and was failing to address significant problems.
These concerns triggered a ‘post-behavioural revolution’. Despite its name, this ‘revolu­
tion was not concerned to displace behaviouralism, but to 'propel political science in new
directions’ (Easton 1969: 1051). Some of these new directions moved the field towards a
further realization of positivist and behaviouralist goals, such as the trend towards 'positive
political theory’ or rational choice theory. Positive political theory assumes that rational self-
interest, *as opposed to attitudes, which are the subject of study in much behavioral research',
provides the motivational foundation for behaviour; and that individual self-interested
rational action com bines to produce collective political outcomes (Amadae and Bueno de
M esquita 1999: 270). But while the post-behaviouralist revolution moved behavioural
research forward, it also set in m otion trends that moved the field in non-positivist direc­
tions, and encouraged the emergence of an array of theoretical approaches that represented
a self-conscious rejection of behavioural and positivist assumptions. Normative theory,
which we will consider in Chapters 3 and 6, witnessed a re-birth, and often self-consciously
as a response to the influence of behaviouralist research. In addition, there emerged a set of
approaches based on non-positivist assum ptions and associated with ‘interpretivism’,
including constructivism , fem inism, post-m odernism , and critical theory.
The behavioural revolution set in motion an important process of discussion and debate
within political research about the methods and goals of the field. It began a discussion on the
desirability and possibility of attaining reliable, empirical, causal knowledge about political life.
It promoted more methodologically self-conscious research; and, though much behavioural
research originally focused on what might be characterized as a narrow range of questions, it
also succeeded in broadening the research domain, as behavioural researchers, seeking insights
P H IL O S O P H Y O F S O C IA L S C IE N C E

from the theories, research methods, and findings of other disciplines, opened the way to
greater interdisciplinarity in the field. Behaviouralism established an emphasis on research
based on empirical observation, testing involving systematic evidence, and falsifiable and
causal explanation. By emphasizing the im portance of research that is capable of replication by
others, behaviouralism makes researchers more precise about what they want to know, what
explanation they are advancing, and how they intend to demonstrate it.
We will gain a better understanding of this revolution, and of both its positivist and n on ­
positivist legacy, as we explore the basic tenets and contours of positivist thought.
Positivism began as a movement to establish a sound basis for social-scientific inquiry.
This is a fundamentally im portant issue in political research. Political researchers want to be
able to offer credible answers to im portant questions, and they are concerned to ensure that
the research practices and m ethods they employ enable them to do this. Positivism offers a
particular approach to resolving these issues. It maintains that it is possible to arrive at factual,
reliable, and objective answers to questions about the social world by employing the m ethods
used in the natural sciences. Depending on your point of view, this position may strike you
as highly controversial or as plain com m on sense. A large num ber of researchers in our field
react to positivist thought in one or the other of these two ways. Consequently, it is likely that
positivism will continue to occupy a central place in our field, both in providing a foundation
for research and in stimulating the articulation of alternative methodological positions.
The term ‘positivism’ was invented by the French philosopher Auguste Comte (1798-1857)
to describe what he saw as the last of three phases in the development of society and its
search for truth. It was Com tes view that society had passed through a theological stage and
then a metaphysical stage; and that now it had entered into a final ‘positive stage in which
the search for truth is characterized by the systematic collection of observed facts. The term
‘sociology’, which refers to the scientific study of the social world, was also his invention.
Both terms expressed the same belief: that the social world could be explained using similar
methods to those used to explain natural phenomena.
Ih is view of social science methodology, in com m on with the other approaches to be
discussed in this chapter, commits us to a num ber of ontological and epistemological claims.
The nature and implications of these claims and their relationship to a positivist m ethodol­
ogy will become clear as we identify and discuss the basic tenets of positivism.
We begin discussion of these tenets by first considering the classical positivist tradition, and
then focusing on the development of positivist thought through the movement o f‘logical posi­
tivism’ and Karl Popper’s critique of logical positivist tenets. In discussing these developments,
our purpose is not to provide an intellectual history of positivism: the ideas of classical positiv­
ism were not superseded by those advanced by logical positivists; nor were those associated with
logical positivism supplanted or displaced by the ideas of Karl Popper. In other words, the devel­
opment of positivism over time did not always or usually lead to the wholesale rejection of
previous ideas, but rather to an expansion of the array of positions associated with it.

Classical positivism

\hc first tenet ol positivism—one implied by our previous discussion—is naturalism . Natu­
ralism is the idea that there are no fundamental differences between the natural and the social
sciences. Note that this idea entails an ontological presupposition about the social world: if
 F O R M S OF K N O W L E D G E

there is no difference between the social and natural sciences, it must be because there is no
fundam ental difference between the social and natural worlds. Both claims provide positivism
with a basis for building a larger edifice of thought concerning the nature and goals of social-
scientific inquiry. As we shall see, positivism maintains that, since the social sciences are no
different from the natural sciences, they should have the same structure and logical character­
istics as the natural sciences. We II return to this notion in a moment when we discuss the third
tenet of positivism. But first let’s consider a second tenet of positivism: empiricism.
Kmpiricism is a philosophical theory of knowledge which claims that what we know of
the world is lim ited to what can be observed. Knowledge is only that which originates in
sensory experience: there is no a priori knowledge, no knowledge of reality that is acquired
prior to sense experience. So, an empiricist epistemology com mits positivism to ihe view
that social reality can only be known through what is observed and that knowledge of the
social world is therefore limited to phenom ena that can be observed by the senses. Positivists
m aintain that social science should be empirical, based on evidence that is visible in the
world. Its goal should be to gain knowledge of social reality through concepts which apply to
or derive from what is observable and measurable.
Additional tenets o f positivism provide further elaboration of its position concerning the
basis of knowledge and the form it takes. Consider a third tenet of positivism: that the goal
of social science is to explain and predict social phenom ena by means of laws. Ihe German
logician Carl Gustav Hempel (1905-1997) argued that if the discovery of laws is necessary
in the physical or natural sciences, then laws must be necessary also in social science. If the
social world is like the natural world, then, like the natural world, it also must be regular,
systematic, and law-governed. There are regularities in, and ultimately laws of, social and
political processes; and we can explain social events and phenom ena by means of law-like
generalizations that have the same status as natural scientific laws.
The possibility o f discovering laws in the social sciences is one of the key issues on which
positivism and its critics divide. As we shall see, there is considerable debate concerning
w hether social laws exist. Some non-positivist approaches insist that there is a difference “ in
kind” between the subject m atter of natural and of social science, which precludes the use of
laws in the explanation of hum an behavior and makes it impossible to establish social laws’
(M cIntyre 1994: 131). We will be considering this view later in the chapter.
We have said that positivism holds that the social world is regular, systematic, and law-
governed, like the natural world; that social phenom ena can be explained and predicted by
m eans of laws that have the same status as natural scientific laws; and that the purpose of
social science, therefore, is to discover these laws. But how do we go about discovering laws
o f social life? Classical positivist thought m aintains that laws can be discovered through
systematic investigation of observable events and happenings, and by means of inductive
reasoning. Induction is a means of reasoning that begins with specific observations and
m easures. It moves to an identification of patterns and regularities and to the formulation of
som e tentative hypotheses that can be explored; and it ends by developing some general
conclusions or theories. An inductive approach to social inquiry is, as we noted earlier, one
in which ‘knowledge is arrived at through the gathering of facts that provide the basis for
laws’ (Brym an 2004: 11). We will have m ore to say about induction and other means of rea­
soning (i.e. 'deduction and ‘abduction’) further on in our discussion of the development of
positivist thought.
P H IL O S O P H Y O F S O C IA L S C IE N C E

We are still discussing the third tenet of positivism: the view that explanation of social
phenom ena should proceed by the discovery of laws. But, for positivism , there is another
key element in social science explanation: explanation must not only proceed with reference
to law-like generalizations, it m ust also establish a cause-effect relationship between events
in the world. Positivism sees the social world as com prising phenom ena that are causally
related to each other; consequently, to explain a social outcom e we are required to show the
factors or conditions that com bined to bring it about or caused it to be m ore likely to occur
in the circumstances.
Virtually all social research is concerned to discover causes. But there are different con­
ceptions of causation. The positivist conception of causation is an em piricist conception
which was introduced by the Scottish philosopher, economist, and historian, David Hum e
(1711-1776). Most of us probably carry in our m inds an idea of causation as a relation
between two events, the cause and the effect, which expresses some type o f ‘necessary con­
nection’ between them. But Hume pointed out that we cannot directly perceive causal rela­
tionships. He points out that ‘when we look about us towards external objects, and consider
the operation of causes, we cannot in any instance discover a power, necessary connexion,
or quality which binds the effect to the cause and renders one an infallible consequence of
the other’ (1966: 51). Instead, we observe only the ‘constant conjunction of events; we
observe only that one thing follows the other. O ur experience of observing this ‘c onstant
conjunction’ between events conveys to our minds a necessary relation between these events.
So the causal conclusions we reach are based, not on ‘knowledge of causal mechanism s and
the generative properties of things’ but only ‘on the observation of how a certain event is
followed again and again by a certain other event’ (Ekstrom 1992: 108).
According to this conception, then, causation is constituted by facts about empirical
regularities among observable variables. There is no underlying power or necessity deriving
from the laws of nature. All we can do is observe that one thing follows another with regular­
ity; and, because of this observation, we develop a psychological expectation that Y will
occur whenever X does. But we cannot know that X is the cause of Y by observing that X is
constantly followed by Y. Consequently, in establishing the basis of causal explanations,
positivists are concerned with observing empirical regularities rather than in discovering
causal mechanisms. This is a subjective conception of causation: causation as a perceived
regular association among variables. An objective conception of causality, one involving
causal necessity or causal mechanisms is, according to positivism, metaphysical. This objec­
tive conception of causation features prom inently in the critique of classical positivism artic­
ulated by logical positivism, a subject to which we will turn next. But before moving on, we
need to briefly note a fourth tenet of positivism: that it is possible to make a distinction
between facts and values.
Positivism m aintains that we can gain knowledge of the social world through applica­
tion of the scientific m ethods used in the natural sciences. According to this fourth tenet
of positivism, the pursuit of knowledge through these m ethods can be value-free or
objective, because statem ents of fact (confirm ed by the senses) can be distinguished from
normative statements. Science is concerned with the discovery of facts, whereas values
relate to ethics or policy studies. Ihe argum ent that it is possible to distinguish between
facts and values, and to treat facts as independent of the observer and of his or her val­
ues, represents a key difference between positivists and adherents of alternative
approaches. However, we will leave discussion of this issue for the time being, since we
Figure 2.1 Induction
Sounr I f IH h i m AHX,

will he exploring it in some tlc lail in ( .hapter <. Instead. wc turn to a consideration (.1 the
further developm ent ol positivist thought as a result o! logical positivism and Karl l \ r
per's critique of it.

E m p iricism and logic as th e b asis o f truth claim s

I he ideas ol classical positivism were developed In a movement that adopted the name
logical positivism ’, as well as by the highly mlluential critiqm- ol Kail Popper.
Logical positivism began in the early twentieth century as a movement within philos
ophy. Inspired by developm ents m twentieth century logic and m athem atu s. its goal was to
introduce logical reasoning and mathematic s as sources ol knowledge m addition to einpir
icism. It advanced the idea that social inquiry should combine induction (based on empiri
cism) and deduction (in the lorm ol logic) as methods ol reasoning
We have previously discussed induction as a means ol discovering laws Induction, you
will recall, is a process ol reasoning from particular tacts to a general conclusion As figure
2.1 shows, in induction we begin with particular observations or cases and then develop
generalizations about them. D eduction works the other way around. As figure 2.2 shows,
deduction moves Irom broader generalizations and theories to specific observations. We
start, not with an observation, but either with a theory that has already been continued or
with a logical argum ent, and then we draw out the m eaning or implications this has lor
explaining some particular case or phenomena.
To digress Irom our discussion of logical positivism lor a moment, it should be noted that,
in practice, researchers do not use solely one m ethod or the other Scientific inquiry typically
involves a process of continuous interaction between theory and observation, in which the
researcher moves from observation to theory (induction) and Irom theory back to observa

b o x 2.2 In d u c t io n , D e d u c tio n , R e tr o d u c tio n

Induction observation ® theory

Deduction theory c* observation

Introduction observation ■theory

P H IL O S O P H Y O F S O C IA L S C IE N C E

Theory

Hypothesis

Figure 2.2 Deduction

Source Trochim2006.

tion (deduction). Box 2.2 illustrates how this process contrasts with and combines induction
and deduction. The compiling of evidence (induction) leads the researcher to theory (deduc­
tion); and once a hypothesis is formed, the researcher brings it ‘backward’ for readjustm ent
or redefinition. The term ‘retroduction’ describes this interaction of induction and deduc­
tion in an evolving, dynamic process of discovery and hypothesis formation.
We have said that logical positivism introduced the idea that social inquiry should combine
both induction and deduction. It also established ‘verification (of statements or propositions) as
the goal of social science research. Verification was held to be the main criterion for establishing
truth claims and a means of defining a clear line of division between science and metaphysics.
Both of these tenets of logical positivism became the target of a critique by Karl Popper
(1902-94), a philosopher of science who also wrote extensively on social and political philosophy.
Popper’s critique had a decisive impact on social-scientific thought. In fact, its influence was so
great that logical positivisms most important contribution to social science, it might be argued,
is the role it played in having served as the focus of this critique. This does not diminish its con­
tribution: in the quest to establish a sound basis for scientific inquiry, logical positivism raised
important questions about the concepts and practices of science which continue to have rele­
vance for social-scientific inquiry today. Moreover, while Popper was a critic of logical positiv­
ism, there are also many affinities between his views and those held by logical positivists.
Logical positivists had argued that both inductive and deductive m ethods of reasoning
should be used to acquire knowledge of social phenom ena. But Popper argued that induc­
tion must be rejected entirely. Moreover, the argum ent he advanced for rejecting induction
also provided grounds for rejecting verifiability as a basis for establishing truth claims.
Popper elaborates these arguments in his book, Logik der Forschung, published in 1934,
and later published in English under the title The Logic o f Scientific Discovery (1959). The
book addresses two issues. The first is what David Hume calls ‘the problem of induction’. The
problem is whether experience can provide the basis for gaining general theoretical knowl­
edge. Since experience is particular, while knowledge is general or even universal, how do we
achieve universal knowledge on the basis of particular experience? How can we reach gen­
eral statements of scientific law on the basis of experiences that are necessarily limited and
particular? Popper argues that no matter how m any experiences we have of observing som e­
thing, this does not permit the deduction of a general statement of scientific knowledge.
Ihe reasoning that leads him to this conclusion begins with David Hume’s argum ent
about the limits of inductive reasoning. Hume argued that since we cannot observe the uni­
verse at all times and in all places, but are only able to observe particulars, we are not justified
in deducing general laws based on inductive evidence. Poppers now famous story of the
 F O R M S OF K N O W L E D G E

black swan illustrates what happens when we attempt to formulate laws based on observa­
tion. Ih e story is that, once upon a time, Huropeans thought that all swans were white because,
having found nothing but white swans for thousands of years, Huropeans concluded on the
basis ot their experience that all swans were white. But one day Huropeans went to New
Zealand (as Popper had), and there they found black swans. What this story tells us is that no
m atter how many observations confirm a theory, it only takes one counter-observation to
falsify it: only one black swan is needed to repudiate the theory that all swans are white. And
since it only takes a single unforeseen or seemingly improbable event to invalidate a gener­
alization based on em pirical observation, then empirical observation alone cannot generate
‘laws’. Popper therefore concludes that, rather than endeavouring to discover laws through
induction, what scientists should be doing is testing theory deductively.
Poppers critique of induction leads him to reject another tenet of logical positivism: the
notion that scientists should seek to verify hypotheses. Popper argues that, since a single
exception to the rule destroys inductively generated theory, then conclusive verification of a
hypothesis is not possible. So Popper proposes that we reverse the logical positivist assum p­
tion about verifiability: he argues that rather than continually attem pting to prove a theory,
scientists should attem pt to disprove it. Since we cannot verify a hypothesis , our aim should
be to falsify it. We should form ulate propositions in such a way as to enable them to be
refuted. By doing this, it will be possible for us to show a theory to be wrong; and we can then
introduce a new theory which better explains the phenom ena. 'Ihis, Popper argues, is how
we achieve progress in science.
The notion of falsifiability is the basis of Popper’s argument concerning the second issue
he addresses in The Logic o f Scientific Discovery: the problem of demarcation. Ihis refers to
the problem of determ ining how to differentiate science from non-science. It is a key problem
in the philosophy of science and the subject of ongoing debate. For Popper, it is falsifiability
—and not verifiability, as logical positivists argued—that defines the boundary between
science and pseudo-science or metaphysics. Anything non-falsifiable is outside science.
C onsider religions and ideologies in this regard. Religions and ideologies are logically
consistent statem ents which provide a guide for understanding the world. But they cannot
be proved false: potentially disconfirm ing or anom alous facts do not prove them false, but
are incorporated within them . A scientific theory, however, must state what evidence would
disconfirm it or prove it to be false. If you cannot think of anything that might disconfirm a
theory, then it is not a theory at all but a set of self-verifying statem ents—an ideology.
To sum up: in rejecting induction, Popper was rejecting the idea that observation pro­
vides the basis for the form ulation of scientific theories. Theories cannot be derived from
observation (induction), because at any tim e a single observation can disconfirm the theory.
Popper concludes that social inquiry must proceed deductively, through a process in which
observations are not the basis of theories, but are derived from and used to ‘test’, or falsify,
them . According to Poppers notion of falsifiability, we endeavour to falsify hypotheses. We
reject those which are falsified and we continue to test those that are not until they become
so thoroughly tested that we can consider them to be ‘confirm ed’, though it remains possible
that som e day som eone may falsify or significantly m odify them.
Two objections have been m ade to this formulation. The first objection is to the distinc­
tion which Popper seem s to m ake between facts and theories. Popper seems to assume that
the observations or facts that we pursue as a m eans of testing theories can be established
P H I L O S O P H Y O F S O C IA L S C IE N C E

independently of the theory that they are m eant to test. We consider the debate concerning
this issue in some detail in C hapter 3. A second objection is that Poppers notion of falsifia-
bilty is at odds with how scientists actually go about developing and testing theories. Do
researchers seek to disprove or falsify their own theories? Do they discard their theories
when they are confronted with disconfirm ing evidence? We will consider Thom as Kuhns
and Imre Lakatos’ answers to these questions, and the further evolution of Popper s falsifica-
tionist position that developed as a response to them , in C hapter 3.
Here, however, we have still to consider a further question: how do we use deductive reason­
ing to discover laws of social life as a basis for explanation? Previously we have discussed the
classical positivist approach to explanation: inductive reasoning based on systematic investiga­
tion of observable events and happenings. As we have seen, logical positivists maintain that both
induction, based on empiricism, and deduction in the form of logic could be used to discover
laws. Popper argues that we can establish laws of social life as a basis for explanation only through
deduction. But, what is the process through which deduction operates as a means of explaining
social phenomena? For the answer to this question, we turn, again, to Carl Gustav Hempel.
Hempel maintains that explanation in the social and natural sciences is the same, not only
because both involve the search for and discovery of law-like generalizations, but because
the social and natural worlds are subject to laws in the same way (see Hempel 1994). The
logic and function of laws, what Hempel calls ‘general laws’, are the same. In both the natural
and social sciences, individual events can be subsum ed within hypotheses about general
laws of nature: what this means is that to explain some fact is to cite some law or laws and
other conditions from which the fact can be deduced.
Hempel formalizes this definition of explanation in his deductive-nom ological m odel.
A deductive-nomological explanation is deductive because the phenom enon to be
explained (explanandum) is logically deducible from that which does the explaining (the
explanans)-, and it is nomological because the explanans includes at least one law (‘nom os’ is
the Greek word for law). According to this model, then, som ething is explained when it is
shown to be a m ember of a more general class of things, when it is deduced from a general law
or set of laws. A full explanation of an event requires that we give an account of how a phe­
nomenon follows deductively from a well-confirmed general law. For instance, ‘To explain
fully why an actor votes (a “fact”) we must do more than just isolate the particular cause of this
particular event (for example, the intensity of the voter’s concern about unemployment). We
must subsume this act of participation under a “law” that explains why, under certain condi­
tions, the voter had to vote: “persons with intense preferences for candidates or issues”, every­
thing else being equal, will become “active in politics’” (Milbrath 1965: 53; quoted in Schwartz
1984: 1123). Given the general law, the particular case in question was to be expected.
But, how do we confirm a regularity or generalization that what we take to be a ‘law’ is, in
tact a law? A regularity might be true, accurate, or supported by evidence; but it might be
only accidentally true’: true only as a result of circum stance or coincidence. Explaining how
to distinguish law-like generalizations from those that are ‘accidental’ is one of the central
problems in the philosophy ol science. However, in general, we can say that a law expresses
a necessary connection between properties, while an accidental generalization does not. If
a necessary connection exists between its properties, then we should be able to test a law by
its ability to predict events. II we predict som ething on the basis of a law and find that the
prediction was true, then the law can be said to be confirmed.
 f O R M S OF K N O W L E D G E

Ihis is what (,arl Hempel proposes that we do in his hypothetico deductive model of
confirm ation. We confirm that the generalization is a law (rather than an accidental gener
alization) by treating it as a hypothesis and then we test the hypothesis by deducing from the
hypothesis a sufficient num ber of explicit predictions of further phenomena that should be
observable as a consequence of the hypothesis. Observations that run contrary to those
predicted are taken as a conclusive falsification of the hypothesis; observations which are in
agreem ent with those predicted are taken as corroborating the hypothesis. It is then suppos­
edly possible to com pare the explanatory value of competing hypotheses by looking to see
how well they are sustained by their predictions.
An exam ple of what is regarded as a law or. at least, a law like generalization in our
field is Duverger's Law. Ihe sociologist. M aurice Duverger, proposed that the plurality
rule for selecting the w inner of elections favours the two party system. Duverger offers
two theoretical explanations for why a plurality rule election system tends to favour a
tw o-party system . Ihe first is the m echanical effect’ of under representing losing parties;
and the second is a ‘psychological factor’: voters don't want to waste their votes on losers
(Riker 1982: 761). W illiam Riker explains: ‘when the definition of winning forces cand i­
dates to m axim ize votes in order to win (as in plurality systems), they have strong motives
to create a tw o-party system; but when the definition of w inning does not require them
to m axim ize votes (as in runoff and proportional systems), then this motive for two par­
ties is absent’ (Riker 1982: 755).-’
To sum up: the deductive-nom ological model holds that an observed phenom enon is
explained if it can be deduced from a law-like generalization. Ihe hypothetico-deductive
m odel confirm s that a generalization is a law by treating the generalization as a hypothe­
sis, and testing it by its deductive consequences. To explain some fact is to cite a law or
laws plus other relevant conditions from which the cxplattatuium may be deduced (the
deductive-nom ological m odel of explanation). To confirm a hypothesis is to deduce some
observed phenom enon from the hypothesis plus other relevant known conditions (the
hypothetico-deductive model of confirm ation).
We have traced the development of positivist thought through a consideration of the basic
tenets of classical and logical positivism and the argum ents advanced by Karl Popper. Wc
turn now to approaches that emerged as a challenge to positivist thought and research.

Challenges to positivist approaches

within the social sciences
There are a num ber of approaches to social inquiry that challenge the positivist position and
that articulate a fundam entally different basis for inquiry. Here, we focus on two alternative
positions—those represented by scientific realism and interpretivism.

Scientific realism
Scientific realism is concerned to elaborate a non-positivist version of science, one that its
adherents claim is more scientific than positivism. Their message, as Ruth Lane puts it (Lane
1996: 373), is that we don’t have to be positivists to be scientific!’
P H I L O S O P H Y O F S O C IA L S C IE N C E

Scientific realism appears to be sim ilar to positivism in som e ways because it accepts a
num ber of assum ptions of positivism that other non-positivist approaches reject. For
instance, scientific realism assumes, like positivism, that the social and natural worlds are
essentially similar, and that the social and natural sciences are therefore fundam entally sim ­
ilar, as well. These assum ptions are based on another shared assum ption: realism . Realism
holds that the world exists independently of our knowledge of it, that reality has an inde­
pendent existence (it exists independently of hum an beings and their perceptions), and that
it impacts directly upon the hum an m ind without any reflection on the part of the hum an
knower. We can therefore gain objective knowledge of the world because our knowledge of
it is directly determ ined by an objective reality within the world.
So, positivism and scientific realism share some key assum ptions. However, there is a key
difference between the two approaches—and it is an im portant one! Let s recap for a m om ent
before stating this difference. Both approaches m aintain that the subject m atter of scientific
research and scientific theory exists independently of our knowledge of it, that we can there­
fore gain objective knowledge of it, and can treat ‘facts’ as independent of the observer and
of his or her values. Now, where the two approaches differ is that, while positivists m aintain
that reality consists of only that which we can directly observe, for scientific realists, reality
consists of observable elements as well as observable ones.
You will recall that positivists assume that statem ents not based on observable data are
metaphysical. Scientific realists break decisively with this assum ption. They assum e that
there are knowable, m ind-independent facts, objects, or properties that cannot be directly
observed but which are, nonetheless, real. They argue that unobservable elements of social
life, such as structural relations between social phenom ena, are crucial to an understanding
and explanation of what goes on in the world. They point out that the central role of unob­
servable elements in shaping outcomes is one of the features that makes the social world
similar to the natural world; that this ontological conception of the social world is not m et­
aphysical, but more scientific, and more closely aligned with the tenets of the natural sci­
ences, than the positivist conception.
Consequently, for scientific realists, the goal of scientific research is to describe and
explain both observable and unobservable aspects of the world. But how do we know
these unobservable elem ents exist? According to scientific realism, we know they exist
because we can observe their consequences: unobservable elem ents of social life can be
treated as ‘real’ if they produce observable effects. To posit the existence of unobservable
entities to explain observable outcom es is consistent with well-established scientific prac­
tice. We treat gravity and subatomic particles as real because, even though we cannot see
them, we can see their effects. Similarly, there are many elem ents in social and political life
that are not directly observable—social structures, capitalism , society—but they have
observable effects; and because their effects are observable, researchers in our field treat
them as real.
Cliven these assumptions, it follows that, for scientific realists, scientific knowledge does
not take the form solely of empirical regularities, and scientific research cannot be solely
concerned with the goal of formulating law-like generalizations based on observations. To
state this differently, if scientific realists reject the notion that only entities of which we have
direct sensory experience are real’, then they cannot depend on an epistemology that places
emphasis on direct observation for pursuing knowledge of the social world.
 F O R M S OF K N O W L E D G E 37

It follows that scientific realists also cannot accept the empiricist I H um ean) conception of
causality that positivists employ. Recall that positivists treat causation as constituted by lads
about em pirical regularities among observable variables, and seek to establish causal rela
tionships by observing these regularities rather than by discovering causal mechanisms.
I hey treat the notion that causal m echanism s produce outcomes in social life as m etaphysi­
cal, since we are unable to have knowledge of causal mechanisms through direct observation
only. But scientific realists assume that unobservable elements are part of realitv and are
knowable, and so they treat causal m echanism s and causal powers as real', as a legitimate
object of scientific investigation, and as fundam ental to explanations ot social outcomes 1or
scientific realists, explaining social outcomes entails providing an account of the causal
m echanism that brought about a given outcome; and with developing empirically lustified
theories and hypotheses about causal mechanisms.
A causal m echanism can be defined as ‘a series ot events governed by lawlike regularities
that lead from the explanam to the explanamlum (Little 1991: 15); or the pathwav or proc­
ess by which an effect is produced or a purpose is accomplished' (C .erring 2007: 17K) C liarles
I illy identifies three sorts of mechanism. Environmental mechanisms are externally gener­
ated influences on conditions affecting social lile', cognitive m echanisms operate through
alterations of individual and collective perception’; and relational mechanisms alter connec­
tions am ong people, groups, and interpersonal networks' (2001: 24). Michael Ross defines
an environm ental m echanism —the nature of a governm ent’s resource base—to explain the
apparent link between oil exports and authoritarian rule. He calls this mechanism a rentier
effect': ‘when governm ents derive sufficient revenues from the sale of oil, they are likely to
tax their populations less heavily or not at all, and the public in turn will be less likely to
dem and accountability from —and representation in —their government' (2001: 332).
Explanations of a variety of political outcom es might link them to the effect of increases or
decreases in the governm ent’s resource base (see e.g. Chapter 6 ). Cognitive mechanisms
have been identified to explain ethnic conflict as, for instance, changing conceptions of
racial, ethnic, gender, religious, or class differences (e.g. Hoffmann 2006). Relational
m echanism s, such as governm ental absorption and destruction of previously autonom ous
patron-client networks, or bureaucratic containm ent of previously autonom ous military
forces, have been held to effect the likelihood of civil war, the level of domestic violence, and
even the prospect that a given state will engage in international war’ (Tillv 2001: 38). Robert
Gilpin (1981) has argued that there is a tendency for a disjuncture to arise between the costs
and benefits of hegemony, and that, when it does, the hegemonic state begins to decline.
Gilpin identifies a num ber of mechanism s that cause this disjuncture. One is the law of the
increasing costs of war’: military techniques tend to rise in cost, and the increasing cost of
war produces a fiscal crisis within the hegemonic state. Another is the law of expanding
state expenditures’: private and public consum ption grows faster than the GNP as a society
becom es m ore affluent (the rich indulge increasingly in lavish consumption; the poor begin
to clam our for welfare).
Those w ho em phasize the im portance of m echanism s in causation have different views
about the nature or types of social m echanism s that operate to produce social outcomes.
These are linked to different assum ptions about what we should treat as the basic unit of
analysis in social inquiry. So, for instance, those who treat individuals as the basic unit of
social analysis favour agent-based m odels, or individual-level m echanism s to explain
P H IL O S O P H Y O F S O C IA L S C IE N C E

outcom es. This is characteristic of rational choice approaches, which assum e that the
instrum ental rationality of individuals is the causal m echanism that produces social o u t­
comes. Structural models, on the other hand, attem pt to dem onstrate that there are struc­
tural or institutional m echanism s that cause social outcom es. In C hapter 4 we will be
exploring the differences between the individualist and collectivist (or holist) ontologies
on which these different m odels are based. We will also discuss how what Peter H edstróm
and Richard Swedberg (1998) call ‘social m echanism s’ produce outcom es through m acro­
micro interactions and linkages.
We have been discussing the assum ptions of scientific realism with regard to basic ques­
tions concerning the nature of the social world, forms of knowledge, and the goals of social
science. We have said that, for scientific realists, the goal of scientific research is to describe
and explain both observable and unobservable aspects of the world. It still rem ains to say
how scientific realists establish that claims regarding unobservable social phenom ena are
true.
Scientific realists argue that knowledge of unobservable elements of social life can be
obtained through the development of theoretical constructs. But how can we know w hether
our theories about unobservable elements of social life are true? The answer is that we can
accept as true the theory or hypothesis which, from am ong those that have been advanced
to explain a phenom enon, offers the best explanation. The ‘best’ explanation or hypothesis is
the one that, based on various ‘rules of m ethod’, explains a fact better than other available
hypotheses. For instance, it may be ‘best’ because it has been tested and not refuted, while the
others have not; because it accounts for more, or better meets the standard of explanation we
accept for other phenomena. ‘If a theory is certified by such rules of m ethod, a scientist is
rationally justified in accepting the theory’ (Sankey 2008: 28). Scientific realism m aintains
that we can accept that a theory is true if there is rational justification for accepting it to be
true; and it is rational to accept as true the best available explanation of any fact.
This position is summed up in the phrase ‘Inference to the Best Explanation’. Scientific
realists maintain that inference from some data to the ‘best explanation’ justifies our accept­
ance of a hypothesis as true. By inference we mean the reasoning involved in the process of
drawing conclusions based on facts or logical premises; and, according to scientific realists,
the kind of inference that justifies our accepting a hypothesis as true emerges from a type of
reasoning called ‘abduction’. Contem porary philosophers use ‘inference to the best explana­
tion’ and ‘abduction’ interchangeably.
Abductive reasoning is prior to and distinct from induction and deduction. Abduction
starts with a hunch that a set of seemingly unrelated facts are connected in some way. The
hunch or hypothesis can then be affirmed by induction or deduction. Abduction may be
used to explain singular events rather than, as in inductive reasoning, to form generaliza­
tions on the basis of a large number of token instances; and, unlike induction, it can employ
both observables and unobservables to explain events. Abductive reasoning requires that we
choose Irom among competing explanations the best available explanation: the one that best
explains a particular event or phenom enon given all the available evidence. The abduction is
provisional: new evidence may later underm ine it. But it is reasonable for us to believe it if it
is the best explanation we have. We may later find out that the explanation is wrong and then
it will no longer be reasonable for us to believe it; but it remains the case that it was not wrong
or unreasonable tor us to have believed it prior to our finding out that it was wrong. W hen a
 F O R M S OF K N O W L E D G E

detective infers that a m urder was com m itted by a suspect, he does so because this hypoth­
esis provides the best explanation for the m urder; and it is the best' explanation because it
fits better with the forensic evidence, and/or provides a better account of motive and oppor­
tunity. Later inform ation may reveal this explanation to be false; but this does not make it
unreasonable for the detective to have made the original inference.
We now have introduced some of the key terms of reference in a continuing debate about
the scientific status of unobservable elements of social life. Because political researchers
continually refer to unobservables such as ‘society’ and ‘structures of power' to explain polit­
ical events and processes, we will be returning to this debate in later chapters. But, before
moving on from discussion of scientific realism to a consideration of other non-positivist
approaches, it is worth noting a related position that has emerged within the field of politics,
critical realism .
We have said that scientific realism, like positivism, assumes that there exists a reality
separate from our description of it. Critical realism represents a move away from this posi­
tion. As we have seen, scientific realism is committed to identifying the unobservable struc
tures that work to generate observable outcomes. Ibis, as Roy Bhaskar points out, is critical
in that it opens up the possibility or our being able to change our world (1998: 2). But critical
realism also rejects the view, accepted by scientific realists and associated with what adher­
ents of critical realism call ‘naive realism’, that the external world is as it is perceived. Instead,
it holds that perception is a function of the hum an mind, and that we can therefore only
acquire knowledge of the external world by critically reflecting on perception. While some
political researchers see the term s ‘scientific realism’ and critical realism’ as synonymous
(see e.g. Brown 2007: 409), this would seem to be a position that moves us further in the
direction of the interpretivist approaches that we will be discussing next. Some examples of
how a critical realist position informs political research are in Chapter 13, where we discuss
critical discourse analysis, a type of textual analysis inspired by, and to a large degree consist­
ent with, a critical realist philosophy of science.
We have been discussing scientific realism, an approach to social inquiry that provides an
alternative to positivism. As we have seen, its main difference with positivism is that it does
not place emphasis on direct observation in pursuing knowledge of the world; rather, it
assumes that reality consists of both observable and unobservable elements. We have also
seen that there are some respects in which scientific realism and positivism are more similar
than dissimilar. For instance, both agree that the world exists independently of our knowledge
of it. This assum ption has im portant implications for what we treat as legitimate knowledge
and how we conduct research. We turn now to a set of approaches that break decisively with this
assumption, and promote ontological and epistemological positions that stand in diametric
opposition to those of positivism.

Interpretivism
Interpretivism m aintains that the social world is fundam entally different from the world of
natural phenom ena, and so we cannot understand it by em ploying the m ethods used to
explain the natural world. It argues that it is impossible for us to gain knowledge of the social
world by searching for objective regularities of behaviour that can be sum m ed up in social-
scientific laws analogous to the laws of physics, because the social world does not exist
P H IL O S O P H Y O F S O C IA L S C IE N C E

independently of our interpretation of it. The social world is what we experience it to be: it
is subjectively created. The task of social science, then, is fundam entally different from that
of natural science, because the objects of the social sciences are different from those found
in the natural world. Social phenom ena are socially or discursively constructed; so we
cannot explain and predict social phenom ena by m eans of laws. The prim ary goal of social
science m ust be to achieve an understanding of hum an behaviour through an interpretation
of the meanings, beliefs, and ideas that give people reasons for acting.
Lets consider the implications of this view for how we conduct political research.
Recall our earlier discussion about behaviouralism . Behaviouralist research is positivist
and empiricist. Its concern is with the question of what political actors do and why they do
it. It seeks to discover the causes of behavioural outcom es by understanding the m otivations
of political actors. It uses public-opinion polls and social surveys to learn about the beliefs,
attitudes, and values that motivate behaviour; or rational choice theory to explain how indi­
vidual self-interested rational action motivates behaviour. However if, as interpretivists
contend, people act on the basis of the meanings they attach to their own and to others’
actions, then understanding hum an behaviour requires an understanding of these m ean­
ings. Consequently, social science must be concerned, not with discovering causes of social
outcomes, but with piecing together an interpretation of the m eanings of a social outcom e
or production.
Intepretivists seek to understand hum an behaviour through interpretation and inter­
pretive theory. These are form s of social science ‘that em phasize understanding the
m eaning that social behaviour has for actors’ (G ibbons 2006: 563). These form s include a
m ultiplicity of approaches, most notably herm eneutics, cultural anthropology, verstehen
social theory, critical theory, and post-structuralism . In what follows, we will focus on
herm eneutics as a m eans of highlighting the differences between interpretivist and p osi­
tivist approaches. We will then consider how approaches based on interpretivist and
positivist assum ptions analyse a specific area of political inquiry.
‘Hermeneutics’ originally referred to a m ethod used to interpret theological and legal
texts. In fact, ‘the literal English translation of the Germ an word “herm eneutics” is interpre­
tation’ (Gibbons 2006: 563). Today, ‘herm eneutics’ refers to theories and m ethods that are
used in the interpretation of texts of all kinds. These texts include not just written docu­
ments, but any object or practice that can be treated as a text and which can, therefore, be the
subject of interpretation. But can human beings and their actions be treated as a text and the
subject of hermeneutical interpretation? Interpretivists argue that they can. Herm eneutics
can be used to study behavioural outcomes because, if behaviour is a product of the m ean­
ings and intentions employed by social actors, then the social scientist endeavouring to
understand that behaviour is involved in an interpretive exercise not unlike that engaged in
by the translator of a text.
Ihe philosopher Charles Taylor elaborates this argument in an influential essay entitled
Interpretation and the Sciences of Man’ (1994). Taylor explains that any field of study can be
the object of hermeneutics if it meets two requirements. First, it must contain an object
or a field of objects that is a text, or a ‘text-analogue’. Second, this text must be ‘confused,
incomplete, cloudy' or seemingly contradictory’; that is, it must be in some way ‘unclear’.
When these criteria are met, hermeneutical interpretation can be used ‘to bring to light an
underlying coherence or sense with respect to the objects defined by the field of study and,
 F O R M S OF K N O W L E D G E

in th is way, enab le us to un d erstand them (Taylor 1994: 181). D oes the study of politics meet
th ese criteria ? Yes. W e can treat the behaviou r we are concerned to un d erstand — the actions
of a g ov ern m en t, or the behav iou r of m em b ers of a group tow ards one an o th er— as texts';
a nd, sin ce the m otives and goals o f this behaviou r are often unclear or at odds with the p ro ­
n o u n cem en ts of the p olitical actors involved, we can use in terpretative m ethods in order to
m ak e sen se of th is behaviour.
In terp retiv ists argue that it is necessary not only to em ploy a herm eneutical approach,
but a lso to reject em p iricist scien tific m ethod s for studying hum an behaviour. Km piricist
m e th o d s treat social reality as co n sistin g only of what C harles Taylor calls brute data'. By
‘b ru te data’, Taylor m eans ‘data w hose validity can n o t be q u estio ned by ottering another
in terp reta tio n or read ing, data w hose cred ib ility can n ot be founded or un derm ined by fu r­
th er rea so n in g (la y lo r 1994: 184). Ih ese data captu re p olitical behaviou r in volving actions
that have an id en tifiab le physical end state. W hen actors raise th eir hands at a m eetin g at the
ap p rop riate tim e we can give this actio n a ‘bru te data’ d escrip tion and say that the actors are
‘v oting for th e m o tion . However, the actio n may have m eanings for the actors that are not
cap tu red by the ‘b ru te data d escrip tion o f it. It m ay be the case that when an actor votes for
a m o tio n , she is a lso expressing loyalty to her party or defen d ing the value o f free speech
(T ay lor 1994: 190). But a ‘b eh av iou ral’ (o r positive, or em p iricist) political scien ce deals
on ly with b ru te d ata and th eir logical c on sequ en ces, and avoids addressing the m eanin g o f
p olitica l behaviou r.
As Ta y lo r p o in ts ou t, b ru te data cap tu res m o re than b ehav iou r that has an id en tifiab le
e n d sta te: it also ca p tu res th e su b jectiv e reality o f in d iv id u als’ beliefs, attitu des, and values
‘as attested by th eir resp o n ses to c e rta in form s o f w ords, or in som e cases, th eir overt n o n ­
v erba l b e h a v io u r’ ( 1 9 9 4 : 1 9 8 -9 ) . But w hile th ese data captu re su b jectiv e m eanin gs, there
a re n o n -su b je c tiv e (in te rsu b je ctiv e and c o m m o n ) m ean in gs con stitu tiv e o f social reality
th a t th ey c a n n o t ca p tu re such as, for in stan ce , in ter-su b jectiv e m eanin gs and co m m o n
m e a n in g s. In te r-su b je c tiv e m e an in g s are m ean in gs that do not exist only in the m ind s o f
a g en ts but are ro o ted in and co n stitu tiv e o f s o cial relatio n s and p ractices such as paying
ta x es and voting. C o m m o n m eanin gs involve recognition or consciou sn ess o f shared beliefs,
a sp ira tio n s, goals, values, and a c o m m o n referen ce p oin t for public life o f a society.
C o m m o n m e a n in g s are th e b asis o f com m u n ity , in that th ey are expressed by collectiv e
a sp ira tio n s, a c tio n s, and feelings (1 9 9 4 : 197). Taylor argues that we need to study th ese
n o n -s u b je c tiv e (in te r su b jec tiv e and c o m m o n ) m e an in g s in ord er to com p reh en d p o liti­
ca l issu es su ch as so c ia l c o h e sio n , stability, d isord er, and legitim acy. M oreover, th ey are
c ru c ia l for ‘a sc ie n c e o f c o m p arativ e p o litic s’: w ithout th em , we ‘in terp ret all o th e r s o c ie ­
ties in th e c a te g o rie s o f o u r ow n (1 9 9 4 : 2 0 0 ), ren d erin g in visible im p o rtan t d ifferences
a m o n g s o c ie tie s and m a k in g c o m p a ris o n im p ossib le.
To stay with this point for a m om ent longer, analyses based on positivist epistemological
assu m p tion s, like th ose offered by rational choice theory, depend on an abstract description
o f h u m an agency, on e that pays little attention to differences across social, cultural, and his­
to rical settings. Rational ch oice theory seeks to explain social phenom ena as the o utcom e o f
p u rp osive rationality, and o f m aterial and structural factors exercising causal influence on
individuals. Its c o n ce rn is to show how a given outcom e is the result of purposive choices by
individuals w ithin a given set o f m aterial and structural circum stances. As Daniel Little puts
it: ‘A gents like these in stru ctu res like those, produce outcom es like these* (Little 2 0 0 9 ). But
PHILO SO PHY OF SOCIAL SCIENCE

in terpretive approaches see in dividuals as un iqu e, and hu m an activ ities, a ctio n s, and social
form a tion s as u n iqu e h istorical exp ressions o f hu m an m ean in g and in ten tio n . C onsequen tly,
they are co n ce rn e d , not with ab stract d escrip tion s o f hu m an agents, b ut w ith detailed
in terpretive w ork on specific cultures.
These d ifferences betw een positiv ist and in terpretiv e ap p roaches can be illu strated by
referen ce to a key area o f research in our field: political p articip ation . P ositivist stu d ies typ i­
cally equate political p articip ation with voting. However, in terp retiv ists w ould argue that a
particular voter m ay ‘not u n derstand v oting as particip ation at all, in co n trast, say, to party
activ ism . (It is n eith er practically nor logically im possib le th at an a cto r could say, “N o, I do
not p articipate in politics, but I do vote”) ’ (Sch w artz 1 9 8 4 :1 1 1 8 ). But positiv ists ten d to treat
p articip atory acts as ‘b ru te facts’, as having an ‘ob jectiv e’ on tological status: as ex istin g ‘in
som e sense “in th e w orld” separate from th e th eoretical stance o f th e ob serv er or o f th e p ar­
ticip a n t’ (Sch w artz 1984: 1119).
Joel Schw artz p oin ts out that th ere is no o b jectiv e p oin t o f view from w hich to d escrib e
and understand p articip ation, that p articip ation ‘is a “su b jectiv e” p h e n o m en o n m u ch like
“ju stice” and “v irtue”’ (Sch w artz 1984: 1119). C onsequen tly, ‘any successful attem pt to
d escrib e and explain p articip atory a cts m ust b egin , no t by im posin g the o b se rv e rs th e o re ti­
cal fram ew ork on to the data, but rather with a sen sitivity to th e fram ew orks o f th e p a r tic i­
pants them selves’ (Schw artz 1984: 1120). W h ile positivist studies typically equate p olitical
p articipation with voter tu rn out, particip ation involves a variety o f p olitical acts. By im p o s­
ing th eir ow n concep t o f ‘participation’, research ers are prevented ‘from seeing th e plural
form s that p articipation in fact takes in the world. W h eth er acts (o f an A m erican v oter or
dem onstrator, a French revolutionary, a M uslim revolutionary, a Solid arity m em b er, and so
on) coun t as acts o f participation depends on th ose actors’ su b jectiv e un d erstand in g o f w hat
they are doing’ (Schw artz 1984: 1117).
In sum , interpretivism m aintains that all social action is fram ed by a m eaningful social
world. To understand, explain, or predict patterns o f hum an behaviour, we m ust first u n d er­
stand the m eanings concrete agents attribute to their env iron m en t (social and natural); the
values and goals they possess; the choices they perceive; and the way they interpret oth er
individuals’ social action. Social scien ce is, th erefore, fundam entally d ifferent from natural
science, and it is the im portance o f m eaning that distinguishes social scien ce from natural
science. H um ans act because o f what things m ean, so an understanding o f hum an behaviour
requires that we develop an understanding o f m eanings and in tentions em ployed by social
actors.
M any researchers have pointed to the ten d en cy to cast p ositiv ism and in terp retiv ism
approaches as ‘two grand trad itions in social scien ce ep istem o lo g y ’ and to exaggerate the
d ifferences betw een them (P ollin s 2 007: 9 3). Positivism and in terp retiv ism have different
ontological and epistem o logical com m itm en ts, but they don’t necessarily represent o p p o s­
ing or com petin g traditions. R esearchers w orking in b oth trad itions gen erally follow the
sam e m ethod ological conv en tion s, and so can un d erstand what th ose w orking w ithin the
oth er tradition are doing. R esearchers depend upon different assu m ptions, and may be
interested in and test different qu estions. But w hile they may ‘be tackling different kinds of
q u estio ns’, practical investigation of these qu estions often leads th em to sim ilar m e th o d o ­
logical tasks' (1-innem ore and Sikk ink 2001: 395). Ted H opf argues that th ere is, in fact, ‘a
 F O R M S OF K N O W L E D G E

ce rta in m e th o d o lo g ica l u n ity ’ betw een th e two trad itions. I h e m ethod o logical co n v en ­
tio n s th ey sh are in clud e th e follow ing:

a. clea r d iffe re n tia tio n o f prem ises from con clu sio n s,

b. a ckn ow led g em en t that sam plin g strategies m atter,

c. reco g n itio n that so m e stan dard s o f valid ation m ust be established for the sources o l evidence
used;

d. d iffe re n tia tio n of causes from corre la tio n s;

e. reco g n itio n th at th e sp ectre ol spurio usn ess h au nts all correlatio n s;

f. a c cep tan c e of d ed uctive logic;

g. b e lie f in th e need for the con testa b ility o f findings (2 0 0 7 : 36)

Th ese shared m ethod o log ical conv en tion s may. in fact, be seen as reflecting a com m on
research p ractice founded in the h yp othetico-deductiv e m ethod. As Brian Hollins puts it:
som e research ers assess w hether the in form ation they have gathered tits with the in terp reta­
tion they have posited ’, and oth ers consid er the fit of com petin g interpretations with the
facts they have gath ered ’, but 'in either case they are practicin g the hypothetico-deductive
m eth od (P ollin s 2 0 0 7 : 100). In fact, according to D agfinn lollesd al, the herm eneutic
m eth od that we discu ssed in our consid eration of interpretivism , above, is actually the
hyp oth etico-d ed u ctiv e m e th o d ’ applied to m aterials that are ‘m eanin gful’, i.e. m aterial that
expresses an a c to r ’s b eliefs and values' ( 1994: 233). In terpretation-hypotheses can be judged
by ded u cin g co n seq u e n ces from them and con fron tin g them with data, such as, for instance,
a given text and related w orks bearing on it.
So interp retiv ists and positivists do not necessarily use different approaches to gathering
relevant ev id en ce. H owever, they do ditt’e r in th eir concep tion o f what constitutes ex p lan a­
tion (recall o u r d iscu ssion , above, about political participation). Ihey also differ in their
un d erstand in g o f evidence.
T h e d iffe r e n c e s b etw ee n p o sitiv ist and in te rp re tiv ist c o n c e p tio n s o f b o th e x p la n a ­
tio n a n d e v id e n c e m ig h t b e d e sc rib e d by d e fin in g a d istin c tio n b etw een e x te r n a l’ and
'in t e r n a l’ e x p la n a tio n and ev id en ce. E x tern a l ex p la n a tio n s are a sso ciated with p o sitiv ­
ist r e se a rc h : th ey ten d to w ork via co r re la tio n s o r d e d u c tio n s on th e b asis o f ascribed
re a so n s, an d so need no t c o n c e rn th em se lv es w ith a c to r s ’ u n d ersta n d in g s o f th e world.
In te r p re tiv e e x p la n a tio n s, on th e o th e r h an d , are in te r n a l’ in th e sen se o f th e ir b ein g
c o n c e r n e d w ith th e w orld o f m e a n in g s in h ab ited by th e a c to r (H a m p s h e r-M o n k and
H in d m o o r 2 0 0 9 : 4 8 ) . T h e d is tin c tio n can be applied to d ifferen t typ es o f ev id en ce, as
w ell: ‘e x te r n a l e v id e n c e ’ c o n s is ts o f e m p iric a l e v id en c e ab o u t th e b eh av iou r, and th e
e ffe c ts o f th e b eh a v io u r, o f p a r tic u la r a c to rs; w hile in te r n a l’ o r in terp retiv e ev id en ce
c o n s is ts o f e v id e n c e a b o u t th e b e lie fs o f acto r s w ho se a c tio n s c o m p ris e th e p h e n o m en a
to b e e x p la in e d .
T o hig hlig h t th ese d istin ctio n s. le ts com p are th e analysis offered by a specific positivist
ap p roach (ra tio n a l ch o ic e th eory) and a specific in terpretivist approach (constru ctiv ism )
w ith resp ect to a p articu lar area o f p olitical in quiry: the eruption o f eth n ic con flict w ithin
th e fo rm er Yugoslavia.
PHILO SO PHY OF SOCIAL SCIENCE

The analysis of ethnic conflict a positivist (rational choice)

and interpretivist (constructivist) approach
A nu m b er o f ration al ch o ice and co n stru ctiv ist exp lan ation s have b een offered for w hy e th ­
nic con flicts erupted in Yugoslavia in the 1990s. B o th types o f ex p lan ation have b een c o n ­
cerned to offer an alternative to the cultural, an cien t hatred s’ ex p lan ation for th e w ar
betw een the Serbs and C roats. As m any people have noted, th e vast m a jo rity o f S erb s and
C roats lived tog ether peacefu lly until the spring o f 1991, w hen C ro atia declared its in d e ­
pend ence. Th ere is no ev iden ce to suggest the ex isten ce o f d eep and w idespread hatred in
relations am ong Serbs and C roats d uring the sixty-year h isto ry o f Yugoslavia. M oreover,
even if evid ence could be found for the persistence o f ‘an cien t h atred s’, th ey still c an n o t
explain a key ‘puzzle’: why relations am ong co m m u n ities that had b een living tog eth er
peacefully b ecam e polarized 50 quickly, b efore finally d issolving in to savage v iolen ce.
R ational ch oice th eory is th e study o f strategic p olitical in teractio n s, o f how people (agen ts
or players) d eterm ine strategies in different situations. It explains o u tcom es as th e result o f
rational choices m ade by individuals w ithin a given set o f m aterial and stru ctu ral c ir c u m ­
stances. It shows that, given a particu lar set o f c ircu m stan ces, the strategic in teractio n s o f
agents will produce p redictable, law -like o utcom es. M uch o f the analysis o f strategic political
in teraction s focuses on how individual actors m ake decisions in gam e-like situations. A
‘gam e’ is any situation in w hich a fixed set o f agents or players, with a fixed set o f strategies
available to th em , com pete against one another, and receive a payoff as a result o f the stra te­
gies they and their fellow actors choo se. The assu m ption is that all players know all possible
outcom es and have preferences regarding these possible ou tcom es based on the am o u n t o f
value or utility they derive from each o f them . All players behave ration ally; and th ey m ake
rational decisions about what strategy to pursue based on a calcu lation o f the costs and b e n ­
efits o f different strategies for achieving th eir preferred outcom e.
In a series o f articles (1 9 9 4 , 1995, 1998), Jam es Fearon used ratio n al-ch o ice assu m ptions
as a basis for exploring the causes and cond ition s o f eth n ic co n flict and war.
In order to explain the ‘puzzle’ o f the rapid p olarization o f Serbs and C roats in C roatia in
1991, James Fearon develops a g am e-theo retic m odel: a m odel o f how groups o f people
interact when confronted with a situation o f uncertainty, based on assu m ptions o f gam e
theory. Fearon points out that, just m onths before the eruption o f w ar betw een Serb s and
C roats in C roatia, jou rn alists had reported that ‘m ost people seem ed to have had no use for
or interest in the exclusivist argum ents pushed by the m inority o f ex trem ists’ (1 9 9 8 : 114).
‘W ith the exception o f a relatively sm all nu m ber o f extrem ists . . . Serbs and C roats in the
mixed population areas recognised that war would be costly and viewed it as unnecessary.’
But in spring 1991, ‘Serbs and C roats who had resisted the extrem ists appeals finally opted
for division and war’ (Fearon 1998: 115).
Fearon argued that, following the collapse o f the Soviet U nion, eth n ic conflict erupted in
Yugoslavia as a result o f a ‘com m itm ent problem ’. The problem arises w'hen ‘two political
com m unities find them selves without a third party that can guarantee agreem en ts betw een
them (1995: 2). If, in a new state, eth nic m inorities don’t believe that the state can guarantee
that the ethnic m ajorities will not infringe on their rights, they will prefer to fight for succes-
 F O R M S OF K N O W L E D G E

E xp lanation consisten t with the h y p othetico-deductiv e m odel here consists o f (1) a set of
initial d ete rm in in g c on d ition s (circu m stan ces p ertain ing at particular tim es and places);
and (2 ) a gen eral law or laws w hich co n n ect these cond ition s to the type o f events to be
explained ( h yp otheses w hich are capable o f b eing confirm ed or discon tin u ed bv suitable
em p irical find ings). First, Fearon specifies the set o f initial condition s The com m itm ent
p rob lem , Fearon tells us. arises w henever three cond ition s hold: (1) the groups interact in
anarchy, w ithout a th ird party able to guarantee and enfo rce agreem ents betw een them ; (2)
on e o f th e groups anticipates that its ability to secede or oth erw ise withdraw from toint
a rra n g em en ts will d eclin e in the near future; and (3) for this group, fighting in the present is
p referable to th e worst p olitical ou tco m e it could face if it choo ses continu ed interaction
(1 9 9 5 : 10). Secon d , on the basis o f a gam e-th eo retic m odel he develops o f the com m itm ent
p rob lem , Fearon gen erates hypotheses about what m akes eth n ic war m ore or less likely. Ihe
key m e ch a n ism s in clud e (1 ) the expected change in size in the relative m ilitary power
b etw een groups that would result from form ation o f a new stale; (2) the relative size o f the
e th n ic m in o rity ; (3 ) w hether m ajority and m inority groups’ costs for fighting are low; and
(4 ) w h eth er in stitu tio ns can be created that give m inority groups political power that is at
least p rop ortion al to th eir num bers.
F earon th en applies th e m odel to the war in C roatia in 1 9 9 1 -2 . W hen C roatia declared its
in d ep en d en ce from Yugoslavia, m inority Serbs living in C roatia faced the prospect o f being
in a state with no cred ib le guarantees on th eir political status, or e con om ic and even physical
security. ‘If th e co m m itm en t problem m odel does capture som eth ing o f what was going on
in C ro a tia , th en we m ight expect to find ev iden ce o f C roatian leaders trying to work out
g uarantees with Serb lead ers’ (1 9 9 8 : 1 19). This, he finds, occu rs on num erous occasion s.
C ro a tia n Presid en t Tud jm an m et with the leader o f the Serbs in C roatia, Jovan Kaskovic, 'to
d iscu ss th e issue o f c o m m itm en t to guarantees o n the Serb s’ status and "cultural autonom y”’
(1 9 9 8 : 119). But despite T u d jm an’s ettorts to constru ct a credible set o f guarantees for the
S erb m inority, his efforts to solve the com m itm en t problem were ultim ately unsuccessful.
W ith th e ‘p rosp ect o f en terin g the new state o f C roatia with no credible guarantees on their
p olitical status, o r eco n o m ic or even physical secu rity’, the prospect o f a war then appeared
b etter to th e Serb s than th e p rospect o f fighting later, by w hich tim e the C roatian state would
have grow n stron g er (F earon 1998: 116).
T h e ev id en ce, th en , con sists in show ing that th ere is a ‘fit’ betw een th e d eductions o f the
th eo ry and th e ob serv ed b ehaviou ral o u tcom es; that the ou tcom e is consisten t with the
th eo retica l p red ictio n s. Fearon also e nd eavours to d em onstrate that his e xplanation offers a
b e tte r ’fit’ with th e facts than o th er exp lanations do. Finally, he argues that th e b asic c o m ­
m itm e n t p rob lem th at he d escrib es ‘appears eith er to lurk or to have caused in terethn ic
v io len ce’ in o th e r c ases, as well: in A zerbaijan, G eorg ia, M oldova, U kraine, Estonia, Z im b a ­
bw e, So u th A frica , and N orth ern Ireland (1 9 9 5 : 21).
L e ts con sid er how an interpretivist approach, con stru ctiv ism , explains the rapid ethnic
p olarization th at o c c u rre d in the form er Yugoslavia in the 1990s.
C onstructivism is an approach that has had an important influence on political inquiry.
C onsistent with interpretivist assumptions, constructivism maintains that reality does not exist
as som ething independent o f us and is not, therefore, merely discovered by us. it is socially, and
actively, constructed. C onstructivists assum e that social p henom ena are social constructs in the
sense that their shape and form is imbued with social values, norm s, and assumptions, rather
PHILO SO PHY OF SOCIAL SCIENCE

than being the product o f purely individual thought or m eaning. W e live in ‘a w orld o f ou r m a k ­
ing’, as N icolas O n u f (1 9 8 9 ) has put it. A ctors are not totally free to ch o o se th eir circu m stan ces,
but m ake choices in the process o f in teracting with oth ers and, as a result, b ring historically,
culturally, and politically d istinct ‘realities’ into being. In this respect, the w orld o f p olitics is a
social construction rather than som eth ing that exists in dependently o f h u m an m eanin g and
a ction. States and oth er social institutions take specific historical, cultural, and p olitical form s
that are a product o f hum an in teraction in the social world.
In contrast with positivist approaches w hich em phasize a single o b jective reality, th e idea o f
social con stru ction suggests difference across context. It is not only the m aterial env iron m en t,
but also the cultural and institutional env iron m en t that provides in centives and d isincentiv es
for behaviour. Society is m ore than just the site o f strategic in teraction to pursue p re-defined
interests in a rational, u tility-m axim izing m anner. It is a constitutive realm , an env iron m en t
that form s and influences the identities and interests o f actors and m akes th em w ho th ey are.
M oreover, social in teraction also influences the identity o f actors. The properties o f actors are
not intrinsic to them : they are socially contingent, they depend on social in teraction : b arg ain ­
ing/negotiating, arguing, com m un icatin g in general. B oth the identities and interests o f actors
are constituted (form ed, influenced) through in teraction and by the institutionalized norm s,
values, and ideas o f society. Sin ce the interests and identities o f actors are not g iven— but result
from social in teraction— they cann ot be abstracted from the social con d ition s w hich produce
them ; and they are subject to change as a result o f political processes.
C onsisten t with these assu m ptions, con stru ctiv ism , like r a tio n al-ch o ice approaches,
rejects explanations o f nationalist and e th n ic p henom ena as the ou tco m e o f essential cultural
identities. But, unlike ration al-ch oice approaches, it sees nationalist and eth n ic co n flict as a
phenom enon that has assum ed a variety o f form s acro ss space and tim e; and it em phasizes
the role o f identities that are m ultiple and fluid and p olitically m alleable: in fluen ced by su r­
rounding structures and ‘c on stru cted ’ for political purposes.
M urat Som er (2 0 0 2 , 2001) addresses the sam e ‘puzzle’ as Fearon: the rapid eth n ic p o la ri­
zation that occurred in Yugoslavia in the 1990s. Som er em phasizes th e sign ificance o f public
discourses in form in g individuals’ eth n ic identities, and in suppressing and reviving d o m i­
nant perceptions o f eth n ic identities. He argues that eth n ic co n flict is a result o f processes o f
ethnic identity constru ction in the public arena that con stru ct a divisive im age o f identities.
Public eth nic activities and expressions are im m ediately observed and they im m ediately
affect the decisions o f others.
F.thnic polarization changes the dom inan t im ages o f eth n ic categories in society th rou gh
cascades o f individual reactions. A cascadin g process changes b ehaviou r and attitudes and,
once begun, is very difficult to stop. In Yugoslavia, eth n ic polarization in public d iscou rses
was engineered by eth nic entrepreneurs who constru cted and prom oted a divisive im age o f
ethnic identities as mutually exclusive and in com patible with b elon ging to the sam e nation,
lh is triggered a ‘cascade process’ which resulted in the creation o f a critical mass o f opin ion
around a new im age o f eth nic identities. People who secretly held this divisive view, as well
as people who now felt com pelled to support it, jum ped on the bandw agon. H ence, the d iv i­
sive im age becam e the norm , and it becam e inappropriate, even blasphem ou s, to defend
in terethnic m ixing and brotherhoo d ’ (Som er 2001: 128).
Som er draws a distinction between public and private eth nic polarization and highlights
the way people publicly ‘falsif y' their private beliefs. D uring the com m u n ist era, state policies
 F O R M S OF K N O W L E D G E

had a im ed at erad icatin g the public expression o f the divisive im age o f eth nic relations in the
c o u n try ; but they had insufficiently encouraged its elim in ation in private'. C onsequently, the
public d iscou rse in Yugoslavia had exerted pressure for downward falsification to discou r­
age people from open ly expressing th eir eth n ic prejudices. Ih is downward preference falsi­
fication con cealed , to m ost observ ers, the private im portance o f the divisive im age';
consequently, even analysts who had a fair idea about the private significance o f the divisive
im age w ere surprised by the severity o f polarization' in the 1990s (2001: 136) D uring the
1990s, the d om inan t public d iscou rse em phasizing unity and b rotherhood' turned into one
that em phasized radical eth non ation alism ' (2001: 136). But this public polarization far
exceed ed private polarization (2 0 0 1 : 143). C onsequently, there was widespread upward
eth n ic preference falsification, the exaggeration o f public support for the divisive im age, as
th e new nationalist regim e exerted pressure for p eople— including liberal and moderately
toleran t ind iv id uals— to th ink and act in an eth nically intolerant m anner.
S o m e r uses survey research w hich in d icates decreases in self-id en tification with the
o v e ra rc h in g Yugoslav id en tity betw een 1981 and 1991. Ih e respondents were anonym ous,
so th ese surv ey s w ere able to cap tu re ch an ges in people’s private p references. D u ring the
1 9 8 0 s th ere w as a strik in g upsurge in ‘th e public exp ression of th e divisive im age' (Som er
2 0 0 1 : 1 43). B ut, ‘in 1 989, w hen public polarization had reached an advanced state, an o n y ­
m o u s su rv eys c o n tin u ed to reveal that in tereth n ic toleran ce levels were high by global
sta n d a rd s’. So, ‘w hile th e public d isco u rse was b eco m in g in creasingly m ore divisive and
less to lera n t o f in tereth n ic d ifferen ces, private attitu des rem ain ed qu ite toleran t of
in te r e th n ic differences'. In fact, ‘the highest levels o f toleran ce were found in B osn ia, the
site o f th e m o st v iolen t c rim es' (S o m e r 2 0 0 1 :1 4 4 ). ‘D esertio n and call-u p evasion were
very c o m m o n d u rin g th e civil war when public support for the divisive im age was at its
p ea k ’ (S o m e r 2 0 0 1 : 144).
L e ts sum up by com p a rin g the tw o approaches to un derstandin g eth nic polarization in
Yugoslavia. B o th highligh t the sign ificance o f social and political in stitutions in form in g
ind iv id u als’ eth n ic identities. But Fearon, like oth er ration al-ch oice th eorists, tends to stress
th e stru ctu ra l and c o n stra in in g features o f in stitutions. Som er, on the oth er hand, em p h a­
sizes th eir socia l and cogn itiv e aspects. B oth are constru ctiv ist in the sense that they see
ch a n g es in an actor's id entity co n stru ctio n s as likely to o cc u r in m o m en ts o f crisis and
d ilem m a . But Fearon, c on sisten t with rational ch o ice approaches, em phasizes the role o f
stra teg ic c a lcu la tio n in id entity co n stru ctio n , while Som er em phasizes cognitive features,
such as n o rm s o f b ehav iou r and in ter-su b jectiv e understandings (th ough these don t nece s­
sarily o p erate to the ex clu sion o f the calculative elem ent stressed in the rational choice
ex p la n a tio n s offered by Fearon and oth ers).
L e ts co n sid er how the two analyses illustrate the d istin ction betw een in ternal’ and ex ter­
n a l’ ex p lan ation and ev id en ce that we previously discussed. The analyses that Fearon and
S o m e r offer are con sisten t w ith th e assu m ptions, respectively, o f rational ch o ice and c o n ­
stru ctiv ist a pproaches regard ing acto rs’ interests: ration al-ch oice th eories assum e that agents
a ct on th e b asis o f fixed in terests and preferences; constru ctiv ists assum e that interests can
on ly develop from th e im age an acto r holds o f h im self and o f oth ers, that identities are the
so u rce o f in terests (and , th erefo re, th e b asis o f actio n ) (e.g. W endt 1994; Ringm ar 1996).
F ea ro n m o d els ex te rn a l 'b eh a v io u r’, and th en seeks e v id en ce by way o f d eductive fit with
th at m o d el. E m p irica l ev id en ce co n sists o f statem e n ts and activ ities o f C roatian leaders
PHILO SO PHY OF SOCIAL SCIENCE

th at in d icate a c o n c e rn for p rov id in g th e S erb m in o rity w ith a c o m m itm e n t to g u aran tee

th eir status and cu ltu ral autonom y. But th ere is n o d ire ct ev id en ce o f th e e x iste n c e o f a
c o m m itm en t prob lem am o n g th e S erb p op u lation , o f strate g ic b eh a v io u r on th e p a rt o f
in d ivid uals o r groups, o r o f th e relatio n sh ip b etw een b e lie f and a c tio n . D ire ct ev id en ce
c o n c e rn in g w hether th e in dividual o r group ch o ice s w h ich led to e th n ic p o la riz a tio n w ere
m ad e for th e reasons stated in his m o d el m igh t be im p ra c tic a l o r im p o ssib le to o b tain .
In stead , the ‘test’ o f th e m odel involves (1 ) d ed u cin g th e fa cto rs th at, in th e given c ir c u m ­
stances, m ight be ex p ected to lead g roups to reso rt to v io len ce; and (2 ) o b serv in g th e o u t­
com es. W h ere we ob serv e th at circu m s ta n ce s favour th e b eh av io u rs th at th e m o d el tells us
are m o st likely to o ccu r, we can in fer th at th ere is a lin e o f cau se and effect th at relates th o se
c ircu m sta n ces and th e o u tcom e. S o m er id en tifies a m e ch a n ism , th e ca sca d e p ro cess,
w hich lin ks p opu lar b eliefs, pub lic p o litical d isco u rse, and relatio n s a cro ss groups. H e
th en co m b in es survey data w ith detailed ex a m in a tio n o f th e h isto ric a l ev en ts to p rov ide
ev id en ce o f th e ch an gin g n ature o f public and private view s and suggest th e relatio n sh ip
betw een them .
Both analyses have im plications for p olitics and policy: w hich o f th e ex p lan ation s for
eth n ic con flict we ch oo se to favour has im p lications for how people relate to each o th e r and
how governm ents act. Th e ch oice o f w hich set o f assu m ptions will provide th e b est startin g
point for your own research on a specific p olitical issue o r p roblem is on e w hich you will
need to carefully consider.

Conclusions
This chapter has begun our consideration of some fundamental ontological, epistemological, and
methodological problems posed by social inquiry. These problems mostly branch out from one
central question: are the methods of the social sciences essentially the same as, or essentially different
from, those of the natural sciences? We have reviewed the basic tenets of three different answers to
this question: positivism, scientific realism, and interpretivism. How these answers differ is presented
in Box 2.3.
All of these define a position with respect to how we study and conduct research in the social
sciences. As Box 2.3 shows, all are based on fundamentally different assumptions about how we can
know and develop knowledge about the social world; and all of them remain important perspectives
for contemporary social research. The question of whether and how we can pursue a systematic and
rigorous study of social phenomena in the way that scientists pursue study of the natural world and,
more generally, philosophical presuppositions about 'reality' implicit in social science research, bears
on how we design or structure an approach to research into political phenomena, and the claims we
can make with respect to the findings that result from that research.
At the heart of the debate among these perspectives is the question of what sort of knowledge we
can gain about social phenomena. This question is also central to the controversy that we take up in
Chapter 3, which is the debate about whether the knowledge produced through the study of the social
world is or can be objective'; and in Chapter 4, where we consider the question of what is the social'.
Reality- is constantly being defined for us-by political scientists, by historians, by politicians in
their speeches, by media analysts in their news reports The ability to identify the underlying structure
of assumptions or the implicit theory which shapes a given account of reality, whether presented by
scholars, politicians, or journalists, allows us to become more active analysts of contemporary politics,
rather than depending on the analysis of others
 F O R M S OF K N O W L E D G E 49
[B O X 2 3 Positivism, Scientific Realism, and Interpretlvism Compared
Positivism Scientific realism Interpretlvism
Ontology What
is the njture of
the serial world7

Epistem ology:
What sort of
knowledge of the
social world is
possible7

•ir mj .-.Ml,, ,<y ,

Causality: what A reflection u!

do we mean by we think jbuu! :m
'causes'? established by disc
observable rcyuljr ’»».uunr/i,-, iim v .,’ s ‘or U>"

MtthoMofr Through direct through d rtxt observa f n t t * r . i- theory artd

How can we gain observation tion and logic applied to»1uj, strategics The
knowledge of the t. ■ix.)!h observable and v« ui world is like a text
social world? unobservable structures and has to be interpreted
to discover hidden
meanings and subtexts

Questions
• What is involved in providing an explanation of social phenomena’ How is explanation distinct from
and related to interpretation?
• What place does the concept of law7have in social-scientific explanation’

• What does causality mean? How, according to different conceptions of causality, do we establish
that something causes something else7
• How is describing, interpreting, and explaining human action different from describing, interpreting,
and explaining non-human events?
• Should the primary goal of social science be to provide law-like explanations, capable of supporting
predictions7Or are law-like explanations impossible, or unnecessary, within social science7
PHILO SO PHY OF SOCIAL SCIENCE

Guide to Further Reading

March, D. and P. Furlong (2002), A Skin not a Sw eater Ontology and Epistemology in Political
Science', in D. Marsh and G. Stoker (eds), Theory and M ethods in Political Science, 2nd edition
(Basingstoke: Palgrave), 17-41.
The authors discuss positivism, interpretivism, constructivism, and also realism, as different
approaches to ontology and epistemology in political science, and illustrate their differences with case
studies.

Martin, M. and L C. M cIntyre (eds) (1994), Readings in the Philosophy o f Social Science (N e w York:
M IT Press).
This volume brings together a collection of important texts on the disputed role of general laws in
social-scientific explanation (Part II), and on interpretation and meaning (Part III).

Gibbons, M. T. (2006), Hermeneutics, Political Inquiry, and Practical Reason: An Evolving

Challenge to Political Science'. American Political Science Review 100:4 (Novem ber), 563-71.

Lane, R. (1996), 'Positivism, Scientific Realism and Political Science'. Journal o f Theoretical Politics
8(3): 361-82.
This article explores the implications of scientific realist principles for political science, political
research, and political theory, providing examples of a scientific realist approach in studies utilizing
a variety of theoretical approaches, including rational choice, new institutionalism, and comparative
politics.

Little, D. (1991), Varieties o f Social Explanation: An Introduction to the Philosophy o f Social Science
(Boulder, Colorado: Westview Press), chapter 2 ('Causal Analysis'), pp. 13-38.

Russo, F. (2009), Causality and Causal Modelling in the Social Sciences (New York: Springer).
This book offers an overview of debates, and it provides a valuable analysis of reasoning about
causation by looking at the causal arguments advanced in specific social science studies.

Sankey, H. (2008), Scientific Realism and the Rationality o f Science (Aldershot: Ashgate), chapter 1.
This chapter provides a clear exposition of the doctrines of scientific realism, which distinguishes
between core and optional doctrines: and the principal arguments that have been advanced for
scientific realism.

References
Amadae. S M and B Bueno de Mesquita (1999), Bryman, A. (2004), Social Research Methods (Oxford:
The Rochester School The Origins of Positive Oxford University Press).
Political Theory'. Annual Review of Political Colomer.J. P. (2005). 'It's Parties that Choose
Science 2 269-95
Electoral Systems (or. Duverger's Laws Upside
Benoit, K (2007). Electoral Laws as Political Down)'. Political Studies 53: 1-21
Consequences Explaining the Origins and
Easton. D. (1969). The New Revolution in Political
Change of Electoral Institutions' Annual Review
Science'. American Political Science Review 63(4)
of Political Science 10 (June): 363-90 (December): 1051-61.
Bhaskar. Roy (1998). The Possibility of Naturalism:
Ekstrom, M. (1992). Causal Explanation of Social
A Philosophical Critique of the Contemporary
Action: The Contribution of Max Weber and of
Human Sciences. 3rd edition (Hemel Hempstead
Critical Realism to a Generative View of Causal
Harvester Wheatsheaf)
Explanation in Social Science'. Acta Sociologica:
Brown Chris (2007) Situating Critical Realism', Journal of the Scandinavian Sociological
Millennium Journal of International Studies 35(2) Association 35 107-22
(March) 409-16
 FORMS OF KNOWLEDGE

Fearon. D Ja m « (1994), Ethnic War as a Hopf. T (2007). The Limits of Interpreting Evidence
Commitment Problem', paper presented at the in R N Leibow and M I bchbach (eds) Theory
annual meeting of the American Political Science and Evidence in Comparative Politics and
Association, New Yorit. http //www Stanford International Relations (Basingstoke Palgrave/
edu/-jfearon/papers/ethcprob pdf Macmillan). 55-84
---(1995), Rationalist Explanations for War1, Hume. D (1966), Enquiries Concerning the Human
International Organization 49(3) 379-414 Understanding (Oxford Clarendon Press)
---(1998) Commitment Problems and the Lane. R (1996). Positivism Scientific Realism and
Spread of Ethnic Conflict1, in David Lake and Political Science.Journal of Theoretical Politics
Donald Rothchild (eds), The International Spread 8(3) 361-82
of Ethnic Conflict (Princeton, NJ: Princeton Little. D (1991). Varieties of Social Explanation An
University Press). 107-26 Introduction to the Philosophy of Social Science
Finnemore, M and K Sikkink (2001), Taking (Boulder. CO Westview Press)
Stock The Constructivist Research Program ---(2009). McIntyre and Taylor on the Human
in International Relations and Comparative Sciences 16July httpy/undersUndingsociety
Politics', Annual Review of Political Science 4 blogspot com/2009/07/mac intyre and taylor -
391-416 on-human sciences html
Fellesdal. D. (1994), Hermeneutics and the May, T (2003), Social Research Issues. Methods
Hypothetico-Deductive Method’, in Michael And Process. 3rd edition (Buckingham Open
Martin and Lee C. McIntyre (eds). Readings in Umvervty Press)
the Philowphy o f Social Science (Cambridge. MA
McIntyre, L C (1994). Complexity and Social
MIT Press)
Scientific Laws', in Michael Martin and Lee C
Gerring. J. (2007). Review Article The Mechamsmic McIntyre (eds). Readings m the Philosophy of
Worldview: Thinking Inside the Box' Bntiih Social Science (Cambridge. MA MIT Press).
Journal o f Political Science 38 161 -79 131-44
Gibbons. M. T (2006). 'Hermeneutics. Political Onuf. N (1989). World of Our Making Rules and
Inquiry, and Practical Reason An Evolving Rule m Social Theory and International Relations
Challenge to Political Science'. American Political (Columbia. SC University of South Carolina Press)
Science Review 100 4 (November). 563-71 Pollms. B (2007) Beyond Logical Positivism
Gilpin. R. (1981). War and Change in World Politics Reframing King. Keohane. and Verba. in R
(Cambridge: Cambridge University Press) N Lei bow and M I Uchbach (eds). Theory
and Evidence in Comparative Politics and
Grix.J (2002). The Foundations of Research
International Relations (Basingstoke Palgrave/
(Basingstoke: Palgrave/Macmillan)
Macmillan). 87-106
Hampsher-Monk. I and A. Hindmoor (2009).
Popper. Karl. (1959). The Logic of Scientific Discovery
'Rational Choice and Interpretive Evidence:
(London Hutchinson of London)
Caught between a Rock and a Hard Place?1
Political Studies 58(1) 47-65 Riker. W H (1982). The Two-Party System and
Duverger's Law An Essay on the History of
Hedstrom. P.and R Swedborg. eds (1998),
Political Science'. American Political Science
Social Mechanisms: An Analysis Approach to
Review 76 (December) 753-66
Social Theory (Cambridge Cambridge
Ringmar. E (1996). Identity. Intern! and Action
University Press).
(Cambridge: Cambridge University Press)
Hempel. C. G. (1994), The Function of General
Ross. M. (2001). Does Oil Hinder Democracy?-
Laws in History, in Michael Martin and Lee
C. McIntyre (eds). Readings in the Philosophy World Politics 53: 325-61
o f Social Science (Cambridge. MA: MIT Press). Sankey. N (2008). Scientific Realism and the
43-54. Rationality of Science (Aldershot Ashgate)

Hoffmann. M.J. (2006). -Social (De)Construction Schwartz. J. D (1984). Participation and

The Failure of a Multinational State', in Jennifer Multisubjective Understanding: An Interpretrvrst
Anne Sterling-Folker (ed ). Making Sense of Approach to the Study of Political Participation'.
International Relations Theory (Boulder, CO: Journal of PoliVcs 46:1117-41.
Lynne Reiner), 123-38.
52 PHILO SO PHY OF SOCIAL SCIENCE

Somer, M. (2001). 'Cascades of Ethnic Polarization: Science (Cambridge, MA: MIT Press),
Lessons from Yugoslavia', The ANNALS of the 181-212.
American Academy of Political and Social Science Tilly, C. (2001), 'Mechanisms in Political
573(1): 127-51. Processes', Annual Review of Political Science 4:
---(2002). Insincere Public Discourse, Trust, 21-41.
and Implications for Democratic Transition: Trochim, W. M. (2006), The Research Methods
The Yugoslav Meltdown Revisited 'Journal Knowledge Base. 2nd edition. Available at: http://
for Institutional Innovation. Development and www.socialresearchmethods.net/kb/.
Transition 6 (December): 92-112.
Wendt, A. (1994), 'Collective Identity Formation
Taylor, C. (1994), 'Interpretation and the Sciences and the International State', American Political
of Man', in Michael Martin and Lee C. McIntyre Science Review 88(2) (|une): 384-96.
(eds), Readings in the Philosophy of Social

Endnotes
1. Note the difference between the terms 'behaviourism' and behaviouralism': 'behaviourism' is a school of
psychology which studies observable behaviour, rather than 'unobservable' behaviour such as mental
processes and intentions, and emphasizes experimentation and causal analysis. 'Behaviouralism' is the
term adopted by political scientists. The key tenet of behaviouralism is that only observable behaviour
may be studied.
2. In recent years some researchers have modified Duverger's Law by suggesting that 'it is the number of
parties that can explain the choice of electoral systems, rather than the other way round' (Colomer 2005:
1; see also Benoit 2007).
 Objectivity and Values

e Chapter Summary
Thu chapter explores a key debate m the philosophy of social science whether it .
possible to separate facts and values in social science research The debate raises fun
damental questions about how values influence social scientific inquiry the role an<
responsibilities of the researcher m social scientific inquiry and the ways m wtuct
areas of '.tudy are shaped by the norms and values of research communities anc
environments Consequently the answers researchers give to these questions hav<
important implications for how they approach research and understand its findings
Among the questions this chapter explores are the following

• Can social science be value free ?

• Are the findings of the natural sciences less biased than those of the social
sciences7

• To what extent, and in what ways do values present problems for the analysis of
the social world7

• To what extent, and with what effect do a researchers own values intrude into
research7

• To what extent do the values associated with a set of shared social practices
intrude into the search for knowledge7

Introduction

Th e a im o f th is chap ter is to explore the q u estion o f w hether social scien ce can be ‘value-free’.
P olitical research ers want to be able to offer credible answers to im portant questions. Ihey
are co n c e rn e d , th erefo re, to em ploy research practices and m ethod s that enable them to do
this. T h e qu estio n o f w hether and to what extent values influence scientific research bears
d irectly on th is c o n c e rn . Sch olars in all areas o f research are con cern ed with th is question;
and, w hether or not they follow on going d ebates on this issue in the philosophy o f social
scien ce, th eir views tend, eith er im plicitly or explicitly, to reflect the different answers and
p o sitio n s that th ese debates have generated.
T h e answ ers p h ilo so p h ers o f sc ie n ce give to th e q u estio n o f how values in flu en ce s c ie n ­
tific resea rch d iffer acc o rd in g to how th ey answ er a larger and m ore fun dam en tal qu es­
tio n : w h e th er it is ev en p ossible to d istin g u ish betw een facts and values in scien tific
in qu iry. T h ese are d ifficu lt and u n settlin g q u estio n s. They not on ly com p licate o u r efforts
to p ro d u ce resea rch fin d ings th at are un biased and, th ere fo re, reliable; th ey also raise
d o u b ts a b o u t w h e th er it is p o ssib le to prod u ce un biased research at all.
In C h a p ter 2 we no ted that th e d istin ctio n betw een facts and values is a central tenet o f
p ositiv ism . Positivism m ain tain s that facts are fun dam en tally different and distinguishable
PHILO SO PHY OF SOCIAL SCIENCE

from values, and that they exist in d ep end en tly o f th e o b serv er and th e o b se r v e rs values.
Scien ce is ‘v a lu e -fr e e — it is co n ce rn e d with facts, no t values; and any in tru sio n o f v alues in
th e research p rocess co n tam in ates th e o b jectiv e c h a ra cter o f scien ce.
Recall that th is m eth od o logical p osition is th e b asis o f an o th er ten et o f p ositiv ism : n atu ­
ralism ’. N aturalism claim s that th e social w orld is no d ifferent from th e world o f natural
p henom ena, and social scien tists should th erefo re approach th e study o f so cial p h e n o m en a
in the sam e way, accord in g to the sam e scien tific principles, th at scien tists use to study n atu ­
ral phenom ena. To be scien tific, social in qu iry m ust seek com p reh en siv e and system atic
explanations o f events, and it m ust be ob jectiv e or value-free. The d istin ctio n b etw een facts
and values is what differentiates scien ce from n o n -sc ien c e, and w hat d efin es th e b o u n d ary
betw een the ‘scien tific’ study o f society and ideology. C onsequen tly, argu m en ts th at suggest
it is not possible to distingu ish betw een facts and values would seem to u n d erm in e th e p o s­
sibility o f un d erstand in g the social world th rou gh scien tific study— th e possibility, th at is, o f
social science.
The aim o f th is ch ap ter is to ex p lore th e q u estio n o f w h e th er s o c ia l in q u iry can be
‘v alu e-free’. H ow we answ er th is q u estio n dep en d s on w h e th er we th in k it is p o ss ib le to
d isting u ish and keep separate th e realm s o f facts and values. In th e fo llo w in g in tro d u c ­
tion to the fact/value debate, we, o f necessity, p resen t a so m ew h at stylized v ersio n o f it.
We first co n sid er how th is issu e has b een add ressed in th e study o f p o litics. W e th en c o n ­
sider various ways that ‘v alues’ in tru d e in to s o c ia l-sc ie n tific in q u iry : how th e o b s e r v e r ’s
values and oth er b iases in flu en ce research , how th e act o f o b se rv a tio n itse lf in tru d es on
the o b ject o f study, and the ex ten t to w hich o b serv a tio n is ‘th e o ry -la d e n ’, i.e. sh ap ed by
shared socia l p ra ctices and in sep arab le fro m the in te rp re ta tio n s to w h ich th o se p ra c tic e s
give rise.

Normative and empirical theory in political research

Ihe in fluen ce on political research o f p ositiv ism and its fa c t-v a lu e d istin c tio n is reflected
in the in stitu tio n a liz a tion o f a div ision b etw een em p iric a l and n o rm ativ e research and
th eory, th is d ivision is pred icated on th e assu m p tion th at it is p o ssib le to sep arate q u e s­
tions about what is (em p irical qu estio n s) from qu estio n s ab o u t w hat sh ou ld o r ou gh t to
be (norm ativ e q u estio n s). A ccord in g to p o sitiv ism , it is th is ab ility to sep arate em p iric a l
and norm ative qu estio n s that m akes it p o ssible for so cial sc ie n c e to be o b je c tiv e and
value-free.
l-.mpirical th eory is c o n c e rn e d w ith q u estio n s th at can b e answ ered w ith em p iric a l
data (data gathered th rou gh o b serv atio n s o f th e w orld arou n d u s). N o rm a tiv e th e o ry is
theory that co n c e rn s itse lf with q u estio n s o f what ou gh t to b e th e case. It is o fte n a s s o c i­
ated with m oral issues in p olitics; with q u estio n s c o n c e rn in g h u m an rig h ts, d is trib u ­
tive ju stice, and in terv en tio n , and with q u estio n s o f w hat is m o rally ju st: w hat is a ju st
governm ent, in tern a tion al order, d istrib u tio n o f reso u rces, way o f trea tin g th e e n v iro n ­
ment lor future g en erations? W h en , if ever, is war ju st, and w hat is ju st co n d u c t in war?
Positivists insist that propositions concerning such questions cann ot be ad d ressed by em p ir-
theory and subjected to form al em p irical tests; and th at h y p o th e ses ab ou t w hat is
 O B JE C T IV IT Y A N D V A I U I S ss
p ra c tic e ten d s to reflect th is view: n o rm ativ e q u estu .n s ten d to be addressed by p olitical
th e o r is ts in ways th at do no t involve rig oro u s, m eth od o logically in form ed em pirical
stu d y ’; w hile em p iric a l qu estio n s tend to be addressed by research ers co n ce rn e d with the
d isc o v ery of fa c ts’; but no t with u n d ertak in g ’rigoro us, philosop h ically in form ed nor
m ativ e stu d y ’ ((¡e r r in g and Yesnow itz 2006: 103) Sch olars will assu m e, for in stance, that
fed era lism and civil so cie ty are good for d em ocracy, or stron g parties are good tor gov
e rn a n c e . but will m ak e n o attem pt to provide em p irical proo! o f the assu m ption Authors
o f em p iric a l stu d ies will suggest im plicitly that th eir p articu lar o h icct «»I studv has a
b ro a d e r sig n ific a n c e , that the su b ject affects so cie ty ’, but fail to m ake ex plicit how. or to
w hat e x te n t, it d oes so ( ( ¡e r r in g and Yesnowitz 2 006: 107).
In recent years, critic s of positivist social scien ce have challenged this disciplinary
d ivide. Ih ey argue that the n o tion of 'norm ativ e th eo ry ’ assum es, wrongly, that we can
d istin g u ish betw een qu estio n s about what is and th ose con cern ed with what ought to hr,
and that all th eory is n o rm ativ e theory. Ihey argue that, though not all th eorv is expressly
co n c e rn e d with reflectio n on norm ativ e issues, no th eory can avoid norm ative assum p
tio n s in the sele ctio n o f what data is im p ortan t, in in terp reting that data, and in a rticu la r
ing why such research is sig n ifican t’ (C o ch ra n 1999: 1). R obert C ox neatly sum s up this
overall p o sition : ‘all th eory is for som e o n e and for som e p u rpose’. All th eories reflect values
(C o x 1981: 128). 'Ih e claim that all th eory is norm ativ e in this sense, is also a claim about
th e im p ossib ility o f separatin g facts and values. We begin our exploration o f th is issue in
th e next sectio n .

Values, the researcher, and the research process

|()|ne can do a good job predicting whal a ttudy will find by knowing (hr preference» of I he
scholars who undertook it.
Oervis 2002 I8H)

I h e statem ent, quoted above, expresses a perhaps com m only held view. It it generally
assum ed that a research er’s values have at least som e influence, not only on the conclusions
reached, but on every aspect of research, including the selection of research questions, data
gathering, ob servations, and cond ucting experim ents. But does this preclude the possibility
o f separating facts and values, and of pursuing objective scientific study o f the social world?
Som e scholars argue that it is impossible for social inquiry to be objective and value free;
oth ers argue that objectivity in the social sciences, while perhaps difficult to maintain, is
possible.
I h e sociologist M ax W eber (1 8 6 4 - 1 9 2 0 ) developed a somewhat com plicated and highly
influential position on this question. W eber argued that ( I ) there is a distinction between
facts and values: questions o f value are independent of or separable from questions o f fact;
but that it nevertheless rem ains the case that (2 ) value-neutrality is not possible in the social
sciences; however, despite this, (3 ) we must strive for a value-neutral social science. l * t ’s
co n sid er each o f these points a bit further.
W eb er argu ed th at facts can be distinguished from values b ecause know ing the facts
o f s o m e th in g is not th e sam e th in g as know ing its value; and know ing the value of
PHILO SO PHY OF SOCIAL SCIENCE

so m e th in g d o es n o t lead us to a fa ctu a l d e s c r ip tio n . W e c a n th e re fo re d istin g u ish

b etw ee n q u e stio n s o f fa c t an d q u e stio n s in v o lv in g v alu es. S o c ia l sc ie n tis ts sh o u ld deal
o n ly in q u e stio n s o f fa c t an d re m a in e th ic a lly n e u tra l o n q u e stio n s o f v alu es. H e e m p h a ­
sized , how ev er, th at v alu es are u n av o id ab le in th e p r a c tic e o f s o c ia l sc ie n c e . V alu es e n te r
in to th e se le c tio n o f p ro b le m s, th e d e te rm in a tio n o f co n c lu s io n s , th e id e n tific a tio n o f
fa c ts, an d th e asse s sm e n t o f ev id en c e. ‘T h e re is n o a b so lu tely “o b je c tiv e ” scie n tific a n a l­
ysis. . . . o f “so c ia l p h e n o m e n a ” in d e p e n d e n t o f s p e c ia l an d “o n e -s id e d ” v ie w p o in ts’
(1 9 9 4 : 5 3 5 ).
Research ers focu s on ‘on ly th o se segm ents o f reality w hich have b e c o m e sig n ifican t to us
because o f th eir valu e-relev an ce’. O n ly a sm all p o rtio n ‘o f c o n c rete reality is in terestin g and
significant to us, b ecause on ly it is related to th e cultural values w ith w h ich we approach
reality’ (1 9 9 4 : 5 3 9 ). O n ly th ose th in g s th at we perceive to b e m ean in g fu l will b ec o m e
o b jec ts o f in vestigation for us. U n d ertakin g research on so m eth in g presuppo ses th at it is
co n n ected in som e way to our values. A research er s ow n ‘poin t o f v iew ’ also in flu en ces the
concep tu al sch em e that he or she co n stru c ts in ord er to pursue inquiry. So research ers
in evitably b rin g th eir ow n presuppo sitio ns in to th e research field. They c a n n o t rem ain n e u ­
tral; th ere is no ‘view from now here’. W eb er argues, how ever, th at th ou gh m an y e lem en ts o f
research are sub jectiv e and v alue-relevan t, it is n o n eth eless possible to co n d u ct research in
a v alue-n eutral m an n er if research ers adhere to the n o rm s and p ractice o f good so cial s c i­
ence, and if they ‘m ake relentlessly clea r’, b oth to th eir au d ience and th em selv es, ‘w hich o f
th eir statem ents are statem ents o f logically deduced or em p irically ob serv ed facts and
which are statem ents o f p ractical ev aluations’ (W eb er 1948: 2). As long as research ers
adhere to scien tific standards, th en social scien ce can m ake referen ce to values w ithout
m aking value jud gem ents. It can provide a factu al and ob jectiv e assessm en t o f a cto rs and
th eir practices, an objectively true accou n t o f how th e w orld fu n ctio n s, w ithout attem pting
to tell us what goals we ought to pursue, w hat d irection p olicy o ught to take, or w hat values
ought to be prom oted.
W eb er’s views have rem ain ed in flu en tial and co n tin u e to in fo rm d iscu ssio n s o f th e role
o f values in scien tific inquiry. M any co n tem p o ra ry sch o lars follow W eb er in argu ing that
social in qu iry can be o b jectiv e , d espite th e in tru sio n o f a resea rc h er’s ow n values in th e
research process; th at, w hile ‘im p o rtan t p ractical o b sta c les’ are freq u en tly e n co u n tered in
social scien ce research , that d oesn ’t m ean that it is intrinsically im possible to secu re u n b i­
ased co n clu sio n s in the social scien ces (N agel 1994: 5 8 3 ). The ph ilosop h er, E rn est N agel,
argues that w hile the sele ctio n o f prob lem s m ay be v alu e-o rien tate d , ‘It is no t clear . . .
why the fact that an in vestigator sele cts the m aterial he studies in light o f p rob lem s w hich
in terest him and w hich seem to him to b ear on m atters he regards as im p o rta n t’, p resents
any greater challenge lo the logic o f social in q u iry th an for that o f any o th er b ran ch o f
in quiry (1 9 9 4 : 57 2 ). Nagel cites as an exam ple a research er w ho believ es th at ‘a free e c o n ­
om ic m arket em bod ies a cardinal hu m an value’ and w ho w ishes to in q u ire in to w hich
activ ities are im portant to its op eratio n . W hy, Nagel asks, sh ould we assu m e that his
interest in a free m arket econ om y will bias his ev aluation o f th e ev id en ce on w hich he
bases his c on clu sion s about what is needed for its op eratio n ? The fact that th e in terests o f
the research er d eterm in e what he selects for in vestigation ‘by itself, rep resents no o b s ta ­
cle to the successful pursuit o f ob jectiv ely co n tro lled in q u iry in any bran ch o f study’
(1 9 9 4 : 572).
 O B JE C T IV IT Y A N O V A LU E S

Nagel ack n ow led ges that research ers co n cern ed with social phenom ena are often
c o m m itte d to social values and that these enter into the assessm ent of evidence and the
c o n te n t o f co n clu sio n s. But he argues that steps can be taken to identify a value bias
w hen it o c c u rs , and to m in im ize if not to elim inate com pletely its perturbing effects'
(1 9 9 4 : 5 7 3 ). O n e way to do this is for social scientists to abandon the pretence that they
are free fro m all bias' and state their value assu m ptions as explicitly and fully as they
can ( 1 9 9 4 : 5 7 2 ). He ack n ow ledges that un con sciou s bias and tacit value orientations
a re rarely o v e rco m e by d evou t resolu tions to elim inate bias’; but there are ways, he
a rgu es, to reco g n iz e and c o rre ct for prejudices. For in stan ce, researchers can sift w ar­
ran ted beliefs and retain only those proposed conclusions . . . that survive critical
e x a m in a tio n by an indefinitely large com m u n ity of students, w hatever th eir value prei
e re n ce s o r d o ctrin a l co m m itm e n ts’ (1 9 9 4 : 5 7 4 ). W hile he recognizes that this m ech a­
nism m ay not w ork as effectively as in the natural sciences, he insists nonetheless that we
are not w arran ted on th ese g rou n d s to con clu d e that reliable knowledge of hum an affairs
is u n attain ab le (1 9 9 4 : 5 7 4 ).
Though m uch consideration focuses on the bias that a researcher may pass on to a subject,
it m ay as likely be the case that bias arises in the interaction between researcher and subject.
This likelihood can arise in all research, but it has received a good deal of attention in relation
to eth n ograp h ic research. In ethnographic research, the researcher, while becom ing m ore
im m ersed in the co m m u n ity of study, may find it difficult to remain objective. In some cases,
researchers co m e to identify so m uch with the subjects that they lose the capacity for objec
tive criticism (th is is often referred to as going native ). Ih is source of bias in ethnographic
research is discussed in C hapter 12.
W hile consciousness o f biases and concerted effort to limit their impact may overcom e
som e o f the difficulties involved in separating facts and values, some biases that influence
research findings m ay be m ore difficult for researchers to identify and control.
O n e bias o f this sort is the influence w hich a research er’s expectations exercise on the
results o f an inquiry. R esearchers call this type o f bias the R osenthal Effect', after psy­
ch o log ist R ob ert R osenthal. Rosenthal condu cted a series o f studies that were designed to
d ete rm in e w h eth er the e x p ectation s o f research ers can bias their results. In one study, two
g rou p s o f subjects w ere show n photographs o f people and asked to rate the photographs
a cco rd in g to w h eth er the people in them had recently exp erienced success or failure. The
exp e rim e n te rs w ho adm in istered the test were told what result their group, on average,
was exp e cte d to p rod u ce. Each tim e this experim ent was replicated the results for each
gro u p co n fo rm e d to the e x p ectation s that had been com m un icated to the experim enters.
A se co n d study tested the effect of exp erim en ters’ expectations on the perform ance of
an im als; a th ird e xp erim en t tested the effect of a teach er’s expectations on student p er­
fo rm a n ce . The results o f all the studies showed the sam e thing: that the exp ectation s o f the
exp e rim e n te rs influenced the behaviour o f their subjects. F u rth er study led to the c o n clu ­
sion th at e x p e rim e n te r bias influences human subjects through subtle and com plex audi­
to ry and visual cu e s’; and it influences animal subjects through the way they handle and
o b se rv e th em (M a rtin 1 9 9 4 : 5 8 5 - 7 ) . So, a researcher s exp ectation about the results o f an
e x p e rim e n t affects the o u tco m e o f the exp erim en t. But this type o f bias, M ich ad M artin
argu es, d o es n o t p reclude the possibility o f v alu e-free inquiry. Experim ental techniques
th at m in im ize c o n ta c t betw een exp erim en ter and subjects, and statistical m ethods that
PHILO SO PHY OF SOCIAL SCIENCE

can c o r re c t for bias, m ean th at th e R o se n th a l E ffect need n o t fo reclo se th e p o ssib ility o f

o b je c tiv e so cia l sc ie n c e research (M a rtin 1994: 5 9 3 ).
In sum , th ese and o th er sch o lars con clu d e, like W eber, that even th ou gh values d o in flu ­
en ce research , it is n ev erth eless still possible for social scien ce to b e o b jectiv e and v alue-free.
Th ere are im p ortan t p ractical d ifficu lties in secu rin g v alu e-free and un biased fin d ings in the
social scien ces, but th ese are not in su rm ou n tab le.
However, w hile it m ay be p ossible to lim it or e lim in ate a r esea rch ers ow n values o r biases,
a m ore difficult issue to address is the ten d en cy for th e act o f ob serv atio n itself to c o n ta m i­
nate the su b ject o f study and p roduces biased and un reliable conclu sion s. Th ere are a n u m b er
o f related a rgum en ts about how th is operates.
O n e a rg u m en t is th at h u m an b eh av io u r ch an g es w hile we study it, b ecau se th o se we
are stu d ying alter th eir b eh av io u r w hen th ey kno w th ey are b e in g o b serv ed . T h is p h e ­
n o m en o n is kno w n as th e ‘H eisen b erg E ffec t’. T h e 'H eisen b erg E ffe ct’ deriv es from
W ern er H eisen b erg ’s P rin c ip le o f In d e te rm in a cy in p a rticle p hy sics. H e ise n b e rg s p r in c i­
ple a sserts ‘th a t th e v ery p ro cess o f try in g to o b serv e tin y p a rtic le s in sid e th e ato m alters
th e situ a tio n b ein g stu d ied ’ (M o rg an 1994: 3 0 ). S o m e p o litic a l scie n tists have p o in ted out
that th e b eh a v io u r o f g o v ern m en t officials and b u reau crats c h an g es w hen th ey kno w th ey
are u n d er o b serv a tio n . Th is, in fact, ‘is on e o f th e ju stific a tio n s for hav in g a free press’: th e
scru tin y o f th e press is sup posed to d ete r p o litic ia n s ‘fro m ta k in g a c tio n s th at w ould be to
th eir benefit but co stly to the p u b lic in te re st’ (M o rg an 1994: 3 0 ).
The p h ilo so p h er C h arles Frankel fou n d m an y in sta n c e s o f th e ‘H eisen b erg E ffe ct’ in
the c o u rse o f w ork in g as a U nited States A ssistan t S e c re ta ry o f S tate in ch arg e o f ed u c a ­
tion and cu ltu re from 1965 to 1967. Fran k el d iscov ered th at ‘w hen he sen t ou t letters
m erely to get in fo rm a tio n ab ou t w hat U S officials were d o in g w ith regard to p a rtic u la r
p rog ram m es, he got b ack rep lies in d ic a tin g th at th e officials to w ho m he had w ritten had
chan ged what th ey w ere d oin g after receiv in g his le tte r ’ (M o rg an 1994: 3 7 ). H e rep o rted
that m erely in an effort to in fo rm h im self, he had ‘ap p aren tly p rod u ced ch an g es in p o l­
icy ’, i.e. in the p h e n o m en a w hich he was stu d yin g (F ra n k el 1969: 83 ; qu o ted in M organ
1994: 37).
The problem , then, is that hum an b ehaviou r changes w hile, and because, we are tryin g to
study it. However, there is a variety o f tech niq ues for dealing with this difficulty. For instance,
som e researchers in our field get around the p roblem by u sing ‘con ten t analysis’ (see C hapter
13). By system atically analysing an official’s statem ents for clues to his percep tio ns and a tti­
tudes, researchers can get m aterial on decision -m ak in g w ithout interview ing the d ecisio n ­
makers.
But the problem o f altering behaviou r as a result o f studying it has an o th er facet: b eh av ­
iour may change as a consequence, not only o f the process, but o f the results, o f studying it.
Ihe problem here is how we can know if the th eories we use to explain ou tcom es are true if
they them selves have the effect o f altering the p ractices that produce th ose outcom es. An
example of this is the efiect d eterrence th eory had on weapons deploym ents, con tin gen cy
planning, a rm s-con trol proposals, and relations am ong allies. If th eory alters p ractice, it
becom es, in eflect, a sell-lulfilling prophecy. How then can we claim to be able to separate
and distinguish between the realm s of fact and value? G enerally speaking, the physical sc i­
ences do not lace this problem : planets do not move differently because o f the process or the
results of studying them. But in the social scien ces, th eory can be a self-fu lfilling prophecy.
 OBJ t C T I V I T > A N D V AL UE S 59

It, u s in g th e a u th o r it y o t s c i e n c e , y o u c o n v in c e p o lic y m a k e r s th aT . tor in s ta n c e , w ar is me\

li a b le g iv e n th e a n a r c h i c a l n a tu r e o t th e in te r n a t io n a l s\ s te in , th a n it i> ine\ lia b le an d it
b e c o m e s im p o s s i b le to test th e p r o p o s itio n s c ie n litic a llv (M o r g a n W 4 I '1 2 i.
1 h e r e a r e tw o o th e r a r g u m e n ts c o n c e r n in g how th e a d o r p r o c e s s o l o b s e r v a tio n c o m p r o
m is e s o u r a b ility to s e p a r a te l a d s a n d \ a lu e s P e rh a p s th e b e st was to m a k e J e . i t th e ir s ub
s ta n c e a n d im p li c a tio n s is by b r ie fly r e v is itin g K a rl P o p p e r s f a ls if ic a t io n s ! th e s is i
w h ic h w e d is c u s s e d in ( .h a p t e r 2. P o p p e r a r g u e d th a t f a ls if ic a tio n , n o t v e r ific a tio n , s h o u ld
b e th e a im o t s c i e n c e ; th a t s o c ia l in q u ir y m u st p r o c e e d d e d u c tiv e lv . th ro u g h a p r o c e s s m
w h ic h o b s e r v a t io n s a re d e r iv e d fr o m a n d u sed to test th e o r ie s , bu t th a t th e a im o l our te s ts
m u s t b e to fa lsity ', r a th e r th a n v e rity , th e m W e r e ie d th o s e w h ich a re fa ls ifie d but c o n tin u e
to test th o s e th a t a r e n o t u n til th e y b e c o m e so th o r o u g h ly te s te d th a t we van c o n s id e r th e m
to b e c o n fir m e d ', th o u g h it r e m a in s p o s s ib le th at s o m e d a v s o m e o n e ma\ falsify o r sig m ti
c a n tly m o d if y th e m .
O n e o b je c t i o n th a t h a s b e e n m a d e to th is a r g u m e n t is th a t P o p p e r s a r g u m e n t a b o u t fal
s if ic a tio n s e e m s to a s s u m e th a t o b s e r v a tio n s o r t a d s w e u se to test th e o r ie s ».an b e e s ta b
lis h e d in d e p e n d e n tly o t th e t h e o r y th a t th e v a re m e a n t to test Hut I h o m a s K u h n ( 1 ^ 2 ) an d
o t h e r s h a v e a r g u e d th a t o b s e r v a tio n is 'th e o r y la d e n th a t o u r o b s e r v a tio n o t t a d s c a n n o t
b e s e p a r a te d fr o m th e t h e o r e tic a l n o ti o n s w h ic h g iv e in te llig ib ility to w hat we o b s e r v e
O b s e r v a t i o n is th e in te r p r e ta tio n ot a p h e n o m e n o n in th e lig h t ot s o m e f/ itv n a n d o th e r
b a c k g r o u n d k n o w le d g e . In ta c t, w e m ig h t say th a t a p h e n o m e n o n is an a lr e a d y in te r p r e te d
r e g u la r ity ( o r e v e n t) , l o r in s ta n c e , a s K u h n e x p la in s ( 1 9 6 2 l i t ) , w h e n w e lo o k at th e pic
tu r e s h o w n in P ig u r e 3.1 w e d o n o t m e r e ly o b s e r v e c u r v e d lin e s o n a p a g e w e s ee e ith e r a

r a b b it o r a d u c k .

Figure 3.1 Duck or rabbit7

Source Jastrow.J (1899)
PHILO SO PHY OF SOCIAL SCIENCE

Can we perceive w hat th ese lin es prod uce irresp ectiv e o f an act o f in terp retatio n ? W h ile
th e sam e d ata lend th em selv es to different in terp retation s, th e p erceptu al ex p erien ce and
th eoretica l in terp retation are sim ultaneou s: th e ‘facts’ do no t present th em selv es to us in d e­
p en d ently o f o u r th eoretical beliefs. W h at we ob serv e th is o b je ct to be is d ep en d en t on how
we d escrib e it. That is, if we are told th at ‘th is is a p ictu re o f a duck’, we will likely see a duck;
if we are told that it is a rabbit, we will probably see a rabbit. The reality is no t in d ep en d en t
o f a d escrip tion o f th at reality.
A seco n d o b je c tio n to P o p p er’s n o tio n o f falsifiab ility h ig h lig h ts a n o th e r p ro b lem w ith
th e fa ct-v a lu e d istin c tio n . P op per assu m es th at th e scie n tific e n te rp rise invo lv es th e r ig ­
orous, sy stem a tic ap p licatio n o f r easo n , so th at w hen th e o rie s are falsified by e x p e rie n c e
th ey are th row n out, irresp ectiv e o f how m an y im p o rta n t p eop le have in vested th e m ­
selves and staked th eir care ers in p ro m o tin g th em . But som e p eop le q u estio n w h e th er th is
assu m p tion is w arran ted. Is th is actu ally th e way scien tists go ab ou t d ev elop in g and te s t­
ing th eories? Is it a ctu ally th e case th at research ers seek to d isprov e o r falsify th e ir ow n
th eories? Do th ey d iscard th eir th eo ries w hen th ey are co n fro n te d w ith d isco n firm in g
evid en ce?
The p hilosoph er o f scien ce, T h om as K u hn, argued th at Pop per p resents an idealization o f
true scien ce; that, rather than un relenting c riticism and testing, scien ce has tended tow ards
paradigm atic con fo rm ity and co nserv atism . In The Structure o f Scientific Revolutions (1 9 6 2 ),
Kuhn argues that scien ce is a social in stitutio n. It con sists o f a co m m u n ity w ithin w hich
there is a com m on view — w hat Kuhn calls a ‘parad igm ’: a co n cep tu al schem e, ab out w hich
som e com m un ity o f scholars agree, and w hich defines th e o b jects o f investigation and the
ways in w hich they are to be investigated. The im p lications o f this argum en t for th e fact-
value d istinction and, thus, for the o b jectivity o f scien tific know ledge, has m ade K u h n s b o o k
a focus o f controversy for nearly h a lf a century.
In the next section we consid er Th om as Kuhn’s in fluen tial argum en ts co n ce rn in g para­
digm s and how they change throu gh scien tific ‘revolutions’. A fter reflectin g o n th eir im p lica­
tions for the possibility o f value-free social inquiry, we th en explore the im p lications o f Im re
Lakatos’ rival notion o f ‘scien tific research program m es’.

Values and social practice

So far, the d iscu ssion has focu sed on how a re se a rc h e r’s v alues in tru d e on th e research
process. But to ad equ ately address th e q u estio n o f h ow values in flu en ce so cia l research
we need also to co n sid er the in flu en ce o f values asso ciated w ith a m o re g en eral set o f
shared social p ractices.
Social-scientific research is often portrayed as b eing unaffected by the con text and en v i­
ronm ent in which it is conducted. However, researchers live w ithin a given society, and they
work within a system o f academ ic incentives and d isincentives that reward som e activities
and discourage others. It would seem unreasonable to suppose that this system , including
the gate-keeping functions of the academy, the creation and en fo rcem en t o f discip linary
norm s, rewarding good behaviour and pu n ishing— or at least ign orin g— bad behav iou r’
(Cierring 2001: 5), public and private funding, publication outlets, universities, governm ent,
and the mass public, docs not affect researchers and their work. The q u estion is w hether such
 OBJECTIVITY AND VALUES

in flu en ces invalidate the possibility of a neutral social science. In the next section, we discuss
tw o in fluen tial perspectives on this question.

Thomas Kuhn and scientific revolutions

Ih om a s Kuhn ( 1 9 2 2 - 9 6 ) is widely considered to have been one of the most influential
p hilosoph ers o f s cien ce of the tw entieth century, perhaps the m ost influential' (Bird 2009)
H is b o o k , ih e Structure o f Scientific Revolutions is one o f the most cited academ ic books of
all tim e' ( Stanford Encyclopedia o f Philosophy). By highlighting the value laden social prac
tices w ithin w hich scien tific in quiry takes place, the book presents a com pelling challenge to
the n o tion that it is possible to separate facts and values in scientific scien ce research.
In Ih e Structure o f Scientific Revolutions (1 9 6 2 ), Kuhn looks at the history of science and
con clu d es that the growth o f knowledge is not a logical process but a social one Science does
not sim ply progress by stages based upon neutral observations, and through cum ulative,
g rad ual know led ge a cq u isitio n . S cien ce is an essentially social in stitutio n. Scien tists are
socialized , not to pursue un relenting criticism and testing, but to accept the reigning values,
beliefs, concep ts, and rules o f their profession. C onsequently, there is no theory independ
e n t’ view o f the w orld; the world appears to us in the context o f th eories we already hold: it
is ‘th eo ry -la d en ’.
Kuhn argued that scien ce tends to be dom inated by 'paradigm s'—conceptual schem es
about w hich som e c om m u n ity o f scholars agree, and which define the ob jects o f mvestiga
tion and the ways in w hich they are to be investigated. O n ce a paradigm is established, it
d irects scien tific investigation through the operation o f what Kuhn calls 'norm al science'.
Th e aim o f ‘no rm al scien ce’ is not 'to call forth new sorts o f phenom ena', but ‘to force nature
into the p reform ed and relatively inflexible box that the paradigm supplies'; indeed 'those
that will not fit the b ox are often not seen at all’ (1962: 24).
Th ough K u hns b ook was con cern ed with the natural scien ces, it caused a great crisis o f
ob jectiv ity in b oth the natural and the social scien ces. Kuhn's argum ents threw open to qu es­
tion th e status o f scien ce as a rational and progressive developm ent towards ’truth', challeng­
ing the d istin ctio n betw een natural and social scien ce, and betw een social scien ce and
ideology.
As we have previously discussed, positivists m aintain that if the study o f politics is to be sci­
entific, it must adopt the sam e scientific principles that the natural sciences use to study natural
phenom ena. But Kuhn challenges the notion that the natural sciences are objective or value-
free; that scientists apply the neutral instrum ent o f ‘scientific m ethod' to independently existing
reality, that the growth o f scientific knowledge is piecem eal and cum ulative; and that theories
are discarded as ‘falsified’ if they fail (B all 19 7 6 :1 5 8 ). Based on his reading o f the history o f sci­
ence, Kuhn argues that the validity o f truth-claim s is grounded in the consensus o f som e schol­
arly com m u n ity; that what scien ce m eans by 'truth' is the consensus o f those it defines as
com peten t and full m em b ers o f its com m unity. Academic jo urnals, hiring, and curricula are
orientated tow ards certain explanatory fram ew orks and conceptual schem as, and research
using o th er fram ew orks and schem as is undervalued and m arginalized within the profession.
These argum ents challenge the distinction in science between fact and value a n d conse­
quently, ou r faith in the authority o f ‘science. Science is essentially sociaL It consists o f a co m ­
munity within which there is a com m on view. This com m on view is what Kuhn calls a ‘paradigm!
PH ILO SO PH Y OF SOCIAL SCIENCE

Paradigms and paradigm change

In The Structure o f Scientific Revolutions, K u hn d e sc rib e s th e d ev elo p m en tal p attern o f
sc ie n c e in term s o f p arad igm sh ifts. For K u hn, th e h isto ry o f scie n ce is th e h isto ry o f tr a n ­
sitio n from o n e p arad igm to an o th e r via ‘rev o lu tio n ’. W h a t K u hn calls a ‘p arad ig m ’ is a
co n c e p tu a l sch e m e, ab o u t w h ich som e co m m u n ity o f sc h o la rs agree, and w h ich defin es
th e o b je c ts o f in vestigation and th e ways in w h ich th ey are to b e investigated . A p arad igm
sh ift o c c u rs w hen an an o m aly o r an o m a lies arise w hich v iolate th e assu m p tio n s o f th e
reig n in g parad igm .
Kuhn elab orates a m odel o f the developm ent o f scien ce that focuses on th e con solid ation
o f paradigm s, and p rocesses w hich shift th e c o m m itm en t o f a com m u n ity o f sch o lars from
one paradigm to another. The m odel starts by elab oratin g the features o f what Kuhn calls
pre-paradigm atic scientific investigation. Th is is a pre-scien tific phase in th e d evelopm ent o f
a scien tific field in which investigation is rand om and diverse. The ab sen ce o f a p aradigm or
som e candidate for one m eans that ‘all o f th e facts that could possibly p ertain to the dev elop­
m ent o f a given scien ce are likely to seem equally relevant’. As a result, ‘fact-g ath erin g is a far
m ore nearly random activity th an the on e that subsequen t scien tific developm ent m akes
fam iliar’ (K u hn 1962: 15).
D uring the period o f pre-parad igm atic in vestigation, a variety o f sch o ols o f th ough t c o m ­
pete for our a ttention. But in the next step in the m odel, on e o f the pre-parad igm atic sch o ols
trium phs, leading to the establishm ent o f a paradigm . It is then that scien tific investigation
becom es highly directed and focused. Paradigm s generate ‘particular coheren t trad itions o f
scien tific research’, reflected in textb ooks, d epartm ents, degrees, appointm ents, prom otions,
jou rnals, conferen ces, and ho nours and awards (1 9 6 2 : 10). The establishm ent o f a paradigm
in augurates a period o f what Kuhn calls 'norm al science', o f scien ce that operates w ithin the
lim its o f a paradigm and co n ce rn s itself with testing and w orking on ‘puzzles’ via bits o f
research that are im plied by the paradigm . It consists p rim arily o f developing the paradigm
‘by extend ing the know ledge o f th ose facts that the paradigm displays as p articularly rev eal­
ing, by in creasing the extent o f the m atch betw een th ose facts and the paradigm ’s p red ic­
tions, and by further articu lation o f the paradigm itse lf’ (1 9 6 2 : 24). This involves b oth
em pirical and th eoretical work. The e m pirical work consists o f fact gath ering, im proving the
accuracy and scope o f factual d eterm inations, m atchin g facts with theory, co m in g up with
quantitative measures, and applying the paradigm to new areas and different settings. N or­
mal th eoretical work involves using the th eories generated by the paradigm to predict, and
developing new' or m ore precise applications o f them . But w hether em pirical or th eoretical,
the work o f norm al scien ce does not raise qu estions about the paradigm itself. It treats the
paradigm as a given; it does not pursue critical exploration o f its core assum ptions. It
engages, instead, in a strenuous and devoted attem pt to force nature into the conceptual
boxes supplied by’ the paradigm (1962: 5). N orm al scien ce proceeds on the assum ption ‘that
the scientific com m unity knows what the world is like’, and scien tists aggressively defend
that assum ption.
It is possible, however, for research w ithin the fram ew ork o f the parad igm , or real world
events and processes, to bring to light ano m alies for the paradigm . An anom aly is so m e ­
thing which deviates from the general rule; som eth ing which is in consistent with the basic
assum ptions. But anom alies are not grounds for rejectin g a parad igm ; they generally do
 O B JE C T IV IT Y A N D V A LU ES

n o t even cau se a critic a l review o f it. However, if the anom alies grow, or it an anom aly is
very fu n d a m en ta l— if it th reaten s to u n d erm in e the very foun dation o f the p arad igm —
th is m ay lead to a crisis. A n ano m aly m ay also evoke a crisis it it has som e im m ediate
p ra ctica l im p ort. In that case, the field starts to revolve around the problem and a search
for its so lu tio n , and the field may con seq u en tly b egin to look different. However, scientists
resist chan ge. 'Ih e y will set the prob lem aside, or devise m o d ificatio ns in order to sh o e­
h o rn th e an o m a lo u s fact in to the ex istin g paradigm . And even w hen the crisis begins to
b lur th e p a ra d ig m — to loosen th e rules o f n o rm al research in the held — th eorists will not
d eclare th e crisis-rid d e n p aradigm invalid unless th ere is som e alternative paradigm avail
ab le to take its place.
O n c e a cand id ate for a new paradigm em erged, th ere is a battle over its acceptance, finally,
th ere is a transition from a paradigm in crisis to a new one from which a new tradition o f
n o rm al scien ce em erges. Ih is constitutes a revolution in the field. As in a political revolu­
tion , in a ‘scien tific revolution’

1. a m a lfu n ctio n leads to crisis;

2. th ere is a loosen in g o f rules as basic in stitutio ns are rejected,

3. c o m p etin g cam p s form : there are those who seek to defend the old order, and those who
seek to in stitu te so m eth in g new;

4. th e com p etin g cam p s seek support by resorting to tech niques o f mass persuasion;

5. sc ien tists tran sfer th eir loyalties from the old to the new paradigm , and this in augurates
a new era o f n o rm alcy o r ‘norm al scien ce’.

F igu re 3.2 d epicts these steps and how they are related.

Paradigm is established

Generates new
[ N«wparadigm ]

Anomalies build

over the others

Competition among
contending candidates
for a successor

Figure 3.2 Thomas Kuhn. The Structure o f Scientific Revolutions

P HILO SO PHY OF SOCIAL SCIENCE

Th e im p lication s o f th is m o d el are sign ificant for th e way we u n d erstand scien tific claim s.
T h e n o tio n o f'n o r m a l scien ce, as elab orated by K u hn, suggests th at th e r ation ality o f scien ce
is lim ited ; that ju st as ad heren ts o f an id eology o r religion will discard an ano m aly o r d isco n -
firm in g fact or in co rp o rate it w ithin the ideology, scien tists w ill ‘often suppress fu n d am en tal
novelties b ecau se th ey are necessarily subversive o f th eir ‘b asic co m m itm e n ts’ (1 9 6 2 : 5).
Sim ilarly, the shift from on e paradigm in favour o f an o th er is no t th e result o f a ration al d e ci­
sion but so m eth in g rather m o re like a con v ersion ex p erien ce. P arad igm s shift w hen s c ie n ­
tists chan ge th eir m ind s about fun dam en tal m atters.
Th is represents a clear challenge to the p ositivist ideal o f scien tific progress. Positivists see
the ch oice betw een tw o co m p etin g th eories as resolved by a ‘cru cial ex p erim en t’ w hose
results sup ports on e th eo ry and refutes th e other. But, for K uhn, th e sh ift from o n e p a ra ­
digm to a n o th er n either results from no r prod uces cum ulative know ledge b ecau se different
paradigm s prod uce incom m ensurable know ledge. Paradigm s are in co m m en su rab le b ecause
they d isagree about what the data actu ally are; so parad igm ch o ice can never be settled by
logic and ex p erim en t alone. It involves, at least in part, an act o f faith.
In sum , a parad igm con sists o f a set o f th eo ries th at sh are, not on ly c o m m o n a ssu m p ­
tions, but in co m m en su rab le co n ten t with resp ect to som e o th er paradigm . Th is is b ecau se
p aradigm s do no t co n tain th eo ry -n eu tral facts that can serve as a b asis o f com p ariso n . All
ob serv ation is th eory -lad en : ‘W h en A ristotle and G alilei look ed at sw in ging stones, th e first
saw constrained fall, the secon d a p en dulum ’ (K u hn 1962: 121). They relied on such d iffer­
ent on tological a ssu m ptions that th ey were in co m m en su rab le; so no neu tral m easure could
be found to assess on e against the other. The arbiter, Kuhn argues, is peer consensu s. And
the new ‘paradigm ’, or way o f look in g at the world, m ay not co n tain any o f th e ideas o f the
old one.

Kuhnian paradigms and political studies: the case of development theory

As previously noted, Kuhn’s m odel o f scien tific progress was developed to d escrib e the n a t­
ural, not the social, scien ces. Kuhn considered the social scien ces to be ‘pre-p arad ig m atic’
and u n likely to ever b e c o m e ‘m atu re s c ie n c e s’, in th e sen se o f th e p ara d ig m -d riv en
investigation that c h a ra c teriz ed th e natu ral s cie n c e s (K u h n 1962: 1 6 4 - 5 ) . H ow ever,
m any social sc ien tists saw K u hn’s e la b o ra tio n o f th e role o f p arad ig m s in s c ie n tific
dev elop m ent, ‘as an e x tre m ely sugg estiv e d escrip tio n o f th e ir ow n d isc ip lin e s’ (G u ttin g
1984: 1). C o n sequ en tly, th e idea that p arad igm s d irect and sh ape research qu ick ly
beca m e ‘widely a ccep ted in the so cial s c ie n c e s’ (B e r n ste in 1981: 4 2 5 ). T h e n o tio n had a
‘profoun d in flu en ce’ on p o litical sc ie n c e (B a ll 1976: 15 1 ). P o litical scie n tists applied the
term 'p arad igm ’ to sp ecific m e th o d o lo g ica l and th e o re tica l sc h o o ls o f th o u g h t such
as b ehav iou ralism and ration al ch o ice th eory, and d irected th eir e n e rg ie s tow ards
esta blish in g p arad igm s in In tern a tio n a l R elatio n s and C o m p arativ e P o litics (S ch m id t
2 0 0 2 : 9).
Ih e term paradigm ’ has been defined in a variety o f ways, even by Kuhn him self. In the
study ol politics, the term is applied to world views or scho ols o f thought that provide a
theoretical architecture for pursuing the study ol the political world. Each paradigm c o n ­
sists ot a set ot assu m ptions that provide a basis tor the developm ent o f th eo ries that
 OB J t C
65
e x p la in p a r t ic u la r e v e n ts o r c a te g o r ie s o l e v e n ts , a n d th a t e n a b le us to d e c id e . (ru m a m o n g
th e u n iv e r s e o l p o liti c a l p h e n o m e n a , w h ic h a re im p o r ta n t to stu d v D iffe re n t b e h a v io u rs
a n d p r e s c r ip t i o n s to r a c t io n How fr o m d if fe r e n t s e ts o ! a s s u m p tio n s It vou a s s u m e p e o p le
a r e c a p a b le o f a ltr u i s m , m u tu a l a id , a n d c o ll a b o r a t i o n , you m ig h t b e in fa v o u r of p u r s u in g
c o o p e r a t io n a n d c o ll e c t iv e a c tio n to re s o lv e p r o b le m s a n d a d v a n c e g lo b a l w elfa re It vou
a s s u m e th a t h u m a n s a r e d r iv e n , first a n d f o r e m o s t , In an in s tin c t to r p o w e r, \our c h o ic e s
m ig h t b e m o r e fa ta lis tic a n d s u m v a l i s t . It vo u a s s u m e th a t p e o p le are b a s ic a lly a p r o d u c t o l
th e m a te r ia l c o n d it io n s a n d s o c ia l r e la tio n s w h ic h d o m in a te th e ir life . th e n , it vou w ish to
c h a n g e th e w o r ld , y o u m a y s e e k to c h a n g e th e m a te r ia l e n v ir o n m e n t w h ich s h a p e s h u m a n
th o u g h t a n d a c t io n I h is g e n e r a l u s a g e o f th e te r m p a r a d ig m in th e stu d v of p o litic s is
s u m m a r iz e d in B o x V 1

I .e t s c o n s id e r an e x a m p le o l h o w th e n o tio n of p a r a d ig m h a s b e e n a p p lied to a sp ecific

a r e a o f in q u ir y w ith in p o litic s : th e stu d y o f d e v e lo p m e n t

Paradigms and the study of development

S in c e W o rld W a r II, th e field of d e v e lo p m e n t s tu d ie s h a s b e e n d o m in a te d In tw o th e o r e tic a l

p e r s p e c tiv e s : m o d e r n iz a tio n the« >ry a n d d e p e n d e n t v th e o r y ' I h e s e per spec ti \ es ha \ e b e e n
f r e q u e n t ly r e fe r r e d to as p a r a d ig m s ' ( B o d e n h e im e r i y ~ l ; f o s te r ( a r te r l irr> , V a le n z u e la
a n d V a le n z u e la 1 9 7 9 ; I’a v lic h ( u d d e s 2<MM) B a rb a ra ( ie d d e s a r g u e s th a t, w h ile nei
th e r p e r s p e c tiv e h a s o p e r a te d as a ‘ K u h n ia n h e g e m o n ic p a ra d ig m ', thev h a w 'm o s t ol th e
o t h e r f e a tu r e s K u h n a ttr ib u te s to p a r a d ig m s ' ( 2 0 0 V (-> 7). I a d i is c o m p r is e d o l c o lle c tio n s ol
t h e o r i e s , h y p o th e s e s , a p p lic a tio n s , a n d fa v o u re d m e th o d o lo g ie s ( 2(H) ^ 21 t In e a c h , a set of
t h e o r i e s s t r u c t u r e s r e s e a r c h , in c lu d in g id e n tif y in g w hat q u e s tio n s n eed to b e a d d re s se d an d
w h a t c o n s t it u t e s e v id e n c e . M o d e r n iz a tio n th e o r y is b a s e d o n 1 ib e ra l an d n e o 1 ib e ra l
a s s u m p tio n s , w h ile d e p e n d e n c y th e o r y is b a s e d o n M a rx is t a n d n e o .M arxist a s s u m p tio n s
H ach c o n s e q u e n t ly d e s c r ib e s a n d e x p la in s th ir d w o r ld ' u n d e r d e v e lo p m e n t in s tr ik in g ly dil
f e r e n t w ays.
A p a r a d ig m f o r th e s tu d y o f d e v e lo p m e n t e m e r g e d in th e 1 'm te d S ta te s d u r in g th e
im m e d ia t e p o s t - W o r ld W a r II d e c a d e s , a n d it r e f le c te d th e s o c ia l c o n te x t o l th a t p la c e a n il

b o x 3.1 Paradigms in Political Research

Paradigms
• arise within a specific social context

• consist of a set of assumptions about the nature of an area of study

• help to explain events and processes, and provide a guide to action or policy
• lose their usefulness when events or circumstances arise which violate the assumptions of the
paradigm, and are supplanted by other paradigms, containing different assumptions
PHILO SO PHY OF SOCIAL SCIENCE

tim e. U S foreig n p o licy follo w ing W orld W ar II was c o n c e rn e d to e n su re th at th e ‘d ev elo p ­

ing w orld ’ w ould n o t be draw n in to th e Sov iet co m m u n ist b lo c ; and it w as th is c o n ce rn
th at a n im a ted th e U S g o v e rn m e n t to e n list so c ia l sc ie n tis ts to study an d d ev ise ways
o f p ro m o tin g ca p ita list e c o n o m ic d ev elop m en t and p o litica l stab ility (G e n d z ie r 1 985;
D ia m o n d 19 9 2 ; Sim p so n 19 9 9 ). By th e early 1950s, th e study o f ‘d ev elo p m en t’ was e sta b ­
lish ed as a key sc h o la s tic p r o je c t and , th ro u g h g en ero u s fu n d in g an d in s titu tio n a l
in d u cem en ts, it c o n tin u ed to attra ct a steady and e v er-exp an d in g flow o f research and
w ritin g fro m acro ss th e so cial scien ces. W ith in a cou p le o f d ecad es, th e study o f d ev elo p ­
m ent had m ark ed ly conv erged arou n d a c o m m o n set o f a n aly tic co n v en tio n s and g en eral
th em es. Th ese co n v en tio n s and th em es eventu ally b eca m e th e b asis for w hat is referred to
as ‘m o d ern iz a tio n th e o ry ’.
M o d ern iz a tio n assu m ed th at u n d erd ev elop m en t was a stage in d ev elo p m en t th ro u g h
w hich all n a tio n s m u st pass, and th at p rogress w ould ev en tu ally co m e a b o u t th ro u g h th e
spread o f m o d ern ca p italism to b ackw ard , tra d itio n a l areas. B ut, co n s iste n t w ith stran d s
o f n eo -L ib era l th ou g h t cu rren t in p o st-W o rld W ar II A m e ric a , it m ain ta in ed th at d ev el­
op m ent was not so m eth in g that cou ld be ach ieved th ro u g h m o re -o r -le ss natu ral p r o ­
cesses o f e c o n o m ic g row th , but o n ly th ro u g h th e in terv e n tio n o f m o re a d v anced co u n tries.
Prob lem s o f d ev elop m ent, it m ain tain ed , result fro m w eakn esses in th e v ario u s fa cto rs o f
p rod u ction — land, labour, and cap ital— w ithin the d eveloping co u n tries; and th e way th ese
weaknesses could be o v ercom e is th rou gh trade, foreign in vestm ent, and aid from ad vanced
capitalist coun tries. U ltim ately, and th rou gh co n ta ct w ith th e d eveloped m arket e co n o m ies,
the diffusion o f capital and tech n o lo g y w ould prod uce ec o n o m ic dev elopm ent in th e b a c k ­
ward coun tries.
By the end o f the 1960s, a key anom aly had em erged for the m o d ern ization paradigm :
‘m od ernization did not appear to be occu rrin g . The paradigm had generated th e e x p ec ta ­
tion that the d evelopm ent o f ind ustrial capitalism in ‘th ird world’ cou n tries would enable
them to achieve the cond ition s o f life that exist in th e advanced in dustrial cou n tries o f ‘th e
W est’. However, it soon b ecam e clear that converging levels o f in d ustrialization had not p ro ­
duced a conco m itant convergen ce o f in com es; and that the divide in in com e and wealth
between the ‘developed’ and ‘developing’ w orlds was, in fact, w idening. For w hole societies
and large populations w ithin th em , industrial ‘developm ent’ appeared to b e p roducin g c o n ­
ditions o f life that were worse and not better.
In the late 1960s, a theoretical perspective em erged that offered an explanation for this
anomaly. Ihe overall conception it advanced was o f capitalist developm ent as a w orldwide
process that, as the result o f the appropriation o f surplus by advanced co u n tries from th ose
in the ‘developing’ world, delivers unequal benefits and produces structurally different
developm ental outcom es. This perspective form ed the basis o f what cam e to be know n as
dependency th eory ’.
D ependency theory not only challenged m any o f the core assu m ptions o f m odern ization
theory, it turned m odernization theory on its head. W hile m od ern ization th eorists focused
on the benefits o f free trade, foreign investm ent, and foreign aid, d epen dency th eorists
argued that international m arket relations o ccu r in a fram ew ork o f uneven relations betw een
developed and underdeveloped coun tries and work to reinforce and reproduce these rela­
tions. Dependency theorists argued that the in ternational econ om y is under the m o n op olis­
tic control of the developed econom ies. Free, unregulated in ternational trade and capital
 O B JE C T IV IT Y A N D V A LU E S

m ovem ents tend to favour the already well-endowed and work to accentuate, not dimmish,
international inequalities. A m ore equitable distribution of benefits, they argued, could not
o c c u r within the international capitalist system; and. in consequence, third world' countries
will be unable to m ove beyond limited industrialization.
But m od ern isation theorists and other critics soon pointed to a key anomalv for depend
en cy th eory: the dem onstrated capacity of some third world’ countries to industrialize and
achieve rapid econ o m ic growth. Ihe response of dependency theorists was to insist that,
though the m an u factu rin g sector in many third world’ countries has grown dramatically,
these sectors are controlled by m ultinational corporation s with headquarters in the advanced
cou n tries; and. w hatever benefits they may bring in the form of managerial and technology
cal know -how , they take m ore than they give and make it impossible tor these countries to
achieve the conditions o f life that exist in the advanced countries. While critics acknowledge
that m any problem s of development persist in countries that have achieved substantial
industrial grow th, they argue that the explanation that dependency theorists offer for them
is unconvincing, that it fails to sufficiently account for different developmental outcom es,
vastly exaggerates the pow er of the international system to shape outcom es in developing
cou n tries, and system atically ignores the role of internal factors in producing and reprodu­
cin g these problems.
The debate rem ained at a stalemate. Adherents of both perspectives failed to undertake
the so rt o f vigorous c riticism , em pirical testing, o r theoretical investigation that might open
the way to radical revision, synthesis, or consensus. Instead, modernization theorists and
d ep en d en cy theorists continued to reassert their com m itm ent to the core assumptions of
th eir respective paradigm s: one set of theorists continually urged the necessity for greater
m arket freedom ; an oth er set continued to focus on the enduring legacy of colonialism and
the co n tin u ed relevance o f im perialism . W'ith developm ent theory at an impasse, many
th eorists sim ply u ndertook to reduce expectations of development, so that, by the second
half o f the 1990s, the norm al usage o f the term “developm ent" had mutated to mean a m o d ­
est in crease in “ in d u strialisation”... an improved capacity to produce textiles, sports shoes,
furniture, o r even a solid G D P based on the export of resources, agricultural products, or
low -tech industries (M ason 1997: 40 9 ).
The field o f developm ent studies appears to m any observers to have been driven, less by
the findings o f rigorou s scientific investigation, than by incom m ensurable and irreconcila­
ble ideological co m m itm en ts. Barbara Geddes has argued that inquiry into development
has been ch aracterized by the succession o f one untested theory after another, bringing us
no clo ser to tru th but, instead, em broiling us in fruitless, often ideologically weighted
d ebates’ (2 0 0 3 : 2 2 3 ). The problem with the two perspectives, she argues, is not so much
th eir in cap acity to resolve anom alies, as the paradigm atic conform ity that led researchers to
ignore discon firm in g evidence. In each paradigm , evidence was available that should have
called th em into question at the tim e of their creation’. Yet, it took ‘decades to notice the
p leth ora o f inconvenient facts’ (2 0 0 3 : 7 - 8 ) , because researchers failed to make use of all
a v a ila b le inform ation* in the form ulation o f theories, and con su m ers were willing to
acce p t th eo ries w ithout strong supporting evidence (2 0 0 3 : 17). Analysts used evidence
selectively ‘to develop, su p port, or test theories’: the m odernization paradigm used evi­
d en ce from only a few N orth A tlantic countries’, while earlier versions o f dependency th e­
o ry ‘ign ored readily evidence o f rapid grow th’ in developing countries’ (2 0 0 3 : 1 7 -1 8 ).
P HILO SO PHY OF SOCIAL SCIENCE

D ep en d en cy th eo ry was revised in resp on se to ev id en ce o f in d u strial grow th in d ev eloping

c o u n tries. H ow ever, ‘cro ss-n a tio n a l studies th at challenged th e im p licatio n s o f th e revised
d ep en d en cy parad igm were ign ored ’ ( 2 0 0 3 :1 5 ) . She no tes th at sch o lars find p ersuasive and
are draw n to a th eo ry b ecau se o f its intuitive, em o tio n al, and id eological appeal. T h ey prefer
on e th eory o ver a n o th er less for its ex p lan atory ab ilities th an b ecau se o f its co n s iste n cy with
th eir ow n in terests, b eliefs, values, and personal m o rality ; and w hen a th eo ry fits w ith th eir
p ersonal ex p erien ce o r id eological c o m m itm en ts, th ey are less in clin ed to dig deep for dis-
co n firm in g ev id en ce. W h at research ers see is essentially related to th eir th eo ries and beliefs.
A vailable in fo rm a tio n that is in co n sisten t w ith a research er’s p reco n cep tio n s is disregarded.
In th is way, parad igm s b eco m e , in effect, a m eans o f exclud in g o th er in terp retatio n s and
th eir policy im plications.
T h e se re fle c tio n s are c o n s iste n t w ith K u hn’s n o tio n o f s c ie n c e as ‘s o c ia l’, as are G ed -
d es’ a c c o u n t o f th e sh ift from on e d ev elo p m en tal m o d el to an o th er. B a rb a ra G ed d es
o b serv es th at ‘E a ch p a rad igm lo st its pow er to s tru c tu re resea rch w hen e v id en ce o f g ross
in c o n sis te n c y b etw ee n e x p e c ta tio n s a risin g fro m th e o rie s w ith in th e parad ig m an d real-
w orld ev ents b e c a m e im p o ssib le to ig n o re’ (2 0 0 3 : 7 - 8 ) . W h en th is o c c u r re d , m o d e r n i­
zation th eo ry was o v erth ro w n by d ep en d en cy th e o ry in th e sam e way th at p o litica l
reg im es are top pled : by ‘w ell-o rg an ized , c o h e re n t, m o b ilized o p p o s itio n ’. T h e su b s e ­
q u en t d em ise o f d ep en d en c y th e o ry was due to its ow n ‘in te rn a l c o n tra d ic tio n s and
in a b ility to deal w ith th e in co n v e n ie n t fa c ts th row n up by th e w o rld ’; and , like th e c o l­
lapse o f a regim e, th e fall o f th e d ep en d en cy p arad igm was follow ed by a p erio d o f ‘c h a o s
and co n te n tio n ’ (2 0 0 3 : 7 ).
W h ile K u hn’s a rg u m en ts co n tin u e to p rov oke reflec tio n on th e n atu re o f th e so c ia l-
scien tific enterprise, a second, con trastin g con cep tion o f th e nature o f scien ce and how it
advances has also proved highly influential. But, b efore tu rn in g to a consid eration o f this
second perspective, it is w orth noting a position that takes seriously Kuhn’s no tion o f ‘n o r­
mal scien ce’ and calls for a ‘p ost-n orm al scien ce’.
The notion o f ‘post-n orm al scien ce’ was advanced by Silvio Funtow icz and Jerom e Ravetz
(1 9 9 2 , 1993) to d escribe an approach that is com p lem en tary to, but different from , co n v en ­
tional or ‘norm al’ science. They argue that, while ‘norm al scien ce’ is appropriate in situations
with low levels o f un certainty and risk, it is not suitable in situations when either decision
stakes or system un certainties are high. W ith low levels o f un certainty and risk, the standard
techniques and procedures o f scien tific inquiry, and the peer review process for ensuring the
quality and validity o f results, rem ain appropriate. But Funtow icz and Ravetz argue that
‘traditional problem -solving strategies o f scien ce . . . need to be enrich ed to solve the p rob ­
lem s that our scien ce-based industrial civilization has created’. They call for a ‘d em ocratiza­
tion o f science that entails, not only in corp orating m ultiple view points into the process o f
inquiry, but extending the peer com m unity. Their notion o f p ost-norm al scien ce is sum m ed
up in this excerpt, below.

Ihe problem situ atio ns that involve p o st-n o rm a l scien ce arc o n es w here, typically, facts arc
un certain, values in d ispute, stakes high, and d ecisio n s u rgent. B ecause applied s cien ce and
profession al con su lta n cy are in ad eq uate, so m e th in g ex tra m ust be added o n to th eir p ractice
w hich bridges the gap betw een scien tific ex perts and a c o n ce rn ed public, 'th is is p o st-n o rm al
s u en c e. com p risin g a dialogue a m on g all the stakeho ld ers in a p rob lem , regard less o f th eir
W h ile th is p erspective shows how Kuhn's argum ents continu e to inform thinking about the
nature o f the so cia l-scien tific enterprise, it leaves unanswered qu estions about the research
m eth od s, and processes o f validation, that are appropriate for post norm al science Let's
tu rn , now, to a con trastin g concep tion o f the nature o f science

Imre Lakatos and scientific research programmes

In his essay, ‘Falsification and the M ethod ology of Scientific Research Program s (1 9 7 0 ),
th e p h ilo so p h e r o f scien ce, Im re Lakatos (1 9 2 2 - 7 4 ) . em phatically rejects Kuhn's m odel
o f scien tific p ro g ress. Kuhn's m odel, he argues, is relativistic and reduces science to
'm o b -p s y c h o lo g y ' (1 9 7 0 : 178). Lakatos' aim , th erefore, is to elaborate a m odel of scien
tific ch a n g e that highlights how scien tific change involves, at every step, a process that
is b oth critic a l an d ration al. He replaces Kuhn's paradigm ' with the notion of a uten
tific research program me’, and describ es how the in crem en tal, cum ulative, and progres
sive a rtic u la tio n o f scien tific research p rogram m es lead to the grow th of scientific
k n ow led ge. H e rejects the notion that there is anything like norm al scien ce’ where
e v e ry o n e ag rees on the basic assum ptions defining a subiect of investigation and the
w ay in w h ich it is to be investigated. He argues that there are always alternative, co m p e t­
ing th e o rie s in existen ce. W h at Kuhn calls ‘n orm al science', he argues, is nothing but a
rese a rch p ro g ra m m e that has achieved m on op oly; but research program m es rarely
ach ie v e c o m p le te m on o p o ly and, w hen they do, they do so only for relatively short p e r­
iod s (1 9 7 0 : 1 5 5 ).
A ‘scientific research p rogram m e’ consists of two main elements: the 'hard core and the
‘protective belt’. The relation o f these two elem ents is defined by two methodological rules:
the ‘positive heu ristic’ and the ‘negative heuristic’. Box 3.2 provides a definition of these
term s, as well as a n um ber o f other term s relating to them.
The hard core o f a research p ro g ram m e consists o f very general hypotheses. It is a ‘hard’
c o re b ecau se o f itsprotective belt o f auxiliary assum ptions. The negative heuristic o f a
rese a rch p ro g ra m m e is a convention or m eth od ological rule that stipulates that scientists
sh ou ld n o t q u estion the h ard co re o f the p rogram m e, that they should not m odify or
positive heur­
a b an d on it, an d should avoid paths o f research that bear d irectly on it. The
istic tells scien tists to pursue research related to the hypotheses that m ake up the p rotec­
tive belt; an d p rovid es general guidelines for changing o r augm en ting them in the face of
an an om aly. The p rotectiv e belt can be m odified o r changed w ithout abandoning the
70 P H IL O S O P H Y OF S O C IA L S C IE N C E

Imre Lakatos and the Methodology of Scientific Research

b o x 3.2
Programmes: Basic Terms

Scientific research programme r v.>- f j r-ieory, which remains constant (the hard
.;<■ i ; /.¡tn its protective belt of auxiliary hypotheses.

Hard core B
- - ‘ " jj i (,^c-st or

Protective belt : <;r ’ ' (j s t i' * t / i ot pse t ased on the

Auxiliary hypotheses ¿.-- ra w : ::v

Heuristic

Negative heuristic .

Positive heuristic ,v:;jst f^eores and aux ¡.ary rypoth-

Anomaly r. . sp ,. ,f, ,et of

Progressive research
programme

Degenerating research
programme

p ro g ra m m e itself, so w h e n w e get a fa ls ify in g o b s e rv a tio n , w e s h o u ld a ttrib u te it to p r o b ­

lem s w ith a u x ilia r y h yp o th eses that c o n s titu te the p r o g r a m m e ’s p ro te c tiv e belt. W e are
tree, in lact, e n c o u rag ed , to q u e stio n , m o d ify , an d ch an g e these h yp o th eses in o rd e r to
protect the h ard core.
A research p ro g ram m e changes, then, by m o d ify in g a u x ilia ry assu m p tio n s an d h y p o th ­
eses in the program m e's prote ctive bell. Ih ese are tested, ad justed and re adjusted, an d even
c om pletely replaced. As f igure 3.3 shows, this process m ay pro d u ce a prob lem -sh ift. Lakatos
was deep ly h ostile to Kuhn's arg um ent that pa rad ig m shifts d e pe nd u ltim a te ly on the c o n ­
sensus o! lhe scientific c o m m u n ity . H e saw this as tan tam o u n t to a claim that scien tific
change is non ration al, essentially a process o f a r r iv in g at 'tru th by co n sen sus’ (L a k a to s
19/0: 92). ( .onsequently. Ins co n cern was to show that there are o bjective reasons for w h v
scientists lavo u r one theory, or research pro g ram m e , over an oth er; that the process bv w h ich
we choose- Irom am on g c o m p e tin g research prog ram m e s is a ration al process (an d not a
political battle, as is suggested in K u h n s a c c o u n t ).
 O B JE C T IV IT Y A N D V A LU ES

NEGATIVE
HEURISTIC PrograMfe«
probtent-sNft

\
Positive

Hturh

\ /

Figure 3.B Imre Lakatos' methodology of scientific research programmes

A progressive problem shift leads to the discovery of novel facts. If a research program m e
increases in content through progressive problem-shifts, it is progressive; if it fails to do this,
it is degenerating. H ere is how Lakatos describes these processes:

Let us take a series o f theories, I I , 12. 1 3 ... where each subsequent theory results Irom
ad ding au x iliary clauses to . .. the previous theory in order to accom m odate some anom aly,
each th eory h avin g at least as m uch content as the unrefuted content of its predecessor Let
us say that such a series o f theories is theoretic ally progressive (or lonstitutes a iheoretn ally
progressive problem shift') if each new theory has some excess em pirical content over its predc
c e ss o r.. . Let us say that a theoretically progressive series of theories is also ertipirn ally progrès
sive (or constitutes un em pirically progressive p roblem shift ') if some of this excess empirical
content is also corroborated . . . Fin ally, let us call a problem shift progressive if it is both
th eoretically and em p irically progressive, and degenerating if it is not (l^ katos IV70: 118;
emphasis in orig in al)

In g en eral, we can assess the relative m erits o f com p etin g research program m es by c o m ­
p arin g the d egrees to w hich they are progressive or degen eratin g— the degree to which
th e p ro b lem -sh ifts th rou gh w hich each is developing are progressive or degenerative.
A gain , and as Figu re 3 .4 show s, we do this by d eterm ining w hether the problem shift
leads to the a ccep tan ce o f a theory which has produced a higher ratio of novel facts to
an o m alies.
R esearch p ro gram m e A is preferable to com peting research program m e B if either A is
m o re progressive than B o r is degenerating less than B. Although not everyone will com e to
reject a research p rog ram m e at the sam e tim e, if the changes m ade to the protective belt to

Theory #1

Ratio of Ratio of
novel facts novel facts
to anomalies < toanomalies

Figure 3.4 Imre Lakatos’ methodology of scientific research programmes: comparing theories
PHILO SO PHY OF SOCIAL SCIENCE

save th e hard core from falsify ing o b serv atio n s are co n tin u ally ad h o c and do little to
in crease th e scop e o f th e research p rogram m e, th e research p ro g ram m e as a w hole will
eventually be rejected.
In sum , scien ce is b oth progressive and rational. The m eth od o logy o f research p rogram m es
leads to scien tific th eories that get closer and closer to the truth. T h e d ecision to ab andon one
research program m e in favour o f an o th er is not no n -ratio n al (as Kuhn appears to assert); it
is at every step a critical, considered decision in the light o f available altern ative th eo ries
against w hich th e progressiveness (or degen eration ) o f successive p ro b lem -sh ifts is gauged.
Like Kuhn, Lakatos was w ritin g about the natural scien ces. But his n o tio n o f scien tific
progress through the articu lation o f scien tific research p rogram m es’ was soon p ro g ram ­
m atically extended to the field o f politics. So lets con sid er how a L akatosian p erspective
d escrib es the developm ent o f p ost-W orld W ar II developm ent theory.
In his survey o f the history o f post-W orld W ar II developm ent theory, R ich ard H iggott
argues that the two d om inan t approaches to d ev elopm ent— m o d ern ization th eory and what
he refers to as ‘M arxist’ th eory (‘broadly defined’) — are best u n derstood as Lakatosian
‘research p rogram m es’. The ‘study o f u n derd evelopm en t’ H iggott explains, ‘has been m arked
by a polarization into [these] two broad based scho ols o f th ou gh t’ (1 9 8 3 : 14), But n eith e r o f
them are ‘tig htly-paradigm atic’ in the sense popularized by Kuhn (H iggot 1 9 8 3 :4 3 ). N eith er
achieved ‘an ascen dancy w hich would allow for a process o f no rm al and rev olutionary sc i­
ence’ (1 9 8 3 : 6). M oreover, there was no ‘paradigm shift’ from on e to the other.
Recall that, in Barbara G eddes’ accou n t, m odernization theory had b een ‘top pled’ and
replaced by d epen dency th eory; that, subsequently, dependency theory had collapsed due to
its own ‘internal con trad iction s and inability to deal with the in conv enient facts’; and that
this collapse was followed by a period o f ‘chaos and con ten tion ’ (G eddes 2 003: 7). But H ig­
gott insists that the crisis w ithin m o dern ization th eory led, not to a paradigm shift, but to
internal restru ctu rin g. C onsequen tly, with the rise o f dep en d ency theory, th ere was no
‘wholesale m igration o f scholars’ from one to the oth er (1 9 8 3 : 9), no ‘transferen ce o f a lle­
giance, or conversion, such as o ccu rs in a Kuhnian paradigm shift (1 9 8 3 : 4 2 ). Nor, in Hig-
g ott’s account, did d epen dency th eory collapse and inaugurate a period o f ‘chaos and
contention. A ccording to H iggott, depen dency th eory arose in the 1960s as a ‘cru de radical
alternative’ to m odern ization theory (1983: 5), but it exhibited ‘highly p lu ralistic ten d encies’
(1 9 8 3 : 4 6 ), and ultim ately served as a springboard to a richer, m ore sophisticated M arxist
analysis (1983: 52).
Consequently, neither perspective ‘was killed o ff (or falsified)’. Rather than one su cceed ­
ing the other, the two traditions co-existed and, not only endured, but prospered. At tim es
they seem ed at risk’ especially in the 1960s and early 1970s when testing ‘appeared to be
degenerative or “content decreasing’”. But both proved to be ‘extrem ely durable’ because ‘the
essencc o f these research program m es’ was ‘basically sou nd’; and the m odifications in tro ­
duced within each perspective were progressive adjustm ents to th eir th eoretical hard cores
(1 9 8 3 :8 ).

The role of values in research: Kuhn and Lakatos compared

It was Lakatos in ten tion to elaborate a contrasting model to the one Kuhn had developed.
However, in their respective' accounts of how science actually operates, the two m odels
 OBJECTIVITY AND VALUES

appear m o re sim ilar than dissim ilar. Both assum e that scientists work within a framework
o f th eoretical assu m ptions; and that scien tists m aintain a fram ew orks central set of prop osi­
tions by conv en tion . To this extent, it seem s right to argue, as Kuhn does, that the Kuhnian
n o tion o f paradigm ' and the Lakatosian conception of research program m e describe the
sam e p h en om en on . Kuhn argues that.

in discussing research conducted within a tradition 1 have repeatedly insisted that it depends,
in part, on the acceptance of elements which arc not themselves subject to attack Irom within the
tradition and which can be changed only by a transition to another tradition, another paradigm
Lakatos, I think, is making the same point when he speaks of the hard core <>t research pro
grams,' the part which must be accepted in order to do research at all and which can be attacked
only after embracing another research program (Kuhn 1970: H7)

But, while both agree that scientists maintain a tradition’s core assumptions, they offer dif­
ferent explanations for why scientists do this. Kuhn argues that the defence of a research
trad itio n s c o re assum ptions is often due to a tendency on the part of scientific com m unities
to en co u rag e paradigm atic conform ity; Lakatos insists that it is the outcom e of rational
m eth od ological decisions. Both invoke the history ol science to support their claims. Based
on the historical evidence they present, it could be argued that Kuhn's evidence suggests
that the com m itm en t of a scientific com m unity to a research tradition does not always have
an entirely objective basis. O n the other hand. Lakatos does not really attempt to show that,
historically, it is actually the case that scientists direct criticism away from the hard core of
a research p ro gram m e because they make a m ethodological decision to do so. It would
seem , then, that while Kuhns argum ent is concerned with what, historically, scientific prac
tice has been, Lakatos is m aking a norm ative argum ent about what scientific progress
should entail.
Kuhn points to what he sees as a second way in which the two models appear to be similar:
the description o f what Lakatos calls the degenerating stage’ in the evolution of a research
pro gram m e. Lakatos ch aracterizes this as a stage in which the program m e ceases to lead to
new d iscoveries, and ad hoc hypotheses accru e to it that do little to increase its scope. Kuhn
w rites that ‘I can n o t m yself tell the difference between what he has to say about this im por­
tant stage and what I have said about the role of crisis in scientific development’ (1970: 139).
But Lakatos insists that it is their differing accounts of what ensues from degeneration or
crisis that distinguishes his m odel from Kuhn’s. For Kuhn, the shift from one scientific ‘par­
adigm ’ to an oth er reflects political and cultural factors; for Lakatos, theory development
throu gh ‘research p rogram m es’ is a rational process. Scientists make a rational determ ina
tion o f w hen to give up on a particular theory. W hen it stops incorporating new facts it can
be considered to be d egenerating and, thus, abandoned for a theory that offers an increase of
em p irical con ten t.
Lakatos was hostile to what he saw as the subjectivist aspects and implications o f Kuhns
m odel and, in particular, the prom inent role that Kuhn ascribes to subjectivity in the choice
am o n g co m p etin g theories and in the shift from one paradigm to another. However. Lakatos
d oes not entirely escape his ow n critique: subjectivity appears to play a prom inent role in his
ow n m odel, as well.
A ccord in g to L akatos, a th eory is to be judged on the basis o f w hether it is able to put
forw ard novel facts*, so m e degree o f excess em pirical content over its predecessor. But
PHILO SO PHY OF SOCIAL SCIENCE

progress ca n n o t always be judged in these term s, b ecause not all em pirical facts are o f
equal significance. Novel facts can be either trivial o r im p ortan t, and w hich th ey are
depends on the values and n orm ative co n cern s o f the individuals m aking the assessm ent.
It is up to the individual scientist to judge w hether a p ro g ram m e that is not generating
progressive problem -shifts is m erely exp erien cin g a tem p o rary lull o r has begun to d egen ­
erate; and this, Lakatos says can , in any event, only be known retrospectively. Thus a scien ­
tist’s ‘th eo ry ch oice is as m uch a subjective ju dgem ent as the d ecision to chan ge p aradigm s
is in K uhns account.
In com p arin g these supposedly contrastin g views o f the nature o f the scientific enterprise,
what becom es apparent is that, irrespective o f w hether research employs a positivist o r inter-
pretivist approach, or addresses em pirical or norm ative questions, the research ers subjec­
tive valuations are likely to enter into the research process. No approach or type o f research
ensures a value-free process o f inquiry; none can free researchers from the need to be explicit
and self-critical concerning their own underlying assum ptions and values.

Conclusions
Where does this consideration of the debate about the fact/value distinction leave us?
W e began our discussion in this chapter by considering how this distinction has been
institutionalized in political research in the division between empirical and normative research and
theory. As John Gerring and Joshua Yesnowitz argue, the separation between normative theory and
empirical analysis generates problems for both sides of the divide. As a result of the division between
empirical and normative research, both lack relevance to important problems (2006:104). Empirical
study of social phenomena 'is meaningless if it has no normative import; and it is misleading if its
normative content is present, but ambiguous', if we don't know how it matters. Normative arguments
that propose or justify one value system over another will lack relevance if they make no attempt to
relate to the facts of actual practice or public life. In sum, good social science 'is both empirically
grounded and relevant to human concerns' (2006.133). Normative theorizing 'must deal in facts' and
empirical work 'must deal in values' (2006:108). While we must be sensitive to the difference between
normative and empirical questions and statements, we must also recognize that they are not
independent of each other, and that there are costs in keeping them separate.
The chapter then explored various ways that 'values' intrude into social-scientific inquiry. It began by
considering whether, how, and with what effect, a researcher's own values intrude into research. It then
focused on how the act of observation itself intrudes on the object of study. Finally, it took up the
question of whether and how research is shaped by shared social practices and is inseparable from them.
To explore this last issue, we considered two different views concerning the role of social factors
and practices in scientific inquiry. Thomas Kuhn claims that scientific investigation often reflects
relations of power, professional ambition, and wider political and cultural factors; that scientists
often accept a theory based, not on observable evidence, but on political and cultural factors. Imre
Lakatos emphatically rejects this view of established scientific practice. He argues that scientific
inquiry is rational and leads to progressive theory development.
These differing views are really the product of different concerns. Kuhn is concerned with the
question of how science advances in practice. Lakatos is chiefly concerned with a normative question:
how should science advance7 Kuhn attempts to produce a factual account of science. Critics are right
to be concerned with its normative implications But rather then shooting the messenger, we would
be better off treating the message as a cautionary tale about how subjectivity and paradigmatic
conformity can intrude into research Lakatos elaboration of a methodology of scientific research
programmes is less a model of how science is actually done, than a proposal for how it should be
 OBJECTIVITY AND V A IU IS

done It contribute* to our thinking about what procedure* would allow for a more consistently
objective assessment of research findings and pursuit of theory development
Ernest Nagel has argued that steps can be taken to identify a value bias when n occurs and to
minimize if not to eliminate completely its perturbing effects (1994 S71) To do th.s we need to
proceed with an awareness of the various ways that values intrude into the resea.ch p.cxess With ,h,v
awareness, we can then be in a position to pursue the ideal that Lakatos elaborates a process of
inquiry that, at every step, is both critical and rational

Questions
• Is political research value free'

• To what extent, and in what wayy do values pretent problems for political analysis'

• According to Kuhn, what is scientific progress' Under what circumstances does scientific progress
occur7 How is Kuhn s view of scientific progress different from the conventional wisdom about how
scientific knowledge develops7

• What is a paradigm? Is the concept of a paradigm relevant for the social sciences' Why or why not'

• How is Kuhn's conception of a scientific revolution similar to a political revolution'

• What is Lakatos' notion of a research programme' What aspects of Kuhn s model does it correspond
to and differ from '

• Is the notion of normal science applicable to the study of politics'

Guide to Further Reading

Ball, T. (1976), 'From Paradigm« to Research Programs: Toward a Post Kuhnian Political Science ,
Am erican Journ al o f Political Science 20(1) (February) 151-77

Critical Review; Special Issue: Rational Choice Theory and Politics 9(1 -2) (1995).
This issue contains a wide-ranging symposium on the criticisms launched against rational choice iheory
by Donald P Green and Ian Shapiro, in Pathologm o f Rational Choice Theory A Critique of Application
in Political Science (New Haven Yale University Press. 1994) At least si* of the contributions both pro
and con, based themselves in part on their interpretations of Lakatos' theory of research programmes

Fleck, L (1979), Genesis and Development o f a Fact (Chicago: University of Chicago Press)
A fascinating study of the genesis of a theory in medical science that attempts to identify the process
by which an idea achieves the status of a fact Fleck shows that, among other things, a fact must be
plausible, and that its plausibility is rooted in a given era A fact must also be suitable for publication.
i.e stylistically relevant within a given culture

Gening, J. and J. Yesnowta (2006), A Normative Turn in Potibcal Sdencer Potty M (l) (January): 10-11
The authors provide arguments concerning how and why a more normatively orientated study of
politics can contribute to theoretical development, empirical inquiry, and disciplinary unity

Lakatos. I. and A. Musgrave (eds) (1970), Criticism and the Growth of Knowledge (Cambridge:
Cambridge University Press).
This volume arose out o f a symposium on Kuhn s work held in London in 1965 The book begins with
Kuhn's statement of his position, followed by seven essays offenng criticism and analysis, and finally
by Kuhn's reply

Martin. M .a n d L C M cIntyre (eds) (1994). Readings in the Philosophy of Social Science

(Cambridge, MA: MIT Press).
This volume brings together a collection of important texts on objectivity and values (Part VII)
PHILO SO PHY OF SOCIAL SCIENCE

Riley, G. (ed.) (1974), Values, Objectivity, and the Social Sciences (Reading, MA: Addison W esley
Longman Publishing Co).
A collection of articles on research objectivity, values, and partisanship by prominent social scientists.

Science, Technology & Human Values 36:3 (M ay 2011). Special issue on 'Post-Normal Science'.

Taylor, C. (1994), Neutrality in Political Science', in Michael M artin and Lee C. M cIntyre (eds),
Readings in the Philosophy o f Social Science (Cambridge, MA: M IT Press), 547-70.
In this influential article, the philosopher Charles Taylor argues that value-neutrality in the social
sciences, and the possibility of separating facts and values, is a myth: it is a myth that researchers
consider the facts of something and, on the basis of the facts, move to a judgement or valuation of them

Vasquez.John A. et al. (1997), Forum on T h e Realist Paradigm and Degenerative vs Progressive

Research Programs', American Political Science Review 91 (December): 899-934.

e References
Ball, T. (1976), 'From Paradigms to Research Geddes, B. (2003), Paradigms and Sand Castles
Programs: Toward a Post-Kuhnian Political Theory Building and Research Design in
Science’, American Journal of Political Science Comparative Politics (Ann Arbor: University of
20(1) (February): 151-77. Michigan Press).
Bernstein, H. R. (1981), 'Marxist Historiography and Gendzier, I. (1985), Managing Social Change: Social
the Methodology of Research Programs', History Scientists and the Third World (Boulder, CO:
and Theory 20(4) (December): 424-49. Westview).
Bird, Alexander, Thomas Kuhn', in Edward N. Gerring, J. (2001), Social Science Methodology: A
Zalta (ed ). The Stanford Encyclopedia of Criterial Framework (Cambridge: Cambridge
Philosophy (Fall 2009 Edition), http://plato.stanford. University Press).
edu/archivcs/fall2009/cntries/thomas-kuhn/. --- and J. Yesnowitz (2006), 'A Normative Turn in
Bodenheimer, S. (1971), The Ideology of Political Science?' Polity 8:(1) (January): 101-133.
Developmentalism: The American-Paradigm- Gutting, G. (1984), 'Paradigms and Hermeneutics:
Surrogate for Latin American Studies (Beverly A Dialogue on Kuhn, Rorty, and the Social
Hills, CA: Sage). Sciences', American Philosophical Quarterly 21(1)
Cochran, M. (1999), Normative Theory in International (lanuary): 1-15.
Relations (Cambridge: Cambridge University Press). Higgott, R. (1983), Political Development Theory: The
Cox. R. (1981), 'Social Forces, States and World Orders: Contemporary Debate (Beckenham, Kent: Croom
Beyond International Relations Theory". Millennium: Helm).
Journal of International Studies 10(2): 126-55. Jastrow,J.(1899). The Mind's Eye', Popular Science
Diamond. S. (1992), Compromised Campus (New Monthly 54: 299-312. From wikipedia, http://
York: Oxford University Press). en.wikipedia.org/wiki/file: Duck-Rabbitjllusion.
Foster-Carter, A (1976), From Rostow toGunder JPg-
Frank: Conflicting Paradigms in the Analysis of Jervis, R (2002), 'Politics. Political Science, and
Underdevelopment', World Development 4(3) Specialization. PS', Political Science and Politics.
(March): 167-80. 35(2) Oune): 187-9.
Frankel, C (1969), High on Foggy Bottom: An Kuhn, T. (1962), The Structure of Scientific Revolutions
Outsider's Inside View of the Government (New (Chicago: University of Chicago Press).
York: Harper & Row).
--- (1970). 'Notes on Lakatos’. Proceedings of the
Funtowicz, S O and J R Ravetz (1992), Three Types Biennial Meeting of the Philosophy of Science
of Risk Assessment and the Emergence of Post Association ( 1970): 137-46
Normal Science . in S Krimsky and D Golding Lakatos, I. (1970). Falsification and the
(eds). Social Theories of Risk (Westport, CT Methodology of Scientific Research Programs',
Praeger), 251-74
in I. Lakatos and A. Musgrave (eds). Criticism
-- (1993). Science for the Post-Normal', Futures and the Growth of Knowledge (Cambridge:
25(7). 739-55 Cambridge University Press). 91-196.
Martin, M (1994). The Philosophical Importance of Schmidt B (2002). On the History and
the Rosenthal Effect', in Michael Martin «nd leeC. Historiography of International Relations m
McIntyre (eds). Readings in the PMotophy of Soaal W Carinae. T Rase, and B Smmorn (eds).
Science (Cambridge. MA MIT Pré»), 585-96 Handbook of Internationa/ Mabont (London
Mason. M. (1997). Development and Dnorder A Sage). 3-22
Hntory o f the Third World unce ) 945 (Hanover Stmpson. C. (ed) (1999). Universities and fmp*e
and London: University Press of New England) (New York New Press)
Morgan. P M (1994), Theories and Approaches to Valenzuela. A. and S Valenzuela (1979).
International Pohvci What Are We to Think? Modernization and Dependence Alternative
4th edition (New Brunswick. NJ Transaction Perspectives in the Study of Latin American
Publishers) Development in J VilUmJ (ed) Tramnational
Nagel. E. (1994). The Value-Onented Bias of Social Capttalitm and National Drvrlopmmt (Susse*
Inquiry, in Michael Martin and lee C. McIntyre Harvester Preu). 31-65
(eds). Readings in the Philosophy of Social Science Weber. M (1940). The Methodology of the Social
(Cambridge, MA. MIT Press). 571-84 Sciences (New York The Free Press)
Pavlich, G. (1988). Re-Evaluating Modernisation and ---(1994). Objectivity in Social Science and Social
Dependency in Lesotho'. TheJourno/ of Modern Policy, m Michael Martin and Lee C McIntyre
African Studies 26(4) (December) 591 -605 (eds). Readings in the Philosophy of Social Science
(Cambridge MA MIT Press). 535-46
Popper. Karl (1959). The Logic of Scientific Oncovery
(London: Hutchinson of London)
 bB Methodological
1^ Individualism and Holism

Chapter Summary
This chapter considers the perennial debate in the philosophy of social science con­
cerning 'individualism' and holism' These represent contrasting views about the na­
ture of the social world and how we can gain knowledge of it. The chapter explores
the various positions-ontological, epistemological, and methodological-which the
terms 'holism' and individualism' are used to represent, and their implications for
how we pursue research and develop explanations of political phenomena. The ques­
tions which we consider include the following:

• What is society ? Is it something distinct from and more than the individuals that
comprise it?

• What is the proper unit of sociological analysis?

• Should explanation in social research give primacy to individuals or to social

collectives?

• Do social explanations need to reduce' to arguments about the actions of

individuals?

• How do structures and agents relate to produce outcomes in politics and

international relations?

Introduction
Ihis chapter addresses the third in a series o f key debates co n ce rn in g how we know about
and study the social world. Previously we considered the debate about w hether know ledge
of social phenom ena can be gained through a process sim ilar to that w hich is used to gain
knowledge about the natural world: w hether we can explain social phenom ena in the way
that scientists explain natural phenom ena, or w hether it is possible only to interpret what
people do and why (C hapter 2). We then focused on debates about how values influence
social-scientific inquiry, w hether it is possible to define a d istinction betw een facts and val­
ues, and w hether pursuit of knowledge o f the social world can be v alue-free and objective
(Chapter 3).
Ibis chapter explores debates co n ce rn in g 'in div idu alism ’ and 'h olism ’ in social inquiry
and their im plications lor how we condu ct political research. Individualism and holism
represent different positions with respect to qu estions o f ontology, epistem ology, and
m ethodology. Ihe debates we consider thus involve three in terrelated issues, l'irst is the
issue ot social ontology. \\ hat are the basic elem ents that m ake up the social world? W hat
 METHODOLOGICAL IN D IVIDUALISM AND HOLISM

sorts of things co n st,tu te social facts ? W hat arc the sorts of thing» we are investigating
w hen we d o research on. for exam ple, nat.ons. political partl<ri> or , lakses, Can tKM
these social w holes or collectives as m ore than their individual constituents, as actors in
their own right? The second issue concerns epistemology. In pursuing knowledge of the
social w orld, what should we treat as the basic unit of sociological analysis? W hat consti­
tutes legitim ate know ledge of social phenom ena? Ihese ontological and epistemological
issues are related both to each other and to a third issue: the nature of social explanation. Is
m e th o d o lo g ica l individ u alism or m eth od ological holism m ore appropriate to explana­
tion in social research? Should social explanation give prim acy to individuals or to social
collectives?
We will begin ou r consideration of these questions in this chapter by disc ussing the ontol
ogical and epistem ological positions that the term s individualism' and holism' are used to
define. Ihe chapter then discusses methodological individualism and holism, debates con
cern in g these positions in social science research, the normative concerns (i.e. the politics of
individualism and collectivism ) that have intruded on these debates, and difficulties in keep
ing norm ative issues separate from analytic ones. Finally, the chapter considers how meth
od ological individualism and holism are reflected in the debate concerning the nature of.
and relationship between, social structures and human agents (‘the structure-agency prob­
le m ) in the field of politics.

Individualism and holism

Before con sid erin g the debate about individualism and holism as methodological positions,
we need to distinguish it from two different but related, areas of debate: the debate about
ontology—about what is, or what is real; and debates about epistemology—about what co n ­
stitutes knowledge o f the social world. Recall how we defined these term s in Chapter 2.
‘on tology’ is the study of what exists and the nature of what exists; epistemology' is con
cern ed with w hat is knowable. I h e ontological question that we focus on in this chapter
co n ce rn s the nature of. and the basic elements that make up. the social world. Ihe epistem ol­
ogical question that we con sid er co n cern s what sort of knowledge of the social world is
possible.
As on tological positions, individualism and holism represent two different answers to a
fundam ental question regarding the nature of society: are societies reducible to the indi­
viduals w ho m ake them up? O r is society som ething m ore than, and distinct from, the sum
o f its individual m em bers? A ccording to ontological individualism, all that exists in the world
are individual p ersons.: W e cannot say, for instance, that the world consists of entities such
as universities as well as teachers, students, and adm inistrators; it contains only individual
persons. The social entity, university’, is nothing m ore or less than the individuals who are
involved in university activities. Though we talk about institutions such as the state, the
ch u rch , o r the university, these entities are really just individual people organized in a p ar­
ticu lar way. W e talk about social groups, but these are simply collections of individuals
co-existin g. The individual elem ents draw on resources around them , but they are nonethe­
less independent entities that have self-contained properties. Thus, in studying social insti­
tutions o r g roups, analysts m ust study individuals.
P HILO SO PHY OF SOCIAL SCIENCE

O ntological holism claim s that social w holes are m o re th an, and d istin ct from , the sum of
th eir individual constituents. The w hole affects and is affected by th e q u alities o f its c o n s titu ­
ents. These constitu ents are not, as individualism m aintains, in d epend en t en tities th at have
self-con tain ed properties: they are in ternally related in th e sense that each is im bu ed with,
and constituted by, th e qu alities o f oth ers. So we can n o t con sid er th e w hole as sim ply
consisting o f independent individuals sequentially sum m ed together, one after the other. M uch
— perhaps all— hu m an in teraction consists o f actio ns, and gen erates o u tcom es, that can n o t
b e com preh ended and explained as a sum o f individual actio ns.
O nto log ical individualism and h o lism entail corresp on d in g epistem ological p osition s:
claim s co n ce rn in g how we know about and what constitutes know ledge o f the social world.
An individualist epistem ology claim s that, sin ce only individuals exist, all that it is possible
for us to know is what individuals do. Social scien ce is, therefore, th e study o f individual
behaviour, o f how individuals act, and o f individual attribu tes, beliefs, percep tio ns, and a tti­
tudes. A holist epistem ology m aintains that, b ecause individuals are part o f a social w hole,
part o f a system o f relations that constitu te th em , an individual actio n is not fully intelligible
until the w hole o f w hich it is a part is taken in to accoun t. As we shall discuss, fu rth er along
in this chapter, one m ight adopt an on tological p osition w ithout necessarily accep ting the
corresp ond in g epistem ological position. For instance, on e m ight con ced e that the social
world consists o f social entities, such as in stitutio ns or classes (on tological h o lism ), but still
m aintain that explanations m ust be reducible to statem ents about individuals (ep istem o l­
ogical individualism ).

Methodological individualism and holism

Thus far we have discussed the claim s o f ontological individualism and holism (different
views o f what really exists), and o f epistem ological individualism and holism (different
views about what constitutes knowledge o f the social world). We turn now to a consideration
o f m e th o d o lo g ica l individualism and holism .

Methodological individualism
Debates about m ethodology in the social scien ces have to do with the principles and p ro ce­
dures o f inquiry that can provide us with legitim ate knowledge o f the social world.
M eth o d o lo g ica l in d iv id u alism claim s that, since all that it is possible for us to know are the
actions o f individuals, then explanations o f social p henom ena such as classes, power, or
nations must ultimately be explicable in term s o f facts about individuals. Unless we can
account for an outcom e in term s o f individuals and their desires and beliefs, we do not have
an explanation o f that outcom e.
The case for m ethodological individualism is generally based on two key claim s. 'Ih e first
is what has been called ‘the doctrin e of reducibility’. A ccording to this d octrin e, statem ents
referring to holistic sociological entities can be reduced to statem ents referring only to in d i­
viduals and their actions or dispositions. To illustrate how this reduction can be made, c o n ­
sider this statement: the party voted unanim ously to accept the proposed platform ’. It is
possible to replace this sentence with one that does not use the holistic term ‘party’. We can
 IN D IVIDUALISM ANO HOLISM m
assum e that the party consists of n mem bers; and we can then state that nl decided to
e ndorse the proposed platform , n2 decided to endorse the proposed platform, n3 decided to
e ndorse the proposed platform . . . ' . etc. Ihe individualist would argue, therefore, that by
saying that the p arty unanimously took some action, we are simply saying that each indi
vidual m em b er o f which it is com posed took that action.
A second claim used to establish the case for methodological individualism is that any
exp lanation o f a given social phenom enon is only final and satisfactory once it is provided in
language w hich refers solely to individuals and their actions. Much of the discussion ol this
claim has focused on what J. W. N. Watkins called rock-bottom ' explanations and half-way
ones: those that do, and those that do not. specify what Talcott Parsons called the action
fram e o f reference (Parsons 1937). Watkins m aintains that we will only have a 'rock-bottom'
explanation for a given social phenom enon once we have an explanation solely in terms ot
individuals and their dispositions (1957: 106).
W atkins identifies two areas, however, in which methodological individualism does not
work, Ih e first involves probability situations ‘where accidental and unpredictable regulari­
ties in hum an behaviour have a fairly regular and predictable result’. I hese statistical regu
larities in social life— such as the generally stable rate of automobile accidents annually— are
inexplicable in individualistic terms' (W atkins 1994: 443). I h e existence of these statistical
regularities, W atkins m akes emphatically clear, does not 'support the historicist idea that
defenceless individuals like you and me are at the chance m ercy of the inhuman and un con ­
trollable tendencies o f o u r society' (W atkins 1994. 44 3 ). We can control these regularities
‘insofar as we can alter the conditions on which they depend. l;or example, we could obvi­
ously abolish road accidents if we were prepared to prohibit m otor traffic' (Watkins 1994:
4 4 4 ). A second area in which methodological individualism does not work ‘is where some
kind o f physical con n ection between people’s nervous systems short-circuits their intelligent
con trol and causes autom atic . . . bodily responses'. For instance, individuality may 'get sub
m erged beneath a collective physical rapport' at revivalist meetings or am ong panicking
crow ds (W atkins 1994: 44 4 ).
So, with these exceptions, Watkins maintains that we shall not have arrived at rock-bottom
explanations o f . . . large-scale |social] phenom ena until we have deduced an account of
them from statem ents about the dispositions, beliefs, resources, and interrelations of indi­
viduals’ (W atkins 1957: 1 0 5 -6 ).
However, there have been challenges to both claims used to establish the case for m eth­
od ological individualism. C ritics argue that the 'doctrine of reducibility — that social terms
‘reduce’ to individual on es— runs into the problem of'm ultiple realizability (Little 199J:
1 9 0 - 9 5 ; Kincaid 1996: 1 4 5 -5 5 ; Sawyer 2 002; Zahle 2 003). In the example of the political
party, it appears that there is an equivalence between social and individual terms, and that
the term ‘p arty’ can be reduced to individual terms. But critics argue that there are cases
w here social term s refer to events o r entities that can be realized by a multitude o f different
configurations o f individuals'. Social term s such as revolution’, prim ary group, power elite’,
'p eer group, and ‘bu reau cracy’ can be realized by any num ber of different individual co n ­
figurations. Even specific institutions (the UK bureaucracy, the US power elite) can experi­
en ce significant changes in the configuration o f individuals realizing them (Kincaid 1994:
5 0 0 ). The problem o f multiple realizability’ provides the basis for the methodologist holist
argu m en t that w here a single m acro-level generalization is instantiated by several m icro-level
PHILO SO PHY OF SOCIAL SCIENCE

m echanism s, th e m acro-lev el relationship will b e invariant u n d er a b road er range o f cir c u m ­

stances than any particu lar m icro -m ech an ism and will con sequ en tly offer a b etter ex p lan a­
tion. A rgum en ts that seek to establish the case for a holist m e th od o logical p osition also
challenge the secon d claim used to establish th e case for m e th od o logical indiv id u alism —
that an explanation o f social p henom ena is only fin a l and satisfactory if it refers solely to
individuals and th eir actio ns. W e will explore this challenge to m eth od o log ical in d iv id u al­
ism fu rth er w hen we tu rn , in the next sectio n , to a con sid eration o f m e th od o logical holism .
Finally, in what G eoffrey H odgson calls a ‘d evastating’ critiq u e o f m eth od o log ical in d i­
vidualism (2 0 0 7 : 2 2 0 ), critics have argued that m eth od o logical individ ualism , in o n e o f its
two versions, has never b een achieved in practice, w hereas its secon d version is tan tam ou nt
to ab and on ing the p osition altogether.
A ccord ing to this argum ent, th ere are two different v ersions o f m e th od o logical in div idu­
alism . The strong (or narrow ) m ethod o logical individualist p osition holds that explanations
o f social phenom ena must be reducible to statem ents about individuals only. A w eaker (or
broad er) version o f th is p osition m aintains that explanations m ay also in clude interactions
am ong individuals, or social relations (U dehn 2 001: 3 4 6 - 9 ). B o th p osition s recognize that
an und erstanding o f individuals m ust involve consid eration o f cultural and in stitutio nal fac­
tors. The strong version holds that social phenom ena m ust, n o neth eless, be explicable in
term s o f facts about in dividuals. H odgson argues that this type o f explanation is ‘un attain a­
ble in practice’ (2007: 220). It has a problem o f in finite regress: the em ergen ce o f the in stitu ­
tions and cultures that in fluen ce individuals must them selves be explained; and these would
at least in part be explained by oth er individuals. As long as we are addressing social p h e­
nom ena, we never reach an end point where there are isolated individuals and no thin g
more. We are involved in an apparently infinite regress, sim ilar to the puzzle ‘w hich cam e
first, the chicken or the egg?’ (H od gson 2007: 219).
As for the weak version o f m ethod ological individualism , that explanation m ay also
include in teraction s am ong individuals, H odgson points out that adm itting ‘in teractiv e rela­
tions between individuals’ in explanations ‘opens the d oor to ‘em ergent p roperties’— to the
possibility that novel properties may em erge, i.e. ‘properties that are not possessed by the
entities taken in isolation’ (H odgson 2007: 220). The no tion o f em ergen ce allows for a c o n ­
ceptualization o f society as not com pletely external to in dividuals; as having an ob jective
character, but one that is located w ithin the p ractices o f individuals. C ertain aspects o f so c i­
ety are not reducible to the individual, even though these aspects o f society are still rooted in
individual practice. This blurs the line separating m ethodological individualism and holism
so that ‘the two d octrin es no longer appear as clear-cut opposites’. Instead, we get ‘a m ix, or
synthesis, o f individualistic and holistic elem ents’ (U dehn 2002: 502).

Methodological holism
Methodological holism assum es that social institutions, collectives, and organizations are
prior to, and fundam entally independent of, individuals and can th erefore be taken as ‘p rim ­
itives in social science explanation: they can serve as the prim ary independent variables
determ ining individual and collective behaviour and outcom es.
Ihe case lor this m ethodological position rests on a num ber o f argum ents. I h e first argu­
ment derives from f.m ile D urkheim s argum ent that there are ‘social facts’ that ‘govern’
 METHODOLOGICAL IN D IVIDUALISM AND HOLISM

individuals. These social facts are not merely an aggregation o ffsets about individuals Ihev
are distinct from facts about individual life; and they function independently of an individ­
ual's use o f them . Ih e y consist of ways of acting, thinking, and feeling external to the mdi
viduai: and they are endow ed with a power of coercion that exercises a check or constraint
on individual action (1 9 9 4 ; 4 3 4 ). The power of external coercion which a social fact exer
cises or is capable o f exercising over individuals may be recognized either by the existence
o f som e specific sanction or by the resistance offered against every individual effort that
tends to violate it (1 9 9 4 : 4 3 4 ). Social (acts include social organizations ( legal and moral
regulations, religious faith, financial systems, etc.') and social currents: ‘great movements ot
enthusiasm , indignation, and pity in a crow d [that] do not originate in any one of the par­
ticular individual con scio u sn esses; that com e to each one of us from without and carry us
away in spite o f ourselves' (1 9 9 4 :4 3 4 ). An example of the coercive force ot social tacts can be
seen, for instance, in education, and in its ‘continuous effort to impose on the child ways of
seeing, feeling, and acting, which he could not have arrived at spontaneously' (1994: 435).
A second argum ent used to establish the case for methodological holism concerns the
existence of em ergent properties’. Ihese are properties that emerge when entities interact
and w hich are ‘novel’ in the sense that they are properties not possessed by the entities taken
in isolation (H od gson 2 0 0 7 : 220). To illustrate how interaction am ong individuals produces
p roperties that are em ergent’, consider the following description of two boys carrying a log.

Ihe boys are fitting their actions to each other And to the object and air involved m a give
and take requiring considerable sensitiveness. Ihe two do not apply lon e in sunrstion, or in
opposite directions; they bring a common lorce to bear simultaneously II one moves somewhat
faster or swerves slightly, the other adapts his movement «.orrespondingly I here is an immedi
ate, direct communication between them through the ob|evt The amount ol movement, timing,
pace, and direction are regulated and continuously checked by the corresponding a».lion of ihe
partner. Here is a unity of action that embrace» the participants and the common t*biecl Ih e
performance is a new product, strictly unlike the sum of their separate exertions Neither boy
would act in just the same way in the absence o f the other, what each contribute* is a function of
his relation to the other in the task. (Asch I9S2: 173-4)

In this exam ple, the in teraction of individuals produces ’novel’ properties— properties that
are not possessed by the individuals taken in isolation. Analogously, society in certain of its
aspects is not reducible to the individuals that com prise it. even though these aspects of
society m ay be rooted in individual practice. In m odern social theory, structures are typi­
cally defined as sets o f interactive relations between individuals'; and these relations might
p rod u ce p roperties that are separate and distinguishable from the individuals themselves.
A n o th er argum ent for the holist m ethodological position relates to the claim advanced by
m eth od ological individualists that any explanation o f a given social phenom enon is only
final and satisfactory on ce it is provided in language which refers solely to individuals and
th eir action s. Recall that J. W. N. W atkins recognized that there are two areas in which m eth­
odological individualism does not work; statistical regularities in social life which are inex
plicable in individualistic term s, and crow d o r situational dynam ics that cause automatic or
instinctive bodily responses (W atkins 1994: 4 4 4 ). But holists argue there are other areas of
research in w hich the rock-bottom ’ explanations dem anded o f methodological indivi­
dualism eith er don't w ork o r aren’t necessary- O ne o f these involves explanations of deep
PHILO SO PHY OF SOCIAL SCIENCE

und erly ing causes o f in tentional states that operate at a su b -in ten tio n al level. For in stance, it
m ight be the case th at a b ehaviou r is generated by a com p u lsio n o r bias th at has no t p en e­
trated in to con sciou sn ess and that consequ ently fu n ction s at a su b -in ten tio n al level. In this
case, an explanation in term s o f in tentional states will not be ‘ro ck b o tto m ’: exp lanation will
need to be sought, instead, at a deeper, m o re fun d am en tal, level, perhaps at th e level o f som e
evolved behavioural/physiological fu n ctio n .}
Th ere are oth er types o f so cial-scien tific in qu iry in w hich ‘r o ck -b o tto m ’ explanations are
not necessary. For in stance, th ere are explanations provided by statistical analyses in w hich
know ledge o f in tentional states does not necessarily con tribu te anything essential to th eir
‘ad equ acy’. As an exam ple, con sid er cu rren t research on civil wars. O n e o f th e stron gest fin d ­
ings in the existin g civil war literature is that p oorer cou n tries are m o re likely to ex p erien ce
civil war (e.g. Fearon and Laitin 20 0 3 ; C ollier and H oeffler 2 0 0 4 ). Studies conclu d e, however,
that in com e inequality betw een individuals does not in crease the likelihoo d o f civil co n flict
(e.g. C ollier et al. 2 0 0 3 ) but, rather system atic social and eco n o m ic in equalities that coin cid e
with eth n ic cleavages or regions w ithin a cou n try (e.g. Stew art 2 002; 0 s t b y 2 0 0 8 ). Total
population size is also know n to affect the p robability o f the on set o f civil war. Population
pressures have been shown to play a role in internal con flict as, for in stance, in th e existen ce
o f ‘youth bulges’, large co h o rts o f youths that may serve as recru itin g grounds for rebel m ove­
m ents if society is unsuccessful in integrating th em (U rdal 2 004, 2 0 0 6 ). These findings lack
m icro-fou nd ations: they are not linked to data on the in tentional states o f th ose w ho par­
ticipate in internal conflicts. But such data are not necessary to m aking th ese findings on
these m acro-level variables a useful and im portant discovery. These findings don’t tell us
anything about the individual d ecisions that lead to the initiation o f civil war, but they have
succeeded in identifying particular m acro-level factors, such as territory and population,
and ruling out others, as having utility in explaining the causes o f in ternal conflicts.
C a u sa l-stru ctu ra l studies are a n o th e r type o f so c ia l-s c ie n tific in q u iry that p rov ides
explanations that are not ‘rock-bo ttom ’ but are, noneth eless, ‘satisfying’. These depend on the
relations betw een various elem ents o f social structure without identifying individual-level
processes that give rise to them . Theda Skocp o l’s, S tates a n d S o c ia l R ev olu tion s (1 9 7 9 ) is a
well-known example. Skocpol argues that social revolution is brought about by factors such
as m ilitary defeat that change the relationship o f state organizations to dom estic political
and social groups. A ncien régim es cann ot respond to external events (‘in ternational m ilitary
threats arising in the m odern era’) and as a result states ex perience ‘revolutionary crises’.
W hen this occurs, ‘revolts from below ’ accom p lish ‘changes in class relations that otherw ise
would not have occu rred ’ (1979: 23). Skocpol identifies a sm all class o f relevant cases, sp eci­
fies the social variables to be em pirically evaluated (state structure, land tenure system s,
lorm s of m ilitary organization), and then d eterm ines w hether there are credible causal
sequences am ong these variables in the several cases. Though it may be possible to provide
explanations such as these with m icro-fou nd alion s (Skocpol does not), this sort o f stru c­
tural explanation, Daniel Little ( 1992) argues is, nonetheless, adequate.
O ther argum ents locus on the advantages o f the interpretivist tech niques favoured by a
holist m ethodological position; and, in particular, their ability to draw attention to p o ten ­
tially im portant contextual factors often overlooked or obscured by positivist research
in strum ents. Positivist instrum ents based on principles o f m ethodological individualism
tend to study separate, self-contained, and hom ogeneous variables that can be com bined
 AM U H O L IS M 85
arithm etically to explain outcom es. In Chapter 2 we said that a variable is 4 characteristic
that can assum e different values or characteristics. But it we treat a vanaWe as separate, inde
pendent, and self-contained, we isolate it from any qualitative, or internal, relationship wuh
o th ers that could m odulate its quality (Ratner 2007: 4 ). Variables, then, will v a n only quan
titatively, and rem ain qualitatively the same Consequently, the order in which we measure
them is irrelevant. So, for instance, in questionnaires, each item is a separate (discrete) ele­
m ent that supposedly taps a discrete attribute; and each response is treated as a separate
elem ent that is accorded equal weight, and can be summed, with the others
However, holism views the constituent elements of a whole, not as independent entities
that have self-contained properties, but as internally related It the whole is not siniplv the
sum o f independent individuals, an individualistic torm ol methodology that explains social
phenom ena as a sum ol individual actions can obscure lust what a researcher might want to
learn. This can be illustrated by contrasting the approach described above, with a herme
neutical, holistic analysis. Instead of treating each response as separate and independent,
this type of analysis exam ines patterns of interrelated responses which indicate the quality
and significance o f each.
C onsider the exam ple of a study designed to determine the level of interest in politics
am on g a population. George Bishop and his colleagues found that people were m ore likely to
think they were 'very interested' in politics following the 1980 presidential campaign in the
United States when they were asked the question immediately after, rather than |ust before, a
set o f questions about who they thought would win the election, how close they thought the
race would be, and whether they personally cared which parly won (he election ( Bishop cl al.
1982). In a sim ilar experim ent, they also discovered that people were less likely to think they
followed ‘w hat’s going on in government and public affairs' when asked about it right after,
instead of just before, a difficult group of questions concerning what they knew about the
record of their m em ber of Congress. Ihey concluded that questions such as these may not
m easure what they are intended to measure: an individuals general interest in politics':
instead, they m ay be measuring, am ong other things, 'whatever response has been made most
plausible and accessible in m em ory by the wording of the question and by the context in
which it is asked' (Bishop et al. 1984: 1 6 0 -1 ). By context' they mean no« just the immediate
questionnaire, but also the electoral environment in which a question is asked. For instance, if

people are asked how interested they are in politics in ihe midst ot an exciting presidential cam
paign. we would expect them to say they are more Interested than If we asked the same question
during a dull, local election campaign. Similarly, we would expect people to think they were
more interested in following a political campaign it they are asked the question shortly afteT the
election than if they are asked about it several weeks later. We would also expect people who
have voted in the election to think they were more interested in the campaign than people were
who did not vote. (Bishop et al 1984: 161)

As this exam ple m akes apparent, context is often crucial, both in shaping the response* to
individual item s and to evaluating them , and, in this regard, a holist. interprettvtst approach
m ay offer im p ortan t advantages over an individualist approach.
However, holistic approaches are prone to analytic weaknesses, as well. W here structure
has been placed at the forefront o f sociological explanation, it has tended to imply a causal
d eterm in ism in which the efficacy o f h um an agency is lost. Structures invariably seem to
PHILO SO PHY OF SOCIAL SCIENCE

exist separately from , but n ev erth eless to d eterm in e, m otivated so cial actio n . Th is leads to
what is perhaps th e m o st ch aracteristic prob lem o f h o listic analyses: th eir ten d en cy to treat
m a c ro -so cia l en tities as if they had a co n crete , m aterial ex isten ce; as analytically in d ep en d ­
ent o f th eir con stitu en t elem ents; in ert, unchanging, and u n m ediated by hu m an agency. The
'reification o f social en tities is apparent, for in stance, in m any d iscu ssion s o f glob alization .
Som e analysts trace th e em ergen ce o f glob alization to th e in terests and activ ities o f specific
groups and g overnm ents. But th e p red om inant ten d en cy is to locate th e o rig in s and driving
force o f g lobalization in m a c ro -so cio lo g ical en tities like m arkets and capital. G lo b alizatio n ,
in th is view, is the ou tcom e o f th e ‘logic’ o f m arkets or o f in tern ation al capital, or o f in stitu ­
tional and tech nological in nov ation and change, or o f the ev olu tion ary d ev elopm ent o f
capitalism and capitalist p rod uction. G lob alization appears, from th is perspective, as a
m o re-or-less natural ou tco m e o f a natural sp atio-tem p oral spread o f m o re-o r-less natural
eco n o m ic and tech n ological developm ents, ‘M arkets’, ‘tech n o lo g y ’, cap ital’, and cap italism ’
are treated as p rim ary and autonom ou s agents in th e p rocess o f glob alization . H um an
agents (individuals, or co lle ctio n s o f individuals organized in groups, gov ernm ents, and
states) appear as m ere bystanders; and can act only w ithin the lim its set by m arkets and th e
‘logic’ o f capital. N ote how this con cep tu alization is reflected in th e follow ing statem ents.
The historian, E ric H obsbaw m , asserts that, as a result o f ‘the transn ation al eco n o m y ’, the
territorial nation state ‘can no longer control m o re than a d im in ish in g part o f its affairs’
(1 9 9 4 :4 2 4 ). Susan Strange enlarges on th is view: ‘im personal forces o f world m a r k e ts . . . are
now m ore powerful than states to w hom ultim ate political authority over society is supposed
to belong. W here states w ere on ce the m asters o f m arkets, now it is the m arkets w h ic h . . . are
the masters over the governm ents o f states’ (1 9 9 6 : 4). The form er Secretary G eneral o f the
United Nations, B outros Bou tros-G h ali, m akes the sam e poin t: ‘individual states have less
and less capacity to influence things, w hile the pow er o f global players— in the realm o f
finance, for in stance— grow and grow w ithout b eing controlled by anyone’ (in M artin and
Schum ann 1997: 185). And, finally, this from form er British P rim e M in ister Tony Blair, in a
radio interview: ‘we are going to live in a m arket o f global fin ance and th ere will be investors
that decide to move their m oney in and out o f c o u n tr ie s___ I’m afraid I’m som eon e who says
look, this is a situation you live and work with and try and prepare yourselves for, but cann ot
really change ( Today Programme, B B C R4, 30.9 .9 8 ; quoted in Held 1998: 26).

The politics of individualism and collectivism

Philosophical discussions about individualism and holism go back to the nineteenth c e n ­
tury. But, in the twentieth century, argum ents about individualism and holism as m e th o d o l­
ogical positions in social analysis generated political controversy w hen they appeared to
conflate these positions w ith norm ative argum ents co n ce rn in g the values o f individualism
and collectivism in political life. Ihe conflation of debates about individualism and holism
as m ethodological doctrines, and norm ative argum ents co n ce rn in g the nature ol ‘the good
society (argum ents upholding the virtues o f com petitive individualism , on the one hand,
and of com m unity and social and econ om ic equality on the oth er), has com plicated discu s­
sion on these issues ever since.
Ihe controversy arose when Karl Popper, in a series of essays entitled ‘ Ih e Poverty o f
H istoriusm (1944a, 1944b, 1945) and in lhc Open Society and its Enemies (1 9 4 5 ), appeared
 METHODOLOGICAL IND IVIDUALISM AND HOLISM

to associate holism with the values o f collectivism in political life. This conflation o f analytic
and norm ative issues seemed to be the basis o f his endorsem ent of m ethodological individu­
alism : he appeared to endorse m ethodological individualism as a counter to holism, which,
he believed, prom oted the adoption of collectivist ideologies in political life and. as he
argued, consequently advanced the cause of totalitarianism. The controversy that was gener
ated by this conflation o f m ethodological and norm ative argum ents has confused debates
about m ethodological individualism and holism in the social sciences. Popper seemed to
invoke ‘m ethodological individualism1 as a means of countering the notion that the social
w orld consists o f collective entities such as classes. But one might adopt methodological indi­
vidualism w ithout accepting ontological individualism; that is. one might concede that the
social world consists o f social entities, such as institutions o r classes, but still maintain that
explanations m ust be reducible to statem ents about individuals. The correspondence of
on tological and epistem ological positions is illustrated with reference to individualism in
Table 4 .1.
The rise o f interest in rational choice theory reignited the controversy. Rational choice
attem pts to explain all social phenom ena in term s of the rational calculations made by self-
interested individuals. It maintains that social interactions are based on the individualistic
com p etition o f self-interested, rational individuals; and that rational individuals do not
co o p erate to achieve com m on goals unless coerced. This challenged a com m on premise in

Table 4.1 The individualistic research tradition

Explanation of tocUl

Ontology Social phenomena arc Sot Ml phennme” ,i .ire

made up of individuals ■
nd'yiduals th<M [ ' S " 1'
(heir physn a! and psyc h< phyv, ai and psyc h and ¡.’«.y' »’il Sla!*", at I k
stales actions interaction
social situation and ph>vral situation andphysira'

Scxjal concepts can in Social phenomena can m Scxiai laws can pnnc pie
Epistemology
principle he defined m terms p-nciple l>e c>pinned m be reduced to law, about

of individuals, their physical mdviduals the? physic a'

and psychic states, actions physical and psychic states and psychic states actions

interaction social situation actions interaction social ■nte'action scxia' vtuation

and physical environment situation, and physical and phys'c a' e^ronment

environment

Social phenomena should Social laws should be

Methodology Social concepts should
be explained in terms of reduced to iaws aDout
be defined in terms of
individuals, their physical individuals, then physical
individuals, their physical
and psychic states actions and psychic states, actions,
and psychic states, actions
interaction social situation interaction, social situation,
interaction, social situation,
and physical environment and physical environment
and physical environment

Sourer Iktahn 2002:499.

PHILO SO PHY OF SOCIAL SCIENCE

the trad ition o f pluralist p olitical thought: that groups o f individuals w ho sh are a co m m o n
interest will have an in centive to p rom ote that in terest; that groups a rise on th e b asis o f c o m ­
m on interests, that they are m aintained th rou gh m e m b er sup port o f group policies, and that
group p olicies are an expression o f underlying co m m o n in terests’ (M o e 1980: 2).
Ih e problem o f collectiv e actio n had already b een revealed to social scien tists b efo re the
rise o f rational ch oice theory. M an cu r O lson had show n, in The Logic o f Collective Action
(1 9 6 5 ), that the existen ce o f co m m o n in terests am o ng individuals does not necessarily p ro ­
duce an incentive to pursue con ce rte d p olitical actio n . The reason, he argued, was that,
rather than w orking to prom ote a c o m m o n interest, individuals are ju st as likely to let o th ers
do the work and to ‘free-rid e’: after all, o n ce a co m m o n in terest (o r collectiv e go o d ’) is
achieved, everyone gains from its provision irrespective o f w hether or not th ey w orked for
it. So social scien tists were already aware o f collectiv e actio n problem s. H owever, ration al
choice th eory provided sim ple, pow erful m odels that showed why, as a result o f th is p ro b ­
lem , rationally calculating individuals will act against collectiv e interests.
'Hie rise o f interest in rational ch o ice th eory gave renew ed im petus to on e o f its core p rin ­
ciples: m ethod ological individualism . A dding to this im petus was the em ergen ce o f the
m ovem ent know n as ‘analytical M arxism ’ and, in particular, Jon E lster’s ration al ch o ice v er­
sion o f it. Elster argued that m uch o f M arxian class analysis ov erlooks the p otential for c o l­
lective a ction problem s; that M arxist th eorists ign ore the incentives that individuals face and
that motivate individual actio n, and so fail to consid er the p ossibility that the w orking class
faces a collective action problem w hen it com es to engaging in rev olutionary activity. W hat
M arxist theory needed, he argued, was a m icro-an alysis o f the in centives that m otivate in d i­
vidual workers to contribu te to achieving a collectiv e good. Rational ch o ice th eoretical tools
such as gam e theory could enable analysts to do this. Using m ethod o logical individualism
to interpret M arxist theory, and tools provided by rational ch o ice theory, Elster sought to
show how M arxist theory could be reconstructed so as to provide explanations o f ‘ex p loita­
tion, struggle, revolution and oth er m acro-social phenom ena (1 9 8 2 : 4 5 3 ), with ‘m icro-
toundations: with an account o f the in tentional states that m otivate individual actio n ’.^
Ihe attempt by Elster and oth er ‘analytical M arxists’ to ground M arxist analysis on p rin ­
ciples o f m ethodological individualism generated consid erable controversy. But it also
inspired a constructive debate on w hether and how to provide explanations o f m acro-social
phenom ena with m icro-fou ndations, a debate that succeeded, as previous d ebates had not,
in keeping the issue separate from the strong (or narrow ) m ethod ological individualist p o si­
tion (that we previously discussed) and oth er efforts at reducing m acro phenom ena to
m icro-foundations.

Coleman's bathtub
In an influential discussion o f how to provide explanations o f m acro-social phenom ena w ith
m icro-foundations, James C olem an introduced a useful visualization o f m a c r o -m ic ro rela­
tions. In a diagram which is often referred to as 'C olem an’s Bathtub’ or ‘b oat’ (because o f its
trapezoidal shape), Colem an depicts (1) causal relations going down from m acro p h e n o m ­
ena (e.g. in stitutions) to the con d ition s o f individual actions, w hich (2) then give rise to
individual that (1) in turn aggregate up to m acro outcom es (C olem an 1990: 8).
C olem an developed his account of these linkages through a discussion o f M ax W eb ers
 METHODOLOGICAL IN D IVIDUALISM AND HOLISM

fam ous explanation, in ihe Protestant Ethic and the Spirit of Capitalism (1905). of how Pro!
estant religious d octrin es contributed to the rise of capitalist econom ic organization.
C olem an argued, m effect, that Weber s argum ent lacked m icro foundations. Social
change com es about through changes in individuals and their interactions, so explanations
o f social change that refer to m acro level factors only are incomplete. More specifically,
m acro phenom ena have to be explained by the interaction of micro and macro-levels. C ole­
man's bathtub' figure, together with a depiction of Weber s argum ent, is shown in Figure 4 1.
W eber explained M acro Factor Y (e.g. capitalism ) in term s of M acro Factor X (Protestant
religious d o ctrin e). C olem an argued that this explanation is incomplete But a lull explana
tion, C olem an argued, must explain how Protestant religious doctrine affected the values ot
individuals (a transition from the m acro-to the micro-level) and how the actions and inter­
action s o f individuals in turn contributed to the rise of capitalism (a transition from the
m icro -to the m acro-level). In sum, a full explanation requires an explanation of the m acro
to -m icro and the m icro -to -m acro transitions: how M acro Factor X creates constraints on
a cto rs (arrow 1); how acto rs choose actions under the constraints (arrow 2); and how the
a ction s accu m u late to the m acro level (arrow 3).
In C h ap ter 2 we discussed causal m echanism s and, in particular, the notion that there are
social m echanism s' that produce social outcom es. Drawing on Coleman's schem e. Peter
H edstrom and Richard Swedberg describe a 'social mechanism' as a process of beginning
with a m a cro phenom enon with m icro implications, followed by ( I )a causal process on the
m icro-level that leads to a new m icro-level; followed finally by (2)a m acro social change that
reflects the m icro -so cial situation (H edstrom and Swedberg 199H). Ihis seems a useful way
to think about how, through m a c ro -m ic ro and m ic ro -m a c ro interactions, these m echa­
nism s op erate to prod u ce outcom es.
W h at m eth od s are there for generating and assessing evidence on causal mechanisms?
A n increasingly popular m ethod for exploring causal mechanism s, primarily in case stud­
ies, is process tracing. A lexander G eorge and T im othy McKeown (1 9 8 5 ) define process trac­
ing as a m eth od to identify the causal chain and causal m echanism s that connect hypothesized
causes and o u tcom es. In process tracing the researcher explores the chain of events or the
d ecision -m ak in g p rocess by which initial case conditions are translated into case outcomes.
The ca u se-effect link that con nects independent variable and outcom e is unwrapped and
divided into sm aller steps; then the investigator looks for observable evidence of each step'

Protestant doctrine —--------------------— ---------------------------- ► Capitalism MACKO-UVCL

(Factor X) (Factor V)

2
Economic behaviour M KRO-UVB.
values ^ —
(constraints on acton) (actions by actors)

F fcu ra tl Macro-micro linkages (Coleman* 'Bathtub', and Wtberi ftototartf B h c and the Sfm* of G&uHsnb
PHILO SO PHY OF SOCIAL SCIENCE

(Van Evera 1997: 6 4 ). Using case studies and w ith in -case analysis, th e research er searches
for evid en ce o f th e causal process a th eory hyp othesizes or im plies in a case, th rou gh ex a m ­
in ing data from histories, archival d ocu m en ts, interview transcrip ts, or o th er sou rces. T h ese
m ight help recon stru ct the seq uence or stru ctu re o f events, o r reveal th e m otives o r p ercep ­
tions o f governm ent officials o r d ecision -m ak ers. In th is way, process tracin g can help to
uncover the m icro-fou n d ation s o f individual behaviour.

Fallacies
W hile there is, as o f yet, no definite solu tion to the m eth od o logical problem o f lin kin g in d i­
vidual and collective actio n, th ere is at least a recognition that, in the ab sen ce o f a solu tion ,
analysts must be alert to falling v ictim to one or an o th er o f two co m m o n fallacies or errors
o f reasoning. The first o f th ese is what is called th e ‘eco lo g ica l falla c y ’: th e e rro r o f in ferrin g
individual ch aracteristics from group c h aracteristics based on aggregate data. For in stance,
and as E lster argued, when we ignore the potential for collectiv e actio n p rob lem s in groups,
we move too easily from an iden tification o f a group interest to the ascrip tio n o f an in d i­
vidual interest. W hen in feren ces about the nature o f specific individual behav iou r are based
solely upon popu lation-level or ‘ecolo gical’ (i.e. group) data, th is can lead to very w rong
conclusions.
The classic exam ple o f an ecological in feren ce problem is provided by W. S. R ob in son. In
a widely cited article, R obin son (1 9 5 0 ) noted that, on the basis o f census data, several g eo ­
graphic d istricts in the U nited States in 1930 showed a positive correlation betw een th e lit­
eracy rate and the p roportion o f im m igrants: the g reater the prop ortion o f im m igrants in the
unit, the higher its average literacy rate. However, w hen individuals instead o f aggregates
were considered, the correlation turned negative: im m igrants were on average less literate.
The positive correlation at the aggregate level was b ecause im m igrants tended to settle in
areas where the population was already m ore literate. M oreover, when R ob in son was w rit­
ing, districts with large percentages o f A frican A m erican s (th en located m ainly in the S ou th)
generally elected segregationist candidates; but, as R obin son dem onstrated, th is relationship
was not reproduced at the individual level: blacks did not vote for segregationist candidates.
Ih is did not mean that the aggregate level relationship was ‘spurious’. D istricts with large
num bers o f African A m ericans really did elect segregationists.
Ihe ecological fallacy appears to be an inherent problem in political studies. The ob serv a­
tion that most arm ed conflicts take place in poor coun tries with low education standards
leads, wrongly, to the conclusion that poor and uneducated young m en are m ore likely to
join a rebel group; the fact that w ealthier states in A m erica are m ore liberal should not lead
us to conclude that liberal voters are richer than conservative voters; that a decrease in
incom e is correlated with an increase in crim e at the county level does not m ean that low
personal incom e leads to an increase in crim in al behaviour. A ggregate-level relationships
are not necessarily reproduced at the individual level!
However, the opposite fallacy— the indiv id ualistic’ or ‘reversed e c o lo g ical’ fallacy — is
just as com m on. Ihis is the error o f deducing con clu sion s about groups (e.g. organizations,
societies) using findings from the individual level o f analysis. So, for in stance, in the case
o f R obin sons study o f electoral outcom es in d istricts with large percentages o f A frican
A m ericans, the (act that A frican A m erican s were not segregationist did not m ean that the
 METHODOLOGICAL IND IVIDUALISM AND HOLISM

district-level linkage between racial com position and segregationaist policies was spurious
In co n te m p o ra ry Fran ce, the vote for the xenophobic National Front tends to be highest
in d istricts with high percentages o f Muslim immigrants'. Ihe ecological fallacy .s to assume
that the im m igran ts are supporting the National Front Ihey arc not ( Inglehart and Wel/el
2 0 0 3 : 6 3 ). Ihe individualistic fallacy is to assum e that, because Muslim im m igrants do not
support the N ational Fron t, the vote for the National Front will not be high m districts with
high p ercen tages of Muslim im m igrants. It is. It we focus too much on the characteristics
o f individuals, we m ay wrongly attribute these ch aracteristics to a group or population. It
m ay be the case that at an individual level, high incom e or other m arkers of material sue
cess are associated with a lower rate of suicide. But this does not m ean that populations or
societies w hich are rich have a lower rate of suicide or better mental health. In tact. U may
be that the op posite is true.
Both fallacies, I'odd Landm an, argues, originate from the same source: the ontological
predisposition o f the researcher' (Landm an 2008a: 43). An individualist ontology predis
poses a researcher to collect data on individuals to provide insight into collective behaviour;
while a holist on tology may focus the attention of a researcher on m acro-social phenomena
in o rd er to better understand the circum stances that shape individual behaviour
W hile explanations that combine m acro- and m icro-foundations would likely be better
able to avoid fallacies associated with individualist and holist methodological positions, they
would also provide m ore com prehensive explanations of political outcomes. We will con
sider this issue in the next section.

Agents and structures in the study of politics

There are two broad tendencies in explanations of political outcom es as they relate to agents
and structures: intentionalism (agential explanations) and structuralism (structural explan
ations). Inten tion alism refers to approaches that give prim acy to the intentions and actions of
individual agents in explaining events. Approaches which pursue this type of agential' expla
nation include rational choice and public choice theories. Ihe term itructurmlUm refers to
approaches that privilege the role of structures in explanations of social phenomena. Fhese
approaches assum e that there are structures that individuals cannot see and of which they
m ay not have any awareness, but that nonetheless shape or influence their actions; that c o n ­
strain, and also provide opportunities for, individual action.
Q uestions relating to the relative im portance of structural and agential factors, and to the
role of both in explaining political events, are at the heart of debates about many political
ou tcom es, with explanations for m any political phenom ena tending to stress either agents or
stru ctu ral factors.
For instance, explanations for the variation in human rights abuse across countries have
tended to be either agential or structural. Agential explanations focus on the intentionality
o f individual and state choices, the strategic interaction o f state and non state actors, and the
hum an rights im plications o f the m ultiple outcom es o f these interactions’. Structural explan­
ations focus on the im pact on hum an rights o f m acro patterns and holistic structures: ‘broad
so cio -e co n o m ic change, institutional differentiation and transition, and particular stru c­
tural con strain ts at d om estic and international levels o f analysis (Landm an 2008b: 4).
PHILO SO PHY OF SOCIAL SC IEN CE

Studies o f rev olu tion s have a lso ten d ed to o ffer e ith e r o n e ty p e o f e x p la n a tio n o r th e o th er.
Thed a Sk o cp o l, in States an d S ocial R evolutions (1 9 7 9 ), argu ed fo r a s tru c tu ra l an d n o n -
v olu n tarist’ study o f rev olu tions. Sam u el P o p k in , in The R ation al P easan t (p u b lish ed th e
sam e year, 1 9 7 9 ), argued for a study o f p easan t rev o lu tio n a ry a c tio n b ased u p o n th e ax io m s
o f ration al c h oice. S k o cp o l s analysis fo cu se s ex clu siv ely u p o n s o c ia l s tru c tu re s, an d P op kin
focuses exclusively up o n in div id ual a ctio n . S k o c p o l v iew s in d iv id u al re v o lu tio n a ry a ctio n
to be a fu n ctio n o f so cia l-stru ctu ra l d y n am ics. Sim ilarly, P op k in v iew s s o cia l ch a n g e (r e v o ­
lu tion ) as a fu n ctio n o f in ten tio n a l m a x im iz in g b eh av io u r o n th e p art o f in d iv id u als. T h e o r ­
etical in co n sisten cies em erge in b o th stu d ies b ecau se o f th e em p h a sis o n o n e set o f facto rs
to the ex clu sion o f th e other. S k o cp o l argues th at a stru ctu ra lly g en era ted c o lla p se o f e ffe c ­
tive peasant sa n ctio n in g en ab les p easan ts to gain su fficien t ta c tic a l fre e d o m to la u n ch c o n ­
c erted attacks again st th e landed elite. B u t S k o c p o ls analysis d o esn ’t a c c o u n t for in d iv idual
c h o ices to revolt; it d oesn ’t ackno w led ge im p ed im e n ts to co lle ctiv e a c tio n . O n th e o th e r
hand, Pop kin’s em p hasis on th e ability o f rev o lu tio n ary org a n iz a tio n s, an d th e p o litical
entrepren eurs w ho lead th em , to provide and cre ate in cen tiv e s th at in d u ce p a rticip a tio n ,
leaves out o f the analysis th e role o f th e so cia l-stru ctu ra l en v iro n m e n t w ith in w h ich th ese
activ ities take place. As Jeffrey B erejik ia n ( 1 9 9 2 ) p o in ts ou t, an u n d e rsta n d in g o f how agen ts
and stru ctu res interact would en ab le us to develop m o re c o m p reh en siv e e x p la n a tio n s o f
socia l change.

The agent-structure problem

In recent years, debate over m eth o d o lo g ical in d iv id ualism and h o lism in th e field o f p olitics
has tended to focus on the q u estio n o f how to d efin e th e relatio n sh ip b etw een stru c tu re s and
agents. This q uestion is at the c ore o f w hat is k now n as th e s tru c tu re -a g e n c y p ro b lem . W h ich
are m ost im portant in explaining political o u tcom es: agen tial facto rs o r stru c tu ra l factors?
W hich o f these have explanatory prim acy? C an th ey b e co m b in ed to p ro d u ce b etter ex p la­
nations o f social phenom ena?
The sub jects o f p olitics are hu m an being s w ho ex ercise ag en cy (p erso n al au tonom y)
w ithin a structure (th e m aterial co n d itio n s w hich define th e range o f ac tio n s available to
actors). A central problem in social and political theory, how ever, is to ex p lain th e ou tco m es
that we observe in term s o f b o th the individual agent and th e so cial stru ctu re w ith in w hich
it acts. As Stuart M cA nu lla points out, ‘W e have an in tuitive u n d erstan d in g o f th e im p o r­
tance o f b oth structural and in tentional factors in social reality’ (2 0 0 2 : 2 7 4 ). T h e prob lem is
to find a way in w hich to define (1) how hu m an agents act in p olitics; (2 ) th e so cial stru ctu re
or system w ithin w hich they act; and (3 ) th e relationship b etw een th e two. It is really the
third o f these that represents the heart o f ‘th e stru ctu re -a g e n t prob lem ’. H ow do we do ju s ­
tice to both sides o f this relationship?
In a highly influential article, ‘The Agent Stru ctu re P rob lem in In tern a tio n a l R elations
lh eory (1 9 8 7 ), A lexander W endt criticized th e two so lu tio n s that had b een offered to this
problem by In ternational R elations th eorists. W endt p oin ts out that N eo -R ea list th eory
and W orld System s th eory had answ ered the on tolog ical q u estio n o f w hich ca m e in to ex ist-
Cn^e a®ents or stru ctu res?— by m aking agents p rio r o r prim itiv e, thus ad o p tin g an
in ivi ua ist onto ogy, or m aking s tru ctu res p rior o r p rim itiv e, w hich is a h o listic on tology.
Both v anan ts are reductionist solu tions. By d efin ing stru ctu re in term s o f th e d istrib u tio n
o f cap ab ilities am o ng the pre-existin g um ts. N eo-R ealism reduces the notion o i structure
to a ttrib u tes o f pre-existin g , individual units that are not theorized.
In W orld Systems theory, structure is treated as analytically independent of its constituent
elem ents. It is conceived of as som ething m ore than, independent of. and existing in the
absence of. interacting individual elements. Agents have no impact on it. So the structure is
inert, unchanging: reified. I here is no theory of the structure, of how it emerges, of what pm
duces it. Wendt argued that we need to conceive o f social structures as ontologkally depend
ent upon, but not reducible to, their units; and develop mediating concepts that can link
stru ctu re and agency in concrete situations. For Wendt, the solution lies in some version of
stru ctu ra tio n ism , a position that has provided the basis for a number of influential theoreti
cal perspectives regarding the structure-agenc y relationship. We consider two of these, below

The dialectical relationship between structures and agents

In con trast to analyses which either give agents or structures primary ontological status,
dialectical approaches attem pt to give weight to both structure and agency without falling
into the traps of structuralism and intentionalism.
In his th eory o f stru ctu ration . A nthony G uldens attem pts to develop a dialectical synthe
sis o f the two. His structuration th eory reconceptualizes the dualism of structure and agency
as a duality o f stru ctu re (1 9 8 4 ). Structuration theory states that structures are themselves
dual in that they are b oth the m edium and the outcom e' of social action (1981: 27) Me thus
recasts the tw o ind epend ent sets o f p henom ena, structure and agency, as concepts which are
d epen d ent upon each oth er and recursively related.
A ccord ing to stru ctu ration theory, structure shapes people's practices, but these practices
con stitu te and reproduce social systems. Agents and structures co exist and co d eterm ine
each other. T h is d ialectical synthesis of the two m akes structures and individuals ontologi
cally equivalent and irred u cible (n ot reducible to the oth er). Ihey are mutually constitutive,
but ontolog ically d istin ct, entities, hach is the effect o f the oth er; they are co-d eterm ined.
However, critics argue that the conception of structure and agency as co-drterm incd. or
mutually constitutive, ends up conflating them. Ihis makes it difficult to examine their inde
pendent properties and the relationship or dialectic between them. Ihe notion that structure
and agency are mutually constitutive— that they are, in fact, the same thing, or two sides of the
sam e coin — makes it very difficult to examine the interrelationship or dialectic between the two.
A second dialectic approach. M argaret A rchers m orphogenetic approach' was elaborated
as an alternative to this conflation o f structure and agency. Archer argues that while rejecting
the ‘upward conflation’ o f m ethodological individualism and the downward conflation of
holism , stru ctu ration ism offers yet another conflation o f structure and agency: a central
conflation’. W e must not conflate the two at all, A rcher argues. Unless we maintain the ana
lytic d istinction between structure and agency and treat them as irreducible to one another,
we can n ot exam in e how they relate to one another over time.
A rch er points out that structures and agents are irreducible because they are fundam en­
tally d istinct. The distinction between them rests on the fact that social structures, unlike
agen ts, d on ’t possess self-aw areness. In w hatever way we conceptualize the so cial’- a s
an o b jectiv e an d e m ergen t stratu m o f reality, o r as a negotiated and objectified social
co n s tru c t— the social rem ains fundamentally different from its com ponent m embers in this
PH ILO SO PH Y OF SOCIAL SCIENCE

la ck o f se lf-co n scio u sn ess. W e m u st no t o n ly keep stru c tu re s a n d a gen ts a n aly tically d istin ct;
we m u st also m ak e it o u r task to d iscov er w hat d ifferen ce ‘th e self-aw aren e ss o f its m e m b e rs
m ak e to th e nature o f th e so cia l’ (A rch er 2 0 0 7 : 4 0 ).
A rch er p roposes that we should un d erstand so cie ty as th e interaction over tim e o f o b jec tiv e
stru ctu re, o n th e on e h and, and individual, su b jectiv e agency, o n th e oth er. T h ou gh sh e m ak es
stru ctu re analytically prior to agen ts— b ecau se th e tem p oral p re-ex isten ce o f stru ctu re is a
co n d itio n for individual a ctio n (1 9 9 5 : 15 )— she argues th at th ere is nev er a m o m e n t at w hich
both stru ctu re and agen cy are n o t jo in tly in play. Stru ctu ral co n d itio n in g (w h ich is tem p o rally
prior, and relatively au tonom ou s, yet p ossessing causal pow ers) shapes s o cial in te ra ctio n ; and
social in tera ctio n , in tu rn , gen erates stru ctu ral elab oration . T h is sch e m e o f Structural C on di­
tioning -» Social Interaction -» Structural Elaboration is stretch ed ou t ov er tim e.
The d ifference betw een th e ‘m o rp h o g en etic a p p roach’ prop osed by A rch er, and stru c tu ra -
tional m o d els, such as that elab orated by A n th on y G id d en s, is rep resented sch e m a tica lly in
Figure 4.2 , below. T h ough b o th c o n ce rn e d w ith th e tim e -s p a c e d im en sio n o f th e stru ctu re -
agent relationship, th e m o rp h o g en etic ap p roach d oes n o t co n flate stru c tu re and agency.
Figu re 4.3 illustrates th e o n to lo g ical d ualism and tem p oral in te ra ctio n d escrib ed in A r c h ­
er’s m o rp h og en etic approach. ‘T ’ represents tim e. ‘T l ’, th en , refers to an in itial p o in t in tim e
in w hich stru ctu res c o n d itio n so cial in teractio n . T h e p erio d o f tim e b etw een ‘T 2 ’ and ‘T 3 ’ is
one in w hich processes o f social in teractio n im p act stru ctu res; so th at at ‘T 4 ’ we find th at, as
a result o f p rior social in teractio n , th e initial stru ctu re s have b een elab orated in so m e way.
T h oug h A r c h e r ’s m o rp h o g e n ic ap p ro ach h as b e e n critic iz e d fo r m a k in g th e te m p o ra l
p re -e x is te n c e o f s tru ctu re a c o n d itio n for in d iv id u al a c tio n , m an y th e o r is ts see th is
‘m o rp h o g e n e tic ’ ap p ro ach as r ep resen tin g a n o th e r step fo rw ard o n th e ro ad to reso lv in g
th e stru c tu re -a g e n c y p ro b lem .

Social practice

Structure Action

Figure 4.2 Structurational models

Source Rose 1999
 METHODOLOGICAL IND IVIDUALISM AND HOLISM

Siryttyral ien^ iisn ing

T1
S « l* ! interact«?^
T2 T3

Svvctur*) eiibofiuon
T4

Figure 4.3 The morphogenetic approach

Source Archer 199S 1S7

Conclusions
Researchers in our field are confronted with two truisms about the social world First individual* arr
often agents whose intentional, self conscious, actions both reproduce and transform social reality
Second, society consists largely of interconnecting social relationships (structures) that condition the
interaction between agents, as well as the outcomes of agent action (Berejikian 199; 647) Howrver
the methodological problems stemming from the simultaneity of individual and collective action have
yet to be solved There is growing consensus that there e*isl macro phenomena-systems of norms
social and political structures, institutions and organizations-but that we must be able to provide
micro-foundations' for entities and causes at the macro-level
All social-scientific theorizing adopts, either implicitly or explicitly a position with respect to the
relationship between agents (the actors) and the structures which shape, give meaning to. or m«ke
possible their actions We have considered influential accounts of how we might understand and
conceptualize this relationship, but debates about methodological individualism and holism the
relationship between macro -‘ and micro factors, between society and the individual and
between structures and agents remain unresolved What then is to be done7 Stuart McAnulla
suggests that 'the debate should not focus upon an effort to find Ihe Holy Grail of a solution Rather,
structure-agency issues should be acknowledged as an unavoidable problem It is an issue on which
we cannot avoid adopting a position we are bound to appeal to some understanding of
structure-agency whenever we offer explanation of political events (McAnulla 2002 273. emphawt
in original) W e cannot seek a solution in the way one might look for an answer to a riddle What
we can do is to continue to look for useful conceptualizations of how structure and agency relate
(McAnulla 2002: 274) This seems a commonsensical way in which to approach these issues in
research.

Questions
• In what sense is a society more than the sum of its individual members*
• Can an explanation of a social phenomenon which only makes use of holistic social terms ever be
fully satisfactory?
• Are there social concepts which cannot be defined by reference to individuals only»

• What difference does the self-awareness of its members make to the nature of the social*

^ Guide to Further Reading

Bimbaum. P,J. Laca. and J. Gaffney (ads) (1990). IndM dualitm : Thceha and Method», tram. John
Gaffney (Oxford: Oxford University Prm).
A collection of essay» by eminent social scientists from several countries on the question of
individualism from historical, methodological, hermeneutical, political, and *ociolo*ical points of view
PH ILO SO PH Y OF SO CIAL SCIEN CE

Hedstrom, P. and R. Swedberg (eds) (1998), Social Mechanisms: An Analytical Approach to Social
Theory (Cambridge: Cambridge University Press).
Papers from a symposium held in Stockholm in 1996, by a number of prominent social scientists,
including Thomas Schellingjon Elster, and Timur Kuran.

Hodgson, G. (2007), 'Meanings of Methodological Individualism', Journal o f Economic

Methodology 14(2) (June): 211 -26.
A discussion of ambiguities, contradictions, and vagueness in how the term 'methodological
individualism' is used.

Little, D. (1990), Varieties o f Social Explanation: An Introduction to the Philosophy o f Social Science
(Boulder, CO: Westview Press), chapter 9 ('Methodological Individualism'), 183-201.

Martin, M. and L C. McIntyre (eds) (1994), Readings in the Philosophy o f Social Science (New York:
MIT Press).
This volume brings together a collection of important texts on reductionism, individualism, and
holism (Part VI).

Udehn, L (2001), Methodological Individualism: Background, History, and Meaning (New York:
Routledge).
This book addresses comprehensively and analytically the large literature on methodological
individualism.

Wendt, A. (1987), The Agent-Structure Problem in International Relations Theory', International

Organization 41(3) (summer): 335-70.
The two most influential structural explanations for international outcomes, Neo-Realism and World
Systems theory are the focus of Alexander Wendt's influential, widely-cited decision. He shows that
each of these theories represents a different approach to solving the agent-structure problem.

References
Archer, M S. (1995), Realist Social Theory: The Collier, P. and A. Hoeffler (2004), 'Greed and
Morphogenetic Approach (Cambridge:Cambridge Grievance in Civil War\ Oxford Economic Papers
University Press). 56(4): 563-95.
---(2007). The Trajectory of the Morphogenetic --- L. Elliott, H. Hegre. A. Hoeffler, M. Reynal-
Approach: An Account in the First-Person', Querol, and N. Sambanis (2003), Breaking
Sociología. Problemas e Práticas 54: 35-47. the Conflict Trap: Civil War and Development
Asch. S (1952), Social Psychology (Englewood Cliffs, Policy (Oxford: Oxford University Press and
NJ: Prentice-Hall) Washington, DC: World Bank), available at
http://econ.worldbank.org/prr/CivilWarPRR.
Berejikian, J (1992), Revolutionary Collective Action
and the Agent-Structure Problem', American Durkheim, E. (1994), 'Social Facts’, in Michael
Political Science Review. 86(3) (September): Martin and Lee C. McIntyre (eds). Readings in
647-57. the Philosophy of Social Science (Cambridge, MA:
Bishop. George F , Robert W. Oldendick. and Alfred MIT Press), 433-40.
J Tuchfarber (1982), Political Information Elster, J. (1982), Marxism, Functionalism and Game
Processing Question Order and Context Effect', Theory', Theory & Society (July): 453-82.
Political Behavior 4(2) (June): 177-200. --- (2007), Explaining Social Behavior: More Nuts
-- (1984). Interest in Political Campaigns: The and Bolts for the Social Sciences (Cambridge:
Influence of Question Order and Electoral Cambridge University Press).
Context. Political Behavior 6(2) (June) 159-69 Fearon, J. D. and D. Laitin (2003), Ethnicity,
Coleman. J (1990). Foundations of Social Theory Insurgency, and Civil War', American Political
(Cambridge, MA Harvard University Press) Science Review 97(1): 75-90.
 METHODOLOGICAL IN D IVIDUALISM AND HOLISM

George. A L and T J McKeown(1985). Case McAnulla. S (2002) 'Structure and Agency m D

Studies and Theories of Organizational Decision MarshandG Stoker (edt) Theory and Methodi
Making1.Advance* in Information Processing m m Political Science, tecond «town (Basingstoke
Organizations 2 21 -58 Palgreve). 271-91
Giddens. A (1981). A Contemporary Cnitqur of Moe T (1900). The Organisation of Interests
Historical Maunaltim (Berkeley. CA University (Chicago University of Chicago Preu)
of California Press)
OHon M (1965) Thr Logic of Collective Action
---(1984). The Constitution of Society Out/me of the Public Good* and the Theory of Groups
Theory of Structuration (Berkeley. CA: University (Cambridge. MA Harvard University Preu)
of California Press)
Ostby G (2008) Poiaruation Horuontal
HedstrOm. P and R Swedberg (1998). 'Social Inequalities and Violent Civil Conflict Journal of
Mechanisms: An Introductory Essay in P Peace Research 4X2) 143 62
Hedstrom and R. Swedberg (eds). Social
Pastom. T (1937). The Unxture of Social Action
Mechannmt: An Analytical Approach to Social (New York Free Press)
Theory (Cambridge Cambridge University Prest).
Popper K (1944a) The Poverty of Histoncism I
1-31
ilonormca 11 86-103
Hobsbawm, E (1994), The Age of Extremes (London
-- (1944b) The Poverty of Hntoncrtm II
Michael Joseph)
iconomca 11 119-37
Hodgson, G (2007). Meanings of Methodological
---(1945), The Open Society and itt tnemm. vol I
Individual nm'.Journal o f Economic Methodology
Plato vol 2 Hegel and Man (London Routtedge
14(2) (June): 211-26
& Kegan Paul)
Inglehart, R and Christian Wefcel (2003). Political
---(1945) The Poverty of Hittoncism III
Culture and Democracy: Analyzing Cron Level
Economca 11 69 89
linkages'. Comparative Politici 36( 1) (October)
Popkin. 5 (1979). The Rational Peasant The Political
61-79
iconomy of Rural Society in Vietnam (Berkeley.
Kincaid. H (1994), Reduction. Explanation, and
CA Univervty of California Prett)
Individualism', in Michael Martin and lee C
Rainer C (2007). Conteitualism versui Powtfvivm
McIntyre (edt). Readings in the Philosophy of
in Crott Cultural Psychology in G Theng.
Social Science (Cambridge, MA MIT Prest),
K Leung and J Adair (edt) Ptnpectlvet and
497-514
Piogitsi m Contemporary Cross Cultural
--- (1996), Philosophical foundations of the Social
Psychology (Beijmg China light Industry Prest)
Sciences (Cambridge. Cambridge Univervty
Robinton. W S (1950) (cologtcal Correlations
Press).
and the Behavior of Individuals American
landman. T (2008a). Issues and Methods m Sociological Review 15 351 7
Comparative Politics. 3rd edition (London
Rote.] (I999J. Towards a Structurstional Theory
Routtedge).
of IS. Theory Development and Cate Study
--- (2008b). Empirical Political Science and llkittratiom. in j Pnes H epetal (eds).
Human Rights', liter Human Rights Review 5(1) Proceedings of the 7th European Conference on
(July): 1-7. Information Systems (Copenhagen Copenhagen
Little. D (1991), Varieties of Social Explanation An Business School)
Introduction to the Philosophy ofSooal Saen a Sawyer. R K (2002). "Nonreductive Individualism
(Boulder, CO: Westview Press). Part l-Superverwence and Wild Disjunction .
--- (1992). Jon Ebter'. in W Samuel* (ed). Philosophy o f the Soool Sciences 32 537-59
New Horuont in £oonomk Thought: Appraisals Sidanius.) andR Kurzban(2003).■Evohitionafy
o f Leading Economists (Aldershot: Approaches to Political Psychology", in D Sears.
Edward Elgar). I Huddy. and R iervrs (ids) Otford Handbook of
Martin, H.-P. and H Schumann (1997). The Global Political Psychology (New York Chtford University
Trap (London: Zed) Press)
Masters, R. (2001). Biology and Politics: linking Skocpd. T (1979). Slates and Social ArvofaMns A
Nature and Nurture'. Annual Revtew o f Political Comparative Anatyus of France. Russia, and Chtna
Science 4.345-69. (New York Cambndm Unrvemty Pre*t)
PH ILO SO PH Y OF SO CIAL SC IEN CE

Somit, A. and S. Peterson (eds) (2003) Human Van Evera. Stephen (1997). Guide to Methods for
Nature and Public Policy: An Evolutionary Students of Political Science (Ithaca, NY: Cornell
Approach (New York: Palgrave/Macmillan). University Press).

Stewart, F, (2002), Horizontal Inequalities: A Watkins, J. W. N. (1957), 'Historical Explanation in

Neglected Dimension of Development', Queen the Social Sciences', British Journal for
Elizabeth House Working Paper Series 81, the Philosophy of Science 8(2) (August):
University of Oxford, available at http://www. 104-17
qeh.ox.ac.uk/pdf/qehwp/qehwps81.pdf. --- (1994), 'Historical Explanation in the Social
Strange, S. (1996), The Retreat of the State: The Sciences', in Michael Martin and Lee C. McIntyre
Diffusion of Power in the World Economy (New (eds), Readings in the Philosophy of Social Science
York: Cambridge University Press). (Cambridge, MA: MIT Press), 441-50.

Urdal. H. (2004), The Devil in the Demographics: Weber. Max (1905), The Protestant Ethic and the
The Effect of Youth Bulges on Domestic Armed Spirit of Capitalism, trans. Talcott Parsons
Conflict. 1950-2000', Social Development Paper (London and Boston, MA: Unwin Hyman,
(Washington, DC: The World Bank). 1930).

--- (2006), A Clash of Generations? Youth Bulges Wendt, A. (1987), The Agent Structure Problem in
and Political Violence', International Studies International Relations Theory', International
Quarterly 50:607-29. Organization 41:3 (summer): 335-70.

Udehn, L. (2001). Methodological Individualism: Zahle.J. (2003), The Individualism-Holism Debate

Background, History, and Meaning (New York: on Intertheoretic Reduction and the Argument
Routledge). from Multiple Realization', Philosophy of the
Social Sciences 33:77-99.
--- (2002), The Changing Face of Methodological
Individualism', Annual Review of Sociology 28:
479-507.

Endnotes
1. See the discussion in Chapter 1 concerning units and levels of analysis. In the study of politics, a 'unit of
analysis' is the entity that a researcher analyses, the 'what' or 'whom’ that is studied, in order to explain a
political event or process. This is not the same as the specific 'unit of observation' on which data are
collected. Units of analysis commonly employed in political studies include individuals, social groups,
legislatures, texts, bureaucracies, and states. To distinguish units of analysis from 'levels of analysis’, recall
that this latter term refers to a conceptual scheme which divides up the world into different aggregations
of social phenomena or 'levels' of social organization. An entity that serves as the unit of analysis for a
study might also be defined in terms of one or another level of analysis. For instance, international
organizations become a unit of analysis when they are selected as the focus of a specific study, and they
are also associated with the international level of analysis. What makes an entity a unit for analysis is its
selection by a researcher for analysis.
2. Epistemological individualism claims, not that only individuals exist, but that knowledge of social
phenomena can only consist of knowledge of separate individual elements.
3. There are several perspectives on how human biological evolution influences political behaviour. See. for
instance. Masters 2001, Sidanius and Kurzban 2003, and Somit and Peterson 2003.
4. Elster has, to some extent, changed his mind about the utility of rational choice theory: 'I now believe that
rational-choice theory has less explanatory power than I used to think. Do real people act on the
calculations that make up many pages of mathematical appendixes in leading journals7 I do not think so'
(2007: 5).
Part 2

How to Do
Research: An
Overview
 Asking Questions: How
to Find and Formulate
Research Questions

Chapter Summary
In this chapter we move frori, .ssues m tt>e philosophy of so< ial v ie<Ke to ac uns.de« altoi
of the research p n x ess This chapter foe uses on ifr;) or* of this pr i* rss ttie tormulat* «
of a well crafted resean h question A researc h question not only initiates It** lese aul
prcxess it is cru( lal to every other step along ttw way The ( tuptei d iv usses why you
research should begin with a resea« h question N r* a lese au h queslion sin «lu re s tt»
research prcxess the difference tietw ren j top« of gefieral question on ttie one tiarxl
and a focused research question on the other wtiere to tmd and how to lorm u ljti
research questions, the various types of questions utvolars asl and Uw role oi ttw litera
turt* review as a source and lationale tor research questions In d iv usung tliese issues
the chapter asks and answers a numt>er oí questions hk ludmg the following

• W hat is a research q u e stio n '

• W hy begin political inquiry with research que stion s'

• W hat kinds of questions stimulate good re se arch'

• H o w can you d evelo p a research question from the e»isting literature relating lo
your area of interest;

• W h.it is the role of the literature review m a research project?

• W hat form does a good research question take7

Introduction
Q uestions arc the engines of intellect, the cerebral machines which «.«invert energy to motion,
and curiosity to controlled inquiry.
(I ischer IV70 1)

M ost p olitical research origin ates from som e general question or problem that arises, either
from th e events, issues, or processes we observe in the w orld around us. or from the theories
and fram ew orks that our field has developed in order to understand them . But the research
process on ly begins with the conversion o f this general question or problem into a w ell-for­
m u lated . c learly focused, research question. As we shall explain, there is an im portant differ
en ce betw een a gen eral qu estion, topic, or idea that you may wish to investigate, and a
research qu estion. It is th e form ulation o f a research question that enables you to channel
your in terest, in tellectual energy, and curiosity into the pursuit o f structured, systematic
inquiry. I h e step you take w hich m oves you from identifying a topic to defining a research
H O W T O DO R E S E A R C H : A N O V E R V I E W

Identify an area of research interest

Specify a research question

Figure 5.1 Step 1

qu estio n, illustrated in Figu re 5.1, can b e ch allen gin g, fru stratin g , and tim e -c o n su m in g . But
on ce it’s accom p lished , you are on your way!
Th is chapter focuses on form u latin g a research q u estio n . It will d iscu ss th e req u irem en ts
for a research qu estio n, w here research qu estio n s c o m e fro m , su rv eyin g th e literatu re to gain
an overview o f the top ic that you are in terested in, th e role o f a ‘literatu re review ’ in prov id ing
b oth the in spiration and ration ale for research q u estio n s, th e differen t typ es o f qu estio n s
scholars ask, and how to form ulate a research qu estio n so th at it addresses an issu e o f s ig ­
n ificance in b o th th e real w orld and th e scho larly c o m m u n ity d evoted to th e study o f p olitics.

T h e re are th re e m is c o n c e p tio n s ab o u t re se a rch q u e stio n s we sh o u ld c le a r up at th e

ou tset.
The first has to do with the role and fu n ctio n o f a research qu estio n . W h e n we say that
good research begins with a w ell-fo rm ulated research q u estio n , th is m ight be in terp reted as
m eaning that a research qu estion is im p ortan t as a m eans o f initiating th e research p rocess.
But a research qu estion plays a co n tin u o u s role th rou gh ou t th e research process. It e sta b ­
lishes the basis for every d ecision a research er m akes in every p hase o f a p ro ject. So, getting
the research qu estion fully in focus not on ly gets you o ff to a g ood start: it in fo rm s every
o th er part o f the research process.
A second m iscon cep tion co n ce rn s the task o f form u latin g a research q u estio n . This m ay
sound like a relatively straightforw ard task— o n e that is easily and q u ickly a ccom p lish ed ;
and m aybe you are w ondering why we need to devote an en tire ch ap ter to d iscu ssin g how to
go about it. However, form ulatin g a good research qu estio n is not as easy as it m ight first
appear. It often dem ands consid erab le tim e and th ough t and, for m any research ers, m ay be
the m ost difficult part o f a research p roject. But how ever m u ch tim e it m ight take to a c c o m ­
plish, on ce you have form ulated a good research q u estio n , the rest o f th e research process
b ecom es far easier and clearer, and moves alon g m u ch faster.
A th ird m isc o n c ep tio n ab ou t research q u estio n s has to do w ith th e ir rele v a n ce to d if­
ferent types o f research . S o m e p eop le m ay th in k th at th e fo rm u la tio n o f a sp ecific
research qu estio n is relevant on ly to q u a n tita tiv e resea rch ; th at q u a lita tiv e research is
m ore o p en -en d ed and d o esn ’t requ ire th e fra m in g o f a research q u e stio n . B ut, ir re s p e c ­
tive o f w hether you are pu rsu in g q u alitativ e or q u a n tita tiv e resea rc h , th ere are g ood
reasons why your research sh ould be b ased on and sh aped by a research q u e stio n . Below ,
we list six reasons why the fo rm u latio n o f a research q u estio n sh ou ld b e step o n e o f the
research process.

1. It fo rce s y o u to get c le a r a b o u t w h a t y o u w a n t to k n o w .

2. By re q u irin g th at it a d d re ss e s a n issu e o f s ig n ific a n c e in th e re a l w o rld , it e n s u re s th at

y o u r a n sw e r, it s y s te m a tica lly d e v e lo p e d , will m a k e it p o s sib le lo r y o u r r e s e a r c h to
c o n trib u te to k n o w le d g e a b o u t im p o r ta n t issu es.
 A S K IN G Q U ES T IO N S

3. By requ irin g that it addresses an issue o f significance to the held of politics. it ensures
that your answ er in som e way c ontribu tes to. rather than merely reproduces, existing
know ledge. 6

4. It org anizes, and is the basis for, everything else that follows.

5. It focuses and narrow s your search for in form ation

6. It enables you to decide what to include and exclude from your research

C o n sid er th e steps involved in condu ctin g research outlined in Box 5 I what u shows is that
every com p on en t o f the process flows from and is directly con n cctcd to a research question
W e will be d iscu ssing each o f these com ponen ts over the course o f this and the following two
chapters.
I h ere arc two th ings that you should note about the outline of research com ponents p re­
sented in B ox 5.1. as well as the discussion throughout this chapter.
hirst, th is outlin e is m eant to provide a general template for form ulating a research ques
tion and stru ctu ring research. Ih e design o f your research will ultimately be shaped by your
topic and the aim s o f your project, and by the guidelines that your thesis/dissertation super
visor or cou rse tu tor requires that you follow The Answer' part of this outline suggests a
focus on hypothesis gen eration and theory testing, an approach not equally relevant to all
types o f research. It m ight be argued, tw). that this is an approach not accepted in the differ
ent form s o f know ledge (positivism /interpretivism ) discussed in Chapter 2 But as Chapter 2
poin ted out, though positivist and interpretivist researchers tend to be guided by different
a ssu m ptions, they also tend to share a com m on set o f research practices founded in the
h yp othetico-d ed u ctiv e m ethod. Som e researchers assess whether the inform ation they have
g athered fits with the interpretation they have posited', and others consider the fit of co m p et­
ing in terp retation s with the facts they have gathered', but 'in either case they are practicing
th e h y p ethetico-d ed uctiv e m ethod' (Pollins 2007: 100). Consequently, they generally follow
the sam e m ethod o logical conventions. But, again. Box S. 1 serves as a starting point and set
o f con sid era tion s to be revised or elaborated consistent with the question and aim s that a n i­
m ate your own p roject.
S econ d , what B ox 5.1 shows is a research process that is structured in an idealized linear
fashion. Th e real p rocess is different. For instance, though researchers may start with 'some
o p eratio n al hu nches about what they expect to find’, they generally do not “ fram e hyp oth­
eses” in any form al sense before they start to work' (Shively 1989: 25). G ood researchers
gen erally are engaged in learnin g and creative re-thinkin g all along the way. As C hapter 6
will d iscu ss, our aim in p resenting th e ou tlin e shown in Box 5.1 is to offer som e clear and
obv ious p roced ures to get you started. As you gain experience, you will develop your own
process o f w ork and way o f fulfilling the requirem ents o f good research.
Irrespective o f th e particular way you choo se to structure your research, a carefully form ul­
ated research qu estion will ensure that your research has a clear purpose. If you are not clear
what qu estion o r qu estions your research is designed to address, you will produce research
that is un focused and, ultimately, uninteresting to others. M ore specifically, a research qu es­
tion ensures that your research will be focused on and clearly linked to the realization o f a
specific aim that contribu tes to our knowledge o f som e significant p roblem or issue. A research
qu estion also provides, as a general question or topic does not, a logic o f inquiry to guide
H O W TO D O R E S E A R C H : A N O V E R V I E W

b o x 5 .i Outline of Research Components

1. The question
A. What do you want to know? What is the central question/problem /issue/puzzle?
B. Why do you think it is worth doing/knowing? What is the rationale fo r pursuing research on this
question?

2. The literature
A. Who else has asked and answered your question7 What (range of) answers to this question are
found in the existing literature relating to it?
B. What are the positive elements in the current literature? What in the literature can you highlight,
underline, expand, extend, improve, build upon, continue?
C. What needs to be done? Delineate the crucial aspects of the problem requiring investigation.
What need to be done in order to provide a better answer to the question than currently exists?
3. Your answer
A. Theoretical Framework. What are the theoretical elements and guiding assumptions o f the study ?
1. What factors or variables of the problem must be investigated in order to answer your
central question?
2 What is/are your hypothesis/es (how are these factors linked)?
a. What is the source of your hypothesis/es? What in theory would lead us to expect the
relation(s) you assert?
b How would you demonstrate the relationships stated by the hypothesis/es?
3. What is the spatial/tem poral domain o f the study7 What is the rationale for defining this
domain for the study?
B. Data and sources
1. What are the data relevant to demonstrating the relationships you hypothesize?
2. What sources are there for these data?

d e cis io n s a b o u t th e d e sig n a n d m e th o d s of th e re s e a rc h . M o re o v e r, a p ro p e rly fo rm u la te d

re se a rch q u e stio n e n su re s th at yo u a re b u ild in g o n e x is tin g k n o w le d g e. A re s e a rc h q u e stio n
m a k e s th e re s e a rch task s p e cific; so th a t, ra th e r th a n Irv in g to g a th e r all th e i n fo rm a tio n yo u
c a n find on a to p ic, you d ire ct y o u r a tte n tio n lo ju st th e in fo rm a tio n th at a d d re ss e s th e q u e s ­
tio n an d h elps you to d e v e lo p an a rg u m e n t in a n sw e r to it. W h e n yo u h av e a w e ll-fo rm u la te d
re se a rch q u e stio n , yo u c a n m a k e d e cis io n s a b o u t w h at in fo rm a tio n sh o u ld b e in clu d e d o r
e x c lu d e d , w h at d a ta yo u will n eed to c o lle c t, w h at to o b s e rv e , o r w h at to ask y o u r in te rv ie w e e s .

Research questions: What are they?

Where do they come from?

W hat is a re s e a rch q u e s t i o n ' A re s e a rch q u e stio n is o n e th at ( 1 ) h a s s ig n ific a n c e lo r a to p ic

o r issu e re la tin g to th e su b je ct m a tte r of o u r field; ( 2 ) is r e s e a rc h a b le (it c a n b e a n sw e re d
th ro u g h c o n d u c tin g re s e a rc h ); a n d ( 3 ) h as n ot yet b e e n a n sw e re d d efin itiv ely. To sav th at a
q u e stio n h as n ot yet b een d efin itiv ely a n sw e re d ' m e a n s th at th e q u e stio n h as n ot b een
addressed; or that the answ ers that have been offered have not succeeded in resolving debate
ab out th e q u estio n ; or that th ere is still m ore that can be done to im prove the answers tha
have so far b een offered by, for instance, providing a m ore system atic, thorough answer, o
by c o n sid erin g the qu estion in light o f different cases or perspectives that m ight provid.
fu rth er illu m in atio n. We will expand on this further along m the chapter
All th ree o f th ese req u irem en ts reflect a basic attribu te and requirem ent o f social scienci
research : th at it ad dresses a topic or issue relating to th e subject m atter of a field of study ir
a way th at c o n trib u tes to our know ledge o f that topic. The first requirem ent com pels you tt
relate a specific top ic that in terests you to a broader issue or area o f inquiry within the fielc
o f p o litics (a I B in B ox 5.1). Ih e requirem ent that a qu estion be researchablc m eans that n
m ust be form ulated in a way that m akes it possible for you to provide an answer to it
th rou g h co n d u c tin g research (#3 in B ox 5.1). To be researchablc it will have to be surti
c ien tly focu sed to p erm it an answ er that can in som e way be tested or dem onstrated, and
free o f erro rs o f reason in g that m ight generate false answ ers or conclusion s Research n
ex p ected to c o n trib u te em pirically, conceptually, or th eoretically to our knowledge of
som eth in g . C onsequ en tly, the th ird requirem ent — that a question must be one to which we
don't k now th e a n s w e r — requires that you e x plicitly locale your research w ithin the fram e
w ork o f an e x istin g social scien tific literature (» 2 in Box 5.1) D oing th is ensures that you
will p roceed w ith an aw areness o f the state o f the art' in term s o f existin g knowledge on the
top ic o r issu e that in terests you and. so, m inim ize the chan ce o f duplicating what has
a lread y b een d one.
W e will d iscu ss each o f these requirem ents a bit further on. when we consider how to
form ulate a g ood research question. First, let's consider how to go about finding a research
q u estio n.

Finding research questions

R esearch topics in o u r field origin ate from our observation o f real-world problem s and
ev ents such as co n flicts, elections, or revolutions. But an im portant source for finding
research questions is the literature in th e field o f politics, as well as social science literatures
relevant to th e su b ject m atter o f ou r field. Please note that, while a topic is usually a state
m en t, a research question is a qu estion: it is a sen tence with a question m ark at the end of it.
B efore we d iscu ss how to use th e literature to find a research question, there are two cave
ats w orth noting.
F irst, th oug h th e literature in ou r field is an im portant source for research questions, you
sh ould no t approach th e literature with too accep ting a fram e o f m ind. As Barbara G eddes
p o in ts o u t. in o u r field

(t)he literature on some subjects contain* only a few argument* generally accepted as true; many
controversies in w hich th e hypotheses on both sides lack both clarity and *mmg empirical »up
port; and large amounts of opinion and conjecture, unsupported by systematic evidence but
nevertheless often referred to as theory (2003:29).

Stu d ents are frequen tly advised to look for ho les in the literature’ in order to find research
qu estio n s. But. if we accept G eddes' ch aracterization o f som e o f th e literature in our field, it
appears th at th is m ay not alw ays b e a good strategy. In f a c t G eddes sees th e literature as
H O W TO D O R E S E A R C H : A N O V E R V I E W

im p orta n t to the form u lation o f a research qu estio n , not b ecau se it suggests gaps th at need
to be filled, but b ecau se it stim ulates ‘in d ig n ation , an n o y an ce, and irrita tio n ’ (G ed d es 2 0 0 3 :
2 9 - 3 0 ) . It should be no ted, too, that o u r u n d erstan d in g o f th e p olitical w orld usually
in creases, not th rou gh fin d ing and filling gaps in ex istin g kn o w led ge— as Patrick D u nleavy
poin ts out, there are often good reasons why gaps rem ain unfilled (2 0 0 3 : 2 7 4 ) — but by well-
crafted research that p roduces in crem en tal ex ten sion s and rev ision s o f it.
In any case, it is w orth em p hasizing that you should approach th e literatu re on any s u b ject
analytically. To be analytical’ is to ask qu estions: to su b ject so m eth in g to q u estio n s in ord er
to d iscov er its m ean in g or essential features. So our advice is to approach th e literatu re with
a qu estio n in g fram e o f m ind: on e that co m b in es respect for th e efforts and a ch iev e m en ts o f
the scholars w ho produced it, with a healthy degree o f scep ticism .
O u r secon d caveat about using the literature as a sou rce for research q u estio n s is that you
should avoid getting drawn into q u estio n s that are politically in sign ifican t. H ere we are
referring to th ose qu estio ns that, as they b eco m e the focus o f dispu te am o n g acad em ic
researchers, lose co n n ectio n with the real-w orld prob lem s that th ey origin ally w ere m eant
to address. R esearch is exp ected to co n trib u te to our know ledge o f a real-w orld prob lem or
issue, either through em pirical research w hich will im prove the th eo ries and m e th o d s rele­
vant to u n d erstandin g that problem and its solu tion , or th rou gh d irectly a d dressing c o n c e p ­
tual or th eoretical sh ortcom in g s in ex istin g th eories and m ethod s. S in ce we do no t yet have
fully effective tools for investigating the m any im p ortant prob lem s that research ers w ant to
address, m uch research in our field is co n ce rn e d with m aking m o re effective th e too ls we use
to understand a given problem . But, as G ary King, R ob ert K eohane, and Sidn ey V erba argue,
research should be driven, not by m ethod s, but by qu estio ns and problem s. R esearch w hich
focuses ‘too m uch on m aking a co n trib u tio n to a scholarly literature w ithout som e attention
to topics that have real-w orld im p ortan ce’, runs the risk o f ‘d escen d ing to politically in sig ­
nificant q u estio ns’. But they also em phasize that ‘attention to the cu rren t p olitical agenda
without regard to issues o f the am enability o f a su b ject to system atic study w ithin the fra m e ­
work o f a body o f social scien ce know ledge leads to careless work that adds little to ou r
deeper u n d erstandin g’ (1994: 17). 'Ihe best research, as King and his co -au th o rs observe,
m anages to do both these things: ‘to be directly relevant to solving real-w orld p rob lem s and
to furth erin g the goals o f a specific scien tific literature’ ( 1994: 18). Box 5.2 sum s up this view.

What Do We Mean When We Say That a Research Project Should

b o x 5.2
Be 'Significant'?

f ir s t : a research project should pose a question that is important' in the real world. The topic should
be consequential for political, social, or econom ic life, for understanding som ething that significantly
affects people s lives, or for understanding and predicting events that m ight be harmful or beneficial.

second, a research project should make a specific contribution to an identifiable scholarly literature by
increasing our collective ability to construct verified scientific explanations of some aspect of the world
(King, Keohane, and Verb,l 1994 15)
 A S K IN G Q U ES T IO N S

W ith th ese caveats duly noted, it should be em phas.zed that reading the literature on a
specific top ic o f interest to you is an im portant m eans o f gam ing knowledge both o f existing
research and cu rren t d ebates, and o f suggesting questions about the topic
Any issue relating to the political world is likely to be the subject of debate. Hxperts are
likely to produce a variety o f conflictin g views about how to define it; where, when, and why
it orig in ated ; how it developed or evolved, what its im pact is. what its likely future trajectory
will be, and what d ecision m akers or specialists can or should do with respect to it If you
plan to pursue research on an issue, you will need to sort through the various opinions and
ju d g em en ts about it and com e to your own conclusions. You will then be able to figure out
how to m ake visible som e d im ension o f the issue that brings added strength to one or
an o th er o f the p osition s represented in that debate, produce a synthesis o f their elem ents, or
show how the d ebate itself is based on erron eous assum ptions or logic. But first vou will need
to get a feel for the d eb ates— for the different positions, and the current state o f play'.
A good way to go about doing this is to survey the literature on the topic that in terests you
W e are not referring here to a literature review, which is som ething you write after vou have
form u la ted your research question (we will be discussing this later in this chapter) A survey
o f literature on your topic helps you to focus your interest and your search for a question on
a narrow er, m ore specific, aspect o f the topic.
L et’s say. for exam ple, that you are in terested in problem s o f dem ocracy, and that you want
to learn m ore about it and m ake it the focus o f a research project. Ihis is a topic o f great
im p orta n ce and interest to researchers throughout the social sciences, and the literature
relatin g to it is im m en se. So. how can you convert your general interest in this large subject
into a w ell-form ulated , clearly focused research question; one that will engage your interest,
ch an n el your energies, and enable you to contribu te to current discussions and debates
about som e aspect o f the subject?
O n e way is to survey the literature about d em ocracy in order to find out what questions
o th er research ers are asking about the topic and what key debates and issues their research
addresses. L ook at the list o f qu estions in Box 5.3. These represent only a small sam ple o f the
q u estio n s that research ers ask about dem ocracy. But cach question alerts you to a different
avenue o f research and d irects your attention to a m ore specific literature and set o f research
q u estio ns. You can then select those you think you m ight be interested in pursuing and com
pile a b ibliography o f previous research and w riting on them. Ih is , then, is the aim of the
survey; to com p ile a startin g bibliography on a narrow er aspect o f the general topic that
in terests you. W ith th is d one, you read the titles on your bibliography (and the abstract or
su m m a ries o f the entries, as well); and this will either inspire ideas about how to extend,
refine, o r critiq u e this literature, or direct your attention to oth er literatures and avenues o f
research.
H ere is a sim ple strategy you can use to get this process going.

1. G o to the library and use the o n lin e catalogue to get the title o f one book on your topic.
W h en you go to th e library sh elf where th e b ook is kept you will discover that it is located in
a n eig hb ou rhoo d populated by m any oth er b ooks on the sam e topic. You can then do a
‘n eigh b ou rh oo d search’ (peruse the titles o f the b ooks on the surrounding shelves) to get a
sen se o f th e variety o f facets o f and e ntry poin ts in to your topic. Pick a b ook whose title seem s
to prom ise eith er an in tro d u ction to an in terestin g facet o f your topic or a good general survey
HOW TO DO RESEA R CH: AN O V E R V IEW

b o x 5 .3 Twenty Questions about Democracy

Democracy
1. What are the problems of democratization in newly decolonized countries?
2. What are the implications for democracy of globalization?
3. Do democratic states have more in common with each other than with undemocratic states?
4. Are there different models of democracy in different cultural contexts?
5. What are the different institutional manifestations of democracy?
6. Is democracy always liberal?

7. Can democracy be imposed upon countries from outside or must it draw upon the established
practices of countries even if these practices are not obviously democratic?
8. To what extent and on what conditions might democracy be imposed (hence undemocratically) by a
foreign power?
9. Are interest groups a threat to democracy7
10 Is the concentration of media ownership a threat to democracy?
12. How does electoral system design affect ethnic conflict regulation in ethnically divided societies?
13. How essential are political parties to representative democracy?
14. What is the relationship between nationalism and democracy?
15. What is a developmental state7 Is democracy relevant to its conceptualization?
16. Can ethmc/religious/linguistic politics and democracy be positively related with each other?
17. Is democracy the best political system for promoting human rights in the developing countries?
18. Can democracy promote interstate conflict resolution?
19. Does democracy require an explicit understanding and justification of certain core values?

ot th e to p ic, an d tu rn to its b ib lio g rap h y o r list of re fe re n c e s. Ih e r e fe re n c e s e c tio n o f o n e w ell-

ch o se n s o u rc e will c o n ta in p a ge s filled w ith titles th a t in d ic a te q u e s tio n s a n d issu es p e o p le are
w ritin g a b o u t. Ih e se will stim u la te id e a s o f h o w yo u m ig h t n a rro w y o u r fo cu s .
In a d d itio n , o r a lte rn a tiv e ly , yo u will w a n t to s e a rc h th e JS T O R d a t a b a s e : h t tp ://w w w .
js t o r .o r g /c g i -b i n /js to r /g e n s e a r c h . JS 'IO R r e p r o d u c e s th e full i m a g e o f a r t ic l e s in o v e r 11 7
s c h o la rly j o u rn a ls , h n te r k e y w o r d s in th e fu ll-te x t s e a r c h . M a k e s u r e th a t th e ‘P o litic a l S c i ­
e n c e Jo u rn a ls b o x is c h e c k e d ; a n d , if y o u r to p ic in v o lv e s s p e c ific g e o g r a p h i c a l a re a s (e .g .
th e M id d le h a st, I.atin A m e r i c a ) o r issu es a n d fa c to rs c e n tra l to o t h e r d is c ip lin e s (e .g . K co n -
o m ic s , A n th ro p o lo g y , P s y c h o lo g y ), yo u m a y w a n t to s e le c t o t h e r j o u r n a l c a te g o r i e s , as w ell.
P e ru se th e re s u ltin g list o f a rtic le s fo r t h re e o r fo u r th a t s e e m p a r t ic u la r l y re le v a n t a n d read
th e ir a b s tra c ts , i n tro d u c tio n s , a n d c o n c lu s io n s . You w ill a lso w a n t to s e a rc h th e W eb o f S c i ­
e n c e ( h t tp ://w e b o f k n o w l e d g e .c o m /). Ih is s o u r c e in clu d e s live d a ta b a s e s c o v e r in g v irtu a lly
all fields o f k n o w le d g e . T w o o f th e m o st u sefu l a re th e S o c ia l S cien ces C ita tio n In d ex (SSC I)
a n d th e Arts & H u m an ities C itation In d ex (A & H C I). W h e n yo u fin d a s c h o la r ly a r t ic l e re l­
e van t to y o u r p a rtic u la r re s e a rch in te re s t, yo u c a n a lso lo c a te all s u b s e q u e n t j o u rn a l a rtic le s
 A S K IN G Q U ES T IO N S

^ lu A1? ChCCk G 0 0 g ,c ( http://scholar.googlc.com/). Always be sure to retain

fu ll bibliographic references ( author, date, title, journal, volume, pages ) for the articles o f in
terest you find. J

2. A particularly useful source to get you started is a survey or state of knowledge' type
article. Ih ese often provide a breakdow n o f the literature on a topic according to us various
en try poin ts, d ebates, and perspectives; and a review o f the evolution o f thinking on the
topic up to and including the most recent and influential works. See if you can spot an article
on your topic w hose title con fo rm s to som ething along the lines of the following Ihe Study
o f (ie n d e r and Politics: Ih e State o f the Art', or Parliam entary versus Presidential Systems
A Survey o f the Literature'. Relevant course syllabi or reading lists, which may be accessible
o n lin e, can also provide this sort o f in troduction to the literature on a topic

3. Pursue the line o f investigation that seem s most interesting to you by getting one of the
relevant b ook s or a rticles you find in a bibliography (or on a course reading list), and then
tu rn to the back o f that book or article to get additional titles (and also to get 'search terms'
that you can put into a search engine).
B ibliog raph ic or R eference M anagem ent Software packages allow you to save and store
the b ibliographic details o f the references you find (articles, books, conferen ce papers, w eb­
sites, etc.). W ith th ese packages you have not only stored the bibliographic details o f the
referen ces you th in k you may want to use, you also have a personal database of references
that you can search and sort; and you can con n ect between your package and Word to create
a custom ized bibliography in your preferred referencing style at the end of your docum ent.
If you chan ge your m ind about what referencing style you want or need to use. you can
qu ickly and easily reform at the references. Som e o f the m ore popular packages are Hndnote.
E n d n ote W eb, RefW orks, Reference Manager. Z otero. and Mendeley.

4. O n c e you have a bibliography o f prom ising b ooks and articles on a particular line of
in vestigation, read the literature you have com piled and th ink, critically and creatively, about
its in sights and how to extend th em , or its weaknesses and flaws and how to overcom e and
co rrec t th em .

B ox 5.4 provides a snapshot o f how a survey o f the literature on a topic reveals num erous
d ifferent d im en sio n s or facets o f a specific topic.
In stitu tional libraries offer abundant support to assist you, including on line and paper-
b ased g uidan ce on searchin g databases and finding research literature, web searching, how
to referen ce (in clu d in g citin g electron ic resources) and m anaging references, and new
d ev elopm ents in search tools and websites.
D efine your topic in term s o f ‘keywords’ that you can use to search various inform ation
sources. T h in k about alternative m eanings and contexts for your keywords. Also think o f ways
o f co m b in in g words to ensure that only the m eaning you want is retrieved. And consider alter­
native spellings o f the words you use. For instance, Am erican and British English use different
spellings for m any words. S o you may want to use a truncation that will capture both spellings
(su ch as behavio*, which will retrieve behavior, behaviour, behavioural, and behavioural).
L ook at ways to lin k your keyw ords. D o you want records w hich contain ail o f your key­
w ords o r are som e alternatives? D o you w ant to relate tw o words so that they are near each
oth er? C a n you focu s your search by excludin g words w hich m ight appear with oth er words
H O W TO DO R E S E A R C H : AN O V E R V I E W

Surveying the Literature to Narrow Your Search for a Research

b o x 5.4
Question

Step 1

1. Identify a research area: c o n flic t

2. Identify a case, particular space of time, or geographic area in which you might want to inve<
Mexico.
3. In Google Scholar type 'Conflict Mexico'. Scan the titles of the listed articles. This identifies q
that scholars are asking about these conflicts, and suggests questions that they either fail to a
address insufficiently, including the following:
a. What are the motivations behind the conflicts?
b. What are the main economic determinants?
c. What are their relationships with political and social issues?
d. What type of conflicts persists in the country?
e What are the root causes of the conflicts?
f What are some of the politico-economic losses and gains?
g What have been the impacts on human development and poverty?
h How does the analysis of the conflicts relate to past and current theoretical discourse anc
i. Which and where are the inflexion points of the conflicts7
j. Have the causes and relationships changed over time? What dimensions have influenced
k. What is the intensity of the conflicts7
I How does (do) the conflict(s) in Chiapas compare to other cases in Mexico? Are there any li
m. What issues have been addressed7 Which ones remain unresolved7
n What politico-economic tools are needed to mitigate and overcome the conflicts7
o What are the lessons learnt for conflict prevention and conflict resolution?
5 Select an aspect or dim ension of the research area and com pile a bibliography on it the rela
of environmental factors to persistent internal conflicts in south-eastern Mexico (the stat
Chiapas, Guerrero, and Oaxaca).

Step 2
6 Begin to assemble a bibliography on this research area by scanning one article, below.

Pick out the particularly interesting titles cited in this article and from am ong the 32 titles that Gc
Scholar shows as ha/m g cited this one, including tins particularly useful looking one:

7 Collect and then read the literaturo you have compiled on that specific aspect or dim ension
researrb area and thml, of ways to oytoncl or refine it
 A S K IN G Q U ES T IO N S

you are searchin g for but in a different context? How you do this and what options are avail
able are not th e sam e with every database, so you will need to use the on screen links and
help system to fam iliarize yourself with how each of them works
Set lim itations on your search as, for instance, by publication date ( how tar hack you want
to search), range (th e types o f publication or docum entation you wish to include), geo
graphical or tim e scope, or coun try o f publication. Be sure to keep a com plete and accurate
record o f your searches. A detailed record o f everything useful you find will enable you to
provide an accurate bibliography at the end of your project.
As part o f a strategy to narrow your search lor a question, you might do an online search
o f fellow ship and grant program m es that provide support lor research in our field Ihere are
various agencies and funding bodies that offer awards and grants to support research on
top ics related to the subject m atter o f our field. And often they issue calls for applications or
proposals. These calls for proposals typically focus on a broad topic and include a break
down o f that topic into a set of narrower issues or questions relating to it Box S.?> shows a
sam ple o f a call for proposals in the field of politics, (iov ernm ent agencies also publish
requests for proposals. Ih ese are likely to describe som e problem that the agency would like
research ers to address, and the approach they would like you to take in investigating it We
would not enco urage you to actually respond to these calls while working towards a degree.
But they m ight serve to spark your in terest in a q uestion and start you on the path that leads
you to a research question.
W e have d iscussed w here and how to find a question. Ihis is the best way we can think of
to get you started. However, as you develop experience and confidence as a researcher, you
will d oubtless be able to find qu estions through other, and perhaps m ore creative, ways.
In any case, on ce you have a qu estion, it is likely that there will still be som e distance to
travel b efore you have form ulated it so that it satisfies the three criteria which define a
research qu estion (on e that has significance for a topic or issue relating to the subject m atter
o f our field, that is researchable. and that has not yet been answered satisfactorily) We take
up th e issue o f how to form ulate a research question, below.

How to formulate a research(able) question

Let s begin ou r con sid eration o f how to form ulate a research question by recalling the three
requ irem ents o f a research question. For the m om ent, we will focus on the first two require­
m ents; we will be d iscu ssing the third requirem ent a bit further along.
At the b eg in n in g o f the research process, the requirem ent that a question be both (1) sig-
nificant (fo r real-w orld problem s or for academ ic debates or th eories); and (2 ) researchabU
(answ erable th rou gh research ) can som etim es be a source o f frustration. This is because a
research er often will start with an interest either in a general subject m atter or broad qu es­
tio n , o r in a qu ite specific case o r problem . Those who begin with an interest in a general
su b ject m atter som etim es have difficulty com ing up with a m ore specific case o r dom ain in
w hich to investigate it; that is, they have difficulty com in g up with a narrower, retearchaW e
q u estio n. C onversely, th ose w ho begin with a narrow er issue or problem som etim es have
d ifficu lty relatin g it to a broader, m ore general area o f interest. So we often have in hand a
112 H O W TO D O R E S E A R C H : A N O V E R V I E W

b o x 5 .5 Calls for Proposals as a Source for Research Questions

Global uncertainties: security for all in a changing world

programme, under the auspices of Research Councils UK
ESRC/AHRC Fellowships on Ideas and Beliefs
The ESRC and AHRC invite full applications from researchers from across the social sciences, arts, and
humanities for a number of high-profile research fellowships, as part of the RCUK Global Uncertainties:
Security for All in a Changing World programme
The cross-Council programme focuses on the nature and interactions of five global issues: conflict,
crime, environmental degradation, poverty and terrorism, and their implications for various concepts and
contexts of security and insecurity. Within this framework, this fellowship call focuses specifically on how
ideas and beliefs of individuals, communities, and nation states relate to these five global phenomena.
Fellowship applications under this call must address one or more of the following key research areas:

a. How do individuals and communities develop their ideas and beliefs about security and insecurity?
b. Why do some ideas and beliefs lead to conflict, violence, or criminal activity? What lessons can
we learn from a) above that provide the basis for countering those ideas and beliefs that reinforce
conflict, violence, and crime?
c. How do issues around the cycle of knowledge production and use interact with the creation,
management, and resolution of insecurities?
d. How are risks and threats communicated, constructed, represented, and received by key actors and
communities, using different media and cultural forms for different audiences, including the use of
language, images, and symbolism?
e Is there an acceptable balance between national security needs and the protection of civil liberties
and human rights7 If so, can one be secured? And how do we balance local needs against global
responsibilities within a security context7
f. How should institutions with responsibility for different aspects of a broad security agenda, including
security forces themselves, evolve to meet new risks and threats?

Proposals addressing the interaction between these questions/areas and applications which challenge
existing policy or practice assumptions and/or consider alternative framing or approaches to addressing
these priority research areas will be welcomed. The full specification document can be found below.
Source. http://people bath.ac uk/esscjs/Research_Office/GU .Fellowships Spec.pdf

sig n ifica n t q u e stio n th at w c d o n ’t k n o w h o w to m a k e r e s e a rc h a b le , o r a r e s e a rc h a b le q u e s ­

tio n w h o s e s ig n ifica n c e for th e field ol p o litics we h a ve n o t yel b e e n ab le to d e t e r m in e o r
d efine.

lo aid th e p ro c e s s ol d e v e lo p in g a s ta te m e n t of th e q u e stio n th a t m e e ts b o th th e se re q u ir e ­
m e n ts, we in tro d u ce a to o l, th e 're s e a rc h va s e ’. My p ro v id in g a v is u a liz a tio n o f th e re s e a rch
p ro c e s s, th e re s e a rch v a s e ’ m a y h elp yo u to lo c a te w h e re y o u r q u e s tio n falls in th e ty p ical
c o n u n d r u m we h ave d e s c rib e d , an d to id e n tify w h at y o u n e e d to d o in o r d e r lo m a k e it s a t ­
isfy b o th re q u ire m e n ts o f a re s e a rch q u e stio n .
 A S K IN G Q U ES T IO N S

To? of vaw the broad question o top«

relating to the vubjea muter ot our («M

Middle of the vase the researchabie question

that *nll enable you to investigate the question

Bottom oi the r o e how the conclusions o* your rn e v c h

help to illuminate the broad question at the top of tf* ***

Flgur* 5_2 The research vase

Th« research vase

M ost research p rojects share th e sam e general structure We can think of this structure as
follow ing th e shape o f th e vase that is pictured in Figure 5 2
T h e vase in th is p ictu re has a broad top and base, and a narrow m iddle Figure 5.2 shows
how d ifferen t a sp ects o f th e research process can be seen as corresp ond in g lo different
p a rts o f th e vase. The top o f the vase represents a large qu estion that engages a broad
th eo retica l p ersp ective or area o f in qu iry relating to the subject m atter o f our held Its nar
row er m id d le rep resents a narrow er, m o re specific qu estion which can be addressed
th rou g h co n d u c tin g research : a qu estio n which is, in oth er words, retearth ab le ' After
p assin g th rou gh th is narrow m iddle, the vase broadens out again Ih u part o f the vase
r ep resen ts th e final stage o f research: th e stage in w hich th e research er reflects on the c o n ­
clu sio n s o f th e research for b oth research ers and p ractition ers (com pleted at the point of
th e p ro cess represented by th e 'm id d le o f the vase ) and what th ese conclu sion s tell us
a b ou t th e b roa d er q u estio n or larger area o f in qu iry w hich is represented by th e top o f the
resea rch vase.
T h e vase, th en, enables us to visualize the first two requirem ents for a research question
a nd how th ey are related to each o ther. The top o f the vase represents the broader question or
terrain w ithin w hich you locate your research, and for w hich it is significant. It should be
p oin ted out that when you ‘fill in' this part o f the research vase you not only satisfy the first
requ irem en t o f a research q u estio n, you also establish part o f the rationale for your question.
L ook again at th e c om p on en ts o f the research process shown in Box 5.1 "The question’
in clud es tw o elem ents: a statem ent o f the question w hich your research will address; and the
ration ale for pursuing research on it. We establish a rationale or justification for pursuing
research on a particu lar qu estion by showing that it m eets th e three requirem ents for a
research qu estio n. T h e ration ale for a question includes a statem ent o f the context for its
co n sid era tio n . D oes it engage a cu rren t real-w orid problem or issue? D oes it arise from an
issu e that is th e focus o f d ebate in ou r field? It also provides a justification o f its importance
f o r study. W h y d o you th in k what th e qu estion asks is w orth knowing? W hy does it m atter
intellectually, theoretically, o r em pirically? So. th e ration ale for pursuing research on a qu es­
tion rests, in p a r t o n a statem ent o f its b roader sign ificance and how its answer will co n trib ­
ute to collectiv e know ledge about p olitics and in ternational relations But a com plete
H O W T O D O R E S E A R C H : A N O V E R V IE W

statem ent o f th e ration ale also will in clu d e an ex p lan atio n o f w hy and in w hat resp ect th e
qu estio n has not yet b een definitively answ ered.
The m iddle o f th e vase represents th e narro w er q u estio n th at a research er form u lates in
ord er to be able to address the top ic o r qu estio n w hich is at th e top o f th e vase and w h ich , as
stated o r in respect to what it asks, is to o b road to answer. T h is n arro w er q u estio n en ab les th e
research er to engage in d irect m easu rem en t or ob serv atio n o f som eth in g . It in d icates the
specific o b serv ation s o f reality in a p articu lar tem p oral and/or spatial settin g, o r th e sp ecific
th eoretical idea or c o n cep tu al system , w hich will be the focu s o f research . Finally, th e bottom
o f the vase represents the stage in the p rocess o f research w here th e c o n clu sio n s o f th e
research are used to reflect b ack on th e broad qu estio n or top ic at th e top o f th e vase.
W e previously noted that it is often th e case that, at th e b eg in n in g o f th e research process,
a research er will have in m in d either a qu estio n that is to o broad to b e research ab le (th e top
o f th e vase), or a qu estion that is very sp ecific and needs to be related to a b ro ad e r issu e or
problem o f sig n ificance in our field (th e m iddle o f the vase). T h ose w ith an in terest in a g en ­
eral su b ject m atter will need to find an em p irical case o r d o m a in — a sp ecific aspect o f th e
problem and/or a settin g (a specific co u n try or g ro u p )— w hich will enab le th em to in v esti­
gate the su b ject m atter and to derive c o n clu sio n s relevant to th e b road er qu estio n ; th o se w ho
start with an interest in a particu lar case will need to id en tify it as a case, in stan ce, o r asp ect
o f a m ore general issue o f interest to scho lars pursuing research in the field o f p olitics.
C o n sid er th is q u estio n : ‘Is th e U S w in nin g its “W ar on T e r ro r ”?’ T h is is a go o d qu estio n :
it is sig n ifican t, the su b ject o f m u ch debate, and o n e to w hich we do no t yet have a d e fin i­
tive answer. But it is also a q u estio n th at in this fo r m is no t p o ssib le to answ er th ro u g h
research , b ecau se it is too b road to be studied directly. I f you were in terested in pu rsu in g
research on th is q u estio n , you would need to express it in a form that is researchable. This
would require you to find an asp ect o f th e ‘W ar on T e r ro r ’— o n e w eapon , o r p o licy — th at
can be the focus o f research , and a settin g in w hich to study it. For in stan ce , you m ight
focus on one in itiative b eing pursued in th e ‘W ar on T erro r’: th e sec u ritiz a tio n o f M uslim
charities. ‘Secu ritizatio n ’ m eans th e rep resen tation o f so m eth in g (in th is case, th e rep re­
sen tation o f M uslim ch arities by g ov ern m en ts) as a p oten tial secu rity p rob lem . In th e case
o f M uslim ch a rities, they have been secu ritized b ecau se th eir fin an ces m ay su p p ort groups
classified by gov ernm ents as co n n ected to illegitim ate violen ce. W ith th is as a focu s, you
m ight ask: ‘W hat has been the im pact o f the secu ritizatio n o f M uslim ch arities in Egypt
sin ce 9/11?’ You m ight hyp othesize that ‘the secu ritizatio n o f M uslim ch arities in Egypt
sin ce 9/11 has increased ten sion s w ithin the co u n try ’. N ow you have developed an e m p iri­
cally testable hyp othesis w hich can co n trib u te to answ erin g the b road er qu estio n with
w hich you began.
A lte rn a tiv e ly , y o u m ig h t h a v e b e g u n th e r e s e a r c h p r o c e s s w ith a n in te r e s t in a n a r ­
ro w e r q u e s tio n a b o u t th e im p a c t o f n e w r e s tr i c ti o n s o n M u s lim c h a r i t i e s in E g y p t. H e re ,
y o u b e g in w ith a s p e c ific r e s e a r c h i n te r e s t ( th e m id d le o f th e v a s e ) , a n d w ill n e e d to ‘fill
in ’ th e ‘to p o f th e v a s e ’ by c o n s i d e r i n g w h a t m o r e g e n e r a l p h e n o m e n o n o r iss u e th e
re s e a rc h q u e s tio n a d d r e s s e s ; h o w y o u r q u e s tio n m ig h t b e lin k e d to a b r o a d e r a r e a o f
c o n c e r n o r in q u iry in o u r field ; w h a t y o u r fin d in g s m ig h t e v e n t u a l ly tell u s a b o u t m o r e
g e n e ra l iss u e s. Ih is, th e n , m ig h t le a d y o u to p o s i ti o n y o u r r e s e a r c h o n M u s lim c h a r it ie s
in h g y p t as p a rt o f a b r o a d e r q u e s tio n c o n c e r n i n g w h e th e r o r n o t th e U S is w in n i n g its
‘ W a r o n T e r r o r ’.
 » « i n t a V U tS T IO N S 115
T o con tin u e, for a m o m en t, w.th the issue o f relating a specific question to a broader ter
rain: you can often identify the broader em pirical or theoretical terrain or context relevant
to your qu estion by translatin g individual nam es into kind nam es For instance, an interest
in the ASfcAN security fram ew ork can provide insight into the m ore general phenom enon
o f regional in teg ra tio n ; a specific question about the n o rth -so u th conflict within Sudan can
eng age an aspect o f the literature on civil wars. A researcher who wants to learn m ore about
th e tribal Pashtuns o f south eastern A fghanistan and north western Pakistan might relate
th is research to a bigger qu estion about the survival and continu ing relevance ot tribes as a
form o f p olitical organization and authority.
But let s look m ore closely at what is entailed in the process ol narrowing down' a broader
qu estion. C onsid er the following question: D oes globalization underm ine the basis of the
nation state? I he q uestion concerns the relationship between globalization' (concept I ) and
th e basis o f a nation state (concep t 2). To m ake this question researchable, each concept will
need to be defined specifically enough to allow us to observe or m easure whether or to wtul
extent on e o f th em (globalization) is adversely affecting (underm ining) the other (the nation
state). W e can do this by choosing one facet of globali/jtion and one leaturr of the basis ol the
nation state that will enable us to explore m ore concretely the relationship between globaJi/a
tion and the basis o f the nation state. We might choose, for instance, to focus on econom ic
in tegration , w hich is a key facet o f globalization; and on state autonomy, which is a key basis
o f th e nation state. I h is will enable us to ask a narrower question: 'W hat im pact does eco
n o m ic in tegration have on state autonom y?' lh rou gh further reading, we might discover that
research ers m easure th e integration o f a country into the world econom y by using an index
that com bin es m easures o f a country's various transnational links and its involvement in
in ternational organizations. Further reading will also enable us to better understand the core
fu n ction s o f th e nation state; and we might decide that a state's ability to manage public
fin ances is a critical indicator o f th e autonom y o f a state, inasm uch as a state cannot enjoy
autonom y if it must depend on external sources to fund its ongoing operations. We continue
to explore ex istin g research and find that researchers have m easured the degree o f a state’s
autonom y and w hether it is increasing or decreasing by using data on the ratio of dom estic to
foreign sou rces o f revenue in a states budget at a given tim e, and changes in this ratio over
tim e. O u r q u estion, th en, m ight be. D o high levels o f integration into the world econom y (a*
m easured by our in d ex) d ecrease a states ability to manage its public finances? So here we
have it. a research qu estion which will enable us to investigate a narrower, more specific ter
rain relating to the broader q u estion about globalization and the bases o f the nation state.
Th ere are tw o o th er issues involved in m aking a question researchable. t h e first has to do
with w hether you are asking the kind o f question you want to ask. I h e second issue has to do
w ith the in clu sion o f errors o f reasoning in the statem ent o f a question. We discuss both o f
th ese issues, below.

Types of questions: getting dear about what you want to know

O n c e you th in k you have a qu estio n, and before proceeding any further with the research
process, it is w orth giving som e thought to what your qu estion is asking and what you will
n eed to d o in ord er to answ er it. Is your qu estion, as stated, asking for som ething different
fro m w hat it is th at you really w ant to know and do?
116 H O W TO DO R E S E A R C H : AN O V E R V IE W

In ord er to get clear ab out w hat you are askin g and w ant to know, it is useful to co n sid er
various types o f qu estio n s that research ers ask. N ot all types o f research qu estio n will be
appropriate to the requ irem en ts o f your d issertation or o th er type o f p ro ject as d efin ed by
your d ep artm ent. For in stance, som e d ep artm en ts m ay require that a d isse rtatio n be b ased
on an explanatory qu estio n. However, though th e req u irem en ts o f y our d ep artm en t m ay
lim it your ch oice am o ng different types o f research qu estio n s, u n d erstan d in g so m eth in g
about all types o f qu estio ns will help you to form ulate your qu estio n and to be clear about
what you will need to do in ord er to answ er it. C o n sid er the types o f qu estio n s listed in B ox
5.6. Each type o f qu estion is asking for a p articu lar type o f in fo rm atio n . For in stan ce , a
descriptive qu estion is on e that asks for in fo rm atio n ab ou t the ch aracteristics o r b ehav iou r
o f som eth ing. Th is is perhaps stating the obvious. But by m ak ing th is exp licit you will be able
to consid er w hether the qu estion you have form ulated is askin g about th e th in g th at you
really are in terested in know ing. A lso, by m aking exp licit what kind o f q u estio n you are ask ­
ing and what that qu estion is asking for, you can begin to th in k ab out what you will be
required to do in order to answ er it. Let’s con sid er each o f th e types o f q u estio n s show n
in Box 5.6.

Descriptive questions
D escriptive questions are concerned with the characteristics o f what has happened, what is
going on, or what exists; and with the characteristics o f how som eth ing behaves. D escriptive
questions often ask ‘W ho, W hat, W here, and W hen’. These can be research qu estions if they are
significant and researchable, and if it can be shown either that people disagree about their

b o x 5.6 Types of Questions, What You Want To Know, and What You Will Do

Type of question: What you want to know: What you will do:

Descriptive The characteristics of Describe the characteristics o f something; model

something or of how it works how it works or behaves

Explanatory ■s of something that Explain what factors or conditions are causally

red or is happening connected to a known outcome

• outcome of r urrent Predict what outcome will occur as a result of a

s oi trends sot of known factors or conditions

Prescriptive
 A S K IN G Q U ES T IO N S

answer or are, in som e way. wrong about the answer on which most o f them agree. For example.
who killed Pakistan’s Prim e M inister. Benazir Bhutto? What national security doctn ne guided
US foreign policy under George W. Bush? When did the takr-otf ’ to industrial development
occu r in England? Where did nationalism , or the nation state m odel, first emerge? But d escnp
tive questions com e in a variety o f forms. Ihe question Have the foreign policies o f h i ’ mem
ber states becom e Europeanized?’ asks whether we can ileu n he the foreign policies of F I'
m em ber states as having becom e Europeanized' Has class dim inished in unportaru c as a basis
for voting?’ is a question which asks whether, in describing the bases of voting preference, class
is less im portant than it previously was. Ihe question. What has been the impact of structural
adjustm ent program m es on third world countries?’ is asking for a description of the im pact of
these program m es. W hen we ask W hat proportion o f people hold various opinions?' we are
asking a descriptive question for which public opinion polls provide an answer. Som e descrip
tive questions can be answered rather easily, as they have a clear factual answer that is readily
accessible and uncontested. O thers, however, are not easily resolved, mav be the subject ol
m uch contestation, and can provide much scope for theoretical and empirical investigation.

Explanatory q u estio ns

Exp lanato ry qu estions generally ask about what is musing or hat m uted an outcom e, or whv
so m eth in g exists or has happened: ‘W hat accoun ts for the resurgence o f “Hindu national
ism ” in India in recent years?' 'W hy are British citizens generally less favourably disposed
tow ards th e EU than their coun terparts in oth er m em ber countries?' 'W hy was there a
reversal o f d em ocratization in G eorgia?' Ihese questions are answered through identif ying
the o u tcom e to be explained (dependent variable) and the factor or factors thought to be
c o n n ected with it (in d ependen t variables); and then showing whal the nature o f the connec
tion o r relation betw een them is.
Exp lanato ry and d escriptive questions are both concerned with conn ectio ns or relations
betw een two o r m ore variables. But while a descriptive question m ight ask what relationship
exists betw een gend er and voting preference, or o f educational achievem ent and political
p articip ation , an explanatory question will be concerned with why that relationship exists.
So, for in stance, a descriptive question m ight ask what proportion o f m ales and fem ales say
th ey would v ote for a D em ocratic o r a Republican candidate in the next U S presidential elec
tion, w hile an explanatory question m ight ask why m ore w om en say they will vote for a
D e m o cra tic cand id ate in the next US presidential election.
But th ere is not alw ays a firm distinction between questions w hich ask for descriptions
and th ose w hich ask for explanations. A question that asks about the causes o f'e th n ic c o n ­
flicts’ is already sugg esting som eth ing about th eir causes ('eth nicity'), and an elucidation o f
th ese causes will, in tu rn , con firm our description o f them as eth n ic’ conflicts. Ih e answer
to th e qu estio n o f w hether people with a college education are m ore apt to vote than those
w ith on ly a grad e sch o ol education, offers b oth a description o f those who vote, and a pos
sib le exp lanation for v oting (W h ite and C lark 1983: 23).
It appears to m any people that explanatory and d escriptive questions are m ore valued by.
respectively, p ositivist and interpretivist researchers. T his is because positivists tend to stress
th e pursuit o f law -like explanations as the goal o f social scien ce research, while interpretiv-
ists ten d to em phasize th e role o f in terpretation (see C hapter 2). But both a r t concerned with
d escrip tion as well as with explanation. W h ere they differ is in th eir concep tion o f what
H O W TO DO R E S E A R C H : AN O V E R V IE W

constitu tes explanation. Th e difference, as Iain H am p sh er-M o n k and A nd rew H in d m o o r

p oin t out, is that p ositivist exp lan ation s are ex tern a l’ in th e sen se th at th ey ten d to w ork via
correla tion s or ded u ctions on th e basis o f ascribed reasons, and so need no t co n c e rn th e m ­
selves w ith a ctors’ u n derstandin gs o f th e w orld; w hile in terp retiv e e x p lan ation s are ‘in te rn a l’
in that they are co n ce rn e d with th e w orld o f m ean in gs in hab ited by th e a cto r (H am p sh er-
M on k and H in d m o o r 2 0 0 9 :4 8 ).

Predictive questions
Predictive qu estions are co n ce rn e d w ith the likely effect o r ou tco m e o f so m eth in g o r th e
tra jecto ry o f existin g trends. In ou r field, research ers tend to b e co n ce rn e d with qu estio ns
about the ou tcom e o f g ov ernm ent policies: ‘W h at will be th e effect o f an in terv e n tio n to
b ring about regim e chan ge in Z im bab w e?’ ‘W h at will be the im p act o f th e “A rab Sp rin g” on
possiblities for d em ocracy in th e M iddle E ast?’ ‘W h at will be th e co n seq u e n ce o f n a tio n a l­
izing insolvent firm s?’ R esearchers are also co n ce rn e d about th e ou tco m e o f trends: ‘W h at
im pact will in creasing urbanization have on Egypt’s un em ploym en t rate?’ ‘W ill c o alitio n
casualties in A fghanistan increase over the next year?’ ‘W ill C h in a and Japan develop m o re
cooperative relations in the next tw enty years?’ ‘W ill eth n ic co n flict in crease in European
cities in the next twenty years?’
Predictive studies use explanations to speculate about w hat w ill likely o ccu r o r b e th e case
in the future. So, for instance, if we analyse ev iden ce and find that event A causes or e xplains
event B (an explanation), we th en can predict that if A con tin u es to in crease, we will likely
have a greater am ount o f B in the future (W h ite and C lark 1983: 2 3). P redictive q u estio ns
require you to develop an argum ent that if certain con d ition s or circu m stan ces prevail, a
certain outcom e is likely to o ccu r or com e into being.

Prescriptive questions
These are qu estions that ask about what we should do, or about what is right or w rong, or
good or bad. A prescriptive question asks: ‘How can we b ring about X (o r prevent Y and Z )? ’
Researchers in our field are concerned with p rescribing what gov ernm ents should u n d er­
take to do in order to bring about or to prevent som eth ing. W h at should th e U N do to b ring
about an end to the war in D arfur? W hat steps should the in ternational c om m u n ity take to
reduce hum an trafficking? W hat can the British governm ent do to stabilize its h ousin g se c ­
tor? W hile prescriptive and norm ative qu estions are often closely con n ected , they are not
necessarily the sam e. Prescriptive qu estions are m ore con cern ed with ascertain in g facts
needed to solve political problem s. The question, ‘Is d em ocracy the best form o f gov ern ­
m ent?’, is a norm ative question; but questions about what governm ents can do to expand
political participation, or to ensure sound electoral processes, or to facilitate peaceful tran s­
fers o f power, are prescriptive questions.

N o rm a tiv e q u estio n s

W hile prescriptive questions are concerned with identifying the best m eans to given ends,
norm ative questions are concerned with d eterm inin g what the ends them selves should or
ought to be. A norm ative question asks: ‘W hat is best, just, right or preferable and what,
therefore, ought to he done (or not done) to bring it about (or prevent it)?’ W hen , if ever, is
war and political violence justified? W hat makes a governm ent or state legitim ate? Are there
 ASKI NG QUI S U O N S

Ir a iiM c iu lt'iit o r u n iv e rs a l h u m a n r ig h t s ' S h o u ld h tiin a n rig h ts lim it th e vsav s t j t o , a n treat

th e ir u t i / e n s ' O n w hat l>asis s h o u ld tr.uk- o ils b e tw e e n li h e r n an d m m i i i U l>e d e e d e d '
W h a t c o n s titu te s p o litic a l e q u a lity an d d u o it r e q u ir e r o m . . m u ei| iialit\ ' I.. w hat ex ten t
s h o u ld lib e r a l t l n i i o i ra i u-s to le r a te illib e ra l * u l h ir e s '
Mox '» 7 s h o w s d ille r e n t lo r m s th a ï des* i ip tu e eu p la n a t. .r s p ie d u t is e p rès, r ip tn e and
M o riii.itiv c <|ue\tions tak i'

box 5.7 Types of Questions

Your question What you wilt do

Dew rib«-

Describe

Describe

Describe

Explain

Explain

Explain

Predict "

Predict1 ■

How i ’ I"' is ii a. !:■' f . ■>< I '"'1

,f ' ’ Prescribe 1
inflLJfru n!<‘

Wh.lt is I»'.' ii» >st .'If." live way rh.K f< -v.> Prescribe :«
can rpspond to torronsm'

How '.ari |»,.v.‘r"rM,-r>ts . ...niff ■

' '■"i
maintaining demm rain fr»***f1<.ms

When, if ever, are war and political v.uU-i Make expUdt the moral intpUctfiore of different
points o< view concerning .1ever war and
justified?
political violence are justified and present reasons wty
your own point of view is to be preferred
H O W TO D O R E S E A R C H : AN O V E R V IE W

Unanswerable questions: fallacies in framing research questions

W e have established that a qu estion m ust b e sufficiently narro w o r sp ecific to p erm it em p ir­
ical in vestigation, and that it should be form ulated as a type o f q u estio n th at asks w hat you
are really interested in answ ering.
H owever, a qu estion that satisfies these co n d itio n s m ay still no t b e research ab le if it c o n ­
tains errors o f logic, or ‘fallacies’ that would gen erate false answ ers or c o n clu sion s. T h e m o st
co m m o n errors o f th is sort are when a qu estion is fram ed so that it

1. ‘begs’ a n o th er qu estio n;

2. presents a false d icho tom y;

3. asks about a fictional event;

4. is m etaphysical;

5. is a tautology.

An exam ple o f each o f these fallacies should suffice to m ake clear the sort o f e rro r it entails
and question that results.
C onsid er this q u estion: ‘W hy was A m erican slavery th e m o st awful that was ever know n’?
As the historian D avid H ackett Fisch er p oin ts out, this is an explanatory qu estio n (a ‘why’
qu estion) that begs a second (descriptive) qu estion: th e qu estion o f w hether A m erican slav­
ery was ‘th e m ost awful that was ever know n’ (F isch er 1970: 8). This secon d qu estion ‘goes
begging’ in the sense that it is not explicitly asked and answ ered, but is sim ply assum ed. In
oth er words, the qu estion is form ulated in a way that really poses two qu estio ns; but it is
explicitly asking— and intending to answ er— only on e o f them .
A question that is constructed so as to present a false dicho tom y is on e that forces us to
ch oo se betw een two answers that are neither m utually exclusive n o r collectively exhaustive.
F isch er describes what happens when qu estions such as these appear on an undergraduate
essay exam in ation : confronted with a qu estion o f this sort, the ‘disgusted undergraduate’ is
forced to m ake a choice betw een ‘unappetizing alternatives, or perhaps to co m b in e th em in
som e ingenious paradoxical contrivan ce o f his own invention’ (F isch er 1970: 10). Som e o f
Fisch er’s exam ples will likely rem ind you o f countless others: ‘N apoleon: E nlightened States­
m an or P roto -Fascist?’; ‘Jacksonian D em ocracy: Myth or R eality?’; ‘Plato: Totalitarian or
D em ocrat?’
Ih e n there is the fallacy o f ‘fictional qu estions’. C onsid er this exam ple: ‘W ould US Presi­
dent Franklin D elano Roosevelt have decided to drop atom ic bom b s on Japan had he still
been in office in August 1945?’ The problem with this question is that there are no secure
grounds on which to base an answer to it. We m ight be able to contrast the views o f R o o se­
velt with those o f his Vice President, Harry T rum an , who succeeded him and w hose d ecision
it was to use these weapons. But how can we know' that, if he had still been president, R o o ­
sevelt would not have done precisely what Harry Trum an did? Em pirical research cann ot
answer this question.
We encounter the sam e problem s with the fallacy o f m etaphysical questions: these are
questions that attempt to ‘resolve a no n-em p irical question by em pirical m eans’ (F isch er
1970:12). C onsider this question: ‘Was W orld War I in ev itable?’ How can you answ er a
question about inevitability through em pirical research? Using words like ‘inevitable’,
 A S K IN G Q U ES T IO N S

unavoidable; and inescapable com m its you to an argum ent that goes bevond what your
research can establish.
Finally, there is the problem o f tautological questions Ihese com e m a vanetv of forms Ihe
most com m on consists o f the statement that things that are X are in tact X . or X is X' Consider
this question: ‘W as G eorge W. Bush unsuccessful because he was moving against the tide of
history? We would not know w hether he was ‘moving against the tide of history except for the
fact that he was unsuccessful. So this question asks if he was unsuccessful hccausc he was unsuc
cessful. Here’s another example: D id the assertion of separate national m m cm cn ts cause the
centrifugal tendencies (i.e. the dispersal of political power away from the central governm ent)
that led to the dissolution o f Yugoslavia?’ Ihis asks if centrifugal tendencies were the cause of
centrifugal tendencies. If you ask. D o mainstream Am ericans believe that racism is unaccept
able?, and further discussion reveals that finding racism unacceptable’ is what you mean by
m ainstream , then what you are asking is whether mainstream Americans are mainstream.

'Researchability': a final consideration

To be researchable, your question must be within your ability to answer it. You need to consider
the m ain practical issues involved in answering your question, including the availability and
accessibility o f data, and additional resources and training you might need. Will you be able to
access people, statistics, or docum ents from which to collec t the data you need to address the
question fully? C an this data be accessed within the limited time and resources you have avail­
able to you? How long would you need to com plete the research necessary to answer the ques­
tion? W hat ethical constraints m ight be at issue? I h e answers to these questions (which we will
consid er further in C hapter 6 ) may lead you to redefine your research questions.
So far, we have consid ered (1 ) the need to form ulate a question so that it relates to an issue
o r area o f in qu iry that is significant for the field o f politics; and (2 ) the various considerations
that ensu re that a qu estion is researchable. These considerations relate to two o f the three
requ irem en ts for a research question. In the next section, we address the third requirem ent
o f a research qu estion: that the question does not yet have a definitive answer. I h e next sec­
tion explains how th e ‘Literature Review* enables you to establish this.

The literature review

T h e c o m m o n ly used term , ‘literature review ’, sou nds like som eth ing m undane and
m e ch a n ica l: an ob lig a to ry ch o re contrived to ensure that you have d one som e reading on
your top ic. Th is is un fortu n ate; and it is also m isleading. A review o f th e e x istin g literature
is not on ly a c ru c ia l part o f th e research process, it is where you d em onstrate your ability to
eng age analytically with p olitics, w here we hear your voice and find out how you th ink and
reason.
A literature review h as a dual fu n ction . First, it explains why. and to w hat extent, a d efin i­
tive answ er to y our q u estio n does not yet exist; so it provides the m eans by w hich you d em ­
on stra te th at y our q u estio n m eets th e th ird requirem ent for a research question. Secon d, and
in th e p rocess o f p erfo rm in g th e first fu n ction , it prepares th e ground for th e elaboration o f
y ou r ow n a r g u m e n t B o x 5 .8 shows how asking and answ ering a few key questions about the
literature ad vances e ach o f th ese fun ction s.
122 H O W TO DO R E S E A R C H : AN O V E R V IE W

b o x 5.8 The Dual Function of a Literature Review

1. Establish that a question has not yet been definitively answered, or that no one else has done
what you are proposing to do, by asking:
A. Who else has asked and answered your question? What (range of) answers have been given to this
question in the existing literature?
B. What is missing, irrelevant, defective, unwarranted, or ambiguous in the existing literature on your
topic?
2. Set the stage for your own argument, by asking:
C. What are the positive elements in the current literature? What in the literature can you highlight,
underline, expand, extend, improve, build upon, continue?
D. What needs to be done? What requires further investigation or rethinking? What would provide
a better understanding of the matter?

'Ihe literature review dem onstrates that the subject has not been adequately dealt with in
the existing literature; and it explains how you intend to address this state o f affairs, either by
doing som eth ing that has not already been done and that prom ises to yield fu rth er insight
into the question; or by doing what has already been d o n e— but better. This may require you
to show that, though the question might already be the su b ject o f m o untains o f articles and
b ooks, the answers that have been offered to it are in som e way in com plete or flawed. W hat
you will need to do is to identify oth er researchers who have asked and answered your qu es­
tion, and then say what in their answ ers is w rong, in com plete, or weak. In add ition to id en ­
tifying the weaknesses, you will also highlight the valuable elem ents in th eir answers. It is on
the basis o f both these positive and less positive elem ents that you build your case for what
you yourself propose to do to contribu te to the literature.
This is the second fun ction o f a literature review: to set the stage for a statem ent o f how
your research is going to address the weaknesses o f ex istin g studies, and/or underline,
expand, extend, improve, or build upon th eir strengths. Students som etim es th ink th ere is
no literature related to their topic. There may not be b ooks and articles that address exactly
the sam e topic, but there will always be related research that will provide a conceptual and
theoretical context lor your research. Ask yourself: ‘W hat is my question a case of? Is there a
broader literature on this type o f p henom ena?’
You should develop your literature review over three stages. There is always a first stage in
which you read, follow leads to oth er prom ising books and articles, and then read som e
more. Vou will he reading analytically — that is, asking a lot o f qu estions about what you are
reading, and also trying to get a sense ol different debates, theories, perspectives, and argu­
ments. Lventually you will need to bring som e kind o f organization to the literature, by su m ­
marizing how the literature breaks down into different theoretical perspectives, entry points,
or arguments. Iliis is the second stage ol developing a literature review, f inally, and as a
result ol the previous two stages, your own argum ent will begin to com e into focus. You
should now have answers to the questions listed under ‘Literature’ in Box 5.1 (O u tline o f
Research C om ponents). At this point you are ready to write the literature review: a d iscu s­
sion ol tlu- best research and writing on your question organized in a wav that sets the stage,
and helps to make the case, lor your own argum ent.
 A S K IN G Q U ES T IO N S

Let'» look m ore closely at each o f the three stage» m the developm ent of a literature rev *w

Stage 1: Reading the literature

Recall ou r previous d iscu ssion about approaching the literature analytically. A» you evaluate
the m a jo r argum en ts o r explanations in the existing literature on your question, ask yourself
qu estio ns about th eir weaknesses. You should query the questions, answers, m ethods, and
con clu sion s o f each co ntribu tion to the literature you read. Scrutinize their ideas, defin i­
tions, con cep ts, in form ation , or logic Ask yourself what is missing, irrelevant, defective,
unw arranted, o r am biguous. Ih ere are num erous questions you can ask about each aspecl of
th e m aterial you read. C onsider, for instance, the questions, below.

1. W hat is the a u th o rs central question or subject o f inquiry* How does the way the ques
tion is asked m ake it different from that which others have asked* How does it illum inate, or
ob scu re, cru cial d im ensio ns o f the p henom enon investigated7 W hat is the author's purpose
(to d escrib e, explain, predict, p rescribe)*

2. W hat is the author's answer or argum ent ? Wrhat factor or factors is the author's analysis
h ighligh tin g? D oes the author define these factors clearly? (W'hat is m eant by technology',
'rapid change', ‘in stab ility’?)

2a. How d oes the author's argum ent or explanation differ from others* ( Note that the title
o f the b o o k or article is usually designed to com m un icate what differentiates this particular
treatm ent o f the su b ject m atter from oth er books and articles, and the chapter titles and sec­
tio n h ead ings provide clues to how the overall argum ent is developed.)

2b. If th e p u rpose o f the inquiry is to explain so m eth in g what are the author's major
hyp otheses? Are they developed deductively from a m ore basic theory* Are they developed
inductively ? W hat is the strength, nature, and generality o f the relationship between the in ­
d epen d ent and depend ent variables?

2c. W hat assu m ptions underlie the account o f events that the author presents? Are the
assu m p tions sou nd ? D oes the argum ent assum e things that need to be dem onstrated?

2d. D oes the b asic argum ent m ake sense? Is the chain o f reasoning sound? If the author
m oves from o n e claim to a n o th er that supposedly follows from the first, does the author
d em on strate a d e a r relationship betw een the two claim s? Are th ere any leaps o f logic, in co n ­
sisten cies, c ircu la r a rgum en ts, or con trad iction s w ithin and am o ng the parts o f the argu­
m ent o r th e su bsid iary argum ents?
3. W h a t m ethods are used to d em onstrate the argum ent? Are th ere any sh ortcom in gs or
flaws w ith respect to th e m ethod s used? Is the tim e fram e to o sh ort, o r artificial; o r the spa­
tial d om a in to o lim ited o r a rtificial, to generate sound conclusion s? Are key d efinitions a m ­
b iguou s, o r to o narrow o r broad?
4. W h a t evidence is p resented in support o f th e claim s that are m ade? Is th e evidence well
selected , o r is it arbitrary ? D oes th e author use m any different kinds o f exam ples, or on es that
are ch o sen sim ply b ecau se th ey support a particular contention o r case? D oes the author
ad d ress exam p les o f w hat m ight b e considered a ‘hard case’ for the argument—cou n ter- e x ­
am p les th at m ig ht u n d erm in e th e central claim s? C an you th ink o f an anom aly— a case that
124 H O W T O DO R E S E A R C H : AN O V E R V IE W

doesn’t fit w ith or isn’t explained by th e th eory? H as th e au thor ign ored , suppressed,
overlooked , o r slighted relevant evid en ce, altern ativ e data s ou rces o r o th e r in terp retatio n s o f
the data?
5. D o the conclusions fully draw th e im p lications o f the overall argu m en t, all p arts o f the
argum ent, and subsidiary argum ents?

Stage 2: Summarizing the literature

You have read widely, but selectively in your top ic area, asked qu estio ns o f ev erything in the
m aterial you have read, and taken careful notes all along th e way. It is tim e, now, to b ring
som e organization to th is literature: to su m m arize it in a way that identifies th e m ajo r
them es, issues, argum ents, p osition s, or perspectives that con stitu te it and that h ighligh ts
th eir m ajor poin ts o f co n tact and division.
You will not be sum m arizin g all the literature relating to your top ic (o r related m ore
broadly to th e focus o f the research ); but only the literature that directly addresses th e sp e­
cific question you are asking. So, for in stance, if you w ere pursuing research on the qu estion
we discussed previously co n ce rn in g globalization and the bases o f th e nation state, you
would not review all the literature on in ternational fin ance (a vast field o f en q u iry ), but
rather the literature that addresses sou rces o f state revenue and the con trol by n ational states
over capital flows.
It m ight be appropriate, depending on your qu estion, to sum m arize the in tellectual p ro ­
gression o f the field, including m ajor debates; alternatively, your q u estion m ight require you
to focus on sources that address the relevant su b ject m atter w ithin a certain tim e period. But,
however you organize your sum m ary, it should include the m ain conclu sion s o f the m ost
im portant and influential argum ents and ideas in the existin g literature relating to your
question. W hat are the principal positions that experts take with respect to causes, co n s e ­
quences, or solutions? D o existing studies tend to divide betw een two or th ree different
approaches or perspectives? Are there different scho ols o f thought? If so, w ho are the m ajor
disputants, and what points o f difference lie betw een them ? W hat are the aspects o f the topic
about w hich experts appear to agree? W hat are the p rincipal insights that have em erged
from the literature?
These questions enable you to organize and sum m arize the literature. They will also raise
additional questions: W hat puzzles, problem s has the literature left unsolved? Is there an
aspect o f the field that is m issing? D o the argum ents and th eories in the literature tend to
focus on exam ples or cases drawn from one part o f the world? Have they been/can they be
applied to other parts o f the world? These are the sorts o f qu estions that will m ove you
towards the third and final stage o f the developm ent o f a literature review.

Stage 3: What still needs to be done? Setting the stage for your own argument

You have read, and you have thought creatively and critically about the argum en ts in the
literature and the findings o f previous research relevant to your topic (first stage). You have
been able to bring som e organization to the literature by breaking it down in to the m ajor
points o f contact and division am ong the various perspectives and argum ents that it
c o n ta in s (stag e two). In the process o f doing this, you have identified strong and weak el*
m en ts in the literature; and th is has led naturally to a consid eration o f what m ight be don
to build on th ose strengths and ov ercom e those weaknesses Consequently, in the proce*
o f a n alysing the positive and less positive elem ents in others' argum ents, your own argu
m ent has begun to com e in to focus. You are now ready to move to the hnaJ stage in th,
d ev elopm ent o f a literature review: w riting a d iscu ssion of the existing literature tha
p resents a recapitulation o f your own process o f evaluation and critique of it and the pra.es
by which your own argum ent began to take shape. In sum: your literature review' discu ssr
th e best literature on your qu estion in a way that shows how a reasoned analysis ol it
stren g ths and w eaknesses leads naturally to a consideration of what vou propose to do a s ,
co n trib u tio n to resolving the qu estion. It leads, naturally, in other words, to the introdu*.
tion o f your ow n argum ent.
H ere are som e basic considerations and ground rules to guide the writing o i your liter*
ture review.

1. A literature review should itself develop an argum ent’—a particular perspective on th»
literature. It should begin with a statem ent o f that argum ent and use evidence to back it up
Th e argum en t o f your literature review is ju st like any oth er argum ent: your interpretation o
th e m aterial you d iscu ss must be backed up with evidence to show that what vou are savinf
is valid. I h e literature review should contain an introduction, the body ot the review con
tainin g the d iscu ssion o f specific sources, and a conclusion.

2. Your d iscu ssion should be organized around ideas, them es, theories, or issues that en
able you to advance the argum ent o f your literature review, and not the sources themselvei
(you are not w ritin g an annotated bibliography—a list o f sources with a discussion o f each
on e o f th em , on e at a tim e).

3. T h e nu m b er o f sources you discuss depends on how many you need in order to per
suasively d em onstrate the argum ent o f your literature review You must satisfy a critical
read er— as for in stance one o f the authors whose work you are challengin g— that your argu
m ent is based on considerable thought and study and that you know what you are talking
about. C om preh ensiv e know ledge o f the literature on a topic is essential to research on that
top ic; and the depth and breadth o f knowledge that your literature review exhibits is what
will give you cred ib ility as a researcher.
4. A d d ress th e best argu m en ts. In clud e th ose with w hich you agree and th ose with
w h ich you d isag ree. D on ’t leave out good argu m en ts in ord er to m ake a case that th ere are
n o g o o d arg u m en ts a lo n g th e lin es o f what you you rself m ay w ant to develop. Don't
d ow n play o r m isrep resen t th e so p h isticatio n and im p licatio n s o f th e argu m en ts advanced
by a stro n g o p p o n en t; and d on ’t go after a w eak o p p on en t (a straw m an ). It may be the
ca se th at th e on ly op p o n en ts are w eak. Be sure, however, you haven’t o v erlooked a stron ger
on e.
5. The types of sources that should be included (books, journal articles, websites1) will
depend on your topic. You are concerned with the state of the best current literature and
previous research on this question. Focus on the articles and books that are most influential
and widely cited. Concentrate your efforts on relevant articles published in the most credible
research journals dealing with the general area of inquiry related to your question. Ideally.
126 H O W TO D O R E S E A R C H : A N O V E R V IE W

th ese should b e research jo u rn a ls that use a b lin d -rev iew system . In a blin d review, authors
subm it potential articles to a jo u rn a l ed itor w ho solicits several review ers w ho agree to give
a critica l review o f th e paper. The paper is sent to th ese review ers w ith n o id en tification o f
the author so that th ere will be no personal b ias (eith er for or again st th e au th or). B ased on
the review ers’ recom m en d ation s, the ed itor can accep t the article, r eject it, o r reco m m en d
that the author revise and resubm it it. A rticles in jo u rn als w ith b lin d -rev iew processes can
be expected to have a fairly high level o f credibility.

6. Select on ly the m o st im p ortant p oin ts in each sou rce th at relate d irectly to th e argu ­
m ent o f your literature review.

7. A good literature review is on e in w hich the m aterial is th oro ughly ‘processed ’: o rg an ­

ized accord ing to an overall argum ent and around key issues or th em es, and rendered in the
w riter’s ow n voice. W h ile th e literature review presents o th ers’ ideas, your voice and your
own perspective should predom inate. Use quotes sparingly and m aintain your ow n voice by
starting and ending the d iscu ssion o f different issues, th em es, or individual sou rces with
your own ideas and words, using the sou rces to sup port what you are saying. Find m odels:
read the literature reviews in books and articles w ritten about your area o f in terest to get a
sense o f what style and organization is effective.

Conclusions
This chapter has been devoted to the first step in the research process: finding and formulating a
research question. We have discussed the following:

• three requirements of a research question. We have said that for a question to be a research
question you will need to:

(1) articulate a rationale for pursuing research on the question in terms of its significance for a
broad issue or area of interest within our field:
(2) formulate the question so that it is researchable-so that it can be answered through research;
and
(3) show that the question has not been definitively answered:
• how to go about meeting these requirements. We introduced a visual tool-the research
vase'-which can help you to conceptualize the first two requirements of a research question.
To meet the third requirement of a research question you must show how others who have asked
this question failed in some way to provide a definitive answer to it The literature review performs
this function. It identifies who else has asked and answered your question, and what (range of)
answers are found in the existing literature relating to it. It highlights the positive elements in the
current literature, and makes an argument about what need to be done in order to provide a
better answer to the question than currently exists;
• different types of questions and what each type commits you to doing;
• the logical fallacies that sometimes find their way into the statement of a question and that lead to
false conclusions

Once you have formulated a research question, you have completed step one of the research process.
You will then be ready to move on to step two: developing an answer to your question. This is the focus
of the next chapter (Chapter 6)
 A S K IN G Q U ES T IO N S

Questions
• What makrs a question researchable ?

• How do you formulate a question so as to maximize its generality >

• How might errors of log* in the formulat.cn of a question generate false answers or cone logons 7

• What is the difference between a survey or summary of the literature and a literature rev*w >

• How are the requ.rements of a research question related to the rattonale or just.fKatKjn «or pursuing
research on a question’

Guide to Further Reading

Brown«, M. N. and S. M. Keeley (1994). Asking the Right Quetbom A Guide to Critical Thinking. 4th
edition (New York; Prentice HaJI)
The authors provide a wide range of questions that can be used to mtef rogaie the e.istmg Inrt aturr
on a topic

Buttolph Johnson. Janet. H. T. Reynolds, and Jason D Mycoff (2008). Polittcal Some* Research
Methods (Washington. DC: CQ Press), chapter I
This chapter discusses the steps involved in specifying a research question and developing a
hypothesis

Geddes, Barbara (2003). Big Questions. Little Answen. How the Questions You Choose Affect the
Answers You Gef. in Paradigm and Sand Castles (Ann Arbor. Ml Michigan University Press),
chapter 2.
This chapter makes the case for caution in using the literature m our field as a source of research
questions, and argues that looking for gaps m the literature may not be a good strategy for finding
research questions

Hedrick, Terry E , Leonard Bickman and Debra J. Rog (1993). 'Defining the Focus of the Research',
in Applied Research Design: A Practical Guide (London: Sage Publications), chapter 2
This chapter provides practical advice about gening clear about the problem or issue that interests
you and identifying the research questions that can address it

8
Hoover, Kenneth and Todd Donovan (2004). The Elements of Social Scientific Thinking. th edition
(Belmont CA. Thomson-Wadsworth), chapter 3.
The authors present strategies for narrowing the focus of a research question, and discuss hypotheses,
operationalizing concepts, and how to relate research questions to broader areas of concern

Kahane, Howard and Nancy Cavender (2005). Logic and Contem porary Rhetoric: The Use
o f Reason in Everyday Life. 10th edition (New York: Wadsworth Publishing), chapters 1
and 2.
These chapters discuss in detail common errors of reasoning or fallacies that prevent dear thinking
and the formulation of good research questions

King. G.. R. Keohane. and S. Verba (1994). Designing Social Inquiry (Princeton. Princeton
University Press), chapter 1
The author? discuss the value of formulating a research question so that it both addresses a real-world
issue and contributes to a scholarly literature in a given field They suggest a variety of ways that the
literature can be used as a source of developing a research question
H O W TO D O R E S E A R C H : AN O V E R V IE W

References
Dunleavy, P. (2003). Authoring a PhD: How to Plan. Pollins, B. M. (2007), 'Beyond Logical Positivism:
Draft. Write and Finish a Doctoral Thesis or Reframing King, Keohane, and Verba', in R.
Dissertation (Basingstoke: Palgrave Macmillan). N. Lebow and Mark Lichbach (eds), Theory
Fischer, D. H. (1970), Historian's Fallacies: Towards and Evidence in Comparative Politics and
International Relations (New York: Palgrave/
a Logic of Historical Thought (New York:
Harper). Macmillan), 87-106.

Geddes, B. (2003). Paradigms and Sand Castles (Ann Shively, W. P. (1989), The Craft o f Political Research,
Arbor, Ml: Michigan University Press). 2nd edition (Englewood Cliffs, NJ: Prentice-Hall,
Inc.).
Hampsher-Monk, I. and A. Hindmoor (2009),
'Rational Choice and Interpretive Evidence: White, L G. and R. P. Clark (1983), Political Analysis:
Caught between a Rock and a Hard Place?' Technique and Practice (Monterey, CA: Brooks/
Political Studies 58(1): 47-65. Cole Publishing Co.).

King, G., R. Keohane, and S. Verba (1994), Designing

Social Inquiry (Princeton, NJ: Princeton:
University Press).

Endnote
1. The internet is a fast and easy way to access information. It is a source of good information, but it is also a
source of half-truths and complete falsehoods. Consequently, you will have to figure out what is a
legitimate source and what is not. In general, authoritative sources are ones that are known to verify their
facts and information to a reasonable extent before publication. Reputable academic journals, for
example, normally only publish articles that have been at least twice blind-reviewed by experts in the
field who agree that the information is essentially correct and that the material is usefully contributing to
a body of knowledge. Major news sources, which are liable under libel legislation and have an interest in
being trusted by their readership, also attempt to meet a high standard of journalism.
There are, however, millions of publications and websites which are geared towards presenting
opinions and rumours rather than research and certifiable facts. These include blogs, 'quasi' news, and
gossip sites. Essentially, the key comes down to verifiable information. Can the source verify the facts
given-or can the author prove their case? In the case of blogs and opinion-based websites or user
content-generated sites, probably not. Examples of academic or authoritative sources online include
articles from major sources (BBC. CNN, AFP), government websites, and reputable NGOs (e.g. Interna­
tional Committee of the Red Cross. Amnesty International, National Union of Journalists). Information
sources such as Wikipedia, About.com, or lnfoplease.com are not academic sources. Wikipedia and other
sites like it do not provide sufficient monitoring of what information goes up, and frequently the
information is wrong and misleading.
 » ■ Answering Research
Questions: Requirements,
Components, and
Construction

Chapter Summary
I he previous chapter was devoted to a disc uss.un of how tu t.nd and formulate j re
starch question fhi-.chapter along with Chapter / Iia uses or, r ,,« u > je .e n .p j r an
Swer to It W eb e g m o u rd .se uss.un .n th.s r hapter b, jddress:ng tf.r tnlii.A .ng ,v.ues

• W hat are the requirem ents and com ponents of an answer to a resrart h
question ’

• W hat is a theory and what is its role m v x iai w ien cr researi. h

• W hat is a theoretical framework '

• W hat is a hypo thesis'

• W hat are the com ponents of a hypothesis >

• W hat types ol hypotheses are there and how do they guide different kinds of
re se arch '1

• W hy do you need and how do you formulate conceptual and o[>et.ition.il

d efinitions of your key term s’

• H o w do you go about answ ering norm ative questions’

Introduction
W e have d iscu ssed Step 1 of the research process: (he form ulation of a well «.rafted
resea rch q u estio n (C h ap ter 5). We tu rn . now. to Step 2 how to develop a hypothesis or
arg u m en t that answ ers it. We d iscu ss the basic co m p on en ts o f a hyp othesis or argum ent
th at answ ers a research qu estio n , what requ irem en ts it must m eet, and w here to find and
how to fo rm u la te one.
W hat does answ ering a research question involve? The various considerations and tasks
involved can best be understood in relation to three basic requirem ents. The first require
m ent is that the answ er be appropriate to the type o f question that is being asked. Different
types o f questions dem an d different types o f answers ( see discussion in C hapter 5 and. also.
B ox 5 .7 ). The secon d requirem ent o f an answer to a research question is that it makes a con
trib u tion to know ledge. Social scien ce research is expected to address a question whose
answ er will con tribu te to collectiv e knowledge in a particular field o f study; so. as you
H O W TO DO R E S E A R C H : AN O V E R V IE W

develop an answ er to your research qu estio n, you m ust ask yourself: W h y sh ould we care
about this answ er or argum ent? H ow does it co n trib u te to the d ev elopm ent o f know ledge
about politics? In oth er words, your answ er m u st m atter.
You con tribu te to the developm ent o f k now ledge in o u r field by relatin g your qu estio n
and answ er to ex istin g theory. The reason for th is is th at in ord er to achieve p rogress in a
field o f study, research m ust be cum ulative; and th eory facilitates cum u lativ e research . It
links one study with another, and helps to ensure that th e w ork o f research ers enters in to
d ialogue with and builds on the w ork o f o th ers. As we d iscu ssed in C h ap ter 5, research ers
develop b oth a qu estion about a top ic, and an answ er to it, th rou gh a critical analysis o f
existing th eory and research relevant to that top ic. All research qu estio n s and answ ers are
in form ed by a set o f ex p ectation s derived from previous research and th eories. R ecall that
th e fu n ction o f your ‘literature review ’ is to articu late the co n trib u tio n you in tend to m ake
by in d icatin g what gap or need in the existin g literature on th e top ic your answ er or arg u ­
m ent will fill. W hat con tribu tion will the research you in tend to pursue m ak e to o u r k n o w l­
edge o f som eth ing? How, specifically, does it con tribu te em pirically, conceptually, or
th eoretically to our know ledge o f that topic? D ifferent types o f research provide different
contributions to knowledge.
In order to contribu te to know ledge in a field o f study, an answ er to a research qu estion
m ust m eet a third requirem ent: it must be clearly and fully specified. It m ust be specific with
regard to (1 ) the factors or elem ents you th in k m ust be taken into con sid eration in ord er to
answer your question; and (2) how you th in k these factors or elem ents are related to each
other. Together, these factors and th eir relations constitute a ‘ hypothesis': a reasoned, clearly
specified hunch or expectation with which you begin your research and w hich helps to guide
and focus your research. An answer or argum ent in response to a research qu estion consists
o f a hypothesis and an investigation o f it. D eveloping a hypothesis e ncourages you to be very
precise about how your answer relates to those that oth ers have offered to your qu estion.
The term ‘hypothesis’ is often treated as applicable only to quantitative research and to a
specific prediction about the nature and direction o f the relationship betw een two ‘variables’
(we will be discussing this term presently). We use the term ‘hypothesis’ in a far m ore ex p an­
sive way. W hat we mean by ‘hypothesis’ is ‘a hunch, assum ption, suspicion, assertion, or idea
about a phenom enon, relationship, or situation’, with w hich research begins and which
becom es the basis o f inquiry (K um ar 2005: 74). H ypotheses can either be tested with ev i­
dence (confirm atory research), or operate as a guide to a process o f discovery (exploratory
research). C onfirm atory research begins with a hypothesis and uses observations to test it.
We begin with a statem ent, on the basis o f a theory, o f what we would expect to find, and then
see whether what we expect is fulfilled. Exploratory research begins with a question and p er­
haps a basic proposition, probes its plausibility against various types o f data, and eventually
generates a hypothesis as a conclusion rather than as a p relim inary to cond u cting the research
itself (Schm ittcr 2008). In this type o f research, we might develop an answer through (1) a
prelim inary hunch; (2) an investigation; and then (3) a m ore concrete hypothesis. W hether
you are addressing a descriptive, explanatory, predictive, prescriptive, or norm ative question,
thinking in terms ol lorm ulating a hypothesis can help you to clarify your argum ent and the
kinds of evidence that will provide a m eaningful dem onstration o f your answer.
We see a hypothesis as a basic infrastructural elem ent whose fun ction , in all research, is
to help to make explicit the argum ent that is being developed. For all types o f research, we
 A N S W E R I N G R E S EA R C H Q U ES T I O N S

th in k it is useful to form ulate a working h y p o th e s is '-a n operational hunch about what vou
exp ect to find. Initially, what argum ent motivates the research? W hat findings might be
expected ? By articulating in advance the contours and logic o f the investigation, a hypothe
sis helps to guide research. As we shall see. different types o f research are guided by different
types o f hypotheses.
We pointed out in C hapter 2 that, irrespective o f w hether researchers see themselves as
positivists o r in terpretivists. practical investigation o f research questions often leads them to
un d ertake sim ilar m ethod ological tasks and research practices A s a number ol scholars
have argued, these com m on practices are founded in the hyp othetic» deductive method
Researchers either assess w hether the inform ation they have gathered tits with the m lerpre
tation they have posited, or they ‘consider the fit ol com peting interpretations with the facts
they have gathered ’; but. in either case, as Brian Pollins points out. they are practicing the
hyp othetico-d ed u ctiv e m ethod ’ (2007: 100).
We want to draw a d istinction between the process ol research and its/>rr.\r»ifijfii>M lh e
process o f research is often circuitous. M ost researchers engage in a dialogue between ideas
and evid en ce (R agin 2 0 0 0 ), m oving back and forth between theory and evidence between
th eorizin g som eth in g that is the focus o f their research, and m ucking about in the dust ol
detail (learn in g m ore about the specific lacts ol the case or question or issue, or the o b serv a­
tions o f oth er scholars that we treat as facts) As W Philips Shively observes, one of the bet
ter-kept secrets’ in our field, is that good researchers usually do not "fra m e hypotheses” in
any form al sense before they start to work, though they may have som e operational hunches
about what they expect to fin d .. . . ’Ihey play with data, im m erse them selves in what other
people have w ritten, argue with colleagues, and th ink’ (Shively 1989: 25). As this chapter
explains, the research process can be conceptualized as a series o f steps. However, in p rac­
tice, the p rocess o f research does not unfold in the sort of linear fashion that these suggest.
W e re-th in k our views as a result o f learning from the research process itself in ways that can
feed back to our previous choices and lead us to revise them.
However, we agree with Shively that starting with 'clear and obvious procedures’ that are
‘m o re m eth od ical and easier to apply for the begin ning researcher' is a good way to learn
how to d o research (Shively 1989: 2 5 -6 ). As students gain m ore experience, they will d ou b t­
less b eco m e m ore creative and develop th eir own process o f work.
Political in quiry encom passes a variety o f types and processes and possible starting
p oin ts. But whatever th e process, we suggest that the presentation of research makes clear the
logic b ehin d your reasoning and the assum ptions upon which you arc relying. All types of
research should be arranged carefully, systematically, clearly, and logically. W hat we read
should be a coheren t narrative w hose story line moves through research question, relevant
literature, hyp otheses, procedures and m ethods, findings, conclusions, im plications.
Th e sectio n s that follow will elaborate on all these points, using the th ree requirem ents o f
an answ er to a research question we’ve just discussed. We discuss the com ponents of, and
answ er to, a research qu estio n; approaches to developing hypotheses and the elem ents that
co n stitu te th em (variables, and th e relationships that can b e established am ong th em ). We
also co n sid er the ways you m ust specify your argum ent or answer: the need to use term s that
m ean w hat you w ant th em to m ean, and that can be un derstood by oth ers in precisely the
way you w ant th em to b e un derstood; the necessity o f providing a conceptualization (or
conceptual definition) o f your b asic term s or variables, as well as an even m ore specific
H O W TO DO R E S E A R C H : AN O V E R V IE W

b o x 6 .1 Outline of Research Components, Revisited

1. The question1
A. What do you want to know? What is the central question/problem fissue/puzzle?
B. Why do you think it is worth doing/knowing? What is the rationale for pursuing research on this
question?

2. T he literature1
A. Who else has asked and answered your question? What (range of) answers to this question are
found in the existing literature relating to it?
B. What are the positive elements in the current literature? What in the literature can you highlight,
underline, expand, extend, improve, build upon, continue?
C. What needs to be done? Delineate the crucial aspects of the problem requiring investigation.
What need to be done in order to provide a better answer to the question than currently exists?

3 Your answer*
A. Theoretical framework. What are the theoretical elements and guiding assumptions o f the study?
1. What factors or variables of the problem must be investigated in order to answer your
central question?
2 What is/are your hypothesis/es (how are these factors linked)?
a. What is the source of your hypothesis/es? What in theory would lead us to expect the
relation(s) you assert?
b. How would you demonstrate the relationships stated by the hypothesis/es?
3. What is the spatial/temporal domain o f the study? What is the rationale for defining this
domain for the study7
B Data and sources
1 What are the data relevant to demonstrating the relationships you hypothesize?
2 What sources are there for these data?

' Discussed in Chapter S

1 Discussed in the present chapter and in Chapter 7

d e fin itio n w hich identifies em pirical relerents lo r them (an o p eration al d efin ition). C hapter
/ w ill con tinu e o u r discussion ol how to answer a research question by focusing on research
design and on data and sources. Box 6.1, w hich reproduces the o u tlin e o f research c o m p o ­
nents we presented in C hapter 3, indicates the elements that we have already covered, and
those that we w ill he covering in both this and the next chapter.

Answers to research questions: general requirements

An answer to a research question ( I ) provides the type of answer appropriate to the question
you are asking, (2) contributes to the developm ent o f know ledge in the field o f p o litics by
relating itself to existing theory and research on a topic; and (3) clearly and fu lly specifies its
key elements and how they are related to each other.
What type of answer does your question require?
A research qu estion, as C hapter 5 em phasized, not only initiates the research process, but i
cru cial to every step along the way Ihe kind of question you ask determ ines the tvpc o
answ er you provide, the research you pursue (confirm atory, exploratory). and the m e ans fr
w hich you pursue it. You must provide the type of answer that your question requires II y«H
were asked in an exam to Describe what the key differences arc in ( hm ese and Iranian ,x»li
cies used to address population growth issues', a response designed to explain wh\ Iran an«.
C h in a have pursued different population growth p olitics would he inappropriate Ihe point
o f course, is that the type o f answer must he appropriate to the type of question lhat mean'
that, since, as we m aintain, an answer to a research question entails an investigation of 4
hypothesis, different types o f qu estions will require different kinds of hypotheses So. let «
review the different types o f questions researchers in our held ask and then consider whai
sort o f h ypothesis m ight he form ulated to order to answer them
In C hapter 5 we identified five different types of question that researchers in our held ask
descriptive, explanatory, predictive, prescriptive, and norm ative questions (We pointed oul
in C hapter 5 that not all departm ents will perm it you to use all these types o! question as the
basis o f a research or d issertation project, som e d epartm ents may require that your disserta
tion be based on an explanatory question; other departm ents may allow other types, or all
types, o f qu estio ns.)
Th ere is a ten d ency to treat hypotheses and hypothesis testing research as applicable only
to explanatory and predictive qu estions Explanatory question* are concerned with what it
causing or has caused an outcom e, or why som ething exists or has happened An eiplan a
tory hypothesis advances a guess as to the cause of. or reason for. som ething by identifying
what factors or cond ition s are conn ected to a known outcom e, they state, for instance, that
X is caused, or m ade possible, by Y. Predictive questions are concerned with what will in cu r
or be the likely ou tcom e o f som ething. A predictive hypothesis generalizes from an under
stand ing o f cu rren t or past events to predict the future outcom e o f current cond ition s or
trends. U sing evid ence on curren t events, accum ulated knowledge of general patterns, and
relevant th eories and th eoretical research, it starts with condition s lhat are thought to be
causal, and p red icts the resulting phenom ena A predictive hypothesu claim s that factor X
will cause event or state Y. that outcom e Y will occu r as a result o f a sel o f known fac tors or
co n d itio n s X , or that in cond ition s A BC , event or state Y will tend to occur
W h ile explanatory and predictive questions are types o f questions most usually associated
with hyp othesis-d riv en research, we m aintain that all research questions and answers are
in form ed by a set o f ex p ectations (hyp otheses) derived from previous research and theories
and th at, for all types o f re s e a rc h -in c lu d in g descriptive, prescriptive, and norm ative
resea rch — it is useful to form ulate a working hypothesis’ or operational hunch about what
you ex p ect to find.
L ets con sid er the sort o f hypothesis that m ight be form ulated in response to a descriptive
qu estio n. Descriptive questions are concerned with the characteristics o f what has happened,
is g oin g on , or exists; or o f how som eth ing behaves. The fun ction o f a hypothesis in descrip­
tive research is to select, based on an assessm ent o f what we th ink will prove m ost relevant,
ju st th ose factors from th e world o f facts and figures, people and events, that we think are
m ost useful for d irectin g and focusing o u r research. W e can n ot sim ply describe everything
relating to a p h en om en on . W e m ust always select and organize, out o f the universe o f
H O W TO DO R E S E A R C H : AN O V E R V IE W

p ossibilities, th ose aspects we th in k will prove im p ortant to explore and highligh t. W e can
start with a th eoretical c o n stru ct or th eoretically in form ed n o tio n , h u n ch , o r ex p ectatio n
that will serve as a descriptive hypothesis ; perhaps an ideal typ e— a set o f c h aracteristics that
we e xpect to find in som e relationship to each oth er in a given case. A d escriptive hyp othesis
m ight state, for in stance, that X has A, B, C characteristics and/or behaves in D, E, F ways.
D escription often involves an effort to set up definitions and classifications, or to m ake sense
o f alternative conceptualizations o f a topic. The basis o f this type o f descriptive analysis might
be a hypothesis that som ething can be meaningfully described , or usefully defined, seen, or
interpreted, as something else ; that a given concept is useful for describing or understanding
som ething; that we shall understand this thing b etter if we see it in this way, or if we interpret it
as being divided into these three types. We m ight call this sort o f d escriptive hypothesis an
interpretive hypothesis. W hen we state, for instance, that all X belong to class Y; or that X can be
interpreted as Y, we are hypothesizing that we can m ake good sense o f X if we classify it, or
interpret it as representing or ‘m eaning’, Y. D oes ‘dem ocracy’ or ‘political participation’ in C hina
mean the sam e thing as these term s do in the United States or Peru? C an we in terpret (m ean in g­
fully describe) older d em ocracies today as facing a crisis o f political p articipation? The ‘test’ o f
this sort o f hypothesis is showing the usefulness o f describing or interpreting X as Y.
H ypotheses can also be form ulated as a basis for addressing prescriptive and norm ativ e
questions. Prescriptive questions ask what we should do to bring about, or prevent, som e o u t­
com e; what course o f action we should follow to achieve a particu lar ob jective or goal. A
prescriptive question m ight ask, for instance, ‘How can the “b rain drain” be reversed in
South A frica?’ A prescriptive hypothesis states that to achieve desired end X, we should do Y.
To investigate this kind o f hypothesis, the researcher will inventory available options, weigh
the pros and cons o f each for achieving the desired outcom e, and, based on this analysis,
advance an argum ent for a policy, using existing th eoretical and em pirical research to verify
the argum ent’s factual and theoretical assum ptions. Normative questions deal with qu estions
o f what is right and wrong, desirable or undesirable, just or unjust in society. A norm ative
hypothesis is an argum ent or suggestive proposition about what ought to be that is advanced
for further discussion, analysis, and investigation. A norm ative hypothesis might state that
the best type o f X is Y; that the basis for deciding or ju stifyin g X ought to be Y, the legitim acy
o f X ought to be based on Y, or that X is just when cond ition s A, B, C are present. Investiga­
tion o f these hypotheses would entail elaborating an argum ent about why this should be the
case and establishing the plausibility and coheren ce o f the argum ent. The ‘test’ o f a n o rm a­
tive hypothesis is the rigour o f logic and internal consisten cy o f the argum ent.
In sum , it is useful to form ulate a ‘w orking hypothesis’ for all types o f research, but the
hypotheses that are formulated must be appropriate to the type o f question being asked. Box
6.2 shows the different kinds o f hypotheses that different types o f questions will require.
Som e examples o f different types o f research questions and the hypotheses that m ight be
ottered in answer to them are shown in Box 6.3. Note that a hypothesis is stated affirmatively,
i.e. not as a question.

Answers that contribute to the development of theory

We have said that an answer to a research question should contribu te to the developm ent o f
knowledge in the field of Politics by relating itself to existing theory and research on a topic.
W hat is a theory and why do we need it (what does it do for us)?
Type of Qufrition ► Form of Hypoth«™

Nofmati*«*

Type of Question ► Research Qu«tion

Prrst rijnive

Normative
136 H O W TO DO R E S E A R C H : A N O V E R V IE W

The w orld is com plex. A th eory is an attem pt to m ake sen se o f th e w orld by in d icatin g that
som e factors are m o re im portant th an o th ers and specify ing relatio ns am o n g th em . A th eo ry
is always a skeletal version o f reality: it enco m passes on ly a p o rtio n o f th e real world, deals
with only a few variables, and m akes sim plifyin g assu m ptions to keep v ariables and cases
m anageable. A good th eory is p arsim oniou s: it attem pts to explain as m u ch as possible w ith
as little as possible (parsim ony). It is also generalizable: it can be applied to a v ariety o f c o n ­
texts or settings.
The nature and fu n ction o f th eory in social research can be brou ght m o re clearly into
focus by contrasting th eory-d riv en research w ith an id eal-typical ‘n o n -th eo re tica l’ research
enterprise. C onsider, for exam ple, a study o f the causes o f W orld W ar I. The n o n -th e o re tic a l’
researcher, u n concerned with either p arsim ony or generalizability, m ight c h o o se to focu s on
any and all factors historians have recorded as present at the tim e o f the ou tb reak o f W orld
W ar I. Here, for exam ple, are a dozen o f the possibly hundreds o f factors relevant to ex p lain ­
ing the outbreak o f the war (note how som e o f these factors are them selves ‘th eo retical’, i.e.
the product o f prior th eorizing): (1) the in ternational b alance o f pow er and the chan ge in the
configuration o f the in ternational system (a rigidified b alan ce-o f-p ow er system , w hich
heightened com petition and ten sion , led to an arm s race and a fear o f losing a llies); (2) B r it­
ish heg em on ic decline and A n glo-G erm an rivalry; (3) G erm an nationalism and oth er
attributes o f the G erm an state, w hich gave it a propensity for aggression; (4 ) A ustria’s stru g ­
gle to survive and its effort to erad icate the threat o f Serb ian nationalism w ithin its em pire,
and in the Balkans generally; (5) Russian geopolitical im peratives (its quest to gain con trol
o f the D ardanelles for access to the M editerranean and thus to the w orlds co m m erce);
(6 ) British fear o f G erm an control over the chan nel ports; (7) France’s am b ition to reacquire
A lsace-L orraine; (8) G erm any’s internal class relations, w hich caused the state to pursue
foreign econ om ic policies that soured relations with Britain and Russia; (9) the b alance o f
power o f eth nic groups within A ustria-H ungary; (1 0 ) the im pact o f the rapid and total
m obilization o f the Russian army, due to the absence o f a plan for partial m o bilizatio n;
(1 1 ) the role and in terests o f m ilitary elem ents in the G erm an gov ernm ent and th eir in flu ­
ence on foreign policy; (12) the personality o f K aiser W ilh elm o f G erm any (th e propensity
he had for war at that tim e, or the qualities o f m ind and personality w hich allowed him to be
manipulated by others into declaring war).
A history o f W orld War I will focus on all o f these, plus many, m any oth ers from am ong
the universe o f factors possibly relevant to understanding why the war occurred. But a th e­
ory of the war focuses on just those few factors w hose relations to each oth er can be shown
to have been decisive in producing the particular outcom e (w ar). T h eory not only m inim izes
the num ber of factors to be treated as analytically relevant to an outcom e, but, sin ce it
requires that we show how the factors are related to one another, it ensures that the selection
ol factors is non-ran dom . Let’s say that, after d evoting som e thought and study to the causes
of World War I, you found your interest increasingly drawn to factors (4) and (1 2 ), a b o v e -
Russian geopolitical am bitions and Kaiser W ilh elm s personality. C ould you show that these
factors not only helped to produce the war, but were also related to each oth er in som e sys­
tem atic and logical way? Later in this chapter we will look at a w ell-know n th eory o f the
causes of World War I, discuss the variables on which the theorist chose to focus, the reason
he chose them, and his argument about how they were related to each oth er and to the o u t­
break of war.
« representing different H a g « in the developm ent of an idea I he h r « stage would be t
p r o p o r tio n a hu nch or g u e u that two or m ore variables are related W hen put forward k*
investigation— staled in a way that enables us to determ ine whether it is right or w r o n g - j
proposition b ecom es a hypothesis So an idea may start as a proposition, a statem ent thai
id entifies key factors that are thought to be related in som e way, it might then be deveiopec
so that th e factors and th eir relations to each other are m ore precisely defined I his mor«
specific statem ent is what we call a hypothesis. Though a hypothesis is ktated in a wav thai
enables us to evaluate, analyse, or investigate, it is (till a provisional and untested idea One«
it has w ithstood repeated tests and has been found to have considerable explanatory power
it b eco m e s a theory.
W e can distingu ish different types o f theory according to scope and level of generali/abil
ity ( grand versus m iddle-range th eory), analytic process (inductive versus deductive th e­
o ry ). and the nature o f the qu estions it addresses (em pirical versus norm ative)
The socio logist. R obert M erton , defined a d istinction between what he called 'grand th e ­
o r y ’ and ‘th e o rie s o f th e m id d le-ran ge (M erton 196H) C.rand theory is what he character
ized as all-inclusiv e system atic efforts to develop a unified theory that will explain all the
observ ed u n iform ities o f social behavior, social organization, and social change’ (1 9 6 8 W )
M erto n argued that grand and abstract theories o f society provided no basis for an em pirical
social scien ce. He called on th eorists to apply them selves to the developm ent of what he
called ’th eories o f th e m iddle range’: th eories that attempted to understand and explain a
lim ited asp ect o f social life, a m ore restricted dom ain or set o f social phenom ena These
exp lan ation s could th en be verified through em pirical research and then perhaps system a­
tized in to th eoretical system s o f broader scope and content.
T h eories also d iffer accord ing to the analytic process that links th eory and research, in
p articu lar w hether th eory guides research (deduction) or w hether th eory is an outcom e of
research (in d u ctio n ). W e discussed th ese different processes in C hapter 2. D eduction moves
from b road er gen eralization s and th eories to specific observations Wre start either with a
th eo ry that h as already been confirm ed o r with a logical argum ent, and then we draw out the
m ea n in g o r im p lications th is has for explaining som e particular case or phenom ena. So, in
ded uctive theory, a hyp othesis is deduced from curren t theory, which is then subjected to
em p irical scrutiny. In d u ctio n , you will recall, is a process o f reasoning from particular facts
to a gen eral conclu sion . W e begin with particular observations or cases and then develop
gen eralization s about th em . Inductive th eory is, th erefore, the o utcom e o f research.
A n o th er analytic p rocess that links th eory and research is w hat has b een called ‘grounded
th e o r y ’ (G la ser and Strauss 1967; C orb in and Strauss 1990). G rounded th eory is an in duc­
tive research strategy. T h e research er starts by collectin g data. C on cep ts and categories are
n o t applied to th e data, but em erge from them . H ypotheses are not tested or form ulated on
th e b asis o f data, but are developed throu gh the in teraction o f th eory and data. Th eory is
th erefo re prod uced th rou gh, and grounded in. data. W hat is m ost em phasized in grounded
th eo ry is th at it is ex p licitly an em ergen t process: th e aim is to discover th e th eory im plicit in
th e d ata, to allow th eo ry to e m erge from the data as opposed to forcing it into preconceived
fram ew orks.
138 H O W TO DO R E S E A R C H : AN O V E R V IE W

Finally, we can distinguish different types o f th eory accord ing to th e nature o f th e qu estions
they address (em pirical versus norm ative). W e have previously discussed (and raised som e
objections to) the distinction between em pirical theory and norm ative th eory (C hapters 1 and
3). Em pirical th eory is con cern ed with qu estions that can be answ ered with em pirical data
(data gathered through observations o f the w orld around us). N orm ativ e th e o ry is con cern ed
with qu estions a bout what is right and wrong, desirable or undesirable, ju st or un just in so ci­
ety. W e will have m ore to say about answ ering norm ative qu estions later o n in this chapter.
So, having reviewed different types o f theory, we can now sum up how the term s ‘prop osi­
tion, ‘hypothesis’, and ‘th eory’ differ from , and are related to, one another. A statem ent p ositing
that two or m ore variables are related is a proposition: a provisional idea that m erits evalua­
tion. In order for us to evaluate its worth, its constituent term s need to be d efined v ery sp ecifi­
cally. O nce you have d one this, you have a hypothesis: a tentative answer or argum ent you wish
to develop (investigate, dem onstrate) in response to your research question. A th eory iden ti­
fies a small num ber o f variables that must be taken into consideration in addressing the qu es­
tion, and how they are related both to each oth er and to the outcom e that is b eing addressed.

Where do hypotheses come from?

Your answer (a hypothesis and investigation o f it) m ust be situated in relation to existing
theory and previous research relevant to your question. A researcher develops a hypothesis
through a critical analysis o f the strengths and weaknesses o f b oth. This provides a way to
identify what requires further investigation or re-thinking: what would provide a better
understanding o f the matter, or m ake visible som e dim ension that brings added strength to
one or another o f the p ositions in a debate; what would produce a fruitful synthesis o f different
perspectives, or reveal erroneous assum ptions or logic in curren t understandings. As we d is­
cussed in Chapter 5, the analysis you undertake o f existing theory and previous research rel­
evant to your topic is presented in what is conventionally referred to as the ‘literature review ’.
Any issue relating to the political world is likely to be the subject o f conflictin g views about
how to define it; where, when, and why it originated; how it developed or evolved, what its
im pact is, what its likely future trajectory will be, and what d ecision-m akers or specialists can
or should do with respect to it. In the process o f sorting through the various opinions and
judgem ents about these issues, you gain an understanding o f the different positions, and the
current ‘state o f play’ regarding a given topic. W ho else has asked and answered your question?
W hat puzzles or problems have they left unsolved? Are there im portant issues or dim ensions
of a phenom enon that they have not addressed? Are there claim s for which there is little sys­
tem atic evidence; or relevant evidence or alternative data sources that have been insufficiently
considered? Are there logical inconsistencies or errors in how variables or a relationship
between two variables are specified, so as to cast doubt on the merit o f the conclusions? Are
concepts defined wrongly, ambiguously, too narrowly or too broadly, or in a way that obscures
crucial dim ensions o f the phenom enon investigated? D o argum ents and theories tend to
focus on examples or cases drawn from only one part o f the world? D o they fail to address or
control for confounding factors or alternative explanation? Is the tim e fram e too short, or
artificial; or the spatial dom ain too limited or artificial? W hat is missing, irrelevant, defective,
unwarranted, or ambiguous? W hat should be further highlighted, underlined, expanded,
extended, improved, built upon, or continued (e.g. through application to other cases)?
 A N S W ER IN G R E S EA R C H Q U IS T IO N S

G radually your own point o f view em erges. You start to .drntify good reasons tor why you
th in k o ne p osition is better, m ore persuasive, m ore accurate or com prehensive than another
You m ight start to develop a list o f reasons for why you think one thing »s true and another
th ing false. Kventually you develop clear, focused, and logically sound reasons lor thinking
what you think. This process is retold in your literature review. Your literature review sets the
stage for your own argum ent or answer by identifying, and developing an argument about,
w eaknesses in the literature relevant to your q uestion that need to be addressed or strengths
that have been insufficiently exploited.

Illustration: how an analysis of existing studies provides the basis for a hypothesis
Ih e process o f reading and analysing literatures related to your research question ».an pro
vide the basis for a hypothesis that escapes the weaknesses o f existing s tu d ie s and builds
upon th eir strengths. Let's illustrate how a researcher s literature review' leads to the devel
opm ent o f a hypothesis about an im portant area o f inquiry in FIR
In States, Scarcity, an d Civil Strife in the Developing World (2 00 6 ). C olin Kahl develops an
argum en t about the relationship between environm ental pressures and civil conflict through
a critica l review o f relevant studies of the issue.
Kahl begin s by ob serv in g that civil strile in the developing world represents perhaps the
g reatest in ternational security challenge o f the early twenty first century', and that a grow
in g n u m b er o f scholars and practitioners' are focusing on the role in these conflicts of rapid
popu lation grow th, env iron m en tal degradation, and com petition over natural resources
(2 0 0 6 : 1 - 2 ) . In ord er to engage with the literature on resource related conflicts. Kahl must
first sort through and organize the various argum ents and ludgemcnts that it contains. It is
this inventory and organization o f ex istin g knowledge on a topic that enables a researcher to
b egin the p rocess o f d eveloping and m aking a case for his or her own point of view. Kahl
conclu d es that the existin g literature on resource-related conflicts can be divided into two
broad perspectives: th ose lin king con flict to scarcity and grievance, and those arguing that
co n flict is driven by abund an ce and greed. He then selects for sustained analysis those
hyp otheses from w ithin each perspective that appear to be most robust and influential: two
hyp otheses lin king co n flict to resource scarcity (th e deprivation hypothesis' and the state
failure hyp othesis’), and tw o that lin k con flict to resource abundance (th e ‘honey pot h ypoth­
esis' and the ‘resource curse hypothesis’). After analysing the strengths and weaknesses of
each o f these, he is th en ready to state his own hypothesis.
Here is a b rie f su m m ation o f K ahl’s key points o f agreem ent and disagreem ent with these
h yp otheses and how his ow n hypothesis addresses their w eaknesses and builds on their
stren gths.
The ‘deprivation hyp othesis’ m aintains that population grow th, environm ental degrada­
tion , and p o o r d istrib u tion o f natural resources create deprivation am ong the poorest d e ­
m ents o f society and, in th is way. in crease th e chan ces o f civil strife. Population and
en v iron m en tal pressures con tribu te to falling wages, unem ploym ent, and landless ness. This
in crea ses poverty and inequality, leads to frustration and grievances, and increases the risks
o f collectiv e violence.
Kahl points to two key weaknesses with this hypothesis. First, it ignores collective action
problems. We discussed the problem of collective action in Chapter 4. The 'problem' to
140 H O W T O D O R E S E A R C H : AN O V E R V IE W

w hich this refers is that the ex isten ce o f co m m o n in terests am o n g individuals d oes not n e c ­
essarily produce an in centive to pursue con ce rte d p olitical actio n ; b ecause, rath er than
w orking to p rom ote a co m m o n in terest, individuals are ju st as likely to let o th ers do th e work
and to ‘free-rid e’. Kahl poin ts out that the ‘deprivation h yp othesis’ d oesn’t take th is prob lem
in to accou n t; it doesn’t provide an explanation o f when and why individuals are w illing and
able to m ake the sacrifices (in land, wages, and tim e) necessary to p articipate in organized
v iolen ce under cond ition s o f resource scarcity. The secon d w eakness K ahl finds w ith th is
hypothesis is that it assigns n o role to the state. T he state, by virtue o f its co n tro l over resources
w ithin a territorial d om ain over w hich it exercises its authority, m ust, Kahl reasons, play an
im portant role in resource-related conflict.
The ‘state failure hypothesis’ escapes th is w eakness by placin g th e state at th e cen tre o f the
story it tells about how, when, and why resource scarcity generates con flict. It argues that
population and environ m en tal pressures con fron t the state with in creased d em ands for
costly investm ents. This u n derm ines state capacity and legitim acy and open s up ‘p olitical
space’ for violence. Kahl argues that th is hypothesis doesn’t go far enough in elu cidatin g the
role o f the state in resource-related conflicts because it assum es that env iron m en tal and
d em ographic pressures lead to conflict only in cou n tries with weak state governance. But
state strength can also provide an avenue for resource-related con flict, Kahl argues, b ecause
a strong state can enable state elites to politically exploit d em ograph ic and env iron m en tal
pressures to eng ineer violence against social groups.
The ‘deprivation hypothesis’ and the ‘state failure hyp othesis’ b oth focus on resource scar­
city. O ther hypotheses focus on situations in which resource abundan ce leads to conflict.
The general argum ent advanced by this perspective is that resource abund an ce leads to c o n ­
flict by encouraging rebel groups to form and fight over valuable natural resources. Th is is
what the ‘honey pot hypothesis’ argues. The problem with this hypothesis, Kahl argues, is
that the ‘greed-based logic’ on which it depends is chiefly applicable to n on-renew able m in ­
eral resources rather than to renewable ones. N on-renew able resou rces— oil, d iam onds,
c opper— are those m ost likely ‘to be im plicated in violent con flicts’ in situations in w hich
there are abundant resources, because these ‘tend to be m uch m ore valuable per unit o f v ol­
ume, geographically concentrated, and easily tradable than m ost renew able resources’. The
‘vast m ajority o f exam ples o f honey p ot-d riv en con flicts’ Kahl argues, ‘revolve around oil,
precious m etals, diam onds, and oth er valuable m inerals’ (2006: 18).
Ihe sam e weakness attaches to what is perhaps the best-know n resource hypothesis about
resource abundance and conflict: the ‘resource curse hypothesis’. A ccording to this hyp oth­
esis, state control o f abundant supplies o f valuable resources contribu tes to corrup t, au th o ri­
tarian governm ents that becom e prim e targets for rebellion. Like the ‘h oney pot hypothesis’,
resource curse accounts apply ‘much m ore to coun tries dependent on the export o f n o n ­
renewable resources than renewable resources’ (2006: 19). Moreover, these accoun ts, Kahl
argues, take insufficient account o f the ways in which social and political factors affect the
relationship between resource endow m ents and violence (2006: 21).
Io sum up these points: Kahl shows that the most im portant hypotheses con cern in g
resource-related conflict are weak in one or m ore o f three ways. First, they don’t provide an
understanding of the role of the state in resource-related conflicts. Hither they assign no role
to the state or. it they do, they tail either to take into account the possibility both o f state
weakness (failure) and state strength (exploitation ), or to provide a sufficient understanding

■ ' ! m M £4
 a n s w e r i n g r e s e a r c h q u e s t io n s

o f the variables that affect Mate capacity. Ih c second weakness with these hypotheses 1» that
they pay insufficient attention to collective action problem s third, thev arc applicable to
cou n tries with non renew able m ineral resources rather than those with chiefly renewable
resources.

Kahl then draws these different lines o f critique together to produce a hypothesis o f his
ow n. I he state, he argues, plays a cru cial role in resource related conflicts in developing
c ou n tries: resource pressures can lead to conflict becauvc they cither push a state toward»
failure or provide th e state with opportu nities for exploitation Scarce resources can lead to
state w eakness (failu re), which, in turn, leads to conflict, resource abundance can contrib
ute to state strength and exploitation , which, in turn, generates rebellion and conflict
W h eth er env iron m en tal pressures push a state towards failure or exploitation depends on
social and political factors which affect the relationship between resource endow m ents and
the state. Kahl conceptualizes these factors in term s o f 'groupness' — the ease with which
co u n tries divide in to eth n o-cu ltu ral, religious, or class factions, which helps to overcom e
collectiv e actio n problem s; and ‘institutional inclusivity or exclusivity', which r e la te s to the
in flu en ce wielded by groups or organizations over the state Kahl hypothesizes that the
p otential for con flict will in crease where there’s a high degree of groupness, and where
exclusive in stitutio ns sh o rt-circu it cooperation and leave state elites free to instigate v io ­
lence. C onversely, the potential for conflict decreases where there is a low degree of g ro u p
ness and w here in stitutional inclusivity facilitates societal cooperation in the face ol a
w eakened state.
Figure 6.1 show s th e m ain hypotheses in the existing literature, and the hypothesis that
Kahl produces as a result o f his analysis o f their strengths and weaknesses. It is useful to
recon stru ct your hypothesis, in the sam e way as the hypotheses in f igure 6 I , using an arrow
diagram to identify key th eoretical variables and the direction of the hypothesized relation
ship betw een them .
You will recall from C hapter 5 that we discussed the need to provide a rationale for your
research qu estion. You need to provide a rationale for your hypothesis, as well. You must
identify th e sou rce o f your h yp othesis—what in th eory or practice gives rise to It. Is it devel
oped deductively from a m ore basic theory? Is it based on assum ptions o f a general 'approach'
(ration al ch o ice, in stitutional. M arxist, etc.). or a blend o f aspects o f two or m ore approaches*
O r is your hyp othesis developed inductively from a body o f em pirical evidence? Are you
hyp othesizing that the lessons drawn from another dom ain o f experience can be usefully
em ployed to address the d om ain relating to your research question?
All hyp otheses are based on a set o f expectations derived from existing theory and previ
ou s research. W h eth er the question you are addressing is descriptive, explanatory, pred k
live, prescriptive, or norm ative, there will be som ething in th eory o r previous research that
provides th e basis for a hypothesis about the way the world works or should work, about
future d evelopm ents or policy proposals. The key com ponen ts o f som e th eory or explana­
to ry approach o r bod y o f research are what lead you to expect the r d atio n (s) your hypothesis
states. T h ese co m p o n en ts com p rise what we refer to as a ‘th eoretical fram ew ork’. This fram e­
w ork is like a story or a set o f assum ptions that co n n ects the dots represented by your key
factors o r e lem ents. W e said previously that th eory ensures that contribu tion s to knowledge
are cum ulative. But th eory also furth ers the research enterprise by providing patterns for the
in terp retation o f data; by supplying fram ew orks w ithin w hich concepts and variables
HOW TO DO RESEARCH: AN OVERVIEW

I The deprivation hypothesis

scarcity

2 The state failure hypothesis

Strains on
state capacity

3 The honey pot hypothesis

m - m - ,
valuable natural
resources

4 The resource curse hypothesis

Kohl's hypothesis

Figure 6.1 Hypotheses about how demographic and environmental pressures generate conflict
 A N S W E R I N G R E S E A R C H Q U ES T I O N S

a cqu ire substantive significance, and by allowing us ‘to interpret the larger meaning ol our
findings for ourselves and others' (H oover and D onovan 2004: 37).
H ypotheses can be investigated in two ways. A hypothesis can be tested with evidence
(con firm a tory research ), or it can operate as a guide to a process of discovery (exploratory
research ). B oth contribu te to the developm ent of theory Hypothesis testing uses logical or
em pirical evid ence to evaluate existing th eories; hypothesis generating research produces
findings that can be used in the developm ent of theory.
H ypothesis testing begins by stating, on the basis o f a theory, what we would expect to
find, and then sees w hether that expectation is fulfilled. H ypothesis testing or confirm atory
research is deductive in that it is driven by a particular hypothesis: the researcher has a spe
cific, focused statem ent in m ind and his/her o bjective is to prove or disprove that hypothesis.
H ypothesis-g eneratin g research begins with a question and perhaps a basic proposition,
exam in es a set o f cases, and com es up with a m ore specific set of propositions Hypothesis-
gen erating or exploratory research is inductive, in that the researcher observes a phenom e­
non in ord er to generate qu estions or hypotheses for subsequent research. Both kinds ol
research are ‘part o f the overall scientific m ethod’ (C iernng 2001: 23). Both types of research
must ultim ately specify the relationship between variables. Both require one to n u k e an
argum ent. B oth are part o f a single, evolving, dynam ic process of discovery and hypothesis
form ation. As we pointed out in C hapter 2. scientific inquiry typically involves a process of
con tin u ou s in teraction betw een theory and observation, in w hich the researcher moves
from ob serv ation to th eory (in d u ction ) and from theory back to observation (deduction).
Figure 6 .2 illustrates this process.

INDUCTION
sum here
i

T
DEDUCTION
su m here

Figure Induction and deduction

HOW TO DO RE SEA RCH: AN O VE RVIEW

At the begin ning o f th is section we stated that in ord er to co n trib u te to know ledge in a
field o f study, an answ er to a research qu estion m ust m eet th ree requ irem ents. T h e first
requirem ent is that the answ er b e appropriate to the type o f qu estio n that is b ein g asked. The
second requirem ent o f an answ er to a research qu estion is that it m akes a c o n trib u tio n to
know ledge in our field by relating your q u estio n and answ er to ex istin g research and theory.
W e have not yet discussed the third requirem ent o f an answ er to a research qu estio n: th at it
must be clearly and fully specified. We address this requirem ent in th e nex t section .

Specifying your argument or answer, the nuts and bolts

Hypotheses can be assessed on the basis o f a variety o f criteria: em pirical data, added value
(in creased understanding), plausibility in com p arison with oth er available hyp otheses, u se­
fulness, internal logic, elegance, parsim ony. But in order to assess a hyp othesis you m ust
clearly and fully specify its com p onen t factors and relations. You m ust not only specify its
essential elem en ts— the variables and th eir relationship to one an o th er; you m ust also m ake
sure you are using term s that m ean what you want th em to m ean, and are d efin ing th em so
that they can be understood by o th ers in precisely the way you want th em to be un derstood.
This requires that you, first, develop a conceptualization , or conceptual d efin itio n, o f the
term s or variables you use. Second, you must develop an even m ore specific, operational,
definition that identifies em pirical referents for them .

The components of a hypothesis

A hypothesis consists o f three elem ents: an independent variable, a dependent variable, and
a proposition (a statem ent about the relationship betw een the variables).

Variables

A variable is a concept or factor that can vary, change, or assum e different values or ch a ra c­
teristics. A factor that is not thought to vary in the context o f the research and, h en ce is not
capable o f contribu tin g to variation in the outcom e, is called a co n stan t’. A variable assum es
at least two different values or characteristics (e.g. high vs low, m ore vs less, present vs
absent). Ihe values o f a variable must be mutually exclusive: each observation must fit in one
and only one category. Take a look at the exam ples in Box 6.4.
In social science research, we deal m ostly with two types o f variables. An in d ep en d en t
variable is a factor thought to influence, affect, or cause variation in ano ther variable. It
always com es before that oth er factor (th e ‘dependent variable ) in tim e and space. A d ep en d ­
ent v ariable is thought to depend upon or be caused by v ariation in an independent variable.
I he relation between these two variables is often affected by an in terv e n in g variable. An
in tervening variable that affects the relationship betw een the independent and dependent
variables by producing an interaction ellect acts as a ‘m oderator’ variable. O ne that tran s­
mits the efiects of the independent variable to the dependent variable is called a ‘m ed iating’
variable. W ithout these mediating variables to act as a conduit, the independent variable
would not allect the dependent variable. Recall our discussion, in C hapter 2, on causal
 145

bo x 6.4 Variables and Values

form r,t j »«-rrifti<-flt

V* 10 etonorim and polit* ol i hartes

Organization y/e

Economic development

Gender

b o x 6 .5 Variables

Variable: a factor or characteristic that can assume different values or

characteristics
A dependent variable IS the variable in w hir h you are prim arily interested and wtnc h you assume
IS d epend ent up o n a set ond variable Our dei>endent variable n some politic a! outi i >me (events cjr
process) we are e nd e avou ring to explain
An independent variable is the variable thought to dire< tiy or indue« tiy i ause <>< mfiuerv e the
depend ent variable W ithin this particular relationship it is independent because it drxn not depend on
the other v aria b le /varia ble s Our independent v aria b le s) are fa c to rs' that bring about the outcom e we
Wish to explain-the factor or set of factors are determ inative of this pofflical outcom e The value of the
dependent v ariab le is influenced (or depends) on the value of the independent variable/variables
An Intervening variable influences the relation betw een two others and thus produces an interact**!
effect ('moderator' variable) or transm its the effects of the IV to the DV (a mediating vanabte)

m e c h a n i s m s .iiul tin 1 n o tio n th.it th e re .ire 'social nu\ h a i m i m th at p nx.hu e mh 1.1 I ouU o m es
In ( ih a p te r 1. w e a d d re s s e d th is issu e a gain in th e c o n te x t <it a disc u ssion c o n c e r n in g how to
p ro v id e e x p la n a tio n s of m a c r o s o cia l p h e n o m e n a w ith m u r o lo u n d a tio n s R em em ber
la m e s C o l e m a n ’s c o n te n tio n th at m a i.ro p h e n o m e n a h ave to he e x p la in e d hv th e in te ra c tio n
o f m i c r o a n d m a c r o levels ( C o l e m a n s h a th tu h ')' ( >ne of th e fu n ctio n s th at m e d ia tin g vari
ab le s m ig h t fulfil is to s h o w h ow m i c r o an d m a c r o p h e n o m e n a a re lin k ed to e a c h o th e r Box
6 .5 s u m m a r iz e s th e ro le of i n d e p e n d e n t, d e p e n d e n t, an d in te rv e n in g va ria b le s
We can illustrate variables, their values, and how they provide the nuts and bolts of an
argum ent with referen ce to the argum ent developed hv C ohn kahl that wc |ust discussed.
Ih c qu estio n Kahl is addressing is: W hat effects do environm ental pressures have on conflict
HOW TO DO RE SEA RCH: AN OVERVIEW

w ithin developing c oun tries. His d epen dent variable (w hat he wants to ex p lain) is th e lik e li­
hood o f con flict. This v ariable can assum e tw o values: increased likelih oo d o r decreased lik e­
lihood. His ind epend ent variable (what he th inks effects th is ou tco m e) is d em ograp h ic and
environm ental stress. These pressures can vary b etw een two values: high and low. He id en ti­
fies an im portant in tervenin g variable that enables th e in d ependen t variable to affect the
d ependent variable. This is the state: a m ediatin g variable that lin ks th e effects o f en v iro n ­
m ental pressures to the likelihood o f conflict. This variable has tw o values: it can be strong
and exploitive, or weak and failing. W h ich o f these two values it assu m es d epen ds on oth er
intervening variables having to do with the social and p olitical in stitu tio ns o f a given c o u n ­
try: ‘groupness’ (values: high/low) and in stitutional inclusivity (high/low). H e argues that
these variables affect w hether env iron m en tal pressures push a state tow ards failure or
exploitation.

Relationships
All types o f research qu estions are con cern ed with co n n ectio n s or relations betw een two or
m ore variables. A descriptive question m ight ask what relationship exists betw een gender
and voting preference, or betw een educational achievem en t and p olitical p articip ation ; an
explanatory question would ask why that relationship exists. A predictive qu estion would
ask how that relationship m ight be expected to change in the future; a prescriptive qu estion
m ight be concerned with what steps m ight be taken to stren gthen a relationship. W h ile we
don’t think o f norm ative political th eory as requirin g a depen dent and in dependen t v ariable,
norm ative questions nonetheless are concerned with co n n ectio n s and relations, and with
w hether they are just or desirable.
There are two types o f relationships in p olitical research: two ways that variables can be
c onn ected or related to each other. In practice, these two types tend to converge.
The first relationship is one o f asso ciatio n . In this relationship, a variable, e.g. ‘un em p loy­
m ent’, is in som e way associated with, related to, or linked w ith an o th er variable, e.g. ‘in fla­
tion’. This term is roughly synonym ous with co rre la tio n (two th ings vary tog ether in a
linear fashion) and with co -v arian ce (th e alleged cause varies with the supposed effect). All
these term s refer to a relation betw een variables such that changes in one variable occur
together with changes in the other. A relationship o f association can be positive or negative.
Two variables are positively related when they change in the sam e d irection : w hen the o cc u r ­
rence o f high values on one variable are associated with the o ccu rre n ce o f high values on the
other, and low values on one are associated with low values on the other. For in stance, a
positive correlation between inflation and unem ploym ent m eans that w hen inflation is
high, unem ploym ent also tends to be high; and when inflation is low, unem ploym ent also
tends to be low. W hen two variables are negatively related, the values o f the variables change
in opposite directions, so that when one variable (in flation ) decreases, the oth er (em p loy­
m ent) increases.
Ihe second type of relationship between two variables is causality. W ith this type o f rela­
tionship, changes in one variable bring about changes in another. To establish that a causal
relationship exists between two variables you must show that four cond ition s have been met.
Ihe first condition is that the hypothesized cause or independent variable (IV ) is tem porally
prior to the effect, i.e. the dependent variable ( I)V ). In oth er words, the IV must precede the
DV in tim e. Th e second condition is that the two variables are correlated or co-vary. O
course, correlation does not establish that two variables are causally related. Ihat is why w,
need the third cond ition : that a causal m echanism or process links the two variables \V<
discussed causal m echanism s in C hapter 2. and the desirability, more generally, of showing
th e m icro-fou nd ations that conn ect variables —the intentions and choices of individuals
or o f social and political units that have acquired the capacity to act collectively You have tc
be able to tell a plausible story that you think probably connect» the independent v ariables!
with the ou tcom e that you are trying to explain, often including an in tervening variable thai
gets us from the hypothesized cause (IV ) to the effect ( D V ) Ihe fourth condition is that the
correlation betw een the IV' and the DV is not spurious. This requires that vou rule out the
possibility o f a variable that is causally prior to both the IN' and the DV. so that the correla-
tion that appears to exist betw een the two variables does so only because a third (anteced ent)
variable is affecting both.
A well know n and puzzling finding about I ’S C ongressional electoral cam paigns «.an
illustrate how a third variable can render a seem ing correlation spurious In a study of the
effects o f cam paign spending in congressional elections, C ary lacobson (1978) found that
in creased spend ing by in cum b en ts correlates with in creased odds o f failure So. it might be
assu m ed from this finding that spending m ore by incum bents som ehow causn electoral fail
ure. However, Jacobson found that there was a third variable that was antecedent to both
spend ing and electoral failure: the nature o f the electoral battle, i.e. w hether it was expec ted
to be tough or easy. In cum b en ts who expect a tough fight spend m ore than incum bents who
expect to win easily. If spending m ore is evidence o f a tough fight, there is nothing puz/ling
at all about finding that increased spending appears to be correlated with electoral failure
The n o tion that som ehow spending m ore by incum bents i auses electoral failure is spurious;
an o th er variable, the tou ghness o f the fight, explains both high spending and electoral
failure.
In sum , w hen we assert that variation in independent variable X muses variation in
depend ent v ariable Y. we are m aking four assertions; (1) the change in X precedes the change
in Y; (2 ) X and Y are correlated; (3) a causal m echanism or process can be identified that
links th e variables; and (4) the correlation between X and Y is not spurious or a
coin cid en ce.
W e said that in p ractice the two types o f relationship we've d iscu ssed —association and
causality— tend to converge. To explain why they do. we need to explain that there arc two
different n o tion s o f causality that social scientists em ploy: d eterm inistic causation and p rob ­
a b ilistic causation. A deterministic causal relation states that if (x) then always/invariably
(y )’. However, we generally tend to see hum an behaviour as, not d eterm ined, but constrained;
consequ ently P IR research usually em ploys a probabilistic notion o f causality rather than a
d ete rm in istic one. A probabilistic causal relation states that if (x) then maybe/som etim es/
probably (y)'. P robabilistic' m eans that w hen the values that an IV takes on increase, this
usually results in the values o f the D V in creasing (or decreasing). This n otion o f cause focuses
th e e fforts o f researchers on finding factors that m ake an effect m ore likdy: for example, the
fin d ing that th e m ore educated a person is, the m ore likely he is to vote. The m ore robust we
m ake o u r c o r r e la t io n s -th e m ore we seek to m eet the condition s o f causality discussed,
abov e— th e m o re th e relation o f correlation betw een tw o variables converges with on e o f
prob abilistic causation. Though a robust correlation can n ot establish why X and Y co-vary.
H O W T O DO R E S E A R C H : A N O V E R V I E W

know ing that X is usually associated with Y can m ake a sign ificant c o n trib u tio n to our
und erstand ing o f p olitical outcom es.

Conceptualization and operationalization

In order for research to be cum ulative, the answ ers we offer to a research qu estio n m u st be
clearly and fully specified. This m eans, am o ng o th er things, that the term s we use m u st m ean
what we want them to m ean and be form ulated so that oth ers will u n derstand th em in p re­
cisely the way we want them to be understood. W e m ust provide b o th a con cep tu alization or
conceptual definition o f th em , as well as a d efin ition that id en tifies em pirical referen ts to
them : an operational definition. Specifying our term s in th is way m akes it p ossible for our
ideas to be assessed and for our research to enter into dialogue with and build on th e w ork
o f others.

Concepts and concept formation: what are you talking about?

A concept is a term applied to a collection or class o f th ings that are to be regarded alike,
b ecause they either share a com m on characteristic or feature (e.g., gender, type o f gov ern ­
m ent) or are instances o f som e general type o f b ehaviou r (war, particip ation ). A con cep t
provides a label for instances or characteristics that are com m on ly found together. W h en we
attach a label or general term (con cep t) to ob serv ation s o r events we can lin k separate o b ser­
vations or events together and form generalizations. W e observe people voting and we refer
to this as ‘participation. We observe that a legislator consisten tly takes certain position s on
political issues and we call him a ‘conservative’.
Translating know ledge into the m ore general catego ries represented by co n ce p ts e n a ­
bles know ledge to cum ulate. We use con cep ts as a sh orth an d to refer to m u ch o f th e su b ­
stantive con ten t and issue base o f political studies: elites, bureaucracy, legislatu res,
ju d iciaries, policy, eth nicity, plu ralism , alien atio n , hum an rights, gov ern m en t, fed eralism ,
public op in ion , election s, power, d ev elopm ent, d em ocracy, cu ltu re, legitim acy, ch arism a,
hegem ony, in stitu tio nalization, ex p loitation , authority, in terests, class, co rp o ratism , civil
society, racism , terrorism , egalitarianism , dependency, con sen su s, w elfare state, social
justice.
But, many co n ce p ts used in ou r field are anythin g but clear. Th ere is often a lack o f
c onsensu s about what co n ce p ts are im p ortan t, what th ey m ean, and how th ey sh ould be
d efined and m easured. W hat on e sch o lar m eans by ‘id eo lo g y ’ m ay be so m eth in g q u ite
d ifferent from what a n o th er person m eans. Th ere is also a lack o f co n sen su s ab out what
concept should be em ployed to d escrib e a given p h e n o m en o n or ou tco m e. R esearch ers
frequently use d ifferent co n ce p ts to un d erstand the sam e situ ation . W hat o n e research er
calls a ‘terrorist o rgan ization ’, an o th er m ight call a ‘social m o v e m en t’; an event that on e
research er calls a ‘rev olu tion’ an o th er m ight call a ‘cou p ’. Social sc ie n c e is ch aracterized
by a con tin u ou s con flict not on ly over how we define co n ce p ts, but w hich o n es we should
use to com preh end a given reality. C onsequen tly, we need to spell out with as m u ch p re ­
cision as possible what a concep t that we use (e.g. ‘id eo lo g y ’) m eans in th e c o n tex t o f our
research. Ihe point is not just to define th e term s we use, but to co n sid er very carefu lly
what term s we should use and how we should define th em .
 A N S W ER IN G R ES EA RCH QU ESTIO NS

Concept formation

I h e te rm \ o m e p t refers tu a m en ta l im a g e (c o n c e p tio n ) that su m m a riz e s a c o llec tio n of s<rm

m gly related o b s e r v a tio n s an d e x p e r ie n c e s ( nrwe/'fu.iii.Miu.H. „1 c o n ce p t form ation'. is the
p ro c e s s o l s e le ctin g th e term by w h ich so m e c o lle c tio n ..I th in g s should k - kn o w n An lo h n C.er
rin g p o in ts o u t, it c o n c e r n s th e m o st basic q u estio n s ot v K ia l s o e r u e l\ mu! itr< *,• fu iliny
abou t'' ( 2 0 0 1 : 3 3 ) M u ch th o u g h t h as be en d evo ted to th e q u estio n ot what d istin g u ish e s a
c o n c e p t fro m o n e that is less g o o d o r less usefu l |ohn ( ir r r m g id en tities rig h t essen tia l crite ria
lo r a g o o d c o n c e p t: la m ih a rity , rrs o n a n c e . |>arsimoin. c o h e r e n ce , d iffe re n tia tio n d ep th . theor
e tica l utility, an d field utility B o x f>.f> show s how ( le rrin g d efin es ea ch 1>t th ese >.riteria
A g o o d c o n c e p t m u st m e d ia te a m o n g th e s e c r ite r ia , usuallv th ro u g h a s eries of tra d e offs
F o r in s ta n c e , yo u d e c id e th a t a g o o d p la c e to sta rt th e p r o c e s s ot c o n c e p tu a liz a tio n is w ith a
te r m th a t m e e ts th e c r ite r io n o l fa m iliarity (e s ta b lis h e d u sa g e) You fin d , h o w ev e r, th at th e
te r m h a s little th eo retic a l utility ( u s e fu ln e s s for th e o r y ) Your n ex t step w ill be to find a term
th a t d o e s a b e tte r |ob o t m e d ia tin g b e tw e e n th e s e tw o c r ite r ia P e rh a p s yo u c o m e up w ith a
n e w te r m lo r s o m e th in g th a t o ffe r s ren'nurue (a rin g ) You m ay d is c o v e r, h o w ev e r, th at th is
c o m e s at to o h ig h a c o s t w ith r e s p e c t to fam iliarity. tlteo reth a l utility, o r o th e r c r ite r ia ¡ ‘a n t
m a n y ( e x p l a in i n g th e m o s t w ith th e le a s t) m u st b e b a la n c e d w ith J r p th (th e n u m b e r o l tea
lu r e s s h a r e d by th e o b s e r v a tio n s o r in s ta n c e s we w ant to reg a rd as a lik e 1 ( i'lic r c ii ,r (th e
internal c o n s is te n c y o f th e in s ta n c e s a n d a ttr ib u te s ) o fte n n eed s to In- c o n s id e r e d m r ela tio n
to d iffe r e n tia tio n (o f th e o b s e r v a tio n s a n d in s ta n c e s fro m sim ila r c o n c e p ts )
We c a n illu s tr a te th e p r o b le m s a n d tra d e o ils th a t r e s e a r c h e r s la c e 111 c o n c e p t f o r m a tio n
b y c o n s id e r in g h o w r e s e a r c h e r s h a v e s tr u g g le d to a r riv e at a s a tis fa cto ry d e fin itio n of a kev
term o f p o litic a l a n a ly s is po w er. B elo w , we p ro v id e a b r ie f su m m a rv o l th e d e b a te a b o u t how
to c o n c e p tu a li z e p o w e r, lh is is n o t in te n d e d as a lu ll disc u s s io n o l th e d e b a te , but onlv as an
il lu s tr a tiv e e x a m p le ot th e p r o b le m s r e s e a r c h e r s in our field lo n t i o n t w lu n a tte m p tin g to
d e f in e k ey c o n c e p ts

box 6 6 Criteria for Conceptual Goodness I

1. Familiarity How f.ri' ^'.1 - s toncepi ito a lay or at a; Jp 'tvi

2. Resonance .'son tCf m nng ,rev:,njtp!?

3 Pireimony

4. Coherence H<isv ii!y a .-log ai , '*Uted! a x •'■v.i'« *<V. at:- ji»-.7

5. Differentiation Ho%v rt’ Ifw - and the most umiUf

concept^7 How fv>u"‘. W -aMe W m r*e p ? >

6. Depth How many ac.om;\r'y’rg property are '^a^ed S t** .rsta"<•<". under defini­
tion?

7. Theoretical utility How useful is the concept within a wider fteW of inferences7

S. FtaM utiUty How useful 5 the concept withm a field of retaed instances and aCnbiAO?

Source fromGtrring 1999; 367.

HOW TO DO RE SEA RCH: AN O VE RVIEW

Conceptualizing 'power': the search for theoretical utility

R esearchers in our field generally agree that ‘pow er’ is a key co n ce p t in p olitical inquiry, and
that it is central to any un derstandin g o f politics. But w hat we m ean w hen we use th e term
‘pow er’ is anything but clear. Is ‘pow er’ a capability, an attribu te, or a relation? A cap acity to
do som eth in g or con trol som ebody, or an actio n? A property o f actors or o f relations? Is it
necessarily conflictual, in tentional, or effective, or all three? W h o possesses and ex ercises it:
individuals, groups, social in stitutions and organizations?
D ifficulties o f this sort arise, in part, because o f the p roblem o f d ifferentiatin g ‘pow er’
from sim ilar or related phenom ena, such as co ercio n , authority, in fluen ce, d om in ation ,
force, in d ucem en t, persuasion, m anipulation. But attem pts to achieve a high d egree o f dif­
ferentiation in the conceptualization o f power often entail a loss o f depth. The term ‘pow er’
must not only be differentiated from other, sim ilar concep ts; it m ust also captu re the m any
different ways that people can exercise pow er over e ach oth er (som e o f w hich c an n o t always
be observed ), and the very diverse sou rces o f power: m aterial resources like m oney, prop ­
erty, w ealth, natural resources, in form ation , m ilitary capabilities, good jo b s; but also in ta n ­
gible factors like charism a, status, prestige, reputation, respect, honour, character.
L et’s consid er these difficulties in relation to two ways in w hich research ers have attem pted
to define ‘pow er’: in term s o f relations betw een actors (pow er as control over actors, agendas,
and preferences); and the outcom e o f th ose relations (pow er as con trol over events and
outcom es).
A conceptualization o f power as control over actors defines ‘pow er’ as th e ability o f one
actor to im pose its will on others. This fits our intuitive no tion o f pow er (fam iliarity ); and it
captures several d im ensions o f power (depth) by including control o f actio ns, agendas, and
preferences. In this conceptualization o f ‘pow er’, to use R ob ert D ahl’s oft-q uoted form u la­
tion, power is the ability o f A to get B to do som eth ing he would not oth erw ise do (D ahl
1957). O ne way this can be achieved is through controlling agendas— stru ctu rin g or d eter­
m ining what issues are discussed, preventing ‘the public airing o f policy c o n flicts’ (B ach rach
and Baratz 1970: 8); keeping potential issues out o f politics, ‘w hether through the operation
of social forces and institutional practises or through individuals’ decisions’ (Lukes 1974:
24). The im portance o f controlling agendas is illustrated by the struggle o f cou n tries in the
South to gain som e control over the agenda relating to N o rth -S o u th issues; the agen da-set-
ting power o f media, NCiOs, or global social m ovem ents, like the anti-g lob alization m o ve­
ment. ‘C ontrol over actors’ can also be achieved through controlling preferences— through
in fluencing what other actors desire. The ability to gel what you want, not through the ‘hard
power’ o f m ilitary and econ om ic strength, but through attraction, is what Joseph Nye has
called ‘soft pow er’ (1990, 2002, 2004). C ontrolling preferences m eans the ability to generate
consensus, through the m anipulation o f public opinion or propaganda (d isin form ation),
through censorship (w ithholding in form ation ), or political correctn ess (th ough t co ntrol).
C ontrolling actors through control over actions, agendas, or preferences encom passes
much o f what we mean when we use the term ‘pow er’. However, this conceptualization
makes it difficult to distinguish between in teraction s in which one actor exercises power
over a nother and those involving exchange relations. How do we know w hether the o utcom e
of an interaction between two actors (e.g. a negotiation) occurred as a result o f the exercise
of power by one actor over another, or because both sides had som eth ing to gain from the
outcom e5, II the UK agrees to the dem and by the US to im pose taxes on offshore activities.
 a n s w e r i n g r e s e a r c h q u e s t io n s

does th is show that the US « e r a s e d power over the UK with respect to ,h ,s „su e. or that
«»m e sort ol exchange occurred in which both benefited? Can we interpret the outcom e
m o re generally, as d em onstrating som ething about current power relations m the interna
tional system , or som eth ing about the nature and content of exchange relations' A control
over actors' d efin ition o f power otten n u k es it difficult to distinguish one from the other
But there is another problem with this conception of power It is tautoK*g.cal vou are
defining power in term s of the outcom e l et s say that we think that some a, lion hv A, tor B
can be explained as a result o f the power that Actor A was able to exercise over Ac tor B We
explain that B did what A wanted it to do because A had the power But we can only know that
A had this power because B did do what A wanted it to do. We only know that A has power
over B by the result. Power' as a concept is only interesting if it can be u sed to explain events
or outcom es; and a conceptualization of power' that is tautological is not theoretually useful
C onsid er the second conception o f power: iontrol over events und ouhom es Bv focusing
not on relations betw een actors, but on the outcom e ol those relations, this conception allows
us to accoun t lor in teraction s between actors in which both actors benefit (exchange), though
they may gain d isproportionately In a conception ol power as control over the outcom es, we
can not only accoun t for shared pow’er or exchange, but also for situations in which relatively
weak cou n tries exercise power through sharing control over mutually consequential out
c om es. For exam ple, during the Cold War. the small, relatively powerless Middle I ast states
exercised power through their shared control over stability m the region Ihey c ould threaten
to act in destabilizing ways, to escalate the Arab Israeli conflict and cause a widescale c o n ­
frontation in the region, or to lose militarily. But though this conception ol power offers
im portant advantages over the 'control over actors' conception , it di*esn't escape the problem
o f tautology. Like the control over actors' conception , it defines power by the outcom e, so
there is no ob jective way ol d eterm inin g who has the power until we see how things turn out
A m ore general criticism o f both conception s is that, in general, most of what happens m
p olitics is not attribu table to the direct exercise o f power by actors, over each other or over
o utcom es. Power is usually exercised in a far less direc t w-ay. Ihis is what the notion of stru t
turul pow er h ighlights: aspects o f power which operate through indirect institutional, unin
tended, or im personally created effects Structural power operates through a network ol
(historically constitu ted ) relationships that benefit som e and disadvantage others (C.u/nni
1 993). Susan Strange (1 9 9 4 ) identified four key structures through w hich power is exercised
the securifv sfrui'furr. w hich provides protection to human beings, the protiui fion strut turr,
the prim ary m eans o f creatin g value and wealth in a society, the mi nee structure, which
d eterm in es w ho has access to m oney, how, and on what term s, and the knowledge strut ture.
w hich 'd eterm in es what knowledge is d iscovered, how it is stored, and who com m unicates it
by what m eans to w hom and on what term s’ (1994: 121). W hen wom en o f c hildbearing age
in p oor cou n tries work at factory jo b s that expose them to potentially dangerous chem icals,
it is a result o f the exercise o f power, not by em ployers, but o f production and financial stru c­
tures that give p oor w om en in these coun tries few oth er ways to survive. The US is able to
in d irectly in fluen ce oth er states' behaviour by controlling the m onetary structures within
w hich they m ust operate.
W e have touched on only a few o f the challenging and vastly complicaled issues and consid­
erations involved in the attempt to conceptualize 'power'. Hopefully, ihe example sufficed to alert
you to the sort o f ambiguity that attaches to term s that are com m only used in our field and the
consequent need to provide a carefully considered conceptual definition o f those that you use.
152 H O W T O DO R E S E A R C H : A N O V E R V I E W

Conceptualizing 'democracy1: mediating between coherence and differentiation

W e achieve b roader know ledge by gen eralizing; and we generalize by applying our c o n c e p ­
tu alizations to a wider range o f o bserv ation s or exp eriences. However, using existin g c o n ­
cepts to cover in stances that lie ‘outside th eir norm al range o f use (G errin g 1 9 9 9 :3 6 0 ), e ither
in application to new cases, or as a result o f changes over tim e within cases— captu red by
G iovann i Sa rto ri’s (1 9 7 0 , 1984) no tion s o f conceptual travelling (th e application o f con cep ts
to new c a ses)— can create problem s o f conceptual stretching (C o llie r and M ah on 1993:
8 4 5 )— the d istortion that o ccu rs w hen a concept does not fit the new case. W e can illustrate
the problem o f conceptual stretch ing’ and efforts to prevent it, by con sid erin g attem pts to
deal conceptually with the chan ging nature o f p olitical d em ocracy over th e past decades.
The great diversity o f new p ost-au th oritarian political regim es established in Latin A m e r­
ica, A frica, Asia, and the form er com m u n ist world in recent decades share im p ortant
attributes o f d em ocracy. But they also differ in significant ways b oth from each oth er and
from the d em ocracies in advanced industrial coun tries.
The term ‘d em o cracy ’ has been applied to th ose political system s that m eet a ‘procedural
m inim um ’ definition o f ‘d em ocracy ’: fully contested elections, full suffrage, the ab sence o f
m assive fraud, and effective guarantees o f civil liberties, including freedom o f speech, a ssem ­
bly, and association (C ollier and Levitsky 1997: 4 3 4 ). Those p olitical system s that have this
procedural m inim um , plus som e furth er differentiating attribu te, provided the basis for
defining the classical’ subtypes o f d em ocracy as, for instance, ‘parliam entary d em o cracy ’,
‘m ultiparty d em ocracy ’, and ‘federal d em ocracy ’. In order to capture the diverse form s o f
dem ocracy that have em erged in recent decades, researchers have defined new sub-types:
authoritarian d em ocracy ’, ‘n eopatrim on ial d em ocracy ’, ‘m ilitary-d om inated d em o cracy ’,
‘lim ited ’, ‘olig archical’, ‘controlled ’, ‘restrictive’, ‘electoral’, ‘illib eral’, ‘guarded’, ‘protected’, and
‘tutelary’ dem ocracy. The goal was to generalize our und erstanding o f d em ocracy to a wider
set o f cases. But because these ‘subtypes’ are cases that do not m eet the defin ition o f the root
term , that do not represent fu ll instances o f the term , ‘d em ocracy ’, it could be argued that
they really refer ‘to a different set o f cases’ than those that we call d em ocracies (C ollier and
Levitsky 1 9 9 7 :4 3 0 ,4 3 8 ).
Rather than referring to form s o f d em ocracy w hich are incom plete, i.e. which are m issing
attributes included in the procedural m inim um defin ition, as different types o f dem ocracy,
som e analysts consider it m ore useful to refer to political system s as exhibiting different
degrees o f d em ocracy (C ollier and Levitsky 1997: 440).
This approach owes m uch to R obert D ahl’s influential study, Polyarchy (1 9 7 1 ). In this
study, Dahl uses the term ‘d em ocracy’ to refer to a political ideal. The term ‘polyarchy’ refers
to actual political system s. Polyarchies are system s o f representative governm ent that ‘to a
relatively high degree’ are characterized by (1) equal chan ces o f all m em b ers o f the c o m ­
munity d eterm inin g the agenda for, as well as the outcom e o f processes o f political d e c i­
sion -m aking; and (2) the effective accoun tability o f the elected officials during the tim e
between elections (1971: 84). The notion that actual political system s can be characterized
in term s o f degrees or gradations o f dem ocracy has been widely accepted. Som e analysts
argue that d em ocracy is always a m atter o f degree’ (B ollen and Jackm an 1989: 612; em p h a­
sis added; see also Bollen 1980, 1993; and Bollen and Paxton 2 000; D ahl 1971, 1989;
C oppedge and Reinicke 1990).
 A N S W E R I N G R E S EA R C H Q U ES T IO N S

But o th ers argue that th is approach to conceptualizing dem ocracy is fundamentally

flawed. Political system s are systems'. G iovanni Sarton argues Ih e v are bounded w hole,
characterized by constitutive m echanism s and principle» that are either present (albeit
im perfectly ) o r absent (albeit im perfectly)' (1987: 184; em phasis added) Regimes cannot be
h a lf-d e m o cra tic’ (Alvarez et al. 19% : 21). Ihe d istinction between dem ocracy and non
d em ocracy should be treated, th erefore, as exhaustive and mutually exclusive categories. Ihe
q u estion o f w hether to conceptualize •democracy1 as a dichotom ous variable (dem ocracy
versus no n -d em o cra cy ) or in term s o f degrees continues to be much debated
As the discussion of power and dem ocracy shows, we must not |ust define our terms, but
th ink carefully about the im plications of how we use them and the (radc-otfs that must be
m ade am ong a variety of considerations that enter into the formulation of good as a concept.

Operational definitions: how will you know it when you see H7

Kven when there is a consensu s about the im portance of a particular concept and how it
should be defined, researchers often will disagree about whether, where, or to what degree it
is o ccu rrin g. For instance, since the 1970s, International Relations scholars have becom e
increasingly preoccupied with the growing interdependence am ong states Researchers gen
erally agree on what the term interdependence' m eans, but they do not always agree about
how significant it is, w hether it is increasing or decreasing, regional or universal, new or old
Even though we may agree on the m eaning of a term , unless we provide an operational' defi
nition o f it— define it in a way that will enable us to determ ine its presence or absence,
stren gth, and e x te n t— we may not know it when we see it So. in addition to providing a
concep tu al d efin ition o f a factor or variable, we need an operational definition, as well.
O p e ra tio n a liz a tio n m eans defining a variable, so that we know it when we see it
Suppose you ask: ‘Is Pakistan a failed state?' You adopt a definition of failed state' used by
o th er researchers: a failed state’ is a state experiencing high state vulnerability or risk of v io­
lence. You now have a conceptual definition of 'failed state’. But how do you know state vul­
n erability’ when you see it? You cann ot leave it to your subjective sense of what constitutes
vulnerability on the part o f a state. You need to decide what will enable you to judge the
degree o f vulnerability o f a state. You might decide that state vulnerability’ can be operation­
ally defined as a state in w hich there exists ( I ) hum anitarian em ergencies as a result o f mas
sive m ovem ent o f refugees or internally displaced persons; (2) sharp and/or severe econom ic
d ecline; (3 ) a progressive deterioration o f public services; (4) the in tervention o f other states
or external p olitical actors. You can then go and get evidence and data on each o f these indica­
tors o f state vulnerability’ in order to reach your conclusions about w hether or not Pakistan
is a failed state.
W e will not be content to know your conclusions: we need to know how you reached
th em . You m ust m ake explicit what set o f actions (operations) will lead another to achieve
th e sam e result when repeated. O th er researchers should be able to follow your procedures
and co m e up with the sam e findings. An operational definition provides travel instructions:
sets o f actio n s (op eratio n s) w hich will reveal to oth er researchers how you arrived at your
findings. It specifies what em pirical indicators or data will enable us to determ ine the pres­
en c e o r a b sence o f a v ariable, or m easure its extent. To operationalize the term speeding, we
154 H O W TO DO R E S E A R C H : A N O V E R V I E W

put a nu m b er on it: 70 mph. This b ecom es a proxy for speeding’. It also suggests th e d ata that
will enable us to d eterm ine w hen speeding’ has occu rred: a speed o m eter reading. Let’s say
you hypothesize that the higher som eone’s social class, th e m o re likely th ey are to b e c o n ­
servative. You m ight ch oo se to use different in com e levels to operatio nalize social class’: less
than $ 4 0 ,0 0 0 is ‘low er’ class, $ 4 0 ,0 0 0 to $7 9 ,9 9 9 is ‘m iddle class’, and $ 8 0 ,0 0 0 and over is
‘upper class’; and to operationalize ‘conserv ativ es’ on the basis o f a series o f qu estio ns about
political issues. Perhaps you are in terested in w hether there is a d ifference in th e ‘personal
a d justm en t’ in the m ilitary o f m en and wom en. How do you know ‘p ersonal ad ju stm en t’
when you see it? You d ecide that those w ho, in answer to the qu estio n ‘H ow have you adjusted
to being in the arm y?’, responded that in general they were in good spirits, that th ey liked
being in the army, and that they were satisfied with th eir arm y jo b s and status, can b e classi­
fied as well adjusted, personally.

Illustration: Lenin's explanation of World War I

Let’s review the com ponen ts o f a hypothesis by consid ering how they co m b in e to produce a
well -known th eory o f the causes o f W orld W ar I.
D uring W orld War I, a book by V ladim ir Ilyich Lenin (1 8 7 0 -1 9 2 4 ) was published, entitled
Imperialism, the Highest Stage o f Capitalism (1 9 1 6 ). The b o o k was inspired by a co n ce rn to
understand the causes o f the war that had engulfed Europe and was escalating into a global
conflagration. In the process o f developing an explanation o f the war, Lenin also developed
an argum ent about capitalism , im perialism , and revolution.
Lenin explains World War I as a consequence o f in tense im perialist rivalry am ong the
m ajor European powers. The source o f his hypothesis was M arx’s notion that the co n trad ic­
tions o f capitalism would lead inevitably to revolutionary crisis and open the way to a transi­
tion to socialism . M arx had pointed to a num ber o f contradiction s o f capitalist developm ent:

1. Technological change under the direction o f capitalism creates u n em ploym en t (a

'reserve arm y o f lab or’). This holds wages in check and depresses the con su m in g pow er o f
the w orking class.

2. U nem ploym ent is an integral and essential elem ent o f the system : w ithout it, capitalists
would lose control over the labour market.

3. As capital accum ulates, the rate o f profit falls. This causes capitalists to interrupt the
accum ulation process, which produces econ om ic crises and d epressions.

4. I he power of the w orking class to consum e d oesn ’t keep pace with the grow th o f
production potential. As a result, production potential rem ains underutilized, and in place
ol actual econ om ic expansion, stagnation and con traction eventually sets in.

fig u re 6.3 shows how Lenin derived a hypothesis from M arx’s theoretical work. Lenin
observed that capitalist nations had avoided the crisis that M arx had predicted through
overseas im perialism . 1 le argued that:

1 finance capital responds to the falling rate of profit by com bin in g into larger econ om ic
units cartels, m onop olies— in order to gain m arket power;

2. finance capitalists invest capital abroad in less developed regions to get a higher rate o f
return;
 a n s w e r in g r e s e a r c h q u e s t io n s

1. SO U K E O f HYPO THESIS:
--........... .........
«CMpkmUun

Figure 6.3 The Source of len in i hypothec» about th* ciutet oi World War I
Sourer Impircd by Gurley 19/S V4 Our thaniit lo D*v»d like *rtw uwd a varutnn erf d man und*«gradu«tt wfnm*i »1 IX LA

3. the flag follows finance capital: because governm ent 1» needed to protect investment, to »cl
up exclusive spheres o f investment. As capital accum ulates and profit rates fall, capitalist
econ om ies in com petition with one another em bark on an effort to divide up the world
in o rder to secure external markets, investment outlets, and »ources o f food and raw
materials;

4. o n ce the en tire world is split up. a re division o f territory can only com e about through
war. In the process o f war, socialist revolutions o ccu r abroad.

The imperialist solution to the problem of overproduction/under-consumption was to

secure external markets and investment outlets abroad This prevented socialist revolution
at home (and. at the same time, made socialist revolution much more likely to occur in the
underdeveloped regions due to the hyper-exploitation of workers there). Lenin’s hypothesis
was that World War 1 was the outcome of efforts by the imperialist powers to territorially
re-divide the world among themselves. The temporal domain to which he intended this
hypothesis to apply was ‘the era of monopoly capitalism!
We can identify the specific component of the hypothesis, as follows. The dependent vari­
able is war in the era of monopoly capitalism; the hypothesized independent variable is
HO W TO DO RE SEA RCH: AN O VE RVIEW

Step 1. Variables

Independent Dependent
variable variable

Intervening variables

Step 2. Relationships between variables

Economic stagnation -> Expansion

l-

War «- Available territory

Figure 6.4 Lenin's hypothesis about the causes of World War I: variables and relationships

econ om ic stagnation. Lenin co n n ects the in dependen t and d epen dent v ariables by in tro ­
ducing two intervening variables. The first is the im perialist expansion th at ensues when
econ om ic stagnation leads to pressures for the acqu isition o f m arkets and in vestm ent outlets
abroad. The second intervening variable is the availability o f territory, w hich d ecreases as
m ore and m ore capitalist coun tries pursue im perialist expansion in order to offset the e c o ­
nom ic stagnation generated by con trad iction s o f capitalist prod uction. Th ere is a positive
relationship between econ om ic stagnation (th e in dependen t variable) and expansion (an
intervening variable): as one increases, so, too, does the other. Th ere is a negative relatio n ­
ship betw een the two intervening variables: as ex pansionism increases, the availability o f
territory decreases. There is also a negative relationship betw een the availability o f territory
and the potential for war: a decrease in the availability o f territory in creases the likelihood o f
war. Figure 6.4 provides a diagram o f these variables and how they are related.
After form ulating this hypothesis, Lenin investigated it by look ing for eviden ce that would
support it or prove it wrong. He presented data and evidence for each o f his key variables,
and relevant conclusions from leading scholars and econ om ists, b oth M arxist and no n-
M arxist, including data on econ om ic stagnation in the m ajor belligerent coun tries, policies
o f im perialist expansion on the part o f those states, and the decrease in the am ount o f avail­
able desirable territory into w hich these states m ight expand.

Answering normative questions

It is som etim es noted that norm ative political th eorists tend not to be as explicit about their
m ethods as are researchers concerned with em pirical qu estio ns— that they are often silent
on questions o f m ethod and approach’ (Leopold and Stears 2008: 1). David Leopold and
Marc Stears maintain that, in fact, ‘the vast m ajority o f students begin ning advanced research
in political theory in the United States and Britain’ em bark on their studies ‘without any
significant training in, and reflection on, the research m ethods that they will have to em ploy
if they are to produce high quality work o f their own’ (2008: 2).
 a n s w ir in g r e s e a r c h q u e s t io n s

We have »aid that the research pri*es& we outline in this chapter it relevant to all type* of
research Here we want to elaborate this point in relation to research that addresses norm a
tive qu estions. In our view, answers to norm ative and em pirical questions can and should be
developed in broadly sim ilar ways.

Empirical and nonnative research

In previous chapters (C hapters 1 and 3). we discussed the d istinction that has been institu
tionalized in political research between em pirical and norm ative research hm pirual
research is thought to con cern questions about the real world', questions that can be
answ ered with em pirical data (data gathered through observations of the world around us),
while norm ative research is seen as concerned with questions concerning what is best. |ust.
right, or preferable in society, which are best answered through lorrm ol argumentative dis
course. But, as we have suggested in previous chapters, the distinction defined between
em pirical and norm ative research is overdrawn Am ong other things, this has likely helped
to o b scu re the extent to w hich generally sim ilar m ethods are used to address both
T h e em pirical/norm ative d istinction is predicated on the assum ption that it is possible to
w holly separate qu estions about what is (em pirical questions) trom questions about what
should or ought to be (norm ative q u estions). However, as we pointed out in Chapter 3. many
people engaged in political research have challenged this assum ption, and this includes
m any norm ative political theorists, as well. In fact, there is an ongoing debate jm o n g politi
cal th eorists c o n ce rn in g the relationship between norm ative political theory and em pirical
in quiry. The d ebate con cern s what many perceive to be the gull between ideal loundational
philosoph ical reflections on. for instance, rights, liberty, equality, or lustice. and the real
world c on d ition s and institutions in w hich problem s relating to these issues arise and must
be resolved. O n e position m aintains that the point o f norm ative political theory is to arrive
at a coheren t ideal. The articulation o f an ideal aids our practical understanding because it
allow s us to m easure the d istance betw een it (th e ideal) and the real world. Ihe other posi­
tion argues that, sin ce the ideal is constrained by what is possible, we need to integrate th e­
ory with a consid eration o f its im plications and likely outcom es for the political world in
w hich real citizens live. A ccording to this view, the task of the political theorist is to investi
gate b oth p olitical principles and the underlying 'facts' o f political and social hie that under­
pin th em (M iller 2 0 0 8 ). The aim o f norm ative political theory is the clarification o f underlying
basic principles or argum ents. But norm ative argum ents that propose or justify one value
system over a no ther will lack relevance it they m ake no reference to related em pirical and
practical issues.
We can better understand the two positions by considering them in relation to theories o f
social justice. Theorists on one side o f the debate argue that the fundam ental principles of
ju stice are logically independent o f issues o f feasibility and questions about human nature
(e.g. M ason 2 0 0 4 . C oh en 2 008); that only by rem oving itself from the world o f em pirical
o bserv ation and the particularities o f tim e and o f place can norm ative political theory per­
form its proper task o f identifying the actual m eaning o f justice', and o f outlining the way» in
w hich ou r w orld could be m ade to fit its dem ands (Stears 2005: 326). An example o f what
m ight be called ideal' theorizing is John Rawls’ Theory o f lustice (1971). In this highly influ­
ential work. John Rawls presents what he identifies as an ideal' theory o f justice. This theory is
H O W T O D O R E S E A R C H . A N O V E R V IE W

‘id e a l’ in th a t it m a k e s tw o id e a liz in g a s s u m p tio n s : th a t ( 1 ) all c o m p l y w ith th e p rin c ip le s o f

ju stic e ; a n d th a t ( 2 ) s o c ia l c o n d itio n s a re s u c h a s to e n a b le c itiz e n s a n d s o c ie tie s to a b id e b y
p rin cip le s o f p o litica l c o o p e r a t i o n . R aw ls says t h a t, b y a b s t r a c ti n g f r o m th e w o rld as it a c tu a lly
is (e .g . ig n o r in g th e p o ssib ility o f la w -b re a k in g , o r c o n d itio n s th a t o v e rw h e lm th e c a p a c i ty o f
a c to rs fo r m o ra l re a s o n in g ), id eal th e o r y a llo w s us to fin d id eal p rin cip le s o f ju s tic e a n d to
d e v e lo p a n u n d e rs ta n d in g o f h o w to re fo rm o u r n o n -id e a l w o rld in lin e w ith th e se
p rin cip le s.
T h e o ris ts o n th e o t h e r sid e o f th e d e b a te a b o u t n o r m a ti v e t h e o r y a n d e m p iric a l in q u iry
a rg u e th a t r e s e a rc h o n s o c ia l ju s tic e h a s b e c o m e d e t a c h e d fr o m th e re a l w o rld ; th a t t h e o r is ts
w h o fo c u s o n th e q u e s tio n o f w h a t an id e a lly ju st s o c ie t y w o u ld lo o k lik e n e g le c t th e e m p i r i ­
ca l re a litie s o f real s o c ie tie s a n d th e im p lic a tio n s o f th e ir t h e o r i z in g fo r p re s s in g p r a c ti c a l
issu es o f s o c ia l ju s tic e c o n f r o n t e d b y s o c ie tie s h e re a n d n ow . T h e y a rg u e th a t c o n s t r u c ti n g a
th e o r y th a t c a n h e lp re s o lv e p ro b le m s o f s o c ia l ju s tic e re q u ire s c o n s i d e r a t io n o f th e n o n ­
id eal c o n d itio n s a n d s o c ia l in s titu tio n s fa c e d b y a c tu a l p o litica l a c to r s ( e .g . D u n n 1 9 9 0 ;
C a re n s 2 0 0 0 ; F a rre lly 2 0 0 7 ; S en 2 0 0 9 ) .
In o u r view , a n a ly tic a lly rig o ro u s th e o re tic a l e n d e a v o u r n e e d n o t b e c o m p r i s e d , b u t r a t h e r
m ig h t b e e n h a n c e d c o n sid e ra b ly , by g iv in g d u e c o n s id e r a tio n to th e d iv e rs e c o m p le x itie s
th a t a rise in re a l, n o n -id e a l s o c ie tie s . D e p e n d in g o n th e q u e s tio n , p o litic a l th e o r i s t s m ig h t
c o m b in e th e u se o f p h ilo s o p h ic a l m e t h o d s o f lo g ic , c a te g o r i z a t io n , o r c o n c e p t u a l a n a ly s is
w ith a n y o f th e m e t h o d s o f d a ta c o lle c tio n th a t r e s e a rc h e rs u se to a d d re s s e m p iric a l
q u e stio n s.

Empirical and normative questions, answers, and methods

R e fle ctin g th e view s c o n c e r n i n g id eal th e o r y a n d e m p iric a l in q u iry d isc u s se d a b o v e , is th e
two
te n d e n c y to th in k o f n o rm a tiv e q u e stio n s as g e n e ra lly fallin g in to o n e o r th e o t h e r o f
b a sic types. T h e first a re questions in which principles, ideals, an d concepts are the prim ary
focus. W hat principles o u g h t to b e a d o p te d a n d e n fo r c e d s u c h th a t c o m p lia n c e w ith th e m will
a c h ie v e s o cia l ju stic e ? W h a t w ou ld an ideally ju st s o c ie ty lo o k like? H o w is th e concept of
ju stic e b est u n d e rs to o d ? T h e o ris ts te n d to a d d re ss th e se s o rts o f q u e stio n s b y e m p lo y in g p h il­
o s o p h ica l m e th o d s o f lo g ic , c a te g o riz a tio n , o r c o n c e p tu a l a n a lysis: b y a n a ly s in g th e lo g ical
s tru c tu re o f a p rin cip le , sy s te m a tiz in g th e re a s o n s th a t c a n v alid ly b e a d v a n c e d fo r an d ag ain st
p a rtic u la r c h o ic e s , o r d e v e lo p in g a c o n c e p tu a l fra m e w o rk . T h e o ris ts m ig h t e n g a g e in a fo rm
o f a rg u m e n ta tiv e d is c o u rs e th a t, by d ra w in g lo g ica l in fe re n ce s a n d s h o w in g h o w th e id eas
p re se n te d w e re log ica lly d e v e lo p e d , c o n v in c e s o th e rs th a t th e c o n c lu s io n re a c h e d is r e a s o n ­
able an d p lausible. T h e o ris ts m ig h t u se an y n u m b e r o f differen t m e th o d s o f c a te g o riz a tio n to
d efin e th e featu res th a t s o m e th in g h a s o r d o e s n ’t h ave, by, fo r in sta n ce , c o m p a r i n g th e s im i ­
la rity o f an in sta n ce to a p ro to ty p e . T h e o ris ts m ig h t b re a k d o w n o r a n a ly se c o n c e p t s in to th e ir
c o n stitu e n t p a rts u sin g c o n ce p tu a l an alysis: ‘an a tte m p t to p ro v id e an illu m in a tin g set o f n e c ­
e s sa ry an d sufficient c o n d itio n s fo r th e ( c o r r e c t ) a p p lic a tio n o f a c o n c e p t ’ (A u d i 1 9 8 3 : 9 0 ) ,
a n d a test o f s u ch sets u sin g h y p o th e tica l e x a m p le s a n d th o u g h t e x p e rim e n ts .
T h e s e co n d typ e o f q u e stio n s n o rm a tiv e p olitical th e o rists ask are questions that involve eval­
uating and criticizing substantive political, social, and economic practices and institutions. W h ich
in stitu tio n s an d p ra c tice s are a p p ro p ria te to im p le m e n t th e p rin cip le s o f d istrib u tiv e ju stice?
D oes eq u ality o f o p p o rtu n ity req u ire in equ ality o f lib erty? T h ese are at least in p art an sw erab le
 A N S W ER IN G R E S EA R C H Q U ES T IO N S

only by em pirical inquiry and data from history and the social sciences Consequently, theo
ru ts w ho address these sort» o f questions often com bine technique* from analytical philosophy
(i.e., tools o f logic, for analysing form s o f language, and increasing argumentative d antv ) with
those from em pirical social science.
T h eorists m ight use any or all m ethods of gathering facts that em pirical researchers use
A th eorist addressing a question concerning existing beliefs might employ m ethods such as
interview s, surveys, or the analysis o f language or m eaning in discourse and texts in order to
chart expressed opin ion s and bring im plicit assum ptions to light Ihe research might engage
with and in corp orate the findings o f sociologists, econom ists, and experts in social policy It
m ight seek to identify statistical correlations, and figure out causal m echanism s, as well Ihe
qu estion m ight con cern how a value—justice, efficiency, national solidarity, welfare, sec ur
ity, d em o c ra c y —ought to be understood. An argum ent might be developed concerning the
im plications o f a particular conception of. or m oral prem ise relating to. this value It might
be developed with reference to som e particular policy or institutional area, and by appeal to
logic and em pirical evidence in varying A researcher might reflect on the relevance and
applicability o f a theory to the clarification and solution o f a norm ative issue or priiblem in
a particular dom ain. For exam ple, the research might involve analysing the usefulness of a
particular theory o f ju stice, or com paring the utility o f several theories, for the study o f prob
lem s relating to globalization, m igration, nationalism . m ulticuJturalism , education, ageing,
chan ging g ender roles, or global inequality.
In general, researchers addressing norm ative and em pirical questions will develop
answ ers to th em in broadly sim ilar ways. As in any research, the norm ative political theorists
finds and form ulates an interesting question, reads what others have written on that ques
tion, engages with this literature and, through that engagem ent, formulates an argument of
his or her own.
W e have previously argued that, irrespective o f what kind of question a rrsearcher
addresses (descriptive, explanatory, predictive, prescriptive, or norm ative), thinking in
term s o f form ulating a hypothesis in answer to it can help to clarify the argument and the
kind s o f ev id en ce that will provide a m eaningful d em onstration o f it. A normahw hypotheiu
advances, for fu rth er d iscu ssion , analysis and investigation, an argument or suggestive prop
osition about what ought to be. Recall that a hypothesis consists o f three elem ents: an inde
pend ent variable, a d ependent variable, and a proposition (a statem ent about the relationship
betw een the variables). W hile we don’t think o f norm ative political theory as requiring a
depend ent and independent variable, norm ative questions nonetheless are concerned with
co n n ectio n s and relations, with d em onstrating relations betw een prem ises and conclusions,
and in d icatin g the nature o f the relations between them . A norm ative hypothesis m ight state
that the best type o f X is Y; that the basis for d eciding or justifying X ought to be Y. the legiti
m acy o f X ought to be based on Y. or that X is just when condition s A. B. C are present.
Investigation o f these hypotheses would entail elaborating an argum ent about why this
should be the case and establishing the plausibility and coheren ce o f the argum ent, with the
help, if relevant, o f em pirical or historical evidence.
W e b egan th is d iscu ssion by observ ing that norm ative political researchers arc often silent
about th e m eth od s they use. it is not a question o f whether political th eorists em ploy m eth ­
ods: th ey do. A variety o f m ethods is used to address norm ative questions. W hat is at issue is
th eir ten d ency not to reflect on the m ethods they use. Irrespective o f what type o f research
160 H O W T O D O R E S E A R C H : A N O V E R V IE W

is pursued, research should be system atic, self-aw are, clear, and transp aren t. Th is enables
oth ers to evaluate th e argum en t w ith respect to th e kin d and n ature o f its prem ises, the
strength o f the in feren tial links betw een p rem ises and co n clu sio n s, and its possible c r iti­
c ism s or refutations.

Conclusions
We have organized our discussion of the various consid erations and tasks involved in developing an
answer to a research question around three basic requirem ents. The first requirem ent is that the
answer be appropriate to the type of question that is being asked. The second requirem ent of an
answer to a research question is that it makes a contribution to knowledge. The third is that your
answer must be clearly and fully specified with regard to the factors or elem ents you think must be
taken into consideration in order to answer your question, and how you think these factors or
elements are related to each other.
W e maintain that for all types of research, answers or arguments in response to research questions
benefit from being formulated as a hypothesis and an investigation of it. W e determ ine w hich ideas
are most useful for understanding social life by turning our ideas into hypotheses and assessing them.
A hypothesis articulates the contours and logic of the investigation. In all research, a hypothesis helps
to make explicit the argument that is being developed and to guide research. Hypotheses can either
be investigated by testing them with evidence (confirm atory research), or through their operation as a
guide to a process of discovery (exploratory research).
Next up: How do you know if your hunch is right? What kind(s) of evidence would give you
confidence that your proposition is probably correct or, alternatively, lead you to conclu de that the
proposition is probably wrong?

Questions
• W hat is the difference between a concept and a variable? How are the two related?
• What is a theory?

• What is the role of theory in political and social scientific inquiry? How does theory contribute to a
furthering of our understanding of the political and social w orld?

• In what ways are the study of concepts important for political and social scientific inquiry?
• What is the function of operationalizing concepts?

© Guide to Further Reading

B ell, Daniel and A vne r De-Shalit (eds) (2003), Form s o f Ju stice : Critical Perspectives on D avid
M iller's P olitical Philosophy (Lanham , M D: Row m an and Littlefield).
Political philosophers have traditionally argued that there is a single, principled answer to the question
of what justice is But David Miller (Principles of Social Justice. Cambridge, MA: Harvard University Press,
1999) has theorized that justice can take many different forms Taking this position as a starting point,
this collection of articles consider whether justice takes one form or many by drawing real-world
implications from theories of justice and examining in-depth social justice, national justice, and global
justice
 a n s w e r in g r e s e a r c h q u e s t io n s

Stondonfc (Lanham, MO Rowman and UtdeAeldV

Gening, John (2001). Social Science Methodology A Catena} framework. Cambridge

Cambridge Univenity Press. Concepts: General Criteria and Stratm n of Definition po
35-86

Goertz. Gary (2005). Social Science Concept! A Utet't Guide (Princeton. Nj Pnnceton University
Press).

Leopold. David and Marc Sturt (eds) (2008) P obtacal Theory; Methods and Approaches (Oatord
Oxford Univenity Press).

Sartor*. Giovanni (ed.) (1984). Social Science Concepts: A Sytutnabc Analyst (New York S^e)
Sartori s th eo retical chapter entitled Guideline* for Concept Analysis eiplam s how words
acqu ire multiple m eaning* and provide* a succinct discussion of the problem s involved in
defining con cep ts and finding suitable term s for them It contains chapters e»ammmg key
term s, e g consensus, developm ent, ethnicity in tegration political culture power and
revolution In each of these chapters, the starting point is a word that has acquired a multiplicity
o i m eanings

Schmltter. P. (2008). The Design of Social and PoNbcal Research'. in D. Della Porta and M Keaor^.
Approaches and M ethodoiogin in the Social Sciences: A Pturaba Perspective (Cambridge:
Cambridge Univenity Preu). 263-95

Schramme. T. (2008). On the Relationship btween Political Philosophy and Impincal Science*'.
Analyte und Kritlk 30:613-26.
Schramm e argue* that the findings o f the empirical sciences m ght play a role m justifying normative
claims in political philosophy He d eicnbes how political theory has becom e a discipline divorced
from empirical sciences, and outline* some functions that etnpincal ttudws might have in political
philosophy

References
Alvarez. M .) A. Cheibub. F limongi. and P fatten K A and P Pailón (2000) "Subjective
Przeworski (1996). Classifying Political Measures of Liberal Democracy' Com parar**
Regimes'. Studm in Comparative International P o ta « / Stud*» 33(1 »(February) 50-86
Development 31 3-36 Careos. J (2000). Culture Crtirrmhip and
Audi. R (1983). The Applications of Conceptual Community (CMonj Orford University Press)
Analysis M etaphiknophy 14(2) (Apnl) 87-106 Cohen. G A (2008) Rncvmq )uttKt and [qualrty
Bachrach. P andM S Barau( 1962). Two Faces of (Cambridge. MA Harvard University Press)
Power', American Political Science Rev*w 56(4) Cother. D and S Levitsky (1997). -Democracy
(December) 947-52 with Adyectrves Conceptual Innovation in
Bollen, K. A (1980). Issues in the Comparative Comparative Research. World Politia 49(3)
Measurement of Political Democracy'. American (Apnl) 430-51
Sociological Review 45 (June) 370-90. -----and J t Mahon (1993) Conceptual’Stretetvng’
----- (1993). liberal Democracy Vafcdity Revisited Alternative Vwws of Categories m
and Method Factors m Cross-National Comparative Analysis. American Potara/ Sam e*
Measures'. American Journal o f PoMcnl Saence Rm w 87(4) (December)- 845-55
37 1207-30 Coppedge M .andW H. Remcke(1990).
Bollen. 1C A. and R. W Jackman (1989). -Democracy. 'Measuring Potyarch/ Stu*et Comparator
Stability, and Dichotomies. American International Deveiopmer* 25 51-72
Sociological Rtview S* 612-21
H O W T O D O R E S E A R C H : A N O V E R V IE W

Corbin. J. and Strauss. A. (1990), 'Grounded Theory Leopold, D. and M. Stears (2008), 'Introduction', in
Research: Procedures. Canons and Evaluative D. Leopold and M. Stears (eds). Political Theory:
Criteria', Qualitative Sociology 13: 3-21. Methods and Approaches (Oxford: Oxford
Dahl, R. A. (1957). The Concept of Power', University Press). 1-10.
Behavioral Science 2:201 -15. Lukes, S. (1974), Power A Radical View (New York:
---- (1971), Polyarchy: Participation and Opposition Palgrave Macmillan).
(New Haven, CT: Yale University Press). Mason, A. (2004), 'Just Constraints', British Journal o f
---- (1989), Democracy and its Critics (New Haven, Political Science 34(2): 251-68.
CT: Yale University Press). Merton. R. K. (1968). Social Theory and Social
Dunn, J. (1990) 'Reconceiving the Content and Structure (Glencoe, IL Free Press).
Character of Modern Political Community1, in Miller, D. (2008), 'Political Philosophy for Earthlings',
J. Dunn (ed.), Interpreting Political Responsibility in D. Leopold and M. Stears (edssew), Political
(Oxford: Polity), 193-215. Theory: Methods and Approaches (Oxford: Oxford
Farrelly. C. (2005), Making Deliberative Democracy University Press), 29-48.
a More Practical Political Ideal', European Journal Nye. J. S. (1990), Bound to Lead: The Changing
o f Political Theory 4(2). 200-8. Nature o f American Power (New York: Basic
---- (2007), 'Justice in Ideal Theory: A Refutation', Books).
Political Studies 55 (December), 844-64. ---- (2002), The Paradox of American Power: Why the
Gerring. J. (1999), “What Makes a Concept Good? A World's Only Superpower Can't Go It Alone (New
Criterial Framework for Understanding Concept York: Oxford University Press).
Formation in the Social Sciences'. Polity 31 (3) ---- (2004), Soft Power: The Means to Success in
(spring): 357-93. World Politics (New York: Public Affairs).
---- (2001), Social Science Methodology: A Criterial Pollins, B. (2007), 'Beyond Logical Positivism:
Framework (Cambridge: Cambridge University Reframing King, Keohane, and Verba', in R.
Press). N. Lelbow and M. I. Lichbach (eds). Theory
Glaser, B. G. and A. L Strauss (1967), The Discovery of and Evidence in Comparative Politics and
Grounded Theory (Chicago: Aldine). International Relations (Basingstoke: Palgrave/
Macmillan), 87-106.
Gurley, John G. (1975), Challengers to Capitalism:
Marx, Lenin, and Mao (Stanford. CA: Stanford Ragin, C. (2000), Fuzzy-Set Social Science (Chicago:
Alumni Association). University of Chicago Press).

Guzzini. S (1993), Structural Power: The Limits Rawls. J. (1971), A Theory of Justice (Cambridge,
of Neorealist Power Analysis', International MA: The Belknap Press of Harvard University
Organization 47(3): 443-78. Press).

Hoover, K. and T. Donovan (2004), The Elements of Sartori, G. (1970). 'Concept Misformation in
Social Scientific Thinking, 8th edition (Belmont. Comparative Politics', American Political Science
CA: WadsworthAhompson). Review 64:1033-53.

Jacobson, G (1978). The Effects of Campaign ---- (1984), 'Guidelines for Concept Analysis', in
Spending in Congressional Elections'. Giovanni Sartori (ed.), Social Science Concepts:
American Political Science Review 72 (June): A Systematic Analysis (Beverly Hills, CA: Sage).
469-91 15-85.

Kahl. Colin H (2006), States. Scarcity, and Civil Strife ---- (1987). The Theory of Democracy Revisited
in the Developing World (Princeton, NJ: Princeton (Chatham. NJ: Chatham House).
University Press). Schmitter, P. (2008). The Design of Social and
Kumar. R (2005). Research Methodology. A Step- Political Research', in D. Della Porta and M.
by-Step Guide for Beginners. 2nd revised edition Keating, Approaches and Methodologies in
(London Sage Publications Ltd.). the Social Sciences: A Pluralist Perspective
Lenin. V I (1916), Imperialism, the Highest Stage of (Cambridge: Cambridge University Press),
263-95.
Capitalism An Outline (New York International
Publishers. 1939)
 A N S W ER IN G R E S EA R C H Q U ES T IO N S

Sen. Amartya (2009). The tdta o ffu títct (Camtondce MA OpporturMty. of P otto* Itm
Harvard Uiwtrvty Pré«) 4(4) 32S-SO
Siwety. W P (1989). The Ca/fc o f Poltxai Ancdfch. 2nd S u w f t.S (19941 S ia n « r i M ortA 2nd «dfton
edition (Engtewood CJrfh Nj Prenbce-Hal. Inc) (London Pmtcr)
Steary Marc (2006). The Vocation o f Political Theory
Principi«, im p u t a i Inquiry and the Politics of
 Research Design

Chapter Summary
Previous chapters have focused on how to find and form ulate a research question
(C hapter 5) and how to develop a hunch about the answer to one (C hapter 6). In this
chapter, w e focus on designing research to investigate your hunch. Our discussion
focuses on the following issues:

• basic principles of research design;

• types of research design;

• selecting cases and units of observation and analysis;

• types of evidence researchers use to answer the questions they ask;

• methods of collecting data;

• sources of data.

Introduction
Y o u h a v e a w e ll-c ra fte d r e s e a r c h q u e s tio n . It is o n e th a t y o u h a v e s h o w n ( 1 ) h a s s ig n ifi­
c a n c e fo r a to p ic o r issu e re la tin g to th e s u b je c t m a t te r o f o u r field ; ( 2 ) is re s e a r c h a b l e (it
c a n b e a n s w e re d th r o u g h c o n d u c t in g r e s e a r c h ) ; a n d ( 3 ) h a s n o t y e t b e e n a n s w e re d d e f i n i ­
tively. Y o u a lso h a v e a h u n c h a b o u t th e f a c to rs y o u th in k m u s t b e ta k e n in to c o n s i d e r a t io n
in o r d e r to a n s w e r y o u r q u e s tio n , a n d h o w y o u th in k th e s e fa c to r s a re re la te d to e a c h
o th e r. T o geth er, th e se f a c to rs a n d th e ir r e la tio n s c o n s t i tu t e w h a t w e c a ll a ‘ hypothesis’: a
re a s o n e d , c le a rly s p e c ifie d h u n c h o r e x p e c t a t io n a b o u t th e a n s w e r to y o u r q u e s tio n th a t
will g u id e a n d fo cu s y o u r re s e a r c h . In C h a p te r 6 w e s a id th a t y o u r r e s p o n s e to a re s e a r c h
q u e s tio n will c o n s is t o f a h y p o th e s is a n d a n i n v e s tig a tio n o f it; a n d w e d is c u s s e d th e b a s ic
c o m p o n e n t s o f a h y p o th e s is , a n d h o w to fo rm u la te o n e . W e tu rn n o w to a d is c u s s io n o f
h o w to in v e s tig a te it.
S o yo u h a ve a h u n c h o r a rg u m e n t a b o u t th e a n s w e r to y o u r r e s e a rc h q u e s tio n . H o w d o
y o u k n o w w h e th e r y o u r h u n c h is rig h t? A n d h o w will y o u c o n v i n c e o t h e r s th a t it is rig h t?
Y o u r s tra te g y fo r p ro v id in g a c o n v i n c i n g ‘t e s t’ o r d e m o n s t r a ti o n o f y o u r h y p o th e s is is w h at
we call a ‘re s e a rch d e sig n ’. A re s e a rc h d e sig n sp e c ifie s th e s o rt o f e v id e n c e y o u n e e d to in v e s ­
tig a te y o u r h y p o th e s is , a n d d e s c rib e s h o w th e e v id e n c e will b e c o lle c te d a n d a n a ly s e d .
In th is c h a p te r w e first d isc u s s th e b a sic p rin cip le s o f re s e a rch d e sig n . W e th en p ro v id e an
ov e rv ie w o f ( 1 ) ty p e s o f re s e a rch d e sig n ; an d ( 2 ) m e th o d s o f d a ta c o lle c tio n . R e se a rc h
d e sig n s an d d a ta c o lle c tio n m e th o d s a re o fte n c o n fu s e d w ith e a c h o th e r, a n d in d e e d th e re is
a c e rta in a m o u n t of o v e rla p b e tw e e n th e m . T h e o v e ra ll s tra te g y y o u will e m p lo y to d e m o n ­
s tra te o r in v e stig a te y o u r a rg u m e n t, is y o u r research design. T h e so rt o f d a ta th at will e n ab le
you to im p le m e n t th is stra te g y , an d w h e re an d h o w th e y will b e o b ta in e d , is y o u r m e th o d o f
 R E S EA R C H D t l IG N

data collection A research design answers the question W hal sort of evidence will m l my
hypothesis? The answer m ight be data o n the perceptions, attitude*, and opinion» of politi
c u n s . A d ata-collection m ethod answers the question 'How and where can I get this in for­
m ation ?’ It m ight specify, for instance, that we will use surveys or interviews, ethnographic
research, o r content analysis, or all four together It will specify whether the data will he col
lected using qualitative or quantitative m ethods, and w h e th e r we will collect them ourselves
or use data collected by others.
In som e fields, the term research design' is also used to refer to the overall plan for address
ing a research problem or question. In this usage, the research devgn include* the definition
o f the research question, the hypotheses to be examined, including the number and type ol
variables to be studied and the relationship between them; and the schedule and budget lor
carryin g out the research. W hile a research design and a data collection m ethod are clotdy
related in the sense that a particular m ethod of data collection must tit with the strategy defined
by a research design, they represent two separate steps or com ponents in the re v a K h pnKess
O ften students will com e to us and say som ething along the lines of the following ’I am
going to be in Egypt over the sem ester break and. as I want to write my th e m on l-gyptian
foreign policy, was w ondering w hether you think it would be a good idea to try and conduct
som e interview s while I am th ere*' In most case* our response will be. T don't know it
depends on the research question you intend to addrc*» and v«hat h y p o th e c you dcxide to
investigate.'1
The point is, you cann ot collect data or gather inform ation until you know precikely whal
sort o f evidence you need. W hat you will need to be looking for is evidence that would c o n ­
firm or d iscon firm the argum ent you plan to advance or investigate, but you won’t know
what sort o f evidence that is, unless you know what argument or hypothesis you are putting
forw ard for Investigation. To call upon the familiar 'toolbox' metaphor, it is as if you said
'There is som eth ing in the basem ent that we want to repair. Should we use a ham m er or a
w rench?' How can we know w hich tool to suggest if we don't know the nature of the problem
and how you th ink you m ight go about solving it?1 h n t. you formulate a research question
and a hu nch about what m ight provide an answer to it; then you can decide whal inform a
tlon you will need to collect in order to confirm or disconfirm your hypothesis O nly then
will you know w hether it would be useful to conduct interviews, and if so. whom you should
Interview and what you should ask.

Basic principles of research design

B efore you b egin your research, you will need to think carefully and creatively about what
jo r t o f research will enable you to answer your question o r test your hypothesis in a convinc­
ing way. Your research design Is th e culm ination o f this thinking. It specifies what will enable
you to draw logical, valid, and reliable conclusion s about your hypothesis. A good research
design has th e follow ing attributes;

1. it specifies the type o f research and tech niq ues o f data collection appropriate to the
o b jectiv e s o f the p ro jcct;
H O W T O D O R E S E A R C H : A N O V E R V IE W

2. it m akes ex plicit th e logic w hich enab les y ou to draw in feren c es— logical co n clu sio n s
based on the in fo rm atio n you collect o r o b serv ation s you m ake;

3. it identifies the type o f ev iden ce that no t on ly c o n firm s your hyp othesis o r argu m en t,
but provides a co n v in cin g ‘test’ o f it;

4. it decreases th reats to the in ternal and external validity o f you r findings;

5. it ensures that your findings are reliable.

D ifferent types o f research design and m ethod s o f data co lle ctio n are no t m o re o r less rig o r­
ous than oth ers; they are only m ore or less appropriate for th e task at hand . A saw is not
b etter or w orse than a w rench; it is only m ore or less useful in relation to a specific task. A
research design is good if it allows research ers to draw valid in feren ces. It should provide a
structure o f in quiry that enables us to draw logical con clu sion s on th e b asis o f know n facts
or prem ises; and it should m ake explicit o f the logic lin kin g o f the facts we know (th at we
collect or observe) to those we don’t (th ose that are th e su b jects o f o u r research qu estio ns
and hypotheses). The process o f using the facts we know to draw logical co n clu sion s about
facts we do not know is what we call in feren ce (K ing, K eohane, and V erba 1994: 4 6 ). This is
the basic logic that underlies m ost th inking. However, in co n d u ctin g research , th e process
must be m ade explicit and follow certain rules.
Your research must be presented in a way that is clear, orderly, and system atic eno ugh so
that som eone else will be able to retrace your steps and arrive at the sam e results and c o n clu ­
sions. Know ledge is the product o f research that is open at every step to ch eck in g and d o u ­
ble-check ing by others. C onsequently, you must be b oth transp aren t and explicit. You must
d escribe what you did (or plan to do) in clear, sim ple language.
The com ponents o f a research project should be designed to provide a conv in cing test o f
your hypothesis. As David de Vaus puts it, rather than seeking evidence that is consistent with
our argum ent or hypothesis, we should seek evidence that provides a compelling test o f it (de
Vaus 2001: 11). You can usually find em pirical support for alm ost any hypothesis. So rather
than m arshalling evidence to fit an argum ent or hypothesis, you should seek evidence that can
‘test’ it against rival perspectives, argum ents, or hypotheses (see the discussion o f ‘literature
reviews’ in Chapter 5) and show why your explanation or in terpretation is to be preferred.
Silence your sceptics by raising the objections o r questions that they m ight raise. A ssum e their
point o f view and argue, on their behalf, against yourself. T h ink carefully about what sort o f
evidence will enable you to make com parisons and ju dgem ents betw een alternative possible
explanations, and to provide the analytic leverage you need to dem onstrate that your hyp oth­
esis is in som e way better, m ore credible, useful, accurate, or illum inating than its rivals.
The better the research design, the m ore it decreases threats to validity. A study is valid if
its measures actually m easure what they claim to, and if there are no logical errors in draw ­
ing conclusions from the data. R esearchers distinguish am ong a variety o f different types o f
validity. We will discuss two types: internal validity and external validity.
In ternal validity often refers to the extent to which we can be confident that the in d epend ­
ent (causal) variable produced the observed effect. The m ore the structure o f a study e lim i­
nates alternative interpretations, the better we are able to draw unam biguous conclusion s
from our results, and ‘the stronger the internal validity o f the study’ (de Vaus 2001: 28). W hat
is at issue is w hether there are factors, other than your independent variables, that m ight be
 R t i t A R C H D IS IC N :.H ?

affecting the outcom e. Is ih c independent variable responsible lor variation in the dependen!
variable? W hal other possible causes might there be tor the relationship between the v an*
ble»? C ould som ething else have been responsible for the variation in the dependent variable’
C ould there be confound ing factors? Ihese are the biggest ihreal to validity: if you do not
design your research in a way that enables you to rule out other tactor* or alternative eiplana
tions. the internal validity o f your study will be threatened External validity reters to the
extent to which results from a study can be generalized beyond the particular study (de Vaus
2 001: 28). C an you generalize your findings? Are your conclusions likely to apply more
widely? Are they applicable to o ther sim ilar situations or cases’
R esearch should be designed to ensure that your findings are reliable R eliability reters to
the repeatability’ or co n sisten cy’ o f your findings A research design is reliable it other
researchers can perform exactly the sam e procedures and com e up with the same results
(your findings are repeatable). A reliable m easure is reliable if it gives us the same result over
and over again (assu m ing that what we are m easuring isn't changing)
Now that we know what a research design does for us. let's consider different types ot
research designs used in political research.

Types of research design

T he types o f research design that are com m on ly used in our field include experim ental
d esig n s, cro ss-sec tio n a l and lon gitu d inal designs, and com parative designs Below, we
d iscu ss ex p erim en ta l designs, which we d iscu ss fu rth er in C hapter 8 Hollowing that is a
b rie f overview o f w h it is conv en tion ally called ‘qu asi-ex p erim en tal' or ‘no n -exp en m e n
tat' d esign s. I h e s e are used in m ost social scien ce research. They lack the ch aracteristics ot
a tru e ex p erim en t, but end eavour to em ploy oth er m eans o f con trollin g the environm ent
and isolatin g th e effects o f particu lar factors or variables We ou tlin e the basic principles
o f cro ss-se c tio n a l and long itu d inal designs. We th en provide a b rief in tro d u ction to com
parative d esign s, w hich we take up in greater detail in C hapter 9

Experimental designs
E xp erim en tal designs are often thought to most closely resem ble the true scientific m ethod
(see C hapter 2 ). and as such they are widely regarded as b eing the m ost effective design for
testin g w hether or not tw o variables are causally related. They m anage to do this thanks to
th e rigorous use o f ex p erim ental control. This helps to overcom e one o f the m ain problem s
that research ers face w hen they want to investigate causal relationships: there are a vast
nu m b er o f p otentially im portant variables that may influence any given political phenom
ena. and in the ab sence o f rigorous controlled exp erim ents (not dissim ilar to those used in
m ed ical resea rch ), it is very difficult to know w hich are responsible for causing the p h e n ­
o m en a un d er investigation. O n e o f the great strengths o f experim ental research is that the
research er can co n tro l the env iron m en t and m anipulate particular variables o f causal
in terest w ith great precision . Indeed, th e d efin ing ch aracteristic o f experim ental research
is in terv en tio n by th e research er in the d ata-gathering process (M orton and W illiam *
H O W T O D O R E S E A R C H : A N O V E R V IE W

2 0 0 8 ). W e call th e data gath ered by in terv e n tio n (o r treatm en t, as it is so m etim es kn o w n )

ex p erim en ta l data.
B roadly speaking th ere are th ree m ain ways o f c arryin g out ex p erim en tal research: in a
laboratory, in the field, and by utilizing natural occu rre n ces. In lab oratory ex p erim en ts, the
su b jects are recruited to a co m m o n location where the ex p erim en t takes place. The la b o ra­
tory e xp erim ent is designed to ensure that the research er has as m u ch co n tro l over th e en v i­
ronm ent to w hich sub jects are exposed as possible. In th is way the ex p erim en tal group and
the control group can be exposed to exactly the sam e env iron m en t except for th e e x p e ri­
m ental in tervention. A ccordingly, any d ifference that is recorded on the p o st-test m easu re­
m ent o f the dependent variable can be confidently attributed to the presence (o r a b sen ce) o f
the in tervention. However, because o f co n ce rn s about th e artificial env iron m en t o f lab o ra­
tory experim ents, som e political scien tists favour the m ore natural settin g o f field ex p eri­
m ents. In field experim ents, the in terventio n by the research er takes place in real-w orld
environ m en ts, and subjects may not even know that they are participating in an exp erim ent.
This obviously raises som e eth ical issues, to do with in form ed con sen t and deceit (w hich we
discuss in detail later in this chapter). But as G erb er and G reen (2 0 0 3 : 94 ) observe, field
experim ents have two m ain strengths. First, random assig nm en t ensures unbiased in feren ce
about cause and effect, and secon d, the natural settings ensure that the results will tell us
som eth ing useful about the real world.
The third m ain type o f experim ental design is often referred to as a natural experim ent.
The natural experim ent relies on naturally o ccu rrin g events as in terventio ns rather than
interventions controlled by the researcher. Even though in this case the research er is not
d oing the intervening, the approach taken with the data is as if the researcher has.
There is often thought to be a trad e-off betw een these different types o f designs. In par­
ticular, it is often argued that laboratory experim ents tend to have relatively high levels o f
in ternal validity but low levels o f external validity, whereas field exp erim ents tend to have
higher levels o f external validity but lower levels o f internal validity. That is, with lab e x p eri­
ments, the controlled setting in w hich the experim ent takes place enables the research er to
have a high degree o f confidence in the effect o f the causal variable on the ou tcom e o f in ter­
est (high internal validity). However, at the sam e tim e, because o f the artificial environ m en t
o f the laboratory setting, it is less certain w hether the findings from the experim ent may be
applied or generalized to real-w orld settings (low external validity). By contrast, sin ce field
experim ents take place in the real world and reproduce as closely as possible the cond ition s
under which different political phenom ena occur, they tend to have higher levels o f external
validity, m eaning the findings can be generalized with m ore confidence.
G iven these different strengths and weaknesses, rather than prioritizin g one approach
over the other, or regarding one approach as inherently sup erior to the other, it m akes m ore
sense to think about how the two approaches can be com bined. For a study to really stand
up, it should be possible to try and exam in e it in both settings. As E lin or O strom (2007:
2 6 - 7 ) says, ‘To test theory adequately, we need to use m ethods that together com bin e ex ter­
nal and internal validity. O ne gains external validity in doing field research, but internal
validity in the laboratory’.
Although experim ental research has certain obvious strengths, particularly when it
com es to testing causal hypotheses, it is still not used particularly widely in political research.
I lowever, there is the potential for it to be used m uch m ore widely than it is. In C hapter 8 we
 R I X A R C H D ESIG N

discuss som e o f the m ain reasons why it is not used m ore widely, with reference lo som e ol
the restrictio n s that eth ical issues and practical considerations play in determ ining what
sort o f q u estio ns can be answered. But we also go on to discuss the potential avenue* lor
future research.
H owever, even if e x p erim en tal research is not used widely, it has still been incredibly
in fluen tial in term s o f shaping how wc can study p olitics M ost ol our research designs
are, in effect, an effort to approxim ate the logic o f experim ental design as closelv as pos
sible and in th e positiv ist hierarchy. Large N qu antitative analysis is often seen as the next
best th in g to d oin g ex p erim en tal research.

Quasi-experimentaJ or non-experimcntal desifns

Quast-experimental'. or non (experimental, deugns are em ployed in most social science
research. In these designs, one characteristic ol a true experim ent will be missing, usually
either rand om ization or the use o f a separate control group. But. m ore generally, we can say
that qu asi-or no n-exp erim ental designs differ from experim ental designs in the degree of
control a researcher has over the subjects of research (that which is being studied) and the
cond ition s in w hich the subjects exist. W hen researchers have com plete control over an
ex perim ent, they allow one and only on e variable to ch an g e They then can trace the effects
o f that chan ge to the dependent variable and measure how it responds Ihe greater the degree
o f control we have over our sub jects and their environm ents, the better able we are to assess
chan ge in the dependent variable (D V ) as a result ot change in the independent variable
(IV ); and the g reater the internal validity o f our research.
Research in the social scien ces rarely perm its com plete control over subjects and their
environ m en ts. Instead, through the use ol quasi or non experim ental designs, we employ
m ethod s o f ob serv ation and data collection w hich, as far as possible, hold all conditions
constant but one. Types of qu asi-expertm en tal designs in d ud e cro w -sectio n al and lo n g itu ­
d in a l stu d ies, and com parative studies. We often use these designs lo increase our ability to
attribu te changes in our D V to changes in our IV a n d thus, to increase the in ternal validity
o f our research.

Cross-sectional and longitudinal datifns

Cross-sectional designs involve analysing a sample, or cross-section, of a population at a
single point in time. Longitudinal designs explore changes or trends over time. In a
longitudinal study, research is repeated with the same sample or set of cases over two or
more intervals. To illustrate how these two designs might differ in how they address the
same question, consider the hypothesis that as countries get richer income distribution
becomes more unequal. A researcher might employ either a cross-sectional research design
or a longitudinal design to investigate this hypothesis. Using a cross-sectional design, a
researcher might select sample of countries with different levels of economic growth (low.
medium, high) and analyse the association between level o i growth and change in
inequality. With a longitudinal design, a researcher might select a country or countries that
have moved from lower to higher levels o i development to see whether income distribution
over time was consistent with the trend suggested by the hypothesis.
H O W T O D O R E S E A R C H : A N O V E R V IE W

A key feature o f a cro ss-sectio n al design is that it is co n ce rn e d with explaining

v ariation betw een places or betw een people at a single p oin t in tim e, rather th an explaining
v ariation w ithin places over tim e. An exam ple o f a c ro ss-sectio n al study is on e c ond u cted by
T im m o n s R oberts, Bradley Parks, and A lexis Vasquez to investigate ‘W h o Ratifies
E nvironm ental Treaties and W h y?’ (2 0 0 4 ). In this study, T im m o n s R ob erts and his c o ­
authors try to explain variation betw een different states’ b ehav iou r w ith respect to in tern a ­
tional env iron m en tal issues. To explore this qu estion th ey con d u ct a cro ss-sectio n al
study on the d eterm inan ts o f w hether or not states have ratified in ternational env iron m en tal
treaties, using data from 177 coun tries. They find that m ost o f th e variance in the ratification
o f e nviron m en tal treaties is accoun ted for by th ree variables: ‘disadvantaged in sertion in to
th e world econ om y ’ (defined in term s o f cou n tries with a narrow ex p ort base), voice and
accoun tability through d om estic in stitutions (th e degree to w hich citizens ch o o se th ose who
govern them and the independent role that the m edia plays in keeping governm ent
accoun table), and civil society pressure (n u m b er o f n ongovernm ental organizations, N G O s,
in the coun try). In particular, ‘the nu m ber o f N G O s in a nation appears virtually syn ony­
m ous with its likelihood to participate in env iron m en tal treaties’ (2 0 0 4 : 3 9). W hile
the d ependent variable for this study is a m easure o f how m any env iron m en tal treaties were
ratified betw een 1946 and 1999, the independent variables (a co u n try ’s position in the world
econom y, voice and accountability, nu m ber o f N G O s per cou n try) are draw n from a single
point in tim e (m ostly from 2000).
W hen data are collected at a single point in time, as in a cro ss-sectio n al study, analysts
must be careful about draw ing any conclusion s about changes over time. In order to ov er­
com e this problem , repeated cro ss-section al studies can be used to in troduce a longitudinal
elem ent into the research design. An exam ple o f a repeated cro ss-section al study is W illiam
M ishler and Richard Rose’s (2 0 0 7 ) investigation into how citizens, socialized by au th oritar­
ian regim es in q u in tessen tial^ a uthoritarian cultures, can learn the attitudes and behaviours
necessary to becom e loyal and effective citizens o f new d em ocratic regim es. W illiam M ish ­
ler and Richard Rose address this question by c on stru ctin g a repeated cro ss-section al study
to test two com petin g hypotheses drawn from two th eories that d om inate the study o f p ro c­
esses o f political learning and relearning.
Cultural theories hypothesize that basic political attitudes inculcated through early life
socialization are deeply ingrained and change only slowly over extended periods. Institu­
tional theories em phasize adult political experiences or political ‘relearning’ based on in d i­
viduals’ rational assessm ents o f contem porary institutions and circum stances. Attitudes and
behaviours are m alleable and adaptable. ‘Thus, later life experiences are expected to play a
greater role in shaping adult opin ion s’ (M ish ler and Rose 2007: 823). The two th eories yield
different predictions: cultural theories 'predict substantial generational differences in p o liti­
cal a ttitu d es. . . that change little over an individual’s lifetim e’; institutional theories ‘predict
substantial changes am ong individuals across either generations or the life cycle’ (M ish ler
and Rose 2007: 824). The authors test these com petin g perspectives using data from a series
o f national probability surveys conducted in Russia between 1992 and 2005. Each survey is
an independent cro ss-section o f the population and not part o f a panel design (one in which
a particular set o f respondents are questioned repeatedly). Although the content o f the sur­
veys changes over tim e to reflect changing cond itions in Russia, they include a core o f qu es­
tions asked consistently over tim e to facilitate com parisons. Their evidence suggested that
 R E S EA R C H D ESIG N 171
‘individual assessm ents o f contem porary political and econom ic experiences. and that Rus
sians lifelong socialisation into an authoritarian culture by an authoritarian regime is not in
itself an insurm ountable obstacle to the developm ent of d em ocracy in Russia (M ishler and
Rose 2 007: 8 32).
W hile repeated cro ss-sectional studies can be used to m easure trends or aggregate change
over tim e, they d on’t m easure individual developm ent or change Ih is type ot design might
th erefore be less satisfactory for investigating, for example, the relationship between indiv
iduals civic orientation s and political participation. As )an Leighly points out. studies that
rely o n cro ss-section al survey data o n individuals' characteristics, such as civic orientations,
as pred ictors o f participation, assum e that positive civic orientations are causally prior to
acts o f participation. But the d irection o f causality can run in the opposite direction: par
ticipation can lead to changes in individuals' political attitudes, and particularly in their
sense o f political efficacy and sophistication (1995: 186) Studies of political participation
th erefore need to look at individuals' acts o f participation, and track how their participation
decisions change, over tim e (1995: 198). This is what longitudinal designs allow us to do
They study processes o f individual developm ent and change over tim e, and the etiects ol
earlier events or characteristics on later outcom es
Th ere are two m ain types of longitudinal design. In a co ho rt' study, the researcher selects
a group o f people w ithin a delin eated popu lation , i.e. having a sim ilar characteristic or
ex p erien ce (e.g. an age or eth nic group), and charts the individuals' developm ent processes
from a particular tim e point. In a panel study, the researcher chooses a sample, often a ran ­
dom ly selected national one, and collects inform ation on it at two or m ore points over tim e
W hile panel studies gather in form ation on the sam e people at each tim e point, the number
o f p articipan ts tend s not to b e constant over tim e, because of attrition. This can introduce
bias into the results. Longitudinal data are particularly useful in answering questions about
the d ynam ics o f change. For exam ple, under what cond itions do voters change political
party affiliation? W hat are the respective roles o f mass m edia and friends in changing politi­
cal attitudes? It is also useful in predicting long-term or cum ulative effects that are norm ally
hard to analyse in a cro ss-section al study.

Comparative designs
Comparative research designs are perhaps the most widely used research design in political
research. In a sense everything is comparative. We can have comparative ex pen mental
designs, comparative cross-sectional designs, and comparative longitudinal designs. We can
even think of the single country case study as a type of comparative design, since it usually
involves a comparison of some type or another, whether it is between regions within the coun­
try or between periods over time. Within the comparative framework it is common to distin­
guish between three main types of research design. "There are large-N studies (where N refers
to the number of countries—or cases-t h a t are compared), small-N studies (involving the
analysis of a small number of countries, typically 2 ,3 .4 but with no real upper limit) and
single-N studies (otherwise known as case studies). These designs are distinguished primarily
in terms of how many countries (or cases) »re compared, but also in terms of how the coun­
tries for analysis are selected. Both aspects of case selection are very important These are
discussed in detail in Chapter 9.
H O W T O D O R E S E A R C H : A N O V E R V IE W

Th ere is often th ough t to be a trad e-o ff betw een th e in -d ep th , intensive know ledge derived
from th e study o f a sm all nu m b er o f cases, on the on e hand, and th e extensive, cro ss-case
know ledge based on the study o f a large n u m b er o f cases, on the o th e r— although th is is
often overstated. Sm all-N studies or sin g le-co u n try case studies are often used to un cover
causal paths and m echanism s and assess specific m e chanism s identified in th eories. Th is is
frequently referred to as p ro cess tracin g . R esearch using a large nu m b er o f cases m ay
observe a strong statistical relation betw een two variables, and m ay use a th eory to d escrib e
these statistical results. But if the research er is unable to observ e d irectly th e key m e c h a n ­
ism s o f the theory, it will be difficult to know if the m echanism s p roducin g the statistical
relation are the sam e as those d escribed in th e theory. S electin g a sm all n u m b er o f cases for
in -d epth investigation can enable a research er to test for the existen ce o f these m echanism s.
The study o f a sm all nu m b er o f cases can also enable a research er to investigate a case that is
th eoretically anom alous. D etailed study o f a case that deviates from th eoretical ex p ectations
may generate findings that lead us to substantially revise or altogeth er discard ex istin g
theories. E xplaining anom alies can also be a sou rce o f new th eories. In -d ep th investigation
o f a sm all nu m ber o f cases can also help in gen erating hypotheses and th eories in developing
fields o f inquiry.
B ecause case study designs can in corp orate a broader history and w ider con text than can
o th er designs, case study research is generally stron g in dealing with two th reats to in ternal
validity: the threats w hich ‘history’ and ‘m aturation’ present to the validity o f a study. ‘H is­
to ry ’ refers to historical or contextual factors, and ‘m aturation’ to natural changes that affect
the relationship betw een the independent variables and the outcom e. C ase study research is
generally w eaker in external validity (i.e. generalizability) because it includes only a sm all
nu m ber o f cases o f som e m ore general phenom enon (G errin g 2007: 4 3). G reater generality
can be achieved by using a larger set o f cases (as, for instance, in com parative case study
d esigns). However, extending the analysis to broader contexts m ight lead to conceptual
stretching and, thus, threaten the conceptual validity o f the study (C o llie r and M ah ony 1996:
69; see C hapter 6 for a discussion o f ‘conceptual stretch ing’).
Research costs will also be a factor in the nu m ber o f cases you select to study. If each case
requires a lim ited am ount o f easily collected in form ation , you m ight include many, or even
all, relevant cases. If you need a great deal o f in form ation , or the in form ation you need is
harder to collect, you will exam in e fewer cases— perhaps only one or two. C ross-national
studies o f war that rely on quantitative m easures that are easily com puted from available
statistical sources (e.g. United N ations annuals or com puterized data banks) m ight include
every coun try in the world. But, a study o f the im pact o f privatization policies on social
inequalities am ong m u lti-ethn ic populations m ight require m ore detailed in form ation that
can be gathered only through close analysis and would lead the researcher to focus on only
a few cases.
You must provide a rationale for why the specific case or set o f cases you selected, from
am ong all those in the larger population, were chosen. R esearchers may select cases because
they are critical (to testing a th eory), revelatory (reveal relationships which cannot be studied
by other m eans) or unusual (throw s light on extrem e cases) (Yin 1984).
But whatever the purpose they are chosen to serve, it is im portant that they are selected
with care, and that the selection is based on the type o f case that will provide the m ost c o n ­
vincing test or investigation o f your hypothesis. Ultimately, the choice o f how many and
 R E S EA R C H D E SIG N

which cases you study and how you will study them will be determ ined bv vour research
q u estion and the h ypothesis that you intend to investigate, though it is b ecom ing more com
m on to integrate these different research designs together, so as not to le a v e out either the
general or the specific.

Types of research, types of design

Km pirical questions (descriptive, explanatory, or predictive questions) are concerned with
how som eth ing works, why things happened the way they did or what might be the outcome
o f current trends or conditions. An answer to a descnptiw question might involve describing
the characteristics o f som ething; or m odelling how it works or behaves. (Common designs for
these questions are case studies that provide an in -depth understanding of a process, event, or
situation. Explanatory questions need a design that enables the researcher to determ ine
w hether one or m ore variables causes or affects one or m ore outcom e variables, and to rule out
oth er feasible explanations. Predictive questions require research designs that enable research­
ers to forecast future developm ents, based on an analysis of current events and relevant th eo­
ries. We can analyse evidence and find that event A causes or explains event B (an explanation).
and then predict that if A continues to increase, we will likely have a greater amount of B in the
future (W hite and C lark 1983: 23). Predictive questions require you to show that if certain
cond ition s or circum stances prevail, a certain outcom e is likely to occur or com e into being.
Prescriptive questions are concerned with finding inform ation to solve political problem s
To investigate a hypothesis about what would be best to do in response to a problem, the
research er will inventory available options, weigh the pros and cons o f each for achieving the
desired outcom e, and, based on this analysis, advance an argum ent for a policy, using e x ist­
ing th eoretical and em pirical research (e.g. case studies). The researcher clearly defines the
qu estion to be answ ered or problem to be resolved; establishes relevant evaluation criteria so
as to be able to com pare, measure, and select am ong alternatives (e.g. cost, net benefit, effec -
tiveness. efficiency, equity, adm inistrative ease, legality, political acceptability); identifies
alternative policies; and evaluates the costs and benefits o f each, and the degree to which
criteria are m et in each o f them .
N orm ative questions are usually settled through reflection on w hich view, when fully
developed, offers the m ost coheren t and convin cing argum ent ( Rawls 1993: 53). The 'test' of
a norm ative hypothesis consists o f two interrelated com ponents: (1) the logical consistency
o f the id eas— the conclu sion and all points leading up to it must follow from the original
prem ises; and (2 ) consid eration o f th e argum ents o f oth er th eorists who have written about
th ese issues, b oth th ose that provide support for your own. and those that raise criticism s
and co n ce rn s that you will need to address if you are going to m ake a convincing case for
your poin t o f view. The appropriate research design, therefore, involves tracking down, as far
as possible, persuasive argum ents that certain conclusion s follow logically from certain
b asic princip les— principles over which, we can reasonably hope, far-reaching agreem ent
can be reached. The m ethod is to render such argum en ts— in respect to both basic and
d erivative p rincip les— as d ear, o pen , and logical as p ossible.*
In norm ative political analysis, th e ch ief con cern is with advancing an argum ent that is
dear, logically sound, and rationally convincing. The researcher finds an interesting ques­
tion, reads w hat o th ers have w ritten on the sub ject, engages with these works, and form s a
174 H O W TO D O R E S E A R C H : A N O V E R V IE W

carefully reasoned argum ent in reaction to th em , with help, if relevant, o f em p irical o r h is­
torical evidence. D epending on the research q u estion, the research m ight b e analytic, critical,
genealogical, decon structive, o r in terpretive; and draw on m eth od s em ployed in analytical or
applied philosophy, or the history o f political thought. The aim m ight be to provide c o n c e p ­
tual fram ew orks for deliberation, to analyse the logical structure o f a principle, or to record
and system atize the reasons that can validly be advanced for and against particu lar choices.
In these cases, th e researcher m ight em ploy th e p h ilosoph ical m eth od o f concep tu al analysis,
logic, and classification. T he aim m ight be to draw out im plications o f a m o ral prem ise w ithin
som e lim ited problem area, appealing to logic and em pirical ev iden ce in varying degrees in
the process. Som e researchers might be con cern ed with the in terp retation o f existin g beliefs.
Their aim is to ascertain what people actually th in k by chartin g expressed op in ion s and
bringing im plicit assum ptions to light. The analysis m ight be co n ce rn e d to un cov er and
exam ine a value—ju stice, efficiency, national solidarity, welfare, security, d em o cracy — that
in form s political policies and institutions and offer an in terpretation o f how that value o ught
to be understood. In these cases, the research m ight em ploy interpretive m eth od s c on cern ed
with w orking out exactly what a text is saying and why it’s saying it in the way it does.

Data-gathering strategies: how the data are collected

Fart o f your research design involves d eterm in in g what data and w hich data sou rces you
will use to answer your research qu estion. You should d escrib e the data required to in vesti­
gate your hypotheses and explain how and w here you will ob tain it. W hat sou rces are there
for these data? Are they generally available to p olitical researchers? D o th e data exist at all?
Are there problem s o f availability, reliability, standardization? C an they be successfully
overcom e?
Data collection involves setting the b oun daries for th e study, and collectin g in form ation
through observations, in terview s, d ocum ents, visual m aterials, and published statistical and
oth er data. You must set the boundaries for the study in advance o f data gathering, including
(a) the tim e fram e, or tem poral dom ain; (b) the place, or spatial dom ain ; (c) the actors or
units that are the relevant focus o f the exam in ation ; and (d) the variables or factors you th ink
arc im portant for arriving at an answer to your research question and that are, in fact, c o m ­
ponents o f your hypothesis (answ er). You are then ready to collect in form ation . This in fo r­
mation can be gathered from a variety o f different sources.
There are many different ways to collect data: through eth nographic research, surveys and
questionnaires, interviews, observations, or the analysis o f existing docum ents and texts.
Data can be gathered using m ore, or less, structured m ethods: for instance, using open-
ended and flexible questions, or questions to be answered by respondents selecting from
am ong a fixed set of choices. The m ain instrum ent for gathering data m ight be the researcher
herself ( it may be that data are not ‘collected ’ at all, but rather are co-prod uced by the observ er
and what is being observed), or instrum ents such as standardized surveys that are a d m in is­
tered in a fashion that m inim izes researcher bias. The m ethod m ight entail gath ering data
through direct contact with the subject, as in the use of focus groups or eth nographic m e th ­
ods, or without direct contact, as in the case of m ailed surveys. You can collect the data
yourself through any of these means; use data collected by others, including statistical
 ■l i t A R C H D iS IG N

sources. oral histories. m em oirs, newspapers, governm ent docum ents. publK opinion sur
veys. and interview s; or use a com bination of both Any data collation method can be used
for any type of research design. No single source has a com plete advantage over the others;
and various com binatio ns of m ethods might be com plem entary and used in tandem Your
hypothesis d eterm ines in large part the kind ol data required and suggests m ethods lor col
lecting it However, practical lim itations (e g tim e, money, skill, ethical concerns) will also
enter into your calculations and choices.
Let’s briefly consid er som e m am data collection m ethods, what each entails and what sort
o f data or evid ence each provides.

Q uestionnaires and surveys

Som etim es we need to know the personal experiences, perceptions, opinions, and altitudes
o f in fluential individuals in order to answer a research question To gel this inform ation we
can use qu estionnaires and surveys that ask people questions about particular lop u s that
can reveal inside views o f the political process. Ihese can be mailed, handed out. or con
ducted in interview form at (tee C hapters It) and 11).
Su rv ey resea rc h is a m e th o d o f g ath erin g d ata from resp o n d e n ts th ough t lo be r e p ­
re se n ta tiv e o f so m e p o p u la tio n , u sin g an in stru m en t co m p o sed ol closed or open-
end ed item s (qu estio n s). It involves selecting the people lo be approached (sam pling),
tra n sla tin g th e broad o b jectiv e s of the study in to qu estio ns that will ob tain the necessary
in fo rm a tio n (q u estio n n a ire d esign ), co lle ctin g data th rou gh qu estio nnaires o r interview s
(field w o rk), and c o d in g and in putting the responses (data processin g), Ih is is a ma|or
form o f data co lle c tio n in th e social scien ces, and is frequently used in p olitical research
Surveys combine a method of obtaining information I mm pet>ple by asking questions and
modern random sampling procedures that allow a relatively small number of people to rep­
resent a much larger population (Schuman and Presser 1996: I ). I hey are a valuable resource
for examining a wide range of topics, and can provide an accurate and reliable insighl into
what ordinary people think about politics and how they do politics. They rely on samples,
which are selected from a particular group of people or other units (such as households,
businesses, schools, etc.) that the researcher wishes to study. If we were to interview every­
body in a population (e.g. all voters, all asylum seekers), it would be incredibly time-con­
suming and expensive. Surveying a representative sample of the population allows the
researcher to make generalizations about the attributes of a given population without having
to actually interview everyone in that population. If the sample is chosen in a haphazard or
subjective way. then there is little hope of making accurate generalizations about the wider
population we are interested in.

Interviewing and focus groups

We have said that one o f the ways researchers try to find out about the social world is to
ask people questions, and that this can be done by asking people to fill in questionnaire*
(surveys). It can also be done through interviews conducted via telephone, internet,
through formal face-to-face interviews, or more informally, in the context o f focus groups
(see Chapter 11).
Interviews can be structured and unstructured. In the structured interview, interviewees
are asked a set o f identical questions in exactly the same way. and are usually asked to select
176 H O W T O D O R E S E A R C H : A N O V E R V IE W

th eir answ ers from a lim ited range o f o ptions. M ore stru ctu red tech n iq u es m ight include
surveys and qu estionnaires. Structured interview s are b etter for m ak ing co m p ariso n s and
less structured interview s m ay b e m o re a ppropriate for early e x p loratory phases o f research .
U nstru ctured interview s are m ore like ord inary conversations: th ere is n o set in terview
structure and in terview ees answ er in th eir own words. T h ese allow for lon ger q u estio n s and
m ore in -d epth probing, and are m ost frequently used in eth n ograp h ic research. T h e in ter­
viewer initiates th e conversation, presents each top ic by m eans o f specific qu estio ns, and
d ecides w hen th e conversation on a top ic has satisfied the research objectives. R esearchers
som etim es use a com bin atio n o f in terview m ethods, using stru ctu red qu estio n s to o b tain
factual in form ation (such as age or in co m e), and u n structu red qu estio ns to prob e deeper
into peoples experiences.

Focus groups
Focus groups involve a form o f un structu red in terview ing th at gen erates d ifferent data
from oth er form s o f interview ing. A focus group is a group o f people selected b ecau se th ey
are believed to be related to som e p h en om en on o f in terest. The research er m eets w ith the
group and facilitates an organized discu ssion related to so m eth in g th e p articip an ts have
exp erience of, or b eliefs about.
D iscu ssion can b rin g out in sights and un d erstan d in g s in ways w h ich sim p le q u e stio n ­
naire item s m ay not be able to tap, such as em o tio n a l and u n co n sc io u s m o tiv atio n s no t
a m en ab le to the stru ctu red qu estio n s o f co n v en tio n al survey research . Th e in te ra ctio n
am o ng focu s group p articip an ts m ay reveal m o re th an th ey w ould in th e m o re form al
in terview settin g. Using focu s groups, research ers can learn how th in g s are d iscu ssed in
a p a rticu la r cu ltu re, test hyp otheses ab ou t b eliefs, m ix people w ho w ouldn’t n o rm ally
m ix, test qu estio n s for future in terview s, or test p o licy ideas to see how citizen s react to
them .

Ethnographic research
Ethnographic research provides data on social phenom ena by placing research ers ‘in the
m idst o f w hatever it is they study so they can exam in e various p henom ena as perceived by
participants’ (B erg 2004: 148). The ob jective o f eth nographic research is to d escrib e the lives
o f people oth er than ourselves with accu racy and detailed observ ation honed by first-hand
experience. It involves participating, overtly or covertly, in people’s daily lives in a way that
can 'throw light on the issues that are the focus o f research’ (H am m ersley and A tkinson
1995: 1). The idea is that a researcher must enter the env iron m en t o f th ose under study to
observe and understand their behaviour (see C hapter 12).
In political research, eth nographic studies have been condu cted with th e purpose o f
developing a better understanding o f different institutions (e.g. the m ilitary, th e B B C , the
World B ank), or cultures (m igrants in the US, fundam entalist religious groups in the E U ),
events (a war, a sum m it), or roles (cam paign worker, news editor).
Participant observation involves the actual participation o f the researcher in the events or
environm ent being studied. This tech nique provides an ‘in sider’ accoun t o f the a ctivities and
daily lives o f the people being studied. By taking part in this way, researchers are able to
construct an account of the way in which the group perceives th eir world. The personal
insights o f the researcher can be cross-validated through repeated, in-d epth interview s with
 R E S EA R C H D ESIG N

a broad cro ss-section o f in fo rm a n t* and through conventional archival research, consuha

tion with expert*, use o f survey*, and oth er technique». The use o f multiple researcher» u an
additional way to ensure reiiabdity.

Discourse/content analysis
C ontent analysis generate» data by analysing docum ents, reports, sta tistic* manuscripts,
and oth er w ritten, oral, or visual m aterials. These m aterials perm it researcher» to access
su b jects that may be d ifficult or im possible to obtain through direct, personal contacl (such
as, for instance, in terview s with decision-m aker») or to increase the sample su e above what
would be possible through either interviews o r direct observation (see Chapter 13)
C ontent analysis involves codin g the content’ of written docum ents, audio transcripts,
radio program m es, television program m es, public speeches, or in ternet pages It allow* us to
explore the beliefs, attitudes, and preferences o f actors (even after they are dead)
C on ten t analysis provides evidence about subjectivity W hat were they (th e actors)
th inking? W hat were their intentions? Q uestions about subjective phenom ena arise in vir
tually all types o f social research. N arrative data (e.g. autobiographies, literature, journals,
d iaries, first-h and accoun ts, new spapers) often provide im portant keys to both process
(and thus m echanism s) and subjectivity. These records provide different sorts o f evidence
Verbal a ccou n ts from politician s, eyew itnesses, journalists, and contem porary histon ans
con stitu te an im portant source o f in form ation for political research These com e in creas­
ingly w ith visual data via photographs, film s, and videos Participants in the political p ro c­
esses we study (e.g. civil servants, m em bers o f advisory coun cils, and representatives of
pressure groups involved in decision m aking processes), generate party program m es, par
liam entary proceed ing s, resolutions, speeches, treaties, pres» conferen ces and press reports,
television and rad io interview s, and corresp ondence. Through content analysis, we can sys­
tem atically analyse these m aterials for clues to d ecision-m ak ers' perceptions and attitudes.
The m ethod is often used in co n ju n ctio n with oth er m ethods in order to establish a firm er
causal link. For exam ple, a researcher m ight explore w hether a nse in racist statem ents in
rad io and telev ision cable shows precedes the rise o f racist attitudes in public opinion polls
(using a survey).
U sing data from ex istin g archives allows you to widen the scope o f your work far beyond
what you could co lle ct for yourself, and m akes possible com parisons and the study o f
tren d s over tim e; and they can be used for purposes qu ite different from th ose for which
th ey w ere orig in ally collected . For in stance, the ESRC D ata Archive holds alm ost 40 0 0
d ata sets, in clu d in g th ose from m any large and im portant governm ent-prod uced surveys
and censu ses, as well as acad em ic research and historical m aterials. The Appendix c o n ­
ta in s a listin g o f data sou rces for p olitical research, including databanks, con sortia, data
arch ives from state and in d ependen t governm ent agencies, p olicy-m akin g o rganization*,
in tern a tio n a l organizations, acad em ic in stitutions, scholars, th in k tanks, and private
org anizations.
It is also a good idea to use multiple sources of data and methods of data collection when­
ever possible. Doing this enables you to approach a research problem from different angles,
something that is called ‘triangulationl Triangulation of evidence increases the reliability of
the data and the process o f gathering it. In the context of data collection, triangulation serves
to corroborate the data gathered from other sources: the use o f different data sources can
178 H O W TO D O R E S E A R C H : A N O V E R V IE W

enable researchers to cro ss-ch eck findings. T riangu lation yields m ore com p lete d ata and
results in m o re credible findings; and it also enables research ers to find agreem en t betw een
different perspectives. It m ight involve the use o f different research m e th od s to study a single
research problem : a com p arison o f th e results o f th ese different m eth od s can enab le a
researcher to identify w hether differences are due to biases in one or an o th er o f th ese m e th ­
ods. It m ight also involve the use o f different research ers to study th e sam e research p roblem
with the sam e people and with the sam e m ethods: if th e sam e results are discovered, the
findings and in terpretation s have stron ger validity b ecause o f th is c o rro b o ratio n . T riangu la­
tion m ight also involve using different th eoretical perspectives to look at th e sam e data:
exam in ing the sam e data from different th eoretical perspectives can enab le the research er to
identify bias.

Ethical research

To conclud e this chapter, we want to discuss the ethical principles to w hich all p olitical
research should conform .
Ethical issues have b ecom e increasingly im portant in guiding research sin ce the late 1990s
or so. Before then, eth ics was not som eth in g that researchers (n o t ju st eth nographers, but
pretty m uch all political researchers) would have had to th in k very m u ch about, as th ere was
little in the way o f ethical scrutiny o f research. But this has now changed. Eth ical scru tin y is
now som eth ing that nearly all research is required to go throu gh, even for graduate and
undergraduate research projects. M ost universities have eth ics com m ittees that issue guide­
lines about ethical practice, and these guidelines or codes are often based on the cod es o f
professional organizations such as the British Sociological A ssociation or the A m erican
Sociological A ssociation.
A lthough one o f the m ain goals o f research is to accum ulate know ledge or develop an
understanding o f a particular p henom enon , it is also recognized that achieving these goals
should not com e at the cost o f all else. And in particular, the social cost (or potential cost) o f
those involved in the research (th e in form ants) needs to be taken into accou n t in order to
ensure that they are not unduly exploited or harm ed. Ethical scrutin y is th erefore designed
to protect research participants. It is also designed to try and protect the university in stitu ­
tions themselves, so that researchers are deterred from engaging in ethically dubious
research activities that could com e back to haunt the institution in the form o f legal action
or unwanted publicity.
Broadly speaking, ethical issues can be sum m arized under six m ain headings: voluntary
participation, inform ed consent, privacy, harm , exploitation , and consequences for future
research. In practice, there is considerable overlap between these different principles.

Voluntary participation

The researcher should always stress that participation in a study is com pletely voluntary, and
that a person who declines to participate will not in cur any penalty. Not only should people
know that they are not required to participate in a study; they should also know that they can
withdraw from a study at any point without penalty (de Vaus 2001: 83).
 R IS E A RC H D ESIG N

Informed consent

This is closely related to voluntary participation Ih e principle o f inform ed consent require*

that the researcher tell participants about the purpose and likdv benefits o f the studv. the
process through which participants were selected, procedures that will be followed, and any
risks for the subject.
'I he issue o f inform ed consent is one o f the most holly debated ethical principles Ihe bulk
o f d iscu ssion focuses o n what is called disguised or covert particip ation Ih is type ol obser
vation happens against the in form ants’ knowledge, and they are thus not provided with an
opportu nity to express w hether they would like to take part in the research or not Most
eth ical cod es o f cond u ct now regard this type o f research as uniusttfied It is otten seen as a
type o f infiltration or spying (Bulm er 1982). and is even thought to contravene the human
rights o f autonom y and dignity o f participants (H am m ersley and Atkinson 2005) l or this
reason, covert studies are rarely carried out anym ore The defence lhal the ends ».an some
lim e justify the m eans lends not to carry much weight
I h e principle o f in form ed consent not only requires that the researcher should make
th eir identity know n, but that they should also make ihe purpose ol their research known,
and provide in form ants with the opportu nity to withdraw from ihe research prix.et» al any
stage. The Social Research A ssociation (SR A ) Ethual (.¡uiJelines (2003: 27) goes as far as to
say:

In qu iries involving hum an subjects should hr S ated a» lar as practicable on the Irerlv given
in fo rm ed c on sen t o f su bjects Ihey should be aware ol th en en titlem ent lo refuse «I any »tage
for w hatever reason and to w ithdraw data supplied In fo rm atio n that would he likely to a lle il a
s u bject's w illingness to p articipate should not he d eliberately withheld

Th e issue o f deception is one that requires careful thought. It might be reasonably argued lhal
any d eception in the cond u ct o f research involving human subjects is inconsistent with
inform ed consent and is unethical. Bui il also may be necessary for participants to remain
un inform ed about the hypotheses under investigation, to as to ensure that the results of the
study will not be biased as a result o f this knowledge ( M cD erm ott 2002: 41) I h is is an issue,
in particular, for experim ental research. If deception is involved, researchers should explain
to participants ‘what the deception was and why it was deem ed necessary for the unbused
collection o f the data. In particular, subjects should be reassured that all their data will be
confidential and that the experim enter will obtain subjects' written perm ission before any of
their inform ation is shared publicly' (M cD erm ott 2 0 0 2 .4 1 ).

Privacy
A third area o f eth ical con cern relates to the right to privacy o f the inform ants. This is partly
related to issues o f inform ed consent, and provides an opportunity for the inform ant to
d ecid e w hat in form ation they are prepared to m ake public. But it also refers to the identity
o f th e in form ant, and th eir right to anonymity. T his is im portant in all areas o f research. but
is particularly im portant w hen there is a risk that in form ation that is provided to the
research er m ight b e used against the in form ant at a later date. Protecting the privacy and
confid entiality o f w hat in form ants td l vou can th erefore be a verv serious issue. At one leveL
H O W T O D O R E S E A R C H : A N O V E R V IE W

in eth nographic research, it is easy to use pseu donym s for th e in form an ts so th at th ey can n o t
be identified by nam e. But th is m ay n o t alw ays b e eno ugh to e nsure privacy. In fo rm an ts m ay
b e identifiable by in form ation you have revealed about th em . So m e eth n ograp h ers th erefo re
prefer to m ake the field site anonym ous.

Harm

A n o th er area o f eth ic a l co n c e rn relates to th e h arm o f p a rticip a n ts. R esearch th at is

likely to h a rm p a rtic ip a n ts is g en erally regarded as u n accep tab le. B u t h arm ca n o c c u r in
a variety o f ways. A lth ou gh p o litical research d o esn ’t ten d to en d an g er in fo rm a n ts
d irectly, th e research p ro cess ca n n o n eth eless have c o n s e q u e n ce s for in fo rm a n ts w h ich
m ay cau se th em h arm or distress. S o m e tim e s th is can co m e ab o u t from th e a ct o f
research . F or exam p le, research on racism m igh t re in fo rce racist p r a c tic e s (see d is c u s ­
sio n by T roy na and C a rrin g to n 1989) or research on fo o tb a ll h o o lig a n s m ig h t in c ite
in fo rm a n ts to v io len ce as a way o f im p ressin g th e research er. But h a rm can also co m e
from th e p u b licatio n o f research , and th e m a k in g p u b lic o f d etails o f p eo p le’s lives o r
o p in io n s that m ay cau se th em d istress or even put th em in d anger fro m th e a u th o ritie s.
The id en tity o f in fo rm a n ts sh ou ld be kept co n fid en tia l; but m o re th an th at, it sh ou ld no t
be possible to id en tify th em .
All o f these issues are discussed in m ore detail in the Social Research C o u n cil’s on lin e
publication, Ethical Guidelines, posted at http://www.the-sra.org.Uk/guidelines.htm #ethic.

Conclusions
Perhaps m ore than any other academ ic discipline political research incorporates a w ide variety of
different methods and approaches. This has both its advantages and disadvantages. It strengthens
the discipline as a w hole w hen this diversity is em braced, and w hen researchers adopt and integrate
the different approaches and engage with research from across the m ethodological spectrum . It
weakens the discipline when this diversity becomes segregated, and w hen researchers from
different m ethodological traditions retreat into their own enclaves and do not engage w ith what
other people are doing in the discipline. In the following chapters w e present chapter-length
treatments of some of the most w idely used and influential methods and approaches in the study of
politics. Each method has its own strength and weakness, but rather than getting bogged down in
petty discussions about w hich method is best, or which method is best for w hich type of topic or
question, the challenge that faces serious researchers is to think about how these m ethods can be
incorporated with each other, and the strengths of one approach used to balance the weaknesses of
another

Questions
• What is meant by research design?

• What are the various types of research designs used in political research?

• W hy should researchers be concerned w ith research design? Can politics be studied rigorously
without attention to issues of research design?

• What are the major components of research designs?

• What are the general criteria for evaluating research designs?

 ■I l f ARC H D E SIG N

Guide to Further Reading

C rtw w D .JohnW (2003). Releaseh Oeugn QualaaOve flnanfttnm «. end Maed Methodt
Approach« (Thouuftd Oaia. CA: Sa«» PuWtubom)
An introduction to different ry p « of research design and ho** to (om bin t qualiUtrvr and
quantitative research strategies

Davwt. R. B. (1994) From Crow-SwtKMUl to Ux^bidHi^ Analywi. n A. Dai* «id R.». Obwm>
(•^*)< Analyzing Social and Poldtcal Change: A Casebook of Method» (Thouund Oafo. CA Sag*
Publication»).
Davies argues that m ore data are required to characterize emptncalfy the dynamic process that lie»
behind the cross seclionai snapshot. and these data can be supplied through longitudinal analysis

d• VMM. David (2001). JUtto/ch Design in ioctai Ketaarch (Thousand Oaks. CA S^e Pubfccabom).
Presents key types of social science research design including case studies cross w ctionaJ
experim ental and longitudinal with a discussion for each of tooh required possible issues and data
analysis

Druckman. J. N . D P Green. J H Kutimski. and A. Lupt& (2006). The grow» and Development ol
Experimental Research mi PoMcal Science American W X k iI Science *e*ew 100(4) 627- )S
This article documents how thinking about experim entation has rvofvrd over the century, and
demonstrates the growing influence of laboratory survey and field experiments

Flyvtycrg, B. (2006). Fhre Mktundentandtngi about Cat* Study AetMrch, Quoin*»** ingutry
12(2) (April): 219-4S
This article discusses five misunderstandings of case study research (a) theoretical knowledge is
m ore valuable than practical knowledge (b) one cannot generalize from a single <avr therefore
the single case study cannot contribute to scientific development (c) the case study is most useful
for generating hypotheses whereas other methods are more suitable tor hypotheses testing and
theory building (d) the case study contains a bus toward verification and (e) it is often difficult to
summarize specific case studies

George, A. L and A. Bennett (2005). Case Studies and Theory Oevetopmenf In the Social Sctencci
(Cambridge. MA. MIT Prm).

Gening. J. (2004), "Whai n a Case Study and Whal It It Good Forr American FoHbcal Saence Review
98 (May): 341-S4.
Thu article clanfies the meaning, and explains the utility of the case study method it argues for the
complementarity of single unit and cross unit research designs

---- (2001), Social Science Methodology: A CrUariai Frammort (Cambridgr Cambridfe UntvonMy
Pn u Y . He—aixh Design: General Criteria (chapter «). Method» (chapter 9). Trratagm of
R«M«rch Design1(chapter 10V

McDermott. R. (2002). -Experimental Method» In Potted idonc*. Annual M em of PaMHrml

ScienceS: 31-61.
This article discusses issues of central concern to exp*""»entalists including impact versus control,
mundane versus experim ental realism , internal versus external validity, deception, and laboratory
versus field experim ents, and sum marues the advantages and disadvantages of experim entation

Mundt,Gerardo(199B). Canomof R w arr h D e riy InQuaBtabveAnafyd«'.StudieiIn

Advances the idea ol muln-method approach to political research

18* H O W T O D O R E S E A R C H . A N O V E R V IE W

Ragin, C. (2000), Fuzzy-Set Social Science (Chicago: University of Chicago Press).

The difference between research designs aim ed at testing th eory and th ose aim ed at building,
developing, and refining theory.

References
Berg, B. L (2004), Qualitative Research Methods Journal of Political Science 51 (4) (October): 822-34.
for the Social Sciences (Boston, MA: Allyn and Morton, R. B. and K. Williams (1999), ‘Information
Bacon). Asymmetries and Simultaneous versus
Bulmer, M. (1982), The Merits and Demerits of Sequential Voting', American Political Science
Covert Participant Observation', in M. Bulmer Review 93:51-67.
(ed), Social Research Ethics (Basingstoke: Ostrom, E. (2007), "Why Do We Need Laboratory
Macmillan), 217-51. Experiments in Political Science?' Paper
Collier, D. and J. Mahoney (1996), 'Research Note. presented at the 2007 American Political Science
Insights and Pitfalls: Selection Bias in Qualitative Association annual meeting, Chicago, IL, 30
Research', World Politics 49(1): 56-91. August-2 September.
de Vaus, D. (2001), Research Design in Social Science Rawls, J. (1993), Political Liberalism (New York:
(London: Sage). Columbia University Press).
Gerber, A. and D. Green (2000), The Effects of Roberts, T. J„ B. C. Parks and A. A. Vasquez (2004),
Canvassing, Telephone Calls, and Direct Mail on ■Who Ratifies Environmental Treaties and Why?
Voter Turnout: A Field Experiment', American Institutionalism, Structuralism and Participation
Political Science Review 94(3) (September): by 192 Nations in 22 Treaties', Global
653-63. Environmental Politics 4:2 2-64.
Gerring, J. (2007), Case Study Research: Principles Schuman, H. and S. Presser (1996), Questions and
And Practices (Cambridge: Cambridge University Answers in Attitude Surveys (Beverly Hills, CA: Sage).
Press). Shively, W. P. (1989), The Craft of Political Research. 2nd
Hammersley. M. and P. Atkinson (1995), edition (Englewood Cliffs, NJ: Prentice-Hall, Inc.).
Ethnography: Principles in Practice, 2nd edition Social Research Association (2003), Ethical
(London: Routledge). Guidelines. Social Research Association, available
King, G.. R. Keohane, and S Verba (1994), Designing at http://www.the-sra.org.uk/documents/pdfs/
Social Inquiry: Scientific Inference in Qualitative ethics03.pdf.
Research (Princeton, NJ: Princeton University Troyna, B. and B. Carrington (1989), 'Whose Side
Press).
are We On?', in R. G. Burgess (ed.), The Ethics
Leighly.Jan (1995), 'Attitudes. Opportunities of Educational Research (Lewes: Falmer Press),
and Incentives: A Review Essay on Political 205-23.
Participation', Political Research Quarterly 48
White, L. and R. P. Clark (1983), Political Analysis:
(March): 181 -209.
Technique And Practice (Monterey, CA: Brooks/
McDermott, R (2002), 'Experimental Methods in Cole Pub. Co.).
Political Science', Annual Review of Political Yin, R. (1984), Case Study Research: Design and
Science 5 (June): 31-61
Methods, 2nd edition (Thousand Oaks, CA: Sage
Mishler. W and R Rose (2007), Generations, Age Publications).
and Time The Dynamics of Political Learning
during Russia's Transformation'. American
 ■( M A R C H O IS IC N

Endnotes
1 Of course. a wouidnl hurt lor ihe Sud an i«1the example above to take « d w u p erf an opporti«tty to
mtervww [o x m w wni officials m Efypt. as a nugM provide useful eiwghti and perhaps gtnt rate *n >dM
lor a research question Bui we woUti nonetheless use the Query as an opporuntfy to make Ihe pom
lhal research questions and hypotheses not method!. sho^d drive research
2 We made the p o r t m Chapter 5 that research should be dnwn not by methods bm by questions and
problem* But lo continue with Ihe lo o t e r metaphor «hare are researched <rf«o * * Hart the research
process by e l ecting they pr Herrad tool Irorw the research methods tootxw and then go looking lor
something to use it on
3 Ncannot be soenoAcaiy demonstrated lhal certain normative standpoints are more correct (w the seme
erf closer lo the truth) than others Normative questions typ<a*r requtfe research thai comtanes (Men
politKal facts with moral arguments In contrast 10 empirical types of research ihe researcher is net
requtfed to produce the fu> factual bans for his/her argument N is often the case that ihe researcher «ril
need to auumetKttwtfNch cannot posubty be M e d a(atf«ftreaifty (Sh»ve*y 19*9 9)
Part 3

How to Do
Research in
Practice
 Experimental Research

Chapter Summary
Thu chapter f u m r n « ih * p/.rxip*«-i ol u p e t.m m u i design and d.u.u%*e%
the issues and problems associated with different aspects cd the appiciac h in dong
so we pay special attention to the issue of .nternal and r » In n a I .al<dir> the common
obstacles assoclated with experimental research and what can be done to try and
avoid or m m im iii them The chapter examines

• basic principles ol experimental research

• field experiments

• laboratory experiments

• natural experiments

• internal and external validity

• problems

• ethnaJ issues

Introduction
The use o f experim ental designs to study political phenom ena has grown considerably in
recent decadev A lthough there is a long history ol experim ental political research, with
som e early pioneering studies dating hack to the 1920s and I9 VH ( m y in particular Harold
G osnell. 1927). in recent years there has been a marked in crease in the use of the approach
Indeed, experim ental research is now perhaps the fastest growing area o f political research
R esearchers in politics have used experim ental designs to study political m obilization (G er
her and G reen 2 000; |ohn and Brannan 2008). voting (lo d g e et al 198V). negative cam
paigning ( W attenberg and Brians 1999). coalition bargaining ( Frechette et al 200S). electoral
system s (M orton and W illiam s 1999); clientelism (W antchekon 2003). culture (H en rk h et
al. 2 0 0 4 ), identity (e.g. H abvarim ana et al. 2007). foreign policy d ecision-m aking (Cieva and
M intz 1997). international negotiations (D ruckm an 1994). justice (Frohlich and O ppcnhe
im er 1992). and deliberation (Sim on and Sulkin 2001)
T h e appeal o f th e approach is easy to appreciate Fx p erim en tal designs are often
th ou gh t to m ost closely resem b le th e true scien tific m ethod (see C hapter 2). and as such
th ey are widely regarded as b ein g the m ost effectiv e design for testin g w hether or not
tw o v ariables are causally related. T h ey m anage to do th is th an ks to the rigorous use o f
e x p erim en ta l c o n tro l. T h is helps research ers to isolate th e im pact o f a specific variable
and to ov erco m e o n e o f th e m ain prob lem s that research ers face when they want to
in vestig a te cau sal relatio n sh ip s. Sin ce th ere is a vast nu m b er o f potentially im portant
H O W T O D O R E S E A R C H IN P R A C T IC E

v ariables th at m ay in flu e n ce any giv en p o litica l p h e n o m e n o n , in th e a b se n c e o f rig o ro u s

co n tro lle d ex p e rim e n ts (n o t d issim ila r to th o se used in m e d ical r e se a rc h ), it is v ery d if­
ficu lt to know w hich v ariable is resp o n sib le for cau sin g th e p h e n o m e n a u n d er in v e sti­
g a tion . O n e o f th e great stren g th s o f e x p e rim e n ta l research is th at th e r e se a rc h e r can
c o n tro l th e e n v iro n m en t and m an ip u late p a rtic u la r v ariab les o f cau sal in te re st w ith
g reat p recisio n . O n e o f th e d efin in g c h a r a c te r istic s o f e x p e rim e n ta l resea rch is in te r v e n ­
tion by th e resea rch er in th e d a ta -g a th e rin g p ro cess. M o r to n and W illia m s (2 0 0 8 )
d esc rib e th is as 'playin g G o d ’. W e call th e data gath ered by in te r v e n tio n (o r tr e a tm e n t,
as it is so m etim es k n o w n ) ex p e rim e n ta l data.
In this chapter we discuss som e o f the m ain issues in ex p erim ental research and som e o f
the m ain ways in w hich the approach has been used in th e study o f p olitics. In the first part
o f this chapter we provide a b rief overview o f the b asic p rinciples o f ex p erim en tal design,
and how experim ents can be used to test causal hypotheses. W e th en go on to d iscu ss som e
o f the m ain ways in w hich experim ental research works in practice. E xp erim en ts can be c ar­
ried out in a n u m ber o f different locations. E xp erim ents can be carried out in a laboratory,
in the field, and by utilizing natural o ccu rren ces. In lab oratory ex p erim ents, the su b jects
(th e participants in the study) are recruited to a co m m o n location w here the in terventio n
takes place. In field experim ents, the in tervention by the research er takes p lace in real-w orld
environm ents, and subjects may not even know that they are p articipating in an exp erim ent.
This obviously raises som e ethical issues (w hich we discuss in detail in C hapter 7). By c o n ­
trast, natural experim ents rely on naturally o ccu rrin g events as in terventio ns rather than
interventions controlled by the researcher.
W ith the rise o f in ternet-based experim ents, the distin ction betw een som e o f these
approaches is b ecom ing m ore blurred (see M orton and W illiam s 20 0 9 ). In ad dition, surveys
are frequently used to carry out qu estion-w ording experim ents, w here som e respondents
are asked a question in a certain way and oth er respondents are asked the qu estion using
slightly different words. These types o f experim ents can be used to exam in e how resp ond ­
ents react to different types o f in form ation , and in form ation fram ed in different ways. We
discuss these types o f experim ents in C hapter 10.
We then discuss som e o f the m ain issues to do with the reliability and validity o f ex p e ri­
m ental research. This has to do with w hether or not we are con fid en t that a causal
relationship really exists between two variables (som etim es know n as internal validity) and
whether or not we are confident that our findings can be generalized to subjects outside the
confines o f the experim ent (som etim es known as external validity). We will also discuss
som e o f the practical and eth ical problem s associated with doin g different types o f e x p e ri­
mental research.

Basic principles of experimental design

Experim ental research relies on the use o f control groups (in w hich no in tervention takes
place) and experim ental groups (in which interventions do take place) and the ran d om
assig nm en t o f subjects (participants) to control and experim ental groups.
The control group provides a point of reference to w hich the effect o f the in tervention can
be compared. The random assignment o f subjects to control and experim ental groups ensures
 E X P E R IM E N T A L R E S EA R C H

(as far as possible) that the two group» are sim ilar to e ad i other (tee Chapter 10 on u rn
pbng). These groups are then treated in the tam e way in every respect apart from the inter­
vention that is earned out o n the experim ental group Any differences that are then observed
betw een th e groups on th e outcom e variable o f interest (the dependent variable) can then be
attributed to the intervention that took place
The classic version o f th e experim ental design com prues five step* (see de Vaus 2001 48):

1 two groups: on e group that is exposed to the intervention (the experim ental group) and
on e group that is not exposed to the intervention (the control group).
2. rand om allo cation o f subjects to the groups before the pre-test.

3. on e p re-interv en tion (p re-test) m easure on the outcom e variable,

4. on e in tervention (test/treatm ent),

5. on e p ost-interv en tion (post test) m easure o n the o utcom e variable.

Table 8.1 provides a conceptual overview o f this m ethod. The colum ns refer to the group» to
w hich subjects are random ly assigned. O nce subjects have been assigned to a control group
and an experim ental group, both groups are pre tested on the outcom e variable of interest,
the dependent variable (Y ) This provides a baseline measure which later results can then be
com pared to. In the next step, a treatm ent (or an intervention) is adm inistered to the
experim ental group but not the control group. The treatm ent refers to the key independent
v ariable (X ) o f interest. This is related to the causal hypothesis w huh we wish to test To see
w hether the hypothesis is confirm ed or not we need to tarry out a post test on both groups
on the dependent variable. If our hypothesis is supported, then we should observe that the
test statistic for our dependent variable Y has changed for the experim ental grtmp (which
received the treatm ent) but has not changed for the control grmip. The test for the effect of
the in tervention is th erefore carried out by com paring changes in the experim ental group
before and after the intervention to changes (if any) in the control group
This classic design is often sim plified in practice by dropping the pre test stagr Valid infer
ences about the causal significance o f the treatm ent variable can still be drawn so long as the
groups are large enough. The key thing is that subjects are randomly allocated to the experi­
mental and control groups. In effect this means that any differences between experimental and
control groups are random and will not account for group differences in outcomes (see de Vaus
2001: 60). W ith this sort o f design, rather than looking at the amount of change between the
groups, the analysis is based on the post test dtffrrmces between the groups. Because subjects
have been randomly allocated to their groups, the post-test differences should be the same as
the difference in change scores o f the experim ental and control groups in the classic set up.

TabteS.1 Experimental devfn __________________________________________________________________________

Control poup t i p w t iw m t y w f _______________

Pr*-t«t Measure o r outcome varabie ; v i M easur? c r o>Morne i* l

No treatmen; Treatmen«
InHfwntion

Post-t«* cn outccKne va<xibie (VI Meawrp c r *anab*f W

H O W T O D O R E S E A R C H IN P R A C T IC E

There is also a nu m b er o f issues to do with case s e le ctio n in ex p erim en tal research . The
first issue is to do w ith how su b jects (o r p articip an ts) are selected for th e study. The secon d
issue is to do with how su b jects are assigned— or allo cated — to co n tro l o r ex p erim en tal
groups w ithin the study. G enerally speaking, research ers w ho carry out ex p erim en ts are
m ore con cern ed with assignm ent than selection , particularly in lab oratory e x p erim ents.
It is the random assignm en t o f sub jects to groups, rather th an th e random selection o f
subjects in the first place, that is m ost im portant for testing causal hyp otheses. For th is rea­
son, m any laboratory exp erim ents do not rely upon representative or rand om sam ples o f
subjects. It can be very difficult, tim e-con su m in g, and expensive to select a rand om sam ple
o f participan ts from acro ss th e cou n try and th en transfer th em to th e lo catio n o f th e lab o ra ­
tory. It is often far m ore conv en ient to recruit people to the study w ho live near th e lab.
In deed, sin ce laboratory exp erim ents frequently take place on university cam pu ses where
acad em ics are based, the m ost conv en ient and accessib le sam ple is often an all-stud en t sa m ­
ple. This is precisely what a lot o f exp erim ental research relies upon. As long as the students
are random ly assigned to treatm ent and control groups, the fact that they are not rep re­
sentative o f the wider population does not un d erm ine the in ternal validity o f th e study (see
D ruckm an and Kam 2011 for an extended d iscu ssion on this top ic). H owever, it m ight c o m ­
prom ise the external validity o f the study and the extent to w hich the findings can be g en er­
alized, at least in so far as the findings can be generalized to non-stu dents.
We will illustrate how these designs are carried out in p ractice with referen ce to som e
specific exam ples. In particular, we focus on laboratory, field, and natural exp erim ents. The
m ain purpose o f this chapter is to provide a b rie f overview o f the three m ain types o f
experim ental design. Politics researchers have condu cted a great deal o f b oth laboratory
research (e.g. on the im pact o f cam paign com m ercials) and field ex p erim en ts (e.g. on the
effects o f canvassing, telephone calls, and direct m ail on voter tu rnout: G erb er and G reen
20 0 0 ; John and Brannan 2 008). And, w here possible, they have also exploited 'natural ex p er­
im ents’ (e.g. on the effects o f gender quotas on fem ale political representation). In the fo l­
lowing sections, we discuss each o f these experim ental design types in turn.

Laboratory experiments
In laboratory experim ents, the subjects are recruited to a com m on location where the
experim ent takes place. The laboratory experim ent is designed to ensure that the researcher
has as much control over the environm ent to w hich sub jects are exposed as possible. In this
way, the experim ental group and the control group can be exposed to exactly the sam e en v i­
ronm ent except for the experim ental intervention. A ccordingly, any difference that is
recorded on the post-test m easurem ent o f the dependent variable can be confidently attrib ­
uted to the presence (or absence) o f the intervention.
The controlled environm ent o f a laboratory is particularly well suited to exam in ing a wide
range o f political phenom ena that might otherw ise be difficult to investigate. In particular,
M orton and W illiam s (2008: 346) suggest that they have three m ain strengths as a setting for
experim ental research. Laboratory experim ents allow the researcher to have a great deal o f
control over what the subject is exposed to. In a natural setting, people m ight be exposed to
many different types of stimuli each day, so it is difficult to investigate what im pact if any
 E X P E R IM E N T A L R E S EA R C H

each specific stim ulus has on their attitudes or behaviour. In a laboratory, diflerero stimuli
can be m anipulated one al a tune, holding everything else constant. Th u allow s causal
hypotheses to be tested with far m ore precision. Second, laboratory experim ents also allow
the researcher to have a great deal o f control over what variables are manipulated, and even
to m anipulate variables that m ight be difficult to vary in the real w orld T h ird laboratory
experim ents allow the researcher lo create environm ents that simply don’t exist in the real
world. For exam ple, they can explore how decision-m aking is influenced by entirely new
voting systems.
However, laboratory experim ents are not without their problem s An example can help to
illustrate b oth the potential strengths and weaknesses of this approach. An area of political
research that has received a great deal o f attention is to do with the study o! m edia effects on
public opinion. In many ways, this type o f research lends itself well to the experim ental set
ting o f the laboratory, since the causal significance of media 'effects on political attitudes is
notoriously difficult lo unravel. M ost people are exposed lo m ultiple, com peting media m es
sages each day. so it is very difficult lo establish what messages m atter and how. O ne particu­
lar aspect o f m edia effects that has received a great deal of attention is the im pact o f negative
cam paignin g by political parties and, in particular, w hether or not attack ads have • d em o­
bilizing effects o n voters d uring an electoral campaign
O n e influential attem pt to answer these questions was carried by Stephen Ansolabehere
and his colleagues. Through a set o f innovative controlled experim ents in laboratory co n d i­
tions. A nsolabehere and colleagues (1 9 9 7 ) exam ine the extent lo which subjects who view
negative cam paign adverts are le u likely to say they will vote than subjects who view more
positive cam paign adverts. In the study, subjects were randomly assigned to different groups.
Su bjects in each group then watched identical public inform al ton broadcasts, Kach broad ­
cast had an advertisem ent break which consisted o f three adverts, and one of these adverts
varied across the groups. In one experim ental group, subjects were exposed to an advert
w hich was political in content and negative in tone; in another experim ental group, subjects
were exposed to an advert w hich was p olitical in content and positive in tone. Subjects in the
control group were not exposed to a political advert, and just watched a product advert. The
results o f th eir study are reproduced in Table 8.2. C ontrolling for a variety o f other factors
that influence w hether or not som eone votes. Ansolabehere and colleagues find that people
w ho are exposed to political adverts w ith positive political messages are about 6 percentage
p oin ts m ore likely to vote than people w ho are exposed to negative political adverts. They
thus conclud e that attack’ adds have a d em obilizing effect.
G iven the random allocation o f subjects to groups, any observed differences between the
experim ental and control group can therefore be confidently attributed to experim ental
intervention. T hese types o f experim ental study therefore have a high level o f internal validity.

TkblaSJ Exposure to negative campMgnmg and l**fchood o# voting

Control gro«?

Intervention Product advertisement Negative poitical aciven Pown» cwtflirai

Post-t«t(Y) 61 X TiVe+y to vote SCT tc .ofe MX lit»*, '.o

Souk* Ad^Wd fromAi**b^ww«ndcoa^u«s( 1997)

H O W TO D O R E S E A R C H IN P R A C T IC E

T h is m eans that we can b e con fid en t th at th e chan ge in th e ou tco m e v ariable really was
b rought about by th e key in d ependen t v ariable (th e in terv e n tio n ), rather th an so m e o th er
factor. That is. it really is th e variation in th e to n e o f th e p olitical adverts that is responsib le for
th e v ariation in w illingness to vote. However, lab oratory ex p erim en ts are often criticized for
lackin g external validity. Sin ce they take place in an 'artificial’ en v iron m en t, th eir findings
ca n n ot b e easily generalized to th e 'real w orld’. W e m ight b e confid ent that exp osu re to attack
ads has a d em obilizin g effect in th e laboratory setting, but d oes th is effect also h old in th e real
world? C ould attack ads b e responsible (at least partly) for a d eclin e in tu rn out?
E xp erim en tal lab oratory research allow s us to test hyp otheses un d er co n tro lled c o n d i­
tio n s d esigned to m ax im ize in ternal validity. However, ex ercisin g a high d egree o f co n tro l
ov er su b jects can often lead to a red u ctio n in th e ex tern al validity o f th e fin d ings and, in
particular, that v ariant o f ex tern al validity th at is called e c o lo g ic a l valid ity . Th is m eans that,
because th ey do not reflect a real-life situation, th e fin dings o f lab oratory ex p erim en ts m ay
not b e generalizable (o r extend ed ) to th e real w orld’. W h en we take people out o f th eir
natural env iron m en t and study th em in th e laboratory, we are ex ertin g som e c o n tro l over
th em . C onsequently, we are possibly lim itin g how m u ch we can gen eralize th e fin dings to all
people in natural settings. The lab oratory study m ay th erefo re lack ecolo gical validity
because th e c ontrolled env iron m en t in w hich it is c on d u cted is so un like th e real w orld that
w hatever results are ob tain ed will b e in applicable to people in n o n -lab o rato ry settin gs. T h e
qu estion, th en, is to w hat extent a lab oratory study is tru e to life. C ou ld it b e replicated any­
where. using natural settings and con d ition s? A n o th er reason why a lab oratory study m ay
lack ecolo g ical validity is b ecause o f reactiv ity . R eactiv ity oc cu rs w hen research study p ar­
ticipants alter th eir b ehav iou r as a result o f b ein g aware o f p articip atin g in a study. R eactivity
th reatens e colo gical validity because, when it o ccu rs, th e results o f th e ex p erim en t m ight be
£rncrah:abli' only to other people who are also being observed.
To illustrate som e o f th ese poin ts, we can return to Steven A nsolabehere's study o f'a tta ck
ads'. O n e co n ce rn with th e external validity o f the findings m ight b e th at, sin ce th e b ehav ­
ioural and attitudinal con sequ en ces o f exposure to th e different types o f cam paign ads w ere
m easured bv a surv ey condu cted shortly after ex posure o ccu rred, it is un clear w hat th e lo n g ­
term im pact o f this was and how it m ight translate in to actu al electoral ou tcom es. If, for
exam ple, in tention to vote d eclin es im m ed iately after exposure to negative advertisin g by 3
percentage points in the lab. does that im ply that the m u d -slin ging sen ate cam paign un der
study low ers actual turn out in the electorate by 3 percentage poin ts? As G reen and G erb er
\2003: 101 'i argue, in tention to vote is not the sam e th in g as actual tu rn ou t; n o r is o n e-tim e
laboratory exposure the sam e th ing as m ultiple attem pted ex posures in th e cou rse o f an
actual cam paign.

Field experiments

A lthough som e scholars prefer laboratory ex p erim ents to field ex p erim en ts b ecause th e lab
offers the researcher tighter control over the treatm ent and how it is presented to subjects,
others take the n ew that the generalizations from these types o f studv will be lim ited unless
treatm ents are deployed, and outcom es assessed, unobtrusively in the field (D ru ck m an et al.
rOOt*'. Because oi concerns about the artificial env iron m en t o f laboratory experim ents.
political scientists have carried out experu nents in reai-worid environm ents Ih e se held
exp erim ent* attem pt to reproduce as closely as possible the conditions under which ditfer-
ent political phenom ena occur, thus increasing the external validity or g rneraluabtlity o f the
findings. As G reen and G erb er (2003: 94) observe, held experim ents have two mam
strengths. First, random assignm ent ensures unbiased inlerence about cause and effect, and
second, the natural settings ensure that the results will tell us som ething useful about the real
world. Field experim entation can therefore be an incredibly powerful tool lor enabling
researchers to draw unbiased and externally valid causal inferences about different social
and political processes.
There is a long history in political research o f field experim entation or » h a t G reen and
G erb er (2 0 0 3 ) term controlled interventions into the political world. Ihev use the term
controlled in tervention since many of the early studies did not assign subiects to treatm ent
and control cond ition s on a purely random basis. However, in m ost other respects, they
closely resem ble field experim ents. An early exam ple o f this kind of controlled intervention
was carried out by H arold G osnell (1 9 2 7 ) o n voter registration and turnout in C hicago p nor
to th e 1924 and 192S elections. G osnell gathered the nam es, addresses, and background
in form ation o f thousands o f voting-age adults living in various C hicago neighbourhoods.
He th en divided these neighbourhoods into blocks, assigning (though not on a strictly ran ­
dom basis) certain b locks to the treatm ent condition of his experim ent, which consisted of
a letter urging adults to register to vote. C om paring the registration and voting rates in his
treatm ent and control group. G osnell found his letter cam paign to have produced a n o tice­
able in crease in political participation across a variety of ethnic and dem ographic groups.
A m o re recent and scien tific exam ple o f th is kind of experim ent was carried out by
Peter John and Tessa B ran nan (2 0 0 8 ) in M an ch ester prior to the 2005 British election
fohn and B ran n an rand om ly selected the nam es of 6 .9 0 0 people from the electoral register
o f a con stitu en cy in M anchester, lh e s u b le ts were then random ly allocated to three
groups (o n e co n tro l and two treatm ent), lh e researchers th en selected one treatm ent
group to receive a telephon e call (th e telephone group); and the oth er treatm ent group to
receive a visit (th e canvassing group). For both groups, the m ain purpose of the contact
was to persuad e th e citizen to vote, b oth by providing reasons why it is im portant and by
attem p ting to respond to any co n ce rn s about the voting process, l h e researchers had no
co n ta c t w ith th e co n tro l group. C om p arin g p articipation across the groups. |ohn and
B ran n an found that tu rn out was significantly higher am ong th e treatm ent groups than it
was a m o n g the co n tro l group, with b oth telephone and canvassing having m uch the sam e
effect on b o o stin g turnout.
These studies not only provide strong evidence about the causes ai turnout in the real
world, but they also provide strong evidence about what factors can increase turnout.
Experimental field research can therefore be a very effective way of evaluating different pol­
icy initiatives and pilot programs, since they provide a direct test of whether the initiative (or
intervention) brings about a direct change in the outcome of interest For this reason, field
experiments can often be used to great effect in collaboration with political parties, policy­
makers, or other agencies who are seriously interested in trying to understand the impact of
some type o f policy or intervention. For example, Wantchekon (2003) earned out a remark­
able field experiment on dientelism in Benin during a national election, in which he was «ble
to persuade political parties to randomize the types of appeal they made to n jten In different
H O W T O D O R E S E A R C H IN P R A C T IC E

villages (betw een p rogram m atic appeals and patron age based c lien telist appeals). H e was
able to do th is because th ere was c on sid erab le in terest am o ng th e lead in g parties in B e n in in
learnin g about th e effectiveness o f altern ativ e cam paign strategies.
W antchekon selected a n u m b er o f villages and th e in h ab itan ts w ere th en ex p osed to
purely clien telist p latform s and cam paign strategies by on e party and purely p rog ram m atic
public policy p latform s and appeals by the o th er party. The type o f appeal th at each p arty put
forw ard varied betw een th e different selected villages. T h e public policy m essage em p h a­
sized general policy goals, such as n ational unity and peace, erad icatin g co rru p tio n , allev iat­
ing poverty, d eveloping agriculture and industry, and p rotectin g th e rights o f w om en and
child ren . The clien telist m essage em phasized specific prom ises m ade to th e village for th ings
like gov ernm ent jo b s or local public goods, such as establishing a new local u n iversity or
providing fin ancial support for local workers.
C om p arin g th ese two ex p erim en tal groups to the c o n tro l group, w here th e platform was
not m anipulated and v oters were exp osed to the usual m ixed platform s o f th e parties,
W antchekon found that p arties w hich adopted the clien telist appeals ten d ed to b e m u ch
m ore successful than the p arties w hich adopted th e program m atic appeals, particu larly for
in cu m b en t cand idates. He also found that som e groups o f people respond ed to th ese appeals
in different ways, and that w om en had a stron ger p referen ce for p ublic g ood s m essages th an
m en.
A n o th er great stren gth o f ex p erim ental research is to try and u n cov er th e way in w hich
variables are related to each oth er when the d irection o f causality is u n certain . For exam ple,
a great deal o f acad em ic research has tried to unpick th e relationship b etw een new spaper
readership and p olitical attitudes and support. D oes the new spaper so m eo n e reads in fluen ce
th eir political preferences? O r do p eoples p olitical p references in flu en ce w hich new spaper
they read? These so-called ch ick en -an d -eg g p rob lem s can be very difficult to solve. P o liti­
cians certain ly believe that new spapers can be very influential, as do the new spapers th e m ­
selves. A fter the C onservative Party v ictory in the 1992 U K election , the Sun (a B ritish
tabloid) brazenly declared ‘It’s the Sun W ot W on it’. Yet hard em pirical ev iden ce to support
th is view is hard to com e by.
O ne way around the chick en-and -egg problem is to carry out exp erim ental research. G er­
ber, Karlan, and Bergan (2 0 0 9 ) report on an experim ent they cond u cted during th e 2005
Virginia gubernatorial election, designed to see if biased in form ation sou rces affected voter
behaviour. W ashington has two national new spapers, the Washington Post and the Washing­
ton Times. W hereas the Washington Post is generally viewed as a liberal new spaper, the Wash­
ington Times is widely considered to be a m ore conservative paper. G erb er and his colleagues
selected a large sam ple o f subjects about a m o nth before the election , and th en, after d is­
counting any people who already read the Post or the Times , random ly assigned sub jects to
one o f three groups. I h e first group received a free on e-m on th subscription to the Post, the
second group received a free on e-m on th subscription to the Times, and the third group
received neither. Subjects com pleted a survey at the begin ning o f the study (p re-test) and
after the election (post-test). G erber, Karlan, and Bergan found that subjects w ho had been
assigned to the (liberal) Washington Post were 8 percentage points m ore likely to vote for the
(liberal) D em ocratic candidate than those not assigned a free newspaper. The results th ere­
fore provide evidence that political biases in new spapers can affect voting behaviour and
political attitudes.
 I

However, held experim ents arc not without their problems either, and in tackling the
problem o f external validity often associated with laboratory experim ents, it u often argued
that they introduce a new problem to do with in ternal validity As we discussed in the previ
ous section, when researcher» exert a lot ol control over a study (such as when thev carry it
out in a laboratory), it creates a degree o f artificiality Ih is reduces the external validity o f the
experim ent and m akes it harder to generalize the findings However, when researchers attempt
to deal with this problem and carry out their experim ents in the held, they have less control
over the »tudy and so can be less sure about the causal significance ol the intervention lh is
can reduce the internal validity of the experim ent 'lb illustrate this pn»blem. n r «.an return
to the series o f studies conducted by the psychologist Robert Rosenthal and his colleagues
that we d iscussed in Chapter 3. Rosenthal and his colleague. Lenore |an>bson. conducted a
study designed to determ ine the im pact of teachers expectations on student perform ance
The research question that Rosenthal and lacobson ( 1V6M) addressed was IVtcs a teachers
expectations affect students’ p erform ance7 Iheir hypothesis was that favourable expect a
tions o f teachers will lead to an increase in students intellectual ability I he dependent van
able was intellectual achievem ent (m easured by IQ tests) I he independent variable was
teachers' expectations (th ose that resulted from the inform ation com m unicated to them by
the researchers). Ih e researchers conducted a pre test ol their dependent variable they had
IQ tests adm inistered to elem entary school students (grades I -6 ) I hey randomly selected
2 0% o f the students who were assigned to serve as the experim ental group, the rem aining
80% o f the students represented the control gntup Ih e teachers were told that the students
in the experim ental group were academ ic 'bloomers', and would show unusual academic
developm ent over the com ing year. At the end of the school year. Rosenthal and laio b ten
retested all o f the students. The IQ s ol students whom they had ih aractcrired as academic
'b loom ers (th e experim ental group) had statistically improved, and they had im proved at a
faster rate th an th ose o f the o th er children (th e control group).
This study appears to meet the conditions of a valid experiment In the ideal experimental
design, we are able to hold constant all variable* but one. and do so in a controlled unchang­
ing environment. But Rosenthal and lacobson did not have complete control over the sub
Jects and their environment, because the experiment occurred in a 'natural' setting.
Consequently, student performance may have been affected by variables other than the
teachers' expectations. Teachers may have introduced concrete changes that enhanced the
learning environment of those students who had been characterized as intellectual bloom­
ers', by, for instance, giving them extra or more difficult work, or choosing them to undertake
more enriching classroom or extracurricular activities. They may have communicated to
parents that their children were more advanced, thus bringing about change* in home envi­
ronments. There is thus sometimes thought to be a trade-off between ensuring high levels of
internal validity and ensuring high levels of external validity.
In order to have confidence in the results of an experiment, it is important to have high
levels o f both internal validity and external validity Given that internal validity can some­
times be a problem in Add experiments and external validity can sometimes be a problem in
laboratory experiments, some critic» argue that, since both designs are flawed there is not
much point in using either method But a more reasoned approach might be to try and
incorporate both types o f design together. For a study to really stand up, it should be possible
to try and examine it in both settings. As Elinor Ostrom (2007: 26-7) says. To teat theory
H O W T O D O R E S E A R C H IN P R A C T IC E

adequately, we need to use m e th od s th at tog eth er co m b in e ex tern al and in tern al validity.

O n e gains external validity in d oin g field research , but in tern al validity in th e lab o rato ry .

Natural experiments
Th e third m ain type o f ex p erim ental design is often referred to as a natural ex p erim en t. The
natural ex p erim en t relies on naturally o ccu rrin g events as in terventio ns rather th an in ter­
v entions controlled by the researcher. A ccord ing to M o rto n and W illiam s (2 0 0 9 ), w hen
natural in terventions o ccu r on a particu lar variable o f in terest, we can som etim es treat the
in tervention as if an ex p erim en talist m anipulated the variable. Even th ough in th is case the
researcher is not doing the in tervenin g, the approach taken w ith th e data is as i f th e research er
has. Such exp erim ents have been utilized by scho lars in a wide variety o f fields, in clud in g
political p articipation (Lassen, 20 0 5 ; K rasno and G reen 2 0 0 8 ), ballot design (G o rd o n and
H uber 2 0 0 7 ; C arm an et al. 2 0 0 8 ), p olitical psychology (van der Brug 2 0 0 1 ), eth n ic p olitics
( A brajano et al. 2 0 0 5 ), com parative p olitics (P o sn er 2 0 0 4 ), and bu reau cracy (W h itfo rd 2 0 0 2 ).
The attraction o f a natural experim ent is that it rem oves m any o f the problem s associated
with carrying out laboratory and field experim ents. First, sin ce a natural exp erim ent is
naturally occurring, there is not the problem o f artificiality associated with the laboratory
experim ent. A nd second, since the intervention occu rs independently o f the actio ns o f the
researcher, there are not the ethical issues which can create obstacles to doing field ex p eri­
m ents. However, for a naturally occu rrin g event to approxim ate a ‘proper ex p erim en t’, it is
essential that the principles o f experim ental research are not violated. In particular, the c o n d i­
tion o f exogeneity is crucial. For a natural experim ent to be valid, there must be variation in a
causal variable that is independent o f all oth er com petin g factors that m ay afFect the outcom e
o f in terest. In practice, this often difficult to achieve. N aturally o ccu rrin g in terventions are
often not entirely random . For exam ple, w hen a coun try changes its electoral system , we m ight
think that this represents a good opportunity to exam in e the im pact o f institutional design on
som e aspect o f political behaviour, such as turnout. We might even regard this as a natural
experim ent, since we have variation in our independent variable o f interest (electoral system )
while many oth er im portant factors appear to stay the sam e. But a problem arises if not all
factors do stay the same. For example, if the decision to change the electoral system is driven
in part by c oncerns about citizen apathy and low levels o f political support, then the variation
in our variable o f interest will be correlated with oth er im portant factors that drive turnout.
It is th erefore im portant to try and ensure as far as possible that the principle o f random
allocation is not violated. This can often be effectively achieved w hen gov ernm ent officials
m anipulate policies or when p olicies are based on near-rand om processes. U nder these c o n ­
d itions, researchers may be able to exploit near experim ental con d ition s to ex am in e a wide
variety o f different issues. For exam ple, M iller, K rosn ick, and Lowe (1 9 9 8 ) exam in ed the
effects o f ballot order (th e order in w hich candidates’ nam es appear on the b allot) on votes
for political candidates in the United States, by e xploiting the fact that, in certain O h io c o u n ­
ties, candidates’ nam es are rotated from one precinct to the next. M iller and colleagues
found that candidates at the top o f the ballot win an average vote share o f 2.5 percentage
points more, with the largest effects tu rn ing up in contests without an in cu m b en t contestan t
and where candidates’ nam es appeared without party affiliations.
 In an unfortunate natural experim ent. C arm an. M itchell, and John* ( 200«) exam ined the
effect! o f ballot design and voting instruction» on the num ber ol »polled vole» (ballot» which
had not been correctly filled in) in Scotland, by exploiting the fact that two different ballot
designs were used in the Scottish Parliam entary election» of 2007 (one which provided full
in structio ns and one w hich provided abbreviated instruction») C arm an and colleague»
found that, even after taking into account other potentially im portant factor», »uch as social
d eprivation, the predicted num ber o f »polled vote* was 65% greater for ballots with abbrevi
ated instructions. C arm an and colleague» call the experim ent unfortunate because the
d écision to abbreviate the ballot instructions was not an intentional one. and onlv cam e to
light after the election, when academ ics and the media began lo investigate why there wrre
so many spoiled ballots.
But natural experim ents can also arise out of specifk policy interventions In an unusual
natural experim ent from In d u , Rikhil Bhavnani (2009) exam ined the im pact of electoral
quotas on w om ens representation in the Panchayat ( village coun cil) In particular. Bhavnani
exam in ed whether electoral quotas for women alter women's chances of winning elections
even after the quotas are withdrawn. He was able to answer this question by exploiting the
fact that random ly chosen seats in local legislatures are set aside for women lor one election
at a tim e. Bhavnani found that the probability o f a woman winning office in « seat which had
previously been reserved for wom en (but was not anym ore) was fivr tim es higher than if the
constituency had not been previously reserved for women

Issues in experimental research

D espite the obvious strengths o f experim ental research for testing causal hypotheses, and
despite the many exam ples o f interesting and valuable research that has been earned out
u sing experim ental m ethods, experim ental designs are «till m uch let* com m on in the social
scien ces than in the natural sciences. Indeed, experim ental research is often viewed with a
great deal o f suspicion, and is som etim es thought to work better in theory than in practice.
T h is view is neatly sum m ed up by Arendt Lnpharl: The experim ental m ethod is the m ost
nearly ideal m ethod for scientific explanation, but unfortunately it can only rarely be used in
political scien ce because o f practical and ethical im pedim ents' (1971 684 5)
The practical and ethical obstacles to experimental research have received a great deal of
attention. One of the main practical objections is that, with respect to many key areas In
politics, researchers simply cannot randomly assign subjects to groups and manipulate the
independent variable in order to measure their impact on the dependent variable. It is one
thing to randomize whether or not voters are exposed to campaign material, but it is another
thing to randomize the foreign policies of governments. The really big political science vari­
ables, like culture, electoral system, level of economic development, ethnic heterogeneity,
cannot be manipulated. As Green and Gerber (2003) note, it is difficult to imagine how one
could randomly assign (»residential and parliamentary regimes for the purpose of evaluating
their relative strengths and weaknesses. Similarly, it is simply not pos*ible to go around ran
domlzing the determinants of war to see which given set of factors is most likely to cause two
countries to start a conflict For these reason*, it u commonly thought that political research
can never fully em brace experim ental m ethod*, and that it will rem ain som ething o f a niche
H O W T O D O R E S E A R C H IN P R A C T IC E

m ethod . Th is view is put forw ard by Sm ith (2 0 0 2 ), w ho suggests th at ex p erim en tal research
can only be cond u cted on a relatively sm all (an d m in o r) fra ctio n o f th e p o litical qu estio n s
that m ost in terest people.
A lthough th ere is som e truth to th is c riticism , it is perhaps w ise no t to ov erstate it. W ith a
bit o f im agination, the p rospects for in novative field ex p erim en ts are m u ch greater th an is
com m on ly assum ed. E xp erim ental research has b een carried o ut o n big issues, such as c rim e
and security (K eizer et al. 2 0 0 8 ), clien telism (W an tch ekon 2 0 0 3 ), p olice d iscrim in atio n
(H eussenstam m , 1971), and the in flu en ce o f th e m edia (G e rb e r et al. 2 0 0 7 ). M oreover, as
G reen and G erb er (2 0 0 3 ) argue, th ere is a balan ce that p olitical research ers need to strike
betw een answ ering 'b ig qu estio ns’ badly and ‘sm all q u estio n s’ well. As th ey put it: ‘If we
th in k o f the expected value o f research as b eing the p rod uct o f th e in trin sic value o f a research
qu estion tim es the probability that know ledge will be advanced by th e ev id en ce flow ing
from that research, this trad e-o ff com es in to sh arp er focus.’ E xp erim en tal research can
th erefore be an extrem ely effective m ethod for h elping to answ er n arrow tractab le q u estio ns.
Th ere are also eth ical issues that m ust be considered w hen co n d u ctin g ex p erim en ts. E th i­
cal issues are relevant to all types o f social inquiry, and we address th em in d etail in C hap ter
7, but they also have special sign ificance for ex p erim en tal research. As G erry Stoker (2 0 1 0 )
points out, eth ical issues tend to be m o re pronou nced in ex p erim en tal research th an in
purely observ ation al research. The act o f in tervenin g to chan ge so m eth in g raises m o re e th i­
cal issues than sim ply ob serv ing w hat is happ enin g, particu larly if that in terven tio n causes
harm or d istress to p articipan ts or relies on som e sort o f d eception. A sem in al psychology
laboratory exp erim ent carried out by Stanley M ilgram (1 9 6 3 ) is probably th e m o st fam ous
exam ple o f this kind o f d eception. It tested the w illingness o f particip an ts to h arm an o th er
person ‘while only follow ing orders’ by getting th em to ad m in ister electric sh o ck s to oth ers
who appeared to be suffering pain. The study raises a nu m b er o f eth ical issues. The first is to
do with d eception: p articipan ts were lied to. They did not really ad m in ister e lectric sh ocks.
The second is that it may have caused p articipan ts distress: p articipan ts m ay have b een upset
to learn that they were capable o f b asically killing so m eon e ju st b ecause a p erson in a p o si­
tion o f authority told them to do so. However, these eth ical challenges are not in su rm o u n t­
able; and there are a nu m ber o f strategies that can be em ployed to try and deal with th em .

Conclusions
Experim ental research has a number of strengths. It allows researchers to investigate causal hypotheses
with a great deal of confidence. There is no other research design that is quite so effective at establish­
ing causal connections, and the principles that underpin experimental research inform much of what
we do in political research. Even if experimentation is not used widely in political research (though this
is beginning to change), most of our research designs are, in effect, an effort to approximate the logic
of experimental design as closely as possible. In particular, the ideas of experim ental control permeate
much of what we do, even if it is dealt with in different ways To compensate for our inability to
completely control any one case, researchers in our field can study many cases through the use of
statistical analysis (see Chapters 14 to 16) We can also try and control for im portant factors using
comparative research (see Chapter 9)
But experimental research also has a great deal of practical and, specifically, policy relevance.
Experimental research can be a very powerful tool for m onitoring and assessing the im plementation of
different policy »atom The»» is cerumty t great deal ol potential tof academics and pohry makers to
*yort rTW* closely together to develop experimental studies And as Stoke* (2010) si^gests pursu**
this strategy wooId realty help to make political research more relevant to politics political actors a«1
citizens The benefit lor academ<s is that they *i**b ie to cdteo data to test hypotheses that they
would not cKherwrvf be able to investigate fo t example Wantchekon would not have bcvn abte to
carry out hn study on clientefesm in Benin without the support of the main political parties who
participated m his experiment The benefit for poi*cy makers <s that they wiU gel valid a r t reliable
evidence on what difference. 4 any th e* policy .ntervention makes Pohc.es can be very expensive to
implement and can aho be very ineffective at bringing about the dew ed change They aie thus upe lo»
experim ent*) research Indeed pM as new medicines and drugs a»e required to undergo rio to u s trial
before they are approved there is a lot to be said to« adopting the same standards It* pokey initiatives
The prospects for experimental research therefore look bright Experiments provide < r^orous and
scientific way to test causal hypotheses They aHo offer the potential to engage with policy makers m
order to assess the effectiveness of new policies With a little imagination experiments can be used to
answer a wide variety of Questions which tackle a range of theoretical and empincal puirtes

Questions
• What are the strengths of experimental research >
• Why is the experim ental approach not used very often vs political research?

• What are the practical and ethical obstacles to experimental reseanh' Mow can these obstacles
be overcome?
• Experimental research can only be used to answer smaA Questions which are of minor interest
to most people Do you agree?
• What kind of research Questions may be particularly wel suited to experimental designs7
• What are the relative strengths and weaknesses ol laboratory and field experiments>

Guide to Further Reading

Dructonan. j. N.. 0 P. Green. J H KiAHnaJd. and A. Lupia (ed»)(J0 U). C M > % Handbook
of Experimentaf PoNbcal Science (Cambridge Cambridge Unfrenfty Pré«)
This book contains a collection of W articles on experimental research in political science covering
topics such as internal and external validity and using students as subjects as we« as detailed chapters
on laboratory experiments M d eipeoments survey experiments, and more The collection aho
includes a number ol chapters that use experiments tn practice to anww Questions of urfntantrve
and theoretical interest

to
Morton. Rebecca 1. and Kenneth C- WWams (2009). From Nature the Lab The Methodology
of Experimental PoHbc*Sde»*ee and the Study of Cau»al*y(Cambridgr Cambrtd*»
University Prm^
Thh booh provides a detailed discussion ol the different ways m which experimental desyns can be
used in political research It provides a comprehensive account of the ddlerent issues researchers
must engage with, and is full of eng*gwig examples to iRustrate thew pomts Is essential raadmg for
anyone Interested in knowing more about experimental research

Stoker. G. (2010). Txptorto* the Prom*» of Experimentation In Poflttof Science: Mtero-

Foundtfional m*%W» m d Policy M m n c« '. M W OMfltc * 100-1*.
This article examines the ways in which experimental research can be mad w ma*mream po*bcal
science and the potential the approach hat to encage with pohcy-retevam research
H O W T O DO R E S E A R C H IN P R A C T IC E

References
Abrajano. M . J. Nagler. and R. Alvarez (2005), ---- , Dean Karlan, and Daniel Bergan (2007), 'Does
'Race Based vs. Issue Voting. A Natural the Media Matter? A Field Experiment Measuring
Experiment'. Political Research Quarterly 58(2): the Effect of Newspapers on Voting Behavior and
203-18. Political Opinions'. American Economic Journal:
Ansolabehere, Stephen and Shanto Iyengar (1997), Applied Economics 1:35-52.
Going Negative: How Political Advertisements Geva, N. and A. Mintz (1997), Decision-Making on
Shrink and Polarize the Electorate (New York: War and Peace: The Cognitive-Rational Debate
Free Press). (Boulder, CO: Lynne Rienner Publishers).
Baumrind, D. (1985), 'Research using Intentional Gordon, Sanford C. and Gregory A. Huber (2007),
Deception: Ethical Issues Revisited', American The Effect of Electoral Competitiveness on
Psychologist 40:165-74. Incumbent Behavior', Quarterly Journal o f
Bhavnani, R. (2009), 'Do Electoral Quotas Work After Political Science 2 (2 May): 107-38.
They Are Withdrawn? Evidence from a Natural Gosnell, Harold F. (1927), Getting-out-the-vote: An
Experiment in India', American Political Science Experiment in the Stimulation o f Voting (Chicago:
Review 103(1): 23-35. University of Chicago Press).
Carman, Christopher, James Mitchell, and Robert Green, D. and A. Gerber (2003), T h e Underprovision
Johns (2008), The Unfortunate Natural of Experiments in Political Science'. Annals o f the
Experiment in Ballot Design: The Scottish American Academy of Political and Social Science
Parliamentary Elections of 2007', Electoral Studies 589:94-112.
27(3): 442-59. H abyarim anaJ., M. Humphreys, D. Posner,
de Vaus, D. (2001), Research Design in Social Science and J. M. Weinstein (2007), W hy Does
(London: Sage). Ethnic Diversity Undermine Public Goods
Druckman, D. (1994), 'Determinants of Provision?'American Political Science Review 10V.
Compromising Behavior in Negotiation: A 709-25.
Meta Analysis', Journal of Conflict Resolution 38: Henrich.J., R. Boyd, S. Bowles, C. Camerer.
507-56. E. Fehr, and H. Gintis (eds) (2004), Foundations
Druckman, J. and Cindy D. Kam (2011), 'Students of Human Society (Oxford: Oxford University Press).
as Experimental Participants: A Defense of the Heussenstamm, F. K. (1971), 'Bumper Stickers and
"Narrow Data Base”', in James N. Druckman, the Cops', Society 8: 32-3.
Donald P. Green, James H. Kuklinski, and John, P. and T. Brannan (2008), 'How Different are
Arthur Lupia (eds), Cambridge Handbook of Telephoning and Canvassing? Results from a
Experimental Political Science (Cambridge: "Get Out the Vote" field experiment in the British
Cambridge University Press).
2005 General Election', British Journal of
---- , D. P. Green, J. H. Kuklinski, and A. Lupia (2006),' Political Science 38: 565-74.
The Growth and Development of Experimental
Keizer, K., S. Lindenberg, and L. Steg (2008), The
Research in Political Science', American Political
Spreading of Disorder', Science 322:1681-5.
Science Review 100(4) (November): 627-35.
Krasno, Jonathan and Donald Green (2008), 'Do
Frechette. G..J. Kagel, and M. Morelli (2005),
Televised Presidential Ads Increase Voter
Behavioral Identification in Coalitional
Turnout? Evidence from a Natural Experiment',
Bargaining: An Experimental Analysis of Demand
Journal o f Politics 70(1): 245-61.
Bargaining and Alternating', Econometrica 73(6):
1893-937. Lassen, David (2005), 'The Effect of Information
on Voter Turnout: Evidence from a Natural
Frohlich, N. and J A Oppenheimer (1992), Choosing
Experiment', American Journal o f Political Science
Justice: An Experimental Approach to Ethical Theory
49:103-18.
(Berkeley. CA: University of California Press).
Lijphart, A. (1971), 'Comparative Politics and the
Gerber. A and D. Green (2000). T he Effects of
Comparative Method', American Political Science
Canvassing, Telephone Calls, and Direct Mail on
Review 65(3): 682-93.
Voter Turnout A Field Experiment', American
Political Science Review 94(3) (September): Lodge. M.. K. M. McGraw, and P. Stroh (1989), 'An
653-63 Impression-Driven Model of Candidate Evaluation',
American Political Science Review S i: 399-419.
 :m

Mdgram. Stanley (1963). Behavwrai Study of Roienthat K and LJacobton( 19681 PygNiaban
Obed«nce Journal of Abnormal end Social *»the Claaroom (New *or* ho* . fcnrftart 4
Piythotogy 67(4^ 371-8 Wei non)
Miier joanne M )on K K/otnxl and L w i Low* Simon A and T S«Atf> ii00 1) Habermaa e>the
(1998). The impact of Candidate Nam* Oder
on Cloction Outcomes Pubfcc Opmon Quarter^ tftecu of Dehberation Pofcticaf Piyehetagy 22
62 291-330 809-26
Morton. R B and fc WJkarm (1999) information Sm » . Bogan M (¿002) VioUd We Make PohtKal
Afymmetrm and Simultaneous « « w Science More oI a Vience or More About
Sequential Voting1American totocal Saence Potator PS Potac* Scwncr and Potted 3S<2)
favM»93 51-67 (>une> 199-201
— (2008) -Experimentation * Pofabcal Science Stoker G (2010) lip*onr«| the Promng of
mjanet Boa-Steflename«r. OawdCofcer. and Upenmercabor n PotMcal Saence Micro
Hanry Brady (edt).The Oxford Handbook of foundabonal tra^to and Pohry Relevance
PokOCal Methodology (Oxford Oriord LkwaruTy PoteKal Stud*» S8 300 19
Pret») vender Bn«. W (2001) Perceptions. Opeuont and
---- (2009). From Noam (o the Lot Tht Party Preference* n the I ace ct a Ra«J World
Methodology ofCapenmMef floptical Science and Event Chemoby* at a Natural Upenment ei
(he Study of CauialKy (Cambridge Cambridge Poetical Pty^twtofy > w n tl of Theoretical
UrMverary Pr«u) f lo it n lX l) S3 »
Ottrom. ihnor (2007). Why do we Need Laboratory Wantchekon L (2003) CLentatam and VoOng
Experiment! m Political Sconce? paper Behewor WtoWPetactSS 399 422
pretented at the 2007 American Pofctnal Scwnce Wallenberg Martin P and Craig L Brians (1999).
Atsociation annual meeting. Ch*a*o II Negative CampMgn A d v isin g OemotMluer or
Power. D (2004). The Political Sahence of CufcnJ Mobduer? Ameocan PofctKal Scwnre Brtwe 93
Difference Why Chewea and Tumbi*aa are ABwa 891-900
m Zambw and Advenanet a» M alw ' Ammten Whaferd A (2002) Oerentrahulion and
Pokocal Science « m m 9B(4) (November) Potbcal Control of the Bttfeaucracy >Mnat af
529-45 Theoretical M e n 14 167 93
 m Comparative Research

Chapter Summary
This chapter exam ines the principles of com parative research design and discusses
the issues and problems associated w ith different aspects of the approach. In doing
so. w e pay special attention to the issue of case selectio n , the com m on sources of
error that are associated w ith com parative research, and what can be done to try and
avoid or m in im ize them. W e exam ine:

• case selection;

• case study;

• small-N com parison;

• large-N studies;

• selection bias;

• measurement.

Introduction
T h is ch a p ter ex a m in es th e p rin cip les and uses o f co m p a ra tiv e resea rc h . T h e c o m p a ra ­
tive m e th o d is on e o f th e m o st w idely used m e th o d s in p o litic a l research and is fr e ­
qu en tly used to study a w ide range o f p o litic a l p h e n o m e n a , fro m d e m o c ra tiz a tio n to
civil war to in stitu tio n a l design to p u b lic policy. T h e sc o p e for co m p a ra tiv e research is
alm o st lim itless. H ow ever, o fte n th e term ‘c o m p arativ e p o litic s’ is used ra th er vaguely to
sim ply m ean th e study o f ‘fo reig n ’ c o u n trie s o u tsid e th e U K and U SA (v an B iezen and
C a ra m a n i 2 0 0 6 ). But th e term ‘c o m p arativ e m e th o d s’ really im p lies so m e th in g m o re
than th is. As A rend L ijp h art (1 9 7 1 ) w rote, th e term ‘c o m p arativ e p o litic s ’ in d ic a te s th e
how o f the analysis, but not th e what. C om p a ra tiv e p o litics is first and fo rem o st a m e th ­
od o lo g ica l a p p roach , rath er than an area o f su b stan tiv e in terest. W h at u n ites and d efin es
com p a ra tiv e stud ies th ere fo re has little to do with th e sort o f q u estio n s asked , but rath er
with the m eth od used. Follow in g Peter M air (1 9 9 6 ) and van B iezen and C ara m a n i
( 2 0 0 6 ), we u n d erstan d th e co m p arativ e m e th o d p rim a rily in te rm s o f th e ru les and
stand ard s and p roced u res for id en tify in g and ex p la in in g d iffe re n c e s and sim ila ritie s
betw een cases (o fte n , but not always, d efin ed in term s o f c o u n tr ie s ), u sing c o n c e p ts that
are applicable in m ore than on e case or cou n try.
Ih ere are many different ways in w hich this can be done. Indeed, the com parative m ethod
actually involves a num ber o f different m ethods. These m ethods can be d istingu ished p ri­
m arily in term s of how many coun tries (or cases) are com pared (denoted by the letter N ),
and how the cases for analysis are selected. Both aspects o f case selection are very im portant.
Broadly speaking, there are three main approaches. There are large-N studies (involving the
analysis of many c a *« ), small-N studies (involving the analysis of a small number of eases,
typically 2. 3.4. but with no real upper limit) and single N studies (otherwise known as case
studies). In this chapter we discuss these different comparative approaches, provide an over
view of the methodological issues that they raise, and evaluate the strengths and weaknesses
of each approach.

The principles of comparison

Comparison, either directly or indirectly, forms an essential part of almost all empirical
analysis. In a sense it is impossible not to compare We do it all the tune, whether we are
aware of it or not. Any discussion of cause and effect relies on comparing the presence of one
factor against the absence of another. Ihis has led some scholars to define the comparative
method rather broadly. For example, tu bnel Almond (1966) equate* the comparative
method with the scientific method generally, and says it makes no sense to speak of • com­
parative politics in political science since if it is a science, it goes without saving that it is
comparative in its approach! According to this understanding, all empirical research is com­
parative. We might even regard the statistical analysis of survey data as a form of compara
Uve analysis, where the unit of analysis is an individual. For example, we can compare young
people with old people to see if there is an association between age and turnout (see Chapter
15). But the comparative method more usually refers to the study of social and political
aggregates rather than individuals In particular, comparative politics is frequently based on
comparing differences (or similarities) between countries. But it can also be used to compare
differences between units within countries, such as regions, organisations. political parties,
pressure groups, or whatever.
Like all methods, comparative methods are only as useful as the purpose to which they are
put. If the research project has clear theoretical goals, then comparative research can be a
valuable tool for helping to answer questions about different political phenomena. But if
there is little theoretical justification for the comparison, or the tusuhcafion i* not made
clear, then comparative research is unlikely to be of much use developing nr testing theory
It is very important then to think carefully about the purpose of the comparison, and to
avoid falling into the trap of )ust comparing different countries for the sake of it. without any
clear theoretical goals guiding the selection of countries to analyse.
Broadly speaking, comparative methods can be used in three main ways: (i) to apply exist
ing theory to new cases; (ii) to develop new theory or hypotheses: and (iii) to test theory.
One o f the key strengths of comparative research is that it helps to broaden our intellec­
tual horizons. We can use comparison to see if what we think is a self-evident truth in one
context also works in the same way in a different context. Comparison therefore has a
number of advantages over the single country case study, and helps us to guard against the
twin dangers of what Rose ( 19 9 1) has labelled foist untquenas and false untvrrudum. False
uniqueness emphasizes the specificity of the case, entirely ignoring the general social forces
at work, and does not move beyond 'thick description! Problems to do with false uniqueness
can sometimes be found in area studies, where researchers etnphasire bow unique—or
exceptional—their chosen country of analysis is, and seal themsebes off from wider engage­
ment with what is being written about in other countries. By contrast false umversalism
H O W T O D O R E S E A R C H IN P R A C T IC E

assu m es th at th e th eo ry tested in on e coun try/con text will be eq ually applicable to oth er

coun tries. For exam ple, w hat is foun d to m atter in th e U nited States, w h eth er it is th e im p o r­
tan ce o f ‘resources for political p articip ation , ‘th e asso ciatio n b etw een so cial d iv ersity and
tru st’, or ‘th e im pact o f lob b y groups on gov ernm ent p o licy ’ is often assu m ed to refer to u n i­
versal truths about the way p olitics works, rather th an co n tex t-sp ecific fin d ings p articu lar to
one country. The o n ly way we can ever establish uniqu eness or univ ersalism is th rou gh c o m ­
parison.
An im portant fu n ction o f com p arison is th erefo re descriptive. D o th e sam e th ings w ork in
the sam e ways across d ifferent contexts? W e can use com p arison to apply th eo ry developed
in one context (o r case) to an o th er co ntext. R okkan (1 9 6 6 : 19) refers to th is as a ‘m icro rep ­
lication, w hich is designed ‘to test out in oth er n ational and cultural settin gs a p rop osition
already validated in one settin g’. T h is type o f approach is o ften d escrib ed as th eo ry -in fir m ing
or th eo ry -co n firm in g (L ijp h art 1971). For exam ple, on e in fluen tial th eo ry in studies on
political behaviou r w hich em erged in the U nited States is that rich people and w ell-educated
people tend to be m ore politically active b ecause th ey possess th e ‘civic skills’ th at reduce
som e o f the obstacles to p articipation (V erba et al. 1995). This th eo ry is no t fram ed in a
co n text-sp ecific way that is p eculiar only to th e U nited States (w here th e research w as carried
out), and so m ight be thought to apply across different con texts and be applicable to oth er
coun tries. O n e useful purpose o f com p arison is th erefo re to try and establish w hether th is is
the case or not, and w hether the general th eory proposed really is a gen eral th eory o r not
(false universalism ). Subsequent research carried out in different cou n tries has found that
there is in fact substantial variation in term s o f how im portant th ese so-called ‘civ ic skills’ are
for political participation. In Britain , for exam ple, although they appear to be im portant,
they are less im portant than in the U S (P arry et al. 1992). A nd fu rth er afield, in India, they
are not im portant at all, and if anything, it is those w ho are the m ost disadvantaged in society
who tend to participate the m ost (Yadav 2 000; M ich elutti 2 008).
C om parison can therefore serve a useful descriptive purpose, and b e used to m ap the
extent to w hich different th eories apply (or can be replicated) acro ss different co ntexts, and
the extent to w hich different social and political p henom ena will o cc u r acro ss different
countries. But at the sam e tim e descriptive com p arison only takes us so far. It is on e th ing to
show that there is variation across contexts, or that som e accou n ts o f p olitical p articipation
work better in one cou n try than in ano ther; it is an o th er th ing to try and develop and test
theories in order to explain these differences.
It is to th is end that com p arativ e m e th od s are m o st freq u en tly used. C o m p a riso n can
be used to try and generate and test hypotheses about the factors that explain these varia­
tions. The advantages o f com parative cro ss-n ation al analysis are m anifold. First, co m p ara­
tive m ethods can be used to merge different levels o f analysis, and link in ternational,
national, and dom estic factors in order to explain a particular political p henom enon . For
example, by com paring in dividuals nested w ithin coun tries, we are able to exam in e the
im pact o f country-level factors (such as institutional arrang em en t) on individual behaviour.
For example, com parative research on voter support for extrem e right-w ing parties shows
that w hether or not som eone votes for the far right is influenced both by individual factors,
such as a persons attitudes towards im m igration and national factors, such as w hether the
person lives in a country that uses proportional representation or not. W ithout com parison,
we would not be able to exam ine the im portance o f these 'aggregate level’ factors, and so our
explanation of a particular phenomenon would be partial and underdeveloped. Compan
»on thus allows the researcher to move beyond a purely individualistic explanation of polit
ical phenomena.

How we compare
The comparative method is primarily a method for case selection l)o we compart many of
few cases? And if we compare few cases, then on what basis do we select them? In the follow
ing sections we discuss the three main forms of comparison (case study, small N. and large
N). In particular, we focus on how we select cases for analysis, the different ways in which
this can be done, and some of the problems and trade-olfs that are involved with each
approach. Whereas small-N studies are better suited to generating hypotheses than testing
hypotheses, large-N studies are able to do both.

Cue study
The great advantage of the case study is that by focusing on a single case, that case can be
intensively examined. For this reason, case studies remain one of the main lormi of research
in Comparative Politics (Geddes 2003; Cieorge and Bennett 200S) (>n the face of it. it might
appear to be a contradiction in terms to talk about a comparative case study, tim e it u not
immediately obvious where the comparison comes from if theTc »s only one case involved
But good case studies are nearly always situated in a comparative context Ihey add rest
theory or issues that have wider intellectual relevance, use concepts that are applicable to
other contexts, and may even seek to make inferences that apply to countries beyond the
original case. For example. Arend Lifphart's ( 1977,1999) studies of consociationahsm in the
Netherlands. Robert Putnams ( 1993) study of social capital in Italy, and Paul Brass's (2003)
study of ethnic violence in India, are all regarded as classic comparative studies', even though
each is only based on the analysis of a single case What sets them apart is that they have all
managed to develop arguments and theories that are relevant to many other contexts, and
not only say something meaningful and interesting about the case in question, but also say
something meaningful about general political phenomena.
Good case studies therefore possess two important characteristics The first is that they
say something Interesting and meaningful about the case that is being studied. For example,
a case study of ethnic violence in India should help to shed light on the sources of conflict in
India, and contribute to the academic literature that has been written on the subvert wtth
reference to the particular case. That is. the findings of the study should be internally valid
However, a good comparative case study should also aim to say something more general
and engage with «rider academic debates that might be applicable to other contexts and
other cases. For example, does the study only shed light on the sources of ethnic violence in
India, or could the Implications of the argument and analysis also be relevant for helping to
explain the sources of ethnic conflict in other parte of the world, such as in Africa? This
Involves setting the case in comparative context, and proposing theories or explanations that
ire also externally valid (at least hypothetically).
H O W T O D O R E S E A R C H IN P R A C T IC E

A lthough th ere is a w idespread assu m p tion that case studies represent a type o f qu alitativ e
research, th is is no t strictly true. C ase studies can b e based o n a w ide v ariety o f d ifferent d ata-
gath ering strategies, such as interview s, surveys, ethnography, fo cu s groups, h isto rical d o cu ­
m ents, policy d ocu m en ts, and speech es, to nam e ju st a few. Indeed, m an y sin g le-co u n try
case studies are based on the qu antitative analysis o f survey data, for exam ple on v oting
b ehaviou r in B ritain , or public attitudes tow ards the en v iro n m en t in G erm any, o r on public
attitudes tow ards im m igrants and eth n ic m in o ritie s in th e N eth erlan ds. In th e follow ing
chapters, we will explore som e o f th ese m e th od s o f data co lle c tio n in m o re detail, so th e
focus h ere is m o re co n ce rn e d w ith issues o f case selection .
It is the e xtent to w hich the case study engages w ith w ider com p arativ e th em es that m akes
it com parative. It can do th is in on e o f tw o m ain ways. First, it can apply th eo ry d eveloped in
one con text to an o th er co n text in ord er to assess w hether th e origin al th eo ry ‘w orks’, thus
add ressing issues to do w ith false u n iqu eness and u n iversalism . O r, secon d , it can seek to
develop a new theory, and gen erate hyp otheses that can be applied to oth er cases. However,
because the case study is o nly based on the analysis o f a single case, a case study can n o t really
be used to d irectly test theory. For this reason, we should always treat the in feren ces m ade
from a case study with a certain degree o f caution. They m ay sou nd plausible, but until they
have been tested on a large nu m b er o f cou n tries, we have no way o f know ing.
Th ese two m ain types o f case study can be used in a v ariety o f ways (L ijp h a rt 1 971; G er-
ring 2 0 0 4 , 2 0 0 7 a ; Yin 2 0 0 9 ). Broad ly sp eaking, we can d istin g u ish b etw een case studies
that (i) provide d escrip tive co n te x tu alizatio n ; (ii) apply ex istin g th eo ry to new co n tex ts;
( iii) ex a m in e ex cep tio n s to th e rule; and (iv ) gen erate new th eory. Issues to do w ith case
sele ctio n and w hat m akes a good co u n try to study depen d upon th e p u rp o se o f th e case
study. You m ust provide a ration ale for why th e sp ecific case or set o f cases you selected ,
from am o n g all th ose in th e larger p op u lation , were ch o sen . R esearch ers m ay sele ct cases
because th ey are critical (to testin g a th eo ry ), or revelatory (reveal relatio n sh ip s w hich
ca n n o t be studied by o th er m ean s) or unusual (th row ligh t on ex tre m e cases) (Y in 2 0 0 9 ).
O n e fu n ction o f the case study is con textu al d escrip tion . Purely d escriptive case studies
do not seek to advance or apply theory, but rather provide a th ick d escrip tion o f a p a rticu ­
lar event or phen om en on in a p articu lar country. These types o f study are m o st closely
associated with eth nography (see C hap ter 12) and can be a valuable sou rce o f data for se c ­
ondary analysis. G iven that th ese types o f study do not engage with th eoretical literature, it
does not m ake m uch difference how the cases are selected for analysis, sin ce th ere is no
attem pt m ade to use the case to m ake any w ider in feren ces. However, th is approach is not
used very often.
M ore com m only, case studies are used to apply theory from one con text to see if it still
holds in ano ther context. R esearch in Politics and IR is full o f exam ples o f this type o f
research. In the field o f electoral behaviour, m any concepts have b een exported from research
on the United States to oth er coun tries. Perhaps one o f the m ost im portant o f these ex p orts’
is the concept o f party identification or party attachm ent, w hich states that som e voters
develop an attachm ent to a political party which is so strong that it form s a part o f their
identity (like a religious, national, or eth n ic identity) and in turn shapes the way in which
they interpret political in form ation and events. This concept was developed in the United
States in the 1950s by Angus C am pbell and his colleagues at the University o f M ich igan, and
was then applied to the British context by Butler and Stokes in the 1960s (see C am pbell et al.
I960; Butler and Stokes 1969). It hat since been applied to many other countries in order to
explain a wide variety of political behaviour
Case studies are an incredibly powerful tool ior examining whether concepts and ihcunes
travei. and whether (or not) they work in the um e way in cate» other than where they were
originally developed. When choosing a cate study to analyte lor these purpose*. Geddes
(2003) »uggesti two main criteria tor ca»e »election. Ihe first is that the case should be rep
resenlalivc of the domains of the theories they are intended to test Ihat is. the «.ase should
provide a fair test of the theory. One must therefore identify the universe of vases to which
the theory (or hypothesis) applies. I he second criterion is that cases used lor testing argu
ments should be different from the cases from which the arguments were induced In this
way. each case study provides a new test of the theory, and contributes to the gradual a«.cu
mulation of knowledge in favour (or against) the theory
Case studies can be used to examine specific outliers or deviant cases, and to examine
countries (or cases) that do not fit existing theory and are known to deviate from established
generalizations. Ihis type of case study is still based on developing existing theory, and seeks
to uncover the reasons why the theory does not apply to the case in question Case studies u4
this type have often focused on countries that are thought to be exceptional lor one reason
or another, such as why there is no socialist political party in the United States, even though
experience of industrialization suggests there should be one. or why there is no democracy
in rich countries like Saudi Arabia, when modernization theory suggests there should be
Rogowslu (2004: M2) notes the central importance ol a deviant case in which the outcome is
unexpectedly wrong: ‘A powerful, deductive, internally consistent theory can be seriously
undermined. . . by even one wildly discordant observation ' George and Bennett (200S 114-
15) suggest that such deviant cases may also yield information about previously unidentified
causal mechanisms that may also operate in other cases (see also Brady and (..oilier 2004).
Case studies of this type can therefore be theory confirming or theory tnhmung ( Liiphart
1971). A theory-confirming case study is one which lends to suppon the original theory, and
shows that it has wider applicability beyond its original context. It thus adds further empin
cal support to the original theory. By contrast, a theory infirmtng case study is one that does
not support the original theory, and shows that it is found wanting in some way. However, it
Is not possible to go as far as to say that a theory-infirraing case study can be used to re^ecl
the original theory, since just because the theory is found not to apply lo a single case, it does
not mean that It does not (or ««rill not) apply to other cases.
For this reason, the idea of what has been termed a crucial case (Eckstein 1975) is contro­
versial. A case is crucial if it can be regarded as crucial for the confirmation or disconfirmation
of a theory. Case selection is based on the likelihood or unlikelihood of the outcome of inter­
est occurring (Bennet and Elman, 2006). According to Gerring (2007b: 232) a most-likely
case is one that, on all dimensions except the dimension o4 theoretical interest, is predicted to
achieve a certain outcome and yet doe* not it is therefore disconfirmatory A least-likely case
is one that, on all dimensions except the dimension of theoretical interat, is predicted not to
achieve a certain outcome and yel does sa It is confirmatory' The ’least-likely’case study relies
an what Levy (2 0 0 2 : 1 4 4 ) has labelled the Srnatra inference: if the theory can make it here, tt
can make it anywhere. However, despite providing a tough empirical test for the theory in
question, it is debatable whether any one-off case can ever be regarded as critical to confirm­
ing or diaconhrm im i a theory (see Gerring 2007b for a detailed d»cu»s»on)
H O W T O D O R E S E A R C H IN P R A C T IC E

But perhaps the m o st im p ortan t use o f th e case study is to gen erate new theory and h y p o th ­
eses in areas w here no th eory exists. The gen eration o f new hyp otheses and th eo ry often
develops from th e analysis o f p articu lar case studies. Su ch case studies can be o f great th e o r­
etical value, as th e ex am ples m e n tio n ed earlier attest. T h ey ten d to w ork b est w hen th e case
selected for analysis provides w hat N aroll (1 9 6 6 ) calls a so rt o f ‘c ru cial ex p e rim e n t’ in w hich
certain variables o f in terest happen to be present in a special way.
Recently, th ere has been a great deal o f discu ssion ab ou t th e m erits o f u sing case studies
to investigate causal m ech an ism s u n d erp in n in g th e asso ciatio n b etw een tw o variables
(B rad y and C o llier (ed s) 2 0 0 4 ; G eorg e and B en n ett 2 0 0 5 ). This is often d escrib ed as p ro cess
tra cin g . It involves look ing for evid en ce o f th e pressures, in centives, m o tiv ation s, and deci-
sion -m a k in g calculus in any given in stan ce o f actio n (G eorg e and B e n n e tt 2 0 0 5 ), often
throu gh the use o f elite interview (see C hap ter 11). It thus refers to a m e th o d o f data co lle c ­
tion and analysis, and although it is frequen tly associated w ith th e case study, it can also be
used for sm all-N analysis. Indeed, qu antitative analysis often tests for th e p resen ce o f causal
m echanism s as well.
C ase stud ies are widely used in com p arativ e p o litics. T h ey have a n u m b er o f key
stren g ths. They allow for a detailed analysis o f p o litical p h e n o m en a, w ith rich tex tu al
d escrip tio n . B ecau se o f th is th ey do no t ten d to o p erate at a high level o f th eo retical
a b stra ctio n and th ere is a g ood m atch b etw een th eo ry and ev id en ce. T h ey th us g en erally
have relatively high in tern al validity. H owever, d espite th ese stren g th s, th ey also suffer
from a nu m b er o f fairly ob v iou s lim itatio n s, p articu larly in term s o f how far th e fin dings
from on e case study m ay b e g en eralized to o th er co n texts. It is no t always apparent w hat
lessons, if any, m ay be learned from a case study un til th e sam e type o f analysis is repeated
elsew here. Indeed, even when apparently sim ilar hyp otheses have b een tested in d ifferent
co n texts, it can still be difficu lt to draw any firm co n clu sio n s ab ou t w h e th er or no t the
results co n firm (o r in firm ) the origin al th eo ry un less th e hyp otheses have b een tested in
exactly the sam e way in each case. D ifferences in th e results m ay be dow n to differen ces in
m ethod , m odel sp ecification (w hich factors are con sid ere d ) or m e asu rem en t (how key
con cep ts are o p eratio n alized ). O n e obv ious way to reduce som e o f th ese p rob lem s, th en ,
is sim ply to in crease th e nu m b er o f cases consid ered in th e analysis. T h is type o f co m p a ra ­
tive research is know n as sm all-N co m p ariso n , w here N refers to the n u m b er o f cases that
are exam ined .

Small-N comparison
A lot o f com parative research is based on sm all-N sam ples. These typically involve th e c o m ­
parison of two or m ore cases, and although there is no real upper lim it to the nu m b er o f
cases that may be exam ined, it norm ally does not exceed m ore than a dozen or so. Sm all-N
com parative studies of this type are widely used in p olitical research, and in som e respects
arc the quintessential form o f com parative analysis. Indeed, Lijph art (1 9 7 1 ) d efines the
com parative approach in term s o f sm all-N analysis (in co ntrast to the statistical approach
of large-N studies). C lassic sm all-N studies include M ichael L ew is-B e ck ’s (1 9 8 6 ) four-
nation study of econ om ic voting in Britain, France, G erm any, and Italy; Sey m our M artin
l.ipset s ( 19^9) study ol the social requisites o f d em ocracy in Europe and Sou th A m erica;
 COMI

Iheld a Skocpoi's (1 9 7 9 ) study o f revolution in Russia. 1-raiKe. and O u n * B am n g to n

M oores (1 9 6 6 ) study o f dem ocracy and dictatorship, and Alm ond and Verbas (1963) studv
o f c iv ic culture in the U K. U nited States. G erm any. Italy, and M exico In addition, there are
nu m erous area studies that rely on sm all-N com parisons to exam ine a variety ol topics,
such as eth nic p olitics in A frica (Posner 2004). party systems in East fcurope (Kitschelt et
al. 1999), and d em ocratic stability in Latin A m erica (M ainw aring and Scully (eds) 1995). to
nam e but a few.
Comparisons of this type, based on the systematic analysis ol several countries, have
many advantages, lhey allow tor the detailed in depth analysis ol the case studv. but at the
same time provide greater scope for contextualuation As such, they leave out neither the
particular nor the general. However, although these small N studies can be extremely useful
for generating new ideas and answering questions, they are also fraught with problems, and
the careless selection of cases for analysis can lead to the production of very misleading
results. When we only analyse a few cates, it makes a great deal of difference which cases we
analyse. Indeed, if we wanted to, it would be very easy to fix the results to get almost any
answer that we wanted by just handptcking three or tour cases that appear to confirm our
hypotheses and ignoring all the other cases which do not.
Therefore to make an argument convincing, a number of important considerations need
to be taken into account, and the justification for case selection needs to be made clear.
If cases are selected in a haphazard wav or without care and attention, then there is the very
real risk of introducing selection bias This is something that the researcher needs to be
acutely aware of. To a greater or lesser extent problems associated with selection bus affect
all small-N comparisons, since we are only dealing with a limited number of cases We there­
fore have to pay special attention to how we select cases in order to try and minimize the nsk
of it occurring.
Small-N comparisons are based on the strategic selection of cases, appropriate to test
a theory. Which cases we select therefore depends upon the theory we are interested tn
testing. But at the same time, there are a number of general principles that guide case
selection. We are rarely if ever interested in comparing two (or more) cases that are
exactly the same in every respect oc completed different tn every respect . Besides the almost
practical impossibility of finding two such countries, even if we could, there would be
little analytical point to such an exercise, since the comparison would be redundant (we
would basically just be looking at the same thing twice). Rather, we are generally interested in
comparing cases that are similar in some respects and different in other respects. We try to
take advantage of the similarities and differences we see in the real world to see if the pat­
tern of observed variations is consistent (or not) »nth our theoretical expectations In this
way. we can test hypotheses that move beyond the purely domestic to examine how the
characteristics of countries and the relationship between countries influence political
phenomena.
There are two main approaches that havr been used to select cases for small N studies
These are the Most Similar Systems Design (MSSD) and Most Dtffcrmt Systems Design
(MDSD) (Prxeworski and Teune 1970). These are based, respectively, on Mills* Method of
Difference and Method of Disagreement. Each approach makes slightly different use of the
observed similarities and differences that we observe between countries, though the logic of
each is quite similar. We discuss each of these in detail below.
210 H O W T O D O R E S E A R C H IN P R A C T IC E

Most Similar Systems Design

The M ost Sim ilar System s D esign is based on selectin g co u n tries th at sh are m an y (th e o r e ti­
cally) im p ortant ch aracteristics, but differ in on e cru cial resp ect (related to th e hyp othesis o f
in terest). This sort o f approach is frequen tly used in area studies (Przew orski and T eu ne
1970: 3 3 ), w here cou n tries w ithin a specific region, such as A frica (o r Latin A m e rica, East
E urope, o r S ou th Asia, etc.), are sim ilar in m any im p o rtan t respects, p erhaps to do w ith th eir
culture, history, level o f e co n o m ic developm ent, and social stru ctu re. T h e shared ch a ra cter­
istics act as a control in ord er to test w hether th e cru cial difference b etw een th e co u n tries is
associated with the v ariation in the d epen d ent variable.
Table 9.1 provides a conceptual ov erview o f th is m ethod . The co lu m n s refer to th e selected
cases and the rows refer to the variables o f interest. W e can see th at th e selected cases are
sim ilar w ith respect to v ariables A, B, and C. These v ariables con stitu te th e co n tro l, and c a n ­
not m eanin gfully e xplain th e variation in th e d epen dent variable, sin ce th ey are th e sam e for
all cases. However, im portantly, the selected cases do vary on variable X . Th is is the variable
related to our key hypothesis, w hich we th in k helps to explain th e variation in our d epen dent
variable Y. If our hyp othesis is supported, th en we should ob serv e that o u r d epen d ent v a ri­
able Y varies in accord ance w ith th e v ariation in X. Thus the presence (o r a b sen ce) o f th e key
explanatory factor is able to accoun t for the variation in th e o u tcom e o f interest.

Example 1: the political salience of ethnic cleavages

W e can illu strate the logic o f th is sele c tio n m e th o d w ith an exam p le. A cla ssic study o f
th is type was carried out by D an iel P osn er (2 0 0 4 ) on th e p o litical sa lie n c e o f e th n ic
cleavages. P osn er w anted to ex p lore th e facto rs th at are asso ciated w ith th e p o litica l s a li­
en c e o f cu ltu ral d ifferen ce, and in p articu lar, th e fa cto rs asso ciated w ith w h e th er d iffer­
ent eth n ic groups w ere p o litical allies or rivals. H is h y p oth esis is th at th e p o litica l sa lien c e
o f eth n ic d ifferen ce is related to the p op u lation size o f th e g roups in q u estio n . W h en the
e th n ic groups are b o th large, th ey c o n stitu te a viab le base for p o litic a l su p p o rt, and so
p o liticia n s will m o bilize the groups, and the cleavage th at d iv ides th em w ill b e co m e
p olitically salient. If, how ever, the groups are b o th sm all, th en th ey w ill go u n m o b ilized
and the cleavage that separates th em will rem ain p o litically irrelev an t. The cu ltu ral d if­
ferences betw een the groups will still ex ist, but th ere will be no p o litica l im p o rta n c e
attached to th em .

Table 9.1 Most Sim ilar Systems Design (MSSD)

Case 1 Case 2

A
 U un g a M ott Sim ilar System s D esign. Posner exam ined the political salience o f ethnic
d ifference* betw een tw o tribes. the C hew as and Tum bukas. who constituted a large propor­
tion o f the electorate in Malawi, but only a im all proportion o f the electorate in Zambia
(Table 9 .2 ). In doing so. Posner was able to control for a wide variety of factors that previous
research suggested was im portant. The key thing to rem em ber is that the selection o i cases
is based, in theory, on the independent variables, not on the d ependent variable. The control
variables are the factors that the eth nic groups in each country share. Ideally, our control
variables should include v ariables that are theoretically im portant, since failure to include an
im portant factor may result in o m itted variable b u s. O m itted variable bias, as the nam e
suggests, o ccu rs when we fad to take into account im portant explanatory factors Ihe con se­
qu ence o f this is that factors that are not really causally im portant may appear im portant
because they are associated with (or have even influenced) the unm easured m issing variable
This can cause spu riou s asso ciatio n (see C hapter 16) The extent to which we can have con
iid en ce that our results will stand up and be robust is therefore dependent in part upon how
adequately we have m anaged to c ontrol for o th er potentially im portant factors
Posner s study controls for four im portant factors, variables A to 1). which are all thought
to in crease the salience o f eth nic differences. Variable A refers to the racial differences
betw een the two groups, variable B refers to the cultural differences between the two groups,
variable C refers to the historical developm ent o f the political system, and variable 1) refers
to the experience o f colonialism . Since these factors are the sam e for each of the countries,
they act as a control. If we control for theoretically im portant variables, and our variable o(
in terest appears to matter, th en we can have greater confidence in the results. Importantly,
then, the variable that relates to the hypothesis we are interested in testing varies across our
cases. In Posner s exam ple, variable X refers to the site o f the eth nic groups.
Following Posner. our expectation is that the variation in the size of the ethnic groups (our
explanatory factor) will be associated with variation in the salience of ethnic deavages (the
thing we are trying to explain), lhat is. when the ethnic groups are both large, the cleavage
that divides them will be politically salient and so they will be rivals, but when the groups arc
both small, the cleavage that separates them will remain politically irrelevant and so they
will be allies. If we observe this pattern, then our hypothesis is supported. If we do not. then

IhtotoM MoilStmriarSystemsDastgn(MSSD) ihtpoMcalsalwnc*of «ihmcdfflctvncn

Zambia Malawi

A
r .v fjii!
B B
* vimiiaMie-,
C
D
E*C t-i Crucial
Low poJiKJl Vl!*<xe ^'g*1 p o llK i1v»i**xe

Notts A • racial d M arm xitN 'M w nroupt

B • cultm l dtffcrancn bctwwn p m p i

E • (0 • «hrec g r a *» are tmdO (1 • |rw*» t n larp)

H O W T O D O R E S E A R C H IN P R A C T IC E

our hyp othesis is no t sup ported . From Table 9.2 we can see th at th e hyp othesis does in deed
receive em pirical sup port, and that th e p olitical salien ce o f eth n ic differen ces is h ig h er in
M alaw i, w here the eth n ic groups are b o th large, th an it is in Z am b ia, w here th e groups are
b oth sm all.
O f course, w hen we are d ealin g with on ly a few cases, th ere are always goin g to b e lots o f
potentially im p orta n t d ifferences betw een th e cases, and it is im p o ssib le to com p letely c o n ­
trol for ev erything that m ight be th eoretically relevant. In th is sen se, as P rzew orski and
Teu ne (1 9 7 0 ) suggest, it is easy to ov erd eterm in e th e depen d ent variable. L ijp h art (1 9 7 1 )
d escrib es th is as th e ‘too m any variables, too few cou n tries’ p rob lem . W h ich ev er cases we
select th ere will always be a nu m b er o f d ifferences betw een th em , w hich m akes it very diffi­
cult to establish w hich are the cru cial d ifferences and w hich are not.
For exam ple, suppose we were interested in exp laining th e sou rces o f d e m o cratic stability.
O u r hyp othesis m ight be that d em ocratic stability is related to the in stitu tio n alization o f
p olitical parties in society. That is, w here p olitical p arties have deep ro o ts and are e m bedded
w ithin society, d em ocracy tends to be m o re stable. To test th is hyp othesis, we m ight c o m ­
pare India and Pakistan, sin ce th ey share m any im p ortan t sim ilarities. T h ey shared th e e x p e­
rience o f colon ial rule under the British , b ecam e in d epend en t at th e sam e tim e, and were
both p oor with low literacy rates and a largely agricultural econ om y at the tim e o f d e m o c ra ­
tization. In m any respects, th en, they w ere sim ilar on th eoretically im p ortan t in d ep end en t
variables. These shared characteristics are unable to explain any d ifferences betw een th e two
coun tries, such as why dem ocracy has persisted in India o ver the past 50 years, but struggled
in Pakistan, with frequent episodes o f authoritarian rule. O u r key in d ependen t v ariable o f
party system in stitutionalization does, however, appear to vary in th e expected way betw een
th e two coun tries. W hereas the C ongress Party in India had a stron g base right acro ss the
country, pen etratin g even the m ost rem ote rural villages, th e M uslim League in Pakistan was
m uch m ore weakly institutionalized.
However, there are also m any p otentially relevant d ifferences betw een the c o u n tries, to do
with leadership (N ehru versus Jin nah), social dem ography and religion, in stitutio nal c a p ac­
ity, in ternational support, and so on. It is difficult to establish, using pair-w ise com p arison ,
w hich o f these differences is the cru cial causal difference, although som e candidates will be
more plausible than others. It is also d ifficult to establish what com b in atio n o f these variables
is im portant. If we were to add m ore cases to our analysis, we m ight be able to elim in ate
som e o f the spurious differences that we observe betw een the two cou n tries, but in reality
there are rarely enough cases to find the right com bin atio n o f sim ilarities and d ifferences we
need to exactly test our theory.

Most Different Systems Design

I h e lo g ic of M ost D ifferen t S y ste m s D e sig n is to se le c t c a s e s th a t a re d ifferen t in m o s t
re s p e cts an d o n ly s im ila r o n th e k ey e x p la n a to ry v a ria b le o f in te re s t. T h u s, r a th e r th a n c o m ­
p a rin g c o u n tr ie s th at a re v e ry s im ila r to e a c h o th e r, w e c o m p a r e c o u n tr ie s th a t a re v e r y d if­
feren t lro m e a ch o th e r, h e n ce th e n a m e . W ith th is a p p ro a c h , th e o b s e rv e d d iffe re n ce s on
im p o rta n t v a ria b le s a ct as a c o n tro l in o r d e r to test w h e th e r th e c ru c ia l s im ila rity th at th e
c o u n tr ie s sh a re is a s so cia te d w ith th e d e p e n d e n t v a ria b le .
TaM»9J Most Mlcrent Syttwrn Dntgn (MOSD)

Cti* 1 Casa 2 Cut

Table 9.3 provide* a conceptual overview of this m ethod. O nce again, the colum ns relcr to
the selected case* and the row* refer to the variable* in the analyst*. This tim e we can see that
the cases are different with respect to variable* A. B, and C These variable* constitute the
control. Im portantly, our case* are all sim ilar with respect to the key independent variable of
interest. X. If our hypothesis is supported, then we should observe that our dependent van-
able Y u also sim ilar across our cases. Ihus the presence o f the key explanatory factor is
associated with the presence o f the o utcom e o f interest

Example 2: revolutions
We can illustrate this approach with the example o f Theda Skocpol*s( 1979) famous study o f revo
lutions. w hich was based ui part upon a Most Different Systems Design, com paring the causes of
social revolution in three very different countries. Russia (1 9 1 7 -2 1 ). France (178 7 -1 8 0 0 ). and
C hina ( 1 9 1 1 -4 9 ) (Table 9.4). (See Geddes (2003) for an extended discussion o f th u example.)
Broadly speaking, Skocpols main argument is that social revolutions occur when external m ili­
tary threats provoke a split in the ruling elite and peasant com munities take advantage o f th u split
and rev olt Accordingly, countries which experience state breakdown and peasant revolt will also
tend to experience social revolutions, even if they differ in many other important respects.
In order to test th is hypothesis, Skocpol selected three countries very different in so cia l
econ om ic, and political terms, but sim ilar in term s o f having autonom ous peasant popula­
tions, and sim ilar in that they all experienced periods of external m ilitary threats w hich pro­
duced splits in their ruling elite. The key thing to rem em ber with d a s * * M D SD s is that the

1*Ma9.4 Mott Different Systems Oevtn (MSSD) the cauiet of revolution

France Ituarta CNna

A Sot A %Ot A 1 0*e*aii

B Mot B Not B
c Not C No* C
X■
EM* spl't x E:-?espf.t V E:'«* tpM 1 C'-j o *
v RrvcJ jto '- f v*mU"Ty
Y RivoJutior v HiVOhjTKT'

Noan. A • political tyitfm

I • tacial and o A val conmt
C « «conomlc cm M o ra
H O W T O D O R E S E A R C H IN P R A C T IC E

selection o f cases should be based on the independent variables, not on th e d epen dent variable.
In the case o f Skocpol’s study, it is debatable w hether th is decision rule was followed (we will
return to this later). In this exam ple, then, the control variables are th e factors that are different
for each country, such as th eir social, econ om ic, and political b ackgrounds. The key in d ep en d ­
ent variables o f in terest are the sam e for each coun try: they all had elite splits, peasant uprisings
and state collapse, and crucially, the o u tcom e variable is the sam e for each cou n try as well: they
all experienced social revolutions, thus con firm in g the initial hypothesis.

Issues in small-N research

The issue o f case selection in sm all-N studies is extrem ely im portant b ecause the cases we
choo se to analyse can influence the answers we get to a p articular q u estion, and this is p articu ­
larly the case w hen we are only analysing a few cases. W h en this occu rs, it is know n as selection
bias. If we do not th ink carefully about case selection, we can end up w ith answ ers that are not
very robust, or are even som ew hat m isleading. That is, we will m ake faulty inferences. The best
way to avoid this happening is to c ondu ct a large-N study and select cases using random p rob ­
ability m ethod s or to take a full census o f cases. But this is not always a viable option. To a
certain extent, then, selection bias, or at least the risk o f selection bias, is always going to be
a danger in sm all-N studies, particularly when the study is designed to try and shed light on a
broader p henom enon , and m ake som e sort o f generalization, however tentatively put.
D iscussion about the relative m erits o f different strategies for selectin g cases in sm all-N
research has been one o f the m ost controv ersial and hotly debated issues in com parativ e
research (see King et al. 1994; Ragin 2 0 0 0 ,2 0 0 8 ; G eddes 20 0 3 ; C o llier et al. 2 0 0 4 ; G eorg e and
B enn ett 2 0 0 5 for d ifferent sides o f the debate). O n e o f the m ain strategies that has been p ro ­
posed is to select cases accord ing to th eir ch aracteristics on the in d epend en t variables (see
King et al. 1994; G eddes 20 0 3 ). Th is strategy has been elevated to alm ost law -like status:
never select on the dependent variable. Som e have even gone as far as to say that selectin g on
the d ependent variable is a type o f'in feren tia l felony’ (A chen and Snidal 1989). This p osition
has been challenged by som e scholars (see C ollier and M ah oney 1996; D ion 1998), but
before outlin ing the criticism s, which we will return to later, it is worth discu ssing the logic
behind the initial d ecision rule.
T h e a rg u m e n t g o e s th a t if w e se le c t o n th e d e p e n d e n t v a ria b le a n d d o n o t p a y s u fficien t
a tte n tio n to th e in d e p e n d e n t v a ria b le s, w e m a y u n w ittin g ly e x c lu d e i m p o rta n t fa c to rs fro m
o u r a n a ly sis, o r fail to c o n tro l fo r th e m a d e q u a te ly . T h is m e a n s th a t w h at lo o k s lik e a s tro n g
c a u sa l re la tio n s h ip b e tw e e n tw o va ria b le s m a y in fact b e s p u rio u s . F o r e x a m p le , s u p p o s e w e
w an t to e x a m in e th e re p re s e n ta tio n o f w o m e n in p a rlia m e n t. W e m ig h t c h o o s e th re e c o u n ­
trie s w h ich h a ve a re la tiv e ly h igh p e rc e n ta g e o f w o m e n in p a rlia m e n t, su c h as S w ed en ( 4 5 % ),
N o rw a y ( 4 0 % ), an d F in lan d ( 4 0 % ), an d seek to id e n tify w h a t fa c to rs th e y h av e in c o m m o n .
S in ce all th re e c a se s h ave e le c to ra l s y s te m s b a se d on p ro p o rtio n a l re p re s e n ta tio n , w e m ig h t
c o n c lu d e th at th is is an im p o rta n t fa c to r, a n d in fe r th a t P R s y s te m s a re c o n d u c iv e to p r o m o t ­
in g th e re p re s e n ta tio n ol w o m e n in p a rlia m e n t. Bu t th is in fe re n ce w ou ld b e w ro n g . Ih e re a re
m a n y o th e r c o u n tr ie s w ith PR sy s te m s th a t d o n o t h a ve a h igh p e rc e n ta g e o f w o m e n in p a r ­
lia m e n t, su ch as Brazil (8 % ), lu rk e y ( 9 % ), a n d U ru g u a y ( 1 5 % ). 'I h c p e rc e n ta g e o f w o m e n in
p a rlia m e n t in S ca n d in a v ia m a y th e re fo re h ave little to d o w ith th e e le c to ra l s y s te m , an d m o re
 C O M P A R A T IV E R E S EA R C H

to d o with the strategy o f political recruitm ent by p olitical elites or the political culture o f the
general public, both o f w hich may be exogenous to institutional design However, because we
have not taken these factors into accoun t when selecting our cases, it is difficult to establish
w hich, if any, are im portant. The problem with not selecting cases on the independent v ari­
able, then, is that it m akes it very difficult to control for potentially im portant variables.
W hen research is th eory-driv en , or at least guided (see C hapters 5. 6. and 7). we should
always have som e hu nch that guides our research, and accordingly we should design our
research in ord er to p rovide an appropriate test’ o f this hunch. Ih erefo re. if our hunch is that
th e type o f electoral system is an im portant factor in d eterm inin g the representation of
w om en in parliam ent, we would want to select cases that allow us to explore this, which
m eans selecting cases accord ing to the independent variables. In sm all-N analysis, this can
be d one e ith er by selecting cases that are very sim ilar to each oth er but vary on the key inde
pendent variable (M ost Sim ilar System s D esign) or selecting cases that are very different to
each oth er but are sim ilar on the key independent variable (M ost Different Systems D esign).
B oth th ese approaches allow us to control for potentially im portant factors, and so reduce
the risk o f o b tain in g spurious relationships.
W hen carryin g out M D SD , it is easy to fall into the trap o f selecting cases on the d epen d­
ent variable. Indeed, M D SD is often in correctly characterized as being based on this strategy.
For exam ple, if we were interested in exploring the cause o f revolutions, we might be tempted
to select th ree cou n tries w hich have experienced revolutions and see whal they have in c o m ­
m on. Th is is, in essence, what Skocpol did in her study o f revolutions, which has been c riti­
cized by G ed d es (2 0 0 3 ). A lthough S kocp o ls study has been incredibly influential, it is also
widely regarded as b eing based on spurious in ference, in so far as the m ain causes o f revolu­
tion do not appear to be generalizable to oth er relevant cases (see G eddes 2003). G eddes
argues that there are lots o f cases that do not confo rm to Skocp o ls theoretical expectations,
w here there have been in stances o f external threat w hich did not lead to a social revolution.
A lthough in som e ways this is part and parcel o f sm all-N research, there are also som e sp e­
c ific issues to do with case selection that may have contributed towards this problem in
S k o cp o ls study. G eddes argues that, in essence, what Skocpol did was select her cases on the
d ependent variable. That is. she didn’t select them because they had all experienced peasant
uprisings and state collapse, but because they had all experienced a social revolution.
However, selection on the dependent variable can just as easily occur in Most Similar Sys­
tem Designs. For example, if would not be wise to select two cases that have populist leaden
and two cases that do not and look to see how they are different, because if you do. you are just
as likely to introduce the same kind of selection errors. In both cases, you risk leaving out or
not properly considering important explanatory factors, which will bias the results. The best
way of ensuring that this doesn't happen (or at least is less likely to happen) is to base the selec­
tion on the independent variables. If prior theory suggests, for example, that institutional
weakness is an important source of populism, then it is a good idea to take this into account
when you select your cases. Thus, whether you are carrying out MSSD or MDSD, in theory the
values of the cases on the dependent variable should not be a criterion for selection, and all
selection decisions should be based on the values of the cases on the independent variables
only. Of course, in practice, we often cannot help but know what value the dependent variable
takes for each of our cases, but the important thing is that this knowledge should not influence
our selection decisions (and if it does, we risk introducing greater selection error).
H O W T O D O R E S E A R C H IN P R A C T IC E

W ith e xploratory research, th ere is som etim es a ju stificatio n for selectin g on th e dep en d ent
variable if the study is in a g enuinely new field and p rior research d oes not suggest any im p o r­
tant explanatory factors. U nder th ese rare c ircu m stan ces, it is n o t p ossible to c o n tro l for th e o ­
retically im portant in dependen t variables b ecause n o such backgro u nd literature exists.
However, in stances o f new research areas that have absolu tely n o co n n e c tio n to previous
research are exception ally rare. A nd even w here th ey do exist, th e research er gen erally still
poses som e guiding th em es or foreshadow ed p roblem s (see C hap ter 12) that th ey w ant to
explore, and so these factors should also feed in to the selection o f cases.
In recent years, there has been a bit o f a qualitative b acklash against th is principle o f selec­
tion, perhaps partly in response to the overtly statistical language in w hich K ing et al. (1 9 9 4 )
originally fram ed their decision rule (see C ollier and M ah oney 1996; Ragin 2 0 0 0 ,2 0 0 8 ; C ollier
et al. 2 004; 8 5 - 1 0 2 ; G eorge and B en n ett 2 005). In essence, what these qualitative m e th o d o lo ­
gists argue is that w hen certain qualitative m ethodologies are em ployed, particularly process
tracing, it is perm issible to select on the d ependent variable. A ccording to G eorge and B en n ett
(2 0 0 5 ), p rocess tracing is fundam entally different from m ethods that rely on co-v ariation , and
th e m ethod ’s contribu tion to causal in feren ce arises from its evidence on th e process w hich
con n ects the cause and the outcom e. Accordingly, to scholars like Collier et al. (2 0 0 4 ) and B e n ­
nett and Elm an (2 0 0 6 ), it is th erefore not susceptible to selection bias, w hich m eans that the
principles o f case selection discussed above are largely irrelevant, and that cases m ay b e selected
on the basis o f the dependent variable (see B enn ett and Elm an 2 0 0 6 for furth er discu ssion on
this point). This is fine up to a p oint, at least in term s o f d escribing what h appens w ithin a given
case (internal validity). But as soon as we are interested in c om parison and m aking w ider in fer­
ences (external validity), it is a good idea to take the issue o f case selection seriously, or else we
can easily end up with m isleading results (see C hapter 10 on sam pling principles).
A second criticism that is often m ade against the principle o f never selecting on the
dependent variable is that there is a d istin ction betw een necessary and sufficient causes.
A ccordingly, selecting on the dependent variable, and look ing to see w hat the c ou n tries have
in c o m m on , can reveal the necessary con d ition s (or causes) for an ou tco m e o f in terest, even
if these causes are not in them selves sufficient to produce or cause the ou tco m e o f interest
(see D ion 1998; B raum oeller and G o ertz (eds) 20 0 0 ; and G o ertz and Starr 2 0 0 3 ). For ex a m ­
ple, with reference to Skocpol, D ion (1 9 9 8 ) d istingu ishes betw een a claim based on a suffi­
cient cond ition for som eth ing to happen, such as ‘state crisis leads to social revolution, and
a claim based on a necessary cond ition for som eth in g to happen, such as ‘social revolutions
arise only if there is a state crisis’. W hereas state crisis w hich does not lead to social revolu­
tion is evidence against the sufficient p roposition, only a social revolution w hich did not
experience state crisis is evidence against the necessary cond ition . Therefore, in order to test
hypotheses based on necessary causes, it is p erm issible to select cou n tries w hich all have the
sam e outcom e on the dependent variable and see what they have in com m on .
Although this is a valid point, it m ight be argued that it is often useful to know w hether
the cond ition is only necessary or w hether it is n ecessary and sufficient, in w hich case sele ct­
ing on the dependent variable is not o f m uch help. It m ight also be argued that it is difficult
to distinguish between a necessary condition and a spurious cond ition . For exam ple, sup­
pose we arc interested in dem ocracy, and want to investigate the claim that ‘d em ocracy only
persists over the long term in m o n o-eth n ic societies’. If we select on the dependent variable
and only choose countries which have been d em ocratic for m ore than 20 years, the chances
 C O M P A R A T IV E R E S EA R C H

are that they will be eth nically hom ogen eous, confirm ing our hypothesis. But this does not
m ean that eth n ic h o m ogen eity is a necessary cause for dem ocratic persistence. It is in fact
not a cause at all. and the apparent coin cid en ce between dem ocratic stability and ethnic
hom ogen eity is spurious, driven by the association betw een eth nic diversity and econom ic
developm ent (see C hapter 16). I h e only way we can try and knock down spurious relation­
ships is to take seriously the idea o f c ontrolling for theoretically im portant factors, and the
m ost effective way in which we can do this is with reference to selecting cases on the basis of
the ind epend ent variables.
'Ih is brings us to a second related point, w hich is often confused with the principle o f not
selecting o n the d ependent variable. Many people believe that it is a bad idea not to have any
variation in th e dependent variable. That is, it is desirable to have negative cases as well as
positive cases, such as cases where there has not been eth nic conflict, as well as cases where
there has been eth n ic conflict. Ih is suggests that a M ost Sim ilar System D esign is preferable
to a M ost D ifferent System s D esign, since the latter is often unfairly criticized as selecting
cou n tries with the sam e outcom e variables. But the logic underlying this supposition is
faulty. Selectin g cases so that there is variation in the dependent variable is still selecting
cases on the dependent variable. Ih e problem s o f selection bias do not com e from the fact
that th ere is no variation in the dependent variable, but from the fact that im portant in de­
pendent variables have not been properly controlled for.
The key then is to control for theoretically im portant independent variables in order to test
a specific hypothesis. W hether the approach used is M D SD or M SSI). failing to control for
theoretically im portant variables can introduce error into the analysis and risk producing
unreliable results. 'Ihe risk o f this o ccu rrin g is no greater when case selection is associated
with sim ilarities in the outcom e variable than when it is associated with differences in the
outcom e variable. However, these problem s can be com pounded. The w orst-cate scenario is
to select on the dependent variable and choo se sim ilar types o f countries with a sim ilar o u t­
com e variable, or conversely to select different types o f countries with a different outcom e
variable. The form er is perhaps m ore com m on in com parative research and is a particular
problem in area studies, where the dependent variable is the sam e for all selected cases. For
exam ple, studies on the rise o f the ‘New IxfV in Latin Am erica may seek to identify the co m ­
m on factors un d erpinning the electoral success o f leftist leaders like Hugo C hiv ez in Vene­
zuela. Evo M orales in Bolivia, and Rafael C orrea in Ecuador. Given that these countries share
many sim ilarities, and that m any o f these sim ilarities may also be shared with other countries
in the region w hich have not experienced the sudden rise o f ‘New Left’ leaders, it would be
easy to ascribe causal significance to sim ilarities between the three countries that are not
really that im portant at all.
When we are dealing with just a small number of cases, selection bias is always likely to be
a problem. We cannot get rid of it completely, and the harsh reality is that we cannot aim to
make robust generalizations when we have a sample of just two or three cases. For this rea­
son small-N studies are generally better at generating theory than testing theory. It is diffi­
cult ever to say with confidence that a theory is good or bad when it is just tested against a
small number of cases. It can apply or not, but until it has been applied to a large number of
cases, it is difficult to draw any strong conclusions. The only way we can have confidence that
our findings can be robustly generalized and are not based on spurious associations is to
gradually extend the analysis and test the key hypotheses across more and more cases.
H O W T O D O R E S E A R C H IN P R A C T IC E

T h is can be d one eith er by rep licating sm all-N stud ies u sing d ifferent cases, o r by c a rry in g
out large-N qu antitative research th at seeks to test m o re widely in sights g en erated from
sm all-N analysis.
O f cou rse, un der m an y circu m stan ces, sm all-N research is th e on ly op tion . O ften th e
popu lation o f in terest is no t th e w hole w orld, but on ly a han d fu l o f c ou n tries. F o r exam ple,
if we are in terested in th e success (o r failu re) o f m arket reform s in fragile d em o cracies, th en
th ere is only a lim ited n u m b er o f cou n tries th at w ould be elig ib le for in clu sio n (W eylan d
2 0 0 2 ). B ut th is does no t chan ge th e b asic p oin t. U ntil th e th eo ry has b een tested on a large
nu m b er o f cases (w h eth er they exist o r n o t), th en th e w ider im p lication s o f th e fin dings m ust
be treated w ith caution.
The com parative in feren ces we can draw from sm all-N studies are th erefo re m u ch m o re
tentative than the in feren ces we can draw from large-N studies. Th is is b ecau se th e standard
for finding su p porting ev iden ce in favour (o r again st) a h yp othesis ten ds to be som ew hat
different. In qu antitative analysis (as we discu ss in C hapters 14 to 16), th e p resen ce (or
a b sence) o f causal relationships are defined probabilistically. That is, for a h yp othesis to be
supported (or rejected ), we m ake a d ecision based on the b alan ce o f probability. However, in
sm all-N studies, it b ecom es m ore d ifficult to talk about p rob abilistic causality. I f we com p are
th ree cases and ob serv e th e ex pected relationship in tw o o f th em , do we r eject or accep t our
w orking hypothesis? G enerally, in sm all-N studies, causality is d ete rm in istic. That is, in
ord er to find ev iden ce to support a hyp othesis, th e ex pected p attern needs to be ob serv ed in
all the cases. Th is obviously creates a problem in that it is q u ite easy to reject a h yp othesis that
is actually true, ju st b ecause on e o f the cases deviates from the exp ected pattern.

Qualitative comparative analysis (intermediate N)

Recently, d iscu ssion o f sm all-N com p arison has m oved in a nu m b er o f different d irection s
with the d evelopm ent o f Q ualitative C om parative A nalysis (Q C A ). Q C A draws on insights
from the m ethodology o f necessary and sufficient cond ition s, B oolean algebra, and fuzzy-
set logic (e.g. Ragin 1 9 8 7 ,2 0 0 0 ,2 0 0 8 ; G oertz and Starr (eds) 2003). Q C A has grown in popular­
ity over the past few years, and there are now a w hole host o f tex tb ooks devoted to e xplaining
th e m ethod (see Ragin 1987, 2 000, 2008; C aram ani 2 009; R ihoux and Ragin (eds) 2009,
http://www.compasss.org). Typically, qualitative researchers exam in e a few cases at a tim e in
detail, and look at how the different parts o f a case fit together, b oth contextually and h is­
torically. A ccording to Ragin, by form alizing the logic o f qualitative analysis, Q C A m akes it
possible to bring the logic and em pirical intensity o f qualitative approaches to studies that
em brace m ore than a handful o f cases, w hich would norm ally be analysed using variable-
orientated, quantitative m ethods. In som e ways, then, Q C A occupies an interm ed iary p o si­
tion between sm all-N analysis and large-N analysis, and is generally used for sam ples that
range from about a dozen or so cases to several hundred.
Boolean m ethods o f logical com parison represent each case as a com bin atio n o f causal
and outcom e cond ition s. Ihese com binatio ns can be com pared with each oth er and then
logically sim plified through a process o f paired com parison. C om pu ter algorith m s can be
used to provide techniques lor sim plifying this type o f data. (Softw are is available on C harles
Ragins website at: http://www.ii.arizona.edu/~cragin/fsQCA.) But when only a few cases
 C O M P A R A T IV E R E S EA R C H

are used, it is also possible to do it by hand. The data m atrix is reform ulated as a tru th table
(see Ragin 1987).
Broadly speaking, th ere are two m ain ways o f doing Q C A . The first is known as crisp set
Q C A , in w hich all th e variables in the analysis are treated as sim ple d ichotom ies (that is. the
hypothesized causal con d ition is cith er present or a bsent). The second is known as fuzzy act
Q C A , in w hich variables are allowed to take different values and are calibrated on an interval
scale betw een 0 .0 and 1.0 (see Ragin 2008: chapter 2). A conventional (or crisp) set is
d icho tom ous: a case is e ither in’ or out' o f a set, for exam ple, the set o f dem ocracies. Ihus. a
conv en tion al set is com parable to a binary variable with two values, 1 (‘in’, i.e. dem ocracy)
and 0 (‘ou t’, i.e. no n -d em o cracy ). A fuzzy set, by contrast, perm its m em bership in the in ter­
val betw een 0 and 1, w hile retain ing the two qualitative stales o f full m em bership and full
n o n-m em b ersh ip . Thus, the fuzzy set o f d em ocracies could include countries which are
‘fully in’ th e set (fuzzy m em bership = 1.0), som e who are alm ost fully in the set (m em b er­
ship = 0 .9 0 ), som e w ho are neither m ore in nor m ore ou t’ o f the set (m em bership = 0.5,
also know n as the crossover p oin t’), som e who are barely m ore out than in’ the set
(m em b ership = 0 .4 5 ), and so on, down to those who are ‘fully out’ o f the set (m em bership =
0 ). It is up to the researcher to specify procedures for assigning fuzzy m em bership scores to
cases, and th ese proced ures must be both open and explicit so that they can be evaluated by
o th er scholars (see Ragin 2 0 0 8 ). D espite these m easurem ent differences, the general analy­
tical principles o f the two approaches are m uch the same. For the sake o f simplicity, we will
focus just on crisp sets to illustrate how the approach can work.
Q C A is a pow erful heu ristic tool as it can be used for several purposes, such as su m m ariz­
ing data, prod ucin g typologies, and elaboratin g new theories or m odels, as well as for testing
existin g th eories, m odels, and hypotheses (R ih oux 2006). In particular, Q C A is well suited
to unravelling causal com plexity in order to detect the different cond ition s (or configura­
tions) that can lead to the sam e ou tcom e occu rrin g (w hich is som etim es called equifinality
or m ultiple causation). The way in w hich it does this is by utilizing what are known as truth

Tibi* 9.5 Hypothetical truth table showing three came* of w cceu h jl itn kei

0 0 ■
0 1 Ü
0 1 1
1 0 c
1 0 1
1 1 c
1 1 ’

No(at A • boom «« product martet

B • threat of sympathy strikes
C ■ U rj* tfnke fund
S • w ccen of strike
N • number of c m
H O W T O D O R E S E A R C H IN P R A C T IC E

tables. T ru th tab les p resent all th e logical possible co m b in a tio n s o f co n d itio n s an d th e o u t­

co m e associated with each co m b in atio n . By ex am in in g all th e possible d ifferent c o m b in a ­
tions o f variables that are associated w ith an ou tco m e o ccu rrin g (o r n o t), th e m ain aim o f th e
approach is to try and sim plify th ese different co m b in atio n s to as few d istin ct (nece ssary )
com bin a tio n s as possible. This is d one u sing so m eth in g called B o o le an algebra (see R agin
2 0 0 8 for a step-by -step guide to how th is w orks in p ractice).
W e can illustrate how this approach works by draw ing on a hyp othetical exam ple. Table 9.5
reproduces Ragins (1987: 96) hypothetical truth table on the factors associated with the su c­
cess o f strike activity. In this exam ple, the aim o f the research is to explore how th ree m ain
factors influence w hether or not a strike is successful. The th ree causal cond ition s are A=
boom ing product m arket; B = threat o f sym pathy strikes; and C = large strike fund. D ata are
collected on 39 strikes, and for each case the causal condition s are (hyp othetically) coded 1 if
they are thought to be present (and 0 otherw ise). Sim ilarly, the outcom e variable is coded 1 if
the strike was successfiil, and 0 if it was not. The first three colu m ns o f the truth table refer to
the independent variables/causal conditions. T he rows refer to all the logical c om binatio ns that
these cond itions can take. W hen there are three independent variables, there are eight possible
com binations in w hich they can b e arranged. In m ore com plex analysis, the rows (representing
com binations o f causal condition s) may be quite num erous. The fourth colu m n refers to the
dependent variable, and shows the recorded outcom e (w hether or not the strike was su ccess­
ful) for each com bination o f conditions. The last colu m n refers to the nu m b er o f cases that
were observed in each row (though these are not really used in the subsequent analysis).
From Table 9.5 we can see that the four different co m b in atio n s o f co n d itio n s all led to a
successful strike (w here S = 1). W e can express these con d ition s in an eq uation as follows:

S = A B C + AbC + A Bc + aB c,

where a capital letter indicates that the causal cond ition in qu estion was present and a low er­
case letter ind icates that it was absent.
The first step in the B oolean analysis o f these data is to try and co m b in e as m any c o m p a t­
ible rows o f the truth table as possible. In a sense, this involves elim in atin g all redundant
term s. If two com binatio ns differ in only on e factor but produce the sam e result, th en the
factor they differ on can be thought o f as irrelevant.
So for exam ple, since A BC and AbC are b oth associated with the p resence o f ou tcom e
S = 1, we may suspect that B is superfluous. That is, when we have A and C together, it d oesn’t
matter w hether we have B or not in order for S to occur. We can th erefore co m b in e the first
two term s in the equation to produce AC.
Similarly, when B is present and C is not present (B e ), it doesn’t appear to m ake any d iffer­
ence w hether or not A is present in order for S to occur. So we can com bin e the term s A Bc
and aBc to produce Be.
We can th erefore express the equation in slightly sim plified term s as:

S= AC + Be.

Ihis final equation states simply that successful strikes o ccu r when there is a b oom in g m ar­
ket tor the product produced by the workers atui a large available strike fund (A C ) or when
there is the threat of sympathy strikes hv workers in associated industries com bined with a
low strike fund ( Be). Ihis finding suggests that strikes can be successful for different reasons.
 COM I

It also suggests that w hereas the unpact o f A and C is conditional on the presence o f each
other, the im pact o f B is cond ition al on the absence o f C. That is. the threat o f sympathy
strikes is taken seriously only when the striking workers badly need the support o f other
workers.
In order to use the truth table presented above, it is necessary to determ ine an output
value for each row that is either 1 or 0. W ith hypothetical data, this is not a problem , but the
real world is not so neat and the use o f truth tables is often com plicated by the presence o f
con tra d ictory o utcom es, ih at is where cases w hich share the sam e causal conditions exhibit
different o u tcom es on the variable o f interest. It is therefore m ore difficult to illustrate how
this m ethod w orks with real data.
N onetheless, th is type o f approach has been used, adapted, and developed for a variety of
purposes and to explore a variety o f different political p henom ena. O ne com m on application
o f the approach is to exam in e the extent to which different pathways (or com binations o f
causal con d ition s) can lead to sim ilar outcom es. Recent exam ples include Rihoux's (2006)
analysis o f the success o f G reen parties in W estern Europe. Veugelers and Magnan's
(2 0 0 5 ) analysis o f the success o f Far Right parties in W estern Europe, and Berg Schlosser's
(2 0 0 7 ) analysis o f the success o f d em ocracy in Africa. Berg-Schlo&ser (2 007) uses Q C A to
identify th e different pathways to d em ocratic success in Africa. H r identifies four main
groups o f coun tries, in w hich each group developed dem ocracy under som ewhat different
cond ition s. For each group, 'loser acceptance' at the lim e o f transition turned out to be a
necessary con d ition for the long term success o f dem ocracy, but by itself this factor was not
sufficient for d em ocracy to prevail, as the negative exam ples o f C ongo-Brazzaville. G am bia,
and oth ers w here d em ocracy was not sustained dem onstrate.
A lthough Q C A provides an interesting approach for exam in ing causal com plexity, it is
also beset with m any o f the sam e problem s that we discussed earlier with reference to case
selection in sm all-N research. There are a num ber o f issues we need to pay attention to when
carry in g out Q C A . The first issue refers to how m any cases we choose to exam ine in our
analysis. In ord er to exam in e w hether the im pact o f a particular factor is necessary or suffi­
cien t. and w hether its im pact is contingent on the presence or absence o f oth er variables, it
is desirable to have cases w hich capture the different possible com binations o f causal co n d i­
tions, so. w herever possible, outcom es are known for each row o f the truth table. O f course,
in reality, som e com bin a tio n s are sim ply not present in the real world. N onetheless, when
very sm all sam ples are used, there are going to be a lot o f gaps in the truth table.
Second, there is also the issue of case selection. As with all comparative research, it is
necessary to define the theoretical population of interest from which the cases are selected.
Sometimes all the countries (or cases) in a given population can be examined, but at other
times a sample will be needed. As soon as we are dealing with a sample, we need to think
about case selection or else our results may end up being very misleading. However, even
when we are analysing an entire population of cases, we should still think about how robust
our findings are. When we use exploratory techniques such as QCA to mine the data for
patterns, we have to be sensitive to the possibility that our findings are driven by quirks or
anomalies in the data that might not be present if we were to try and reproduce our analysis
with different (but similar) cases. For this reason, scholars who use QCA often go on to use
other methods of inquiry to test their arguments more widely, and so will think about the
ways in which QCA can be used to complement large-N research.
H O W T O D O R E S E A R C H IN P R A C T IC E

Quantitative analysis (large-N comparison)

L arge-N com parativ e studies are w idely used in p olitical research . Studies o f th is type fre ­
quently em ploy qu antitative analysis to answ er qu estio ns on top ics as diverse as th e nature
and d irection o f the relationship betw een ec o n o m ic d ev elopm ent and d em o c ra cy (Przew or-
ski et al. 2 0 0 0 ); th e causes o f eth n ic c o n flict (W ilk in so n 2 0 0 4 ) and civil w ar (G led itsch 2 0 0 7 );
the im pact o f party com p etition o n social w elfare (L ip sm eyer 2 0 0 2 ), th e relatio nship betw een
d ifferent electoral and political in stitu tio ns and political stability, co n flic t, and b reak do w n,
to nam e but a few.
The principles o f in feren ce and qu antitative data analysis are d iscu ssed elsew here in this
b o o k (see in particular C hapters 15 and 16) and so will no t b e dealt w ith here. In stead, we
will briefly discuss issues to do w ith selecting cases and data co lle ctio n . Large-N co m p a ra ­
tive research, based on the statistical analysis o f m any c ou n tries, has m an y advantages. It
allows us to rigorously test different hyp otheses and m ake in feren ces ab ou t h ow variables are
conn ected . M any o f the problem s o f selection b ias associated with sm all-N analysis th at we
d iscussed earlier can th erefo re be m itigated by using th is approach. Large-N c om p arison s
also allow us to system atically ex am in e the effect o f m any different variables and to k n o ck
down spurious association s. They also allow us to ex am in e how different factors in teract
with each oth er to produce different con seq u en ces in different contexts.
This allows us to have greater con fid en ce that our results are robust. This con fid en ce
com es in part from sim ply b eing able to analyse a large nu m b er o f cases (w hich are broadly
representative o f the population we are studying) and in part from b ein g able to ex am in e the
im pact o f m any different variables sim ultaneously, so as to rule out plausible alternative
explanations. To rule out plausible alternatives we need to analyse m any cases, or else we
may enco u nter the ‘too many variables not enough cases problem ’ that we d iscu ssed earlier.
However, despite these obvious strengths, analysing a lot o f cases also brings its ow n p ro b ­
lem s. This is often presented in term s o f a trad e-o ff betw een in ternal and external validity. As
Coppedge (1 9 99: 4 6 4 ) puts it, although large-N quantitative analysis is the best m ethod
available for testing generalizations, especially gen eralization s about com plex causal rela­
tionships, it is also often criticized for its ‘th in’ (red u ctio nist or sim p listic) con cep ts and
theories.
Large-N com parative studies involve the tasks o f case selection , data c ollection , and data
analysis. The principle o f case selection for large-N research is fairly clear-cu t, and th ere is a
generally agreed upon set o f p rocedures o f best practice. Ultim ately, the goal o f case sele c­
tion is to achieve a representative sam ple o f the p op u lation o f interest which is o f sufficient
sam ple size to enable robust inferences to be made. The principles o f sam ple design are d is­
cussed in detail in C hapter 10, and m any o f the sam e principles apply to case selection in
com parative research. The m ain advantage o f this approach is that it helps us to overcom e
many o f the problem s to do with selection bias that we discussed earlier in the chapter.
lo test any theory or hypothesis one must first identify the dom ain o f the argum ent or the
universe o f cases to which the hypothesis should apply. This is generally referred to as the
population o f interest. So, for exam ple, if you are interested in exploring why som e coun tries
engage in war whereas others don't, the population o f in terest m a y b e all the coun tries in the
world, since every country either engages in war or doesn’t. However, if you’re interested in
 COMI :h

ex ploring why som e cou n tries engage in war in the face o f provocation, whereas others
d on’t, your population o f in terest may be all the coun tries in the world that have experienced
provocation . (You would need to th ink very carefully about how you could define provoca­
tion and th en how you could m easure it.)
H aving defined the population o f interest, the next step is to select eligible cases for the
study. If the population is very big, it may be necessary to draw a random sam ple (see C hap­
ter 10). However, it is quite com m on in large-N com parative research to simply analyse all
th e cases in the population o f interest, and th ere are even a nu m ber o f studies which have
created data sets for all the cou n tries in the world. The problem with this though is that this
is still rather a sm all sam ple. These studies generally have data on about 192 coun tries, and
if we had a sam ple o f just 192 people in a survey we would be a bit reluctant to draw any
strong conclusion s. The sam e applies to com parative research. The size o f the sample in rela­
tion to the population doesn't m ake any difference to how reliable the findings from our
sam ple are. W hen you have a sm all sam ple there is always the risk o f draw ing faulty in fer­
ences.
C om paratavists often try to get round this problem by looking at data over tim e in order
to try and in crease th eir sam ple size. So, for exam ple, they may collect inform ation on 192
cou n tries at yearly intervals over say a 50-year pen od . w hich would give them a total sample
o f 9 ,6 0 0 . But even qu ite m odest in creases in sam ple size can help to k nock down many faulty
inferences. G ed d es (2 0 0 3 : 8 9 -1 0 5 ) provides a particularly striking exam ple of this, which
b oth highlights the lim itations o f sm all-N research and the advantages o f large- N research.
G ed d es recounts how analysts trying to explain why som e developing countries have grown
econ om ically faster than oth ers have tended to focus their analysis on a few successful new
ind ustrializing cou n tries (N IC s). such as Taiwan, South Korea. Singapore. Brazil, and M ex­
ico. It was noted that these coun tries all had fairly repressed labour forces. Accordingly,
som e scho lars asserted that repressive labour laws and weak u nions constitute a com parative
advantage in in ternational e con om ic com petition. The problem with this sort o f claim is that
it m akes sw eeping g eneralizations from the basis o f a very few cases. The hypothesis sounds
as if it m ight be plausible. And as we d iscussed earlier. sm all-N research is very useful for
developing theories. But to actually test this th eory and see w hether it really d oes stand up.
we need to see w hether it applies m ore generally. To do th is G eddes collects data on e c o n ­
om ic grow th and labour repression for 84 developing coun tries and exam in es w hether the
finding still holds.
The statistical tech niq ues she used to do th is are discussed in C hapters 15 and 16. H ow­
ever, th e m ain point o f interest is that she found that th e th eory did not stand up to wider
generalization. A lthough it was true that Taiwan, South Korea, and the oth ers did have
repressive labour laws and high levels o f e con om ic grow th. G eddes found that there were
m any oth er cou n tries that also had repressive labour regulations, such as Iran and A rgen­
tina, but w hich had not experienced econ om ic growth. Sim ilarty, she also found that there
were m any c ou n tries w hich did not have repressive labour laws, such as Botsw ana, but had
experienced high grow th. In sh ort, the initial assum ption that labour repression in fluences
econ om ic grow th was a faulty inference. It is easy to m ake these sorts o f faulty inferences
w hen we only exam in e a few cases. A nd th e best way to overcom e th is problem is to exam ine
m any cases. I f we are in terested in testing th eories rigorously, and seeing w hether they hold
H O W T O D O R E S E A R C H IN P R A C T IC E

up in the different co n texts to w hich they are th ough t to apply, th en we are in ev itab ly draw n
to large-N analysis.
W hereas alm ost any type o f data can be used in sm all-N research, large-N research gen er­
ally relies upon quantitative data. We call it quantitative data b ecause all th e in fo rm atio n is
assigned a num erical value. Som etim es how we assign th is value is obvious, b ecause the v ari­
able o f in terest is m easured in num bers, such as votes or inflation, o r G D P p er capita, war
casualties, or unem ploym ent. At oth er tim es, how we assign a value is less obvious, and is the
result o f a cod in g decision based on a qualitative jud gem ent to do with how d em ocratic an
expert’ th inks a particular cou n try is. Th ere are m any different sou rces o f data that are used in
large-N com parative research. Surveys are a valuable resource (discu ssed in C hapter 10), and
there are now many cro ss-national surveys that have been c arried out in alm ost iden tical fash­
ion in many different coun tries around the world over a relatively long period o f tim e. These
surveys provide valuable in form ation about how public perceptions o f corru p tion , or support
for d em ocracy, or support for hum an rights vary b etw een different regions. So m e o f the m ost
widely used c ross-national surveys o f this type are the Com parative Study o f E lectoral System s
(C SE S), the W orld Values Survey, Eurobarom eter, Latinobarom eter, and A frobarom eter.
But population surveys are not the only sou rce o f data that large-N stud ies can draw on.
Th ere are now m any cro ss-n ation al data sets that have been com piled from ex p ert surveys
(see C hapter 11), which m easure p erceptio ns about the level o f co rru p tio n in a co u n try
(Transparency In ternational) or the level o f d em o cracy in a co u n try (w hich form s p art o f the
Freedom H ouse and Polity IV m easures o f d em ocracy ). C om parative research also draws on
policy d ocu m ents and speeches (see C hapter 13) to ex am in e the ideology o f gov ernm ents
and political parties (Budge et al. 20 0 1 ; B en o it and Laver 20 0 6 ) and leadership styles and
p opulism (H aw kins 20 0 9 ). These m ethod s o f data co lle ctio n are discu ssed in detail else­
w here in the b ook, and so will not be considered here.
But w hatever data is collected, it is im portant that it is b oth v alid and reliab le. Broadly
speaking, validity refers to w hether the data m easures what it is supposed to m easure. That
is, how closely does the operational in dicator m atch the concept o f interest? For exam ple, if
you are interested in the level o f corrup tion in a country, is a survey qu estion on a b u sin ess­
m an’s perceptions of corru p tion a valid m easure o f this? The issue is not w hether you have
measured accurately or not w hether the businessm an th inks that the cou n try in qu estion is
corrupt or not, but w hether perceptions o f corrup tion are a valid m easure o f actual c o rru p ­
tion. Reliability relers to how well the operational in dicator has been m easured. That is, does
(he question really tap into what the businessm an th inks about the cou n try in qu estion?
The principles o f data collection in large-N research are that in form ation about the world
can be coded or sum m arized in ways that capture m eaning. We start with con cep ts or c o n ­
structs that we want to m easure; we develop operational in d icators for these co n stru cts; and
then we collect data to m easure the operational indicators (see C hapter 6). Som e co n stru cts
are (airly easy to measure. Suppose we are interested in conven tion al political p articipation.
O ur conceptual definition ol political participation m ight be w hether or not people vote
in general elections. O ur operational in d icator— w hich is what we use to m easure our
conceptual definition, might th erefore be official election returns from around the world
which detail what proportion o f the voting-age public actually cast a ballot. This is a pretty
valid measure. I he data clearly taps into our concept o f participation. It is also likely to be a
 C O M P A R A T IV E R E S E A R C H

reasonably reliable m easure, though p ractices o f vote rigging may m ake it less reliable in
som e cou n tries than others.
However, oth er con stru cts are m ore difficult to m easure. Suppose you are interested in
m easuring a co u n try ’s level o f d em ocracy. We must therefore provide a conceptual defim
tion o f what we m ean by dem ocracy. Ih e r e is a vast literature on this, and scholars disagree
about how d em ocracy should be defined. Som e authors favour a thin definition based on
in stitutional arrangem en ts, oth ers favour a thicker or fuller definition based on civil and
political rights. Suppose we go for a thin d efinition o f dem ocracy, in which we define d em o c­
racy as a system o f governm ent in w hich governm ents are selected though com petitive elec
tions. O u r operational d efinition may th erefore be the com petitiveness o f elections, and our
operational in d icator o f com petitiveness m ight be the difference in vote share between the
w inning party or candidate and the second-placed party or candidate. So an election where
the w inner achieved 9 9% o f the vote would be considered relatively uncom petitive, whereas
an election where tw o parties each got approxim ately 30% o f the vote would be considered
highly com petitive. O u r m easure m ight be reliable, but we might not say it is valid, since it
d oesn’t captu re the full definition o f dem ocracy.
O ne o f the m ain challenges in com parative research, particularly large S' comparative
research, is to do with establishing equivalence o f m eanin g That is. whether the theoretical
concepts and em pirical indicators o f those concepts mean the same things and measure the
sam e things in the different contexts to which they are applied. For example, the concept of
populism may m ean different things in different contexts. Ih u s. when we talk about populism
in Latin A m erica, we may be talking about som ething rather different to when we talk about
populism in Europe. Different understandings o f a concept can lead to different measures lo
tap into that concept (see Adcock and C ollier 2001). which in turn can mean that evrn though
we th ink we are talking about (and com paring) the sam e thing, we are actually talking about
two very different things. Under these circum stances, the com parison is no longer particularly
meaningful, because we are no longer com paring equivalent phenom ena.
W hen we are only com parin g a few coun tries, it is p ossible to explore these issues in som e
detail, and take steps to ensure that there is a close m atch between concepts and m easures in
each o f the cases being analysed. However, when we com pare many countries, we face a
nu m ber o f problem s. O n e the o ne hand, our concepts may mean different things in different
contexts, and so our m easures are not valid. O n the o ther hand, in order to establish equival­
en ce our concep ts and m easures may b ecom e so broad, if not stretched (Sartori 1970). that
th eir m eanin g b ecom es diluted. O n the dangers o f con cep t -stretch ing, see C hapter 6. It is
only with careful analysis and detailed substantive knowledge that we can have confidence
that our m easures m ean the sam e things in different countries.

Conclusions
In this chapter we have examined a number of different w ay * d o in * comparative research
Broadly speaUr^. we can distinguish between thee approaches accordmc to the nuntoer of cases
that are compared, and how the cases are »elected Comparative research it therefore primarily
defined as a method of case selection Different approaches lo case selection are associated with
H O W T O D O R E S E A R C H IN P R A C T IC E

various different problem s, and the approach that is preferred m ay therefore depend upon the
purpose to w hich it is used.
In particular, there are two main problem s. The first is to do w ith selection bias. The second is to do
w ith equivalence of meaning. The m ain problem w ith sm all-N studies and single-country case studies
is that there is the real risk o f selection bias. The cases w e select can in fluence the answers w e get. This
means that our findings might not be very reliable, and could just be an artefact of the cases w e have
selected. If w e selected different cases, w e w ould com e up w ith very different conclusions. One w ay to
get round this is sim ply to exam ine m ore cases. But this com es with its ow n problem s. W hen there are
m any cases, it becomes m ore difficult to ensure that the variables under investigation are both
conceptually and em pirically com parable. To ensure com parability we can end up using very general
concepts and measures, w hich becom e abstract and devoid of m eaning, underm inin g the validity of
the measures.
Although these problems affect all com parative research, they should not be overstated. W ith
careful design, w e can take steps to tackle them head on, and by com bining approaches, w e can
attempt to integrate the strengths of one approach w ith the strengths o f another. H owever, it is
w orth bearing in mind that these issues o f case selection are only the first part of the research
process, and having chosen a case (or cases) to study, you then need to think about ho w you are
going to study it/them. These m ethods of data collection and data analysis are discussed in detail in
the following chapters.

Questions
• W hat are the problems w ith selecting on the dependent variable? Have these problem s been
overstated?

• If the cases you select affect the answers you get, does this mean that sm all-N com parison is
inherently less reliable than large-N com parison?

• Under what conditions does it make more sense to adopt a Most Different Systems Design than a
Most Sim ilar Systems Design?

• If a topic is worth investigating, it is worth investigating com paratively? Discuss.

• W hat is concept-stretching? To what extent can it hinder com parative research?

• To what extent is a small-N com parative study a qualitative research design?

• What is the difference between methods for case selection, data collection, and data analysis? Does
what we do at one stage influence what w e do at the next?

• What are the strengths and weaknesses of sm all-N com parison and large-N com parison?
• How does QCA differ from small-N research?

© Guide to Further Reading

Brady, H. E. and D. Collier (eds) (2004), Rethinking Social Inquiry: Diverse Tools. Shared Standards
(Lanham. MD: Rowman Littlefield).
Contains a selection of essays w hich describe the ways in w hich qualitative approaches can contribute
to the study of political phenomena.

Geddes B. (2003), Paradigms and Sand Castles: Theory Building and Research Design in
Comparative Politics (Ann Arbor, Ml: University o f M ichigan Press).
Provides a detailed discussion on case selection and how the cases you choose for analysis can affect
the answers you get
 C O M P A R A T IV E R E S EA R C H

Georg«. A. L and A. Btnnctt (2005). Case Studio and Theory Deveiopmer* m the Sooat Soences
(Cambridge. MA: MfT Pm*).
An im portant discussion of th e place of case studies in social science m ethodology Emphasues the
im portance of withm-case analysis, provides a detailed discussion of process tracing, and of the
concept of typological theories

King, Gary, Robert Keohane. and Sidney Verba (1994). Designing Soool Inquiry: Saem fk Inference
in Qualitative Research (Princeton. NJ: Princeton University Press).
O ne of the classic methodology books Provides a very influential perspective on small-N research
and how it can b e carried out systematically and scientifically

Landman. T. (2008). Issues and Methods in Comparative Poirba (London: Roudedf»)

Provides an engaging treatm ent of issues in comparative research with many examples drawn from
the fields of democracy, human rights, and conflict The book examines how case studies. sm allN
com parisons, and large-N com parisons have been used to answer a variety of different questions m
com parative politics

Ragin. C. C. (1987). The Comparative Method Moving Beyond Qualitative and Quantitative
Strategies (Berkeley, CA. University of California Press)
This book introduces the pnnciples of Quantitative Comparative Analysis and Boolean Algebra using
crisp sets It provides an informative and engaging introduction i0 QCA and how it can be used to
answer a variety of questions in comparative political research

---- (2008). Redesigning Social Inquiry: fuzry Sets and Beyond (Chicago, IL University of Owcago
Pres*).
This book develops Ragin's earlier work on QCA. and provides an in-depth treatm ent of how to
calibrate fuzzy sets It contains lots of helpful information on how to construct truth tables and how
to analyse them

References

Ac hen. C and D Snidal (1989). national Deterrence Brady. H £ and D Co«*» (eds) (2004). Rethink,ng
Theory and Comparative Case Studies' Wortd Social Inquiry Drvmt Took Shored Standard
Po/itici41(2) 143-69 (Lanham MD Bowman Littlefield)

Adcock. Robert and David CoIJter (2001 1 Brass. P (2003). The Production of Hmdu Muriwn
'Measurement Validity A Shared Standard for VioJmrr m Contemporary India (Seattle WA
Qualitative and Quantitative Research. Amentor University of Washington Press)
Political Science ffevxw 95(3f 529-44 Braumoeter. Bear, and Gary Goeru (2000).
Almond. G (1966). PoirticaJ Theory and Political The Methodology of Necessary Condition*
Science'. Amencan Political Science Revww 6 0 Amencan Journal of Political Science 44
8 7 7 -8 (October) 844-58

— and Sidney Verba (1963). The Civk Culture Budge. Ian Ham-Dwter Klwgeman. Andrea
PolttKal Attitudes and Democracy * five Natnm Vofcens. Judith Bara and Enc Tanentoaum
(Princeton. NJ Prtnceton University Press) (2001). Mapping Pohcy Preferences fsbmotn
for Portwv £lecîon and Govemmena 194S-I998
Benoit. Kenneth and Michael Laver (2006).
(Odord Orford Unrvemty Press)
Party Pokey in Modem Democroo« (London
Routtedge) Butler. D and D Stokes (1969) PoÊGcal Change m
Bndan (London Macm«an)
Bennet. A and C Elman (2 0 0 6 )-Qualitative
Research Recent Developments m Case Study Camptod. Angus Ph*p I Convene. Warren MOer.
and Donald Stokes (1960) TTwAmencan Voter
Methods''. Annual Review of Poibtaf Sdena 9
455-76 (New Yortr WMevl
H O W T O D O R E S E A R C H IN P R A C T IC E

Caramani, D. (2008), Introduction to the Comparative Inference in Qualitative Research (Princeton, NJ:
Method with Boolean Algebra, Quantitative Princeton University Press).
Applications in the Social Sciences (Beverly Hills, Kitschett, H., Z Mansfeldova, R. Markowski, and G. Toka
CA: Sage Publications). (1999), Post-Communist Party Systems: Competition,
Collier, D. and J. Mahoney (1996), Insights and Representation, and Inter-Party Cooperation
Pitfalls: Selection Bias in Qualitative Research', (Cambridge: Cambridge University Press).
World Politics 49(1): 56-91. Levy, J. S. (2002), 'Qualitative Methods in
---- ,J. Mahoney and J. Seawright (2004), 'Claiming International Relations', in F. P. Harvey and
Too Much: Warnings about Selection Bias', in H. M. Brecher (eds), Evaluating Methodology in
E. Brady and D. Collier (eds), Rethinking Social International Studies (Ann Arbor, Ml: University
Inquiry: Diverse Tools, Shared Standards (Lanham, of Michigan Press). 432-54.
MD: Rowman Littlefield), 85-102. Lewis-Beck, Michael (1986), 'Comparative Economic
Coppedge, Michael (1999), Thickening Thin Voting: Britain, France, Germany, Italy', American
Concepts and Theories: Combining Large N Journal o f Political Science 30.315-46.
and Small in Comparative Politics'. Comparative Lijphart, A. (1971), Comparative Politics and the
Politics 31(4): 465-76. Comparative Method', American Political Science
Dion, F. D. (1998), Evidence and Inference in the Review 65(3): 682-93.
Comparative Case Stud/, Comparative Politics ---- (1977), Democracy in Plural Societies: A
30(2): 127-46. Comparative Exploration (New Haven, CT: Yale
Eckstein, H. (1975), 'Case Studies and Theory in University Press).
Political Science', in F. I. Greenstein and N. W. ---- (1999), Patterns o f Democracy (New Haven, CT:
Polsby (eds). Handbook of Political Science: Yale University Press).
Political Science: Scope and Theory, vol. 7
Lipsmeyer, Christine S. (2002), 'Parties and Policy:
(Reading, MA: Addison-Wesley), 94-137.
Evaluating Economic and Partisan Influences on
Geddes, B. (2003), Paradigms and Sand Castles: Welfare Policy Spending during the European
Theory Building and Research Design in Post-Communist Transition', British Journal of
Comparative Politics (Ann Arbor, Ml: University Political Science 32 (4): 641-61.
of Michigan Press).
Lipset, Seymour Martin (1959), 'Some Social
George, Alexander L. and Andrew Bennett Requisites of Democracy: Economic
(2005), Case Studies and Theory Development Development and Political Legitimacy', American
in the Social Sciences (Cambridge, MA: MIT Political Science Review 53:69-105.
Press).
Mainwaring, S. and T. Scully (eds) (1995), Building
Gerringjohn (2004), "What is a Case Study, and Democratic Institutions: Party Systems in Latin
What is it Good For?' American Political Science America (Stanford, CA: Stanford University Press).
Review 98(2): 341-54
Mair, P. (1996), 'Comparative Politics: An Overview',
---- (2007a), Case Study Research: Principles and in R. E. Goodin and H.-D. Klingemann (eds),
Practices (Cambridge: Cambridge University A New Handbook of Political Science (Oxford:
Press). Oxford University Press), 309-35.
---- (2007b), Is there a (Viable) Crucial-Case Michelutti, L (2008), The Vernacularisation of
Method7' Comparative Political Studies 40: Democracy (London and Delhi: Routledge).
231-53
Moore B. (1966), Social Origins of Dictatorship and
Gleditsch. Kristian Skrede (2007)., Transnational Democracy: Lord and Peasant in the Making of
Dimensions of Civil War' .Journal of Peace the Modern World (Boston, MA: Beacon).
Research 44(3): 293-309
Naroll. R. (1966), 'Scientific Comparative Politics and
Goertz. G and H Starr (eds) (2003), Necessary International Relations', in R. Barry Farrell (ed ).
Conditions Theory, Methodology and Applications Approaches to Comparative and International Politics
(New York Rowman and Littlefield). (Evanston. IL: Northwestern University Press).
Hawkins. K (2009), Is Chavez a Populist7 Measuring Parry. Geraint. George Moyser, and Neil Day (1992).
Populist Discourse in Comparative Perspective'. Political Participation and Democracy in Britain
Comparative Political Studies 42 1040-67 (Cambridge: Cambridge University Press).
King. Gary Robert Keohane. and Sidney Verba Posner, D. (2004), The Political Salience of Cultural
(1994). Designing Social Inquiry Scientific
Difference: Why Chewas and Tumbukas are
 C O M P A R A T IV E R E S E A R C H 229

Allies in Zam bia and Adversaries in M alawi,' Comparing Nations: The Use o f Quantitative Data
Am erican Political Science Review 98(4) in Cross-National Research (New Haven, CT: Yale
(N ovem ber): 529-45. U niversity Press).
Przeworski, A. and H. Teune (1970), The Logic o f Rose, Richard (1991), 'Comparing Forms of
Com parative Social Inquiry (N ew York: W iley). Com parative Analysis', Political Studies 39:
-----. M ichael E. Alvarez, Jose Antonio Cheibub, 446-62.
and Fernando Lim ongi (2000), D em ocracy and Sartori, G. (1970), 'Concept M isrepresentation in
Development. Political Institutions and Well-Being C om parative Politics', Am erican Political Science
in the World. 1950-1990 (Cam bridge: Cambridge Review, 64:1033-63.
University Press).
Skocpol, T. (1979), States and Social Revolutions.
Putnam , Robert (1993), M aking Dem ocracy Work: A Com parative Analysis o f France, Russia, and
Civic Traditions in M odem Italy (Princeton, NJ: China (C am bridge: Cam bridge University
Princeton University Press). Press).
Ragin, C. C. (1987), The Com parative Method: van Biezen, I. and D. Caram ani (2006), '(Non)
M oving Beyond Q ualitative and Quantitative Com parative Politics in Britain', Politics 26(1):
Strategies (Berkeley, CA: University of California 29-37.
Press).
Verba, S., Kay Schlozm an, and Henry Brady (1995).
-----(2000), Fuzzy Set Social Science (Chicago: Voice and Equality: Civic Voluntarism in American
University of Chicago Press). Politics (Cam bridge, MA: Harvard University
-----(2008), Redesigning Social Inquiry: Fuzzy Sets Press).
and B eyond (C hicago: University of Chicago Veugelers.J. and A. Magnan (2005). 'Conditions of
Press). Far-Right Strength in C ontem porary W estern

Rih oux, B. (2006), 'Governm ental Participation Europe: An Application of Kitschelfs Theory',

and the Organizational Adaptation of Green European Journal o f Political Research 44:8 3 7 -6 0 .

Parties: On Access, Slack, O verload and Distress', W eyland, K. (2002), The Politics o f M arket Reform in
European Jo urn al o f Political Research 45: S69- Fragile Democracies: Argentina, Brazil
S98. W ilkinson, S. (2004), Votes and Violence: Electoral
-----and C. Ragin (eds) (2009). Configurational Competition and Ethnic Riots in India (Cam bridge:
Com parative M ethods: Qualitative C om parative Cam bridge University Press).
Analysis (Q CA) and R elated Techniques Yadav. Y. (2000), 'Understanding the Second
(Thousand Oaks, CA: Sage). Dem ocratic Upsurge: Trends of Bahujan
Rogowski, R. (2004), T h e Role o f Theory and Participation in Electoral Politics in the 1990s’,
A no m aly in Social-Scientific Inference', in Henry in Francine R. Frankel, Zoya Hasan, Rajeeva
Brady and David C ollier (eds), Rethinking Social Bhargava, and Balveer Arora (eds), Transforming
Inquiry: Diverse Tools, Shared Standards (Lanham , India: Social and Political Dynamics o f Dem ocracy
M D: Row m an & Littlefield Publishers, Inc.). (Delhi: Oxford University Press).

Rokkan. S. (1966), 'C om parative Cross-N ational Yin, Robert K. (2009). Case Study Research: Design
Research: The Context o f Current Efforts', and Methods. 4th edition (Thousand Oaks, CA:
in Richard L M erritt and S. Rokkan (eds). Sage).
 Surveys

Chapter Summary
This chapter exam ines the principles of survey research and discusses the issues and
problem s associated w ith different stages of the design process. In doing so, it exam ­
ines questionnaire design, sam ple design, and in terview techniques and discusses the
com m on sources of error that affect survey research and w hat can be done to try and
avoid or m in im ize them. The chapter exam ines:

• questionnaire design;

• measurem ent error;

• sampling design;

• sampling error;

• interview mode.

Introduction
This chapter exam in es the principles o f survey research, and discu sses the issues and c h a l­
lenges involved in carryin g out a good survey. D espite a nu m b er o f w eaknesses, surveys
continu e to flourish and are frequently used in political research to investigate a wide range
o f political phenom ena. They com bin e two things: the ancient but extrem ely efficient
m ethod o f ob tain ing in form ation from people by asking q u estions; and the m o d ern random
sam pling procedures that allow a relatively sm al 1 nu m ber o f people to represent a m uch
larger population (Shum an and Presser 1996: 1). As such, they are a valuable resource for
exam in ing a wide range o f topics, and w hen done well, provide an accurate and reliable
insight into what ord inary people th ink about politics and how they participate in politics.
A successful survey involves m any different steps. Before th in kin g about qu estio nnaires
or sam ples, however, it is first necessary to establish what the m ain aim s o f the research
project are, and to clarify they key concepts, issues, and th eory that you want to exam in e (see
C hapters 5 and 6). The survey will only give you the answ ers to the qu estio ns that are asked,
and it is therefore im portant to have a clear idea about what it is that you want to find out
before the survey is carried out, since it will be too late afterw ards if a new idea com es to you
when you are doing the analysis. Ih e decisions that are m ade at this stage will in fluen ce each
of the subsequent steps that are taken.
Ihe next step— which is the focus o f this chap ter— is to decide how the qu estio nnaire is
going to he designed and how the sam ple is going to be selected. Surveys are about asking
questions, and the type of question you ask can affect the sort o f answ er you get in three m ain
ways. W hat question do you ask? W ho do you ask the question to? And how do you ask the
question? Ihe purpose ol this chapter is to exam in e each of these in turn. The first part o f the
 SU RV EYS

c h a p te r d ea ls w ith what you a sk an d e x a m in e s issu es rela tin g to q u e s tio n n a ire d esig n . I he

se c o n d p a rt d ea ls w ith who you ask an d c o v e rs s a m p lin g d e sig n , and th e th ird p art d eals w ith
how you a sk q u e s tio n s , a n d lo o k s at d iffe re n t typ es o f in te rv ie w te c h n iq u e .

The elements of a good survey

S u rv ey s h a v e b o th a d e sc r ip tiv e p u r p o se an d an a n a ly tica l p u rp o se, T h e d e scrip tiv e p u rp o se

is to try a n d p ro v id e a c c u r a te in fo rm a tio n a b o u t w hat p eo p le th in k an d d o in a giv en p o p u ­
la tio n ; th a t is, to try a n d u n d e rsta n d w h at h a p p en s. T h is is a v alu ab le e n d in itself, and
w h e th e r th e su rv e y is d e sig n e d to p r o d u c e p o p u la tio n e stim a te s a b o u t in c o m e o r in eq u ality ,
tr u s t in p a r lia m e n t, o r sa tis fa c tio n w ith d e m o c ra c y , o r ho w m an y p eo p le th in k te r r o ris m is
a le g itim a te fo r m o f p o litic a l p r o te st, th e a n sw ers to th e se s o r ts o f q u e stio n s ca n b e e x tre m e ly
re v e a lin g . H ow ev er, su rv e y s c a n b e u sed fo r m u ch m o re th an ju s t th is. T h e a n a ly tica l p u r ­
p o se is to p ro v id e d a ta th a t c a n b e u sed fo r th e o r y te s tin g o r h y p o th e sis te s tin g to try and
u n d e rsta n d w hy th in g s h a p p e n . It is o n e th in g to k n o w th a t a c e r ta in p e r c e n ta g e o f th e
p o p u la tio n h a s lo st fa ith in d e m o c r a c y ; it is q u ite a n o th e r to k n o w th e re a so n s why. Su rv ey s
a re a p o w e rfu l to o l fo r tr y in g to an s w e r th e se q u e stio n s. T h e r e s e a rc h e r d ev elo p s th e o r ie s o r
e x p la n a tio n s a b o u t w hy th in g s h a p p e n , an d u ses su rv ey s to c o lle c t d ata to test th o s e e x p la n a ­
tio n s a g a in st w h a t is o b s e r v e d in th e real w orld . I f th e e x p e c ta tio n s are c o r re c t, th en th is
le n d s s u p p o rt to th e th e o ry . I f th e e x p e c ta tio n s are w ro n g , th e re are tw o m a in p o ss ib ilitie s.
E ith e r th e th e o r y is w ro n g , in w h ic h c a s e it is b a c k to th e d ra w in g b o a rd to try an d c o m e up
w ith s o m e th in g m o re p la u sib le , o r th e m e a s u r e m e n t o r te s tin g o f th e th e o r y is w ro n g , an d is
s o m e h o w flaw ed . In th is c a se , th e re m a y b e n o th in g w ro n g w ith th e th e o r y o r e x p la n a tio n
p e r se, b u t th e w ay in w h ic h k e y c o n c e p ts h av e b e e n o p e r a tio n a liz e d (se e C h a p te r 6 ) is b ad .
O b v io u sly , w e w a n t to tr y a n d m in im iz e th e p o s s ib ility o f th e la tte r o c c u r r in g , an d th is
m e a n s w e m u st b e v e r y c a re fu l h o w w e c o lle c t th e in fo rm a tio n th a t w e n ee d , an d h o w we
th e n a n a ly se it.
T h e f o c u s o f th is c h a p t e r is o n h o w su r v e y s c a n b e u se d to c o lle c t in fo r m a tio n . C h a p te rs
14 to 16 d ea l w ith iss u e s o f a n a ly s is , b u t b e fo r e w e re a c h th a t sta g e it is n e c e s s a r y to b e
c o n f id e n t th a t th e d a ta w e w ish to a n a ly s e a re r e a s o n a b ly so u n d . O n e o f th e g re a t s tre n g th s
o f su r v e y r e s e a r c h is th a t it h e lp s us to m a k e g e n e ra l c la im s a b o u t w h at d iffe r e n t s e c t io n s
o f s o c ie ty o r d iffe r e n t s u b - g r o u p s o f th e p o p u la tio n a c tu a lly th in k a n d d o. It th u s giv es
v o ic e to p e o p le th a t m ig h t n o t o th e r w is e b e h e a r d . B u t a m a jo r w e a k n e ss is th a t su rv e y s
c a n , a n d fr e q u e n tly d o , m is r e p r e s e n t w h a t p e o p le th in k a n d d o an d th u s c r e a te m is le a d in g
in f o r m a tio n .
T h e e x te n t to w h ic h su r v e y s m is re p re s e n t th e ‘re a l’ o r ‘tr u e a ttitu d e s a n d b e h a v io u r o f th e
p e o p le th e y s e e k to stu d y c a n b e th o u g h t o f as erro r. T h e p u r p o se o f a g o o d su rv ey is to try
a n d m in im iz e th is e r ro r. In p r a c tic e , it is ra re ly p o s s ib le to e lim in a te all e r r o r (in d e e d , it is
v ir tu a lly im p o s s ib le ), b u t w ith g o o d re s e a rc h d e sig n an d c a re fu l a d m in is tr a tio n , it is p o s s i­
b le to re d u c e e r ro r , a n d p e r h a p s m o r e im p o rta n tly , to b e aw are o f w h at e r r o r r e m a in s.
T h is c h a p te r fo c u s e s o n su r v e y e r ro r. It d e s c r ib e s th e m a in ty p es o f e r r o r th a t a re fo u n d in
su r v e y s, a n d w h e re p o s s ib le p ro v id e s in fo r m a tio n a b o u t w h at to d o in o r d e r to tr y an d
r e d u c e th e m . T h e re a re tw o im p o r t a n t s o u r c e s o f e r r o r th a t w e c o n s id e r. T h e s e are h ig h ­
lig h te d in F ig u re 10.1 (s e e a lso G ro v e s e t al. 2 0 0 4 ) . T h e first is to d o w ith m e a s u r e m e n t e r ro r,
H O W T O D O R E S E A R C H IN P R A C T IC E

Measurement Representation

Figure 10.1 Sources of error in survey research

and refers to the ways in w hich surveys use q u estio ns to try and m easure d ifferent so cial and
political phenom ena, such as p olitical attitudes, opin ion s, and behaviour. T h e secon d is to
do with sam pling error, and refers to the ways in w hich respondents are ch o sen or selected
to com plete the survey and the im plications th is has for the representativeness o f th e sam ple.

What you ask: questionnaire design

Q u estio n n a ire design is a critical com p onen t o f survey research. If qu estio ns are badly
designed, then the results they produce will be flawed, no m atter how large th e sam ple size
or how sophisticated the statistical analysis. Badly phrased qu estio ns or p oorly m easured
concepts can render the research exercise useless b efore it has even begun and waste a lot o f
m oney in the process. It is therefore critical to m ake sure the q u e stio n n a ire does w hat its
supposed to do as well as it can.

Validity: from concepts to survey questions

O ne o f the m ost com m on uses o f surveys is to m easure the attitudes and b ehaviou r o f o rd in ­
ary people. Many o f these p h en om en a— or co n stru cts— that we are interested in m easuring
are not directly observable. We th erefore have to ask qu estions to try and m easure the c o n ­
structs we are interested in. G enerally, it is thought that it is easier to m easure behaviour,
since this is m ore concrete. Fast behaviour is in a sense a ‘fact’; som eon e either voted or they
did n t, they are either a m em ber o f a political party or they are not, and so on. By contrast,
questions relating to altitudes and opin ion s are m ore abstract, and so m ore difficult to pin
down and measure through the use ot one or m ore questions. A lthough there is som e truth
 SU RV EYS

in th is, in re a lity th e re are d ifficu ltie s a sso cia te d w ith m e a su r in g b o th a ttitu d es an d b e h a v ­

iou r. T h e h a rd e r a c o n c e p t is to d e fin e , th e h a rd e r it is to m e a su re , so th e first step is to cla rify
e x a c tly w h at it is th a t w e se e k to e x a m in e (se e th e d isc u ssio n o f c o n c e p tu a liz a tio n and o p e r ­
a tio n a liz a tio n in C h a p te r 6 ). C o n c e p ts th at are vagu e o r m u ltifa ce te d need to be p in n ed
d o w n a n d p e rh a p s b ro k e n up in to sm a lle r p arts.
O n c e w e h av e a d e fin itio n th a t we are h ap p y w ith , th e n e x t step is to th in k ab o u t ho w it can
b e m e a su re d . T h is in v o lv es m o v in g fr o m th e a b s tr a c t to th e sp ecific . T h is is o n e o f th e h a r d ­
est jo b s in su rv e y re se a rc h , a n d it ca n easily go awry. W h a t q u e stio n (o r q u e stio n s) taps in to
th e g e n e ra l id ea th a t y ou a re in te re ste d in e x p lo r in g ? T h e se q u e s tio n s — o r in d ic a to r s — are
th e b rea d a n d b u tte r o f su rv e y r e se a rch . W h e th e r w e are m e a su r in g b e h a v io u r o r attitu d es,
o u r p r im a r y c o n c e r n is th a t th e q u e stio n we u se is b o th v alid an d reliab le. V a lid ity m e an s
th a t th e q u e s tio n m e a su r e s w h at w e th in k it d o es an d re fle cts th e c o n c e p t th at we are try in g
to ad d ress. It is a n issu e o f h o w th e q u e stio n is u sed an d th e p u r p o se it is u sed for. F o r e x a m ­
ple, i f w e a re in te r e ste d in a g e n e ra l c o n c e p t, su ch as so c ia l statu s, we m ig h t use a q u e stio n on
e d u c a tio n a l lev el to tr y a n d m e a su r e th is (d e V aus 2 0 0 2 : 5 3 ). T h e issu e o f v a lid ity is n o t
w h e th e r w e h a v e m e a su re d e d u c a tio n p ro p erly , b u t w h e th e r th is is a su ita b le m e a su re o f o u r
c o n c e p t— s o c ia l sta tu s. T h is is a fr e q u e n t p r o b le m in s o c ia l an d p o litic a l re se a rch . O fte n
q u e s tio n s a re u se d in w ays fo r w h ic h th e y are n o t id eally su ited . T h e re m ay th e re fo re n o t b e
a n y th in g w r o n g w ith th e q u e s tio n p e r se, b u t th e p u r p o se th a t it is u sed fo r m a y n o t b e
a p p ro p ria te . T h e v a lid ity o f a m e a su r e th e re fo re d e p e n d s u p o n h o w w e d efin e th e co n c e p t.
T h is is alw ay s g o in g to b e o p e n to d e b a te , th o u g h is o b v io u s ly m o re s tra ig h tfo rw a rd in so m e
c a se s th a n o th e r s .
T h e re a re n o e a sy w ays to re so lv e th is. B u t th e re are fo u r b a sic w ays in w h ich w e ca n try
a n d a ssess v a lid ity ( s e e M o s e r a n d K a lto n 1 9 7 1 ; d e V aus 2 0 0 2 fo r an e x te n d e d d isc u ssio n ).
T h e first a n d m o s t stra ig h tfo r w a rd is fa c e v a lid ity . F a c e v alid ity sim p ly m e a n s: o n th e fa c e o f
it, d o e s th e q u e s tio n in tu itiv e ly s e e m lik e a g o o d m e a su r e o f th e c o n c e p t? I f th e an sw er is no ,
th e n w e d e fin ite ly h a v e a p r o b le m , b u t w h at is in tu itiv e ly yes fo r o n e p e r so n m a y n o t b e so
fo r a n o th e r . C o n t e n t v a lid ity e x a m in e s th e e x te n t to w h ich th e q u e stio n co v e rs th e fu ll ran g e
o f th e c o n c e p t, c o v e r in g e a c h o f its d iffe r e n t a sp e c ts. C r i t e r io n v a lid ity e x a m in e s h o w w ell
th e n e w m e a s u r e o f th e c o n c e p t r ela tes to e x is tin g m e a su re s o f th e c o n c e p t, o r related c o n ­
ce p ts. C o n s t r u c t v a lid it y e x a m in e s h o w w ell th e m e a su r e c o n fo r m s to o u r th e o r e tic a l e x p e c ­
ta tio n s , b y e x a m in in g th e e x te n t to w h ic h it is a sso c ia te d w ith o th e r th e o r e tic a lly relev an t
fa c to r s. U ltim a te ly , th o u g h , th e m a tte r o f v a lid ity is o n e o f ju d g e m e n t.
R e lia b ility m e a n s th e q u e s tio n is an s w e re d in th e sa m e w ay o n d iffe re n t o cc a s io n s . It is
c le a r th a t a n u n re lia b le q u e s tio n la c k s valid ity. B u t a relia b le q u e stio n n ee d n o t b e v alid ,
b e c a u s e it c o u ld b e m e a s u r in g s o m e th in g o th e r th a n w h at it w as d e sig n e d to m e a su re , an d
re lia b ility s h o u ld n o t c o m p e n s a te fo r lo w v alidity.

Measurement error developing indicators

A w e ll- c o n s tr u c t e d q u e s tio n m a y o r m a y n o t b e v alid . H ow ev er, a b a d ly w o rd ed q u e stio n is
n e v e r v a lid b e c a u s e it w ill n o t b e a g o o d m e a s u r e m e n t in s tr u m e n t. T h e n e x t im p o rta n t issu e
to a d d r e ss th e n , a s sh o w n in F ig u re 1 0 .1 , is m e a s u r e m e n t e rro r. B y m e a s u r e m e n t e r ro r o r
e r r o r in o b s e r v a tio n , w e m e a n th e d iffe r e n c e b e tw e e n w h at w e r e c o rd as th e r e s p o n d e n ts
a n s w e r to a q u e s tio n a n d w h a t is th e r e s p o n d e n t s real o r tr u e a n sw er to th a t q u e stio n .
H O W T O DO R E S E A R C H IN P R A C T IC E

O n the w hole, respondents will try to answ er a qu estio n tru th fu lly and to th e b est o f th eir
ability, but they m ay still not provide an accu rate answ er to th e qu estio n th ey are asked for a
nu m b er o f reasons. It m ay be that th ey sim ply have not u n d erstood th e q u estio n (w hich
refers to p rob lem s o f c o m p reh en sion ) or th ey can n o t rem em b e r th e answ er (p rob lem s o f
recall). H owever, there are also m ore subtle reasons w hy th ey m ay no t provide a ‘tru e’ answer.
O n e reason is to do with the suggestibility o f respondents. S o m e p eople m ay n ot b e sure how
to answ er th e qu estion b ecause th ey have never really th ough t ab out it b efore, and, rath er
than adm it as m u ch, th ey look for cues in th e way in w hich th e q u estio n is asked to give th em
clues how to answer. A no ther reason is th at th ey m ay h old u n fash ion ab le or even un sav oury
views, and thus be unw illing to reveal th em . The follow ing sectio n d iscu sses each o f th ese
potential sou rces o f m easurem en t error in tu rn.

Question wording comprehension

The literature on the w ording o f qu estions is vast (see M oser and K alton 1971 and Tou-
rangeau, Rips and Rasinski 2 0 0 0 for good overview s). R espo ndents will in terp ret th e q u es­
tion as best they can; but the m ore difficult a qu estion is to un derstand, th e m ore likely it is
that respond ents will interpret the qu estion in d ifferent ways, in tro d u cin g co n fu sio n about
what exactly is b eing m easured. We shall con fin e ourselves to som e aspects o f w ording that
are o f general im p ortance in survey research.
A co m m o n error is to ask a general qu estio n w hen a m ore sp ecific q u estio n is needed.
For exam ple, if one is interested in th e extent to w hich th e gen eral public tru st th eir p o liti­
cal representatives, it m ight be tem pting to ask th e follow ing qu estio n: ‘D o you tru st p o liti­
cia n s?’ However, this qu estio n is unsatisfactory. It is not clear w hat tru st m eans in th is
c ontext. The respondent m ight trust politician s not to take b ribes, but m ay not tru st th em
to tell the w hole truth when they are in a tight corn er. Th e qu estio n fails to provide the
respond ent with the necessary fram e o f referen ce to give a m eanin gful answer. Th ere may,
o f course, be a nu m ber o f d istin ct fram es o f reference. O n e m ight b e to tell th e tru th , on e
might be to put the in terests o f the cou n try first, but w hatever th ey are, th ey should be m ade
explicit.
Am biguity m eans that the qu estion can be in terpreted in different ways, and so in effect
respondents may be answ ering different questions. A qu estion can be am bigu ous for a wide
variety o f reasons, and it may not always be obvious that it is am bigu ous until you see how it
is in terpreted by different people. That is why it is so im portant to pre-test q u estio ns by
carryin g out exploratory research and piloting the survey (see C hapter 11 on interview s as a
way o f doing this). O ne particular type o f am biguity arises with d o u b le-b a rrelled q u estio n s.
Ihis is when the question in corporates two distinct qu estions w ithin one. So, for exam ple,
consider the question: ‘Do you agree or disagree with the follow ing statem ent. “Tony Blair
lied to parliam ent about Iraq and should be tried for war crim es’? You m ight th in k he lied
but should not be tried for war crim es, and you m ight th ink he didn’t lie but should be tried
for war crim es, so how do you answer? Better to break the question up into two separate
questions and ask about w hether or not you thought he lied in parliam ent, and w hether or
not he should be tried for war crim es separately.
Related to the double-barrelled question is the question that relies on faulty p resu p p o si­
tion. Ihis m eans that the question assum es som eth ing that is not necessarily true. M oser
 SU RVEYS

a n d K a lto n re fe r to th e se ty p es o f q u e stio n s as p r e s u m in g q u e s tio n s . For ex am p le, c o n s id e r

th e fo llo w in g s ta te m e n t: ‘ D e m o c r a c y is th e b est fo rm o f g o v e r n m e n t b ec a u se it m ak es le a d ­
ers a c c o u n ta b le . T h is is d iffic u lt to d e c id e b e c a u se it rests o n th e p re m is e th at d e m o c ra c y is
in d eed th e b e st fo r m o f g o v e r n m e n t. In th is c o n te x t, it is h ard to kn o w w hat it m e an s if a
r e sp o n d e n t d isa g re e s w ith th e sta te m e n t. It co u ld m e a n , ‘I d isa g ree; d e m o c ra c y is n o t th e
b est fo r m o f g o v e r n m e n t’, o r it c o u ld m e a n ‘I d isa g ree; a c c o u n ta b ility is no t th e m ain reason
why d e m o c r a c y is th e b e st fo rm o f g o v e r n m e n t.’ L ik ew ise, th e q u e stio n ‘W h o did you vote
fo r in th e last G e n e r a l E le c t io n ? ’ p re su p p o se s th at th e re sp o n d e n t v oted . T h ese typ es o fq u e s -
tio n a re b e st a sk e d o n ly a fter a filt e r q u e s tio n h as rev ealed th a t th e re sp o n d e n t did v ote in th e
last e le c tio n , a n d so , o n c e a g a in , are b e st ask ed separately.
T h e n e x t th r e e p r o b le m s in v o lv e th e m e a n in g o f w o rd s o r p h r a s e s in th e q u e s tio n . It is
im p o r t a n t th a t th e q u e s tio n h a s th e sa m e m e a n in g fo r d iffe r e n t r e s p o n d e n ts . V ag u e c o n ­
c e p ts o r v a g u e q u a n tif ie r s s h o u ld th e r e fo r e b e m a d e as c o n c r e te as p o s s ib le . V ag u e q u e s ­
tio n s e n c o u r a g e v a g u e a n s w e rs . I f y o u a sk s o m e o n e w h e th e r th e y ta lk a b o u t p o litic s
r e g u la r ly o r o c c a s io n a lly , th e m e a n in g o f t h e ir a n s w e r w ill b e v ag u e. A s M o s e r an d K a lto n
o b s e r v e , th is c o m m o n c h o i c e o f a lt e r n a tiv e s is s t r ic t ly illo g ic a l s in c e th e w o rd ‘o c c a ­
s io n a l’ r e fe r s to f r e q u e n c y , w h e re a s th e w o rd ‘r e g u la r ’ d o e s n o t. M a n y e v e ry d a y te r m s are
v a g u e , a n d r e s p o n d e n ts m a y in te r p r e t th e m in d iffe r e n t w ays. T h is is a p a r tic u la r issu e
in c o m p a r a t iv e s u r v e y s , w h e re th e sa m e te r m o r w o rd c a n h a v e d iffe r e n t c o n n o ta t io n s in
d if f e r e n t c o n t e x t s . F o r e x a m p le , th e w o rd ‘n a t io n a l is t’ m ig h t h av e p o s itiv e c o n n o ta t io n s
in s o m e c o u n t r ie s , b u t n e g a tiv e c o n n o t a t io n s in o th e r c o u n tr ie s , p a r tic u la r ly w h e n
‘n a t io n a l is t ’ m o v e m e n t s h a v e b e e n lin k e d to a c ts o f v io le n c e o r r e p re s s io n . It is th e r e fo r e
i m p o r t a n t to in v e s tig a te w h e th e r th e q u e s tio n h a s th e s a m e (e q u iv a le n t) m e a n in g in th e
d iffe r e n t c o u n t r ie s w h e re it is b e in g a sk e d . T h is is k n o w n as e s t a b lis h in g e q u iv a le n c e o f
m e a n in g .
S im ila rly , u n fa m ilia r te r m s o r ja r g o n sh o u ld b e av o id ed . T h e aim in a q u e stio n w o rd in g is
to tr y a n d c o m m u n ic a te w ith r e s p o n d e n ts as cle a r ly as p o ss ib le in a lan g u ag e th ey w ill
u n d e rs ta n d . P e o p le w h o w rite su rv e y s a re o fte n e x p e rts in th e s u b je c t, an d th ey m ay o v e r e s­
tim a te h o w fa m ilia r th e r e s p o n d e n ts are lik e ly to b e w ith th e te r m in o lo g y th a t th e y use. So,
ev e n i f it is p e r fe c tly c le a r to th e q u e s tio n d e sig n e r w h at is m e a n t b y fiscal p o licy , d e m o c ra tic
d e fic it, h u m a n ita r ia n in te r v e n tio n , o r b ila te ra l tra d e a g r e e m e n t, it m ig h t n o t b e fo r th e p e r ­
so n a n s w e rin g th e q u e s tio n . M o s e r an d K a lto n (1971) c o u n s e l q u e stio n d e sig n e rs to put
th e m se lv e s in th e p o s itio n o f th e ir ty p ic a l— o r le a st e d u c a te d — re sp o n d e n t. M a n y w ords
w h ic h a re c o m m o n to u n iv e rs ity -tr a in e d su rv e y e x p e rts are far fro m c o m m o n in o r d in a r y
c o n v e r s a tio n . E v en w o rd s lik e h y p o th e tic a l, a g g rav ate, d e p re ca te , an d h u n d re d s m o re are in
th is ca te g o ry , a n d G o w e r s (1954) m e n tio n s m a n y w ord s w h ich can o fte n b e re p laced b y
s im p le r a lte rn a tiv e s , su c h as ‘h elp ’ in ste a d o f ‘a s s is t’; ‘th in k ’ in stea d o f ‘c o n s id e r ; b eg in
in ste a d o f ‘in itia te ’; ‘c o u n tr y ’ in ste a d o f ‘sta te ’; an d ‘e n o u g h ’ in ste a d o f ‘su fficie n t. F o r su rv ey s
o f th e g e n e ra l p o p u la tio n ( r a th e r th a n e lite s, w h ic h is d isc u sse d in C h a p te r 11), M o s e r an d
K a lto n (1971: 321) a d v ise th a t th e first p r in c ip le in w o rd in g is th a t q u e stio n s sh o u ld u se th e
sim p le s t w o rd s th a t w ill c o n v e y th e e x a c t m e a n in g , a n d th a t th e p h r a sin g s h o u ld b e a s sim p le
a n d in fo r m a l a s p o ss ib le . T h e m o re q u e s tio n s s o u n d lik e an o r d in a r y (p o lite ) c o n v e r sa tio n ,
th e s m o o th e r th e in te r v ie w w ill b e.
T o e n s u re th a t r e s p o n d e n ts all u n d e rs ta n d th e q u e s tio n an d in te r p re t it in th e sa m e way, it
is th e r e fo r e r e c o m m e n d e d to ( 1 ) d e fin e te r m s c a re fu lly ; (2 ) av oid u n fa m ilia r, a m b ig u o u s, o r
H O W T O D O R E S E A R C H IN P R A C T IC E

im precise words; (3 ) avoid com p licated w ording/too m an y clauses; (4 ) avoid tw o o r th ree

qu estio ns in one. To help ov ercom e c o m p reh en sion prob lem s, p ilo tin g th e q u estio n n aire is
cru cial. O b serv in g how respond ents answ er qu estio n s can give a clear in d icatio n o f how well
they have un d erstood th em . D o th ey ask for th e qu estio n to b e repeated? D o th ey ask for
clarification? D o they stare b lankly at the interview er?

Recall problems
However, even when respond ents un derstand the qu estio n, m easu rem en t erro r can still
arise in a variety o f ways. A n o th er potential sou rce o f erro r in survey resp onses is to do w ith
m em ory failu re— or recall p rob lem s. This is particu larly relevant for q u estio n s ab out past
b ehaviou r or past actio ns that the respond ent m ay have u n d ertaken, such as w h eth er th ey
voted in the last election , and, if so, for w hich party (see H eath and Joh n s 2 0 1 0 ). S o m etim es
respond ents can n ot rem em ber th e relevant events at all, and som etim es they p artially
rem em ber or in accu rately rem em ber. W right (1 9 9 3 ) suggests that o n e reason for th is is that
less politically engaged respond ents are less likely to retain th e in fo rm atio n o f w ho th ey
voted for in th eir m em ory. V oters sim ply forget w hich party or cand id ate th ey v oted for.
Respond ents struggling to rem em ber how they voted are probably m o re likely to in fer that
they voted for the w in ner (W right 1993; A tkeson 1999). This gives rise to the so-called b a n d ­
w agon effect.
In ad d ition, ano ther type o f m em ory-related p roblem is to do with w hat B elli et al. (1 9 9 9 )
call sou rce-m o n ito rin g error. Th is problem can affect qu estio ns about all sorts o f behaviour,
from tu rnout to protest to acts o f aggression, particularly if th e qu estion is co n ce rn e d with
behaviou r w ithin a specific tim e fram e, such as the past 12 m onth s. R espo ndents can g en er­
ally rem em ber that they have done such an activity, but can n ot recall accurately w hen it was,
and so confu se real p articipation w ithin the specified tim e fram e with earlier exp eriences.
This is also known as telescop in g. As m ore tim e passes sin ce th e event, we m ake m o re errors
in dating w hen it too k place.
There are a nu m ber o f strategies that can be used to try and deal with th ese problem s.
Perhaps the m ost sensible is not to put the resp on d en ts m em ory under too m u ch strain in
the first place by asking them to recall events in the distant past, or to recou nt the nu m b er o f
tim es they have done som eth ing in a fairly lengthy tim e period. For this reason, qu estions
tend to focus on events that have taken place in the past seven days for relatively com m on
activities, such as reading the new spaper, or the past 12 m onth s for less co m m o n activities,
such as taking part in a protest.

Deliberate misreporting
However, even if respondents understand the qu estion and can rem em ber the answer, they
may not always answer entirely truthfully. It is desirable that respondents answ er the qu es­
tion. N on-response (or item n o n -re sp o n se as it is know n w hen it refers to a specific q u es­
tion) can introduce error. Refusals may happen for a variety o f reasons, but are m ore
com m on when the question deals with sensitive issues. Respondents may not want to
declare how much m oney they earn, or w hich political party they support (particularly in
polarized political contexts). For these reasons, sensitive qu estions tend to go at the end o f
 SU RV EYS

q u e s tio n n a ire s , a lth o u g h th is n ee d s to b e b a la n c e d a g ain st issu es re latin g to q u e stio n o rd er

effects.

R e s p o n d e n ts g e n e ra lly w ant to b e h elp fu l, b ut at th e sam e tim e th e re is a ten d en cy , w hich

h a s b e e n fo u n d in m a n y stu d ies, th a t th e y also w ant to p resen t th e m se lv e s in a g o o d ligh t.
T h is is k n o w n as s o c ia l d e s ir a b ilit y b ia s , in w h ich re sp o n d e n ts o v e r -re p o r t soc ia lly ‘d e s ir a ­
b le b e h a v io u r a n d u n d e r-r e p o r t ‘u n d e sira b le b eh av io u r. At o n e e x tre m e , resp o n d e n ts m ay
b e u n w illin g to a d m it th a t th e y h o ld ra cist v iew s, o r v ote for e x tre m e -r ig h t p a rties, an d so
in ste a d w ill o p t fo r a n o th e r a n sw er (p e rh a p s say in g th ey d o n ’t k n o w ). B u t even m o re m o d e r ­
a te v iew s a n d fo r m s o f b e h a v io u r are su s c e p tib le to th e sam e b iases. F o r ex a m p le, su rv eys
fr e q u e n tly fin d th a t r e sp o n d e n ts o v e r -re p o r t th e tru e e x te n t to w h ich th e y have v oted. T h is
in p a rt m a y ste m fr o m a d e sir e to a p p e a r a g o o d c itiz e n an d to hav e fu lfilled o n e s civ ic o b li­
g a tio n s (th o u g h th e e x te n t to w h ich v o tin g is c o n s id e re d a c iv ic o b lig a tio n o b v io u sly varies
a c ro s s c o n t e x t s ) . O n e w ay to c o u n te r a c t th is relativ ely m ild s o cia l d e sira b ility b ias is to load
th e q u e s tio n in a c e r ta in w ay th a t giv es th e r e sp o n d e n t a r e a d y -m a d e e x c u se fo r n o t h av in g
d o n e w h a t th e y m a y th in k is e x p e c te d o f th e m . L o a d in g a q u e stio n m e a n s w o rd in g it in a
way th a t in v ite s a p a r tic u la r r e sp o n s e , as d iscu sse d b elo w w ith r e fe re n c e to lea d in g q u e s ­
tio n s. In th is c a se , th e q u e s tio n is lo a d ed to tr y an d in v ite th e so c ia lly u n d e sira b le resp o n se.
S u d m a n a n d B r a d b u rn ( 1 9 8 2 ) d istin g u ish sev eral stra te g ie s fo r d o in g th is: ra n g in g fro m th e
‘e v e r y b o d y - d o e s - it’ a p p r o a c h to th e ‘r e a so n s fo r d o in g it ap p ro a ch ’. Q u e s tio n s o n v o tin g g e n ­
era lly b o rr o w fr o m th e s e stra te g ie s, a n d a sk s o m e th in g lik e th is, w h ich is ta k e n fro m th e
B r itis h E le c tio n S tu d y (B E S ) :

Talking with people about the general election on M ay 5th, we have found that a lot o f people
didn't m anage to vote. How about you, did you manage to vote in the general election?

T h e se stra te g ie s h elp to r e d u c e th e a p p a re n t se n sitiv ity o f th e q u e stio n . Sim ilarly , e m b e d d in g

o n e se n s itiv e q u e s tio n (fo r e x a m p le , o n v a n d a lism ) a m o n g o th e r m o re sen sitiv e q u e stio n s
(fo r e x a m p le , o n te r r o r is m ) m a y m a k e th e se n sitiv e ite m o f in te re st se e m less th re a te n in g by
c o m p a r is o n .

Question order effects

S o far, th e p r o b le m s d is c u s s e d h av e b e e n relativ ely stra ig h tfo r w a rd , b u t th e re is a n o th e r set
o f m e a s u r e m e n t p r o b le m s th a t is less e a sy to d efin e. O n e o f th e m a jo r p r o b le m s th a t a ffect
su rv e y re s e a rc h is to d o w ith th e o r d e r in w h ic h q u e s tio n s a re a sk e d — q u e s tio n o r d e r e ffe c ts .
A p a rt fr o m s a m p lin g e r ro r , q u e s tio n o r d e r e ffe c ts a re p ro b a b ly th e m o st c o m m o n ly c ite d
e x p la n a tio n fo r a n u n e x p e c te d o r a n o m a lo u s su rv e y f in d in g (S h u m a n an d P re sse r 1 9 9 6 :2 4 ) .
T h e c o n t e x t in w h ic h a q u e s tio n is a sk ed o r its p o s itio n in re la tio n to o th e r q u e stio n s ca n
h a v e a b e a r in g o n th e a n s w e rs th a t a re g iv en to it. T h e m o st o b v io u s m a n ife s ta tio n o f th is is
w h e n r e s p o n d e n ts tr y to a p p e a r c o n s is te n t b y a n s w e rin g o n e q u e s tio n to fit in w ith h o w th ey
a n s w e re d a p re v io u s q u e s tio n . T o sh o w t h e s o r t o f e ffe c t th a t th e o r d e r o f q u e stio n s c a n have,
w e d ra w o n a s u r v e y e x p e r im e n t c a r r ie d o u t b y S c h u m a n an d P r e sse r ( 1 9 9 6 : c h a p te r 2 ), in
w h ic h a lte rn a tiv e v e r s io n s o f a q u e s tio n n a ir e w ere a sk ed to r a n d o m h alv e s o f th e sam p le.
R e s p o n d e n ts w ere a sk e d tw o q u e s tio n s , a b o u t C o m m u n is t jo u r n a lis ts a n d A m e ric a n jo u r ­
n a lists. H o w ev er, th e o r d e r in w h ic h th e y w ere a sk ed th e s e q u e s tio n s v a ried a c ro s s th e tw o
h a lf-s a m p le s . S u rv e y e x p e r im e n t s o f th is ty p e a re fr e q u e n tly u se d in p o litic a l r e se a rc h fo r a
H O W T O D O R E S E A R C H IN P R A C T IC E

Table 10.1 Order effects on Communist and American newspaper reporter items in 1980

Communist reporter first Am erican reporter first

Yes to Communist Yes to American Yes to Communist Yes to American

Reporter reporter reporter reporter

54 7% 74 6% 63 7% 81 9%

Source: Adapted from Schuman and Presser (1996)

wide variety o f purposes, and can be useful for tryin g to id en tify causal relatio nships (see
C hapter 8).

Do you think the United States should let Communist newspaper reporters from other countries
come in here and send back to their papers the news as they see it?
Do you think a Communist country like Russia should let American newspaper reporters come
in and send back to America the news as they see it?

From Table 10.1, we can clearly see that respondents were m uch m ore likely to say that they
would allow C om m un ist reporters into the U nited States after they had answered the question
on the A m erican reporter first. W hen the question on the C om m u n ist reporter was asked first,
only 54.7% o f the respondents thought C om m un ists should be let into the United States, but
when the A m erican question was asked first, this p ercentage increased to 63.7% (a statistically
significant increase). Similarly, respondents w ere less likely to say they would allow A m erican
reporters in to C om m un ist countries after they had answered the C om m u n ist question first,
with the percentage dropping from 81.9% to 74.6% (also a statistically significant change).
Schum an and Presser interpret this to suggest that respondents answer the question in term s
o f pro-A m erican and anti-C om m unist sentim ents, w hichever q uestion is asked first. But hav­
ing answered the first question, they follow a norm o f reciprocity, in which a substantial
num ber feel com pelled to provide an answer that is consistent with th eir previous response.
This type o f problem can affect a wide range o f qu estions, and it is not obvious how to deal
with it. Generally, it is thought that it is preferable to ask the m ore im p ortant q u estio ns first,
so they are not contam in ated by responses to less im portant qu estions. For this reason, in
most N ational Election Studies, qu estions on w hich party respondents voted for generally
tend to com e at the begin ning o f the qu estionnaire. Flowever, the dow nside o f this is that it
risks contam in atin g subsequent qu estions that are aim ed at m easuring why people voted as
they did. To try and reduce the possibility o f this happening, it is perhaps a good strategy to
leave gaps betw een questions w here this type o f co ntam in atio n is thought to be m ost likely.

Question wording effects

A nother group ol problem s arise from what can broadly be term ed q u estio n w ording
effects. Respondents may be m ore (or less) likely to report certain types o f behaviou r or
attitude, depending upon the way in w hich a question is phrased or the answ er is recorded.
A good exam ple of this is the difference betw een o p en -en d ed and closed qu estio n s, and
how response format can influence the answers that are given to the sam e question. A ‘closed’
question provides the respondent with a set ol answers to the question and the respondent
must select the response that reflects their view m ost closely. By contrast, an open -ended
 SU RVEYS

q u e s tio n d o e s n o t p ro v id e th e r e sp o n d e n t w ith any a n s w e r-o p tio n to c h o o se fro m , and th e

r e sp o n d e n t m u st fo rm u la te th e ir ow n re sp o n s e th em se lv es, w hich th e in terv iew er will th en
r e c o rd v e rb a tim .
T h e re a re p ro s a n d c o n s a tta ch e d to e a c h fo rm a t. A m a jo r ad v an tag e o f o p e n -e n d e d q u e s ­
tio n s is th a t th ey g iv e th e r e sp o n d e n t th e o p p o r tu n ity to v o ice e x a c tly w hat th ey th in k . Ih ey
allo w fo r a d e ta ile d a n d re a so n e d re sp o n s e , w h ich can reveal th e lo g ic and ra tio n a le b eh in d
a r e s p o n d e n ts an sw er. A n s w e rs are n o t p ro m p ted , an d so reveal w hat th e resp o n d e n t a c tu ­
ally th in k s . H ow ev er, th e re is a lso a n u m b e r o f d isa d v a n ta g es o f u sin g o p e n -e n d e d q u e s ­
tio n s. I h e y m a y p ro v id e a lo t o f in fo rm a tio n , b u t th is in fo rm a tio n th en nee d s to b e co d ed in
o r d e r to m a k e s e n se of th e an sw ers. T h is ca n b e v ery tim e -c o n s u m in g . M an y r esp o n ses m ay
be irre le v a n t o r o ff-to p ic , a n d s o m e g ro u p s o f p eo p le m ay be m o re likely to resp o n d th an
o th e r s , p a r tic u la rly th o s e w h o are m o re ed u c a ted o r m o re in terested in th e top ic.
F o r th e se r e a so n s, c lo s e d q u e s tio n s are th e stan d ard p ro c e d u re for m o st q u e stio n s. A tti­
tu d e q u e s tio n s a lm o s t alw ays u se a clo se d fo rm a t, w ith th e an sw er c a te g o rie s fo r m in g a scale.
C lo s e d q u e s tio n s p ro v id e th e r e sp o n d e n t w ith a p red efin e d set o f r esp o n s e a ltern a tiv es. T h ey
a re q u ic k a n d ea sy fo r th e r e sp o n d e n t to an sw er an d fo r th e in te rv ie w e r to c o d e, an d d o no t
d is c r im in a te a g a in st th e less ta lk a tiv e o r e lo q u e n t re sp o n d e n ts. T h ey can also rem in d
r e sp o n d e n ts o f a n s w ers th a t th ey m ig h t n o t o th e r w ise hav e retriev ed fro m th e ir m e m o r y
( S c h u m a n et a l., 1 9 8 6 ). H ow ev er, th is k in d o f p r o m p tin g m ay b e u sed by re sp o n d e n ts to save
th e m se lv e s th e m e n ta l e ffo r t o f p r o d u c in g a v alid a n sw er th em se lv es. M o reo v er, it is q u ite
lik e ly th a t th e p r e d e te r m in e d list o f a lte rn a tiv e s w ill ex clu d e so m e issu es th at are g en u in e ly
s a lie n t to so m e p e o p le . T h is is a p r o b le m g iv en su rv ey r e sp o n d e n ts’ g en era l re lu c ta n c e to
in sist o n a r e s p o n s e th a t is n o t e x p lic itly o ffered (a n d th u s le g itim iz e d ) by th e q u e stio n .
T h e s e p o in ts a re w ell illu stra te d by su rv e y e x p e rim e n ts co m p a rin g o p e n an d clo se d v er­
s io n s o f q u e s tio n s o n w h a t r e sp o n d e n ts c o n s id e r to b e th e m o st im p o rta n t p o litica l issu e
f a c in g th e c o u n tr y ( S c h u m a n a n d P r e sse r 1 9 9 6 ). E ig h t a lte rn a tiv e s w ere offered in th e clo se d
f o r m a t, th o u g h r e s p o n d e n ts d id h av e th e o p tio n o f ‘O th e r ’. T h e d is trib u tio n o f r esp o n ses in
th e tw o v e r s io n s is s h o w n in T a b le 10.2.
T h e first p o in t to n o te is th a t, w h e n u n p ro m p te d , 3 6 .5 % o f r e sp o n d e n ts n a m e d o n e o f a
w id e v a rie ty o f p r o b le m s th a t w ere n o t in clu d e d in th e list o f eig h t altern a tiv es. In th e c lo se d
v e r sio n , h a r d ly a n y r e s p o n d e n ts in siste d o n th e se ‘o th e r ’ issu es; in stead th ey c h o s e fro m
a m o n g th e o p tio n s e x p lic itly o ffered . H ow ev er, an d th is is th e se c o n d k ey p o in t, th ey did n o t
c h o o s e b e tw e e n th o s e e ig h t issu es in th e sa m e p r o p o r tio n s as r e sp o n d e n ts to th e o p e n -
e n d e d v e r sio n . In s te a d , th e y o p ted in d is p r o p o r tio n a te ly large n u m b e r s fo r ‘c r im e an d v io ­
le n c e ’ a n d ‘d e c r e a s in g tr u s t in g o v e r n m e n t’. In c o n tra s t, th e p e r c e n ta g e s fo r ‘in fla tio n ’ an d
‘u n e m p lo y m e n t’ a re b a re ly d iffe r e n t a c ro s s th e tw o v ersio n s. It s e e m s th a t e c o n o m ic issu es
c o m e to r e s p o n d e n ts ’ m in d s w ith o r w ith o u t p r o m p tin g , b u t th at c r im e is a p ro b le m o f
w h o se im p o r t a n c e r e s p o n d e n ts n e e d a r e m in d e r. (S e e a lso H ea th an d Jo h n s 2 0 1 0 .)
It is n o t stra ig h tfo r w a rd to asse s s w h ich o f th e se q u e s tio n s p ro d u c e s th e tr u e r p ic tu re o f
issu e s a lie n c e . It m a y b e th a t r e s p o n d e n ts p e rce iv e c r im e to b e a lo c a l issu e, an d so d o n o t
c o n s id e r it w h e n a sk e d th e o p e n -e n d e d v e rsio n o f th e q u e s tio n n a ire . T h u s th e o p e n q u e stio n
p ro v id e s th e n a r ro w e r fr a m e o f r e fe re n c e . W h ic h fo r m a t is p re fe rre d is th e re fo re d ifficu lt to
r eso lv e. T h e a r g u m e n t fo r c lo s e d q u e s tio n s is stro n g e r in th o s e c a se s w h e re su rv ey d esig n ers
a re c o n f id e n t th a t th e ir lis t w o u ld c o v e r m o s t, i f n o t all, o f th e m o re c o m m o n re sp o n s e s. T h e
b e st r o u te to su c h c o n f id e n c e is to u se o p e n q u e s tio n s in a p ilo t su rv ey, an d th e n to b ase th e
c lo se d list o n th e re su lts fr o m th a t p r e -te s tin g .
H O W T O D O R E S E A R C H IN P R A C T IC E

Table 10.2 Com parin g open and closed questions measuring the m ost im portant issue facing the country

Closed O p en

W hich of these is the most W hat do you think is the most

important problem facing the im portant problem facing the
country at present? country at present?

Food and energy shortages 6.2 17

Crim e and violence 361 161

Inflation 13 0 13.6

Unem ploym ent 20 4 19 6

Decreased trust in government 102 3.1

Busing 1.1 1.1

Breakdowns o f morals and religion 95 58

Racial problems 17 25

Other issues/problems 19 365

Total 100 0 100 0

N 422 449

Source: Adapted from Schuman and Presser (1996).

A further exam ple o f q uestion w ording effects relates to th e ton e o f th e wording. This is
illustrated by oth er split ballot experim ents that exam in e support for freedom o f speech in
the United States (Sch um an and Presser 1996). The results from th ese ex p erim en ts show
that w hen respondents were asked if the governm ent should forbid speeches against d e m o c ­
racy, support for freedom o f expression was m u ch higher than when respondents were asked
if the governm ent should allow speeches against d em ocracy.
The reason why logically equivalent qu estions can produce different responses accord ing
to the tone o f wording that is used can o ccu r for a variety o f reasons. In general, th ose who
are m ost susceptible to w ording effects are probably th ose for w hom the qu estion does not
have great resonan ce or im portance. People who design qu estionnaires for political research
tend to be very in terested in the subject, and to have thought a great deal about the issues.
However, these people also tend to be o utliers, and, for many respondents, the su b ject m atter
may not be o f any great personal im portance. These respondents with w eaker attitudes may
therefore be responsive to a variety o f external influences, ranging from a story on the prev i­
ous night’s news to a nuance in the question wording (Z aller 1992; Tourangeau et al., 2000).

Balanced questions
A sim ilar way in which question wording can influence responses is to do with w hether it is
balanced or not. In general, questions should be neutral. In p ractice, th is is difficult to
achieve, and questions may vary considerably in term s o f how neutral or balanced they are.
 SU RV EYS

O n e e x a m p le o f an u n b a la n c e d q u e stio n is w hat is o fte n d e sc r ib e d as a le a d in g q u e s tio n . A

le a d in g q u e stio n in d ic a te s to r e sp o n d e n ts th a t c e r ta in resp o n s e s are m o re u sefu l o r a c c e p t­
a b le to th o s e a sk in g th e q u e stio n (B e ls o n 1981 G en era lly , th ey are to b e avoid ed , sin c e th ev
ca n p ro m p t th e re sp o n d e n t to answ er th e q u e stio n in a c e r ta in way. O n e fa m o u s e x am p le o f
th is ty p e o f q u e s tio n th a t is still w idely used , is an item d esig n ed to m e a su re w h e th er
r e s p o n d e n ts id e n tify w ith a p o litica l party.
I h e o r ig in a l q u e s tio n d e v e lo p e d fo r th e N a tio n a l E le c tio n S tu d ies in th e U n ited S ta tes is:
G e n e r a lly sp e a k in g , d o you u su ally th in k o f y o u r s e lf as a R e p u b lica n , a D e m o c r a t, an In d e ­
p e n d e n t, o r w h a t? ' T h is q u e stio n h as b e e n e x p o rte d to m an y o th e r c o n te x ts, in clu d in g th e
B r itish E le c tio n Study, w h ich a sk s: ‘G e n e r a lly sp ea k in g , d o you th in k o f y o u r s e lf as Lab ou r,
C o n s e rv a tiv e , L ib e ra l D e m o c r a t, o r w h a t? ’
T h e u se o f o r w hat at th e en d m ig h t b e c o n s id e re d so m e w h a t ru d e, ev en th re a te n in g , and
c o u ld b e ta k e n to im p ly th a t n o t h a v in g su ch an id e n tific a tio n is a b it o d d . R esp o n d e n ts m ay
th e re fo re feel p ressu red to say th a t th e y d o id e n tify w ith o n e o f th e p a rtie s. Sim ilarly, m e n ­
tio n in g o n ly th e m a in p a r tie s m a y m a k e r e sp o n d e n ts less lik e ly to say th a t th e y th in k o f
th e m se lv e s a s p a r tis a n s o f o th e r p a r tie s w h ich are n o t m e n tio n e d . C e rta in ly , a ltern a tiv e
fo r m s o f a sk in g th is q u e s tio n in an o p e n -e n d e d fo r m a t rev eal m u c h lo w er lev els o f id e n tifi­
c a tio n (H e a th a n d Jo h n s 2 0 1 0 ) .
B u t q u e s tio n s c a n a lso b e u n b a la n c e d in o th e r w ays. A n d a p e r fe c tly b a la n c e d q u e stio n is
d iffic u lt to a c h ie v e . A c q u ie s c e n c e b ia s refers to a te n d e n c y a m o n g re sp o n d e n ts (o r so m e
ty p es o f r e s p o n d e n t) to a g re e w ith attitu d e s ta te m e n ts p re se n te d to th e m . T h e re are sev eral
d iffe re n t in te r p r e ta tio n s fo r th is. Im p lic it in m a n y o f th e se in te r p re ta tio n s is th a t so m e
g ro u p s o f p e o p le , su c h as th e p o o r ly e d u c a te d , are m o re lik e ly to la c k t r u e a ttitu d es o n issu es
p re se n te d to th e m . C o n v e r s e ( 1 9 6 4 ) la b elle d th e se n o n -a ttitu d e s . S in c e th e y h av e n o real
a n s w e r to th e q u e s tio n , th e y te n d to ju s t ag ree. A c q u ie s c e n c e b ia s is th e re fo re lik e ly to b e
m o re p r o n o u n c e d o n issu es o r attitu d e s th a t a re o f lo w sa lie n c e to th e r e sp o n d e n t, in w h ich
h e o r s h e la c k s c le a r ly d e fin e d v iew s, o r in ca se s w h e re th e q u e stio n is n o t c le a rly u n d e rsto o d .
T h e m o s t o b v io u s re m e d y to th is p r o b le m is to m ix p ro a n d a n ti sta te m e n ts, so th a t at least
th e e x te n t o f th e p r o b le m c a n b e id e n tifie d . R ela ted to a c q u ie s c e n c e b ia s are so m e th in g
k n o w n a s r e c e n c y e ffe c ts , w h ich r e fe r to th e o r d e r in w h ic h r e sp o n s e s are p re se n te d to th e
r e s p o n d e n t in c lo s e d - f o r m a t q u e stio n s. R e c e n c y e ffe cts are said to o c c u r w h e n re sp o n d e n ts
se le c t th e la st o p tio n th e y a re o ffered . T h e y a re g e n e ra lly th o u g h t to b e m o re lik e ly to o c c u r
w h e n th e q u e s tio n is lo n g o r c o m p lic a te d , o r w h e n th e r e sp o n d e n t d o e s n o t h av e a c r y s ta l­
liz e d ’ o p in io n .

Response format
A n o th e r im p o r t a n t a s p e c t o f q u e s tio n n a ir e d esig n rela tes to h o w to r e c o rd th e r e s p o n d e n ts
a n sw er, o r m o re s p e c ific a lly w h at ty p es o f r e sp o n s e o p tio n to p ro v id e th e m w ith . T h e re are
m a n y issu es at sta k e h e re . T h e first c o n c e r n s th e n u m b e r o f r e sp o n s e c a te g o rie s th a t are
p r o v id e d — a n d w h e th e r d ic h o to m o u s (yes/ no) o r p o ly to m o u s (e.g . lev els o f a g r e e m e n t)
r e s p o n s e c a te g o r ie s a re u se d . I f p o ly to m o u s r e sp o n s e s c a le s are to b e u se d , th e n h o w m an y
s h o u ld b e u se d a n d s h o u ld a m id -p o in t b e in clu d e d ? T h e re is little sta n d a rd p r a c tic e an d
th e re is c o n s id e r a b le v a ria tio n in te r m s o f w h at is g e n e ra lly u se d , th o u g h , fo r m o s t to p ics,
fiv e c a te g o r ie s a re p r o b a b ly su fficie n t.
H O W T O D O R E S E A R C H IN P R A C T IC E

Standard qu estio ns generally do not offer a ‘D on ’t kn o w ’ category, but will accep t d on ’t

know as a valid response if it is spon taneou sly offered by th e respondent. T h e ration ale
b ehin d th is approach is to try and p rob e an op in ion . But in som e cases th is p rob in g can
invent data, as the respond ent will be forced in to providing an o p in io n th at th ey do n o t really
hold. Indeed, C onv erse (1 9 6 4 ) suggests th at m u ch o f the e rro r th at oc cu rs in surveys co m es
from the rand om responses that are provided by p eople with n o real opin ion . To d istingu ish
betw een ‘real attitudes’ and 'n on -attitu d es’ it is th erefo re o ften a good idea to provide a q u es­
tion to gauge the resp ond ent’s level o f in terest or know ledge ab out th e top ic b efore (o r after)
asking about th eir o pin ion .

Who you ask: sample design

So far, we have discu ssed issues relating to q u estio n n aire design, and how th e qu estio n s you
ask and th e way in w hich they are asked can affect th e answ ers you get. Equ ally im p o rtan t is
the issue o f sam pling design. This is because surveys rely on sam ples, w hich are selected
from the p o p u latio n s that we seek to study. In sam pling term inolog y, th e pop u lation o f
interest is a particu lar group o f people or oth er units (su ch as ho useho lds, b u sin esses,
scho ols, etc.) that the research er w ishes to study. Thus, for a study on voting b eh av iou r in a
given country, the popu lation o f in terest m ay be all o f th e elig ib le v oters. W e rarely in ter­
view everybod y in the popu lation (w hen we do, it’s called a censu s) b ecause it is incred ib ly
tim e-co n su m in g and expensive, and so we must th erefo re c h o o se w ho we select to in ter­
view. The purpose o f the sam ple survey is to allow the research er to m ake a gen eralization
about the attribu tes o f a given popu lation w ithout having to actu ally in terview ev eryone in
that popu lation to find out what they th in k and do. In ord er to do th is, th e survey relies
upon in terview ing a sam ple o f th e popu lation. How we do th is has a big im p act on the
results we get. If the sam ple is chosen in a haphazard or su b jectiv e way, th en th ere is little
hope o f m ak ing accu rate gen eralization s about the w ider p opu lation we are in terested in.
There are thus three m ain sou rces o f erro r that can in fluen ce how representative th e sam ple
is o f the population (see Figure 10.1). These relate to cov erage e r ro r (w hat we ch o o se our
sam ple from ), sam pling error (how we ch oo se ou r sam ple), and n o n -re sp o n se (how m any
people from our sam ple com plete the survey).

Defining the population: sampling frame

The sam ple fram e is the set o f the target population that has a chan ce to be selected in the
survey sample. O nce we have defined our population o f interest, we must ob tain a sam p lin g
fram e, which contains a com plete (or as com plete as possible) list o f all the popu lation units.
A frequently used sam pling fram e for national surveys is the Post O ffice Address file, which
contains an alm ost com plete list o f addresses in the UK. It is im portant to rem em ber that a
random probability sam ple alone does not guarantee that a given sam ple will be rep resenta­
tive if the sam pling fram e from which the sam ple is drawn does not cover the entire popu la­
tion. U nder-coverage or coverage error can b ecom e a problem if certain segm ents o f the
population are left oH the sam pling fram e. For exam ple, the Electoral Register is one p o ssi­
ble sam pling frame tor registered voters in the UK, but it will not tell us about people who
 SU RV EYS 243

are no t reg iste red to v ote. It is th e re fo re no t su ch a g o o d sam p lin g fra m e lo r th e p o p u latio n

as a w h o le.

Choosing the sample: random probability sampling

A g o o d sa m p le is re p re s e n ta tiv e o f th e p o p u la tio n fro m w h ich it is d raw n. T h is allow s us to

g e n e ra liz e o u r fin d in g s tro m th e su rv ey to th e p o p u la tio n . T h ere are m an y d ifferen t m e th o d s
tor d ra w in g a sa m p le. S in c e th e re is n o way o f k n o w in g w hal o u r p o p u latio n lo o k s like in
ev ery re sp e c t, it is d ifficu lt to assess h o w w ell we hav e m a tc h ed o u r sam p le to th e p op u latio n .
But th e re are s o m e w ays th at w ill ten d to p ro d u c e b e tte r resu lts th an o th ers. T h ere are two
m a in w ays o f d ra w in g a sa m p le: p r o b a b ility sam p les, w h ich are draw n at ra n d o m ; and n o n ­
p ro b a b ility sa m p le s, w h ich are sele cted m o re su b jectiv ely . To b e rep resen tativ e in a sen se
a p p ro p ria te for s ta tistic a l in fe re n c e , a sa m p le m u st b e a p ro b a b ility sam p le, w here ev ery o n e
in th e p o p u la tio n h a s a k n o w n , n o n -z e r o p ro b a b ility o f b e in g sam p led . G en erally , m o st a c a ­
d e m ic su rv ey s follo w th is a p p ro a ch . N o n -p r o b a b ility sam p les are m o re freq u en tly used in
M a rk e t R e se a r c h a n d 'O p in io n P o lls’, an d a lth o u g h th ey are no t reliab le in a sta tistic a l sen se,
th ey d o o fte n p r o d u c e re su lts th a t are re a so n a b ly a c c u ra te . H ow ever, w h en th ey do go w rong,
th ey c a n d o so s p e c ta c u la rly . (S e e B o x 1 0 .1 .)
P r o b a b ility sa m p le s a ssig n e a c h u n it in th e s a m p lin g fra m e a k n o w n n o n -z e r o c h a n c e o f
b e in g se le c te d . T h e im p o rta n t th in g is th at th e p ro b a b ility o f b e in g sele cted is k n o w n for each
u n it; it d o e s n o t hav e to b e eq u a l. F o r e x a m p le , su rv ey r e se a rch e rs freq u en tly o v er-sam p le
sm a ll b u t in te r e s tin g g ro u p s in th e p o p u la tio n so th ey c a n b e a n aly sed in g reater d etail. T h ere
a re fo u r m a in ty p es o f sa m p le d e sig n b a sed o n p ro b a b ility sam p lin g . T h ese are: (1 ) sim p le
ra n d o m sa m p lin g ; ( 2 ) s y ste m a tic sa m p lin g ; (3 ) stra tified sam p lin g ; an d (4 ) clu s te r sam p lin g .
S im p le r a n d o m s a m p lin g (S R S ) is p e rh a p s th e m o st stra ig h tfo rw a rd typ e o f p r o b ab ility
sa m p lin g . E v ery u n it in th e s a m p lin g fr a m e h a s th e same p ro b a b ility o f b e in g sele cted . T h e
u n its are a ssig n e d n u m b e r s , a n d th e n ra n d o m n u m b e r ta b les are u sed to sele ct th e sam p le.
T h is m e th o d , th o u g h , is n o t w ith o u t d ra w b a c k s . T h e first is th at it ca n b e v ery e x p en siv e to
c a rr y o u t w h e n in v e stig a to rs h av e to in te rv ie w th e r e sp o n d e n ts p erso n a lly b eca u se th ey m ay
n ee d to tra v el lo n g d is ta n c e s in o r d e r to a tte m p t an in terview . T h e o th e r d raw b ack is th at

b o x i o. i Sampling Design: How Not to Do It

A fam ous exam ple o f the dangers o f non-probability sampling is the survey by the Literary Digest
m agazine to predict the results of the 1936 US presidential election. The magazine sent out about
10 m illion questionnaires o n postcards to potential respondents, and based its conclusions on those
that w ere returned. This introduced biases in at least two ways. First, the list of those w ho w ere sent
the questionnaire w as based on registers such as the subscribers to the magazine, and of people with
telephones, cars, and various club memberships. In 1936, these w ere mainly w ealthier people w ho
w ere m ore likely to be Republican voters, and the typically poorer people not on the source lists had
no ch ance of being included. Second, only about 25% of the q uestionnaires w ere actually returned,
effectively rendering the sam ple into a volunteer sample. The magazine predicted that the Republican
candidate Alf Landon would receive 57% of the vote, w hen in fact his Dem ocratic opponent, F. D.
Roosevelt, gained an overw helm ing victory with 62% of the vote. The o utcom e of the election was
predicted correctly by a m uch sm aller probability sam ple collected by G eorge Gallup.
244 H O W T O D O R E S E A R C H IN P R A C T IC E

in terestin g but sm all sub-groups o f the p opu lation m ay n ot be selected in sufficient nu m b ers
to ca rry out m eanin gful analysis. It is th erefo re not used very o ften.
System atic sam pling is a slight variation on sim ple random sam pling, and shares m any o f
the sam e strengths and weaknesses. Researchers ch oo se a random starting p oin t on th e s a m ­
pling fram e, and then select respondents at fixed intervals. So, for exam ple, i f th ere are 100,000
units listed on the sam pling fram e, and the aim o f th e survey is to interview 1,000 people, th en
the researcher will start at a random position , for exam ple at nu m b er 8 7 6 , and th en interview
every 100th person. By and large, this produces the sam e results as a sim ple ran d o m sam ple.
The only d ifference that may em erge is if th ere is p eriodicity in the sam pling fram e, so that a
certain type o f person is listed at regular intervals w ithin the sam pling fram e. For exam ple, if
th e sam pling fram e is a database o f m arried couples, and th e data is arranged so that w om en
are followed by their husbands, th en every second person in th e sam pling fram e will b e a
man. Accordingly, the 876th person on the list will be a m an, as will every 100th person that
follows this starting point. T he sam ple that is drawn will th erefore be entirely m ade up o f m en.
Probability sam ple designs can be m ade better by introducing special features to ensure that
the sample adequately represents specific sub-groups o f the population that m ay be o f particular
interest. Stratified sam pling is an example o f this. It involves setting different selection p robabil­
ities for different groups (or strata) in the population. It is frequently used to ensure that ade­
quate num bers o f people are selected from different regions o f a country. For example, a survey
o f British adults based on simple random selection m ight by chance lead to a sam ple containing
very few people from Wales or Scotland, and lots o f people from England. To ensure that the
sample represents the distribution o f the population across these three regions, the sam pling
frame can be stratified (divided up) into separate lists for each country. Stratifying sam ples obvi­
ously requires that the sam pling fram e contains inform ation regarding the stratifying variable
(e.g. country). The sample can then de drawn from each list. The proportion o f the sam ple
drawn from each stratum can either be set to ensure that the sam ple is representative o f Britain
as whole, or can vary by strata so that it over-samples from those regions where fewer people
live. This allows for the possibility o f having a bigger sam ple size from Wales or Scotland than
would otherwise be achieved, which in turn can facilitate the statistical analysis o f people living
in these regions. These unequal selection probabilities can then be corrected by w eighting the
data (see Box 10.2) to ensure that the overall sample is representative o f Britain as a whole.
A lthough this m ethod helps to ensure the representativeness o f the sam ple, it does little to
ease the costs o f carryin g out the survey that can m ake surveys based on sim ple random
selection prohibitive. A technique to try and m ake this cost m ore m anageable is cluster s am ­
pling. W ith cluster sam pling, the units are not sam pled individually but in groups (clusters).
fo r example, a survey may sam ple households and then interview every adult w ithin each
selected household. Although this saves on transp ort costs, it increases sam pling error, since
people living in the sam e household might share certain characteristics, such as social class,
which may not he representative o f people living in oth er households. M ulti-stage cluster
sam pling is an extension ol this approach and is often used for large national surveys. This
involves drawing several samples. At the first stage, the population is divided up into g eo ­
graphic regions, such as electoral constituencies, which form the clusters. These clusters are
then selected using SRS. At the second stage, each o f the selected clusters are then divided up
into even sm aller geographic units, such as electoral wards, w hich are again selected using
SRS. At the third stage, addresses from within each electoral ward are selected, and then
 SU RVEYS 245

b o x 10 .2 Weighting the Data

People often talk about weighting the data, and about weighted and non weighted data, though it is
not always clear what this refers to Broadly speaking, there are two main ways in which this can be
done. The most straightforward form of weighting is to correct for unequal selection probabilities So,
for example, if we deliberately over-sam ple people living in Scotland, Wales, and Northern Ireland (so
that we have a sufficiently large sam ple size to analyse people living in these countries separately), our
com bined sam ple will not be representative of the UK population as a whole To correct this, we apply
weights to the data, w hich adjusts our sam ple and statistically reduces the number of people living
in Wales in our sam ple by the sam e degree to which we over-sampled them at the design stage. This
brings the proportion of people living in W ales w ithin our sam ple back into line with the proportion of
people living in Wales within the UK. This is a relatively straightforward procedure, since the selection
probability is known for every selected case.
The second w ay in w hich weights are often applied is to correct for unit non-response. This is more
problematic, since it involves estim ation. For example, suppose we find that response rates to our survey
are higher am ong o lder people than younger people. O ur achieved sam ple will therefore be skewed
towards o lder people, and will not be representative of the population as a whole in terms of age.
This can introduce bias if young people and old people differ in their political attitudes or behaviour.
To com pensate for this under-representation of young people, an adjustment is made to our sample
w hich statistically increases the proportion of young people w ithin our sample, so that it matches the
proportion of young people in the population as a whole.
The problem with this strategy is that it assumes that non-response is random, and that young people
w ho d o not answ er the survey are essentially the sam e as young p eople w ho d o answer the survey.
However, this assum ption is not always valid. Young people w ho d o not respond to the survey m ay be
quite different in important respects. T hey m ay be less interested in politics, they may be less inclined
to give up their tim e for nothing, o r they m ay simply be occupied with w ork and going o u t In these
instances, applying n on-response weights will do little to reduce any bias that has crept into the survey.

w ith in e a c h h o u s e h o ld a re sp o n d e n t is se le cte d to in terv iew . T h e ad v an tag e o f th is ap p ro ach

is th a t ty p ic a lly o n e in v e stig a to r c a n c a r r y o u t all th e in te rv ie w s w ith in ea ch clu ster, s in ce all
th e se le c te d r e s p o n d e n ts w ill b e w ith in a relativ ely c o n fin e d g eo g ra p h ic a l area.

Non-probability sampling
E v e ry th in g e lse b e in g e q u a l, fo r p o p u la tio n su rv ey s, p ro b a b ility sam p les are alw ays p r e ­
ferred to n o n - p r o b a b ility sa m p les. N o n -p r o b a b ility sa m p les are n o t o n ly m o re likely to
in tro d u c e e r r o r in to th e su rv ey , b u t th e y are lik e ly to d o so in w ays w h ich are d ifficu lt to
p re d ic t. T h is m a k e s it m u c h m o re d ifficu lt to h av e c o n fid e n c e in th e resu lts. T h e m o st c o m ­
m o n fo r m o f n o n -p r o b a b ility s a m p lin g is th e q u o ta sam p le.
Q u o t a s a m p lin g r e q u ire s in v e stig a to rs to in te rv ie w a c e r ta in n u m b e r (q u o ta ) o f r e sp o n d ­
e n ts, w h e re q u o ta s a re set a c c o r d in g to p e r so n a l c h a r a c te r is tic s , su ch as age grou p , sex , and
in c o m e gro u p . T h e s e q u o ta s a re o rg a n iz e d to tr y a n d get a sp read o f d ifferen t typ es o f p eop le,
w h ic h c a n th e n b e w eig h ted to e n s u re th a t th e sa m p le is rep re s e n ta tiv e o f th e p o p u la tio n in
th e se r e s p e c ts . Q u o ta sa m p le s te n d n o t to m a k e e x p lic it u se o f s a m p lin g fra m e s an d so are
s u s c e p tib le to c o v e r a g e e rro r. Q u o ta s a m p lin g is n o n -r a n d o m b e ca u se in terv iew ers can
s e le c t a n y s u b je c t s th e y w a n t th a t fit th e s e le c tio n c rite r ia . T h is is u su ally d o n e u sin g so m e
246 H O W T O D O R E S E A R C H IN P R A C T IC E

(un stated ) form o f purposive or c o n v en ien ce sam pling, in w hich in vestigators approach
th ose people w ho look m o st cooperative. A lthough qu ota sam pling is qu ite co m m o n , and
generally gives reasonable results, it is easy to in tro d u ce biases in th e selection stage, and
alm ost im possible to know w hether th e resulting sam ple is a representative one.
O th e r form s o f n o n-p robab ility sam pling are even m ore unreliable. P u rp o s iv e sam p lin g
involves investigators using th eir ow n ‘ex p ert’ ju d gem en t to select respond ents w hom th ey
consid er to be typical or representative o f th e popu lation o f interest. It suffers from m an y o f
the sam e problem s as qu ota sam ples, sin ce it is v ery easy to in tro d u ce bias in to th e selection .
Sim ilarly, snow b all sam p lin g involves investigators finding respondents w ho m eet som e
criteria o f interest, such as b eing an anim al rights activist or drug user, and th en askin g the
respondent to suggest oth er sim ilar types o f people w ho th ey m ay know for th e research er to
con tact for interview. In this way, the research er is able to build up a sam ple o f respondents.
However, the problem with th is tech niq u e is that the cases that are selected are no t in d e­
pendent o f each other, and are often not in dependen t o f the in itial starting p oin t from w hich
the snowball was generated. For exam ple, using th is sam pling strategy, B eck er (1 9 6 3 )
attem pted to interview drug users th rou gh his co n tacts in the m u sic business. U n su rp ris­
ingly, the resulting sam ple was heavily biased tow ards m u sician s, w ho m ade up h a lf o f his
sam ple. It was not therefore very representative o f the popu lation o f interest. Snow ball sa m ­
ples tend to be very unreliable, and are not a sound basis for m aking w ider in feren ces. H ow ­
ever, they are som etim es used in qualitative research, particularly when the p opu lation o f
interest is hard to reach, and does not have a reliable sam pling fram e (see C hapter 11).
However, perhaps the m ost unreliable o f all sam pling strategies is the v o lu n teer sam ple.
This is the sort o f survey that is frequently carried out by m o rn in g television shows and
internet sites, where viewers or readers are encouraged to express th eir op in ion s on various
topical issues o f interest. For exam ple, consid er what in feren ces you can draw from the fo l­
lowing exam ple o f a Sky News viewers’ poll:

Should Britain jo in the space race?

N o — It's cosm ic showboating 52%

O kay, but let’s spend sensibly 30%
Y es— I d be over the Moon! 18%

P u ttin g a sid e issu es to d o w ith q u estio n w o rd in g , it tells u s v e ry little. W e ’d be u n w is e to in fe r

that the B ritish p u b lic is a g ain st jo in in g the sp ace race. W e’d ev en be u n w is e to in fe r that S k y
vie w e rs are a g ain st jo in in g the sp a ce race. A ll w e le a rn fro m th is typ e o f e x e rc ise are the
o p in io n s o f tho se rea d ers o r v ie w e rs w h o c o u ld be b o th ered to se n d in th e ir re sp o n se . F o r
these reaso n s, v o lu n teer sam p les (even the m o re s e rio u s o n e s) tell u s e ss e n tia lly n o th in g
about the average attitu d es o f the g en era l p o p u la tio n .

Sampling error
Hven tho u gh p ro b a b ility sa m p les ten d to be m o re re p resen ta tive o f th e p o p u la tio n th a n n o n -
p ro b ab ility sa m p les, the y are still affected by d ifferent ty p es o f sa m p lin g e rro r. S a m p lin g
e rro r can o c c u r for a v a riety o f r ea so n s. T w o m a in fa cto rs that are im p o rta n t are to d o w ith
the resp o nse rate that the su rv e y a ch ie ves an d the sa m p le size o f the su rv ey.
 SU RV EYS 247

Ideally, e v e r y o n e th at is sa m p led will c o m p le te th e survey. H ow ever, th is rarely happ ens,

an d n o n -re s p o n s e ca n o c c u r for a v ariety o f reaso n s. S o m e p eop le will refuse to p articip ate, or
w ill b e u n a b le to p a rtic ip a te b e ca u se th ey are to o sick o r b ecau se th ey are away on b u sin ess.
N o n -r e s p o n s e m a y th e re fo re red u ce sam p le size and in tro d u c e bias. T h o se peop le w ho do not
a n sw er th e su rv ey m a y b e so m ew h a t d ifferen t in so m e key resp ects from th o se peop le w ho do
an sw er su rv ey s. If s o m e o n e is no t in terested in p o litics, th ey m igh t no t w ant to spen d an h o u r
a n s w e rin g q u e stio n s a b o u t th e m o st rec en t e le c tio n . Sim ilarly, if so m e o n e is to o sick to be
in terv iew ed , th ey m ig h t b e to o sic k to d o lots o f o th e r th in g s as well, such as go out to vote.
A n e x a m p le o f th is rela tes to th e m e a su r e m e n t o f tu rn o u t. It is no t u n c o m m o n to find that
th e p r o p o r tio n o f r e sp o n d e n ts w h o r e p o rt v o tin g in su rv ey s far e x c eed s th e official e stim ates
o f a c tu a l v o te r tu rn o u t. F o r ex a m p le , in th e 2 0 0 5 B E S , se lf-r e p o r te d tu rn o u t w as m easu red
at 7 9 % , w ell a b o v e th e o ffic ia l tu rn o u t fig u re o f 6 1 % . T h is su b sta n tia l d iscre p a n cy b etw een
su rv ey e s tim a te s o f s e lf-r e p o r te d tu rn o u t an d o fficial e stim a te s o f tu rn o u t is n o t ju s t lim ited
to B r ita in , a n d is c o m m o n to m o st su rv ey s ca rr ie d ou t aro u n d th e w orld (an d has b een a
p a r tic u la r s o u rc e o f c o n c e r n in , fo r ex a m p le , th e U n ited S ta te s). A study by Sw addle and
H ea th ( 1 9 8 9 ) e x a m in in g th e so u rc e s o f th is d is c r e p a n c y fo u n d th at a c o n s id e ra b le p o r tio n o f
it w as d u e to p r o b le m s o f r e sp o n s e b ias (w h e re tu rn o u t a m o n g n o n -re sp o n d e n ts is su b s ta n ­
tia lly less th a n tu rn o u t a m o n g su rv e y r e sp o n d e n ts).
N o n -r e s p o n s e m a y in tro d u c e b ia s i f th o se in d iv id u a ls w h o d o n o t ag ree to b e in terv iew ed
fo r th e su rv e y ( o r c a n n o t b e in te rv ie w e d b e c a u se th e y are aw ay) are less likely to v o te th an
th o s e in d iv id u a ls w h o a g ree to b e in terv iew ed . Sw ad d le an d H eath fo u n d th at, a c c o r d in g to
th e o ffic ia l tu rn o u t reg iste r, tu rn o u t a m o n g th e su rv ey re sp o n d e n ts (8 3 % ) w as su b stan tially
h ig h e r th a n a m o n g th e n o n -r e s p o n d e n ts (7 5 % ). A su b sta n tia l p o r tio n o f th e ap p aren t o v e r­
r e p o rt in tu rn o u t b y su rv ey s is th u s a ttrib u ta b le to issu es o f re sp o n s e an d co v erag e. O n e
o b v io u s w ay in w h ic h su rv e y m e a s u r e m e n t ca n b e im p ro v ed is th e re fo re to en s u re a high
r e sp o n s e rate. H ow ev er, th is is n o t easy to ach ie v e , an d in re ce n t y ears resp o n se rates for
fa c e - to - f a c e su rv e y s h a v e fa lle n to a r o u n d 6 0 % . A lth o u g h re se a rch e rs c a n try to c o r re c t for
n o n - r e s p o n s e b ia s b y w e ig h tin g th e d ata, th is is n o t a p e r fe c t rem ed y an d d o es n o t get rid o f
th e p r o b le m .
S u rv e y o r g a n iz a tio n s th u s d e v o te c o n s id e ra b le tim e an d e n e rg y try in g to m a x im iz e
r e sp o n s e . V a r io u s s tra te g ie s a re e m p lo y ed , su ch as s e n d in g an a d v an ced le tte r to th e r e sp o n d ­
e n ts, v isitin g th e r e s p o n d e n t in p e r s o n , w ith rep eat v isits o n d iffe re n t days at d ifferen t tim e s
if th e r e s p o n d e n t is u n a v a ila b le . O n e stra te g y th a t is b e c o m in g m o re c o m m o n is th e u se o f
in d u c e m e n ts , su c h a s a c a sh in c e n tiv e o r g ift. It is u n c le a r ho w m u c h d iffe re n ce th e se in d u c e ­
m e n ts a c tu a lly m a k e th o u g h . R e se a r c h o n d iffe re n t ty p es o f in d u c e m e n t sh o w th at e n te rin g
th e r e s p o n d e n t in to a p riz e d ra w h as n o im p a c t o n re sp o n s e , w h e reas an u n c o n d itio n a l p ay ­
m e n t ( w h e th e r th e y c o m p le te th e su rv e y o r n o t) is m o st effectiv e. H ow ev er, th is can o b v i­
o u s ly in c r e a s e th e c o s ts o f th e p r o je c t su b stan tially .

How you ask: interview mode

T h e la st sta g e o f s u r v e y d e sig n th a t w e c o n s id e r is h o w th e su rv ey is a d m in iste re d . T h is is an

im p o r t a n t p a r t o f d a ta c o lle c tio n . N o m a tte r h o w w ell a su rv ey is d esig n ed , i f it is p o o rly
e x e c u te d , th e n it w ill p r o d u c e u n re lia b le resu lts. O v e r th e p ast 2 5 y ears o r so th e re h as b een
H O W T O D O R E S E A R C H IN P R A C T IC E

a rapid expansion in the ways— or m o d es— that surveys can b e carried out. T h e m o st
co m m o n ways are face-to -face, by telephone, by post, and, m o re recently, by in tern et. Th e
way in w hich the survey is carried out can in fluen ce b o th how people answ er th e qu estio n s,
relating to issues o f m easurem en t error, and w ho answ ers th e qu estio ns, relatin g to issues o f
sam pling error (see G roves et al. 2 0 0 4 for an extended d iscu ssion ). W e briefly d escrib e the
m ain differences betw een each o f these m o d es o f data c o lle ctio n , and th en d iscu ss som e o f
the issues they raise.
Fa ce-to -fa ce interview s rem ain the gold standard in survey research , and m o st o f th e big
acad em ic and g overnm ent surveys are carried out in th is way. However, th ey are v ery ex p en ­
sive. Respond ents are interview ed by trained investigators w ho ad m in ister th e q u estio n ­
naire personally, usually at the respondent s hom e. Investigators read out th e q u estio ns to the
respondent, show prom pt cards with the responses (for closed qu estio ns) and record the
verbal answers that the respondent provides. T raditionally, these interview s w ere carried out
with pen and paper, but now they tend to be carried out using laptops (com p u ter-assisted
personal in terview ing— C A P I), which can be used to play video clips and speeches. F ace-to -
face surveys can clearly establish who is answ ering the qu estions, and w hether th ey are
doing so freely and com fortably w ithout in terventio n from oth er people. T elephon e in ter­
views are also ad m inistered personally by trained investigators, th ough obviously in a m ore
rem ote way sin ce they take place over the phone. They do not provide response cards and so
lack visual cues. By contrast, both postal surveys and in te rn e t surv eys are self-ad m inistered
by the respondent, who must fill out w ritten versions o f the q u estio nnaire by them selves.
Internet surveys in particular are b ecom ing m ore popular, and are frequently used in a c a ­
d em ic research. They have the advantage o f b eing relatively cheap to adm inister, and th ere is
now a whole host o f w ebsites on the in ternet w hich can be used to ho st the survey at rela­
tively little cost, w hich can m ake it seem an attractive option for researchers with a lim ited
budget.
Besides budget, there are a nu m ber o f oth er im portant factors that need to be considered.
Ih e way in which a qu estionnaire is adm inistered may in fluen ce the way in w hich resp on d ­
ents answer the questions in a num ber o f different ways. First, face-to -face surveys (and to a
lesser extent telephone surveys) can help to alleviate co m p reh en sio n prob lem s. The in ter­
viewer can make sure that the respondent has understood the question properly, and can
repeat a question if necessary. Sin ce personally adm inistered interview s are carried out ver­
bally, they also place less o f a burden on the literacy skills o f the respondent. By co ntrast,
people with reading and w riting difficulties may feel less confident and be less able to co m ­
plete self-adm inistered surveys.
However, there is another side to this. The presence o f an investigator can in som e cases
inhibit the responses that are given. These are known as interview er effects. To illustrate, Schu-
man and ( .onverse (1971) carried out a study on w hether the race o f the interviewer influenced
respondents’ answers towards questions on race relations. They found consistent interviewer
eflects, whereby black respondents were much more likely to say that they trusted w hite people
if they were interviewed by a white investigator than if they were interviewed by a black inves­
tigator. Ihe more anonymous setting of internet and postal surveys may therefore help to alle­
viate these problems, and in the process may help to reduce social desirability bias which can
affect certain sensitive questions, l or these reasons, for sensitive topics, face-to-face surveys
 SU RVEYS

using C A F I o fte n pass th e laptop to th e resp o n d e n t and ask th em to answ er th e qu estio n s on

th e sc re e n by th em selv es. W ith resp ect to m e a su rem en t issues th en , th e m o d e o f in terview that
is p referred m a y vary, an d d ep en d in part u p o n th e typ e o f q u estio n s that are b ein g asked in the
survey, an d w h e th er th ey are co m p le x on th e o n e han d o r sen sitive on th e other.
T h e way in w h ic h a q u e s tio n n a ire is a d m in iste re d ca n also in flu e n ce th e rep re s e n ta tiv e ­
n ess o f th e sa m p le. In p a rtic u la r, p r o b le m s to d o w ith co v erag e e r ro r and n o n -re s p o n s e m ay
v a ry by in te r v ie w m o d e (se e G ro v e s et al. 2 0 0 4 ). T h e b est way to en s u re th at a sam p le is r e p ­
re se n ta tiv e o f th e p o p u la tio n is to d raw a ra n d o m sam p le, in w h ich all m e m b e rs o f th e p o p u ­
la tio n h a v e a k n o w n n o n -z e r o p ro b a b ility o f b e in g sele cted . T o a c h iev e th is it is n ece ssa ry to
d efin e th e p o p u la tio n a n d to s e le ct an a p p ro p ria te sa m p lin g fr a m e fro m w h ich to draw th e
sa m p le. H ow ev er, th e c o v e ra g e o f c o m m o n targ et p o p u la tio n s v aries by d ifferen t m e th o d s o f
d a ta c o lle c tio n , a n d ev en w h e re c o v e ra g e is h ig h , it is n o t alw ays p o ss ib le to find a r e lia ­
b le s a m p lin g fra m e . F o r fa c e - to - fa c e su rv ey s an d p o stal su rv ey s, th e m o st co m p re h e n siv e
n a tio n a l s a m p lin g fr a m e fr o m w h ic h to d raw a sam p le is th e P ost O ffice A d d ress file (o r
e q u iv a le n t). T h is ty p e o f sa m p lin g fra m e ten d s to co v e r p retty m u ch all th e a d d resses in a
c o u n tr y a n d so d o e s n o t in tro d u c e m u c h co v era g e erro r. H ow ev er, o b ta in in g a sam p lin g
fr a m e w ith g o o d n a tio n a l c o v e ra g e is m u c h m o re p ro b le m a tic fo r te le p h o n e in terv iew s and
in te r n e t in te r v ie w s, a n d th is c a n in tro d u c e g re a te r co v era g e error.
F o r te le p h o n e su rv ey s, ev en th o u g h th e p r o p o r tio n o f h o u se h o ld s w ith a te le p h o n e ten d s
to b e q u ite h ig h in d e v e lo p e d c o u n tr ie s (ty p ic a lly o v e r 9 0 % ), it is n o n e th e le ss d ifficu lt to
o b ta in a c o m p re h e n s iv e s a m p lin g fra m e . F o r e x a m p le, th e p h o n e b o o k (Y ello w Pages or
e q u iv a le n t) is n o t v e r y r e p re s e n ta tiv e o f th e p o p u la tio n b e c a u s e o f th e large n u m b e r o f p e o ­
ple w h o a re e x - d ir e c t o r y ( n o t lis te d ). F o r e x a m p le , in th e U K a b o u t 2 5 % o f th e p o p u la tio n
w h o h a v e a te le p h o n e a re n o t lis ted in th e p h o n e b o o k . E v en th o u g h th e ov erall p r o p o r tio n
o f th e p o p u la tio n w h o h a v e a te le p h o n e h a s in c re a se d in r e c e n t y ears, th e p r o p o r tio n o f th e
p o p u la tio n w h o a re n o t lis ted in th e p h o n e b o o k h as also in c re a se d . T h e p h o n e b o o k is th e re ­
fo re n o t a p a r tic u la rly g o o d sa m p lin g fr a m e , an d re se a rc h e rs w h o d o te le p h o n e su rv ey s ten d
to u se r a n d o m d ig it d ia llin g in ste a d . T h is in v o lv es d ia llin g n u m b e r s at ra n d o m . A lth o u g h
th is r e d u c e s c o v e ra g e er ro r , it in v o lv es m a k in g a lo t o f w asted ca lls to n u m b e r s th a t aren ’t in
u se, a n d d o e s n o t d istin g u is h b e tw e e n b u sin e ss a n d re sid e n tia l a d d resses. T h is ca n m ak e it
d iffic u lt to c a lc u la te r e s p o n s e r a tes (se e b e lo w ).
F o r in te r n e t su rv e y s, th e p a st few years h av e s e e n a rap id in c re a se in th e p r o p o r tio n o f
B r itis h h o u s e h o ld s w ith in te r n e t a c c e s s , an d a c c o r d in g to th e O ffice o f N a tio n a l S ta tistic s,
a b o u t 7 0 % o f B r itis h h o u s e h o ld s n o w h av e in te r n e t a cc e ss. B u t th is is still a lo n g w ay aw ay
fr o m fu ll c o v e r a g e , a n d a n y su rv e y c a rr ie d o u t th ro u g h th e in te r n e t w ill leav e o u t a su b s ta n ­
tia l p r o p o r tio n o f th e p o p u la tio n . M o re o v e r, th e so -c a lle d ‘d ig ital d iv id e b e tw e e n th o s e w h o
are o n lin e a n d th o s e w h o a re n o t m e a n s th a t th e re are also lik e ly to b e im p o rta n t d e m o ­
g r a p h ic d iffe r e n c e s b e tw e e n th e tw o g ro u p s, w h ic h m a y in tro d u c e fu r th e r b ias (see R o b in ­
so n , N e u sta d tl a n d K e s tn b a u m 2 0 0 2 ) . M o re o v e r, fo r in te r n e t su rv ey s, n o g o o d sam p lin g
fr a m e h a s b e e n d e v e lo p e d fo r s a m p lin g th e in te r n e t p o p u la tio n (se e C o u p e r 2 0 0 0 , fo r a
re v ie w ). S u rv e y c o m p a n ie s th e r e fo r e o fte n rely u p o n d ra w in g sa m p les fr o m a d a ta b a se o f
v o lu n te e r s , w h ic h c a n th e n b e w eig h ted to m a k e th e sa m p le rep re s e n ta tiv e o f th e p o p u la tio n
w ith r e s p e c t to k e y d e m o g ra p h ic v a ria b le s. T h is s a m p lin g stra te g y m ix e s e le m e n ts o f th e
v o lu n te e r s a m p le a n d th e q u o ta sa m p le , d isc u sse d prev iou sly.
250 H O W T O DO R E S E A R C H IN P R A C T IC E

However, despite th ese problem s, recent research show s th at in som e cases th ere is very
little observ able difference betw een analyses using in -p erso n data and in tern et data (see
Sand ers et al. 2 0 0 7 ). D avid San ders and colleagues carried out an ex p erim en t to c om p are the
two m odes o f data collection follow ing the 2 0 0 5 British gen eral electio n and found that,
although th ere were som e statistically sign ificant, albeit sm all, d ifferences in th e d istrib u ­
tion s o f key explanatory variables, the relative im pact o f th ese v ariables on tu rn ou t and p arty
ch oice were virtually identical for the two types o f data. They conclu d e th at in -p erso n and
in ternet data tell very sim ilar stories about what m atters for tu rn ou t and party p reference in
Britain.
H ow ever, no t all surv eys are d esign ed to study n a tio n a l p o p u la tio n s, and in so m e
ca ses it m ay be p o ssib le to acq u ire good sam p lin g fram es fo r in te rn e t su rv eys. F o r e x a m ­
ple, if th e p op u latio n o f in terest is stu d en ts, th en it m ay b e p o ss ib le to u tilize U n iv e rsity
record s to o b ta in a full list o f stu d en ts’ em ail ad d resses. T h is sam p lin g fra m e w ould have
full cov erag e o f th e p op u latio n o f in terest, and cou ld be used to se le c t a p ro b a b ility -
based sam ple.
The way in w hich a qu estio nnaire is adm inistered can also in fluen ce th e resp o n se rate to
the survey. O n e o f the m ain challenges for all types o f survey research is how to o b tain a high
response rate, since n o n-resp onse can in troduce error if people w ho do not com p lete the
survey differ in key respects from th ose w ho do. It is th erefore im portant to try and achieve
a high response rate, and also, to be aware o f w ho does not respond to the survey and why.
Traditionally, response rates for p opulation surveys have tended to be h igh er for face-to -face
surveys than for telephone surveys and mail and in ternet surveys (de Vaus 20 0 2 ; G roves et
al. 2 0 0 4 ). However, there is also substantial variation across m odes o f in terview depending
upon how m uch effort is m ade to re-co n tact respondents, and accord ing to G roves et al.
(2 0 0 4 : 154), it is not clear w hether there are any in herent differences betw een the m od es that
d irectly affect response rates.
M ode effects are m ore apparent w hen it com es to id en tifyin g th e reasons for n o n-resp onse,
and for distinguishing betw een different types o f n o n-response. For exam ple, with face-to-
face interviews, it is possible to distinguish betw een eligible units and ineligible units. An
ineligible unit m ight be a business address or a d erelict property where no one lives. In g en ­
eral, we are only concerned about the no n-response o f eligible units, so this in form ation is
im portant. However, in mail surveys it can be hard to distinguish betw een the two. If there
is no response, it may be because the potential respondent did not com plete the survey (e li­
gible unit), or it may be because the survey was sent to a derelict property w here no one lives
(ineligible unit). Sim ilarly, with telephone surveys, if the telephone rings but no o ne answers,
it may be b ecause the potential respondent is away or busy (eligible unit) or it may be b ecause
the telephone num ber is not in action (ineligible unit). Lastly, although em ails sent to bad
addresses are frequently b ounced back, it is not always possible to tell w hether it has reached
its intended target.
In sum mary, although there are advantages and disadvantages in using each o f the differ­
ent interview m odes, these are not fixed and vary according to the purpose o f the survey and
the target population. Moreover, m odes o f interview can be com bined. For exam ple, it is
com m on practice to supplem ent face-to-face surveys with a mail back or internet follow-up,
which can be a cheap and reliable way o f gathering m ore data on ce the initial sam ple has
been drawn.
 SU RV EYS

Conclusions
As the examples in this chapter have shown, there are a w ide range of difficulties that face researchers
trying to undertake survey research. These difficulties com e in many forms and can introduce error at
various stages of the research process. But despite these difficulties, and perhaps because w e are aware
of them, surveys remain a valuable source of information. Some difficulties are easier to manage than
others. By using random probability sampling, we can have confidence that our surveys are more
representative of the population at large than surveys conducted by any of the other alternatives. And
this representativeness can give us confidence that inferences w e make from our sample are
generalizable to the w ider population. But ensuring representativeness is o nly part of the challenge
Questionnaire design is not an exact science and the researcher is confronted with many measure­
ment problems. Som e of these problems can be at least partially remedied by careful question
wording, but other problems persist and are harder to get rid of. Questionnaires are a very delicate
instrument. Slight changes in how questions are worded or phrased can influence how people answer.
This means that w e should be very careful how we interpret the results from different survey questions.
For example, if w e are interested in the balance of public opinion for or against the decision to invade
Iraq, w e have to bear in mind that our survey estim ate is influenced in part by how we have asked the
question. A similar q uestion framed in a slightly different way may produce a different estimate. If we
want to know w hether pubic opinion has changed, we therefore have to be careful to only compare
survey questions that have been asked in exactly the sam e way. Even small changes in question
wording could make the o bserved differences an artefact of measurement differences, rather than an
indication of any real difference of opinion. For these reasons, w hen we analyse survey data, we tend to
focus on w hether responses to a particular question vary over time; or whether they vary by different
sections o f society.

Questions
• W hat are the principles o f good questionnaire d esign? Are these principles generally followed in
survey research?

• If the w ay in w hich w e ask questions affects the answers that w e get, can surveys ever be trusted?

• W hat is m easurem ent error? Can it be avoided? If not, w hat should w e d o?

• W hat is the difference between validity and reliability?

• H ow do w e know if a survey question is valid?

• W hat is the d ifference betw een m easurem ent error and bias? W hich should w e be more
concern ed about? W hy?

• W hat are the strengths and w eaknesses of different sampling techniques?

• Are non-probability sam ples really so bad? W hy?

• W hat are the strengths o f survey research? W hat are the weaknesses?

• W hat are the different types of error that affect survey research? W hat can be d one to try and

reduce them ?
• As long as w e are aw are of the error w e make, it doesn't matter if w e make errors. Do you agree?

Guide to Further Reading

de Vaus, David (2002), Surveys in Social Research (London: Routledge).
A detailed and accessible overview of survey design and administration. Contains many easy-to-read
and helpful chapters o n how to carry o ut a survey and how to analyse the results.
H O W T O D O R E S E A R C H IN P R A C T IC E

Groves, Robert, F. Fowler, M. Couper, J. Lepkowski, E. Stinger, and R. Tourangeau (2004), Survey
Methodology (New Jersey: John Wiley and Sons).
A sophisticated examination of survey error and how to minimize it.

Moser, Sir Claus and G. Kalton (1971), Survey Methods in Social Investigation (London: Heinemann
Educational Books).
A classic book on survey design. Provides a thorough examination of survey research, and offers
advice on how to plan, design, and carry out a survey effectively.

Schuman, Howard and S. Presser (1996), Questions and Answers in Attitude Surveys: Experiments
on Question Form, Wording and Context (London: Sage Publications).
A fascinating study reporting the findings from a number of survey experiments designed to
investigate the impact of questionnaire design on the responses that are given.

References
Atkeson. L (1999), “ Sure, I Voted for the Winner!" Robinson, J. P., A. Neustadtl, and M. Kestnbaum
Overreport of the Primary Vote for the Party (2002), The Online "Diversity Divide': Public
Nominee in the NES', Political Behavior 21 (3): Opinion Differences among Internet Users and
197-215 Nonusers’, IT & Society 1:284-302.
Becker, H S (1963), Outsiders: Studies in the Sanders, D.. H. Clarke, M. Stewart, and P. Whiteley
Sociology of Deviance (New York: Free Press). (2007), 'Does Mode Matter for Modeling Political
Belson, W. (1981), The Design and Understanding of Choice? Evidence from the 2005 British Election
Survey Questions (London: Gower). Study1, Political Analysis 15:257-85.

Belli, Robert. Michael W. Traugott, Margaret Young, Schuman, Howard and S. Presser (1996), Questions
and Katherine A. McGonagle (1999), Reducing and Answers in Attitude Surveys: Experiments on
Vote Overreporting in Surveys', Public Opinion Question Form, Wording and Context (London:
Quarterly. 63: 90-108 Sage Publications).

Converse, Philip E (1964), The Nature of Belief ---- Jaco b Ludwig, andjon Krosnick (1986),
Systems in Mass Publics', in David Apter (ed.). The Perceived Threat of Nuclear War, Open
Ideology and Discontent (New York: Free Press of Questions, and Salience', Public Opinion
Glencoe). Quarterly 50: 519-36.

Couper. M (2000). W eb Surveys: A Review of Issues Sudman, Seymour, and Norman M. Bradburn
and Approaches', Public Opinion Quarterly 64: (1982), Asking Questions (San Francisco, CA:
464-94 Jossey-Bass)

de Vaus, David (2002), Surveys in Social Research Swaddle, K. and Heath, A. (1989), 'Official and
(London: Routledge). Reported Turnout in the British General Election
Gowers, E A (1954), The Complete Plain Words of 1987', British Journal of Political Science 19(4):
537-51.
(London Penguin)
Groves. Robert, F Fowler, M Couper. J Lepkowski. Tourangeau. Roger, Lance J. Rips, and Kenneth
E Stinger, and R Tourangeau (2004), Survey A Rasinski (2000), The Psychology of Survey
Methodology (New Jersey John Wiley and Response (Cambridge: Cambridge University
Press).
Sons)
Heath. O and R Johns (2010). Measuring Political Wright. Gerald C. (1993), 'Errors in Measuring
Behaviour and Attitudes', in M. Bulmer (ed ). Vote Choice in the National Election Studies.
Social Measurement through Social Surveys: An 1952-88', American Journal of Political Science
Applied Approach (Aldershot Ashgate). 37:291-316

Moser Sir Claus and G Kalton (1971). Survey Zaller.J (1992). The Nature and Origins of Mass
Methods in Social Investigation (London Opinion (Cambridge: Cambridge University
Heinemann Educational Books) Press)
 Interviewing and Focus
Groups

Chapter Summary
O ne of the ways in which researchers collect information about the social world is to
ask people questions. This can be d one by conducting structured and semi structured
interview s, face-to -face interview s, telephone or o nline interviews, or organizing
focus group sessions. This chapter discusses different types and forms of interview­
ing, how interviews and focus groups (a form of interview) should be carried out; and
coding and analysing interview data. The questions addressed in this chapter include
the following.

• What are the different types and forms of interview?

• How can interview data be used to confirm or disconfirm a hypothesis or

argument?

• H ow d o you plan and conduct an interview?

• W hat do you need to consider w hen designing a questionnaire?

• What are the different types of question that might be included in an interview or
questionnaire?

• H ow d o the type and w ording of questions, and the order in which you ask them,
affect the responses you get?

• What interviewing skills will ensure a more successful outcom e to an interview?

Introduction

Interviews as a
In te r v ie w s a re a p r o m in e n t m e th o d o f d ata c o lle c tio n in p o litic a l re se a rch .
m e th o d o f c o lle c tin g d a ta in r e se a rc h ca n b e d istin g u ish e d fro m surveys, an d also fro m
in te r v ie w s th a t are c o n d u c te d fo r p u r p o se s o th e r th a n re se a rch (su c h as jo b in te rv ie w s o r
m e d ia in te r v ie w s ). T h e re are sim ila r itie s b etw ee n in te rv ie w s an d su rv ey s (d isc u sse d in
C h a p te r 1 0 ). B o th u se q u e s tio n n a ir e s an d in te rv ie w e rs . But th e se s im ila r itie s relate m o stly
to str u c tu r e d in te r v ie w s. T h is ch a p te r fo c u se s m o stly o n se m i-s tr u c tu re d in terv iew s
( in c lu d in g fo c u s g r o u p s). In th e se ty p es o f in te rv ie w s o fte n th e re se a rc h e r w ants to p ro b e,
w h ic h is r a th e r lik e a sk in g le a d in g q u e s tio n s , to get th e in te rv ie w e e to o p e n up an d d isc u ss
s o m e th in g o f re le v a n c e to th e r e se a rc h q u e s tio n . T h is k in d o f te c h n iq u e is g en era lly no t
u se d in su rv e y r e se a rc h . T h e ro le o f th e in te r v ie w e r is th e re fo re so m e w h a t d ifferen t, and
w ith th e in te r v ie w th e in te r v ie w e r is m o re activ e . M o r e g en erally , we c a n say th a t th e aim s
o f in te r v ie w in g a n d su rv e y s c a n a lso b e so m e w h a t d iffe re n t (a lth o u g h th e y are fre q u e n tly
c o m p le m e n ta r y ) . T h e a im o f surveys is to p r o d u c e sta n d a rd iz e d d ata w h ich c a n b e u sed to
m a k e g e n e ra liz a tio n s a b o u t w h at a g iv en p o p u la tio n o f p e o p le th in k a b o u t a p a r tic u la r
H O W T O D O R E S E A R C H IN P R A C T IC E

issue. By con trast, interviewing is m o re c o n ce rn e d w ith o b tain in g detailed, often specialized

in form a tion from a sin gle individual o r sm all n u m b er o f individuals. T h e aim is n o t n e c e s­
sarily to m ake generalization s, b ut to gain valid know ledge and u n d erstan d in g ab o u t what
th e p erson in qu estio n th inks. A lthough b o th interview s and surveys take th e issues o f re li­
ability and validity seriously, it m ay be argued that surveys ten d to p rioritize reliability
(w hich allow s th em to m ake gen eralization s), w hereas in terview s p rioritize validity (w hich
allow s th em to gain a depth o f know ledge).
In th is chapter we will discuss types and form s o f interview s, what con sid eration s enter
into d esigning a schedule o f q u estio ns or a qu estio nnaire, how to co n d u ct an in terview and
organize a focus group, and how to analyse in terview data.

I. Interviews
To decid e w hether interview s will enable you to investigate your hyp othesis you m ust ask
yourself, and answer, a nu m b er o f questions. How will interview s enab le you to investigate/
d em onstrate your hypothesis? W hat, specifically, do you need to find out, th rou gh in ter­
view ing, to enable you to investigate th e relationships stated by your h ypothesis? Is c o n d u ct­
ing interview s the best way to ob tain th is in form ation ? W hat kind o f interview, and what
sort o f qu estions, will enable you to ob tain th is in form ation ? O f w hom will you ask qu estions
and what tech niq ues will you use? U nderstanding th e various types and form s o f in terview
will enable you to answ er th ese questions.

Types of interview
D ifferent types o f interview provide different ways o f collectin g interview data. Interview
data can be collected by cond u ctin g individual face-to -face, telephone, or o n lin e interview s,
or by organizing focus group sessions. These types o f interview vary with respect to the
degree o f p ersonal contact they entail betw een researchers and subjects. However, they all
involve a situation in which a researcher asks respond ents qu estions and then records and
analyses their answers. For all types o f in terview ing, the p rim ary purpose is to prod uce data
that will help answer research questions.

In d ivid u al fa c e -to -fa c e in te rv ie w in g

This type o f interview is the best data-collection type for op en -end ed qu estions and in-
depth exploration o f opinions. The interview er works directly with the respondent and
has the opportunity to probe or ask follow -up questions. By b eing able to ask qu estions o f
subjects personally, the interview er can probe unclear responses, resolve difficulties that
lead to non-responses, and obtain useful in form ation from body language and vocal cues. In
sum , this type o f interview ing can support longer and m ore detailed qu estions, is adaptable,
and is a rich and dense source o f data.

T e le p h o n e inte rv ie w s

This tvpe o f interview enables a researcher to gather in form ation rapidly. M ost m ajor public
opinion polls that are reported are based on telephone interview s (see C hapter 10). Using
 IN T E R V IE W IN G A N D FO C U S G R O U P S

te le p h o n e in te rv ie w s giv es r e se a rc h e rs w ide g eo g ra p h ica l a cc e ss: p eo p le fro m all o v er th e

g lo b e c a n b e in te rv ie w e d i f th e y hav e a c ce ss to a te le p h o n e o r c o m p u ter, as well as p o p u la ­
tio n s th a t are d iffic u lt to rea ch o r in te rv ie w by o th e r m e a n s (e.g. p eo p le liv in g in rem o te
a rea s). L ik e fa c e to -fa c e in te rv ie w s, th ey allo w for so m e p e rso n a l c o n ta c t b etw ee n th e in te r ­
v iew er a n d th e re sp o n d e n t, a n d th e y allo w th e in te rv ie w e r to ask fo llo w -u p q u estio n s. T e le ­
p h o n e in te r v ie w in g su p p o rts lo n g e r q u e s tio n n a ire s th a n m ailed q u e stio n n a ire s, th o u g h not
as w ell as fa c e - to - f a c e in te rv ie w in g .

O n lin e interview s

O n lin e in te rv ie w s are in c re a sin g ly used as a d ata c o lle c tio n m e th o d by so cia l sc ien tists.
T h e se c a n b e a u se fu l te c h n iq u e for d ata c o lle c tio n o n sen sitiv e item s, an d th ey e lim in a te
in te r v ie w e r b ia s. B u t o n lin e in te rv ie w s are a far less p e rso n a l typ e o f in terv iew th an o th e r
ty p es; a n d th e y a re n o t an a p p ro p ria te to o l fo r e x p lo r a to r y stu d ies, w h ich re q u ire in ten siv e
i n te r a c tio n w ith in te rv ie w e e s in o rd e r to g ain b e tte r in sig h t in to an issu e. T h ey ca n b e u sefu l
in s m a ll-s c a le stu d ie s o n sen sitiv e to p ics an d in c o m b in a tio n w ith o th e r to o ls. T h is is a q u ick
a n d ea sy w ay to r e a c h g e o g r a p h ic a lly d isp e rse d in terv iew ees.

Fo cus gro up s

F o c u s g r o u p s g e n e r a t e d if f e r e n t ty p e s o f d a ta fr o m o th e r ty p e s o f in te r v ie w in g . In th is
ty p e o f in te r v ie w , a g r o u p o f p e o p le is s e le c te d b e c a u s e th e y a re r e la te d to s o m e p h e ­
n o m e n o n o f in te r e s t. T h e r e s e a r c h e r m e e ts w ith th e g ro u p a n d fa c ilit a te s an o r g a n iz e d
d is c u s s io n r e la te d to s o m e th in g th e p a r tic ip a n ts h av e e x p e r ie n c e o f, o r b e lie f s a b o u t.
T h e in t e r a c t io n a m o n g f o c u s g r o u p p a r tic ip a n ts m a y r e v e a l m o r e th a n th e y w o u ld in an
in d iv id u a l in te r v ie w , a n d b r in g o u t in s ig h t s a n d u n d e r s t a n d in g s in w ays w h ic h sim p le
q u e s t io n n a i r e ite m s (u s e d in f a c e - t o - f a c e o r te le p h o n e in te r v ie w s o r s e n t th r o u g h th e
m a il) m a y n o t b e a b le to ta p , s u c h as e m o tio n a l a n d u n c o n s c io u s m o tiv a tio n s . U sin g
f o c u s g r o u p s , r e s e a r c h e r s c a n le a r n h o w th in g s a r e d is c u s s e d in a p a r tic u la r c u ltu r e , te s t
h y p o th e s e s a b o u t b e lie f s , m ix p e o p le w h o w o u ld n ’t n o r m a lly m ix , te s t q u e s tio n s fo r
f u tu r e in te r v ie w s , o r te s t p o lic y id e a s to s e e h o w c itiz e n s r e a c t to th e m .
L e t’s c o m p a r e th e a d v a n ta g e s a n d d is a d v a n ta g e s o f d iffe r e n t ty p e s a n d fo r m s o f in te r ­
v ie w in g fo r s p e c if ic p u r p o s e s . In d iv id u a l f a c e - t o - f a c e in te r v ie w s o r fo c u s g r o u p s p ro v id e
m o r e fle x ib ilit y a n d c a n e n a b le r e s e a r c h e r s to p r o b e o r a sk fo llo w -u p q u e s tio n s . H o w ­
e v er, th e y a lw a y s c a r r y th e p o s s ib ility o f in te r v ie w e r b ia s, as w ell as r e s p o n d e n t b ia s o n
s e n s itiv e ite m s . T h e y r e q u ir e sk ill to u n d e r t a k e w ell; th e y c a n a lso b e v e r y e x p e n s iv e , ta k e
a lo t o f tim e , a n d w ill u s u a lly b e le ss fe a s ib le to d o if th e p o p u la tio n yo u w ish to in te r v ie w
is w id e ly d is p e r s e d g e o g r a p h ic a lly . T h e d a ta fr o m b o th ty p e s a re d iffic u lt to a n a ly s e —
p a r tic u l a r ly in th e c a s e o f f o c u s g r o u p s , w h e re th e a n a ly sis m a y b e c o n c e r n e d b o th w ith
th e c o n t e n t o f w h a t w as sa id a n d th e p a tte r n s o f in te r a c tio n . T e le p h o n e in te r v ie w s e n a b le
t h e r e s e a r c h e r to in te r v ie w p e o p le w h o a r e n o t e a sy to a c c e s s , b u t th e in te r v ie w e r d o e s
n o t s e e th e in te r v ie w e e , so b o d y la n g u a g e a n d v o ic e a n d in to n a t io n c a n n o t b e u se d as a
s o u r c e o f e x tr a in f o r m a tio n . M o r e o v e r , th e in te r v ie w e r h a s n o c o n t r o l o v e r th e s e ttin g
a n d a m b ie n c e a n d c a n n o t s ta n d a r d iz e th e s it u a tio n in w h ic h th e in te r v ie w ta k e s p la c e .
A n d , t e le p h o n e in te r v ie w s h a v e to b e re la tiv e ly s h o r t o r e n e r g y le v e ls w ill flag. T e le p h o n e
in te r v ie w s a r e n o t id e a l i f y o u w a n t to a sk le n g th y q u e s tio n s , g iv e th e r e s p o n d e n t d e ta ile d
b a c k g r o u n d f o r a q u e s tio n , o r a s k th e r e s p o n d e n t to s e le c t fr o m a lo n g lis t o f p o s s ib le
H O W T O D O R E S E A R C H IN P R A C T IC E

o p tio n s or answ ers. O n lin e in terview s have b e c o m e in c rea sin g ly popu lar. In cases w here
it is no t p o ssible to sch e d u le a tim e to m eet fa c e -to -fa c e o r to sp eak on th e p h o n e, it
m ight b e th e only way to co n d u ct an in terview . O n lin e in terv iew s p resen t o p p o rtu n itie s
to in terview ind iv id u als based anyw h ere in th e w orld. A nd th ey prov id e m o re tim e for
in terv iew er and in terview ee to c o n s id er th e ir q u estio n s and answ ers, allo w in g for
in crea sed flex ib ility and d eep er r eflectio n . U n lik e m o re p erso n al fo rm s o f in terv iew in g
(in d iv id u al fa c e -to -fa c e , fo cu s group, or telep h o n e in terv iew in g ), th ey allow th e in te r ­
view er to ask qu estio n s th at m ay req u ire th e resp o n d e n t to co n su lt reco rd s in o rd er to
supply answ ers. H ow ever, th ey are a p o o r c h o ic e for co m p lex q u estio n s, and th e
resea rch er c a n n o t p rob e or co lle ct a d d itio n al data. As w ith telep h o n e in terv iew in g , th e
resea rch er has no c o n tro l over w h e th er and how a resp o n d e n t is in te ra ctin g w ith o th ers
w hile c o m p letin g th e qu estio n n aire.

Forms of interview
You will not be able to select am ong th ese different types o f in terview ing— to fully w eigh
th eir advantages and disadvantages for different p u rp ose— unless you are clear about how,
w ithin these types o f interview, the form o f interview ing can vary. W hat we w ant to clarify is:
what are the different form ats that interview s take, including the types o f q u estions asked
and the m an ner in w hich they are asked? W hich o f these form s or form ats are supported by
the different types o f interview s that we ju st discussed?
There are three basic formso f interview that vary with respect to the degree to w hich the
qu estions and responses are standardized across interview subjects. These are structured,
unstructured, and semi-structured form s o f interview. They differ with respect to the form o f
question em ployed (closed or op en ), how they are worded (sh o rter and sim pler, or longer
and m ore com plex), and w hether the sam e q u estions are asked the sam e way and in the sam e
sequence across interview subjects (standardized or variable). S tru ctu red in terview s c o n ­
sist o f a standardized set o f closed and sh orter or sim pler qu estions that are asked in a stan d ­
ardized m an ner and sequence. O f the types o f interview ing we discussed previously, m ailed
qu estionnaires alm ost invariably use this form o f interview ing. U n stru ctu red in terview s
use open, and perhaps lengthier and m ore com plex questions, which m ight vary in the way
and order in which they are asked. Focus groups, w hich involve interview ing people together
in flexible and exploratory group discussion form ats are an exam ple o f a type o f interview
that uses this form o f interview. Perhaps the m ost com m on type o f interview used in p o liti­
cal research is semi-structured interviewing: a form o f interview that com bines elem ents o f
both structured and unstructured interviews.

Stru ctu red in te rv ie w s

Structured interview s often consist o f closed q u estio n s. This m eans that both the q u estio ns
asked and the codin g of answers are standardized. TTiis form o f in terview is p rim arily used
in surveys, and is discussed in detail in C hapter 10. But w hether thousands o f respondents
are interview ed in this way, or just a handful, the p rinciples o f qu estio n n aire design arc
exactly the same. In terv iew ees— or resp ond ents— are asked a set o f identical qu estio ns in
exactly the sam e way and in the sam e order; and they are usually asked to respond by
selecting from a lim ite d range o f possible options or answers. Individual fa ce-to -fa ce and
 IN T E R V I E W I N G A N D F O C U S G R O U P S

te le p h o n e in te r v ie w s c a n u se s tru c tu re d in te rv ie w s, fo c u s g ro u p s c a n n o t use th e m , and

m a iled q u e s tio n n a ir e s g e n e ra lly m u st u se th e m .
A k ey c o n c e r n w ith th e s t r u c t u r e d in te r v ie w h as to d o w ith th e lim ita tio n s o f a sk in g
sta n d a rd iz e d q u e s tio n s a n d p r e s e n t in g a sta n d a rd iz e d list o f o p tio n s fro m w h ich
r e s p o n d e n ts c h o o s e a r e s p o n s e . A lth o u g h th is m e th o d m ay p r o d u c e r e lia b le re su lts, in
th a t e v e r y o n e is p r e s e n te d w ith th e sa m e q u e s tio n an d h as th e sa m e o p p o r tu n itie s to
a n sw er, it is o fte n a r g u e d th a t it d o e s n o t p r o d u c e v alid d a ta . T h e p r o b le m is th at th e sa m e
w o rd c a n m e a n d iffe r e n t th in g s to d iffe r e n t p e o p le . C o n s id e r th is e x a m p le o f a clo se d
q u e s tio n :

How would you rate your course tutor?

E x c e lle n t □
Q u ite g o o d □
Q u ite p o o r □
U sele ss □
D o n ’t k n o w □

T h e p ro b le m w ith th is q u estio n is th at p eop le m ig h t m e an very differen t th in g s by e x cellen t’,

but ev e ry o n e w h o tick s th a t b o x in resp o n se to th e q u estio n will b e in clu d ed in th e percen tage
o f p eo p le ju d g in g th e c o u r se tu to r to b e e x c e lle n t’. T h is raises q u estio n s a b ou t th e v alidity o f the
d ata th a t th e se resp o n ses yield. A s d iscu ssed in C h a p ter 10, th ere are fou r b asic ways in w hich
we c a n try a n d assess v alid ity (see M o se r an d K alto n 1971; de V aus 2 0 0 2 for an ex ten d ed d is­
c u ss io n ). T h e first a n d m o st stra ig h tfo rw ard is face validity. F a c e v a lid ity sim ply asks if, on the
fa ce o f it, th e q u e stio n in tu itiv ely seem s like a g o o d m e asu re o f th e c o n ce p t. I f th e answ er is no,
th en we d efin itely hav e a p ro b le m , b u t w hat is in tu itiv ely yes for o n e p erso n m ay n o t b e so for
an o th er. C o n t e n t v a lid ity e x a m in e s th e e x te n t to w h ich th e q u estio n cov ers th e full range o f
th e c o n c e p t, c o v e rin g ea c h o f its d ifferen t asp ects. C r i te r io n v a lid ity ex a m in es how well th e
new m e a su re o f th e c o n c e p t relates to ex istin g m e asu res o f th e co n ce p t, o r related co n ce p ts.
C o n s t r u c t v a lid ity e x a m in e s h o w w ell th e m e a su re co n fo rm s to o u r th eo retica l e x p ecta tio n s,
by e x a m in in g th e e x te n t to w h ich it is a sso ciated w ith o th e r th eo retic a lly relevant factors. U lti­
m ately, th o u g h , th e m a tte r o f v alid ity is o n e o f ju d g e m e n t. T h e stru ctu ra l fo rm o f in terview m ay
la ck v a lid ity in th a t it fails to fully c ap tu re th e reality o f p eo p les lived ex p erien ce. B ecau se th e
stru c tu re d in terv iew a llow s resea rch ers to b e m o re d etach ed fro m th e p eop le th ey are studying,
it is d ifficu lt fo r th e m to e x p lo re w h at th e ir su b je c ts actu ally m e an an d how th ey actu ally behav e
in real situ a tio n s.

U n structure d interview s

U n s tr u c tu r e d in te r v ie w s o fte n u se o p e n q u e s t io n s , w h ic h allo w fo r lo n g e r an d p e r h a p s
m o r e c o m p le x q u e s tio n s , as w ell as m o r e in -d e p th p r o b in g a n d fle x ib ility , th a n s t r u c ­
tu re d in te r v ie w s . T h e y a r e m o r e lik e o r d in a r y c o n v e r s a tio n s : th e r e is n o se t in te r v ie w
s t r u c t u r e , a n d in te r v ie w e e s a n s w e r in t h e ir o w n w o rd s. T h e in te r v ie w e r in itia te s th e c o n ­
v e r s a tio n , p r e s e n t s e a c h to p ic b y m e a n s o f s p e c ific q u e s tio n s , a n d d e c id e s w h e n th e c o n ­
v e r s a tio n o n a to p ic h a s sa tis fie d th e r e s e a r c h o b je c tiv e s . T h e s e e le m e n t s o f th e in te r v ie w
p r o c e s s w ill lik e ly v a ry w ith e a c h in d iv id u a l in te r v ie w , b a s e d o n th e r e s p o n s e s o f th e
in te r v ie w e e .
H O W TO DO R E S E A R C H IN P R A C T IC E

Semi-structured interviews
Sem i-stru ctu re d in terview s gen erally in volve a sm all n u m b er o f in terview s in w hich th e
in terview er uses a co m b in a tio n o f stru ctu red q u estio n s (to o b ta in factu al in fo rm a tio n )
and u n stru ctu red qu estio n s (to probe d eeper into p eople’s ex p erien c es). T h e in terview er
prepares a schedule o f qu estio ns, as well as su p p lem en tary qu estio n s to exp lore asp ects o f
the response to each qu estio n asked. But th e qu estio n s m ay be m o re g en eral, and th e in te r­
viewer m ay c h o o se to vary the seq u en ce in w hich th ey are asked, and to ask follo w -up
qu estions. U nstru ctured or sem i-stru ctu red in terview s can give greater in sight in to th e
m eanings o f a su b ject’s exp erien ces and h en ce provide m ore valid data. H owever, it is not
as easy to com p are responses o f different groups or o f th e sam e group ov er tim e sin ce each
interview ee will have been asked slightly different qu estio n s to find out w hat th ey th ink.
The data are not standardized and are th us hard to gen eralize from and, as th e results c a n ­
not be qu antified and re-tested , less reliable.
Box 11.1 shows how these types and form s com m on ly co m b in e in political research.

Which type and form of interview to use?

W hen selecting the type o f data collection tech niq u e and form o f in terview to use, there are
a nu m ber o f substantive and practical issues to consider.
G en erally, stru ctu re d in terv iew s are b etter for m ak in g c o m p a ris o n s and less s t r u c ­
tured in terview s may b e m o re a p p rop riate for early ex p lo ra to ry phases o f research .
U ltim ately, it will be th e n ature o f what you w ant to ask resp o n d e n ts th at d e te rm in e s the
type and form o f in terview you sele ct. A re you goin g to need to get lots o f d etail in the
resp onses (op en q u e stio n s); or can you a n ticip ate th e m o st freq u en t or im p o rta n t types
o f responses and develop reaso n ab le clo se d -e n d ed q u estio n s? W ill th e q u e stio n s be
com p lex and require m u ltip le parts or su b -q u estio n s? C an you co n s tr u c t in ad v an ce a

Forms of interview (types of questions, w ording, and sequencing)

Structured Semi-structured Unstructured

(closed, shorter, (combines structured/ (open, longer/more
standardized) unstructured forms) complex, varied)

Types o f interview (d ata co llectio n tech n iq u es)

 IN T E R V I E W I N G A N D F O C U S G R O U P S

set o f q u e s tio n s a n d th e s e q u e n c e in w h ic h th e y w ill b e a sk e d ? O r w ill you n ee d to p r o ­

c e e d in a m o r e e x p lo r a to r y m a n n e r , p e r h a p s e n t a ilin g lo ts o f fo llo w -u p q u e s tio n s th a t
y ou c a n ’t e a s ily a n t ic ip a t e ?

A k ey c o n s id e r a t io n is th e lik e lih o o d o f b ia s an d o f th re a ts to th e v a lid ity an d r e li­

a b ility o f y o u r fin d in g s. A lim ita t io n w ith all ty p es an d fo r m s o f in te r v ie w s is th at
r e s e a r c h e r s a re d e p e n d e n t o n w h at p e o p le tell th e m . S o m e p e o p le m ay h av e p r o b le m s in
r e c a llin g in f o r m a tio n a c c u r a te ly . B u t all p e o p le c o m e to an in te r v ie w w ith b ia se s an d
p r e ju d ic e s ; a n d p e o p le g e n e r a lly a re p r o n e to s o m e th in g k n o w n as th e ‘in te r v ie w e ff e c t’:
th e te n d e n c y fo r in te r v ie w e e s to g iv e m o re ‘s o c ia lly a c c e p ta b le ’ an s w e rs o r an s w e rs th ey
th in k th e in te r v ie w e r w a n ts. T h e b ia s e s a n d p r e ju d ic e s o f in te r v ie w e rs m ay a lso d is to rt
th e in te r v ie w p r o c e s s : th e y m a y n o t a sk q u e s tio n s th a t m a k e th e m u n c o m fo r ta b le , o r fail
to lis te n c a r e f u lly to r e s p o n d e n ts o n to p ic s fo r w h ic h th e y h av e stro n g o p in io n s . P eo p le
m a y n o t tell th e tr u t h b e c a u s e th e y w an t to ‘lo o k g o o d ’ in th e e y es o f th e in te r v ie w e r, o r
i f th e y a re a sk e d s o m e th in g th a t e it h e r th e y d o n ’t k n o w h o w to a n s w e r o r th a t w ou ld b e
e m b a r r a s s in g fo r th e m to a n sw er. S o m e ty p e s o f in te r v ie w are b e tt e r a b le to r e d u c e th e
e ffe c ts o f b ia s e s a n d p r e ju d ic e s . A n o n y m o u s q u e s tio n n a ir e s th a t c o n t a in n o id e n tify in g
in f o r m a tio n a r e m o r e lik e ly to p r o d u c e h o n e s t r e s p o n s e s th a n th o s e id e n tify in g th e
r e s p o n d e n t.
In s e le c tin g th e ty p e o f d a ta - c o l le c tio n te c h n iq u e , y ou w ill n e e d to c o n s id e r a n u m b e r
o f p r a c tic a l c o n s id e r a t io n s , s u c h as c o s t , f a c ilitie s , e q u ip m e n t, a n d tim e . C o s t is o fte n a
m a jo r d e te r m in in g f a c to r in s e le c tin g th e ty p e o f in te r v ie w to c o n d u c t. C o s ts c a n in c lu d e
tra v el e x p e n s e s e n ta ile d in c o n d u c tin g p e r s o n a l in te r v ie w s , o r p o s ta l c o s ts fo r an e x te n ­
siv e m a ilin g . F a c ilitie s a re a n o th e r c o n s id e r a t io n . Y ou w ill r e q u ire a c c e s s to w e ll-e q u ip p e d
p h o n e - s u r v e y in g fa c ilit ie s fo r te le p h o n e in te r v ie w s , a n d a c o m fo r t a b le an d a c c e s s ib le
r o o m to h o s t f o c u s g r o u p s. E q u ip m e n t n e e d s m ig h t in c lu d e a tr a n s c r ip t io n m a c h in e w ith
Dragon
a p e d a l- o p e r a t e d s t a r t - s t o p m e c h a n is m ; o r v o ic e tr a n s c r ip t io n so ftw a re — su c h as
Naturally Speaking s p e e c h r e c o g n it io n s o ftw a re fr o m N u a n c e C o m m u n ic a tio n s — to
r e c o rd r e s p o n s e s , a n d e q u ip m e n t o r r e s o u r c e s to p ay fo r p r o fe s s io n a l t r a n s c r ip t io n .1
F in a lly , w ill y o u h a v e e n o u g h tim e to s e n d o u t m a il q u e s tio n n a ir e s a n d fo llo w -u p r e m in d ­
e rs, a n d to g e t r e s p o n s e s b a c k b y m a il? W ill y o u h a v e e n o u g h tim e to c a r r y o u t all th e
p e r s o n a l in te r v ie w s ?
E a c h o f th e th r e e f o r m s o f in te r v ie w th a t w e h av e d isc u sse d (s tr u c tu re d , s e m i-s tr u c tu re d ,
a n d u n s tr u c tu r e d ) c o lle c t d a ta in slig h tly d iffe r e n t w ays. A n in te r v ie w e r c o n d u c tin g fa c e-
to - f a c e structured in te r v ie w s w ill g e n e ra lly p re p a re an in te r v ie w s c h e d u le : a list o f q u e s ­
tio n s to b e a sk e d o f all in te r v ie w e e s. B y c o n tr a s t, an in te r v ie w e r c o n d u c tin g u n s tr u c tu re d o r
s e m i- s tr u c tu r e d in te r v ie w s w ill g e n e ra lly p re p a re an in te r v ie w g u id e : a set o f q u e stio n s th a t
p ro v id e a list o f ta lk in g p o in ts o r a re a s to b e c o v ered . B elow , w e d isc u ss e a ch o f th e se
a p p r o a c h e s to c o lle c tin g d a ta .

Designing a questionnaire or interview schedule for a structured interview

D e s ig n in g a g o o d q u e s tio n n a ir e o r in te r v ie w sc h e d u le alw ays in v o lv es d e c isio n s r e g ard in g
th e c o n t e n t , f o r m u la tio n , a n d s e q u e n c in g o f th e q u e stio n s.
H O W T O D O R E S E A R C H IN P R A C T IC E

► Content: choosing questions to ask

Take your hypothesis as a starting point. The research p roject should b e designed to provide
a conv in cing test o f your h ypothesis. M ake sure that you are clear about ex actly w hat you do
need to find out and that the qu estions you ask supply you with the in fo rm atio n you need.
Form ulate qu estions w hose answ ers will con tribu te to co n firm in g or d isco n firm in g your
hypothesis. Ensure that your qu estio nnaire or interview schedule is designed to enab le you
to investigate your m ajor analytic categories or variables and ob tain th e m ax im u m qu ality o f
responses. D ecide what qu estions will enable you to op eratio nalize (m easu re o r defin e) your
variables. You m ight c onsider u sing q u estio ns em ployed by oth er research ers to tap the sam e
in form ation .

► Formulating questions
In discu ssing structured, unstructu red, and sem i-stru ctu red form s o f in terview ing, we
referred to two types o f qu estions: op en -end ed q u estio ns (allow ing for com pletely op en as
well as partially categorized answ ers), and closed qu estions. H ere we will look m o re closely
at these types o f questions. (See also our d iscu ssion in C hapter 10.)

Open-ended questions
O p en-en d ed qu estions perm it researchers to ob tain in -depth in form ation on issues with
w hich they are not very fam iliar; opin ion s, attitudes, and suggestions o f respond ents; or
sensitive issues. O p en-en d ed qu estions ask: ‘W hat did you th in k o f the initiative? H ow did
you feel about the d ecision? W hat do you like b est about the proposed p rogram ?’ T h e use o f
open -end ed qu estions can be im proved by preparing a list o f fu rth er qu estions to keep at
hand to ‘probe’ for answ er(s) in a system atic way.

Closed questions

Answers to closed qu estions can be recorded in a nu m b er o f different ways. However, it is

im portant to ensure that the response alternatives provide a sufficient range o f choices, and
that they are m utually exclusive.
O ne o f the m ost sim ple resp onse form ats is to invite resp on d en ts to select b etw een
two fixed altern ativ es. The respond ent is offered two p o ssible resp on ses, on ly one o f
which may be selected (dichotomous questions): for in stan ce , resp on ses m ay take the
form o f Yes/No, True/False or A gree/D isagree. A lthough th ese q u estio n s are relatively
sim ple, for m any top ics th ey are u n likely to cov er th e full range o f resp on ses that a
respond ent may want to give.
These types o f questions can th erefore easily be extended to cover a greater range o f
response alternatives. For exam ple, a respondent may be offered a list o f item s and asked to
select one o f the responses.

Which party did you vote for in the general election?

C o n s e rv a tiv e □
Labour n
 IN T E R V I E W I N G A N D F O C U S G R O U P S

L ib era l D e m o c r a t □
S c o ttis h N a tio n a l P a rty □
P laid C y m r u □
G re e n P a rty □
O th e r p a rty n
B y c o n tra s t, r e sp o n d e n ts m a y also b e p ro v id ed w ith th e o p tio n o f se le ctin g m o re th an o n e
re sp o n s e . F o r th e se q u e s tio n s th e r e sp o n s e is c h o s e n fro m a list o f item s in w h ich any number
o f ite m s m a y b e se le c te d .

Which o f these news sources have you consulted during the past month?

T e le v isio n n ew s p r o g ra m m e s □
R a d io n ew s p r o g ra m m e s □
D a ily n e w sp a p e rs □
In te r n e t n ew s sites □
W e e k ly n ew s m a g a z in e s □
M o n th ly n ew s m a g a z in e s □
O th e r □
A n o th e r ty p e o f re sp o n s e f o r m a t th a t is fr e q u e n tly u sed is o n e w h e re th e ca te g o rie s hav e an
o rd er, a n d a n s w e rs in v o lv e ra tin g , fo r in s ta n c e , th e lev el o f a g r e e m e n t o r d isa g re e m e n t w ith
a sta te m e n t fr o m lo w to h ig h . T h e se are o fte n referred to as n u m e r ic a l ra tin g s ca les , an d th ere
a re a n u m b e r o f w ays in w h ic h th e y c a n b e c o n s tr u c te d . O n e o f th e m o st c o m m o n typ es o f
r a tin g s c a le is k n o w n as th e L ik e r t sc a le. T h is ty p e o f q u e stio n g e n e ra lly p ro v id es a sta te m e n t
th a t r e fle c ts a p a r tic u la r a ttitu d e o r o p in io n an d ask s re sp o n d e n ts to in d ic a te th e ir lev el o f
agreem ent o r disagreement w ith th e sta te m e n t. U su ally re sp o n d e n ts are giv en five d ifferen t
r e sp o n s e a lte rn a tiv e s:

The Government generally treats people like me fairly.

1. S tro n g ly a g ree

2 . A g re e

3. N e ith e r a g r e e n o r d isa g re e

4 . D isa g r e e

5. S tro n g ly d isa g re e .

R a tin g s c a le s c a n a lso b e u se d to m e a su r e a r e sp o n d e n t s lev el o f support o f lev el o f interest

in s o m e th in g .

How much interest do you generally have in what is going on in politics?

1. A g r e a t d ea l

2 . Q u it e a lo t

3. S o m e

4. N o t v e ry m u c h

5. N o n e a t all.
262 H O W TO D O R E S E A R C H IN P R A C T IC E

Finally, we can also ask qu estions w here th e respondent is invited to rank different response
o ptions in ord er o f preference:

Please rank the following items in order o f what seem the most desirable to you, where 1 is
the most desirable and 4 is the least desirable.

M ain tain ing order in the nation □

G iving people m ore say in im portant d ecisions □
F ighting rising prices □
Protecting freedom o f speech □
Q u estion s askin g respond ents to rank item s b eco m e in creasin gly difficu lt as th e n u m b er
o f item s in creases, and th is m ay m ake th e answ ers less reliable. G enerally, resp on d en ts
should not be asked to ord er or rank a series o f m o re th an five item s. F or m o re d etails see
C hap ter 10.

Partially categorized questions

Pre-categorized q u estio ns co m b in e elem ents b oth o f open and closed q u estio ns. R esp o n d ­
ents are asked an open qu estio n, but rather than record in g th eir answ er v erbatim , th e in ter­
viewer selects from a list o f p re-cod ed responses and selects the answ er catego ry that s/he
th ink s com es closest to what the respondent said. A nsw ers can be recorded quickly, and
analysis is easier. But if one p re-categorizes too early, a lot o f in terestin g and valuable in fo r­
m ation may ju st end up in the category ‘O th e r’. Interview ers may try to force th e in fo rm a ­
tion into the categories that are listed and, by m erely tick ing these, ad ditional valuable
in form ation may be lost. Interview ers may stop after receiving the first answer, w hereas
m ore than on e response could be applicable. If a q u estion leads to an in terestin g d iscu ssion ,
it should be w ritten down as com pletely as possible, in addition to b ein g coded. Adequate
space should be provided so that ‘O th e r’ responses can be recorded as close as p ossible to
the respond ents’ own words. O therw ise categorization o f these responses m ay be difficult
afterwards.

Designing an interview guide for semi-structured or unstructured interviews

A s e m i-s tru c tu re d in te r v ie w is a p o w e rfu l re se a rch i n s t r u m e n t w h ic h can h e lp a
re s e a rch e r u n d e rsta n d p e o p le ’s p e r c e p t i o n s , fe e lin g s, o p i n i o n s , e x p e r i e n c e s , u n d e r ­
s ta n d i n g s , v a lu e s , b e lie fs , a t ti t u d e s , e m o ti o n s , b e h a v io u r , f o rm a l a n d in f o r m a l ro le s , a n d
r e la tio n s h ip s . In s e m i - s t r u c t u r e d i n te r v ie w s , th e in te r v ie w e r g u id e s th e in te rv ie w , b u t
p e r m i ts v a r io u s a s p e c t s o f th e to p ic to a r is e n a tu r a lly a n d in a n y o r d e r . T o d o th is ,
r e s e a r c h e r s w ill u s u a lly c o n s t r u c t an 'in te rv ie w ' g u i d e ’: a b a s ic c h e c k lis t o f a r e a s t o b e
c o v e r e d in th e i n te rv ie w in th e f o rm o f q u e s t i o n s . B u t, a lth o u g h th e i n te r v ie w e r c o m e s
p re p a re d w ith a list o f a re a s to b e c o v e r e d , a n d m a y p r o m p t a n d g iv e d i r e c t io n t o th e
in te rv ie w e e , th e in te rv ie w e e is tre a te d a s a n a c ti v e s u b je c t, a n d n o t m e r e l y a r e p o r t e r o f
t a c ts o r e x p e r i e n c e s .

la k e a loo k at th is d raft of an in te rv ie w gu id e o n fa m ily life in b o r d e r a re a s , below'.

 IN T E R V IE W IN G A N D FO C U S G R O U P S

DR A FT INTER VIE W G VIDE

Introduction: purpose o f research, tape recorder/confidentiality, conduct o f interview.

BIOGRAPHY

1. Tell m e about your fam ily background. What was it like when you were a child?
Prompt: Level/main source o f household income
Education o f fam ily members
2. Are there holidays or special fam ily events that your fam ily celebrates?
Prompt: Could you tell me more about these?
3. Tell m e something about your extended fam ily.
Probe: What sort o f connections did your fam ily maintain with this larger group o f
fam ily m em bers while you were growing up?
Probe: How many fam ily m em bers live across the border?

FAMILY PRACTICES

4. Does you fam ily cross the border to see fam ily members?
5. W hat is the pattern o f your cross-border fam ily visiting? How often do you cross the
border to see fam ily members?
Probe: Places you visit
Usual length o f visit
6. Do you celebrate holidays and special occasions with m em bers o f your extended fam ily
who live across the border?
7. Do fam ily m em bers keep in touch with m em bers across the border in other ways?

CROSS-BORDER A CTIVITIES

8. Do you cross the border to participate in fam ily celebrations?

Prompt: Which ones?
9. Do you cross the border fo r other purposes?
Prompt: Can you tell m e briefly about these?
10. H ave there been any m arriages in your fam ily between individuals living on opposite
sides o f the border?
Prompt: Was the m arriage considered a special marriage or was it treated like any other
fam ily marriage?

EXPERIENCE OF CROSS-BORDER VISITS

11. W hat is it like travelling across the border? Does it take a long time to pass through the
border control station?
H O W T O DO R E S E A R C H IN P R A C T IC E

12. What are your impressions o f the places you have visited across the border?
Probe: What do you like/dislike about these places?
A no ther exam ple o f how an interview guide on social m ovem ents m ight be con stru cted is in
B ox 11.2, below.
In cond u ctin g a sem i-stru ctu red or un structu red interview, th e aim o f th e in terview er is
b oth to ask qu estio ns and to listen in a way that enco urages th e interview ee to talk. In p ar­
ticular, the interview er tries to adopt a ton e and m an n er that gives interview ees th e freedom
to open up, reflect, and express them selves. As interview er, you m u st m ake sure not to lead
the interview in a way that closes o ff d iscu ssion by the m an n er in w hich you ask qu estions
and receive responses. H ere are som e general ‘rules o f thum b’:

• Avoid qu estions phrased in a way that suggests or assum es a particu lar kind o f
answer. For in stance, rather than asking w hat the respond ent likes (o r dislikes)
about a candidate, an interview er m ight ask, first, how th e respondent feels about th e
cand id ate; and then follow up with a qu estion about what the respond ent m o st likes (o r
dislikes) about the candidate.

• D o n ’t discou rage long answ ers; instead, verbally and throu gh body language enco urage
the interviewee.

• If you d o n ’t understand som eth in g, c o n tin u e to listen rather th an in terru p t. Y ou

can ask for clarification d uring a break in con v ersation . P rob e in ways th at in d icate
your in terest in w hat a person is saying and enco u rages th e person to explain in m o re
detail.

• D o n ’t com m u n icate, either verbally or through body language, your ju d gem en t about
what the interview ee has told you. M ake sure that your body language does not appear
jud gem ental, or expose your negative jud gem ent. D o n ’t react to provocative statem ents
but accept what the interview ee says.

• Use body language that signals interest (focus on interview ee, eye con tact, nodding,
sm iling).

A sem i- or unstructured interview will generally follow the sam e sequence.

• Introduction. The aim o f the in tro d uction is to explain the research, and the interview
process, and to establish rapport with the interviewee. Explain the purpose o f your
research, what it is, and what you will do with the in form ation you gather. Address
confid entiality issues, and get perm ission to record the interview. Explain what the
interview process will be like and how long it will take.

• Warm up. The aim is to help the respondent to relax. Em phasize that there are no
wrong or right answers; and start with qu estions that are easy to answ er and relatively
im personal.

• Main body of the interview. The m ain body o f the interview consists o f qu estions
relating to the list o f topics contained in the interview guide. These will consist of:

- questions that introduce a topic (W ould you please tell m e about when your interest
in the m ovem ent began? Have you e v e r ...?);
 IN T E R V IE W IN G A N D FO C U S G R O U P S

box 11.2 Example of an Interview Guide on Social Movements

O verarch in g the m es Q uestio n s Possible probes (follow -up questions)

Personal Pieav..-' n y
background youi u/,'

continuously since *rr- tmr- you

joined?

Can you te:i i How many t.mes have you attended?

•neetinas yoi Which ones do you find you have liked
most?
Why was that?

About the If you think about the movement, Raising awareness of key issues?
movem ent what do would you say that its Encouraging wider participation in
major accomplishments have been? public life?
Raising funds for key activities and
services?

What do you think are the main Differences between the movement
strengths of the movement7 and other similar movements?
What has enabled the movement to
develop these strengths?

What do you think are the main What makes you think so?
challenges that the movement Do you think others in the movement

faces? would agree?

Perceptions about In general, why do you think people What type of person becomes a

the other m em bers become active members of the member?

movement? • Have you ever talked with other

members about this?

Do you see members outside of • Why not?

meetings? • What sorts of things do you do or talk

about with other members?

Perceptions about Do you think the movement's • Do you think anything should have

recent activities recent campaign was successful in been done differently?

achieving its aims? • How do you think the membership at

large felt about the campaign?

Do you think the movement should If you had a leadership position in

make any changes in how it pro­ the movement, would you have done

motes its interests? anything differently?

H O W T O D O R E S E A R C H IN P R A C T IC E

- d irect q u estio ns (D o you en joy particip atin g in th ese types o f activ ities?); and

- prob in g These in clude follow-up questions (C ou ld you say m o re

qu estions.
specifying questions (H ow did you react w hen you heard th e new s? W h at
a b o u t...? ),
did you do after th at?); and interpreting questions (D o you m ean th at your o p in io n o f
X changed?).
• Cool-off. W h en the m ain topics have been covered, the in terview er should begin to
prepare for w inding down th e interview. In th is c o o l-o ff’ segm ent o f th e in terview ,
the in terview er moves the d iscu ssion away from ‘heavy’ to p ics— th ose that have b een
m ost difficult to discuss or that have generated ten sio n — and returns to m ore general
questions. The aim is to end the in terview on a lighter and b righ ter note.

• Closure. The interview will conclu d e with askin g the resp ond ent if they have anythin g
they would like to add to w hat they have told you, and if th ere is anythin g they would
like to ask you. You should not appear to be in a hu rry to get away. C on clu d e the
interview by offering the respondent sin cere th anks for taking the tim e to talk w ith you.

Formulating questions
W hichever type o f interview you con d u ct, or qu estion you use, all q u estio ns should be for­
m ulated with the follow ing considerations in m ind.

1. Avoid leading questions. T h e w ord ing o f a q u estio n is e x tre m ely im p o rta n t. R e ­

sea rch ers striv e for o b je c tiv ity in in terv iew in g and, th ere fo re, m u st be care fu l no t to
lead th e resp on d en t in to givin g th e answ er th ey w ould like to receiv e. A lead in g q u e s­
tion is on e that co m m u n ic a tes th e su gg estion to resp o n d e n ts th at c e r ta in resp o n ses are
m o re useful or a ccep tab le th an oth ers. A lead in g q u estio n is fram ed in such a way as to
suggest that on e answ er is ex p ected o r preferred . C h ap ter 10 cited as an ex am p le o f th is
type o f q u estio n on e d ev eloped for th e N ation al E le ctio n Stu d ies in th e U n ited States:
Generally speaking, do you usually think o f yourself as a Republican, a Democrat, an In­
dependent, or what? T h e phrase ‘or what’ at th e end o f th e q u estio n m igh t give th e im ­
pression that not havin g such an id e n tific a tio n is som ew h at o d d , m ak in g it m o re likely
that resp on d en ts will feel pressured to say th at th ey do id en tify w ith o n e o f th e p arties
and less likely to say that th ey th in k o f th em selv es as p artisan s o f p arties w hich are not
m entio ned .

2. Avoid double-barrelled questions. These are qu estions that ask for an answ er on m ore
than one dim ension. For exam ple, a researcher investigating the public response to a new
legislative initiative asks, Do you approve o f the scope and cost-effectiveness o f the measure? If
a respondent answers no’, then the researcher will not know if the respondent disapproves
o f the scope or cost, or both. A question that asks for a response on m ore than one d im ension
will not provide the inform ation you are seeking.

3. Make questions non-confrontational or non-threatening. In order to evoke the truth,

questions must be non th reatening. W hen a respondent is concerned about the co n s e ­
quences o f answering a question in a particular m anner, there is a good possibility that the
answer will not be truthful.
 IN T E R V I E W I N G A N D F O C U S G R O U P S

4. Use simple, direct language. You n e e d to c o n s id e r w hat y o u r q u e s tio n m ig h t m e an

to d iffe r e n t r e s p o n d e n ts . T h e q u e s tio n s m u st he c le a r ly u n d e rs to o d by th e r e sp o n d e n t.
T h e y m u st b e s p e c ific a n d p r e c is e e n o u g h so th at d iffe re n t r e s p o n d e n ts d o n ’t in te rp re t
th e m d ifie r e n tly . C h e c k th a t y o u r la n g u a g e is ja r g o n -fr e e . A void a m b ig u ity , im p re c is io n ,
an d a s s u m p tio n . M o d ify in g a d je c tiv e s c a n h av e h ig h ly v a ria b le m e a n in g s. W o rd s like
usually, often, som etim es, occasionally, seldom, an d rarely m ay no t m e a n th e sa m e th in g
to all p e o p le . D o n o t u se u n c o m m o n w o rd s o r u n fa m ilia r a b b re v ia tio n s . A void a m b ig ­
u o u s o r te c h n ic a l te r m s , a n d lo n g o r v ery g e n e ra l q u e s tio n s . M a k e th e w o rd in g o f q u e s ­
tio n s sim p le and to th e p o in t. M a k in g th e m as b rief as p o s s ib le w ill red u c e
m is u n d e r s ta n d in g s .

5. F in ally, you sh o u ld pre-test questions. D o a te s t-ru n to ch e c k w h e th er th e q u estio n s e n ­

a b le you to o b ta in th e in fo rm a tio n you req u ire and w h e th er in terv iew ers as well as respond
en ts feel at e a se w ith th em . P re -te s tin g q u estio n s e n su re s that th e research in stru m en t
fu n c tio n s th e w ay it is m e a n t to do. P re -te s tin g is u su ally c o n sid ere d on ly in stru ctu re d in te r ­
v iew in g ; b u t it ca n b e usefu l to (m e n ta lly ) pre test q u e stio n s p lan n ed for an u n stru ctu red in ­
terv iew , as well. C o n s id e r ea ch q u e stio n and d ec id e w h e th er all resp o n d e n ts will be ab le to
a n sw er it. A sk th e q u e stio n s to y o u r se lf o r to a frien d and c h e ck w h e th er th e an sw ers you get
are th e ty p e o f resp o n s e s you w ant.

F o r m o re o n p itfa lls to av oid in th e w o rd in g o f q u e stio n s, see C h a p te r 10.

► Question sequence
T h e s e q u e n c e o f q u e s tio n s m u st b e lo g ic a l. T h e in te r v ie w e r sh o u ld g ro u p to g e th e r q u e s ­
t io n s th a t a r e s im ila r , r a th e r th a n ju m p in g fr o m o n e u n re la te d to p ic to a n o th e r . T h e
in te r v ie w e r s h o u ld a ls o u se s c r e e n in g o r ‘f ilte r ’ q u e s tio n s to d e te r m in e w h e th e r th e
r e s p o n d e n t h a s th e r e q u is ite k n o w le d g e to a n s w e r a su b s e q u e n t q u e s tio n . F o r in s ta n c e ,
b e f o r e a s k in g s o m e o n e w h o th e y v o te d fo r in th e last e le c t io n , you w o u ld first n ee d to
a sk w h e th e r th e y v o te d . In th is e x a m p le , i f th e r e s p o n d e n t a n s w e re d ‘n o ’ to th e s c r e e n in g
q u e s tio n , th e in te r v ie w e r w o u ld s k ip th e fo llo w in g q u e s tio n (a b o u t h o w th e r e s p o n d e n t
v o te d ) , o r th e q u e s tio n n a i r e w o u ld d ir e c t th e r e s p o n d e n t to th e c o r r e c t su b s e q u e n t
q u e s tio n .
T h e se q u e n c in g o f q u e s tio n s sh o u ld a lso allow , as m u ch as p o ss ib le, for a ‘n a tu ra l’ c o n v e r ­
sa tio n , ev en in m o re stru c tu re d in te rv ie w s. W ith in e a c h g rou p , tr a n s itio n s b etw ee n q u e s ­
tio n s sh o u ld b e s m o o th . E a c h q u e stio n sh o u ld follo w c o m fo rta b ly fro m th e p rev io u s
q u e s tio n . You sh o u ld sta rt w ith q u e s tio n s d ire ctly related to th e to p ic of th e r e s e a rc h —
p a r tic u la rly q u e s tio n s m o s t lik e ly to s e c u r e th e in terest an d a tte n tio n o f th e r esp o n d e n t.
S ta rt w ith a q u e s tio n th a t is d ir e c tly related to th e s u b je c t o f th e study an d th at will raise th e
r e s p o n d e n ts in te r e st. M o r e se n sitiv e q u e stio n s sh o u ld b e p o sed as late as p o ss ib le in th e
in terv iew .
W it h in e v e r y g r o u p o f q u e s tio n s , g e n e r a l q u e s tio n s sh o u ld c o m e b e fo r e m o re s p e c ific
o n e s . T h e r e a s o n , a s A la n B r y m a n p o in ts o u t, is th a t ‘w h e n a s p e c ific q u e s tio n c o m e s
b e f o r e a g e n e r a l o n e , th e a s p e c t o f th e g e n e r a l q u e s tio n th a t is c o v e r e d by th e s p e c if ic­
ô n e is d is c o u n te d in th e m in d s o f th e r e s p o n d e n ts b e c a u s e th e y feel th e y h av e a lre a d y
H O W T O D O R E S E A R C H IN P R A C T IC E

cov ered it’. F or ex a m p le, i f you first ask a q u estio n a b o u t a resp o n d e n t s s a tisfa c tio n w ith
h is salary, and afterw ard s ask ab o u t th e re sp o n d e n t’s sa tisfa c tio n w ith h is jo b (a m o re
g en era l q u e stio n ), th e resp o n d e n t m ay d isco u n t th e salary issu e, h av in g alread y
ad d ressed it (B ry m a n 2 0 0 4 : 12 2 ). T h is is w hat is c alled q u e s tio n o r d e r e ffe c t. T h e c o n ­
tex t in w hich a q u estio n is asked or th e p o sitio n in w h ich it ap p ears in rela tio n to o th e r
q u estio n s ca n have a b e a rin g on th e answ ers th at are giv en to it. T h e m o st o b v io u s m a n ­
ifesta tio n o f th is is w hen resp o n d e n ts try to ap p ear co n s iste n t by an sw erin g a q u estio n
so th at it fits in w ith how th ey answ ered a p rev iou s q u e stio n . (W e d iscu ss th is in m o re
d etail in C h a p ter 10.)
F or exam ple, G eorg e B ish op and h is colleag u es foun d th at peop le w ere m o re likely to
rep ort that th ey w ere ‘very in terested ’ in th e 1980 U S p resid en tial cam p aig n w hen th ey
w ere asked th is q u estio n im m ed iately after, rath er th an ju st b efo re, a set o f q u estio n s
ab out w ho th ey th ough t w ould w in th e e le c tio n , how clo se th ey th o u g h t th e race w ould
be, and w h eth er th ey person ally cared w h ich p arty w on th e e le c tio n (B ish o p et al. 19 8 4 ).
In a sim ilar ex p erim en t, th ey also d iscov ered that p eople w ere less likely to th in k th ey
follow ed 'w h at’s g oin g on in gov ern m en t and p u b lic affairs’ w hen asked ab ou t it right
after (in stead o f ju st b efo re) a difficu lt group o f qu estio n s c o n c e rn in g w hat th ey knew
a b out th e record o f th eir m e m b er o f C on g ress. They co n clu d ed th at qu estio n s su ch as
th ese ‘m ay not m easure w hat th ey are in tend ed to m easure: an in d iv id u al’s gen eral in te r­
est in p o litic s’; in stead, ‘th ey m ay be m easu rin g , am o n g o th er th in gs, w hatev er resp on se
has been m ad e m ost p lau sible and accessib le in m e m o ry by the w ording o f th e qu estio n
and by the c o n tex t in w hich it is asked ’ (B ish o p et al. 1984: 1 6 0 -1 ). By ‘c o n te x t’ th ey m ean
n ot ju st the im m ed iate qu estio n , but also th e e le cto ral en v iro n m en t in w hich a qu estio n
is asked. For in stan ce, if people are asked how in terested th ey are in p o litics in th e m idst
o f an ex citin g presid en tial cam p aign, we w ould ex p ect th em to say th ey are m o re in te r ­
ested than if we asked th e sam e q u estio n d u rin g a dull, local elec tio n cam p aign . Sim ilarly,
we w ould ex p ect people to th in k th ey w ere m o re in terested in follo w ing a p o litical c a m ­
paign if they are asked th e q u estio n sh o rtly after th e elec tio n th an if th ey are asked ab out
it several weeks later. We w ould also ex p ect people w ho have voted in th e electio n to
th in k they w ere m ore interested in th e cam paign th an people were w ho did not vote
(B ish op et al. 1984: 161).
As these exam ples m ake apparent, context is often cru cial, b oth in shaping the responses
to individual item s and in evaluating them . This applies to all types o f interview alike.

Conducting a face-to-face interview

In terview in g skills

Interview ing requires skill. A good interview er (1) puts the in terview ee at ease without
b ecom ing overly fam iliar; (2) tries to m inim ize the social distance betw een him /herself and
the interview ee by dressing in a culturally acceptable and sim ple style; (3) is at ease and
never in a hurry; (4) listens to answers, shows interest in what the in terview ee savs, and
never shows any disapproval o f the in form ation received during the in terview ; (5) is sen si­
tive to the respondents' m ood, body language, tim e constraints, and different cultural
norm s; (6) probes and cro ss-checks in a thorough but sensitive m an ner; (7) m akes sure the
environm ent is tree ot noise and oth er people.
 IN T E R V IE W IN G A N D F O C U S G R O U P S 269

R eco rd ing the interview

If an in te rv ie w e e h es ita te s o r refu ses to ag ree to au d io rec o rd in g , th e o n ly so lu tio n is to take

n o tes. U sually, in te rv ie w e e s forget q u ic k ly that th e r e c o rd e r is o n , but if th ey ap p ear d is ­
tu rb e d d esp ite th e ir c o n s e n t th at it b e u sed, it sh o u ld b e stop p ed . H ow ever, ev en if you are
u sin g a ta p e -re c o r d e r (or, m o re likely, a d ic ta p h o n e ), you sh ou ld tak e n o te s o f im p o rta n t
n o n -v e rb a l e v en ts o r o b se r v a tio n s. You sh o u ld tak e n o tes in a d iscreet way w ithou t in te r ­
ru p tin g th e flow o f c o n v e r sa tio n .

Keeping co n tro l o v er the interview

T h e in te r v ie w e r h as to e n s u re th at th e in te rv ie w b eg in s w ell, and in tro d u c e th e research to

in te rv ie w e e s in a c le a r a n d c o m p re h e n siv e way. T o en s u re th at th e in fo rm a tio n prov id ed by
d iffe re n t r e sp o n d e n ts w ill b e c o m p a ra b le , th e in te rv ie w e r m u st m ak e su re th at all to p ics are
a d eq u a tely co v ered . R e sp o n se s th at go o n to o lo n g o r o f f tra c k sh o u ld b e p o litely stop p ed or
ste e re d b a c k o n tra c k . T h e in te rv ie w e r m ig h t say so m e th in g in th is c ase like: 'T h a n k you, th is
is in te re stin g . D o you m in d if 1 go b a ck to th e p rev io u s q u e stio n ? T h is (s p e c ify ) is no t yet
fu lly c le a r to m e.’

P robing

O p e n - e n d e d q u e s tio n s te n d to b e v e r y g e n e ra l (e .g . 'W h a t d o you th in k a b o u t . . .? W h y

d o y o u feel th a t w a y ? ’). C o n s e q u e n tly , r e s p o n d e n ts m ay a n s w e r in to o g e n e ra l a w ay o r
u se a d je c tiv e s th a t a r e to o g e n e ra l to c o n v e y a c le a r r e s p o n s e . In a d d itio n to b e in g u n c le a r
o r in c o m p le te , r e s p o n s e s m ig h t b e ir re le v a n t to th e q u e s tio n , o ffe r n o a n s w e r at all, o r
a p p e a r u n tr u th f u l. W h e n e v e r a r e s p o n s e is in a d e q u a te o r u n s a tis fa c to r y , th e in te r v ie w e r
c a n u se p r o b in g . P r o b in g c o n s is ts o f a s k in g q u e s tio n s to e n c o u r a g e fu r th e r c o n v e r s a tio n ,
w ith o u t in f lu e n c in g th e a n s w e r. F o r in s ta n c e , w h e n a r e s p o n s e is u ndcar o r incomplete,
th e in te r v ie w e r m ig h t g a in c la r ity o r a d d itio n a l in fo r m a tio n by a s k in g fo r a m o r e s p e c ific
r e s p o n s e , o r a n e x p la n a t io n o f a te r m . F o r in s ta n c e , th e in te r v ie w e r m ig h t ask : ‘C o u ld
y o u e la b o r a te a b it ? ’ ’C o u ld y o u m e n tio n m o re p o s s ib i liti e s ? ’ 'W h y d o y ou th in k s o ? ’ ,
'W h y d o y o u feel th a t w a y ? ’ , o r 'W o u ld yo u say s o m e m o re a b o u t t h a t? ’ O r, i f th e r e s p o n d ­
e n t h a s s t a r te d to g iv e an e x a m p le b u t d id n o t fin is h , th e in te r v ie w e r m ig h t p r o b e w ith
q u e s tio n s s u c h as: 'W h e n w as t h a t ? ’ , 'W h y d id y ou d o t h a t? ’, ‘H ow o fte n d o e s th a t h a p ­
p e n ? ’ I f th e r e s p o n s e is irrelevant to th e q u e s tio n , th e in te r v ie w e r m ig h t say s o m e th in g
lik e : ‘T h a t is n o t e x a c tly w h a t I m e a n t to a sk a b o u t’ a n d r e p e a t th e q u e s tio n , s lig h tly
e la b o r a te d .
A n a n s w e r m ig h t b e non-responsive, e it h e r b e c a u s e th e r e s p o n d e n t d id n o t u n d e rs ta n d
th e q u e s tio n o r b e c a u s e th e q u e s tio n to u c h e d o n se n s itiv e in fo r m a tio n th a t th e r e s p o n d ­
e n t h e s ita te d to d iv u lg e . In c a s e o f n o n -r e s p o n s e , th e in te r v ie w e r sh o u ld rep ea t o r re p h r a se
th e q u e s tio n w ith o u t s u g g e s tin g a r e s p o n s e . I f it a p p e a rs th a t th e q u e s tio n w as n o t u n d e r ­
s to o d , th e in te r v ie w e r m ig h t say : 'P e r h a p s , I w a sn ’t re ally c le a r in th e w ay I a sk ed th e q u e s ­
tio n . W h a t I m e a n t to a sk w as . . .’ , fo llo w e d by th e q u e s tio n , p h r a se d slig h tly m o re
e la b o r a te ly . I f th e r e s p o n d e n t s e e m s h e s ita n t to a n s w er, th e n th e in te r v ie w e r ca n stre ss
a g a in th a t th e in f o r m a tio n w ill r e m a in in c o n f id e n c e , a n d th e n re p e a t th e q u e s tio n . I f th e
r e s p o n d e n t still o b je c t s to c o n t in u in g , th e in te r v ie w e r c a n o ffe r to sk ip th e q u e s tio n .
S o m e tim e s an in te r v ie w e r m ig h t su s p e c t a r e s p o n s e is not truthful b e c a u s e v a rio u s p a r ts
o f a n a n s w e r c o n t r a d i c t e a c h o th e r o r b e c a u s e th e r e s p o n d e n t s e e m s c o n c e r n e d to giv e th e
270 H O W T O D O R E S E A R C H IN P R A C T IC E

‘d esirab le answer. The in terview er sh ould co m m u n ica te, no t su sp icio n , b ut an in terest in

c la rify in g th e situ ation , and say: ‘I d idn’t ask th e q u estio n clearly en o u g h ’, o r ‘I did n’t
un d erstand th e answ er’. S/he th en m ight say: ‘H ow sh ould I in terp ret th is’? S u m m arize th e
c o n tra d icto ry statem en ts: ‘You said X . You also said Y ’; and th en wait for th e resp on d en t
to con tin u e.

Ending the interview

At the end, the in terview er should not only su m m arize th e interview , but also respond to
q u estio ns that cam e up during the interview , give advice ( if n ecessary or asked fo r), and
give an opportu nity for fu rth er qu estio ns o f th e respondent. Su ch ‘after-in terv iew ’ d iscu s­
sions and qu estio ns should always be recorded, like all sp on taneou s in fo rm atio n , b ecause
d iscu ssion s can shed light on com p licated , not yet fully clear issues from m an y p reced ing
interview s.

Conducting online interviews

Synchronous online inter­
O n lin e interview s can be either syn chro nous or asyn chronous.
views resem ble a traditional research interview in that they take place in 'real time’. The
interview er and interview ee are on line sim ultaneously, in a virtual env iron m en t that allows
real-tim e com m u n ication betw een two or m ore users. An exam ple is an in ternet chat room ,
in w hich qu estions and answers are posted, read, and answered in ‘real tim e’. Asynchronous
interviews take place in non-real time, for exam ple using em ail. A syn chronous o n lin e in ter­
viewing can also be condu cted via bulletin boards, d iscu ssion groups, or web/internet
forum s.

Email interviews
An em ail interview can be an effective form o f interview ing. M any p eople p refer em ail in ter­
views because it is not necessary for a syn chronous tim e to be scheduled. It m ight be the only
way to cond u ct an interview (1) with a busy (or reclusive) public figure, with w hom it is not
possible to schedule a tim e to m eet face to face or to speak on the phone; or (2) if the in c o n ­
venience and expense o f travelling to an interview site would be prohibitive.
In em ail interview s, you should send an initial note to in tro d u ce you rself and to in form
the individual o f your affiliation, what in form ation you are seeking, and how the in fo rm a ­
tion will be used. Indicate the length o f tim e and the general tim e fram e in w hich you
foresee the interview process taking place. Ask the individual to specify a tim e for you to
send qu estions that will be conven ient and that will perm it a tim ely response. Agree with
the interview ee a follow -up exchange for clarification s or follow -up q u estio ns if the
responses you receive are not clear or if you do not understand so m eth in g that the in ter­
viewee writes. The researcher then sends out an em ail w hich contains the interview qu es­
tions either in the body o f the em ail or as a word attachm ent to the em ail. The participan t
responds to the interview qu estions, either in the body o f the em ail or in a word docu m ent
and returns the com pleted answers to the researcher. O ften the in terview will take place
over a period o f tim e and qu estions are sent in stages so that the interview ee is not ov er­
whelmed with a long list o f questions at the start o f the process.
 IN T E R V IE W IN G A N D FO C U S G R O U P S

Hmail is th e m o st fa m ilia r m o d e o f o n lin e in te r a c tio n , an d is relativ ely sim p le in term s o f

te c h n o lo g ic a l r e q u ire m e n ts , F m a il in terv iew s avoid all th e p ra c tic a l d ifficu lties of rec o rd in g
e q u ip m e n t a n d tr a n s c r ib in g m a c h in e s ; and th ey e lim in a te travel and tra n scrip tio n costs.
Ih e y p r esen t o p p o r tu n itie s to in terv iew in d iv id u als based a n y w h ere in th e w orld ; and th ey
ten d to re d u c e in te rv ie w e r b ias. Ih e y also giv e in terv iew ees tim e to th in k ab ou t q u estio n s
and to g iv e c o n s id e re d resp o n ses. Ih e a b ility to red raft and ed it q u e stio n s and answ ers
allo w s for in c re a se d fle x ib ility and d eep er re fle ctio n . It m ay b e, too , that p eop le ten d to be
m o re o p e n w ith o th e r s in c y b e rs p a c e th an in rea l-w o rld c o m m u n ic a tio n . Ih e lack o f visual
c lu e s ca n e n c o u ra g e c a n d id in te rch a n g e s an d o p e n up co n v e rsa tio n in d ir e c tio n s w h ich
o th e r w is e m ig h t b e a v oid ed . Ih o s e in d iv id u a ls w h o are shy an d re tice n t ab ou t sp eak in g in
f a c e - to - f a c e g ro u p in te r a c tio n s m ay fin d th at th e v irtu al e n v iro n m e n t e n co u ra g e s th em to
o p e n up a n d en g a g e in m o re c a n d id e x ch a n g es.
T h e re a re, h o w ev er, a lso d isa d v a n ta g es in c o n d u c tin g e m ail in terv iew s. Ih e m e d iu m
re q u ire s fa irly sim p le q u e stio n s, as th e re is less o p p o r tu n ity fo r th e in terv iew er to p ro b e or
cla rify . G re a te r m o tiv a tio n a n d in te re st w ill b e req u ired on th e p art o f in terv iew ees: th ey
h av e to b e w illin g to ta k e th e tim e a n d e ffo rt to sit in fro n t o f a c o m p u te r s cre e n and read tex t,
th in k , ty p e, a n d m a in ta in a lo g ic a l th rea d o f a n sw erin g . R e sp o n d e n ts w ill n o t b e ab le to
e x p re ss th e m se lv e s at a n y th in g n e a r th e rate th ey c o m m u n ic a te in sp e e ch , so less in fo r m a ­
tio n c a n b e e x c h a n g e d w ith in a giv en tim e fram e. M o reo v er, w h ile re sp o n d e n ts m ay feel
b e tte r a b le to d isc u ss se n sitiv e issu es in an o n lin e e n v iro n m e n t th an in a f a c e -to -fa c e in te r ­
view , th e y m ig h t a lso b e less o p e n if th e y have c o n c e r n s a b o u t p riv acy o r se n se a la c k o f
c o n tr o l o v e r th e u se o f p e r so n a l d ata. T h e la ck o f tim e r e s tr ic tio n an d th e slo w in g d ow n o f
th e c o m m u n ic a tio n p r o c e ss m a y lead , n o t to g re a te r o p e n n e ss , b u t to la ck o f en g a g e m e n t o r
to th e p r o d u c tio n o f ‘so c ia lly d e sir a b le ’ a n sw ers, ra th e r th an th e m o re s p o n ta n e o u s r esp o n ses
ty p ic a l o f s y n c h ro n o u s in te r v ie w s (b o th o n lin e an d face to fa ce ). T h e delayed in te r a c tio n
an d th e in a b ility to s p o n ta n e o u s ly d ir e c t th e flow o f c o n v e r sa tio n , o r to p ro m p t an d p ro b e
p a r tic ip a n ts , m a y lea d to a p a u c ity o f d ata c o m p a re d to s y n c h ro n o u s in terv iew s. Fin ally,
th o u g h e m a il a n d o th e r ty p es o f o n lin e in te r v ie w in g p ro v id e r e se a rc h e rs w ith o p p o r tu n itie s
to in te r v ie w in d iv id u a ls b a sed an y w h e re in th e w orld , n o t e v e ry o n e h as a c c e ss to th e in te r ­
n et. Y o u n g , w ealthy, a n d ed u c a te d p e o p le ten d to use c o m p u te r s m o re th a n th e eld erly, p oor,
a n d u n e d u c a te d .

Interviewing via video link and web camera

A s y n c h ro n o u s o n lin e in te r v ie w in g ca n a lso b e c o n d u c te d v ia v id eo lin k o r w eb c a m e ra .
C o n f e r e n c in g so ftw a re is a v a ila b le fo r fa c ilita tin g o n lin e s y n c h ro n o u s in terv iew s in a c h a t­
r o o m -ty p e e n v iro n m e n t. T h e so ftw a re p a ck a g e s c a n b e d o w n lo a d e d b y th e p a rticip a n ts.
T h e y p ro v id e a la rg e ‘c h a t’ w in d o w in w h ich th e d ia lo g u e is d isp layed . B e n e a th th is is a
s m a lle r w in d o w w h e re u se rs ty p e th e ir te x t, a n d p ress r e tu r n ; s e c o n d s later th e c o n trib u tio n
is d isp la y ed , p re fix e d w ith th e ir n a m e.
U sin g s y n c h r o n o u s c h a t d iffe r s in a n u m b e r o f im p o r ta n t w ays fr o m m o re tr a d itio n a l
in te r v ie w in g . M is s in g a re th e v isu a l, n o n -v e r b a l c u e s w h ic h th e in te r v ie w e r w ou ld n o r ­
m a lly rely o n in o r d e r to b u ild r a p p o rt a n d g a in th e tr u s t o f th e in te r v ie w e e . T h e in te r v ie w e r
m u st, th e r e fo r e , g iv e th o u g h t to w ays o f e s ta b lis h in g ra p p o rt w ith in te rv ie w e e s. V ir tu a l
H O W T O D O R E S E A R C H IN P R A C T IC E

interview s also do not perm it the natural flow o f conv ersation in th e way that telep h on e or
fa ce-to -fa ce interview s do. C onsequen tly, th e nu m b er o f qu estio ns sh ould be lim ited. The
n u m b er o f o p en -en d ed qu estio ns should be lim ited, as well. M ore th an a few q u estio n s in
total will m ake th e in terview begin to feel lab oriou s. If m ore in fo rm atio n is needed , th e
in terview er can ask in terview ees if they are willing to do a follow -up pho ne interview . If
you will be asking confidential or p ersonal qu estio ns, you m ust co n sid er how to handle
privacy issues.
M ake sure you un d erstand th e eq u ip m en t and p rocess b efore th e in terview and, in
particu lar, th e type o f m icro p h o n e and w hat ad ju stm en ts m ay need to be m ade in using
it. Test your eq u ip m ent and, if u sing a w eb cam , p r actice using it b efo re th e interview . G et
a FAQ sh eet on how to use the eq u ip m ent and respond to p ossible glitch es that m ight
arise. C h o o se an area with a neu tral b ackgro u n d , and m ak e sure th e areas arou nd you are
clea r and orderly. You will want to speak directly to th e cam e ra so that it appears you are
speaking to the in terview ee. P ositio n your m aterials in clo se p ro xim ity to th e c a m e ra so
that w hen you glan ce at your no tes, you will no t appear to be lo o k in g away from th e c a m ­
era. C on sid er how to elim in ate all p otentially d istractin g no ises, as th e m icro p h o n e on a
w ebcam can m ag nify the slightest sou nd. If u sing Skype, you m ay e x p erien c e som e tim e
lag when you’re talking. M ake sure you talk clearly so your v oice will be easily picked up
by the m icrop h on e.
F ocus groups, w hich we discuss later on in this chapter, can also be condu cted on line.
O nline focus groups can be condu cted using listserv , which is one o f the key softw are appli­
cations for m anaging m ailing lists. O n e advantage o f this is that it elim in ates the need to set
up m utually convenient chat tim es. However, the in terview er does not com m u n icate sy n ­
chronously with participants. R espondents can post their replies at any tim e. This reduces
the level o f group in teraction and the sense o f im m ediacy. And the fact that the facilitator
cann ot always play an active role in m oderating the discussion interview m eans that there is
a greater likelihood o f exchanges being unfocused or off -topic.

Elite interviews
M uch political research is c o n ce rn e d with u n d erstan d in g p olitical in stitu tio n s and
d e cisio n -m a k in g p rocesses. In terv iew in g p olitical elites is a key m eans o f o b tain in g
in form a tion about m any asp ects o f th ese p h en om en a. P olitical elites are people ‘w ho
ex ercise d isp rop ortio n ately high in flu en ce on the ou tco m e o f ev ents or p o licies in your
research area’ (P ierce 2 0 0 8 :1 1 9 ). We can id en tify th ree broad pu rp oses for w hich in te r­
view ing elites can be useful.
The first purpose is to obtain new inform ation . By virtue o f the position s they hold,
elites may have access to inform ation that might not oth erw ise be available to a researcher.
However, a second purpose o f elite interviews is to confirm the a ccuracy o f in form ation that
has previously been collected from oth er sou rces’. As O isin Tansey explains,

W he n d o cum ents, m e m o irs, and seco ndary so urccs p ro vid e an in itia l o ve rvie w o f the events
or issues un der e xa m in a tio n , in te rv ie w s w ith key players can be used to co rro b o ra te the e a rly
fin ding s. In tins w ay, in te rv ie w s c o n trib u te to w ard the research goal o f tria n g u la tio n , w here
 IN T E R V IE W IN G A N D FO C U S G R O U P S 273

c o lle c te d clati» a r c c ro s s c h e c k e d th ro u g h m u ltip le s o u rc e s to in c re a s e the fin d in g s ' ro b u s tn e s s

(T a n s e y 2 0 0 7 : 7 6 6 )

fin a lly , elite in te rv ie w s ca n e n a b le a r e se a rch e r to m ak e in fe re n ce s ab ou t th e b eliefs or

a c tio n s of a w id er p o p u la tio n o f p o litic a l elites. W h e n an aly sts ran d o m ly select a sam ple of
elites to in te rv ie w fro m a m o n g a b ro a d e r p o p u la tio n , th ey can g en era liz e from th e fin d in gs
of th at sa m p le to th e w id er g ro u p (see e.g. A b erb a c h and R o c k m a n 2 0 0 0 ).
O n e of th e b ig gest c h a lle n g e s o f c o n d u c tin g elite in terv iew s is g ettin g th e ind iv id u als you
select to a g ree to an in terview . W h at you will need to d o is to w rite to p ro sp ectiv e in terview
su b je c ts. I h is sh o u ld b e a b rief letter on th e m o st p restig io u s, n o n in fla m m a to ry letterhead
you have a c c e ss to ’ (A b e rb a c h an d R o ck m a n 2 0 0 2 :6 7 4 ) . 'I h e letter sh ou ld state your p u rp ose,
but not in great d eta il. It sh o u ld su gg est, in a su b tle way, w hy th e in terv iew w ould in so m e way
b en efit th e p ro sp e c tiv e in terv iew ee. R o g er P ierce ( 2 0 0 8 : 120) lists a few reaso n s why elites
m ig h t w ant to a g ree to an in terview . T h ese m igh t in clu d e:

• ’you hav e p ersu a d ed th em o f th e im p o rta n c e o f y o u r resea rc h and th eir p o ten tial

c o n t r i b u tio n ’;

• th ey feel th a t th e ir p o s itio n o b lig a te s th em to a g ree to in terv iew s an d th at req u ests for

in te rv ie w s u n d e rlin e s th e ir sta tu s;

• th ey w a n t an o p p o r tu n ity e ith e r to ‘set th e re c o rd s tra ig h t’, o r to reflect o n th e p o licies,

p o litic s, o r in s titu tio n s w ith w h ich th ey are o r h av e b een asso cia ted .

Y o u r in itia l le t te r s h o u ld a ls o in f o r m th e r e c ip ie n t o f y o u r in te n t io n to fo llo w up w ith a

p h o n e c a ll to se e if a tim e m ig h t b e a r ra n g e d to m e e t. You m u st th e n fo llo w up th e le tte r
w ith a p h o n e c a ll. P r e p a r e fo r an e x c h a n g e w ith th e a p p o in tm e n t s s e c r e ta r y in w h ich
you w ill d e liv e r a c h a r m i n g a n d p e r s u a s iv e a c c o u n t o f y o u r s e r io u s n e s s o f p u r p o s e . B e
p e r s is t e n t a n d ‘ in s is t fir m ly , b u t p o lite ly (a n d w ith a c o n v in c in g e x p la n a t io n ) ’ th at n o
o n e b u t th e ta r g e te d in d iv id u a l ‘w ill d o fo r th e in te r v ie w ’ (A b e r b a c h an d R o c k m a n
2 0 0 2 : 6 7 4 ).
W e h a v e n ’t said a n y th in g a b o u t h o w to c h o o s e in terv iew su b je c ts. R o g er P ierce sugg ests
th a t you ‘a im h ig h e r ra th e r th a n low by a p p r o a c h in g th e “A -L ist elite s” ra th e r th an lesser
elite s w h o m you m a y c o n s id e r to b e m o re a p p ro a c h a b le (P ie rc e 2 0 0 8 : 1 2 1 ). But, as G eo rg e
an d B e n n e tt p o in t o u t, o fte n lo w er-lev el o fficia ls m ay b e b e tte r in terv iew so u rce s, giv en th eir
d a y -to -d a y in v o lv e m e n t w ith p o litica l p ro ce sse s (G e o rg e an d B e n n e tt 2 0 0 5 : 10 3 ). As w ith all
r e se a rc h d e c is io n s , you m u st let y o u r r esea rc h q u e stio n and h y p o th e sis d e te rm in e your
c h o ic e o f in te r v ie w su b je c ts . W e w ould like to p o in t o u t, ho w ev er, th at you sh o u ld no t h e s i­
ta te to p u rsu e ‘A -lis te r s’ th a t you feel w ould c o n trib u te to you r resea rc h . A -lis ters are alw ays
h ap p y to re c e iv e re q u e s ts fo r th e ir tim e an d a tte n tio n ; th e y may, in fact, feel u n easy w h e n ­
ev er su c h re q u e s ts b e g in to th in out.
L e t’s m o v e o n . O n c e y o u h a v e a n in te r v ie w s c h e d u le d , yo u m u st d e c id e o n a fo r m a t.
E lite in te r v ie w s te n d to be s e m i- s tr u c tu r e d and u se o p e n -e n d e d , r a th e r th a n
c lo s e d - e n d e d , q u e s tio n s . B u t th e r e a re a n u m b e r o f f a c to r s y o u sh o u ld c o n s id e r in
d e c id in g w h a t ty p e o f q u e s tio n to u se . O n e c o n s id e r a t io n is th e d e g r e e o f p r io r r e s e a rc h
o n th e s u b je c t o f c o n c e r n . T h e m o r e th a t is k n o w n , th e e a s ie r it is to u se c lo s e d - e n d e d
H O W T O D O R E S E A R C H IN P R A C T IC E

q u estio n s. C on v ersely , th e m o re u n c h a r te d th e te r r a in , th e m o re lik e ly you w ill w an t to

use e x p lo ra to ry , o p e n -e n d e d q u e stio n s. A se co n d c o n s id e ra tio n is ‘re sp o n s e v a lid ity .
O p e n -e n d e d q u estio n s in cre a se th e v a lid ity o f th e resp o n s e s b e c a u se th e y p ro v id e a
g rea ter o p p o rtu n ity fo r resp o n d e n ts to o rg a n iz e th e ir an sw ers w ith in th e ir ow n fr a m e ­
w orks. T h e th ird m a jo r c o n s id e ra tio n is th e re c e p tiv ity o f re sp o n d e n ts. ‘E lite s e s p e ­
c ia lly — b u t o th e r h ig h ly ed u cated p eo p le as w ell— d o n o t lik e b e in g put in th e s tr a it-ja c k e t
o f c lo se d -e n d e d q u estio n s. T h e y p refer to a r tic u la te th e ir v iew s, e x p la in in g w hy th ey
th in k w hat th ey th in k .’ R e fle ctin g on th e ir e x p e rie n c e in in te rv ie w in g elite s, Jo el A u e r­
b a ch and B ert R o ck m a n n o ted th at c lo se d -e n d e d q u e stio n s o fte n e lic ite d q u e stio n s in
retu rn ab o u t w hy we used th e resp o n se c a te g o rie s we u sed o r w hy we fra m ed th e q u e s ­
tio n s th e way we d id ’ (A u erb a ch and R o ck m a n 2 0 0 2 : 6 7 4 ).
Th ough elite interview s often rely o n o p en -en d ed q u estio n s, a co m b in a tio n o f o p en - and
closed-end ed q u estio ns m ight provide research ers w ith b etter o p p o rtu n ities to o b ta in th e
in form ation they are seeking. For exam ple, Sh aron Rivera and h er colleagu es used a c o m ­
bination o f op en -end ed and closed -en d ed qu estio n s to in terview R ussian elites. They
started out with five op en -end ed q u estio ns, th en follow ed with a couple o f clo se d-e n d ed
qu estions, and th en alternated betw een op en - and closed-en d ed qu estio n s for th e re m a in ­
der o f th e interview. W ith th is seq u encing, they were able to realize a n u m b er o f advantages.
By starting with open -end ed q u estio ns, they found it easier to elicit answ ers to closed-
ended questions. As they describ e it: ‘W e had dem onstrated respect for th e com p lexity o f
th eir views through the open -end ed qu estions and thus had “earn ed ” th e right to ask q u es­
tions posed exclusively from our fram e o f reference. Also, the c losed-en d ed qu estio ns p ro b ­
ably allowed respondents to recover a bit from the m ore d em anding o p en -en d ed qu estion
form at’ (Rivera et al. 2002: 686).
In addition to, or as part o f the p rocess of, preparing qu estions for the interview, you must
th oroughly research the subject. M ake sure you know as m uch about him or her as possible.
You should also check on line for im ages so that you will be able to recognize your in ter­
viewee by sight. Arrive at the interview with your qu estionnaire, and/or in terview schedule
or guide, and an audio recorder. W hether or not you record the interview, you will want to
take notes (to record non-verbal cues, as well as your own reflections); but you should not
make the note-taking obvious, or let it deter you from m aintaining eye contact. In co n d u ct­
ing the interview, you will need to give space for the respondent to open up new areas, while
m aking sure to cover the topics you feel you need to address. End the interview ten m inutes
early. The most revealing inform ation is expressed by interview ees when the interview
appears to have been formally ended. After the interview, type up your notes and transcrib e
your recording as soon as possible.

Expert interviews and surveys

E x p e rt in te rv ie w s an d s u r v e y s m ig h t b e th o u g h t o f a s a ty p e o f 'e l i t e i n te r v ie w ’. B u t th e
ty p e o f d a ta r e s e a rc h e rs seek to c o lle c t t h ro u g h e x p e r t in te rv ie w s a n d s u r v e y s w ill u s u ­
ally d iffer q u ite s u b s ta n tia lly t ro m th e d a ta th e y m ig h t o b ta in th r o u g h th e u se o f e lite
in te rv ie w s . E x p e rt s u r v e y s a re w id e ly u se d in c o m p a r a t iv e re s e a rc h to m e a s u r e a w id e
variety of d iffe re n t p o litica l p h e n o m e n a , an d c a n b e a n a ly s e d b o th q u a lita tiv e ly an d
q u a n tita tiv e ly .
 IN T E R V IE W IN G A N D FO C U S G R O U P S

In an e x p e r t in te r v ie w o r su rv ey , th e r e s e a r c h e r id e n tifie s in d iv id u a ls w ith s p e cia liz e d

k n o w le d g e o r e x p e r t is e re la tin g to a p a r tic u la r issu e a n d ask s th em a c o m m o n set o f
q u e s tio n s , e it h e r d ir e c tly in f a c e - t o - f a c e in te r v ie w s , o r th ro u g h th e a d m in is tr a tio n o f a
p o s ta l su rv ey . R a th e r th a n p r o b in g to d is c o v e r w h at se le c te d in d iv id u a ls th in k , an d w hy
th e y th in k w h a t th e y th in k , th e a im o f e x p e r t in te r v ie w s is to e lic it s p e c ific in fo rm a tio n
fr o m in d iv id u a ls w ith s p e c ia liz e d k n o w le d g e o r e x p e r tis e on a p a r tic u la r issu e. E x p e rt
su rv e y s p e r f o r m th e sa m e fu n c tio n . E x p e r t su rv e y s e n a b le s r e s e a rc h e rs to s u m m a r iz e th e
c o n s e n s u s ju d g e m e n t o f e x p e r ts o n a g iv en issu e an d d o so in a s y ste m a tic w ay ( B e n o it
an d L a v er 2 0 0 6 : 9 ).
O f c o u r se , th e v a lid ity o f e x p e r t-o p in io n data w ill d ep en d o n th e qu ality o f th e ex p erts.
E x p e rt ju d g e m e n ts are v alid a n d relia b le in so far as th e e x p e rts are w e ll-in fo rm e d ab ou t th e
s u b je c t in q u e stio n . S o m e stu d ies u se a large p o o l o f e x p e rts; o th e r s rely o n a sele cted few.
T h e use o f m u ltip le e x p e rts in c re a se s th e v a lid ity o f th e data (D o ru s s e n et al. 2 0 0 5 : 3 1 7 ). But
in so m e in sta n c e s k n o w led g e m ay b e lim ite d to a h a n d fu l o f o b se rv e rs o r p a rtic ip a n ts, eith e r
b e c a u se o n ly a few p e r s o n s a re p riv y to th e relev an t in fo rm a tio n (e.g. g o v e rn m e n t p o sitio n s
in b e h in d - c lo s e d - d o o r s n e g o tia tio n s ) o r b e ca u se o n ly a few p e rso n s hav e th e relev an t e x p e r ­
tise (H o o g h e et al. 2 0 0 9 : 4 ).
E x p e r t - o p in io n d a ta h a s b e e n u se d w id ely in stu d ies o f th e E u ro p ea n U n io n . A n ex am p le
is th e D o m e s tic S tru c tu r e s a n d E u r o p e a n In te g r a tio n (D O S E I) p r o je ct. T h e p u rp o se o f th e
p r o je c t w as to d e te r m in e th e p o s itio n o f th e E U m e m b e rs o n th e d raft C o n stitu tio n for th e
E u ro p e a n U n io n . A s p a rt o f th e D O S E I p r o je c t, c o u n tr y e x p e rts w ere asked to assess th e
n a tio n a l p o s itio n o n th e d ra ft C o n s titu tio n . A to ta l o f 7 7 e x p e rts w ere in terv iew ed , v ary in g
fro m o n e to six e x p e rts fo r a n y p a r tic u la r p o litic a l a c to r ’ (D o ru s s e n et al. 2 0 0 5 : 3 1 6 ). T h e
p r o je c t c o lle c te d a w ea lth o f in fo r m a tio n o n all th e relev an t p o lic y p o sitio n s o f all th e m ain
a c to r s w ith r e sp e c t to th e d ra ft E u r o p e a n C o n s titu tio n .
E x p e r t s u r v e y s a ls o h a v e b e e n u se d to id e n tify th e p o lic y p r e fe re n c e s o f p o litic a l p a r ­
ties. W h a t th e s e su r v e y s se e k to d o is to o b ta in a c o n s e n s u s o f s p e c ia lis ts a b o u t p a r tic u la r
p o lic y p o s itio n s in o r d e r to id e n tify w h e re p a r tie s s ta n d o n n a tio n a l issu es. In a stu d y th at
re v e a le d th e p o t e n tia l g a in s th a t c o u ld b e p r o d u c e d fr o m u sin g e x p e r t su rv e y s fo r th is
p u r p o s e , F r a n c is C a s tle s a n d P e te r M a ir ( 1 9 8 4 ) u se d a p o s ta l su rv e y to a sk e x p e rts in 16
c o u n tr ie s to lo c a t e p a r tie s in t h e ir ‘o w n ’ c o u n tr ie s o n a te n -p o in t s c a le b e tw e e n L eft and
R ig h t. K e n n e t h B e n o it a n d M ic h a e l L a v e r ( 2 0 0 6 ) c o n d u c te d a s im ila r su rv e y c o v e r in g 4 7
c o u n tr ie s . T h e ‘e x p e r t s ’ th e y su r v e y e d w ere ty p ic a lly a c a d e m ic s p e c ia lis ts in th e p o litic a l
p a r tie s a n d e le c t o r a l p o litic s o f t h e ir ‘o w n ’ c o u n tr ie s . B e n o it a n d L a v er d e fin e d fo u r s u b ­
s ta n tiv e p o lic y d im e n s io n s as a b a s is fo r c o m p a r in g p o lic y s c a le s a m o n g su rv e y e d c o u n ­
tr ie s . F o r e a c h p o lic y d im e n s io n ( e c o n o m ic p o licy , s o c ia l p o licy , th e d e c e n tr a liz a tio n o f
d e c is i o n - m a k in g , a n d e n v ir o n m e n ta l p o lic y ) , th e y u se d a s c a le r u n n in g fr o m 1 to 2 0 ,
w ith th e lo w e r p o s itio n in d ic a tin g th e ty p ic a lly ‘le ft’ p o s itio n an d th e h ig h e r v alu e th e
tr a d itio n a lly ‘r ig h t’ p o s itio n .
E x p e rt su rv e y s a re a lso u se d to c o lle c t a w id e v a rie ty d ata in c o m p a ra tiv e re se a rc h o n to p ­
ics su c h as d e m o c ra c y , c o r r u p tio n , h u m a n rig h ts, an d p o litic a l re p re s sio n , to n a m e b u t a few
(se e C h a p te r 9 ) . O fte n h a rd d a ta o n th e se issu es is e ith e r u n re lia b le o r u n av ailab le. F or
e x a m p le , v e r y few g o v e r n m e n ts k e e p u p -to -d a te r e c o rd s o n h o w m a n y p o litic a l p riso n e rs
th e y to r tu re . N o r is th e re m u c h in th e w ay o f o fficia l d a ta o n h o w c o r ru p t b u sin e ss p ra c tic e s
are. T o g e t r o u n d th is p r o b le m re s e a rc h e rs o fte n rely o n ev a lu a tio n s o f d iffe re n t p o litica l
276 H O W T O D O R E S E A R C H IN P R A C T IC E

p h e n om en a by ex p erts in th e relevant field. So, w ho b etter to assess how co rru p t b usin ess
p ra ctices are in different c o u n tries th an b u sin essm en w ho frequ en tly travel arou n d th e
world? Tran sp aren cy In tern atio n al has now b een carry in g ou t ex p ert surveys on th is issue
for a n u m b er o f years. It asks b u sin essm en a series o f q u estio n s a b ou t busin ess p ractices and
corru p tio n in different c ou n tries. T h e d ata d oes no t th erefo re strictly m easure how corru p t
a co u n try is. R ath er it m easures p ercep tio n s o f corru p tio n . But th e m o re b u sin essm en agree
on w hether o r not a co u n try is corru p t, th e higher w e can regard th e v alidity o f th e m easure.
W e d iscu ss th is issue o f the in ter-co d er reliability in m o re d etail in C h ap ter 13, b ut th e p rin ­
ciples are m uch the sam e in expert interview s.
An exam ple helps to illustrate th is point. Exp ert surveys are frequ en tly used to m easure
th e level o f d em o cracy in a country. Tw o o f th e m o st w idely used sou rces o f data on th is are
Freed om H ouse and Polity IV. B o th draw on expert surveys (to differing degrees). E xp erts
(defined as cou n try specialists) are asked to evaluate th e ex ten t to w hich sp ecific co u n tries
m eet different criteria. For exam ple, on e qu estio n o n the p olitical e n v iron m en t asks ‘To w hat
extent are m edia outlets’ new s and in form ation con ten t d eterm ined by the gov ernm ent o r a
particular partisan interest? ( 0 - 1 0 p o in ts)’. A lthough th ere is a certain am o unt o f su b jectiv ­
ity in herent in this approach (see C hapter 4 ) ideally, if all the experts are unbiased and well
qualified to answer this qu estion, then they should all agree and give th e sam e rating. U nder
these circu m stan ces in ter-cod er reliability is high, and we can th in k that percep tio ns are a
valid m easure o f actual experience.

II. Focus groups

T h e fo cu s g r o u p c a n b e c o n s id e re d a fo rm o f g r o u p in te rv ie w . H o w e v e r, it d iffers fro m a
s ta n d a rd g r o u p in te rv ie w in tw o w ays. F irs t, it in volv e s a g r o u p o f in te rv ie w e e s w h o a re
s e le c te d o n th e b asis o f t h e ir h a v in g h a d , o r n o t h a d , s o m e p a r t ic u la r e x p e r i e n c e , o r h a v in g
a p a rtic u la r a ttrib u te . S e c o n d , ra t h e r th a n a s k in g s u b je cts to re s p o n d to th e s a m e q u e s tio n ,
a s in a g ro u p in te rv ie w , th e fo cu s g r o u p se ss io n is o r g a n iz e d to a d d re s s a th e m e o r to p ic.
A lan B ry m a n c h a r a c te riz e s a fo cu s g r o u p as c o m b in in g a g r o u p in te rv ie w w ith a fo c u s e d
in te rv ie w ( 2 0 0 4 : 3 4 6 ) .
F o cu s g ro u p s a re a fo rm o f in te rv ie w in g in v o lv in g a g r o u p o f six to ten p e o p le w h o m eet
in a c o n f e r e n c e -r o o m - l ik e se ttin g , a n d a re in te rv ie w e d t o g e th e r in a flexib le an d e x p l o r a t o r y
g ro u p d isc u s sio n fo rm a t. M e m b e rs of th e fo cu s g r o u p a re s e le c te d b e c a u s e th e y s h a r e s o m e
a ttrib u te o r are related to s o m e p h e n o m e n o n o f in te re s t. T h e y m ig h t b e c h o s e n , fo r in sta n ce ,
b e ca u s e th e y are all b e tw ee n th e age s of 3 5 a n d 5 0 , o r all h a ve a ch ild c u r re n tly s e rv in g in th e
a rm y an d b ased o v e rs e a s, o r a re all fem ale p o lice office rs. I h e p e o p le s e le c te d to fo rm a
g ro u p ca n be u n k n o w n to e a c h o th e r, o r a lre a d y fo rm s o m e s o rt of p re -e x is tin g o r n atu ra l
g ro u p — p eop le w h o k n ow ea ch o th e r th ro u g h clu b s o r so c ia l c e n tre s , w h o w o rk to g e th e r, o r
a re c la ssm a te s.

K ath er th a n a c tin g as an in te rv ie w e r, th e r e s e a rc h e r p e rfo rm s th e ro le o f m o d e r a to r o r

facilitator. Ih e m o d e r a to r in itiates th e se ssio n an d m o d e r a te s th e e n su in g d isc u s sio n . Ih e
fa c ilita to r g e n e ra lly c o m e s e q u ip p e d w ith a relativ ely sm all n u m b e r o f q u e stio n s an d a w ill­
in gn ess to relin q u ish a la rge r d e g re e of c o n tro l o v e r th e p ro c e e d in g s th an w ou ld n o rm a lly be
tlu iasi. in an in divid u al in terv iew , an d to allow th e d isc u s sio n to d e v elo p an d m o v e in
 IN T E R V I E W I N G A N D F O C U S G R O U P S

d ifferen t d ir e c tio n s w ith a c e r ta in d eg ree o f sp o n tan e ity . T h e em p h a sis is on in te ra ctio n s

b etw ee n p a r tic ip a n ts ra th e r th a n b etw ee n th e r e se a rc h e r and p a rtic ip a n ts. T h e p u rp o se is to
ex p lo re p e o p le s id ea s in a p u b lic se ttin g so th at th e in terv iew er ca n o b se rv e how th ey react
to ea c h o th e r ’s id ea s an d ho w th e ir o p in io n s are fo rm ed .
W h y d o fo c u s grou p in terview s rath er th an in dividual in terview s? In terv iew in g individuals,
eith e r face to face o r ov er th e telep h o n e, o r th rou gh m ailed qu estio n n aires, helps research ers
lea rn a b o u t how p eop le feel. T h e fo cu s grou p is a g ood tech n iq u e for ex p lo rin g social in te ra c ­
tion s, and ho w th ese sh ap e attitu d es and op in io n s. E xch an g es am o n g p articip an ts can lead to
far m o re p ro b in g and re flectio n th an is p ossible in in dividual in terview s o r q u estio n n aires, and
m ay prov id e m o re robu st and rev ealin g resp o n ses to th e issues w hich are th e su b ject o f the
fo cu s group. T h e d isc u ssio n ca n also b rin g in to fo cu s d im en sio n s o f an issue not previously
co n sid ere d by th e in terview er. T h e u n iqu e c o n trib u tio n o f a focu s group am o n g th e array o f
in terv iew m e th o d s is th at it en a b le s a research er to study ‘th e ways in w hich individuals c o l­
lectiv ely m a k e sen se o f a p h e n o m e n o n and c o n s tru c t m e an in g s arou nd it’ (B ry m a n 20 0 4 : 34 8 ).
In C h a p te r 2 we c o n s id e re d ho w u n d e rsta n d in g s em e rg e as a result o f in te r a c tio n and
d isc u ssio n as in d iv id u a l p a r tic ip a n ts c o n tra s t an d fit th e ir in d iv id u al v iew s w ith th o se o f
o th e rs. T h e u n d e rsta n d in g s th a t e m e rg e s fro m in te r a c tio n a m o n g in d iv id u als rep resen t, no t
ju s t th e su m o f th e ir sep a ra te v iew s, b u t so m e th in g ‘no vel, n o t p o ssessed by th e in d iv id u als
ta k e n in is o la tio n a n d th a t w ou ld n o t h av e e m e rg e d in th e a b se n c e o f th e ir in te r a c tio n ’. In te r ­
a c tiv e r e la tio n s b e tw e e n in d iv id u a ls c a n p ro d u c e p ro p e rtie s th a t are sep arate an d d is tin ­
g u ish a b le fr o m th e in d iv id u a ls th em se lv es.
W h e n a g ro u p o f p eo p le w ith s im ila r in terests d iscu ss an issu e to g eth er, th ey are likely to
p ro d u c e r ic h e r in sig h ts a n d a w id er ran g e o f in fo rm a tio n th an in d iv id u al resp o n ses o b ta in ed
privately. O n e p e r so n ’s c o m m e n t is likely to p rov ok e a re a c tio n fro m th e o th e r p a rticip a n ts
an d g e n e ra te m o re v iew s. S in c e m e m b e rs o f fo c u s gro u p s are sele cted fo r so m e sh ared
a ttrib u te o r e x p e rie n c e , th e y are likely to feel m o re c o m fo rta b le ex p ressin g th e ir ideas and
feelin g s; a n d , b e c a u se th e y a re n o t req u ired to an sw er sp ecific q u estio n s, resp o n d e n ts m ay b e
m o re s p o n ta n e o u s, a n d a c c u r a te , w h e n e x p ress in g th e ir v iew s. H ow ever, a m a jo r d e te rm i­
n a n t o f fo c u s g ro u p su c c e s s a n d th e q u a lity o f th e ir resu lts is th e skill o f th e m o d erato r. A lso,
fo c u s g ro u p s a re n o t r ep resen ta tiv e o f th e g en era l p o p u la tio n ; an d th e u n stru ctu re d n atu re o f
th e re sp o n s e s in fo c u s g ro u p d isc u ssio n s m ak es c o d in g , an alysis, an d in te rp re ta tio n difficu lt.
H e n c e , th e r esu lts o f fo c u s g ro u p d iscu ssio n s sh o u ld b e treated as ex p lo rato ry , ra th er th an
c o n c lu siv e .

Organizing and conducting a focus group session

O rg a n iz in g a n d c o n d u c tin g a fo c u s g ro u p se ss io n in vo lv es th e fo llo w in g steps.

1. Identify the m ajor objective o f the focus group session. Ju st as is th e ca se w ith

q u e s tio n n a ir e s o r in te rv ie w g u id e s, fo c u s g ro u p s m u st b e d esig n ed to p ro d u ce data
r e le v a n t to a h y p o th e sis.

2. Develop questions. A se ss io n sh o u ld last b e tw e e n 6 0 an d 9 0 m in u tes. T h is is tim e

en o u g h to a llo w th e m o d e ra to r to ask at m o st five o r six q u e stio n s. M a k e su re th ese
are fo r m u la te d so a s to e lic it th e in fo rm a tio n you w an t to b e ab le to g a th e r d u rin g th e

se ss io n .
278 H O W T O D O R E S E A R C H IN P R A C T IC E

3. Membership. F ocu s groups are usually co n d u cted w ith six to ten m e m b ers w ho have
som e sim ilar attrib u te o r ex p erience. Select m e m b ers w ho are likely to be p articip ativ e
and reflective, and w ho d o n ’t know each oth er.

4. Plan the session. H old the session in a co n fe re n ce ro o m , w ith p a rticip an ts sittin g in a

c ircle o r arou nd a sem in ar tab le so th at ev eryone can see ev eryon e else. Prov id e n am e
tags for m em b ers, and refresh m en ts. The agenda w ould n o rm ally lo o k so m eth in g like
this: ( 1 ) w elcom e the p articip an ts; (2) review th e agen da; (3) ex p lain th e goal o f th e
m eetin g, and discu ss the ground rules; (4) have g roup m em b ers in tro d u ce th em selv es;
(5 ) qu estio n s and answ ers; (6) w rap up.

5. Running the session. The m o d erator will norm ally begin a session by in tro d u cing him/
herself, the purpose o f the research, the form at o f the session, and som e g round rules,
such as allowing everyone to speak and only o n e person to speak at a tim e. The m oderator
will also want to ask participants to fill out a form with som e basic in form ation about
them selves: age, occupation, etc. In addition to an in form ation sheet, p articipan ts should
also be a given a consent form to sign b efore b egin ning the focus group.

All m em b ers should participate as m uch as p ossible. If on e o r two people are d om in atin g the
m eeting, then call on oth ers. Tell th e individuals you want to hear from the oth ers. D on’t
look at the individuals when you ask a qu estion. Raise your hand as if to say stop w hen the
individual tries to talk, and look at som eon e else. You m ight say som eth in g like ‘L ets have
som eone else go first.’ If the participant continu es to d om inate the session, you m ight sug­
gest using a round-table approach, including going in one d irection around the table, giving
each person a m inute to answer the question. If the dom ination persists, note it to the group
and ask for ideas about how the participation can be increased.
D on’t be ab ru pt with a d o m in atin g p articip an t, as it m ay d isco u rag e p articip a tio n
from oth ers. And if you feel your efforts to b road en th e d iscu ssio n have led to a w ith ­
drawal o f p a rticip atio n by the d o m in atin g in d ividual, try to re -estab lish rap p ort. If n o th ­
in g else w orks, take a b reak and privately e ith er d iscu ss your c o n c e rn s w ith th e in dividual
or ask the individual to leave. The d iscu ssio n sh ould rem ain focu sed , m ain tain m o m e n ­
tu m , g en erate useful in fo rm atio n , and achieve closu re on q u estio n s. A fter each q u estio n
is answ ered, the m o d erato r should reflect back a faithful su m m ary o f what was said.

6 . R eco rd th e session with eith er a il a u d io o r a u d io -v id e o reco rd er. T h e fo cu s g r o u p s e ssio n

s h o u ld be r e c o rd e d , u sin g m ic ro p h o n e s s tro n g e n o u g h to p ick up v o ice s a r o u n d th e
tab le o r c irc le . It will be im p o rta n t to h a v e a r e c o r d , n o t ju st o f w h a t w as said , b u t o f h o w
it w as said a n d w h o said it. So th e re sh o u ld b e s o m e o n e ta k in g n o te s , as w ell.

Il l Analysing interview data

D a ta a n a ly s is o l in t e r v ie w d a ta e n ta ils th re e m a in ste p s: d a ta r e d u c tio n , c o d in g , a n d d r a w in g

c o n c lu s io n s . In d e e d th e se steps ol a n a ly s is a re c o m m o n to a ll ty p e s o f d a ta , th o u g h h o w w e
go abo ut d o in g it in p r a c tic e v a r ie s s o m e w h a t a c c o r d in g to th e ty p e o f d a ta w e h a v e c o lle c te d
(lo r e x a m p le , see the d is c u s s io n o l te x tu a l a n a ly s is in C h a p te r 13 ).
 IN T E R V I E W I N G A N D F O C U S G R O U P S 279

BOX 11 3 Different Processes of Data Analysis

Structured interviews Unstructured interviews

A ra.'yv s s "g r e : sta*. s: :s a - ; s ana .sec s.-stemat ca crg a -c -g anc

- :e :" - e : -g - f c — a: ~ -.g categcr.es. themes,
a " ~ c : is : r ¿ : \ z a::e-~ i anc 'e.at-onsnips

Result l e r c to s*-nnmar.2e 2ane"'.$ of s n^a^i es Results are r.-ceoth exoiaoat;o-'s for panerns of
-I. anc/or sign fcan ce of any oenaviour

1. Data reduction

‘D a ta r e d u c tio n refers to th e p ro c e ss o f se le c tin g , fo c u sin g , sim p lify in g , ab stra ctin g , and

tr a n s fo rm in g th e d a ta th a t a p p ear in w ritten up field n o tes o r tr a n sc r ip tio n s’ (M ile s and
H u b e r m a n 1 9 9 4 : 1 0). D a ta r e d u ctio n can b e ach iev e d by n o tin g red u n d a n c ies in th e d ata
a n d d isc a r d in g all b u t th e m o st in te re stin g an d c o m p e llin g sta tem e n ts c o n c e rn in g a p a r tic u ­
lar issu e o r th e m e . N ot o n ly d o th e d ata n eed to b e c o n d e n se d for th e sake o f m an ageab ility,
th ey a lso h av e to b e tr a n s fo rm e d so th e y ca n b e m a d e in tellig ib le in te rm s o f th e issu es b ein g
a d d ressed .
D a ta r e d u c tio n is so m e w h a t m o re s tra ig h tfo r w a rd fo r s tru c tu re d in terv iew s th an for
s e m i- s tr u c tu r e d o r u n s tr u c tu re d in te rv ie w s, sin c e m u ch less d ata are c o lle c te d . S tru ctu re d
in te r v ie w s g e n e ra lly a sk q u e s tio n s th at call fo r relativ ely sh o rt, sim p le, o r q u a n tifia b le
a n sw ers w h ic h th e in te r v ie w e r c a n ea sily reco rd . T h e m a in goal in th e an aly sis o f s t r u c ­
tu red in te r v ie w d a ta is to q u a n tify th e d ata an d th e n s u b je c t it to s ta tistic a l an aly sis so th at
h y p o th e se s c a n b e te s te d . T h e d ata fr o m u n s tr u c tu re d in te rv ie w s is n o t as easy to c o d e o r
q u a n tify a s d a ta o b ta in e d fr o m su rv ey s. G re a t c a re n ee d s to b e ta k e n in d ev elo p in g c o d e
fr a m e s to re c o rd th e d iffe re n t ty p es o f a n sw ers, an d a tte n tio n also n ee d s to b e paid tow ard s
h o w in te r v ie w e rs ’ q u e s tio n s o r p r o b e s v a ry a c ro ss d iffe re n t re sp o n d e n ts. S in c e th e re is
o fte n su b s ta n tia l v a ria tio n in h o w q u e s tio n s are ask ed , th e d ata ca n n o t b e tra c k e d as easily
o v er tim e , a n d th e re su lts o f d a ta a n a ly sis are n o t as easily c o m p a re d , sin c e in e ffect w hat is
o fte n b e in g tr a c k e d o r c o m p a re d is a n sw ers to d iffe re n t q u e stio n s. T h e m o re s tru c tu re d a
q u e s tio n th e n , th e e a s ie r it w ill b e to an aly se. D a ta fo r fo cu s g ro u p s are d ifficu lt to a n a ly se —
p a r tic u la rly i f th e a n a ly sis is c o n c e r n e d w ith b o th th e co n te n t o f w h at w as said an d th e
p a tte rn s o f in te r a c tio n . O p e n q u e s tio n s p e r m it free r e sp o n s e s, w h ic h are rec o rd ed o r w r it­
te n d o w n as fa r as p o ss ib le in th e r e s p o n d e n ts ’ o w n w ord s. T h is m ass o f d ata h as to b e
o r g a n iz e d a n d so m e h o w m e a n in g fu lly re d u ce d o r re co n fig u re d . B o x 11.3 giv es a sn ap sh o t
o f h o w th e a n a ly sis a n d re su lts o b ta in e d fr o m se m i-s tr u c tu re d an d u n str u c tu re d , and
str u c tu r e d in te r v ie w d a ta d iffer.
T h e first step in d a ta r e d u c tio n is to tr a n s c r ib e th e ta p e s o r n o te s o f th e in terv iew s. As p art
o f th is p r o c e ss, y o u sh o u ld d ev elo p a ‘p ro file’ o f ea ch in terv iew ee: a su m m a r y o f th e in te r ­
v ie w e e s b a c k g ro u n d , e x p e rie n c e , an d o p in io n s. W h e n e v e r p o ssib le, in terv iew ee’s o p in io n s
H O W T O D O R E S E A R C H IN P R A C T IC E

should appear in these profiles in their own words, to avoid the possibility that rewriting
them may change their substance or emphasis.
R eco rd in g and tra n sc rib in g in terview s resu lts in an un w ieldy am o u n t o f v erbal data.
O n e o f th e c e n tral tasks o f se m i-stru ctu re d and u n stru ctu red in terv iew data analysis is to
red uce th e a m o u n t o f data to a m o re m an ageab le level by id en tify in g and e x tra c tin g th e
m o st im p o rta n t, m ean in gfu l, and in terestin g parts o f th e in terv iew text. This sh ou ld b eg in
as a p rocess o f ‘d iscov erin g w hat’s in th e m aterial, rath er th an startin g ou t w ith defin ite
hyp otheses in m ind. W h en th e tran scrip ts have b een co m p leted , read th em , first, as a
w hole to no te your gen eral im p ressions. Read carefu lly th ro u g h each in terv iew w ithou t
takin g notes. W h en you fin ish read ing you m ight w ish to w rite dow n a few gen eral n o tes
or im pressions.
N ext, review the in terview transcripts, again, th is tim e look in g for very sp ecific th in g s—
for sim ilarities and differences and pattern s and th em atic co n n ectio n s in th e data. M ake
m arginal notes, h ighligh tin g key words and th em es. For focus groups, m ake a no te o f m ajor
op in ion s and attitudes that are expressed by the groups. U nd erlin e or circle th ose item s that
you th in k will prove to be the m ost im p ortant or m eanin gful. Th is is th e start o f p roducin g
a list o f term s that will be subsequently used for coding. C od es em erge from a process o f
refining the m arginal notes you m ake on transcripts.

2. Coding
C od ing m eans breaking down interview m aterial and assigning th em to different categories
accord ing to the variable to which they relate. In codin g in terview data you nam e and c a te ­
gorize p henom ena through close exam in ation o f data. C losed qu estions are pre-coded.
O pen qu estions will require a codin g fram e— a list o f categories and cod es for e ach qu estion
used to analyse responses. Eventually, you will perhaps want to reduce the nu m b er o f code
words and categories o f in form ation ; but as you begin this process you should not worry
about placing too many initial restrictions on yourself.
C onsid er w hether you are using two or m ore w ords or phrases to d escribe the sam e thing;
or w hether an item m ight be coded in m ore than one way. Som e responses m ay refer to m ore
than one issue or idea. These should be given m ore than one code word so that the response
can be included in each relevant area. C onsider, too, w hether your codes m ight be replaced
with concepts and categories in the literature relating to your research question and h yp oth­
esis. Look to see if certain codes tend to be associated with oth er codes, and w hether you can
perhaps code for these conn ections.
W h e n yo u c o d e th e t r a n s c r ip t s , y o u m a rk s e c t io n s o f th e t r a n s c r ip t in a w ay th a t i n d i ­
c a te s w h at th e in te rv ie w e e o r fo cu s g r o u p p a r tic ip a n t is ta lk in g a b o u t. F o r in s t a n c e , e v e r y
tim e a p a rtic ip a n t m e n tio n s a ‘n e w r e p o r t ’, th e re s e a r c h e r m a rk s ih e s e c t io n to in d ic a te
th is. U sin g c o d e w o rd s will m a k e th is fa s te r, e .g . N E W S R E P (i.e . N E W S R E P O R T ) . S o , in
th e e n d a tra n s c rip t will h a v e a list o f c o d e w o rd s ru n n in g d o w n th e s id e o f th e p ag e. T h is
m a k e s it e a s ie r to id e n tily s e c tio n s o f in te re s t la te r o n , as all th a t will b e n e e d e d w h en
lo o k in g at th e issu e ot X 1» Z is to lo o k d o w n th e t r a n s c r ip t s a n d ta k e all th e r e s p o n s e s
m a rk e d X Y Z .

Ihe in fo rm a tio n p rov id e d by op e n q u e stio n s n e ed s to b e c o d e d in o r d e r to m ak e s en s e o f

the a n sw ers. Ihis ca n be very tim e -c o n s u m in g . M an y re s p o n se s m a y be u n co d a b le ; a n sw ers
 IN T E R V I E W I N G A N D F O C U S G R O U P S

m ay be irrelev a n t o r o ft-to p ic , an d so m e grou p s o f p eop le m ay b e m o re likely to respond

th a n o th e rs, p a rtic u la rly th o se w ho are m o re ed u cated o r m o re in terested in th e top ic. E ach
c o h e re n t in terv iew se g m e n t sh o u ld b e giv en a co d e n u m b e r to in d ica te th e co n ce p t, c a te ­
g ory, th e m e , o r a rg u m e n t it relates to. T h ese seg m en ts can th en b e co p ied and separated
fro m th e tr a n sc r ip ts (w ith e ith e r real o r e le c tro n ic sciss o rs) and put b ack to g eth er th e m a ti­
cally to serv e as th e raw m a te ria l for an a n aly sis o f th e g en eral fin d in gs o f th e study and th eir
im p lic a tio n s for p o lic y -o r d e c isio n -m a k in g .
T h ro u g h o u t th is p ro c e ss it is im p o rta n t to reflect on w hat sig n ifica n c e th ese data have for
y ou r r esea rc h q u e stio n and h y p o th e sis. T h e re se a rc h e r m u st m ak e c h o ic e s ab ou t w hich
a sp e c ts o f th e d ata sh o u ld b e em p h a siz ed , m in im iz e d , o r set asid e com p letely. At all stages o f
th e re se a rc h p ro c e ss, th e r e se a rc h e r w ill b e c o n c e rn e d to p ro b e th e g en eralisab ility , relia b il­
ity (th e c o n s is te n c y o f fin d in g s/ resu lts), an d v alid ity ( i f th e study in fact in vestigates what
w as in te n d e d ).
T ra d itio n a lly , c o d in g in v o lv ed g a th e rin g to g e th e r th o se p o r tio n s o f a tex t b e lo n g in g to a
g iv en la b el o r n a m e . R e s e a r c h e r s used d iffe re n t c o lo u re d h ig h lig h te rs fo r ea ch co d e and
th en c u t a n d p a sted to g e th e r s im ila r ly c o lo u re d fra g m e n ts u sin g s c iss o rs and paste. Today,
th e re is c o m p u te r so ftw a re th a t p e r fo rm s th is task. C o d in g o f data can b e d o n e u sin g o n e o f
th e c o m p u te r -b a s e d a n a ly sis p ro g ra m p a ck a g es, e.g. N U D *IS T , N V iv o, o r A tla s.ti). H o w ­
ever, w h e th e r c o m p u te r so ftw a re o r h ig h lig h te r s are used , th e p ro cess re m a in s th e sam e. It
is th e r e s e a rc h e r w h o d e fin e s an d n a m e s th e c a te g o rie s o f d ata. T h e resu lt o f b o th co d in g
a n d a n a ly sis d e p e n d s ex c lu siv e ly u p o n th e r e s e a rc h e r ’s in te r p re ta tio n o f m e a n in g s h id d en
in th e d ata.
U sin g th e r esea rc h q u e stio n an d h y p o th e sis as a guide, ev ery lin e, paragraph , o r o th er s e c ­
tio n o f tex t is c o d e d fo r relev a n t th em es. A s th e m e s are dev elop ed , th e research er assig ns a
w o rk in g d e fin itio n to ea c h co d e. T h at way, in g oin g th ro u g h th e tra n scrip ts, th e d efin itio n is
c o n tin u a lly b e in g ch a llen g ed , an d s o m e tim e s n ew co d es m u st b e d evelop ed b ecau se th e p ro p ­
ertie s d o n o t fit th e tex t. A lso, c o d e s th at are rarely used are d ism issed and so m e c ateg o ries are
b ro a d e n e d to a c c o m m o d a te th e lo st co d e. Im p o r ta n t to n o te is th at th is type o f analysis is not
lin ear, b u t circ u la r. C o n sta n t c o m p a ris o n (see G la se r and Strau ss 19 6 7 ) m ean s th at the
r e se a rc h e r m u st c o n tin u a lly c o m p a re th e ca teg o ries an d c o d es o f n ew tra n sc rip ts w ith e x ist­
in g c a te g o rie s an d c o d e s in o rd e r to m o re fully d ev elop th e p ro p erties o f th e o v erarch in g c a t­
eg o rie s fo r th e in d iv id u a l c o d es. T h is p ro cess is o n g o in g u n til satu ratio n is reached : th at is, n o
n ew c o d e s o r ca te g o rie s e m erg e an d c o d in g m o re tra n sc rip ts w ould o n ly p ro d u ce a rep etitio n
o f th em es.
B e aw are th a t e x tra c tin g and co d in g frag m en ts will rem ov e th em from th e co n text w ithin
w h ich th ey ap p eared and d isru p t o r lose th e n arrativ e flow o f w hat w as said (B ry m an 2 0 0 4 :4 1 1 ).

3. Analysis
O n c e s e q u e n c e s o f te x t a re m a rk e d w ith c o d e s (c o d in g ), se q u e n c e s o f tex t m a rk ed w ith each
c o d e a re c o lle c te d to g e th e r (re tr ie v in g ). T h e an aly sis sh o u ld th en c o n n e c t up th e co d e s to
e a c h o th e r in o r d e r to b rin g in to fo cu s a w eb o f m e a n in g s.
D ra w in g c o n c lu s io n s in v o lv es step p in g b a c k to co n s id e r w hat th e a n aly sed data m e a n and
to a ssess th e ir im p lic a tio n s fo r th e q u e stio n s at h an d . V e rifica tio n , in teg rally lin k ed to c o n ­
c lu s io n d ra w in g , e n ta ils re v isitin g th e d ata as m a n y tim e s as n e c e ssa ry to c ro s s -c h e c k o r
k.* m H O W T O D O R E S E A R C H IN P R A C T IC E

I: In which direction should the EU develop itself in the future in your opinion, specifically concerning
the relationship between the central EU institutions, member states, and regions?

R: I am a kind of EU-minimalist. For the EU is about making life easier, a little more flexible, and to
realize the four freedoms that they have been dealing w ith for a long tim e now. For me the EU
should create a dynamic that makes an econom ic and [pause] yes, make sure so happens. The EU that
have moved in other directions as well, and in my opinion this has made things worse, made it more
difficult and at least at certain areas made it more troublesome. If I were to decide I would have led the
EU back to basics. M ore of what used to be, in particular the thoughts that were dominating at the end of
the 80s and beginning of the 90s. The chase in deepening the cooperation on other areas, I believe has
created more pain than gain for EU as an organization.

I: So if the trend is more political power to Brussels you are against this?

R: Absolutely. Absolutely. I mean that all experience shows that the nation state, also in Europe, has
come to stay.

i l l ~ r ~
The EU that have If I w ere to decide I The chase in
deepening all
moved in other would have led the
cooperation experience
directions as well, EU back to basics.
on other shows
and in m y opinion M ore of w hat used to
areas, I that the
this has made things be, in particular the
believe has nation
worse, made it more thoughts that were
created more state, also
difficult and at least dom in ating at the
pain than in Europe,
at certain areas made end o f the 80s and
gain for EU. has com e
jt m ore troublesom e beginning of the 90s.
to stay.

Right trajectory o f EU W rong trajectory o f EU Nation state

Figu re 11.1 A n exa m p le o f data reduction

Source: Folkestad 2008:6.

v e r ify th e se e m e rg e n t c o n c lu s io n s , to v a lid a te fin d in g s b y c r o s s -c h e c k in g w ith o th e r q u e s ­

tio n s a n d in fo r m a tio n fr o m o th e r r e s p o n d e n ts . ‘T h e m e a n in g s e m e r g in g fr o m th e d a ta h av e
to b e tes ted fo r th e ir p lau sib ility , th e ir stu rd in e s s , th e ir “c o n fir m a b ility ”— th a t is, th e ir v a lid ­
ity ’ (M ile s a n d H u b e r m a n 1 9 9 4 : 1 1 ) . V a lid ity m e a n s s o m e th in g d iffe r e n t in th is c o n t e x t th a n
in q u a n tita tiv e e v a lu a tio n , w h e re it is a te c h n ic a l te r m th a t r e fe rs q u ite s p e c ific a lly to w h e th e r
a g iv en c o n s tr u c t m e a s u r e s w h a t it p u r p o r ts to m e a s u r e . H e r e v a lid ity e n c o m p a s s e s a m u c h
b ro a d e r c o n c e r n fo r w h e th e r th e c o n c lu s io n s b e in g d ra w n fr o m th e d a ta a re c re d ib le , d e f e n ­
sib le , w a rra n te d , a n d a b le to w ith s ta n d a lte rn a tiv e e x p la n a tio n s .
F o lk e sta d ( 2 0 0 8 ) is in te r e s te d in e x p lo r in g E u r o s c e p t ic is m a m o n g v a rio u s p o litic a l p a r tie s
in N o rd ic c o u n tr ie s a n d C e n t r a l E a s te r n E u r o p e . H e d e c id e d th a t c o n d u c tin g q u a lita tiv e
in te r v ie w s w ith elite s in p o litic a l p a r tie s m ig h t p ro v id e h im w ith in s ig h t in to th is p h e n o m ­
e n o n . T o e x p lo r e th e c o n t e n t o f E u r o s c e p tic is m a m o n g th e r e s p o n d e n ts th a t h e in te r v ie w s,
h e fo r m u la te s c o n c e p tu a l c a te g o r ie s c o n c e r n i n g th e tr a je c to r y o f th e E U , d ra w n fr o m th e
ca te g o rie s th a t K o p e c k y a n d M u d d e ( 2 0 0 2 ) u se d to c o n c e p tu a liz e s u p p o rt fo r a n d o p p o s i­
tio n lo E u r o p e a n in te g r a tio n .
 IN T E R V I E W I N G A N D F O C U S G R O U P S

F o lk e sta d re d u c e d th e m a te ria l fro m th e in te rv ie w in to th re e ca te g o rie s : ‘right and

w ro n g tr a je c to r y o f th e E U , a n d a m o re o n to lo g ic a l categ o ry , “N a tio n -s ta te ”. W h e r e a s th e
tw o first c a te g o rie s sh o w th e a n sw er th e re sp o n d e n t is g iv in g to th e q u e stio n (rig h t and
w ro n g d ir e c tio n s o f th e E U ), th e last c a te g o ry tells us so m e th in g ab o u t w hat k in d o f p o lit­
ica l o r g a n iz a tio n / p o lity h e v iew s as th e m o st p r o p e r ’ (F o lk e sta d 2 0 0 8 : 7 ). T h e d ata re d u c ­
tio n p r o c e ss is sh o w n in F ig u re 11.1.

Conclusions
As w ith a ny m etho d o f data co llectio n, the use o f in terview s o r focus groups should be appropriate
both to the research questio n and to the hypothesis yo u are investigating. You cho o se interview ing as
a m eans o f data co llectio n because you are co nfiden t that the data that you co llect through this
m ean s w ill p ro vid e a c o n vin cin g test o f y o u r hypothesis, and w ill enable you to draw logical c o n c lu ­
sions ab o ut y o u r hypo thesis. Q u estio ns should be designed so that the responses a ctually m easure
w h a t they cla im to.
As w e discussed in C h a p ter 7, it is a good idea to use m ultiple sources o f data and m ethods o f data
co llectio n w h e n e ve r po ssible. D o ing this enab les you to ap proach a research problem from different
angles. This is called 'triangulation'. T riangulatio n o f eviden ce in creases the reliability o f the data and
the process o f gathering it. In the co ntext o f data co llectio n, triangulatio n serves to co rroborate the
data gathered fro m o th e r so urces: the use o f different data sources can enable researchers to
cro ss-ch e ck findings. Triangulatio n y ields m o re co m p lete data and results in m ore credible fin dings;
and it also e nab le s researchers to find agreem ent betw een different perspectives. It m ight involve the
use o f differe nt research m etho ds to study a single research problem : a com parison of the results of
these d iffere n t m etho ds can e n able a researcher to identify w he the r differences are due to biases in
o ne o r a n o th e r o f these m etho ds. It m ight also in vo lve the use of d ifferent researchers to study the
sam e research p ro blem w ith the sam e peo ple and w ith the sam e m ethods: if the sam e results are
disco vere d , the fin ding s and in terp retatio n s have stronger va lid ity because o f this corro bo ratio n.

Questions
• W h a t are the advantages and disadvantages o f sem i-structu red in terview s co m pared to structured
in te rvie w s and surveys?

• So m e researchers are ve ry critical o f the w idespread use o f structured interview s. W hy?

• W h y m ight a re sea rch e r p refer to use a structured rather than an unstructured in te rvie w to gather
d ata?
• W h a t are the d ifferent types o f questions that m ight be part o f a q uestio nnaire and w hat sorts of
in fo rm a tio n do th e y en a b le yo u to o b tain?

• In w h a t w a ys does the o rd e r in w h ic h yo u ask q uestio ns affect the responses you get?

• W h a t co n sid era tio n s w o u ld guide y o u r ch o ice o f w h a t type o f in te rvie w to co nduct?

• H o w can in te rv ie w data be used to co n firm o r d isco n firm a hypothesis o r argum ent?

© Guide to Further Reading

A b erb a ch .J.J. Chesney, and B. Rockman (1975), Exploring Elite Political Attitudes: Some
Methodological Lessons', Political M ethodology 2 (1975): 1 -27.
Uses data fro m in te rv ie w s w ith A m erica n p o litical elites to illustrate open-ended interview ing .
D iscu ssio n o n co d in g te chniq ue s fo r in te rv ie w data analysis.
284 H O W T O D O R E S E A R C H IN P R A C T IC E

Denzin, N. K. and Y. S. Lincoln (eds) (1994), Handbook of Qualitative Research (Thousand Oaks, CA:
Sage Publications).
This volum e consists of 36 chapters on qualitative m ethods in social research. It covers historical and
philosophical perspectives, as well as detailed research methods. Extensive coverage is given to data
collection and data analysis, and to the 'art of interpretation' o f findings obtained through qualitative
research

Gubrium, J. F. and J. A. Holstein (eds) (2002), Handbook of Interview Research: Context and Method
(Thousand Oaks, CA: Sage Publications).
A com prehensive handbook covering virtually all forms of interviewing.

Hammer, D. and A. Wildavsky (1993), Th e Open-Ended, Semistructured Interview: An (Almost)

Operational Guide', in A. Wildavsky (ed.), Craftways: On the Organization of Scholarly Work
(New Brunswick, NJ: Transaction Publishers), 57-101.
Examines the procedures associated w ith sem i-structured interviewing and provides useful discussion
about the relationship between the researcher and the respondent.

Krueger, R. A. (1998), Moderating Focus Groups (Thousand Oaks, CA: Sage Publications).

Kvale, S. (1996), Interviews: An Introduction to Qualitative Research Interviewing (Thousand Oaks,

CA: Sage Publications).
Covers all aspects of the interview process, including the theoretical and philosophical foundations of
interviewing, and the influence those foundations have on the process, the content, and the analysis
of qualitative interviews.

MacNaughton, P. and M. Jacobs (1997), 'Public Identification with Sustainable Development

Investigating Cultural Barriers to Participation', Global Environmental Change 7:5-26.

McLellan, Eleanor, Kathleen M. MacQueen, and Judith L Neidig (2003), 'Beyond the Qualitative
Interview: Data Preparation and Transcription', Field Methods 15 (February): 63-84.
Outlines the consequences of inappropriate or inadequate preparation of transcripts from recordings
and offers practical considerations that can help researchers systematically organize and analyse
textual data.

M organ, D. (1997), Focus G roups as Q ualitative Research, 2nd edition (N e w b ury Park, CA: Sage
Publications).
Compares participant observation and individual interviews: strengths and weaknesses; uses of focus
groups.

O 'Connor, H„ C. Madge, R. Shaw, and J. W ellens (2008), 'Internet-based Interview ing', in

N. Fielding, R. M. Lee, and G. Blank (eds), The SAG E H andbook o f Online Research M ethods
(London: Routledge), 271 -89.

S lap in.J. and S.-O. Proksch (2008), A Scaling M odel fo r Estim ating T im e Series Policy Positions
from Texts', A m erican Jo urn al o f P olitical Science 52(8): 705-22.

Sudm an, S., and N. Bradb urn (1983), A sking Questions: A Practical Guide to Questionnaires (San
Francisco, CA: Jossey-Bass).

'Sym posium : Interview Methods in Political Science', PS: Political Science an d Politics 35(4)
(D ecem ber 2002): 663-88. Beth L. Leech, 'Asking Questions: Techniques for Sem istructured
Interviews' (pp. 665-8); Kenneth G oldstein, 'Getting in the Door: Sam pling and Com pleting
Elite Interviews' (pp. 669-72); Joel Aberbach and Bert Rockm an, ‘Conducting and Coding Elite
Interview s (pp. 673-6); Laura W oliver, 'Ethical Dilem m as in Personal Interview ing' (pp. 677-8);
Jeffrey M. Berry. Validity and R eliability Issues in Elite In terview ing' (pp. 679-82); Sharon
W erning Rivera. In terview ing Political Elites: Lessons from Russia' (pp. 683-8).
 IN T E R V IE W IN G A N D FO C U S G R O U P S

Weiss, R. (1994), Learning from Strangers: The Art and Method o f Qualitative Interview Studies
(N ew York: Free Press, 1994).
Offers step -b y-step in trod uctio n to m ethod of qualitative interview ing: sam ple selection,
de ve lo p m e nt of an in te rvie w guide, the co nduct of the in terview , data analysis, and p reparation of the
data. In cludes e xam ples of successful and less successful in terview s.

Software
Friese, S. (2004), Software Overview. Available online as a PDF at: http://www.quarc.de/software
overview_table.pdf.
A useful table co m p arin g six so ftw are packages.

Matheson, J. L (2007), T h e Voice Transcription Technique: Use of Voice Recognition Software to

Transcribe Digital Interview Data in Qualitative Research', The Qualitative Report 12(4): 547-60.
Retrieved from http://www.nova.edu/ssss/QR/QR12-4/matheson.pdf.

Rettie, R. (2005, W inter), Exploiting Freely Available Software for Social Research', Social Research
Update, 48. Retrieved 4 September 2006, from http://sru.soc.surrey.ac.uk/SRU48.html.

References
A berbach.J. D. and B. A. Rockman (2002), Glaser, Barney G. and Anselm Strauss (1967), The
'Conducting and Coding Elite Interviews',PS: Discovery o f Grounded Theory (Chicago: Aldine).
Political Science and Politics 35(4) (December): Hooghe, L., R. Bakker, A. Brigevich, C de Vries, E
673-6. Edwards, G. Marks, J. Rovny. and M. Steenbergen
Benoit. Kenneth and M ichael Laver (2006), (2009), Reliability and Validity of the 2002
Party Policy in M odern Democracies (London: and 2006 Chapel Hill Expert Surveys on Party
Routledge). Positioning', http://web.m e.com/rovny/Site/

Bishop, George F„ Robert W. Oldendick, and Academic_files/HoogheEtAI EJPR ju ly2009.pdf

Alfred J. Tuchfarber(1984), 'Interest in Political Kopecky, P. and C. Mudde (2002), T h e Two Sides
Cam paigns: The Influence of Question Order and of Euroscepticism: Party Position on European
Electoral Context', Political Behavior, 6(2): 159-69. Integration in East Central Europe', European

Brym an, A. (2004), Social Research Methods. 2nd Union Politics 3: 297-326.

edition (Oxford: Oxford University Press). Miles, M. B. and A. M. Huberman (1994), Qualitative
Data Analysis, 2nd edition (Newbury Park, CA:
Castles, F. and P. M air (1984), Left-Right Political
Sage Publications).
Scales: Some Expert Judgements', European
Journal o f Political Research 12: 73-88. Moser, C. A. and G. K. Kalton (1971), Survey
Methods in Social Investigation (London:
de Vaus, David (2002), Surveys in Social Research
Heinemann).
(London: Routledge).
Pierce, R (2008). Research Methods in Politics: A
Dorussen, H., H. Lenz, and S. Blavoukos (2005),
Practical Guide (London: Sage Publications).
'Assessing the Reliability and Validity of Expert
Interviews'. European Union Politics 6(3): 315-37. Rivera, S. W „ P M. Kozyreva, and E. G. Sarovskii
Folkestad, Bjarte (2008), Analysing Interview (2002), Interviewing Political Elites: Lessons from
Data: Possibilities and Challenges', Eurosphere Russia', PS: Political Science and Politics 35(4)
Online W orking Paper Series, W orking Paper (Decem ber): 683-8
No. 13 (Decem ber); at http://eurospheres.org/ Tansey, O. (2007), 'Process Tracing and Elite
files/2010/08/Eurosphere_Working_ Paper 13
Interviewing: A Case for Non-probability
Folkestad.pdf. Sampling', PS. Political Science and Politics 40(4)
George, A. L. and A. Bennett (2005), Case Studies (October). 765-72.
and Theory Development in the Social Sciences
(Cam bridge, MA: M IT Press).
H O W T O D O R E S E A R C H IN P R A C T IC E

Endnote
1. There are various types of computer-assisted interviewing software. Computer-assisted telephone
interviewing (CATI) is usually employed in conjunction with random-digit dialling (RDD) as a means of
approximating random sampling. Computer-assisted personal interviewing (CAPI) allows the interviewer
to enter responses directly into an uploadable database using a hand-held computer. Audio-CASI uses
voice technology to prompt the respondent with items and receive responses by voice.
 Ethnography and
Participant Observation

Chapter Summary
This chapter examines the principles of ethnography and participant observation: what
they are. how (if) they became standardized as a research method, what form of evi­
dence they constitute, and what place they occupy in the study of Politics. We look at the
strengths of ethnographic fieldwork, and we try to identify what kind of material it pro­
duces and what aspects of social life it can reveal. W e shall also discuss its weaknesses,
especially issues of subjectivity, reliability, and generalizability. Topics discussed are:
• ethnography;

• participant observation;

• sampling;

• access;

• key inform ants;

• recording observations.

Introduction
A lthough particip an t ob serv ation was first developed by social an thropologists and so c io lo ­
gists, in recen t years it has b eco m e a m u ch m o re w idely used research tool across the social
scien ces, and atten tion to political eth nograp hies in particu lar has grown (see Auyero 2006
and Sch atz 2 0 0 9 ). Political eth n ograp h ies have been carried out in a w ide variety o f contexts,
from th e study o f political in stitu tio ns and organizations, such as p olitical parties and par­
liam en tarian s (F en n o 1978; Searing 1994), the ju d iciary (L atou r 2 0 1 0 ), local elites (D ahl
1 9 6 1 ), in tern a tion a l o rganizations (W eaver 2 0 0 8 ), and N G O s, to the study o f social m o ve­
m ents (B lee and C u rrier 2 0 0 6 ) and in form al netw orks, such as terrorist groups, the m afia,
drugs cartels, and b ettin g syn dicates (P arnell and K ane (eds) 2 0 0 3 ). Political eth nography is
also b ec o m in g m o re widely used in th e study o f In tern ation al R elations (IR ) (see Schatz
2 0 0 9 : 3 0 8 ). G illespie led a team o f eth n ograp h ers studying how British citizens, and British
M u slim s in particular, ex p erien ced secu rities and in secu rities d uring the W ar o n T error (see
G illespie, G ow and H oskin s 2 0 0 7 ; G illespie and O ’L ough lin 2 0 0 9 ). Political eth nographies
have a lso b een ca rried out by B arn ett (2 0 0 6 ) on th e U S m ission to th e U N ; by Pouliot (2 0 0 7 )
on d ip lo m a cy and secu rity; and H o p f (2 0 0 2 ) on Russian foreign policy.
P olitical eth n og ra p h ies have also b een w idely used in com parativ e research and have
ex a m in ed th e p olitical attitu des and behav iou r o f ord in ary p eople living in different parts o f
th e w orld, for exam ple in India (M ich elu tti 2 0 0 8 ), Senegal (Sch affer 1998), U ganda (K arl-
strom 1 9 9 6 ), and C h ile (Paley 2 0 0 1 ). P olitical eth n ograp h ies o f this type have shed light on
H O W T O D O R E S E A R C H IN P R A C T IC E

how m argin alized social groups in teract w ith m o d ern izatio n , g lob alization , and d em ocracy,
and how gov ernm ental p olicies and d evelopm ent p rogram m es c o m e to be reinterpreted ,
and accepted (o r no t) on the ground.
True, m any o f th ese issues can be, and frequen tly are, studied using o th er m eth od s, b ut at
the sam e tim e th ere is no substitu te for g etting out o f th e arm chair, getting y our han ds dirty,
and ob serv ing first hand what it is that you are w ritin g about in its natural settin g. Indeed,
one o f the key stren gths o f particip an t ob serv ation is that it provides th e research er with
first-h and ex p erien ce o f the su b ject they are w ritin g about. As R ich ard F en n o (1 9 9 0 : 56)
wrote, as long as p olitical scien tists con tin u e to study politician s, som e o f us certain ly will
want to collect data through repeated in teraction w ith th ese politician s in th eir natural h a b ­
itats’. This chapter exam in es the principles o f the eth nograp hic m ethod , its application to
political research, and the issues that co n fro n t research ers w ho use th is approach.
Ethnography is often d escribed as an approach rather than a specific m ethod. Indeed,
eth nographers can and frequently do em ploy a variety o f m ethod s, from p a rtic ip a n t o b se r ­
v atio n, to archival analysis, interview s, and surveys. The term eth nograp hy’ com es from
Latin and literally m eans w riting about people, w here ethnos m eans people or folk and
graphia m eans writing. The term th erefore en com passes and recognizes two im portant
c om ponen ts o f the approach. The first is to do with the study o f people, and how data are
collected , and the second refers to how this data is th en recorded and analysed (or w ritten
up). In this chapter we focus on both these com ponen ts. First, we exam in e the m ethod o f
participant observation , w hich is the m ethod o f data collection m ost closely associated with
the eth nographic approach. Secon d , we exam in e how the observ ation s and data from the
fieldwork are recorded and reported.

The principles of ethnography and participant observation

E th n o g ra p h y a n d p a rtic ip a n t o b s e r v a t i o n c a n b e d e fin e d in a n u m b e r o f w ay s, a n d th e se
d e fin itio n s fre q u e n tly o v e rla p w ith e a c h o th e r , a n d th e tw o t e rm s a re fre q u e n tly u se d i n te r ­
ch a n g e a b ly . A c c o r d in g to D e la m o n t ( 2 0 0 4 : 2 1 8 ) ‘p a r t ic ip a n t o b s e r v a t i o n , e t h n o g r a p h y
a n d fie ld w ork a re all u se d i n te r c h a n g e a b l y ...t h e y c a n all m e a n s p e n d in g lo n g p e rio d s
w a tc h in g p e o p le , c o u p le d w ith ta lk in g to th e m a b o u t w h a t th e y a re d o in g , th in k in g an d
sa y in g , d e sig n e d to se e h o w th e y u n d e rs ta n d th e ir w o rld ’. W ith re fe re n c e s p e c ific a lly to
e th n o g ra p h y , B re w e r ( 2 0 0 0 : 6 ) w rite s th a t: ‘e t h n o g r a p h y is th e s tu d y o f p e o p le in n a tu r a lly
o c c u r r in g s e ttin g s o r “ field” by m e th o d s o f d a ta c o lle c tio n w h ic h c a p t u r e th e ir s o c ia l
m e a n in g s a n d o r d i n a r y a c tiv itie s , in v o lv in g th e r e s e a r c h e r p a r t ic ip a t i n g d ir e c tly in th e s e t ­
t in g , if n o t a lso th e a c tiv itie s , in o r d e r to c o lle c t d a ta in a s y s te m a tic m a n n e r ’. L ik ew ise
H a m m e rs le y an d A tk in s o n ( 2 0 0 7 : 3 ) d e s c rib e th e e t h n o g r a p h i c a p p ro a c h in te r m s o f th e:
e th n o g r a p h e r p a rtic ip a tin g , o v e rtly o r c o v e r tly in p e o p le ’s d a ily lives fo r an e x te n d e d
p e rio d of tim e , w a tc h in g w hat h a p p e n s , lis te n in g to w h at is s a id , a s k in g q u e s t i o n s — in fa c t,
co lle c tin g w h a te v e r d a ta are a v a ila b le to th ro w lig h t on th e issu es th at a re th e fo cu s o f
re s e a rch . B o th th e se d e fin itio n s of e t h n o g ra p h y in clu d e r e fe re n c e to m u ltip le fo rm s o f d ata
c o lle c tio n , l o s o m e e x te n t th en e th n o g ra p h y re fe rs to a m u l ti - m e th o d a p p ro a c h . It e n c o m ­
p asses a p lu ra lity ol m e th o d s , all o f w h ich a re u sed in c o n ju n c tio n w ith e a c h o t h e r to
g e n e ra te a d e e p , first-h a n d u n d e rs ta n d in g o f w hat p e o p le d o in th e ir s o c ia l e n v iro n m e n t.
 E T H N O G R A P H Y A N D P A R T IC IP A N T O B S ER V AT IO N

The m ain m ethod that is em ployed to do this is participan t observation . In order to

observe what people do in p ractice, participan t observ ation involves ‘research based on the
close-u p, o n -th e-g ro u n d ob serv ation o f people and in stitutio ns in real tim e and space, in
which the investigator em bed s herself near (or w ithin) the phenom enon so as to detect how
and why agents on the scen e act, th in k and feel the way they do’ (W acqu ant, 2003: 5). The
d istinctiv e feature o f participan t o b serv ation , and one o f the great strengths o f the approach,
is that data c o lle ctio n is carried out in real tim e. This m eans that the eth nographer has a
direct, first-h and op p ortu n ity to ob serv e what people actually do, what they actually say to
each other, and how they actu ally in teract with different in stitutio ns or political processes,
rather than ju st relying on w hat people say that they do, which is not always the sam e as
what they actu ally do (as we d iscu ssed with referen ce to survey m easurem en t error in
C hapter 10).
P articip ant ob serv atio n (and eth nography m ore generally) th erefore has a n u m ber o f
c h a ra cteristics that overlap with oth er m eth od s we have considered in th is b ook (such as
surveys, focu s groups, and in terview s) and a nu m b er o f c h aracteristics that are distinctive,
p articu larly with resp ect to th e role o f o b serv ation . W h ereas surveys (see C hapter 10) are
b ased on th e a n cien t art o f askin g qu estio ns to find out what people th ink, say, and do,
p articip an t o b serv a tio n is b ased on so m eth in g rath er d ifferent. It recognizes that what p eo ­
ple say th ey do, and w hat th ey actu ally do, can be, and frequen tly are, different. A ccord ­
ingly, to get a ‘tru e sen se o f w hat people th in k and say and do, it is not enough to m erely ask
people q u estio n s and record th eir answ ers; it is also n ecessary to ob serv e what people do in
p ractice.
P articipant o b serv a tion , as th e term im plies, th erefore involves the researcher living, and
in teracting , and p articip atin g in th e daily life o f the people (often referred to as in fo rm an ts)
who she or he is studying. Th is can involve long periods o f research, o f m any m onths, if not
years, in th e field (th e research settin g). This prolonged period o f tim e spent in the field
allow s th e research er to em bed them selves in th e settin g which they are studying and really
get to know th eir in form an ts, un derstand th eir behaviour, and get detailed in form ation
about w hat th ey th in k and say and do. As Ingold (2 0 0 8 ) puts it, th is involves the researcher
g etting out o f th e a rm chair, and actu ally in teractin g with the people w hom he or she is w rit­
ing about.
Th is research on th e ‘everyday p ractices’ o f hu m an behav iou r has a nu m b er o f key
stren gths. It is p articu larly useful for studying sensitive topics, w here people may not d irectly
reveal th eir real th ou gh ts or o p in ion s to a stranger. O n e such sensitive top ic is w itchcraft,
w hich, although an im p o rtan t part o f social life in m any parts o f th e world, is generally not
som eth in g th at people feel co m fo rtab le talking about to strangers. Adam A sh forths (2 0 0 5 )
bo o k Witchcraft, Violence, and Democracy in South Africa draws on th ree years o f fieldwork
in Sow eto (So u th W est T ow n ship ), an A p artheid -bu ilt b lack suburb o f Johannesbu rg. In
ord er to do his fieldw ork, A sh forth learnt Zulu, Sotho, X h o sa, and the special brands o f
E nglish Sow etans speak; he a lso learned to play th e violin Zulu style, and was ready to defend
his ad opted b ro th ers and sisters from recu rren t th reats o f attack. Th rough first-h and ob ser­
v ation, person al in terv e n tio n , and lon g discu ssion s with in form an ts A sh forth was able to
build up a d etailed d escrip tion o f life un der th e shadow o f ex tre m e violen ce. This helped him
to arriv e at tw o rev ealing co n clu sio n s, first, that no on e can m ake sen se o f local Sou th A fri­
can p o litics w ithout u n d erstan d in g th e en o rm o u s part played by fears a bout, accu sations of,
H O W T O D O R E S E A R C H IN P R A C T IC E

and reaction s to w itchcraft in Sow eto’s (and , by ex ten sio n , Sou th A frica’s) everyday p olitics;
secon d , that n o o n e can ho pe to deal with Sou th A frica’s d ev astating A ID S e p id em ic o r build
local-lev el d em o cracy w ithout co n fro n tin g w itchcraft d irectly (see T illey 2 0 0 6 for an
extend ed d iscu ssion o f th ese findings).
Participant observation is also well suited to u npicking difficu lt-to -d efine or m u ltifaceted
political p henom ena, where oth er research in strum en ts such as surveys, in terview s, o r focus
groups may provide too blunt an in strum en t to fully capture the diversity and m eanin g o f a
concept. For exam ple, p articipant observ ation is able to shed light on and capture how Indians
(M ich elutti 2 0 0 8 ), Senegalese (Sch affer 1998), U gandans (K arlstrom 1996), o r m em b ers o f
C hilean social m ovem ents (Paley.2001) understand w hat d em ocracy m eans.
But perhaps the core stren gth o f th e m ethod is to do with the act o f o b serv ation . P a rtici­
pant observ ation allows the research er to o b serv e w hat people do on a d ay-by-day basis, and
how th eir behaviou r changes in response to different stim uli o r different events. F or e x a m ­
ple, using eth nographic, in -d ep th in terview s, and d o cu m en t data on new and em ergin g
social m ovem ent groups (SM G s) in Pittsburgh for 20 m o nth s before and after th e 2 0 0 4 US
presidential election, Blee and C u rrier (2 0 0 6 ) ex am in e how SM G m em b ers th in k ab ou t e lec­
tions and how groups decide to respond to national electoral cam paigns. Particip ant o b ser­
vation also allows the research er to observ e first hand how and why unplanned or
spontaneous events em erge— such as a riot or a fight, a strike that develops in a pitched b a t­
tle, or a sim ple quarrel that escalates and gets out o f hand. Th is kind o f research is often
d escribed as process tra cin g (see C hapters 4 and 9). It involves look ing for ev iden ce o f the
pressures, incentives, m otivations, and decision -m ak in g calculus in any given in stance o f
a ction (G eorg e and B enn ett 2 005; Parsons 20 1 0 ). M oreover, by ob serv ing people over a
prolonged period o f tim e, it is possible to build up a m uch m ore reliable im pression o f what
they really th ink than would be possible after ju st a b rie f m eetin g or interview.

Theory and hypothesis testing

A k ey p u rp o s e o f p a rtic ip a n t o b s e rv a tio n is s im p ly to d e s c rib e w h a t p e o p le t h in k , say, an d
d o. By s p e n d in g a lo n g tim e in th e field, re p e a te d ly talk in g to in fo rm a n ts o v e r tim e an d in
d ifferen t so c ia l se ttin g s, a n d o b s e rv in g w h at p e o p le d o , th e m e t h o d is ab le to p ro v id e a t h ic k
d e s c r i p ti o n o f th e s o c ia l a n d p olitica l lives o f th e in fo rm a n ts . In d o in g so , it a im s to p ro v id e
a valid a n d reliab le re p re s e n ta tio n o f h o w th e s u b je cts u n d e r stu d y b e h a v e an d th in k p o liti­
cally. In s o m e s e n s e , th e n , th e m e th o d is c o n c e r n e d w ith u n d e rs ta n d in g , w ith m e a n in g , an d
with in te rp re ta tio n . T h is s o rt o f d a ta is v a lu a b le in its o w n rig h t. C o lo n ia l e th n o g ra p h ie s ’ o f
d ifferen t c o m m u n itie s in In d ia p ro v id e a rich a n d d e ta ile d d e s c rip tio n o f s o c ia l life in th e
n in e te e n th c e n tu ry , w h ich w ou ld o th e rw is e h a v e b e e n lo s t. E t h n o g ra p h ie s c o n s titu te a rich
s o u rc e of h isto rica l d a ta , a n d in a d d itio n c a n giv e v o ice to p e o p le w h o o th e rw is e w ou ld n o t
be h e a rd , an d sh in e a light in to c o r n e r s o f th e p o litica l w o rld th a t w ou ld o t h e rw is e g o u n d e ­
te cte d .

But at th e s a m e lim e , th is e th n o g ra p h ic d e s c rip tio n a n d fo cu s o n rich te x tu a l d etail is

often th o u g h t to c re a te a tra d e -o ff in te rm s of h o w ta r fin d in g s c a n b e g e n e ra liz e d to o th e r
c o n te x ts an d to shed light on th e o ry m o re gen erally. E th n o g ra p h y a n d p a rtic ip a n t o b s e r v a ­
tio n have th e re lo re often b een d e rid e d as on ly c a p a b le o f p r o d u c in g ‘th ic k d e s c rip tio n ’.
A c co rd in g to 1 la m m e rs le y an d A tk in son ( 2 0 0 7 ) , e th n o g ra p h ic re s e a rch c o m p ris e s a s tro n g
 E T H N O G R A P H Y A N D P A R T IC IP A N T O B S ER V AT IO N

em phasis on ex ploring the nature o f social p henom ena, rather than setting out to test
hypotheses about them . Sin ce ethnography tends to focus on the specific rather than the
general, it is th erefore often regarded as an inappropriate m ethod to use to test general
hypotheses.
W h eth er or not eth nography can be used to test th eory is a debate which has rum bled on
for m any years. Indeed, as far back as the 1960s, Edm und Leach com plained about the te n ­
dency o f eth nog rap hers to focus on the specific rather than the general and lam ented, most
o f my colleagues are giving up in the attem pt to m ake com parative generalisations; instead
they have begun to write im peccably detailed historical eth nographies o f particular peoples
(L each 1961: 1). To be sure, eth nography can be used in a purely descriptive, inductive way
(and often is), but equally it can be, and som etim es is, also used m ore deductively.
As we d iscu ss in C hapter 2, it is easy to overstate the differences betw een inductive and
deductive approaches. But, in general, eth nographic research always begins with som e p rob­
lem or set o f issues. At the very least, it starts from what M alinow ski (1 9 2 2 ) referred to as
foreshad ow ed p ro b lem s. S o m etim es this is based on w ell-developed theory, som etim es it
will be m ore exploratory, and som etim es it will lead to the form ulation o f specific hyp othe­
ses that are to be tested, or at least in vestigated.
Participant ob serv a tion can be used for each o f these purposes. That it is frequently not
says m o re about the people w ho use the m ethod than it does about the m ethod itself. Thus,
participan t o b serv ation is co n ce rn e d with producin g d escriptions and explanations o f spe­
cific political p h en om en a , developing th eories w hich m ay apply m ore generally, and apply­
ing th eory to d ifferent social settings. It is a useful tool for generating th eory in areas o f
research for w hich little is know n, or w here previous th eory sheds little light. It is also a u se­
ful tool for applying theory, w hich is perhaps one o f its m ost frequent uses. Ethnographers
draw on insights from previous research and attem pt to apply them to new contexts. E th n o ­
graphic stud ies frequen tly take w ell-developed ideas (o r th eories) about hierarchy or power
or division o f la b ou r or social stratification , to nam e but a few, and see w hether these ideas
or th eories apply in different contexts. If the th eory is found to apply, and to provide a good
fram ew ork for analysis, th en the study in som e sense corrob orates o r builds upon and d evel­
ops the ex istin g theory. But if the th eory d oes not provide a conv in cing fram ew ork, then the
study can be used to con sid er and reflect upon why it does not, and perhaps to propose som e
lim ita tion s to th e orig in al theory.
For exam ple, E liasop h s (1 9 9 7 ) eth nography o f d ifferent public and private spaces in a
tow n on the U S Pacific coast is a fantastic ex planation o f how political apathy is produced by
contextu al factors, and built up slow ly th rou gh different phases and contexts o f peoples
lives. She tests N o elle -N eu m a n s ‘spiral o f silence theory, an explanation for why individuals
feel un able to express th eir political views publicly, by spen ding years in this town getting to
k now groups and ind ivid uals, b eing a participan t ob serv er in social clubs, cake bakes, and
tow n hall m eetin g s, as well as talking to people in private.
The p rocess o f applying th eory m ay over tim e, with repetition, lead (in d irectly) to the
testing o f th eo ries, in that som e th eories m ay be found not to apply so m any tim es that they
lose th eir relev ance and can essentially be falsified, while oth er th eories are found to apply to
m any d ifferent co n tex ts and so are given m ore em pirical support. But the key point is that
th e testin g o f th eo ry co m es from th e gradual accu m u lation o f know ledge, rather than from
any p a rticu la r o n e -o ff ethnography.
2 92 H O W TO D O R E S E A R C H IN P R A C T IC E

Participant observation methodology

T h e m a in r e s e a rc h i n s tru m e n t to r d a ta c o l le c ti o n in p a r tic ip a n t o b s e r v a t i o n is th e in d iv id u a l
r e s e a r c h e r w h o a c tu a lly u n d e rta k e s th e e x e rc is e . 'T here is th u s an in h e r e n tly p e r s o n a l c o m ­
p o n e n t to th e r e s e a rc h p ro c e s s . B e lte r re s e a r c h e r s m a k e lo r b e t te r d a ta c o l le c to r s , a n d b e tte r
d a ta a n a ly s e rs. A s B e rn a rd ( 2 0 0 6 : 3 4 4 ) o b s e rv e s , p a rtic ip a n t o b s e r v a t i o n is a c r a f t, a n d as
w ith all c ra fts , b e c o m i n g a skilled a rtis a n at p a r t ic ip a n t o b s e r v a t i o n ta k e s p r a c ti c e . It a lso
ta k e s ta le n t a n d sp e c ific p e rs o n a l skills. P a rtic ip a n t o b s e r v a t i o n in v o lv e s g e t t in g i n f o r m a n t s
to o p e n u p, b e in g a llo w e d in to th e ir liv es, a n d h a v in g th e o p p o r tu n i t y to o b s e r v e a n d r e c o r d
p e rs o n a l a n d p riv a te e v e n ts first h a n d . T h e e x te n t to w h ic h th e e t h n o g r a p h e r is ab le to d o
th is th e re fo re d e p e n d s in la rg e p a rt u p o n th e e t h n o g r a p h e r ’s p e rs o n a l c h a r a c t e r is t ic s , s u c h
as w h e th e r th e y a re frie n d ly a n d a p p ro a c h a b le , ab le to in sp ire tru s t a n d c o n f i d e n c e , o r
w h e th e r in fo rm a n ts th in k th e y a re a r ro g a n t a n d s ta n d o ffis h .
A b o v e all e lse, fie ld w ork in v o lv e s th e toil a n d d e d ic a tio n o f th e e t h n o g r a p h e r , w h o h as to
s p e n d t h o u s a n d s o f h o u rs in th e field, l-'ieldw ork relies oil th e te n a c ity an d p e r s e v e r a n c e o f
th e e th n o g ra p h e r, an d th e p a tie n c e to sit th ro u g h lo n g , b o rin g m e e tin g s in th e h o p e th at
s o m e th in g in te re s tin g m ig h t c o m e up. F t h n o g r a p h e r s c a n s p e n d m o n th s fee lin g as if n o t h ­
in g is h a p p e n in g an d n o p ro g re s s is b e in g m a d e , o n ly lo r th in g s to s u d d e n ly ta k e off. It c a n
o fte n feel like field w ork is 9 9 " » p e rs p ira tio n fo r th a t 1% in sp ira tio n th at u n lo c k s th e d o o r to
s o m e sp e cific issu e o f in te re s t. D o in g fie ld w ork is a lo n g an d a r d u o u s b u sin e ss , o ften in a
v e ry d ifficu lt e n v iro n m e n t, o fte n a m o n g s tra n g e a n d u n fa m ilia r p e o p le. It c a n b e lo n e ly an d
a lie n a tin g , a n d is th e m o st p e rs o n a lly in volv e d an d c h a lle n g in g m e th o d o f d a ta c o lle c tio n
th at th e re is. Put sim p ly, d o in g field w ork is to u g h .
l o r m a n y of t hese re a son s, t he idea o f s peci fic m e t h o d o l o g i c a l t r a i n i n g in p a r t ic ip a nt
o b s e r va t i o n was, lor a l on g t ime , not t ak en terri bl y seriousl y. As W h i l i n g recall s b e i n g told
by a Vale I 'm v e r s i t y pro f es s o r in t he 1 93 0 s , m e t h o d 'wa s a subj ect to b e d is c u s s e d c a su al ly at
b r e a k fa st ’ ( W h i t i n g 1 9 S2: 15M. not s o m e t h i n g w o r t h y o f a s e m in a r. Th e skills for d o i n g
f ieldwork w ere re g ar de d as my s t e r i o u s a nd u nt e a c h a b le ( B e r n a r d 2 0 0 6 : 3 4 3 ) . S inc e s o m u c h
d e p en ds u po n the p er s on al c h a r ac t er i st i c s of t he e t h n o g r a p h e r , it is not really pos si bl e to
know how s o m e o n e is go i ng to r e a d to the pre s su re s of d o i n g fi eldwor k until t hey a re out in
lhe lield. ac tual ly d o i n g it. b i b l i og r ap h er s w er e t he re fo re e x p e c t e d to basi call y just gel on
with it. a nd learn how to d o part ic ip an t o bs e r va t i o n o n t he job.
Al t ho ugh m u c h ol this still hol ds, it d oe s p e r h a ps re p re s en t s o m e t h i n g of a r o m a n l i c i z a -
tion ol what it m e a n s to d o fieldwork. Pa rt ic ip an t o b s e r v a t i o n is b ot h a h u m a n i s t i c m e t h o d
a nd a sc i cntihc o ne ( Be r n a r d 2()0(v 3 4 2 ) , an d as s uc h it ent ail s a n u m b e r o f m e t h o d o l o g i c a l
pri nci ples that c an a t i e d the quality, reliability, an d validity ol t he d at a that is c o lle c te d . In
recent vears, i heie has been a grow ing r e c og ni t io n ol this, a n d t he re are now m a n y t e x tb o o k s
s pe ut ka ll v on e t hn og ra p hi c m e t h o d s (see l l a m m e r s l e y a nd A t ki ns on 2 0 0 7 for a p a r t i a l
la 'Tv in depth e x a mp l e I. and m an y u n d e r g r a d u a t e a nd g r a du a te c o u r s e s that a d d re ss di ffer­
ent m e t h o d ol o gi ca l and ethical c o n s i d er at io n s c o n n e c t e d to the pr oc e ss o f c o l le c ti n g and
analv s mg data.

P a i t u i p a n t o b s e i v a t i o n i nvol ve s t he r e s e a i c h e i ma k i n g a nu m b er of me t hodo l o gi c al
dec i s i ons , aii d t he de c i s i o ns t hat are m a d e c a n i n f l ue n c e t he qual i t v ol t he dat a t hat is col
l eded l h e s e de c i s i o ns i . ue t he e t h n o g r a p h e i evei v dav. a n d wh e r e a s it is not po s s i bl e to
di s c us s . i l l ot t h e m m detai l , it is n o n e t h e l e s s i mp o i t a n t to hi ghl i ght s o me ol t he ma i n i ssues
 E T H N O G R A P H Y A ND P A R T IC IP A N T O B S ER V A T IO N

that need to be con fron ted , and discuss th eir im plications for the quality o f data that is g en ­
erated. Broadly speaking, particip an t ob serv ation involves three interrelated steps. I he first
step refers to ca se s e le c tio n and gain ing access to the field site. Participant observation
relies (usually) on the detailed analysis o f a single case study. The criteria and justification
for the ch o ice o f case (and type o f case) that is selected is very im portant, since the type o f
cases you ch o o se to analyse can in fluen ce the answ ers you get to a particular research qu es­
tion (see C hapter 9 ). This p roblem is even m ore o f an issue when only one case is being
exam in ed . The secon d step refers to issues related to carryin g out research in the field and
colle ctin g data. The role th e investigator adopts, the con tacts they m ake, the inform ants
they ob serv e and speak to, and the ways in w hich this is done, all have a substantial bearing
on the quality, reliability, and validity o f the data that is generated. Finally, the third step
refers to record in g ob serv a tion s, and how the data that is collected is w ritten up, coded, and
used for analysis.
In som e sense, th ese different issues are analogous to the sam pling and m easurem ent
issues that we discu ssed with referen ce to survey research in C hapter 10, and to a greater or
lesser extent the sam e issues underpin all em pirical research, w hether it is qualitative or
quantitative. W e have to be aware about how we select our cases, how we collect our data and
cod e it. The d ecisions we m ake at each step can have a strong bearing on what our final
results lo o k like. Participant ob serv ation may p rioritize internal validity over external valid­
ity, but it certa in ly d oes no t ign ore th e latter. But even to be internally valid, it is necessary to
pay special a ttention to how th e data is collected and coded w ithin the dom ain o f study. It
would be p rob lem atic, to say th e least, if ano ther eth nographer carried out research on
exactly th e sam e top ic in th e sam e location at the sam e tim e and cam e to very different c o n ­
clusions.

Choosing a research site

In a sense, all eth nograp hies are a form o f case study (th ough the reverse is not true). And,
as with any case study, th e selection o f the case (or field site) poses a nu m ber o f challenges,
and th ere are m any issues that need to be considered. H am m ersley and A tkinson (2007: 28)
suggest that case selection can proceed in one o f two general ways. First, the issue, or research
q u estio n, or foreshadow ed p roblem com es first, and a case o r settin g is then selected in order
to investigate th e top ic. O r secon d, in som e special circu m stan ces, the setting itself may
c om e first. A sp on taneou s op p ortu nity m ay arise to investigate an interesting situation or
group o f people, and th e research q u estio ns spring from the nature o f that setting. For e x am ­
ple, Pieke (1 9 9 5 ) recou nts how five m o n th s into his fieldwork in Beijing, he w itnessed the
em erg en ce o f the 1989 C h in ese People’s M ovem ent. He thought this p resented an interesting
opportu nity, and so started to c arry out research. The setting thus cam e first, and his research
qu estio n s w ere thus largely driven by his ch o ice o f case, rather than the oth er way round.
H owever, in stan ces o f th is type o f ‘op p ortu n istic research’ (R iem er 1977) are relatively rare,
and m o re usually th e ch o ice o f research settin g (at least initially) is guided by the research
qu estio n and th e issues or hyp otheses that the eth nograp her wants to investigate, though, o f
cou rse, th ese m ay chan ge som ew hat d uring the cou rse o f fieldwork.
294 H O W T O D O R E S E A R C H IN P R A C T I C E

O n e o f th e k e y p r o b le m s th a t e th n o g r a p h e r s fa c e is th a t th e y s o m e tim e s fin d th a t th e field

site th e y h a v e s e le c te d is n o t an a p p r o p r ia te a r e n a to in v e stig a te w h a t th e y h a d in itia lly h o p e d
to e x p lo re . T o a g r e a te r o r le ss e r e x te n t th is p r o b le m is r ela tiv ely fr e q u e n t, a n d it is w id ely
a c c e p te d th a t th e c o lle c tio n o f p r im a r y d a ta o fte n p lay s a n im p o r t a n t ro le in th e d e v e lo p ­
m e n t o f th e r e s e a rc h q u e s tio n . W h a t a n e th n o g r a p h e r in itia lly s e ts o u t to in v e stig a te m a y n o t
th e re fo re a lw a y s b e e x a c tly th e sa m e as w h a t th e y e n d up in v e stig a tin g . A lth o u g h th is isn ’t
a lw ay s a p r o b le m , a n d in d e e d it c a n s o m e tim e s le a d to th e e la b o r a tio n o f n e w a n d e x c itin g
r e s e a rc h q u e s tio n s , i f t h e in itia l q u e s tio n o r issu e w as th o u g h t to b e w o r th in v e s tig a tin g in
th e first p la c e , th e n it is w o r th s e r io u s ly th in k in g a b o u t h o w th is c a n b e d o n e .
In p a r tic u la r, th e r e a re tw o in te r r e la te d issu e s th a t n e e d to b e a d d r e sse d . T h e fir s t is to
e sta b lis h th e ty p e o f s ite th a t th e r e s e a r c h e r p la n s to stu d y, a n d s e c o n d , o n c e th is is d o n e , to
d e c id e w h ic h s ite in p a r tic u la r to s e le c t fo r in v e s tig a tio n . T h a t is, b e f o r e s e le c tin g th e site , it
is first n e c e s s a r y to th in k a b o u t th e p o p u la tio n o f p o t e n tia l site s th a t w o u ld b e e lig ib le ( o r
a p p r o p r ia te ) c a s e s fo r s e le c tio n . T h e m o r e th o u g h t th a t is p u t in to th is at th e p la n n in g sta g e ,
th e less r is k th e r e is o f e n d in g up in a p la c e th a t is ill-s u ite d to e x a m in in g th e d e s ir e d issu e s.

Types of research site

E a r ly e th n o g r a p h ic s tu d ie s u sin g p a r tic ip a n t o b s e r v a tio n te n d e d to b e t h e p r e s e r v e o f a n t h r o ­

p o lo g is ts , w h o o fte n w o r k e d in sm a ll, is o la te d , s e lf- c o n ta in e d c o m m u n itie s th a t h a d little to
d o w ith th e o u ts id e w o rld a n d sta te s tr u c t u r e s . H o w e v e r, e th n o g r a p h e r s h a v e in c r e a s in g ly
tu r n e d th e ir a tt e n t io n b e y o n d v illa g e s e t tin g s to in v e s tig a te s o c ia l a n d p o litic a l p h e n o m e n a
in a w id e v a rie ty o f s e ttin g s , fr o m u r b a n e th n o g r a p h ie s to e th n o g r a p h ie s o f p o litic a l p a r tie s ,
in s tit u tio n s , a n d s o c ia l m o v e m e n ts . In d e e d , s o m e o f th e m o s t in - d e p th e th n o g r a p h ie s rely
o n fie ld w o rk in m u ltip le s e ttin g s . F o r e x a m p le , M ic h e lu t ti s ( 2 0 0 8 ) stu d y o n h o w d e m o c r a t ic
id e a s a n d p r a c tic e s w e re r e in t e r p r e te d a m o n g th e Y ad a v s, a h is to r ic a lly m a r g in a liz e d c o m ­
m u n ity in In d ia , w as c a r r ie d o u t in a n e ig h b o u r h o o d w h e re m a n y Y ad av s liv e d , in th e t e m ­
p le s a n d r e lig io u s fe stiv a ls th a t th e y a tte n d e d , in th e p o litic a l p a r tie s to w h ic h th e y w ere
a ffilia te d , a n d in th e c u ltu r a l a s s o c ia tio n s o f w h ic h th e y w e r e m e m b e r s . A lt h o u g h th e m a in
r e s e a r c h s e ttin g w as th e u r b a n a r e a w h e re th e y liv ed , th e o th e r s e t tin g s s t e m m e d fr o m th is ,
lin k in g th e lo c a l a r e a to th e w id e r r e g io n , th e sta te , a n d e v e n to th e n a t io n . In th is se n s e ,
e th n o g r a p h y is o fte n d e s c r ib e d as h o lis tic , in th a t it e x a m in e s h o w w h a t p e o p le say a n d d o
p la y s o u t in d iffe r e n t s e t tin g s a n d d o m a in s o v e r tim e . L in k e d to th is , r e c e n t d is c u s s io n s o f
m u lti- s ite d a p p r o a c h e s se e k in a s im ila r w ay to lin k th e lo c a l to th e r e g io n a l a n d e v e n th e
g lo b a l ( M a r c u s 1 9 9 5 ; C o m a r o f f a n d C o m a r o f T 2 0 0 3 ) . M u lti-s it e d a p p r o a c h e s c a r r y o u t fie ld ­
w o rk in d iffe r e n t r e s e a rc h site s. F o r e x a m p le , r e s e a r c h o n th e P u n ja b i d ia s p o r a m ig h t e a r n ­
o u t r e s e a rc h in th e P u n ja b , o n p e o p le liv in g th e re , a n d th e n c a r r y o u t r e s e a r c h in a d iffe r e n t
site in S o u th a ll, in th e U K , w h e re la rg e n u m b e r s of P u n ja b is live.
B y c o n t r a s t, o th e r s tu d ie s , p a r tic u la r ly in p o litic a l r e s e a r c h , h a v e te n d e d to h a v e a m o r e
s p e c ilit lo t u s a n d a re o lte n p r im a r ily b a s e d o n fie ld w o rk in o r g a n iz a t io n s a n d in s tit u tio n s ,
sin h .is p o litic ,il p a r tie s , th e c iv il s e r v ic e , p a r lia m e n t, th e F I ’, a n d N l i O s , o r lo o s e n e tw o r k s ,
s m h .is g a n g s , r e lig io u s e x tr e m is t s , o r fr e e d o m lig h te r s . M e m b e r s a n d p a r tic ip a n ts m av no t

.ill liv e in th e sa m e p la n -, a n d so an - p r im a r ily lin k e d th r o u g h th e o r g a n iz a t io n s . F o r th e s e

i . i s i s , 11 it- 111.1111licld sih- is i h e i e l o i e th e o r g a n iz a t io n , siu h as th e loc a I p o litic a l p ar tv h e a d

<|ii.n I n s. a n d o il ic i i c m '.iii li s it e s m av th e n sii-m 11o m th is ( s e e . loi e x a m p le . B a le a n d P a n n s

 E T H N O G R A P H Y A ND P A R T IC IP A N T O B S ER V AT IO N

2 002 study o f New Z ealan d ’s G reen Party). The definition o f the prim ary research site and
ju stification tor why one type ot site is appropriate rather than another is therefore a crucial
first step, which needs to be related explicitly to the research question and objectives.

Selection of research site

Having defined the type o f research site that is appropriate for the research project, the next
step is to decide w hich specific site is to be selected for the study. O ften practical consid era­
tions form a p rom in ent basis for selection , and although these are im portant, they should
not be entirely divorced from th eoretical and m ethodological considerations. The factors
that m ake access to a site easier m ight also m ake it peculiar in som e im portant respects,
which m eans that it m ight not be very representative or typical o f the population o f interest.
For exam ple, although there is a great deal o f acad em ic interest in the shanty tow ns, or bar­
rios, that surround m ajor cities in Latin A m erica, these locations are often dangerous and
v iolen t places w hich m ake it risky for a research er to ju st turn up. There is a ten dency am ong
research ers, th erefore, to focus on the b arrios which are relatively safe, and where there are
active N G O s and religious organizations that can help with access arrangem en ts and in tro ­
d uctions. A lthough th is paves th e way for som e incredibly valuable research, at the sam e
tim e b a rrio s w hich have these organized links with civil society are likely to be som ew hat
different from th e on es w here N G O s and religious organizations are not present. This is an
im p ortan t con sid eration w hen th in kin g about how far findings from the research may be
generalized.
A useful startin g poin t for guiding case selection should therefore be to try and define the
p opu lation o f elig ible cases for selection . W hat are the th eoretical criteria for selecting a
p articu lar case? A nd w hat oth er p otential cases m ight fit these criteria? Is the case chosen
b ecause it is in tend ed to be typical or b ecause it is an outlier? For exam ple, if the research
p roject is o n H in d u -M u slim relations in India, are you interested in exam in ing a neighb our­
ho od th at is ch aracterized by particularly high levels o f con flict, or particularly low levels o f
co n flict? T h in kin g about th ese q u estio ns will help to identify the population to which your
fin d ings m ight apply, and set th e param eters for potential generalizations that may stem
from th e research.

Access to research site

H aving selected a site to con d u ct fieldw ork, one o f the m ost difficult steps in carryin g out
p a rticip a n t ob serv a tion is gain ing access to the research site. It is all very well to have a clear
idea ab ou t th e type o f place w here you w ant to do research, and even to have selected a
p oten tial site. B ut i f it is n o t p ossible to gain access to the field site, th en it is b ack to the d raw ­
ing b oard . A ccess issues can create p rob lem s in two m ain ways. First, if access is not possible,
it m ay in tro d u ce response b ias sim ilar in nature to the selection bias discussed above. Places
w here you c a n n o t gain access m ay be different in im p ortan t respects from the places where
you can (o r ca n m o re easily) gain access.
Seco n d , access issues can also affect the way in w hich you carry out research. How you
en ter th e field can shape how you are perceived by in form ants, and so your poin t o f entry
m ay have lo n g -term co n seq u e n ces for your research p roject. O n e ob stacle eth nographers
296 H O W T O D O R E S E A R C H IN P R A C T I C E

frequently face is to overcome the suspicion of the people they are studying. For many
informants, the purpose of ethnography is hard to fathom. W hy would anyone want to
study us? It is sometimes much easier to assume that the investigator is a spy rather than a
political scientist. W ith time and hard work, these suspicions can be overcom e, but the
extent to which this is possible and the am ount of time it takes may be influenced by the
level of distrust in the first place. And on this score, first impressions can count for a lot. In
this section we discuss some of the strategies that can be employed to gain access to the
research site, and the impact that these strategies may have on the way in which research
is conducted.
The issues surrounding access vary in a num ber of ways according to the object of
investigation. This is often discussed in term s of whether the research setting is open or
closed or a public or non-public setting (Ham m ersley and Atkinson 2 0 0 7 ). The distinc­
tion between the two types of setting is relevant for gaining access because a closed setting
may require the perm ission of som eone in order to gain access. Access to closed settings
(non-public), such as formalized groups or institutions, political parties, law courts, the
police, the European Union, UN, development organizations, NGOs, to nam e just a few,
present a num ber of problems, since som e sort of formal perm ission m ay be required in
order to carry out research. Brym an (2 0 0 4 : 2 9 7 ) suggests a num ber of strategies for gain­
ing access to closed (organizational) settings, such as using friends, contacts, or colleagues
to try to get support from som eone within the organization. In particular, it is highly likely
that you will need perm ission to do research from a gatekeeper— senior m anagem ent in
the organization. Gatekeepers m ay be w ary about letting researchers snoop about in the
organization, particularly if they have concerns as to w hether the organization will be
presented in a good light or not. Similarly, governm ent perm issions, or perm its, are often
needed to carry out research in tribal areas or reservations, and these perm its can be dif­
ficult to com e by, particularly if the research is on a sensitive topic. It is often then a good
idea to offer gatekeepers som ething in return, such as a report or su m m ary of your main
findings. In dealing with gatekeepers, Brym an also suggests that it is a good idea to p ro ­
vide a clear explanation of the aims and m ethods o f the research, and the am ount of tim e
it will take (and take away from people doing their job or other duties). It is also im portant
to negotiate. Full access may not be forthcom ing, but with careful com prom ise it m ight be
possible to have som e sort of limited access that nonetheless enables the researcher to
investigate their topic. As Schatz (2 0 0 9 : 3 0 7 ) writes, access is a sliding scale’ on w hich the
political ethnographer strives to achieve the n ea rest p o s sib le v a n tag e p o in t to study a given
problem ’.
W h e r e p e r m i s s i o n is n e e d e d , it is o b v i o u s l y i m p o r t a n t t o g e t t h e p e r m i s s i o n g i v e r — o r

g a t e k e e p e r — o n b o a r d . H o w e v e r , it is a l s o i m p o r t a n t to b e a r in m i n d h o w t h e c o m p r o m i s e s

y o u m a k e ( i f a n y ) a n d t h e r e l a t i o n s h i p y o u d e v e lo p w it h t h e g a t e k e e p e r w ill a f f e c t v o u r

r e s e a r c h , a n d h o w it w ill s h a p e h o w y o u a r e p e r c e i v e d b y i n f o r m a n t s o n c e y o u h a v e g a in e d

access. I h e e x t e n t to w h i c h t h is is a n is s u e m a y d e p e n d u p o n t h e t y p e o t s e t tin g « a n d th e

h i e r a r c h y w it h in t h e o r g a n i z a t i o n , a n d h o w t h e g a t e k e e p e r is v ie w e d I n t h o s e w it h w h o m

y o u will Ik- d o i n g r e s e a n h . l o r e x a m p l e , it y o u a r e w o r k i n g m a t a c t o r v , a n d h a v e b e e n g iv e n

p e r m i s s i o n to d o i c s e a n . Ii b y a n u n p o p u l a r m e m b e r o l s e n i o r m a n a g e m e n t , t h e n t h e r e i> t h e

u s k th a t b y a s s o u a t i o n y o u w ill In- v ie w e d by th e w o ik e i.s in a n u n ta \ o u t a b le lig h t , a n d

p e r h a p s s e e n a s a sp y
 E T H N O G R A P H Y A N D P A R T IC IP A N T O B S ER V AT IO N

Access to open settings (public sites) is in theory m ore straightforward, since there aren’t the
sam e bureaucratic obstacles that need to be negotiated. But in practice it is beset by many o f the
sam e issues that face gaining access to private settings, particularly when the object o f study is
defined as a hard-to-reach population. If the site is a neighbourhood, or village or other public
space, then it is not necessary to obtain formal perm ission to be there, and so the researcher can
arrive, check out the location, and sim ply start observing what is going on. To facilitate this proc­
ess, researchers often choose field sites where they have som e prior personal link or contact.
However, som e types o f open public sites can present substantial access problem s, par­
ticularly if the group in qu estion is hard to reach, or involved on the periphery o f legal activ ­
ity, or indeed com pletely im m ersed in it, such as m afia, terrorist groups, or freedom fighters.
For these in accessible groups, it is often necessary to develop co ntacts who can provide an
in tro d u ction to the group in question. However, o b tain ing these con tacts is far from straigh t­
forw ard. They rely on trust and so are often only m ade after m any years o f hard work.

The role of the ethnographer

O n e obvious way to reduce som e o f these access problem s is to do away with the need for
p erm ission in the first place, and for the investigator to assum e a cov ert role, in which they
do not reveal the true purpose o f th eir research, or even that they are a researcher at all. This
strategy m eans that the research er does not face the problem s o f having to persuade people
to give th eir perm ission to be studied, and the researcher d oesn’t have to face awkward qu es­
tions about why they w ant to do the research either. This distin ction betw een ov ert research
and cov ert o b serv ation also in fluen ces the way in which data is collected, and the role that
the eth n ograp h er adopts in th e field. Broadly speaking, there are three m ain types o f role the
research er can adopt: the participan t, the participant observer, and the observer.
T he participan t is a fully fu n ctio n in g m em b er o f the social setting and his or her identity
is not know n to the in form ants. The participan t is a covert observer. It is widely d ocum ented
that the a ct o f b eing studied can alter the b ehaviou r o f the su b jects under investigation. This
is kno w n as rea ctiv ity . O n e way to reduce this reactivity is for the researcher to adopt a co v ­
ert role so that the in form an ts don’t know they are b eing studied. The behaviou r o f in fo rm ­
ants will th erefo re be m o re natural, and this can often lead the research er to gain m uch closer
access. For exam ple, H um phreys (1 9 7 0 ) pretended to be gay in order to enter the world o f
th ose he was studying.
The p articip an t ob serv er is the m ost co m m o n approach used by the investigator in eth n o ­
g raphic research . It is m u ch the sam e as the participan t approach, in that the researcher
p articip ates d irectly in th e in form an ts’ lives, but differs in that it is overt, and the in form ants
are aware o f th e research er’s identity. By contrast, the ob serv er (or o b serv er-p articip an t) is
focused on ob serv a tion rather th an p articipation. The research er acts m ainly as an in ter­
viewer, or c a rries out un ob tru siv e ob serv ation .
A lthough th ere are som e obvious attraction s to the covert approach, particularly when
th e research er is c o n ce rn e d that perm ission m ight not be given if asked for, or w hen the
research er is c o n c e rn e d that know ledge about th eir identity m ight alter the way in which
in fo rm a n ts relate to th em , th ere is also a nu m b er o f eth ical issues related to carryin g out
c ov ert research to do w ith d ecep tion and in form ed c on sen t (see C hapter 7 ). It is th erefo re an
approach that is no t w idely used anym ore.
298 H O W T O D O R E S E A R C H IN P R A C T I C E

Doing participant observation

Once in the field, there is a number of issues that the researcher faces in terms of how they
can do participant observation, and how this will influence the quality of data that is gener­
ated. In participant observation, the main instrument of data collection is the individual
ethnographer. The reliability and validity of the data that is collected therefore depend in
large part upon how they behave, who they talk to, what they observe, and where they go. In
addition, there are also all sorts of personal issues—such as the gender, ethnicity, and age of
the researcher—that may influence how they are perceived by their informants, and shape
their role as a researcher within the community. This can introduce considerable subjectivity
into the research process, which may bias the conclusions that are reached (see Chapter 3 for
an extended discussion of this).
Having gained access to the field site, a requisite of participant observation is to build and
develop a relationship with the people who are being studied. Part of this comes simply with
time and contact. One of the most important factors that influences the quality and reliability
of the data generated is therefore related to the amount of time spent in the field. All other
things being equal, the ethnographer is likely to have much more reliable data if they have been
doing fieldwork for 12 months than if they have only been doing fieldwork for 12 days. It is only
by observing events and talking to people over a relatively long period of time that the ethnog­
rapher can build up a picture of how the community under study really thinks and behaves.
But the amount of fieldwork that is needed can also vary substantially according to the
researchers familiarity with the method of participant observation, the topic of study, and
the location of fieldwork and the people. Since many of the skills of doing participant obser­
vation are learned on the job, the first major piece of fieldwork by an ethnographer might be
as long as 18-24 months, but subsequent fieldwork might be much shorter. With practice,
the ethnographer will be able to collect data more efficiently, they will have a better eye for
what to look out for, and possess more finely tuned research skills. Moreover, if the ethnog­
rapher is going back to the same place as a previous fieldwork, then much of the background
data is already in the bank. They already know the people and the language, and so can col­
lect good data in just a matter of days or weeks.
In a d d it i o n , in o r d e r to d e v e lo p a r e la tio n s h i p w ith i n f o r m a n t s , it is n e c e s s a r y fo r th e
e t h n o g r a p h e r to le a r n th e la n g u a g e o f th e p e o p le th e y a r e s t u d y in g . T h is is a c o r e sk ill in
its o w n r ig h t , a n d th e lin g u is t ic c o m p e t e n c e o f e t h n o g r a p h e r s is a k e y p a r t o f b e in g a b le to
d o r e s e a r c h e ffe c tiv e ly . A lt h o u g h s o m e e t h n o g r a p h e r s u s e in t e r p r e t e r s , to p ic k up s e n s i ­
tiv e d a ta it is a d v a n t a g e o u s to k n o w th e lo c a l la n g u a g e . 'F lu e n c y in th e lo c a l la n g u a g e
d o e s n ’t ju s t im p r o v e y o u r r a p p o r t ; it in c r e a s e s th e p r o b a b i lity th a t p e o p le w ill tell yo u

s e n s itiv e th in g s . . . a n d th a t e v e n il p e o p le tr y to p u t o n e o v e r o n y o u , y o u ’ll k n o w a b o u t it'

( B e r n a r d 2 0 0 6 : 3 6 1).

Choosing informants

< ) n c ol llu j.*i t-.it s t ic n g t h s o ! p .irtu ip a tio n is th at it is th o u g h t to h av e h ig h le v e l* ot in t e r n a l

v a lid ity II us m e .m s lli.it llu ' u 's i . m I it' i i an b e t o n tu le n t th a t th e In u lm g s t i o m th e t e s e a u h
. III \ .iliil W illi i c l e i c i u I lo tilt d o n u m ot sliu K (h e ld s ite ), e \ en it the\ a re less . o n tid e n i th at

tin f in d in g s v .in In in i . ill/<•>I 11 i o i i w ulcl\ lo o il ic i i «oteli lia I site s I low e \ e i, I h is p o in t »..in

 E T H N O G R A P H Y A ND P A R T IC IP A N T O B S ER V AT IO N

easily be overstated, and should not be taken for granted. There are a nu m ber o f issues that
influence the internal validity o f eth nographic research, and, without careful attention, it is
easy to produce data that are neither internally nor externally valid.
As with surveys, doing participant observation involves sam pling. Kven just hanging
around and chattin g to people is a kind o f c onv en ience or v olu n teer sam ple (see Chapter
10). Ideally, the eth nographer would speak to everyone in the population o f interest (field
site), and ob serv e everyone equally. However, in p ractice this is rarely possible. The eth n o g ­
rapher m ight speak to som e people m ore than oth ers and spend tim e with som e people
m ore than oth ers. The sam ple o f in form ants that the eth nographer prim arily relies upon
may th erefore on ly be a relatively sm all sub section o f the entire population. This is particu ­
larly the case for urban eth nographies, where hundreds if not thousands o f people may live
tog ether in relative proxim ity, or for eth nographies o f relatively large organizations that
m ight con tain m any m em bers. In these situations, the eth nographer needs to think very
carefully about w ho they spend tim e with and why, as error and bias can easily be introduced
into the sam ple, w hich will m ean that it is not representative o f the population o f study, and
so the in ternal validity o f the findings will be com prom ised.
For example, suppose a researcher is conducting an ethnography o f the House o f C om m ons,
and is doing participant observation am ong British MPs. Part o f the research will involve w atch­
ing what M Ps do and say in public places, but part o f the research will also be carried out in more
private settings, where in-depth conversations and interviews with M Ps can take place. Perhaps
also the researcher will track different M Ps, and get perm ission to observe who they have lunch
with, who they talk to and m eet, and so on to shed light on the links and contacts that M Ps have
with business and lobby groups and different social organizations. Basically, the ethnographer
will try, as m uch as possible, to live the life o f an M P in order to get a detailed first-hand im pres­
sion o f what they do. But undoubtedly the researcher will spend m ore tim e with som e M Ps than
others. Perhaps they will only spend tim e with backbench M Ps because senior m inisters and
shadow m inisters may be less prepared to give full access to the researcher because they are
concerned about leaks, confidentiality, secrecy, or are sim ply just too busy. So already there is
som e selection bias, and what started o ff as a study o f M Ps is now becom ing a study o f backbench
M Ps. But, even am ong these backbench M Ps, som e might be m ore helpful and accom m odating
and m ore prepared to give up their tim e. There is always a tem ptation to be drawn towards the
cooperative inform ants and to spend tim e talking to the people whom you get on with best, who
are m ost helpful and approachable, and who m ake the jo b o f the ethnographer easier.
So m e in form a n ts m ay th erefo re be m ore useful or helpful than oth ers, and certain in fo rm ­
an ts— or key in fo rm a n ts — m ay b eco m e m ore central to the collection o f data than others.
Th ese key in form a n ts can be a great asset. They may understand the purpose and aim s o f the
research , and be able to d irect th e eth nograp her to situations, or places or people, w ho may
be useful for research. B ut they also carry risks. O v errelian ce on key in form ants can create a
different set o f p rob lem s, to do with the reliability and validity o f the in form ation that is
gen erated . As B ry m an ( 2 0 0 4 :3 0 0 ) puts it, a reliance on key in form ants may m ean that ‘rather
th an seeing social reality th rou gh th e eyes o f m em b ers o f the social setting, the research er is
seeing socia l reality th rou gh th e eyes o f the key in fo rm an t’. It is th erefo re v ery im portant for
th e eth n og ra p h e r to con tin u ou sly b ear in m ind the representativeness o f th eir in form ants,
and to actively seek out different voices and speak to th ose w ho m ay be less forth com in g , as
well as th o se w ho are help fu l and am e n ab le. T h e co n s e q u e n ce o f failin g to do so (o r not
H O W TO D O R E S E A R C H IN P R A C T IC E

d oin g so adequately) is to in tro d u ce e rro r and bias, w hich can u n d erm in e th e in tern al v alid ­
ity o f the study.

Studying with informants

B ecause the nature o f p articipan t o b serv ation relies on so m u ch social in teractio n , Ingold
characterizes it as study with a particu lar group o f people, rather th an ju st a study o f people.
The social in tera ction is thus an im p ortant, but also p otentially prob lem atic, asp ect o f par­
ticipant ob serv ation , as the research er aim s to b eco m e accepted’ and ‘tru sted ’ by th o se that
they are w orking with. As W hyte (1 9 9 3 : 3 03) argues, ‘if people accep t you— you can ju st
hang around and you can learn the answ ers in th e lon g run w ithout even having to ask q u es­
tions’. Building trust is thus a key part o f th e research process. It creates an env iron m en t
where in form ants can speak frankly about sensitive and private issues w ithout em b arrass­
m ent, w here they have the confid ence to express th eir ow n views rather th an th e views they
th ink they should hold or are culturally expected to hold, all o f w hich allow s th e eth n o g ra ­
pher to collect rich, detailed, valid data. In addition, th e m o re an eth n ograp h er is accepted
by the com m un ity he o r she is studying, the less likely in form ants are to respond to th e p res­
ence o f the ethnographer. Rather than sticking out like a sore th um b and co nstantly draw ing
attention to them selves, the eth nographer can slip into the background, b eco m e part o f the
m undane, and observe quietly what goes on around them . But if the eth nograp her is not
really trusted — or viewed with suspicion— th en there is a greater risk that the ethnographer,
ju st by th eir very presence, can influence what happens in the research site. In form an ts will
be guarded around the ethnographer, will change th eir behaviou r when they th in k they are
being observed, and act in an artificial’ way.
This is known as reactivity, which can be a m ajor source o f error in eth nographic research.
Reactivity occurs when the very presence o f the researcher alters the way in which in fo rm ­
ants act. There are two m ain strategies for d ealing w ith this (see Bernard 2 0 0 6 :4 2 5 ). The first
is training. It is not possible to elim inate observer bias com pletely, but th ere is a w ealth o f
evidence to suggest that training and experience m akes people b etter— m ore reliable and
m ore accu rate— observers (H artm an n and W ood 1990). The second is to build trust. W hen
inform ants are com fortable in the presence o f the ethnographer, they b ecom e m ore
unguarded and less likely to change their behaviour w hen the eth nographer is around. This
helps to reduce (though never entirely elim inates) the problem o f reactivity. B ecom ing
accepted and gaining trust is thus an im portant part o f doing fieldwork. In doing so, the
ethnographer will have greater opportunity to observe and talk about things that are per­
haps on the borders o f social acceptance or legality— and enter worlds o f crim inality, c o r­
ruption, usury. Inform ants will also be m ore likely to reveal personal in form ation and allow
the ethnographer to ask direct and personal questions, exam ine sensitive areas o f research,
and be invited to private settings.
But this acceptance can also be a double-edged sword. Becom ing accepted within one
part ot the com m unity of study can also isolate the researcher from oth er groups. For e x am ­
ple, a researcher interested in studying H ind u -M u slim relations in India m ight find that as
they becom e accepted by Hindus, they b ecom e distrusted by Muslim s. Their research th ere­
fore becom es skewed—or biased— towards one side o f the story. Sim ilarly, during Stacey
et al.s (19/n) research on local party politics, the research team hired offices in the same
 E T H N O G R A P H Y A ND P A R T IC IP A N T O B S ER V AT IO N

building as the local Labour Party headquarters. This gave the researchers good access to
Labour Party workers, and they soon built up good relations with them . But in the process
of doing this, they realized that they were now being viewed with suspicion by C onservative
Party w orkers, w ho thought that the researchers were too close to Labour, and som ehow ‘on
th eir side’. These problem s can com pound the access problem s discussed previously, if the
group that the research er is perceived to be closer to is also the group that they gained access
through.
A related problem associated with building trust and acceptan ce is to do with the nature
o f the research er’s relationship with the people they are studying. Although the eth n ogra­
pher needs to cultivate good relations in ord er to collect good data, they also need to be able
to m aintain som e em otion al d istance from the people they are studying so they retain the
capacity for critical analysis. As the research er becom es m ore im m ersed in the com m un ity
o f study, they m ay find it harder to be o b jectiv e (see C hapter 3). In extrem e circum stances
ob serv ers m ay even ‘go n a tiv e’ and identify so m uch with the sub jects that they lose the
capacity for ob jectiv e criticism . They may even b ecom e an advocate for the group and ca m ­
paign and lobby on th eir behalf.
It can b e in cred ib ly difficult to stay neutral and im partial when the act o f data collection
involves so m u ch close social in teraction in order to gain the trust o f inform ants. I h is can
m ake it difficult to present a ‘warts and all’ betrayal o f real life, and the eth nographer may not
want to present th eir in form an ts in a bad light, particularly if they would like to go back to
th e field site to do research in the sam e place again. The practical difficulties o f m aintaining
this b alan cin g a ct can be hard to pull off, and there is no easy solution. W hile it is desirable
to b eco m e accepted w ithin the group, th ere can also be unintended consequences o f doing
this. Indeed, it can so m etim es feel like an act o f betrayal and raise eth ical issues. The art o f
gain ing accep ta n ce is often describ ed as a euphem ism for ‘im pression m an agem ent’ or
d eception.

Identity of the researcher

The ability o f th e eth n og rap h er to build a relationship with in form ants, and the type o f rela­
tionsh ip th at they are able to build, depends not only on what the eth nographer does in the
field, but also to a certa in extent on w ho the eth nograp her is. There is a large reflexive litera­
ture on how the id entity o f the eth nograp her in fluen ces how in form ants perceive them . In
particular, m u ch o f th is d iscu ssion has focused on the sex o f th e ethnographer, but oth er
social ch a ra cteristics, such as age, ethnicity, nationality, and even m arital status and w hether
or not th e e th n og rap h er has child ren , have all been identified as having an im portant im pact
on how th e research er is received by in form an ts and the kinds o f a ccess to different social
w orlds th at th ey are granted.
As far b ack as th e 1930s anthrop ologists such as M argaret M ead had already started to
h ighligh t th e im pact th at the g en der o f th e eth n ograp h er could have on the collection o f data
in term s o f th e co n seq u e n ces it had on (i) access to in form ation ; and (ii) on how o th ers p er­
ceive th e eth nog rapher. Som e social ch aracteristics have a fairly restrictiv e im pact on what
th e eth n o g ra p h e r can o r ca n n o t study. M en, for exam ple, would not be able to carry out
particip a n t ob serv a tio n with M uslim w om en in purdah, sin ce th ey would be prohibited
from b ein g in th e sam e room as th em . Sim ilarly, m u ch o f th e d iscu ssion on gender has also
H O W TO D O R E S E A R C H IN P R A C T IC E

exam in ed the ob stacles that gender stereotypes create for w om en research ers, w ho have
often been restricted to the study o f d om estic realm s involving o th er w om en, ch ild ren , and
elderly people (H am m ersley and A tkinson 2 0 0 7 ). The g en d er o f th e research er can th us b o th
help and restrict access to different areas o f social life, d epen d ing upon th e cultural n o rm s o f
the in form ants. W hat you can and can’t do as a m an or w om an (o r w here you can and can’t
go) is m ore fixed in som e cultures th an in o th ers, and in m any cultures th ere is a certain
am ount o f elasticity in gender roles that creates lots o f individual variation.
The g ender o f the research er can also have a less obvious im pact, and in flu en ce the way in
w hich in form ants respond to questions.
In addition, the eth nicity and nationality o f the researcher can also in fluen ce how in fo rm ­
ants react. Som etim es it can actually be an advantage if the eth nographer belongs to a different
eth nic or national group than the inform ants. For exam ple, H annerz (1 9 6 9 ) carried out
research in a black ghetto in the United States, and com m ented that although he was often
jokingly referred to as ‘blue-eyed blond devil’ the fact that he was Swedish rather than A m e ri­
can helped to distance him from oth er whites, who were not favourably received by his in fo rm ­
ants. Similarly, an Indian Brahm an studying Indian D alits, or an upper-class English m an
studying Welsh coalm iners m ight take m uch m ore ‘baggage’ into the field with them than
would som eone from a different country less closely tied to the people they were studying.
The extent to w hich the eth nographer shares social characteristics with th e in form ants
can th erefore be both a help and hind ran ce to building a relationship, depen ding upon the
context. It is not therefore that the eth nographer should allow th ese factors to d eterm in e
what they do in the field, but rather that they should be aware o f how they m ight influence
both what they can do and how people respond to them .
C ritics o f ethnographic research often challenge it as a m ethod w hich is in herently su b jec ­
tive (see C hapter 3). Accordingly, these subjective influences can affect what type o f data is
collected, how it is collected, and from whom it is collected , all o f w hich m eans that the data
collected and the findings that com e from them may tell us m ore (or m ore than we would
ideally like) about the researcher who carried out the p roject than about the people or place
that were actually being studied. This creates a serious problem in term s o f rep licab ility,
which is one o f the hallm arks o f robust research (see C hapter 2). If different researchers
studying the sam e phenom enon in the sam e place com e to different conclusion s, how can
the findings from either be trusted? In what sense, if any, can the findings from eth nographic
research therefore be regarded as internally valid?
Political ethnographers have long struggled to answer these questions (Schatz (ed.) 2009;
Wedeen 2010). For som e the task becom es too daunting, and so they retreat ever further into
self-reilexivity, producing ‘descriptive analysis o f the m ost lim ited, self-referential s o r t—
explanation phobic (C om arott 2010: 526) and reducing ‘ethnography to a solipsistic literary
practice, one so obsessively reflexive as to be of no interest to anybody outside o f itself’
(C.omaroH 2010: 325). For others, the only way to get around these problem s is to incorporate
ethnographic research within other m ethodologies, so for scholars like Laitin (2003, 2006)
narrative approaches (such as ethnography) are by themselves inadequate, but when co m ­
bined with large-N statistical work and formal models . . . can help generate robust findings’
(Wedeen 2010: 259). See also discussion in Pachirat (2009) and H op f(2006). Both these views
represent somewhat extrem e positions. And whereas it is probably true that subjectivity is
inherent in ethnography, as it is to a greater or lesser extent in all social research, particularly
 E T H N O G R A P H Y A N D P A R T IC IP A N T OBS ER V AT IO N

qualitative research, this should not be overstated. As we discuss in Chapter 3, no approach or

type o f research ensures a com pletely value-free process o f inquiry, and none can therefore
free researchers from the need to be explicit and self-critical concerning their own underlying
assum ptions and values and behaviour, and how this might influence the research process.
But this self-criticism is only o f any use if it helps to guide the research process.

Recording observations
E thnography involves two d istinct but related processes: collecting data through participant
o b serv ation , and record in g and w riting up that data. The quality o f data that is collected in
an eth nography rests not only on how the fieldwork was conducted, but also on how the
fieldwork was recorded. Som e form s o f data can be relatively easy to record. For exam ple, an
interview or speech can be digitally recorded and transcribed (see C hapter 11). But other
types o f data are m o re difficult to record, and the distin ction betw een recording or preserv­
ing data and analysing that data b ecom es blurred. Indeed, in m any ways it is alm ost un a­
voidable that research ers will in terpret th eir data— and to a certain extent construct their
data— b ecause as th ey w rite up and record what they have seen or heard, the process of
record in g d ata is filtered th rou gh the ethnographer. It is not th erefore a sim ple record or
r epresentation o f ob jectiv e in form ation , but a con stru ctio n o f that in form ation that is in part
in terp retation , part analysis, and part ob serv ation . In this section we exam in e how different
types o f d ata can be record ed in preparation for analysis.

Fieldnotes

F ie ld n o te s are th e tra d itio n a l way o f record in g ob serv ation al and interview data in e th ­
nography. O rig in a lly th ey w ere h an d w ritten in field jo u rn a ls, but now th ere is a wide v ari­
ety o f c o m p u ter softw are program s that can be used to record and organize fieldnotes. For
a d etailed d isc u ssio n a b ou t th e p ro d u ctio n o f field n otes, see E m e rson et al. (1 9 9 5 ) and
S a n jek (1 9 9 0 ). T h e w ritin g o f field n otes is so m eth in g that needs to be done with great care
and a tte n tio n . P o o r q u a lity n o te -ta k in g can sev erely h am p er th e research p rocess, and
u n d e rm in e th e study. It is im p o rta n t th at n o tes are co m p reh en siv e, but at th e sam e tim e
field n otes are always selectiv e. It is no t possible to record ev erything, and so the eth n o g ra­
p her m u st p rio ritiz e w hat is in terestin g or relevant to th e research p roject. As H am m ersley
and A tk in so n ( 2 0 0 7 ) n o te, takin g g ood fieldn otes is not ju st about w riting down what you
see and hear, b u t a b o u t k n o w in g w hat to w rite dow n , how to w rite it dow n, and w hen to
w rite it dow n.
Th e when is ex trem ely im p ortant. As a general rule, th e quality o f notes d im inishes with
tim e, and th e resea rch er should th erefo re aim to m ake notes as soon as possible after the
ob serv ed a ctio n or con v ersation has taken place. D etails can easily be forgotten or confused,
and th e lon ger th e passage o f tim e betw een th e event and the record o f the event, the greater
the lik elih oo d th at th is will happen. Ideally, th en, notes should be m ade in real tim e, as
events or con v ersa tion s are takin g place. B ut th is is not always practical, and can be d isrup­
tive to th e a ctiv ity o f ca rry in g out particip an t ob serv ation . It is often necessary to wait until
a tim e w hen th e resea rch er is no t d irectly involved in carry in g out the research, and can
spend tim e w ritin g up th e d a y s activ ities and expandin g u pon any b rie f notes that have been
H O W TO D O R E S E A R C H IN P R A C T IC E

jotted dow n d uring th e cou rse o f th e day. B ern ard ( 2 0 0 6 :3 8 7 ) reco m m en d s sp en d in g tw o to

th ree ho urs p er day d oin g this.
In som e sense, field jo u rn als and recordings provide th e raw data th at th e research er then
uses for analysis, though the d istin ction betw een data and analysis is no t alw ays clear-cut.
B ernard (2 0 0 6 ) distingu ishes betw een d escriptive notes and analytical notes, th ou gh in p rac­
tice there is likely to be considerable overlap betw een th e two. In preparation for w riting up
and analysing field notes, there are now m any softw are program s that can be used to in dex
and cod e the data. It is im portant to draw o n the full breadth o f the fieldw ork data w hen w rit­
ing up the results, rather than selectively using only the data and evidence that is conv en ient
to the argum ent, and m odern softw are program s can help to ensure that th is is d one m ore
systematically.
O n e o f the m ain p roblem s with eth nograp hic research is th at it is very difficult for the
reader to know w hether th is has been done, w hich m akes it difficu lt to assess th e reliability
and validity o f the research. In recent years th ere has b een som e d iscu ssion o f w hether
eth nograp hic data should be d eposited in archives and m ade available, sim ilar to th e stan d ­
ard practice with survey and in terview data. This w ould help to rem ove th e veil o f secrecy
that surrounds m uch o f fieldw ork and eth n og rap h ic data co lle ctio n . The advantage o f th is
is that it would create a firm er d istin ction betw een data and analysis. People w ould b e able
to see how data was collected , what data was collected (and what was no t) and how it was
then analysed. This would create a stockpile o f evidence and in form ation that could be
reanalysed— perhaps with differing conclusion s from the original au th ors. This could help
m ake better use o f data w hich are already collected. D oing this would no doubt im prove the
transparency o f data collection , and reduce the probability o f eth nographers ‘m ak ing up’
evidence to fit their argum ent.
For exam ple, the 2 0 0 4 - 7 Sh iftin g Secu rities eth nography p roject, led by M arie G illespie
(m entioned earlier), w hich investigated ord inary p eoples un derstandin gs o f security, was
required by its funding body to deposit all eth nograp hic transcrip ts and reports on a secure
website ( http://ww w.m ediatingsecurity.com). As a collab oratio n betw een 18 e th n o g ra ­
phers, the project website becam e a resource for the p roject team itself to exchange in fo r­
m ation, prelim inary papers, and o f course the notes and reports o f oth ers. This allowed for
the coord ination o f research qu estions and em ergin g hypotheses across the team . It also
enabled the team leader to pay attention to issues o f internal validity w ithin the study, since
patterns and outliers could be observed as findings cam e in across the 18 individual e th ­
nographers’ studies. Moreover, it allowed researchers w orking across the fields o f political
and social science to access this data, com pare it with th eir own studies, and use that data
to develop their own projects. It was im portant that the Shifting Secu rities website required
secured access. For ethical reasons (see C hapter 7), this enabled the project leader to restrict
the transcripts and reports to selected guest researchers and m aintain ethical fidelity to the
in form ants.
However, there is also considerable resistance and opposition to this developing trend.
O ne objection is based purely on a matter o f scale. Dingwall (1992: 169) wrote, ‘it is no more
possible to reproduce all the data than it is for a film -m aker to show every inch o f film. . .' .
But even selective appendices present problems. There is the issue o f confidentiality, and
whether or not the identity of inform ants would be com prom ised or not. There is also the
issue of how the data would be used, and whether it is (or could be) m eaningful to people
 E T H N O G R A P H Y A N D P A R T IC IP A N T O B S ER V A T IO N

who did not w rite the notes and do the research them selves. Despite these objections, there
is grow ing pressure am ong research cou n cils and funding bodies to m ake at least som e
form s o f eth nog rap hic data m ore publicly available. But the m onograph is still an im portant
sou rce o f data in its own right.

Conclusions
Ethnographic methods and participant observation have tended to be overlooked as a research tool
in politics and IR. However, this is beginning to change and in recent years there has been a renewed
interest in political ethnography. Ethnography and participant observation are well suited to the inves­
tigation of political phenom ena that other more formal techniques are ill-equipped to examine, such
as hard-to-define topics or hard-to-access groups, like guerilla movements, revolutions, riots, racisim.
and terrorism.
Participant observation is also well suited to uncovering what people actually do, particularly
when, for whatever reason, what people say they do might be different from what they actually do. If
you want to know how a riot broke out, you can ask people afterwards, speak to witnesses, and even
speak to participants themselves. But this will provide a different, potentially less reliable source of
data, than actually observing first hand what happens. Participant observation can produce two
im portant sorts of data. The first is to do w ith what people say. This can either be in the form of
answers to questions (like surveys and interviews and focus groups), or revealed statements that come
to light in the course of other events. But perhaps the key advantage of participant observation over
all other methods is the observational data that it reveals.
However, despite these undoubted strengths of participant observation, it is very difficult to assess
the quality of data generated. The problem with participant observation is that it is very difficult to tell
w hether the data has been collected rigorously or not. W e generally only know how long the
researcher was in the field, the location of the field site (though sometim es this is anonym ized), and
the group of people among w ho the researcher w orked. M ore methodological appendices would
help to establish the reliability and validity of the research, and how far it could be generalized to
other contexts. Some of these problems can be overcom e. Carrying out com parative ethnography, or
replicating existing ethnographies in other contexts, can help to establish the extent to which findings
can be generalized. Com binin g ethnography with other methods of data collection can be used to
derive testable hypotheses from the ethnographic data, w hich can then be examined more generally,
using, for exam ple, survey data.

Questions
• W hat are the particular strengths of ethnographic study? W hen might it be an appropriate method
of analysis? W hen might it not be so helpful?
• To w hat extent does ethnography reflect the ethnographer's own w orldview rather than that of the
people it purports to represent?
• Is ethnography only concerned w ith description? How can it be used to develop explanations?

• Is ethnography alw ays biased? If so, does it matter? If not, how can w e avoid or m in im ize bias?

• A re the lim its of ethnographic generalization overstated?

• Should ethnographers deposit their fieldnotes in data archives?

• W hat are the ethical obstacles to doing participant observation? Are these an unnecessary incon­
venience?
• H ow might ethnography and survey data be m eaningfully com bined?
H O W TO DO R E S E A R C H IN P R A C T IC E

Guide to Further Reading

Bernard, H. (2006), Research M ethods in Anthropology: Qualitative an d Quantitative Methods
(Lanham, MD: Altamira Press).
A very detailed and com prehensive account of the different aspects o f ethnography, and the ways in
which ethnographic research can be com bined w ith other research strategies and methods. Although
w ritten prim arily for anthropologists, it is full o f engaging examples and is an extrem ely useful
resource for all students interested in political ethnography.

Fenno, R. (1978), Home Style: House Members in their Districts (New York: Harper Collins).
This is one of the classic ethnographies in political research. Fenno conducted research w ith 18
members of Congress as they travelled around their constituencies, spending tim e w ith them, and
observing what they did and who they spoke to on a day-to-day basis. The book offers fascinating
insights into how congressmen interact with their constituents, and how this translates into their
legislative effectiveness.

Hammersley, M. and P. Atkinson (2007), Ethnography: Principles in Practice (London: Routledge).

A very clear and com prehensive book on the different aspects of ethnographic research. It contains
everything you need to know, w ith informative and detailed chapters on specific issues such as access,
field relations, writing ethnography, and ethics. A must read.

Michelutti, L (2008), The Vernacularisation o f Democracy: Politics, Caste and Religion in India
(London: Routledge).
This is a fascinating ethnography of everyday practices and beliefs about dem ocracy among a
historically marginalized caste group in Northern India. The book clearly illustrates what ethnography
can contribute to the study of politics. Michelutti examines how culture and social practices inform
politics and vice versa In doing so. she examines how ideas about dem ocracy get ingrained in
domains of life, such as marriage, kinship, and religion, which in turn serves to 'vernacularize'
democratic politics.

Schatz, E. (ed.) (2009), Political Ethnography: What Immersion Contributes to the Study o f Power
(Chicago and London: University of Chicago Press).
This book contains a collection of essays on how ethnographic approaches have been used to study
a variety of issues in political research, with chapters on topics such as ethnic nationalism, civil war,
public opinion, and authoritarianism.

© References
Auyero, J (1999). From the Client's Point(s) of View: Bernard. H. (2006), Research Methods in
How do Poor People Perceive and Evaluate Anthropology: Qualitative and Quantitative
Political Clientelism?'. Theory and Society 28 Methods (Lanham, MD: Altamira Press).
297-334
Blee, K and A Currier (2006), How Local
Ashforth A (2005). Witchcraft. Violence, and Social Movement Groups Handle a Presidential
Democracy in South Africa (Chicago: University Election’. Qualitative Sociology 29:
of Chicago Press) 261-80
Bale T and C Dann (2002). 'Is the Grass Really Brewer. J (2000), Ethnography (Buckingham: Open
Greener The Rationale and Reality of Support University Press)
Party Status A Case Study. Party Politics 8(3):
Bryman. A (2004), Social Research Methods (Oxford
349 65
Oxford University Press)
Barnett M (2006j. fyewiMrss to a Genocide The
Comaroff. J L (2010). The End of Anthropology.
United Nntiom and Rnanda (Ithaca NY Cornell
Again Toward a New In/Discipline'. American
Anthropologist 112(4) 524-38
 E T H N O G R A P H Y A N D P A R T IC IP A N T O B S E R V A T IO N

Comaroff.J. and J. Comaroff (2003), 'Ethnography 1955 and 1999 (Ithaca, NY: Cornell University
on an Awkward Scale: Postcolonial Press).
Anthropology and the Violence of Abstraction'. ---- (2006), 'Ethnography and Rational Choice in
Ethnography 4:147-79.
David Laitin: From Equality to Subordination to
Dahl, R. (1961), Who Governs?: Democracy and Absence', Qualitative Methods 4(1): 17-20
Power in an American City (New Haven, CT: Yale Humphreys, L. (1970), Tearoom Trade (London:
University Press). Duckworth).
Delamont, S. (2004), 'Ethnography and Participant Ingold, T. (2008), Anthropology is Not Ethnography1.
Observation', in C. Seale, G. Gobo.J. Gubrium, Proceedings of the British Academy 154: 69-92.
and D. Silverman (eds), Qualitative Research
Karlstrom, M. (1996), 'Imagining Democracy:
Practice (London: Sage Publications).
Political Culture and Democratization in
Dingwall. R. (1992), 'Don't Mind H im -He'sfrom Buganda', Africa 66: 485-505.
Barcelona: Qualitative Methods in Health
Laitin, D. D. (2003), The Perestroikan Challenge to
Studies', in J. Daly, I. MacDonald, and E. Willis
Social Science’, Politics and Society 31(1):
(eds) Researching Health Care: Designs. Dilemmas
163-84.
and Disciplines (London: Routledge).
---- (2006), 'Ethnography and/or Rational Choice:
Eliasoph. Nina (1997), Avoiding Politics: How
A Response from David Laitin', Qualitative
Americans Produce Apathy in Everyday Life
Methods 4(1): 26-33.
(Cambridge: Cambridge University Press).
Latour, Bruno (2010), The Making of Law: An
Emerson, R„ R. Fretz, and L Shaw (1995), Writing
Ethnography of the Conseil d'Etat (Cambridge:
Ethnographic Fieldnotes (Chicago: University of
Polity).
Chicago Press).
Leach, E. (1961), Rethinking Anthropology (London:
Fenno, R. (1978), Home Style: House Members in
Athlone Press).
their Districts (New York: Harper Collins).
Malinowski, B. (1922), 'Introduction: The Subject,
---- (1990), Watching Politicians: Essays on
Method and Scope of this Inquiry', Argonauts
Participant Observation (Berkeley, CA:
of the Western Pacific: An Account of Native
Institute of Governmental Studies, University of
Enterprise and Adventure in the Archipelagoes of
California). Melanesian New Guinea (London: Routledge).
George, A. and A. Bennett (2005), Case Studies 2-25.
and Theory Development in the Social Sciences
Marcus, G. (1995), 'Ethnography in/of the World
(Cambridge, MA: MIT Press). System: The Emergence of Multi-Sited
Gillespie, M. and B. O'Loughlin, B. (2009), 'News Ethnography', Annual Review o f Anthropology
Media, Threats and Insecurities: An Ethnographic 24:95-117.
Approach', Cambridge Review o f International Michelutti, L (2008), The Vernacularisation of
Affairs 22(4): 667-86. Democracy: Politics, Caste and Religion in India
---- , J. Gow, and A. Hoskins (2007), 'Shifting (London: Routledge).
Securities: News Cultures Before and Beyond the Pachirat, T. (2009). The Political in Political
Iraq Crisis 2003: Full Research Report', ESRC End Ethnography: Dispatches from the Kill Floor',
of Award Report, RES-223-25-0063 (Swindon: in E. Schatz (ed ). Political Ethnography. What
ESRC). Immersion Contributes to the Study of Power
Hammersley, M. and P. Atkinson (2007), (Chicago and London: University of Chicago
Ethnography: Principles in Practice (London: Press).
Routledge). Paley.J. (2001), Marketing Democracy: Power
Hannerz. U. (1969). Soulside: Inquries in Ghetto and Social Movements in Post-Dictatorship
Culture and Community (New York: Columbia Chile (Berkeley, CA: University of California
University Press). Press).

Hartmann. D. and D. Wood (1990), 'Observational Parnell, P. and K. Kane (eds) (2003), Crime's
Methods', in A. Bellack, M. Hersen, and A. Kazdin Power: Anthropologists and the Ethnography
(eds), International Handbook o f Behaviour of Crime (New York and Basingstoke: Palgrave
Modification Therapy (New York: Plenum). Macmillan).

Hopf. T. (2002), Social Construction o f International Parsons. C. (2010). 'Constructivism and Interpretive
Politics: Identities and Foreign Policies, Moscow Theory, in D. Marsh and G. Stoker (eds), Theory
H O W T O D O R E S E A R C H IN P R A C T IC E

and Methods in Political Science (Basingstoke: Stacey, M„ E. Batestone, C. Bell, and A. Murcott
Palgrave Macmillan). (1975). Power, Persistence and Change: A Second
Pieke. F. (1995). Witnessing the 1989 Chinese Study o f Banbury (London: Routledge and Kegan
People's Movement, in C. Nordstrom Paul).
and A. Robben (eds), Fieldwork Under Fire: Tilley. C. (2006), Afterword: Political Ethnography
Contemporary Studies of Violence and Survival as Art and Science', Qualitative Sociology 29:
(Berkeley and Los Angeles: University of 409-12.
California Press). Wacquant, L (2003), 'Ethnografeast: A Progress
Pouliot, V. (2007), 'Pacification without Report on the Practice and Promise of
Collective Identification: Russia and the Ethnography', Ethnography 4:5 -14.
Transatlantic Security Community in the Weaver, C. (2008), Hypocrisy Trap: The World
Post-Cold War Era', Journal of Peace Research Bank and the Poverty o f Reform (Princeton, NJ:
44: 605-22. Princeton University Press).
Riemer, J. (1977), Varieties of Opportunistic Wedeen, L (2010), 'Reflections on Ethnographic
Research', Urban Ufe 5:467-77. Work in Political Science', Annual Review of
Sanjek, R. (1990), 'Ethnograph/, in A. Barnard and J. Political Science 13:255-72.
Spencer (eds), Encyclopedia o f Social and Cultural Whiting, J. (1982), 'Standards for Psychocultural
Anthropology (London: Routledge). Research', in E. Hoebel, R. Currier, and S. Kaiser
Schaffer, F. (1998). Democracy in Translation: (eds), Crisis in Anthropology: View from Spring Hill
Understanding Politics in an Unfamiliar Culture (New York: Garland).
(Ithaca, NY: Cornell University Press). Whyte, W. (1993), Street Corner Society: The
Searing, D. (1994), Westminster's World: Social Structure o f an Italian Slum (Chicago
Understanding Political Roles (Cambridge. MA: and London: Chicago University Press). First
Harvard University Press). published 1943.
 Textual Analysis

Chapter Summary
Texts have always been a m ajor source of information and evidence for political
researchers. This chapter discusses two forms of textual analysis that have become
increasingly prom inent in Politics and International Relations research discourse
analysis and content analysis. It also offers a brief discussion of using documents,
archival sources, and historical writing as data. Among the questions the chapter ad­
dresses, are the following:

• W hat is a discourse?

• W hat can an analysis of discourse reveal about the social w orld?

• W hat is content analysis?

• H ow do discourse analysis and content analysis differ?

• W hat are the differences between qualitative and quantitative content analysis?

• W hat procedures are involved in both quantitative and qualitative content

analysis?

• W hat special considerations need to be made in using documents, archival

sources, and historical w ritin g as data?

Introduction

G o v ern m en t reports, political pam phlets, new spapers, and oth er texts have long provided
im p ortan t sou rces o f in fo rm atio n for students o f politics. However, in recent decades, grow ­
ing aw areness o f th e im p o rtan ce o f language and m eanin g for p olitical analysis and o f the
pow er o f the m ass m edia has produced a dram atic upsurge o f interest in textual analysis,
n ot on ly in P olitics and In tern ation al R elations, but th rou ghou t th e social scien ces. In p o lit­
ical research , two form s o f textual analysis have b ecom e particularly prom in ent: discou rse
analysis and co n ten t analysis.
D is c o u r s e a n a ly sis is a qu alitativ e typ e o f analysis that ex p lores th e ways in w hich
d isc o u rse s give le g itim a cy and m e a n in g to so cial p ra ctice s and in stitu tio n s. Discourses
c o n s ist o f e n s em b les o f id eas, co n c e p ts, and categ o ries th rou gh w hich m e an in g is p ro ­
d uced and rep ro d u ced in a p a rtic u la r h isto rica l situ atio n . The elem en ts o f a d isco u rse
ca n b e b ro u g h t to lig h t th ro u g h an alysin g th e lan guage, sem io tics (laten t m e an in g in
te x t), and c o n v e n tio n s fou n d in a v ariety o f w ritten , oral, and visual ‘tex ts’. But w hile
tex tu a l a n a ly sis ca n rev eal th e e lem en ts o f a d isco u rse, th e meaning th at th ey prod u ce or
rep ro d u c e ca n o n ly b e u n d e rsto o d in rela tio n to so m e b ro ad e r co n tex t. C on sequ en tly,
d isc o u rse a n a ly sis is c o n c e rn e d w ith an alysin g, no t ju s t th e tex t itself, but th e relatio n o f
a tex t to its c o n te x t (its so u rc e , m essage, ch a n n e l, in ten d ed au d ie n ce, co n n e c tio n to o th er
3 10 H O W T O D O R E S E A R C H IN P R A C T IC E

tex ts and ev en ts), as well as th e b ro a d e r r elatio n s o f pow er and a u th o rity w h ich sh ap e

th at c o n tex t.
In contrast to d iscou rse analysis, c o n te n t a n aly sis is co n ce rn e d w ith the study o f th e text
itself, rather than with the broad con text w ithin w hich it was produced. T h is analysis can be
either quantitative or qualitative. The aim o f quantitative content analysis is to draw in fer­
ences about the m eanin g and in tention o f a text th rou gh an analysis o f th e usage and fre­
q u ency o f words, phrases, and im ages, and th e pattern s th ey form w ithin a text. Qualitative
content analysis is a m ore interpretive form o f analysis co n ce rn e d w ith u n cov erin g m e a n ­
ings, m otives, and purposes in textual content.
Q ualitative content analysis and d iscou rse analysis have m u ch in co m m o n . But a discou rse
analyst and a content analyst will tend to approach the exploration o f an issue in som ew hat
different ways.
C onsid er, for exam ple, issues o f race and eth nicity, w hich th e upsurge in id en tity p olitics
(th e politicization o f eth n ic, regional, gender, and sexual id en tities) has m ade a key focu s o f
research in recent years. A d iscou rse analyst ex ploring th ese issues m ight ex am in e how
eth n ic and ‘racial’ inequality is expressed, enacted and reprodu ced by d iscou rse as on e o f
the practices o f a racist so cie ty ’ (van D ijk 2 004: 3 5 4 ). In his studies o f racism , Teun van D ijk
investigates a recurrin g set o f ‘underlying m ental m odels and social rep resentations’ in
‘influential public discou rses’ (th ose o f elites and elite in stitu tio n s), and attem pts to u n d er­
stand the cond ition s, consequ ences, and fu n ction s o f th ese d iscu rsive elem ents by ex p lor­
ing the contexts in w hich they are produced (van D ijk 2 0 0 2 ,2 0 0 4 ). A con ten t analyst would
be m ore typically c oncerned with identifying e th n o cen tric and racist representations in the
content o f mass m edia, literature, and film. For exam ple, in order to find out how th e m edia
represented W hites, Blacks, and Latinos as crim e victim s, Travis D ixon and D aniel Linz
(2 0 0 0 ) cond u cted a content analysis o f a random sam ple o f television news p rogram m es
aired in Los Angeles and O range C oun ties. T h eir exam in ation o f the frequen cy o f certain
words, phrases, and im ages, revealed that Whites are m ore likely than A frican A m erican s
and Latinos to be portrayed as victims o f crim e on television news; and that Blacks and Lati­
nos are m ore likely than W hites to be portrayed as lawbreakers than as crim e v ictim s. W ith
these data as a basis, the authors were th en able to explore the th eoretical im plications o f
these findings.
W hat this example shows is that, despite sim ilarities, d iscourse analysis and content analysis
offer researchers a means o f asking and answering different types o f questions. In the next sec­
tion, we discuss these two forms o f textual analysis in greater detail, in cluding the types o f
research questions they address, what they can reveal about the social world, and the p roce­
dures that each entails.

Discourse analysis
Discourse analysis is an interpretive and constructivist form o f analysis that draws on diverse
theoretical and m ethodological approaches from linguistics, anthropology, and sociology.
As an approach to understanding political phenom ena, discourse analysis is in terpretive.
In com m on with interpretivist approaches (sec Chapter 2), it assum es that people act on the
basis ot beliefs, values, or ideology that give meaning to their actions; and that to understand
 T E X T U A L A N A L Y S IS

political behaviour, we must know about the m eanings that people attach to what th ey’re
doing. C onsequently, its aim is to reveal the m eanings that the political world has for agents
who participate in it and that give people reasons for acting.
But discou rse analysis is also con stru ctiv ist: it assum es not only that people act towards
ob jects, including people, on the basis o f the m eanings which those o b jects have for them , but
that these m eanings are socially and discursively constructed. The aim o f discourse analysis,
therefore, is not only to reveal m eanings through an exam ination o f the language and d is­
course we em ploy in our in teraction s, but to uncover how discursive practices construct
m eanings through the production, dissem ination, and consum ption o f various form s o f
texts, including form al written records, T V program m es, advertisem ents, and novels.

Speech act theory, post-structuralism, and critical discourse analysis

To fu rth er elaborate on th is constru ctiv ist d im ension o f discou rse analysis, it is useful to
briefly consid er som e key ideas on which the constructiv ist understanding o f discourse is
based.

Speech act theory

The n o tio n that m ean in gs are d iscu rsively con stru cted is based on a fun dam en tal assu m p­
tion o f d isco u rse analysis: that language is a m edium orientated tow ards actio n and fu n c ­
tio n ; th at w hen we speak (o r w rite), we do so, not only in order to say som eth in g, but also
to do som eth in g . This id ea— that language is used as m uch to do th ings as to m ake state­
m e n ts— lies at the h eart o f sp eech a ct th eo ry , an approach to the exp lanation o f language
p ion eered by th e p h ilosop h ers John L. A ustin and John Searle in the 1960s. ‘The issuing of
an u ttera n ce’, w rote Joh n A ustin, ‘is th e p erfo rm in g o f an a ctio n ’ (1 9 6 2 : 6 ); or, as the
A ustrian p h ilosop h er, Ludwig Jo s e f Joh an n W ittgen stein put it: ‘W ords are deed s’ (1 9 8 0 :
4 6 ). All d isco u rse-a n a ly tic ap p roaches are based on th is fun dam en tal idea. Thus, discou rse
a nalysis is c o n c e rn e d w ith language, no t as an ab stract system , but as so m eth in g that p e o ­
ple use to do th ings. People use language to p rom ise, th reaten , insult, plead, and dem and.
M ore broadly, ce rta in th in g s b eco m e real th rou gh language: ‘attitu d es’, poin ts o f view, par­
ticu la r ‘su b jec t p o sitio n s’ (ways in w hich o b je cts or people take up or are placed in p a rtic u ­
lar p o sitio n s), en tities, sta tes-o f-affair (w hat is going o n ), events, processes, activ ities, and
relatio nship s.
So people use language to create different kinds o f social and cultural m eaning, and to
Discourse is ‘a system o f texts
c o n stru ct different a ccou n ts, or versions, o f the social world.
th at b ring s o b jec ts in to b ein g ’ (H ardy 2001: 26). It con stru cts social reality by helping to
c o n stru c t w hat is know able, sayable, and doable within a particular historical context. To
highligh t th e d istin ctio n furth er, we can th in k o f language as a set o f rules o f gram m ar and
logic that can prod uce in finite possible sen tences (language) or propositions (logic). H ow ­
ever, a d iscou rse is a system , or e nsem ble, o f ideas, concep ts, and categories, that is linked to,
and fu n ctio n s w ithin, a specific context. C onsequently, we can say that discou rse analysis is
an ex p loration o f ‘language in co n tex t’. It is not interested sim ply in how som eth in g is repre­
sen ted by language, but in how it is actu ally con stru cted by the ways in w hich it is intelligible
and legitim ate to talk ab out it in a p articu lar tim e and place.
H O W T O D O R E S E A R C H IN P R A C T IC E

Post-structuralism
The m ost im p ortant representative o f p ost-stru ctu ralist th inking, M ich el Foucau lt ( 1 9 2 9 -
8 4 ), focused attention on the various ways d iscou rses system atically form th e o b je c ts o f
w hich they speak’ (Foucau lt 1 9 7 2 :4 9 ). Th is w ork suggested a variety o f chan nels o f in qu iry
to explore the co n stru ctio n o f social reality throu gh d iscou rse, including:

1. how ways o f talking about a top ic are em bed ded in sets o f pow er relations;

2. how these pow er relations are supported by in stitu tio n s (asylu m s, gov ernm ents,
prisons, and scho ols) in particular historical con texts; and

3. how these in stitutio nal and historical con figu ration s o f discou rse con stru cted new
kinds o f hu m an subjects.

A particularly resonan t avenue o f in qu iry that Foucault op en ed up was the ex p loration o f

how reiterated key words and statem ents that recur acro ss texts o f all kin ds enab le and
d elim it fields o f know ledge and inquiry, and govern what can be said, th ough t, and d one
w ithin th ose fields. In his Archaeology o f Knowledge, Foucault analyses the h isto ry or g en ­
esis o f the ‘discursive p ractices’ that lay claim to revealing know ledge, show ing how the
d evelopm ent o f know ledge is intertw ined with the m ech an ism s o f (p olitical) power. ‘D is­
course’ can be understood in this con text as a system o f statem ents, expressing rules, roles,
and boun d aries that form a body o f knowledge. For in stance, we can speak o f ‘discou rses o f
p olitics’, ‘d iscou rses o f dem ocracy ’, or ‘authoritarian p olitics’. In Encountering Development
(1 9 9 4 ), A rturo E scobar traces the em ergen ce o f a ‘developm ent discou rse’ (‘d evelopm ental-
ism ’) after W orld W ar II, and shows how this d iscou rse c on stru cted a particu lar rep resen ta­
tion o f the Third W orld that, over tim e, ‘created a space in w hich on ly certain th ings could
be said and even im agined’ (1994: 39).
Poststructuralist theory questions w hether there are essential hum an subjects, individual
agents, and social realities independent o f their dynam ic historical con stru ction in social
and cultural discourses. For instance, Foucault asks w hether the natural and social worlds
are indeed knowable, accessible, and analysable without recourse to the constitutive forces
o f discourse. He did not think that there were definite underlying structures that could
explain the human condition ; and he thought that it was im possible to step outside o f d is­
course and survey the situation objectively.
Building on this perspective, p ost-structu ralist discourse theory assum es that the way
people talk about the world does not reflect som e objective truth about the world, but the
success o f particular ways o f thinking and seeing. These ways o f thinking and seeing b ecom e
invisible because they are assumed to be truthful and right. In this way people’s thought
processes themselves can com e to represent and reinforce particular regim es o f power and
coercion.

Critical discourse analysis

Posl-siructural thinking made im portant contribu tion s to the field o f discourse studies.
However, it came under attack for its tendency to treat reality as little m ore than som ething
Lonsiituted in and through discourse, and to ignore pre-existing social structures and power
 T E X T U A L A N A L Y S IS 313

relationships. I his line of critiq ue was developed in what cam e to be known as critical d is­
course analysis’.
C ritical d iscou rse analysis (C D A ) is critical’ because il seeks to expose conn ectio ns
betw een language, power, and ideology (Fairclough 2 0 0 1 :4 ). It is principally concerned with
the role o f discou rse in enacting, reproducing, and resisting social power abuse, dom inance,
and inequality (Van D ijk 2001: 300, 352).
H ypotheses linking discourses to power can be investigated through an analysis o f how
powerful groups control public discourse, and o f how discourse controls the m inds and actions
o f less powerful groups, and the social consequences o f such control (Van D ijk 2001: 355).
D iscursive pow er— control over a d iscou rse— is a cru cial constituent o f social power, and
a m ajor m eans o f reprodu cing d om in an ce and hegemony. M em bers (and particularly lead­
ers) o f m o re pow erful social groups and institutions

have m o re o r less exclu sive access to, and co n tro l over, o n e o r m ore types o f p ublic d iscourse.
Thus, p rofesso rs co n tro l sch o larly d iscourse, teachers ed ucatio n al d iscourse, jo u rn a lists m edia
d iscou rse, law yers legal d iscou rse, and p o litician s policy and o th e r public political discourse.
T h ose w ho have m ore c o n tro l over m o re — and m ore in flu en tia l— d iscourse (and m ore d is­
c ou rse p rop erties) are by that d efin itio n also m ore pow erful. (V a n D ijk 2 0 0 1 : 35 6 )

C on trollin g peoples m ind s is ano ther cru cial constituent o f social power. Those groups who
con trol the m ost influential discou rses also have m ore chan ces to control the m inds and
a ctio n s o f oth ers, because people tend to accept beliefs, knowledge, and opin ion s ‘through
d iscou rse from what they see as authoritative, trustworthy, or credible sources, such as
scholars, experts, professionals, or reliable m edia . . . ’. M oreover, ‘in som e situations p artici­
pants are obliged to be recipients o f discou rse, e.g. in education and in m any jo b situations’
(V an D ijk 2 0 0 1 : 35 7 ).
C D A d eveloped in a sso ciatio n w ith a critiq u e o f p o st-stru ctu ral and o th er d iscou rse
a na ly tic ap p roach es. But it has also been th e target o f critiq u e. A key asp ect o f C D A that
d istin g u ish es it from o th er form s o f disco u rse analysis is th e degree to w hich the analyst
d ep en d s on a p rio r th eo retical p ersp ective to analyse the data. For C D A , you start with a
th eo ry o f d o m in a n ce o r pow er, th en you explore m aterials in ord er to d iscov er how d is­
c o u rse p ro m o tes or ch allen ges o n e group’s pow er over an o th er (Fairclou gh 2 0 0 1 ). As Ted
H o p f p o in ts o ut, c ritic a l d isco u rse a nalysis ‘is in fact a p olitical th eo ry as m u ch as a m ethod
o f in q u ir y ’. It assu m es th at language is a m edium w ithin w hich p rev ailin g c on figu ration s
o f pow er are articu la ted and reprodu ced. D isru p tin g and ch allen gin g th ese pow er rela­
tio n s ‘is on e o f th e c en tra l featu res o f w hat we call p olitics. This m eans that th e m ean in g o f
any given tex t for DA often p oin ts to som e un d erly ing p olitical prob lem or q u estio n ’
( H o p f 2 0 0 4 : 3 0 ). T h is d ep en d en ce on prior th eo ry b rings it in to co n flict w ith th o se d is­
c o u rse analy sts w ho w ant to stay clo se r to th e m aterials in th eir data (see e.g. W eth erall
1 9 9 8 ; B illig and S c h e g lo ff 1999).

Analysis
W e have said th at a discourse is ‘a system o f texts that brings o b jects in to b ein g’ (H ardy 2001:
2 6 ) w ithin a p articu lar in stitutio nal and h istorical c o ntext. The goal o f discourse analysis is to
exp lore th e relationship betw een discou rse and reality in a particular context. Thus, the
314 H O W T O D O R E S E A R C H IN P R A C T IC E

hyp othesis to b e investigated will typically b e that th at th ere is a c o -v a riatio n o r association

betw een a d iscou rse and a given context.
T o investigate th is h yp othesis, th e analyst will ch o o se a d iscrete b od y o f w ritten w ork
(e.g. a new spaper over a specific period o f tim e, a set o f sp eech es) and c o n d u ct an analysis
o f what reality its language, m etap ho rs, and/or sym bols help to c o n stru ct. Texts are selected
to enable the analyst to explore a hyp othesized asso ciatio n betw een tex t featu res and c o n ­
tex t features, betw een som e chan ge in the c o n tex t or circu m stan ces and a system atic chan ge
in d iscou rse features that are the focus o f in terest in a study (L em k e 1998). T h e analysis
entails retrod u ctio n: reasoning backw ard from a p articu lar d iscu rsiv e p rod u ction to esta b ­
lish structure from its em pirical m an ifestation s. ‘It asks w hat th e co n d itio n s o f possibility
are o f th is or that particu lar d iscu rsive p rod u ction (LafFey and W eldes 2 0 0 4 : 2 8 ). (See
C hapter 2.)
Establishing a co-v ariation or association betw een d iscou rse and con text is on ly a first
step. A second step is to provide details o f the process th rou gh w hich th e pow er o f a d is­
course has d em onstrable effects.
Let’s give som e furth er consideration to each o f these two steps.

1. The context

All approaches to discou rse assu m e th at, to un derstand th e c o n stru ctiv e effects o f d is­
courses, research ers must place th em in th eir h isto rical and social co n texts. As we stated
previously, d iscou rse analysis is th e study o f language use in con text. D isco u rses co n stru e
aspects o f the world in in herently selective and reductive ways. The qu estio n to ask, th e re ­
fore, is: 'W h y this particu lar selection and red u ctio n, and why here and no w ?’ Thus, d is­
course analysis ‘produces its greatest insights w hen rich contextu al in fo rm atio n can be
factored into the analysis o f each text or episod e’ (L em ke 1998: 1185). But how do we
d eterm ine the relevant contextual factors o f a text or d iscou rse event? ‘C o n te x t’ is difficult
to d eterm ine with any precision . It is difficult to know what aspects o f co n text are p o te n ­
tially relevant to a textual analysis.
Let’s first consid er the different sorts o f con texts that can be explored. We can d istin ­
guish betw een local and broad con texts (T itsc h er et al. 2 0 0 0 ). The local co n text in cludes
the im m ediate task and situation, the sou rce, m essage, ch an n el, and intended au d ience o f
the com m u n ication . The broad con text consists o f cultural norm s and assu m ptions,
knowledge, beliefs and values, the resources and strategies ch aracteristic o f a co m m u n ity ’s
general cultural resources. A no ther way o f ch aracterizin g different con texts are through
the use o f the term s micro-discourse (specific study o f language) and macro-discourse.
Language use, discourse, verbal in teraction , and com m u n ication belong to the m ic ro ­
level of the social ord er; power, d om inan ce, and inequality betw een social groups are typ i­
cally term s that belong to a m acro-level o f analysis’ (van D ijk 2001: 3 5 4 ), the broad,
societal curren ts that are affecting the text, b eing studied. Teun van D ijk explains that ‘In
everyday interaction and experience the m acro and m icro level (and interm ed iary “m eso-
levels ) form one, unified whole, lo r instance, a racist speech in parliam ent is a d iscou rse
at the m icro-level of social in teraction in the specific situation o f a debate, but at the sam e
tim e may enact or be a constituent part o f legislation or the reprodu ction at racism , at the
macro level' (van Dijk 2001: 3341
 T E X T U A L A N A L Y S IS

In his studies o f discourse and racism , van D ijk focuses on communicative, interactional, and
societal contexts. W hat he is interested in investigating is how ‘influential public discourses’
(those o f elites and elite institutions) reflect sim ilar underlying mental models and social repre­
sentations’, and ‘sim ilar ways o f social interaction, com m unication, persuasion, and public o pin­
ion form ation’ (van D ijk 2002: 157). To understand the conditions, consequences, and functions
o f these discursive elem ents, he explores the com m unicative, interactional, and societal contexts
in which they are produced. Figure 13.1 represents the way in which these contexts operate.
Th ese co n tex ts to g eth er in clude biased or stereotypical new s, produced by jo u rn alists
and o th er profession als, u n d er the co n tro l o f ed itors, in m edia organizations. In the wider
society, th e co n ten t o f th ese texts is p rod uced, and also reprodu ced, by m em b ers o f many
d ifferent profession al and o th er social groups, and as part o f daily rou tines and p ro ce­
dures: legislative d ebates con d u cted by p o litician s; tex tb ook s, lessons, and scho larly pub ­
lica tio n s p rod uced by tea ch ers and scho lars. ‘R acist so cie ties and in stitu tio n s produce
ra cist d iscou rses, and racist d iscou rses reprodu ce th e stereotypes, preju d ices and id eo lo ­
gies th at are used to defend and legitim ize w hite d o m in an ce’ (van D ijk 20 0 4 : 354).

Social, Political Economic Context

Discursive Practices
(the production, distribution, consum ption of texts)

Texts

Context: Existence of Racism

Discursive Practices:
Pressures on Journalists

Texts: Newspapers

Figure 13.1 Text and context in discourse analysis

Source: Fairdough (1995: 59).
H O W T O D O R E S E A R C H IN P R A C T IC E

2. Analysing the process thorough which the power of a discourse

has demonstrable effects
In addition to establishing an association betw een discou rse and co n text, d iscou rse analysis
also seeks to show that d iscou rse has dem onstrab le effects. In ord er to do th is, d iscou rse
analysts have sought to d elin eate the process th rou gh w hich d iscou rse b eco m e s naturalized
so as to b ecom e co m m o n sen se— th e m echanism s th rou gh w hich it ‘fixes’ m ean in gs and
becom es naturalized. These m echanism s can best be u n derstood as a social process with two
d istinct d im ensions: articulation and in terpellation (W eldes 1996: 28 4 ).
The term ‘articulation’ refers to ‘the process th rou gh w hich m eanin g is prod uced out o f
extant cultural raw m aterials or linguistic resources’ (W eldes 1996: 2 8 4 ). ‘A rticu lation fixes
m eanings ‘through a process o f repeatedly establishing association s betw een different ele ­
m ents, so that these e lem ents com e to be seen as in herently or necessarily co n n ected and the
m eanings they produce com e to seem natural, to be an accu rate d escrip tion o f reality’
(W eldes 1996: 285).
In an investigation o f the process o f articulation , the research er would, first, id en tify the
m ain signifying elem ents o f the discou rse and how they are articulated to each other. For
instance, Stuart Hall has shown how n eo -lib eral discou rses w ere origin ally c on stru cted out
o f a chain o f discursive elem ents lin king ‘big g ov ernm ent’, ‘un em ploym en t’, and ‘welfare
state’, on the one hand; and ‘deregulation’, ‘free m arkets’, and ‘privatization’, on the o th er (H all
1988: 50). Investigation o f the process o f articulation would, th en, focus on how d iscou rses
are articulated to institutional form s (th in k tanks, political parties, and to m edia outlets) and
partake o f their power (H all 1988: 4 6 -7 ) .
In order to be successful, the representations that articulation establishes need to be
accom panied by a second dim ension o f the social process through w hich d iscou rses ‘fix’
m eanings and b ecom e naturalized: the process o f ‘interpellation’. It is through this process
that the acceptance o f specific representations is achieved.
W hat interpellation entails is the acceptan ce o f the ‘subject p osition s’ (th e placing o f p eo ­
ple in particular positions) that discourse constructs. Investigating processes o f in terp ella­
tion involves exam ining how a discourse constructs the identities o f subjects and o b jects,
and how these identities are then em bodied in the spoken, w ritten, and sym bolic texts, the
encounters, interactions, and inform al talk, o f in stitutional bureaucracies.
An example o f an analysis which explores processes through w hich the power o f a d is­
course has d em onstrable effects is N orm an Fairclough’s study o f discourse and social change.
Fairclough is concerned to explore the ways in which discourse '(re)co n stru cts’ social life in
processes o f social change (Fairclough 2005a). D iscourse, he argues, is a cru cial and irred u c­
ible’ elem ent in social change (Fairclough 2005b: 41). To construct an account o f how d is­
course operates as an elem ent o f social change, he identifies four ‘m om en ts’ in the dialectical
relationship ol discourse and social change (Fairclough 2005b : 4 2 - 3 ):

1 the translating and condensing o f com plex realities into new discourses constructed
through the articulation of elem ents of existing discourses ('em ergen ce');
2 . the contestation am ong discou rses— part o f the contestation between strategies
and between groups of social agen ts—which may lead to particular discourses (and
strategies) becom ing hegem onic (‘hegem ony’);
 T E X T U A L A N A L Y S IS

3. the d issem in ation o f discou rses and their recontextu alization in new organizations,
in stitutions, or fields, or at new scales (‘recontextu alization’); and

4. the ‘e nactm en t o f d iscou rses as new ways o f (in ter)actin g , their inculcation as new
ways o f being, or identities, th eir m aterialization in features o f the physical world’
( ‘op era tio n a liz a tion ’).

F airclough applies this schem a to an investigation o f the ‘discourse m o m en t’ in ‘the em er­

gence o f a new regim e o f in ternational security and the use o f force, as evidenced recently in
Kosovo, A fghanistan, and Iraq’: the effort to develop and diffuse a new hegem onic discourse
o f in ternational relations and in ternational security. The focus o f Fairclough’s analysis is the
con tribu tion o f one key ‘player’ in this p rocess: the U K Prim e M inister, Tony Blair. To explore
B la irs con tribu tion to elaboratin g a new d o ctrin e o f ‘in ternational com m u n ity’, he analyses
‘“d o ctrin a l” speeches w hich elaborate policy’ that Blair delivered in April 1999, April 2002,
and January 2 0 0 3 (Fairclough 2005b : 41).
W hile his analysis is not able to reveal all four ‘m o m en ts’ in the d ialectic o f discou rse and
social change, it does suggest what avenues o f research this m ight entail. He finds that the
texts he exam in es reveal, to som e extent, the m o m en t o f emergence o f a new discourse
throu gh articu lating elem ents o f ex istin g ones; and that the second m om en t, hegemony, is
revealed in asm uch as B lair is a m ajor in ternational statesm an and o p in io n -fo rm er— though,
as Fairclough notes, a w ider range o f m aterial would need to be analysed over a longer period
o f tim e ‘to get a sen se o f heg em on ic struggles over in ternational relations and international
secu rity ’. The third m o m en t, recontextualization, m ight be revealed in a fu rth er study ch a rt­
ing the ‘tra jecto ries o f the em ergen t h eg em on ic d iscou rse in its structural and scalar d is­
sem in ation ’. This m ight entail ‘investigating how the em ergent hegem on ic discourse o f
in ternational relations and in ternational security b oth “colon izes” and is appropriated’
w ithin gov ernm ent policy and m edia texts. He suggests, further, that the operationalization
o f a chan ge in h eg em on ic d iscou rse ‘in new strategies, institutions, exchanges’ might be
revealed in case studies o f processes o f policy form ation and im plem entation by, for ex am ­
ple, studying ‘th e process o f decision m aking and im plem entation in the procurem en t o f
new m ilitary hardw are’ (Fairclough 2005b : 59).

Validity and reliability

Judgem ents o f validity focus on the extent to w hich a plausible case has been m ade that patterns
in the m eanin g o f texts are constitutive o f reality in som e way. D oes an in terpretation ade­
quately accoun t for observations in relation to relevant contextual factors? D oes it m inim ize
potential researcher bias, and provide explanatory coherence within a larger theoretical frame?
In a d d ition to p lausibility, the v alidity o f a d iscou rse-an alytic study can be judged in term s
o f its cred ib ility — its ability to im part ‘c oh eren ce’ to a text, show ing how it fits together in
term s o f con ten t, fu n ction s, and effect; and its ‘fru itfu lness’— its ability to provide insights
that m ay prove useful (P o tter and W etherell 1994). A discou rse-an alytic study should d em ­
on strate a careful read ing o f the text; provide an in terpretation that is clearly related to the
textual ev id en ce; and present an analysis w hich aim s to be credible, plausible, coheren t, and
fruitfu l. R esearch should be open and transp aren t b oth about the textual eviden ce under
review and ab out th e basis o f th e claim s m ade about it (Rapley 20 0 8 ).
H O W T O D O R E S E A R C H IN P R A C T IC E

Content analysis
C ontent analysis involves th e system atic analysis o f textu al in fo rm atio n . In P IR , research ers
tend to study e lection m an ifestos, new s m edia, and political leaders speech es. But th ere is a
wide variety o f texts that researchers m ight ch o o se to analyse, including:

(1 ) official documents: g ov ern m en t rep orts and ad m in istrativ e reco rd s, p ro g ra m m e

ev aluations, d escrip tio n s o f p ro g ram m e activ ities, legal rep o rts, ju d ic ia l d ecisio n s,
c om p any accou n ts; record s from sch o ols, h o sp itals, law c o u rts; ju d icia l d ecisio n s;
and tra n scrip ts o f speech es, co n v ersatio n s, d iscu ssion s, and oral answ ers to
q u estio ns;
(2 ) cultural documents: new spaper articles or ed itorials, m agazines, T V p rogram m es,
film s, videos, art works, photographs, advertising copy; reports from jo u rn als,
m agazines, and new spapers); and

(3) personal documents: letters, diaries, and em ails.

C ontent analysis is an unobtrusive m ethod o f data c ollection . G ath erin g data th rou gh u n o b ­
trusive m eans has a nu m ber o f advantages over obtrusive m eth od s o f data co lle ctio n , such
as surveys, unstructured and sem i-structu red interview s, focus group discu ssion s, e th n o ­
graphy, and participant observation . The ch ie f advantage is that they can reduce bias. For
instance, in all types and form s o f interview s, people can be expected to com e to th e in ter­
view with biases and prejudices. M oreover, all are generally prone to the ‘interview effect’:
the tendency to give ‘socially acceptable’ answers or ones that they th in k the in terview er
wants. People may not tell the truth because they w ant to ‘look g ood’ in the eyes o f the in ter­
viewer; or because they are asked som eth ing either that they don’t know how to answ er or
that would be em barrassing for them to answer. O f course, the researcher also com es to the
interview with biases and prejudices, and these can distort the interview process, as well.
In addition to these problem s, obtrusive m ethods are also prone to the ‘H eisenberg E ffect’,
which we discussed in Chapter 3. This is the ten dency for people to change th eir behaviou r
when they know they are under observation. We described, as an illustration o f this, the
experience o f C harles Frankel, who worked as a United States Assistant Secretary o f State in
the 1960s. W hen Frankel sent out letters merely to get in form ation about what US officials
were doing with regard to particular program m es, he got back replies in dicatin g that ‘the
officials to whom he had written had changed what they were doing after receiving his letter’
(M organ 1994: 37). Merely in an effort to inform him self he had ‘apparently produced
changes in policy’, i.e. in the phenom ena w hich he was studying (Frankel 1 9 6 9 :8 3 ; quoted in
Morgan 1994: 37).
Researchers in our field can get around these problem s by using unobtrusive m ethods o f
data collection. Using content analysis, researchers can get m aterial on decision-m ak ing
without interviewing the decision-m akers. Ih e y can system atically analyse an official’s state­
ments for evidence concerning his or her perceptions and attitudes. They can analyse tran ­
scripts of a public hearing rather than depend on wrhat governm ent officials rem em ber or
choose to tell them about those hearings. In addition to reducing bias, analysing textual
inform ation can also enable researchers to gain access to subjects that may be difficult or
 T E X T U A L A N A L Y S IS

im possible to research through direct, personal contact; and they can study larger popula­
tions and m ore d ocu m ents than would be possible through either interviews or direct
observation.
In sum , content analysis m ight be a way to reduce som e types o f bias and investigate a
wider range o f topics am ong a larger population o f people. But w hether content analysis is
the appropriate analysis for your research project will depend on what evidence you need in
order to investigate your hypothesis, and w hether the evidence is at least partially em bodied
in texts.

Qualitative and quantitative content analysis

C on ten t analysis can be either quantitative or qualitative. We tend to agree with Klaus Krip-
pend orf, that ‘ [u]ltim ately, all reading o f texts is qualitative, even when certain ch aracteris­
tics o f a text are later converted into n u m bers’ (2004: 16). However, quantitative and
qualitative content analysis differ in the types o f qu estions they address, as well as in the
proced ures they use b o th to analyse text and to record, process, and report data. W hich
m ethod you em ploy will depend on your research question and the hypothesis you intend to
investigate.
T he d evelopm ent o f quantitative content analysis was inspired ‘by the need to develop a
m ore ob jectiv e and system atic m ethod for analysing the rapidly in creasing volum e o f co m ­
m u n ication s produced by governm ents, com panies, and oth er organizations’ (Burnh am et
al. 2 0 0 4 : 2 3 6 ). It has b een generally con cern ed with the m an ifest c o n ten t o f co m m u n ica­
tions. This is the co n ten t that is easily o b serv able— that resides on the surface o f co m m u n ica­
tion. A n exam ple w ould b e to cou n t the nu m b er o f tim es a particular word, phrase, or im age
o ccu rs in a com m u n ica tio n . Q uantitativ e analyses generally are con cern ed to tell us ‘How
o fte n ?’ or ‘How m an y?’
H owever, no t all research qu estio ns can be answered by focusing on the m anifest or sur­
face co n ten t o f texts. Th e m eanin g o f the text, or the variables o f interest to the researcher,
m ay no t reside on th e surface o f the con ten t but ‘b etw een the lin es’. Qualitative content
analysis is m o re co n ce rn e d with this laten t c o n ten t. Q ualitative content analysis assum es
that it is p ossible to ex p ose the m eanings, m otives, and purposes em bedded w ithin the text,
and to in fer valid hid d en or underlying m eanings o f interest to the research er (W eber 1990:
7 2 - 6 ) . It is gen erally m o re sen sitive to the context in w hich texts are produced, and better
able to tell us ab out m eanin gs, no rm s, values, m otives, and purposes. However, as we shall
d iscu ss later in th is chapter, recen t advances in com puter-aided content analysis have also
enabled qu antitative analyses o f con ten t to un cover latent d im ensio ns o f texts.
Irre sp ectiv e o f w h e th er you c h o o se to c o n d u ct a qu antitative or a qu alitativ e con ten t
analy sis, you will need to a d dress th e sam e co n sid eratio n s b efore em barkin g on a con ten t
a n alysis. You w ill need to d ecid e w hat p op u lation o f d o cu m en ts w ould provide ev iden ce
relev ant to y ou r h y p oth esis, w hat c o n ten t you will ex am in e and why, w hat kin ds o f data
are requ ired , w h e th er th ey are av ailable or th ey need to b e co lle cted , the kin d o f analysis
th at is requ ired , and th e reso u rces needed. A nd, irresp ectiv e o f w hether your analysis is
qu a n tita tiv e o r q u alitativ e, th e p roced u res th at you w ill need to follow to co lle ct data are
g en era lly th e sam e.
310 H O W T O D O R E S E A R C H IN P R A C T IC E

Steps
Q uantitative and qualitative co n ten t analysis generally involves th e sam e fou r steps. W e can
th in k o f each o f these steps as relating to a specific qu estio n that th e research er m ust ask and
answer.

Step one
Question: What set of documents is germane to your research question, and what sample from this set
will you analyse?

Answer: Select both the population of texts you will use and how much of this material is to be
analysed.

The first step is to select the material to be analysed. This requires th at you first identify the
population o f texts (d ocu m ents or o th er c om m u n ication s) that will provide ev iden ce appro­
priate to an investigation o f your hypothesis. O n ce the popu lation o f relevant texts has been
identified, you will need to b e sure that all o f the d ocu m ents that you need from th is pop u la­
tion are available and accessible. If som e can n ot be located — if, for in stance, th ere are m iss­
ing years in a series o f annual reports that you w ant to investigate— you m ay risk in tro d ucing
bias into your analysis. O n ce you identify the population o f relevant d ocu m en ts and ensure
that they are available, you must decide w hether to analyse th e full set or a partial set o f the
m aterial. If the docum ent population is too large to be analysed in its entirety, you will need
to select a representative sample o f the m aterial to investigate and analyse. Probability sam ­
pling may be the right choice if your in tention is to g eneralize from the sam ple to the p opu la­
tion. N on-probability sam pling can be used if generalization is n ot n ecessary or if probability
sam pling procedures are not practical.
D epending on the aim s o f the analysis, you may choo se textual data that belongs to a
single sem antic dom ain, e.g. political speeches, or that cuts across a nu m b er o f sem antic
dom ains. Your data m ight consist o f one particular text type (party m anifestos, annual
reports) or include texts belonging to different text types. The texts you select m ight contain
only texts produced by a specific speaker or author, either o f the sam e, or different, types; or
they may be produced by different p ersons— as in the case o f open responses to qu estions in
surveys. The data can be obtained from an existing archive, or they m ay need to be collected.
There exist a large num ber o f electron ic text archives containing text data from a large v ari­
ety o f sources, and online text databases with a wide variety o f text m aterial w hich can be
directly accessed and downloaded. In addition, full texts o f a variety o f publications are
available online.

Step two

Q u estio n : W hat w ill you exam ine these texts for?

A n sw er: Define categories.

O nce you have selected both the p opu lation o f texts you will use and how m u ch o f th is
m aterial you will analyse, the secon d step is to define the categories or top ics o f in terest
that you will search lor in the m aterial you have selected to analyse. For a q u antitative
 T E X T U A L A N A L Y S IS

c o n ten t analy sis, th is req u ires you to be c lear ab out th e variables you want in fo rm atio n
a b out: th e su b je c ts, th in g s, or ev ents that vary and that help to answ er your research
qu estio n . T h e n you m ust id en tify categ o ries for each v ariable. For exam ple, you m ight
id en tify th ree c a teg o ries related to th e v ariable attitu d e tow ards the w ar’: negative, n e u ­
tral, and positiv e. O r you m ight often d efin e five categ o ries for the variable ‘attitu de
tow ards th e in c u m b e n t’: g reatly d islike, m o d erately dislike, in d ifferen t to, m o derately
like, and greatly like.

Step three

Question: What segments of the text will contain what you are searching for?

Answer: Choose the recording unit.

The third step is to choose the recording unit (unit o f co n te n t)— the portion or segm ent o f text
to w hich you will apply a category label. There are five recording units norm ally used in
con ten t-an aly tic studies.

1. A single word or symbol. This is generally the sm allest unit that is used in content
analysis research , and is an appropriate unit if you are studying th e use o f language.W hen
w ords are the record in g unit, researchers are usually interested in investigating the choice o f
certain words or cou n tin g the frequen cy with w hich they occur. Word sense—words that
convey values (positive, negative, indifferent) or ideological p osition s— is a variation on
words as record in g units; and they can be counted ju st as if they were words.

2. A sentence or paragraph.
3. A theme. The b oun d ary o f a th em e delineates a single idea, or ‘a single assertion about
som e su b jec t’ (H olsti 1969: 116). This m ight be the recording unit in research on propa­
ganda, values, attitudes, and beliefs.

4. A character. This m ight be the recordin g unit in studies o f fiction, dram a, m ovies,
radio, and oth er form s o f en tertain m en t m aterials.

5. A n item or whole text. This m ight be an entire article, film , b ook, or radio program m e;
a new spaper item , a m agazine article, a web page or book, an episode o f a T V program m e,
or a tra n scrip t o f a rad io interview.

Q uantitativ e co n ten t analysts are generally m ore likely to divide texts into sm aller seg­
m ents: to cou n t individual words, or exam in e phrases, w ord-strings, sen tences, or p ara­
graphs. Q ualitative analysts tend to study docum ents in th eir entirety. In all cases, selection
o f record in g un its m ust be based upon the nature o f the variables and the textual m aterial to
be cod ed . However, trad e-offs am ong different options should also be considered. A record ­
ing un it th at is ob jectively id en tifiab le— one that has obvious physical (e.g. a whole text) or
sem an tic bou n d aries (paragraphs, sen tences, or w ords)— m akes th e c o d e rs task relatively
easy. However, w hile a record in g unit that is ob jectively identifiable has its advantages, it
m ay no t properly enco m p ass th e categories b eing in vestigated. Som etim es a paragraph
em braces to o m an y ideas for th ere to b e con sisten t assignm en t o f the text segm ent to a single
category. In fact, th e larger th e size o f the recordin g unit, the m ore difficult and subjective is
H O W T O D O R E S E A R C H IN P R A C T IC E

the w ork o f cod in g, b ecause as the size o f the unit expands, so d oes th e lik elih oo d th at the
unit will enco m pass m ultiple variables.

Step four
Question: How will you identify, and signal the presence in your recoding units of, the categories you
are looking for?

Answer: Creating (a) a coding protocol; and (b) a code for each variable, or a tag for each theme or
topic, you are looking for; and (c) marking the text with the codes or tags.

O n ce th ese steps are com p leted , you are ready for th e fo u rth step: co d in g . T h is involves
(a) creating a protocol for id en tifyin g th e target v ariables and categ o ries; (b ) creating codes
that will signal th eir presen ce in th e tex t; and (c) coding th e tex ts using th e p ro to co l and
cod es.

(a) Creating a protocol involves d eveloping a set o f rules to ensure the reliability o f the c o d ­
ing. The protocol will reflect a set o f decisions that ensures that the researcher will cod e things
consistently throughout the text, in the sam e way every tim e. For instance, the researcher has
to decide whether to code only a predefined set o f categories, or w hether relevant categories
not included in the set can be added as they are found in the text. Using a predeterm ined
num ber and set o f concepts allows the researcher to exam in e a text for very specific things.
O n the other hand, providing for flexibility allows new, im portant m aterial to be incorporated
into the coding process that could be significant for the findings o f the analysis. M uch q u an ­
titative analysis uses predeterm ined categories, though advances in com puter tech nology (to
be discussed later) have made it possible for quantitative analysis to uncover latent categories,
rather than using predefined categories. M ost qualitative content analysts prefer not to pre-set
categories but to allow categories to em erge out o f the data (Bry m an 2004: 183).

A nother decision is whether to code only for a given word, or also for w ords that imply that
word. Can words be recorded as the sam e when they appear in different form s? For exam ple,
‘agree might also appear as ‘agreem ent’. The researcher needs to determ ine if two words are
sim ilar enough that they can be coded as being the sam e thing, i.e. ‘agreeing words’. R esearch­
ers concerned to draw conclusions about the im portance o f a topic in the print or broadcast
media might measure newspaper space (colum n inches) or radio or television air time.
'These two decisions might best be understood as com pleting a sequence o f steps in which
you (1) ensure your analysis will be focused on the variable o f interest to you (Step O ne,
above), (2) determ ine the possible values each variable can take (Step Two, above); and (3)
provide operational definitions o f the variable’s values that specify what phenom ena you
must observe to identify its existence (Step 3).

(b) O nce you have created a protocol, you can then create a code, or short tag, for each
variable’s categories. Codes are sim ply abbreviations, or tags, for segm ents o f text. Typically,
a code will be an abbreviated version of a category. For exam ple, a researcher coding three
categories related to the variable attitude towards the war’— negative, neutral, and positive—
might label these categories attwarn’ (for negative), attwarO’ (for neutral), and attwar/)’ (for
positive).
 T E X T U A L A N A L Y S IS

(c) The final step is coding: m arking recording units with the appropriate tags that you’ve
designed to identify the categories being sought in the text. ( Before you start, make sure each
docum ent from the set o f working d ocum ents is recorded in a log and given a unique
num ber. As the cod in g proceeds, record additional in form ation , such as the nam e o f the
coder, the date it was coded, and unusual problem s.)

Coding
C od in g involves the identification o f passages o f text (or oth er m eaningful phenom ena,
such as parts o f im ages) and applying labels to them that indicate they are exam ples o f som e
th em atic idea. The cod in g process enables researchers quickly to retrieve and collect together
all the text and oth er data that they have associated with som e th em atic idea so that they can
be e xam in ed tog ether, and different cases can be com pared. We can distinguish betw een two
broad approaches to coding: th ose using a priori codes, and th ose relying on grounded
codes. A priori cod es are based on a research hypothesis or a range o f sources relating to it,
such as previous research or theory, or top ics from your interview schedule. Relying on a
priori cod es is w hat is often referred to as closed cod in g’. G rounded codes em erge from the
data as the research er reads it. The researcher puts aside presuppositions and previous
know ledge o f the su b ject area and con cen trates, instead, on finding th em es in the data. This
is often referred to as 'open cod in g’. The m ost co m m o n way to go about developing grounded
cod es is throu gh co nstant com p arison ’. W hat this m eans is that every tim e a researcher
selects a passage o f tex t and cod es it, it is com pared with all the passages that have already
been cod ed that way. The research er asks: ‘W hat is this ab ou t?’ and ‘How does it differ from
the preced ing or follow ing statem en ts?’ A list o f the cod es is drawn up, with a short defin i­
tion attached to each one. E ach tim e you find a passage that appears as though it m ight be
cod ed w ith an existin g code, you can ch eck the codin g fram e or list to be sure that it fits with
the d efin itio n. I f th ere isn’t an appropriate code, or the text doesn’t fit with the d efinitions,
th en you can create a new one. Eventually, you m ay w ant to sort codes into groups. You may
find several cod es group tog ether as types or kinds o f som eth ing; that they refer to different
ways that people react to, categorize, or cause som eth in g and so m ight be seen as d im en ­
sion s o f that th in g (Strauss and C o b in 1990).
O f cou rse you can m ove from one cod in g approach or strategy to another. For instance,
som e research ers suggest a th ree-stage cod in g process. In the first, ‘open cod in g’ phase, the
research er carefully reviews a sm all sam ple o f the docum ents, m aking general notes about
the broad th em es that ch aracterize each individual docu m ent, as well as the entire set of
texts. In the secon d stage, all the d ocu m ents are reviewed with these th em es in m ind. Pat­
tern s are labelled, and passages are ‘tagged’ as b elon ging to one or m ore categories. In the
th ird stage, th ese labels and tags are checked and re-checked to ensure that they are applied
properly (N eum an and R ob son 20 0 7 : 3 3 7 - 4 2 ) .

Manual and computer-assisted coding

Textual m aterial can be coded m anually, i.e. reading throu gh the text and m anually w riting
dow n co n ce p t o ccu rre n ces; or it can be coded on the com puter throu gh the use o f various
c om p u ter program s.
324 H O W T O D O R E S E A R C H IN P R A C T IC E

In m anual codin g, th e cod er tran scrib es no tes and interview s and cop ies tran scrip ts and
im ages; and th en m akes m ultiple copies o f ev erything, as each item m ay represent an e x a m ­
ple o f m ore th an one th em e or analytic idea. In op en cod in g, p erhaps u sing colou red h ig h ­
lighters, th e cod er m arks the text by circling or un d erlinin g words, o r ru n n in g lin es down
the m argins to indicate different m eanings. In closed cod in g, th e c o d er sim ply m arks the
boun d aries o f the recordin g unit and w rites th e cod e in the m argin o f th e d o cu m en t, p er­
haps using different colou red pens for each variable. N ext, the c o d er will cut up th e tr a n ­
scripts and collect all the text fragm en ts that are e xam ples o f sim ilar th em es o r analytic ideas
(open cod in g), or that are coded the sam e way (closed co d in g ), in to piles, envelopes, o r fold ­
ers. E ach group o f fragm ents can then be set out and re-read to d iscover m ore sp ecific p at­
terns or features o f significance.
C od ing can also be perform ed autom atically with com p u ter softw are that cod es specific
parts o f text according to the particular categorization schem e you con stru ct. A large
num ber and variety o f increasingly u ser-friendly com p u ter softw are program s have been
developed for this purpose.
Q uantitative content analysis program s allow you to ex am in e v ery large a m o u n ts o f data,
and a wide range o f texts, qu ickly and efficiently. A m o ng the w idely used p rogram s for
quantitative content analysis are: G eneral In quirer, V B P ro , W ordsm ith, Textpack, T A C T —
Text, Analysis C om pu tin g T ools, and T ex tstat— a freew are program for the analysis o f texts.
There are also softw are program s that have been developed specifically for qu alitative c o n ­
tent analysis, including Atlas/ti, NUD*IST, and HyperQual. These allow the research er to
identify the recording unit in the text and assign the text to a cod in g catego ry that has been
defined either in advance or in the analysis process. They allow m ultiple cod in g o f in d i­
vidual passages, and m ultiple coders to w ork on a single cod in g task w hile m ain tain in g
id entification o f the cod er for calculation o f reliability. A wide variety o f reports can be
generated from these packages, including coun ts o f cod es with illustrative q u otation s from
the text.
C om puter Assisted Q ualitative Data Analysis Softw are (C A Q D A S) allows researchers to
use com puter-based directories and files rather than physical files and folders, and to use
word processors to annotate texts. C A Q D A S packages, like Q SR N Vivo, help to organize and
analyse data in docum ents, as well as in pictures, audio, and video.
Dictionary-based content analysis program s provide a ‘basic handful’ o f text analysis fu n c­
tions, in cluding word and category frequency coun tin g and analysis, sortin g, and sim ple
statistical tests. 'W ord frequen cy’ refers to how often each word occu rs in a docum ent. M ost
operating systems (W indow s, M ac O SX , Unix/Linux) have utilities to perform basic word
counting and sorting. But software packages allow you to exclude ‘stopwords’, com m on
words like 'in’ and ‘the’ which add little m eaning but get in the way o f the analysis. They can
produce a key words in context (K W IC ) con co rd an ce— a list o f the principal words in the
docum ent, in alphabetical order, and their im m ediate context. They also include lem m atiza-
tion, which involves com bining all words with the sam e stem , such as intend, in tended,
intends, intending, intent, intention, intentions, etc. For category frequencies, program s will
group synonyms into categories and then show how many tim es each category o ccurs in the
docum ent, l or example, the l inguistic Inquiry and Word C ount (I.IW C ) d ictionary maps
onto the category death the following words: ashes, burial*, buried, bury, casket*, cem et*,
co llin ', crem at", dead d eath ', d ecay', d ecease', deteriorat*, die, died, dies, drow n*, dying
 T E X T U A L A N A L Y S IS 325

fatal, fun eral*, grave’ , grief, griev*, kill*, m o rtal*, m o urn *, m urder*, suicid*, tcrm in at* to
L IW C category 59, death. The asterisks tell the program to treat all words m atching a stem
or stem word as belonging to the sam e category (Lowe 2007: 2). By collecting into a single
category d ifferent words or phrases that represent a concept, category counts helps to make
m anifest the latent content in texts.
A no ther set o f program s contains annotation aids (e.g. W in M A X -9 7 FR O ; Q SR N U D* 1ST;
A T L A S .ti). T h e se c o n s ist o f an e le ctro n ic v ersion o f th e set o f m argin al no tes and cro ss-
references that researchers use when m arking up transcripts by hand in order to analyse
them and d iscover patterns.
An in stru ctiv e exam ple o f the purposes for w hich researchers use the system atic analysis
o f political texts is the w ork o f the C om parative M anifestos Project (C M G ). The C M G is a
large and influential quantitative content analysis project concerned with m easuring the
policy position s o f p olitical parties. The C om parative M anifesto P roject produced qu antita­
tive con ten t analyses o f p arties’ election program s from 51 parliam entary d em ocracies cov ­
ering all d em ocra tic elections sin ce 1945. This has b ecom e a widely used data set for party
positions.
The C om parative M an ifestos P roject (previously know n as the M anifesto Research
G rou p) was form ed in 1 979.' Th e C M P u n dertook to m easure the political preferences of
p arties acro ss tim e and space through a com parative content analysis o f political parties’
election m an ifestos. Party m an ifestos are the program m es political parties issue for elec­
tions in som e parliam entary dem ocracies. They are authoritative party policy statem ents
w hich set out b oth th eir strategic d irection and the legislative proposals they intend to pur­
sue should they win sufficient support to serve in governm ent. The political researchers
involved in the p roject w ere interested in addressing two questions: (1) what political issues
divided p ost-w ar p olitical parties; and (2) were they converging or diverging in ideological
or p olicy term s? A nthon y D ow ns’ influential m odel o f tw o-party com petition leads to the
ex p ecta tion o f party conv ergen ce to the policy position espoused by the m edian voter
(D ow ns 1957: 1 1 2 -1 9 ). M R G researchers were con cern ed to find out w hether this actually
o ccu rred .
To estim ate the policy p osition o f a particular party on a particular m atter at a particular
electio n , th e C M P used a trained hu m an cod er to m anually code the party’s election m an i­
festo.2 First, a classification sch em e was developed con sistin g o f 57 p olicy-cod in g catego­
ries. Th e un it o f analysis that was used was a ‘qu asi-sen ten ce’. A q u asi-sen ten ce is defined as
‘an arg u m en t or p hrase w hich is the verbal expression o f on e idea or m eanin g’ (K lingem ann
et al„ 2 0 0 6 : x x iii). Sin ce lon g sen ten ces m ay con tain m ore than one argum ent, break ing up
text in to qu a si-sen ten ce s enables research ers to isolate individual ideas so that they can be
analysed. C o d in g con sisted o f allo catin g every q u asi-sen ten ce contained in a party’s m a n i­
festo to on e, and on ly on e, o f th e 57 categories (on e o f w hich was 'u n cod ed ’). A left-rig h t
p osition m easure is calcu lated by groupin g issue categories in to ‘right’ and ‘left’ categories
and su b tra ctin g on e from the other.
O n c e tex t un its w ere allocated to each category and coun ted, the C M P then defined the
relative sa lien ce for th e party o f th e policy area defined by each category as the percentage o f
all text un its allocated to that category. The cod in g schem e derived from the d ocum ents
them selves. R esearchers found that relative em phases were the way in w hich B ritish parties
expressed them selves. P arties com p ete with each oth er by em phasizing different policy
H O W TO D O R E S E A R C H IN P R A C T IC E

priorities, rather than by directly opposing each oth er on th e sam e issues. So, for in stance,
political op p osition was expressed by em phasizing peace, as opposed to m ilitary stren gth,
freedom as opposed to planning. So the analysis was no t c o n ce rn e d w ith co u n tin g positive
or negative references m ade to different policy areas, but th e relative emphasis p arties placed
on these categories.
The C M P data are gen erated by p arty m a n ifesto s co d ed o n ce , and o n ce only, by a sin g le
hu m an cod er. T h ou gh m an ual co d in g has adv antages, th e co d in g p ro to c o l h ad d isa d v a n ­
tages. The m an ual an alysis o f p arty m a n ifesto s is an e x tre m ely tim e -c o n s u m in g and
costly process. M oreover, th ere is a p o ten tial for co d in g b ias b ecau se h u m an c o d e rs are
in ev itably aware o f th e au th orsh ip o f th e tex ts th ey are co d in g . T h ese flaws in its m an u al
cod in g e nco u raged C M P research ers to develop c o m p u ter p ro g ram s fo r th e c o n te n t a n a l­
ysis o f party p o sition s. A m ajo r step forw ard was th e d ev elo p m en t o f a fully au to m ated
text analysis program for m easu rin g p o licy p o sitio n s o f tex ts called W o rd sco re s (L av er et
al. 2 0 0 3 ). W ord scores uses referen ces tex ts and referen ces v alues in o rd er to p red ict p o l­
icy position s. The b asic idea is th at we can estim ate p o licy p o sitio n s by co m p a rin g tw o
sets o f texts: ‘referen ce tex ts’, d o cu m en ts for w hich we know th e p o licy p o sitio n s (e.g. by
relying on expert surveys), and ‘virgin tex ts’, d o cu m en ts ab ou t w h ich we d o n o t kno w
anythin g apart from the w ords th ey co n tain . A m o re recen t in n o v atio n in q u an titativ e
content analysis o f party p osition s, Wordfish, is a p rogram th at uses a sta tistical m o d el o f
word cou n ts, rather th an an ch o rin g d o cu m en ts. Wordfish estim ates th e p o licy p o sitio n s
o f p olitical actors based on th e assu m p tion th at w ords are d istrib u ted acc o rd in g to a pois-
son d istrib u tion (th e prob ability o f a n u m b er o f ev ents o cc u rrin g in a sp ecified in terv al)
(P rok sch and Slapin 20 0 9 ).
Com puterized coding is easier, m ore flexible, less costly and lab our intensive; and by
rem oving the human factor from the codin g process, it can sign ificantly en h an ce th e relia­
bility o f the content analysis. Thus, efforts to develop program s that resolve sh ortcom in g s in
these m ethods are continu ing; and because o f the salience o f the w ork generated by the
CMP, the innovations o f C M P researchers in this area have b ecom e on e o f the prim ary
im petuses behind the move to com puterized content analysis in political research, m ore
generally.

Analysis
Analysis is the process o f m aking sense and attaching m eaning to the data we have gathered,
and applying the resulting knowledge to our research question.
O nce the coding is done, the researcher exam ines the data for patterns and insights rele­
vant to the key research issues. The codes m ight be com bined or sorted into fam ilies for
more meaningful analysis. The data is analysed either to describe the target v ariable(s), or to
identify themes or relationships between variables. The researcher then attem pts to draw
conclusions and generalizations by linking the data back to the research question. Irrespec-
ti\e of whether the data are quantitative or qualitative, the analysis will involve an attem pt to
interentially link the textual data to the specific events, behaviour, or p henom ena that are of
interest to the researcher.
In a quantitative content analysis, analysis involves ex am in in g nu m erical data in rela­
tion to pre-operationalized variables, and draw ing in feren ces based on the frequency.
 T E X T U A L A N A L Y S IS

am ount, salience, or in tensity o f a category (i.e. the in tensity o f a person’s op in ion s or

attitu d es). Ih e in feren ces drawn will be based on what resides on the surface o f c o m m u ­
n ication , i.e. what has been said, and on the o b serv ation o f patterns that are explicitly
present, but w hich may have been hard to see ju st by reading all the texts. 'Ih e researcher
may also be able to d iscov er relatio nship s am ong th ose patterns, and to identify much
larger trend s or ideas from the p attern s; and, if the research involves chartin g changing
language or o th er textual features over tim e, the research er can draw in feren ces about
political change. Q u alitativ e data is n o n -n u m erical (w ords, im ages). Its analysis is c o n ­
ceptual, and involves id en tifyin g th em es that em erge from the data. Q uantitativ e content
analysts rep ort th eir data largely in n u m eric form , be it statistical, graphical, tabular, or
figurai. They may use frequ en cy scores to gen erate ‘word clou d s’ (or ‘tag clou ds’), which
use the freq u en cy o f w ords in a given piece o f text to generate a visual representation o f
the d o cu m en t. Q u alitativ e co n ten t analysts rely on qu otation s and narrative as th eir p ri­
m ary m od es o f presen tation . Som e also draw concep t m aps, charts, diagram s, or oth er
figures to visually represent the pattern s in th eir data. The d ifferences in how qu antitative
and qu alitativ e resea rch ers typ ically con d u ct a co n ten t analysis are sum m arized in Box
13.1, below.

b o x 1 3. i Quantitative and Qualitative Content Analysis

Q uantitative Q ualitative

Objects of observation Manifest content word usage. Latent content meanings, motives.
sequences purposes

Recording units (units of Segments of text W hole texts

observation)

Procedures of observation Counting, rating logging Thememg, tagging, m.emomg

Discovery of patterns ( a , , : , - ! - , , ,g analysis Developed througnou! process

Presentation of data G-ap:-.'.. •anif-. statistics, figures Quotations, concept maps, narrative

Data format N ^ n , r : , , : . f r , , , u f.:u y a .,o u r ,. tJon " rnenca: (words, images)

Data reduction Vanabies 'operationalised a priori) Themes (em ergent)

Substance of data M eaning is inherent M eaning is contingent

Data recording Standardised instrument Variable instrument

Data processing Mathematical Conceptual

Data reporting Statistical, graphical Verbal

Standards o f evidence Probability Plausibility

Source: Adapted from W esley (2011 : table 6.2).

H O W T O D O R E S E A R C H IN P R A C T IC E

Validity and reliability

As with any m ethod o f data c ollection , research ers u sing con ten t analysis m u st b e co n ce rn e d
with the validity o f th eir analysis and the reliability o f th eir results.
In quantitative studies, validity refers to the ex ten t to w hich we can draw un am b iguo us
conclusion s from ou r results, and w hether our con clu sion s are likely to apply to o th e r s im i­
lar situations or cases. A study is valid if its m easures actu ally m easure w hat you claim they
m easure, and if your in feren ces follow from the data. R eliability refers to th e ‘r ep eatab ility’
or ‘con sisten cy ’ o f your findings. A study is reliable if anyone else follow ing th e sam e p ro ce­
dures would get the sam e results. The reliability o f a con ten t analysis study d epen ds on th ree
elem ents. The first is coder stability: does the sam e cod er con sisten tly re-co d e th e sam e data
in the sam e way over a period o f tim e? Secon d , is reproducibility: do th e c od in g sch em es lead
to the sam e text b eing coded in the sam e category by two or m ore co d ers?5 D ifferent people
should cod e the sam e text in the sam e way. The third elem ent is objectivity. In te rc o d e r
reliab ility reveals ob jectivity by show ing the extent to w hich different cod ers, each cod in g
the sam e content, com e to the sam e codin g d ecisions (R ou rke et al. 2 001: 13).
There are a num ber o f com m on sources o f unreliability:

1. a docum ent is poorly w ritten or vague;

2. word m eanings, category d efinitions, or o th er rules in the cod in g in stru ctio n s are
am biguous;

3. there is a lack o f objectivity in the process o f category d efin itio n and in th e cod in g o f
sections o f text;

4. the cod er m akes mistakes.

In qualitative analysis, the results o f a study are valid and reliable to the degree that they are
plausible to others: i.e. if the researcher explains how s/he cam e up with the analysis in a way
that the reader can make sense of.
Both quantitative and qualitative content analysts should m ake both th eir data and raw
m aterials available for verification (i.e. codin g databases, m em os, and the original d o c u ­
m ents). In quantitative content analysis, which uses a standardized codin g in stru m en t, this
is most efficiently accom plished through the publication o f the codin g m anual, including a
com prehensive list o f coding rules. Q ualitative analysts m ust provide their readers with a
detailed account of the coding 'protocol’, including how conclusion s were reached (A ltheide
1996: 2 5 -3 3 ). As Holliday suggests, all research ‘needs to be accom p anied by accou n ts o f
how it was really d o n e ... [Analysts must] reveal how they negotiated com plex procedures to
deal with the "m essy” reality o f the scenarios being studied' (2007: 7).

Using documents, archival sources, and historical

writing as data

It is worth briefly considering docum ents, archival sources, and historical w riting as sources
o! data. It is usual to distinguish between primary and secondary sources, with docum ents
and archival material of various sorts falling within the form er category, and historical w rit­
ing belonging to the latter.
 T E X T U A L A N A L Y S IS 329

Primary sources refer to those m aterials w hich are w ritten or collected by those who actu ­
ally w itnessed events w hich they describe. A prim ary source provides d irect.or first-hand
evidence about an event, o b ject, or person; and shows m inim al or no m ediation between the
docum ent/artefact and its creator. Exam ples o f prim ary sources include letters, m anuscripts,
diaries, jou rn als, new spaper and m agazine articles (factual accoun ts), speeches, interviews,
m em oirs, d ocum ents and records produced by governm ent agencies, recorded or tran ­
scribed speeches, interview s with participants or witnesses o f an event, or with people who
lived d uring a particular tim e; photographs, maps, postcards, posters, audio or video record­
ings, research data, and ob jects or artefacts such as works o f art, buildings, tools, and
weapons.
P articipants in the political processes we are concerned to understand generate official
records in the form o f party p rogram m es, parliam entary proceedings, resolutions, speeches,
treaties, press conferen ces and press reports, television and radio interviews, and co rres­
pon d ence. These com e from a variety o f participants, including civil servants, m em bers o f
advisory cou n cils, and representatives o f pressure groups involved in decision-m ak ing
processes. P rim ary sou rces are also called archival data because they are kept in m useum s,
archives, libraries, or private collection s. There are m any different types o f archival sources.
Som e d ocu m en ts (e.g. governm ent surveys and research p rojects) are produced with the
aim o f research in m ind ; oth ers (e.g. diaries) are produced for personal use. The archive
rep ositories (o r record offices) m aintained by national and local governm ents contain a
wide range o f official record s, but also considerable qu antities o fp r iv a te ’ m aterial o f p oten ­
tial value to researchers. O th e r types o f archives containing m aterial such as autob iogra­
phies, m em oirs, or oral histories are found in university and oth er libraries or m ore
specialized locations. D ocu m e n ts vary in th eir degree o f accessibility, from closed (e.g.
secret police files) or restricted (e.g. m edical files and confidential corporate reports), to
open -arch iv al (e.g. census reports) and open -publish ed (e.g. governm ent budget statistics).
To assess the e v id en tiary value o f a prim ary source, it is im portant to consider its intended
aud ience, and th e circu m sta n ces in w hich it was produced. A lexander G eorge and Andrew
B en n ett coun sel research ers to ask four qu estions w hen assessing prim ary docum ents: (1)
who is speaking; (2 ) to w hom are they speaking (even unsolicited d ocum ents for personal
use are addressed to an a ud ience); (3) for what purpose are they speaking; and (4) under
what circu m sta n ces (2 0 0 5 : 99)?
A great deal o f p olitical research relies on secon d ary sou rces. Secondary sources are
m aterials prod uced som etim e after an event happened. They con tain in form ation that has
b een in terp reted , co m m en ted , analysed, or processed in som e way. Biographies, h istories,
en cy clop ed ia s, new spaper a rticles that in terp ret, or jo u rn a l articles and b ook s w ritten by
so cia l scien tists are all secon d ary sou rces. U ltim ately, all sou rce m aterials o f whatever type
m u st be assessed critically. But here we want to focus on the p articu lar issues that arise for
politica l research th at relies on th e w ork o f h istorian s for data and evidence.
M any p olitical research ers draw on historical studies to find out what history can tell us
ab out con tem p o ra ry events, to develop typologies o f p olitical phenom ena, or to accoun t for
different patterns o f political and so c io -eco n o m ic developm ent. Political researchers who
draw on h istorical sou rces for evidence m ust consid er two related issues.
T he first issue co n c e rn s th e extent to w hich we can treat the work o f historians as reliable
rep orts o f past p olitical or social realities. P olitical researchers need to recognize that the
H O W T O D O R E S E A R C H IN P R A C T IC E

work o f historians can n ot be treated as u np rob lem atic sou rces o f facts, and th at it is no t p o s­
sible to use an historical accou n t as a ‘th eoretically n eu tral’ backgro u nd narrative. H istorical
a ccoun ts con tain errors, biases, ex clusion s, and exaggerations. T h ey reflect th e h isto rian ’s
personal com m itm en ts and, m ore generally, th eir im plicit th eories o f o r persp ectives on
hum an behaviour. R ecall ou r d iscu ssion in C hapter 3 ab out th e argu m en t, advanced by
T h om as Kuhn and oth ers, that ob serv ation is ‘th eory-lad en ’. A ccord in g to th is argum en t,
ou r observation o f ‘facts’ can n ot be separated from the th eoretical n o tion s w hich give in tel­
ligibility to what we observe. W h at we call ‘observ ation’ is th e in terp retation o f a p h e n o m e­
non in the light o f som e theory and oth er b ackground know ledge.
Consequently, as Ian Lustick points out, ‘the w ork o f h istorians... c an n o t legitim ately be
treated by oth ers as, an u n problem atic b ackground narrative from w hich th eoretically n e u ­
tral data can be elicited for the fram ing o f p roblem s and the testin g o f th eo ries’ (L ustick
1996: 6 0 5 ). He points out that what we th in k we know about a period is not th e ‘result o f an
objective sifting and reporting o f what prim ary sou rces and artifacts co n tain , but the result,
first and forem ost, o f im aginative con stru ction s o f lives and events, w hich, woven in to p ar­
ticu lar overarching narratives, seem ed natural, conv in cing, or useful to th ese h istorian s as
vehicles for the claim s they wished to advance’ (Lustick 1997: 606).
This raises a second issue that political researchers m ust c o n fro n t w hen th ey use historical
studies as evidence. If there is no th eoretically neu tral historical record, if different authors
offer vastly different interpretations about the sam e historical events, how are th e b ack ­
ground historical narratives w hich we use in historically grounded research to be chosen
from am ong the available accounts? In the ab sence o f a single ‘h istorical record ’ on w hich we
can rely, what set o f rules can guide us in distingu ishing ‘accurate’ from ‘in accu rate’ h isto ri­
cal accounts?
The problem for political researchers is to choo se from the available sou rces in a way that
avoids ‘selection bias’— i.e. a bias towards th ose accoun ts which fit with the argum ent being
investigated and defended. The danger o f selection bias arises w henever we must choo se
am ong conflicting, or partially conflictin g, available historical accoun ts. The ‘nub o f the
issue’, as Ian Lustick points out, is that the search for available studies in ord er to form an
evidentiary base for an argum ent ‘may well entail, and can logically be supposed to entail, a
heavy selection bias toward works by historians using im plicit th eories about how events
unfold and how people behave very sim ilar to the th eory under consideration by the social
scientist (Lustick 1996: 607). It is natural that we will find m ost in terestin g th ose accounts
that seem to best fit with the concepts and categories that our argum ents employ. The chosen
in terpretation will likely coin cide with or be supportive o f the argum ent that is being ‘tested’.
We tend to adopt sources that suit the theory that we are testing, because we tend to find
most convincing those accounts that fit with our theory (Lustick 1996: 614).
How do we choose from am ong these differing accoun ts o f the past? Lustick suggests
several strategies to address this issue. H rst, include an analysis o f p atterns across the range
of historical accounts relating to your topic. Second, look for regularities that appear across
otherwise different or contradictory accou n ts— accounts based on different approaches or
on different archival sources, or which develop different perspectives or reach different c o n ­
clusions. Third, note alternative versions, other sources that are available and that contradict
those on which you rely. i.e. those that tell a different story. Justify your choices. You can lim it
the amount ot additional space this might take by ‘lim iting use o f this technique to elem ents
 T E X T U A L ANA LY SI S

o f the background narrative that are either particularly controversial w ithin existing h istori­
ography or that are particularly salient for the theoretical argum ent under review’.
W e conclud e with two points regarding the use o f docum ents, archival sources, and his­
torical w riting as data.
The first p oin t is th at th e ch o ice o f sou rces, as in every ch o ice m ade in the research
process, d epen d s on w hat is required for developing a m eanin gful and persuasive in vesti­
g ation o f your research q u estio n . The secon d poin t is the need for self-con sciou sn ess in
the sele ctio n o f sou rce m aterial. You m ust be c ritical and rigorous, b oth in term s o f how
you do your ow n research and how you evaluate the research o f oth ers. R esearchers must
be self-aw are and c ritic a l a b ou t the ch o ices they m ake, and m ake the con sid eration s that
en ter in to th o se ch o ices clea r and tran sp aren t to oth ers.

Conclusions
This chapter has outlined two main forms of textual analysis. Each of the two forms of textual analysis-
discourse analysis and content analysis-provides insights into political phenomena: discourse analysis
through examining how discursive practices construct the identities of subjects and objects and
exercise power; qualitative content analysis through exploring the meanings, motives, and purposes of
political action embedded w ithin texts: and quantitative content analysis by drawing inferences about
opinions or attitudes from an analysis of the usage and frequency of words, phrases, and images, and
the patterns they form within a text. Despite their very real differences, sim ilar standards apply to both
forms of textual analysis. Both must be trustworthy in their treatment of documents. Both must be
concerned w ith the validity and reliability of their procedures and conclusions.
Box 13.2 sum m arizes the differences and sim ilarities among these forms and approaches to textual
analysis.
W hich of the two form s of textual analysis is 'better' than the others can only be determined in
relation to a specific research project, and w ill depend on the research question, what sort of analysis
will provide a useful response to it, and what data are needed and from whom . As w e have previously
em phasized, techniques of data collection are always employed in the service of a research question.
As w ith any method of data collection, the use of discourse and content analysis should be appropri­
ate to the research question and hypothesis you are investigating. The data collected by these means
do not 'speak for themselves'. They are only interesting and significant to the extent that they provide
a means of investigating a research question and hypothesis. You choose these means of data
collection because you are confident that the data that they provide will enable you to investigate and
draw logical conclusions about your hypothesis.
Unlike asking people questions (e.g. through surveys or in interviews), using texts to collect data has
the advantage of being non-intrusive. Researchers do not face the problem of influencing their data
source through the questions they ask. and they can study past policy positions as they were recorded
at the tim e. Once recorded, texts do not change.

Questions
• W hat do discourse and content analysis offer to political analysis?
• H ow do you know w hen you've identified a discourse? W here are its boundaries? Do discourses
overlap? In w hat context or set o f conditions does a discourse exist?
• H o w can you evaluate w hether, w hen, and how political texts have effects on political life?
H O W TO DO R E S E A R C H IN P R A C T I C E

box 13.2 Discourse Analysis and Content Analysis Compared 1

Discourse analysis Content analysis

Qualitative Quantitative

Ontology Constructionist-assumes that There is no 'nherc meaning Realist-assumes that an

real'ty is socially constructed in the text. :in.-anii independent reality
construct'."!'; !ri a [,M'ticuiar
r ti *<t . " l i r e
marcher a!J

Epistemology Meaning :s fixed and

reflects reality ;r: ways
’• " ' f .- u s ,- o f that can be ascertained
through the i_.se of

Data source

' t. lu tu , i texts ter exarrole over

Method , „ Quantitative
............................ - ■ ■...

Categories ........
' . .. .. .; A^ f _ ; ....... .. . . .

Inductive,/
deductive

Subjectivity/
objectivity

Role of
 T EX T U A L ANA LY SI S

Validity in the form of demon­ The results are valid to the Validity is in the form of
strating a plausible case that degree that they show how accuracy and preci­
patterns in the meaning of patterns in the meaning sion-demonstrating that
texts are constitutive of reality of texts are constitutive of patterns in the texts are
in some way reality accurately measured and
reflect reality

Reflexivity Necessarily high-author is Considers the extent to Not necessarily high—

part of the process whereby which the author plays a role author simply reports on
meaning is constructed in making meaning, and dif­ objective findings
ferent ways a meaning might
be consumed

Source: Adapted from Hardy et al. (2004:20-1).

• W ith what general type of research questions is content analysis concerned?

• W hat is the difference between manifest and latent content? What are the implications of this
distinction for content analysis?

• If politics is about power, and language has power, is political analysis a matter of analysing
language?

• In what ways might your analysis address the issue of selection bias when using historical writing as
data?

Guide to Further Reading

Discourse analysis
d aym an, S. and J. Heritage (2002), The News Interview: Journalists and Public Figures on the Air
(Cambridge: Cambridge University Press).

Fairclough, N. (2002), Analysing Discourse: Textual Analysis for Social Research (London:
Routledge).
An introduction to discourse analysis, drawing on a variety of texts, from political speeches and
television news reports to management consultancy reports and texts concerning globalization, to
illustrate key issues in discourse analysis.

Gee, J. P. (2005), An Introduction to Discourse Analysis: Theory and Method, 2nd edition (New York:
Routledge).
This is an introduction to discourse analysis that presents both a theory of language-in-use and a
method of research. Its aim is to demonstrate how language, both spoken and w ritten, enacts social
and cultural perspectives and identities.

Hopf, T. (2004), 'Discourse and Content Analysis: Some Fundamental Incompatibilities',

Qualitative Methods Newsletter (Spring): 31 -3.

Howarth, D. R. Norval.J, Aletta, and Y. Stavrakakis (eds) (2000), Discourse Theory and Political
Analysis: Identities, Hegemonies and Social Change (M anchester Manchester University Press).
Case study chapters show how discourse analysis can be applied.

Milliken, J. (1999), T h e Study of Discourse in International Relations: A Critique of Research and

Methods', European Journal o f International Relations 5(2) (June): 225-54.
H O W TO D O R E S E A R C H IN P R A C T I C E

Multi-Author Symposium (2004), Discourse and Content Analysis', Qualitative M ethods (Spring):
15-38.

Raymond, G. (2000), T h e Voice of Authority: The Local Accomplishment of Authoritative

Discourse in Live News Broadcasts', Discourse Studies 2 : 354-79.

Ricento, T. (2003), T h e Discursive Construction of Americanism', Discourse & Society 14(5):

611-37.

Shenhav, S. R. (2006), 'Political Narratives and Political Reality', International Political Science
Review 27(3): 245-62.

Symposium: Discourse Analysis and Content Analysis (2004), Newsletter o f the American Political
Science Association Organized Section on Qualitative M ethods 2(1): 15-39.

Content analysis
Hopkins, D.. and G. King (forthcoming), 'A M ethod of Automated Nonparametric Content Analysis
for Social Science', American Journal o f Political Science, available at http://gking.harvard.edu/
files/abs/words-abs.shtml.

Johnston, A. I. (1995), Cultural Realism: Strategic Culture and Grand Strategy in Chinese History
(Princeton, NJ: Princeton University Press).
A highly effective example of content analysis using ancient Chinese texts to assess beliefs and
predicted behaviour.

Kohlbacher, Florian (2005), T h e Use of Qualitative Content Analysis in Case Study Research',
Forum: Qualitative Social Research 7, available at http://www.qualitative-research.net/index.
php/fqs/article/viewArticle/75/153.
A basic introduction is given to qualitative content analysis as an interpretation method for qualitative
interviews and other data material. Useful for understanding how content analysis can contribute to
qualitative case study research.

Krippendorff, K. (2004), Content Analysis: An Introduction to its M ethodology, 2nd edition

(Thousand Oaks, CA: Sage Publications).

— and M. A. Bock (eds) (2008), The Content Analysis Reader (Thousand Oaks, C A: Sage
Publications).
Fifty-one papers grouped around the following topics: the history and conception of content analysis,
unitizing and sampling, inferences and analytic constructs, coders and coding, categories and data
language, reliability and validity, computer-aided content analysis.

Lewis, R. B. (2004), 'N Vivo 2.0 and Atlis.ti 5.0: A C om parative R eview of Tw o P opular Q ualitative
Data-Analysis Programs', Field Methods 16(4): 439-69.

M ayring, Philipp (2000), Qualitative Content A nalysis’, Forum: Qualitative Social Research 1,
available at http ://w w w .qualitative-research.net/index.php/fqs/article/view /1089.
The author describes an approach of systematic, rule-guided qualitative text analysis, including
the central procedures of qualitative content analysis, inductive development of categories, and
deductive application of categories

Monroe, B .L .M . P. Colaresi, and K. M. Quinn (2008), "Fightin" W ords: Lexical Feature Selection
and Evaluation for Identifying the Content of Political Conflict1, Political Analysis 16: 372-403.
 T E X T U A L A N A LY SI S

---- and P. A. Schrodt (2008), 'Introduction to the Special Issue: The Statistical Analysis of Political
Text, Political Analysis 16:351 -5.

Neuendorf, Kimberly A. (2002), The Content Analysis Handbook (Thousand Oaks, CA: Sage
Publications).
Covers the history of content analysis, sampling message units, handling variables, reliability, and
use of NEXIS for text acquisition. Also covers PRAM, software for reliability assessment with multiple
coders.

Riffe, D., S. Lacey, and F.G. Fico (2006), Analyzing M edia Messages: Using Quantitative Content
Analysis in Research (Mahwah, NJ: Lawrence Erlbaum).
A com prehensive guide to conducting quantitative content analysis. Provides step-by-step instruction
on designing a content analysis study; and detailed discussion of measurement, sampling, reliability,
data analysis, and validity.

References
Altheide, D. L. (1996), 'Reflections: Ethnographic ---- (2005a), 'Critical Discourse Analysis,
Content Analysis', Qualitative Sociology 10(1): Organizational Discourse, and Organizational
65-77. Change’, Organization Studies 26: 915-39.
Austin, J. L (1962), How to do Things with Words ---- (2005b), 'Blair's Contribution to Elaborating a
(Oxford: Clarendon Press). New Doctrine of “International Community",
Benoit, K„ M. Laver, and S. Mikhaylov (2009), Journal of Language and Politics 4(1): 41 -63.
Treating Words as Data with Error: Uncertainty Foucault, M. (1972), The Archaeology of Knowledge
in Text Statements of Policy Positions', American and the Discourse on Language (New York:
Journal o f Political Science 53(2) (April): 495- Pantheon).
513. Frankel, C. (1969), High on Foggy Bottom: An
Billig, M. and E. Schegloff (1999), 'Critical Discourse Outsider's Inside View of the Government (New
Analysis and Conversation Analysis: An Exchange York: Harper and Row).
between Michael Billig and Emanuel A Schegloff, George, A. and A. Bennett (2005), Case Studies
Discourse & Society 10(4): 543-82. and Theory Development in the Social Sciences
Bryman, A. (2004). Social Research Methods, 2nd (Cambridge, MA: Belfer Center for Science and
edition (Oxford: Oxford University Press). International Affairs).

Burnham, P., K. Gilland, W. Grant, Z. Layton-Henry Hall, S. (1988), The Hard Road to Renewal: Thatcherism
(2004), Research Methods in Politics (Basingstoke: and the Crisis of the Left (London: Verso).
Palgrave Macmillan). Hardy, C. (2001), 'Researching Organizational
Dixon, T. and D. Linz (2000), 'Race and the Discourse', International Studies in Management
Misrepresentation of Victimization on Local and Organization 31(3): 25-47
Television News', Communication Research 27(5) Hardy, C., B. Harley, and N. Phillips (2004),
(October): 547-73. 'Discourse Analysis and Content Analysis: Two
Downs, A. (1957). An Economic Theory o f Democracy Solitudes?’ Qualitative Methods: Newsletter of the
(New York: Harper). American Political Science Association Organized
Section on Qualitative Methods. Symposium:
Escobar, A. (1994), Encountering Development
Discourse and Content Analysis (Spring): 19-22.
The Making and Unmaking o f the Third World
Holliday, Adrian (2007), Doing and Writing
(Princeton, NJ: Princeton University Press).
Qualitative Research. 2nd edn (Thousand Oaks:
Fairclough, N. (1995), Media Discourse (London: Sage Publications).
Edward Arnold).
Holsti, O. R. (1969), Content Analysis for the Social
---- (2001), Language and Power (London:
Sciences and Humanities (Reading, MA: Addison-
Longmans). Wesley).
336 H O W TO D O R E S E A R C H IN P R A C T I C E

Hopf, T. (2004), 'Discourse and Content Analysis: Proksch, S.-O., and J. B. Slapin (2009), 'Position
Some Fundamental Incompatibilities', Taking in European Parliament Speeches'. British
Qualitative Methods: Newsletter o f the American Journal o f Political Science 40(3): 587-611.
Political Science Association Organized Section on Rapley, T. (2008), Doing Conversation, Discourse, and
Qualitative Methods. Symposium: Discourse and Document Analysis (London: Sage Publications).
Content Analysis (2004): 31 -3.
Rourke, L , T. Anderson, D. R. Garrison, and W.
Klingemann, H.-D.. A.V olkensJ. Bara, I. Budge, Archer (2001), 'Methodological Issues in the
and M. McDonald (2006). Mapping Policy Content Analysis of Computer Conference
Preferences II: Estimates for Parties, Electors, Transcripts', International Journal o f Artificial
and Governments in Eastern Europe, European Intelligence in Education 12:8-22.
Union and OECD 1990-2003 (Oxford: Oxford
Searle, J. R. (1969). Speech Acts: An Essay in the
University Press).
Philosophy of Language (Cambridge: Cambridge
Krippendorf. K. (2004), Content Analysis: An University Press).
Introduction to its Methodology, 2nd edition
Strauss, A. and J. Corbin (1990), Basics o f Qualitative
(Thousand Oaks, CA: Sage Publications).
Research: Grounded Theory Procedures
Laffey, M.and J. Weldes (2004), 'Methodological and Techniques (Newbury Park. CA: Sage
Reflections on Discourse Analysis', Methods: Publications).
Newsletter of the American Political Science
Titscher, S.. M. Meyer, R. Wodak, and E. Vetter
Association Organized Section on Qualitative
(2000), Methods o f Text and Discourse Analysis
Methods (Spring): 28-30.
(London: Sage Publications).
Laver, M.. K. Benoit, and J. Garry (2003), 'Estimating
van Dijk, T. A. (2001), 'Critical Discourse Analysis',
the Policy Positions of Political Actors Using
in D. Tannen. D. Schiffrin. and H. Hamilton
Words as Data', American Political Science Review
(eds), Handbook of Discourse Analysis (Oxford:
97(2). 311-31.
Blackwell), 352-71.
Lemke.J. L. (1998), Analysing Verbal Data:
---- (2002), 'Discourse and Racism', in D. Goldberg
Principles, Methods and Problems', in
and J. Solomos (eds), The Blackwell Companion
B. J. Fraser and K. Tobin (eds), International
to Racial and Ethnic Studies (Oxford: Blackwell).
Handbook of Science Education (Dordrecht:
145-59.
Kluwer), 1175-89.
---- (2004), 'Racist Discourse', in E. Cashmore
Lowe, W. (2007), Software for Content Analysis-
(ed.), Routledge Encyclopedia o f Race and Ethnic
A Review, 22 March, available at http://kb.ucla.
Studies (London: Routledge), 351 -55.
edu/system/datas/5/original/content_ analysis,
pdf Weber, R. P. (1990), Basic Content Analysis, 2nd
edition (Newbury Park, CA: Sage Publications).
---- (2008). Understanding Wordscores', Political
Analysis 16(4): 356-71. Weldes, J. (1996), 'Constructing National Interests',
Lustick, I. (1996). History, Historiography, and European Journal of International Relations 2(3).
275-318.
Political Science: Multiple Historical Records and
the Problem of Selection Bias', American Political Wesley, Jared J. (2011), 'Observing the Political
Science Review (September) 605-18 World: Quantitative and Qualitative
Morgan. P M (1994). Theories and Approaches to Approaches', in Keith Archer and Loleen
International Politics What Are We to Think? Youngman-Berdahl (eds). Explorations: A
4th edition (New Brunswick, NJ Transaction Navigator's Guide to Research in Canadian
Publishers) Political Science. 2nd edition (Toronto: Oxford
University Press)
Neuman. W L. and K Robson (2007), Basics of
Social Research Qualitative and Quantitative Wetherell, M (1998), 'Positioning and Interpretive
Approaches (Toronto Pearson) Repertoires: Conversation Analysis and Post­
structuralism in Dialogue', Discourse & Society
Potter. J and M Wetherell (1994). Analysing
9(3) 387-412
Discourse. in A Bryman and R Burgess (eds).
Analysing Qualitative Data (London Routledge). Wittgenstein. L (1980) Culture and Value (Oxford:
47-66 Blackwell).
 T E X T U A L A N A LY SI S 337

Endnotes
1. The MRG was formed by Ian Budge and David Robertson, both at that time in the Department of
Government, University of Essex. It was constituted formally as a Research Group of the European
Consortium for Political Research (ECPR). which obtained funding to support most of its work in the
1980s.
2. The full coding process is described in Benoit et al. (2009).
3. The simplest and most common method of reporting intercoder reliability is the percent agreement
statistic. This statistic reflects the number of agreements per total number of coding decisions. 'Percent
agreement after discussion' refers to reliability figures that were obtained through discussion between
coders. Holsti's (1969) coefficient of reliability (CR) provides a formula for calculating percent agreement:
CR = 2m /nl + n2, where:
m = the number of coding decisions upon which the two coders agree
n l = number of coding decisions made by rater 1
n2 = number of coding decisions made by rater 2.
A Quantitative Analysis:
Description and Inference

^ Chapter Summary
This chapter provides an introduction to quantitative analysis w ith a fo cus on descrip­
tion and inference. W e discuss the different ways in w hich we can sum m arize data
for a single variable from our sample and use it to make inferences about the wider
population from which the sample was drawn. In particular, we focus on two of the
key building blocks in quantitative research to do w ith measures of central tendency
and measures of dispersion. The chapter contains:

• overview of analysis;

• univariate analysis;

• levels of measurement;

• methods of analysis;

• descriptive statistics;

• central tendency;
• dispersion;

• inferential statistics.

Introduction
Q u a n tita tiv e m e th o d s a re o n e o f th e m o st w id ely u se d te c h n iq u e s in p o litica l r e s e a rc h . T h e y
a rc u sed to a n sw e r a w id e v a rie ty o f q u e stio n s, to d o w ith d e m o c r a c y a n d d e m o c r a t iz a t i o n
(P rz e w o rsk i et al. 2 0 0 0 ; G le d itsch a n d W a rd 2 0 0 6 ) , civ il w a r (G le d its ch 2 0 0 7 ) an d e th n ic
co n flict (W ilk in s o n 2 0 0 4 ) , a n d to d o w ith p u b lic o p in io n (W Ie z ie n 1 9 9 5 ; B a rtle et al. 2 0 1 1 ) ,
e le ctio n s, an d v o tin g b e h a v io u r (C la rk e et al. 2 0 0 4 , 2 0 0 9 ; Fra n k lin 2 0 0 4 ) . W h a te v e r to p ic yo u
are in te re ste d in stu d y in g in p o litica l re s e a rc h , it is h a rd to av o id c o m i n g a c ro s s lite ra tu re
th at uses q u a n titativ e m e th o d s . It is th e re fo re an i m p o rta n t m e th o d to u n d e rs ta n d . Yet for
m an y stu d e n ts this is an ofT-pu ttin g p ro s p e ct. To th e u n tr a in e d eye, q u a n tita tiv e re s e a rc h c an
seem in tim id a tin g an d difficult to u n d e rs ta n d — c lo se r to th e stu d y o f s ta tis tic s th an p o litics,
'let in reality th in g s are n ot as difficult as th e y first a p p ear. W h a t ca n lo o k c o m p lic a te d o n th e
s u rla c e is a ctu a lly relatively s tra ig h tfo rw a rd , ev e n se n sib le , in p ra c tic e , an d fa m ilia rity w ith
lust a lew basic c o n ce p ts an d te ch n iq u e s c a n tak e yo u a v e ry lo n g w ay in a s h o rt s p a c e of
tim e. It is th e re ln re well w o rth th e effort o f try in g to le a rn h o w to u se an d u n d e rs ta n d q u a n ­
titative m e th o d s , even il this in volves s o m e initial d is c o m fo rt. It will n ot o n ly h elp you
 Q U A N T IT A T I V E AN A LY SI S

understand the work o f others, but, perhaps m ore rewardingly, it will also enable you to do
your own original research yourself. And these data-analysis skills are hard transferable
skills that are also highly valued in the work place, and can help you get a jo b after you
graduate.
In our ex p erien ce stud ents are som etim es apprehensive about learning qu antitative
m ethod s. Yet in our ex p erien ce this apprehen sion is also soon overcom e. Indeed, learning
how to do qu antitative research is often an in credib ly rew arding ex perience. It is not only
rew arding to ov ercom e your in itial anxieties, but in the process, new p ossibilities also
open up, providing you with new skills w hich you can take with you in your own research
and in your future career. M oreover, th e actu al statistical com p o n en t o f qu antitative
resea rch is actu a lly a relativ ely m in o r part o f th e research p rocess. G ood qu antitative
research is based upon good research design, in terestin g hyp otheses, and a careful reading
and u n d erstand in g o f the relevant literature. These research skills are com m on to all types
o f p olitical research.
This chapter, and the chapters that follow, provide an in troduction to the principles o f
quantitative research and a step-by -step guide on how to use and interpret a range o f c o m ­
m only used tech niques. W e start with the basics. The first part o f the chapter looks at the
build ing b lock s o f qu antitative analysis. W e focus on different ways in w hich data can be
sum m arized , b oth graphically and with tables. We in troduce two im portant measures: the
m ea n and the stan d ard d ev iation. In the second part o f the chapter, we m ove on to in feren ­
tial statistics, and discuss how we can m ake generalizations. In doing so, we introduce the
con cep t o f confid ence intervals, often known as the m argin o f error.

Descriptive and inferential statistics

The research process involves the dual goals o f description and explanation (see King et al.
1994: chapter 2). E ach is essential. W e can n o t con stru ct m eaningful explanations w ithout a
sou nd know ledge o f what it is that needs to be explained. For exam ple, there is a great deal
o f research w hich d escrib es the p olitical landscape. By d escrib ing the world around us, c o l­
lecting data and in form ation , we build up a picture o f what is usual, or strange, or salient in
som e way, and w hat it is that is interesting to explain. M any o f the q u estions we are in terested
in are th erefore descriptive ones, the answers to which tell us som eth ing m eaningful about
th e w orld we live in. Have people b ecom e less politically engaged? Is the m edia b ecom ing
m ore pow erful? A re politician s corrup t? Is inequality increasing? How m uch pow er do
d om estic gov ernm ents have over eco n o m ic policy? Is there global w arm ing? W hat is g lo ­
b alization? Has it increased ? To w hat extent do Iraqis support d em ocracy? To w hat extent do
A fghan is support the Taliban? These sorts o f q u estions are all descriptive (see C hapter 5).
S o m etim es we are interested in describ ing the present, such as how m uch people know
ab out politics, and som etim es we are interested in d escrib ing patterns over tim e, such as
w hether Iraqis’ support for d em ocracy has increased or decreased sin ce the U S -U K -le d
invasion. Related to th ese descriptive qu estions, we then develop explanations that we can
seek to test. W hy are som e people m ore politically engaged than others? W hy has Iraqi sup­
p ort for d em o cra cy changed sin ce th e invasion?
H O W T O DO R E S E A R C H IN P R A C T I C E

In the following chapters we will e xam in e the statistical a ssociation b etw een tw o variables
(b iv a ria te analysis) and th ree or m o re variables (m u ltiv ariate an alysis). Th is allow s us to
test hypotheses about causality, and answ er qu estions about why th ings happen. But before
getting into explaining why som eth ing happens, it is first a good idea to d escrib e what hap­
pens in the first place.
D escription in quantitative research com prises two parts. The first is to describ e th e data
th at we have co lle cted . W h e th e r it is fro m su rv ey data (C h a p te r 1 0), e x p e rt in terv iew s
(C h a p ter 11), tex tu al data (1 3 ), or co m p arativ e aggregate data (C h a p te r 9 ), we can use
descriptive statistics as a way o f sum m arizing our sam ple data for a particular variable. But
we can also use this in form ation to try and m ake g eneralizations about the w ider p o p u latio n
from which the sam ple was drawn; that is, to m ake inferences. W ith in feren tial statistics, we
can go from just talking about how m any people in our sam ple support the far-righ t British
N ational Party (B N P ), to m aking inferences about w hat we th in k the true level o f support for
the BN P is in the population as a whole. O ur ability to m ake this jum p from describ ing our
sam ple to m aking generalizations about the population rests on how we have collected the
data in the first place. This is why p robability sam ples are so im portant, because they create a
statistical link between sam ple data that we have and the p opulation that we want to m ake an
inference about.

Levels of measurement
Quantitative analysis requires that the inform ation or evidence that we have collected is
converted into num bers. Som etim es it is fairly obvious what these nu m bers refer to, such as
when we are recording a respondent s age or a cou n try’s G D P per capita. However, at oth er
tim es it is not so intuitive, such as when we are recording a respondents religion or a c o u n ­
try’s electoral system. How we in terpret the num bers we assign to different variables then
depends upon what the num bers actually refer to, and w hether they are in a sense 'real n u m ­
bers’ or arbitrary codes for distinct categories.
There are different m ethods o f sum m arizing data and the m ethod that we use depends
upon the type o f variable we are exam ining, and how this variable is measured. We can d is­
tinguish between these different types o f variable according to what is known as their level
o f m easurem ent. Broadly speaking, there are three levels o f m easurem ent: nom inal (also
called categorical), ordinal, and interval (also called continu ous or scale). W e describe each
of these below.
Nominal com es from the Latin for nam e. A no m in al variable is one w here the num bers
assigned to the variable are interesting only in so far as the lab els— or n am es— that are
attached to them are interesting. To in terpret the variable, we must know what the values
refer to and the nam es o f the different categories. For exam ple, consider the following qu es­
tions from the British Flection Study. We can see that each response to the question is
assigned a number, but these num bers do not refer to anything oth er than the label that
they represent.

I alkin^ to people about the general election on June 7th . wc have found that a lot of people
ilkln t manage to \ote I low about yo u ' Did you manage to vote in the general election?
 Q U A N T I T A T I V E A N A LY SI S

1. Yes, voted.

2. No.

[II- YKS) W h ich party did you vote lor in the general election?
[D O N O T P R O M P T ]

1. C onservative

2. Labour

3. Liberal D em ocrat

4. Scottish N ational Party

5. Plaid C ym ru

6. G reen Party

7. O th er Party.

The fact that people w ho didn’t vote are coded 2 and people that did vote are coded 1 does
not m ean that n o n-v oters are twice as apath etic as voters. A lthough each o f these responses
is assigned a n u m erical value, the nu m b er is o f no interest itself, we are only interested in
what the nu m b er refers to.
By con trast, in the case o f o rd in al variables, the nu m bers assigned to the different response
categories do have som e m eaning. They have an order. C onsid er the exam ple below:

L et’s talk for a few m inutes about politics in general. H ow m uch interest do you generally have
in what is going on in politics?

1. N on e at all

2. N ot very m u ch

3. Som e

4. Q uite a lot

5. A great deal.

O n c e again, each response catego ry is assigned a value, but th is tim e we can in terpret the
values acco rd in g to th e ord er in w hich th ey are arranged. W e can th in k o f the n u m bers as
referrin g to levels o f p olitica l in terest, so so m eon e w ho replies that they have not very m uch
in terest in p olitics is cod ed 2, and from this we can see that th ey have less in terest in p olitics
th an so m eo n e w hose resp onse is cod ed 4. But th ere is a lim it to the extent that we can draw
m ean in g fu l co n clu sio n s ab out how m u ch m ore or less in terested they are. We can n o t say
th a tn so m eo n e w ho is cod ed 4 is tw ice as interested in politics as so m eon e w ho is coded 2.
N or can we say th at th e d ifferen ce in in terest betw een 4 and 5, w hich is 1, is the sam e as the
differen ce in in terest b etw een 1 and 2, w hich is also 1. In sh ort, we can o n ly m ake m e an in g ­
ful sta tem e n ts ab ou t the ord er o f th e responses, not the m ag nitu de o f the differences
b etw een th em .
W ith in te r v a l— or sca le— variables, th e n u m bers do m ake sense. Indeed, th ere is no d is­
tin c tio n betw een th e value and th e label. A n exam ple o f th is type o f variable is age.

N ow , a few q uestions ab ou t yo u rself and yo u r b ack grou n d . W h a t was you r age last birthday?
A ge in y e a r s __
342 H O W T O DO R E S E A R C H IN P R A C T I C E

If som eone is 34 years old, they are coded as 34. The n u m b ers have an order, so so m eo n e w ho
is 4 0 is older than som eone w ho is 30, and th e n u m bers also have a m agnitu de th at we can
in terpret, so s om eone w ho is 6 0 is tw ice as old as som eone w ho is 30. T he d ifference betw een
25 and 30 is 5 years, and this distance is the sam e as th e difference betw een so m eo n e w ho is
45 and 50. For this reason, interval variables are often referred to as real nu m bers.

Summarizing the data

Next we turn to how the data from these different types o f variables can b e sum m arized.
There are a num ber o f ways in which we can do this. We can use tables, figures, and statistics.
We can look at the freq u en cy d istrib u tio n — which d escrib es the entire d istrib u tion o f
responses, and sum m arizes the nu m ber o f cases for each given response code. W e can also
sum m arize various aspects o f this d istribution , relating to m easures o f cen tra l t e n d e n c y -
such as averages— and m easures o f d isp ersio n or variation — such as the standard deviation.
The appropriate m ethod for sum m arizing data depends upon the level o f m easurem en t. We
discuss each o f these below.
Exam ining the distribution o f variables also serves as a way o f checking the data. O n e ser­
ious problem that affects quantitative analysis is m issin g data— or item non-response. For a
variety o f reasons, data for som e cases in our data set may be unavailable. R espondents may
not have answered som e questions, or may have answered but given ineligible responses.
Missing data o f this kind can introduce error or bias into the analysis if those people who did
not answer the question are different in som e im portant respect from those who did answer.
It is not straightforward what to do about missing data, but there are a num ber o f strategies
that are available (see Box 14.1). The other thing to keep an eye out for is sparse categories,
which are much easier to deal with. This occurs when relatively few o bservations are found for
a particular response category. This can create problem s when analysing the data, and so the
best thing to do if this problem arises is to com bine responses into larger, m ore general catego­
ries.

Tables
Frequency tables are the norm al tabular m ethod for presenting distributions o f a single v ari­
able. I he tabic provides inform ation on the distribution o f responses across all response ca te ­
gories. They are therefore most appropriate when there are not too many different response
categories (otherwise the tables becom e too big to easily interpret), and so are mainly used for
nominal and ordinal variables. Tables should be clear and easy to understand. A good table
presents the relevant inform ation in a straightforward and transparent way. It should contain:

• a cle a r sell e x p la n a to ry title;

• c le a r labels;

• an a p p ro p ria te level o f p re c is io n ( ro u n d p e rc e n ta g e s to n e a re s t w h o le n u m b e r);

• sam p le s i/e ;

• a d ata sou rce .

 Q U A N T I T A T I V E A N A LY SI S 343

box 14.1 Missing Data

Missing data is a common problem in survey research (and other types of data collection) Running
frequency tables allow you to identify the scope of the problem Missing data can take different forms
Item non-response occurs when a respondent (for whatever reason) does not provide a valid answer
to a question. It can reduce the sample size (especially in a multivariate analysis or when combining
variables when several small amounts of missing data may accumulate) and risks introducing selection
bias. Selection bias occurs when the sources of missingness' are not random, and are structured so that
some types of people (or people holding some types of attitude or opinion) are less likely to provide a
valid response than others. The structure of missing responses may vary from one variable to another,
so there is no straightforward remedy that can be applied. The strategy for how to deal with this should
depend in part on your theory about what has caused it. Below we discuss the common approaches to
dealing with item non-response.
Listw ise deletio n is often the default choice, and sim ply involves deleting all missing values from the
analysis. The danger with this approach is that you are throwing away inform ation, reducing sample
size, and more im portantly, may be introducing bias if the missingness is not random . However, in cases
w here there are only a small number of missing values, this is probably the most practical solution.
However, w hen there is a large num ber of missing values, it may be advisable to retain missing as a
separate category.
M ore sophisticated techniques involve trying to predict what respondents would have answered
to the question from inform ation that is a vailable elsewhere in the questionnaire. This is known as
imputation. The sim plest form of im putation is ju st to set the missing values to the m idpoint or mean.
This approach is quick and easy, but makes best sense if you believe the missing is random . In general,
how ever, this is rarely the case, and setting missing to any constant value is regarded as unsophisticated
and som ewhat questionable. M ore advanced m ethods of im putation use regression methods (see
C hapter 16) to im pute valu es on the basis o f other inform ation provid ed by the respondent elsew here in
the questionnaire. Although this is statistically sounder, it can becom e quite technical, and is only useful
in so far as good predictors are available.

All tables should be clearly labelled and easy to understand. They should have a self-explana­
tory title, and be properly referenced with the source o f data. I h e response codes should be
clearly labelled, and the sam ple size should be reported. If there is m issing data, this needs to be
reported too. If there is a large am ount o f m issing data, it is im portant to be clear w hether you
are reporting percentages based on what is often called the total per cent (w hich includes m iss­
ing values in the calculations) or the valid per cent (w hich does not include m issing values).
For exam ple, Table 14.1 sum m arizes the responses to the variable on vote ch o ice that was
asked in th e 2 0 0 5 B ritish E lection Study. It is a no m in al variable. The first colu m n con tain s a
list o f all th e m ain response catego ries (th e nam es o f the m ain p olitical parties in B ritain ).
The secon d co lu m n (F req u en cy ) con tain s in form ation on the nu m b er o f respond ents from
th e survey w ho selected each response. At the b ottom o f this colu m n in the last row, labelled
‘T otal’, we can see that 4 ,161 people were in terview ed for the survey. O f th ese people, 1,198
said th ey voted L abour, 8 6 7 said th ey voted C onserv ative, and 1,079 said they did not vote.
W e can use th ese nu m b ers to calculate th e percentage o f the sam ple that provided each
response. Th is is d one in the next colu m n , labelled ‘Per c en t’. The percentages are based on
th e total cou n t. So, for exam ple, 1,198 people out o f 4,161 said they voted Labour, and th is
co rresp o n d s to 2 9% o f o u r sam ple (1 ,1 9 8 divided by 4,161 m u ltip lied by 100 = 29).
H O W TO DO R E S E A R C H IN P R A C T I C E

Table 14.1 Voting behaviour in the 2005 British General Election

Frequency Per cent Valid p ercen t

1198 29 40
Labour
867 21 29
Conservative
645 16 21
Liberal Democrats
Scottish National Party (SNP) 137 3 5

69 2 2
Plaid Cymru
20 0 1
Green Party
United Kingdom Independence Party (UKIP) 45 1 1

British National Party (BNP) 6 0 0

Other 36 1 1

Did not vote 1079 26

Don't know 13 0

Refused 46 1

Total 4161 100 100

Source British Election Study 2005.

We should note, however, that som e respondents did not provide an answ er to th e question.
Som e people did not vote, so if we are in terested in the relative shares o f the vote that each
party received, we should exclude these people from the analysis. A lso som e people did not
know, or could not rem em ber, w hich party they voted for, and for whatever reason, som e
people refused to answ er the qu estion. These ‘m issin g values’ are a co m m o n p rob lem in
survey research, and can pose a bit o f a headach e in term s o f how we treat th em . D o we
regard them as valid answers, or do we discou nt them from our analysis?
M issing data can introduce error, since we cann ot be sure what th ose people actually did.
We can hope that they are a random selection o f the sam ple. But it is also possible that they
are more likely to be certain types o f voter. For exam ple, those that refused to answ er the
question may be m ore likely to be C onservative voters (th e so-called ‘shy T ories’) or BN P
voters (who may be unwilling to reveal their true behaviour because o f s o cial d esirab ility
bias; see Chapter 10), and so excluding them will bias our results, sin ce it m eans that we are
undercounting som e response categories. W hen the num ber o f m issing cases is sm all (as in
this case), this error or bias will also be sm all, but when the num ber is relatively large, it can
be a problem and we need to think carefully about what to do about it.
The most straightforward option (though not always the m ost appropriate) is sim ply to
discount the missing values from the analysis. We can then re-calculate the percentages to
refer to just the people who provided valid responses. This is done in the next colum n,
labelled valid per cent. Here the base excludes all those who said that they did not vote and
all those who did not know w ho they voted for or who refused to answer. We can see that the
percentages in this colum n are som ewhat different. Now our base is just 3,023. So, for e x am ­
ple, 1,198 out of 3,023 respondents said they voted Labour, w hich corresp onds to 40%
(1,198/ 3,023' 100 = 40).
We should also note that we have very few responses for som e o f the m inor parties. For
example, only 1% of our sample reported voting for the Green Party and UKIP, and less than
 Q U A N T I T A T I V E A N A LY SI S 345

1% reported voting for the BNP. Sparse categories such as these can pose a nu m ber o f p rob­
lem s for analysis, and it is generally advisable to com bine them together where possible. In
th is in stance, they could be in corporated w ithin the ‘O th er party’ category.
The nex t exam ple we c o n sid er is from a qu estio n designed to m easure w hether resp on d ­
ents th in k o f th em selv es as b ein g ideologically left w ing or right wing. The frequen cy
d istrib u tio n is p resented in T able 14.2, and the ex act survey qu estio n is reprodu ced under
the table. The variable is ord in al. A gain, we can su m m arize the data in term s o f the fre ­
qu ency, th e p er cen t, and th e valid per cen t, in the sam e way as b efore. However, we
sh ould n o te th at th ere are q u ite a lot o f people w ho selected th e 'D o n ’t kno w ’ op tion. P eo ­
ple m ay no t have u n d erstood th e qu estio n , or m ay have b een un able to place them selves.
Th is in a sen se is a valid answer. N ot ev eryon e th in k s o f th em selv es as b eing on the left or
th e right, and for a n alytical pu rp oses we th erefo re need to th in k carefully about how we
treat th is data.
O n e o p tio n is sim ply to ig n ore th e ‘D o n ’t know s’ and exclud e th em from th e analysis.
F or d escrip tiv e p u rp o ses, we m igh t be in terested in the percen tag e o f th o se w ho are able
to place th em selv es on th e le ft-rig h t scale. This is rep orted in th e c o lu m n labelled ‘Valid
p er c e n t’. H ere we have to b e care fu l to be clear ab out w hat th e data now refer to. They no
lo n g er refer to th e id eo lo g ica l d isp o sitio n o f ev eryon e (th e B ritish adult p o p u latio n ), but
on ly to th o se peop le w ho w ere ab le to place th em selv es. So, for exam p le, o f th o se w ho
w ere ab le to p lace th em selv es on a le ft-rig h t scale, 36% placed th em selv es in th e cen tre,
a nd ju s t 1% placed th em selv es on th e far left, and 2% placed th em selv es on th e far right.
W ith o rd in a l v a ria b les we can also lo o k at th e cu m u lativ e p er cen t. The cu m u lativ e per

Table 14.2 Left-right ideology self-placement, British adults in 2005

Frequency Per cent Valid per cent Cumulative per cent

OLeft 36 1 1 1

1 41 1 1 2

2 124 3 3 6
3 241 6 7 12
4 366 9 10 23

5 1295 31 36 59

6 532 13 15 74

7 454 11 13 87

8 295 7 8 95

9 78 2 2 97

10 Right 82 2 2 100

Don't know 613 15

Refused 4 0

Total 4161 100

Note: Question: In politics, people sometimes talk about parties and politicians as being on the left or right. Using the 0
to 10 scale on this card, where the end marked 0 means left and the end marked 10 means right, where would you place
yourself on this scale?"
Source. British Election Study 2005
H O W TO D O R E S E A R C H IN P R A C T I C E

cen t is a usefu l way o f aggregatin g d ifferen t resp on ses. It adds up th e p ercen tag es in o rd er
for each response category. From th is, we can clearly see th at 2 3 % o f resp o n d e n ts w ho
were able to place th em selv es on a le ft-rig h t scale placed th em selv es to th e left o f ce n tre
(givin g answ ers 0 to 4 ).
W ith interval variables it is generally not feasible to report the data in tables, sin ce th ere
will be a lot o f different response categories, and so the table will require a lot o f rows. A table
o f age, for exam ple, in w hich respondents’ age varies from 18 to 100 would require 82 rows,
which would go on for several pages. Th ere w ould also b e a lot o f sparse catego ries. T h e pur­
pose o f tables is to present data in a clear and transp arent way. Too m any rows can distract
from this. Moreover, it would not be very easy to interpret. O n e o ption then is to recod e data
in to bands. So rather than displaying the d istribution for all ages, we can band th e ages in to
age groups (e.g. 1 8 -3 0 years old, 3 1 - 4 0 years old, 4 1 - 5 0 years old, 5 1 - 6 4 years old, and 65
years and over). There are a nu m ber o f different ways in w hich data like th is can be grouped
together. The data can be grouped into equal intervals, so that each band con tain s, say 20%
o f the sam ple; it can be grouped into equal intervals, so that each band spans, say, 15 years;
or they can be grouped into distinctive categories o f th eoretical in terest, so that band s refer
to specific groups o f interest, such as young adults, or retired people. It doesn’t really m atter
which approach is used, but it is advisable to avoid having g roups with either a very sm all or
a very large sample size.

Graphs and figures

Sam ple d istrib u tion s can also be displayed graphically. T h is is often a m o re accessib le way
o f presenting data, and can be easier to analyse. The p u rpose o f g raphs and figures is to
present the data as clearly and accurately as possible. People can often get very creative
when it com es to displaying graphs, but it is wise not to get carried away, and to focus on
ju st trying to present the in form ation in the clearest term s possible. In m ost cases, a b a r
c h a rt or h isto g ram is the best option for su m m arizin g the d istrib u tion o f a single v ariable.
Line graphs are also frequently used for sum m arizin g data over tim e. However, th e alter­
natives, such as pie charts, generally are not to be recom m end ed . They m ay look pretty,
but it can often be difficult to distinguish betw een the relative sizes o f the different ‘slices’.
However, this is not such a problem with histogram s and bar ch arts, w hich generally
present the data m ore clearly.
Ihe choice between bar charts and histogram s depends upon the level o f m easurem ent o f
the variable. For nom inal and ordinal variables bar charts are used, and for interval variables
histograms are used. In practice, bar charts and histogram s look quite sim ilar to each other.
Ihe principal difference is that bar charts have a space between the bars to indicate that the
response categories are d istin ct— or d iscre te — w hereas with histogram s the bars all touch
each other to in dicate that the response categories are continu ou s.
Figure 14.1 presents the data from Table 14.1 as a bar chart. A long the x-ax is (th e
horizontal axis), the different response catego ries are labelled. Ih e y-axis (th e vertical
axis) records the share of the vote in per cent. It is im portant to clearly label the axis so
that the reader can interpret the in form ation . Presen tin g the data in this way conveys all
 Q U A N T I T A T I V E A N A LY SI S

Labour Conservatives Liberal SNP Plaid Cymru

Democrats

Party voted for

Figure 14.1 Bar chart o f party vote in the 2005 British General Election
Source: British Election Study 200S.

Left-right self-placement

Figure 14.2 Bar chart o f left-right self-placem ent

Source: British Election Study 2005.

th e relev a n t in fo rm a tio n , an d clearly sh ow s w h ich p arties receiv ed th e m o st v otes and

w h ich th e least.
Fig u re 1 4.2 d ep ic ts a b a r ch a r t fo r le ft-r ig h t self-p la cem en t. T h is v ariable is ord in al,
an d so th e re a re s p a ces b etw ee n th e b ars, a lth o u g h 0 - 1 0 scales are often treated as in terval
v a ria b les a n d so it co u ld b e p resen ted as a h isto g ra m . P re se n tin g th e data in th is way is
in so m e r e sp e c ts a n im p ro v e m e n t o n ju s t p re se n tin g th e data in ta b u la r fo rm , s in c e we
no w g et a c le a r e r se n se o f th e o v erall d istrib u tio n o f th e v ariab le. W e c a n see th a t th e
H O W T O D O R E S E A R C H IN P R A C T I C E

m o st c o m m o n resp o n ses are in th e c e n tre o f th e d istrib u tio n , an d th e le a st c o m m o n

resp o n ses are in th e ta ils o f th e d is trib u tio n (th e e x tre m e v alu es at e ith e r e n d ). In fa c t,
th e v a ria b le lo o k s q u ite lik e a n o r m a l d is t r ib u t io n , w h ich is so m e th in g w e w ill b e c o m ­
in g b a c k to la ter on .
F ig u re 14.3 d ep icts a h isto g ra m for th e B r itish N atio n al P a rty ’s (B N P ) sh are o f th e v ote
in th e co n s titu e n c ie s w h ich it c o n teste d fo r th e 2 0 0 5 B r itish G e n e ra l E le c tio n . T h is is an
ex am p le o f a g g reg a te d ata. S in c e sh are o f th e v ote is an in terv a l-lev el v ariab le, we display
th e d ata in a h isto g ram , w here th e b ars to u c h ea c h o th e r to in d ica te th a t th e data are
co n tin u o u s.
As we discussed w ith referen ce to Table 14.1, very few survey resp ond ents rep orted h av ­
ing voted for the B ritish N ation al Party (th e far-righ t party in B ritain ) in th e 2 0 0 5 election .
Indeed, nationally only 0.7% voted for the BNP. Survey data is th erefo re no t well placed to
analyse the factors associated with the B N P vote, sin ce th ere is no t en o u gh in fo rm atio n to
draw sensible conclusion s. A n alternative approach th en is to ex am in e th e official electio n
returns, and see how well the party did in each co n stitu en cy w hich it contested. This is
reported in Figure 14.3. The B N P stood in 119 con stitu en cies, and in th ese co n stitu en cie s its
sh are o f th e vote ranged fro m a low o f 0 .8 % to a h igh o f 17% . W e can see fro m Fig u re 14.3
that in 29 constituencies the B N P got around 4% o f th e vote, and in on ly seven co n stitu en ­
cies did it m anage to get 10% or m o re o f the vote. M oreover, we can see th at in on e co n stitu ­
ency it did particularly well, gaining 17% o f the vote. This constituency, B arking, is clearly
exceptional, and is separated from m ain b od y o f the distrib ution . It can th erefo re b e regarded
as an ou tlier.
Presenting the data in this way we can get a sense o f how the level o f B N P sup port varies
across constituencies. W e can get a rough idea o f how well it did on average (ab ou t 4%
appears to be the m ost c om m on value) and a rough idea o f how m u ch its v ote varied, but to
make clearer statem ents about these types o f observ ation we need to em ploy special statis­
tics. These are discussed in the next section.

BNP share of the vote (%)

Figure 14.3 Histogram or BNP share of the vote in constituencies in the 2005 British General Election
Source The British Parliamentary Constituency database. 1992-2005
 Q U A N T IT A T I V E AN A LY SI S

Summarizing the distribution using statistics

As well as sum m arizin g the entire distribution , we can also choose to sum m arize som e spe­
cific aspect o f the distribution. This allows us to m ore easily make com parisons, either over
tim e or betw een groups o f people, coun tries, or places. In m aking these com parisons, it is
useful to have in form ation on the centre o f the distribution , and how scattered or dispersed
the data are around this point. These m easures are known as m easures o f central tendency
and m easures o f d ispersion.

Measures of central tendency

There are three m easures o f central ten dency: the m ode, the m edian, and the m ean. Each
m easure provides a sum m ary o f the central value’ o f the distribution in som e sense, although
how they do this varies som ew hat. These m easures are generally only appropriate for ordinal
and interval-level data, sin ce for n o m in al-level variables it does not really m ake m uch sense
to talk about the central value because the values do not have any underlying order to them .
H ow ever, for n o m in a l v ariables, we are often in terested in th e m o de, w hich refers to the
m ost co m m o n value, or the largest group. So we m ight say that the m odal religion in India
is H ind u ism , or the m odal eth n ic group in Kenya is Kikuyu, or the m odal political party in
the U K parliam ent is the C onservative Party.
To illustrate how th ese m easures o f central ten d ency are calculated, consid er the follow ­
ing exam ple. Su ppose we w ere in terested in the financial status o f students, and decide to
ask each o f the stud ents in a class how m u ch m o ney th ey receive from all sou rces per year.
Th ere are 11 stud ents in th e class, and each is asked to estim ate th eir total annual in com e in
p ou nd s (to th e nearest th ou san d ). From th is we o b tain the follow ing values. This is our
d istrib u tion :

4,6,8,8,10,10,10,12,14,16,20
T h e m o d e is sim p ly th e m o st c o m m o n v alue in th e d istrib u tio n . W e can see th at th ree
s tu d en ts have an in c o m e o f ab ou t £ 1 0 ,0 0 0 . T h is is th e m o st co m m o n value in th e d is tri­
b u tio n and so is th e m o d e. It is p o ssib le to have m o re th an on e m o d e in a d istrib u tio n ;
in su ch ca ses, th e d istrib u tio n is b im o d al or m u ltim o d al. T h e m e d ia n refers to th e m id ­
d le v alu e o f th e d istrib u tio n . It is d efin ed as th e p o in t w hich div id es th e o b serv ed values
in to tw o eq ual p a rts. A ll th e valu es are arran g ed in ascen d in g o rd er (as ab ov e) and th e
m e d ia n is sim p ly th e o n e in th e m iddle. S in c e we have 11 stu d en ts, in th is case the
m e d ia n refers to th e v alue o f th e six th o b se rv a tio n , w h ich is also £ 1 0 ,0 0 0 . If th ere is an
ev en n u m b er o f o b se rv a tio n s in th e d istrib u tio n (an d so th ere is no t o n e value th at is
d ire c tly in th e m id d le ), th e m e d ian is calcu la ted as th e d iffe re n c e b etw een th e tw o m id ­
d le v alues.
W h erea s th e m o d e and th e m ed ian can be directly ob serv ed from th e d istrib u tio n , the
m ean n eed s to b e c alcu lated (th o u g h m o st co m p u ter program s will do th is for you). The
m ean is th e m easu re o f cen tra l ten d en cy th at m o st o f us th in k a b out w hen we hear the term
average’. It is sy m bo lized in tw o different ways, acco rd in g to w hether we are d escrib in g th e
p op u la tio n m ea n o r th e sam ple m ean. For th e tim e being , we ju st co n sid er th e sam ple
m ean. T h is is d eno ted as X (p ron ou n ced x-b a r), w here x is the variable o f interest (in th is
H O W T O D O R E S E A R C H IN P R A C T I C E

ca se, stud ent in c o m e). T h e m ean is th e m easu re o f cen tra l te n d e n cy th at is used m o st often
in qu antitative analysis, and is o n e o f th e b u ild in g b lo ck s for m o re ad v an ced qu an titativ e
analysis tech n iq u es th at we w ill e n co u n ter in th e follo w ing chapters. It is calcu lated by a d d ­
ing up all th e values o f x th at are ob serv ed in th e d istrib u tio n , and th en div id in g th e to tal
by th e n u m b er o f o b serv atio n s (stu d en ts). It is expressed as follow s:

where

X = the m ean;
I = ‘th e sum o f ’ (pronou nced sigm a);
X' = th e values o f x to be sum m ed;
n = the nu m b er o f ob serv ation s (sam ple size).
U sing the above exam ple, we can plug in all the values o f x, add th em up, and th en divide
by the n u m ber o f observ ation s (1 1 ). This gives us:

- _ 4 + 6 + 8 + 8 + 10+ 1 0 + 1 0 + 12 + 14 + 16 + 20
x - h

_ 118
X = — = 10.72.
11

So, in the above exam ple, the m ean is £ 1 0 ,7 2 0 . W h en the m o d e, th e m ed ian , and th e m ean
are all the sam e, we have a p erfect n o rm al d istrib u tio n . However, it is no t u n c o m m o n to
find that som etim es the different m easures produce very d ifferent values. U n d er th ese
circu m sta n ces, it is natural to ask w hich m easure is th e best, and it is easy to su p pose that
it m ust be the m ean, sin ce this appears to be the m o st scien tific. But th is is no t always the
case.
The m ean can be sensitive to extrem e outliers, especially when we are dealing with rela­
tively sm all sam ples. Suppose the day after we collected the above data, a new student entered
the class w ho had a trust fund w hich gave h im an annual in com e o f £2 m illion. Including this
person in our calculation dram atically changes the m ean. W hereas the m ode and the m edian
are still £1 0 ,0 00, the m ean is now a whopping £ 1 6 6,676. Clearly, the m ean no longer appears
to be very typical o f what the average student has to live on. It is therefore always im portant
to check for the presence o f outliers (particularly when the sam ple size is relatively sm all), as
these can dramatically distort the results and m ake what is highly unusual appear typical.
But at the sam e tim e, the problem with the m edian is that it is qu ite a blunt m easurem en t
instrum ent. If we are interested, for exam ple, at look ing at how public op in ion has changed
over time, we might prefer a m ore sensitive m easurem ent in strum en t. For exam ple, Euro-
barom eter routinely asks people lo place them selves on a 0-10 left-rig h t scale, where 0 is
left and 10 is right (as discussed earlier). Figure 14.4 plots the m edian position sin ce 1973.
We can see that in the 1970s the median position fluctuated betw een 5 and 6 , and sin ce the
late 1980s has rem ained constant at 5, suggesting that there has not been a great deal o f
change in public attitudes. By con trast, Figure 14.5 plots the m ean p osition sin ce 1973.
From this measure we can see that since the late 1980s there has been a gradual shift to the
lelt in public opinion.
 Q U A N T IT A T I V E AN A LY SI S

6.2

î ", m
4.8

1970 1975 1980 1985 1990 1995 2000 2005 2010

Figure 14.4 Left-right self-placem ent in the U K-m edian position of adult population, 1973-2007
Note: Question: In politics, people sometimes talk about parties and politicians as being on the left or right Using the 0 to 10
scale on this card, where the end marked 0 means left and the end marked 10 means right, where would you place yourself
on this scale?'
Source: Eurobarometer.

6.2

6 — Mean

4.8
1970 1975 1980 1985 1990 1995 2000 2005 2010

Figure 14.5 Left-right self-placem ent in the U K -m e a n position of adult population, 1973-2007
Source: Eurobarometer.

C h o o sin g w hich m easure o f central ten d en cy to report depends upon the kind o f in fo rm a­
tion you w ant to convey. Th e m o d e is appropriate if you are talking about data m easured at
th e n o m in a l level, o r i f you are sim ply interested in what is the m ost co m m o n value or the
largest group. The m ed ian is appropriate if th ere are ex trem e outliers, w hich m ay d istort the
m ean. M o st gov ern m en t p u b lication s use the m edian rather th an th e m ean to report average
hou seho ld in co m e for ju st th is reason. O therw ise, the m ean is generally appropriate. The
m ean is th e on ly m easure th at is m athem atically based. It uses all o f th e in form ation in the
d istrib u tion , and is on e o f th e m ain b uilding b locks for m o re advanced analysis.
H O W T O D O R E S E A R C H IN P R A C T I C E

Measures of dispersion
M easures o f cen tral ten d en cy provide a pow erful way o f su m m arizin g large am o u n ts o f data.
They are useful for su m m arizin g w hat is th e average o r typ ical person or value. B u t it is also
useful to get an idea o f how individuals o r o b jects in th e d istrib u tion d iffer fro m o n e a nother.
S om etim es two different distrib u tion s can have th e sam e m ean (o r m ed ian ) b u t b e v ery d if­
ferent from one an o th er in term s o f th e v ariability or spread w ithin th e d istrib u tio n . A n
exam ple illustrates this. Suppose students taking in tro d u ctio n to qu antitative m e th o d s are
random ly assigned to one o f th ree classes. Each class has 12 students, and at th e end o f th e
course th e students are given a test that is m arked out o f 10. The m arks for each stud ent in
each class are show n below :

Mark out of 10

Class 1: 6 6 6 6 6 6 6 6 6 6 6 6

Class 2: 4 4 5 5 5 6 6 7 7 7 8 8

Class 3: 1 2 2 3 4 4 8 9 9 10 10 10

For each class, the m easures o f central ten d ency based on the m ean and the m ed ian are the
sam e— 6 — w hich represents a pass m ark. However, th e variation in th e m arks is different in
each o f the classes. In Class 1, everyone gets the sam e m ark, perhaps b ecau se th ey have ju st
been taught the bare m inim um to pass, and this has b een dru m m ed into th em over and over
again during the course. In Class 2, th ere is m ore v ariation in the m arks— som e students do
well w hereas others fail, but th ere is not m u ch difference betw een th e top m arks that are
achieved and the bottom m arks. In C lass 3, there is m ore variation still. The students ten d to
do either very well or very badly, perhaps because the teach er focuses m ore on the able stu ­
dents and leaves the students who are struggling behind.

This exam ple shows how im portant it is to exam in e b oth the variation and th e m ean. O n
the face o f it, we m ight assum e that there is not m uch difference betw een each o f the classes
since they all have the sam e m ean and m edian exam m ark. However, from in sp ectin g the
variation o f m arks w ithin each class, we m ight com e to the con clu sion th at th e classes are in
fact quite different from one another. And, depending upon how confident we felt in term s
o f our own ability to do quantitative research, we m ight have strong op in ion s about which
class we would rather go in.
This example clearly illustrates the intuition behin d calculating m easures o f d ispersion.
As with the m easures o f central tendency, variation— or disp ersio n— can be m easured in a
num ber o f ways that vary in statistical com plexity. The three m ost com m on ly used m easures
of dispersion are: the range; the interquartile range, and the standard deviation. These are
discussed below.
Ihe range is the m ost straightforw ard m easure o f dispersion. It sim ply calculates the d if­
ference between the sm allest and the largest values. So with reference to the above exam ple,
the range for Class 1 is zero ( 6 - 6 = 0), the range for Class 2 is 4 ( 8 - 4 = 4), and the range for
Class 3 is 9 ( 10-1 = 9 ). Hven this sim ple m easure gives us a better feel for the data than we
would have from considering just the mean alone. We know that there is a lot o f variability
in the marks o f students in Class 3, and none at all in C lass 1. So if we w ere a student who was
 Q U A N T I T A T I V E A NA L YS IS 353

particularly apprehensive about taking a quantitative m ethods course, we might feel m ore
confid ent if we had the teach er for Class 1.
However, one drawback o f the range is that, since it depends on the extreme values o f the
distribution, it is very sensitive to outliers. O ne or both o f these values might be atypical, so the
range is not always a particularly trustworthy measure. The in terquartile range (1QR) avoids
this problem by ignoring the tails o f the distribution, and only exam ines the dispersion within
the middle 50% o f the distribution. It thus divides the distribution into quartiles and com pares
the value o f the first quartile (the value which divides the ordered set o f observations into the
sm allest 25% and the largest 75% ) and the third quartile (the value which divides the ordered
set o f observations into the sm allest 75% and the largest 25% ). This can be thought o f as a related
technique to the median, w hich divides the data into the sm allest 50% and the largest 50%.
Table 14.3 provides a worked exam ple with data from C lass 3. We can see that all the stu­
dents in the b ottom qu artile got less than 3 out o f 10 on th eir test, and that all the students in
the top qu artile got full m arks o f 10 out o f 10. The interquartile range (th e m iddle 50% )
th erefore stretch es from m arks o f 3 out o f 10 to 9 out o f 10, giving a range o f 6.
The stand ard deviation is related to the m ean, and is by far the m ost widely used measure
o f dispersion. R ath er th an ju st consid ering the d ifference betw een two points o f the d istrib u ­
tion (as w ith the range and IQ R ), it utilizes all the values in the distribution and describes
how far on average each value deviates from the m ean. W hen all the values in the distrib u­
tion are th e sam e (as in the exam ple from Class 1), the difference betw een each value and the
m ean will be zero. H owever, when the values are very different from the m ean (as in Class 3),
the stand ard d eviation will be relatively large.
The stand ard d eviation is calculated with th e follow ing form ula,

~*)2

Table 14.3 Calculating the interquartile range

Student X (Test score)

1 1
2 Lower quartile
2
3 2
4 3

5 4

6 4 Interquartile range
7 8

8 9

9 9

10 10
Upper quartile
11 10

12 10
354 H O W TO DO R E S E A R C H IN P R A C T IC E

where

s = the standard deviation o f th e sam ple;

x = the value for each o b serv ation in th e d istrib u tion ;
X = the sam ple m ean;
N = th e nu m b er o f ob serv ation s (sam ple size).
In oth er words, the standard deviation is the square root o f the sum o f th e squared differences
betw een each value in the distribution and the m ean, divided by the sam ple size m inus one.
This sounds a bit o f a m outhful, but is actually fairly straightforw ard if we consid er each step in
turn. Table 14.4 provides a worked exam ple for calculating the standard deviation, using the
distribution o f test scores from Class 3, discussed above. We have 12 students in our sam ple,
and for each student we have recorded their test score. These test scores are th e values for xi,
and are listed in colum n 2 for each student. For exam ple, we can see that student 1 got ju st 1 out
o f 10 on the test. We are interested in how far these scores deviate from the m ean. W e already
know that the m ean is 6, so for each student we calculate how far away th eir score is from 6.
This is done by subtracting the m ean from each individual score. This is show n in colu m n 3.
For student 1 this c om es out as - 5 (1 - 6 = - 5 ). The p roblem is that som e o f th ese d ev iations
are positive and som e are negative, so if we sum m ed th em all up, th e positives and negatives
would cancel each oth er out and we would be left w ith zero. To get rid o f th e different signs,
we therefore square the deviations. This is presented in colu m n 4. N ext we add up these
values. This com es to 144. This value is know n as the sum o f th e squares. W e th en divide this
value by our sam ple size m inus one (11) w hich gives us 13.09. This value is know n as the
variance. The last step involves taking the square root o f th is num ber, w hich gives us the
standard deviation. This is 3.62. Som e students got scores w hich were close to the m ean,
other students got scores which were far away from the m ean, but what th e standard dev ia­
tion tells us is that, on average, the deviation from the m ean was 3.62 points.

Table 14.4 Calculating the standard deviation

Student X (Test score) (X - X ) (X - x )2

1 1 -5 25
2 2 -4 16
3 2 -A 16
4 3 •3 9
5 4 ■2 A
6 4 -2 A
7 8 2 A
8 9 9
9
10
A 16
11
A 16
12
16
Sum (Total)
144
Sum /N -1
144/1 1 -- 13 0°
Square root
3 62
 Q U AN T I T A T I V E ANA LYS IS

Like the m ean, the stand ard d eviation is one o f the corn erston es o f quantitative analysis. It
can be used to su m m arize the dispersio n o f a single variable w ithin our sam ple. It can also
be used to com p are th e d ispersio n am ong two or m ore sam ples. This is often o f m ore analy­
tical in terest. It can be hard to know how to in terpret the size o f a standard deviation when
there is on ly one value. Is it big or is it sm all? But when we have repeated values, we can
look at w hether th e stand ard deviation changes, and if so, w hether it gets bigger or smaller.
This is often o f c on sid erab le an alytical in terest. M easures o f v ariation have many practical
application s in p olitical research , and can be used to tell us about extrem ism , polarization,
and inequality. For exam ple, sup pose we are interested in ideological extrem ism . There is
a great deal o f talk th ese days about id eological convergen ce, m eanin g that voters have
m oved tow ards th e m id d le ground. O n e way o f ex am in in g th is is ju st to look at the m ean,
as we did ea rlier in Figu re 14.5. From th is figure, we can see that the centre has shifted, but
that it appears to shift in a cy clical fashion. However, it doesn’t tell us w hether voters have
converged or not. A n o th er way o f ex am in in g th is is to look at the standard deviation o f
le ft-rig h t scores. Th is tells us how clustered or spread out the scores are. Figure 14.6 plots
the stand ard d ev iation o f le ft-rig h t self-p lacem en t. There d oesn’t appear to have been a
great deal o f chan ge ov er tim e. The first value in th e series appears to be a bit o f an outlier,
and sin ce th en , th ere have b een som e ju m p s up and dow n, w ith perhaps a slight downward
ten d ency, sugg estin g th at som e id eological conv ergen ce has taken place. However, we
sh ould perhap s b e w ary o f draw ing stron g con clu sion s.

Descriptive inference
So far we have consid ered ways in w hich we can describ e our sam ple. But we are also in ter­
ested in m ak in g in feren ces about the popu lation from w hich the sam ple was drawn. This
allow s us to m ak e gen eralization s, and it is this ability to m ake robust generalizations about
th e w ider p opu lation th at is on e o f the key strengths o f qu antitative analysis. In order to
m ake g en eralization s, we use th e data from our sam ple to tell us som eth in g — or m ake in fer­
en ce s— ab out th e w ider w orld. In feren tial statistics allow us to build upon what we have

2.5

2.4

2.3

c 22
1 21
1 2
■o
2 19
" 1.8
1.7

1.6
1.5
1970 1975 1980 1985 1990 1995 2000 2005

Figure 14.6 Left-right self-placement in UK. standard deviation, 1970-2007

Source: Eurobarometer trendfile
H O W TO D O R E S E A R C H IN P R A C T IC E

covered so far and to m ake estim ates ab ou t popu lation values. Th ese e stim ates often tak e th e
form o f a c o n fid en c e in terv al, w hich is a range o f values w ithin w hich we are fairly sure th e
‘real’ popu lation value is c ontained. This idea o f the c o n fid en ce in terval is p articu larly u seful
w hen we are interested in com p arin g values over tim e o r betw een co u n tries, and helps us to
decid e w hether changes or d ifferences are ‘m ean in gfu l’ o r not.
B efore o u tlin in g how in feren tial statistics w ork in p ractice, it is w orth d iscu ssin g a little o f
the th eory behin d the a pproach. W e do no t go in to g reat detail here, sin ce o u r focu s is o n the
analysis and in terpretation o f th ese statistics rather th an how th ey w ork, but useful su m ­
m aries can be found in Agresti and Finlay (2 0 0 9 ). O u r sam ple is in terestin g. W e ho p e it is
reliable, and we hope that it is representative so th at we can m ake in feren ces ab ou t th e p o p u ­
lation from w hich it was drawn. But how exactly can we do th is? The c e n tra l lim it th e o re m
is what allows us to lin k our sam ple to the p opu lation . The cen tral lim it th eorem is perhaps
the m ost im portant th eorem in statistical theory. It states that if repeated ran d om sam ples
are draw n from a popu lation, th e sam p lin g d istrib u tio n o f the sam ple estim ate (fo r e x a m ­
ple, C onservative vote share) will approach norm ality. A n exam ple helps to illustrate this.
Suppose we interview 100 people and find that 45 said th ey w ould vote C on serv ative if
there were an election tom orrow . W e can th in k o f this as our sam ple estim ate. W e ho pe that
our sam ple estim ate is a good approxim ation o f th e true p opu lation value, and that in the
real world som ew here around 45% o f the population would indeed vote C onserv ative. But
if we interviewed a different 100 people, we m ight find that on ly 4 4 people said th ey would
vote C onservative, and if we interview ed a different 100 people again, we m ight find that
only 40 people said they would vote C onservative, and so on. Th is m ight sou nd like a p o ten ­
tial problem , but the beauty o f the central lim it th eorem is that it isn’t. We know that each o f
our sam ples is likely to produce a slightly different estim ate, and we also know that if we take
a large num ber o f random sam ples (o f sufficient sam ple size), m ost will com e up with a p er­
centage estim ate close to the true population value. In only a few sam ples will th e sam ple
estim ate be way o ff the m ark. In fact, the d istribution o f the sam ple estim ates (know n as the
sam pling distribution ) would approxim ate a ‘n o rm al’ d istrib ution (see Figure 14.7).

Conservative vote share (%)

Figure 14.7 The normal distribution
 Q U A N T I T A T I V E AN A LY SI S

We can th ink o f the sam pling d istribution as being a distribution o f all the possible sample
estim ates that we could draw from a population using the sam e sam ple size. If we were to
take all these possible sam ple estim ates, sum them and divide by the num ber o f samples, we
would have the exact value o f the population. The elegance o f the sam pling distribution is
that even if the population is not norm ally distributed, repeated sam ples will generate esti­
m ates that are approxim ately norm ally distributed. And, as with all distributions, the sam ­
pling d istrib ution will have a m ean, and this m ean will be the true population value. The
peak o f the sam pling d istrib ution th erefore corresp onds to the true population value for
C onservative share o f the vote (45% ).
Now, we h o p e th at o u r sam ple estim ate is ex actly the sam e as th e popu lation value or
at least p retty c lo se to it, but in tru th we do not know w here it is. O u r sam ple estim ate
could p o ten tia lly be o n e o f th e e x tre m e estim ates right o ff in the tails o f the d istrib u tio n ,
a lon g way away from th e tru e value o f th e p op u lation . H owever, if it was, we could c o n ­
sole ou rselv es th at we m u st have b een p retty un lucky to have draw n such a duff sam ple
if we had d o n e ev ery th in g right in th e sam p lin g p hase o f the research (see C hap ter 10).
C ru cia lly , it is th is id ea o f how lucky or no t we w ould have to be to draw a du ff sam ple
that u n d erp in s in feren tia l sta tistic s. So how do we decid e w hether th e sam ple we have
d raw n is o n e o f th e sam p les th at is clo se to th e tru e p op u lation value or on e that is far
away?
To answ er th is q u estio n we need to know ab ou t so m eth in g called the standard error.
T h is ca n b e used to tell us how likely it is th at o u r sam ple estim ate is close to the true
p o p u la tio n value. As w ith all d istrib u tio n s, th e sam pling d istrib u tion has a standard d ev i­
a tion. W h erea s th e m ean o f th e sam p ling d istrib u tio n is the sam e as the popu lation value,
th e sta n d a rd d ev ia tion o f th e sam p lin g d istrib u tio n is sm aller th an the p opu lation stan d ­
ard d ev ia tio n . Th is is b eca u se th ere ten d s to be m o re v ariation betw een values for in d i­
vid uals th a n th ere d oes b etw een sta tistics su m m arized for sam ples. Th is m akes intuitive
sen se. The m ean in c o m e in B rita in is a rou nd £ 2 0 ,0 0 0 p er year. In dividuals e arn in g £ 5 0 ,0 0 0
per year or m o re are u n c o m m o n , th ou gh no t u n heard o f (ap p roxim ately 5% in 2 0 0 7 ).
H ow ever, a ra n d om sam ple o f 1000 p eople w ith a m ean in co m e o f £ 5 0 ,0 0 0 per year would
b e ex c ep tio n a lly rare.
The stand ard dev iation o f the sam pling distrib u tion has a special nam e: the standard
error. Prob ab ility th eo ry tells us that in 95% o f sam ples that we draw, the population value
will be w ithin two (actu ally 1.96) standard errors o f the sam ple estim ate. W ith nom inal v ari­
ables, we ca n estim ate th e stand ard error o f a p rop ortion . W ith interval (and at a stretch
ord in al) variables we can estim ate the standard error o f a m ean. The way in w hich we do this
is slightly different, but in b oth cases the size o f the standard erro r is partly a fun ction o f
sam ple size. Th ese poin ts are not on ly o f statistical interest, but are o f great practical use as
well. W e can u tilize the idea o f th e sam pling distrib u tion and the standard erro r to m ake
in feren ces ab out p opu lation s. This is discu ssed below.

Population estimates
A s s o o n as w e m o v e in to th e realm o f in fe re n tia l s ta tistic s, th e re are a w h o le h o st o f
th in g s w e c a n d o. T h e fir s t, m o st b a sic a p p lic a tio n it th at we can m ak e g u esses ab ou t
th e tr u e p o p u la tio n v a lu e th a t we are in te re ste d in . W e can d o th is fo r n o m in a l, o rd in a l,
H O W TO D O R E S E A R C H IN P R A C T IC E

and in terv a l data. F or n o m in a l v a ria b les, we ca n m ak e a g u ess a b o u t th e p o p u la tio n

p ro p o r tio n , su ch as w hat is th e p r o p o r tio n o f th e p o p u la tio n th a t w ill v o te C o n se rv a tiv e .
F or o rd in a l and in terv al v ariab les, we c a n also m ak e a g u ess a b o u t th e p o p u la tio n m e a n ,
su ch as w hat is th e m ean p o sitio n on th e le f t -r ig h t sca le o f th e p o p u la tio n as a w h o le.

Confidence intervals for proportions

O n e c o m m o n use o f surveys and o p in ion polls is to track likely voting in ten tio n s in th e g e n ­
eral election. This is som eth in g we frequen tly read about in th e new spapers. F or exam ple, a
survey may report that if th ere w ere a general election tom orrow , 4 4% o f th e sam ple say th ey
would vote C onservative. This is in terestin g to know, but m o re in terestin g is w hat it tells us
about our estim ate o f what th e true level o f sup port for th e C on serv ative p arty is in th e
population as a whole. W e th in k it is probably a round 44% , but we would b e unw ise to state
that it is exactly 4 4% . W hat we have is an estim ate.
It is co m m o n to accom p an y th is estim ate w ith a m arg in o f error. S u rv ey co m p a n ie s
often say that th e m argin o f e rro r is plus or m in u s 3 p ercen tag e p oin ts. T h is m e an s th at
th e b est guess is 44% , but th e exp ected tru e v alue co u ld b e b etw een 41 and 4 7 % (4 4 % plus
or m inus 3 percentage p o in ts). T h is is qu ite a big range. D u rin g e le ctio n c am p aig n s, c o m ­
m en tators can get q u ite ex cited ab out sm all ch an g es in th e p olls, b ut m o st o f th e tim e,
when a pa rty ’s sh are o f th e vote appears to go up o r dow n by a few p ercen tag e p o in ts, it is
still well w ithin the m argin o f error. It does no t m ean th at th e u n d erly in g level o f su p p o rt
for th e party in the pop u lation has chan ged , but rath er ju st reflects w hat is kn o w n as
sam pling v ariation .
The tech nical term for this m argin o f erro r is a con fid en ce interval. Th is sectio n describ es
the calculation behind a confid ence interval, though in p ractice it is not som eth in g that you
need to know how to calculate, since m ost statistical softw are program m es will do it for you.
To calculate a confidence interval we draw on probability theory, and the ideas o f th e sa m ­
pling distribution and the standard error discussed above. Sin ce we know that 95% o f the
tim e the population value will fall w ithin 1.96 standard errors o f the sam ple estim ate, we can
calculate the range within w hich we th ink the true population value lies. To estim ate th is we
need to calculate the standard error. Because we are dealing with a no m in al variable, we
calculate the standard error for a proportion. But the procedure is m uch the sam e as that for
calculating the standard error for the mean.
T h e fo r m u la fo r th e s ta n d a r d e r r o r o f a p r o p o r tio n is:

_ fT/j, ) ( i - p >
‘ ' = \ N
w h e re

Si. = th e sta n d a rd e r r o r o f th e p r o p o r tio n ;

/•* = th e s a m p le p r o p o r tio n ;
iV - th e sa m p le size.

H a vin g c a lcu la te d th a t s ta n d a rd e r r o r , w e c a n n o w e s tim a te th e r a n g e w ith in w h ic h th e p o p -

u a lio n valu e is lik ely to be. I h is is o u r c o n fid e n c e in te r v a l. I f w e s p e c if y a 9 5 % c o n fi d e n c e
 Q U A N T I T A T I V E ANA LY SI S

in terval, it m eans that we th ink our range will correctly contain the true population value
95% of the tim e. This is expressed below:

If we assum e our sam ple size is 1,000, and the percentage who say they will vote C on serv a­
tive is 44% , then we can plug in the num bers as follows.

ci = 0 .4 4 ± 0 .03.

The 9 5% confid ence interval is therefore the sam ple proportion (0 .4 4 ) plus or m inus 0.03. So
in percentage term s we can say that we th ink the true level o f C onservative support lies som e­
w here betw een 4 1% and 4 7% (44% plus or m inus 3 percentage points). This is our best guess.
O f course, we don’t know w hether our guess is correct or not. We may be wrong. But we can
say that m ost o f the tim e our guess will be correct. In fact, we can say that 95% o f the tim e our
guess will be right. This, o f course, leaves us with the possibility o f being wrong 5% o f the
tim e, w hich will happen w hen we draw one o f those dodgy sam ples right off in the tails o f the
sam pling d istribution . This is know n as sam pling error, w hich is the culprit frequently
blam ed when surprisin g o n e -o ff polls are recorded w hich seem out o f step with what else has
b een recorded.
T h ere is th us a trade-off" betw een precision and confid ence. If we were not happy to be
c o rre c t 9 5 % o f th e tim e, we could extend our co n fid en ce interval. Prob ab ility th eory tells
us that 9 9 % o f th e tim e the p opu lation value will be w ithin 2.58 standard errors o f the
m ean. T h is will give us a co n fid en ce interval for the true level o f C onserv ative sup port o f
b etw een 4 0 % and 4 8 % (4 4 % plus or m inus 4 p ercentage p oin ts). Now we have m ore c o n fi­
d en ce th at o u r estim ate will co n tain th e true p opu lation value, but our estim ate is w ider
and less precise. F or th is reason, th e 95% co n fid en ce in terval is generally accepted as the
d efault p osition .
H aving established a con fid en ce interval for one prop ortion , we can then use this to
m ak e a c om p a rison with th e con fid en ce interval o f an o th er prop ortion , to see if the d iffer­
en ce, say betw een th e L abour and C onserv ative share o f the vote, could ju st be down to
sam pling v ariation or w hether it represents a real ‘statistical difference’. So, for exam ple, if
ju st b efore an electio n we ca rry o ut a survey and find that the C onserv atives are on 44% and
L ab ou r are on 4 1 % , how con fid en t can we be that th is represents a ‘real’ difference in public
o p in ion ?
O u r sam p le e stim a te for L ab ou r m ay be 4 1 % , w hich is som ew h at low er th an o u r e s ti­
m ate o f th e C o n se rv a tiv e vote sh are, b ut b efo re ru sh in g to any co n c lu sio n , th e im p o rtan t
th in g is to in sp ect th e co n fid e n c e in terval for L ab o u r’s vote sh are. The 95% c o n fid en ce
in terv a l for L a b o u r vote is b etw een 38% and 44% (4 1 % plus or m in u s 3 percen tage
p o in ts). T h ere is th u s so m e ov erlap b etw een o u r tw o estim ates. It could b e th at th e tru e
p o p u la tio n v alue is at th e to p en d o f th e L ab o u r estim a te and th e b o tto m end o f th e
H O W TO DO R E S E A R C H IN P R A C T I C E

C o n se rv a tiv e estim a te, in w h ich c a se th ere m ay b e n o real d iffe re n ce in th e ir r e sp ectiv e

sh ares o f th e vote.
Th is leads us to a slightly different qu estio n th at we w ant to answer. R ath er th an try in g to
find out what the true level o f sup port for each party is in th e pop u lation , we now w ant to
know w hether th ere is a real d ifference betw een th eir vote shares. D o we have en o u gh e v i­
d ence to reject the idea that they have equal sup port? Th is in tro d u ces us to th e idea o f
hypothesis testin g and the null hyp othesis (w hich we e x am in e in m o re d etail in C h ap ter 15).
O u r null hypothesis is that in the p opu lation th ere is n o d ifferen ce b etw een th e L ab o u r share
o f the vote and the C onserv ative share o f th e vote.

Confidence intervals for the difference between two proportions

A lthough it can be interesting to m ake estim ates about the tru e lev el— or p ro p o rtio n — o f a
particular variable in the popu lation, it is often m ore in terestin g to know ab ou t chan ge or
com parison. For exam ple, we m ight be interested to see if L ab ou r sup port chan ged when
Brow n too k over from Blair. Alternatively, we m ight b e interested to see if tru st in p olitician s
changed after the 2 0 0 9 expenses scandal. O r we m ay w ant to know if public sup port for
d em ocracy is higher in A fghan istan or Iraq.
Obviously, w hen we m ake these sorts o f com p arisons, we have to rely o n survey data that
has been asked in exactly the sam e way, sin ce even sm all changes in qu estion w ording can
influence the response (see C hapter 10). However, th ere are now a nu m b er o f surveys that
have been carried out in the sam e way over tim e and betw een cou n tries. O n e such survey
series is the Audit o f Political Engagem ent, carried out by the H ansard Society, w hich tracks
British public attitudes tow ards politics over tim e. In the w ake o f th e p arliam entary expenses
scandal o f May 20 0 9 , when there w ere widespread allegations that m any M Ps were m isusing
or abusing their official allow ances, there were w idespread co n ce rn s that th e reputation o f
M Ps in general had been dealt a severe blow. However, oth ers argued that M Ps had never
been held in particularly high regard, and so the scandal ju st reinforced w hat the public
thought they already knew, rather than changed th eir opin ion . To help us look at th ese d if­
ferent hypotheses, we can com pare public attitudes tow ards M Ps from b efore the scandal
broke with attitudes after. Table 14.5 presents data collected in 2 0 0 6 and 2009.
C om paring the two colum ns, for N ovem ber 200 6 and N ovem ber 2 009, we can see there
is not much change in the percentage o f people who are very or fairly satisfied with the way
in w hich M Ps do their jo b . There has been a slight decline in the percentage w ho are neither
satisfied nor dissatisfied, and a slight increase in the p ercentage o f those that are fairly or very
dissatisfied. O verall, the percentage who said they were dissatisfied (either fairly or very)
with the ways in which M Ps in general are doing their jo b increased from 36% to 44% .
F ro m th is it lo o k s like th e re m a y h ave b e e n a sligh t c h a n g e . B u t is th is c h a n g e re a l, o r c o u ld
it ju st be th e resu lt o f sa m p lin g v a r ia tio n ? M ay b e th e tru e level o f d is s a tis fa c tio n in th e p o p ­
u lation is 4 0 % , an d th e first sa m p le u n d e re stim a te d it a little b it, a n d th e s e c o n d sa m p le
o v e re s tim a te d a little bit. T h is is a d istin c t p ossib ility , a n d o n e th a t w e w o u ld w an t to ru le o u t
b efore in fe rrin g th at an y real c h a n g e h ad ta k e n p la ce in p u b lic p e rc e p tio n s a b o u t h o w well
(o r h ow b ad ly ) B ritish M P s a re d o in g th e ir job . T o try a n d d istin g u is h b e tw e e n s a m p lin g
v a ria tio n an d real v a ria tio n w e u se te sts o f sta tis tic a l sig n ifica n c e . T h is d ra w s o n th e id ea o f
 Q U A N T I T A T I V E ANA LY SI S

Table 14.5 Are you satisfied or dissatisfied with the way MPs in general are doing their job?

N ovem ber 2006 Novem ber 2009

Very satisfied 2
Fairly satisfied 29
Neither satisfied nor dissatisfied 2/ /'i
Fairly dissatisfied 26
Very dissatisfied 10
Don't know 6
Dissatisfied (fairly + very) 36 44
N 1282 11'/.
Source. Hansard Society. Audit of Political Engagement 7.

confid ence intervals discu ssed above. We can estim ate a confidence interval for dissatisfac­
tion w ith M Ps for each o f the surveys. This is done below.
C on fid en ce interval for first survey in 2006:

36 *(1 0 0 - 36)
= 3 6 ± 1.96,
V 1282
= 3 6 ± 2 .63
= {3 3 .4 , 3 8 .6 }.

C o n fid e n ce in terval for secon d survey in 2009:

/44 *(1
(1 0 0 -4 4 )

1—
= 4 4 ± 2 .8 6
= { 4 1 .1 ,4 6 .9 } .

N ote th at we can express th e con fid en ce intervals in term s o f percentages or proportions.

This tim e we express th em in term s o f percentages. W e can see that the two confid ence in ter­
vals do not overlap. The top band o f the interval in 2 0 0 6 is lower than the lowest band o f the
in terval in 2 0 0 9 . Th is suggests that th ere has b een som e real change. However, we can go one
step fu rth er th an th is, and m ake an estim ate about how m u ch change we th in k has taken
place. In ou r tw o surveys, we observed an in crease in levels o f d issatisfaction o f 8 percentage
poin ts. W h a t we w ant to know is th at if th ere had been no real in crease in the population,
how likely is it th at we w ould have drawn two sam ples in w hich we observed an in crease o f
8 percen tag e points?
T h e way we do th is is to create a con fid en ce in terval for the difference betw een ou r two
p rop ortion s. O u r sam pling d istrib u tion is now th e difference betw een the two sam ple p ro ­
p o rtio n s. T h is d ifference, as p er th e central lim it th eorem , will b e norm ally distributed. We
k now th at 9 5 % o f th e tim e th e ‘tru e’ difference in the popu lation will lie w ithin 1.96 standard
H O W T O D O R E S E A R C H IN P R A C T I C E

erro rs o f the sam ple d ifference, and w hat we w ant to kno w is th e lik e lih o o d o f th is range
includ ing 0, m eanin g th at th ere has b een n o chan ge. W e can calcu late th is in m u ch th e sam e
way as we did b efore, but th e on ly slight difference is in how we calcu late th e stand ard erro r
o f the sam pling d istrib ution . To do th is we take in to a cco u n t b o th sam ples ( S I and S 2 ). T h e
form ula for the standard e rro r o f th e difference betw een tw o p ro p o rtio n s is:

P 'd - p ,) p,2( i - p 2)

”l M2

S£ _ (3 6 * 6 4 ) + ( 4 4 »5 6)

J \ 1282 + 1156

14768
SE = . -------
* \ 2438

SEd = >/l%

= 1.40.

Having obtained an estim ate o f the standard error o f the d istrib u tion , we can now calculate
th e confid ence interval in the sam e way:

ci = 8± 1.9 6*1.40
= 8 ± 2.74
= {5.3, 10.7}.

We can see our confidence interval ranges from 5.3 to 10.7. This is how m u ch chan ge we
th ink there has been in the level o f d issatisfaction with M Ps betw een our two surveys. We
can see that 0 is not contained w ithin this confid ence in terval. O u r estim ate is that th ere has
been change o f som ew here betw een 5.3 percentage points and 10.7 percentage points. We
are confident that this represents real change. And we will be co rrect in th is estim ate 95% o f
the time.

Confidence intervals for means

W ith ordinal and interval data, we can estim ate th e m argin o f e rro r associated with our
sam ple m eans using the sam e general logic. To do this we calcu late a statistic called the
standard error of the m ean. This is based on the standard deviation o f th e sam ple. As
before, we can say that 95% o f the tim e the true value o f the p opu lation will fall w ithin plus
or m inus 1.96 standard errors o f the sam ple mean.
Ihe formula for the standard error o f a m ean is:
 Q U A N T I T A T I V E ANA LY SI S

where

SE. = the standard error o f the m ean;

5 = the sam ple standard deviation;
N - the sam ple size.

The 95% confid ence interval can then be calculated in the sam e way.

These confid ence intervals are particularly useful if we are in terested in w hether things have
changed over tim e or not. Suppose we are interested in the ideology o f the electorate in B rit­
ain (as we d iscussed previously), and w hether there has been any real m ovem ent over tim e.
U sing Eurobarom eter, which has asked the sam e question sin ce 1973, we can calculate the
confid ence interval for each sam ple estim ate and com pare them to each other. This is shown
in Figure 14.8. The circles indicate the m ean and the ‘w hiskers’ indicate the 95% confidence
interval for the m ean.

6.5 -

4.5 -

I I I I I I I 1—
I I— T i i i r r i i i i i r i i i i i i i i

I I I I
Year

Figure 14.8 M ean left-right self-placem ent w ith 95% confidence intervals
H O W TO D O R E S E A R C H IN P R A C T I C E

T he first th ing to n o tice is th at th e first p o in t in th e tim e series appears to b e so m eth in g o f

an outlier. It m ay be that betw een 1973 and 1976 th e electo rate u n d erw en t a d ram atic sh ift
to th e right, but it is probably m o re plausible to put th e a n o m alou s estim ate o f 1 973 d ow n to
sam pling error. Sin ce th en , we can see th at th ere have b een som e flu ctu ation s up and dow n.
By and large, the con fid en ce in tervals for two ad jacen t estim ates ten d to overlap, in d icatin g
little sh o rt-term change, but over th e long run we can see that, for exam ple, th e e lecto rate is
significantly m o re to the left in th e m id -1 9 9 0 s th an it was in th e late 1980s. T h e advantage o f
utilizing con fid en ce intervals is that we are less likely to o v er-in terp ret sm all chan ges, and so
have a m o re robust basis for assessing differences. A nalysis o f th ese d ifferen ces o v er tim e can
be m ore fully explored using tim e series analysis.

Conclusions
This chapter has provided an introduction to the first steps in analysing quantitative data. In doing so,
w e have encountered some of the building blocks of quantitative analysis: the m ean, the standard
deviation, and the central limit theorem . The latter provides the link between samples and popula­
tions which allows us to make inferences. Description is an im portant part o f political research. It
enables us to set the scene and describe the nature of the problem that we m ay then seek to go on to
explain. Descriptive inference allows us to link the sam ple to the population. It not only enables us to
make generalizations, but also allows us to assess whether things have changed o r w hether there are
significant differences between two sample estimates. This is valuable w hen w e move to wards
examining how things change over tim e or vary between countries. However, this univariate
descriptive analysis is also often ju st the first step in the analysis. Often w e are interested in developing
explanations about w hy things vary or w hy they differ and this takes us towards developing more
complicated models, which include different explanatory factors. W e turn to this in the next chapter.

Questions
• W hat is the difference between description and explanation?

• How can quantitative methods be used to describe different political phenom ena? W hat examples
can you think of?

• What makes for good description?

• What is the difference between description and inference?

• What are the different levels of measurement? Give examples.

• What is missing data? What problems can it cause? W hat can be done about it?

• How do we decide what is typical? What are the different measures of central tendency and how do
they differ from each other?

• When is it appropriate to use the median? The mode? And the mean?
• What do we mean by a confidence interval?
• What issues influence our ability to make inferences?

© Guide to Further Reading

Agresti, A. and B. Finlay (2009), Statistical M ethods fo r the Social Sciences, 4th edition (U pper
Saddle River, Nj: P earson-Prentice H all).
 Q U A N T IT A T I V E AN A LY SI S 365

Covers the basics of statistical description and inference, as well as more advanced topics on
regression methods, including multiple regression, analysis of covariance, logistic regression, and
generalized linear models.

Johnson, Janet Buttolph, Henry Reynolds, and Jason Mycoff (2007), Political Science Research
Methods, 6th edition (Washington, DC: CQ Press).
Provides an introduction to quantitative analysis and covers some more advanced topics, with lots of
examples from political research.

Carlson, James and Mark Hyde (2002), Doing Empirical Political Research (Boston, MA: Houghton
Mifflin).
Provides an introduction to quantitative analysis and covers some more advanced topics, with lots of
examples from political research.

de Vaus, David (2002), Surveys in Social Research (London: Routledge).

A detailed and accessible introduction to quantitative methods, with easy-to-read chapters on how to
carry out analysis and interpret the results.

Field, A. (2009), Discovering Statistics Using SPSS: (And Sex and Drugs and Rock 'n' Roll) (London: Sage
Publications).
Provides an easy-to-follow introduction to quantitative analysis and step-by-step instructions on how
to use the SPSS statistical software package to do your own analysis.

Rowntree, Derek (2000), Statistics without Tears: An Introduction for Non-Mathematicians

(Harmondsworth: Penguin).
An easy-to-read introduction to statistical methods.

e References

Agresti, A. and B. Finlay (2009), Statistical Methods ---- and Michael D. Ward (2006), 'Diffusion and
for the Social Sciences, 4th edition (Upper Saddle the International Context of Democratization',
River, NJ: Pearson-Prentice Hall). International Organization 60(4): 911 -33.
Bartle.John, Sebastion Dellepiane, and James A. King, Gary, Robert Keohane, and Sidney Verba
Stimson (2011), The Moving Centre: Preferences (1994), Designing Social Inquiry: Scientific
for Government Activity in Britain: 1950-2005.' Inference in Qualitative Research (Princeton:
British Journal o f Political Science 41:259-85. Princeton University Press).
Clarke, Harold, D. Sanders, M. Stewart, and Przeworski, A., M. Alvarez, J. Cheibub, and
P. Whiteley (2004), Political Choice in Britain F. Limongi (2000), Democracy and Development:
(Oxford: Oxford University Press). Political Institutions and Well-Being in the World.
1950-1990 (Cambridge: Cambridge University
---- (2009), Performance Politics and the British
Voter (Oxford: Oxford University Press). Press).
Wilkinson, S. (2004), Votes and Violence: Electoral
Franklin, M. (2004), Voter Turnout and the
Competition and Ethnic Riots in India (Cambridge:
Dynamics of Electoral Competition in Established
Cambridge University Press).
Democracies since 1945 (Cambridge: Cambridge
University Press). Wlezien, Christopher (1995), The Public as
Thermostat: Dynamics of Preferences for
Gleditsch, Kristian Skrede (2007), Transnational
Spending1, American Journal of Political Science
Dimensions of Civil War”, Journal o f Peace
39(4): 981-1000.
Research 44(3): 293-309.
 Patterns of Association:

• Bivariate Analysis

Chapter Summary
This chapter exam ines the association between two variables, and builds upon the
previous chapter by introducing ideas about association and causality. In doing so,
it provides a link between description and explanation. In particular, w e fo cus on
hypothesis testing and significance tests; and how w e can describe the pattern of as­
sociation between two variables. W e in troduce tw o of the most w idely used statistical
analysis techniques in political research, ordinary least squares (OLS) regression and
cross-tabulation. The chapter considers:

• bivariate analysis;

• cross-tabulation;

• significance testing;

• null hypothesis;

• chi square;

• correlation.

Introduction
In C hapter 14 we exam ined the different ways in w hich we can d escrib e the d istrib u tion o f
one variable using un iv ariate statistics. This chapter builds on these ideas. It is on e th in g to
show that people vary in term s o f th eir left-rig h t ideology or env iron m en tal aw areness, or
that som e countries are m ore dem ocratic or m ore prone to war than oth ers, it is quite a no ther
thing to try and explain or account for these variations. W hy are som e people m o re c o n ­
cerned about the environm ent than others? Is it because o f th eir age? T h eir class? Their
education? W hy are som e people m ore left wing than others? If we find that w orking-class
people are m ore left wing than m iddle-class people, we m ight say that there is an association
between class and left-rig h t attitudes. If we are interested in explaining the causes o f id eol­
ogy, then we might hypothesize that class is an im portant factor. But we m ight also be in ter­
ested in exploring why this is the case, and unpicking the causal m echanism s. W hy are
working-class people m ore left wing? Is it because o f their location in the labour m arket? Is
it because of their socialization experiences? O r is it simply a function o f political m o biliza­
tion and persuasion by the parties that they support?
Bivariate analysis allows us to explore these issues, and exam ine how two variables are
related. Bivariate analysis includes a num ber of different m ethods. The type o f m ethod that
is appropriate depends upon the level o f m easurem en t o f our variables. As we discussed in
 Bl VAR I ATE AN A LY SI S

C hapter 14, we use different m ethods for exam in ing the relationship between categorical
variables (n om in al or o rd inal) and continu ous variables (interval). W hen both our variables
are interval level, we can use O LS regression (or correlation ); and when both our variables
are categorical, we can use cross-tabulation s. W hen our dependent variable is interval and
our in d epend en t variable is categorical, we can do a com parison o f m eans, as discussed in
C hapter 14. A lthough the m ethod s used are som ew hat different, they serve the sam e basic
purpose. They allow us to answer the follow ing questions: Is there a relationship between
two variables? A nd if so, what form does the relationship take? How can we describe it?
T he chapter is divided in to th ree m ain sections. The first section provides an in troduction
to the logic o f bivariate analysis and discusses the difference betw een association and cau ­
sality. The next sectio n exam in es the association betw een two interval-level variables using
c orrela tion and regression, and the final section exam in es the relationship betw een two
categ o rical variables using cro ss-tab ulation s.

The principles of bivariate analysis

B iv ariate analysis is an essential tool for getting to know your data, and helping you to
id en tify p attern s o f asso cia tio n betw een two variables. The variable that refers to the phe­
n o m en o n that we are tryin g to explain is often know n as the d ep en d en t v ariab le (or
r esp o n s e v a ria b le ). The v ariable th at we th in k is im p ortan t for explaining the dependent
variable is know n as the in d ep en d en t v ariab le, or ex p la n a to ry v ariab le (see C hapter 6).
So, for e xam ple, if we are in terested in eth n ic co n flict in India, ou r depen dent variable may
be w h e th er a p a rticu la r region ex p erien ced a v iolen t clash betw een H indus and M uslim s
(o r no t) w ithin a certa in p eriod o f tim e (say five years). This is the th ing that we want to
exp lain. W hy are som e regions m o re prone to co n flict than oth ers? O ur in dependen t v ari­
able refers to a fa c to r th at we th in k m ay be im p ortan t for exp laining this. A ccord ing to
A sh utosh V arsh n ey’s (2 0 0 2 ) w ork on the su b ject, one im p ortant factor refers to the stru c­
ture o f civil so cie ty and, he argues, regions w here th ere are dense in tra -e th n ic social n e t­
w orks th at lin k d ifferent c o m m u n ities tog eth er are less prone to c o n flict than regions
w here th e tw o c o m m u n ities are m o re socially segregated. W e can sp ecify th is hypothesized
rela tio n sh ip as a d iagram :

X ----------------------------------------- ► y
where:

Y is level o f eth n ic c o n flict in a region (ou r depend ent v ariable); and

X is level o f in tra -e th n ic civil society in a region (ou r in dependen t or explanatory
v ariable).

Bivariate analysis is a way o f establishing w hether or not there is indeed a relationship between
two variables, as the th eory predicts. It allows us to com pare our theoretical expectations
against evidence from the real world to see if the th eory is supported by what we observe or
not. In its sim plest bivariate form , we have o ne dependent variable (w hich we are interested in
trying to explain), such as eth n ic conflict, and on e independent variable (w hich we th ink influ­
ences, o r causes, o r helps to explain in som e way our dependent variable), such as the structure
H O W TO D O R E S E A R C H IN P R A C T I C E

o f in tra-eth n ic civil society. W h en we have m ore th an o n e in d ependen t v ariable (i.e. m o re than

one explanatory factor), we m ove from bivariate analysis to m ultivariate analysis (w hich we
deal with in the next chapter), but m any o f the analytic principles are m u ch th e sam e.
Th e hard work in quantitative analysis is not to be found in ca rry in g out th e a nalysis itself.
T h is is relatively straightforw ard, and o n ce a few gen eral tech n iq u es have b een acqu ired , th ey
can be applied in a wide variety o f ways. R ath er th e hard w ork is th eo retical, d eveloping
hypotheses and in terestin g research qu estio ns that engage w ith p rio r r esearch b ut a lso tak e it
in new d irection s (see C hapter 5).
Regardless o f w hich tech n iq u e we use (cro ss-tab s o r regression), th e p rincip les o f b ivariate
analysis are m uch the sam e. Indeed, the p rinciples o f qu antitative m e th od s in gen eral are
m uch the sam e. We start o ff with a research q u estio n — or idea. That is, we seek to d escrib e,
or explain or predict, som e social or p olitical p h e n o m en o n (o u r dep en d ent v ariables), and
we have certain th eoretical ex p ectations about w hat facto r (or factors) m ay b e im p ortan t in
helping us do this (ou r in dependen t variables) (see C hapters 5 and 6). W e th en co lle ct data
and develop ind icators to m easure ou r different con cep ts (see C hapters 7 and 10). Th en,
finally, we analyse the data in order to e stablish w hether our th eoretical e x p ectatio n s are su p ­
ported by w hat we observe in the real w orld o r not. This last part is relatively straigh tforw ard,
and involves ob tain ing answers to th ree basic questions.

Is there a relationship between the variables?

The first question that we w ant to answ er is w hether th e variables are related to, or a ssociated
with, each oth er or not. Two variables are associated if know ing som eth in g ab out on e v ari­
able helps us to know som eth ing about th e oth er variable. In form al term s, we can say that
variables are associated if the cond ition al d istrib ution o f one variable v aries acro ss th e levels
o f the other. For exam ple, if econ om ically underdeveloped cou n tries tend to be a u th o ritar­
ian and econ om ically developed cou n tries tend to be d em ocratic, we could say that e c o n ­
om ic developm ent and d em ocracy are associated; that is, the level o f d em o cracy varies
across levels o f econ om ic developm ent. In oth er words, we first want to assess w hether the
sample associations that we observe are statistically significant, or not.
U nderpinning significance testing is the idea o f hypothesis testing, and the nu ll h y p o th e­
sis. The null hypothesis is the hypothesis that we em pirically test, and is norm ally the o p p o ­
site o f what we theoretically expect to observe. In this sense, it has p arallels with Karl P op pers
'falsificationist thesis’, which we discussed in C hapters 2 and 3. This principle o f falsification
lies at the heart o f hypothesis testing in quantitative m ethods. We do not therefore test to see
if there is a relationship between two variables, but rather test to see if we can reject (or falsify)
the hypothesis that there is no relationship. So, for exam ple, if we are interested in the rela­
tionship between econ om ic developm ent and dem ocracy, we may th ink that rich coun tries
tend to be more dem ocratic than poorer countries. We do not directly test this though.
Rather, our null hypothesis is that there is no association betw een econ om ic developm ent
and democracy. If we find evidence to reject the null hypothesis, then we can reject the idea
that there is no relationship, and infer that there probably is a relationship. W h eth er we reject
(or tail to reject) the null hypothesis is incredibly easy to decide. We will discuss this in m ore
detail later in the chapter, but the standard practice is sim ply to reject the null hypothesis if
som ething called the p value is less than 0.05. I h e p value refers to the probability that the null
 BI V AR I AT E A NA LYS IS

hypothesis is true. So a p value o f 0.05 tells us that there is a 5% chance that the null hypoth­
esis is true. It the probability o f it being true is very low (by convention less than 5% ), then we
assum e that it is probably false.

How can we describe the relationship between the variables?

It we find ev id en ce to suggest that two variables are statistically associated, we could then go
on to d escrib e the pattern o f this association. Principally, we are interested in w hether the
pattern o f the relationship con fo rm s to our theoretical expectations or not (i.e. that richer
cou n tries are m ore d em ocra tic). How we d escribe this association varies according to the
m ethod that we use (cro ss-tab s or regression). W ith cro ss-tab s we describe the association
with referen ce to percentages, and with regression we d escribe the association with refer­
en ce to so m eth in g called the lin e o f b est fit. But in either case we are principally concerned
with w hether th e pattern that we observe is in the expected direction predicted by our
theory.

How good is our explanation?

Fin ally th e th ird q u estio n th at we need to bear in m ind is to do with how g ood — or how
ro b u st— we th in k o u r find ings are. Put simply, are they conv in cing? O n what grounds
could they be challenged ? Invariably, with bivariate analysis our explanation will be partial
sin ce we are c o n sid erin g th e im pact o f on ly one variable on the depend ent variable. We
th en need to th in k ab out w hat oth er variables or factors may be im portant, and w hether
ou r results still stand up w hen we take th ese into accoun t. This we deal with in the next
chapter.

Association and causality

In m an y cases, id en tify in g patterns o f association betw een two variables helps us to make
claim s about causality. W e can th ink about this in d eterm inistic term s, or in probabilistic
term s (see C h ap ter 6 ). However, it m ust be rem em bered that ju st because two variables are
associated with each other, it does no t m ean that the relationship is causal. And trying to
d em on stra te that a relationship is causal is not straightforw ard. A causal relationship can be
th ou g h t o f as on e w here X — our in d epend en t variable— causes Y— our dependent variable.
F or exam ple, does ec o n o m ic developm ent in fluen ce (or cause) dem ocracy? D oes eth nic
d iversity in flu en ce (o r cause) civil war? For a relationship to be considered causal, we need
to establish th ree c on d ition s. First, th ere m ust be statistical association . If two variables are
not associated w ith each other, th en th ere can n ot be a causal relationship. However, associa­
tion by itself is no t en o u gh to suggest causality. It is a n ecessary but not sufficient condition .
S econ d , th ere m ust b e th e c o rrect tem poral ordering. For variable X to cause variable Y, X
m u st b e tem p orally p rio r to Y. So m etim es th is is easy to establish. It is clear, for exam ple, that
a p erso n s age is tem p orally p rio r to th eir d ecision to go out to vote. We could not argue that
a b sta in in g m akes you younger. But at oth er tim es, it is less clear cut, and it is m ore difficult
to d isen ta n g le th e tem poral ord ering, or d irection o f causality. For exam ple, does econ om ic
H O W TO DO R E S E A R C H IN P R A C T I C E

developm ent m ake a co u n try m o re d em o cratic? O r could it be th e o th e r way rou n d ? D o es

b eing dem ocra tic m ake a co u n try rich er? T he th eo ry m ay be u n c erta in , and q u estio n s o f th is
type bedevil em pirical research. D oes a cause b or d oes b cau se a? O r in d eed is th ere a r e c ip ­
rocal relationship?
Th ird , we m u st ru le out altern ativ e e x p lan atio n s. W e w ill retu rn to th is p o in t in th e
n ex t ch a p ter w hen we co n sid er m u ltiv ariate analysis. H ow ever, ev en w h en th e se r e q u ire ­
m en ts are all m et, we can still o n ly m ak e ten tativ e cla im s a b o u t cau sality, an d p erh ap s
w hat we have achieved can b etter be d escrib ed as ‘ro b u st a sso c ia tio n ’. U ltim ately, th e n ,
even if we develop th eo retica l h y p o th e ses th at are cau sal in natu re, w h en we c o m e to
analysing th e data, all we can do is see w h e th er th e p attern o f a sso cia tio n is c o n s iste n t
with our ex p ec ta tio n s or not.

Data considerations for bivariate analysis

Bivariate analysis in cludes a nu m b er o f d ifferent m eth od s. The type o f m e th o d th at is
appropriate d epends upon the level o f m easu rem en t o f o u r v ariables. In th e follo w ing s e c ­
tions, we exam in e the association betw een two in terval-lev el variables using O L S reg res­
sion, and then consid er the association betw een two categ o rical-lev el variables using
cro ss-tab ulation s.

Bivariate analysis for interval data

There are a nu m ber o f ways in w hich we can ex am in e th e relatio nship b etw een tw o in te rv a l-
level variables. The m ost com m on tech niq u e is sim ple lin ear regression (O L S regression ),
and Pearson’s correlation . Both tech niq ues are closely related, and we d iscu ss th em in detail
below. The w orkhorse for b oth tech niq ues is the scatter plot. Bivariate analysis w ith in ter-
val-level data can best be depicted th rou gh the use o f figures (at least w hen th ere is a sm all
am ount o f cases), as this creates a clear visual representation o f how th e two v ariables are
related (or not). The pattern o f this relationship can th en be sum m arized using statistics.
This section focuses on sim ple lin ear regression, w hich is on e o f th e m o st widely used
statistical tech niq ues in p olitical research , and w hich can be used b o th for biv ariate and
m ultivariate analysis. The first part o f th is section ex am in es how scatter plots can be used to
depict the relationship betw een two interval-level variables. The secon d part th en look s at
how we can sum m arize this relationship using the lin e o f best fit— o th erw ise know n as the
regression line. The third part then moves on to qu estio ns o f s ign ificance, and d iscu sses
how we can make inferences about w hether the sam ple asso ciatio n exists in th e w ider
population.

Bivariate analysis using scatter plots: describing the relationship

i igures p rov ide an in tu itive w ay o f e x a m in in g th e re la tio n sh ip b e tw een tw o in terv al-lev el
variab les w hen we h ave relatively few ca se s. T h e y visu ally re p re s e n t th e im p o rta n t c h a r a c t e r ­
istics of th e relatio nsh ip , an d just by e y e -b a llin g th e s ca tte r p lot, w e c a n im m e d ia te ly g et a
rou^h idea of w h e th e r th e re is a re la tio n sh ip o r n o t, a n d if th e re is, w h e th e r it is p o sitiv e o r
 B I V A RI A T E AN A LY SI S

negative and w hether it is weak or strong. In this section, we exam ine how figures can be used
to depict the association between two interval-level variables, and how this association can be
sum m arized using the line o f best fit. The line o f best fit is also known as the regression line,
which form s the basis o f sim ple linear regression analysis, which we will exam ine in more
detail.
To illustrate these techniques let us consider the relationship between the level o f econ om ic
developm ent o f a coun try and its level o f dem ocracy. A great deal o f research has exam ined
this topic, and the research has gone in m any different and sophisticated directions, but the
starting point for m uch o f the analysis is the sim ple observation that rich countries tend to be
m ore d em ocratic than poor countries. To exam ine this relationship we need to develop
em pirical indicators for each o f our variables. To m easure econ om ic developm ent is fairly
straightforw ard. There are a nu m ber o f measures we could use, but perhaps the m ost straight -
forw ard is sim ply G D P per capita. To m easure a coun try’s level o f dem ocracy is slightly more
p roblem atic (see C hapter 6 on conceptual d efinitions). D em ocracy is a difficult concept to
define, and w herever there is definitional ambiguity, there is also likely to be controversy over
m easurem ent. There are a nu m ber o f widely used indicators to m easure dem ocracy, from
sim ple d icho tom ies o f w hether a coun try is dem ocratic or not, based on w hether it holds free
and fair elections (Przew orski et al. 20 0 0 ), to m ore extensive scales based on political rights
(Polity IV; M arshall and Jaggers 20 1 0 ) and civil rights (Freedom H ouse 2 010). For this e x am ­
ple we opt for Polity IV, sin ce it distinguishes betw een different levels o f d em ocracy (from
very authoritarian to fully d em ocratic) on a scale o f 0 to 10 (w here 10 is the m ost d em ocratic).
T he next step is to collect in form ation for different coun tries for each o f these variables.
E ach co u n try th erefo re represents a case, and for each case we collect in form ation on its level
o f d em o cra cy (m easured as its P olity IV score) and its level o f econ om ic developm ent (m eas­
ured as G D P p er capita in U S dollars). To sim plify this exam ple we start by consid ering just
ten cou n tries. The data for th ese ten cou n tries are show n below in a data m atrix, sim ilar to
how data are stored in m o st statistical softw are packages. E ach row represents a case, and
each colu m n represents a variable.

Country X—GDP Y—Level of

per capita democracy

Chad $871 1
Cambodia $1,446 3
Georgia $2,664 5
Ecuador $3,203 6
Paraguay $4,426 7
Bulgaria $5,710 8
Venezuela $5,794 7
Brazil $7,625 8
Costa Rica $8,650 10
Mexico $9,023 8

From the raw data, it is difficult to ascertain w hether there is m uch o f an association between a
cou n try ’s score o n each o f the variables. But this b ecom es m uch clearer if we represent the data
graphically. O u r expectation is that th ose coun tries with low levels o f econ om ic developm ent
will a lso have low levels o f dem ocracy, and that th ose cou n tries with high levels o f econ om ic
H O W TO D O R E S E A R C H IN P R A C T I C E

d evelopm ent will have high levels o f dem ocracy. To see if th is ex p ectatio n is fulfilled we can
sim ply plot th e values for each variable against each oth er on a scatter plot. T h ese values can
be thought o f as the coord inates for each country, w hich form th e basis o f c o rrelatio n and
regression. This is done in Figure 15.1, w here th e ho rizon tal axis (th e x -ax is) refers to a c o u n ­
try’s level o f econ om ic developm ent and the v ertical axis (th e y -axis) refers to a c o u n try ’s level
o f d em ocracy. E ach dot represents a country, and its position on the scatter plot relates to its
score on each o f the variables. Just from eye-b alling the results, we can see that ou r e x p e c ta ­
tion appears to be confirm ed. The cou n tries with relatively high levels o f eco n o m ic dev elop­
m ent, like C osta Rica and M exico, also tend to have relatively high levels o f d em ocracy, and
th e coun tries with relatively low levels o f eco n o m ic developm ent, like C had and C am b o d ia,
tend to have relatively low levels o f dem ocracy.
We can phrase this in a slightly different way, and say that th ere appears to b e a positiv e
relationship betw een the level o f e c o n o m ic developm ent o f a c o u n try and its level o f d e m o c ­
racy. That is, cou n tries w ith higher levels o f ec o n o m ic d ev elopm ent ten d to have higher levels
o f dem ocracy. We can see this qu ite clearly from th e scatter plot, but we can also provid e a
sum m ary m easure o f th is relationship by fitting to th e data so m eth in g c o m m o n ly k now n as
the line o f best fit, or the regression line (see Figure 15.2).
The regression line gives a neutral, im partial way o f identifying the best fitting line to a scatter
o f dots in a two-dim ensional space. This m ethod involves identifying a line that m inim izes the
sum o f squared vertical deviations from itself. The deviations are the distance between the line o f
best fit and each observation. The regression line cuts through the m iddle o f all the observations.
Som e o f the observations are above the line, like Bulgaria, som e o f the observations are below the
line, like Chad, but the best fitting line m inim izes the distance shown by the arrow between the
line and the observations (also known as the residuals). This procedure is called least squares and
is often called ordinary least squares’ or OLS for short. The calculation o f the line o f best fit is not
particularly interesting from a substantive point o f view, and there is no need to go into it in
detail, since practically all software packages will fit the line for you. But the general principle is
sim ple enough. The im portant thing is to be able to interpret the line o nce it has been fitted.

♦ Bulgaria ♦ Brazil
♦ Paraguay ♦ Venezuela
♦ Ecuador
► Georgia

♦ Chad
0 \-------------------- t
50 S2 S4 $6 $8

X -G D P per capita (US$1000s)

Figure 15.1 Scatter plot of econormc development and level of dem ocracy
 B I V AR I A TE A N A LY SI S

The regression line can be represented with the following equation:

Y = a + bX

where

Y is the dependent variable (th e one on the vertical axis);

X is the independent variable (th e one on the horizontal axis);
a is the intercept;
b is the slope.

Som etim es this equation is written with a ‘h at’ over the Y to indicate that the dependent
variable is not an exact fu n ction o f X, and is a predicted value for which there is associated
error (m eaning that not all o f the observations fit perfectly onto the line o f best fit). Either
way, the m ain points o f interest are the values for the intercept (som etim es called the c o n ­
stant) and the slope coefficient and how these can be interpreted.
From the above exam ple, the values that we obtain for our line o f best fit can be expressed
as follows:

Y= 1.93 + 0 .79X .

The in tercep t, or the constant as it som etim es know n, is the point w here the line o f best fit
crosses the y-axis. It thus refers to the value o f Y w hen X is zero. Here, the intercept o f 1.93
refers to th e pred icted level o f d em ocracy for a cou n try with a G D P per capita o f zero. In
m ost cases, this value is not o f m uch analytical interest, since a value o f zero on the in d e­
p en d ent v ariable does not m ake m uch th eoretical or e m pirical sense. We are therefore rarely
interested in d iscu ssing the size o f the constant, but it is im portant if we are interested in
p re d ic tio n , and m a k in g a guess ab ou t Y for som e given value o f X , w hich we will retu rn
to later.

£
o

X-GDP per capita (USSIOOOs)

Figure 15.2 Scatter plot of economic development and level of democracy, with line of best fit
374 H O W T O D O R E S E A R C H IN P R A C T I C E

O f far m o re substantive interest is the slope o f th e line, w hich describ es th e fu n ctio n al rela­
tionship betw een x and y, and is som etim es know n as th e s lo p e c o e ffic ie n t o r reg ressio n c o e f­
ficien t, depicted by the letter b. The slop e co effic ie n t can tell us a n u m b er o f things. First, if th e
slope coefficient is zero, then the slope o f the line is flat, parallel to th e x-ax is, and th ere is n o
relationship betw een the two variables. In oth er words, an in crease in th e value o f o u r in d e­
pendent variable has no im pact on the value o f th e depen dent variable. Secon d , th e slope c o e f­
ficient can also tell us w hether the relationship is positive or negative: a positive value for the
slope coefficient indicates that as values o f X increase, values o f Y also tend to increase, w hereas
a negative value for the slope coefficient indicates that as values o f X in crease, values o f Y tend
to decrease. A nd third, it also allows us to q uantify the m agnitude o f this in crease (o r d ecrease)
and specifically it tells us the nu m ber o f units that Y changes for each unit chan ge in X . So, in
this example, a slope coefficient o f 0.79 tells us that for every unit in crease in X o f $ 1 ,0 0 0 (sin ce
G D P per capita is measured in thousands o f dollars), the predicted or estim ated level o f d e m o c­
racy in a coun try increases by 0.79 points on the Polity IV scale (w hich ranges from 0 to 10).
The value o f the regression coefficient m ight appear sm all, but its size is partly a fu n ctio n
o f the scale on which the in dependen t variable (and d epen d ent variable) is m easured . For
exam ple, if G D P per capita were m easured in dollars rather th an th ousan ds o f dollars, the
size o f the coefficient would be 1,000 tim es sm aller. It is often helpful th erefo re to illustrate
the substantive im pact o f the in dependen t variable by calcu latin g the p redicted value o f Y
for different values o f X. So we m ight want to com p are the predicted level o f d e m o cracy for
a relatively w ell-off country, say o ne with a G D P per capita o f $ 8 ,0 0 0 and for a relatively p o o r
country, say one with a G D P per capita o f $2,0 0 0 . W e can th en plug th e relevant nu m b ers
in to our equation (rem em berin g to take into accou n t th e value o f th e in tercep t) as follows:

Y = 1 .9 3 + 0 .7 9 X .

The predicted level o f dem ocracy for a w ell-off cou n try with G D P per capita o f $ 8 ,0 0 0 :

Y = 1 .9 3 + 0 .7 9 * 8
Y = 1 .9 3 + 6 .3 2
V" = 8 .2 5 .

Fhe predicted level o f dem ocracy for a poor cou n try with G D P per capita o f $ 2 ,000:

Y = 1 .9 3 + 0 . 7 9 * 2
Y = 1 .9 3 + 1 .5 8
r = 3 .5 i.

A n in cre a s e in G D P p e r c a p ita fro m $ 2 ,0 0 0 to $ 8 , 0 0 0 is a s s o c ia te d w ith an in c re a s e

in d e m o c r a c y o f 4 .8 p o in ts , fro m 3 .5 to 8 .3 . O r, to p u t it a n o t h e r w ay, th e p r e d ic te d level o f
d e m o c r a c y in a re la tiv e ly w e ll-o ff c o u n tr y is m o re th a n tw ic e a s h ig h as th e p r e d ic te d level
ot d e m o c r a c y in re la tiv e ly p o o r c o u n tr y .

Is there a relationship? Significance testing with interval variables

l o r tw o variables to he a sso cia te d , th e n , th e slop e o f th e line m u st b e d ifferen t fro m zero . If th e
s op e is positive, then we o b se rv e an u p w a rd -slo p in g line, as s h o w n in F ig u re 15 .3 ( a ), in d ic atin g
 BI VA RI A TE A NA LYS IS

(a) Positive relationship (b) Negative relationship (c) No relationship

Figure 15.3 Different types of linear relationship

that as the value o f the independent variable increases, the predicted value o f the dependent
variable also tends to increase. If the slope is negative, then we observe a downward-sloping
line as shown in Figure 15.3(b ), indicating that as the independent variable increases, the
predicted value o f the dependent variable decreases. If the slope coefficient is zero, however,
then the line o f best fit is flat, as shown in Figure 15.3(c), indicating that as the independent
variable increases, there is no linear change in the predicted value o f the dependent variable.
This raises the q uestion o f what counts as a slope coefficient that is different from zero. How
big does the coefficient need to be? If our slope coefficient is, say, 0.2, can we infer that there
is a real relationship? To have confidence that the relationship really exists, we have to estab­
lish how likely it is that the slope also exists in the population. To help us do this we draw on
the p rinciple o f sign ificance testing, which builds directly on what we discussed in Chapter 14.
So far we have used sim ple lin ear regression as a descriptive tool to describe the pattern o f
the relationship betw een two variables in our sam ple. But linear regression m odelling is
m o re pow erful as a tool o f in feren ce: it allows us to m ake estim ates o f the param eters for the
m od el in the population. W e use inferential statistics because we want to say som ething
about th e p opu lation from w hich our sam ple is drawn. That is, in the world at large, do rich
co u n tries tend to be m o re d em ocratic than p oorer countries?
In ord er to answ er this qu estion we carry out a sign ificance test. The m ethod we use to test
for sig n ifican ce in regression is called the t t e s t . The t test (or t ratio) enables us to assess how
likely it is that the sam ple association we observe could ju st be the result o f sam pling varia­
tion. Even if we had data on all the coun tries in the world, we would still treat this as a sample,
sin ce w hat we are really in terested in is the relationship betw een the variables, and the co u n ­
tries in the world are in a sense a sam ple o f the values these variables can take. The only
param eter o f the sim ple lin ear regression m odel for w hich we will d escribe m ethods o f statis­
tical in feren ce is th e slope coefficient b. Tests and confid ence intervals for population values
o f th e in tercept a are alm ost never substantively in teresting, so they will not be considered
here. Sig n ifican ce testing involves a nu m b er o f steps. The first step is to clarify exactly what it
is th at is to b e tested. T h is is know n as stating the null hypothesis. Th e second step involves
m ak ing a d ecision on w hether the null hypothesis can be rejected or not. To do this we refer
to so m eth in g th at is know n as the test statistic (or t ratio), w hich provides a sum m ary m eas­
ure o f th e d ifferen ce betw een what we observe and what we would expect to observe if there
was no relationship. Finally, we m ake a decision on w hether this difference is statistically
sig n ifica n t o r not, w ith referen ce to the p value, w hich refers to the probability o f obtain ing
m H O W T O D O R E S E A R C H IN P R A C T I C E

th e ob serv ed t value if th e null h yp othesis w ere true. T h e standard p ra c tic e is to r e je c t th e null

hypothesis if the p value is less th an 0 .05. W e discu ss each o f th ese steps below.
T he null hyp othesis is th at th ere is n o lin ear relatio n sh ip b etw een th e ex p lan ato ry variable
X and the response variable Y in th e p opu lation . T h is co rresp o n d s to a slop e c o e fficie n t w ith
a value o f zero. G raphically, th is co rresp o n d s to a regression lin e in th e p op u lation w h ich is
flat. The null hyp othesis can thus b e expressed in w ords as:

H o: Th ere is no lin ear association betw een X and Y in th e popu lation .

O n a point o f clarity, we do not ever prove a null hypothesis to b e true o r false. R ath er we c o l­
lect evidence to make an inform ed decision based on the b alance o f probability. B ecause we are
dealing with probabilities rather than absolutes, there is always a risk that we m ake an error. In
particular, there are two m ain types o f error that we can make. These are know n as the Type I
error and the Type II error. A Type I error is w hen we in correctly reject a null h ypothesis that is
actually true. The way to reduce these errors is ju st to raise the threshold for failing to reject the
null hypothesis, from the 0.05 level to the 0.01 level. The problem with this approach though is
that it risks introducing what is know n as the Type II error, w hich is failing to reject a null
hypothesis that is actually false. Type I errors are conventionally considered m ore serious than
Type II errors, so what we m ost want to avoid is rejecting the null hypothesis unnecessarily. T his
implies that we will maintain the null hypothesis unless data provide strong enough evidence to
justify rejecting it, a principle which is som ewhat sim ilar to Pop pers in junction to ‘keep a theory
until falsified’, discussed in Chapter 3, or even the ‘in nocen t until proven guilty’ principle o f law.
D espite our dislike o f Type I errors, we never try to avoid th em com pletely. The on ly way
to guarantee that the null hypothesis is never in correctly rejected is never to r eject it at all,
w hatever the evidence, w hich is obviously n ot a sensible d ecision rule for em pirical research.
Instead, we decide in advance how high a probability o f Type I error we are w illing to to ler­
ate, and then use a test procedure with that probability. The conv en tion is that we are p re­
pared (if not happy) to be wrong 5% o f the tim e, and so we use a 5% level o f sign ifican ce to
make decisions from a test. The null hypothesis is then rejected if the sam ple yields a test
statistic for w hich the p value is less than 0.05. This is our critical value.

The test statistic

To make som e sensible statem ent about w hether the slope in the popu lation is flat or n ot, we
need to rely upon statistical and probability th eo ry From our sam ple, we have an estim ate o f
the population slope, but it is only that: an estim ate. W hat we w ant to know is w hether there
is enough evidence to suggest that our estim ate is significantly different from zero. If there
is, we can reject the null hypothesis and infer that in the population there probably is a rela­
tionship between the two variables. But if we do not have enough evidence to do this, we fail
to reject the null hypothesis. The logic un derpinning the t test is quite intuitive. We have an
estimate of the slope coefficient in the population. We want to know if this is significantly
different from zero, but our coefficient is only an estim ate. The true value may be a little
higher or a little lower. To see w hether it is significantly higher or lower than zero we utilize
som ething called standard error, and from this we can calculate a confid ence interval in
much the same way as we did in Chapter 14. The 95% confidence interval for the slope co e f­
ficient is calculated as the coefficient plus or m inus 1.96 multiplied by its standard error. So,
 B I VA RI A TE AN A LY SI S

for exam ple, if the slope coefficient is 3; and the standard error is 2, our 95% confidence
interval is betw een 6 .92 a n d -0 .9 2 (3 plus or m inus 3.92). Since 0 is contained within that
confid ence interval, we cann ot rule out the possibility that the real population param eter is
zero, and so we would fail to reject the null hypothesis. Thus, as a shorthand, when the slope
coefficient is less than twice the size o f its standard error, we fail to reject the null hypothesis
(th is is why it is im portant always to report the standard errors o f the regression estim ates).
The test statistic is sim ply the slope coefficient divided by its standard error. Ihe formula
is show n below:

t= - L
se(P)

where

P is the estim ated coefficient; and

sê(P) is the estim ated standard error o f the coefficient.

The t statistic can be thought o f as is a m easure o f the difference betw een what we would
expect to see if the null were true (e.g. the slope would be z ero), and what we actually observe.
Larger values o f t th erefore indicate that our sam ple does not co nfo rm to the expected pat­
tern o f no association . U sing probability theory, we can establish how likely it is that we
would ob serv e a t o f the given value if the null hypothesis were actually true. The p value
gives us th e exact probability o f draw ing a sam ple with a test statistic o f the observed value
or greater from a popu lation in w hich there is no association betw een the two variables (e.g.
slope is zero). By conv en tion , if the probability is lower than 0.05, we reject the null hyp oth­
esis. W h a t we are saying is that the probability o f o btain ing a t value o f this size if the null
hyp othesis is true is so sm all that we are confident that it must therefore be false. But this is
som ew hat d ifferent to proving that it is false. The key word here is confidence. We can never
know for sure w hether the null hypothesis is true or not. W e know that 95 tim es out o f a 100
it will be the c o rrect d ecision to reject the null hypothesis, but we also know that five tim es
out o f a 100 we will reject a null hypothesis that is actually true.

The strength of the relationship: a question of fit

So far we have exam in ed w hether (or not) we th in k there is a relationship betw een two
in terval-lev el variables, using sign ificance tests, and how we can d escribe the form o f that
relationship, with referen ce to the d irection o f the relationship and the m agnitude o f the
coefficien t. Th ere is, how ever, a third aspect o f the relationship that it is often helpful to c o n ­
sider. This is so m etim es know n as th e stren gth o f the relationship, but is perhaps better
d escrib ed as th e fit. If we were to say that th ere was a stron g relationship betw een two v ari­
ables, we could un d erstand th is in two different ways. First, we could th in k o f a stron g rela­
tionsh ip as b ein g on e w here ou r in d ependen t variable has a ‘big’ im pact on our dependent
v ariable, that is, o n e w here th e slope o f the regression line is very steep. But we could also
th in k o f a stron g relationship as b eing on e w here our lin e o f b est fit produces v ery little error,
th at is, w hen all th e o b serv a tion s are clustered very tightly around the line.
m H O W T O D O R E S E A R C H IN P R A C T I C E

Fig ure 15.4 The 'fit' of the regression line

The line o f best fit allows us to p redict values o f Y for given values o f X . However, we a lm ost
always make som e errors in o ur prediction o f Y. The errors are called the residuals, and are the
distance betw een the line o f best fit and the actual ob serv ation s that the lin e passes throu gh.
The m ore a ccurate the guesses o f Y, the sm aller the residuals, and the better we can say th e line
fits the data. This is illustrated in Figure 15.4, w hich shows two graphs with exactly the sam e
values for the intercept and slope coefficient. In the first graph, we can see that the ob serv a­
tions are quite spread out around the regression line, w hereas in the second graph, all th e data
points are very close to the regression line. We can d escribe the difference betw een th ese two
regression lines in term s o f which one provides the b etter ‘fit’ to th e data. B o th Pearson s co r­
relation and R-square are measures for sum m arizing this fit.
R-square (R 2) is one o f the m ost com m on ly reported statistics in qu antitative political
research. O ne way to th ink a bout it is that it provides a su m m ary m easure o f how m u ch b e t­
ter off we are in our predictions o f Y using the regression lin e than som e oth er estim ation
tech nique (basically, guessing). It d escrib es the prop ortion o f the v ariation in th e d epen dent
variable that our model is able to ‘explain. If we had no in form ation about how X is related
to Y, then our best guess o f Y would sim ply be its m ean value, 7. As Figure 15.5 show s, for a
given value o f X, the value o f Y may be quite a long way away from the observed value o f Yt.
We can sum m arize this error from guessing by adding up the squared d istance betw een
each value o f Y and the m ean value o f Y (Y- Y). This is known as the total sum o f squares
(T SS ), and can be regarded as a m easure o f the total variation in our depen d ent variable:

TSS = % Y ~Y)

Out of this total variation, we want to know how m uch our m odel explains. Using the regres­
sion line, we make predictions o f V, denoted ?. Som etim es our predictions will be good and
Y will be close to the observed values of Y; som etim es our predictions will be bad and Y will
be a long way away trom the observed values o f Y. We can break this down into what our
model explains— the explained variation, and what it does n o t— the unexplained variation.
The explained part ot the model can be thought of as the difference between our predicted
 B IV AR I AT E A NA LYS IS

value o f Y and the m ean value o f Y. This is known as the ‘M odel sum o f squares’, denoted
SSM , or the ‘R egression sum o f squares’, RegSS:

SSM = %Y
The unexplained part o f the m odel (or the error) can be thought o f as the difference between
our predicted value o f Y and the observed value o f Y. This is known as the ‘Error sum of
squares’, denoted SSE , or the ‘Residual sum o f squares’, denoted ResSS:

SSE = '%Y,-Y)2.
These sou rces o f v ariation are related to each oth er so that:

Total variation o fY = Variation explained + Unexplained variation by regression

TSS = SSM + SSE.

The R -sq uare statistic is sim ply defined as the p roportion o f the total variation that the model
explains. It can thus be expressed as follows:

R-square = SSM/TSS.

This is know n as a P R E m easure (P roportion al reduction in error). It is the proportion o f the

total variation o f Y explained by the regression model. R-square ranges between 0, when the
regression explains n othin g (th e slope coefficient is 0), and 1, when there is perfect association
(all the observed values o f Y fall exactly on the line o f best fit). The form al interpretation o f the
R-square is: ‘T he proportion o f variance in Y that can be explained by the variance in X ’. So if
we o b tain an R-square o f 0 .5 0 , we can say that 50% o f the variance in Y can be explained by the
variance in X. Or, that our m odel explains 50% o f the variation in the dependent variable.

Fig ure 15.5 Explained and unexplained variation

H O W TO D O R E S E A R C H IN P R A C T I C E

This can also be put in a slightly different way, and we could say that we m ake 5 0% few er errors
when guessing Y while know ing X, as com pared to guessing Y w hen not kno w ing X.

Pearson's correlation
Closely related to R-square is Pearsons correlation coefficient, or Pearsons r as it is o ften know n.
W hen Pearsons r is squared, it becom es R-square, and can be interpreted as a PR E measure, as
discussed above. But unlike R-square, Pearsons r can also take negative values and ranges from - 1
to +1. This m eans it can provide a useful sum m ary about the direction o f the relationship between
two variables (w hether it is positive or negative), as well as about the strength o f the relationship
(in terms o f its fit to the data). W hen there is perfect negative association between two variables,
Pearsons r is - 1 ; when there is perfect positive association between two variables, Pearsons r is
+1; and when there is no linear association between two variables, Pearsons r is 0.

Example 1: democracy and development

T o illu stra te h o w all th e s e ste p s fit to g e th e r, w e c a n re t u r n to th e e x a m p le o f th e re la tio n s h ip

b e tw e e n d e m o c r a c y a n d e c o n o m i c d e v e lo p m e n t th a t w e i n tr o d u c e d e a rlie r , a n d e x a m i n e
th e a s s o c ia tio n w ith real d a ta . D o e s d e m o c r a c y v a r y b y a c o u n t r y ’s level o f e c o n o m i c d e v e l­
o p m e n t? If so , h o w ? T h e d a ta set th a t w e u se to a n s w e r th is q u e s tio n is th e c r o s s - n a ti o n a l
d e m o c r a c y d a ta set c o m p ile d b y P ip p a N o rris , w h ic h c o n t a in s v a lid in f o r m a t i o n o n 1 4 5
c o u n tr ie s . E a c h c o u n tr y th e re fo re re p re s e n ts a c a s e , a n d fo r e a c h c a s e w e h a v e in f o r m a t i o n
on th e level o f d e m o c r a c y ( m e a s u r e d as P o lity IV s c o r e ) a n d level o f e c o n o m i c d e v e lo p m e n t
( m e a s u re d as G D P p e r c a p ita ). W e c a n s p e c ify th e m o d e l th a t w e e s tim a te a lg e b ra ic a lly :

Y = a + fc(X (+ e

w h e re :

Y re fe rs to level o f d e m o c r a c y m e a s u re d b y P o lity IV ( o u r d e p e n d e n t v a r ia b le );
X re fe rs to G D P p e r c a p ita in U S $ 1 ,0 0 0 s ( o u r i n d e p e n d e n t v a r ia b le );
e is th e e r r o r te rm .

O u r null h y p o th e s is is th a t th e re is n o a s s o c ia tio n b e tw e e n a c o u n t r y ’s level o f e c o n o m i c

d e v e lo p m e n t an d level o f d e m o c ra c y . T h a t is, th e tru e p a r a m e t e r o f th e s lo p e c o e fficie n t b is
ze ro . O u r first task th e n is to e sta b lish w h e th e r th e null h y p o th e s is c a n b e re je c te d o r n o t. T o
d o th is, w e e s tim a te th e m o d e l u sin g a sta tis tic a l so ftw a re p a ck a g e , s u c h as S P S S. T h e resu lts
are p re s e n te d in T ab le 15 .1 . Ih e tab le c o n ta in s q u ite a lot o f in fo rm a tio n , a n d it is i m p o rta n t
to be c le a r a b o u t w h at e a ch p ie ce o f d a ta re fers to. T h e first ro w in th e m a in b o d y o f th e tab le
is labelled c o n s ta n t, an d th e v alu e in th e c o lu m n lab elled 'b c o e ffic ie n t’ re fe rs to th e v a lu e o f
th e c o n s t a n t — o r th e in te rc e p t. 'Ib is sim p ly tells us th e p o in t at w h ich th e lin e o f b est fit
cro s se s th e y-a x is. A lth o u g h m o st s o ftw a re p a ck a g e s will a lso p ro v id e a d d itio n a l i n f o r m a ­
tio n , su ch as th e sta n d a rd e rro r, th e t s ta tis tic , an d th e p rob a b ility, n o n e o f th e se a re o f m u c h
in terest w ith re fe re n ce to th e in te rce p t.
Of m o re su b sta n tiv e in terest is th e ro w labelled ‘G D P p e r c a p ita ’. O u r first task is to e s ta b ­
lish w h e th e r the null h y p o th e sis that th e re is n o a s s o c ia tio n b e tw e e n e c o n o m i c d e v e lo p m e n t
and d e m o c ra c y ».an be rejected o r n ot. lo m ak e a d e cis io n on th is w e in sp e ct th e a s s o c ia te d
 B I V AR I AT E ANA LY SI S

t statistic and the p value. The t statistic is 4.15, and the p value is < 0.0005. The p value tells
us the probability o f observing a t statistic o f 4.15 or greater if the null hypothesis were true.
In this instance, the probability o f the null being true is very sm all indeed, and m ore im por­
tantly, is m uch less than our critical c u t-o ff point o f 0.05. We can therefore confidently reject
the null hypothesis o f no association , and infer that there probably is an association between
the two variables.
H aving established that th ere is a sign ificant relationship, we next turn our attention to
d escrib in g the pattern o f th is relationship. We do this with referen ce to the slope coeffi­
cien t, w hich is som etim es reported as the unstand ardized beta coefficient. The slope c o e f­
ficien t o f 0 .2 3 tells us that th ere is a positive association betw een econ om ic developm ent
and d em ocracy , and, m o re precisely, that the predicted level o f d em ocracy in a coun try
in crea ses by 0 .2 3 poin ts on the Polity IV scale for every U S $ 1 ,0 0 0 in crease in a co u n try ’s
G D P p er capita. This is co n sisten t with our p rior ex p ectation s, and the eviden ce does sug­
gest that on average rich co u n tries tend to be m ore d em ocratic than poor coun tries.
Finally, we can assess how well this m odel fits the data by considering the R-square, which
is reported in the note to the table. The R-square is 0.11, w hich tells us that 11% o f the vari­
ance in d em ocra cy scores can be explained by the variance in G D P per capita. O n the face o f
it, th is m ight not seem very m uch. In term s o f explaining why som e coun tries are m ore
d em ocra tic than oth ers, we have only scratched the surface, and there may be m any other
reasons apart from eco n o m ic developm ent for why som e coun tries are m ore dem ocratic
than oth ers. To gain a fuller understanding o f the sou rces o f this variation we may therefore
want to explore the im pact o f oth er factors, w hich leads us to m ultivariate analysis, w hich we
will deal with in the next chapter.
O f course, we have to b ear in m ind that w hat we have established is a statistical association
betw een the tw o variables, and this is very different from establishing a causal relationship
betw een e c o n o m ic developm ent and dem ocracy, even if that is what the th eory suggests.
Statistical a ssociation is, o f course, the first cond ition o f establishing causality, but we might
have serious qu estio ns a b out w hether the oth er two cond ition s o f correct tem poral ordering
and elim in a tion o f alternative explanations have been satisfactorily dealt with. We do not
know th at if a co u n try w ere to b eco m e richer, it it would also b ecom e m ore dem ocratic (to
investigate that, we would need to look at changes over tim e) or even why rich coun tries are
m o re d em o cra tic th an p oorer ones. Bivariate analysis thus often leads to m ore questions
th an it can answer, and constitutes the first step in a m ore involved in vestigation. We will
retu rn to som e o f th ese p oin ts in the next chapter.

Table 15.1 Democracy and development, OLS regression, parameter estimates

Un-standardized
Std error t p value
beta coefficient

Constant 1.96 061 3 20 0 002

GDP per capita 023 0 06 4 15 0000

Notes: Dependent variable: Polity democracy 10-pt score. 2000. N = 145; R-square = 0.11.
Source Pippa Norris. Cross-national democracy project
H O W T O D O R E S E A R C H IN P R A C T I C E

Bivariate analysis for categorical data: cross-tabulation

So far we have exam in ed th e biv ariate asso ciatio n b etw een tw o in terval-lev el variables.
However, m any o f the social and political p h e n o m en a that we are in terested in ex p lo rin g are
not m easured on an in terval scale. R esearch on v oting b ehav iou r, p olitical p rotest, d e m o ­
cra tic breakdow n, and civil war often deal with dep en d ent variables th at are n o m in a l, and
focus on w hether som eon e votes for on e p articu lar p olitical party o r ano ther, o r w h e th er a
cou n try h as exp erienced d em o cratic breakdow n or civil war, or not. In deed, d e m o cra c y can
also be thought o f in no m in al term s: eith er a co u n try is d e m o cratic o r it is not. To ex am in e
these types o f variables, th en , we need to use a different tech n iq u e. C ro ss-ta b u la tio n s (or
cro ss-ta b s) are th e appropriate m eth od o f biv ariate analysis w hen you are in terested in th e
association betw een tw o catego rical (n o m in al or ord in al) variables. C ro ss-ta b s w ork b est
when the variables have a lim ited n u m b er o f categories. Too m any catego ries m ak e th e tables
unwieldy and difficult to analyse, and also run th e risk o f in tro d u cin g sparse o r em pty cells
(see C hapter 14). But even if variables do have a lot o f categories, such as in terval-lev el v a ri­
ables like age, it can often b e helpful initially to split th em in to categories, such as age groups,
and look at the cro ss-tab s. A lthough cro ss-tab s involve analysing th e relatio nship betw een
two variables in a slightly different way from the m eth od o f sim ple lin ear regression d is­
cussed above, the analytic principles are very sim ilar. It enables the research er to answ er two
m ain questions. Is there a relationship betw een the two variables? A nd if so, w hat d oes that
relationship look like? How can we d escrib e it?

Bivariate analysis using tables: describing the relationship

The essence o f cro ss-tab s is to com pare the frequen cy d istrib ution s (see C hapter 14) for two
or m ore categories. These are som etim es know n as cond ition al d istribution s. They are c o n ­
ditional because the distribution o f the dependent variable depends upon the level o f the
explanatory variable. To illustrate the logic o f cro ss-tab s we can th in k about the relationship
betw een age and turnout. In recent years, there has been a great deal o f academ ic research on
the relatively low levels o f electoral p articipation am ong young people, and what, if anything,
a ccounts for this. To exam ine this relationship we can look at survey data collected by the
British Election Study in 2005. Since age is an interval-level variable we need to recode it into
different age bands, transform ing it into an ordinal variable. By contrast, turnout is a n o m i­
nal variable. People either voted or they didn’t. W hen a variable is b in ary — that is, w hen it
contains only two categories (voted/didn’t v ote)— it is often referred to as a du m m y variable.
We can then proceed to see whether (and how) patterns o f turnout vary by age group.
Since we are interested in trying to explain tu rnout, this is our dependent variable. O ur
independent variable, or explanatory variable, is age. It is conven tion that the dependent
variable goes along the rows, and the independent variable goes down the colum ns. Turnout
is therefore the row variable; and age is the colum n variable (see Table 15.2). The in tersection
of a row and a colum n is the cell o f a table. 'Ihe frequencies in the internal cells d escribe the
joint distribution: they show how many cases have each possible com binatio n o f the row and
colum n variables. Ihe (irst step is just to aggregate the cell counts for each different co m b in a­
tion of responses. For example, the 2005 BF.S interviewed 3,154 people. We can see that
overall 2 ,4 :>9 people voted and 695 people did not vote (see co lu m n labelled 'T o ta l'). In
 B I V AR I A TE ANA LY SI S 383

Table 15.2 Age and turnout in 2005 British General Election, cell counts

18-39 years old 60 years old

4 0 -5 9 years old Total
and over

Did not vote 350 22A 1/1 (!,

Voted 652 947 860 /A : j
Total 1002 1171 981 ,r ,4
Source: British Election Study 2005

ad d ition, we can see that 6 52 o f the youngest age group voted, com pared with 947 o f the
m iddle age group and 8 6 0 o f the oldest age group.
Tables organized in this way are not very easy to interpret, and it is difficult to tell which age
groups are the m ost likely to vote, since the size o f the different age groups varies. To help us
m ake these com parisons m ore easily we therefore convert the cell counts into percentages, just as
we did in the previous chapter. T here is a num ber o f ways in which we can calculate percentages,
and it is im portant to be clear about w hether we are calculating (or interpreting) row percentages
or colu m n percentages. Each type o f percentage refers to som ething slightly different, so it is
im portant not to get confused. C olum n percentages are the standard practice. These are read
down the table in colum ns. These are reported in Table 15.3. So, for example, we can see that
65% o f 1 8 -3 9 -y ea r-o ld s voted in the election. W hen interpreting the table, it is useful to draw
com parisons betw een the different cells. For example, with reference to Table 15.3, we can see
that tu rnout appears to increase with age. W hereas only 65% o f 18-39-year-old s reported hav­
ing voted, 81% o f 4 0 -5 9 -y ea r-o ld s and 88% o f the over 6 0s reported having done so. Reported
turn out am ong the 4 0 - 5 9 age group was therefore 16 percentage points higher than it was
am ong the younger age group. This difference appears to be quite substantial.
Row percentages are som ewhat different. These are not reported in the table, but can easily be
calculated from the cell counts in Table 15.2. In this example, a row percentage would refer to the
percentage o f voters w ho fall into each age group. So, for example, we can see that 2,459 people
voted in the election. O f these 2,459, 652 were aged 1 8 -3 9 , which equates to 27%. By contrast,
39% o f voters were aged 4 0 - 5 9 , and 35% o f voters were aged 60 or over. We have to be careful
what substantive conclusions we can draw from these percentages. We can see that the largest
n u m ber o f voters are in the 4 0 -59-y ear-o ld age group, but this is not because people in this age
group are the m ost likely to vote (which they are not, as we saw in Table 15.3), but because this
age group is sim ply the largest in term s o f size, and so constitutes the biggest proportion o f voters.

T able 15.3 Age and turnout in 2005 British General Election, colum n percentages

60 years old
18 -3 9 ye ars old 4 0 -5 9 ye ars old All
and o ver

35 19 12 22
Did not vote
65 81 88 78
Voted
1002 1171 981 3154
N

Source: British Election Study 2005,

H O W TO DO R E S E A R C H IN P R A C T I C E

W e can also use cro ss-tab s to lo o k at th e asso ciatio n b etw een tw o o rd in al variables in
m uch the sam e way. Table 15.4 rep orts th e co lu m n p ercen tages for a cro ss-ta b b etw een ed u ­
catio nal a ttain m en t and sup port for th e death penalty. R esp o n d en ts were asked w h eth er
they agreed or d isagreed with the statem ent that th e death pen alty is n ev er ju stified . O v erall,
m ore people d isagreed w ith th is statem ent than agreed w ith it (4 7 % vs 3 8 % ), sugg estin g th at,
on balance, public op in ion in B ritain is in favour o f the death penalty. H owever, th ere was
consid erab le v ariation in responses by ed u cation level. W e can see that 57% o f h ighly ed u ­
cated people agreed that the death penalty was never ju stified , com p ared with ju st 2 9% o f
people with low er ed ucational q u alifications. S in ce b oth variables are ord in al, we can say
that support for the death penalty appears to d ecrease as level o f ed u cation in creases.

Is there a relationship? Statistical significance for cross-tabs

So far we have used cross-tabs as a descriptive tool to describe the observed pattern o f association
between two variables in our sample. In som e instances, we may observe a clear pattern, where
it is obvious (or at least appears so) that the distribution o f responses varies by sub-group. But, in
other instances, the observed pattern may be less clear-cut, and there may be only sm all differ­
ences between the sub-groups, or hardly any difference at all. Under these circum stances, it is not
so obvious whether there is a ‘real’ relationship or not. D o the differences that we observe reflect
real differences that exist within the population, or could they just be the result o f sam pling error?
To answer these questions we need to carry out tests o f sign ificance, based on so m eth in g
called chi square. As we discussed above with referen ce to regression, th ere are different
ways o f carryin g out these statistical tests and the lest that we em ploy depends upon the type
o f variable we are exam in ing and the type o f analysis that we are carryin g out. However, all
significance tests serve the sam e underlying purpose, w hich is to provide us with the in fo r­
m ation to make a decision based on probability as to w hether or not we th in k the relatio n ­
ship we observe in our sam ple reflects a real relationship that exists in the population.
To illustrate these points, Table 15.5 shows two hypothetical examples. Exam ple 1 depicts
what it would look like if there were a perfect relationship between a persons social class (m e as­
ured in terms of whether they are working class or middle class) and which party they reported
voting for at the last election (just focusing on w hether they voted for Labour or C on serv a­
tives). By contrast. Example 2 depicts what it would look like if there were no relationship. From
Example 1 we can see that all the working-class respondents voted for Labour and none voted
for the Conservatives, whereas all the middle classes voted for the Conservatives and none

Table 15.4 Attitudes towards the death penalty by level of education, colum n percentages

Education

High Interm ediate Lower Total

Agree
Neither agree
nor disagree
Disagree
 B I V A RI A TE ANA LYS IS

Table 15.5 Association, hypothetical example, column percentages

Exam ple 1: Perfect association Exam ple 2: No association

Working class (%) Middle class (%) Working class (%) Middle class (%)

Labour 100 fj
Conservative 0 100 A',
Total 100 100 100

voted for Labour. This is an exam ple o f perfect association. If we know som ebody’s class— that
is, w hether they are m iddle class or w orking class— we can perfectly predict which party they
will vote for. By contrast, Exam ple 2 depicts an example o f (perfect) no association. We can see
that exactly the sam e proportion o f m iddle-class and working-class voters support each party.
K now ing som eone’s class does not therefore help us at all in predicting which party they vote
for. The cond itional distributions o f each class are exactly the same, and there is thus no asso­
ciation betw een the two variables.These two examples provide extrem e scenarios. In practicc,
we are never likely to find the first example in the real world, but what we want to know is
w hether our sam ple is sim ilar or not to the second example. If we think that it is similar, then
we conclud e that there is probably not a relationship between the two variables, but if we think
that it is different from Exam ple 2, then we can conclude that there probably is a relationship.
As before, to establish w hether our sam ple is different or not from what we would expect
to see if th ere was no relationship betw een the two variables, we carry out a significance test.
The m eth od we use to test for sign ificance in cro ss-tab s is called chi square, often denoted as
X2. C hi square enables us to assess how likely it is that the sam ple association we observe
could ju st be the result o f sam pling variation. A lthough the C hi-sq u are statistic that we use
to test for sig n ificance in cro ss-tab s is different from the t test that we discussed with refer­
en c e to b ivariate analysis w ith interval data, the analytical principles are m uch the sam e and
we go th rou gh the sam e b asic series o f steps.

• State null hyp othesis.

• C alcu late statistic and in terpret.

• A ccep t o r r eject null hypothesis.

The null hyp othesis is stated in exactly the sam e way as we discussed earlier. Using the e x am ­
ple d iscu ssed above, the null hyp othesis would be:

H 0: In th e p opu lation , th ere is no association betw een som eone’s class and w hich party
th ey vote.

I f we ca n find sufficient ev id en ce to reject the null hypothesis (at the 0.0 5 level), then we can
in fer th at in th e p opu lation th ere probably is an association betw een the two variables. The
logic u n d erp in n in g chi square is qu ite intuitive. It allows us to com pare what we observe
with w hat we would ex p ect to ob serv e if th ere was no association . W e com pare the table that
we o b serv e in our sam ple w ith a th eoretical table o f what we would expect to observe if there
was n o relatio nship betw een th e variables (as show n in Exam ple 2 o f Table 15.5). To do this,
w e c o m p a re s o m e th in g c a lled th e ob serv ed fre q u e n cie s and th e ex p ected freq u en cies.
H O W T O D O R E S E A R C H IN P R A C T I C E

S tarting w ith a table with ju st th e cell freq u en cies, we co m p are th e o b serv ed freq u en cies (O )
that we ob serv e in ou r sam ple, w ith th e o n es th at w ould b e ex p ected u n d er th e null h y p o th ­
esis o f n o a ssociation (E ). T h e ch i-sq u are test is an ex cellen t w orkh orse for testin g w h e th er
th e d ifferences betw een O and E are statistically sig n ifican t o r not.
Table 15.6 shows th e ob serv ed d istrib u tion o f freq u en cies for a c ro ss-ta b o f class by vote
choice. The cell cou n ts refer to th e ob serv ed frequ en cies. T h e exp ected freq u en cies are c a l­
culated for th e m iddle class and w orking class, assu m in g th at th e d istrib u tio n o f resp o n ses is
exactly the sam e as for the sam ple overall. S in ce th e total p ro p o rtio n o f L ab ou r v oters in th e
sam ple is 0 .3 9 6 , we would th erefo re e x p ect ( if th e null w ere tru e) 4 0% o f th e w orking class to
vote L abour and 40% o f the m iddle class to vote L abour. T h erefore th e ex p ected freq u en cy
o f w orking-class Labour v oters would b e 1,078 (th e total n u m b er o f w orking class in th e
sam ple) m ultiplied by 0 .3 9 5 8 (th e total p rop ortion o f L ab ou r voters in th e sam p le), w hich
gives 4 2 7 . This can be calculated in a sim ilar way for each cell in th e table.
C hi square com pares the difference betw een th ese exp ected and ob serv ed values. Th e
form ula for calculating chi square is w ritten below.

where:

f o is observed frequencies; and

f e is expected frequencies.

Although you will never need to, b ecause c om puter program s will d o it for you, it is very easy
to calculate chi square. We can go th rou gh each o f the steps with referen ce to Table 15.7. The
colu m n labelled f o (observed frequen cies) is what we ob serv e in our sam ple. The colu m n
labelled f t (expected frequen cies) is the frequen cy distrib u tion we would exp ect if th e null
hypothesis was true. The next colu m n is sim ply the d ifference betw een th ese two values, and
the colum n after that is the d ifference squared, so we get rid o f the negative n u m bers (w hich
would otherw ise cancel them selves out when we add th em all tog ether). The final colu m n
divides the squared difference by the expected frequen cies and then we sim ply add up these
values for each cell and the total is chi square.

Table 15.6 Class and vote, observed frequencies

Class

Middle Working Total

Labour Ovi S28 1162
Conservative
Liberal
>'S 6S7
Other
84 184
Total
K>/8 29S6
 BI V AR I AT E ANA LY SI S

Table 15.7 Calculating chi square

U 1, w (U S (1 ,- IM
Middle Labour 634 735 -101 10/5!
Working Labour 528 427 101 10701
M iddle Conservative 695 590 105 11025
W orking Conservative 238 34 3 105 110/5 •/■■/:
M iddle Liberal 429 415 14 IV . V ./
W orking Liberal 228 242 1/! '69 ':/ 0
M iddle other 100 116 16 7r 6 /
W orking other 84 68 16 'r -j , ;r.
I 2936 2936 43320 Z 05 /

C learly, a sm all chi square suggests that there probably isn’t a relationship, since this m eans
that the observed pattern is quite sim ilar to the expected pattern assum ing no association,
and a large chi square suggests that th ere probably is a relationship betw een the two varia­
bles, but how do we decide what value o f chi- square constitutes large and what sm all? This
is w here prob ability th eory com es to the rescue. C hi square has a know n distribution,
although th e shape o f th is d istrib ution varies according to how m any cells we have in our
table (know n as d egrees o f freedom ). W e can thus com pare our obtained chi-square statistic
against th e ch i-sq u a re d istrib u tion for the appropriate n u m ber o f degrees o f freedom . If the
prob ability o f getting ou r obtain ed value o f chi square is very sm all (typically below 0.0 5 ),
th en we can reject th e null hypothesis. In this sense, the logic o f sign ificance testing is the
sam e as discu ssed previously with reference to the t test, only we are using a different in stru ­
m ent (chi sq uare), w hich has a different type o f distribution.
W e should note, however, that ch i-squ are tests provide an overall m easure o f w hether or
n o t th ere is a statistical a ssociation betw een our two variables. It thus provides a global test
for th e table as a w hole. If we are interested in w hether a particular cell w ithin the table is
sig n ificantly different from what we expect, then we need to look at som eth in g else. The
adjusted residuals enable us to see in w hich particular cell o f the table the O bserved and
E xp ected freq u en cies sign ificantly differ (at the 5% level). Adjusted residuals greater than
plus or m inus 1.96 thus in d icate a sign ificant d ifference.

Example 2: Ethnic diversity and civil war

As before, we can put all th ese different steps o f analysis tog ether to exam in e a real-w orld
p rob lem . R esearch on th e causes o f civil w ar has received a great deal o f attention in PIR. The
causes are h otly contested . W h ereas Fearon and Laitin (2 0 0 3 ) insist there is no lin k betw een
eth n ic h eterogen eity and co n flict, oth ers find ev iden ce that eth n ic cleavages may increase
the risk o f co n flict (E llin gsen 20 0 0 ; C ed erm an and G irardin 2 0 0 7 ). Part o f the reason for the
con trov ersy is that different scho lars use d ifferent data sets, and different operational defin i­
tio n s o f key variables. However, th e fact that th ese different m easures tend to produce d iffer­
ent results should give pause for th ough t, and raise c o n ce rn s about the re liab ility and
v a lid ity (see C h ap ter 10) o f th e different indicators.
H O W T O D O R E S E A R C H IN P R A C T I C E

To exam in e th is association , we use data on civil w ar from th e U ppsala/PR IO arm ed c o n ­

flict data p roject (G led itsch et al. 2 0 0 2 ), sup plem en ted w ith ad d itional data from G led itsch
(2 0 0 7 ). The data set provides in form ation on 177 cou n tries acro ss ev ery year from 1946 to
2 0 0 2 , d etailing the in cid en ce o f arm ed co n flict involving m o re th an 25 casualties. To m easu re
the eth n ic com p osition o f th e different c ou n tries, we follow G led itsch (2 0 0 7 ) and ex am in e
the share o f the population w hich is not in the d om in an t eth n ic group. Th e eth n ic disp ersio n
m easure is therefore 100 m inus the percentage sh are o f th e d om in an t eth n ic group, b ased on
data provided by V anh anen (2 0 0 1 ), recoded into four equal catego ries rangin g from so cie ties
w hich are very eth nically hom ogen eous (cod ed 1) to very eth n ically diverse (cod ed 4).
U sing th is data, we ex am in e th e relationship betw een eth n ic d iv ersity and civil war. Th e
results are presented in Table 15.8. O u r null hyp othesis is th erefo re th at th ere is n o a sso cia ­
tion betw een the level o f eth n ic diversity in a c o u n try and w hether o r no t it ex p erien ced civil
war. O u r first task its to establish w hether o r n o t we can reject th e null h yp othesis. T o do th is,
we inspect the ch i-squ are statistics. Th e value o f the Pearson chi square is 3 3 0 on 3 degrees
o f freedom . The p robability o f o b tain in g a chi square o f th is size or g reater i f th e n ull h y p o th ­
esis were true is less th an 0 .0 0 0 5 (SP SS actu ally says it is 0 .0 0 0 , but it is no t tech n ically zero,
so we ju st say p < 0 .0 0 0 5 ). This is clearly very sm all, and well below th e con v en tion al c u t-o ff
m ark o f 0 .05. We can th erefore confidently reject the null hyp othesis and in fer th at th ere
does appear to be a significant relationship betw een eth n ic diversity and civil war.
Having established that we th in k a relationship does exist betw een th e two v ariables, ou r
next task is to d escribe the pattern o f the relationship to see if it co n fo rm s to o u r e x p e c ta ­
tions. To this end, we discuss the percentages in th e table. First, we can see that in very
hom ogeneous societies there does not appear to be a v ery high likelihoo d o f civil w ar taking
place. In very hom ogen eous societies, only 2.6% o f the cases exp erienced an in cid en t o f civil
war. By contrast, in m ore eth nically m ixed societies, the in cid en ce o f civil w ar appears to be
higher. A m ong quite ho m ogen eous societies, 20% have exp erienced civil war; in quite
diverse societies, 18% have experienced civil war; and in very d iverse societies 21% have
experienced civil war. These percentages are all substantially higher than w hat we observed
for very hom ogeneous societies, lending som e support to our initial th eoretical hyp othesis
that civil war tends to be higher in eth nically diverse coun tries.
But interestingly, there does not appear to be a great deal o f difference in levels o f civil war
between countries that are quite homogeneous (20% ) and countries that are quite diverse (18% ) or
very diverse (21 %). This suggests that the relationship is not linear. It doesn’t appear to m atter how
big the dominant ethnic group is, it just matters whether or not there is a dom inant ethnic group.

Table 15.8 Civil war and ethnic diversity, column percentages

Very Quite Quite

hom ogeneous hom ogeneous diverse Very diverse Total
societies societies societies societies

No civil war
Civil war
N
 B IV A R IA T E A NA LYS IS

These results appear to con firm ou r expectations. E thnic diversity, or m ore specifically, the
presence o f a d om inan t eth n ic group, does appear to increase the likelihood o f civil war.
However, it is also im portant not to o ver-interpret these results. To be sure, diverse countries
tend to ex p erien ce m o re civil war than hom ogen eous countries, but even am ong diverse
c ou n tries the vast m ajority (around 80% ) do not experience civil war, so the relationship is
far from d eterm in istic. W e m ay th erefore want to investigate furth er why som e ethnically
diverse cou n tries d escen d into civil war, w hereas others do not. W hat other factors are
im portant? How does eth n ic m orph ology interact with econ om ic conditions, institutional
structures, and security relations with neighb ouring countries? To answer these questions,
we need to exam in e oth er factors, and it is this aim o f developing m ulti-causal explanations
that leads us to m ultivariate analysis, which we consider in the next chapter.

Conclusions
This chapter has provided an introduction to statistical analysis using bivariate analysis. We have
introduced the key concept of hypothesis testing, which enables us to see whether two variables are
statistically associated and if so, how, using sim ple linear regression and cross-tabs Bivariate analysis
provides a useful starting point for examining how variables are related, and is an im portant building
block for developing explanations and causal statements about w hy things happen. But it does not tell
us the full story. Inevitably, bivariate analysis leads us to ask more questions. In the real world, social
and political phenom ena are rarely mono-causal. If we are interested in w hy things happen, or why
certain factors are influential in some conditions rather than others, the picture rapidly becomes more
com plicated and w e are com pelled to look for additional explanations, w hich brings us to multivariate
analysis, w hich w e will discuss in the next chapter.

Questions
• Draw an arrow diagram depicting the relationship between two variables.

• W rite a brief note to accom pany the arrow diagram, and state clearly which is the dependent vari­
able and w hich is the independent variable, and w hy you think they may be related.

• W hat is the null hypothesis? H ow is it tested?

• W hy should w e treat the results from bivariate analysis with caution?

• W hat is the difference between causality and association?

• H ow do w e know w hether a relationship between two variables is causal or not?

• In cross-tabs, w hen might it be useful to report the row percentages and w hen might it be useful to
report the colum n percentages?
• In sim ple linear regression, what is more substantively interesting: the magnitude of the slope coef­
ficient o r the valu e of R-square? W hat does each tell us?

© Guide to Further Reading

Agresti, A. and B. Finlay (2009), Statistical Methods for the Social Sciences. 4th edition (Upper
Saddle River, NJ: Pearson-Prentice Hall).
Covers the basics o f statistical description and inference, as w ell as m ore advanced topics on
regression m ethods, including m ultiple regression, ANOVA and repeated measures ANOVA, analysis
o f covariance, logistic regression, and generalized linear models.
390 H O W T O D O R E S E A R C H IN P R A C T IC E

Johnson, Janet Buttolph, Henry Reynolds, and Jason Mycoff (2007), Political Science Research
Methods, 6th edition (Washington, DC: CQ Press).
Provides an introduction to quantitative analysis and covers som e m ore advanced topics, w ith lots o f
exam ples from political research.

Carlson, James and Mark Hyde (2002), Doing Empirical Political Research (Boston, MA: Houghton
Mifflin).
Provides an introduction to quantitative analysis and covers som e m ore advanced topics, w ith lots of
examples from political research.

de Vaus, David (2002), Surveys in Social Research (London: Routledge).

A detailed and accessible introduction to quantitative m ethods, w ith easy-to-read chapters on how to
carry out analysis and interpret the results.

Field, A. (2009), Discovering Statistics using SPSS (and Sex and Drugs and Rock 'n' Roll) (London:
Sage Publications).
Provides an easy-to-follow introduction to quantitative analysis and step-by-step in structions on how
to use the SPSS statistical software package to do your ow n analysis.

Rowntree, Derek (2000), Statistics without Tears: An Introduction for Non-Mathematicians

(Harmonsdsworth: Penguin).
An easy-to-read introduction to statistical methods.

© References
Cederman, L and L. Girardin (2007), 'Beyond Conflict, 1945-99: A New Dataset, Journal of
Fractionalization: Mapping Ethnicity onto Peace Research 39(5): 615-37.
Nationalist Insurgencies', American Political
Marshall, Monty G. and Keith Jaggers (2010), Polity
Science Review 101:173-85.
IV Project: Political Regime Characteristics and
Ellingsen, T.(2000), 'Colorful Community or Ethnic Transitions, 1800-2010, Center for Systemic
Witches' Brew? Multiethnicity and Domestic Peace, available at http://www.systemicpeace.
Conflict during and after the Cold War1.Journal org/polity/polity4.htm.
of Conflict Resolution 44(2): 228-49.
Przeworski, A., M. Alvarez, J. Cheibub, and F.
Fearon.J. and D. Laitin (2003), 'Ethnicity, Insurgency, Limongi (2000), Democracy and Development:
and Civil War', American Political Science Review Political Institutions and Well-Being in the World,
91(1): 75-90. 1950-1990 (Cambridge: Cambridge University
Freedom House (2010), Freedom in the World, 2010 Press).
edn , available at http://www.freedomhouse.org. Vanhanen, T. (2001), 'Domestic Ethnic Conflict
Gleditsch, Kristian Skrede (2007). Transnational and Ethnic Nepotism: A Comparative Analysis',
Dimensions of Civil War'. Journal of Peace Journal of Peace Research 36(1): 55-73.
Research 44(3): 293-309. Varshney, A. (2002), Ethnic Conflict and Civic Life:
Gleditsch, N , P Wallensteen, M Erikson. M. Hindus and Muslims in India (New Haven. CT:
Sollenberg. and H Strand (2002), Armed Vale University Press).
 A Guide to Multivariate
^ Analysis

© Chapter Summary
This chapter examines how we can develop and test multi-causal explanations of
political phenomena. It builds on the previous chapter by extending the principles of
bivariate analysis to take into account more than one independent variable In doing
so, w e exam ine two of the most widely used statistical analysis techniques in political
research, OLS regression and logistic regression. This chapter includes:

• causality and association;

• statistical control;

• spurious relationships;

• indirect causality;

• interaction effects;

• linear regression;

• logistic regression.

Introduction

This chapter exam in es m ultivariate analysis, which builds directly on m any o f the principles
d iscu ssed in C hapter 15. M ultivariate analysis involves exam in ing the pattern o f associations
betw een m ore than on e independ ent variable and the dependent variable. Political p henom ­
ena are rarely m o no-cau sal. If we are interested in trying to explain ‘why’ things happen, then
m ore often than not we need to develop explanations that take into accoun t m ore than one
ex p lan atory factor. Bivariate analysis allows us to get an initial sense about the structure o f the
relationship betw een two variables. However, this may not accurately reflect the ‘true nature
o f th e relationship for a variety o f reasons. U nd ertaking m ultivariate analysis allows us to
ex a m in e the im pact o f m ultiple factors on our d ependent variable o f interest, and to com pare
the exp lanatory pow er o f rival hypotheses. It also allows us to exam in e w hether the associa­
tion betw een on e in d ependen t variable and the dependent variable still holds up when we
c o n tro l for— or take into a cco u n t— oth er factors, and to explore w hether the im pact o f one
v ariable also depen d s upon the level o f ano ther variable. Investigating these issues helps us to
d evelop fuller explanations for why different political phenom ena occur, and provides a
r igorous test o f our explanations to see if they still stand up in the face o f close scrutiny.
As we have seen in previous chapters, the way in w hich we carry out quantitative analysis
d ep en d s upon th e types o f v ariables that we are interested in exam in ing. The sam e applies to
m u ltiv ariate analysis. In th is ch ap ter we con sid er two different m ethods: O LS regression
H O W TO DO R E S E A R C H IN P R A C T IC E

(w hich is an ex ten sion o f w hat we discu ssed in th e p rev ious ch ap ter for in terv al-lev el
d epen d ent v ariables) and logistic regression (w hich is ap p rop riate w hen o u r d ep en d en t
variable is catego rical, specifically a dich o tom y ). T h ese m e th o d s o f analysis are am o n g th e
m ost w idely used qu antitative tech n iq u es in p olitical research , and u n d erstan d in g h ow th ey
can be used and interpreted provides a stron g fou n d ation for c ritically ev alu ating m u ch o f
th e research that is published in leadin g acad em ic jo u rn a ls. The first part o f th is ch ap ter
exam in es th e principles o f m u ltivariate analysis, and th e d ifferent typ es o f an alytical q u es­
tion to w hich they can be applied. Th e secon d part ex am in es m u ltiv ariate analysis w hen ou r
depen d ent v ariable is interval level, using O L S regression th at b u ild s d irectly upon th e top ics
covered in C hapter 15. The third part th en c on sid ers m u ltiv ariate analysis w hen o u r d ep en d ­
ent variable is categorical, using logistic regression, w hich builds on cro ss-tab s.

The principles of multivariate analysis: statistical control

M ost social and political phenom ena are m u lti-cau sal, w hich m eans th at in sights garnered
from bivariate analysis, where only on e causal variable is consid ered , are going to be p artial,
at best. M ultivariate analysis allows us to exam in e the im pact o f m u ltip le causal factors on
our dependent variable o f interest in ord er to see w hich factors m atter o r w hich m atter m ost.
It thus provides a useful tool for developing a m ore com plete u n d erstand in g o f th e top ic o f
enquiry. M ultivariate analysis also allows us to have con fid en ce in ou r results, and to assess
w hether or not they are robust. D o our results still stand up when we in clude o th er p o te n ­
tially im portant factors?
Bivariate analysis provides a good starting point for exam in ing the stru ctu re o f relatio n ­
ships betw een variables. It allows us to develop causal explanations and provide tentative
answers to our research questions. But establishing c ausation is n o toriously d ifficult, and we
are probably never able to do it com pletely. At best, we therefore aim for ‘robust association ’.
This requires three condition s. The first is statistical association . The secon d is appropriate
tim e ordering. And the third is the elim in ation o f oth er possibilities. From bivariate analysis,
we can establish the first two cond ition s, but not the third. This last point is very im portant.
To have confidence in our findings we have to su b ject th em to rigorous testing. W e have to
be sceptical. If we observe a bivariate relationship, we always have to th in k about w hether
that relationship is real, or w hether there are potentially o th er factors that could accou n t for
it. This is where m ultivariate analysis com es to the fore.
Suppose we are interested in exploring social trust. A great deal o f research has exam in ed
the benefits o f living in a com m unity with high levels o f social trust, and the factors that are
associated with it. We might have three different hypotheses. O ur first hypothesis m ight be
that social trust tends to be lower in ethnically diverse com m un ities, sin ce people are m ore
likely to trust those who are like them . W hen there are many people from different eth n ic
backgrounds, the level of social trust in the com m un ity therefore tends to be lower, sin ce, in
the words of Robert Putnam , people ‘hunker down’ (Pu tnam 2007). O ur second hypothesis
might be that social trust tends to be lower in econ om ically deprived areas, sin ce neighb ou r­
hoods with econom ic problems are m ore likely to be run down and lack the nice civic spaces
that foster social trust, and so people may becom e socially withdrawn. And finally our third
hypothesis might be that social trust tends to be lower in areas with high levels o f crim e,
 M U LT IV A R IA T E A NALYS IS

sin ce crim e and the fear o f crim e stop people going out on to the street and interacting with
each other, and th erefo re fosters social isolation. From a bivariate analysis, we might find
sup port for each o f these hypotheses: that ethnically diverse areas tend to have lower levels
of social trust than eth nically hom ogen eous areas, that econ om ically deprived areas tend to
have low er levels o f social trust than econ om ically prosperous areas, and that neighbour­
ho od s w ith high levels o f crim e have lower levels o f trust than neighbourhoods with low
levels o f crim e.
A m u ltiv ariate m od el allows us to exam in e the relative im pact o f all these variables on
social tru st w hen we ex am in e th em sim ultaneously. That is, when we control for— or take
in to a c c o u n t— a n eig h b o u rh o o d s level o f econ om ic deprivation and crim e level, does its
level o f eth n ic d iversity still in flu en ce social trust? A no ther way o f thinking about this is,
sup posing we are com p a rin g two neighb ourhoo ds with exactly the sam e level o f econ om ic
d epriv ation and c rim e, but one neighb ou rhoo d has high levels o f e th nic diversity and the
oth er has low levels o f eth n ic diversity, would their respective levels o f social trust vary? If
th eir levels o f socia l trust did vary, then we could say that controlling for econ om ic depriva­
tion and c rim e, th ere is an a ssociation betw een eth n ic diversity and trust. We can express the
m u ltiv ariate m o d el that we w ant to test sym bolically, using arrow diagram s, as shown in
F igu re 16.1.
M u ltiv ariate analysis allow s us to com pare the im p ortance o f these factors against each
oth er, and to e x a m in e th eir in d epend en t im pact on the d ependent variable. For exam ple, we
m ig h t say th at eth n ic diversity and eco n o m ic deprivation have no independent im pact on
levels o f s o cia l tru st, o n ce we take into accou n t levels o f crim e. Ethnically diverse neigh­
b o u rh o o d s m ig h t ten d to b e relatively poor, and p oor areas m ight have higher levels o f
c rim e , but it is on ly c rim e that is directly associated with levels o f trust. That is, am ong
n eig h b o u rh o o d s w ith high levels o f c rim e, th e level o f trust does not vary by eth n ic diversity
o r e c o n o m ic d epriv ation . D iverse, poor, safe com m u n ities have the sam e level o f trust as
ho m o g en eo u s, rich , safe c o m m u n ities. The key th ing is w hether the com m u n ities are safe
o r n o t, no t how rich o r d iverse th ey are. In th is scen ario, eth n ic diversity and econ om ic
d ep riv ation have no in d ep en d en t effect on social trust when we con trol for, or take into
a c co u n t, levels o f crim e.
T h e idea o f statistical co n tro l is one o f th e m ost im portant ideas in quantitative analysis.
T h e d riving force b eh in d statistical con trol is to rule out theoretically plausible alternative
ex p la n a tion s for th e a ssociation betw een two variables to see if the association still stands up
w hen we take in to a c c o u n t— or con trol for— oth er variables. If we can rule out th e alternative

Figure 16.1 M ultivariate m odel: predicting social trust

394 H O W T O DO R E S E A R C H I N P R A C T I C E

explanations, then th is gives us m o re con fid en ce that th e way we have specified th e re la tio n ­
ship is the correct one. We can th erefo re have co n fid en ce that ou r results are robu st, w hich
ultim ately is what we strive for. We want to be sure that th ey stand up. A nd th e on ly way we
can do this is to see if we can kno ck th em down. This is a key s tren gth o f qu antitative analysis.
W e are not in terested in ch erry -p ick in g e v iden ce to sup port our a rgu m en t, but are interested
in finding evid ence that m ight co n trad ict ou r argum ent. So if we th in k th at th e level o f crim e
is really im portant for predictin g a n eig h b o u rh o o d ’s level o f trust, we have to th in k ab ou t all
the reasons why it m ight not be. Is it crim e, or the fear o f crim e, that drives social tru st? W h at
factors could accoun t for the apparent association betw een levels o f crim e and social trust?
If we can find oth er factors that can accou n t for this statistical asso ciatio n , th en we know our
initial explanation is not very conv in cing. But if we can’t, then we can have co n fid en ce that it
might be reasonable.

Specifying different types of relationship

As the above exam ple illustrates, when we m ove from bivariate analysis to m u ltiv ariate
analysis, we can som etim es uncover a different pattern o f association s from w hat we initially
observed. M ultivariate analysis can help to k no ck down spu riou s relationships or even
uncover real relationships that are hidden w hen we only ex am in e two variables. But it can
also allow us to exam in e how variables are related to each other, and d istingu ish betw een
direct and indirect causal influences, and uncover how the im pact o f on e variable is c o n tin ­
gent on the presence (or absence) o f ano ther variable. Below we discuss each o f th ese in turn.

C o n fo u n d in g v a ria b le s a n d sp u rio u s rela tio n sh ip s

C onfounding variables are ones that m ight obscure the ‘true relationship betw een two v ari­
ables. C onfounding variables are associated with both the probable cause and the ou tcom e o f
a specific political phenom enon, but have no direct causal influence them selves. A real-life
example o f this is the classic study o f student adm issions to the University o f C alifornia.
Overall, it appeared that wom en had a less favourable chan ce o f b eing accepted to the U niver­
sity than men, but this proved to be spurious once the subject they applied for was taken into
account: women applied for m ore com petitive subjects than m en, but w ithin each subject,
their acceptance rates were at least as good as m en’s. Thus the original association betw een sex
and acceptance was spurious, and sex can be regarded as a confoundin g variable.
W e ca n f u rt h e r i llustrate t he l ogic o f a s p u r i o u s r e l a ti o ns h ip wi th r e f e r e n c e t o t he e x a m ­
ple d i s c us se d e a rl ie r b e t w ee n a n e i g h b o u r h o o d s e t h n i c d i v e rs it y a n d level o f s o c ia l t r us t.
W e m a y o b s e r v e a r el at io ns hi p b e t we e n e t h n i c d ive r s it y a n d s oc ia l t r us t ( d e p i c t e d in F i g ­
ure 16. 2) . H ow e ve r, w h e n we i n t r o d u c e a t hi rd va r i ab le , say e c o n o m i c d e p r i v a t i o n , t he
si gn ifi ca n ce ol t he r el at io ns hi p b e t w e e n d iver s it y a n d t ru st d i sa p p e a r s . Ihat is, a m o n g
n e i g h b o u r h o o d s with the s a m e level o f e c o n o m i c d e p r i v a t i o n , e t h n i c d iver si ty h as n o
i mp ac t on social trust. Ihe re a s o n l or t he a p pa r e n t b iva r i at e a s s o c i a t i o n , t h e n, is t hat e t h ­
nic di versit y is a c o n f o u n d i n g va riabl e, a nd is a s so c i a t e d wit h an u n m e a s u r e d t h ir d v a r i a ­
ble, w i n d ! , S e c o n o m i c d e pr i va t io n ( see Fi g ur e 16 .3 ) . F t h ni c a l l y d iv er s e n e i g h b o u r h o o d s
ten d to he p o o r e r th an h o m o g e n e o u s are a s , hut, i m po r t an t l y , this is not r eg a r d e d as a
Causal r e la ti o ns hi p
 M U LT IVA R IA T E ANA LY SIS

| Diversity j ----------------------

Fig ure 16.2 Bivariate relationship between diversity and trust

Deprivation

Figure 16.3 Diversity as a confounding variable

Sp u riou s relationships can also w ork the oth er way round, and there can be a spurious
lack o f a sso cia tio n ’. This is w hen a ‘real relationship’ betw een two variables is masked by a
th ird variable. O bviously, to get a true idea o f how variables are causally related we want to
kn o ck dow n any spu riou s relationships. This is why we have to think carefully about our
m o d el s p e c ific a tio n , and w hether or no t our explanation includes all the theoretically rele­
vant causal factors that we can th in k of. Failure to control for a theoretically im portant vari­
able ca n m ea n that we a ttrib u te undue causal im p ortance to oth er variables. This is also
know n as o m itte d v a ria b le bias.

Intervening variables and indirect causality

In terv en in g (o r m ed iatin g) variables have a rather different role— they attem pt to m odel the
m e ch a n ism s th rou gh w hich, say, social class affects support for a particular political party.
P ossible in terven in g v ariables m ight be in com e or le ft-rig h t values. For exam ple, we might
find from a b ivariate analysis that th ere is a stron g association betw een class and party sup­
po rt (see F igu re 16 .4 ), but th at th is association weakens (or even disappears) when we c o n ­
trol for ideology. It m ight b e tem pting to th in k that the association betw een class and vote is
spu riou s, as d iscu ssed above, and that class doesn’t m atter causally. But the key point that
d ifferentiates an in d irect a ssociation from a spurious association is that we can plausibly
postu late a causal seq u en ce w hereby different classes hold different ideological positions,
w hich in tu rn affects o n e’s propensity to vote for a particular party (see Figure 16.5). To be
sure, th ese causal seq u en ces are largely a m atter o f theory, but no th eory could possibly sug­
gest th at, say, le ft-rig h t attitu des d eterm in e o n es class, although w orking-class people may
well b e d isp rop ortio n a tely left wing.

Fig ure 16.4 Bivariate relationship between class and vote

H O W TO D O R E S E A R C H IN P R A C T IC E

Figure 16.5 Ideology as an intervening variable

A lthough th e d istin ction betw een in d irect relatio nship s and spu riou s relatio n sh ip s can
look very sim ilar on the surface, th ey have different th eo retical im p lications. To distingu ish
betw een a confo u nd in g variable and a m o d eratin g v ariable we need to establish th e tem p ­
oral order o f what com es causally prior. So if we th in k d iversity causes depriv ation , we m igh t
th in k that there is an in d irect relationship betw een eth n ic diversity and tru st, and th at e c o n ­
om ic deprivation is th erefore a m ediatin g variable. However, if we th in k th at e co n o m ic d ep ­
rivation is causally p rior to eth n ic diversity, and that p oor areas tend to be m o re eth n ically
diverse b ecause im m igrants are m o re likely to only b e able to afford to live in p o o r areas (or
are m ore likely to b e allocated ho u sin g in p o o r areas), th en we w ould regard eth n ic diversity
as a confo und in g variable and say that th e apparent relationship betw een eth n ic diversity
and trust is spurious.

Moderator variables and interaction effects

So far we have assum ed that the in tro duction o f a third variable works in m u ch th e sam e way
for everyone. That is, we are dealing with uniform effects. But it is also often th eoretically
plausible that som e factors have greater influence on som e groups o f people (or cases) than
others. An in tera ctio n effect is when the im pact o f an independent variable is c on d ition al on
the presence o f another independent variable, or m oderator variable. So, for exam ple, su p ­
pose we want to investigate the relationship betw een som eone’s policy preferences and th eir
vote choice. We might th ink that those w ho hold left-w ing preferences are m ore likely to vote
Labour and those who hold right-w ing preferences are m ore likely to vote C onservative.
There is thus a bivariate relationship betw een ideology and vote (see Figure 16.6). But we
might also think that the form o f this relationship varies across different groups. In particular,
we might th ink that the relationship betw een ideology and vote choice is stronger for people
who have high levels o f political knowledge (sin ce perhaps they are m ore aware o f the id eo ­
logical differences between the parties), and weaker am ong people w ith low levels o f political
knowledge (since perhaps they are less aware o f the ideological differences betw een the par­
ties). The im pact o f ideology on vote choice is therefore contingent upon political knowledge,
which can be regarded as a m oderator variable. We can sum m arize this type o f relationship
symbolically, as shown in Figure 15.7, which shows that it is the com bination o f ideology and
knowledge together (the interaction) that is im portant for predicting vote choice.

| Ideology j -------------------------------► j y 0 te

Figure 16.6 Bivariate relationship between ideology and vote

 M U LT IV A R IA T E A NALYS IS 397

Ideology

Knowledge

Fig ure 16.7 Knowledge as a moderating variable

W e now turn to how these different types o f relationships can be exam ined in practice, and
discuss the principles o f m ultivariate analysis using O LS regression and logistic regression.

Multivariate analysis using OLS regression

W h en our depen d ent variable (th e th ing that we are trying to explain) is interval level (or
o rd in a l if th e v ariable ap p roxim ates a no rm al d istrib u tio n ), th e appropriate m eth od o f
analysis is O LS regression, as d iscussed in C hapter 15. M ultivariate analysis using O LS is just
a sim ple ex tension o f the sim ple lin ear regression m odel, and doesn’t involve any new m ajor
c on ce p ts o r tech niq ues. W e specify and estim ate the m odel in m uch the sam e way (only with
m o re in d epend en t variables) and in terpret the results with reference to the significance,
d irection , and m ag nitu d e o f the different param eter estim ates. In this section we exam ine
how we can specify a m ultivariate m odel and interpret the results from m ultiple regression,
w ith referen ce to th e sign ificance and d irection o f the param eter estim ates, and the fit with
the data. W e also show how th e results can be presented, using figures to plot the predicted
values o f th e d epen d ent variable for different com bin atio n s o f the independent variables.
In C hapter 15 we exam ined the association between two interval-level variables, but we can
also use O LS w hen one (or m ore) o f our independent variables is m easured at the n om inal level.
To do this, though, it is necessary to transform the nom inal variable into what are known as
du m m y variables. These are b inary variables. Even if the nom inal variable contains m ore than
two categories, we can break it down into a series o f ‘dum m ies’. An exam ple o f a nom inal-level
variable is an individuals religious identity. The variable may contain seven categories: Protes­
tant, C ath olic, M uslim , H indu, Jewish, O ther religion, and None. To estim ate the association
betw een religious identity and, say, support for the Iraq W ar using O LS regression, we need to
recod e th e religious ID variable into seven different dichotom ous variables, where each variable
is cod ed 1 for the category o f in terest and 0 for everyone else. See below:

PROT: (1) Protestant (0) Everyone else

CATH: (1) Catholic (0) Everyone else

MUS: (1) Muslim (0) Everyone else

HIND: (1) Hindu (0) Everyone else

JEW : (1) Jewish (0) Everyone else

OTH: (1) Other religion (0) Everyone else

NONE: (1) No religious ID (0) Everyone else.

H O W TO DO R E S E A R C H IN P R A C T IC E

T hese variables (apart from o n e) can th en be en tered in to th e regression eq u atio n . T h e v a ri­

able that is not entered in to th e eq u ation co n stitu tes th e referen ce categ o ry (fo r e x am p le No
religious ID ’). T h e coefficien ts for each o f th e oth er variables can th en b e in terp reted as
d escrib ing how m u ch m o re (or less) th e group in q u estio n sup ports th e Iraq w ar co m p ared
to the referen ce category (people with n o religiou s ID ).
To illustrate th e different steps we need to take to estim ate and in terp ret a m u ltiv ariate
m odel using O L S regression, we build on the exam ple we con sid ered in C h ap ter 15 to do
w ith the factors th at are associated w ith a co u n try ’s level o f d em ocracy.

Example 1: democracy
A great deal o f research has exam in ed the factors that are associated w ith th e level o f d e m o c ­
racy in a country, and th ere are th ree co m m o n ex p lan ation s for w hy so m e co u n tries are
m ore d em ocratic th an oth ers. These refer to eco n o m ic d ev elopm ent, eth n ic diversity, and
state form ation. The classic (1 9 5 9 ) study by Seym ou r U p set, suggested th at rich co u n tries
tend to be m ore d em ocratic th an p o o rer on es sin ce eco n o m ic d ev elopm ent gen erates th e
wealth, social cond ition s, and cultural values w hich are cond u civ e to lib eral d em ocracy.
M ore recently, it has also been argued that eth n ically d iverse cou n tries ten d to b e less d e m o ­
cratic th an m ore h o m ogen eous cou n tries, sin ce eth n ic d ifferences can divide so cie ty and
m ake com prom ise and consensu s m ore difficult. Finally, it has also b een argued th at c o u n ­
tries that have only recently gained in d epend en ce m ay be less d e m o cratic th an co u n tries
that have b een independent a long tim e, sin ce the in stitu tio ns o f th e state m ay be w eaker and
less organizationally developed.
B efore carryin g out a m ultivariate analysis, it is always a good idea to lo o k at how th e in d e­
pendent variables are related to each other. The assum ption is that th ey are in d ep en d en t—
that is, they m easure different things, b oth conceptually and em pirically. I f th is assu m ption
is violated, then there could be a problem o f m u ltico lin earity . This o ccu rs w hen tw o in d e­
p endent variables are so highly correlated that they do not m eanin gfully m easure different
things. For example, if we had two independent variables, such as G D P per capita and the
percentage o f the population with a high-speed in ternet co n n ectio n at ho m e, we m ight find
that the two variables are very highly correlated. That is, rich cou n tries all have a high degree
of internet penetration, and all coun tries with high levels o f in ternet c o n n ectio n are rich.
Under these circum stances, it b ecom es alm ost im possible to estim ate the in dependen t effect
o f each factor (sin ce they are not really em pirically different from on e a no ther). It is th erefore
advisable to either drop one o f the variables from the analysis, or to co m b in e the two v aria­
bles together to form one com posite m easure, say o f e con om ic and tech nological develop­
ment. There is no fixed cu t-o ff point for how correlated is too correlated, but generally when
the correlation coefficient between two independent variables is m ore than plus or m inus
0.7, there is a problem.
Table 16.1 reports the Pearsons correlation coefficient (see Chapter 15) between each pair o f
variables used in the analysis. The asterisk denotes whether the bivariate relationship is signifi­
cant at the 0.05 level. We can see that, at the bivariate level, each o f the independent variables is
significantly associated with the level o f democracy. The correlation coefficient between G D P
per capita and dem ocracy is 0.4, which is significant and positive, indicating that richer co u n ­
tries tend to he more democratic than poorer countries. The correlation coefficient between
 M U LT IV A R IA T E ANALYS IS

Table 16.1 Factors associated with democracy, bivanate correlation, Pearson's

GDP Ethnic Year of Polity IV

per capita d iversity Independence dem ocracy
GDP per capita

Ethnic diversity

Year of independence -0

Polity IV dem ocracy 0 (,/A ' __

Motes: 'Significant at p =0 05 level.

eth nic diversity and d em ocracy is -0 .2 7 , which is also significant, but this time negative, indicat­
ing that ethnically diverse countries tend to be less dem ocratic than ethnically homogeneous
countries. Third, the correlation coefficient between year of independence and dem ocracy
is - 0 .2 4 , w hich is significant and negative, indicating that countries which have becom e inde­
pendent recently are less dem ocratic than countries which becam e independent a long time ago.
Tu rn in g to the correlation s betw een the independent variables, we can see that there is a
sig n ificant negative correlation betw een G D P per capita and eth n ic diversity (Pearson’s
r = - 0 .4 6 ) . This in d icates that rich cou n tries tend to be less eth nically diverse than poor co u n ­
tries. Sim ilarly, th ere is a sign ificant negative relationship betw een G D P per capita and year
o f in d ep en d en ce (P earson’s r = - 0 .3 3 ) , in dicatin g that richer coun tries tend to have achieved
in d ep en d en ce earlier than p oorer coun tries. Finally, there is not a significant correlation
betw een year o f in d ep en d en ce and eth n ic diversity (Pearson’s r = 0 .14). Im portantly, though,
n o n e o f th ese c orrela tion s is sufficiently large to in dicate that th ere may be a problem with
m u ltico linearity. N ote, with correlation (un like regression), the relationship is sym m etrical.
Th is m eans that its value does not depend upon w hich variable is considered the in depend­
ent v ariable and w hich is consid ered th e depen dent variable. Thus, we can see that the c o r­
relatio n betw een G D P per capita and eth n ic diversity is exactly the sam e as the correlation
betw een eth n ic diversity and G D P per capita.
F rom th e b ivariate analysis reported above, we can see that there is a sign ificant negative
a sso cia tio n b etw een eth n ic diversity and level o f dem ocracy. That is, m ore eth nically diverse
co u n tries ten d to have low er levels o f dem ocracy. But how m u ch can we infer from this
biv ariate asso cia tio n ? W ould we be justified in con clu d in g that eth n ic diversity inhibits
d em o c ra c y in so m e way? W e m ay have th eoretical argum en ts why it m ight, and the data
m ig h t b e c o n sisten t w ith th ese ex p ectation s, but to have con fid en ce in th e result, we need to
ru le out th e possibility that oth er factors are responsible for the apparent correlation .
F or exam ple, we m ay suspect that th ere is in fact no d irect causal relationship betw een
eth n ic d iv ersity and d em ocracy. Rather, it m ay be th e case that eth nically diverse coun tries
ten d to b e ra th er p o o rer th an m o re ho m ogen eou s cou n tries (as we have also seen from the
co rrela tio n s rep orted above) and th at it is th is correlation that is d riving the apparent asso­
c ia tio n b etw een eth n ic diversity and dem ocracy. A ccordingly, w hen we co n tro l for e co n ­
o m ic d ev elop m ent, we m ay find that the a ssociation betw een eth n ic diversity and dem ocracy
d isapp ears. T o e x a m in e th ese possibilities, we can use O L S regression to e x am in e the im pact
on d e m o c ra c y o f all th ree in d ep en d en t variables. W e can sp ecify the m odel sym bolically (as
sh ow n in F ig u re 1 6.8) and algeb raically as follow s:
H O W T O D O R E S E A R C H IN P R A C T IC E

Fig ure 16.8 M ultivariate model predicting dem ocracy

Y = a + b,Xx+ b1X1 + biX)

where

X I refers to eth n ic diversity on a fractio n alizatio n scale (ran g in g fro m 0 to 1, w here 0

is hom ogen eous and 1 is diverse);
X 2 refers to p eriod o f ind epend en ce (w here 0 = P re -W W 2 and 1 = P o s t-W W 2 );
X 3 refers to G D P per capita in U S$ 1,000s.

Now, ra th er th a n ju s t hav in g o n e slo p e c o e ffic ie n t b to e stim a te (as we did w ith sim p le

lin ea r reg ression in C h ap ter 1 5), we have th re e slo p e c o e ffic ie n ts to e stim a te . T h e s e are
lab elled b l, b2, etc. T h e se c o e ffic ie n ts are k n o w n as p a r tia l r e g re s s io n c o e f f ic ie n ts .
T h e y are p a rtia l b ecau se th e ir m ag n itu d e d ep en d s u p o n th e o th e r in d e p e n d e n t v a ria ­
bles in th e eq u a tio n . T h a t is, th ey in d ica te how m u ch th e d ep en d en t v a ria b le c h a n g es for
a o n e -u n it in crea se in th e in d ep en d en t v ariab le, h o ld in g all th e o th e r v a ria b les in th e
m od el c o n sta n t. To give a c lear idea o f how th e in d ep en d en t v a ria b les are related to each
o th er and to d em o cracy , we w ill no t e stim a te th e full m o d el all in o n e go, b u t w ill g r a d ­
ually build it up by in tro d u c in g each in d ep en d en t v ariab le in to th e m o d el o n e at a tim e.
W e thus estim a te th ree m od els. T h e first m o d el ju st c o n ta in s e th n ic d iv ersity as an in d e ­
p en d ent variable, th e seco n d m o d el c o n ta in s e th n ic d iv ersity and p erio d w h en gain ed
in d ep en d en ce; and th e th ird m o d el c o n ta in s all th ree in d ep en d en t v a ria b les o f e th n ic
diversity, p eriod w hen gain ed in d ep en d en ce, and G D P p er cap ita. T h e resu lts are p r e ­
sented in Table 16.2.
In terpreting tables using O LS regression is relatively straigh tforw ard . The key th in g s to
look for are w hether the in dependen t variables are sign ifican t or n o t (u su ally at th e 0.0 5
level) and w hether the im pact o f the variable is in the exp ected d irectio n (p ositiv e or n eg ­
ative). W e can also in sp ect th e m ag n itu d e o f th e c o e ffic ie n t to see if it is su b stan tiv e ly
interesting.
Model 1 reports the effect o f eth nic diversity when it is the only explanatory variable in the
model. We can see that the coefficient is significant at the 0 .05 level (in d icated by an asterisk)
and the sign o f the coefficient is negative, w hich is in the expected direction . That is, m ore
ethnically diverse countries tend to be less dem ocratic. The m agnitude o f the coefficient is
k =-7.67. This tells us that, for every one point increase on the fraction alizatio n scale (w hich
ranges from 0 to 1), the expected level o f dem ocracy in a coun try declin es by 7.67 points on
the Polity IV scale (which ranges from 0 to 10). Taken on its own like this, the m agnitude o f
the coefficient refers to the total effect o f eth nic diversity on dem ocracy. W hen we include
additional variables, part of this effect may be accounted for by other factors, and so what we
 m u l t iv a r ia t e a n a l y s is

Table 16.2 Predict,ng the level of dem ocracy: OLS regression. unstandard,zed beta coefficients

M odel 2 Model 3
Standard Standard
B coefficient . . Standard
error error

Constant

Diversity

Independence

GDP

R-square 016

Adjusted R-square 0 15

Notes. * significant at <0.05 level, N = 145.

Source'. Pippa Norris. Democracy data set.

will be left w ith is th e partial effect; that is, the im pact o f eth n ic diversity controlling for the
im pact o f th e o th e r v ariables in the m odel.
The last row s o f th e tab le rep ort th e R -sq uare (see C hapter 15) and the adjusted R-square.
As we add ex tra in d ep en d en t variables to th e m odel, the adjusted R-square will only
in crea se if th e new term im proves th e m odel m ore than would be expected by chance. The
ad ju sted R -sq u a re is th ere fo re usually som ew hat low er than the R-square. The in terp reta­
tion o f th e ad ju sted R -sq u are is also som ew hat different from that o f R -square, and no
lon ger refers to th e ex p la in e d ’ v arian ce, sin ce th is has been adjusted by the nu m ber o f v ari­
ables in th e m o d el. It th ere fo re takes in to accou n t th e p arsim ony o f the m odel, and is higher
for a m o d el w hich ex p lains a given am o u n t o f v ariation in the dependent variable using
on ly a few in d ep en d en t variables than for on e that explains the sam e am ount o f variation
but uses m o re in d ep en d en t v ariables. It is th erefo re often a good idea to report b oth m eas­
ures. T h e R -sq u a re o f 0 .0 9 in d icates th at eth n ic diversity explains 9% o f the variation in
levels o f d em ocra cy .
In M o d e l 2 , we no w h av e tw o in d e p e n d e n t v a ria b le s — e th n ic fr a ctio n a liz a tio n , and
w h e th er th e c o u n try achieved in d ep en d en ce after W orld W ar II or not. We can see that both
term s are sig n ifica n t and in the ex p ected d irectio n . That is, eth n ically diverse cou n tries still
ten d to b e less d e m o c ra tic th an eth n ically h o m ogen eou s cou n tries, even when we take into
ac c o u n t w hen th ey achieved in d ep en d en ce. That is, th e relationship betw een eth n ic diver­
sity a nd d e m o c ra c y h o ld s b o th for c o u n tries th at achieved in d epend en ce b efore W orld War
II and for co u n trie s th at achieved in d ep en d en ce after W orld W ar II. In ad dition, we can see
th a t c o u n trie s th a t a ch ieved in d ep en d en ce after W orld W ar II ten d to be less d em ocratic
th a n c o u n trie s w h ich achieved in d ep en d en ce b efore it (b = -3 .9 9 ), con trollin g for eth n ic
diversity.
H ow ever, in terestin gly, we also n o te som e chan ge in th e m agnitu de o f th e coefficient for
th e e th n ic d iv ersity term . It d ecreases from b = -7 .6 7 , when it is the on ly variable in the
m o d el, to b = -5 .3 1 , w hen we co n tro l for th e effect o f in d epend en ce. This tells us that part o f
H O W TO D O R E S E A R C H IN P R A C T IC E

the apparent asso cia tio n b etw een eth n ic d iv ersity and d e m o c ra cy is really a fu n c tio n o f
in d epend en ce, and w hen we take in to a cco u n t w hen th e co u n try achieved in d ep en d en ce,
eth n ic diversity d oes no t appear to b e qu ite so im p o rtan t. T h is is b ecau se th o se c o u n trie s
w hich achieved in d ep end en ce after W orld W ar II ten d to be m o re eth n ica lly div erse th an
th ose cou n tries w hich achieved in d ep en d en ce before.
Finally, in M od el 3, we ex am in e th e relative im p act o f all th ree in d ep en d en t v ariables
sim ultaneously. This is our final m odel. W e now see th at, co n tro llin g for w hen a co u n try
a chieved in d ep end en ce and its level o f G D P p er capita, th e eth n ic d iv ersity o f a c o u n try has
n o sign ificant im pact on its level o f d em ocracy. By co n trast, th e effects o f in d ep en d en ce and
G D P per capita are b o th sign ificant and in th e ex p ected d irectio n . C o u n tries w h ich b eca m e
in d epend en t after W orld W ar II tend to be less d e m o cratic th an co u n tries w hich b eca m e
in d epend en t b efore it (c o n tro llin g for th e o th er variables in th e m o d el), and rich c o u n tries
tend to be m ore d em ocratic th an p o o r cou n tries (co n tro llin g for th e o th e r variables in th e
m od el). C on trollin g for G D P th erefo re com pletely w ipes out th e effect o f e th n ic diversity.
W h en we know how rich a cou n try is and w hen it achieved in d ep en d en ce, it d o esn ’t m ak e
any difference to our p rediction o f how d em o cratic a co u n try is if we kno w how eth n ically
diverse it is.
H ow th en do we acco u n t for th e apparent asso cia tio n b etw een e th n ic d iv ersity and
d em o cra cy at th e biv ariate level? W ell, on th e face o f it, th is appears to b e a ca se o f s p u ri­
ous co rrela tio n . A lthough eth n ically d iv erse co u n tries are less d e m o cra tic th an h o m o g e ­
neous on es, th is isn’t really anythin g to do w ith th e ir level o f e th n ic diversity, and is m o re
to do with th eir level o f eco n o m ic dev elopm ent. It ju st so hap p en s th at e th n ica lly div erse
co u n tries tend to be p o o rer th an eth n ically h o m o g en eo u s c o u n trie s, and it is th is c o r r e la ­
tion that drives th e apparent a sso cia tio n b etw een e th n ic div ersity and d em o cracy . O f
cou rse, if we could th in k o f som e th eo retically p lau sib le reaso n for w hy th e e th n ic h e te r o ­
g eneity o f a co u n try m ight in flu en ce its level o f e co n o m ic d ev elop m en t, th en we could
m ake a case for eth n ic diversity having an in d irect effect on levels o f d em o cracy . B u t, in
the ab sence o f th is th eory (w h ich appears u n likely), th e asso cia tio n is p ro b ab ly m o re
likely to be spurious.

Measures of fit
The R-square for the final m odel is 0.19. We can interpret this in a nu m b er o f ways. First, we
can say that our model explains 19% o f the variation in levels o f d em o cracy betw een the
countries. Or, to put it another way, we can say that if we are interested in p redictin g a c o u n ­
try s level o f dem ocracy, we will m ake 19% fewer errors if we know th eir scores on the in d e­
pendent variables than if we ju st guess based on the m ean. W e m ight not th in k this is very
good. To be sure, there are num erous oth er factors that we could include in our m odel that
might improve its predictive power. But at the sam e tim e the im p ortan ce o f R-square should
not be exaggerated. R-square is rarely the m ost im portant part o f the m o d el’s results. W e may
be particularly interested in R-square if the regression m odel is fitted solely for the purpose
of predicting future observations o f the dependent variable. M ore often, however, we are at
least as, or more, interested in exam in ing the nature and strength o f the association s betw een
the dependent variable and the explanatory variables and testing specific hypotheses, in
w h ich case the regression coefficients are the m ain param eters o f interest.
 M U LT IVA R IA T E ANA LY SIS

A s d is c u s s e d a b o v e , w e c a n in te rp re t th e re g re ss io n coe fficie n ts in te rm s o f th eir sig n ifi­

c a n c e a n d th e ir sig n , a n d w h e th e r o r n o t th e y s u p p o rt o u r th e o re tic a l e x p e c ta tio n s . E n te rin g
th e v a r ia b le s in a s te p -w is e fa sh io n a lso allo w s us to in sp e ct h ow th e p a tte rn s of a s so cia tio n
c h a n g e w h e n w e c o n t r o l fo r a d d itio n a l v a ria b le s, w h ich c a n sh e d light o n d ire ct an d in d irect
c a u s a l m e c h a n i s m s , as w ell as u n c o v e rin g s p u rio u s a s s o c ia tio n s . Bu t w e m a y also be in te r­
e s te d in m a k in g s o m e s ta te m e n t a b o u t w h ic h va ria b le a p p e a rs to b e th e m o st im p o rta n t.
W h a t f a c to r h a s th e b ig g e st im p a c t o n th e d e p e n d e n t v a ria b le ? W h a t m a tte r s m o st for a
c o u n tr y s level of d e m o c r a c y ? Its e c o n o m i c d e v e lo p m e n t? Its le n g th o f tim e as an i n d e p e n d ­
e n t s ta te ? Its e th n ic d iv e rs ity ?

Interpreting the magnitude of the coefficients

W e ca n n o t m ake d irect com p arisons betw een the regression coefficients for the different
in d epend en t variables because each variable is m easured on a different scale. Since the
regression coefficient tells us how m uch the dependent variable changes for each unit
in crease in the in d ependen t variable, the m agnitude o f the coefficients are all based on d if­
ferent scales, m ak ing d irect com p arison difficult. For exam ple, independence is m easured as
a d um m y variable, G D P per capita is m easured in U S $1,000s, and eth nic diversity is m eas­
ured in term s o f fraction alizatio n. W e can n ot say that each o f these units is equivalent, and
so d irect com p a riso n betw een the sizes o f the coefficients is essentially m eaningless.
H owever, it is often useful to be able to discuss the relative im pact o f different variables.
E ven if we c a n n o t d irectly com p are w hich variables ‘m atter the m o st’, there are a num ber o f
tech n iq u es that we can use to try and do this indirectly. We could ju st enter the variables in
th e m o d el on e by on e in step-w ise fashion and look at which variables lead to the largest
in crea se in R-square. However, the problem with this approach is that how m uch each v ari­
able appears to ‘explain will partly depend upon the order in which it is entered into the
m o d el, and variables w hich are entered early will m o re easily b e able to explain a lot o f v ari­
a tion th an variables entered tow ards the end.
A lternatively, we could ju st com pare how m uch difference it m akes to the dependent var­
iable w hen we in crease each in dependen t variable from its m in im u m value to its m axim um
value. So, for exam ple, co n tro llin g for all the oth er factors in the m odel, what is the expected
d ifferen ce in levels o f d em o cracy betw een the p oorest cou n try and the richest cou n try c o m ­
pared w ith th e ex p ected difference betw een the m ost eth nically h o m ogen eous coun try and
th e m o st eth n ica lly d iverse coun try? Th e problem with th is approach, th ough , is sim ilar to
th e prob lem s associated with look in g at the range, w hich we discussed in C hapter 14 with
referen ce to m easures o f variation . A ccordingly, sin ce how m uch each variable appears to
m a tter d ep en d s upon the ex tre m e values o f the d istrib u tion , this approach is very sensitive
to ou tliers. O n e or b o th o f th ese values m ight be atypical, giving a m isleadin g im pression o f
h ow m u ch th e in d ep en d en t v ariable really m atters.
O n e w ay we ca n get rou n d th is prob lem is to base o u r co m p ariso n s on th e standard
d ev ia tio n in stea d (see C h ap ter 14). From p rob ability th eo ry we know that a given p ro p o r­
tio n o f c a ses fall w ith in o n e stan d ard d ev iation o f th e m ean. If we th en com p are the am o unt
o u r d ep en d en t v a ria ble c h an g es in relatio n to a on e stand ard dev iation in crease for each o f
o u r in d ep en d en t v a riables, th en we are co m p a rin g th e chan ge in th e dep en d ent v ariable in
r ela tio n to b road ly eq u iv alen t c h an g es in th e d istrib u tio n o f o u r ex p lan atory variables.
H O W TO D O R E S E A R C H IN P R A C T IC E

This ‘b roa d eq u iv a len ce’ is often d iscu ssed in te rm s o f w hat are k n o w n as stan d ard ized
b eta c o efficie n ts, or stand ard ized reg ressio n co e fficie n ts, o fte n d en o ted (ca p ita l) B. M o st
sta tistica l softw are p rogram s ro u tin ely in clu d e th ese m e asu res as p a rt o f th e o u tp u t.
‘Stan d ard izin g’ a variable involves su b tra ctin g its m ean fro m ea ch in d iv id u al v alu e, and
d ivid in g by th e stand ard d ev iation . T h e m e tric for a stan d ard ized r eg ressio n c o e fficie n t is
th erefo re stand ard d ev iation un its. W e can th ere fo re draw c o m p a ris o n s b etw ee n th e
chan ge in Y for a broad ly eq u iv alen t ch an g e in th e d istrib u tio n o f X (m e a su red in stan d ard
d ev iation un its).
The final m odel is reprodu ced in Table 16.3, with in fo rm atio n on th e u n stan d ard ized b eta
coefficient, the standard error, th e standardized b eta co efficie n t, th e t value, and th e p value
for each o f th e in d ependen t variables. W e can see th at th e on ly term s for w hich th e p value
is below 0 .0 5 are in d ep end en ce and G D P p er capita. (R ecall th at th e p value prov id es th e
probability o f o b tain in g a test statistic o f th e observ ed size o r g reater i f th e null h y p oth esis is
true.) From the standardized b eta coefficien ts, we can get an idea o f th e relative im p o rta n c e
o f each variable in explaining v ariation in dem ocracy. G iven th at th e eth n ic d iv ersity term is
not significant, th ere is no need to discu ss it furth er. H owever, we can see th at th e sta n d a rd ­
ized regression coefficient for th e in d ep end en ce term (B = -0 .2 1 ) is o f alm ost th e sam e m a g ­
nitude as the coefficient for the G D P per capita term (B = 0 .2 2 ). B o th term s th ere fo re appear
to have m uch the sam e relative im pact.

Presenting the results

Finally, it is often helpful to present results graphically, as th is can illustrate th e im p act o f
certain key variables, and provide a clear visual representation o f th e results for p eople w ho
may shy away from looking at tables filled with nu m b ers and asterisks. W e can do th is in

Table 16.3 Predicting the level of dem ocracy: OLS regression, parameter estim ates

Standard S tandardized
e rro r B co efficien t

Constant

Ethnic diversity
0 IS
Independence
0 02
GDP per capita

R square

Adjusted R-square

Nc;r< ’ signifiant at 0 OS IcvpI

Scarce Pippa Norns Democracy data set
 M ULT IVA R IA T E ANA LY SIS

| ----- Pre-WW2 ------Post-WW2

Figure 16.9 Estimated impact of GDP and period of independence on level of democracy

m u ch th e sam e way as we did for bivariate regression (see C hapter 15), and use the regres­
sion eq uation to p red ict values o f Y, depicted algebraically as follows:

Y = a + blXl + b2X2 + bJXJ.

F ro m o u r analysis, we have estim ates for each o f the different p aram eters, and so can
s u b stitu te th e v alues o f ea ch u n stan d ard ized regression coefficien t in to the eq u ation as
follow s:

y = 3 . 1 - 3 . 3 X , - 2 . 9 X 2 + 0 .9 X 3.

T he last step is sim ply to plug in different values for X. W e can then predict values o f Y for
c o u n tries that gained in d ep end en ce after W orld W ar II (w hen X2 - 1) at different levels o f
G D P p er capita, and do th e sam e for c ou n tries that e x perienced in d epend en ce before World
W ar II (w hen X2 = 0) in the sam e way. The predicted values o f Y can then be illustrated
graphically, as sh ow n in Figu re 16.9.

Multivariate analysis using logistic regression

S o far we have consid ered m u ltivariate analysis for when our depen dent variable is m eas­
u red at th e in terval level. However, m an y p olitical p h en om en a that we are interested in are
n o t m easu red in th is way. For exam ple, th ose d oin g research on v oting b ehaviou r are often
in terested in w h ich party p eople vote for (Labour, C onserv ative, Lib D em , or o th er), o r even
w h e th er p eople vote at all in th e first place (vote o r not v ote). Sim ilarly, we may b e interested
in w h e th er o r no t a c o u n try has ex p erien ced civil war or eth n ic co n flict or terrorism in a
given p erio d o f tim e (m easu red in term s o f w hether th ey have or have no t). Indeed, research
on d e m o c ra c y often treats d em o c ra c y as a catego rical variable, and d istingu ishes betw een
H O W TO D O R E S E A R C H IN P R A C T IC E

w hether a cou n ty is d em o cratic or n o t, o r even w h e th er a c o u n try is a lib eral d em o cracy ,

electoral d em ocracy , o r som e o th er type o f ‘d em o c ra cy w ith ad jectiv e’.
W h en our d epen d ent v ariable (th e th in g th at we are try in g to e x p lain ) is a b in a ry v ariable,
the appropriate m u ltivariate tech n iq u e to use is lo g istic regression. O u r in d ep en d en t v a ria ­
bles can be m easured at eith er th e n o m in al, ord in al, or in terval level. T h is b u ild s on so m e o f
the ideas and tech niq u es we d iscu ssed w ith referen ce to cro ss-ta b s in C h ap ter 15, and is
sim ilar in som e respects to th e princip le o f O L S regression discu ssed above. W e use lo g istic
regression to exam in e the patterns o f association betw een o u r d ep en d en t variable and d iffer­
ent in d epend en t variables. W e first look to see w hether th e asso ciatio n is sig n ifican t, th en
look at the d irection o f th e asso ciatio n — w hether it is positive o r n egative— and th en , lastly,
we can m ake som e substantive p oin t about th e m ag nitu d e o f th e e ffect— th at is, w h e th er it is
large or sm all.
But logistic regression also in volves th e in tro d u ctio n o f som e new ideas. T h e m ain an aly ­
tical d ifference w ith O LS regression is that th e logistic regression co efficie n ts have a d ifferent
nu m erical in terpretation . This is b ecause th ey refer to odds and p rob abilities o f c h an g e in
the depend ent variable, rather th an sim ple in creases or decreases in th e d ep en d en t variable.
W hereas the coefficient in O LS refers to th e amount o f probable change in th e d ep en d en t
v ariable— that is, how m uch we ex p ect th e d epen d ent v ariable to chan ge for a o n e -u n it
increase in the in d ependen t variable, th e coefficien t in logistic regression refers to th e prob­
ability o f change in the dependent variable.
W ith O L S regression, the coefficients refer to th e im pact that a o n e -u n it in crease o f th e
independent variable has on the value o f the depen dent variable. W ith logistic regression,
the coefficients do not refer to the value o f the d epen dent v ariable— becau se it can on ly be
two values— either zero or one. Instead, the coefficient refers to the o d d s o r p rob ability o f the
d ependent variable taking one value rather than the other. So, for exam ple, if o u r d ep en d ent
variable is w hether a cou n try ex perienced civil war (cod ed 1) or not (cod ed 0 ), th e co e ffi­
cient for an interval-level variable tells us w hether the prob ability o f a co u n try ex p erien cin g
this outcom e (or the odds o f them having a civil war) in crease or decrease for every un it
change in the independent variable. By contrast, the coefficient for a catego rical-lev el v a ri­
able tells us w hether the odds o f exp eriencin g civil war are higher or low er for th e catego ry
in question in relation to the reference category.
If it w e re sim p ly a q u e stio n o f d e a lin g w ith c h a n g e s in th e p ro b a b ility o f th e d e p e n d e n t
v a ria b le ta k in g o n e v alu e o v e r th e o th e r, th e n th e o u tp u t fro m lo g is tic re g re s s io n w o u ld b e
q u ite in tu itive a n d e a sy to u n d e rs ta n d . B u t th is is n o t th e c a s e . F o r t e c h n ic a l r e a s o n s , lo g is tic
re g re ssio n in volv es tra n s fo rm in g p ro b a b ilitie s in to s o m e t h in g ca lle d od d s ra tio s a n d log
odds ratios. T h e s ta tis tic s b e h in d h o w th e s e a re c a lc u la t e d , a n d h o w th e p a r a m e t e r s a re
e s tim a te d , are q u ite c o m p lic a te d , a n d a re b e y o n d th e s c o p e o f th is c h a p t e r (b u t se e A g re sti
2 0 0 2 for a d e ta ile d in tr o d u c t io n ). W e th e re fo re fo c u s o n h o w th e y c a n b e i n te rp re te d , w h ic h
is far m o re s tra ig h tfo rw a rd .

T h e w o rk h o rse s o f lo g i s t i c re g re ss io n a re o d d s a n d o d d s ra tio s . T h e id e a b e h in d o d d s is
fairly s tra ig h tlo rw a rd , an d is c lo s c ly lin k e d to p ro b a b ilitie s a n d p e r c e n ta g e s . S o fo r e x a m ­
ple, ii a su rv e y rev e a ls th a t 7 8 % o f th e p o p u la tio n v o te d in an e l e c t io n , w e c a n c o n v e r t th is
p e rc e n ta g e in to a p ro b a b ility th a t s o m e o n e p lu c k e d at ra n d o m w o u ld h a v e v o te d . T h is
p rob a b ility is e x p re s se d as 0 .7 8 . W e c a n p h ra s e th is in a d iffe re n t w ay, a n d a sk w h a t a re th e
o d d s th at a p e rs o n p lu ck ed at ra n d o m v o te d ? T h e o d d s a re c a lc u la te d as th e p ro b a b ility o f
 M U LT IV A R IA T E A NALYS IS 407

having voted against the probability o f not having done so. This can be expressed as: p(v)/
p ( l- v ) = 0.78/0.22 = 3.5 5 :1 . The odds o f voting o f 3.55:1 tell us that for every one person
w ho did not vote, there were 3.55 people who did vote.
O dds ratios extend this type o f analysis by sim ply com paring the odds o f two different
groups. There are m any different ways in which we can do this, but one com parison that we
m ight be interested in is the odds o f young people voting com pared to old people. From Table
16.4 we can calculate the odds o f voting for each o f these age groups in the sam e way as above.
The odds o f voting am ong young people can therefore be calculated as: 0.651/0.349 = 1.865.
A nd the odds o f voting am ong old people are: 0.877/0.123 = 7.130. From these odds we can see
that the odds o f voting are higher am ong old people (7 .1 3 :1 ) than they are am ong young peo­
ple (1 .8 6 : 1). The odds ratio sim ply tells us how much higher (or lower) the first set o f odds are
c om pared to the last set (our reference category). So, the ratio between the odds o f voting for
old people com pared to the odds o f voting for young people (our reference category) can be
calculated as: 7.130/1.865 = 3.82 (see Table 16.5). This odds ratio tells us that the odds o f voting
am ong old people are nearly four tim es higher than the odds o f voting am ong young people.
O d d s ratios can take values betw een 0 and infinity, but they are always positive num bers.
If the odds ratio is 1, th en odds o f voting are the sam e for both groups. If the odds ratio is less
than 1, th en th e odds o f voting am ong old people is less than the odds o f voting am ong
youn g people (w hich is our referen ce category). And if the odds ratio is greater than 1, the
od d s o f v oting for old people are greater than the odds o f v oting for young people. Odds
ratios are th erefo re no t sym m etrical. For exam ple, if our reference category is young people,
th en an od d s ratio o f 4 tells us that the odds o f voting are four tim es higher am ong old people
th an am o n g y oung people. However, if our referen ce is old people, then an odds ratio o f 0.25
tells us th at th e od d s o f voting are four tim es less (on e-q u arter) am ong young people than
a m o ng old people.
O d d s ra tio s are a v ery usefu l way o f d e scrib in g th e asso ciatio n betw een two variables.
T h ey a lso form th e b a sis for in terp retin g lo g istic regression coefficien ts. A lthough the
o u tp u t fro m m o st sta tistic a l softw are p ackages d oes rep ort th e od ds ratios, som eth in g

Table 16.4 Voting by age group, colum n percentages

18-39 yrs 40-59 yrs 60+ yrs All

Did not vote 35 19 12 28

81 88 78
Voted 65

Source: British Election Study, 2005.

Table 16.5 Voting by age group, odds ratios

18-39 yrs 40 -5 9 yrs 60+ yrs

2 29 382
Voted (REF)

Source: British Election Study, 2005.

H O W TO D O R E S E A R C H IN P R A C T IC E

called th e log od d s ra tio s are usu ally rep o rted w h en w ritin g up th e resu lts in ta b les. T h e se
are based o n th e od d s ra tio s, b u t are tra n sfo rm e d u sin g s o m e th in g c alled th e lo g fu n c tio n ,
so th ey are c en tred on zero. L og od d s w ith p o sitiv e valu es th e re fo re in d ic a te th a t th e lik e ­
lih o o d o f v o tin g is h ig h er a m o n g th e group in q u e stio n th a n it is a m o n g th e r e fe re n c e
category, w hereas log od d s w ith neg ativ e v alues in d ica te th at th e lik e lih o o d o f v o tin g is
low er a m o n g th e group in q u estio n th an it is am o n g th e refe re n ce categ o ry . It is th e re fo re
im p o rta n t always to b e clea r w h e th er y ou are r e p o rtin g (o r in te rp re tin g ) lo g o d d s o r od d s
ratios.
A lthough the in terp retation o f th ese coefficien ts m ight lack th e in tu itiv e sim p licity o f
the O L S regression coefficients, logistic regression is really n o m o re com p licated to use or
und erstand than O L S regression. The princip les o f statistical c o n tro l are m u ch th e sam e; as
are the principles o f hypothesis testing. As w ith O L S regression, we are p rim arily in terested
in w hether our in d ependen t variable has a sign ificant asso ciatio n with th e d ep en d en t v a ri­
able (at th e 0 .0 5 level or a bove) and, if so, w hether th e d irectio n o f th e relatio nship is p ositive
(in d icated by positive log odds) or negative (in d icated by negative log od d s). W e can th en
m ake som e substantive in terp retation ab out th e m ag nitu d e o f th e effect, but th is is often
m ost easily d one by tran sfo rm in g th e log odds back in to prob abilities (w hich we w ill do later
in this chapter).

Example 2: turnout
In recen t years, th ere has b een a great deal o f c o n c e rn ab o u t d e c lin in g lev els o f tu rn o u t
in B rita in and m any o th e r ad v anced d e m o c ra cie s . In p a r ticu la r th e re h as b e e n c o n c e rn
ab ou t th e low p a rtic ip a tio n rates o f you n g peop le. V ariou s d iffe re n t e x p la n a tio n s have
been prop osed in ord er to try and ex p lain w hy p eop le vote o r do n o t v ote, and how th is
relates to th eir age. In th is s e c tio n , we fo cu s on th ree d iffe re n t sets o f fa cto rs th at are
widely th ou gh t to be asso ciated w ith w h e th er o r no t p eop le vote. T h e first set o f fa cto rs
focu ses on the so cia l c h a r a c te r istic s o f v oters, su ch as th e ir age, sex , and ed u ca tio n .
T h e re is a large b od y o f research th at ex a m in es p a rticip a to ry eq uality, and th e e x te n t to
w hich people from d ifferen t so cial b a ck g ro u n d s p a rtic ip a te in p o litic s. It is o f d e s c r ip ­
tive in terest to know w h e th er p articip a tio n rates vary by so cia l b ack g ro u n d , but it is also
o f analy tical in terest to know w h eth er th o se p eople w ith high levels o f ‘civ ic r e so u r c e s’
such as ed u cation are m o re likely to vote, and w h e th er peop le a c q u ire th e ‘h a b it’ o f v o t­
ing as they get older. T h e secon d fa c to r relates to how attach ed peop le feel to p o litic a l
p arties. If p eople have a stro n g a tta ch m e n t to a p a rticu la r p o litic a l party and feel th at it
‘rep resents’ them in som e way, th en we m ay e x p ect th at th ey will be m o re likely to vote.
Third, we can also lo o k at facto rs related to th e stru ctu re o f p o litica l c o m p e titio n , such
as w hether the electio n is co m p etitiv e and th e race is ex p ected to be c lo se , and w h e th er
th ere is m uch policy d ifferen ce at stake in term s o f how d ifferen t th e p arties are view ed
as b eing from each other.
By e x a m in in g w h ich o f th e se fa c to rs in flu e n c e t u r n o u t , a n d w h ic h f a c to r s in flu e n c e
tu rn o u t th e m o s t, w e c a n try a n d ge t a s e n s e o f w h a t is m o s t im p o r t a n t. Is tu r n o u t s o m e ­
th in g a b o u t v o te rs? S o m e th in g a b o u t p a rtie s ? O r s o m e t h in g a b o u t c o m p e t it i o n ? H o w d o
th e se d ifferen t ( a c to rs relate to an d in te ra c t w ith e a c h o th e r ? T o w h a t e x te n t c a n th e a n sw e rs
to th e se q u e stio n s shed light o n t u rn o u t d e clin e , a n d w h y y o u n g p e o p le h a v e b e c o m e
 M U LT IVA R IA T E ANA LY SIS

in creasing ly unlikely to vote? I h e follow ing analysis attem pts to provide som e answer to
th ese q u estio ns.
To exam in e how these factors are related to a persons likelihood o f voting, we analyse the
com bined B ritish election surveys, from 1964 to 2005, which gives us data on individuals’
social cha ra cteristics and strength o f attachm ent to political parties, and data on elections to
do with the expected closeness o f the race. We can specify the model algebraically as:

Logit = a + b, X , + b2X 2 + b ,X , + X, + b,X, + bbXh,

w here ou r depen d ent variable is the log odds o f som eone voting, and:

X I refers to an individual’s sex (1 = m ale; 0 = fem ale);

X 2 refers to an individual’s age (1 = under 30 years old; 0 = 30 years old and over);
X 3 refers to an individual’s level o f education (1 = university educated; 0 = school);
X 4 refers to an individual’s strength o f party identification: from 0 (n o attachm ent) to 4
(stron g atta ch m en t);
X 5 refers to an individual’s perception o f the policy difference betw een the parties: from
0 (n o d ifference) to 1 (som e difference) to 2 (big difference);
X 6 refers to the average gap in the opin ion polls betw een the two m ain parties’ share
o f the vote (m easured in percentage points).

T his m o d el can be in terpreted in m u ch the sam e way as the m ultiple linear regression model
as lon g as we rem em ber that the dependent variable refers to the log odds o f voting, not Y or
p rob abilities. W e are not tryin g to predict the value o f the dependent variable, but the likeli­
h o od (in term s o f log od d s) o f the dependent variable taking one value (voting) rather than
a n o th er (n o t voting).
H aving specified our m od el, we state the null hypothesis for each independent variable,
m ak e a d ecisio n on w hether we reject (o r fail to reject) the null hypothesis, and then if we
reject, d iscu ss th e pattern o f the association betw een the independent and dependent v ari­
able and w hether or n ot it co n fo rm s w ith our th eoretical expectations. T hese steps are exactly
th e sam e as for O L S regression, though the way we carry them out is slightly different.
T h e n u ll h y p o th e sis is that, for exam ple, con trollin g for the oth er variables in the m odel,
th ere is n o a sso cia tio n betw een a p erson’s age and w hether or not they voted. If the null
h yp othesis is tru e, th en th e partial regression coefficient (th e log o dds) for the age term will
b e zero. T h e sig n ifica n ce test for this hyp othesis is m uch the sam e as the oth er significance
tests we have d escrib ed earlier. W e look to see if the coefficient is significantly different from
z ero by c o m p a rin g the m ag nitu de o f the coefficient w ith its standard error. However, the way
we do th is in log istic regression is slightly different from how we did it in lin ear regression.
R a th er th an calcu late th e test statistic, we calcu late som eth in g called the z statistic, where:

P
se0)

and ¡J is th e estim ated coefficien t and sê(p) is th e estim ated standard error o f the coefficient.
T h e sq uare o f th e z statistic is often know n as th e W ald statistic (rep orted in m ost statistical
410 H O W TO D O R E S E A R C H IN P R A C T IC E

softw are packages), w hich has a ch i-sq u are distrib u tion . T h e associated p value th ere fo re
indicates the probability o f ob tain in g a W ald statistic o f th e ob serv ed size or g reater if th e null
hypothesis were true. If th e p value is less th an 0 .0 5 , we can reject th e null hyp othesis at th e 5%
level and go on to d escrib e th e p attern o f the a ssociation . R ecall th at in a logistic regression we
are in effect investigating how th e likelihoo d o f v oting varies a cco rd in g to th e range o f p red ic­
tors included in the m odel.
The results from th is m odel are presented in Table 16.6. F or th e tim e b ein g we w ill ju s t
focus on the sign ificance o f th e p aram eter estim ates and w h eth er th e a sso cia tio n is positiv e
or negative. Table 16.6 rep orts b oth the log odd s ratios and th e odds ratios. It is usual p ra c­
tice to report the findings in term s o f the log odds ratios, w hich have th e statistical advantage
o f b eing centred on zero, m aking the d irectio n o f the asso ciatio n clearly ev id en t (p ositiv e or
negative). W e find sup port for ou r ex p ectatio n th at social ch aracteristics, such as sex, age,
and ed ucation, m atter for w hether an individual will vote o r n o t, even w hen we co n tro l for
all the oth er variables in the m odel. For each o f th e social ch a ra c teristic variables, we can
reject the null hypothesis o f no association , sin ce th e associated p values are all below 0.05.
W e ind icate w hich coefficients are sign ificant by an asterisk. From th e d irectio n o f th e c o e f­
ficients, we can see that m en tend to be less likely th an w om en to vote (b = -0 .1 1 ); people
under 30 tend to be less likely than people over 30 to vote (b = -0 .5 5 ); and p eople w ho have
been educated to degree level or above tend to be m ore likely to vote th an th o se w ho have
not (b = 0.4 6 ).
Th ere is also a sign ificant and positiv e a sso cia tio n b etw een tu rn o u t and stren g th o f
party a ttach m en t. W e have treated stren g th o f p arty a tta ch m e n t as an in terv al-lev el v a ri­
able, and so the coefficien t tells us th at for ev ery un it in crease in th e stren g th o f a tta c h ­
m ent (on a 0 to 4 scale), th e log od ds o f so m eo n e v oting in crease (b = 0 .6 3 ). P eo ple w ith a

Table 16.6 Predicting turnout (logistic regression, parameter estimates)

Log odds Standard Wald Df P-value Odds ratios

ratios error

Sex: Female (REF)

Male 011’
■ 0 048 6 46 1 0011 091
Age: Over 30 yrs old (REF)

Under 30 yrs old ■0 :»s- 0 040 192 66 1 0 000 0 58

Education: No degree (REF)

Degree " „A - 0 071 43 10 1 0 000 1 59

Strength of party ID 1 0 000 1 88
Policy difference
1 0 00(1 1 S9
Poll gap between parties
1 0 000 0 98
Constant
C 000 18 /

21 279 rhi squ.ire 1 S40 degrees of freedom 6. p value <0 000S. 2LL 1761 3; Nagelkerke R Square 0 12
British [ lection Study ¿00S. cumulative data set
 M U L T IV A R IA T E A N A LYS IS

stron g party atta ch m en t th erefo re tend to be m ore likely to vote than people with only a
weak a tta ch m e n t, or no a ttach m en t at all. There is also evidence to support our hypothesis
that th e p olitical co n text m atters. There is a sign ificant and positive association between
tu rn ou t and p ercep tio n s o f policy d ifference betw een the m ain parties (b = 0 .3 3 ). Sin ce the
policy d ifference v ariable is also m easured on an interval scale, this indicates that for every
unit in crea se in the p ercep tio n o f policy difference, the log odds o f som eone voting
b ec o m e h ig h er (b = 0 .3 3 ). People w ho th in k that th ere is a great deal o f policy difference
b etw een the p arties are th erefo re m o re likely to vote than people who only th ink there is a
sm all d ifferen ce, or no d ifference at all. Finally, th ere is a significant and negative asso cia­
tion b etw een th e gap betw een the m ain two parties’ share o f the vote in the opin ion polls
and tu rn o u t (b = -0 .0 2 ). For every one p ercentage point difference betw een the C o n se rv a­
tive and L ab ou r sh are o f th e v ote in op in ion polls, the log odds o f som eone voting decrease
by 0 .0 2 . This in d icates that people are less likely to vote w hen a landslide v ictory is p re­
d icted for o n e party th an w hen th e race is expected to be very close.

Measures of fit
W e are often interested in tryin g to establish ‘how well’ we have m anaged to explain our
p h e n o m en o n o f in terest. D oes ou r m odel o f turn out provide a com prehensive explanation
for why people vote or not. To try and answer this qu estion, we often refer to what are
k now n as m easures o f fit. As with O LS regression, there is a nu m ber o f m easures that can be
used to d escrib e how well the logistic regression m odel fits the data— or explains the
o b serv ed v ariation in the data. However, they lack the intuitive appeal o f R-square, are
som ew hat m o re c u m b e rso m e to interpret, and th ere are d isagreem ents about w hich m eas­
ure is b est. The logistic regression equivalent o f R-square is know n as pseudo R-square. The
fact th at it is prefaced w ith th e word ‘pseudo’ in dicates that it is not really the sam e as n o r­
m al R -sq u are, as it d oes not represent explained variation’. Rather, it refers to the p ropor­
tio n a l im p rov em ent in th e fit to th e data. To calculate th e pseudo R-square we com pare the
m o d el’s fit to th e d ata w ith th e null m o d el’s fit to the data (w hen we only include the constant
and do no t in clud e any in d epend en t variables), using som eth in g called the log likelihood
(L L ) o r - 2 L og lik e lih o o d ( - 2 L L ), as it is often reported. Th ere are different ways o f doing
th is, and d ifferent m easures o f pseudo R -sq uare are based on slightly different calculation s,
so it is alw ays im p o rta n t to be clear about w hich m easure is b ein g used. The above exam ple
rep o rts th e N agelkerke R-Sq uare. This is 0 .1 2 , in d icatin g that the m odel im proves the fit to
th e d ata by 12% .

Interpreting the magnitude of coefficients

As alread y m en tio n ed , it is difficult to m eanin gfully in terpret the m agnitu de o f th e coeffi­
cien ts, and to c om p a re w hich factors m atter m o re than oth ers. A lthough th ere is no direct
eq u iv alen t to th e stand ard ized b eta coefficient th at we discu ssed with referen ce to O LS, we
ca n get an idea ab out th e relative m agnitu de o f th e different p aram eter estim ates by com p ar­
ing th e size o f th e log od d s ratio to its standard error. But a m o re intuitive way to illustrate
th e relative im p o rta n ce o f different v ariables is to conv ert th e log odds b ack into predicted
p rob a bilities, w hich are m u ch easier to interpret.
412 H O W T O D O R E S E A R C H IN P R A C T IC E

F ro m th e results o f o u r analysis, th e pred icted log od d s o f so m e o n e v o tin g ca n b e ex p ressed

by p lugging th e values o f th e d ifferent p aram eter estim ates in to o u r eq u a tio n fo r th e m o d el,
as follows:

2 3 5
Logit = 0.6 3 - 0 .1 1 * X, - 0 .5 5 * X + 0 .4 6 * X + 0 .6 3 * X< + 0 .3 3 * X - 0 .0 2 * X 6.

W e c a n th en use th is eq uation to p red ict th e log odd s o f so m eo n e v oting, b ased o n differen t

values for each o f the in d ep end en t variables. T h e m o st straigh tforw ard way o f d o in g th is is
sim ply to hold each value o f X at its sam ple m ean. T h en , o n e by o n e, we can ca lcu late th e
pred icted log odds o f v oting for a on e standard dev iation in crease in th e value o f ea ch in d e ­
pendent variable (rem em b e rin g to return its value b ack to th e m ean o n ce th e c alcu la tio n has
been perform ed ). For each v ariable, th en , w e have c alcu lated th e pred icted log od d s w h en all
variables are held at th eir m ean, and th e pred icted log odds w hen th e v ariable in q u estio n
has in creased by on e standard d eviation. W e can tran sfo rm th e p red icted log od d s in to p ro b ­
abilities using th e follow ing fu n ction :

P = ----------------------------------------------- ,
1 + e x p (-(fl + biX1+ ........btXb)

w here exp is the inverse log fu n ction (o r ex p onen tiated fu n c tio n as it is often called ).
A lthough th is looks like a bit o f a fiddle, it is relatively straigh tforw ard to d o in p ackages
such as Excel, and som e statistical softw are packages such as STATA will even do it for you.
The change in predicted probability o f v oting (o r chan ge in p redicted percen tage) can th en
be neatly depicted in a graph, as show n in Figu re 16.10.
From Figure 16.10 we can clearly see that party I D — th e s tren gth o f so m eo n e’s attach m e n t
to a political party— has the largest im pact on the predicted prob ability o f voting. H old in g
all oth er variables at th eir m ean, the pred icted level o f tu rn ou t in creases by 5.5 percentage
points for a on e-standard-deviation in crease in a person’s stren gth o f party attach m e n t. The

Young person etei

Poll gap

Policy difference

Party ID

Degree
—
Male

-4 00 - 3 00 - 2.00 -100 0.00 1.00 2.00 3.00 4.00 5.00 6.00

v ^ l b l l 6 10 Change m PredlCted lu rn° ut f0r a one-s,andard-deviation change in the independent

 M U L T IV A R IA T E A N A LYS IS

next m ost im portant variable is age, and then perceptions of policy diflerence. By contrast,
the im pact o f sex is relatively weak (even though it is significant). From this we can see that
how attached people are to political parties, and how much difference they think there is
betw een them in policy term s is strongly related to how likely som eone is to vote. By co n ­
trast, civic resources, measured in term s o f whether som eone has been to university or not,
is m ore weakly associated with turnout. This suggests that factors to do with parties and
policy (at least in term s o f how they are perceived) may be rather m ore im portant than fac­
tors to do with voters (in term s o f their civic resources). People interested in increasing the
level o f turnout may th erefore want to think about what parties stand for, and how this is
com m u n icated to the electorate.

Interaction effects
Up to now we have assum ed that each independent variable has the sam e im pact on the
depen d ent variable, regardless o f the oth er factors, such as a p ersons age. So, for example,
that b oth old and young people respond to the policy difference betw een the parties in the
sam e way; that is, that the im pact o f policy difference on turnout is the sam e for b oth groups.
But we m ight th in k that the picture is m ore com plicated than that. For exam ple, we might
th in k that young people are m ore likely to be influenced by the political context than older
people, sin ce they have not acquired the ‘habit o f v oting’. This suggests that different groups
o f people respond to d ifferent factors in different ways. W hen we have strong theoretical
reasons for suspectin g this, we can fit an in teraction term , w hich tests to see w hether the
im pact o f on e variable significantly varies across levels o f ano ther variable.
In teraction term s can be difficult to in terpret correctly, but they are frequently used in
qu antitative research and so are well w orth understanding. W hen done appropriately, in ter­
actio n term s can also be used in an incredibly revealing way to help shed light on com plex
em pirical problem s. They help us to exam in e heterogeneity, and don’t im pose a on e-size-fits
all exp lanation on p olitical phenom enon . Im plicitly, we m ake argum ents that involve in ter­
actio n term s all the tim e. For exam ple, G eddes (1 9 9 9 ) states that no th eory o f d em ocratic
tra n sition explains all real world variation. Th is is clearly true, but it does not m ean that we
have to th row up our hand s and give up trying to explain transitions. Rather, we can try to
develop exp lanations that take into accou n t this diversity, so, for exam ple, we can recognize
that p opu lar pressure from below does not in fluen ce transition s to d em ocracy in all cases,
but it ten d s to be o f m u ch stron ger in fluen ce in sin gle-party authoritarian regim es than
m ilita ry regim es. T h ere is thus an in teraction betw een the im pact o f civil society on d em o ­
c ra tic tra n sition and prior regim e type.
W e ca n illustrate th e princip les o f how to specify and in terpret in teractio n effects with the
follow ing exam ple o f electoral tu rn out. V arious scholars have suggested that people’s early
political exp erien ces have im p ortant lon g-term effects (e.g. Franklin 2 0 0 4 ). A ccordingly, it
is often said th at voting is a habit that is either acqu ired (or no t) at a relatively young age.
Peo p les early exp erien ces with electio n s shape th eir future behaviour. W h en th ere is plenty
at stake, youn g people go out to vote and acqu ire th e ‘h ab it’ o f voting, w hich lasts later in to
life. H owever, w hen th ere is no t m u ch difference betw een the parties, and they are viewed as
b ein g all th e sam e, th ey do not go out to vote, and thus acqu ire the habit o f not voting. The
key p oin t h ere, th en , is th at o n ce th e habit o f voting (or not voting) has b een acqu ired, older
H O W TO D O R E S E A R C H IN P R A C T IC E

people should be relatively im m u n e to th e s h o rt-te rm vagaries o f th e p o litical co n tex t. If

they have acqu ired th e habit o f voting, th ey w ill tu rn out to v ote m o re o r less regardless o f
w hether th ere are sub stantial differences b etw een th e p arties o r n o t. By co n tra s t, y ou n ger
people, w ho have no t acqu ired th e habit o f voting, w ill be far m o re resp onsiv e to th e p e r c e p ­
tion o f how d ifferent the p olitical parties are. W e can test th is ex p ecta tio n by fittin g an in te r ­
actio n term betw een age and policy d ifference. The in tera ctio n is illu strated in F igu re 1 6.11,
and show s that we estim ate an additional p aram eter for th e in tera c tio n b etw een a p e rso n s
age and th eir p ercep tio n o f policy d ifference. H ere age is a m o d erato r variable, and we e x a m ­
ine w hether th e association betw een p ercep tio n o f p o licy d ifferen ce and tu rn o u t varies
accord ing to w hether so m eo n e is un der 30 years old o r not.
W e can also specify th e m odel algebraically, w here th e in teractio n term is d efin ed as a
m ultiplicative term betw een age and p olicy d ifference:

Logit = a + blXi + b2X2 +fc3X , + b4X< + b5Xs +b6X6 + b7X2X,,

where our dependent variable is the log odds o f som eo n e voting, and:

X I refers to an in dividuals sex (1 = m ale; 0 = fem ale);

X 2 refers to an individual s age (1 = un der 30 years old; 0 = 30 years old and ov er);
X 3 refers to an individuals level o f education (1 = university educated; 0 = sch o o l);
X 4 refers to an individual’s strength o f party id en tification: from 0 (n o a tta ch m e n t) to
4 (stron g attachm ent);
X 5 refers to an individual’s p erception o f the policy d ifference b etw een th e parties:
from 0 (n o difference) to 1 (som e difference) to 2 (b ig d ifference);
X 6 refers to the average gap in the op in ion polls betw een two m ain p arties’ share
o f the vote (m easured in percentage points).

The results o f this model are reported in Table 16.7. We can see that all th e independent v ari­
ables which are entered by themselves are significant and in the sam e direction as before. W e
will therefore limit our focus to the discussion o f the in teraction effect. G reat care needs to be
exercised when interpreting interaction effects, since they can often cause confu sion . There are
three parameters o f interest. We have what are known as two parent term s— these are the v ari­
ables that appear in the model on their own, and an in teraction term — which is when these
variables appear in com bination with each other. We can see that each o f the parent term s is
significant and in the sam e d irection as before (see Table 16.7). W e can also see that the in terac­
tion term is significant (p < 0 .05). This tells us that there is evidence that the association betw een
policy difference and turnout is som ewhat different for young people and older people.

Figure 16.11 Mult.variate model predicting turnout, with an interaction term

 M U L T IV A R IA T E AN A LYS IS 415
Table 16.7 Predicting turnout with an interaction term between age and policy difference (logistic regres­
sion, param eter estimates)

Log odds Standard Wald df P value Odds ratios

erro r
Male -0 10' 0 04 6 34 l ....“ —

Degree 0 46' 0 07 42 08

Strength of party ID 0 63' 0 02 83646

Policy difference 0 27- 0 03 70 1 0 000

Gap in opin ion polls -0 02- 0 00 34 28 1 0 000 0 98
Young person -0 79' G 08 100 23 0 000 0*6

Young person by Policy

0 19* 0 05 11 97 1 0 001 121
difference

Constant 071 0 06 127 89 1 0 000 2 03

Note: N = 21,279.
Source: British Election Studies, 1964-2005.

S in ce old people are th e referen ce category, the in teraction term s denote how m uch bigger
(o r sm aller) th e m ag nitu d e o f the policy difference coefficient is for young people. The
in tera c tio n term betw een age and policy difference th erefore tells us w hether the associa­
tion b etw een p olicy d ifference and tu rn out is the sam e for young people (in w hich case the
c o efficie n t for th e in tera ctio n term would be zero), stron ger (in w hich case it would be
p o sitiv e), or w eaker (in w hich case it would be negative). Sin ce the in teraction term is p os­
itive, we can co n clu d e that the a sso ciatio n betw een policy difference and tu rn out is stronger
for youn g people th an it is for old people. The estim ated size o f the policy difference co e f­
ficien t for youn g p eople is th erefo re the size o f th e p arent term coefficient (b = 0 .2 7 ) plus the
size o f th e in tera c tio n term coefficien t (b = 0 .1 9 ), w hich gives b = 0 .46, w hereas the e stim ­
ated size o f th e p olicy d ifference coefficien t for old people is ju st the size o f the parent term
c o e fficie n t (b = 0 .2 7 ).
W e ca n illustrate w hat th is m eans m ore clearly by using th is in form ation to predict levels
o f tu rn o u t fo r young people and older people at different levels o f policy difference. We do
th is in th e sam e way as before, and hold all the oth er variables at th eir m ean. We then c a lcu ­
late th e pred icted prob ability o f v oting for a young p erson when there is the perception o f no
d ifferen ce b etw een th e p arties (cod ed 0 ) and when th ere is th e perception o f big differences
b etw een th e p arties (cod ed 2 ). W e do the sam e for older people. The results are presented in
T able 16.8.
T able 16.8 show s that w hen th ere is a p erceptio n o f big policy differences betw een the
p arties, p eople over 3 0 y ears old are ju st slightly m o re likely to vote (0 .9 1 ) than people under
3 0 ( 0 .8 6 ). O r, put a d ifferent way, hold ing all oth er factors con stan t, the predicted difference
in th eir tu rn o u t rates is ju s t 5 percentage points. However, when th ere is a perception o f no
d ifferen ce b etw een th e parties, th e predicted prob ability o f v oting is m u ch higher for old
p eople (0 .8 4 ) th an it is for young people (0.71 ), even if b o th groups are less likely to vote than
H O W TO D O R E S E A R C H IN P R A C T IC E

Table 16.8 Predicted probabilities

Parties are all the sam e (0) Big differences betw een the parties (2)

Under 30 years old 071 086

30 years old and over 084 091

when th ere are b ig p olicy d ifferences. In th is in stan ce, h o ld in g all o th er fa c to rs co n s ta n t, th e

pred icted d ifference in th eir tu rn o u t rates is now 13 percen tage p oin ts.
W e can a lso see th at, for o ld p eople, th e p red icted level o f tu rn o u t is 7 p erc e n ta g e p o in ts
hig h er w hen th ey th in k th ere are b ig d ifferen ces b etw ee n th e p a rtie s co m p a red w ith w h en
th ey th in k th ere aren’t any d ifferen ces at all. By c o n tra s t, for you n g p eo p le, th e p red icted
level o f tu rn o u t is 15 p ercen tag e p o in ts higher. F ro m th is we can in fer th a t w h e th er th e
parties are all th e sam e or no t does no t m atter so m u ch to o ld er p eo p le— th ey are q u ite
likely to vote in any event, p erhaps b ecau se th ey have b een so cialized in to v o tin g fro m an
earlier age w hen th ere was m o re at stake d u rin g e lectio n s. H ow ever, for y ou n g p eo p le, it
m akes m o re d ifferen ce how different th e p arties are p erceiv ed to b e — w h en th ey are v ery
d ifferent, young p eople are alm o st as likely as old p eop le to v ote, b u t w hen th ey are m u ch
the sam e, th en young people are far less likely to vote and are also m u ch less likely to v ote
than old er people.
The results from th is sim ple m od el shed som e light on recen t d iscu ssion s ab ou t tu rn ou t
decline. First, we can see that im portant factors in d ecidin g w h eth er people vote o r no t are
to do w ith th eir level o f party id en tification, and the stru ctu re o f the p olitical co n text. O v er
the past few years party iden tification has d eclin ed, th e difference betw een th e p arties has
narrowed, and we have had a nu m b er o f election s that were landslides (but n o tab ly no t the
one in 2 0 1 0 ). All oth er things b eing equal, this can lead to a substantial d eclin e in levels o f
turnout. However, m ore interestingly perhaps, we can th in k that th ese chan ges have had a
disproportionate effect on young people. Sin ce young people are m ore in fluen ced by the
political context, the policy convergen ce under New L ab ou r betw een the tw o m ajo r p arties
had a greater im pact on them than it had on older people, and so th ey are corresp ond in gly
less likely to vote. Part o f the reason, th en, why tu rn out am ong young p eople h as b een so low
over the past few elections may be because the parties have b eco m e m ore sim ilar to each
other. It is not that young people have changed per se— they have alw ays paid close attention
to the contest— but because the parties have changed. And as they have b eco m e m o re sim i­
lar, there is less incentive to vote.

Presenting and reporting data

W h a te v e r th e typ e o f an a lysis th a t is b e in g u n d e rta k e n , it is im p o r ta n t to p re s e n t a n d r e p o r t

th e in fo rm a tio n in a c le a r an d a c ce s sib le w ay. T ab les m u s t c o n ta in all th e re le v a n t i n f o r m a ­
tion an d be c le a r an d easy to u n d e rsta n d for th e re a d e r. In p a rtic u la r, it is im p o r ta n t th a t th e y :

• h a ve a cle a r title, w ith c le a r i n fo rm a tio n a b o u t th e d e p e n d e n t v a r ia b le a n d th e ty p e o f

an alysis u sed (lo g istic re g re ss io n , O I.S re g re s s io n , e t c .) ;
 M U LT IV A R IA T E AN A LYS IS

• ind icate clearly the coefficient (unstandardized and/or standardized beta coefficients
for O LS; log odds ratios and/or odds ratios for logistic regression);

• have clear labels for the independent variables (always give a self-explanatory label);
• ind icate what the ‘reference category’ is for categorical variables;

• give in form ation that enables the reader to judge the level o f significance o f the
p aram eter estim ates (asterisks and/or standard errors);

• report all param eter estim ates (in cludin g non-sign ificant ones);

• provide som e in d ication o f goodness o f fit (R -sq uare for O LS; pseudo R-square, chi
square for logistic regression;

• report the sou rce o f the data;

• report the base (sam ple size) for the actual analysis.

W hen w riting up the analysis, it is good practice to avoid causal language o f ‘effects’, ‘drivers’,
and so on and use language sim ply as if you were reporting patterns o f association. C on cen ­
trate on w hether the results confirm or refute your prior theoretical expectations, with refer­
ence to the d irection o f the param eter estim ate as well as on the level o f significance. We have
com pared the sizes o f the param eter estim ates only in the vaguest and m ost general terms. If
we want to provide a m eaningful interpretation, we can calculate and present the predicted
probabilities (for logistic regression) or the predicted values o f y (for O LS regression).

Conclusions
M ultivariate analysis is a powerful tool for exploring the relationship between different variables. It
allow s us to develop fuller accounts of w hy things happen, and to see whether these accounts are
robust or not. M ultivariate analysis also allows us to exam ine different types of relationship. We can
use m ultivariate analysis to debunk spurious relationships, or to illustrate indirect causal m echan­
isms. W e can also use it to specify interaction effects, and to exam ine how the impact of one
variable depends upon the level of another. These techniques are in credibly powerful, and can be
used to good effect to explore a w ide range of political phenom ena. But, at the same tim e, the
results from m ultivariate analysis should alw ays be analysed w ith caution. Do the results really
support the argum ent that the author is using them for? Is there a strong link between the
theoretical expectations, and how this theory has been operationalized and tested? Have any
p o tentially im portant variables been left out? H ow might the inclusion of these variables change
the results? Have the variables been m easured in a satisfactory way? These questions dom in ate all
q uantitative studies, and it is up to the researcher to dem onstrate that they have taken steps to
answ er them as best they can.

Questions
• W hat is the difference between a confounding variable and a mediating variable? Illustrate your
answ er w ith exam ples.
• W hy should w e treat the results from bivariate analysis w ith caution? W hy might the findings not
stand up in a m ultivariate analysis?

• W hat is the difference between causality and association?

• H o w do w e know if a relationship between two variables is causal or not?

H O W TO DO R E S E A R C H IN P R A C T IC E

Guide to Further Reading

Agresti, A (2007), An Introduction to Categorical Data Analysis (New York: John W iley & Sons).
Presents a non-technical introduction to topics on categorical data analysis, such as odds ratios and
logistic regression.

- . and B. Finlay (2009). Statistical Methods for the Social Sciences, 4th edition (Upper Saddle River,
NJ: Pearson-Prentice Hall).
Covers the basics of statistical description and in ference, as w ell as m ore advanced topics on
regression methods, including m ultiple regression, A N O VA and repeated m easures A N O VA , analysis
of covariance, logistic regression, and generalized linear models.

Carlson, James and Mark Hyde (2002), Doing Empirical Political Research (Boston, MA: Houghton
Mifflin).
Provides an introduction to quantitative analysis and covers som e m ore advanced topics, w ith lots of
examples from political research.

de Vaus, David (2002), Surveys in Social Research (London: Routledge).

A detailed and accessible introduction to quantitative m ethods, w ith easy-to-read chapters on how to
carry out analysis and interpret the results.

Field, A. (2009), Discovering Statistics using SPSS (and Sex and Drugs and Rock 'n' Roll) (London:
Sage Publications).
Provides an easy-to-follow introduction to quantitative analysis and step-by-step instructions on how
to use the SPSS statistical software package to do your own analysis.

Johnson, Janet Buttolph, Henry Reynolds, and Jason Mycoff (2007), Political Science Research
Methods, 6th edition (Washington, DC: CQ Press).
Provides an introduction to quantitative analysis and covers som e m ore advanced topics, w ith lots of
examples from political research.

Rowntree, Derek (2000), Statistics without Tears: An Introduction for Non-Mathematicians

(Harmondsworth: Penguin).
An easy-to-read introduction to statistical methods.

References
Agresti, A (2002), Categorical Data Analysis (New Upset. Seymour Martin (1959). Some Social
York: John Wiley & Sons) Requisites of Democracy: Economic
Franklin. M (2004). Voter Turnout and the Dynamics Development and Political Legitimacy', American
of Electoral Competition in Established Democracies Political Science Review 53: 69-105.
since 7945 (Cambndge Cambridge University Press)
Putnam. Robert D. (2007), E Pluribus Unum:
Geddes Barbara (1999). What Do We Know about Diversity and Community in the Twenty-First
Democratization after Twenty Years7’ Annual Century The 2006 Johan Skytte Prize Lecture',
Review o f Political Science 2 115-44 Scandinavian Political Studies 30(2): 137-74.
 A P P E N D IX

Finding and Citing Sources

for Political Research

I. Typ«s of Sources also include guides to reference works and bibliographies

Print Materials http://www.worldcat.org/search?qt-worldcat_org_
aII&q=historical+literature.
Many of these materials can now be accessed online.
Book Review Digest Indexes reviews appearing in about 11X)
Bibliographic and Reference Materials
periodicals; includes brief excerpts. http://www.ebscohost.
Social Sciences Citation Index (SSCI) An index to works com/public/book-revicw-digest/.
cited in current social and behavioural science literature.
The SSCI provides access to current information and ret­ Books in Print The online version includes full text reviews
rospective data from 1956 onwards. Fields covered include from Choice, Library Journal. New York Times, and other
anthropology, history, library science, law, linguistics, phil­ review sources, http://www.booksinprint2.com.
osophy, psychology, political science, public health, sociol­
International Bibliography o f the Social Sciences Contains
ogy, urban studies, and women's studies. The SSCI adds
bibliographic information from over 2,600 journals pub­
approximately 60,000 new cited references per week.
lished in 100 countries and over 6,000 books per year, in
Arts & Humanities Citation Index (A&HCI) Provides ac­ the fields of anthropology, economics, political science,
cess to current information and retrospective data from 1975 and sociology. Coverage is from 1951 to the present, with
onwards. Fields covered include archaeology, architecture, quarterly updates, http://www.ibss.ac.uk.
art, Asian studies, folklore, history, language, linguistics, phil­
Dissertation Abstracts Online Subject, title, and author
osophy, radio/television/film, religion, and theatre. A&HCI
guide to virtually every American dissertation accepted at
adds approximately 15,250 new cited references per week.
an accredited institution since 1861; selected Master's theses
http://science.thomsonreuters.com/mjl/scope/scope_ssci/.
have been included since 1962. In addition, since 1988, the
W orldCat The world’s most comprehensive bibliography. database has included citations for dissertations from 50 Brit­
It covers virtually all the material catalogued by any of sev­ ish universities; and citations and abstracts from Section C,
eral thousand member libraries, including books, serials, Worldwide Dissertations (formerly European Dissertations).
archival materials, maps, visual materials, computer files, http://library.dialog.com/bluesheets/html/bl0035.html.
and internet resources, http://www.worldcat.org.
ProQuest Dissertations and Theses Database Citations to
Library o f Congress Catalog (LC) The LC is arguably the dissertation and theses from around the world from 1861
world’s most comprehensive library. Its web page includes to the present day, together with 1.2 million full text disser­
tations that are available for download in PDF format. The
links to the catalogues of several hundred other research
libraries, primarily in North America, the UK, and Aus­ database offers full text for most of the dissertations added
tralia. Available at http://catalog.loc.gov/. since 1997 and strong retrospective full text coverage for
older graduate works, http://www.proquest.com/.
Library o f Congress Classification Outline Useful for iden­
The Statesm an’s Yearbook Information on every country
tifying the LC call numbers that you should use to browse
in the world, covering key historical events, politics, econ­
to find a particular kind of material. Available at http://
omics, trade, and infrastructure. http.V/www.statesmans-
lcweb.loc.gov/catdir/cpso/lcco/lcco.html.
yearbook.com/.
G uide to Historical Literature (American Historical
Handbooks, Yearbooks, and Encyclopaedias that provide
Association) The basic principle of organization is geo­
information on various facets of the political world
graphic, with some reliance on topical and chronological
subdivisions. The geographic sections include brief over­ Political H andbook o f the World An overview of the pol­
views of scholarly trends, and helpful annotations; some itical system and situation in countries around the world.
420 A P P E N D IX

Europa W orld Yearbook Snapshots of the political, econ­ Reader's G uide to P eriodical Literature Indexes general
omic, and social conditions of countries worldwide. periodicals (magazines, including newsmagazines, but not
scholarly journals), http://www.hwwilson.com/databases/
World Encyclopaedia o f Political Systems an d Parties Readersg.htm.
Explanation of executive, legislative, and judicial systems,
political parties, and electoral systems around the world. Africa Digest Weekly record of African events (since
1961), with index.
Public Opinion and Polling Around the World: A Histori­
cal Encyclopedia Africa Research Bulletin (Political and Cultural Series)
Indexed monthly summary of African political develop­
World Encyclopaedia o f Parliam ents an d Legislatures ments (since 1964).
Country-by-country analysis of politics, parties, elections,
and parliaments around the world. Asian Recorder A weekly digest of Asian events (since
1955), with index.
The Encyclopaedia o f Democracy Articles on the concepts
of democracy, important figures, and country and regional The M iddle East an d North A frica. A survey and directory
assessments of the state of democracy in the world. of the countries of the Middle East (since 1948) and North
Africa (since 1964), with a listing of events.
Encyclopaedia o f W orld Political Systems A country-by-
country survey of the political environment of a country BBC W orldw ide M onitoring: Intern ation al Reports The
and how that has affected governments. BBC’s Monitoring Service systematically tracks broad­
casts from 120 countries in 50 languages. The Summary
Encyclopaedia o f Nationalism The role of nationalist of World Broadcasts (SWB) was a separate part o f the
movements in politics, with entries for specific countries. service until the end of March 2001, when it was dis­
continued. Alter that date, the information previously
International Encyclopaedia o f Elections Forms of elec­
carried in the SWB became part of the International
tions and election processes.
Reports. Accessible through LexisNexis: http://www.
Newspapers, News Services, News Digests, and News Maga- Iexisnexis.com.

Some of the sources, below, provide news summaries. Oth­ Academic and Trade Journals
ers are indexes to newspapers or to periodicals which re­ fSTOR Reproduces the full image of over 117 scholarly
port or analyse current events. Others tap various sources journals, starting in each case with volume 1. The purpose
of current opinion and comment, or contain references to is to provide an archival collection of core scholarly jour­
more general sources. nals, not current access; most articles are at least two years
old. Full text searching is available. Runs of additional ti­
Newseum Allows you to find daily newspaper front pages tles are gradually being added to the database. http://www.
from 68 countries across the globe. http://www.newseum. jstor.org.
org.
Project MUSE Includes the full image of all articles in over
World News Connection Provides English-language transla­ 100 scholarly journals, including all Johns Hopkins Uni­
tions of material from thousands of media sources. Included
versity Press journals. Full text searching, as well as subject
are political speeches, television and radio broadcasts, and
searching, is available, http://muse.jhu.edu/.
articles from newspapers, periodicals, and books. Online
coverage: 1996 to the present, http://wnc.dialog.com/. For Web o f Knowledge This source includes five databases
earlier coverage, see the Foreign Broadcast Information covering virtually all fields of knowledge. Two of the most
Service. Daily Report. Transcripts of foreign radio broad­ useful are the Social Sciences Citation Index (SSCT) and the
casts, including news and comment. Arts & Humanities Citation Index (A&HCI). With these
databases, you can discover networks of scholarly articles
Facts on File A weekly news digest (since 1941), indexed an­
relevant to your particular research interests. Beginning
nually and with an attached atlas, http://www.fofwcb.com.
with any article (hat you have found valuable, you can lo­
Keesing’s Contemporary Archives A monthly rccord of cate all subsequent journal articles that cite it. Registering
world events (since 1931) with indexes, http://www.kees- tor Current Contents Connect, which is part of the Web of
ings.com. Knowledge, allows you to set up a personal alerting pro­
file for journals that you select. This profile automatically
Annual Register o f World Events http://annualrcgister. emails you the latest tables of contents for your chosen
chadw\ck.com. journals as they appear. Registering also enables you to set
up citation alerts that notify you when future articles cite
Newspaper Indexes A good index to a major newspaper key articles you have found to be of special interest. Cur­
provides an annual register of events. See, for instance. Vie rent (.ontents Connect includes a personal home page that
■New )ork lim es index, or Ihe Times (London) Index.
displays all your selected titles and allows you to see the
 A P P E N D IX

latest table of contents by clicking on a title. http://wokinfo. Find the name ol the country you are researching in
com/products_tools/products/. the index to be directed to abstracts of articles on that
country.
Social Sciences Full Text An index to English-language
periodical articles. Subjects include anthropology, area Education Resources information Center (ERIC) Provides
studies, economics, international relations, law, political access to more than 13 million bibliographic records of
science, and sociology. Online coverage from February journal articles, books, research syntheses, conference pa­
1983, with abstracts from lanuary 1994, and some full pers, technical reports, policy papers, and olher education-
text articles, http://www.ebscohost.com/academic/social- related materials, with hundreds of new records added
sciences-full-test/ multiple times per week. If available, links to full text are
included, http://www.eric.ed.gov/.
Ulrich's Periodicals Directory on the Web Provides de­
tailed information on more than 300,000 periodicals of all EServer Journals Collection Provides links to popular jour­
types: academic and scholarly journals, e-journals, peer- nals. Organized by subject. http://journals.eserver.org/.
reviewed titles, popular magazines, newspapers, newslet­
ters, and more, http://ulrichsweb.serialssolutions.com/. Government Reports and Legal Documents
PolicyFile: Public Policy Research and Analysis Indexes
Public Affairs Information Service Bulletin Indexes nu­ and abstracts of publications, covering a complete range
merous periodicals in the field of public affairs and public of public policy research from such organizations as the
policy: journals, pamphlets, government documents, some American Enterprise Institute, Brookings Institution,
books, http://www.csa.com/factsheets/pais-set-c.php. Carnegie Endowment for International Peace, IMF, World
Bank, CS1S, Rand Corporation, and many others. Where
Columbia International Affairs Online (CMO) Allows
available, access to home pages and full text is provided.
search engine access to working papers from a wide range of
http://www.policyfile.com/.
think tanks and to tables of contents and article abstracts from
the most recent issues of about 35 international affairs jour­ Public Affairs Information Service (PAIS) A selective list­
nals. Also contains information on forthcoming scholarly ing, with abstracts, of journal articles, books, and docu­
conferences in international affairs, http://www.ciaonet.org/. ments. Subjects include economics, business, political
science, sociology, demography, and international law and
Fulltext Sources Online A systematic list of periodicals relations, with an emphasis on public-policy-orientated lit­
available online in full text, along with the names of the erature. Includes titles published in six languages, http://
databases that contain them. Includes foreign- as well as vrww.csa.com/factsheets/pais-set-c.php.
English-language periodicals, http://www.fso-online.com.
LexisNexis Provides the full text of hundreds of publica­
Historical Abstracts Selectively indexes over 2,000 journals tions, including taw journals, wire services, country econ­
in the field o f history, with abstracts. Also indexes selected omic reports, government publications, magazines,
books. Online coverage from 1955. http://www.ebscohost. newspapers, news digests, and industry newsletters and
com/public/historical-abstracts. periodicals, http://www.lexisnexis.com.

Humanities Full Text An index of English-language per­ Internet-only Sources

iodicals. Subjects include history, area studies, language
Annotated List o f Reference Websites (Library of Con­
and literature, philosophy, and religion. Online coverage:
gress) Covers many fields of the social sciences and hu­
index from February 1984, abstracts from March 1994. and manities. http://lcweb.loc.gov/rr/main/alcove9.
some full text from 1995.
Internet Scout Project Tracks useful websites and assesses
International Political Science Abstracts Provides cita­ their intellectual reliability. The Scout Report Archives is
tions and abstracts for a worldwide selection of journal and a searchable and browseable database of over seven years’
yearbook articles in the fields of international relations, pol­ worth of the Scout Report and subject-specific Scout Re­
itical science, and public administration. Online coverage: ports. It contains about 12,000 critical annotations of
1989 to present. carefully selected internet sites and mailing lists. Each
annotation analyses the site’s general content, attribution
Social Sciences Index Indexes social science periodicals
(authors, etc.), currency, availability, accessibility, and pres­
(scholarly journals).
entation. http://scout.wisc.edu.
H istorical Abstracts Notes and abstracts of historical arti­
Social Science Information Gateway A guide to a large
cles on the twentieth century. This journal began in 1955;
number of online scholarly sites, screened by experts.
in 1971, it spun off a second annual volume, Twentieth-
Includes breakdowns by anthropology, economics,
Century Abstracts. Search it by country name. geography, politics, sociology, and other fields. Orientated
primarily towards British scholarship, http://www.ariadne.
Intern ation al Political Science Abstracts Bimonthly ab­
stracts of political science journals and yearbook articles. ac.uk/issue2/sosig/.
422 A P P E N D IX

Weblogs/Blogs A type of interactive journal where writers Data from official statistical agencies
post and readers respond. They vary widely in quality of These data are published on a yearly or quarterly basis by
information and validity of sources. Prestigious journal­ national and international statistical agencies.
ists and public figures may have blogs, which may be more
credible than most. National Statistical Offices a n d D ata Archives such as
the Council for European Social Science Data Archives
Message boards, discussion lists, and chat rooms Some are (Cessda). http://www.cessda.org/.
useful and well researched, others are not.
Official Statistics in Europe: web guide to socio-econom ic
M ultim edia The internet has a multitude of multimedia surveys http://www.mzes.uni-mannheim.de/projekte/
resources, including online broadcasts and news, radio, mikrodaten/drafts/index.html.
and television broadcasts, interactive talks, public meet­
ings, images, audio files, and interactive websites. Statistical yearbooks published by independent agen­
cies and organizations, such as the statistical yearbooks
Internet Data and Archive Resources from the Encyclopaedia Britannica, which provide data
Data sets When you investigate a data set and its code­ on governments, elections, economics, and demography;
books, consider the following questions: Who collected the yearbooks from S1PRI (Stockholm International Peace
the data? What is the unit of analysis? How many Research Institute) provide, among other things, data on
cases are there in the data set? What was the sampling military expenditures and warfare; the World H andbook o f
method? For how many variables was data collected in Political and Social Indicators (Yale University).
each case? What (dependent or independent) variables
does it contain that are of particular interest to you? International statistical agencies such as the IMF, the
Can this data help to answer the question(s) you are IBRD (International Bank for Reconstruction and De­
addressing? velopment), and the OECD (Organisation for Economic
Co-operation and Development). The IMF provides inter­
Websites with links to lots of data sets national financial statistics, and trade statistics; the OECD
http://gamet.acns.fsu.edU/-phensel/data.html#index provides historical statistics, employment statistics, and
http://www.psr.keele.ac.uk/data.htm also economic surveys for individual countries; and the
http://www.paulhensel.org/data.html 1LO (International Labour Organization) is a source of la­
bour force statistics.
Political Resources on the Net Listings of political sites
available on the internet, sorted by country, with links Websites o f Policy-M aking Organizations There are many
to parties, organizations, governments, media, and more policy organizations, think tanks, private research insti­
from all around the world. http://www.politicalresources. tutes, and NGOs that produce relevant papers and data.
net.
Bank fo r International Settlements http://www.bis.org.
Intute Social Sciences A selective catalogue of thousands
of websites in the social scienccs, hosted in the United Bretton Woods http://www.brettonwoods.org.
Kingdom. Users can browse by topic and region or search
by keyword. Each entry has been reviewed and annotated. International Labour Organization (ILO) http://www.ilo.
The focus is on high-quality sites that provide information org/global/lang- -en/index.htm.
directly rather than just linking to other sites. A good re­
source for international social scienccs data. http://www. International M onetary Fund (IMF) http://www.imf.
intute.ac.uk/socialscienccs. org. International Financial Statistics Online (IFS Online)
is the principal statistical publication of the International
Internet Crossroads in Social Science Data http://www. Monetary Fund. Tables for each Fund member country in­
disc.wisc.edu/newcrossroads/indcx.asp. Offers hundreds clude data on the country's exchange rates, Fund position,
of annotated links to online data sources. Searchable by international liquidity, money and banking accounts, in­
keyword or browsable by category, the site includes links terest rates, prices, production, international transactions,
to government and non-government sites concerned government accounts, national accounts, and population.
with domestic and international economics and labour, Selected series are published in area and world tables. Data
health, education, geography, history, politics, sociology, may be downloaded as HTML, MS Excel, comma-delim-
and demography The site is maintained by the Data and itcd, or tab-delimited files. Some series begin with 1945.
Program Library Service at the University of Wisconsin. Updated monthly.
Madison
Organisation fo r Economic Co-operation and Devel­
Inter-Iniversity Consortium fo r Political and Social Re­ opment (OFCD) Includes many of the data sets avail­
search UCPSR. University of Michigan) A publiclv avail­ able from the OECD. This is primarily useful for wealthy,
able archive of datasets constructed by political scientists
developed countries, though comparison with other coun­
http:'.■'www.icpsrumich.edu ■'icpsrweb/ICPSR/. tries is sometimes included, http://www.occd.org.
 A P P EN D IX

Third World Network http://www.twnsidc.org.sg. multi govern mental institutions, etc. Includes also politi
cal parties. http://www.gksoftxom/govl/
United Nations (UN) http://www.un.org. United Nations
Common Database (UNCDB): a broad-based statistical The World Factbook (from the CIA) https://www.cia.gov/
resource, it draws data Irom a wide variety of UN and UN- library/publications/the-world-factbook.
related (e.g. FAO, ILO. WHO) organizations. UN Com­
mission on International Trade Law http://www.un.or. Ihe ESRC Data Archive Ihe l’SRC Data Archive is a re­
al/uncitral.UN Conlcrence on Trade and Development source centre whose holdings consist mainly of the data
(UNCTAD) http://www.unctad.org. from past surveys. It holds almost 4,000 data sets, includ­
ing those from many large and important government
World Bank http://www.worldbank.org. GDF Online produced surveys and censuses, as well as academic research
(Global Development Finance Online): Global Develop­ and historical materials, http://www.esds.ac.uk/.
ment Finance (GDF) is the World Bank's annual report on
external financing prospects for developing and transition
countries. It tracks the yearly movement of international II. Citing Sources
capital flows to developing countries, and analyses policy Cite While You Write (Cite While You Write') is a feature of
issues for developing countries. GDF provides statistical EndNote, a software package that facilitates proper citation.
data for 137 countries. Data may be displayed as charts or
maps and exported. World Development Indicators: World The Internet Public Library's Style and Writing Guide Sec­
Development Indicators includes nearly 800 statistical in­ tion Resources by subject, style, and writing guides avail­
dicators related to social and economic development. It is able at http://www.ipl.org/.
organized in six sections: World View, People, Environ­
M odem Language Association (MLA) has a brief overview
ment, Economy, States and Markets, and Global Links. The
of what MLA style is. The MLA Style Crib Sheet provides
tables cover 152 economies and 14 country groups—with
extensive descriptions of general MLA style notes, text, and
basic indicators for a further 55 economies. Data can be
block quotations, page formatting, Ml.A text citations, and
downloaded for further manipulation. http://data.world-
the MLA works cited, http://www.mla.org/.
bank.org/data-catalog/world-development-indicators.
The Online Writing Center (OWL) at Purdue An excellent
W orld Economic Forum/Davos http://www.wef.org.
overview, with information, example papers, and individ­
ual work; cited examples, http://owl.english.purdue.edu/
W orld T rade Organization (WTO) http://www.wto.org.
owl/resource/747/08.
Information on National Institutions
College o f DuPage Library, Citing Sources http://www.
Web Sites o f N ation al P arliam ents (provided by the cod.edu/library/research/Citenet.htm.
Inter-Parliamentary Union), http://www.ipu.org/english/
parlweb.htm. The Chicago M anual o f Style Online: Chicago-Style Cita­
tion Quick Guide http://www.chicagomanualofstyle.org/
Websites fo r N ational Banks (provided by Bank for Inter­ tools_citationguide.html.
national Settlements), http://www.bis.org/cbanks.htm.
Online! Citation Styles by A. Harnack and F.. Kleppinger
Governments on the W WW Comprehensive database of http://www.bedfordstmartins.com/online/citex.html.
governmental institutions on the World Wide Web: par­
liaments, ministries, offices, law courts, embassies, city Learning APA Style http://www.apastyle.org/learn.
councils, public broadcasting corporations, central banks.
Glossary

Abduction: a process of inference that involves Chi-square test: a statistical test used to test the
selecting from among competing explanations the one null hypothesis that there is no relationship between
that best explains a particular event or phenomenon, given two variables Often used in coniunction with
all the available evidence. Abduction produces a cross-tabs.
hypothesis that can then be affirmed through either
induction or deduction. Closed question: a question employed in a structured
interview or questionnaire in which respondents are asked
Access: refers to how the researcher enters the chosen to select their answers from a limited range of options
field site and starts to do fieldwork. Also called a 'fixed choice' question.

Acquiescence bias: the tendency of some respond­ Closed setting: research settings with restricted access
ents to automatically agree with every question they are or which require permission to access.
asked.
Code: a short alphanumeric term that refers to the
Aggregate data: describes data referring to aggregates category of a variable and often the location of a text
or collectivities rather than individuals. passage.

Association: two variables are associated when values Coding: marking a text segment with a code.
of one variable vary systematically with values of the other
variable. Comprehension problems: occurs when, for a
variety of reasons, the respondent does not fully
Bandwagon: Bandwagon effects are said to occur when understand the question that is being asked of them.
people say they did something (for example, vote for a
particular party), not because they necessarily did, but
Concept-Stretching: has to do with defining concepts
in very broad terms in order to ensure that competing
because they want to go along with the majority position.
definitions or interpretations of the concept overlap with
Bar chart: a way of graphically displaying nominal or each other.
ordinal data.
Confidence interval: range of values in which the true
Behaviouralism: the application of positivism and population value is thought to lie.
empiricism to the study and explanation of outcomes in
Confirmatory research: a type of research that
politics and international relations.
involves confirming’ an existing assumption or theory.
Bivariate analysis: analysis of the relationship
Confound: A variable that is associated with both the
between two variables.
probalde cause and outcome of a phenomenon, but has no
Case: refers to the unit of analysis. direct causal influence.

Case selection: refers to issues to do with how and why Constant: see intercept
a particular field site or case is chosen for study.
Construct validity: an approach used for assessing the
Case study: the intensive study of one case. validity of a measure which is based on the extent to which
the measure is associated with other theoretically relevant
Causality: a relationship between two variables in factors.
which changes in one variable bring about changes in
another. Constructivism: an approach which maintains that
reality does not exist as something independent of us, but
Central limit theorem: states that when samples are is socially, and actively, constructed.
large (above about 30), the sampling distribution will take
the shape of a normal distribution.
Content analysis: analysis of the content of a text in
order to uncover its meanings and intentions.
Central tendency: a measure of central tendency is a
Content validity: an approach used for assessing the
way of summarizing the central value in a frequency
validity of a measure which is based on the extent to which
distribution. See also mean, median, and mode.
the measure covers the full range of the concept, covering
Ceteris paribur. 'all else being equal'. each of its different aspects.
G LO SSA R Y

Continuous: a continuous variable is measured on a Deductive-nomological model: an explanatory

continuous scale to varying degrees of precision, where model, also known as the 'covering law model’, which
within the range of the variable any value is possible, such holds that explanations of individual events or actions are
as distance in metres or weight in kilos. derived from laws that express universal empirical
associations.
Control group: in experimental research, subjects in
the control group do not receive the intervention and so Dependent variable: refers to the outcome or
are used as a point of comparison in order to gauge the phenomenon we are trying to explain. Often denoted
effect of the intervention on the outcome of interest. as Y.

Correlation: the term refers to whether there is a linear Deviant case: a type of case study which is based on the
relationship between two variables. analysis of a case which is known to deviate from
established generalizations and does not fit the existing
Co-variance: a relationship between two variables such theory.
that one varies with the other.
Discrete: a discrete variable can only take on certain
Coverage error: occurs when the sampling frame from fixed values within its range and cannot be subdivided.
which the sample is drawn does not completely match the
population of interest. Dispersion: refers to the extent to which data values are
clustered together or spread out. See also standard
Covert research: when the researcher does not reveal deviation.
his or her identity to the informants.
Discourse analysis: a qualitative type of analysis that
Covering law: an explanatory model that holds that explores the ways in which discourses give legitimacy and
something is explained when it is shown to be a member meaning to social practices and institutions.
of a more general class of things: when the particular case
is deduced from a more general law or set of laws. See Double-barrelled question: a question that is really
deductive-nomological model. two questions in one.

Covert participation: see Covert research. Dummy variable: a dichotomous variable which can
take one of two values (0 or 1).
Crisp set: refers to dichotomous variables that are used
in Qualitative Comparative Analysis. In a crisp set Ecological fallacy: the assumption that relationships
analysis, a case is either in or out of a particular set. between variables at the aggregate level imply the same
relationships at the individual level.
Criterion validity: An approach used for assessing the
validity of a measure which is based on how well the new Ecological validity: the extent to which the
measure of the concept relates to existing measures of the conditions simulated in the laboratory reflect real-life
concept, or related concepts. conditions.

Critical realism: a position that holds that perception Empirical: refers to things that can be experienced
is a function of the human mind, and that we can therefore through the five senses (of seeing, hearing, touching, etc.).
only acquire knowledge of the external world by critically Empirical research means research based on finding
reflecting on perception. things out by experience or the sense organs: hearing,
seeing new primary data (newspaper items or peoples
Cross-sectional designs: involve making observations words in interviews or questionnaires).
of a sample, or cross-section, of a population or phenom­
enon at a single point in time and analyzing it carefully. Empiricism: the view that knowledge of the world is
limited to what can be observed. Concepts apply to or
Cross-tabulation: a table that can be used to depict the
derive from an experience. Empiricism believes that all
relationship between two variables.
knowledge originates in empirical observation.
Crucial case: a type of case study which is based on the
Epistemology: a branch of philosophy which studies
analysis of a case which is thought to be crucial for the
the nature, sources, and limits of knowledge.
confirmation or disconfirmation of a theory.
Equivalence of meaning: has to do with whether the
Data analysis: the processing, interpretation, and concept under investigation means the same thing to
analysis ol findings.
people in different contexts.
Data collection: how information is gathered.
Experimental group: subjects in the experimental
Deduction: reasoning trom a general or logical proposi­ group do receive the intervention.
tion to ,1 specific or particular outcome.
Explanatory variable: See independent variable.
 glossary

Exploratory research: a type of research typically

Histogram: a way of graphically displaying data lor
performed when the researcher does not have a hypothesis
interval or continuous data.
or does not have specific assumptions concerning the
problem or question. Historicism: the idea that a society ,s m ille d along a
pre determined route by historical laws which cannot be
External validity: the extent to which findings can be resisted.
generalized to other similar situations.
Holism: the claim that the whole of something is distinct
Falsifiability: a concept in philosophy of science,
from and not directly explicable in terms of its parts.
popularized by Karl Popper, that for a proposition to be
Social facts have to have social causes that are irreducible
scientific, it must be logically possible to show, through to facts about individuals.
observation or experiment, that it is false.
Hypothesis: a proposition put forward for empirical
Face validity: an approach used for assessing the testing.
validity of a measure which is based on whether or not the
measure intuitively seems like a good measure of the Hypothetico-deductive method: this is an
concept. application of two operations: the formation of hypotheses
and the deduction of consequences from them in order to
Faulty presupposition: see presuming questions. arrive at beliefs which—although they are hypothetical
Field: the place where research is carried out. are well supported, through the way their deductive
consequenccs fit in with our experiences and with our
Field experiments: experiments which take place in other well-supported beliefs (Follcsdal 1994: 234).
real-world settings.
Independent variable: refers to a factor we think may
Fieldnotes: how the ethnographer records the data that influence or cause the dependent variable. Often denoted
they observe in the field. as X.

Filter: a filter question is used in surveys to filter out Indicator: empirical measure of an abstract concept.
respondents for whom more detailed questions may not
Individualism: the claim that the basic units of society
apply and who therefore should not be asked.
are individuals. Social phenomena are (he combined
FOCUS group: a form of group interview organized to results of individual actions.
address a specific phenomenon of interest, a specific
Induction*, reasoning from particular facts or
theme or topic, involving a group of people selected
observations to a general conclusion.
because they are related to, have experience of, or beliefs
about the topic or theme in question. Interaction effect: when the impact of one
independent variable on the dependent variable varies
Foreshadowed problems: refers to the questions or according to the level of another independent variable.
issues that the researcher intends to explore during their
fieldwork. Intercept: the point at which the line of best fit crosses
the y-axis.
Frequency distribution: displays the number of
times each value of a variable occurs in a given sample. Inference: the process of drawing logical conclusions
on the basis of premises assumed to be true.
Fuzzy set: refers to multi-value variables that are used
in qualitative comparative analysis. In a fuzzy set analysis, Informants: refers to the people who are being studied.
a case can be either fully in or fully out of a particular set, Although this sounds a bit conspiratorial, the term implies
or somewhere in between. something more than just respondent, since whether
unwittingly or not informants provide information about
Gatekeeper: someone who controls access to closed the topic under study.
settings.
Informed consent: the principle that researchers must
Going native: when the researcher loses objective obtain consent to carry out research from the people they
distance from the people that they are studying. are studying.

Grand theory: an attempt to construct a total Intentionalism: the methodological precept that
theoretical system covering all aspects of social life, rather explanation of social phenomena must give an account of
than an explanation of particular instances, societies, or the intentional states—the aims, beliefs, attitudes, and
phenomena. expectations—that motivate individual action.

Grounded theory: an inductive research strategy in Intercoder reliability: the degree of coding
which theory is produced through, and grounded in, data. consistency between two or more coders.
G LO SSAR Y

Internal validity: has to do with whether or not we are Line graph: a way of graphically displaying data for
confident that a causal relationship really exists between repeated measures of the same variable over time.
two variables.
Line of best fit: a straight line representing the best
Interpretivism: maintains that knowledge of the social linear relationship between two variables. See also
world cannot be gained by testing hypotheses and regression line.
developing science'; and that interpretation should be the
listserv: a list management tool consisting of a set of
basis of inquiry.
email addresses for a group in which the sender can send
Interquartile range: refers to the range of values of a one email and it will reach a variety of people.
variable within the middle 50% of the distribution.
Logical positivism: a movement that sought to apply
Interval: a variable where the values of the variable can logic and mathematics to the discovery of empirically veri­
be rank ordered, and the differences between two values are fiable causal laws.
important and can be quantified and directly compared.
Log odds ratios: a statistical transformation of the
Intervening variable: a variable that helps to explain odds ratios.
or account for the relationship between two other variables.
Longitudinal designs: explore changes or discern
Intervention: intervention in the data-gathering trends over time.
process occurs when the researcher manipulates or alters a
variable of causal interest (such as what the subject is Manifest content: meanings that reside on the surface
exposed to). of communication and are therefore easily observable.

Interviewer effects: are said to happen in survey Mean: a measure of central tendency that is computed
research when respondents answer identical questions in by adding the values for all the cases and dividing by the
different ways depending upon the characteristics or number of cases.
identity of the person who asked them.
Measurement error: occurs when, for a variety of
Interviews: a method of data collection in which a reasons, a respondent s answer to a question does not
researcher asks questions of participants. represent their true answer.

Interview schedule: a list of questions to be asked of Median: a measure of central tendency that refers to the
all interviewees. middle value in a rank-ordered set of cases.

Item non-response: occurs in survey research when Methodological holism: the claim that properties of
the respondent docs not provide a valid answer to a a system as a whole cannot be deduced by the properties
particular question. of its components alone. The system as a whole determines
how the parts behave.
Key informant: an informant that the ethnographer
comes to rely upon as a valuable source of information Methodological individualism: the methodologi­
and knowledge.
cal precept that explanations of social phenomena—
Laboratory experiments: experiments conducted in classes, power, nations, or other social phenomena—must
a facility that provides controlled conditions. be reducible to the characteristics of individuals.

Large-N: study of a large number of cases through Methodology: the principles and procedures of
statistical analysis. inquiry used by researchers in a particular discipline.

Latent content: meanings that do not reside on the Missing data: where a variable does not have a valid
surface of communication and are therefore not easily value the data is said to be missing.
observable.
Mode: a measure of central tendency that refers to the
Law: a statement of relationship (e.g. ‘A leads to B'), most common value of a variable.
which is acccpted as having been universally verified by
observation. Model specification: refers to the way in which a
hypothesis is tested and what variables arc included in the
Leading question: a question that encourages a
spec,Ik response.
Most Different Systems Design: method of
Level of measurement: refers to how a variable is comparison based on analysing cases which arc very
measured, and how the dillerent values of the variable
different from each other with respect to theoretically
relate to each other. See also nominal, ordinal, interval.
relevant independent variables.
 G LO SSA R Y 429

Most Similar Systems Design: method of Omitted variable bias: occurs when the causal
comparison based on analysing cases which are very significance ot variables is exaggerated because important
similar to each other with respect to theoretically relevant factors have been omitted from the analysis and not taken
independent variables.

Multicolinearity: occurs in multiple regression when Ontology: the study of what exists and the nature ot
two independent variables are highly correlated with each what exists.
other, making it difficult to disentangle the independent
impact of either variable. Open-ended question: a question where respondents
are free to give their own answer and no response
Multivariate analysis: analysis of the relationship alternatives are provided.
between three or more variables.
Open question: a question employed in a structured
N: denotes the number of cases in the analysis. Also interview or questionnaire which allows respondents to
referred to as sample size. respond by expressing themselves in an open-ended,
detailed manner.
Natural experiments: experiments in conditions
which occur naturally, in so far as the researcher is not Open setting: research settings with public access
active in the data-gathering process. which do not require permission to enter.

Naturalism: the view that the social and behavioural Operationalization: process of turning abstract ideas
sciences should have the same structure and logical into empirical indicators.
characteristics as the natural sciences; that the primary
goal of the social sciences—like that of the natural Ordinal: a variable where the values or categories of the
variable can be ranked, but the differences between the
sciences—is to explain and predict social phenomena by
categories cannot be quantified or directly compared.
means of laws; and that only that which exists can, at least
in principle, be investigated scientifically (contrary to Outlier: refers to a value that is markedly different to the
‘scientific realism’—see below). other values in the distribution.
Nominal: a variable where the values or categories of Overt research: when the researcher reveals his or her
the variable cannot be ranked or the differences between identity to the informants.
the categories quantified.
Partial regression coefficients: a measure of the
Non-probability sampling: refers to sampling relationship between two variables controlling for the
methods which do not use random selection. effect of other variables.

Non-response bias: is introduced into a sample when Participant observation: type of fieldwork in which
people who do not complete the survey are systematically the researcher becomes a regular participant in the
different in some respect from those people who do day-to-day activities of the people whom they are
answer. studying.

Normal distribution: is a probability distribution. The Population: refers to the collection of units from which
graph of the distribution is symmetrical and bell shaped. the sample is drawn and to which findings can be
Sometimes known as the bell curve. generalized.
Normative theory: theory concerned with questions Positivism: the view that advocates pursuing knowledge
about what is right and wrong, desirable or undesirable, of the social world through the discovery of universal laws
just or unjust in society. and the falsification of theories. Positivists treat the social
world as if it were the world of natural phenomena. They
Null hypothesis: that in the population there is no
assume that all that we know of the world is given to us in
association between two variables.
experience. Experience concerns events and happenings.
Objective: objectivity means striving as far as possible Positivists systematically investigate these events and
to reduce or eliminate bias in the conduct of research. happenings so as to reveal their underlying regularities.

Odds: refers to the likelihood of one outcome occurring Presuming questions: is a survey question which
compared to the likelihood of an alternative outcome assumes something that is not necessarily true.
occurring.
Process tracing: a method of within-case analysis to
Odds ratios: compare the odds of an event occurring in evaluate causal processes.
one group compared with another. An odds ratio of 1
indicates that the odds of a particular outcome are equal in
Proposition: a statement which links two or more
variables.
both groups.
G LO SSARY

Purposive sampling: is a non-probability sampling Replicability: the replicability of a study refers to

method which involves investigators using their own expert’ whether or not it can be carried out in more or less the
judgement to select respondents whom they consider to be same way with a fresh group of subjects in another sample
typical or representative of the population of interest. at a different point in time to check the validity and
reliability of research.
Qualitative: research that tends to be based on the
discursive analysis of more loosely coded information for Research design: a logical structure of inquiry that
just a few cases. specifies the sort of test or evidence that will convincingly
confirm or disconfirm a hypothesis.
Quantitative: research that tends to be based on the
statistical analysis of carefully coded information for many Reliability: the repeatability' or consistency' of
cases or observations. research findings.

Questionnaire: pre designed lists of closed questions Response rate: the response rate of a survey refers to
designed to collect data from a large sample of respond­ the percentage of people who completed the survey out of
ents and to be completed by respondents themselves. the total number of people selected for the survey.

Questionnaire design: the way in which the survey Response variable: see dependent variable.
questions are constructed and put together.
Sample: refers to the cases which have been selected for
Question order effects: occurs when the order in analysis. A sample is a smaller subset of the population of
which questions are asked influences the answers that are interest. The way in which a sample is drawn affects the
given to them. extent to which the sample can be used to make valid
inferences about the population of interest.
Question wording effects: occurs in survey research
when slight variations in the way in which questions are Sampling distribution: refers to the distribution of
worded can have a significant impact on how people possible values for a statistic that we would expect to get if
respond. we drew repeated samples from a population.

Quota sampling: is a non-probability sampling Sampling error: the extent to which the sample
method which involves investigators selecting a certain estimate differs from the true population value.
number or quota of respondents to interview, according to
the personal characteristics of the respondent, such as Sampling frame: the list of units in the population
their age group, sex, and income group. from which the sample will be drawn.

Random assignment: the experimental method for Sampling variation: the extent to which the value of a
deciding which subject goes in which group. variable varies in samples taken from the same population.

Random digit dialling: is a sampling method for Scientific realism: the view that scientific research is
selecting people to take part in telephone surveys by based, not only on evidence that is visible in the world,
generating telephone numbers at random. but on unobservable facts, objects, or properties that
exist independently of our direct knowledge of them, and
Random sample: sampling method in which all units that the goal of scientific inquiry is to describe and
in the sampling frame have a known non-zero probability explain both observable and unobservable aspects of the
of being selected.

Range: the range of scores is the value of the smallest Selection bias: occurs when our findings arc not very
score subtracted from the largest score. reliable and may owe more to the cases wc have selected
Reactivity: has to do with whether subjects are conscious than any real relationship between the variables wc arc
examining.
ot being studied and alter iheir behaviour accordingly.

Recall problems: refers to error in survey research to Simple random sampling: is a probability sampling
do with memory failure, where respondents cannot method where each member of the population has the
remember whether or not they have done something or same chance of being selected.
when they did it.
Single-N: in-depth study of a single case.
Regression: a statistical technique for describing the
relationship In-tween two or more variables where the Slope coefficient: derived from the line of best fit.
dependent variable is interval level Describes the amount the Y variable increases for a
one-unit increase in the X variable. Also called the b
Regression coefficient: see slope coefficient. coefficient or the regression coefficient.
Regression line: see line of best fit.
Small-N: in-depth study of a small number ol cases.
 G LO SSA R Y

Snowball sampling: is a non probability sampling Theories of the middle-range: theories that, while
method where- respondents who have already taken part in they may involve abstractions, are constructed with
the survey put lorward their Iriends or acquaintances to reference to phenomena that are observable
also take part in the study.
Theory: a proposition which has been elaborated and/or
Social desirability bias: occurs when a respondent withstood repeated testing
provides an answer to a question which they think will
present them in a good light, rather than one which Thick description: combines description and
reflects their true feelings. interpretation, and attempts to describe the meaning of an
event and the motivations of the people involved.
Sparse categories: occurs when a table cell only
contains a few cases. Can produce unstable results. Treatment: see intervention.

Speech act theory: an approach to the explanation of Truth table: is used in QCA to display all the logical
language premised on the idea that language is a medium possible combinations of conditions and the outcome
orientated towards action and function. associated with each combination.

Spurious association: occurs when an apparent Ttest: a statistical test used to test the null hypothesis
association between two variables is caused by an that there is no relationship between two variables.
unmeasured third variable. Spurious associations are not Type I error: is said to occur when we reject a null
causal. hypothesis that is actually true.
Standard deviation: a measure of dispersion of data Type II error: is said to occur when we fail to reject a
points around the mean. null hypothesis that is actually false.
Standard error: the standard deviation of the sampling Unit of analysis: refers to the entity that you are
distribution. studying (such as individuals or countries).
Structuralism: an approach that privileges the role of Univariate statistics: refer to statistical measures used
structures over that of agency in explanations of social to summarize some characteristic of a single variable.
phenomena.
Unstructured interviews: these use open, and
Structurationism: the view that agents and structures perhaps lengthier and more complex questions that might
are not ontologically distinct entities; they co-exist and vary in the way and order in which they are asked.
co-determine each other.
Validity: whether a question measures the concept that
Structured interviews: these consist of a standardized it is supposed to measure.
set of closed and shorter or simpler questions that are
asked in a standardized manner and sequence. Variable: a property of an object that can take on
different values.
Subject: refers to the participant in the study.
Variance: refers to the average amount of variability
Surveys: a method of data collection whose goal is to (spread) of a set of data.
gain specific information about either a specific group or a
representative sample of a particular group, and that Verifiability: the principle that a proposition can be
includes at least one question which is either open-ended true either by definition, or by empirical verification.
or close-ended and employs an oral or written method for Volunteer sample: is a self-selecting sample.
asking these questions.
Weighting the data: is a technique that is often used
Tails: the tails of a distribution refer to those values in survey research, which corrects for non-response or
which lie at the extreme ends of the distribution. unequal selection probabilities to try to make the sample
Telescoping: occurs in survey research when more representative of the population.
respondents think that things that happened a long time
ago happened more recently than they really did.
 Index

A behaviouralist research 40 categorical data, cross tabulation

abduction, and scientific realism 38 behaviour 382-8
acquiescence bias 241 Heisenberg effect 58 causal-structural studies 84
agent-structure problem 92-3 Rosenthal effect 57 causality
Giddenson 93-5 bibliographies 107-9 deterministic 147
relationship between agents and Bishop, on:, context of study 85 probabalistic 147
structures 93-5 bivariate analysis 366-90 causation
agential explanations 91-5 association and causality 369 causal mechanism, defined
Almond, on comparative research categorical data, cross-tabulation 37.49
203 382-8 Hume on 30
annotation aids 325 data considerations 370 positivist concept 30
answering research questions ethnic diversity and civil war 387 cause and effect 30
129-63 interval data 370 central tendency 349-51
concepts 148-53 Pearson's correlation 380 chi square 384, 387 (tab.)
general requirements 132-44, 160 significance testing with interval civic skills 204-5
(tab.) variables 374-5 civil conflict 139-41
grande theory vs theories of statistical significance for cross­ civil wars
middle-range 137 tabs 384 and ethnic diversity 387, 388
operational definitions 153 strength of the relationship 377-9 (tab.)
process vs presentation 131 test statistic 376 population pressures 84
specifying 144-56 using scatter plots 370-3 class and vote, observed frequencies
three steps 130 using tables 382-3 386 (tab.)
types of answer 133-4 Blair, on:, doctrine of international clientelism, Benin, Wantchekon on
types of questions 133-5 community 317 193-4
see also questions Boolean logical comparison 218-20 closed questions 238, 240
appendix 419-22 British adults (2005), left-right coding 322-6
Archer, on:, agent-structure ideology self-placement 345 validity and reliability 328
problem, morphogenetic (tab.), 347(fig.) cohort studies 171-2
approach 93-4 ‘brute data’ 41 Coleman’s bathtub 88-90
archival sources, as data 328-31 collective action, Olson on 88
arguments and conflict 1-2 commitment problem, ethnic
articulation 316 conflict 44-5
Arts & Humanities Citation Index calculations Communist and American news
(A&HC1) 108 chi square 387 (tab.) items (1980), wording
association interquartile ranges 353 (tab.) effects 238
hypothetical example 385 (tab.) standard deviation 354 (tab.) Comparative Manifestos Project
patterns 366-90 capitalism (CMP) 325
spurious 211 and dependency theory 66-8 comparative research 16,202-29
variables, correlation and co­ and development theory 66 Almond on 203
variance 146-8 global (transnational) economy 86 case studies 205-8
association and causality, bivariate case selection, experimental research data sources 224-5
analysis 369 190 descriptive function 204-5
assumptions, and empiricism 55 case studies hypotheses 204
comparative research 205-8 large-N comparison 222-5
democracy and development 152 principles 203-5
B designs 172-3 small-N comparison 208-9
band-wagon effect 236 generation of new theory 208 Computer Assisted Qualitative
bar charts 346 internal/external validity 205 Data Analysis Software
outliers or deviant cases 207 (CAQDAS) 324
barrios 295
theory-confirming vs computer-assisted personal
behavioural revolution 2 7-8
theory-infirming 207 interviewing (CAPI) 248
behaviouralism 27-8
turnout 408-11 concept-stretching 225
vs behaviourism 52
434 IN D E X

concepts 148-53 discourse/content analysis 177-8 descriptive questions 116

anomaly concept 62-3 ethnographic research 177 research design 133-4,173
conceptualizing democracy 152-3 focus groups 176-7 deterministic causality 147
formation 148-53 from existing archives 178 deterrence theory 58
positivist concept 30 interviewing 176 development
power, definition 150-1 large-N comparison 222-5 case studies 152
survey question validity 232-3 multiple sources (triangulation) economic 371
conceptual goodness 149 178 ‘development discourse’, Escobar
conceptual stretching 152 quantitative data 224-5 on 312
conceptual travelling 152 questionnaires and surveys 175-6 development theory 64-9
confidence interval 356, 359-60, 362 research design 175-8 and capitalism 66
confirmation, hypothetico-deductive data reduction, interviews 279-80, democracy 221
model 35 282 Geddeson 65,67, 72
confounding variables and spurious data sources modernization 66
relationships 147,394, 395 archival, historical writing, and QCA 221
(fig) documents 328-31 deviant cases 207
consent 179-80 ‘brute data’ 41 dictionary-based content analysis
constructivism 45-6,311 comparative research 224-5 324
content analysis 177-8, 318-28 data weighting 244-5 Dion, on necessary and sufficient
data collection 318 databases 108-9 cause 216
dictionary-based 324 JSTOR database 108 discourse analysis 309, 310-17
latent content 319 death penalty, attitudes, by education articulation 316
qualitative and quantitative 310, 384 (tab.) goal 313-14
319-23,326-7 deception 180 interpellation 316
steps 1—4 320-3 deduction 31 text and context 314-15ffif
validity and reliability 328 and social phenomena 34 validity and reliability 317
vs discourse analysis 331 -2 (tab.) testing hypotheses vs content analysis 331-2 (tab.)
context of study. Bishop on 85 143(faJ discourse and social change,
covert participation 179-80 deductive-nomological model, of Fairclough on 316-17
crisp set QCA 218-19 explanation 34-5 discursive power 313
critical discourse analysis 312-13 deletion, listwise 343 ‘discursive practices’, Foucalt
critical realism 39 democracy on 312
Croatia Bollen on 152 dispersion, measures 352-3
independence declaration 44-5 case studies, using OLS regression distribution
war 1991-2 45 398-401 normal 348, 356
cross-sectional designs 170-2 Dahl on 152 using statistics 349-64
cross-sectional research design 170-2 defining 225 diversity, as confounding variable
cross-tabs development, OLS regression 381 395 (fig.)
regression 369 (tab.) documents, as data 328-31
statistical significance 384 development theory 221 double-barrelled questions
cross-tabulation 382-8 and GDP 371, 374 234
questions on 108 duck vs rabbit (Ulus.) 59
democracy levels Durkhcim, on social facts 82-3
D OLS regression 401 (tab.), 404 Duverger's law 35
Dahl, on democracy 152 (tab.)
data parameter estimates 404 (tab.)
interval data 370 unstandardized beta E
longitudinal data 171-2 coefficients 401 (tab.) ecological fallacy 90-1
missing 343 dcmocralic stability, India compared ecological validity 192
dala analysis 14 with Pakistan 212 economic development 371
categorical data, cross-tabulation dependency theory 66-8
level of democracy 372 (fig.)
382-8 dependent variable 144-6, 216, 367
coding 280- 1 electoral behaviour, party attachmenl
necessary and sufficient cause 216 206
interviews 278-82 selection on 215-16 Klster, on Marxism 88
presenting and reporting data deprivation hypothesis 139-40, 142 email, interviews 270-1
descriptive inference 355
data collection 14, 18, empirical theory, defined 54
descriptive and inferential statistics
empirical vs normative research 4-5,
339
contení analysis 31H 54-5
 IN D EX 435

basic principles of experimental tocus groups 255-6, 276-8

and assumptions 55 design 188-90, 189 (tab.) interviews 255-6, 276-8
classical tend of positivism 29 organizing 277-8
and logic, basis of truth claims control group 189 formulating questions 266-8
31-5 controlled interventions 193 open/dosed 260-2
epistemology 26, 49 (lab.) designs 168-9 l:oucalt, on:
holist 80 ecological validity 192 'discursive practices' 3 12
individualist 80 effects of ballot design and voting post-structuralism 312
individualistic research tradition instructions 197 Irankel, on Heisenberg effect 58
87 (tab.) field experiments 192-6 Funtowicz and Ravctz,
equivalence of meaning 225 five steps 189 on'postnormal science' 6-9
influence of newspapers 194 fuzzy set QCA 218-19
fictional questions 120 internal and external validity
of logic 120-1 levels 195-6
margin of error 358 issues 197-8 G
measurement error 17 laboratory experiments 190-2 gatekeepers 296
of reasoning 120-1 media effects on public opinion GDP, and democracy 371, 374
sampling error 17, 246 191 Geddes, on:
sources, in surveys 232 (fig.) natural experiments 196-7 development theory 65,
tautology 121 negative campaigning, and 67,72
see also fallacies likelihood of voting 191 large-N vs small-N quantitative
Escobar, on ‘development discourse' (tab.) analysis 223
312 principle of random allocation MDSD215
ethical research 178-80 196-7 generation of new theory, case
covert participation 179-80 reactivity 192 studies 208
deception 180 teachers' expectations, and Giddens, on agent-structure
experimental research 193 student performance 195 problem 93-5
harm 180, 193 experimenter bias 57-9 global capitalism, transnational
informed consent 179-80 expert opinion, interviews 274-5 economy 86
privacy 180 explanation, deductive-nomological globalization, and nation state 115
Social Research Association model 34 glossary 425-31
(SRA) Ethical Guidelines explanatory questions 116, 117, graphs and figures, quantitative
(2003)180 133, 173 analysis 346-9
voluntary participation 179 groupness 141
ethics see normative questions
ethnic cleavage, Posner on 210-12.
210 (tab.) face-to-face interviews 248, 254-5, H
ethnic conflict, Yugoslavia 44-8 269-70 Hall, on articulation 316
analysis 44-8 facts vs values 30-1,53-77 hegemony, economic costs 37
commitment problem 44-5 Martin on 57-8 Heisenberg effect 58
polarization 45-6 Nagel on 56-7 Frankel on 58
two approaches to understanding social practice 60-74 Hempel
47 tenet of positivism 30-1, 53-4 deductive-nomological model of
ethnic diversity and civil war 387 Weber on 55-6 explanation 34-5
ethnographic research, data Fairclough, on: hypothetico-deductive model of
collection 177 discourse and social change confirmation 35
ethnography 287-308 316-17 hermeneutics 40
choice/access of research site 4 ‘moments’ 316-17 Higgott, on Marxist theory 72
293-305 fallacies 90-1, 120-1 histograms 346
definitions 288 false uniqueness and false historical writing, as data 328-31
participant observation 17-18, universalism 203-4 Hobsbawm, on transnational
287-308 falsifiability. Popper on 33, 59-60 economy 86
principles 288-93 faulty presuppositions 234-5 Hodgson, on methodological
evidence fictional questions 120 individualism 82
external vs internal 43 ,4 7 -8 field experiments 192-6 holism
negative 167 fieldnotes 303-5 methodological 82-6
expectations, fulfilment (Rosenthal fieldwork, participant observation Popper on 86
effect) 57 292-8 holist epistemology 80
experimental research 187-201 fit see measures of fit holist ontology 91
IN D E X

honey-pot hypothesis 140,142 inferential statistics 340 J

House of Commons research 299 informants 299-301 jargon 235-6
Hume, on: informed consent 179-80 John and Brannan, on increasing
causation 30 intentionalism91 turnout 193
induction 32 interaction effects 396 JSTOR database 108
hypotheses 3 1 -3 ,1 3 0 -1 ,1 3 3 -5 and turnout 413-15
comparative research 204 intercept 373
components 144-8 intermediate-N comparison, K
definition 130-1 qualitative comparative
Kahl, on:
deprivation 139-40,142 analysis 218-22
civil conflict 139-41
existing studies providing basis internal/external validity, case
hypotheses 139-42
139-44 studies 205
key informants 299
falsify vs verify 33 international community, Blair on
key words in context (KWIC)
foreshadowed problems 291 317
concordance 324
generation of conflict 142 (fig.) international relations, vs politics 4
knowledge, forms of 9 -1 1 ,2 5 -5 2
honey-pot 140,142 interpellation, discourse analysis
Kuhn on science as a social
induction and deduction 143 (fig.) 316
institution 60-74
Kahl’s 139-42 interpretivism
null 368-9, 385 positivism and scientific realism
origins 138-44 compared 49 (tab.) L
rationale 141 shared methodological
laboratory experiments 190-2
resource curse 140-1,142 conventions 43
Lakatos, and scientific research
state failure 139-40,142,153 vs positivism 39-43
programmes 6 9-72
testing 143,222 interquartile ranges 353 (tab.)
Lakatos and Kuhn compared 72-4
variables 144-8 interval data 370
large-N comparison, quantitative
hypothetical truth table, Ragin on interval questions 262
analysis 222-4
219 (tab.), 220 interval variables 341, 374-5
advantages 222
hypothetico-deduclive model of intervening variable 144-6, 396 (fig.)
case selection 222-3
confirmation 35,43 interventions, experimental research
data sources 224-5
193
vs small-N quantitative analysis,
interviews 247-50, 253-86
Geddes on 223
analysing 278-82
Latin America, 'New Left’ 217
ideologies 33 choice of 258-60
imputation 343 law/law-likc generalization
computer-assisted personal
positivism 29
independent variable 189,215-16, interviewing (CAPI) 248
367 prediction of events 34
data analysis 278-82
testing 34
necessary and sufficient cause design 260-6
216 laws 29
elite 272-4
India email 270-1 of expanding state expenditure 37
compared with Pakistan, expert 274-5 left-right ideology self-placement,
democratic stability 212 British adults (2005) 345
face-to-face 248, 254-5,269-70
electoral quotas 197 focus groups 255-6, 276-8 (tab.). 347 (fig.)
individualism formulating questions 266-8 Lenin, on causes of World War I
characteristics, and group guide 265 (tab.) 154-6. 155-6 (fig.)
characteristics 91 interviewer effects 248-9 linear relationship 375 (fig.)
doctrine of reducibility 81-2 online 255, 270 Linguistic Inquiry and Word Count
methodological 80-2 postal and internet surveys 248 (LIWC) dictionary 324
ontological 78-9 question order effects 268 listwise deletion 343
vs collectivism. Popper on 86-8 questionaire administration literature review 121-6
individualist epistemology 80 248-50 reading 123-4
individualistic fallacy 90-1 random digit dialling 249 summarizing/executing 124-6
individualistic research tradition 87 recording 269 literature survey 103, 107
induction 29-30, 31 schedule 260 keywords 109
Hume on 32 screening questions 267 strategies 109-11
rejecting. Popper on 33 skills 269 survey articles 109
testing hypotheses 143 (fig.) structured/unstructured 256-8 logical positivism 31
inlcrence surveys 247-50, 274-6 Popper on 32
descriptive 355 telephone 255 logistic regression 405-15
and scientific realism 38 video link and web 271-2 longitudinal data 171-2
 IN D EX 437

M using logistic regression 405-15 open-ended questions 238. 240

macro-micro phenomena 89 case studies, turnout 408-11 open/dosed (public/non-public)
Malawi, ethnic cleavage 211 (lab.) interaction effects 4 1 3 -15 setting 296
manual and computer assisted interpreting the magnitude of ordinary least squares (OLS)
coding 323-6 coefficients 411-12 regression 372, 381.
Marxist theory measures of fit 411 398-401
Hlster on 88 using Ol.S regression 397-404 outliers, case studies 207
Higgott on 72 interpreting the magnitude of
mean 347 the coefficients 403
difference of 362-3 measures of fit 402
measurement, quantitative analysis results: presentation 404 paradigms 61-4
340 Kuhnian 64-5
measurement error 17 paradigm change 62-4
measures of fit N in research 65
using logistic regression 4 11 Nagel, on distinction between facts participant observation 17-18.
using Ol.S regression 402 and values 56-7 287-308
media effects on public opinion 191 natural experiments 196-7 definitions 288
median 349 naturalism 28 fieldwork 292-3
methodological holism 82-6 necessary and sufficient cause, Dion methodology 292-3
‘emergent properties’ 83 on 216 participation
methodological individualism negative campaigning, and likelihood covert 179-80
80-2 of voting 191 (tab.) voluntary 179
Hodgson on 82 Neo-Realist theory 92 participation, political 224-5,
and holism 78-98 new industrializing countries (NICs) 408-10
Watkins on 81,83 223 electoral behaviour 206
methodology 26 'New Left’, Latin America 217 see also turnout
individualistic research tradition newspapers 194 patterns of association 366-90
87 experimental research 194 see also bivariate analysis:
research process 14-20 non-governmental organizations multivariate analysis
misreporting, deliberate 236-7 (NGOs) 170-1 Pearson's correlation 378, 380, 399
missing data 343 non-probability sample design 245 (tab.)
mode 349 normal distribution 348, 356 (fig.) political elite, interviews 272-4
model sum of squares 379 normative hypothesis 159 politics, vs international relations 4
moderator variables, and interaction normative political analysis 174 Popkin, on social revolution 92
effects 396 normative questions 116, 118,
morphogenetic approach, agent- 156-60, 174 falsifiability 33, 59-60
structure problem 93-4 answers and methods 158-60 holism 86
Most Different Systems Design empirical vs normative research logical positivism 32
(MDSD) 212-13, 215-17 157-8 population, defining 222
Most Similar Systems Design normative research design 134,174 population pressures, civil wars 84
(MSSD) 209-12,215-17 normative research process 116, 118 positivism
vs MDSD 217 normative theory 138 challenges to 35-48
MPs defined 54-5 analysis of ethnic conflict 44-8
expenses scandal 360 normative vs empirical research 4-5, interpretivism 39-43
public satisfaction/dissatisfaction 54-5 scientific realism 35-9
361 (tab.) null hypothesis 368-9, 385 classical tenets 28-31
women in Parliament 214 empiricism 29
multivariate analysis 391 -418 explanation and prediction by
confounding variables and o laws 29
spurious relationships 394 observation 290 facts vs values 30-1
intervening variable 144-6 recording 303-5 naturalism 28
and indirect causality 395 odds ratios 406-8 induction 29-30
moderator variables and OLS regression 372, 381, 398-401 interpretivism and scientific
interaction effects 396 Olson, on collective action 88 realism compared 49 (tab.)
presenting and reporting data online interviews 255,270 invention 28
416-17 ontological individualism 78-9,91 law/law-like generalization 29
specifying different types of ontology 26, 49 (tab.) logical 31
relationship 394-7 individualistic research tradition vs interpretivism 6 -7 ,3 9 -43
statistical control 392-3 87 vs scientific realism 35-9,49 (tab.)
IN D E X

design 232 recall problems (memory failure) 236

Posner, on ethnic cleavages 210-12,
and surveys 175-6 recontextualization 317
210 (tab.)
surveys 232 reflexivity 332
Post Office Address file 242-3
post-structuralism 312 questions, research design 173-4 regression 369
descriptive 133-4,173 logistic 405-15
Foucalt on 312
postal interviews 248 explanatory 133,173 OLS 398-401
‘postnormal science', Funtowicz and normative 134,174 regression coefficient 374
Ravetz 6-9 predictive 133,173-4 regression line 371 -2
power prescriptive 134,174 fit 378 (fig.)
concept 150-1 requirements of an answer 160 regression sum of squares 379
concept formation 150 (tab.) reliability 224
predictions surveys 234-46 coding 328
democracy levels, OLS regression types 119 (tab.) content analysis 328
401 (tab.). 404 (tab.) wording effects 238 discourse analysis 317
turnout, logistic regression 410 questions, research process 101-28, religions 33
(tab.). 415 (tab.) 116 (tab.). 129-63 replicability, defined 302
predictive questions 116,118 calls for proposals, utilizing 112 research components, outline 132
research design 133, 173-4 descriptive 116 research design 9 ,19,164-83
prescriptive questions 116,118 errors of reasoning 120-1 attributes 166-7
research design 134, 174 explanatory 116,117 basic principles 166
presuming questions 235 first step: formulation 102-4, case study designs 172-3
primary and secondary sources 329 111-21 comparative designs 172-3
privacy 180 interval 262 components 104
probabalistic causality 147 normative 116,118 control (over subjects) 169-70
process tracing 89-90,172 partially categorized 262 cross-sectional designs 170-2
proportion predictive 116,118 data collection 175-8
difference between 360 prescriptive 116,118 defined 104-11
standard error 359 significance/insignificance 106 ethical research 178-80
Protestantism and capitalism, Weber questions, surveys experimental designs 168-9
on 89 acquiescence bias 241 finding 105-11
balanced 240-1 internal and external validity 167
double-barrelled 234 large-N studies 172
Q leading 241 longitudinal designs 170-2
qualitative comparative analysis loading 237 process tracing 89-90, 172
218-22 open-ended and closed 238-40 quasi-/non-experimental designs
crisp vs fuzzy set QCA 218-19 (tab.) 169-70
intermediate-N comparison order effects 237-8 questions 173-4
218-22 presuming 235 research costs 173
qualitative content analysis 319-20, response format 241 -2 selection of cases 173
327 (tab.) wording 234-6 small-N studies 172
quantitative analysis wording effects 238-40 types 168-74
description and inference 338-65 quota sampling 245-6 research issues 3-4
descriptive and inferential empirical vs normative research
statistics 339 4-5,54-5
distribution using statistics R politics vs international relations 4
349-64 R-square 378 positivism vs interpretivism 6-7
graphs and figures 346-9 race and ethnicity 310 quantitative vs qualitative
large-N comparison 222-4 Ragin, on hypothetical truth table research 7-8,14-16
levels of measurement 340 2¡9 (tab.). 220 research positions/aims
tables 342-6 random allocation 196-7 pluralism 3
quantitative content analysis 310, random digit dialling 249 problem-driven 3
319, 326-7,327 (tab.) random probability sampling 243-6 relcvance 3
quantitative data, large-N research rational choice theory 4 1, 87-8 (tab.) research process 8-9
222-4 and eruption of ethnic conflict design of research 9
quantitative vs qualitative research 44-8 formulation of research
7-8, 14-16 Rawls, on justice 157-8 questions 9
question order eU'ects 268 reactivity knowledge 9-11
questionnaires defined 297, 300 methodology 14-20
administration, interviews 248-50 experimental research 192 overview 11-14
 INDEX 439

research questions see questions single-N studies, research design survey articles 10V
'research vase' 112-14 172 surveys 230-52
researcher, gender/identity 301-2 Skocpol, on: ambiguity 234
resource bias 37 MDSD 213, 215 avoiding unfamiliar terms/jargon
resource curse hypothesis 140-1, 142 social revolution 84,92,213 235 6
response rates 247 slope coefficient 374, 376 deliberate misreporting 236-7
retroduction 31 small-N comparative research 208-9, elements of a good survey
revolution, and MDMS 213 (tab.) 214-18 231-2
Rosenthal effect 57 generation of theory 217 faulty presuppositions 234-5
Most Different Systems Design interviews 247-50, 274-6
(M DSD) 212-13,215 Post Office Address file 242-3
s Most Similar Systems Design questionnaires, design 232
sample design 242-7 (MSSD) 209-12 questions 234-46
how not to do it 243-5 replication 217 recall problems (memory failure)
non-probability 245 research design 172 236
probability 244-5 vs large-N research 218 response format 241-2
quota sampling 245-6 snowball and volunteer sampling sample design 242-7
random probability sampling 246,299 sampling frame 242-3
243-6 social desirability bias 237 sources of error 232 (fig.)
response rates 247 social facts. Durkheim on 82-3 validity, from concepts to survey
sampling error 17, 246 Social Research Association (SRA) questions 232-3
snowball and volunteer sampling Ethical Guidelines (2003) systematic sampling 244
246 180
systematic sampling 244 social revolution
weighting the data 244-5 Popkin on 92
sample mean 349 Skocpol on 84,92,213 (tab.) t test 376
sampling error 17,246 Social Sciences Citation Index tables
sampling frame 242-3 (SCSCI) 108 bivariate analysis 382-3
scatter plots 370-3 (fig.) social trust 393 (fig.) cross-tabulation 382-8
science sources, finding and citing 419-23 quantitative analysis 342-6
‘normal’ 62-4 South Africa, witchcraft 289-90 tautological errors 121
as a social institution, Kuhn on speech act theory 311 Taylor, on hermeneutics 40
60-74 spiral of silence theory 291 teachers’ expectations, and student
scientific realism spurious association 211 performance 195
abduction and 38 squares, model sum of 379 telephone interviews 255
goals 38 standard deviation 354, 354 (tab.) telescoping 236
inference and 38 standard error 357, 362 test statistic 376
interpretivism and positivism state failure hypothesis 139-40,142, textual analysis 309-33
compared 49 153 discourse analysis 309, 310-17
vs positivism 35-9 statistical analysis see bivariate theory
scientific research programmes analysis; multivariate application/testing 291
Lakatos on 69-72 analysis definition 136-7
terminology 70 (tab.) statistical significance, cross-tabs development 134-8
Scottish Parliamentary elections 384 grounded 137
2007, effects of ballot design statistics normative 138
and voting instructions central tendency 349-51 see also hypotheses
197 descriptive inference 355-7 Tilly, on causal mechanism 37
screening questions for interviews difference between two transnational economy 86
267 proportions 360-2 Hobsbawm on 86
secondary sources 329 differences of means 362-3 truth claims, empiricism and logic
security 304 dispersion 352-5 31-5
selection bias 330 population estimates 358 truth table, hypothetical 219 (tab.)
significance, cross-tabs 384 proportions 358-9 turnout
significance testing, with interval test statistic 376 age and policy difference (logistic
z statistic 409 regression, parameter
variables 374-5
structural explanations 91 -5 estimates) 415 (tab.)
significance/insignificance, questions
in research process 106 agent-structure problem 92-3 case studies 408-11
structuralism 91 increasing 193
silence, spiral of (theory) 291
structurationism 93 and interaction effects 413-15
simple random sampling 243-6
IN D E X

u causality 146-7 Weber, on:

confounding variables and distinction between facts and
unfamiliar terms/jargon 235-6
uniqueness, false, and false spurious relationships 147, values 55-6
394 Protestantism and capitalism 89
universalism 203
US ‘war on terror’ 114 independent and dependent Wendt, on agent-structure problem
144-6,215-16 92-3
interval variables 341, 374-5 women in Parliament 214
intervening 144-6, 395-6 Wordfish 326
moderator 395-6 wording of questions for surveys
validity 224
moderator variables and 234-6,238-40
coding 328
interaction effects 396 World Systems theory 92-3
construct 233
ordinal 341 World W ar I
content 233
relationships 146-8 factors relevant to outbreak 136
content analysis 328
criterion 233 and values 145 Lenin’s explanation of causes
video link and web interviews 271-2 1 54-6,155-6 (fig.)
discourse analysis 317
ecological validity 192 voluntary participation 179
experimental research 192 volunteer sampling 243,246, 299
face 233 voting behaviour in 2005 British X
internal/external General Election 344 (tab.) x-bar, sample mean 349
case studies 205 and negative campaigning 191
levels 195-6 (tab.)
trade-off 222 Y
surveys 232-3 Yugoslavia, eruption of ethnic
values see facts vs values
w conflict 44-8
van Dijk, on discourse analysis Wantchekon, on clientelism, Benin
315 193-4
variable bias 211 Washington, newspapers 194 z
variables 144-8 Watkins, on methodological z statistic 409
association 368-9 individualism 81,83
Zambia, ethnic cleavage 211 (tab.)
correlation and co-variance
146-8
 Developing a solid understanding of the research enterprise will help you to
appreciate opposing ideas and arguments, find answers to complex questions, make
your own arguments strong and convincing, and evaluate the research of others.

The study of politics can be confusing and

controversial, but developing research skills will
online resource centre
w w w .oxfordtextbooks.co.uk/orc7halperin/
enable you to make sense of conflicting arguments
and interpretations, contribute to knowledge about
the political world, and engage yourself in the This book is accompanied by an Online
search for answers and solutions to important Resource Centre, which includes:
political questions. For students:
► web links
Sandra Halperin and Oliver Heath take a step-by-
step approach to the subject, guiding the reader For registered adopters of the text:
through how to ask and answer questions about ► additional case studies
the world of politics. They present practical advice ► seminar questions and exercises
on how to be critical and rigorous, both in terms of
evaluating the research of others and conducting
independent research, providing you with skills
Sandra Halperin is Professor of
that are genuinely transferable; applicable to any
International Relations at Royal Holloway,
endeavour you choose to pursue.
University of London

Oliver Heath is a Senior Lecturer in

Politics at Royal Holloway, University
of London

Cover image: © Shutterstock; Fotolia

OXFORD ISBN 9 7 8 - 0 - 1 9 - 9 5 5 8 4 1 -A

U N I V E R S IT Y P R E S S

9 II7 8 0 1 9 9 II55 8 4 1 4 I