02-Gilbert-3332-02.

qxd 1/12/2006 2:15 PM Page 7

Reviewing Existing
Research
Annette Boaz with Adrienne Sidford
2
• Literature reviews can be approached systematically to locate as much relevant
literature as possible. This chapter shows you how.

• Reviews are needed to identify what is already known, to bring together results
from different studies, and to provide a starting point for new research.

• Reviewing involves defining the topic, identifying sources, evaluating the

sources, synthesising and reporting.

Whereas single studies are often based on small samples and offer different results, a
review can go beyond individual studies to identify trends and patterns in research
findings. A literature review can be defined as follows:

A systematic, explicit and reproducible method for identifying, evaluating and interpreting
the existing body of recorded work produced by researchers, scholars and practitioners.
(Fink, 1998: 3)

Of course, there is a huge volume of research out there on almost any given topic, so
any attempt to review the literature needs to be guided by a clear plan of action. There
are different ways of going about doing a review, and this chapter will help you to
decide on an approach and put together a plan.

Why you might do a review

There are a number of circumstances in which you might choose to conduct a review
of the literature. Researchers (including PhD students) may need to summarise existing
[Link] 1/12/2006 2:15 PM Page 8

8 From postgraduate to social scientist

research to provide a platform for further work. For students, the literature review
often takes the form of a discrete chapter in a thesis which seeks to summarise the
‘body of knowledge’ relevant to the chosen topic. Other academic studies take a similar
approach, devoting a section of the final research report to a summary of the relevant
literature. This ‘foundation laying’ literature review is perhaps the most common
application of research review techniques.

In addition, research commissioners, such as funding councils and government depart-

ments, sometimes require literature reviews as ground-clearing exercises at the start of
programmes of research. Reviews can also be used to survey a field in order to identify
research gaps or directions and to help develop an agenda for future funding programmes.

These functions, though important, have in the past relegated research reviewing to the
methodological wilderness, with limited debate about its role, conduct and quality.
Increasingly, however, reviews have been commissioned in their own right for quite dif-
ferent purposes. Researchers and research commissioners are promoting reviews as an
opportunity to go beyond the individual study (with all its weaknesses) and to seek out
and synthesise the huge body of research that already exists on almost any topic. Reviews
have become a crucial part of the ‘what works?’ agenda in the UK, with its focus on find-
ing out which public policy interventions are the most effective in bringing about change
(Davies et al., 2000). As a result, literature reviews are finding their place as a mainstream
research methodology. They offer new analytical insights and fresh ideas, and skills in
reviewing will soon be an indispensable part of the researcher’s methodological portfolio.

Why might you do a review?

• To understand what you know already about a given topic

• To identify gaps in the evidence base or generate a research agenda
• To build a platform for future work
• To complement primary research
• To ‘go beyond’ individual studies on a given topic
• To use the individual studies as data to explore a topic or question

The need for more ‘recycling’ of existing research is now widely accepted. At the very
least it makes sense to check whether a research study has, in effect, been done already.
[Link] 1/12/2006 2:15 PM Page 9

Reviewing existing research 9

It is sometimes surprising just how much research there is, even in relation to quite
specific topics. For example, a team at Glasgow University in Scotland conducted a
literature review on the topic of gentrification – the rejuvenation and renovation of
run-down urban areas by the middle class (usually resulting in the displacement of
resident lower-income communities). This might be assumed to be a relatively small
and specialised field of study. However, the search identified research from all over the
world, including Australia, Europe and North America. The research team identified 17
existing literature reviews on the topic, in addition to the mass of primary research
(Atkinson, 2002).

Literature reviews should be undertaken to the same high standards as primary

research. Reviewers need to demonstrate clarity of purpose, use appropriate methods
and demonstrate objectivity and rigour in their work. Review findings and methods
should also be clearly reported.

By taking each step of the review process in turn we shall consider how to conduct a
good-quality review that is something more than a dry summary of the state of knowl-
edge on a given topic. The key steps of the review process are:

1. Defining the review topic or question

2. Locating sources

3. Judging the quality of studies identified

4. Synthesising the studies identified

5. Reporting the findings of the review

It is worth noting that, while the stages are presented as a linear process, most ‘real
world’ reviews do not follow a simple progression through stages 1 to 5. For example,
searching often continues (or should continue) throughout much of the review process
as the reviewer develops a closer understanding of the topic.

Review methods
At first glance there seems to be a suite of literature review methods from which to
choose. Common terms you might have come across include literature review, scoping
review, systematic review, narrative review, meta-analysis and rapid review. However,
[Link] 1/12/2006 2:15 PM Page 10

10 From postgraduate to social scientist

on close inspection, the terminology is confusing and unhelpful. The list does not
constitute a menu of methods.

There is a growing interest in conducting reviews more systematically and transpar-

ently (Boaz et al., 2002). There are a number of organisations, such as the international
Cochrane and Campbell Collaborations, which support the conduct of systematic
reviews in health, social welfare, criminal justice and education (see the Further
Reading section at the end of this chapter for more details), and various organisations
have produced ‘how to’ guides and training for individuals interested in using this
approach. These include the CRD report 4 (NHS Centre for Reviews and
Dissemination, 2001) and the EPPI-centre review companion ([Link]/EPPI
WebContent/downloads/[Link]).

This chapter does not seek to replicate these comprehensive resources. Instead it dis-
cusses some general principles that might apply to anyone conducting a good-quality
literature review. Illustrations are drawn from two reviews of the literature on men-
toring (Table 2.1). Mentoring is used as an intervention in a wide range of settings,
including education, youth justice and the workplace (Boaz and Pawson, forthcom-
ing). Mentors act as role models, counsellors or teachers, providing support to mentees
with less experience. The mentoring reviews discussed here describe themselves dif-
ferently (as a literature review and a meta-analysis), but they have common features,
as we shall see below.

Defining the topic

Focus
When embarking on a review of the literature it is important to think carefully about
its focus. Reviewers too often try to cover everything there is to know about a given
topic, an over-ambitious goal that usually results in a review with little space to reflect
on the literature identified. While the first of the mentoring reviews addresses a rela-
tively tightly defined topic, the second sets out to answer a whole set of questions,
including: what is mentoring, does it work, what makes it work and how is it viewed
by different stakeholders? The sheer volume of literature identified presents a daunt-
ing task to the reader.

As with good primary research, systematic review guides promote the development of
a precisely defined review question in the hope that this will lead to a more focused
[Link] 1/12/2006 2:15 PM Page 11

Reviewing existing research 11

Table 2.1 Two reviews of the mentoring literature

Type of review (author

Bibliographical details description) Main findings

Dubois, D., Holloway, B., A ‘meta-analysis’ There is evidence that mentoring

Valentine, J. and programmes are effective interventions
Cooper, H. (2002) (although the effect is relatively small). For
Effectiveness mentoring programmes to be as
of mentoring successful as possible, programmes
programs for youth: need to follow guidelines for effective practice.
a meta-analytic review Programme characteristics
American Journal of that appear to make a difference in
Community promoting effective practice include
Psychology 30(2): ongoing training for mentors, structured
157–197 activities, frequent contact between mentors
and mentees and parental involvement.

Hall, J.C. (2003) A ‘literature review’ Mentoring is an ill-defined and contested

Mentoring and Young concept. The US evidence suggests that
People: A Literature mentoring is an effective intervention,
Review although the impact may be small.
The SCRE Centre: Successful mentoring schemes are likely
University of Glasgow, to include: programme monitoring, screening
51pp of mentors, matching, training, supervision,
structured activities, parental support and
involvement, frequent contact and ongoing
relationships. The UK literature concludes that
mentoring needs to be integrated with other
activities, interventions and organisational
contexts. Most mentors are female, white and
middle-class, and report positive personal
outcomes, including increased self-esteem.

review. In its original form in medicine, reviewers have been encouraged to formulate
their review question in the way shown in Figure 2.1.

For example, are young offenders (the population) who undergo a mentoring pro-
gramme (the intervention) less likely to re-offend (the outcome)? How do they fare
compared to other young people who have not been involved in the mentoring
programme (comparison, control and/or context)? While many of the questions
reviewers address in social policy areas are more complex than this, it is some-
times useful to apply this simple formula (called PICO – Population, Intervention,
Control, Comparison and/or Context and Outcomes) when thinking about a review
question.
[Link] 1/12/2006 2:15 PM Page 12

12 From postgraduate to social scientist

Population Intervention Outcome

Control, comparison and/or context

Figure 2.1 Population, intervention, control, comparison and/or context,

and outcomes

Scope
It is also useful to consider a number of other dimensions that will have an impact on
a review’s scope, and thus on its findings. For example, will the review contain litera-
ture from one country or from a range of countries? This can be an issue of real sig-
nificance. Consider, for example, what the implications might be for a UK review of
gun crime that relied only on literature from the USA, where policies towards gun
ownership are very different from those in other countries.

It is also important to think carefully about the time period covered by the review.
The coverage needs to be congruent with the review question. For example, a review
of the impact of mobile phones on communication skills might only consider looking
at the literature from the last ten years (although it has been argued that the first
mobile phone was invented in 1924! – see [Link]/mobile_phones_
[Link]). If the time period has not been dictated by the review question, there
has to be some explicit reason why this is the case (e.g. lack of time, resources etc.).
The Dubois mentoring review examines literature from 1970 to 1998 and the Hall
review focuses on research published after 1995. Neither reviewer is explicit about
their reasons for choosing a particular time period.

Most reviews only include research (as opposed to evidence contained in policy docu-
ments and articles in newspapers and magazines). Non-research evidence is often side-
lined on grounds of quality. Reviewers are concerned that the evidence in newspapers
and policy documents cannot be trusted. However, there have been some attempts to
open up a debate about the quality and contribution of different types of evidence
(Grayson et al., 2004). In particular, non-research evidence can often provide useful
information about context and can help to fill gaps in the research evidence base.

Sometimes reviews even focus on specific sorts of research, usually reflecting the
terms of the review question. For example the first review in Table 2.1 includes
[Link] 1/12/2006 2:15 PM Page 13

Reviewing existing research 13

only studies that evaluate interventions because the review question focuses on the
impact of mentoring. However, the second review looks at a wide range of review
questions and pulls together lots of different types of research to address them.

Defining the topic: the mentoring review

The Hall review sought to address the following list of questions: What is a
mentor/mentoring? What works/doesn’t work? What is the evidence of positive
outcomes for young people? Is there a case for regulating mentors? What are the
views and experiences of mentors and mentees? What are the views and experi-
ences of commissioning bodies and/or employers? The review included
research from the UK and ‘other relevant countries’ from 1995 onwards.

The Dubois review focused on the impact of mentoring programmes on youth.

The review included evaluation studies that used either before and after com-
parisons or control groups. The review focused on research published between
1970 and 1998. All the studies seem to be from the USA.

It is important that all these details about the scope of the review are written down in
a protocol or project plan. This has the advantage of recording the reviewer’s deci-
sions for all to see, thus promoting transparency. For example, the scope of a review is
often progressively narrowed for practical reasons, such as lack of time and resources
(translation costs for large numbers of papers can be high, for example). It is impor-
tant to articulate these issues to allow readers to place the review findings in context.
It also allows the reviewer to revisit his or her decisions later on in the review process.

Locating sources
Searching
Searching for the raw material of a review is about much more than pulling down a
set of books off the shelf. In order to find as much literature as possible that is rele-
vant to the topic or question, a research reviewer needs to employ a range of tools and
techniques. The search might typically include: a key word search of relevant data-
bases, internet searches (discussed in more detail in Chapter 3), hand searches of key
journals and a search through the bibliographies of papers, reports and conference
proceedings. These more formal approaches are often complemented by informal
[Link] 1/12/2006 2:15 PM Page 14

14 From postgraduate to social scientist

contact with key individuals working in the field under review. This strategy is
particularly useful for picking up unpublished research and work in progress.

Database searching is the most common approach to identifying the literature.

Electronic bibliographic databases are particularly strong in the peer-reviewed acade-
mic literature, but can also cover the grey literature (unpublished research and/or
research from non-academic sources) and PhD theses.

Some well-known social science databases

• ASSIA (Applied Social Science Index and Abstracts)

• ERIC (Educational Resources Information Centre)
• IBSS (International Bibliography of the Social Sciences)
• Sociological Abstracts
• Criminal Justice Abstracts
• Worldwide Political Science Abstracts
• INSIDE (British Library database)

Searching for mentoring studies

The Hall review searched the following databases: ERIC (Educational

Resources Information Centre), BEI (British Education Index) and ERSDAT
(the Educational Research in Scotland Database), looking for the key word
‘mentor’. The reviewer also searched relevant websites, checked lists of refer-
ences in other literature and contacted key individuals in the field, requesting
copies of any relevant documents.

The Dubois review searched the following databases: PsycINFO, ERIC, Medline
and Dissertation Abstracts, using both subject (index) terms (e.g. mentor) and free
text words (e.g. the names of popular mentoring schemes). The reviewers also
searched the internet using several search engines, and checked the reference
sections of the studies identified for inclusion to check that nothing had been missed.

Reviewers in the social sciences can face significant problems as a result of the multi-
plicity of electronic databases (small and large, general and specialist, free and priced),
[Link] 1/12/2006 2:15 PM Page 15

Reviewing existing research 15

their variable coverage of the different kinds of publication media in which useful
social science evidence appears, and the need for skills and experience to carry out
thorough searches (Grayson and Gomersall, 2003). These difficulties make the sup-
port of information specialists of particular importance, and reviewers often draw on
the support of information scientists and librarians in designing and carrying out
searches. However, courses are also available for researchers and reviewers interested
in developing their own search skills. For example, the NHS Centre for Reviews and
Dissemination at York University in the UK provides sessions on literature searching
as part of its three-day course on systematic review and critical appraisal, and the
ESRC Evidence Network also runs day courses on search skills. There are also some
developments designed to improve access to social policy research, including the
launch of a new database entitled ‘Social Policy and Practice.’

It is unwise in any review to rely solely on the results of database searches. Careful
reviewers will hand search key journals and other bibliographic sources, and scan the
bibliographies of all retrieved papers, reports and books to pick up related material.
They will often complement searches of bibliographic databases by looking at research
databases (e.g. the ESRC’s REGARD database) or at conference proceedings (e.g. via
the British Library’s INSIDE database) to identify relevant ongoing and unpublished
work. Contacting professional networks and experts in the field to check for gaps and
to identify any very recent or unpublished research can also be very productive.

To facilitate the search for relevant material a search strategy is recommended, making it
clear to readers of the review how the studies included were identified. The search strategy
is likely to outline the key words used to search databases. For example, the search strategy
for the Hall review on mentoring and young people looked for papers that included the
search term ‘MENTORS’, but excluded a long list of terms alluding to other types of men-
toring, such as ‘NURSING EDUCATION’, ‘TEACHER IMPROVEMENT’ and
‘GRADUATE STUDY’. It might also include a list of databases, contacts to be followed
up, websites to be checked, etc. The strategy is likely to be an evolutionary process, prob-
ably beginning with fairly broad searches to establish the main outline of the topic, followed
by a series of more detailed searches as the reviewer develops a closer understanding of the
issues. Searching may continue throughout much of the review process and should be care-
fully documented, with details of the sources and search terms used.

Sifting
Using a broad set of search terms on a number of databases is likely to identify a very
large number of papers. Many will be irrelevant to the review and an initial sift can
[Link] 1/12/2006 2:15 PM Page 16

16 From postgraduate to social scientist

often be done on titles. For example, a literature search for a systematic review on new
roads (including bypasses) identified 23,000 studies, many of which were concerned
with coronary artery bypass operations rather than road bypasses (Egan et al., 2003)!
Papers that look promising based on their titles can be checked for relevance through
abstracts or summaries, although it is important to remember that the quality of
abstracts in social science databases is often inadequate and the choice of ‘promising’
papers should be liberal. The roads reviewers ordered 700 papers, of which 100 were
relevant to the review. Of these, 32 met the inclusion criteria (the reviewers were only
interested in the impact of roads on human health so environmental studies, for exam-
ple, were excluded). In a review such as this, which has a tightly defined question and
quality criteria, the reviewer will often only need to read a small subset of the papers
identified through the initial searches. A PhD student doing a review for thesis pur-
poses, or a researcher doing a scoping review with a broader question (and no quality
threshold) may well have to read a great deal.

In order to manage the task of sifting and sorting, many reviewers use a data extrac-
tion tool to record basic information about the individual studies identified, such as
title, author and key findings. The advantage of completing a sheet about each study
is that it acts as a record for the reviewer and can be a useful resource for anyone aim-
ing to update the review. At the very least, a reviewer should make a note of the full
reference for each study, either in a word-processing package or in a reference organ-
ising programme such as Reference Manager or EndNote.

Judging quality
A reviewer may decide to include all the literature relevant to his or her review topic.
Here ‘fitness for purpose’ (or fit with the review topic or question) is the primary con-
cern when deciding which studies to include in the review. This is the approach used
in the Hall review on mentoring.

In particular, some reviews aim to include in scope all the relevant literature identi-
fied through searching in order to understand what is known about a given topic.
Similarly, a review aiming to identify gaps in the evidence base or to generate a
research agenda is often less concerned with the quality and more concerned with the
coverage of existing research. While occasionally a reviewer might focus on the stud-
ies that offer the most explanatory power or the clearest conclusions, they have rarely
explicitly excluded studies on methodological grounds.
[Link] 1/12/2006 2:15 PM Page 17

Reviewing existing research 17

In contrast, systematic reviewing has brought with it a renewed interest in the

methodological quality of the studies identified as potentially relevant for review.
This approach involves the use of quality criteria, frameworks and checklists to sift
out poorer quality studies (e.g. studies with inadequate sample sizes). Where possible,
review teams try to employ more than one person in the appraisal process.

However, choosing to exclude studies on the basis of methodological quality is not

without its challenges. Perhaps the biggest problem is the lack of transparency in the
reporting of primary studies. Often crucial methodological details are missing from
research papers and reports, whether as a result of oversight, the pressure of word lim-
its or the perceived low importance of methodological details.

A further difficulty can arise if the majority of the evidence relevant to the review ques-
tion is deemed to be of poor quality. Here reviewers get caught in a trade off between the
quality of the research, the relevance of the research to the review question and the cur-
rent availability of evidence. One review team, considering the evidence on water fluori-
dation, got around this problem by reporting a wide range of evidence of different quality,
but presenting it with a quality mark of either A, B or C to allow the reader to decide
whether to consider the poorer quality results (NHS Centre for Reviews and
Dissemination, 2000). Many review groups seek to address the issue of poor-quality evi-
dence on their chosen topic by setting out a clear agenda for future research at the end of
the review. Even if they cannot come to firm conclusions on the topic, reviews offer a valu-
able opportunity to reflect on the quality of, and learn from, the conduct of other research.

For some research methodologies (such as randomised controlled trials), quality crite-
ria, tools and checklists abound, while for others it is difficult to find a critical
appraisal tool to use (NHS Centre for Reviews and Dissemination, 2001). Within the
social sciences there is a lack of consensus on what counts as good-quality research
in some areas, such as qualitative research, while some researchers argue that the
appraisal of quality is inappropriate and unhelpful (Pawson, 2004).

Whether or not reviewers use formal criteria or checklists, their own judgement of qual-
ity is a crucial element in any review. This is informed by their private understanding of
what should count as a useful and reliable piece of work. However, seeking to articulate
this process of judgement (for the benefit of the reader) is a challenge for the reviewer.

The two mentoring reviews discussed here took different approaches to appraising
the quality of the literature.
[Link] 1/12/2006 2:15 PM Page 18

18 From postgraduate to social scientist

Appraising the quality of mentoring studies

The Hall review sifted the papers on their relevance to the review.

The Dubois review team first sifted papers on their relevance to the review
question. Second, as the review was concerned with effectiveness, they excluded
studies that did not evaluate the effectiveness of mentoring programmes using
specific research methods: either studies with a comparison group (controlled
studies) or studies that compared participants before and after participation in
the programme (pre and post-test studies). No details are given of any quality
checklists or criteria applied to these studies.

Synthesising results
Perhaps one of the biggest challenges facing the reviewer is how to make sense of
the mass of literature identified, retrieved and neatly piled ready for analysis. Most
reviewers hope to pull together the research into a chapter or report that says some-
thing additive, that is, the plan is to produce a review that is greater than the sum of
its individual parts (or studies).

The extent to which the studies are integrated in a synthesis depends in part on the
purpose of the review (for example, a review might aim to give an overview of the cur-
rent literature, and thus just order and assess coverage rather than integrate) and in
part on the methods used (for example, a meta-analysis combines data from individ-
ual studies).

Perhaps the most straightforward approach to bringing the literature together is to

describe each study briefly and draw some conclusions. This is less of a synthesis,
more a description of the literature. Sometimes, it might take the form of an annotated
bibliography, or it may consist of paragraphs of text describing the different studies in
turn. The moment of synthesis then rests in a section that draws conclusions from this
descriptive text.

However, some reviewers have tried to achieve a greater degree of integration. For
example, the most common methodology for synthesis in medical research reviews
[Link] 1/12/2006 2:15 PM Page 19

Reviewing existing research 19

is meta-analysis (Egger et al., 2001), a way of numerically combining the results of

existing studies in order to calculate the size of the overall effect for an intervention.
However, meta-analysis is only possible where there are appropriate and comparable
quantitative data from more than one study. This is rarely the case in social policy
research, where multiple outcomes lead to diverse measures, and where quantitative
data may be entirely lacking.

Synthesising mentoring studies

The Hall review organised and synthesised the literature around a set of ques-
tions provided by the review commissioners. All the literature relevant to each
question was described under question headings. For example, in addressing the
question ‘Does mentoring work?’, the review draws on a similar evidence base
to the Dubois review (and the Dubois review itself), whereas in addressing the
question ‘How is mentoring viewed by different stakeholders?’, the review
draws together a very different literature.

The Dubois mentoring review synthesises the literature in a meta-analysis of

the 55 relevant studies identified. The review team had to find a way of pooling
together very different outcome measures (such as school achievement, atten-
dance and employment) in order to conduct a meta-analysis.

Where the quantitative data cannot be integrated in a meta-analysis, or where the

research identified is largely non-numerical, other forms of synthesis are necessary. It
is possible to organise the content of the review in other ways, for example, around
themes or questions (as is the case with the second mentoring review), around meth-
ods (What do all the surveys tell us about the attitudes of mentees to mentoring
schemes? What do all the case studies tell us about the ways in which mentoring
schemes are set up and run?), or around some sort of taxonomy or model. While these
methods do not necessarily call for the same level of integration, they add a degree of
organisation to the information presented.

The difficulties of applying meta-analysis and other methods of synthesis developed

within a medical context have encouraged social scientists to develop new approaches,
including Realist Synthesis and Meta-ethnography. Realist Synthesis is a theory-driven
[Link] 1/12/2006 2:15 PM Page 20

20 From postgraduate to social scientist

approach (Pawson, 2002) that aims to use the literature to expose and articulate
underlying assumptions about how a policy or programme is intended to work.
Rather than passing a straightforward ‘it works’ or ‘it doesn’t work’ verdict on a policy
or programme, Realist Synthesis aims to explain how an intervention (such as men-
toring) works, why it works and in what circumstances it works. As such, the review
involves both theoretical thinking and an empirical testing of this thinking through an
exploration of the literature.

First developed by Noblit and Hare (1988), and further developed by Campbell et al.
(2003), meta-ethnography uses a qualitative, ethnographic approach to synthesise
qualitative research. This method involves identifying key concepts from the individ-
ual papers and seeking them out in the wider literature. Noblit and Hare argue that
this approach can be used to go beyond individual studies, comparing, challenging and
providing new insights.

In the social sciences, narrative descriptions of the research identified remain the
most common form of synthesis, and these new methods are still in a relatively early
stage of development. Synthesis remains one of the most intellectually challenging
aspects of the review process, and one that invites the reviewer to think creatively
about how to present the mass of literature in a meaningful way to potential
readers.

Reporting
The final hurdle involves the important tasks of reporting and communicating the
review findings. Reviewers are encouraged to be as transparent as possible about
the process they used to conduct the review. Methodological information does not
need to be given in the opening chapters of the report, but should be included
(often in detail as an appendix). Furthermore, a full bibliography of papers refer-
enced in the review should be included. Where the reviewer has used data extrac-
tion forms, these are sometimes reproduced at the back of the report in an
appendix or in a separate volume. One of the advantages of such clear reporting is
that the review can be updated by the reviewer or by someone else interested in
the topic.

The approach to writing up and sharing the findings of a review will depend on the
purpose for which the review was undertaken. For example, some reviews will
be conducted for largely internal purposes, such as for a PhD or to inform a new
[Link] 1/12/2006 2:15 PM Page 21

Reviewing existing research 21

project. However, where a review has been commissioned or has things to say to a
wider audience than the review team, it is important to consider carefully the
communication of its findings. Reviews can be over-long, turgid documents, as thick
as telephone directories (but less accessible). At the very least, the document needs
to be navigable and to have a short, well-written summary. The Findings series pro-
duced by the UK Joseph Rowntree Foundation is frequently cited as an example of
good practice ([Link]). Short papers, for example in journals and magazines
read by practitioners, can also be a useful method of disseminating the findings of a
review.

Often it is difficult to anticipate the future uses of a review. As literature reviews can
be invaluable sources of primary literature for future researchers, it is recommended
that a copy is sent to the national copyright library (such as the British Library in
the UK).

In addition to short summaries and articles, there may be a need to present a review
orally to interested parties. Increasingly, formal reports and presentations are also
complemented by web-accessible versions. Writing and communicating research find-
ings are discussed in more detail in Chapters 8 and 9.

Management of a review
A protocol is a map or plan of the review process that provides an opportunity to think
about crucial issues, such as the resources of time, skills and labour needed to com-
plete the work (NHS Centre for Reviews and Dissemination, 2001). A protocol
should not be set in stone and can be revisited and changed as the review progresses.
Issues of project management are discussed in more detail in Chapter 6.

While literature reviews have traditionally been seen as a one-person, desk-based

activity, they can also be conducted by teams. The advantage of this approach is that
a mix of expertise can be brought together. For example, a review team might include
an information specialist, a subject specialist and a methodologist with knowledge of
review methods. More than one person can be involved in independently appraising
the studies in order to try to reduce bias. Like primary research, reviews also often
benefit from the support of an advisory group. For example, this was the case for the
second review in Table 2.1. Where the review is a lone pursuit, advice from an infor-
mation specialist, in particular, can be a real advantage.
[Link] 1/12/2006 2:15 PM Page 22

22 From postgraduate to social scientist

Case study: A student’s perspective on reviewing literature

Adrienne Sidford

I think that most researchers and students can look back on a moment where they
got really ‘lost’ when doing a review. I was looking at evaluation within both policy
and public health domains. Before beginning my study I needed to have evidence
to support the evaluation method being used, as well as evidence setting the con-
text of previous evaluations.

The review soon became much bigger than me, growing as I was side-tracked. My
questions for the review were vague, partly because I had little understanding of
the area. The more I read the more I became embroiled in the ongoing debate sur-
rounding evaluation and the views of different researchers’ paradigms regarding
the discipline/methodology (positivist vs constructionist). I became bogged down in
the fine detail and was trying to solve a debate that will continue to permeate
through the social sciences. Also, my initial plan for the review was not allowing me
to highlight central themes and I became more frustrated. I found I was unable to
tease out the story as I was not really sure what I wanted. I had not really accepted
that although my main reason for doing the review was to look at how evaluation
had been carried out in public health and policy, it was also to provide a rationale
for the method that my study was using. Unsurprisingly, my review was going around
in circles!

After some discussions with my supervisors, I took a break from the review, reflect-
ing on the problems I had encountered. I came back to it able to see the whole pic-
ture, discussed it with colleagues and managed to focus the review to get it to
answer all my questions. Writing reviews is both a painful and exciting process,
involving a lot of detective work. The reward is a fuller, more powerful argument for
the study I am conducting, and a greater understanding of where my study sits
within this body of knowledge.

Conclusions
Literature reviewing is a research method and quality considerations apply as they do
to all research methods. A review should be conducted in a rigorous and transparent
manner and address a clearly defined question or topic. It should be based on a thor-
ough and fully documented search. The reviewer should consider the studies identi-
fied in terms of their fitness for the purpose of the review. The reviewer might also
appraise the studies and include only those of good methodological quality. A method
[Link] 1/12/2006 2:15 PM Page 23

Reviewing existing research 23

must be chosen (or devised) of pulling the results of the different studies together to
produce a review that is more than an untidy pile of primary studies. Finally, the
reviewer must also consider how best to communicate the findings to different audi-
ences. The entire process can be improved by the use of a protocol outlining the
proposed review approach, including the search strategy.

This may sound like a lot of work, but well-conducted literature reviews can have
something powerful to offer a range of different users, including researchers them-
selves, policy makers and practitioners. A literature review goes beyond the common
reliance on commissioning a new study or relying on one piece of research. Recycling
research is likely to be of more and more use to the research community in the future.

Further reading
Davies, H.T.O., Nutley, S.M. and Smith, P.C. (2000). What Works? Evidence-based Policy and
Practice in Public Services. Bristol: The Policy Press.

Fink, A. (1998). Conducting Research Literature Reviews: From Paper to the Internet. Thousand
Oaks, CA: Sage.

Relevant organisations and websites

Campbell Collaboration: [Link]

Building on the experience of the Cochrane Collaboration, Campbell has been set up to carry
out reviews of interventions in the fields of education, criminal justice and social welfare. The
website currently includes guidance on protocol construction, specimen protocols and other
information. The first Campbell Review on Scared Straight initiatives can be downloaded
from both the Campbell and Cochrane websites.

Cochrane Collaboration: [Link]

The Cochrane Collaboration prepares, maintains and disseminates the results of systematic
reviews of research on the effects of health care. The Cochrane Library is a quarterly updated
electronic database of reviews. The Cochrane Manual and the Reviewer’s Handbook are
available online.

ESRC UK Centre for Evidence-based Policy and Practice:[Link]

The Centre’s EvidenceNetwork website is designed to act as a starting point for

accessing key literature and information resources on evidence-based policy and
practice. This is a useful site for identifying electronic databases, internet portals
[Link] 1/12/2006 2:15 PM Page 24

24 From postgraduate to social scientist

and other sources of information for literature reviews, and includes some basic advice on
search techniques.

Evidence for Policy and Practice Information and Coordinating Centre: [Link]

The Centre was originally commissioned by the Department for Education and Employment
to provide a resource for those wishing to undertake systematic reviews in the field of
education. A database of systematic reviews, and useful publications on systematic review
methodologies are accessible via this site.

NHS Centre for Reviews and Dissemination: [Link]/inst/crd

CRD carries out systematic reviews on selected topics in the health-care field and maintains a
database of reviews (DARE). A number of useful documents, including Undertaking Systematic
Reviews of Research on Effectiveness (CRD Report No. 4), are accessible online.