Appraisal tool for Cross-Sectional Studies (AXIS)

Critical appraisal (CA) is used to systematically assess research papers and to judge the reliability of the study being presented in
the paper. CA also helps in assessing the worth and relevance of the study [1]. There are many key areas to CA including
assessing suitability of the study to answer the hypothesised question and the possibility of introducing bias into the study.
Identifying these key areas in CA requires good reporting of the study, if the study is poorly reported the appraisal of suitability
and bias becomes difficult.

The following appraisal tool was developed for use in appraising observational cross-sectional studies. It is designed to address
issues that are often apparent in cross-sectional studies and to aid the reader when assessing the quality of the study that they are
appraising. The questions on the following pages are presented in the order that they should generally appear in a paper. The aim
of the tool is to aid systematic interpretation of a cross-sectional study and to inform decisions about the quality of the study being
appraised.

The appraisal tool comes with an explanatory help text which gives some background knowledge and explanation as to what the
questions are asking. The explanations are designed to inform why the questions are important. Clicking on a question will
automatically take you to the relevant section in the help text. The appraisal tool has areas to record a “yes”, “no” or “don’t know”
answer for each question and there is room for short comments as well.
Contents
Appraisal of Cross-sectional Studies....................................................................................................................................................3
Introduction...........................................................................................................................................................................................4
Aims..................................................................................................................................................................................................4
Methods.................................................................................................................................................................................................4
Study Design.....................................................................................................................................................................................4
Sample Size Justification..................................................................................................................................................................4
Target (Reference) Population..........................................................................................................................................................4
Sampling Frame................................................................................................................................................................................4
Census...........................................................................................................................................................................................5
Sample Selection...............................................................................................................................................................................5
Non-responders.................................................................................................................................................................................5
Measurement Validity & Reliability.................................................................................................................................................5
Statistics............................................................................................................................................................................................6
Overall Methods................................................................................................................................................................................6
Results...................................................................................................................................................................................................6
Basic Data.........................................................................................................................................................................................6
Response Rate...................................................................................................................................................................................6
Internally Consistent Results............................................................................................................................................................6
Justified Discussions and Conclusions.............................................................................................................................................7
Aim................................................................................................................................................................................................7
Selection Bias................................................................................................................................................................................7
Non-response................................................................................................................................................................................7
Confounding..................................................................................................................................................................................7
Non-significant Results.................................................................................................................................................................7
Limitations........................................................................................................................................................................................8
Other......................................................................................................................................................................................................8
Conflicts of Interest...........................................................................................................................................................................8
Ethical Approval...............................................................................................................................................................................8
References.............................................................................................................................................................................................8
Appraisal of Cross-sectional Studies

Don’t know/
Question Yes No
Comment
Introduction
1 Were the aims/objectives of the study clear?
Methods
2 Was the study design appropriate for the stated aim(s)?

3 Was the sample size justified?

Was the target/reference population clearly defined? (Is it clear who the
4
research was about?)

Was the sample frame taken from an appropriate population base so that
5
it closely represented the target/reference population under investigation?

Was the selection process likely to select subjects/participants that

6
were representative of the target/reference population under
investigation?
7 Were measures undertaken to address and categorise non-responders?

Were the risk factor and outcome variables measured appropriate to the
8
aims of the study?

Were the risk factor and outcome variables measured correctly using
9 instruments/measurements that had been trialled, piloted or published
previously?

Is it clear what was used to determined statistical significance and/or

10
precision estimates? (e.g. p-values, confidence intervals)

Were the methods (including statistical methods) sufficiently described

11
to enable them to be repeated?
Results
12 Were the basic data adequately described?

13 Does the response rate raise concerns about non-response bias?

14 If appropriate, was information about non-responders described?

15 Were the results internally consistent?

16 Were the results presented for all the analyses described in the methods?
Discussion
17 Were the authors' discussions and conclusions justified by the results?

18 Were the limitations of the study discussed?

Other
Were there any funding sources or conflicts of interest that may affect the
19
authors’ interpretation of the results?

20 Was ethical approval or consent of participants attained?

Introduction should be presented. This should be concluded giving a

rational as to why the current study is being presented and
The introduction serves to establish the context of the work what the aims and/or hypothesis under investigated are [2,3].
that is about to be presented in the text of the paper.
Relevant primary literature should be discussed and Aims
referenced throughout the introduction. The history and
The aim(s) of the study tells us if the study addresses an
current understanding of the problem being researched
appropriate and clearly focused question. If the aim is not Question 3 asks if sample size justification was reported, but
clearly stated or not stated at all, it will be difficult and in it should also be clear what methods were used to determine
some cases impossible to assess the extent to which the the sample size. In some cases clustering of observations
study objectives were achieved. Ideally, an aim should be within groups can occur (e.g. patients within hospitals or
stated both at the beginning of the abstract and at the end of livestock within herds) and this should be taken into account
the introduction [3]. If the answer to question 1 is no, then it if sample size has been determined. It should be clear
will make it difficult to assess some of the other questions in whether the inferences drawn actually relate to the attributes
the critical appraisal process. for which the sample size was calculated [7]. If sample size
justification isn’t given or restrictions make it difficult to
Methods reach the desired sample size then this should be declared in
the text.
The methods section is used to present the experimental
study design of the paper. The methods should be described
Target (Reference) Population
clearly in easy to understand language and clearly identify
measures, exposures and outcomes being used in the study The target or reference population is the overall population
[4]. More specific issues are addressed below. that the research is directed towards. When doing a cross-
sectional study, a target population is the overall population
Study Design you are undertaking the study to make conclusions about or
the population at risk of acquiring the condition being
Question 2 is used to assess the appropriateness of using a
investigated [8–10] e.g. the total female population in the
cross-sectional study to achieve the aim(s) of the study.
UK, or all dogs in the USA with cardiovascular disease.
Cross-sectional studies are observational studies that provide
(See Figure 1) Question 4 asks if this is clearly defined in
a description of a population at a given time, and are useful
the study. It is important that this is understood both by the
in assessing prevalence and for testing for associations and
researcher and the reader; if it is not clearly defined then
differences between groups [5]. Examples of cross-sectional
inferences made by the researcher may be inappropriate.
designs include point-in-time surveys, analysis of records
and audits of practice [6]. The reader should try and
Sampling Frame
decipher if a cross-sectional study design is appropriate for
the questions being asked by the researcher. As a reader you need to 1determine if the sample frame being
Figure
used is representative of the target population. The study
Sample Size Justification population should be taken from the target population; units
from this study population have information that is
Sample size justification is crucial as sample
accessible and available which allows them to be placed in
size profoundly affects the significance of the
the study. The sampling frame is the list or source of the
outcomes of the study. If the sample size is too
study population that the researcher has used when trying to
small then the conclusions drawn from the study
recruit participants into the study (Figure 1). Ideally it
will be under powered and may be inaccurate.
should be exactly the same composition or structure as the
This can occur by failing to detect an effect which truly
target population. In practice it is generally much smaller,
exists (type II error) sometimes referred to as a “false
but should still be representative of the target population.
negative”. The probability of a type I error is also taken into
Generally, for convenience, the sampling frame is a list of
account when determining sample size. A type I error is
units that are within the target population e.g. list of
drawing significant conclusions when no real difference
exists and is a function of the p-value (see Statistics section
below) sometimes referred to as a “false positive”.

telephone owning households, computerised patient records

etc. A sample of units is selected from the study population
to take part in the study and is generally only a small
proportion of the study population (see Sample Selection
below) - this proportion ratio is known as the sampling
fraction. It is very important that the sampling frame is
representative of the target population as results from the
study are going to be used to make assumptions about the
target population [8–10].
Convenience sampling can be carried out in some situations lead to inferences been made about the health of the worker
and are used because the participants are easy to recruit. population but is only relevant to healthy workers and not ill
Convenience samples generally lead to non-representative or workers. A veterinary example of this is a researcher trying
biased samples and therefore cannot be used to make to do a cross-sectional study to determine health factors in
assumptions about the characteristics of the target the general dog population and decides to sample from a
population [11]. Convenience samples are often used for local park. Unfortunately there is a tendency to over select
pilot or analytical studies where the need for a representative healthy animals as sick animals will tend to be left at home
sample is not required [12], however the authors should and not taken for a walk. This will in turn lead to inference
make this clear in the text. been made about the health of the dog population but is only
relevant to healthy dogs and not sick dogs.
Census
Self-selection is another example of selection bias that can
A census is where the target population and the study
be introduced and should be assessed [13]. For example,
participants are the same at the time the census is taken. In
when using a postal questionnaire to examine eating habits
theory questions 5, 6 and 7 don’t apply to census studies.
and weight control, people who are overweight might read
However even if a study is described as a census it should be
the survey and be less inclined to complete and return the
very clearly stated where the study participants have been
survey than those with normal weight leading to over
recruited from, and the reader should make the decision if
representation of people with normal weight. Similarly, if
the study truly is a census. A census may include all the
using a postal questionnaire to examine mastitis levels on
population from the sample frame, but not all the target
cattle farms, farmers that have a high somatic cell counts
population; in this scenario questions 5 to 7 need to be
(SCC) might be less inclined to complete the survey than
addressed.
those with normal or low SCC leading to over representation
of farms with good SCC (see Non-responders below).
Sample Selection
Question 6 is used to establish how the researchers got from Non-responders
the sample frame to the participants in the study. It
Non-response in cross-sectional studies is a difficult area to
examines the potential for selection bias and how the
address. A non-responder is someone who does not respond
researcher developed methods to deal with this. The sample
either because they refuse to, cannot be contacted, or
selection process is important in determining to what extent
because their details cannot be documented. As a rule, if
the results of the study are generalizable to the target
participants don’t respond it is often difficult and sometimes
population. For question 6 we are looking in depth at how
impossible to gain any information about them. However
the sample (study participants) was selected from the
other baseline statistics may exist that can be used as a
sampling frame. It is important to know if there were any
comparator to assess how representative the sample is [14]
inclusion or exclusion criteria used, as inappropriate criteria
e.g. age, sex, socio-economic classification. Methods used,
can dramatically shift how representative the sample is of
if any, should be well described so that the results from the
the target population [8,10,13].
analyses can be interpreted. This is important as non-
Selection bias can occur if every unit in the sample frame responders may be from a specific group, which can lead to
doesn’t have an equal chance of been included in the final a shift in the baseline data away from that group. This shift
study [11,14]. Randomisation is used to ensure that each can lead to results that don’t represent the target population.
participant in the sampling frame has an equal chance of In some situations the sampling frame doesn’t have a finite
being included in the sample. If methods of randomisation list or a fully defined baseline population. This also makes it
are not used, not described or are not truly random, this may difficult, and in some cases impossible, to quantify non-
lead to a non-representative sample being selected and response and it may be inappropriate to do so in these
hence affect the results of the study [10,11]. situations. If the researchers are using non-defined
populations this should also be declared clearly in the
There are many other situational issues to take into account materials and methods section [15,16].
when determining if the population in the sample is likely to
represent the target population. Often these issues are Measurement Validity & Reliability
outside the control of the researcher, but sometimes are
Measurement validity is a gauge of how accurately the study
overlooked. One such issue is the healthy worker effect
measurements used assess the concepts that the researcher is
which is a well-known phenomenon in human cross-
attempting to explore. Measurement reliability is a gauge of
sectional studies [13]. An example of this is, a researcher
the accuracy of the measurements taken or the procedures
trying to do a cross-sectional study to determine health
used during the study. Question 8 is used to address the
factors in a factory population and decides to sample from
concepts of measurement validity, and is specifically aimed
workers at work on a particular day. Unfortunately there is a
to address the appropriateness of the measurements being
tendency to over select healthy workers as ill workers may
used.
tend to be at home on the day of selection. This will in turn
The importance of measurement validity is that it gives difference for the reader when interpreting the results and
weight to applying the statistical inferences from the study the discussion as they may be unsure if the correct methods
to members of the target population. If inappropriate are being used.
measures are used in the study it could lead to
misclassification bias and it will be difficult to determine to Results
what extent the study results are relevant to the target
population [12,17]. The results section of a paper is solely for the purpose of
declaring the results of the data analysis and no opinion
Question 9 is an attempt to gauge the measurement should be stated in this section. This gives the reader the
reliability of the study measures. Measurements must be opportunity to examine the results unhindered by the
able to be reproduced and produce identical results if opinion of the researcher. It is important for the reader to
measured repeatedly, so that the measurements would be form their own ideas or opinions about the results before
exactly the same if performed by another researcher. With progressing to the discussion stages.
this in mind, the measurements must be of international or
globally accepted standards (e.g. IU standards) where Basic Data
possible and appropriate. If they are being used for the first
time they must be trialled, or in the case of questionnaires, Question 12 asks for a description of the basic data. Basic
they should be piloted before being used. descriptive analysis aims to summarise the data, giving
detailed information about the sample and the measurements
Statistics taken in the study. The basic data gives an overview of the
process of recruitment and if the sampling methods used to
While interpretation of statistics can be quite difficult, a recruit individuals were successful in selecting a
basic understanding of statistics can help you to assess the representative sample of the target population. If the
quality of the paper. Often
many different methods can be Figure 2
used correctly to test the same
data, but as there is such a
wide range available, knowing
what tests are most appropriate
in particular situations can be
hard to decipher. There is an
expectation that the researcher
has this understanding or has at
least sought statistical
assistance to ensure that the
correct methods are used. Therefore for question 10 the sampling methods are unsuccessful in selecting a
emphasis for the reader is that the statistical methods, representative sample of the target population, those
software packages used and the statistical significance levels participants included in the study can often be different to
are clearly stated even if the paper is just presenting the target population; this leads to inaccurate estimates of
descriptive statistics. The statistical significance level is prevalence, incidence or risk factors for disease. Descriptive
usually described as a p-value. In most cases the p-value, at data of the measurements taken in the study give an
which the null hypothesis is rejected, is set at 0.05. The overview of any differences between the groups, and may
higher the p-value is set the greater the possibility of give insight into some of the reasons for statistical
introducing a type I error. Confidence intervals should also inferences that are made later in the paper.
be declared with p-values or instead of p-values as an
indication of the precision of the estimates. It is usual to Response Rate
present a confidence interval of 95% which means that the As stated previously it can often be difficult to deal with
researchers were 95 per cent confident that the true non-responders. Question 13 requires that there is some
population value of the outcome lies between these intervals. attempt made to quantify the level of non-response by the
This can be used to compare groups where an overlap would researchers and asks the reader to interpret if the response
suggest no difference and a gap between confidence rate is likely to lead to non-response bias. Question 14 is
intervals would suggest a difference (Figure 2). examining if any information on non-responders was
available and if so were they comparable to those that did
respond as this could help in answering question 13. Non-
Overall Methods response bias occurs if the non-responders are substantially
Question 11 asks if the methods are sufficiently described to different to the rest of the population in the sample [15].
enable them to be repeated. If there are sections or even
small pieces of information missing it could make a great
Internally Consistent Results
Question 15 is an exploration of the basic data and asks that controversial discoveries from the study and may therefore
the reader spends some time exploring the numbers given in be giving a biased overview of the research conducted.
the results; in the text, figures and tables. Information about
the level of missing data should also be declared in the Selection Bias
results. It is important to check that the numbers add up in
the tables and the text. If the study has recruited 100 There is an expectation that the researcher discusses
participants, the tables and the text should include data about selection biases and takes these into account when
100 participants. If not, the missing data should be clearly interpreting the results of the study. This also gives a clear
declared and the reason for its non-appearance explained. view of whether the researcher has an overall understanding
of the study design. (See notes on selection bias in the
Comprehensive Description of Results methods section).

It is important to check that all the methods described Non-response

previously lead to data in the results section (question 16). Was there an interpretation of the results that included non-
Sometimes the results from all analyses are not described. If response? This is particularly important if the response rate
this is noted it will be unclear whether the researcher found was low, as non-responders may be a specific group, and
non-significant results or just didn’t describe what was lead to a shift in the baseline data (See notes on non-
found. If there are results missing that you would expect to response in the methods section).
find, there is a concern that these missing results may not
have been what the researcher wanted to see and hence the Confounding
authors have omitted them. It is also important that the
significance level declared in the methods is adhered to. As Confounding is a major threat to the validity of practical
the reader, it is important to watch out for phrases such as inferences made from statistical analyses about cause and
“tended towards significance” in the text, and if these are effect. Confounding occurs when the outcome of interest is
used to pay close attention to the results. associated with two different independent variables and one
of those variables is closely associated with the outcome
Discussion only because it is closely associated with the other variable
(confounder). This can sometimes be accounted for using
The discussion of a paper should summarise key results of statistical methods however sometimes these associations
the study objectives. It should give an overall interpretation are missed because the confounder isn’t measured or isn’t
of the results of the study keeping in mind the limitations considered to be a confounder in the analyses. What then
and the external validity of the document. The discussion happens is an erroneous conclusion is made; that the
section should also address both significant and non- variable might have a causal relationship with the outcome.
significant findings of the study and make comparisons The researcher should consider confounding both in the
with other research, citing their sources [2,4]. analyses and in the interpretation of the results [18]. An
example would be where in a study on cancer a researcher
Justified Discussions and Conclusions concludes that increased alcohol intake causes lung cancer;
In question 17 there is an expectation that the researcher however there was confounding in the sample that the
gives an overall summary of the main findings of the study researcher didn’t discover. People in the study that were
and discusses these in detail. It is important that the reader inclined to drink more alcohol were also inclined to smoke
considers the study as a whole when reading the researcher’s more (the confounder) and smoking was the cause of lung
conclusion. If the researcher’s conclusion is different or is cancer not increased alcohol intake. Similarly, a study was
more definitive than the study suggests it should be, it can undertaken to examine surgical deaths in cats. The
be an indication that the researcher has misunderstood their researcher concluded that cats that had gaseous anaesthesia
own study or has other motives or interests for coming to were more likely to die during surgery than those that had
that conclusion. just injectable anaesthesia. There was confounding in the
sample: cats that underwent surgery using gaseous
It is up to the reader to explore the discussion fully in order anaesthesia were more likely to be ill or undergoing major
to answer question 17. The following points should be taken surgical procedures (the confounders) and this was the cause
into account: for cats being more likely to die during surgery and not the
use of gaseous anaesthetics.
Aim
Non-significant Results
In the discussion section the researcher should discuss all
results that pertain to the overall aim of the study, even if Discussing non-significant results is as important as
they are not significant. If some results are overlooked in the discussing significant results and should also be included in
discussion it could suggest that the researcher either doesn’t the discussion, especially if they have a direct association
believe the results, or doesn’t want to draw attention to with the aim being investigated. Non-significant results can
be influenced by factors associated with study design and
sample size. If there are biases introduced during the study NY [etc.]: Erlbaum 2008. 317–
design this can lead to non-significant results that in reality 86.http://library.wur.nl/WebQuery/clc/1877609 (accessed 26
may be significant (this can work the other way around as Nov2012).
well). If there are only small differences between groups, 6 Rothman KJ. Chapter 4: Types of Epidemiologic Study. In:
non-significant results may be apparent because the sample Epidemiology: An Introduction. New York: : Oxford
University Press 2002. 57–93.
size is too small (see sample size justification). Again it is
important that the researcher has a clear understanding of 7 Woodward M. Chapter 8: Sample Size Determination. In:
Epidemiology: Study Design and Data Analysis, Second
this and conveys that in the discussion. Edition. Chapman and Hall/CRC 2004. 381–426.

Limitations 8 Groves RM, Fowler FJJ, Couper MP, et al. Chapter 1: An

Introduction to Survey Methodology. In: Survey methodology.
Hoboken, NJ: 2004. 1–
In question 18 we explore whether limitations are discussed.
37.http://www.loc.gov/catdir/toc/wiley041/2004044064.html
Unfortunately all forms of research have some limitations. http://www.loc.gov/catdir/bios/wiley047/2004044064.html
The question here is whether the researcher has an http://www.loc.gov/catdir/description/wiley041/2004044064.
understanding of the limitations involved in their study html
design. If this issue is not explored, this is cause for concern 9 Lohr SL. Chapter 6: Coverage and Sampling. In: Leeuw ED,
that the limitations don’t stop at the design and that the Hox JJ, Dillman DA, eds. International handbook of survey
methodology. New York, NY [etc.]: Erlbaum 2008. 97–
researcher has a poor understanding of the study as a whole. 112.http://library.wur.nl/WebQuery/clc/1877609 (accessed 26
Nov2012).
Other 10 Woodward M. Chapter 1: Fundamental Issues. In:
Epidemiology: Study Design and Data Analysis,
Conflicts of Interest Second Edition. Chapman and Hall/CRC 2004. 1–32.
11 Groves RM, Fowler FJJ, Couper MP, et al. Chapter 3: Target
It is very important that conflicts of interest or bodies
Populations, Sampling Frames and Coverage Error. In:
involved in funding the study are declared in the text Survey methodology. Hoboken, NJ: 2004. 67–
(question 19). This can give an impression as to background 92.http://www.loc.gov/catdir/toc/wiley041/2004044064.html
reasons for carrying out the study. Where studies are funded http://www.loc.gov/catdir/bios/wiley047/2004044064.html
http://www.loc.gov/catdir/description/wiley041/2004044064.
by a specific agency the researcher may unconsciously html
interpret in favour of the agencies’ ideals; if the researcher
12 Dohoo IR, Martin SW, Stryhn H. Capther 2: Sampling. In:
has worked in a specific area their own ideas and beliefs Veterinary Epidemiologic Research. Charlottetown, Canada: :
may affect the interpretation of the results. It is up to the Ver Books 2009. 27–52.
reader to identify these and come to the conclusion as to 13 Rothman KJ. Chapter 5: Bias in Study Design. In:
whether these conflicts of interest are relevant or not. This Epidemiology: An Introduction. New York: : Oxford
can be declared in different areas of the text and should be University Press 2002. 94–112.
stated. 14 Bruce N, Pope D, Stanistreet D. Quantitative Methods for
Health Research: A Practical Interactive Guide to
Ethical Approval Epidemiology and Statistics. 1st ed. Wiley-Blackwell 2008.
15 Groves RM, Fowler FJJ, Couper MP, et al. Chapter 2:
Question 20 deals with ethical approval and participant Inference and Error in Surveys. In: Survey methodology.
consent. It is important that these are sought before carrying Hoboken, NJ: 2004. 39–
out research on any animal or person. 65.http://www.loc.gov/catdir/toc/wiley041/2004044064.html
http://www.loc.gov/catdir/bios/wiley047/2004044064.html
http://www.loc.gov/catdir/description/wiley041/2004044064.
html

References: 16 Lynn P. Chapter 3: The Problem of Nonresponse. In: Leeuw

ED, Hox JJ, Dillman DA, eds. International handbook of
1 Young JM, Solomon MJ. How to critically appraise an article. survey methodology. New York, NY [etc.]: Erlbaum 2008.
Nature Clinical Practice Gastroenterology & Hepatology 35–96.http://library.wur.nl/WebQuery/clc/1877609 (accessed
2009;6:82–91. doi:10.1038/ncpgasthep1331 26 Nov2012).

2 Lebrun J-L. Scientific Writing 2.0: A Reader and 17 Hox JJ. Chapter 20: Accommodating Measurement Errors. In:
Writers Guide. World Scientific 2011. Leeuw ED, Hox JJ, Dillman DA, eds. International handbook
of survey methodology. New York, NY [etc.]: Erlbaum 2008.
3 Masic I. How to Search, Write, Prepare and Publish the http://library.wur.nl/WebQuery/clc/1877609 (accessed 26
Scientific Papers in the Biomedical Journals. Acta Nov2012).
Inform Med 2011;19:68–79.
doi:10.5455/aim.2011.19.68-79 18 Rothman KJ. Epidemiology: An Introduction. New York: :
Oxford University Press 2002.
4 Von Elm E, Altman DG, Egger M, et al. The Strengthening
the Reporting of Observational Studies in Epidemiology
(STROBE) statement: guidelines for reporting observational
studies. The Lancet 2007;370:1453–7. doi:10.1016/S0140-
6736(07)61602-X
5 Leeuw ED, Hox JJ, Dillman DA. Data Analysis. In:
International handbook of survey methodology. New York,