Internal 2012 - Software and model selection challenges in meta-analysis

[Poster title]
[Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4
1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]
Meta-analysis overview
Two-stage meta-analysis
One-stage meta-analysis
Summary
Software and model selection challenges in
meta-analysis
MetaEasy, model assumptions, homogeneity and IPD
Evan Kontopantelis David Reeves
Centre for Primary Care
Institute of Population Health
University of Manchester
Health Methodology Research Seminar
Manchester, 6 Nov 2012
Kontopantelis, Reeves Software and model selection challenges in meta-analysis

[Poster title]
Summary
Outline
1 Meta-analysis overview
2 Two-stage meta-analysis
3 One-stage meta-analysis
4 Summary

[Poster title]
Summary
The heterogeneity issue
Outline
4 Summary

[Poster title]
Summary
Timeline
‘Meta’ is a Greek preposition meaning ‘after’, so
meta-analysis =⇒ post-analysis
Efforts to pool results from individual studies back as far as
1904
The first attempt that assessed a therapeutic intervention
was published in 1955
In 1976 Glass first used the term to describe a "statistical
analysis of a large collection of analysis results from
individual studies for the purpose of integrating the
findings"

[Poster title]
Summary
Meta-analysing reported study results
A two-stage process
the relevant summary effect statistics are extracted from
published papers on the included studies
these are then combined into an overall effect estimate
using a suitable meta-analysis model
However, problems often arise
papers do not report all the statistical information required
as input
papers report a statistic other than the effect size which
needs to be transformed with a loss of precision
a study might be too different to be included (population
clinically heterogeneous)

[Poster title]
Summary
Individual Patient Data
IPD
These problems can be avoided when IPD from each
study are available
outcomes can be easily standardised
clinical heterogeneity can be addressed with subgroup
analyses and patient-level covariate controlling
Can be analysed in a single- or two-stage process
mixed-effects regression models can be used to combine
information across studies in a single stage
this is currently the best approach, with the two-stage
method being at best equivalent in certain scenarios

[Poster title]
Summary
Outline
4 Summary

[Poster title]
Summary
When Robin ruins the party...

[Poster title]
Summary
Heterogeneity
tread lightly!
When the effect of the intervention varies signiﬁcantly from
one study to another
It can be attributed to clinical and/or methodological
diversity
Clinical: variability that arises from different populations,
interventions, outcomes and follow-up times
Methodological: relates to differences in trial design and
quality
Detecting quantifying and dealing with heterogeneity can
be very hard

[Poster title]
Summary
Absence of heterogeneity
Assumes that the
true effects of the
studies are all
equal and
deviations occur
because of
imprecision of
results
Analysed with the
ﬁxed-effects
method

[Poster title]
Summary
Presence of heterogeneity
Assumes that
there is variation in
the size of the true
effect among
studies (in addition
to the imprecision
of results)
Analysed with
random-effects
methods

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Methods and performance
ˆτ2
= 0
Outline
4 Summary

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Challenges with two-stage meta-analysis
Heterogeneity is common and the fixed-effect model is
under fire
Methods are asymptotic: accuracy improves as studies
increase. But what if we only have a handful, as is usually
the case?
Almost all random-effects models (except Profile
Likelihood) do not take into account the uncertainty in ˆτ2.
Is this, practically, a problem?
DerSimonian-Laird is the most common method of
analysis, since it is easy to implement and widely available,
but is it the best?

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Challenges with two-stage meta-analysis
...continued
Can be difﬁcult to organise since...
outcomes likely to have been disseminated using a variety
of statistical parameters
appropriate transformations to a common format required
tedious task, requiring at least some statistical adeptness
Parametric random-effects models assume that both the
effects and errors are normally distributed. Are methods
robust?
Sometimes heterogeneity is estimated to be zero,
especially when the number of studies is small. Good
news?

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Outline
the MetaEasy add-in
metaeff & metaan
ˆτ2 = 0
4 Summary

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Organising
Data initially collected using data extraction forms
A spreadsheet is the next logical step to summarise the
reported study outcomes and identify missing data
Since in most cases MS Excel will be used we developed
an add-in that can help with most processes involved in
meta-analysis
More useful when the need to combine differently reported
outcomes arises

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
What it can do
Help with the data collection using pre-formatted
worksheets.
Its unique feature, which can be supplementary to other
meta-analysis software, is implementation of methods for
calculating effect sizes (& SEs) from different input types
For each outcome of each study...
it identiﬁes which methods can be used
calculates an effect size and its standard error
selects the most precise method for each outcome

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
What it can do
...continued
Creates a forest plot that summarises all the outcomes,
organised by study
Uses a variety of standard and advanced meta-analysis
methods to calculate an overall effect
a variety of options is available for selecting which
outcome(s) are to be meta-analysed from each study
Plots the results in a second forest plot
Reports a variety of heterogeneity measures, including
Cochran’s Q, I2, HM
2
and ˆτ2 (and its estimated conﬁdence
interval under the Proﬁle Likelihood method)

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Advantages
Free (provided Microsoft Excel is available)
Easy to use and time saving
Extracted data from each study are easily accessible, can
be quickly edited or corrected and analysis repeated
Choice of many meta-analysis models, including some
advanced methods not currently available in other software
packages (e.g. Permutations, Proﬁle Likelihood, REML)
Unique forest plot that allows multiple outcomes per study
Effect sizes and standard errors can be exported for use in
other meta-analysis software packages

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Installing
Latest version available from www.statanalysis.co.uk
Compatible with Excel 2003, 2007 and 2010
Manual provided but also described in:
Kontopantelis E and Reeves D
MetaEasy: A Meta-Analysis Add-In for Microsoft Excel
Journal of Statistical Software, 30(7):1-25, 2009
Play video clip

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Stata implementation
MetaEasy methods implemented in Stata under:
metaeff, which uses the different study input to provide
effect sizes and SEs
metaan, which meta-analyses the study effects with a
ﬁxed-effect or one of ﬁve available random-effects models
To install, type in Stata:
ssc install <command name>
help <command name>
Described in:
metaan: Random-effects meta-analysis
The Stata Journal, 10(3):395-407, 2010

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Many random-effects methods
which to use?
DerSimonian-Laird (DL): Moment-based estimator of both
within and between-study variance
Maximum Likelihood (ML): Improves the variance estimate
using iteration
Restricted Maximum Likelihood (REML): an ML variation
that uses a likelihood function calculated from a
transformed set of data
Proﬁle Likelihood (PL): A more advanced version of ML
that uses nested iterations for converging
Permutations method (PE): Simulates the distribution of
the overall effect using the observed data

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Performance evaluation
our approach
Simulated various distributions for the true
effects:
Normal
Skew-Normal
Uniform
Bimodal
Created datasets of 10,000
meta-analyses for various numbers of
studies and different degrees of
heterogeneity, for each distribution

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Performance evaluation
our approach
Compared all methods in terms of:
Coverage, the rate of true negatives when the overall true
effect is zero
Power, the rate of true positives when the true overall effect
is non-zero
Conﬁdence Interval performance, a measure of how wide
the (estimated around the effect) CI is, compared to its true
width.

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Homogeneity
Zero between study variance
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
zero between study variance
Coverage
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
Power (25th centile)
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
Overall estimation performance

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Coverage performance
Small and large heterogeneity under various distributional assumptions
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
H²=1.18 − I²=15% (normal distribution)
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
H²=1.18 − I²=15% (skew−normal distribution)
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
H=1.18 − I=15% (bimodal distribution)
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
H=1.18 − I=15% (uniform distribution)
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
Coverage
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
Coverage

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Power performance
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
CI performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
Confidence Interval performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
Overall estimation performance (medians)
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=0, ku=3
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk=2, ku=9
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE
sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1
Overall estimation performance (medians)
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
FE DL ML
REML PL PE

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Coverage by method
Large heterogeneity across various between-study variance distributions
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Zero variance Normal Skew−normal
Bimodal Uniform
H=2.78 − I=64%
FE − Coverage
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
DL − Coverage
0.70
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
ML − Coverage
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
REML − Coverage
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PL − Coverage
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PE − Coverage

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Power by method
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
FE − Power (25th centile)
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
DL − Power (25th centile)
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
ML − Power (25th centile)
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
REML − Power (25th centile)
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PL − Power (25th centile)
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PE − Power (25th centile)

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
CI performance by method
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
FE − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
DL − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
ML − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
REML − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PL − Overall estimation performance
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PE − Overall estimation performance

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Which method then?
Within any given method, the results were consistent
across all types of distribution shape
Therefore methods are highly robust against even severe
violations of the assumption of normality
Choose PE if the priority is an accurate Type I error rate
(false positive)
But low power makes it a poor choice when control of the
Type II error rate (false negative) is also important and it
cannot be used with less than 6 studies

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Which method then?
For very small study numbers (≤5) only PL gives coverage
>90% and an acccurate CI
PL has a ‘reasonable’ coverage in most situations,
especially for moderate and large heterogeneiry, giving it
an edge over other methods
REML and DL perform similarly and better than PL only
when heterogeneity is low (I2 < 15%)
The computational complexity of REML is not justiﬁed

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
Bring on the champagne?
Does not necessarily mean homogeneity
Most methods use biased estimators and not uncommon
to get a negative ˆτ2 which is set to 0 by the model
We identiﬁed a large percentage of cases where the
estimators failed to identify existing heterogeneity
In our simulations, for 5 studies and I2 ≈ 29%:
30% of the meta-analyses were erroneously estimated to
be homogeneous under the DL method
32% for REML and 48% for ML-PL

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
ˆτ2
= 0
What does it mean?
In these cases coverage was substandard and was over
10% lower than in cases where ˆτ2 > 0, on average
The problem becomes less profound as the number of
studies and the level of heterogeneity increase
Better estimators are needed
There might be a large number of meta-analyses of
‘homogeneous’ studies which have reached a wrong
conclusion

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Outline
4 Summary

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Forest plot
One advantage of two-stage meta-analysis is the ability to
convey information graphically through a forest plot
study effects available after the ﬁrst stage of the process,
and can be used to demonstrate the relative strength of the
intervention in each study and across all
informative, easy to follow and particularly useful for
readers with little or no methodological experience
key feature of meta-analysis and always presented when
two-stage meta-analyses are performed
In one-stage meta-analysis, only the overall effect is
calculated and creating a forest-plot is not straightforward
Enter ipdforest

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Outline
A practical guide
ipdforest methods
ipdforest examples
Power
4 Summary

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
The hypothetical study
Individual patient data from randomised controlled trials
For each trial we have
a binary control/intervention membership variable
baseline and follow-up data for the continuous outcome
covariates
Assume measurements consistent across trials and
standardisation is not required
We will explore linear random-effects models with the
xtmixed command; application to the logistic case using
xtmelogit should be straightforward
In the models that follow, in general, we denote ﬁxed
effects with ‘γ’s and random effects with ‘β’s

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Model 1
fixed common intercept; random treatment effect; fixed effect for baseline
Yij = γ0 + β1jgroupij + γ2Ybij + ij ij ∼ N(0, σ2
j )
β1j = γ1 + u1j u1j ∼ N(0, τ1
2)
i: the patient
j: the trial
Yij: the outcome
γ0: fixed common intercept
β1j: random treatment
effect for trial j
γ1: mean treatment effect
groupij: group membership
γ2: fixed baseline effect
Ybij: baseline score
u1j: random treatment
effect for trial j
τ1
2: between trial variance
ij: error term
σ2
j : within trial variance for
trial j

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Model 1
fixed common intercept; random treatment effect; fixed effect for baseline
Possibly the simplest approach
In Stata it can be expressed as
xtmixed Y i.group Yb || studyid:group,
nocons
where
studyid, the trial identifier
group, control/intervention membership
Y and Yb, endpoint and baseline scores
note that the nocons option suppresses estimation of the
intercept as a random effect

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Model 2
fixed trial specific intercepts; random treatment effect; fixed trial-specific effects for
baseline
Common intercept & fixed baseline difficult to justify
A more accepted model allows for different fixed intercepts
and fixed baseline effects for each trial:
Yi j = γ0j + β1j groupi j + γ2j Ybi j + i j
β1j = γ1 + u1j
where
γ0j the fixed intercept for trial j
γ2j the fixed baseline effect for trial j
In Stata expressed as:
xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4
|| studyid:group, nocons
where Yb‘i’=Yb if studyid=‘i’ and zero otherwise

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Model 3
random trial intercept; random treatment effect; ﬁxed trial-speciﬁc effects for baseline
Another possibility, althought contentious, is to assume trial
intercepts are random (e.g. multi-centre trial):
Yi j = β0j + β1j groupi j + γ2j Ybi j + i j
β0j = γ0 + u0j
β1j = γ1 + u1j
wiser to assume random effects correlation ρ = 0:
i j ∼ N(0, σ2
j ) u0j ∼ N(0, τ2
0 )
u1j ∼ N(0, τ2
1 ) cov(u0j , u1j ) = ρτ0τ1
xtmixed Y i.group Yb1 Yb2 Yb3 Yb4 ||
studyid:group, cov(uns)
cov(uns): allows for distinct estimation of all RE
variance-covariance components

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Model 4
random trial intercept; random treatment effect; random effects for baseline
The baseline could also have been modelled as a
random-effect:
Yi j = β0j + β1j groupi j + β2j Ybi j + i j
β0j = γ0 + u0j
β1j = γ1 + u1j
β2j = γ2 + u2j
as before, non-zero random effects correlations:
u0j ∼ N(0, τ2
0 ) u1j ∼ N(0, τ2
1 )
u2j ∼ N(0, τ2
2 ) cov(u0j , u1j ) = ρ1τ0τ1
cov(u0j , u2j ) = ρ2τ0τ2 cov(u1j , u2j ) = ρ3τ1τ2
xtmixed Y i.group Yb || studyid:group Yb,
cov(uns)

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Model 5
Interactions and covariates
A covariate or an interaction term can be modelled as a
ﬁxed or random effect
Assuming continuous and standardised variable age we
can expand Model 2 to include ﬁxed effects for both age
and its interaction with the treatment:
Yi j = γ0j +β1j groupi j +γ2j Ybi j +γ3agei j +γ4groupi j agei j + i j
β1j = γ1 + u1j
xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4
age i.group#c.age || studyid:group, nocons
If modelled as a random effect, non-convergence issues
more likely to be encountered

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
General
ipdforest is issued following an IPD meta-analysis that
uses mixed effects two-level regression, with patients
nested within trials and a
linear model (xtmixed)
or
logistic model (xtmelogit)
Provides a meta-analysis summary table and a forest plot
Trial effects are calculated within ipdforest
Can calculate and report both main and interaction effects
Overall effect(s) and variance estimates are extracted from
the preceding regression

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Process
ipdforest estimates individual trial effects and their
standard errors using one-level linear or logistic
regressions
Following xtmixed, regress is used and following
xtmelogit, logit is used, for each trial
ipdforest controls these regressions for fixed- or
random-effects covariates that were specified in the
preceding two-level regression
User has full control over included covariates in the
command (e.g. specification as fixed- or random-effects)
But we strongly recommend using the same specifications

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Estimation details
In the estimation of individual trial effects, ipdforest
controls for a random-effects covariate (i.e. allowing the
regression coefficient to vary by trial) by including the
covariate as an independent variable in each regression
Control for a fixed-effect covariate (regression coefficient
assumed constant across trials and given by the coefficient
estimated under two-level model) is a little more complex.
Not possible to specify a fixed value for a regression
coefficient under regress and the continuous outcome
variable is adjusted by subtracting the contribution of the
fixed covariates to its values in a first step prior to analysis
For a binary outcome the equivalent is achieved through
use of the offset option in logit

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Reported heterogeneity
For continuous outcomes, ipdforest reports, I2 and H2
M,
based on the xtmixed output
For binary outcomes, an estimate of the within-trial
variance is not reported under xtmelogit and hence
heterogeneity measures cannot be computed
Between-trial variability estimate ˆτ2 and its conﬁdence
interval is reported under both models
Random-effects model approach is more conservative,
accounting for even low levels of τ2
Described in:
A short guide and a forest plot command (ipdforest) for
one-stage meta-analysis
The Stata Journal, in print

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Depression intervention
We apply the ipdforest command to a dataset of 4
depression intervention trials
Complete information in terms of age, gender,
control/intervention group membership, continuous
outcome baseline and endpoint values for 518 patients
Results not published yet; we use fake author names and
generated random continuous & binary outcome variables,
while keeping the covariates at their actual values
Introduced correlation between baseline and endpoint
scores and between-trial variability
Logistic IPD meta-analysis, followed by ipdforest

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Dataset
. use ipdforest_example.dta,
. describe
Contains data from ipdforest_example.dta
obs: 518
vars: 17 6 Feb 2012 11:14
size: 20,202
storage display value
variable name type format label variable label
studyid byte %22.0g stid Study identifier
patid int %8.0g Patient identifier
group byte %20.0g grplbl Intervention/control group
sex byte %10.0g sexlbl Gender
age float %10.0g Age in years
depB byte %9.0g Binary outcome, endpoint
depBbas byte %9.0g Binary outcome, baseline
depBbas1 byte %9.0g Bin outcome baseline, trial 1
depC float %9.0g Continuous outcome, endpoint
depCbas float %9.0g Continuous outcome, baseline
depCbas1 float %9.0g Cont outcome baseline, trial 1
Sorted by: studyid patid

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
ME logistic regression model - continuous interaction
ﬁxed trial intercepts; ﬁxed trial effects for baseline; random treatment and age effects
. xtmelogit depB group agec sex i.studyid depBbas1 depBbas2 depBbas5 depBbas9 i
> .group#c.agec || studyid:group agec, var nocons or
Mixed-effects logistic regression Number of obs = 518
Group variable: studyid Number of groups = 4
Obs per group: min = 42
avg = 129.5
max = 214
Integration points = 7 Wald chi2(11) = 42.06
Log likelihood = -326.55747 Prob > chi2 = 0.0000
depB Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
group 1.840804 .3666167 3.06 0.002 1.245894 2.71978
agec .9867902 .0119059 -1.10 0.270 .9637288 1.010403
sex .7117592 .1540753 -1.57 0.116 .4656639 1.087912
studyid
2 1.050007 .5725516 0.09 0.929 .3606166 3.057303
5 .8014551 .5894511 -0.30 0.763 .189601 3.387799
9 1.281413 .6886057 0.46 0.644 .4469619 3.673735
depBbas1 3.152908 1.495281 2.42 0.015 1.244587 7.987253
depBbas2 4.480302 1.863908 3.60 0.000 1.982385 10.12574
depBbas5 2.387336 1.722993 1.21 0.228 .5802064 9.823007
depBbas9 1.881203 .7086507 1.68 0.093 .8990569 3.936262
group#c.agec
1 1.011776 .0163748 0.72 0.469 .9801858 1.044385
_cons .5533714 .2398342 -1.37 0.172 .2366472 1.293993
Random-effects Parameters Estimate Std. Err. [95% Conf. Interval]
studyid: Independent
var(group) 8.86e-21 2.43e-11 0 .
var(agec) 5.99e-18 4.40e-11 0 .

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
ipdforest
modelling main effect and interaction
. ipdforest group, fe(sex) re(agec) ia(agec) or
One-stage meta-analysis results using xtmelogit (ML method) and ipdforest
Main effect (group)
Study Effect [95% Conf. Interval] % Weight
Hart 2005 2.118 0.942 4.765 19.88
Richards 2004 2.722 1.336 5.545 30.69
Silva 2008 2.690 0.748 9.676 8.11
Kompany 2009 1.895 0.969 3.707 41.31
Overall effect 1.841 1.246 2.720 100.00
Interaction effect (group x agec)
Hart 2005 0.972 0.901 1.049 19.88
Richards 2004 0.995 0.937 1.055 30.69
Silva 2008 0.987 0.888 1.098 8.11
Kompany 2009 1.077 1.015 1.144 41.31
Overall effect 1.012 0.980 1.044 100.00
Heterogeneity Measures
value [95% Conf. Interval]
I^2 (%) .
H^2 .
tau^2 est 0.000 0.000 .

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Forest plots
main effect and interaction
Overall effect
Kompany 2009
Silva 2008
Richards 2004
Hart 2005
Studies
0 2 3 4 5 6 7 8 9 101
Effect sizes and CIs (ORs)
Main effect (group)
Overall effect
Kompany 2009
Silva 2008
Richards 2004
Hart 2005
Studies
0 .2 .4 .6 .8 1.2 1.41
Interaction effect (group x agec)

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
ME logistic regression model - binary interaction
ﬁxed trial intercepts; ﬁxed trial effects for baseline & age cat; random treatment effect
. xtmelogit depB group i.agecat sex i.studyid depBbas1 depBbas2 depBbas5 depBba
> s9 i.group#i.agecat || studyid:group, var nocons or
Mixed-effects logistic regression Number of obs = 518
Group variable: studyid Number of groups = 4
Obs per group: min = 42
avg = 129.5
max = 214
Integration points = 7 Wald chi2(11) = 42.67
Log likelihood = -326.24961 Prob > chi2 = 0.0000
depB Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
group 1.931763 .5702338 2.23 0.026 1.083151 3.445233
1.agecat .7389353 .213031 -1.05 0.294 .4199616 1.300179
sex .7173044 .154959 -1.54 0.124 .469698 1.095439
studyid
2 1.029638 .5608887 0.05 0.957 .3539959 2.994823
5 .828577 .6082301 -0.26 0.798 .1965599 3.492777
9 1.266728 .6812765 0.44 0.660 .4414556 3.634794
depBbas1 3.196139 1.515334 2.45 0.014 1.261999 8.094542
depBbas2 4.625802 1.923028 3.68 0.000 2.047989 10.44832
depBbas5 2.354493 1.692149 1.19 0.233 .5756364 9.630454
depBbas9 1.902359 .7183054 1.70 0.089 .9075907 3.987446
group#agecat
1 0 .9277371 .3558053 -0.20 0.845 .4374944 1.967331
1 1 1 (omitted)
_cons .6175744 .2800575 -1.06 0.288 .253914 1.502076
Random-effects Parameters Estimate Std. Err. [95% Conf. Interval]
studyid: Identity
var(group) 2.24e-19 1.24e-10 0 .

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
ipdforest
modelling main effect and interaction
. ipdforest group, fe(sex i.agecat) ia(i.agecat) or
Main effect (group), agecat=0
Hart 2005 3.594 1.133 11.402 19.88
Richards 2004 2.814 1.034 7.657 30.69
Silva 2008 3.750 0.585 24.038 8.11
Kompany 2009 1.010 0.475 2.147 41.31
Overall effect 1.792 1.079 2.976 100.00
Hart 2005 1.166 0.376 3.617 19.88
Richards 2004 2.566 1.007 6.543 30.69
Silva 2008 1.935 0.340 11.003 8.11
Kompany 2009 3.551 1.134 11.126 41.31
Overall effect 1.932 1.083 3.445 100.00
Heterogeneity Measures
value [95% Conf. Interval]
I^2 (%) .
H^2 .
tau^2 est 0.000 0.000 .

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Forest plots
main effect for each age group
Overall effect
Meyer 2009
Lovell 2008
Willemse 2004
Mead 2005
Studies
0 2 4 6 8 10 12 14 16 18 20 22 24 261
Overall effect
Meyer 2009
Lovell 2008
Willemse 2004
Mead 2005
Studies
0 2 4 6 8 10 121

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Power calculations
to detect a moderator effect
The best approach is through simulations
As always, numerous assumptions need to be made
effect sizes (main and interaction)
exposure and covariate distributions
correlation between variables
within-study (error) variance
between-study (error) variance - ICC
Generate 100s of data sets using the assumed model(s)
Estimate what % of these give a signiﬁcant p-value for the
interaction
Can be trial and error till desired power level achieved

[Poster title]
Summary
A practical guide
ipdforest methods
ipdforest examples
Power
Research on power calculations with simulations
The Stata Journal (2002)
2, Number 2, pp. 107–124
Power by simulation
A. H. Feiveson
NASA Johnson Space Center
alan.h.feiveson1@jsc.nasa.gov
Abstract. This paper describes how to write Stata programs to estimate the power
of virtually any statistical test that Stata can perform. Examples given include
the t test, Poisson regression, Cox regression, and the nonparametric rank-sum
test.
Keywords: st0010, power, simulation, random number generation, postfile, copula,
sample size
1 Introduction
Statisticians know that in order to properly design a study producing experimental
data, one needs to have some idea of whether the scope of the study is sufficient to give
a reasonable expectation that hypothesized effects will be detectable over experimental
error. Most of us have at some time used published procedures or canned software
to obtain sample sizes for studies intended to be analyzed by t tests, or even analysis
of variance. However, with more complex methods for describing and analyzing both
continuous and discrete data (for example, generalized linear models, survival models,
selection models), possibly with provisions for random effects and robust variance es-
timation, the closed-form expressions for power or for sample sizes needed to achieve
a certain power do not exist. Nevertheless, Stata users with moderate programming
ability can write their own routines to estimate the power of virtually any statistical
test that Stata can perform.
2 General approach to estimating power
2.1 Statistical inference
Before describing methodology for estimating power, we first define the term “power”
and illustrate the quantities that can affect it. In this article, we restrict ourselves
to classical (as opposed to Bayesian) methodology for performing statistical inference
on the effect of experimental variable(s) X on a response Y . This is accomplished by
calculating a p-value that attempts to probabilistically describe the extent to which the
data are consistent with a null hypothesis, H0, that X has no effect on Y . We would
then reject H0 if the p-value is less than a predesignated threshold α. It should be
pointed out that considerable criticism of the use of p-values for statistical inference has
appeared in the literature; for example, Berger and Sellke (1987), Loftus (1993), and
COMMENTARY Open Access
Simulation methods to estimate design power:
an overview for applied research
Benjamin F Arnold1*†
, Daniel R Hogan2†
, John M Colford Jr1
and Alan E Hubbard3
Abstract
Background: Estimating the required sample size and statistical power for a study is an integral part of study
design. For standard designs, power equations provide an efficient solution to the problem, but they are
unavailable for many complex study designs that arise in practice. For such complex study designs, computer
simulation is a useful alternative for estimating study power. Although this approach is well known among
statisticians, in our experience many epidemiologists and social scientists are unfamiliar with the technique. This
article aims to address this knowledge gap.
Methods: We review an approach to estimate study power for individual- or cluster-randomized designs using
computer simulation. This flexible approach arises naturally from the model used to derive conventional power
equations, but extends those methods to accommodate arbitrarily complex designs. The method is universally
applicable to a broad range of designs and outcomes, and we present the material in a way that is approachable
for quantitative, applied researchers. We illustrate the method using two examples (one simple, one complex)
based on sanitation and nutritional interventions to improve child growth.
Results: We first show how simulation reproduces conventional power estimates for simple randomized designs
over a broad range of sample scenarios to familiarize the reader with the approach. We then demonstrate how to
extend the simulation approach to more complex designs. Finally, we discuss extensions to the examples in the
article, and provide computer code to efficiently run the example simulations in both R and Stata.
Conclusions: Simulation methods offer a flexible option to estimate statistical power for standard and non-
traditional study designs and parameters of interest. The approach we have described is universally applicable for
evaluating study designs used in epidemiologic and social science research.
Keywords: Computer Simulation, Power, Research Design, Sample Size
Background
Estimating the sample size and statistical power for a
study is an integral part of study design and has profound
consequences for the cost and statistical precision of a
study. There exist analytic (closed-form) power equations
for simple designs such as parallel randomized trials with
treatment assigned at the individual level or cluster
(group) level [1]. Statisticians have also derived equations
to estimate power for more complex designs, such as
designs with two levels of correlation [2] or designs with
two levels of correlation, multiple treatments and
attrition [3]. The advantage of using an equation to esti-
mate power for study designs is that the approach is fast
and easy to implement using existing software. For this
reason, power equations are used to inform most study
designs. However, in our applied research we have routi-
nely encountered study designs that do not conform to
conventional power equations (e.g. multiple treatment
interventions, where one treatment is deployed at the
group level and a second at the individual level). In these
situations, simulation techniques offer a flexible alterna-
tive that is easy to implement in modern statistical
software.
Here, we provide an overview of a general method to
estimate study power for randomized trials based on a
* Correspondence: benarnold@berkeley.edu
† Contributed equally
1
Arnold et al. BMC Medical Research Methodology 2011, 11:94
http://www.biomedcentral.com/1471-2288/11/94

[Poster title]
Summary
MetaEasy can help you organise your meta-analysis and
can be especially useful if you need to combine continuous
and binary outcome.
Methods implemented in Stata under metaeff and metaan
A zero ˆτ2 is a reason to worry; heterogeneity might be
there but we cannot measure or account for in the model
If ˆτ2 > 0, even if very small, use a random-effects model
The DL method works reasonably well, under all
distributions, especially for low levels of heterogeneity
Proﬁle likelihood, which takes into account the uncertaintly
in ˆτ2, works better when I2 ≥ 15%

[Poster title]
Summary
A few different approaches exist for conducting one-stage
IPD meta-analysis
Stata can cope through the xtmixed and the xtmelogit
commands
The ipdforest command aims to help meta-analysts
calculate trial effects
display results in standard meta-analysis tables
produce familiar and ‘expected’ forest-plots
The easiest way to calculate power to detect complex
effects is through simulations

[Poster title]
Appendix
Thank you!
Key references
Comments, suggestions:
e.kontopantelis@manchester.ac.uk

[Poster title]
Appendix
Thank you!
Key references
References
Kontopantelis E, Reeves D.
A short guide and a forest plot command (ipdforest) for one-stage
meta-analysis.
The Stata Journal, in print.
Performance of statistical methods for meta-analysis when true study
effects are non-normally distributed: A simulation study.
Stat Methods Med Res, published online Dec 9 2010.
metaan: Random-effects meta-analysis.
The Stata Journal, 10(3):395-407, 2010.
MetaEasy: A Meta-Analysis Add-In for Microsoft Excel.
Journal of Statistical Software, 30(7):1-25, 2009.

Internal 2012 - Software and model selection challenges in meta-analysis

More Related Content

Viewers also liked

Similar to Internal 2012 - Software and model selection challenges in meta-analysis

More from Evangelos Kontopantelis

Recently uploaded

Internal 2012 - Software and model selection challenges in meta-analysis