See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/51058719

Dimensionality Assessment of Ordered Polytomous Items With Parallel

Analysis

Article in Psychological Methods · June 2011

DOI: 10.1037/a0023353 · Source: PubMed

CITATIONS READS
1,056 2,146

2 authors:

Marieke E. Timmerman Urbano Lorenzo-Seva

University of Groningen Universitat Rovira i Virgili
165 PUBLICATIONS 5,741 CITATIONS 154 PUBLICATIONS 7,505 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Personalidad: medición y contextos de aplicación View project

UPFRONT-study View project

All content following this page was uploaded by Marieke E. Timmerman on 18 July 2018.

The user has requested enhancement of the downloaded file.

Psychological Methods © 2011 American Psychological Association
2011, Vol. 16, No. 2, 209 –220 1082-989X/11/$12.00 DOI: 10.1037/a0023353

Dimensionality Assessment of Ordered Polytomous Items With

Parallel Analysis
Marieke E. Timmerman Urbano Lorenzo-Seva
University of Groningen Rovira i Virgili University

Parallel analysis (PA) is an often-recommended approach for assessment of the dimensionality of a

variable set. PA is known in different variants, which may yield different dimensionality indications.
In this article, the authors considered the most appropriate PA procedure to assess the number of
common factors underlying ordered polytomously scored variables. They proposed minimum rank
factor analysis (MRFA) as an extraction method, rather than the currently applied principal
component analysis (PCA) and principal axes factoring. A simulation study, based on data with
major and minor factors, showed that all procedures consistently point at the number of major
common factors. A polychoric-based PA slightly outperformed a Pearson-based PA, but conver-
gence problems may hamper its empirical application. In empirical practice, PA-MRFA with a 95%
threshold based on polychoric correlations or, in case of nonconvergence, Pearson correlations with
mean thresholds appear to be a good choice for identification of the number of common factors.
PA-MRFA is a common-factor-based method and performed best in the simulation experiment. PA
based on PCA with a 95% threshold is second best, as this method showed good performances in the
empirically relevant conditions of the simulation experiment.

Keywords: number of common factors, exploratory factor analysis, minimum rank factor analysis,
principal component analysis, polychoric correlation, tetrachoric correlation

Determining the dimensionality of a set of variables plays a Boomsma, 2010), including Kaiser’s eigenvalue-over-one crite-
critical role in scale construction and psychological theory build- rion (Kaiser, 1960), the scree test (Cattell, 1966), and certain
ing. In scale construction, dimensionality is an important issue variants of parallel analysis (Buja & Eyuboglu, 1992; Glorfeld,
because a sound interpretation of scale scores requires unidimen- 1995; Horn, 1965; Humphreys & Ilgen, 1969). Methods associated
sionality of the variables that make up the scale (Hattie, 1985). In with common factors are the use of goodness-of-fit measures
psychological theory building, the dimensionality of the variables (Browne & Cudeck, 1992) and a different variant of parallel
under study is often a key issue to settle. For example, the theory analysis (Humphreys & Ilgen, 1969).
of the Big Five personality traits has found a large number of Parallel analysis (PA) appears to be among the most recom-
adherents, but claims for traits beyond the Big Five have been mended methods (Fabrigar, Wegener, MacCallum, & Strahan,
made (Becker, 1999; De Raad & Barelds, 2008). In this case, the 1999; Hayton, Allen, & Scarpello, 2004; Thompson, 2004). This
theory on distinguishing traits is the primary focus, and the dimen- recommendation is based on generally good performances in com-
sionality issue plays a crucial role in the research. parative simulation studies (Humphreys & Montanelli, 1975;
In deliberations about dimensionality, both substantive consid- Peres-Neto, Jackson, & Somers, 2005; Zwick & Velicer, 1986), as
erations and formal criteria should play a role. To the latter end, well as in empirical applications (Steger, 2006). Notwithstanding
numerous formal criteria have been put forward. Those methods the recommendations to use PA in empirical practice, its applica-
can be categorized as being associated with principal component tion is not straightforward. Since its original proposal (Horn,
analysis (PCA) or common factor analysis. The most popular 1965), different modifications of PA have been put forward, re-
methods appear to be associated with PCA (ten Holt, van Duijn, & sulting in different PA procedures (Buja & Eyuboglu, 1992). As
different procedures generally yield different results, it is impor-
tant to know, given the nature of the observed data, which proce-
This article was published Online First April 18, 2011. dure is most likely to properly indicate the dimensionality.
Marieke E. Timmerman, University of Groningen, Groningen, the Neth- In this article, we discuss the key aspects to consider in selecting
erlands; Urbano Lorenzo-Seva, Research Center for Behavior Assessment, an optimal PA procedure that suits the properties of data observed.
Rovira i Virgili University, Tarragona, Spain. Herewith, we are interested in a PA procedure suitable for finding
This research was supported by grants to Urbano Lorenzo-Seva from the the number of common factors of ordered polytomous items,
Research and Information Society of the Catalan Ministry of Universities
which are items scored on a limited number of ordered categories.
(2009SGR1549) and from the Spanish Ministry of Education and Science
(PSI2008-00236/PSIC).
Such choices are motivated by the idea that the model used in
Correspondence concerning to this article should be addressed sent common factor analysis fits well to a fundamental scaling princi-
to Marieke E. Timmerman, Heymans Institute for Psychology, Psychomet- ple, namely, that a series of observed item scores are indicative of
rics and Statistics, Grote Kruisstraat 2/1, Groningen 9712TS, the Nether- a latent trait (Borsboom, 2006) and that ordered polytomous items
lands. E-mail: m.e.timmerman@rug.nl make up the vast majority of psychological data.

209
210 TIMMERMAN AND LORENZO-SEVA

Plan of the Article result in empirical alphas higher than nominal alphas and, thus, in
a loss of statistical power for increasing dimensionalities. The size
We start the discussion on PA procedures by introducing the of the power loss is data dependent and cannot be generally
originally proposed PA (Horn, 1965). This procedure is used to predicted. Because successive eigenvalues of a correlation matrix
assess the number of principal components but could potentially be are directly related to the explained variance of successive princi-
applied to determine the number of common factors. Subse- pal components, the preceding implies that Horn’s PA is theoret-
quently, we discuss two procedures designed to assess the number ically well founded to use for assessment of the significance of the
of common factors. We explain that the PA procedure based on first principal component. For the remaining successive principal
principal axis factoring (Humphreys & Ilgen, 1969) may yield a components, Horn’s PA suffers from loss of statistical power.
tendency to indicate too many common factors. We put forward an Horn’s PA has also been used as a method to assess the number
alternative based on minimum rank factor analysis (ten Berge & of common factors (e.g., Finch & Monahan, 2008; Reddon, Mar-
Kiers, 1991) and explain the rationale behind its use in PA. ceau, & Jackson, 1982). Because a direct relationship between
Subsequently, different ways to obtain the sampling distributions eigenvalues of a correlation matrix and the number and nature of
of eigenvalues are treated. Finally, the use of Pearson or poly- common factors is lacking, Horn’s PA is weakly founded for this
choric correlations in PA for ordered polytomous items is dis- purpose.
cussed. The comparative quality of various PA procedures is Horn’s PA has been extensively studied, both on the basis of
examined in a simulation study. The empirical implications of the empirical data sets (Cota, Longman, Holden, & Fekken, 1993;
findings are discussed. Hubbard & Allen, 1987) and Monte Carlo simulation (Glorfeld,
1995; Peres-Neto et al., 2005; Turner, 1998; Velicer, Eaton, &
Fava, 2000; Zwick & Velicer, 1986). The performance of Horn’s
Procedures for Parallel Analysis PA appears to be generally good, and the method has appeared to
outperform competitive methods, such as Bartlett’s test (Bartlett,
Horn’s Parallel Analysis to Assess the Number 1950) and the scree test (Cattell, 1966).
of Components

The central idea of the original PA (Horn, 1965) is that true PA Procedures for Assessment of the Number of
dimensions are associated with eigenvalues that are larger than the Common Factors
ones associated with dimensions derived from random data.
Horn’s PA can be viewed as a modification of the eigenvalue- PA based on principal axes factor analysis. Humphreys and
over-one criterion (Kaiser, 1960), where sampling variability is Ilgen (1969) noted that Horn’s PA for assessment of the number of
taken into account. The latter is important because even if the principal components (PCs) is not well suited for use in assessing
eigenvalues at the population level equal one, as is the case for the number of common factors (CFs). They proposed a PA pro-
independent variables, the eigenvalues of the correlation matrix cedure to assess the number of CFs. In this procedure, the eigen-
resulting from finite samples are larger than one (Buja & Eyubo- values are obtained from the reduced correlation matrix, with
glu, 1992; Horn, 1965). estimates of the communalities on its diagonal. As a communality
The procedure followed in Horn’s PA boils down to the follow- estimate, the squared multiple correlation R2j from the regression of
ing. First, the sampling distributions of the successive eigenvalues the jth variable with all other variables is taken, as in principal axes
under the null hypothesis of independent variables are acquired. factor analysis (PAFA). Therefore, we denote this PA procedure as
Then, each empirical eigenvalue associated with the observed PA-PAFA; it appears to have received relatively little study (Hum-
correlation matrix is compared with the distribution of random phreys & Ilgen, 1969; Humphreys & Montanelli, 1975).
eigenvalues at the same position (i.e., the first empirical eigenvalue Comparing the two PA procedures from a theoretical perspec-
is compared with the distribution of the first random eigenvalue, tive shows that PA-PAFA may have a tendency to indicate larger
the second empirical eigenvalue to the distribution of the second dimensionalities than Horn’s PA (Buja & Eyuboglu, 1992; Steger,
random eigenvalue, and so on). The number of dimensions indi- 2006). To see why this difference between Horn’s PA and PA-
cated equals the number of dimensions related to empirical eigen- PAFA occurs, one should note two important observations. First,
values larger than a given threshold in the distribution of random with increasing sample sizes, the eigenvalues of random correla-
eigenvalues. Horn (1965) proposed using the average as the thresh- tion matrices approach one, whereas the eigenvalues of random
old. Because this threshold appeared to yield a tendency to over- reduced correlation matrices approach zero. This implies that the
estimate the dimensionality, the use of a more stringent criterion eigenvalues associated with randomly generated data in Horn’s PA
was advocated, like the 95% quantile (Buja & Eyuboglu, 1992; are systematically higher than those in PA-PAFA. Second, reduced
Glorfeld, 1995). The 95% threshold appears to be most popular to correlation matrices, as used in PA-PAFA, are often non-Gramian.
date. Such a matrix has negative eigenvalue(s), implying that its highest
An important question is what the rationale behind Horn’s PA eigenvalue(s) must be larger than the highest eigenvalue(s) of the
is. It might be tempting to interpret Horn’s PA as an inferential unreduced correlation matrix. Hence, a PA based on a comparison
method (i.e., a method to assess the significance of each dimen- of eigenvalues of a reduced correlation matrix of observed data
sion), as Glorfeld (1995) insinuated. As Buja and Eyuboglu (1992) with those of randomly drawn data may result in higher indicated
explained, this interpretation is appropriate only for the first eigen- dimensionalities than a PA based on nonreduced correlation ma-
value because of the inherent dependencies between successive trices. In empirical examples, PA-PAFA indeed appears to over-
eigenvalues. The dependencies between successive eigenvalues factor (Buja & Eyuboglu, 1992; Steger, 2006). However, PA-
 PARALLEL ANALYSIS OF ORDERED POLYTOMOUS ITEMS 211

PAFA has not been properly examined in a simulation study in (Buja & Eyuboglu, 1992; Glorfeld, 1995; O’Connor, 2000; Zwick
which the seriousness of this drawback could be established and & Velicer, 1986).
the potential usefulness of the method in some particular condi- To avoid the at-that-time expensive Monte Carlo simulations,
tions could be determined. researchers developed alternative methods. Regression equations
From a theoretical point of view, we observe a fundamental in different variants were proposed with which to predict the
problem with PA-PAFA. A prerequisite for justifying the compar- eigenvalues of random data, both for correlation matrices (Allen &
ison of the eigenvalues of the two types of correlation matrices Hubbard, 1986; Lautenschlager, Lance, & Flaherty, 1989; Long-
(i.e., those of the observed data and those of the randomly gener- man, Cota, Holden, & Fekken, 1989) and reduced correlation
ated data) is that they have a common interpretation. This is the matrices (Montanelli & Humphreys, 1976). Doubt has been cast on
case for Horn’s PA, because the eigenvalues of all correlation the precision of those formulas (Lautenschlager, 1989), although the
matrices involved are proportional to the percentage of explained Longman et al.’s (1989) approach appeared to yield satisfactory
variance of the PC concerned. For PA-PAFA, this property does precision (Cota, Longman, Holden, & Fekken, 1993). Also, the use of
not hold, because the eigenvalues considered in PA-PAFA stem interpolation for tabled eigenvalue distribution information was pro-
from reduced correlation matrices. Thus, the eigenvalues related to posed (Cota, Longman, Holden, Fekken, & Xinaris, 1993; Lauten-
the observed data and to the randomly generated data appear to schlager, 1989). Although the precision of this approach was reported
lack a common basis for interpretation. However, the possibility to be satisfactory (e.g., Buja & Eyuboglu, 1992), the need to use
that PA-PAFA may perform well in certain conditions has not yet approximate tables has vanished with the increase of available com-
been ruled out. putational power.
PA based on minimum rank factor analysis. As an alter- The robustness of the eigenvalue distributions to departures
native to PA-PAFA for assessment of the number of CFs, one may from normality have been examined in simulation studies (Buja &
compare properties of observed and randomly generated data with Eyuboglu, 1992; Dinno, 2009; Glorfeld, 1995). Those studies
a common interpretation. In particular, we propose to compare the consistently have shown that the eigenvalue distribution appears to
proportions of explained common variance (ECV) of successive be rather insensitive to departures from normality. To use PA for
CFs of observed data with the ECV distributions obtained from ordered polytomous variables, Liu and Rijmen (2008) have pro-
randomly generated data. The logic underpinning this approach is posed drawing from the multinomial distribution, but the merits of
similar to that of Horn’s PA while possible differences in com- this approach have not been evaluated so far.
munalities across observed variables are accounted for. Because of A further development has been the proposal to use a nonpara-
possible non-Gramian reduced correlation matrices when the metric approach to obtain the eigenvalue sampling distribution
squared multiple correlations are used as a communality estimate, with a permutation of the raw data (Buja & Eyuboglu, 1992). This
the ECV cannot be computed from the PAFA solution. approach would yield more precise information than drawing from
To estimate the ECV of the successive CFs, we propose to use a multivariate normal distribution in a case in which the empirical
minimum rank factor analysis (MRFA; ten Berge & Kiers, 1991). and hypothesized population distributions would differ from each
MRFA is used to find communality estimates such that, given a other. Although Buja and Eyuboglu (1992) considered the use of
predefined number of CFs, the amount of unexplained common permutation to be not of primary importance, referring to the
variance is minimized, subject to the constraint that the reduced robustness of PA to departures from normality, permutation has
correlation matrix is Gramian. This implies that the eigenvalues of been recommended for PA, especially in cases where the observed
the reduced correlation matrices obtained from MRFA can be used data distribution clearly deviates from normality (O’Connor, 2000;
to estimate the ECV per CF, both for the observed and the Steger, 2006). However, it is unclear to what extent permutation is
randomly generated data. In each MRFA, we used the maximal favorable over drawing from a multivariate normal distribution for
possible number of CFs, which is equal to the number of variables ordered polytomous items in empirical practice.
minus one. In a preliminary simulation study, this number hardly
affected the estimated ECVs and did not affect the number of CFs Pearson or Polychoric Correlations?
indicated. In what follows, the just-proposed PA procedure will be
denoted as PA-MRFA. The usual correlations in a PA are Pearson correlations. Implic-
itly, this approach is based on a linear factor model, implying that
a linear relationship between the factor scores and expected values
Obtaining the Sampling Distributions of Eigenvalues of the observed items is assumed. This assumption may be grossly
violated in the case of ordered polytomous items. In those cases,
In a PA, the sampling distributions of the eigenvalues under the Pearson correlations are known to underestimate the strength of
null hypothesis of independent variables are required. Because an relationship between variables (Olsson, 1979b).
analytic solution to solve for the sampling distributions of eigen- The detrimental effects of analysis of ordered polytomous vari-
values of correlation matrices is not available, different approaches ables with a linear factor model on parameter estimates and fit
have been taken to obtain those sampling distributions. statistics in the context of a confirmatory factor analysis have been
Horn (1965) proposed the use of Monte Carlo simulation. He well studied (e.g., Dolan, 1994; Muthén & Kaplan, 1985). Dolan
recommended computing K random correlation matrices on the (1994) advised against this approach in the case of ordered poly-
basis of K data sets drawn from a multivariate normal distribution. tomous items with less than five response categories. The studies
In the early years of PA, one resorted to K ⫽ 1 (e.g., Horn, 1965; that examined the performance of linear-factor-model-based meth-
Humphreys & Ilgen, 1969), but with the increased availability of ods when applied to ordered polytomous items suggest that the
computational power, larger values for K (e.g., 1,000) were applied number of factors generally will be overestimated. This effect has
212 TIMMERMAN AND LORENZO-SEVA

been reported for Kaiser’s criterion (Bernstein & Teng, 1989) and tomous items could possibly be improved through the use of a
for the chi-square statistic (Olsson, 1979b), especially in the case permutation of the sample data rather than draws from a multivar-
of items that are skewed in opposite directions. An adaptation of iate normal distribution. Because the assumption of multivariate
the scree test displayed poor performance when applied to binary normality is clearly untenable for ordered polytomous items, the
items (Roznowski, Tucker, & Humphreys, 1991). For binary null distribution of eigenvalues could be approximated better on
items, Horn’s PA performed reasonably well for detection of the basis of the sample distribution rather than the multivariate
unidimensionality (Weng & Cheng, 2005), but the performance normal distribution. Finally, when common factors are searched
deteriorated with increasing skewness of items. for, it may be advantageous to use a PA-MRFA approach, or
The polychoric correlation (e.g., Olsson, 1979a) is an alternative possibly PA-PAFA approach, rather than Horn’s PA.
to the Pearson correlation for ordered polytomous variables. Under
the assumption that the ordered polytomous variables represent
underlying bivariate normally distributed variables, the polychoric Software to Perform PA
correlation is a maximum likelihood estimate for the Pearson
correlation between the underlying variables. The tetrachoric cor- Although PA is not implemented in popular statistical software
relation (Pearson, 1901) is a special case of the polychoric corre- packages like SPSS, SAS, Stata (StataCorp, 2009), or Mplus
lation for binary variables. (Muthén & Muthén, 1998 –2010), PA can be applied rather easily
In the case of ordered polytomous items, it is fairly standard to through freely available software. O’Connor (2000) provided syn-
apply a linear factor model to a polychoric correlation matrix. This taxes for SPSS, SAS, and MATLAB, whereas Ledesma and
procedure is not without problems, as such a correlation matrix Valero-Mora (2007) offered the stand-alone program ViSta-
may be non-Gramian, and the sampling error of polychoric corre- PARAN. The syntaxes and ViSta-PARAN implemented the same
lations may be relatively large in small sample sizes. The sampling PA procedures, namely Pearson-based Horn’s PA and PA-PAFA,
error greatly depends on the item difficulty (Roznowski et al., using normal draws or permutation, with mean and any percentile
1991). Roznowski et al. (1991) tested an adaptation of the scree thresholds. The program PARAN is available for use with R (R
test based on tetrachoric correlations and concluded that tetracho- Development Core Team, 2005) and Stata (Dinno, 2009). PARAN
rics probably were not dependable in cases with a wide range of offers Pearson-based Horn’s PA, PA-PAFA, and three other CF-
difficulties, except for sample sizes “substantially larger than based PA variants (i.e., based on principal component factors,
2,000” (p. 124). iterated principal factors, and maximum likelihood factors), using
In the context of PA, Weng and Cheng (2005) investigated draws from a uniform distribution, with mean and any percentile
whether it would be advantageous to obtain the empirical eigen- thresholds. Watkins (2010) provided a stand-alone Windows and
values from the tetrachoric correlation matrix rather than the Mac programs to compute the mean thresholds for a Pearson-based
Pearson correlation matrix of the observed data. They examined Horn’s PA using normal draws. Finally, we implemented PA
Horn’s PA, with random sampling from a multivariate normal procedures functions in FACTOR (Lorenzo-Seva & Ferrando,
distribution. Thus, in this PA procedure, the empirical correlation 2006), a stand-alone program for Windows used to perform dif-
matrix and random correlation matrices differed in nature. In a ferent types of exploratory common factor analyses. The proce-
simulation study with binary variables generated from a unidimen- dures included are Pearson- and polychoric-based Horn’s PA,
sional common factor model, the empirical eigenvalues of Pearson PA-PAFA, and PA-MRFA, using normal draws or permutation,
correlations—surprisingly— generally outperformed the ones with with mean and any percentile thresholds. The program, a demon-
the tetrachoric correlations. Weng and Cheng (2005) attributed this stration, and a short manual are available at http://psico.fcep
to the large sampling error of tetrachoric correlations in the case of .urv.es/utilitats/factor/.
small sample sizes, especially with largely varying item difficul-
ties.
Unlike Weng and Cheng (2005), Cho, Li, and Bandalos (2009) Simulation Experiment
proposed to use, in applying Horn’s PA, the same type of corre-
lation matrices for both the empirical and random eigenvalues. In Determining which of the PA procedures would perform best as
a simulation study, they examined the relative performance of an indicator of the number of common factors of ordered polyto-
Pearson correlations and polychoric correlations for items with mous items would be of interest. On the basis of the theoretical
two and three response categories. PA based on random Pearson considerations and results from research discussed in the previous
correlations appeared to perform at least as well as PA based on sections, we hypothesized that our newly proposed PA-MRFA
polychoric correlations. However, this study was limited in that would outperform Horn’s PA and PA-PAFA in determining the
only symmetrical distributions and less than optimal sample sizes number of common factors. We expected PA-PAFA to yield a
(maximally 800) were considered. tendency to overfactor. Furthermore, we hypothesized that for
Although the attempts to use polychoric correlations rather than ordered polytomous items, PA would show improved perfor-
Pearson correlations in PA have not been very successful so far, mances when the sampling distributions of the eigenvalues were
the use of polychoric correlations in PA should not be dismissed obtained through permutation, rather than random sampling from
on the basis of those results only. Because the potential benefits of a multivariate normal distribution. Finally, we conjectured that the
polychoric correlations are to be expected particularly with items use of polychoric correlations rather than Pearson correlations
of varying difficulties and large sample sizes, those conditions might increase the performance, especially in large sample sizes
should be included in studies in which polychoric-based PA is and conditions in which the item difficulties vary considerably
evaluated. Furthermore, the performance of PA for ordered poly- across items (i.e., skewness of items).
 PARALLEL ANALYSIS OF ORDERED POLYTOMOUS ITEMS 213

To study the comparative performance of various PA proce- 0.50, 0.00, and 0.50, respectively, and in the minor present con-
dures in indicating the number of common factors when ordered dition as 0.50, 0.10, and 0.40, respectively.
polytomous items are involved, we performed a simulation study. The design of the loading matrices implies that the expected
major, minor, and unique variances are equal across variables. The
Generation of Simulated Data weights in Equation 1 were determined as wma ⫽ ␴ma 2
⫽ 0.50,
wmi ⫽ ␴mi/E(⌳mi,1⌳ mi,1⬘), wun ⫽ ␴un, with E(⌳mi,1⌳mi,1⬘) repre-
2 2

The simulated data were generated according to a linear CF senting the expected rowwise sum of squares of the elements of the
model, where the resulting continuous variables were categorized first row of ⌳mi. This implies that the nonzero loadings of the
to yield ordered polytomous observed variables. The linear CF
0.5
major factors (i.e., wma ⌳ma) were fixed at 0.71. In previous PA
model included both major and minor factors, as this could be the research, this magnitude appeared to be associated with generally
case with real-world data. In the simulation study, the presence of reasonable PA performances, while differences across the various
minor factors was manipulated. The condition with major factors conditions were shown (Weng & Cheng, 2005).
only yields an unambigious definition of the correct dimensionality Previous studies on PA (Cho et al., 2009; Velicer et al., 2000;
(as will be discussed in detail in the section Quality Criteria). The Weng & Cheng, 2005; Zwick & Velicer, 1986) have revealed that
condition with minor factors present represents a realistic mimic of its performance is affected by sample size, number of variables per
empirical cases (MacCallum & Tucker, 1991). factor, number of factors, the distribution of the variables, the
Starting from the idea that a CF model with a limited number of magnitude of the loadings, and, for polytomous variables, the
CFs will never fit exactly at the population level, we considered number of response categories. In our study, we manipulated those
each variable to be composed of a part consistent with the CF factors for which differential performances across PA procedures
model and another that is not. The latter is called model error and could be expected, that is, the sample size, number of variables per
is represented by means of several minor factors (based on the factor, number of factors, the distribution of the variables, the
middle model by Tucker, Koopman, and Linn, 1969). Researchers number of response categories, and the presence of minor factors.
previously have taken this approach to assess the performance of By manipulating the minor factors, we were able to compare PA
PA (Humphreys & Montanelli, 1975; Zwick & Velicer, 1986) and performances in optimal conditions (i.e., major factors only) with
in other CF model research (e.g., MacCallum & Tucker, 1991; those in empirically more plausible conditions.
Stuive, Kiers, Timmerman, & ten Berge, 2008). The various levels of the factors in the experimental design were
In the simulation study, the population correlation matrix of the selected such that they represented conditions present in empirical
continuous variables R*pop was taken as research or such that they were expected to differentiate between
PA procedures. The sample size was varied as 100, 200, 500, and
R *pop ⫽ wma ⌳ma ⌳⬘ma ⫹ wmi ⌳mi ⌳ⴕmi ⫹ wun IJ, (1) 2,000. The values of 100 and 2,000 are considered to be extremely
low and high sample sizes, respectively. A sample size as large as
where ⌳ma (J ⫻ Qma) and ⌳mi (J ⫻ Qmi) are major and minor 2,000 was included because tetrachoric correlations require large
loading matrices, respectively, with Qma and Qmi the number of sample sizes to achieve an acceptable sampling error in case item
major and minor factors, and J the number of observed variables; difficulties greatly vary (e.g., Roznowski et al., 1991); a sample
IJ (J ⫻ J) is the identity matrix, reflecting the covariance matrix of size as low as 100 may occur in empirical practice. The number of
the unique parts of the variables; and wma, wmi and wun are weights major factors was varied at Qma ⫽ 1, 3, and 6, representing low,
to manipulate ␴ma 2
, ␴mi
2
, and ␴un
2
, the variances of the major, minor medium, and high numbers of factors, respectively. The number of
and unique parts of the correlation matrix, respectively. observed variables per major factor was varied at 5 and 16. The
We obtained each simulated continuous data matrix X* (N ⫻ J), value of 5 was chosen to represent a slightly overidentified model
with N being the sample size, by randomly drawing N vectors from (for the major factors), and 16 was chosen to represent a clearly
a multivariate normal distribution N(0, R*pop). Subsequently, we overidentified model. Note that in the condition with minor factors
obtained each element xnj of the polytomous simulated data matrix present, the total number of factors (i.e., Qmi ⫹ Qma) was not
X (N ⫻ J) from the element xnj* of the continuous data matrix X* identified in the case of M ⫽ 5, whereas in the remaining condi-
using prespecified thresholds ␶c (␶c ⫽ ␶0, . . . ,␶C, with C being the tions, the factors were overidentified. The number of categories
number of categories), with xnj ⫽ c if ␶c⫺1 ⬍ xnj* ⱕ ␶c. (C) of the ordered polytomous variables was selected to be 2, 4,
To keep the simulation study feasible, we kept the following and 5. The number 2 defined the condition in which any differ-
properties of the simulated data constant: The major loading ma- ences between Pearson- and polychoric-based PA procedures were
trix ⌳ma (J ⫻ Qma) related each variable uniquely to a single major expected to become manifest most clearly. The conditions with
factor, as each row consisted of a single element equal to 1, and 0s C ⫽ 4 and 5 were included, because they are considered to be the
otherwise. Such simple structure was used in previous PA research critical numbers for a nonlinear and a linear factor model, respec-
(e.g., Cho et al., 2009), and complexity of structure did not appear tively (Dolan, 1994) and hence may also be critical for a poly-
to crucially affect PA performance (Zwick & Velicer, 1986). The choric- and Pearson-based PA procedure, respectively. Besides,
elements of minor loading matrix ⌳mi (J ⫻ Qmi) were randomly C ⫽ 5 is frequently used in empirical practice. The distribution of
drawn from a uniform distribution on the interval [⫺1, 1]. The the variables was manipulated to be uniform and skew. In the
number of minor factors Qmi was fixed at Qmi ⫽ 3 ⫻ Qma. Each uniform condition, the thresholds ␶c (␶c ⫽ ␶0,. . .,␶C) were chosen
major common factor was associated to M, observed variables; such that the expected proportion of observations in category c was
hence, J ⫽ M ⫻ Qma. We manipulated the conditions of the minor equal to C⫺1, with C representing the number of categories. In the
factors presence by varying the variances of the major, minor, and skew condition, for each single factor, half of the variables were
unique parts, which were taken in the minor absent condition as skewed in the opposite direction, so as to mimic differences in item
214 TIMMERMAN AND LORENZO-SEVA

difficulty across variables; the thresholds were chosen such that the conditions, we estimated polychoric correlations (Olsson, 1979a),
expected proportion of observations in categories c ⫽ 1,. . ., C which are well known for being difficult to converge (e.g., Baba-
were [0.80, 0.20] and [0.20, 0.80] for C ⫽ 2; [0.05, 0.15, 0.75, kus, Ferguson, & Jöreskog, 1987). The most difficult conditions
0.05] and [0.05, 0.75, 0.15, 0.05] for C ⫽ 4; [0.05, 0.10, 0.20, 0.60, appeared to be Cs ⫽ 4 and 5, Qmas ⫽ 3 and 6, M ⫽ 16, and N⫽
0.05] and [0.05, 0.60, 0.20, 0.10, 0.05] for C ⫽ 5. Since each 200; in those conditions, half of the matrices initially generated
simulated continuous variable in X* is standard normally distrib- were not well suited. In those conditions, the ratio between the
uted, a threshold ␶c was taken as z*, the value for which number of observed variables and the number of observations was
P(Z ⬍ z*) ⫽ pcum,c, with pcum,c the expected cumulative propor- unfavorable.
tion of observations in categories 1, . . . , c.
In a preliminary study, it appeared that the size of the correlation
Analyses of Simulated Data
matrices was a key factor for computing time. Herewith, the
number of factors indicated by different variants of PA-MRFA and
In total, we used 18 PA procedures, which were based on two
Horn’s PA, which we considered to be the most promising meth-
correlation types (Pearson and polychoric), two random sampling
ods, appeared to be highly stable across replicates, alleviating the
techniques (permutation and randomly drawing from a multivari-
need for a large number of replicates. To keep the total computing
ate normal distribution), three extraction methods (Horn’s PA,
time within acceptable range, we used 50 replicates; furthermore,
PA-PAFA, and PA-MRFA), and two thresholds (the 95th percen-
we used a completely crossed design for the conditions with one
tile and the mean). For the Pearson-based PA procedures, all 12
and three major factors (associated with maximally 48 variables)
possible combinations—2 (sampling technique) ⫻ 3 (extraction
only. This resulted in 2 (number of major factors) ⫻ 3 (number of
method) ⫻ 2 (threshold)—were considered. For the polychoric-
response categories) ⫻ 4 (sample size) ⫻ 2 (number of variables
based PA procedures, random categorical data are needed, and
per factor) ⫻ 2 (minor factors presence) ⫻ 2 (distribution) ⫻ 50
therefore we used only permutation as a sampling technique,
(replicates) ⫽ 9,600 simulated data sets. For the condition with six
yielding six combinations—3 (extraction method) ⫻ 2 (threshold).
major factors, we only selected those conditions that appeared to
The 18 PA procedures were applied to each simulated data
be difficult in the study with one and three major factors (i.e., those
matrix, except to those with a sample size of 100, for which only
conditions for which recovery of the dimensionality appeared in
the 12 Pearson-based procedures were used. As noted earlier, the
less than 100% for the best performing PA procedure). The diffi-
nonconvergence problems of the polychoric correlation algorithms
cult conditions appeared to be those with minor factors present and
are extremely great in case of such a small sample size.
with C ⫽ 2 answering categories. Hence, this yielded to another 1
In the random sampling step of each PA, we used 500 replicates
(six major factors) ⫻ 1 (two response categories) ⫻ 4 (sample
to obtain the sampling distribution. This number of replicates is
size) ⫻ 2 (number of variables per factor) ⫻ 1 (minor factors
considered sufficient to yield precise p values (Buja & Eyuboglu,
present) ⫻ 2 (distribution) ⫻ 50 (replicates) ⫽ 800 simulated data
1992). If we could not estimate a polychoric (or tetrachoric)
sets.
correlation matrix (i.e., the corresponding algorithm did not con-
For each simulated data set, we had to estimate a polychoric
verge) or if the correlation matrix appeared to be non-Gramian and
correlation matrix. For binary observed variables (i.e., C ⫽ 2), we
the smoothing procedure did not yield a Gramian matrix, a differ-
estimated tetrachoric correlations using computationally simple
ent random data matrix was generated. To keep the study feasible,
method of Bonett and Price (2005), who showed that it yields
we repeated this process maximally 25 times. That is, if 25 random
acccurate approximations. For the ordered polytomous variables
samples in a row did not produce a proper polychoric (or tetra-
(i.e., Cs ⫽ 4 and 5), we estimated polychoric correlations using the
choric) correlation matrix, we used 3 (extraction methods) ⫻ 2
two-stage iterative procedure (Olsson, 1979a). We used 100 iter-
(thresholds) and reported the associated six PA analyses to be
ations at maximum, which is well above the recommended 50
nonconvergent.
iterations (Uebersax, 2010).
A proper correlation matrix is Gramian. Therefore, if an esti-
mated tetrachoric or polychoric correlation matrix appeared to be Quality Criteria
non-Gramian, we adapted the estimated correlation matrix to a
Gramian matrix using the nonlinear smoothing procedure (Devlin, To assess the quality of the PA analyses, we considered the
Gnanadesikan, & Kettenring, 1975; Devlin, Gnanadesikan, & degree of recovery of the correct number of common factors. A
Kettenring, 1981). key question in the conditions with minor factors present is what
If no proper polychoric correlation matrix could be estimated, actually should be considered the correct number of common
either because the algorithm used to estimate the polychoric cor- factors. The investigators who conducted earlier simulation studies
relations did not converge or because the smoothing procedure did of PA that included major and minor factors (Humphreys &
not result in a Gramian matrix, a different data set was simulated. Montanelli, 1975; Zwick & Velicer, 1986) took the perspective
In a preliminary study, the convergence problem for a sample size that minor factors represent model error. Therefore, they consid-
of N ⫽ 100 appeared so large (convergence for only 1 out of 100 ered the number of major factors to be the correct number of
generated data matrices) that we omitted computing polychoric common factors. On the other hand, from a theoretical perspective,
correlations in conditions with N ⫽ 100. In the remaining condi- the sum of the numbers of major and minor factors should be
tions, the convergence problems in computing the polychoric considered to be the correct number of common factors, because
correlations were largest in the conditions with the number of this is the rank of the reduced population correlation matrix (i.e.,
categories Cs ⫽ 4 or 5, the number of major factors Qmas ⫽ 3 or *
Rpop ⫺ wunIJ). However, because PA is known to suffer from a
6, and the number of items per major factors M ⫽ 16. In those loss of statistical power for increasing dimensionalities, it is highly
 PARALLEL ANALYSIS OF ORDERED POLYTOMOUS ITEMS 215

questionable whether PA will indicate the total number of factors, factors is indicated correctly in 18.8%– 69.8% of the cases. In a
rather than the major factors only. In our study, we considered both substantial percentage of cases (10.9%–38.4%), neither the num-
the recovery of the number of major factors and of the total ber of major nor the total number of factors was correctly indi-
number of factors. cated. Furthermore, it is salient that the PA-PAFA variants in some
For each simulated data set, the results of each of the 18 PA cases (2.9%–10.9%) indicated even more than the number of total
procedures were recorded (i.e., the numbers of indicated factors or factors. This tendency toward overfactoring is in line with the
a nonconvergence). We assessed the quality of the various PA theoretical considerations and empirical evidence on the properties
procedures by considering comparative performance in correctly of PA-PAFA, which were discussed earlier. From the previous
recovering the number of major factors and the number of total evidence, it can be concluded that the PA-PAFA procedures gen-
factors (i.e., the sum of the numbers of major and minor factors). erally showed poor performance in the recovery of the number of
Furthermore, possible tendencies of PA procedures for overfactor- factors. Therefore, we do not discuss PA-PAFA further.
ing and the factors influencing the quality were examined. A further comparison of Horn’s PA and PA-MRFA showed that
the use of polychoric rather than Pearson correlations appeared to
Results reduce the correctly indicated number of major factors consider-
ably. This reduction was mainly due to the nonconvergent solu-
Performances of PA procedures across all conditions. To tions, which occurred in as much as 63.0% of the data matrices for
give a first impression of the quality of recovery for each PA which polychoric correlations were computed. Furthermore, it is
procedure, we summarized different performance aspects across salient that neither PA-MRFA nor Horn’s PA showed clear ten-
all conditions in Table 1. Table 1 contains the percentages of dencies to indicate the number of total factors (0.0% of the cases)
correctly indicated numbers of major factors, the percentage of or to overfactor (maximally 1.0% of the cases). Apparently, the
correctly indicated number of total factors (only for the condition relative size of the minor factors in the study (with each minor
with minor factors present), the percentages of incorrectly indi- factor accounting for maximally 3.33% of the total variance) was
cated number of factors (i.e., neither major nor total number of so small that the loss of statistical power of PA for increasing
factors), the percentages of overfactoring (i.e., more factors than dimensionalities is shown.
the total number of factors), and the percentages for which no Performance of Horn’s PA and PA-MRFA in indicating the
convergence (of the polychoric correlation matrix of the random number of major factors. From Table 1, we can conclude that
data) could be reached. Horn’s PA and PA-MRFA performed particularly well in indicat-
Table 1 shows that of the three extraction methods examined, ing the number of major factors, rather than the number of total
PA-MRFA performed best in indicating the correct number of factors. To obtain an insight into the comparative performances
major factors, followed very closely by Horn’s PA, and a distant within the various conditions, we considered the percentages of
third was PA-PAFA. When comparing the performance of the 95% correctly indicated number of major factors per level of the ex-
threshold and the mean threshold across all PA variants, we found perimental factors. The percentages for the conditions with major
that the use of the 95% threshold yielded slightly better perfor- factors only are shown in Table 2, and those for the conditions with
mance than the mean threshold for Horn’s PA, PA-PAFA, and major and minor factors are shown in Table 3. We presented the
polychoric-based PA-MRFA. However, for the Pearson-based PA- results separately to avoid distortion related to the not-completely-
MRFA variants, the mean threshold resulted in slightly better crossed experimental design and to show explicitly the effects of
performances than the 95% threshold. For all PA variants consid- the presence of minor factors on the performance. The percentages
ered, the use of permutation as the sampling technique compared presented for the polychoric-based solutions were computed on the
with the use of draws from a normal distribution did yield highly basis of only those matrices for which convergence was reached.
similar performances. A thus-performed comparison between the performances of the
PA-PAFA appeared to perform poorly across the six variants Pearson- and polychoric-based variants enabled us to answer the
examined. Among the six PA-PAFA variants, the number of major empirically relevant question of whether it was worthwhile to use

Table 1
Performance (in Percentages) of the Parallel Analysis Procedures Across All Conditions
Horn’s PA PA-PAFA PA-MRFA

Pearson correlation Polychoric Pearson correlation Polychoric Pearson correlation Polychoric

correlation– correlation– correlation–
Normal Permutation permutation Normal Permutation permutation Normal Permutation permutation
Variable 95% M 95% M 95% M 95% M 95% M 95% M 95% M 95% M 95% M
Major factors correct 95.2 93.0 95.2 93.0 36.5 36.1 69.8 56.6 69.7 56.6 25.0 18.8 99.0 99.3 99.0 99.3 36.8 36.7
Total factors correct 0.0 0.0 0.0 0.0 0.0 0.0 3.4 5.0 3.4 5.1 1.0 1.5 0.0 0.0 0.0 0.0 0.0 0.0
Incorrect no. of factors 4.8 7.0 4.8 7.0 0.5 0.9 26.8 38.4 26.9 38.3 10.9 16.7 1.0 0.7 1.0 0.7 0.2 0.2
Overfactoring 0.6 1.0 0.0 1.0 0.5 0.9 6.2 10.9 6.2 10.9 2.9 7.3 0.0 0.1 0.0 0.0 0.0 0.0
Nonconvergence 0.0 0.0 0.0 0.0 63.0 63.0 0.0 0.0 0.0 0.0 63.0 63.0 0.0 0.0 0.0 0.0 63.0 63.0
Note. PA ⫽ parallel analysis; PA-PAFA ⫽ PA based on principal axes factoring; PA-MRFA ⫽ PA based on minimum rank factor analysis; normal and
permutation ⫽ sampling methods; 95% ⫽ 95% quantile criterion threshold; M ⫽ mean criterion threshold.
216 TIMMERMAN AND LORENZO-SEVA

Table 2
Percentages Correctly Indicated Number of Major Factors for Conditions With Only Major Factors
Horn’s PA PA-MRFA

Pearson correlation Polychoric Pearson correlation Polychoric

correlation– correlation–
Converged
Normal Permutation permutation Normal Permutation permutation
No. of polychoric
Variable 95% M 95% M 95% M 95% M 95% M 95% M data sets PA (%)
Factors (no.)
1 99.92 99.29 99.92 99.29 100 99.90 99.79 99.88 99.83 99.88 99.90 100 2,400 42.12
3 97.33 96.29 97.33 96.29 100 100 99.21 99.46 99.25 99.42 100 100 2,400 22.25
Categories (no.)
2 95.87 93.38 95.88 93.38 100 99.90 98.75 99.12 98.81 99.06 100 100 1,600 65.06
4 100 100 100 100 100 100 99.75 99.88 99.81 99.88 99.57 100 1,600 14.69
5 100 100 100 100 100 100 100 100 100 100 100 100 1,600 16.81
Skewed
No 99.96 99.83 99.96 99.83 100 99.89 99.79 99.87 99.79 99.87 100 100 2,400 37.83
Yes 97.29 95.75 97.29 95.75 100 100 99.21 99.46 99.29 99.42 99.84 100 24,00 26.54
Sample size (no.)
100 99.58 98.08 99.50 98.08 na na 98.92 99.08 99.00 99.00 na na 1,200 na
200 99.58 98.75 99.58 98.67 100 99.81 99.58 99.83 99.58 99.83 99.81 100 1,200 43.08
500 99.58 98.83 99.67 98.92 100 100 99.92 99.92 100 99.92 100 100 1,200 45.33
2,000 95.75 95.50 95.75 95.50 100 100 99.58 99.83 99.58 99.83 100 100 1,200 40.33
Variables (no.)
5 99.54 98.46 99.54 98.42 100 100 99.08 99.46 99.17 99.42 99.90 100 2,400 42.63
16 97.71 97.13 97.71 97.17 100 99.81 99.92 99.87 99.92 99.87 100 100 2,400 21.75
Minor factors: No 98.62 97.79 98.62 97.79 100 99.94 99.50 99.67 99.54 98.65 99.94 100 4,800 32.19
Note. PA ⫽ parallel analysis; PA-PAFA ⫽ PA based on principal axes factoring; PA-MRFA ⫽ PA based on minimum rank factor analysis; normal and
permutation ⫽ sampling methods; 95% ⫽ 95% quantile criterion threshold; M ⫽ mean criterion threshold; na ⫽ not available (missing by design).

the polychoric PA variant, given that convergence could be peared larger with increasing numbers of factors, sample sizes, and
reached. The percentages of converged polychoric correlation ma- number of variables per factor and with skewed rather than sym-
trices and the total number of simulated matrices per condition are metrically distributed variables. For PA-MRFA, the polychoric-
presented in the last two columns. based procedure appeared slightly more accurate. The differences
We will first discuss the comparative performances of the various in performances between the polychoric- and Pearson-based vari-
Horn’s PA and PA-MRFA procedures. As can be seen in Tables 2 and ants appeared much smaller than those for Horn’s PA, and these
3, the results based on permutation and based on draws from a normal differences are difficult to attribute clearly to certain experimental
distribution are remarkably similar, with a maximal observed differ- factors.
ence of 0.28% between performances in the same condition and with In the condition with major factors only (see Table 2), the
the same PA procedure. This contrasts to our hypothesis that permu- performances of the Pearson-based procedures in conditions with
tation-based PA procedures would be advantageous in the case of four- and five-ordered categories ranged from near perfect (PA-
ordered polytomous items. For reasons of parsimony and clarity, we MRFA with C ⫽ 4) to perfect (Horn’s PA with Cs ⫽ 4 and 5;
will refer to the results of only a single sampling technique in what PA-MRFA with C ⫽ 5), as shown in Table 2. As expected, the
follows. The polychoric-based results were obtained on the basis of condition with two-ordered categories appeared slightly more dif-
permutation only. To enable a fair comparison with the Pearson-based ficult than with four- and five-ordered categories for the Pearson-
results, we will focus on the permutation-based results. based Horn’s PA and PA-MRFA procedures. The use of the
As can be seen in Tables 2 and 3, for each Horn’s PA procedure, polychoric correlations appeared advantageous, in cases for which
the 95% threshold appeared to yield slightly more accurate results convergence could be reached, since the polychoric Horn’s PA and
than the mean threshold in most conditions. This is in line with earlier PA-MRFA procedures succeeded in a near perfect to perfect
findings (Buja & Eyuboglu, 1992; Glorfeld, 1995). For PA-MRFA, recovery.
the same pattern was found only for polychoric-based solutions that The presence of minor factors yielded a clear deterioration in
included both major and minor factors (i.e., Table 3); for the Pearson- performance for all Horn’s PA and PA-MRFA procedures, as can
based results, the mean threshold tended to slightly outperform or be be seen by comparing Tables 2 and 3. Horn’s PA procedures
comparable to the 95% threshold. Unless indicated otherwise, we will appeared to be affected to a larger extent than PA-MRFA proce-
refer to the 95% thresholds results in what follows. dures. In what follows, we focus on the relatively difficult condi-
For Horn’s PA, the polychoric-based procedures yielded results tions with minor factors present.
that were the same or slightly more accurate than the results from As can be seen in Table 3, an increase in the number of variables
the Pearson-based procedures in all conditions. The gain in using per factor from five to 16 yielded a decrease in performance for the
the polychoric-based procedure over the Pearson-based one ap- Horn’s PA-based procedures. Further inspection of the results
 PARALLEL ANALYSIS OF ORDERED POLYTOMOUS ITEMS 217

Table 3
Percentage Correctly Indicated Number of Major Factors for Conditions With Major and Minor Factors
Horn’s PA PA-MRFA

Pearson correlation Polychoric Pearson correlation Polychoric

correlation– correlation–
Converged
Normal Permutation permutation Normal Permutation permutation
No. of polychoric
Variable 95% M 95% M 95% M 95% M 95% M 95% M data sets PA (%)
Factors (no.)
1 93.08 88.96 93.17 88.96 98.25 95.66 99.62 99.63 99.62 99.63 99.70 99.47 2,400 54.67
3 93.25 90.21 93.25 90.17 95.38 94.01 98.71 99.25 98.87 99.21 97.09 96.58 2,400 24.33
6 86.87 84.38 87.00 84.50 99.26 98.53 93.71 96.14 93.71 95.86 100 99.75 800 51.00
Categories (no.)
2 90.12 85.88 90.21 85.83 97.73 95.66 96.57 97.74 96.74 97.65 99.93 99.79 2,400 60.54
4 94.06 91.13 94.12 91.13 95.71 92.14 99.81 99.88 99.81 99.88 95.24 94.29 1,600 26.25
5 93.69 91.00 93.69 91.06 99.54 99.54 99.88 100 99.88 99.94 100 99.77 1,600 26.94
Skewed
No 94.89 92.79 94.93 92.79 96.30 93.51 99.27 99.78 99.31 99.75 98.36 98.03 2,800 43.46
Yes 89.64 84.89 89.71 84.89 99.26 98.25 97.67 98.25 97.78 98.18 99.91 99.63 2,800 38.82
Sample size (no.)
100 96.86 94.43 96.86 94.50 na na 95.46 96.85 95.46 96.69 na na 1,400 na
200 98.86 96.57 98.86 96.57 99.85 99.11 99.14 99.14 99.14 99.07 99.85 99.41 1,400 48.14
500 97.00 93.07 97.07 93.07 99.49 98.34 99.79 100 99.86 100 99.74 99.74 1,400 55.86
2,000 76.36 71.29 76.50 71.21 94.33 90.67 99.29 99.93 99.50 99.93 97.87 97.40 1,400 60.50
Variables (no.)
5 98.64 98.11 98.64 98.04 100 99.93 97.11 98.71 97.25 98.61 100 99.93 2,800 53.25
16 85.89 79.57 86.00 79.64 93.48 88.07 99.89 99.33 99.89 99.33 97.42 96.68 2,800 29.04
Minor factors: Yes 92.27 88.84 92.32 88.84 97.70 95.75 98.47 99.02 98.55 98.96 99.09 98.78 5,600 41.14
Note. PA ⫽ parallel analysis; PA-PAFA ⫽ PA based on principal axes factoring; PA-MRFA ⫽ PA based on minimum rank factor analysis; normal and
permutation ⫽ sampling methods; 95% ⫽ 95% quantile criterion threshold; M ⫽ mean criterion threshold; na ⫽ not available (missing by design).

revealed that this was completely attributable to a number of Horn’s PA and from 98.86% to 97.07% to 76.50% for N ⫽ 200 vs.
indicated factors larger than the number of major factors. In N ⫽ 500 vs. N ⫽ 2,000, respectively for Pearson 95% threshold
contrast, the Pearson-based PA-MRFA procedures performed Horn’s PA based on permutation in Table 3).
slightly better, with 16 variables per factor. The latter is not The deterioration in performance with increasing sample size
surprising, because the common factors can be identified better could possibly be due to an increase in power to detect minor
with more variables per factors, and MRFA estimates CFs. How- factors. If this were the case, it would result in indicated numbers
ever, this advantage was not maintained for the polychoric-based of factors between the number of major factors and the total (major
PA-MRFA procedures. The slightly worse performance of PA- and minor) number of factors. A close inspection of the indicated
MRFA in conditions with higher numbers of variables per factor number of factors revealed that they appeared about symmetrically
might be related to the fact that a limited-information approach is distributed around the true number of major factors for Pearson-
used in polychoric PA-MRFA. That is, polychoric correlations are based and polychoric-based PA-MRFA and polychoric-based
estimated pairwise, rather than on the basis of all items simulta- Horn’s PA. In contrast, Pearson-based Horn’s PA appeared to
neously. It might be that the degree of inaccuracy of thus-obtained indicate either the number of major factors or from one to three
polychoric correlation matrices increases with increasing numbers more (hence, lower than, and never more than, the total number of
of pairs and hence with the number of observed variables. factors). At first sight, those observations suggest that, unlike
A sample size of 200 appears to be sufficient to have a reason- Pearson- and polychoric-based PA-MRFA and polychoric-based
able performance of PA-MRFA. An increase in sample size Horn’s PA, the power of Pearson-based Horn’s PA to detect minor
yielded a slight decrease in performance for PA-MRFA (i.e., the factors indeed increased with increasing sample size. However,
percentage of correctly indicated major factors slightly dropped also in the condition with major factors only, Horn’s PA showed
from 99.85% to 99.74% to 97.87% for N ⫽ 200 vs. N ⫽ 500 vs. a decrease in performance with increasing sample size, which was
N ⫽ 2,000, respectively, for polychoric 95% threshold PA-MRFA completely due to overfactoring (0.3%, 0.4%, 0.4%, and 4.2% for
in Table 3 and from 99.86% to 99.50% for N ⫽ 500 vs. N ⫽ 2,000, sample sizes 100, 200, 500, and 2,000, respectively). Thus, there is
respectively, for Pearson 95% threshold PA-MRFA based on per- no compelling evidence that the power to detect minor factors
mutation in Table 3). For Horn’s PA, a similar pattern was found, increases with larger sample size for any of the PA procedures
but the decrease in performance appeared larger than for PA- examined.
MRFA (i.e., the percentages of correctly indicated major factors A comparison of the polychoric- and Pearson-based procedures
dropped from 99.85% to 99.49% to 94.33% for N ⫽ 200 vs. N ⫽ showed that the use of polychoric correlations yielded an improved
500 vs. N ⫽ 2,000, respectively, for polychoric 95% threshold performance, except for a slight decrease in performance for
218 TIMMERMAN AND LORENZO-SEVA

PA-MRFA with four categories. The slightly larger gains in case threshold appears best for Horn’s PA and the polychoric PA-
of two categories (compared with Cs ⫽ 4 and 5) suggest that MRFA. For Pearson PA-MRFA, the mean threshold tends to be
tetrachoric correlations may perform slightly better than the poly- comparable or slightly more favorable than the 95% threshold.
choric ones. Furthermore, the use of five- rather than four-ordered For polytomous items, the use of polychoric correlations ap-
categories yields an increase in performance. pears worthwhile considering. In our study, we found that for
From the previous evidence, it follows that polychoric-based indicating the number of common factors, polychoric correlations
procedures generally outperform their Pearson-based variants, pro- appeared to be more reliable than Pearson correlations. However,
vided that convergence is reached. In practice, the convergence of the Pearson-based variants are much needed in empirical practice.
polychoric correlations may pose severe problems. In our experi- The convergence problems of the polychoric approach prevent its
ment, the convergence rate across all 10,400 simulated data ma- general application to empirical data. Furthermore, polychoric
trices appeared to be as small as 37.01%. As can be concluded PA-MRFA appeared to be not always superior to Pearson PA-
from the last columns of Tables 2 and 3, tetrachoric correlation MRFA, namely, in the condition with four-ordered categories,
matrices converged more often than polychoric ones. Furthermore, where there was a slight decrement in performance. To improve
the problems tended to be smallest in computing tetrachoric cor- PA-MRFA for ordered polytomous items, it might be useful to
relations on the basis of a large sample size (e.g., N ⫽ 2,000) and develop a PA procedure based on a full-information approach to
for small numbers of observed variables (e.g., five). common factor analysis for polytomous items (Lee, Poon, &
Bentler, 1990).
To summarize, to assess the number of common factors of a
Discussion
polytomous data set in empirical practice, we suggest using the
polychoric 95% threshold PA-MRFA or, if convergence cannot be
The results of the simulation experiment show that performance
reached, the Pearson mean threshold PA-MRFA. It may be wise to
of PA-PAFA was inferior to those of Horn’s PA and PA-MRFA
apply the polychoric (or Pearson, in case of nonconvergence) 95%
and yielded a tendency to overfactor, as was expected. In contrast
threshold Horn’s PA as well. If PA-MRFA and Horn’s PA should
to our hypothesis, the use of permutation as the sampling technique
happen to indicate different numbers of factors, our simulation
rather than the use of draws from a normal distribution did yield
results may be of help for judging the potential source of this
similar performances, even in difficult conditions as in the case of
difference. For example, with a presumed large number of vari-
two categories. Thus, PA appears to be rather robust to deviations
ables per factor, PA-MRFA is more likely to be correct than
from normality and also in the context of ordered polytomous
Horn’s PA.
items. This finding is in line with earlier claims on the robustness
The recommendations provided are based on our simulation
of PA (Buja & Eyuboglu, 1992; Dinno, 2009). Our conjecture that
results. Although we aimed at selecting the conditions that repre-
a polychoric-based PA procedure would outperform its Pearson-
sent the typical cases found in empirical practice, it is not guar-
based variant was largely supported: Provided that convergence
anteed that we covered all relevant conditions. In particular, one
was reached, the polychoric variants indeed appeared to perform
may question whether the improved performance of the poly-
slightly better, except for PA-MRFA with four-ordered categories,
choric-based procedures with increasing numbers of response cat-
where polychoric performance was sometimes slightly less than
egories is maintained with more than five categories. Furthermore,
for Pearson variants. Furthermore, in line with our expectations,
the performance in cases of considerably more than six major
polychoric-based PA procedures tended to have slight advantage
factors would be worth examining.
over Pearson-based procedures particularly in large sample sizes
The manipulation of the presence of minor factors enabled us to
and with varying item difficulties, although both worked rather
examine whether PA procedures tend to indicate the number of
well in these conditions.
major factors or the total number of factors. Our results indicate
that the PA procedures considered consistently point at the number
Concluding Discussion of major factors, rather than the total number of factors. This
suggests that factors that are weakly related to the observed vari-
PA is among the most recommended methods for assessment of ables may be more difficult to detect with PA procedures. More-
the number of factors in empirical practice. Our simulation results over, the presence of minor factors appeared to cause deterioration
support this recommendation, provided that one is interested in of the PA performance in indicating the number of major factors,
finding major common factors and one applies an appropriate PA where both overfactoring and underfactoring were found. From a
procedure. On the basis of theoretical considerations and our theoretical point of view, those findings mean that the PA proce-
simulation results, we would advocate using the PA-MRFA pro- dures offer proper estimations of the rank of the reduced popula-
cedure. This procedure is well suited to assessment of the number tion correlation matrix only when the condition number of the
of common factors and showed regular behavior in the simulation population correlation matrix is reasonably low. From an empirical
study. However, the current popularity of Horn’s PA is not alarm- perspective, one might argue that this property is advantageous,
ing, because this method showed good performances in conditions because one will recover only the substantial factors and ignore the
that potentially could arise in empirical practice. In contrast, the factors that are weakly related to the observed variables. An
use of PA-PAFA should be discouraged. important remaining question for study is how small the “substan-
As a sampling technique, both draws from a multivariate normal tial” factors may be while still ensuring proper performance of the
distribution and permutation can be used, as they yield similar PA procedures and to what extent this depends on other charac-
results. In empirical practice, the normal draws may be favorable, teristics of the data, such as the number of ordered response
as this method is easiest to implement. As a threshold, the 95% categories and the number of variables per factor.
 PARALLEL ANALYSIS OF ORDERED POLYTOMOUS ITEMS 219

The simulated data were generated according to a common descriptors. Journal of Personality and Social Psychology, 94, 347–364.
factor model for polytomous items. This model perfectly matches doi:10.1037/0022-3514.94.2.347
the polychoric PA-MRFA procedure. It is of interest how robust Devlin, S. J., Gnanadesikan, R., & Kettenring, J. R. (1975). Robust
PA is to the data-generation mechanisms involved. We conjecture estimation and outlier detection with correlation coefficients. Bi-
ometrika, 62, 531–545. doi:10.1093/biomet/62.3.531
that PA is rather robust, because the performances of Horn’s PA
Devlin, S. J., Gnanadesikan, R., & Kettenring, J. R. (1981). Robust
and the Pearson-based variants, which do not fully match the estimation of dispersion matrices and principal components. Journal of
common factor model for polytomous items, appeared to be rather the American Statistical Association, 76, 354 –362. doi:10.2307/
close to that of the polychoric PA-MRFA procedure. For a thor- 2287836
ough assessment of the general applicability of PA in empirical Dinno, A. (2009). Exploring the sensitivity of Horn’s parallel analysis to
practice, it would be useful to examine the performance under the distributional form of random data. Multivariate Behavioral Re-
alternative models as well. The normal ogive and logistic item- search, 44, 362–388. doi:10.1080/00273170902938969
response models are closely related, or even equivalent, to the Dolan, C. V. (1994). Factor analysis of variables with 2, 3, 5 and 7
common factor model for polytomous items (Takane & de Leeuw, response categories: A comparison of categorical variable estimators
1987). Therefore, it would be much more interesting to consider a using simulated data. British Journal of Mathematical and Statistical
Psychology, 47, 309 –326.
nonparametric model, like the Mokken model (Mokken, 1971;
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J.
Sijtsma & Molenaar, 2002), as an alternative model to study the (1999). Evaluating the use of exploratory factor analysis in psycholog-
robustness of PA. ical research. Psychological Methods, 4, 272–299. doi:10.1037/1082-
989X.4.3.272
Finch, H., & Monahan, P. (2008). A bootstrap generalization of modified
References parallel analysis for IRT dimensionality assessment. Applied Measure-
ment in Education, 21, 119 –140. doi:10.1080/08957340801926102
Allen, S. J., & Hubbard, R. (1986). Regression equations for the latent Glorfeld, L. W. (1995). An improvement on Horn’s parallel analysis
roots of random data correlation matrices with unities on the diagonal. methodology for selecting the correct number of factors to retain.
Multivariate Behavioral Research, 21, 393–396. doi:10.1207/ Educational and Psychological Measurement, 55, 377–393. doi:
s15327906mbr2103_7 10.1177/0013164495055003002
Babakus, E., Ferguson, C. E., Jr., & Jöreskog, K. G. (1987). The sensitivity Hattie, J. A. (1985). Methodology review: Assessing unidimensionality of
of confirmatory maximum likelihood factor analysis to violations of tests and items. Applied Psychological Measurement, 9, 139 –164. doi:
measurement scale and distributional assumptions. Journal of Market- 10.1177/014662168500900204
ing Research, 24, 222–228. doi:10.2307/3151512 Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention
Bartlett, M. S. (1950). Tests of significance in factor analysis. British decisions in exploratory factor analysis: A tutorial on parallel analysis.
Journal of Psychology, 3, 77– 85. Organizational Research Methods, 7, 191–205. doi:10.1177/
Becker, P. (1999). Beyond the big five. Personality and Individual Differ- 1094428104263675
ences, 26, 511–530. doi:10.1016/S0191-8869(98)00168-8 Horn, J. L. (1965). A rationale and test for the number of factors in factor
Bernstein, I. H., & Teng, G. (1989). Factoring items and factoring scales analysis. Psychometrika, 30, 179 –185. doi:10.1007/BF02289447
are different: Spurious evidence for multidimensionality due to item Hubbard, R., & Allen, S. (1987). An empirical comparison of alternative
categorization. Psychological Bulletin, 105, 467– 477. doi:10.1037/ methods for principal component extraction. Journal of Business Re-
0033-2909.105.3.467 search, 15, 173–190. doi:10.1016/0148-2963(84)90047-X
Bonett, D. G., & Price, R. M. (2005). Inferential methods for the tetra- Humphreys, L. G., & Ilgen, D. R. (1969). Note on a criterion for the
choric correlation coefficient. Journal of Educational and Behavioral number of common factors. Educational and Psychological Measure-
Statistics, 30, 213–225. doi:10.3102/10769986030002213 ment, 29, 571–578. doi:10.1177/001316446902900303
Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, Humphreys, L. G., & Montanelli, R. G. (1975). An investigation of the
71, 425– 440. doi:10.1007/s11336-006-1447-6 parallel analysis criterion for determining the number of common fac-
Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model tors. Multivariate Behavioral Research, 10, 193–205. doi:10.1207/
fit. Sociological Methods & Research, 21, 230 –258. doi:10.1177/ s15327906mbr1002_5
0049124192021002005 Kaiser, H. F. (1960). The application of electronic computers to factor
Buja, A., & Eyuboglu, N. (1992). Remarks on parallel analysis. Multivariate analysis. Educational and Psychological Measurement, 20, 141–151.
Behavioral Research, 27(4), 509–540. doi:10.1207/s15327906mbr2704_2 doi:10.1177/001316446002000116
Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Lautenschlager, G. J. (1989). A comparison of alternatives to conducting Monte
Behavioral Research, 1, 245–276. doi:10.1207/s15327906mbr0102_10 Carlo analyses for determining parallel analysis criteria. Multivariate Behav-
Cho, S., Li, F., & Bandalos, D. (2009). Accuracy of the parallel analysis ioral Research, 24, 365–395. doi:10.1207/s15327906mbr2403_6
procedure with polychoric correlations. Educational and Psychological Lautenschlager, G. J., Lance, C. E., & Flaherty, V. L. (1989). Parallel
Measurement, 69, 748 –759. doi:10.1177/0013164409332229 analysis criteria: Revised equations for estimating the latent roots of
Cota, A. A., Longman, R. S., Holden, R. R., & Fekken, G. C. (1993). random data correlation matrices. Educational and Psychological Mea-
Comparing different methods for implementing parallel analysis: A surement, 49, 339 –345. doi:10.1177/0013164489492006
practical index of accuracy. Educational and Psychological Measure- Ledesma, R. D., & Valero-Mora, P. (2007). Determining the number of
ment, 53, 865– 876. doi:10.1177/0013164493053004001 factors to retain in EFA: An easy-to-use computer program for carrying
Cota, A. A., Longman, R. S., Holden, R. R., Fekken, G. C., & Xinaris, S. out parallel analysis. Practical Assessment, Research, and Evaluation,
(1993). Interpolating 95th percentile eigenvalues from random data: An 12, 1–11.
empirical example. Educational and Psychological Measurement, 53, Lee, S., Poon, W., & Bentler, P. M. (1990). Full maximum likelihood analysis of
585–596. doi:10.1177/0013164493053003001 structural equation models with polytomous variables. Statistics & Probability
De Raad, B., & Barelds, D. P. H. (2008). A new taxonomy of Dutch Letters, 9, 91–97. doi:10.1016/0167-7152(90)90100-L
personality traits based on a comprehensive and unrestricted list of Liu, O. L., & Rijmen, F. (2008). A modified procedure for parallel analysis
220 TIMMERMAN AND LORENZO-SEVA

for ordered categorical data. Behavior Research Methods, Instruments, Stata (Version 11) [Computer software]. (2009). College Station, TX:
& Computers, 40, 556 –562. doi:10.3758/BRM.40.2.556 StataCorp.
Longman, R. S., Cota, A. A., Holden, R. R., & Fekken, G. C. (1989). A Steger, M. F. (2006). An illustration of issues in factor extraction and
regression equation for the parallel analysis criterion in principal com- identification of dimensionality in psychological assessment data. Jour-
ponents analysis: Mean and 95th percentile eigenvalues. Multivariate nal of Personality Assessment, 86, 263–272. doi:10.1207/
Behavioral Research, 24, 59 – 69. doi:10.1207/s15327906mbr2401_4 s15327752jpa8603_03
Lorenzo-Seva, U., & Ferrando, P. J. (2006). FACTOR: A computer pro- Stuive, I., Kiers, H. A. L., Timmerman, M. E., & ten Berge, J. M. F. (2008).
gram to fit the exploratory factor analysis model. Behavior Research The empirical verification of an assignment of items to subtests: The
Methods, Instruments, & Computers, 38, 88 –91. doi:10.3758/ oblique multiple group method versus the confirmatory common factor
BF03192753 method. Educational and Psychological Measurement, 68, 923–939.
MacCallum, R. C., & Tucker, L. R. (1991). Representing sources of error doi:10.1177/0013164408315264
in the common-factor model: Implications for theory and practice. Takane, Y., & de Leeuw, J. (1987). On the relationship between item
Psychological Bulletin, 109, 502–511. doi:10.1037/0033-2909.109 response theory and factor analysis of discretized variables. Psy-
.3.502 chometrika, 52, 393– 408. doi:10.1007/BF02294363
Mokken, R. J. (1971). A theory and procedure of scale analysis: With ten Berge, J. M. F., & Kiers, H. A. L. (1991). A numerical approach to the
applications in political research. The Hague, the Netherlands: Mouton. approximate and the exact minimum rank of a covariance matrix.
Montanelli, R. G., & Humphreys, L. G. (1976). Latent roots of random data Psychometrika, 56, 309 –315. doi:10.1007/BF02294464
correlation matrices with squared multiple correlations on the diagonal: ten Holt, J. C., van Duijn, M. A. J., & Boomsma, A. (2010). Scale
A Monte Carlo study. Psychometrika, 41, 341–348. doi:10.1007/ construction and evaluation in practice: A review of factor analysis
BF02293559 versus item response theory. Psychological Test and Assessment Mod-
Muthén, B., & Kaplan, D. (1985). A comparison of some methodologies eling, 52, 272–297.
for the factor analysis of non-normal Likert variables. British Journal of Thompson, B. (2004). Exploratory and confirmatory factor analysis: Un-
Mathematical and Statistical Psychology, 38(2), 171–189. derstanding concepts and applications. Washington, DC: American
Muthén, L. K., & Muthén, B. O. (1998 –2010). Mplus user’s guide (6th Psychological Association. doi:10.1037/10694-000
ed.). Los Angeles, CA: Authors. Tucker, L. R., Koopman, R. F., & Linn, R. L. (1969). Evaluation of factor
O’Connor, B. P. (2000). SPSS and SAS programs for determining the analytic research procedures by means of simulated correlation matri-
number of components using parallel analysis and Velicer’s MAP test. ces. Psychometrika, 34, 4(Pt.1), 421– 459. doi:10.1007/BF02290601
Behavior Research Methods, Instruments, & Computers, 32, 396 – 402. Turner, N. E. (1998). The effect of common variance and structure pattern
doi:10.3758/BF03200807 on random data eigenvalues: Implications for the accuracy of parallel
Olsson, U. (1979a). Maximum likelihood estimation of the polychoric analysis. Educational and Psychological Measurement, 58, 541–568.
correlation coefficient. Psychometrika, 44, 443– 460. doi:10.1007/ doi:10.1177/0013164498058004001
BF02296207 Uebersax, J. (2010). User guide for POLYCORR 1.1 (Advanced version).
Olsson, U. (1979b). On the robustness of factor analysis against crude Retrieved from http://www.john-uebersax.com/stat/xpc.htm
classification of the observations. Multivariate Behavioral Research, 14, Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication
485–500. doi:10.1207/s15327906mbr1404_7 through factor or component analysis: A review and evaluation of
Pearson, K. (1901). On lines and planes of closest fit to systems of points alternative procedures for determining the number of factors or compo-
in space. Philosophical Magazine, 2, 559 –572. nents. In E. Helmes (Ed.), Problems and solutions in human assessment:
Peres-Neto, P. R., Jackson, D. A., & Somers, K. M. (2005). How many Honoring Douglas N. Jackson at seventy. (pp. 41–71). New York, NY:
principal components? Stopping rules for determining the number of Kluwer Academic/Plenum.
non-trivial axes revisited. Computational Statistics & Data Analysis, 49, Watkins, M. (2010). Monte Carlo PCA for parallel analysis [Computer
974 –997. doi:10.1016/j.csda.2004.06.015 software]. Retrieved from http://www.public.asu.edu/⬃mwwatkin/
R Development Core Team. (2005). R: A language and environment for Watkins3.html
statistical computing (Reference Index Version 2.x.x) [Computer soft- Weng, L., & Cheng, C. (2005). Parallel analysis with unidimensional
ware]. Vienna, Austria: R Foundation for Statistical Computing. binary data. Educational and Psychological Measurement, 65, 697–716.
Reddon, J. R., Marceau, R., & Jackson, D. N. (1982). An application of doi:10.1177/0013164404273941
singular value decomposition to the factor analysis of MMPI items. Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for
Applied Psychological Measurement, 6(3), 275–283. doi:10.1177/ determining the number of components to retain. Psychological Bulle-
014662168200600303 tin, 99(3), 432– 442. doi:10.1037/0033-2909.99.3.432
Roznowski, M., Tucker, L. R., & Humphreys, L. G. (1991). Three approaches to
determining the dimensionality of binary items. Applied Psychological Mea-
surement, 15, 109–127. doi:10.1177/014662169101500201 Received October 28, 2009
Sijtsma, K., & Molenaar, I. W. (2002). Introduction to nonparametric item Revision received January 20, 2011
response theory. Thousand Oaks, CA.: Sage. Accepted January 26, 2011 䡲

View publication stats