JOURNAL OF RESEARCH IN SCIENCE, MATHEMATICS AND TECHNOLOGY EDUCATION

DOI: 10.31756/jrsmte.223

http://www.estej.com/

The Development and Validation of a 21st Century Skills Instrument:

Measuring Secondary School Students’ Skills

Alpaslan Sahin
Harmony Public Schools, USA
Mirim Kim
Myeongsun Yoon
Texas A&M University, USA

Abstract: Due to the rapid change in technology and information dissemination, the qualities and skills employers
and colleges demand in the 21st century have changed. To help higher education institutions and workforce to
identify and measure their prospective students and employees’ skills respectively, we designed an instrument for
secondary grade students to self-assess their 21st century skills. After successful piloting, validation of the final
instrument was done with 282 high school students from a public high school in Texas. We utilized exploratory
factor analysis and investigated construct validity for the instrument using principal axis factoring with Promax
rotation and Kaiser normalization. We found that the original 48 items developed for the instrument were loading the
four factors as theorized in our model. Finally, confirmatory factor analysis (CFA) models for four scales were
separately investigated. Maximum likelihood estimation method was used for all analyses though Mplus8.2 (Muthén
& Muthén, 1998-2017). We came up with 5 factors and 43 items. Researchers, K-12 educators, postsecondary
educators, and employers may benefit from the development of this instrument.
Keywords: 21st century skills; Exploratory factor analysis; Instrument development; Principal axis
factoring; Confirmatory factor analysis.

late 20th century is not very long ago,

Introduction
differences in skills needed between then and
In today’s world, with the evolution of every now are profound, well known, and well
aspect of human life pioneered by documented. For example, collaboration was
technological and scientific advances, it is not a type of skill that was needed as critically
indispensable to have transformed the in the 20th century but it has become an
expectations and standards for the skills and essential skill in the 21st century. When
competencies that individuals should possess contemplating about the essential skills that
for both work and daily life. The changes in are in the rise in the 21st century, it is
standards and expectations are indeed a important to analyze the way we understand
paradigm shift in skills needed for life—a some relevant constructs and how they have
shift that occurred in transitioning from the altered and/or transcended their original
20th century to the 21st century. Although the essence. For instance, cognition has evolved
86 | S A H I N , K I M , & Y O O N

thanks to research on neurosciences and teach/learn, with what to teach/learn, when to

cognitive and developmental psychology teach/learn, how to teach/learn, how to assess
resulting in implications for teaching and learning, and so forth. Reformed-based
student learning (Mintz, 2014). Cognition teaching and standards movements are some
moved towards the forefronts of learning policies have come to life as a response to
theories (e.g., Piaget, 1950) way before the these developments.
turn of the new century, however, it was not
until recently that it truly made it to the The key question is then: What are the “new”
educational practice, after utilization of skillsets that are needed now or in the near
technological advances, educational future? Also, how do we define them? In
technology, in particular (Saettler, 2004). defining 21st century skillsets, many parties
should be involved in the process since the
The shift also rendered educational goals and new set of skills is overarching and more
objective to be revisited and reformed sophisticated than they were in the last
because they could not stay static. Bloom’s century (Dede, 2010). It is a collaborative
Taxonomy (Bloom, 1984) for educational task with involvement from educators,
goals, for example, which have been applied education experts, researchers, workforce
by many K-16 educators for decades, has leaders, and business partners to define and
been revised by cognitive psychologists, illustrate the skills, knowledge, expertise, and
educational theorists and researchers, and support systems that students need to succeed
assessment specialists. Rather than simple in work, life, and citizenship (Partnership for
nouns of the original taxonomy, verbs and 21st Century Learning [P21], 2009; 2016). In
gerunds were added to the names of very basic and commonly acceptable terms,
categories to underline the dynamism 21st 21st century skills refer to certain core skills
century brought in (Anderson et al., 2001, and competencies such as collaboration,
Churches, 2008). Moreover, we live in a problem-solving, digital literacy, knowledge
digital age in which technology is in key subjects such as mathematics and
progressively ubiquitous. This is why many science, and social and leadership skills
new educational terms have been introduced students nearing end of compulsory
with the turn of the century such as “digital education need to succeed in work, life, and
literacy.” This necessitates reforms in what to citizenship (Binkley et al., 2012; P21, 2016;
 J. of Res. in Sci. Math. and Tech. Edu.| 87

Rich, 2010). The next section describes 2010). Also, basic skills and its application
frameworks that are used to define 21st go hand in hand (Rotherham & Willingham,
century skills and how learning should look 2010; Silva, 2009).
like within the framework.
Several researchers reviewed and
Theoretical Framework
synthesized different frameworks for 21st
Inculcating 21st century skills is not about century skills (e.g., Dede, 2010; Mishra &
teaching new skills, but in fact, breathing life Kereluik, 2011; Voogt & Roblin, 2012). The
and giving importance to the otherwise most common frameworks—found in the
dormant skills that were not much in demand literature and also included in these different
in the 20th century (Silva, 2009). The advent syntheses and in this study—were P21
of technology automated the mundane and (2009), enGauge (North Central Regional
menial jobs, thus, resulting in the decrease Educational Laboratory [NCREL]& Meriti
amount of people required to put in effort for Group, 2003), International Society for
such work (Levy & Murnane, 2004). Even in Technology in Education (2016), and
existing jobs, there arise cases where there is Organisation for Economic Cooperation and
no written solution and in such situations Development (OECD; 2005). Although there
professionals have to improvise and adapt is some inconsistency in the origin of
using their knowledge and experience to different frameworks (i.e., goals, intentions)
figure out a remedy (Dede, 2010). Most of the and the implications they extend (i.e.,
workforce is now looking for employees that practice), it is fortunate to see greater
can perform more than trivial tasks. They alignment across different frameworks in
must be able to comprehend their role and terms of what overarching skillsets are and
realize its functioning (Silva, 2008). There why they are important (Dede, 2010; Voogt
are different arguments as to what skills & Roblin, 2012). The underlying skillsets
constitute the 21st century skills and common across all these different
furthermore, in what manner should they be frameworks can be grouped into four main
taught. There are many proponents for categories: (a) learning and innovation skills
development of new standards, assessments, (Ongardwanich, Kanjanawasee, & Tuipae,
and curriculum that reflect the needs of the 2014; Saavedra & Opfer, 2012; Lombardi,
21st century (Rotherham & Willingham, 2007; Pacific Policy Research Center
88 | S A H I N , K I M , & Y O O N

[PPRC], 2010), (b) life and career skills Lombardi, 2007). Effective and interactive
(Bell, 2010; Ongardwanich, Kanjanawasee, communication and collaboration skills are
& Tuipae, 2014; P21, 2016), (c) also crucial as learning is fundamentally a
interdisciplinary themes (literacy; social activity in many ways (Dron &
Association of American Colleges and Anderson, 2014; Trilling & Fadel, 2009).
Universities, 2007; NCREL & Meriti Group, Thus, individuals should be able to articulate
2003; OECD, 2005; Schneider, 1997), and ideas effectively using different
(d) information, media, and technology skills communication means and work in harmony
(Ongardwanich, Kanjanawasee, & Tuipae, and productively in groups (Pacific Policy
2014; PPRC, 2010; P21, 2016). These Research Center [PPRC], 2010).
categories are explained below and are used
as the framework for the 21st Century Skills Life and career skills basically refer to social
Instrument (21CSI) developed in this study. and emotional competencies needed to
The 21CSI instrument developed in this navigate through the complex life and work
study draws on these four overarching environments (P21, 2016). Being able to
constructs each of which has up to four sub- work effectively even in ambiguous
constructs. environments and to adapt to a variety of
context and situations (flexibility and
Learning and innovation skills are critical to adaptability), understanding and embracing
be creative and a lifetime learner as they may cultural and social difference (social and
be also called as “survival skills” (Savedra & cross cultural skills), transcending mastery
Opher, 2012, p. 8). The new living standards, skills and self-monitoring (initiative and self-
world of work, and contemporary citizenship direction), and working with the interest of
constantly demand higher levels of thinking, the larger community in mind and inspiring
communication, and collaboration. In this others by example (leadership and
new era, individuals are expected to develop responsibility) are all important constructs
and create new ideas and to respond to new related to life and career skills (Bell, 2010;
and diverse perspectives (creativity and PPRC, 2010; P21, 2016).
innovation; P21, 2016). Critical thinking and
problem solving are now considered the new Interdisciplinary themes transcend the
basics of learning (Anderson et al., 2001; traditional core subject areas (i.e, reading,
 J. of Res. in Sci. Math. and Tech. Edu.| 89

social sciences, math, and science) and critically and use them effectively, and
emphasize the contemporary literacy understand social, political, and economic
elements such as civic literacy, issues surrounding the use of information
environmental literacy, and global (PPRC, 2010).
awareness. There is the need for students to
be able to participate in the civic society by It should be noted that besides the four
understanding and actively contributing to constructs there are specified above, all the
civic decisions (civic literacy; OECD, 2005). frameworks include in some form or other for
Environmental issues, whether caused by the students to possess basic and holistic
human or occurred naturally, present forms of literacy which includes but not
significant problems to humanity and nature. limited to science, mathematics, arts,
It is very important for students to understand languages, and social sciences.
and discuss environmental issues and
propose and evaluate a variety of solutions to Purpose of the Research

these problems (environmental literacy; Our goal of this study is to develop and

Schneider, 1997). In addition, in today’s validate an instrument designed to measure

world, students should be able to work with secondary school students’ self-assessment

and exchange knowledge exchange with of 21st century skills they may have

people from diverse background. However, developed during K-12 school years.

achieving this goal in the most successful Although several indicators and ways of

way relies on having a global awareness, measuring students 21st century skills were

which requires knowing, respecting, and offered and developed (see Griffin & Care,

understanding different cultures, religions, 2015; Ongardwanich, Kanjanawasee, &

lifestyles, ideologies, and political contexts Tupae, 2014; Soh, Osman, & Arsad, 2012;

(global awareness; NCREL & Meriti Group, Trilling & Fadel, 2009), there is a scarcity of

2003). instruments that delve into students’

perceptions about their own skills and that are

Information, media, and technology skills are comprehensive enough to cover all aspects of

used by effective citizens and workers to the 21st century skills framework introduced

determine the extent of information needed above. Moreover, there have been various

and access it, evaluate sources of information attempts in order to measure the advanced
90 | S A H I N , K I M , & Y O O N

skills required in the modern era (CRESST, Methods

CWRA, River City, IBD, Key Stages 3 ICT Participants
Literary Assessment, NAEP). However, we Participating school was an urban public
are in a situation where we do not need more charter high school in west Texas where
tests but better assessments (Silva, 2008). majority of students are Hispanics in west
st
There is a need, not only for 21 century Texas. The participating school was
skills, but a viable measurement to assess representing the city demographics and
them on a large scale. The tests listed may be majority of its students were Hispanics. First,
intensive in their approach, but they have we distributed parent consent forms to
drawbacks in the form of expense, logistics students who are younger than 18 years old.
and administration. Out of 400 students, 321 returned the parent
consent forms. We sent the instrument to 321
In addition, existing instruments either deal 9-12 grade. Of those, 282 (88%) of them
with only some subdomains of this completed the survey. The study participants
framework (e.g., digital literacy; see Calvani, were 44% male, 82% Hispanic, 5% black,
Cartelli, Fini, & Ranieri, 2009) or are limited 11% white, and 3% Asian. Their grade
to certain contexts (e.g., Osman, Soh, & demographics were 30% 9th, 24% 10th, 24%
Arsad, 2010). In response to this need in the 11th, and 22% 12th grade students.
field, the present study details the steps used
to develop and validate an instrument to Development of the Instrument
measure students’ self-perceptions on their The cross-sectional survey method was used
acquisition of 21st century skills. The 21CSI to identify 21st century skills elements for
instrument developed in this study will secondary grade students. Through an
provide a more convenient and sufficient way extensive review of the related literature and
of evaluating secondary grade students’ 21st focus group discussions, we came up with the
century skills by asking them how they see constructs that best represent what secondary
what they learned during their secondary grade students need to learn and can learn
years. through their high school education in this
era. The focus group crew included math and
science educators, science and STEM
university professors. Each group read the
 J. of Res. in Sci. Math. and Tech. Edu.| 91

literature chosen for the study and reviewed included in the instrument. Another one
the instruments related to the focus of the suggested including several negative format
study. Then, they discussed their groups of type questions would increase the reliability
skills with each other groups and agreed on of the survey. Our language expert
constructs and subconstructs after 2 two- recommended us to have multiple shorter
hour meetings. Finally, they came together items instead of one long item.
discuss their item sentences. This took two
half-day study sessions. All of these study After addressing the recommendations, the
groups yielded four factors,11 sub-factors, final instrument constructed included 48
and 37 items. items for self-assessment of secondary grade
students’ 21st century skills in four domains:
Then, we sent the instrument to three The first section of the survey, learning and
nationally known researchers (George innovation skills (LIS), is comprised of five
Washington University, Texas A&M sections: creativity and innovation (3
University, and University of Texas-Austin) questions), critical thinking and problem
with expertise on 21st century skills and solving (5 questions), communication (4
STEM education for content validity questions), initiative and self-direction (4
(Lawshe, 1975). Three experts were asked to questions), and leadership and responsibility
provide comments and suggestions for each (3 questions). The second main construct is
construct and items on its content and life and career skills (LCS) with three
category. We also sent the instrument to a sections—flexibility and adaptability (4
linguistic expert (Duquesne University) for questions), social and cross-cultural skills (4
language validity. We were provided questions), and collaboration (4 questions).
comments and feedbacks regarding the The third construct, interdisciplinary themes
appropriateness of each item for both native (IT), has three sections: global awareness (3
and international secondary grade students’ questions), civic literacy (3 questions), and
language level. Upon receiving all those great environmental literacy (4 questions). The
inputs, the research group met one more time fourth construct is information, media, and
to go over the comments and suggestions to technology skills (IMTS) with seven items.
re-write the items. For instance, one expert All questions ask participants’ beliefs about
worried about too much affirmative questions how confident they are on the statements
92 | S A H I N , K I M , & Y O O N

given under each sub-construct (e.g., “I can extraction and rotation indicated that four
analyze and evaluate alternative points of factors explain 63.53% cumulatively of the
view”). The responses to these 48 questions total variance in the data. Then, we re-did the
in total are on a 7-point Likert-scale (7=Very PCA with direct oblimin rotation with 4
Confident to 1=Not Confident at All). factors.

Data Analysis Because we found correlations greater than

Confirmatory factor analysis (CFA) models .32, we used one of the oblique rotations—
for four scales were separately investigated. Promax with Kaiser normalization as part of
Maximum likelihood estimation method was our exploratory factor analysis (EFA) with
used for all analyses though Mplus8.2 principal axis factoring as common factor
(Muthén & Muthén, 1998-2017). analysis to uncover the latent constructing
underlying the variables. We successfully
Results
achieved a simple structure (Brown, 2009)
First, we ran a PCA with scree plot without that explained all the original 48 items with
any rotations to determine the number of new four factors that was proposed in the study.
and reduced factors that will come out. We
extracted 6 factors which had eigenvalues Then, confirmatory factor analysis (CFA)
greater than 1. These 6 factors account for models for four scales were separately
68% of the variance cumulatively. According investigated. Maximum likelihood
to the Kaiser-Guttman rule, factors with estimation method was used for all analyses
Eigenvalues greater than 1 should be through Mplus8.2 (Muthén & Muthén, 1998-
accepted. 2017).

To cross check, we also examined the scree

plot (see Figure 1). According to the scree
plot, it shows the sharp descent of the
Eigenvalues of factor 1 to factor 4, and a
leveling off from factor 5 onwards suggesting
that four factors should be rotated in the
survey items. The results of the factor
 J. of Res. in Sci. Math. and Tech. Edu.| 93

Figure 1. Scree plot of eigenvalues.

A scale of learning and innovation skills direction; (5) Leadership and responsibility.
(LIS) is constructed with five subscales: (1) A single factor model that is 19 items loaded
Creativity and innovation; (2) Critical on the single factor was fitted to the data (see
thinking and problem solving; (3) Table 1).
Communication; (4) Initiative and self-
94 | S A H I N , K I M , & Y O O N

Table 2

Global Fit Indices for CFA Models

Chi2 CFI RMSEA SRMR

Learning and Innovation Skills (19)
Single factor 399.487 (152) .928 .076 .041
Life and Career Skills (10)
2-factor 116.166 (34) .972 .093 .037
Modified 2-factor* 84.057 (33) .974 .074 .028
Interdisciplinary Themes (7)
Single factor 27.007 (14) .991 .058 .016
Information, Media, and Technology Skills (7)
Single factor 65.623 (14) .959 .116 .031
Modified single factor* 35.681(13) .980 .085 .023
Note. * A residual correlation was freed based on the modification indices

The original item set of Life and career skills three items loaded on the LCS-b factor. Due
(LCS) scale had 15 items under four to the low fit of RMSEA (.093), we freed a
subscales: (1) Flexibility and adaptability; (2) residual correlation between FA4 (Dealing
Social and cross-cultural skills; (3) positively with setbacks and criticism) and
Collaboration; (4) Global awareness. COL4 (Valuing individual contributions of
However, some of items were dropped from others for collaborative work) based on the
the final set of items based on the EFA result, modification indices. The global fit for the
because three items (i.e., FA2, SSCS2, and modified 2-factor model was fair (CFI = .974,
SSCS3) had cross-loadings on more than two RMSEA = .074, SRMR = .028, in Table 2).
other factors not relevant to the LCS factor, A factor correlation between two factors was
and two items (i.e., GA2 and GA3) had a .938, and the freed residual correlation was
cross-loading over .30. A 2-factor model .347.
representing ‘Life and career skills-a and
‘Life and career skills-b’ factors was tested. Interdisciplinary themes (IT) and
Seven items loaded on the LCS-a factor and Information, media, and technology skills
 J. of Res. in Sci. Math. and Tech. Edu.| 95

(IMTS) scales are constructed by seven media forms) and IMTS7 (Effectively
items, respectively. A single factor model utilizing technology in variety of ways as they
was tested for both constructs, and the IT relate to life), and the freed residual
single factor model was fitted to the data well correlation was .339. The global fit of the
(CFI = .991, RMSEA = .058, SRMR = .016, modified single factor model was better (CFI
in Table 1). = .980, RMSEA = .085, SRMR = .23, in
Table 1).
On the other hand, the IMTS single factor
model showed the low fit of RMSEA. Finally, each item was assessed for its
Modification index suggested a residual internal consistency. All factors had high
correlation between IMTS2 (Using various Cronbach alpha reliability (Table 2).

Table 2
21CSS Instrument Cronbach Alpha Reliability Scores
# of items Factors Internal Consistency
19 Learning and innovation skills .954
7 Life and career skills-a .924
3 Life and career skills-b .797
7 Interdisciplinary themes .934
7 Information, media, and technology skills .921

21st century skills instrument at a local high

Discussion and Conclusions
school in Texas. We found that the finalized
Due to rapid change in technology and 43 items for the instrument successfully
information dissemination systems, the measured the high school students’ core
qualities and skills employers and colleges skills with 5 factors and 11 skills.
demand in the 21st century have changed. To
help higher education institutions and With the evolution of every aspect of human
workforce teams to identify and measure life pioneered by technological and scientific
their prospective students’ and employees’ advances, it has become indispensable to
skills respectively, we designed and tested a have changed and improved expectations and
96 | S A H I N , K I M , & Y O O N

standards for the skills and competencies that Although some of these were either subject
individuals should possess for both work and specific or focus only a few of several
daily life (Dede, 2010; P21, 2016; National dimension of 21-CSs (Greiff & Kyllonen,
Center on Education and the Economy, 2016; Griffin & Care, 2015), only few of
2007). As a response to this transformation in them have developed an instrument that was
skills and competencies, in this study, we close to what we wanted to accomplish (e.g.,
developed and validated the 21CSI designed Ongardwanich, Kanjanawasee, & Tuipae,
to measure secondary school core students’ 2015; Soh, Osman, & Arsad, 2012). The
21st century skills. Although, our advantage of the current 21CSI instrument is
study/instrument is similar to what that we accomplished to measure 11
Ongardwanich, Kanjanawasee, and Tuipae research-recommended 21st century skills all
has developed in 2014, our instrument at once.
measures several more skills with less items
including Global citizenship, Civic, and Limitations and Future Research

Environmental literacy. Also, our instrument One of the limitations of this research is about

is designed to measure high school students’ its sample size. The sample size we had was

self-assessment of their 21st century skills not enough to measure five factor CFA

development after each year. model. Future study with bigger sample size
might have better fit of the 5-factor CFA

Our goal was to develop an instrument that model, then factor relationships can be

was an easy-to-use/administer and cost- examined. Another limitation and

effective way of surveying students about improvement area would be about its

their skills. Another aim was to work on an language. Although we consulted with a

instrument that was comprehensive enough linguistic expert about its language and

to cover all aspects of the 21-CSs framework length, the language of the instrument may

introduced above. There have been various need more attention to simplify its language.

attempts to measure the advanced skills

required in the modern era (e.g., CRESST, Future steps may include translation or trans-

Key Stages 3 ICT Literary Assessment, adaptation of this instrument to native

NAEP; Ongardwanich, Kanjanawasee, & languages of students from an international

Tuipae, 2015; Soh, Osman, & Arsad, 2012). audience since 21-CSs are considered
 J. of Res. in Sci. Math. and Tech. Edu.| 97

universal. Future studies may also include to mathematics performance or science

explore predictive value of this measure on performance. Longitudinal studies would be
students’ future career choices. Cross also interesting to see how 21-CSs help
validation of this instrument with other students succeed in college and in life after
methods of measuring these skills is also college. 21CSI may also help schools and
needed. schools district to ensure that all students
receive the most empowering forms of
Implications learning that can get them to be college-
This instrument may help both researchers ready. Higher education may also benefit
and practitioners. Researchers may use this to from this form of investigative lenses onto
extend and expand research efforts in whether their students have the skills needed
investigating students’ 21-CSs and factors to find a good job and succeed in life after-
affecting these skills. Particularly interesting college (Association of American Colleges
area of investigation would be linking 21- and Universities, 2007).
CSs to academic achievement such as

References

Anderson, L. W., Krathwohl, D. R., Airasian, P., Cruikshank, K., Mayer, R., Pintrich, P., ...
Wittrock, M. (2001). A taxonomy for learning, teaching and assessing: A revision of
Bloom’s taxonomy. New York. Longman Publishing.
Association of American Colleges and Universities. (2007). College learning for the new global
century: A report from the National Leadership Council for Liberal Education &
America's Promise. Washington, DC: Author. Retrieved from
https://www.aacu.org/sites/default/files/files/LEAP/GlobalCentury_final.pdf
Bell, S. (2010). Project-based learning for the 21st century: Skills for the future. The Clearing
House, 83(2), 39–43.
Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M.
(2012). Defining twenty-first century skills. In P. Griffin, B. McGaw, & E. Care (Eds.),
Assessment and teaching of 21st century skills (pp. 17–66). New York: Springer.
98 | S A H I N , K I M , & Y O O N

Bloom, B. S. (Ed.) (1984). Taxonomy of educational objectives (Handbook 1: Cognitive

domain). New York: Longman.
Brown, J. D. (2009). Choosing the right type of rotation in PCA and EFA. JALT Testing and
Evaluation SIG Newsletter, 13(3), 20–25.
Calvani, A., Cartelli, A., Fini, A., & Ranieri, M. (2009). Models and instruments for assessing
digital competence at school. Journal of E-learning and Knowledge Society, 4(3), 183–
193.
Churches, A. (2008). Bloom’s digital taxonomy. Retrieved from
http://burtonslifelearning.pbworks.com/f/BloomDigitalTaxonomy2001.pdf
Dede, C. (2010). Comparing frameworks for 21st century skills. In J. A. Bellanca & R. S. Brandt
(Eds.), 21st century skills: Rethinking how students learn (pp. 51–76). Bloomington, IN:
Solution Tree Press.
Dron, J., & Anderson, T. (2014). Teaching crowds: Learning and social media. Edmonton,
Canada: Athabasca University Press.
Fan, S., & Le, Q. (2011). Developing a valid and reliable instrument to evaluate users’
perception of web-based learning in an Australian university contexts. Journal of Online
Learning and Teaching, 7(3), 366–379.
Greiff, S., & Kyllonen, P.C. (2016). Contemporary assessment challenges: The measurement of
21st century skills. Applied Measurement in Education, 29(4), 243–244.
Griffin, P., & Care, E. (Eds.). (2015). Assessment and teaching of 21st century skills: Methods
and approach. New York: Springer.
International Society for Technology in Education. (2016). ISTE standards for students.
Retrieved from http://www.iste.org/
Lawshe, C. H. (1975). A quantitative approach to content validity 1. Personnel
Psychology, 28(4), 563–575.
Levy, F., & Murnane, R. J. (2004). The new division of labor: How computers are changing the
way we work. Princeton, NJ: Princeton University Press.
Lombardi, M. M. (2007). Authentic learning for the 21st century: An overview. In D. G.
Oblinger (Ed.), Educause Learning Initiative: Advancing learning through IT innovation
(pp. 1–12). Retrieved from https://library.educause.edu/resources/2007/1/authentic-
learning-for-the-21st-century-an-overview
 J. of Res. in Sci. Math. and Tech. Edu.| 99

Mintz, S. (2014, March 5). Five ways that 21st and 20th century learning will differ: Proficiency,
data, science, and more. Inside Higher Ed. Retrieved from
https://www.insidehighered.com/blogs/higher-ed-gamma/five-ways-21st-and-20th-
century-learning-will-differ
Mishra, P., & Kereluik, K. (2011). What 21st century learning? A review and a synthesis. In M.
Koehler & P. Mishra (Eds.), Proceedings for the 22nd International Conference for the
Society for Information Technology & Teacher Education (pp. 3301–3312). Chesapeake,
VA: Association for the Advancement of Computing in Education.
National Center on Education and the Economy. (2007). Tough choices or tough times: The
report of the new commission on the skills of the American workforce. Washington, DC:
Author. Retrieved from http://www.ncee.org/wp-content/uploads/2010/04/Executive-
Summary.pdf
North Central Regional Educational Laboratory, & Meriti Group. (2003). enGauge® 21st
Century Skills: Literacy in the Digital Age.
Organisation for Economic Cooperation and Development. (2005). The definition and selection
of key competencies: Executive summary. Paris, France: OECD.
Osman, K., Soh, T. M. T., & Arsad, N. M. (2010). Development and validation of the Malaysian
21st century skills instrument (M-21CSI) for science students. Procedia-Social and
Behavioral Sciences, 9, 599–603.
Pacific Policy Research Center. (2010). 21st century skills for students and teachers. Honolulu:
Kamehameha Schools. Retrieved from
http://www.ksbe.edu/_assets/spi/pdfs/21_century_skills_full.pdf
Partnership for 21st Century Learning. (2009). Results that matter: 21st Century skills and high
school reform. Tucson, AZ: Author.
Partnership for 21st Century Learning. (2016). Framework for 21st century learning. Washington,
DC: Author. Retrieved from www.p21.org
Piaget, J. (1950). The psychology of intelligence. Routledge.
Rich, E. (2010). How do you define 21st-century skills? One question. Eleven answers.
Education Week, 4(1), 32–35.
Rotherham, A. J., & Willingham, D. T. (2010). “21st-Century” Skills. American Educator, 17.
100 | S A H I N , K I M , & Y O O N

Saavedra, A. R., & Opfer, V. D. (2012). Learning 21st-century skills requires 21st-century
teaching. Phi Delta Kappan, 94(2), 8–13.
Saettler, P. (2004). The evolution of American educational technology. Greenwich, CT:
Information Age Publishing.
Schmidt, D. A., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S. (2009).
Technological pedagogical content knowledge (TPACK) the development and validation
of an assessment instrument for preservice teachers. Journal of Research on Technology in
Education, 42(2), 123–149.
Schneider, S. (1997). Defining environmental literacy. Trends in Ecology & Evolution, 12(11),
457.
Silva, E. (2008). Measuring skills for the 21st century (Education Sector Reports). Washington,
DC: Education Sector.
Silva, E. (2009). Measuring skills for 21st-century learning. Phi Delta Kappan, 90(9), 630–634.
Tabachnick, B. G. & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Upper Saddle
River, NJ: Pearson Allyn & Bacon.
Trilling, B. & Fadel, C. (2009). 21st century skills: Learning for life in our times. Retrieved from
http://21stcenturyskillsbook.com/blog/q-a/
Voogt, J., & Roblin, N. P. (2012). A comparative analysis of international frameworks for 21st
century competences: Implications for national curriculum policies. Journal of Curriculum
Studies, 44(3), 299–321.

APPENDIX

21st Century Skills Instrument (21CSI)

43 items
Please state your level of confidence in performing/fulfilling the following tasks.
(1=Not confident at all, 3=Somewhat confident, 5=Confident, 7=Very confident)

A. Learning and Innovation Skills (19)

1. Creativity and Innovation
1.1. Developing new and innovative ideas.
 J. of Res. in Sci. Math. and Tech. Edu.| 101

1.2. Responding to new and diverse perspectives.

1.3. Working on information from a variety of perspectives when developing ideas.
2. Critical Thinking and Problem Solving
2.1. Reasoning effectively in making complex decisions.
2.2. Asking relevant questions during the problem-solving process.
2.3. Solving complex problems.
2.4. Analyzing and evaluating alternative points of view.
2.5. Providing feedback on decisions and processes.
3. Communication
3.1. Articulating thoughts and ideas using written communication skills.
3.2. Articulating thoughts and ideas orally and nonverbally (e.g., gestures) in a SMALL
group environment.
3.3. Articulating thoughts and ideas orally and nonverbally (e.g., gestures) in a LARGE
group environment.
3.4. Listening effectively to interpret meaning.
4. Initiative and Self-Direction
4.1. Going beyond basic mastery skills to gain new expertise.
4.2. Going beyond basic curricula to expand my own learning.
4.3. Defining and prioritizing tasks without direct oversight.
4.4. Completing tasks without any supervision.
5. Leadership and Responsibility
5.1. Working with the interest of the larger community in mind.
5.2. Inspiring others by example.
5.3. Capitalizing on the strengths of others to achieve a common goal.

B. Life and Career Skills-a (3)

1. Flexibility and Adaptability
1.1. Assuming and fitting in different roles, schedules, and contexts.
1.2. Performing tasks successfully without praises or other external rewards.
1.3. Dealing positively with setbacks and criticism.
102 | S A H I N , K I M , & Y O O N

C. Life and Career Skills-b (7)

2. Social and Cross-Cultural Skills
1.1. Understanding and embracing cultural and social differences.
1.2. Conducting myself in a respectful and professional manner.

3. Collaboration
2.1. Working effectively and respectfully with people from diverse backgrounds.
2.2. Including others’ perspectives when making decisions.
2.3. Assuming shared responsibility for collaborative work.
2.4. Valuing individual contributions (of others) for collaborative work.

4. Global Awareness
3.1. Understanding the thinking of people from different backgrounds (e.g., cultures,
religions, ideologies, and life styles).

D. Interdisciplinary Themes (7)

1. Civic Literacy
1.1. Making sense of governmental processes locally or globally.
1.2. Understanding the local or global implications of civic decisions.
1.3. Participating in civic life (e.g., by influencing decision-making).
2. Environmental Literacy
2.1. Understanding complex environmental issues.
2.2. Evaluating proposed environmental plans.
2.3. Assessing the risks of proposed environmental plans.
2.4. Understanding how individual decisions affect the environment at local or global
scales.

E. Information, Media, and Technology Skills (7)

1.1. Understanding ethical issues surrounding the production of information on the
media.
1.2. Using various media forms.
 J. of Res. in Sci. Math. and Tech. Edu.| 103

1.3. Critiquing the inclusion or exclusion of opinions or factual information on the

media.
1.4. Effectively creating and delivering media products.
1.5. Recognizing when and to what extent information is needed.
1.6. Evaluating and use the needed information effectively.
1.7. Effectively utilizing technology in variety of ways as they relate to life (e.g., web
tools, games, and software).

Author name: Alpaslan Sahin, Ph.D.

University, Country: Harmony Public Schools, US
Email [email protected]

Please Cite: Sahin, A., Kim, M., & Yoon, M. (2019). The Development and Validation of a 21st Century Skills
Instrument: Measuring Secondary School Students’ Skills. Journal of Research in Science, Mathematics and
Technology Education, 2(2), 85-103. doi: 10.31756/jrsmte.223

Received: April 12, 2019 ▪ Accepted: May 7, 2019