ISSN 1799-2591 Theory and Practice in Language Studies, Vol. 3, No. 2, pp.

254-262,
February 2013 © 2013 ACADEMY PUBLISHER Manufactured in Finland.
doi:10.4304/tpls.3.2.254-262

Mixed Method Research: Instruments,

Validity, Reliability and Reporting Findings
Mohammad Zohrabi (Corresponding
​ niversity of Tabriz, Iran
author) U

Abstract― ​ The mixed method approaches have recently risen to prominence. The reason that more
researchers are opting for these types of research is that both qualitative and quantitative data are
simultaneously collected, analyzed and interpreted. In this article the main research instruments
(questionnaire, interview and classroom observation) usually used in the mixed method designs are
presented and elaborated on. It is believed that using different types of procedures for collecting data
and obtaining that information through different sources (learners, teachers, program staff, etc.) can
augment the validity and reliability of the data and their interpretation. Therefore, the various ways of
boosting the validity and reliability of the data and instruments are delineated at length. Finally, an
outline of reporting the findings in the mixed method approaches is sketched out. It is believed that
this article can be useful and beneficial to the researchers in general and postgraduate students in
particular who want to start or are involved in the process of conducting research.

Index Terms​―research methods, instruments, reliability,

validity

I. I​NTRODUCTION

There are various procedures of collecting data: tests, questionnaires, interviews, classroom
observations, diaries, journals, etc. Quite often, quantitative designs use tests and closed-ended questionnaires
in order to gather, analyze and interpret the data. However, the qualitative methods mostly make use of
interviews, diaries, journals, classroom observations and open-ended questionnaires to obtain, analyze and
interpret the data. On the other hand, mixed method approaches usually use closed-ended questionnaires
(numerical data), interviews and classroom observations (text data) to collect information. In order to triangulate
the data, the researchers can obtain information through different procedures to heighten the dependability and
trustworthiness of the dada and their interpretation. This article only investigates the virtues and weaknesses of
the questionnaires, interviews and classroom observations because of their importance and widespread use.
Generally, before and after collecting the data, the researchers need to consider the validity and reliability of
their data. Therefore, the various ways of magnifying the validity and reliability of the data will be described in
detail. In the end, the researchers need to put their study together and report it. The final section illustrates how
the inquirers can write and report their study.

II. R​ESEARCH ​I​NSTRUMENTS

Generally, there are various procedures of collecting data. The main instruments used in the mixed
method researches consist of closed-ended, open-ended questionnaires, interviews and classroom
observations. These different ways of gathering information can supplement each other and hence boost the
validity and dependability of the data. In the main, the quantitative data are obtained through closed-ended
questionnaires and the qualitative data through open- ended questionnaires, interviews and classroom
observations. The items of the questionnaires are mainly developed based on the research objectives and
research questions.

A. Questionnaires
Questionnaires are doubtless one of the primary sources of obtaining data in any research endeavor.
However, the critical point is that when designing a questionnaire, the researcher should ensure that it is “valid,
reliable and unambiguous” (Richards & Schmidt, 2002, p. 438). On the whole, questionnaires can appear in
three types:
1- closed-ended (or structured) questionnaires 2- open-ended (or unstructured) questionnaires 3- a mixture of
closed-ended and open-ended questionnaires. As a matter of fact, closed-ended questionnaires provide the
inquirer with quantitative or numerical data and open- ended questionnaires with qualitative or text information.
In this regard, Blaxter et al. (2006, p. 170) divide questionnaires into “seven basic question types: quantity or
information, category, list or multiple choice, scale, ranking, complex grid or table, and open-ended.” Generally,
a questionnaire might make use of one or several types of these question forms.
The fact is that each type of questionnaire has its own strengths and weaknesses. Seliger and Shohamy
(1989) are of the opinion that closed-ended questionnaires are more efficient because of their ease of analysis.
On the other hand,

© 2013 ACADEMY PUBLISHER

THEORY AND PRACTICE IN LANGUAGE STUDIES 255

Gillham (2000, p. 5) argues that “open questions can lead to a greater level of discovery.” He too admits the
difficulty of analyzing open-ended questionnaires. In this regard, Alderson and Scott (1996, p. 53) acknowledge
the usefulness of qualitative data but state that “their open-ended nature made it more difficult to compare
reports of discussions and interviews ...” The important issue in open-ended questions is that the responses to
these types of “questions will more accurately reflect what the respondent wants to say” (Nunan, 1999, p. 143).
Therefore, it is better that any questionnaire include both closed-ended and open-ended questions to
complement each other.
It is safe to say that appropriately developed questionnaires have many advantages. The following
statements are some of the advantages of the questionnaires (Seliger & Shohamy 1989; Robinson, 1991;
Lynch, 1996; Nunan, 1999; Gillham, 2000; Brown, 2001).
- They are one of the efficient means of collecting data on a large-scale basis. - They can be sent
simultaneously to a great number of people. - The inquirer can fairly easily gather data in field sites. -
Respondents‟ anonymity makes them to share information more easily. - When similar questions are
administered simultaneously to a large number of people the acquired data are more identical, correct and
standard.
- They are a time-efficient way of collecting data from many people. - Closed-ended questionnaires can easily
be analyzed in a straightforward way. - They are cost-efficient. On the other hand, questionnaires have some
disadvantages which should be kept in mind whenever and wherever they are used (Gillham, 2000; Brown,
2001).
- Sometimes the answers are inaccurate and questionable. - There is usually low return rate when sent by post
or email. - Ambiguity and unclearness of some questions might lead to inaccurate and unrelated responses. -
Some questions may cause misunderstanding. - Wording of the questions might affect the respondents‟
responses. In general, there are different ways of administering questionnaires which each one has its own
advantages and disadvantages. For instance, whenever the respondents are not within the reach of the
evaluator, they might be sent by post. However, the return rate of this procedure is rather low. They can also be
sent over internet. In this procedure the return rate is a little higher than by post because it is easier for the
respondents to return them. Moreover, they might be administered through the telephone. In this procedure the
respondent is rather obliged to answer the questions. Finally, they can be administered face-to-face. In this
procedure too the respondents are relatively compelled to answer the questions. So that the return rate is high
and any ambiguous question can be clarified on the spot by the researcher (Gillham, 2000).
On the other hand, Brown (2001, p. 6) divides administering questionnaires into two methods. The first
one is the self-administered questionnaire which is usually mailed out to the indented respondents. This
procedure has some major defects. That is, the respondents usually hardly return the questionnaire. Also, if any
misunderstandings arise, or if there be any vague questions, the researcher is not available to clarify them.
Finally, the researcher has no idea how the questions were answered. The second type is the group
administered questionnaire. In this procedure the questionnaire “is administered to the groups of individuals all
at one time and place ...” (ibid). It is believed that this method of administering the questionnaire is more
preferable than the self-administered one. That is, the return rate is high, the researcher is present to explain
any unclear questions and the researcher knows the conditions under which the questionnaires were filled out.

B. Interviews
The second main type of data to be collected in the mixed method design is the interview. Burns (1999, p.
118) contends that “Interviews are a popular and widely used means of collecting qualitative data.” To this end,
the researcher wants to get firsthand information directly from some knowledgeable informants. The inquirer
intends “to obtain a special kind of information” (Merriam, 1998, p. 71) and investigates for himself/herself what
is going on in the respondents‟ mind. The point is that the researcher cannot observe the informants‟ feelings
and thinking, so that interviewing is a key to understand what and how people perceive and “interpret the world
around them” (ibid: 72). In this juncture, Flick (2006, p. 160) adds that the purpose of interview “is to reveal
existing knowledge in a way that can be expressed in the form of answers and so become accessible to
interpretation.”
In general, interview can be conducted in two forms: person-to-person and group or collective formats.
Merriam (1998) believes that both of these forms of interview are a kind of goal oriented conversation. On the
whole, Johnson and Turner (2003, p. 308) list the strengths of the interview encounter as follows:
- Good for measuring attitudes and most other content of interest. - Allow probing by the
interview. - Can provide in-depth information. - Allow good interpretative validity. - Very
quick turnaround for telephone interviews. - Moderately high measurement validity for
well-constructed and well-tested interview protocols.

© 2013 ACADEMY PUBLISHER

256 THEORY AND PRACTICE IN LANGUAGE STUDIES

- Relatively high response rates often attainable. - Useful for exploration and confirmation. Weaknesses: -
In-person interviews expensive and time-consuming. - Perceived anonymity by respondents possibly low. -
Data analysis sometimes time-consuming for open-ended items. There are several different types of interviews.
The decision to choose one over another depends on the purpose of the research, the type of data,
phenomenon under study, etc. Merriam (1998, p. 72) asserts that our choice rests on “determining the amount
of structure desired.” Patton (1990, p. 288-9) divides interviews into four major types:
- Informal conversation interview - Interview guide approach - Structured open-ended interview - Closed, fixed
response interviews The informal conversation interview is usually conducted without any predetermined
questions and without any order. The questions emerge from the natural flow of conversation. This type of
interview is exploratory in nature and indeed difficult for the new researchers. At the other end of the continuum
is the structured open-ended interview. In this type of interview the questions are predetermined with almost
fixed order. However, this form of interview is too rigid and “adhering to predetermined questions may not allow
you to access participants‟ perspectives and understandings of the world” (Merriam, 1998, p. 74). Also, the
closed or fixed response interview is similar to a closed- ended questionnaire in which the respondent only
answers the interviewer‟s questions in a fixed format and order. In fact, the respondent is barely free to
express himself/herself. This type of interviewing is too mechanistic and bores the interviewee. However, at
midposition on the continuum is the interview guide approach. In this form of interview, the topics and questions
are specified but they can be reworded in any sequence based on the situation. One of the advantages of the
interview guide approach is that the collected information “can later be compared and contrasted” (Fraenkel &
Wallen, 2003, p. 456). In this approach data collection is rather systematic and conversational.
Furthermore, interviews can often be specified based on the amount of control that the inquirer exercises
over the conversation. In this juncture, Burns (1999, p. 119) presents three types of interviews: structured,
semistructured and unstructured.
It can be argued that Patton‟s (1990) and Burns‟ (1999) proposals are relatively similar. However,
Patton‟s list contains one more option. These two types of interviews can be illustrated as follows:
Patton Burns Informal conversational interview = unstructured Interview guide approach =
semistructured Structured open-ended interview = structured Closed, fixed response interview __ Clearly, the
most preferred type of interview is the semistructured interview guide approach. This type of interview is flexible
and allows the interviewee to provide more information than the other ones. This form of interview is neither too
rigid nor too open. It is a moderate form in which a great amount of data can be elicited from the interviewee.
The important issue in any interview encounter is the type of the questions that are asked. The interviewer
can prepare questions according to the focus of the study. In order to obtain more relevant data from the
interviewee, the researcher ought to ask good questions. To this end, Flick (2006, p. 160) proposes that “the
different types of questions allow the researchers to deal more explicitly with the presuppositions they bring to
the interview in relation to aspects of the interview.” Furthermore, in order to obtain more data, the interviewer
should use familiar language. Fraenkel and Wallen (2003) believe that technical jargons can be used in the
scientific circles and the interviewer should use ordinary everyday language during the interview process.
Merriam (1998: 79) cautions against using the following type of questions:
Multiple questions (e.g. How do you feel about the instructors and classes?) Leading questions (e.g. What
emotional problems have you had since leaving your job?) Yes-or-no questions (e.g. Has returning to school
been difficult?) Generally, multiple questions confuse the respondents. Merriam (ibid, 78-9) points out that
“Leading questions reveal a bias or an assumption that the researcher is making, which may not be held by the
participants. These set the respondent up to accept the researcher‟s point of view.” Finally, the yes-or-no
questions hardly provide any useful and relevant information.
The researcher should in the mean time try to find out some knowledgeable and appropriate informants.
In order to obtain more valid and reliable information, the interviewer should select the respondents more
carefully. There are different ways of identifying suitable informants. First of all, the researcher can find a
knowledgeable person based on his/her experience and on-site observations. The second way is asking other
participants to introduce a reliable informant. Finally, the person being introduced can also be asked to
introduce other key informants in the field.
In any interview endeavor the interviewer should explain to the respondents the purpose of the interview.
Flick (2006, p. 169) states that you should attempt “to create a good atmosphere in the interview and to give
room to your

© 2013 ACADEMY PUBLISHER

THEORY AND PRACTICE IN LANGUAGE STUDIES 257

interviewees to open up.” The interviewer should try to consider the following points during the interview
process (Merriam, 1998; Fraenkel & Wallen, 2003; Johnson & Turner, 2003; Flick, 2006):
- The interviewees should be provided with scope to express their opinions. - The interviewer should be
nonjudgmental and neutral during the interview. - The interviewer should be respectful, natural and
nonthreatening. - The interviewer should create rapport. - The interviewer should not interrupt. Generally, the
interview can be recorded in three ways (Merriam, 1998). The first method is to tape record the interview. The
second way is to take notes. In this method the interviewer manages to write the important points during the
interview process. The final method is to attempt to write down every detail of the interview immediately after
the interview. This method of recording data is not recommended because it is difficult and rather an incorrect
and inaccurate way of collecting information.

C. Classroom Observations
Observation is a preplanned research tool which is carried out purposefully to serve research questions
and objectives. When using this method, the researcher observes the “classroom interactions and events, as
they actually occur” (Burns, 1999, p. 80). Flick (2006, p. 219) also contends that observation “is an attempt to
observe events as they naturally occur.” More importantly, observation enables the researcher to combine it with
questionnaires and interviews to collect “relatively objective firsthand information” (Johnson & Turner, 2003, p.
314). To this end, Merriam (1998, p. 96) believes that observation is a kind of data triangulation in order to
“substantiate the findings.” In this regard, Fraenkel and Wallen (2003, p. 453) state that the observers “study the
subjective factors objectively.” However, Nation (1997, p. 276) asserts that the researchers try to study the
“representations of behavior rather than the behavior itself.”
Certainly, like every other research instrument, observation has its own strengths and weaknesses.
Observational data represent a firsthand picture of the events, is carried out in a natural field setting and enable
the researcher to obtain contextual factors. However, analyzing observational data is time-consuming,
observing large population is difficult, there is the possibility of observer bias, reactivity and investigator effects
on the students and instructors, etc. On the whole, in order to carry out observation, the observers need to
determine the setting. Then, they should identify what to be documented. Next, the inquirers ought to gain “an
initial, general presentation of the field” (Flick, 2006, p. 217). After that, the researchers might attempt to
conduct focused and selected observations which are pertinent to the research objectives and questions.
In the main, observation can take place through two methods: nonparticipant and participant. In
nonparticipant observation, the observer only watches and records the classroom activities without any
involvement. Burns (1999, p. 82) expresses that the inquirer‟s goal “is to remain aloof and distant and to have
little or no contact with the subjects of the research.” Also, Fraenkel and Wallen (2003, p. 451) confirm that
“researchers do not participate in the activity being observed but rather sit on the sidelines and watch.”
However, in participant observation the observers enter the classroom and integrate with the students directly.
Burns (1999, p. 82) is of the opinion that “The researcher becomes a member of the context and participates in
its culture and activities.” In this regard, Flick (2006, p. 220) emphasizes that the observers “dive headlong into
the field.” Nevertheless, participant observation has attracted some criticism in the field. For instance, Merriam
(1998) notes that the researcher loses sight of the students and their activities. The observer becomes too
much involved in the classroom processes which consequently cannot concentrate on selected behaviors and
activities.
In this process the crucial issue is what to be observed. It is clear that several classes cannot be observed
at a short period of time. Therefore, the researcher must be selective and choose randomly in order to maintain
the representativeness criterion. Inside the classroom the observer must pay particular attention to details in
order to record them objectively without personal bias. Fraenkel and Wallen (2003, p. 452) propose two types of
observations: “narrow focus and broad focus observations.” Narrow focus observation usually concentrates on
a single element and broad focus on an overall picture of what is happening in the classroom. In this regard,
there are several factors which determine and influence what should be observed. These factors involve the
observer‟s goal, the research objectives, research questions, the inquirer‟s field of study, practical issues,
researcher‟s personal impressions, the topic under study and so on (Merriam, 1998). Therefore, according to
these considerations the researcher can select and observe the classroom setting, the students‟ and
instructors‟ activities and behaviors, etc. In the main, the observer can either enter the class and observe what
is happening or prepare a checklist of activities and tick accordingly. To implement the latter option, Ross
(1996, p. 173) sets forth four categories of classroom activities: “student activities, sources of input to students,
student behavior and the distribution of classroom time.”
The next essential point in the observational process is the amount of time spent on observation. Almost
often the duration of observation depends on the research objectives and questions. To this end, Merriam
(1998, p. 98) maintains that “There is no ideal amount of time to spend observing.” In this regard, Fraenkel and
Wallen (2003, p. 452) distinguish between “single observation of limited duration and multiple observations.”
Mainly, it is clear that the more time spent observing, the better and reliable information can be obtained. Flick
(2006, p. 219) emphasizes that “How far this aim can be fulfilled remains doubtful.”

© 2013 ACADEMY PUBLISHER

258 THEORY AND PRACTICE IN LANGUAGE STUDIES

One of the main drawbacks of the observation is the reactivity problem. That is, the students react
differently because of the researcher‟s presence in the classroom. To overcome this problem, Johnson and
Turner (2003, p. 312) suggest that reactivity may “decrease significantly after the researcher has been
observing for a while.” The point is that at the first session the observers had better not record or take notes.
That is, they should only sit and observe and let “the students become accustomed to their presence and go
about their usual activities” (Fraenkel & Wallen, 2003, p. 453). Therefore, it is preferable that the observers
begin to focus on the classroom, students‟ and instructors‟ activities and behaviors from the second and third
sessions onwards.
One more problem in the observation process is the observer bias. That is, the researchers‟ viewpoints
and backgrounds may considerably affect what they see. The observers are human beings and possess some
peculiarities which have a tremendous effect on how they observe. Therefore, it is imperative that the observers
try hard to remain nonjudgmental and “control their biases” (Fraenkel & Wallen, 2003, p. 453).
The final issue in the observation process is the recording of information: “This written account of the
observation constitutes field notes” (Merriam, 1998, p. 104). Therefore, the observers should attempt to write
down their observations either during or soon after the end of the class, i.e. the sooner the better. The
researchers had better “take extensive field notes during and after the observation sessions” (Johnson & Turner
2003: 313). The inquirers should prepare an appropriate form and put the information in it. In this regard, Nation
(1997: 279) adheres to “designing a suitable data sheet.”
The data collection should be continued till the saturation point is obtained. The following checklist is an
example of how also used to collect information through the classroom observation process. One of the
advantages of this checklist is that it is fairly detailed and the observer can easily tick the appropriate boxes
according to his/her observations. This method of collecting information is used to complement field notes.

III. V​ALIDITY

The principles underlying naturalistic and/or qualitative research are based on the fact that validity is a
matter of trustworthiness, utility and dependability that the evaluator and the different stakeholders place into it.
As Merriam (1998, p. 202) states in qualitative research “reality is holistic, multidimensional and ever-changing.”
Therefore, it is up to the researcher and research participants who attempt to build validity into the different
phases of the research from data collection through to data analysis and interpretation. In the main, validity is
concerned with whether our research is believable and true and whether it is evaluating what it is supposed or
purports to evaluate. In this regard, Burns (1999, p. 160) stresses that “validity is an essential criterion for
evaluating the quality and acceptability of research.” Generally, researchers use different instruments to collect
data. Therefore, the quality of these instruments is very critical because “the conclusions researchers draw are
based on the information they obtain using these instruments” (Fraenkel & Wallen, 2003, p. 158). Thus, it is
imperative that the data and the instruments to be validated. On the whole, the following miscellaneous
procedures can be used to validate the instruments and the data.

A. Content Validity
Content validity is related to a type of validity in which different elements, skills and behaviors are
adequately and effectively measured. To this end, the research instruments and the data might be reviewed by
the experts in the field of research. Based on the reviewers‟ comments the unclear and obscure questions can
be revised and the complex items reworded. Also, the ineffective and nonfunctioning questions can be
discarded altogether. In addition, the questions could be face validated by these persons.

B. Internal Validity
Mainly, internal validity is concerned with the congruence of the research findings with the reality. Also, it
deals with the degree to which the researcher observes and measures what is supposed to be measured. On
the whole, to boost the internal validity of the research data and instruments, the researcher might apply the
following six methods recommended by Merriam (1998): triangulation, member checks, long-term observation
at research site, peer examination, participatory or collaborative modes of research and researcher‟s bias.
1- Triangulation. In order to strengthen the validity of evaluation data and findings, the investigator should
try to collect data through several sources: questionnaires, interviews and classroom observations. Gathering
data through one technique can be questionable, biased and weak. However, collecting information from a
variety of sources and with a variety of techniques can confirm findings. Therefore, if we obtain the same
results, we can become sure that the data are valid. Certainly, through triangulation we can gain qualitative and
quantitative data in order to corroborate our findings.
2- Member checks. Through member checks the results and interpretations are taken back to the
participants in order to be confirmed and validated. Therefore, the results and interpretations of interviews might
be handed over to the interviewees in order to confirm the content of what they have stated during the interview
encounter. In this way the plausibility and truthfulness of the information can be recognized and supported.

© 2013 ACADEMY PUBLISHER

THEORY AND PRACTICE IN LANGUAGE STUDIES 259

3- Long-term observation. Repeated observations over an extended period of time can naturally enhance
the validity of research data and findings. The researcher should try to visit different classes in order to obtain
the intended information. The observation continues as long as the saturation point is achieved.
4- Peer examination. In peer examination process the research data and findings are reviewed and
commented on by several nonparticipants in the field. However, these peers need to be familiar with the subject
under study and possess enough background information in it. Therefore, the researcher might ask two or three
experienced experts who have taught the same course for some years to review and comment on the interview,
classroom observation and questionnaire data and findings. It is certain that the plausibility of data analysis and
interpretations by these peers can tremendously augment the validity of the research.
5- Participatory or collaborative modes of research. This means that the researcher should try to involve
most of the participants in all phases of inquiry. The purpose “is to arrive at evaluation conclusions as a result of
a consensus among persons from different perspectives in relation to the program” (Lynch, 1996, p. 62).
Clearly, it is very difficult for the researcher to conduct a study single-handedly. But sharing ideas with different
students, learners, teachers and instructors can strengthen the research findings and interpretations. The
researcher should, therefore, try to involve the students, ex-students, language instructors, subject instructors
and program staff in the different phases of the study in order to enhance the validity of the research. Their
varied ideas and views can be constructive and useful.
6- Researcher‟s bias. It is clear that every researcher has his/her own particular values, beliefs and
worldviews. The point is that the investigator should try to collect, analyze and interpret data as impartially as
possible. The inquirer needs to be explicit, critical and faithful at different phases of the inquiry process.
Therefore, the researcher should try to remain as nonjudgmental and clear as possible throughout the research
process. He/She should try to stick to the ethical rules and principles, perform the evaluation as accurately as
possible and report the findings honestly.

C. Utility Criterion
In addition to the above-mentioned six criteria of checking and contributing to the validity process, we can
add the utility criterion. Lynch (1996, p. 63) asserts that “Utility refers to the degree of usefulness the evaluation
findings have for administrators, managers and other stakeholders.” This criterion intends to inquire whether or
not the research works. That is, utility criterion asks whether the evaluation endeavor generates enough
information for the decision-makers with regard to the effectiveness and appropriateness of the program.
Clearly, when the evaluation process provides the different stakeholders with proper and ample information, it
can be surmised that the utility criterion has been met and consequently achieved validity requirement.
D. External Validity
One more issue to be considered is the external validity. External validity is concerned with the
applicability of the findings in other settings or with other subjects. As Burns (1999, p. 160) notes “How
generalisable to the other contexts or subjects is our research.” In fact, it might depend on the underlying
similarities between our context and other contexts. Nunan (1999, p. 17) puts emphasis on the research design
and states that “Is the research design such that we can generalize beyond the subjects under investigation to
a wider population?”

IV. R​ELIABILITY

One of the main requirements of any research process is the reliability of the data and findings. In the
main, reliability deals with the consistency, dependability and replicability of “the results obtained from a piece of
research” (Nunan, 1999, p. 14). Obtaining the similar results in quantitative research is rather straightforward
because our data are in numerical form. However, in qualitative approaches to research achieving the identical
results are fairly demanding and difficult. It is because the data are in narrative form and subjective. To this end,
Lincoln and Guba (1985, p. 288) point out that instead of obtaining the same results, it is better to think about
the dependability and consistency of the data. In this case, the purpose is not to attain the same results rather
to agree that based on the data collection processes the findings and results are consistent and dependable.
Merriam (1998, p. 206) believes that “the human instrument can become more reliable through training and
practice.” In general, Lincoln and Guba (1985) and Merriam (1998) suggest that the dependability of the results
can be ensured through the use of three techniques: the investigator‟s position, triangulation and audit trial.
1- The investigator‟s position. In order to increase the reliability of the research, the investigator needs to
explain explicitly the different processes and phases of the inquiry. Therefore, the researcher should elaborate
on every aspect of the study. He/She should describe in detail the rationale of the study, design of the study
and the subjects.
2- Triangulation. The researcher should use different procedures such as questionnaires, interviews and
classroom observations to collect data. Also, this information needs to be obtained through different sources
such as learners, students, ex- students, language instructors, subject instructors and program staff. Therefore,
collecting varied types of information through different sources can enhance the reliability of the data and the
results. In this way the replication of the study can be carried out fairly easily.

© 2013 ACADEMY PUBLISHER

260 THEORY AND PRACTICE IN LANGUAGE STUDIES

3- Audit trial. In order to fulfill this procedure, the researcher should describe in detail how the data are
collected, how they are analyzed, how different themes are derived and how the results are obtained.
Therefore, this detailed information can help replicate the research and contribute to its reliability.

A. External Reliability
On the whole, external reliability is concerned with the replication of the study. As Burns (1999, p. 21-20)
states “Could an independent researcher reproduce the study and obtain results similar to the original study?” It
is believed that the external reliability of the research can be increased if the inquirer pays heed to five
important aspects of the inquiry (LeCompte & Goetz, 1982; Nunan, 1999). These five aspects include the status
of the researcher, the choice of informants, the social situations and conditions, the analytic constructs and
premises and the methods of data collection and analysis.
1- The status of the researcher. This aspects requires that the researcher‟s social position with regard to
the participants of the study to be clarified.
2- The choice of informants. This aspect demands that the investigator describe the participants as fully
as possible. Therefore, if any independent inquirer desired to replicate the study, it could be done fairly easily.
3- The social situations and conditions. The study might be conducted in an academic situation, therefore,
in that case the social situation and condition is fairly constant and uniform for most of the learners. That is, all
the learners have equal opportunity to study and learn English language. However, it is clear that there are
grave differences among the learners from the proficiency, social and economic point of view.
4- The analytic constructs and premises. The main terms, constructs, definitions, units of analysis and
premises should be delineated and their underlying assumptions elaborated on explicitly. Therefore, the
identification and description of constructs and premises can ease the process of replication and consequently
enhance reliability.
5- Methods of data collection and analysis. The different procedures of collecting data need to be explicitly
explained. The main methods of gathering information in mixed method research are questionnaires, interviews
and classroom observations. Mainly, the quantitative data are analyzed through descriptive statistics and
qualitative data by means of descriptive and thematic interpretations.

B. Internal Reliability
Internal reliability deals with the consistency of collecting, analyzing and interpreting the data. Internal
reliability might be obtained when an independent researcher on reanalyzing the information comes to the
similar findings as the original researcher. Burns (1999, p. 21) asserts that “Would the same results be obtained
by other researchers using the same analysis?” In the present study, in order to guard against threats to
internal reliability, the researcher has used the four basic strategies suggested by LeCompte and Goetz (1982)
and elaborated on by Nunan (1999). These are: the use of low inference descriptors, multiple

researchers/participant researchers, peer examination and mechanically recorded data. ​1- Low inference

descriptors. Low inference descriptors are “easily observable and can be readily quantified (i.e. counted
​ or
measured)” (Richards & Schmidt, 2002, p. 239). For example, „asking factual questions‟ can be counted and
measured. On the other hand, high inference descriptors are categories “of behavior which cannot be observed
directly but which has to be inferred” (ibid). For example, it is difficult to observe and measure the students‟
motivation and interest. These elaborate descriptions and explanations enhance the internal reliability and any
independent observer can observe and replicate them rather easily.
2- Multiple researchers/participant researchers. As Nunan (1999, p. 60) acknowledges “In much research
this is not feasible, because a research team consisting of several members can be extremely expensive.”
However, he suggests that the researcher can ask the experienced participants to help him/her verify and
confirm in the data collection, analysis and interpretations. Therefore, the researcher can involve two or more
participants in the analysis, interpretation and validating conclusions.
3- Peer examination. Based on Lecompte and Goetz (1982) this method can be implemented through
applying and utilizing other researchers‟ findings. The investigator can use other researchers‟ results and
conclusions in his/her report. Thus, the researcher can utilize some relevant studies in his/her research in order
to enhance the internal reliability.
 4- Mechanically recorded data. The interviews can be recorded and preserved, therefore, the reanalysis
or the replication of the data can be rather easily implemented by any independent investigator. This procedure

can increase the internal reliability of the data and findings. ​V. R​EPORTING THE ​S​TUDY

Generally, the outcomes of the whole research process should be reported to the relevant audience.
Lynch (1996, p. 174) states that “The final product of an evaluation is a report, which can take many different
forms, depending on the audiences and goals ...” In this regard, the researcher describes the procedures that
he/she performed and explains in detail the results or outcomes of the study. As Creswell (1994, p. 169)
emphasizes the important issue is that “how this outcome compares and contrasts with theories and the
literature.” That is, the results ought to be collated and examined in comparison with other related studies,
theories and so on. The researcher should try to expound the study, the results

© 2013 ACADEMY PUBLISHER

THEORY AND PRACTICE IN LANGUAGE STUDIES 261

and his/her interpretations as clearly as possible to the relevant audience. In this way the readers can make
sense of the whole study “and what the study means to them and to the language teaching profession as a

whole” (Brown, 2001, p. 12). ​Mainly, the researcher must try not to hurt the different stakeholders. That is,

sometimes the results of the study may be ​ provocative and create some controversial issues. However, Lynch
(1996, p. 9) believes that “The critical issue is how to communicate the findings of the evaluation honestly and
successfully.” It is safe to say that the researcher should consider the social and political atmosphere of the
context and situation and act accordingly. This does not mean that he/she might make some compromises.
Rather, the researcher should be considerate and take into account every aspect before reporting the
outcomes. Creswell (1994, p. 169) argues that the inquirer should impart his experiences so that “allow readers
to vicariously experience the challenges he encounters and provide a lens through which readers can view the
subject‟s world.” On the whole, the researcher should report the results based on the pre-agreed goals and
objectives of the study.
Generally, there are different formats for reporting the results. The inquirer should, therefore, report the
findings according to the specific design in which the data were gathered and analyzed. Brown (2001, p. 253)
divides the research design into four parts: “purely statistical, statistical with some qualitative, qualitative with
some statistics, and purely qualitative.” The quantitative research report is usually presented in numerical forms
along with the relevant tables, diagrams and figures. However, the qualitative research report appears in
narrative form and its organization is fairly flexible. As Creswell (1994, p. 168-9) puts it “the results will be
presented in descriptive, narrative form rather than as a scientific report.” The important issue in this process is
to represent the outcomes as completely and clearly as possible. In qualitative research the researcher makes
every effort to recount the process of research. That is, the process is more important than the sheer product.
As Brown (2001, p. 257) asserts this account of the “story may differ in structure from project to project and
report to report.”
On the other hand, a study might be a mixture of qualitative and quantitative researches. To this end, the
organization of the report might be in a hybrid form. Therefore, if the design is a mixed approach a combination
of descriptive and statistical report forms might be rendered. However, Brown (2001, p. 259) cautions that “you
may need to decide whether it is primarily a statistical study or mainly qualitative in nature. Therefore, the
researcher should know which of the approaches is the predominant one and accordingly prepare a report on
those bases. By and large, if the research approach is a mixed design, the following outline might be a
workable option to report the evaluation process and outcomes:
I. Introduction
A. Background/Literature Review B.
Statement of Purpose C. Research
Questions II. Methods
A. Participants 1. Sampling 2.
Characteristics of the Participants B.
Materials C. Procedures III. Results IV.
Discussion V. Conclusions

VI. C​ONCLUSION

This article discussed and elaborated on the strengths and weaknesses of three different procedures of
data collection: questionnaires, interviews and classroom observations. As was explained in the introduction
section of this paper, there are other ways of gathering information. However, these three (questionnaires,
interviews and classroom observations) instruments were selected because of their extensive utilization and
their application in mixed method approaches to research. By using these instruments, the researchers can
obtain both quantitative and qualitative data. In addition, it was argued that any researcher needs to enhance
the validity and reliability of the data. Therefore, these two ways of extending the steadiness and stability of the
data were delineated in detail. Finally, an overall outline of writing the research paper and/or report was
presented and discussed.

R​EFERENCES

[1] Alderson, J. C. & Scott, M. (1996). Insiders, outsiders and participatory evaluation. In J.C. Alderson & A. Beretta (Eds.),
Evaluating second language education ​(pp. 25-60). Cambridge: CUP. [2] Blaxter, L., Hughes, C. &
Tight, M. (2006). How to research. Berkshire: Open University Press. [3] Brown, J. D. (2001). Using
surveys in language programs. Cambridge: CUP. [4] Burns, A. (1999). Collaborative action research for
English language teachers. Cambridge: CUP. [5] Creswell, J. W. (1994). Research design: Qualitative and
quantitative approaches. Thousand Oaks, CA: Sage.

© 2013 ACADEMY PUBLISHER

262 THEORY AND PRACTICE IN LANGUAGE STUDIES

[6] Flick, U. (2006). An introduction to qualitative research. London: Sage. [7] Fraenkel, J. R. & Wallen, N. E. (2003). How to
design and evaluate research in education. Fifth ed. New York: McGraw-Hill. [8] Gillham, B. (2000). Developing a
questionnaire. London: Continuum. [9] Johnson, B. & Turner, L. A. (2003). Data collection strategies in mixed methods
research. In A. Tashakkori & C. Teddie (Eds.).
Handbook of mixed methods in social and behavioral research ​(pp. 297-319). Thousand Oaks, CA: Sage. [10]
LeCompte, M. & Goetz, J. (1982). Problems of reliability and validity in ethnographic research. ​Review of Educational
Research, 5​ 2(1), 31-60. [11] Lincoln, Y. S. & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, Calif.: Sage.
[12] Lynch, B. K. (1996). Language program evaluation: Theory and practice. Cambridge: CUP. [13] Merriam, S. B. (1998).
Qualitative research and case study applications in education. San Francisco: Jossey-Bass. [14] Nation, J. R. (1997).
Research methods. New Jersey: Prentice Hall. [15] Nunan, D. (1999). Research methods in language learning. Eighth
printing. Cambridge: CUP. [16] Patton, M. Q. (1990). Qualitative evaluation and research methods. Second ed. Newbury
Park, CA: Sage. [17] Richards, J. C. & Schmidt, R. (2002). Longman dictionary of language teaching and applied linguistics.
Third ed. London:
Longman. [18] Robinson, P. C. (1991). ESP today: A practitioner‟s guide. New York: Prentice Hall. [19] Ross, S.
(1996). Program-defining evaluation in a decade of eclecticism. In J. C. Alderson & A. Beretta (Eds.). ​Evaluating
second language education ​(pp. 167-195). Cambridge: CUP. [20] Seliger, H. W.
& Shohamy, E. (1989). Second language research methods. Oxford: OUP.

Mohammad Zohrabi ​is an assistant professor. He has got his PhD from UKM, Malaysia. He has taught various
courses both at undergraduate and graduate level at the University of Tabriz, Iran. He has published various articles in
international journals and produced 4 books: ​A Dictionary of Research Terms in Applied Linguistics,​ ​A Dictionary of
Language and Linguistics​, ​Reading English in Action​, and ​Active Reading Comprehension.​ His research interests include:
program evaluation, material writing and evaluation, first and second language acquisition, teaching reading and writing
skills, English for academic purposes, English for general purposes, and English for specific purposes.

© 2013 ACADEMY PUBLISHER