Download as PDF, PPTX
EM-03-Internal and External Validity - Experimental Methods - Purdue 2019.pdf
- 1. EM 03 –Internal & External Validity Page i Experimental Methods: Design & Analysis 03 – Internal & External Validity Trenton D. Mize Department of Sociology & Advanced Methodologies (AMAP) Purdue – Fall 2019
- 2. EM 03 –Internal & External Validity Page ii Contents INTERNAL/EXTERNAL VALIDITY.......................................................................................................................................... 47 Goals of an experiment according to Zelditch ................................................................................................. 47 Scope Conditions........................................................................................................................................................... 48 Internal, External, and Construct validity.......................................................................................................... 49 External validity and the experimental method..............................................................................................50 Construct validity ...........................................................................................................................................................51 Construct validity and the experimental method ...........................................................................................52
- 3. EM 03 –Internal & External Validity Page 47 Internal/External Validity “The purpose of the laboratory experiment is to create certain theoretically relevant aspects of social situations under controlled conditions.” (Zelditch 1969) “Disconfirming predictions, when they have been derived from an explicit theory, are valuable because they can show which assumptions or conditions in the theory need improvement.” (Webster and Sell 2014) Goals of an experiment according to Zelditch 1. To create states of affairs difficult to discover in natural settings 2. To produce controls and contrasts that are difficult to find… in natural settings 3. To replicate events that seldom recur under the same conditions in natural settings 4. To isolate a process from the effects of other processes that confused our understanding of it
- 4. EM 03 –Internal & External Validity Page 48 Scope Conditions “Scope conditions tell the kinds of situations in which a theory claims it can describe what happens. If a situation meets its scope conditions, the theory should be able to predict accurately.” (Webster and Sell 2014) 1. Example: status characteristics theory a. Task group b. Collectively oriented i. Common goal c. Task oriented d. Observable distinctions between individuals 2. Would this theory possibly apply outside of its intended scope? To actors other than individuals (e.g. organizations?)
- 5. EM 03 –Internal & External Validity Page 49 Internal, External, and Construct validity 1. Traditionally, experiments are billed as having excellent internal validity a. How certain you can be that a study tests the relationships it aims to test i. Construct validity is necessary (see following slides) ii. Causality is fundamental; how certain we are that the IV causes the observed change in the DV 2. Traditionally, experiments are billed as having low external validity a. Most often thought of in terms of generalizability: how certain you can be that the relationships you observe in a given sample would be the same in a different setting, sample, population, time, etc. i. Lucas (2003) critiques the common assumption that we should only consider generalizing to the larger population ii. Key to generalizing across time, place, and context is theory 1. Therefore, we should attempt to generalize to theory 2. Internal validity is fundamentally important to knowing whether your theory is supported or not
- 6. EM 03 –Internal & External Validity Page 50 External validity and the experimental method 1. Traditionally, experiments are often seen as worth the tradeoff of questionable external validity for the unrivaled gains in internal validity a. Note this is mostly said with lab experiments in mind b. Survey experiments and audit studies challenge this conventional wisdom even when discussing external validity in the more commonly used sense (i.e. generalizing to a population or conducting studies in the “real world”) 2. Focusing on internal validity as the degree to which your study provides theoretical support/lack of support challenges the common wisdom that experiments have low external validity a. Experiments (when designed well) can be highly externally valid tests of theory 3. Findings from different researchers and different methods are connected through theory a. How else do we understand what ethnographies, interviews, survey studies, experiments, and other methods tell us about a common topic?
- 7. EM 03 –Internal & External Validity Page 51 Construct validity 1. Construct validity is how well a given set of measures represent the underlying concepts they are meant to quantify a. E.g. Does your “competence scale” actually measure competence? Does it instead measure intelligence? Are these different? 2. In an experiment you create the independent variable and manipulate it. You also design the dependent measures a. Therefore, the researcher determines how well the IV and DV represent the concepts of interest; you are not limited to what a survey institute decided to collect / how they ask the questions, etc. b. This is a huge plus for experiments, but also makes your job hard i. An under-appreciated aspect of being a good experimentalist is being a good survey methodologist
- 8. EM 03 –Internal & External Validity Page 52 Construct validity and the experimental method “When testing theories, all measures are indirect indicators of theoretical constructs, and no methodological procedures taken alone can produce external validity.” (Lucas 2003) 1. Ever been frustrated by the measures available in secondary data? a. E.g. Does the General Social Survey (GSS) ask questions exactly how you would want them to? In a manner ideal for your research question? 2. If a given study is not internally valid—because: a. The measures are not accurate indicators of the IVs and/or DVs of interest (i.e. no construct validity) b. And/or because the IV is not really a cause of the DV (i.e. not a causal effect) i. In the case that a study is not internally valid—is the question of external validity important? 1. i.e. Without internal validity should we even care about assessing external validity? Should we care about the sample characteristics? Etc.?