Downloaded 173 times
![Statistical Significance
A result is statistically significant if it is unlikely to occur
when the null hypothesis (H0) is true
(Sufficient to reject the null hypothesis [H0])
• An effect is significant if we decide to reject the null
hypothesis after conducting a hypothesis test
• Z = 0.87, p > 0.05
does not fall in the critical region, fail to reject the null
• Z = -2.50, p < 0.05
falls in the critical region, reject the null](https://image.slidesharecdn.com/chapter8hypothesistesting-170706210310/75/Hypothesis-testing-15-2048.jpg)
More Related Content
Hypothesis testing
- 1.
- 2. What is aHypothesis Test? • A hypothesis test is a statistical method that uses sample data to evaluate a hypothesis about a population • A standardized methodology we use to answer a research question of interest • The most commonly utilized inferential procedure • Incorporates the concepts of z-scores, probability, and the distribution of sample means • The general goal of the hypothesis test: • To rule out chance (sampling error) as a plausible explanation for the results from a research study
- 3. The Logic ofHypothesis Testing 1. We have a hypothesis about a population • Typically about a population parameter: μ = 50 2. Based on our hypothesis, we predict the characteristics our sample should have • M should be around 50 3. Obtain a random sample from the population • Sample n = 20 from the population and compute their M 4. Compare the sample statistic with our hypothesis about the population • If the values are consistent, the hypothesis is reasonable • If there is a large discrepancy, we decide the hypothesis is unreasonable
- 4. The Unknown Population •Typically, research involves an unknown population • After we administer “tutoring treatment” to our sample of students, what is the new μ? • Remember, σ should remain the same • The focus of our hypothesis is the unknown population
- 5. The Research StudySample • Theoretically: • We would treat the entire population, randomly sample from the treated population, and evaluate the M • Realistically: • We sample from the original population, treat the sample only, and evaluate the M
- 6. The Purpose ofthe Hypothesis Test 1. There does not appear to be a treatment effect The difference between the sample and the population can be explained by sampling error 2. There does appear to be a treatment effect The difference between the sample and the population is too large to be explained by sampling error To decide between two explanations:
- 7. The Hypothesis Test:Step 1 State the hypothesis about the unknown population • The null hypothesis (H0) • States that there is no change in the general population before and after an intervention. • In the context of an experiment, H0 predicts that the independent variable (IV) will have no effect on the dependent variable • The scientific/alternative hypothesis (H1) • States that there is a change in the general population following an intervention • In the context of an experiment, H1 predicts that the independent variable (IV) will have an effect on the dependent variable
- 8. SAT Example • Populationparameters of a normally distributed variable are μ = 500 and σ = 50 • I believe my prep course will have some effect on this mean value • My hypotheses: • Null (H0): Even with treatment, the mean SAT score will remain at 500 • Alternative (H1): With treatment, the mean SAT score will be different from* 500 *Note: Here we do not specify a direction (increase/decrease); just some difference 500: 500: 1 0 SAT SAT H H
- 9. The Hypothesis Test:Step 2 Set the criteria for a decision • Determine before the experiment what sample means are inconsistent with the null hypothesis • The sampling distribution of the mean for the population is divided into two parts: 1. Sample means likely to be obtained if H0 is true (sample means close to the null value) 2. Sample means that are very unlikely to be obtained if H0 is true (sample means that are very different from the null value)
- 10. The Alpha (α) Level a.k.a.the Level of Significance The α level establishes the criterion, or “cut-off”, for making a decision about the null hypothesis. The alpha level also determines the risk of a Type I error (TBD in next section) • How we define which sample means have “high” or “low” probabilities • Typically set at: • α = 0.05, • α = 0.01 • α = 0.001 • Separates the typical values from the extreme, unlikely-to- occur values (in the critical region)
- 11. The Hypothesis Test:Step 3 Collect data and compute sample statistics • After we state they hypotheses and establish our critical regions: • Randomly sample from the population • Give the sample the treatment/intervention • Summarize the sample with the appropriate statistic (e.g.: the M) • Compute the test statistic M M z The z-score (the test statistic) forms a ratio comparing the obtained difference between the sample mean and the hypothesized population mean versus the amount of difference we would expect without any treatment effect (the standard error)
- 12. The Hypothesis Test:Step 4 Make a decision •Use the z-score obtained for the sample statistic and make a decision about the null hypothesis (H0) based on the critical region. Either: • Reject the null hypothesis • The sample data fall in the critical region • This event is unlikely if the null hypothesis is true, so we conclude to reject the null • This does NOT mean that we have proven the alternative • Fail to reject the null hypothesis (retain the null) • The sample data do not fall in the critical region • We do not have enough evidence to show that the null hypothesis is incorrect (NOT that the null hypothesis is “true”)
- 13. Making your decision: One mo’time! • Use the z-score obtained for the sample statistic and make a decision about the null hypothesis (H0) based on the critical region IF: My sample mean has a corresponding z = 0.87 • My treatment effect is not convincing. I do not have enough evidence and fail to reject the null. IF: My sample mean has a corresponding z = -2.50 • This is an unlikely event if the null is true, so I reject the null. I have convincing evidence my treatment works and has an effect. Fail to reject H0 Reject H0 Reject H0
- 14. An Analogy forHypothesis Testing Research Study 1. H0: There is no treatment effect 2. Researchers gather evidence to show that the treatment actually does have an effect 3. If there is enough evidence, the researchers reject H0 and conclude that there is a treatment effect 4. If there is not enough evidence, the researcher fails to reject H0 Jury Trial 1. H0: Defendant did not commit a crime 2. Police gathers evidence to show that the defendant really did commit a crime 3. If there is enough evidence, the jury rejects H0 and concludes that the defendant is guilty of the crime 4. If there is not enough evidence, the jury fails to find the defendant guilty Both are trying to refute the null hypothesis (H0) They are not concluding that there is no treatment effect; simply that there is not enough evidence to conclude there is an effect They are not concluding that the defendant is innocent; simply that there is not enough evidence for a guilty verdict
- 15. Statistical Significance A resultis statistically significant if it is unlikely to occur when the null hypothesis (H0) is true (Sufficient to reject the null hypothesis [H0]) • An effect is significant if we decide to reject the null hypothesis after conducting a hypothesis test • Z = 0.87, p > 0.05 does not fall in the critical region, fail to reject the null • Z = -2.50, p < 0.05 falls in the critical region, reject the null
- 16. The z-Score asa Test Statistic • A Test Statistic = A single, specific statistic (converted from sample data) that is used to test the hypothesis • The z-score formula as a recipe • Make a hypothesis about the value of µ. This is H0. • Plug the hypothesized value in the formula along with the other values • If the results is a z-score near zero (where z-scores are supposed to be), fail to reject H0. M M z
- 17. The z-Score Formulaas a Ratio If z is large, the obtained difference is larger than expected by chance • For example: z = 3 indicates the obtained difference is three times larger than what would be expected by chance M M z Sample mean – hypothesized population mean Standard error between M and µ Obtained difference Difference due to chance OR 3 1 3
- 18. Uncertainty and Errorsin Hypothesis Testing
- 19. Type I Errors(α) • A Type I error occurs when the sample data appear to show a treatment effect when, in fact, there is none • We reject a null hypothesis (H0) that is actually true • We falsely conclude that the treatment has an effect • Caused by unusual, unrepresentative samples that fall into the critical region despite a null effect of treatment • Type I errors are very unlikely
- 20. Type II Errors(β) • A Type II error occurs when the sample does not appear to have been affected by the treatment when, in fact, it has (the treatment does have an effect) • We fail to reject the null hypothesis (H0) that is actually false • We falsely conclude that the treatment had no effect • Commonly the result of a very small treatment effect • Although the treatment does have an effect, it is not large enough to show up in our data
- 21. The Relationship Betweenα & Errors • The alpha (α) level is the probability that the test will lead to a Type I error • If α = .05: The probability that the test will lead to a Type I error is 5% • If α = .01: The probability that the test will lead to a Type I error is 1% • How does changing α influence Type II errors? • As we make α smaller, we decrease the chance of making a Type I error. • However, we also make it harder to reject the null; thus increasing the risk of a Type II error
- 22. Critical Region Boundaries Ifα = .05, z = +/-1.96 If α = .01, z = +/-2.58 If α = .001, z = +/-3.30
- 23. THE HYPOTHESIS TEST AnExample
- 24. The Study: Prenatal alcoholexposure on birth weights A random sample of n = 16 pregnant rats is obtained. The mother rats are given a daily dose of alcohol. At birth, one pup is selected from each litter to produce a sample of n = 16 newborn rats. The average weight for the sample is M = 15 grams. It is known that regular newborn rats have an average weight of μ = 18 grams, with σ = 4
- 25. Step 1 State thehypotheses and select the alpha (α) level • For this test, we will use an alpha level of α = .05 • We are taking a 5% risk of committing a Type I error H0 :malcoholexp =18 H1 :malcoholexp ¹18 Even with alcohol exposure, the rats still average 18 grams at birth or Alcohol exposure does not have an effect on birth weight Alcohol exposure will change birth weight
- 26. Step 2 Set thedecision criteria by locating the critical region • The 3-step process: If we obtain a sample mean that is in the critical region, we conclude that the sample is not compatible with the null hypothesis and we reject H0
- 27. Step 3 • Collectthe data, and compute the test statistic • In this case, the z-score: z = M -m sM = 15-18 4 16 æ è ç ö ø ÷ = -3 4 4 æ è ç ö ø ÷ = -3 1 = -3.00
- 28. Step 4 Make adecision • Our z-score (-3.00) exceeds the critical value of +/-1.96 • It is in the critical region • Our interpretation: • This event is unlikely if the null is true Reject the null hypothesis. Alcohol does have a significant effect on the birth weight of rats.
- 29. Things to KeepIn Mind • State the hypotheses: • Your hypotheses lays out the whole point of the test • Phrased in terms of the population parameters • When setting critical values: • What are they? • Are they positive? Negative? Both? • Results • What do we conclude about the null? • State your results in scientific language: The treatment with alcohol had a significant effect on the birth weight of newborn rats, z = -3.00, p < .05. • Conclusions • What do we conclude about the point of the study?
- 30. Factors Influencing aHypothesis Test 1. The size of difference between M and μ • Numerator of z-score • Bigger difference, more likely to reject 2. The variability of the scores (σ) • Denominator of z-score (standard error) • More variability, less accurate, less likely to reject 1. The number of scores in the sample (n) • Denominator of z-score (standard error) • Bigger sample, more accurate, more likely to reject
- 31. Assumptions of HypothesisTesting with z-Scores 1. Random sampling • Ensures a representative sample 2. Independent observations • No consistent relationship between observations • The occurrence of the first has no bearing on the second 3. The value of σ is unchanged by the treatment • In effect, the treatment adds/subtracts a constant from every value in the distribution • Remember, when this occurs, the mean will change but not the SD 4. Normal sampling distribution (n > 30 or normal population) • To justify using the Unit Normal Table
- 32.
- 33. • Two-tailed test •Critical region involves means that are either higher or lower than expected by chance • There are two “tails” to this distribution • One-tailed test • Critical region involves only the area you specify • Specifies an increase or decrease • There is one “tail” to this distribution
- 34. • One-tailed test:• Two-tailed test: H0 :m £100 H1 :m >100 100: 100: 1 0 H H
- 35. CONCERNS ABOUT HYPOTHESIS TESTING Measuringeffect size
- 36. Effect Size Intended toprovide a measurement of the absolute magnitude of a treatment effect, independent of the size of the sample(s) being used. • Statistical significance does not necessarily mean the difference is substantial or meaningful • It allows us to conclude the null hypothesis is unlikely and offers support for the alternative hypothesis • BUT cannot make any real claims about the treatment effect found • If the standard error is very small, even a very small treatment effect may result in rejecting the null hypothesis • With a large enough sample, you will find a significant effect for almost any sized difference
- 37. Measuring Effect Size •(0.0 < d < 0.2) = Small effect size • (0.2 < d < 0.8) = Medium effect size • (d > 0.8) = Large effect size Cohen's d = mean difference standard deviation = mtreatment -mno.treatment s We don’t know this estimated Cohen's d = mean difference standard deviation = Mtreatment -mno.treatment s so we use this
- 38. Cohen’s d usesσ to measure effect size estimated Cohen's d = mean difference standard deviation = Mtreatment -mno.treatment s (let's just call this) Cohen's d = 15 100 = 0.15
- 39. Cohen’s d usesσ to measure effect size estimated Cohen's d = mean difference standard deviation = Mtreatment -mno.treatment s (let's just call this) Cohen's d = 15 15 =1.00
- 40.
- 41. Power The probability thatthe test will correctly reject a false null hypothesis. •Will identify a treatment effect if one truly exists •Factors that affect power: • Effect size • A larger effect size more likely to reject H0 power increases • Sample size • Larger sample size more likely to reject H0 increased power • Alpha level • Lower α less likely to reject H0 less power • One- versus two-tailed test • One-tailed more likely to reject H0 more power