Research Methods 1 Handouts, Graham Hole,COGS - version 1.0, September 2000: Page 1::
HYPOTHESIS TESTING:
The story so far...
Step 1:
(a) Scores often tend to be normally distributed.
(b) Any given score can be expressed in terms of how much it differs from the
mean of the population of scores to which it belongs (i.e., as a z-score).
Step 2:
(a) Sample means tend to be normally distributed around the population mean
(thanks to the "Central Limit Theorem").
(b) Any given sample mean can be expressed in terms of how much it differs from
the population mean.
(c) "Deviation from the mean" is the same as "probability of occurrence": a sample
mean which is very deviant from the population mean is unlikely to occur.
Step 3:
(a) Differences between the means of two samples that come from the same
population are also normally distributed. Most samples from the same population should
have similar means; and hence most differences between sample means should be small.
(b) Any observed difference between two sample means could be due to either of
two possibilities:
(i) they are two samples from the same population, that happen to differ by chance
(the "null hypothesis", "HO"); OR
(ii) they are not two samples from the same population, but instead come from two
different populations (the "alternative hypothesis", HA).
Assume "innocent until proven guilty": we retain the null hypothesis unless the
weight of evidence allows us to reject it in favour of the alternative hypothesis.
Large difference between sample means: reject HO, and
assume HA is true.
Small difference between sample means: have no reason to reject HO.
How big must a difference be, before we reject HO?
Type 1 error: reject HO when it is true. (Also known as ��, "alpha"). i.e., we think our
experimental manipulation has had an effect, when in fact it has not.
Type 2 error: retain HO when it is false. (Also known as ��, "beta"). i.e., we think our
experimental manipulation has not had an effect, when in fact it has.
Research Methods 1 Handouts, Graham Hole,COGS - version 1.0, September 2000: Page 2::
What we actually have in practice...
Interpretation under the null hypothesis: samples come from the same population:
Interpretation under the alternative hypothesis: samples come from different populations:
mean of
sample A
mean of
sample B
population mean
mean of
sample A
mean of
sample B
mean of population A
mean of population B
mean of
sample A
mean of
sample B
Research Methods 1 Handouts, Graham Hole,COGS - version 1.0, September 2000: Page 3::
Any observed difference between two sample means could in principle be either
"real" or due to chance - we can never tell for certain. Large differences between samples
from the same population are unlikely to arise by chance.
Small differences between samples are likely to have arisen by chance.
The 0.05 Significance Level:
The problem is that reducing the chances of making a Type 1 error increases the
chances of making a Type 2 error, and vice versa.
Psychologists therefore compromise between the chances of making a Type 1
error, and the chances of making a Type 2 error:
We set the probability of making a Type 1 error at 0.05.
When we do an experiment, we accept a difference between two samples as "real",
if a difference of that size would be likely to occur, by chance, 5% of the time, i.e. five
times in every hundred experiments performed.
Summary of a typical experimental procedure and analysis:
(a) Perform your experiment; find the mean of each sample (i.e., experimental
condition or group), and the difference between these means.
(b) Assume the null hypothesis - that the experimental manipulations have had no
effect , and the two samples are still two samples from the same population.
(c) Assess the probability of obtaining by chance a difference between sample
means as large as the one you have found. (In practice, consult a table).
(e) If this probability is 0.05 or smaller, then reject the null hypothesis, in favour of
the alternative hypothesis that the difference between the sample means has not arisen by
chance.
(f) If this probability is larger than 0.05, retain the null hypothesis - no reason to
assume that the differences between sample means aren't just due to chance.
Research Methods 1 Handouts, Graham Hole,COGS - version 1.0, September 2000: Page 4::
Directional and non-directional hypotheses:
Non-directional (two-tailed) hypothesis:
Merely predicts that sample means A and B will be significantly different from each other:
Directional (one-tailed) hypothesis:
More precise - predicts the direction of difference (i.e., either predict A is bigger than B, or
predict B is bigger than A).
possible differences between sample means
A>B differences this extreme (or more)
occur by chance with p = 0.025.
A = B
A > B
A < B
A<B differences this extreme (or more)
occur by chance with p = 0.025.
A>B differences this extreme (or more)
occur by chance with p = 0.05.
possible differences between sample means
A = B
A > B
A < B