The p-value for the sign test

Ana Marta Lisboa Vila Fernandes

[email protected]

Instituto Superior Técnico, Lisboa, Portugal

December 2015

Abstract
The sign test is a non-parametrical statistical procedure used to evaluate hypothesis about the q-th
continuous population quantile. Denoted χq , with 0 < q < 1, the q-th continuous population quantile,
on the two-sided sign test the null hypotesis states H0 : χq = χ0 while in the alternative H1 : χq 6= χ0 .
The test procedure is based on the sign differences between the sample observations and χ0 . Considering
the statistic Sn that accounts the frequency of the sign (+) on the sample of n observations, it is clear
that Sn ∼ Bin(n, p) with p = 1 − q. When the observed value of the statistic sn is not close to [np],
the sample favours to the alternative hypothesis. A decision rule can be conducted based on the p-
value. Nevertheless, the commonly used p-value formula was originally applied on two-sided tests for
statistics with continuous distributions. In the case of discrete or continuous and asymmetric statistics,
the usual p-value formula can lead to incorrect values. For the two-sided sign test incoherent p-values
can be obtained under particular conditions of the binomial statistic. The main goal of this thesis is to
address in which situations the two-sided sign test p-values meaning is not clear. In order to solve this
problem, there are some alternatives proposed in the literature and their advantages and disadvantages
were analyzed. Also, a new p-value formula was introduced and its efficiency was compared with the
alternative methods for the two-sided sign test.
Keywords:Binomial Distribution, P-value methods, Power of the Sign Test, Two-sided Sign Test.

1. Introduction %) may be not attainable. To illustrate the occur-

rence of p-value greater than one, taking up the
The great boost for the realization of this thesis specific case of the random variable X ∼ N (10, 2)
was the fact that the common p-value formula ap- and the sample of 6 observations generated with the
plied to two-sided sign tests may return probabili- statistical package R (R Core Team, 2014):
ties greater than one. Nevertheless this happening
occurs under particular conditions, which depend H0 : χ 21 = 10 versus H1 : χ 12 6= 10.
on the symmetricity or asymmetricity of the bino-
mial statistic distribution and its concretization for The sign test statistic is S6 ∼ Bin(6, 12 ) and the
the sample. Therefore this fact is often ignored or observed statistic value is s6 = 3. Using the usual
even not noticed by whom applies the test. For the formula for calculating the p-value, the obtained
two-sided tests, and taking the observed value of value is:
the statistics as the reference, it is common prac-
tice to compute the p-value as twice the probability p-value = 2 × min{P (S6 ≤ 3), P (S6 ≥ 3)}
on the tail with lower probability. However, prob- = 2 × 0.6563
lems can arise when the test statistic is discrete.
= 1.3125 > 1.
In the continuous case, all points of the real line
has an associated probability that can be attain-
The main goal of this thesis is to warn of situa-
able. In the case of discrete the scenario is differ-
tions that produce p-values greater than one in the
ent. To illustrate this, let us assume two consecutive
two-sided sign test. Furthermore we also introduce
points n and m = n + 1 under the null hypothesis
a new alternative to compute the p-value, which
of the test statistics, with probabilities pn and pm ,
led to satisfactory results as compared to existing
respectively, such that pn 6= pm . In this case, since
alternatives in the literature.
there are no intermediate points between n and m,
the probability p such that pn < p < pm , is never 2. P-Value
attainable. This can raise problems in hypothesis Under the null hypothesis H0 , the p-value is the
tests, for example a fixed significance level (e.g. 5 probability of observing a result equal to or more

1
extreme than the observed value of the test statis- form the calculations by adopting the properties of
tic. To calculate a p-value it is necessary to know at the Neyman-Pearson methodology. The conclusion
least three conditions: null hypothesis under test, is given from the Fisher perspective, rejecting the
distribution function of the test statistic and a pos- null hypothesis and taking the decision according
sible ordering of the data to allow identification of to the comparison of the pre-specified level of sig-
the data greater than χ0 . This ordering is conve- nificance and the p-value obtained. It is a messy
niently organized in terms of a test statistic. Usu- procedure since these approaches are developed ac-
ally as a p-value is a frequentist concept, the sample cording to different principles and aiming to differ-
space is defined in terms of possible outcomes that ent responses, and may even result in antagonistic
can occur from an infinite repetition of the random conclusions. The main differences identified in these
experience. two approaches are the hypothesis on test and as-
sociated error rates. Fisher mainly focuses on the
2.1. The Ronald Fisher p-value type I error and in the probability of rejecting the
Fisher (1920) was the work that formally introduced null hypothesis when it is in fact true. Neynam and
the notion of p-value and its calculation. In the con- Pearson added to the Fisher’s concern the type II
text of classical inference, Fisher’s approach focuses error. For those statisticians, the probability of not
only in one hypothesis and as final conclusion the rejecting the null hypothesis when it is false should
hypothesis rejection or lack of evidence for that. Ac- also be controlled
cording to Fisher (1935), “Every experiment may be
said to exist only in order to give the facts a chance 3. The sign test
of disproving the null hypothesis”. The definition The sign test is a nonparametric alternative to eval-
of the p-value according to Fisher is similar to the uate hypotheses about the quantiles, location pa-
currently used, but the notions of how it should be rameters of distributions. The sign test may be ap-
used and interpreted are a bit different: first of all, plied to one sample or two paired samples. In this
it is a measure of evidences in a single experiment, paper only the first situation is addressed, namely
used to reflect the credibility of the null hypothesis the test for the quantile χq of a distribution X. The
in the data. Secondly, as a measure of evidences, stated null hypothesis, H0 , is that q − th quantile of
the p-value should be combined with other sources X, χq , is equal to χ0 . The distribution of X does
of information of the phenomenon under study. In not require to be continuous or symmetric, as men-
short, in Fisher’s point of view the p-value is an in- tioned Chakraborti and Gibbons (2004, page 62).
dex that calculates the degree of evidence against However, this test is usually applied when the pop-
the null hypothesis. It says that a p-value ≤ 0.05 ulation X is continuous. For simplicity, in this pa-
(0.05 × 100% significance) is a standard level that per we will deal only with samples from continuous
allows us to conclude that there is evidence against populations. Assuming that the random variable X
the hypothesis tested, although not as an absolute as distribuition function F (x), is a continuous and
rule. strictly increasing function. This test is based on
the fact that under H0 approximately nq × 100%
2.2. The Neyman-Pearson p-value of the n sample observations will be below χ0 and
Jerzy Neyman and Egon Pearson of Neyman and n(1 − q) × 100% will be greater then χ0 . Taking in
Pearson (1928) developed an alternative theoretical to account the differences (xi − χ0 ), i = 1 . . . , n, if
framework. In 1928, they published a background the number of positive signs is approximately equal
paper on the theoretical foundation for a process to n(1 − q) then H0 should not be rejected.
they called hypothesis test. They introduced the
3.1. Hypothesis testing
idea of alternative hypothesis and type II error as-
Consider the case of a continuous population X
sociated to it. In this approach, the error rate must
with distribution function F (x), and a concrete
be given priority to data collection, defining by α
sample (x1 , . . . , xn ) to perform a test on the
the error rate of type I and by β the error rate of
parameter χq . The two-sided test states the hy-
type II. The fact of conduting the test with em-
potheses:
phasis in the Type I error value is related with re-
peatability concept of the sample. For Neyman and H0 : χq = χ0 versus H1 : χq 6= χ0 , (1)
Pearson, the improper rejection of H0 can be done
in α × 100% of the decisions and therefore they pro- with 0 < q < 1.
pose to attribute small values to α.
3.2. Test statistic
2.3. Comparison between the two approaches As previously mentioned under the validity of H0
It is interesting that these two approaches are not it is expected that in the n sample values approxi-
competitors, as they address the problem in a differ- mately nq of those values are smaller than χ0 and
ent perspective. Frequentists analysis usually per- approximately n(1 − q) of those values are greater

2
than χ0 . Denote by Sn the random variable con- 4. Alternatives for p-value calculation
sisting of the number of observations greater than The purpose of this section is to present some alter-
an χ0 in a random sample of size n from X. Us- native formulas for calculating the p-value of two-
ing the indicator function, of follows that Sn = sided sign tests. As previously mentioned, in the
P n
i=0 I(xi − χ0 ) equals the number of positive signs two-sided hypothesis testing usually the p-value is
in the sample of differences (xi − χ0 ), where: defined as twice the value of the smaller p-value
 associated to the two one-sided tests. This duplica-
1 , if xi > χ0 tion is particularly significant and informative only
I(xi − χ0 ) =
0 , if xi ≤ χ0 . for continuous statistics. In contrast, when the dis-
tribution of the test statistic is discrete, as in the
Since P (I(X − χ0 ) = 1|H0 ) = 1 − q, it follows that sign test, there is no consensus about the calcula-
Sn |H0 ∼ Bin(n, 1 − q). Note that the distribution tion of the p-value for two-sided tests. Although
of Sn under H0 does not dependent on the popu- the literature that addresses this issue is not exten-
lation distribution, F (x), from which the sample is sive, some authors have demonstrated concern with
originally. To simplify the notation, we will consider this issue. That was the case of Gibbons and Pratt
(1 − q) = p, so that Sn |H0 ∼ Bin(n, p). (1975) and Kulinskaya (2008), which indicate alter-
native formulas for calculating the p-value. In this
3.3. The rejection region section, we presente the alternatives used in Gib-
For the two-sided test the rejection region results bons and Pratt (1975) and Kulinskaya (2008). Fur-
on the union of two sets for the values of the Sn thermore, we introduce and apply a new formula
statistic. Taking a fixed significance level α, the denoted by dist for calculating the p-value. The
rejection region of the test is Sn ≤ cα/2 or Sn ≥ formulas considered by Gibbons and Pratt (1975)
c∗α/2 . The two constants cα/2 and c∗α/2 are given and Kalinskaya (2008) are confronted and compared
by: cα/2 is the largest integer number such that with the alternative dist for several binomial statis-
cα/2 n i
Σi=0 (i )p (1 − p)n−i ≤ α/2 and c∗α/2 is the smallest tics, with symmetric, asymmetric, unimodal and bi-
integer that satisfies Σni=c∗ (ni )pi (1 − p)n−i ≤ α/2. modal distributions of two-sided sign tests.
α/2
The usual p-value is calculated as: 4.1. Method of Placing
One method that can be used to obtain the p-value
p-valueusual = 2×min{P (Sn ≤ sn |H0 ), P (Sn ≥ sn |H0 )} is .the method of placing (plac), wherein given the
observed value of the statistic, sn , selects an equal
3.4. Statistical Power number of values in the two tails of the distribution.
The power of a test, γ, is the probability of getting In the case of a binomial statistic, assume first that
values in the rejection region when the hypothesis the observed number of successes, sn , is a value
H1 is true. Obviously, the value of γ depends on in the right tail of the distribution. The p-value
the following factors: is defined as the sum of the probabilities in values
larger or equal to sn or smaller or equal to (n − sn ),
1. Discrepancies between the assertions estab- entailing that,
lished in H0 and H1 ,
p-valueplac = P (Sn ≥ sn ) + P (S ≤ n − sn ).
2. The rejection region of test, Similarly, it s is a value in the left tail of the dis-
n
3. Significance level, tribution, then:

4. Sample size. p-valueplac = P (Sn ≤ sn ) + P (Sn ≥ n − sn ).

Example: Assuming a binomial statistic with
The probability γ is usually taken as a criterion to n = 10 and p = 0.6, under H0 , and take up s10 = 3
evaluate test performance, selecting the tests with for the observed value of the statistic. The prob-
high values of γ. This is natural, since γ is the com- ability function of S10 |H0 ∼ Bin(10, 0.6) is pre-
plementary of the probability of making a type II sented in Figure 1. Note that this is a unimodal
error, γ = 1 − β. When H1 consists a composite and slightly right-skewed distribution. In Table 1
hypothesis, γ is a function of the values stipulated the probability function of S10 is presented. As the
in H1 . In order to be able to calculate the power value s10 = 3 is the fourth largest value in the left
of a test, it is necessary to know the statistic distri- tail of the distribution, the correspondent value in
bution under H1 . In contrast to many other non- the right tail counting down from 10 is the value 7.
parametric tests, the power function for the sign Thus the p-value is given by:
test is very simple to determine. In this test there
is the advantage that Sn |H1 is also a random vari- p-valueplac = P (S10 ≤ 3) + P (S10 ≥ 7)
able with binomial distribution. = 0.055 + 0.382 = 0.437.

3
 are {0, 1, 2, 3} ∪ {9, 10}. Thus,

p-valuepml = P (S10 ≤ 3) + P (S10 ≥ 9)

= 0.055 + 0.046 = 0.101.
0.20

4.3. Conditional method

The calculation formula of the conditional p-value,
f(s)

0.10

proposed by Kulinskaya (2008), is given by:

P (Sn ≤ sn )
0.00

p-valuecond = × I{sn <ν}

P (Sn ≤ ν)
0 2 4 6 8 10
+ I{sn =ν}
s P (Sn ≥ sn )
+ × I{sn >ν} . (2)
P (Sn ≥ ν)
Figure 1: Probability function of the Bin(10, 0.6) where sn is the observed value of the statistic and ν
distribution. is a location measure, such as the mean, the median
or the mode of Sn .
Equation 2 can be rewritten as:
Since P (S10 ≥ 7) is about 7 times larger than
P (S10 ≤ 3), there is an increase of the probability P (Sn ≤ sn ) P (Sn ≥ sn )
p-valuecond = min{ , },
regarding to p-valueusual = 2 × P (S10 ≤ 3) = 0.11 P (Sn ≤ ν) P (Sn ≥ ν)
(approximately 4 times).
since both produce the same results. Took up for
the parameter ν the median of Sn , denoted by w,
s P (S10 = s|H0 )
since the median is robust measure of location.
0 0.0001
Example: Returning to the example of the statis-
1 0.0016
tic S10 |H0 ∼ Bin(10, 0.6) with median w = 6, since
2 0.0106
s10 = 3 < w entails that:
3 0.0425
4 0.1115 P (S10 ≤ 3)
5 0.2007 p-valuecond = = 0.0887 .
P (S10 ≤ 6)
6 0.2508
7 0.2150 The p-value produced by this calculation formula
8 0.1209 is lower than p-valueplac and p-valuepml . Note that
9 0.0403 considering the significance level α fixed at 0.1, the
10 0.0060 p-value cond would lead to the rejection of the hy-
pothesis H0 : χ0.4 = χ0 contrary to the decision
based on the p-valueplac and p-valuepml . To eval-
Table 1: Probabibity function of the Bin(10, 0.6) uate the performance of this conditional formula,
distribution. assume that the observed value of the statistic is
s10 = 8 > w. It follows that:

4.2. Method Principle of Minimum Likelihood P (S10 ≥ 8)

p-valuecond =
With the principle of minimum likelihood method, P (S10 ≥ 6)
pml, the p-value is the sum of the probabilities in = 0.2642 .
all values of Sn whose probability does not exceeds
P (Sn = sn ). Therefore, If the observed value of the statistic equals the me-
dian of S10 , s10 = 6 = w, then:
n
X p-valuecond = 1 .
p-valuepml = P (Sn = i|H0 ) ×
i=0
Few observations can be taken when this method
I{P (Sn =i|H0 )≤P (Sn =sn |H0 )} .
is applied to the sign test. In the first analysis, the
mean and median of the binomial distribution can
Example: Considering the previous example with take values that do not belong to the set of natu-
S10 |H0 ∼ Bin(10, 0.6) and s10 = 3. The probability ral numbers, N, and in Kulinskaya (2008) there is
of the value 3, as shown in Table 1, is P (S10 = no reference to this situation. When this occurs,
3|H0 ) = 0.0425. The set of points in the range of the wisest course is to consider the integer part of
S10 , whose probabilities that do not exceed 0.0425 the location parameter. Another observation can

4
be made about the statistic Sn |H0 ∼ Bin(n, 21 ) by the sum of the probabilities presented in blue:
when n is even. In this case since P (Sn ≤ ν) =
P (Sn ≥ ν) = 21 , the p-valuecond equation equals to p-valuedist = P (Sn ≤ 3) + P (Sn ≥ 9)
p-valueusual : = 0.101.
P (Sn ≤ sn ) P (Sn ≥ sn ) The p-value with the method dist can be computed
p-valuecond = min{ , }
P (Sn ≤ ν) P (Sn ≥ ν) as:
= 2 × min{P (Sn ≤ sn ), P (Sn ≥ sn )} w+d−1
X
= p-valueusual . p-valuedist = 1 − { P (Sn = i) × I{sn <w}
i=sn +1
However, for Sn |H0 ∼ Bin(n, 21 ) with n even, both
n −1
sX
P (Sn ≤ ν) and P (Sn ≥ ν) are not equal to 0.5.
+ P (Sn = i) × I{sn >w} }.
This occurs because Sn is a discrete statistic and
i=w−d+1
some probabilities are not attainable. Therefore,
when n is even the cond method is not equal to the The dist method is very similar to the pml method,
usual method. therefore sometimes both methods lead to the same
results.
4.4. New Proposal - Distance Method
Bearing in mind the studied methods, we will de- 4.5. Comparison and analysis of the p-value meth-
velop a new method (dist) to compute the p-value ods
that is consistent with the hypothesis H0 and that The aim is compare the p-values obtained by the
does not result in p-values greater than one. Con- five methods, usual, plac, pml, cond and dist,
sider the test statistic Sn |H0 ∼ Bin(n, p) of the when the distribution of the statistic Sn given H0 is
sign test and w = χ 12 the median of Sn as the symmetric or asymmetric. To carry out this study,
central measure of location. Suppose that the ob- we developed computational implementations, with
served value of statistic in a two-sided test to the software R, of the p-value methods. The analysis
q-th quantile of X is sn and d let denote the eu- consists in obtaining the p-value with the different
clidean distance between sn and w, d = |w − sn |. methods of the two-sided test: H0 : χq = χ0 vs
The p-valuedist is given by the sum of probabili- H1 : χq 6= χ0 , and q = 0.1, 0.25, 0.5, 0.75 and 0.9.
ties of the points whose distance to w is larger or The goal is to verify in which methods, usual or
equal than d. This procedure is especially intuitive plac, the p-value is greater than one and in which
when seeking to interpret the p-value as the degree cases the decision of the test can change. In Tables
of agreement or disagreement between the observed 2 and 3 we report the p-values obtained by the five
value of the statistic and the median of Sn . methods for samples with size n = 6 and n = 7,
Example: Return to S10 |H0 ∼ Bin(10, 0.6) with respectively. The results of the study can be sum-
w = 6 and sn = 3, so that d = |w − sn | = 3. In marized as follows:

1. plac method

• When the distribution of Sn is quite

asymmetric, the p-valueplac can lead to
0.30

false decisions if sn is unlikely to occur.

0.20

• If Sn asymmetric and sn = w then the

f(s)

p-value is smaller than one.

0.10

n
• When n is an even number and sn = 2,
then p-valueplac > 1.
0.00

• If n is an odd number and the observed

0 2 4 6 8 10
value of the statistic is sn = b n2 c or sn =
s d n2 e, then the p-value is equal to one.
• This method produces acceptable results
when the statistic Sn has symmetric or
Figure 2: Probability function of S10 ∼ Bin(10, 0.6) slightly asymmetric distribution.
illustrative of the distance method.
2. pml method
Figure 2, we represent in blue the probability value
in points whose distance to the median w is equal or • In samples with even or odd size and bi-
larger than d and in dashed (green) the probability nomial statistics with symmetric distribu-
value in the other points. The p-valuedist is given tions p-valuepml = p-valueusual . If n is

5
 p-value
q s6 usual plac pml cond dist • If Sn |H0 has an asymmetric distribution,
0.1 0 0.0000 0.5314 0.0000 0.0000 0.0000
1 0.0001 0.8858 0.0001 0.0001 0.0001 on the right (left), where every point i
2 0.0025 0.9854 0.0013 0.0013 0.0013
3 0.0317 1.0146 0.0158 0.0158 0.0158 that satisfies P (Sn = i) ≤ P (Sn = sn )
4 0.2285 0.9854 0.1143 0.1143 0.1143
5 0.9371 0.8858 0.4686 0.4686 0.4686 is located at the right (left) of sn it is ver-
6 1.0629 0.5314 1.0000 1.0000 1.0000
p-valueusual
0.25 0
1
0.0005
0.0093
0.1782
0.5386
0.0002
0.0046
0.0003
0.0056
0.0002
0.0046
ified that p-valuepml = 2 .
2 0.0752 0.8682 0.0376 0.0457 0.0376
3 0.3389 1.1318 0.1694 0.2061 0.1694 • When sn is the mode of Sn then
4 0.9321 0.8682 0.6440 0.5670 0.6440
5 1.0679 0.5386 1.0000 1.0000 1.0000 p-valuepml = 1.
6 0.3560 0.1782 0.3474 0.3333 0.6440
0.5 0 0.0313 0.0312 0.0313 0.0238 0.0312
1
2
0.2188
0.6875
0.2188
0.6875
0.2188
0.6875
0.1667
0.5238
0.2187
0.6875
3. cond method
3 1.3125 1.3125 1.0000 1.0000 1.0000
4 0.6875 0.6875 0.6875 0.5238 0.6875
5
6
0.2188
0.0312
0.2188
0.0312
0.2188
0.0313
0.1667
0.0238
0.2187
0.0312
• If Sn is symmetric and n is an even num-
0.75 0 0.3560 0.1782 0.3474 0.3333 0.6440 ber, the p-value returned by the cond
1 1.0679 0.5386 1.0000 1.0000 1.0000
2 0.9321 0.8682 0.6440 0.5670 0.6440 method is never equal to p-valueusual .
3 0.3389 1.1318 0.1694 0.2061 0.1694
4 0.0752 0.8682 0.0376 0.0457 0.0376 This is a consequence of P (Sn ≥ w) and
5 0.0093 0.5386 0.0046 0.0056 0.0046
6 0.0005 0.1782 0.0002 0.0003 0.0002 P (Sn ≤ w) never being equal to 0.5.
0.9 0 1.0629 0.5314 1.0000 1.0000 1.0000
1 0.9371 0.8858 0.4686 0.4686 0.4686
2 0.2285 0.9854 0.1143 0.1143 0.1143 • In general this method returns coherent
3 0.0317 1.0146 0.0158 0.0158 0.0158
4 0.0025 0.9854 0.0013 0.0013 0.0013 p-values. However the associated values
5 0.0001 0.8858 0.0001 0.0001 0.0001
6 0.0000 0.5314 0.0000 0.0000 0.0000 are relatively lower than p-valueusual .

4. dist method
Table 2: P-values of the test statistic S6 , with
S6 |H0 ∼ Bin(6, p), for the test H0 : χq = χ0 vs • The dist method sometimes produces
H1 : χq 6= χ0 obtained by the usual, plac, pml, similar results to the pml method. How-
cond and dist methods . ever, it is noted that when the test statis-
tic has a symmetric distribution and n is
p-value
q s7 usual plac pml cond dist an odd number p-valuedist 6= p-valuepml ,
0.1 0 0.0000 0.4783 0.0000 0.0000 0.0000
1 0.0000 0.8503 0.0000 0.0000 0.0000 except when sn corresponds to the limits
2 0.0004 0.9745 0.0002 0.0003 0.0002
3 0.0055 1.0000 0.0027 0.0052 0.0027 of the median class.
4 0.0514 1.0000 0.0257 0.0492 0.0257
5
6
0.2994
1.0434
0.9745
0.8503
0.1497
0.5217
0.2869
1.0000
0.6280
1.0000 • If the distribution of Sn is asymmetric
7 0.9566 0.4783 1.0000 0.5625 0.6280
0.25 0 0.0001 0.1335 0.0001 0.0001 0.0001
with q > 0.5 whenever sn < w and
1 0.0027 0.4463 0.0013 0.0024 0.0013
2 0.0258 0.7693 0.0129 0.0232 0.0129
P (Sn = sn ) < P (Sn = w + d) or sn > w
3
4
0.1411
0.4872
1.0000
1.0000
0.0706
0.3771
0.1271
0.4389
0.2040
0.6885
and P (Sn = sn ) > P (Sn = w − d), then
5
6
1.1101
0.8899
0.7693
0.4463
1.0000
0.6885
1.0000
0.5882
1.0000
0.6885
p-valuedist 6= p-valuepml . The same hap-
7 0.2670 0.1335 0.2040 0.1765 0.2040
0.5 0 0.0156 0.0156 0.0156 0.0101 0.0078
pens with q < 0.5, whenever sn > w and
1
2
0.1250
0.4531
0.1250
0.4531
0.1250
0.4531
0.0808
0.2929
0.0703
0.2891
P (Sn = sn ) < P (Sn = w − d) or sn < w
3
4
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
and P (Sn = sn ) > P (Sn = w + d).
5 0.4531 0.4531 0.4531 0.2929 0.2891
6
7
0.1250
0.0156
0.1250
0.0156
0.1250
0.0156
0.0808
0.0101
0.0703
0.0078
• Whenever p-valuedist 6= p-valuepml it also
0.75 0
1
0.2670
0.8899
0.1335
0.4463
0.2040
0.6885
0.1765
0.5882
0.2040
0.6885
happens that p-valuedist is different from
2
3
1.1101
0.4872
0.7693
1.0000
1.0000
0.3771
1.0000
0.4389
1.0000
0.6885
the p-value returned by the other meth-
4
5
0.1411
0.0258
1.0000
0.7693
0.0706
0.0129
0.1271
0.0232
0.2040
0.0129
ods.
6 0.0027 0.4463 0.0013 0.0024 0.0013
7 0.0001 0.1335 0.0001 0.0001 0.0001 • The dist method returns coherent p-
0.9 0 0.9566 0.4783 1.0000 0.5625 0.6280
1 1.0434 0.8503 0.5217 1.0000 1.0000 values that are never greater than one.
2 0.2994 0.9745 0.1497 0.2869 0.6280
3 0.0514 1.0000 0.0257 0.0492 0.0257
4
5
0.0055
0.0004
1.0000
0.9745
0.0027
0.0002
0.0052
0.0003
0.0027
0.0002
5. Simulation Study
6
7
0.0000
0.0000
0.8503
0.4783
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
We will condute a simulation study in order to eval-
uate the performance of the previously presented
methods to compute the p-value and power calcu-
Table 3: P-values of the test statistic S7 , with lations of the sign test:
S7 |H0 ∼ Bin(7, p), for the test H0 : χq = χ0 vs
H1 : χq 6= χ0 obtained by the usual, plac, pml, H0 : χq = χ0 versus H1 : χq 6= χ0 ,
cond and dist methods.
with q = 0.1, 0.25, 0.5, 0.75 and 0.9. To perform
that, we simulated N = 10000 samples with n data
an even number the method pml never from symmetric or asymmetric distributions. The
returns p-values greater than one, con- n data sets were generated, in the R program, as
trary to what may happen with the usual originating from reduced normal distribution and
method. Gama(3, 0.1). Furthermore, we also used generated

6
 H0 : χ
data from a neighborhood-ε of the F = N (0, 1) dis- 0.5 =0
p-value usual plac pml cond dist
tribution, with 0 < ε < 1, as: > 1
≤ 0.01
3072
0
3072
0
0
0
0
0
0
0
≤ 0.05 339 339 339 339 339
≤ 0.1 339 339 339 339 339
Fε (F ) = {G : G = (1 − ε)F + εH} , (3) n = 10
> 1 2431 2431 0 0 0
≤ 0.01 32 32 32 32 32
and H = N (2, 4). The sample size n was taken as ≤ 0.05 221 221 221 221 221
≤ 0.1 221 221 221 1102 221
n = 6, 7, 10, 11, 20, 21, 30 and 31. The q-th quantiles n = 20
> 1 1768 1768 0 0 0
values, χq , were obtained in R and assigned to χ0 . ≤ 0.01 26 26 26 118 26
≤ 0.05 389 389 389 389 389
For each method we recorded the relative frequency ≤ 0.1 389 389 389 1141 389
of p-value ≤ α, with α = 0.01, 0.05 and 0.1, on > 1 1415 1415
n = 30
0 0 0
the 10000 generated samples. Furthermore, we have ≤
≤
0.01
0.05
54
409
54
409
54
409
54
409
54
409
recorded also the frequency of p-value > 1 for the ≤ 0.1 998 998 998 998 998

usual and the plac methods.

The aim of this study was check whether there Table 4: Relative frequency of simulated samples
are significant differences among the methods. To from F = N (0, 1) with p-value > 1, p-value ≤ 0.01,
assess the power test, another simulation was per- p-value ≤ 0.05 and p-value ≤ 0.1 for the median
formed by establishing in H1 contiguous hypotheses test, when n = 6.
to H0 . Here we considered the test H0 : χq = χ0
versus H1 : χq = χ0 + ∆, where χ0 refers to the
q−th quantile of X ∼ N (0, 1). Now, under H1 , the H0 : χ
0.5 =0
samples were generated from F1∗ = N (∆, 1) when p-value usual plac pml cond dist
> 1 1 1 0 0 0
∆ > 0, or from F2∗ = N (∆, 1) case ∆ < 0. For each ≤ 0.01 0 0 0 0 168
≤ 0.05 168 168 168 168 168
sample, we evaluated γ(∆) = P (reject H0 |H1 ) with ≤ 0.1 168 168 168 1300 1300
n = 11
the five methods. The study of the sign test power > 1 0 0 0 0 0
≤ 0.01 13 13 13 130 130
with data sets from asymmetric distributions was ≤ 0.05 130 130 130 665 665
≤ 0.1 665 665 665 665 665
not performed because in this case it is not easy to n = 21
> 1 2 2 0 0 0
change the q−th quantile without changing the dis- ≤ 0.01 69 69 69 69 69
≤ 0.05 271 271 271 271 271
tribution variance. The purpose of this simulation ≤ 0.1 753 753 753 753 753
n = 31
was to assess the methods that conduce to small > 1 2 2 0 0 0
≤ 0.01 30 30 30 107 107
values of the probability of type II error. ≤ 0.05 274 274 274 274 274
≤ 0.1 705 705 705 705 705

5.1. Simulation of the p-value

• Generation of N = 10000 data sets with di-
Table 5: Relative frequency of simulated samples
mension n = 6, 7, 10, 11, 20, 21, 30 and 31 from
from F = N (0, 1) with p-value > 1, p-value ≤ 0.01,
the distribution F = N (0, 1).
p-value ≤ 0.05 and p-value ≤ 0.1 for the median
In tables 4 and 5 we reported the results obtained to test, when n = 7.
the two-sided tests for the median of X ∼ N (0, 1).
For each value of n and each method, the number of
samples (from 0 to 10000) that checked the p-value
conditions listed in the first column of these tables With the five methods, we calculated γ(∆i ) =
are presented. P (reject H0 |H1 ). Under H1 , as the the data comes
from F1∗ or F2∗ , it follows that S1∗ n ∼ Bin(n, p∗1 )
5.2. Power Simulation
with p∗1 = P (X > χ0 |F1∗ ), and S2∗ n ∼ Bin(n, p∗2 )
As it is well known, a statistical test with low power
with p∗2 = P (X > χ0 |F2∗ ). Note that the power
to detect that the sample is consistent with H1
functions of these tests, except for the median test
causes indecision about the validity of H0 . In order
of X, does not exhibit any symmetry around the
to compare the power value of the sign test pro-
value χ0 . We evaluated the average powers ob-
duced by the methods usual, plac pml, cond and
tained in the N = 10000 simulated samples. For the
dist, we considered the two-sided test:
usual and plac methods the samples which gives
H0 : χq = χ0 vs H1 : χq = χ0 + ∆i , rise to γ(∆i ) > 1 were removed, since these two
methods can return a probability larger than one.
where χ0 indicates the q-quantile of X ∼ N (0, 1)
Figures 3 and 4 present the average power obtained
and ∆i = ±0.25, ±0.5, ±1.5, ±2, ±2.5 and ±3. Un-
for the median test of F .
der H1 we generated:
• N = 10000 samples with dimension n = 6. Interpretation of simulation results
6, 7, 10, 11, 20, 21, 30 and 31 from F1∗ =
N (∆i , 1), ∆i > 0, or from F2∗ = N (∆i , 1), Then, we present the results obtained in the simu-
∆i < 0. lation of the p-value and power simulation.

7
 1.0
n=6 n=7
or asymmetric, behaviour of the distribution

1.0
of X. The cardinal samples that verify each
of the conditions about the p-value obtained in
0.8

0.8
the q−th quantile test were similar, which leads
0.6

0.6
Potência

Potência
us to conclude that the results are independent
0.4

0.4
of the asymmetry of the distribution of X.
0.2

0.2
USUAL USUAL
COL COL
PVM PVM
COND COND
• Regarding the conditions, p-value ≤ 0.01,
0.0

0.0
DIST DIST

0.5 1.0 1.5

∆
2.0 2.5 3.0 0.5 1.0 1.5

∆
2.0 2.5 3.0
p-value ≤ 0.05 and p-value ≤ 0.1, as larger
is the sample size n the larger is the number of
n=10 n=11 samples that verify such conditions. The op-
posite happens with the p-value > 1 condition.
1.0

1.0

USUAL USUAL
COL COL
PVM PVM
COND COND
DIST DIST
0.8

0.8

• It is notorious that p-valueusual > 1 occurs

0.6

0.6

more often than p-valueplac > 1.

Potência

Potência
0.4

0.4

• For the median test, when n is odd, the number

0.2

0.2

of samples that verify the p-value > 1 condition

0.0

0.0

0.5 1.0 1.5 2.0 2.5 3.0 0.5 1.0 1.5 2.0 2.5 3.0
is neglegible on the 10000 simulated samples.
∆ ∆

• When H0 establishes that the quantiles of F

Figure 3: Power functions for the methods usual,
are equal to the quantiles of F = N (0, 1), it was
plac, pml, cond and dist for the median test with
observed that the results of the test to quan-
H1 : χ0.5 = 0 + ∆i , ∆i = 0.25, 0.5, 1.5, 2, 2.5 and 3.
tiles q = 0.75 and q = 0.9 are obviously the
Sample size equals to n = 6, 7, 10 and 11.
most affected. The contamination of samples
n=20 n=21
with observations generated from N (2, 4) does
not affect the smaller quantiles of F . The in-
1.0

1.0

USUAL USUAL
COL COL
PVM
COND
DIST
PVM
COND
DIST
troduction of noise in the samples can decrease
0.8

0.8

the sign test performance.

0.6

0.6
Potência

Potência

6.2. Summary results of the power simulation study

0.4

0.4

• In general, the usual method proved to be a

0.2

0.2

poor method, with smaller average power than

0.0

0.0

0.5 1.0 1.5 2.0 2.5 3.0 0.5 1.0 1.5 2.0 2.5 3.0
the other methods.
∆ ∆

• As larger is the difference between the expected

n=30 n=31 value of the distributions stipulated under H0
and H1 , the greater will be the power of the
1.0

1.0

USUAL USUAL
COL COL
PVM PVM
COND COND
DIST DIST
test. This increase of γ was observed in all
0.8

0.8

methods, except for the usual method which

0.6

0.6
Potência

Potência

falls steeply (γ ≈ 0) with the increase of |∆i |,

0.4

0.4

showing the instability of the method. One rea-

0.2

0.2

son for this to occur is the number of samples

that are removed of the power study because
0.0

0.0

0.5 1.0 1.5 2.0 2.5 3.0 0.5 1.0 1.5 2.0 2.5 3.0

∆ ∆ verified P (rejectH0 |H1 ) > 1.

Figure 4: Power functions for the methods usual, It was found that as larger is the fixed value for
plac, pml, cond and dist for the median test with |∆|, greater is the number of removed samples
H1 : χ0.5 = 0 + ∆i , ∆i = 0.25, 0.5, 1.5, 2, 2.5 and 3. from the study. The samples are generated ac-
Sample size equals to n = 20, 21, 30 and 31. cording to the distribution of H1 and the num-
ber of observations in the sample of X above
χ0 increases when ∆ > 0 and decreases when
6.1. Summary results for the p-value simulation ∆ < 0.
study
As a consequence, the observed value of the
• In general, the sample size n influences the de-
test statistic sn coincides more times with the
cision of the sign test. Different decisions are
median of S1∗ n or of S2∗ n , respectively, when
obtained for each value of n.
∆ > 0 or ∆ < 0.
• The p-value results for the q− th quantile tests The usual method returns p-values greater
were not influenced by the shape, symmetric than one when sn = w, therefore the number

8
 of samples that satisfy γ > 1 increases with test when selected instead of usual. The usual
increasing |∆|. method rejects the hypothesis H0 less often.
• Good results were recorded with dist method, • If the Sn distribution is fairly asymmetric, the
in general, for small values of |∆i |, proving dist application of the usual, dist and pml methods
to be the method with the greatest average can lead to changes on the decision of the test.
power. Furthermore, in general, the increase
of ∆i lead to greater or equal powers than in 7.1. Achievements
the other methods. From the alternative methods studied, plac, pml,
cond and dist, it is not possible to support in favor
• For small values of |∆i |, the average of γ be- of a particular method, since all of them have their
longs to [0.55, 0.7]. When ∆i increase, the av- on limitations. However, the plac method proved
erage of γ also increases, getting very close to to be an inconsistent method and therefore is not
one, with the exception of the usual method. recommended.
• When the q−th quantile increases, we observe a The results obtained by the dist method at power
slight decrease in the average of γ when ∆ > 0 simulation of the q−th quantile test for the stan-
and a slight increase where ∆ < 0. dard normal were quite satisfactory. Typically,
this method produces a greater average power than
7. Conclusions other methods.
After all the extensive analysis of the five p-value It was possible to concluded that, apart from
calculation methods, usual, plac, pml, cond and some exceptional cases, low powers are obtained
dist, the main features of each were appointed. The with the usual method, enhancing the inconsis-
usual and plac methods are the only ones that tency of the method. In short, the study in this
return p-values greater than one. In the case of paper gives to evidence the weaknesses of the usual
the usual method, p-valueusual > 1 when sn = w. method. Although the usual method is the most
Whenever Sn has a median class, the p-valueusual = used to perform statistical tests, it did not show to
1 when sn equals one of the limits of this class. be appropriate to determine the p-value in the two-
In the plac method, the p-value is greater than sided test. In contrast, the dist method proved to
one when n is even and sn = n2 . Another note be a good option for the p-value calculation.
about this method is that it only returns accept-
able p-values when Sn is a symmetric or a weakly 7.2. Future work
asymmetric distribution. Regarding the methods As proposal for future work we suggeste to address
pml, cond and dist, these never produce p-values the justification for the choice of the euclidean dis-
greater than one. The p-valuepml = p-valueusual tance, d = |sn − w|, used to calculate the p-value
when the test statistic is symmetric. When Sn is by the dist method, or an alternative metric that
asymetric, there are situations where p-valuepml is optimizes the method. The realization of an equal
equal to one half of p-valueusual . When the ob- study for the Wilcoxon test or another test that has
served value of the statistic corresponds to the mode a test statistic with discrete or continuous asymmet-
of Sn , p-valuepml = 1. Comparing the methods ric distribution, is another possible topic for future
cond and usual it can be observed that, in general, work. This is a challenge, since the test statistic
p-valuecond < p-valueusual . When the value of sn of the Wilcoxon test is not as simple as the bino-
corresponds to the median of Sn then p-valuecond = mial distribution of the sign test. Furthermore, the
1. The recently introduced method, dist method, p-value is greater than one when sn = w and there-
usually produces the same results that are obtained fore the determination of the median of Sn for the
by the pml method. The dist method showed a Wilcoxon test is also an hurdle to overcome.
very positive performance, producing coherent re-
Acknowledgements
sults. Based on the comparison study of the p-value
I warmly thank to Prof. Isabel Rodrigues for her
calculation methods and simulation study, setting
support, availability and dedication. It was a plea-
α × 100% in the usual significance levels (1%, 5%
sure working with her. I also want to thank my
and 10%), we observe that:
family and friends for believing in me and giving
• When the distribution of Sn is symmetric, the me strength in every time I did not believe in my-
cond and dist methods may return p-values self. I got it!
relatively lower than other methods. Thus, for
the usual significance levels, the cond and dist References
methods reject more times the hypothesis H0 . Fisher, R. A. (1920). A Mathematical Exami-
nation of the Methods of Determining the Ac-
• Case the distribution of Sn is asymmetric, all curacy of an Observation by the Mean Error
methods can change the decision of the sign and by the Mean Square Error. Monthly No-

9
tices of the Royal Astronomical Society, 80,
758-770.
Fisher, R. A. (1935). The Design of Experi-
ments. Hafner Press, New York, 9th ed.
Gibbons, J. D. and Chakraborti, S. (2004).
Nonparametric Statistical Inference. Marcel
Dekker, New York, 4th ed.

Gibbons, J. D. and Pratt, J. W. (1975). P-

values: Interpretation and Methodology. The
American Statistician, 20, 20-26.
Kulinskaya, E. (2008). On two-sided p-
values for non-symmetric distributions.
arXiv:0810.2124.
Neyman J. and Pearson, E. S. (1928). On
the use and interpretation of certain test cri-
teria for purposes of statistical inference: part
I. Biometrika, 20, 175-240

R Core Team (2014). R: A Language and En-

vironment for Statistical Computing. Vienna,
Austria: R Foundation for Statistical Comput-
ing.

10