E-value

---

Statistical hypothesis testing is a method of statistical inference that may be performed according to the frequentist or Bayesian paradigms. The e-value provides a Bayesian evidential measure of how strongly observed data support statistical hypotheses classified as sharp hypotheses. As in traditional methods, these statistical hypotheses refer to the parameters of the adopted statistical model, that supposedly describe the data generating process. However, when sharp hypotheses are satisfied, the parameters of the statistical model must belong to a subset of the parameter space that usually has fewer degrees of freedom than the parameter space itself. Point hypotheses are a very useful kind of sharp hypotheses.

Since the Bayesian school typically uses Bayes factors or posterior probabilities to test hypotheses, this may imply a technical difficulty as it requires the adoption of mixed priors on the parameter space. This means that the prior distribution should attach some probability mass to specific (or a countable number of) points of the parameter space and the remainder of the probability mass is continuously distributed on the rest of the parameter space.

The e-value is a Bayesian measurement derived from the Full Bayesian Significance Test (FBST), a procedure designed to test sharp hypotheses which does not require the use of a special kind of prior and is based on the full posterior distribution of the parameters, i.e. it does not require marginalization of nuisance parameters.

Basic concepts

A statistical hypothesis is any conjecture about the parameters of a (parametric) statistical model, a family of probability distributions that supposedly describes a collection of random variables, that represents observed data. In this article, $\mathbf{𝐗}$ denotes the collection or vector of random variables to be observed in the experiment and $\mathfrak{𝔛}$ the set of all possible values this vector may assume, also called sample space. The observed random variables are usually real-valued and, for this reason, $\mathfrak{𝔛}\subseteq {\mathbb{ℝ}}^n$, where $n\in \mathbb{\mathbb{N}}$ is the sample size or number of variables observed in the experiment. Lowercase $\mathbf{𝐱}$ denotes a sample point or data set observed from the experiment. The random vector is assumed to be a random sample (the random variables are independent and identically distributed or i.i.d.) from a parametric statistical model

${ℱ}=\{f(\mathbf{𝐱}|\theta ):\mathbf{𝐱}\in \mathfrak{𝔛},\theta \in \mathrm{\Theta }\}$

in which $f$ is a probability density or mass function of a specific statistical model and $\mathrm{\Theta }$ is the parameter space, set of all possible values of the parameters of the model (usually a subset of ${\mathbb{ℝ}}^k$, $k\in \mathbb{\mathbb{N}}$ the dimension of such a space. Given these assumptions, a sharp hypothesis is defined by any subset of the parameter space ${\mathrm{\Theta }}_H\subset \mathrm{\Theta }$ such that the dimension of ${\mathrm{\Theta }}_H$ is smaller than $k$, the dimension of $\mathrm{\Theta }$.

Example

For the linear regression model $y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i,i=1,...,n$ with ${\epsilon }_i$ i.i.d. $N(0,{\sigma }^2)$, the parameter space has dimension three since it is composed of all the vectors $({\beta }_0,{\beta }_1,\sigma )\in \mathbb{ℝ}\times \mathbb{ℝ}\times {\mathbb{ℝ}}_{+}$.

In this case, the hypothesis $H:\sigma >2$ also has dimension three and can be tested following the Bayesian approach using posterior probabilities for essentially any kind of prior adopted on the parameter space. On the other hand, hypotheses such as $H:{\beta }_0+{\beta }_1=1$ have dimension two and therefore are sharp.

Calculation

Following the Bayesian paradigm, let $\pi (\cdot )$ be a probability prior distribution over $\mathrm{\Theta }$ and $L(\cdot \mid \mathbf{𝐱})$ the likelihood function derived from the observed data set $\mathbf{𝐱}$ and the statistical model ${ℱ}$. We denote the posterior distribution of $\theta$ on $\mathrm{\Theta }$ by

${\pi }_n(\theta )\propto \pi (\theta )\cdot L(\theta |\mathbf{𝐱})$.

To compute the e-value supporting some hypothesis $H$, it is necessary to specify a distribution on the parameter space called reference density, $r(\theta )$, that guarantees that the e-value is invariant to reparameterizations, even if the reference density is an improper density function. (See also Improper prior.)

With this density the relative surprise function is obtained as the ratio between the posterior density and the reference density, that is, $s_n(\theta )={\pi }_n(\theta \mid \mathbf{𝐱})/r(\theta )$. If the reference density is, for instance, proportional to any given constant, the surprise function will be, in practical terms, equivalent to the posterior distribution. It is possible to compute the e-value using other reference densities such as uninformative or diffuse distributions. (See also Non-informative prior.)

The following step is to maximize the relative surprise function on ${\mathrm{\Theta }}_H$, the subset of the parameter space that defines the hypothesis being tested, and compute the maximum value of the relative surprise function on this region:

${\theta }^{*}=\arg \underset{\theta \in {\mathrm{\Theta }}_H}{\max }\left\{\frac{{\pi }_n(\theta )}{r(\theta )}\right\},s_n^{*}=\underset{\theta \in {\mathrm{\Theta }}_H}{\max }\left\{\frac{{\pi }_n(\theta )}{r(\theta )}\right\}=\left\{\frac{{\pi }_n\left({\theta }^{*}\right)}{r\left({\theta }^{*}\right)}\right\}.$

Thus, given a sharp hypothesis $H:\theta \in {\mathrm{\Theta }}_H$, the highest relative surprise set or simply tangent set of the hypothesis given data $\mathbf{𝐱}\in \mathfrak{𝔛}$ is

${\mathrm{\Theta }}_n^{*}=\{\theta \in \mathrm{\Theta }:s_n(\theta )>s_n^{*}\}.$

The tangent set is the subset of the parameter space with points whose relative surprise, $s_n(\theta )$, is larger than the relative surprise of any point in ${\mathrm{\Theta }}_H$, being tangential to ${\mathrm{\Theta }}_H$ in this sense. The e-value favoring a sharp hypothesis ${\mathrm{\Theta }}_H$ is then defined as the posterior probability of the complementary set, regarding the parameter space, of the tangent set.

The Bayesian e-value ($ev$) supporting the sharp hypothesis ${\mathrm{\Theta }}_H$ is

$ev=1-\underset{{\mathrm{\Theta }}_n^{*}}{\int }{\pi }_n(\theta )d\theta .$

Hence, the e-value takes as evidence supporting the hypothesis considers the posterior probability of all points of the parameter space whose relative surprise is at most as large as its supremum over ${\mathrm{\Theta }}_H$. By this definition, a large value of $ev$ means that ${\mathrm{\Theta }}_H$ lies in a region of large posterior probability, implying that the data strongly support the hypothesis.

Examples

Testing the fairness of a coin

As an example, let us consider an experiment to test if a coin, when flipped into the air, is fair (it has equal probability of landing heads or tails). Denoting by $\theta$ the probability of landing heads, this means that $H:\theta =0.5$ is the hypothesis under analysis. Let $n$ be the number of times the coin was flipped and $\mathbf{𝐱}$ the number of times it landed heads.

The parameter space is the unit interval $\mathrm{\Theta }=[0,1]$ and, assuming the tosses are statistically independent and have always the same probability of landing heads, a suitable statistical model for $X$ is the binomial distribution. Adopting as prior and reference density the uniform distribution on $\mathrm{\Theta }$, the surprise function, $s_n(\theta )$ is proportional to the likelihood function

$s_n(\theta )\propto {\theta }^{\mathbf{𝐱}}(1-\theta {)}^{n-\mathbf{𝐱}}.$

If, for instance, $n=20$ and $\mathbf{𝐱}=14$, the tangent set is the interval displayed in the figure and the e-value is the area under $s_n(\theta )$ with values smaller than $s^{*}=s_n(0.5)$. In this case this area can be computed using the beta distribution function and it results in approximately 0.061.

Hardy-Weinberg principle

A sample of $n\in \mathbb{\mathbb{N}}$ individuals of the same species is randomly selected from their population. Let $x_1$ and $x_3$ represent the two homozygote sample counts of a given genetic locus, and $x_2=n-x_1-x_3$ the heterozygotic individuals in the sample, such that $\mathbf{𝐱}=(x_1,x_2,x_3)$. If $\mathbf{𝐱}$ is described by a trinomial model, the parameter space is the standard simplex:

$\mathrm{\Theta }=\{\theta =({\theta }_1,{\theta }_2,{\theta }_3)\in {\mathbb{ℝ}}^3:{\theta }_1+{\theta }_2+{\theta }_3=1,{\theta }_i\ge 0,i=1,2,3\}$

where ${\theta }_i$ is the population frequency of individuals with genotype $i=1,2,3$. The Hardy-Weinberg principle states that, under equilibrium, these genotype frequencies will obey the relationship ${\theta }_3={(1-\sqrt{{\theta }_1})}^2$. Hence, the hypothesis to test the principle is $H:\mathrm{\Theta }\in {\mathrm{\Theta }}_H$ being

${\mathrm{\Theta }}_H=\{\theta \in \mathrm{\Theta }:{\theta }_3={(1-\sqrt{{\theta }_1})}^2\}$

a subset of the parameter space with zero Lebesgue measure. As in the preceding example, we use as prior distribution over $\mathrm{\Theta }$ the uniform distribution on the standard simplex, in this case the Dirichlet distribution with its three parameters equal to one, and as reference density also the uniform distribution on the simplex. Hence the surprise function ($s_n(\theta )$) is proportional to the likelihood

$s_n(\theta )\propto {\theta }_1^{x_1}{\theta }_2^{x_2}{\theta }_3^{x_3}$

In this example there is a closed-form solution $\underset{\theta \in {\mathrm{\Theta }}_H}{\sup }{s}_n(\theta )$, and the computation of $ev$ is carried by Monte Carlo integration from independent vectors sampled from a Dirichlet distribution with parameters $x_1+1,x_2+1$, and $x_3+1$. Assuming, for instance, $n=20,x_1=5$, and $x_3=5$, the estimated e-value is 0.91. The problem of testing the Hardy-Weinberg equilibrium law using Bayes factors requires the specification of a mixed prior on ${\mathrm{\Theta }}_H$ and frequentist tests are the Chi-square goodness-of-fit test (with continuity correction) and Fisher's exact test.

Properties and limitations

The FBST e-value satisfies important properties for statistical inferential procedures such as, the likelihood principle, invariance under reparameterizations (of the parameter space), its computation does not require the use of asymptotic approximations and it may be conceived as a formal Bayesian test. Other properties and applications of the e-value are discussed in the literature.

The computation of the e-value, however, may present practical difficulties. The maximization of the surprise function, $s_n(\theta )$ on the restricted parameter space ${\mathrm{\Theta }}_H$ is a problem of constrained optimization which, in several applications, does not have a closed-form solution, requiring the use of numerical optimizers. Another difficulty is the integration of the posterior distribution on a subset of the parameter space, the tangent set ${\mathrm{\Theta }}_n^{*}$, which, even in simple cases, may require the use of numerical techniques (such as Markov chain Monte Carlo) that provide an accurate estimates (but not exact value) of $ev$. It is also possible to use approximation techniques based on Laplace approximations.

Another practical problem regards the search for a threshold value for $ev$, below which one may reject the hypothesis being tested. Since $ev$ is formally a statistic whose distribution can be derived from the adopted statistical model, one possibility is to use its sampling distribution of $ev$ to find this threshold value. Other technical problems with the procedure were recently examined in the literature.

References

This article "E-value" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:E-value. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.

---