Hoeffding's inequality

Template:Short description In probability theory, Hoeffding's inequality provides an upper bound on the probability that the sum of bounded independent random variables deviates from its expected value by more than a certain amount. Hoeffding's inequality was proven by Wassily Hoeffding in 1963.

Hoeffding's inequality is a special case of the Azuma–Hoeffding inequality and McDiarmid's inequality. It is similar to the Chernoff bound, but tends to be less sharp, in particular when the variance of the random variables is small. It is similar to, but incomparable with, one of Bernstein's inequalities.

Statement

Let Template:Math be independent random variables such that $a_i\le X_i\le b_i$ almost surely. Consider the sum of these random variables,

$S_n=X_1+\cdots +X_n.$

Then Hoeffding's theorem states that, for all Template:Math,

$\begin{array}{rl}\mathrm{P}(S_n-\mathrm{E}\left[S_n\right]\ge t)& \le \exp (-\frac{2t^2}{\sum _{i=1}^n(b_i-a_i{)}^2})\\ \mathrm{P}(|S_n-\mathrm{E}\left[S_n\right]|\ge t)& \le 2\exp (-\frac{2t^2}{\sum _{i=1}^n(b_i-a_i{)}^2})\end{array}$

Here Template:Math is the expected value of Template:Mvar.

Note that the inequalities also hold when the Template:Mvar have been obtained using sampling without replacement; in this case the random variables are not independent anymore. A proof of this statement can be found in Hoeffding's paper. For slightly better bounds in the case of sampling without replacement, see for instance the paper by Template:Harvtxt.

Generalization

Let

$Y_1,\dots ,Y_n$

be independent observations such that

$\mathrm{E}(Y_i)=0$

and

$a_i\le Y_i\le b_i$

. Let

$ϵ>0$

. Then, for any

$t>0$

,

$$P(\sum _{i=1}^n{Y}_i\ge ϵ)\le \exp (-tϵ+\sum _{i=1}^n{t}^2(b_i-a_i{)}^2/8)$$

Special Case: Bernoulli RVs

Suppose $a_i=0$ and $b_i=1$ for all i. This can occur when X_i are independent Bernoulli random variables, though they need not be identically distributed. Then we get the inequality

$\begin{array}{rl}\mathrm{P}(S_n-\mathrm{E}\left[S_n\right]\ge t)& \le \exp (-2t^2/n)\\ \mathrm{P}(|S_n-\mathrm{E}\left[S_n\right]|\ge t)& \le 2\exp (-2t^2/n)\end{array}$

or equivalently,

$$\begin{array}{rl}\mathrm{P}((S_n-\mathrm{E}\left[S_n\right])/n\ge t)& \le \exp (-2n{t}^2)\\ \mathrm{P}(|(S_n-\mathrm{E}\left[S_n\right])/n|\ge t)& \le 2\exp (-2n{t}^2)\end{array}$$

for all $t\ge 0$. This is a version of the additive Chernoff bound which is more general, since it allows for random variables that take values between zero and one, but also weaker, since the Chernoff bound gives a better tail bound when the random variables have small variance.

General case of bounded from above random variables

Hoeffding's inequality can be extended to the case of bounded from above random variables.

Let Template:Math be independent random variables such that $\mathrm{E}{X}_i=0$ and $X_i\le b_i$ almost surely. Denote by

$\begin{array}{r}{C}_i^2=\{\begin{array}{ll}\mathrm{E}{X}_i^2,& \mathrm{i}\mathrm{f}\mathrm{E}{X}_i^2\ge b_i^2,\\ \frac{1}{4}{(b_i+\frac{\mathrm{E}{X}_i^2}{b_i})}^2,& \text{<mrow data-mjx-texclass="ORD">otherwise</mrow>}.\end{array}\end{array}$

Hoeffding's inequality for bounded from above random variables states that for all $t\ge 0$,

$\mathrm{P}(\left|\sum _{i=1}^n{X}_i\right|\ge t)\le 2\exp (-\frac{t^2}{2\sum _{i=1}^n{C}_i^2}).$

In particular, if $\mathrm{E}{X}_i^2\ge b_i^2$ for all $i$, then for all $t\ge 0$,

$\mathrm{P}(\left|\sum _{i=1}^n{X}_i\right|\ge t)\le 2\exp (-\frac{t^2}{2\sum _{i=1}^n\mathrm{E}{X}_i^2}).$

General case of sub-Gaussian random variables

The proof of Hoeffding's inequality can be generalized to any sub-Gaussian distribution. Recall that a random variable Template:Math is called sub-Gaussian, if

$\mathrm{P}(|X|\ge t)\le 2e^{-c{t}^2},$

for some $c>0$. For any bounded variable Template:Math, $\mathrm{P}(|X|\ge t)=0\le 2e^{-c{t}^2}$ for $t>T$ for some sufficiently large Template:Math. Then $2e^{-c{T}^2}\le 2e^{-c{t}^2}$ for all $t\le T$ so taking $c=\log (2)/T^2$ yields

$\mathrm{P}(|X|\ge t)\le 1\le 2e^{-c{T}^2}\le 2e^{-c{t}^2},$

for $t\le T$. So every bounded variable is sub-Gaussian.

For a random variable Template:Math, the following norm is finite if and only if Template:Math is sub-Gaussian:

$\Vert X{\Vert }_{{\psi }_2}:=\inf \{c\ge 0:\mathrm{E}\left(e^{X^2/c^2}\right)\le 2\}.$

Then let Template:Math be independent sub-Gaussian random variables, the general version of the Hoeffding's inequality states that:

$\mathrm{P}(|\sum _{i=1}^n(X_i-E[X_i])|\ge t)\le 2\exp (-\frac{c{t}^2}{\sum _{i=1}^n\Vert X_i{\Vert }_{{\psi }_2}^2}),$

where c > 0 is an absolute constant.

Equivalently, we can define sub-Gaussian distributions by variance proxy, defined as follows. If there exists some $s^2$ such that $\mathrm{E}[e^{(X-\mathrm{E}[X])t}]\le e^{\frac{s^2{t}^2}{2}}$ for all $t$, then $s^2$ is called a variance proxy, and the smallest such $s^2$ is called the optimal variance proxy, and denoted by $\Vert X{\Vert }_{\mathrm{v}\mathrm{p}}^2$. In this form, Hoeffding's inequality states$$\mathrm{P}(|\sum _{i=1}^n(X_i-E[X_i])|\ge t)\le 2\exp (-\frac{t^2}{2\sum _{i=1}^n\Vert X_i{\Vert }_{\mathrm{v}\mathrm{p}}^2}),$$

Proof

We quote:

Template:Math theoremThe proof of Hoeffding's inequality then follows similarly to concentration inequalities like Chernoff bounds.Template:Math proof

This proof easily generalizes to the case for sub-Gaussian distributions with variance proxy.

Usage

Confidence intervals

Hoeffding's inequality can be used to derive confidence intervals. We consider a coin that shows heads with probability Template:Mvar and tails with probability Template:Math. We toss the coin Template:Mvar times, generating Template:Mvar samples $X_1,\dots ,X_n$ (which are i.i.d Bernoulli random variables). The expected number of times the coin comes up heads is Template:Math. Furthermore, the probability that the coin comes up heads at least Template:Mvar times can be exactly quantified by the following expression:

$\mathrm{P}(H(n)\ge k)=\sum _{i=k}^n(\genfrac{}{}{0ex}{}{n}{i})p^i(1-p{)}^{n-i},$

where Template:Math is the number of heads in Template:Mvar coin tosses.

When Template:Math for some Template:Math, Hoeffding's inequality bounds this probability by a term that is exponentially small in Template:Math:

$\mathrm{P}(H(n)-pn>\epsilon n)\le \exp (-2{\epsilon }^{2}n).$

Since this bound holds on both sides of the mean, Hoeffding's inequality implies that the number of heads that we see is concentrated around its mean, with exponentially small tail.

$\mathrm{P}(|H(n)-pn|>\epsilon n)\le 2\exp (-2{\epsilon }^{2}n).$

Thinking of $\bar{X}=\frac{1}{n}H(n)$ as the "observed" mean, this probability can be interpreted as the level of significance $\alpha$ (probability of making an error) for a confidence interval around $p$ of size 2Template:Mvar:

$\alpha =\mathrm{P}(\bar{X}\notin [p-\epsilon ,p+\epsilon ])\le 2e^{-2{\epsilon }^{2}n}$

Finding Template:Mvar for opposite inequality sign in the above, i.e. Template:Mvar that violates inequality but not equality above, gives us:

$n\ge \frac{\log (2/\alpha )}{2{\epsilon }^2}$

Therefore, we require at least $\frac{\log (2/\alpha )}{2{\epsilon }^2}$ samples to acquire a $(1-\alpha )$-confidence interval $p\pm \epsilon$.

Hence, the cost of acquiring the confidence interval is sublinear in terms of confidence level and quadratic in terms of precision. Note that there are more efficient methods of estimating a confidence interval.

See also

• Concentration inequality – a summary of tail-bounds on random variables.
• Hoeffding's lemma
• Bernstein inequalities (probability theory)

References

• [1].

 Template:Refend