View source for Pooled variance

{{Short description|Method for estimating variance of several different populations}}
{{More citations needed|date=July 2019}}
In [[statistics]], '''pooled variance''' (also known as '''combined variance''', '''composite variance''', or '''overall variance''', and written <math>\sigma^2</math>) is a method for [[estimation theory|estimating]] [[variance]] of several different populations when the mean of each population  may be different, but one may assume that the variance of each population is the same. The numerical estimate resulting from the use of this method is also called the pooled variance.

Under the assumption of equal population variances, the pooled sample variance provides a higher [[precision (statistics)|precision]] estimate of variance than the individual sample variances. This higher precision can lead to increased [[statistical power]] when used in [[statistical test]]s  that compare the populations, such as the [[t-test|''t''-test]].

The square root of a pooled variance estimator is known as a '''pooled standard deviation''' (also known as '''combined standard deviation''', '''composite standard deviation''', or '''overall standard deviation''').

== Motivation ==
In [[statistics]], many times, data are collected for a [[statistical independence|dependent variable]], ''y'', over a range of values for the [[statistical independence|independent variable]], ''x''. For example, the observation of fuel consumption might be studied as a function of engine speed while the engine load is held constant. If, in order to achieve a small [[variance]] in ''y'', numerous repeated tests are required at each value of ''x'', the expense of testing may become prohibitive. Reasonable estimates of variance can be determined by using the principle of '''pooled variance''' after repeating each [[Statistical hypothesis testing|test]] at a particular ''x'' only a few times.

==Definition and computation==

The pooled variance is an estimate of the fixed common variance <math>\sigma ^2</math> underlying various populations that have different means.

We are given a set of [[sample variance]]s <math>s^2_i</math>, where the populations are indexed <math>i = 1, \ldots, m</math>,

:<math>s^2_i</math> = <math>\frac{1}{n_i-1} \sum_{j=1}^{n_i} \left(y_{i, j} - \overline{y}_i \right)^2. </math>

Assuming uniform [[sample size]]s, <math>n_i=n</math>, then the pooled variance <math>s^2_p</math> can be computed by the [[arithmetic mean]]:

:<math>s_p^2=\frac{\sum_{i=1}^m s_i^2}{m} = \frac{s_1^2+s_2^2+\cdots+s_m^2}{m}.</math>

If the sample sizes are non-uniform, then the pooled variance <math>s^2_p</math> can be computed by the [[weighted average]], using as weights <math>w_i=n_i-1</math> the respective [[degrees of freedom (statistics)|degrees of freedom]] (see also: [[Bessel's correction]]):

:<math>s_p^2=\frac{\sum_{i=1}^m (n_i - 1)s_i^2}{\sum_{i=1}^m(n_i - 1)} = \frac{(n_1 - 1)s_1^2+(n_2 - 1)s_2^2+\cdots+(n_m - 1)s_m^2}{n_1+n_2+\cdots+n_m - m}.</math>
The distribution of <math>\frac{s_p^2(\sum_i n_i - m)}{\sigma^2}</math> is <math>\chi^2(\sum_i n_i - m)</math>.

'''Proof.''' When there is a single mean, the distribution of <math>(y_1 - \bar y, \dots, y_n - \bar y)</math> is a gaussian in <math>\Delta_{n-1}</math>, the <math>(n-1)</math>-dimensional simplex, with standard deviation <math>\sigma</math>. Where there are multiple means, the distribution of <math>(y_{1, 1} - \bar y_1, \dots, y_{1, n_1} - \bar y_1, \dots, y_{m, 1} - \bar y_m, \dots, y_{m, n_m} - \bar y_m)</math> is a gaussian in <math>\Delta_{n_1-1} \times \dots \times \Delta_{n_m-1}</math>.

=== Variants ===

The unbiased least squares estimate of <math>\sigma^2</math> (as presented above),
and the biased maximum likelihood estimate below:

:<math>s_p^2=\frac{\sum_{i=1}^N (n_i - 1)s_i^2}{\sum_{i=1}^N n_i },</math>

are used in different contexts.{{Citation needed|date=November 2010}} The former can give an unbiased <math>s_p^2</math> to estimate <math>\sigma^2</math> when the two groups share an equal population variance. The latter one can give a more [[Efficiency (statistics)|efficient]] <math>s_p^2</math> to estimate <math>\sigma^2</math>, although subject to bias. Note that the quantities <math>s_i^2</math> in the right hand sides of both equations are the unbiased estimates.

==Example==

Consider the following set of data for ''y'' obtained at various levels of the independent variable&nbsp;''x''.

{| border="1" cellspacing="0" cellpadding="5" align="center"
! ''x''
! ''y''
|-
| 1
| 31, 30, 29
|-
| 2
| 42, 41, 40, 39
|-
| 3
| 31, 28
|-
| 4
| 23, 22, 21, 19, 18
|-
| 5
| 21, 20, 19, 18,17
|}

The number of trials, mean, variance and standard deviation are presented in the next table.
{| border="1" cellspacing="0" cellpadding="5" align="center"
! ''x''
! ''n''
! ''y''<sub>mean</sub>
! ''s''<sub>''i''</sub><sup>2</sup>
! ''s''<sub>''i''</sub>
|-
| 1
| 3
| 30.0
| 1.0
| 1.0
|-
| 2
| 4
| 40.5
| 1.67
| 1.29
|-
| 3
| 2
| 29.5
| 4.5
| 2.12
|-
| 4
| 5
| 20.6
| 4.3
| 2.07
|-
| 5
| 5
| 19.0
| 2.5
| 1.58
|}

These statistics represent the variance and [[standard deviation]] for each subset of data at the various levels of ''x''. If we can assume that the same phenomena are generating [[random error]] at every level of ''x'', the above data can be “pooled” to express a single estimate of variance and standard deviation. In a sense, this suggests finding a [[mean]] variance or standard deviation among the five results above. This mean variance is calculated by weighting the individual values with the size of the subset for each level of ''x''. Thus, the pooled variance is defined by

: <math>s_p^2 = \frac{(n_1-1)s_1^2+(n_2-1)s_2^2 + \cdots + (n_k - 1)s_k^2}{(n_1 - 1) + (n_2 - 1) + \cdots +(n_k - 1)}</math>

where ''n''<sub>1</sub>, ''n''<sub>2</sub>, . . ., ''n''<sub>''k''</sub> are the sizes of the data subsets at each level of the variable ''x'', and ''s''<sub>1</sub><sup>2</sup>, ''s''<sub>2</sub><sup>2</sup>, . . ., ''s''<sub>''k''</sub><sup>2</sup> are their respective variances.

The pooled variance of the data shown above is therefore:

: <math>s_p^2 = 2.764 \, </math>

==Effect on precision==

Pooled variance is an estimate when there is a correlation between pooled data sets or the average of the data sets is not identical. Pooled variation is less precise the more non-zero the correlation or distant the averages between data sets.

The variation of data for non-overlapping data sets is:

:<math>
 \sigma_X^2 =\frac{ \sum_i \left[(N_{X_i} - 1) \sigma_{X_i}^2 + N_{X_i} \mu_{X_i}^2\right] - \left[\sum_i  N_{X_i} \right] \mu_X^2 }{\sum_i N_{X_i} - 1}
</math>
where the mean is defined as:
:<math>
    \mu_X = \frac{ \sum_i N_{X_i} \mu_{X_i} }{\sum_i N_{X_i} }
</math>

Given a biased maximum likelihood defined as:

:<math>s_p^2=\frac{\sum_{i=1}^k (n_i - 1)s_i^2}{\sum_{i=1}^k n_i },</math>

Then the error in the biased maximum likelihood estimate is:

:<math>\begin{align}
\text{Error} & = s_p^2 - \sigma_X^2 \\[6pt]
& =\frac{\sum_i (N_{X_i} - 1)s_i^2}{\sum_i N_{X_i} } - \frac{1}{\sum_i N_{X_i} - 1} \left( \sum_i \left[(N_{X_i} - 1) \sigma_{X_i}^2 + N_{X_i} \mu_{X_i}^2\right] - \left[\sum_i N_{X_i} \right]\mu_X^2 \right)
\end{align}</math>

Assuming ''N'' is large such that:

:<math>
\sum_i N_{X_i} \approx \sum_i N_{X_i} - 1
</math>

Then the error in the estimate reduces to:

:<math>\begin{align}
E & =- \frac{\left( \sum_i \left[N_{X_i} \mu_{X_i}^2\right] - \left[\sum_i N_{X_i} \right]\mu_X^2 \right)}{\sum_i N_{X_i}}\\[3pt]
& =\mu_X^2 - \frac{\sum_i \left[N_{X_i} \mu_{X_i}^2\right] }{\sum_i N_{X_i}}
\end{align}</math>
Or alternatively:
:<math>\begin{align}
E & =\left[ \frac{\sum_i N_{X_i} \mu_{X_i}}{\sum_i N_{X_i}} \right]^2 - \frac{\sum_i \left[N_{X_i} \mu_{X_i}^2\right] }{\sum_i N_{X_i}}\\[3pt]
& =\frac{\left[\sum_i  N_{X_i} \mu_{X_i} \right]^2 
 -  \sum_i N_{X_i} \sum_i \left[N_{X_i} \mu_{X_i}^2\right]
}{\left[\sum_i N_{X_i} \right]^2}
\end{align}</math>

==Aggregation of standard deviation data==
{{Unreferenced section|date=June 2011}}
{{cleanup merge|Standard deviation|date=August 2016}}
Rather than estimating pooled standard deviation, the following is the way to exactly aggregate standard deviation when more statistical information is available.

===Population-based statistics===
The populations of sets, which may overlap, can be calculated simply as follows:
:<math>\begin{align}
                                       &&N_{X \cup Y} &= N_X + N_Y - N_{X \cap Y}\\
\end{align}</math>
The populations of sets, which do not overlap, can be calculated simply as follows:
:<math>\begin{align}
 X \cap Y = \varnothing &\Rightarrow &N_{X \cap Y} &= 0\\
                           &\Rightarrow &N_{X \cup Y} &= N_X + N_Y
\end{align}</math>

Standard deviations of non-overlapping ({{nowrap|''X'' ∩ ''Y'' {{=}} ∅}}) sub-populations can be aggregated as follows if the size (actual or relative to one another) and means of each are known:
: <math>\begin{align}
   \mu_{X \cup Y}   &= \frac{ N_X \mu_X + N_Y \mu_Y }{N_X + N_Y} \\[3pt]
   \sigma_{X\cup Y} &= \sqrt{ \frac{N_X \sigma_X^2 + N_Y \sigma_Y^2}{N_X + N_Y} + \frac{N_X N_Y}{(N_X+N_Y)^2}(\mu_X - \mu_Y)^2 }
  \end{align}</math>

For example, suppose it is known that the average American man has a mean height of 70&nbsp;inches with a standard deviation of three inches and that the average American woman has a mean height of 65&nbsp;inches with a standard deviation of two inches. Also assume that the number of men, ''N'', is equal to the number of women. Then the mean and standard deviation of heights of American adults could be calculated as
: <math>\begin{align}
    \mu    &= \frac{N\cdot70 + N\cdot65}{N + N} = \frac{70+65}{2} = 67.5 \\[3pt]
    \sigma &= \sqrt{ \frac{3^2 + 2^2}{2} + \frac{(70-65)^2}{2^2} } = \sqrt{12.75} \approx 3.57
  \end{align}</math>

For the more general case of ''M'' non-overlapping populations, ''X''<sub>1</sub> through ''X''<sub>''M''</sub>, and the aggregate population <math display=inline> X \,=\, \bigcup_i X_i</math>,
: <math>\begin{align}
    \mu_X    &= \frac{ \sum_i N_{X_i}\mu_{X_i} }{ \sum_i N_{X_i} } \\[3pt]
    \sigma_X &= \sqrt{ \frac{ \sum_i N_{X_i}\sigma_{X_i}^2 }{ \sum_i N_{X_i} } + \frac{ \sum_{i<j} N_{X_i}N_{X_j} (\mu_{X_i}-\mu_{X_j})^2 }{\big(\sum_i N_{X_i}\big)^2} }
  \end{align}</math>,
where
: <math>
    X_i \cap X_j = \varnothing, \quad \forall\ i<j.
  </math>

If the size (actual or relative to one another), mean, and standard deviation of two overlapping populations are known for the populations as well as their intersection, then the standard deviation of the overall population can still be calculated as follows:
:<math>\begin{align}
    \mu_{X \cup Y} &= \frac{1}{N_{X \cup Y}}\left(N_X\mu_X + N_Y\mu_Y - N_{X \cap Y}\mu_{X \cap Y}\right)\\[3pt]
 \sigma_{X \cup Y} &= \sqrt{\frac{1}{N_{X \cup Y}}\left(N_X[\sigma_X^2 + \mu _X^2] + N_Y[\sigma_Y^2 + \mu _Y^2] - N_{X \cap Y}[\sigma_{X \cap Y}^2 + \mu _{X \cap Y}^2]\right) - \mu_{X\cup Y}^2}
\end{align}</math>

If two or more sets of data are being added together datapoint by datapoint, the standard deviation of the result can be calculated if the standard deviation of each data set and the [[covariance]] between each pair of data sets is known:
:<math>\sigma_X = \sqrt{\sum_i{\sigma_{X_i}^2} + 2\sum_{i,j}\operatorname{cov}(X_i,X_j)}</math>

For the special case where no correlation exists between any pair of data sets, then the relation reduces to the root sum of squares:
:<math>\begin{align}
             &\operatorname{cov}(X_i, X_j) = 0,\quad \forall i<j\\
 \Rightarrow &\;\sigma_X = \sqrt{\sum_i {\sigma_{X_i}^2}}.
\end{align}</math>

===Sample-based statistics===
Standard deviations of non-overlapping ({{nowrap|''X'' ∩ ''Y'' {{=}} ∅}}) sub-samples can be aggregated as follows if the actual size and means of each are known:
:<math>\begin{align}
    \mu_{X \cup Y} &= \frac{1}{N_{X \cup Y}}\left(N_X\mu_X + N_Y\mu_Y\right)\\[3pt]
 \sigma_{X \cup Y} &= \sqrt{\frac{1}{N_{X \cup Y} - 1}\left([N_X - 1]\sigma_X^2 + N_X\mu_X^2 + [N_Y - 1]\sigma_Y^2 + N_Y\mu _Y^2 - [N_X + N_Y]\mu_{X \cup Y}^2\right) }
\end{align}</math>

For the more general case of ''M'' non-overlapping data sets, ''X''<sub>1</sub> through ''X''<sub>''M''</sub>, and the aggregate data set <math display=inline> X \,=\, \bigcup_i X_i</math>,
:<math>\begin{align}
    \mu_X &= \frac{1}{\sum_i { N_{X_i}}} \left(\sum_i { N_{X_i} \mu_{X_i}}\right)\\[3pt]
 \sigma_X &= \sqrt{\frac{1}{\sum_i {N_{X_i} - 1}} \left( \sum_i { \left[(N_{X_i} - 1) \sigma_{X_i}^2 + N_{X_i} \mu_{X_i}^2\right] } - \left[\sum_i {N_{X_i}}\right]\mu_X^2 \right) }
\end{align}</math>

where
:<math>X_i \cap X_j = \varnothing,\quad \forall i<j.</math>

If the size, mean, and standard deviation of two overlapping samples are known for the samples as well as their intersection, then the standard deviation of the aggregated sample can still be calculated. In general,
:<math>\begin{align}
    \mu_{X \cup Y} &= \frac{1}{N_{X \cup Y}}\left(N_X\mu_X + N_Y\mu_Y - N_{X\cap Y}\mu_{X\cap Y}\right)\\[3pt]
 \sigma_{X \cup Y} &= \sqrt{  \frac{[N_X - 1]\sigma_X^2 + N_X\mu_X^2 + [N_Y - 1]\sigma_Y^2 + N_Y\mu _Y^2 - [N_{X \cap Y}-1]\sigma_{X \cap Y}^2 - N_{X \cap Y}\mu_{X \cap Y}^2 - [N_X + N_Y - N_{X \cap Y}]\mu_{X \cup Y}^2}{N_{X \cup Y} - 1} }
\end{align}</math>

==See also==
* [[Chi-squared distribution#Asymptotic properties]]
* Used for calculating [[Effect size#Cohen.27s_d|Cohen's ''d'' (effect size)]]
* [[Variance#Distribution_of_the_sample_variance|Distribution of the sample variance]]
* [[Pooled covariance matrix]]
* [[Pooled degree of freedom]]
* [[Pooled mean]]

==References==
*{{cite journal |author=Killeen PR |title=An alternative to null-hypothesis significance tests |journal=Psychol Sci |volume=16 |issue=5 |pages=345–53 |date=May 2005 |pmid=15869691 |pmc=1473027 |doi=10.1111/j.0956-7976.2005.01538.x }}
==External links==
* [http://goldbook.iupac.org/P04758.html IUPAC Gold Book – pooled standard deviation]
* [https://web.archive.org/web/20020624174749/http://www.isixsigma.com/dictionary/Pooled_Standard_Deviation-295.htm]
* [http://web.psych.utoronto.ca/~psy379/Stats%20PPT.pdf &ndash; also referring to Cohen's ''d'' (on page 6)]


{{DEFAULTSORT:Pooled Variance}}
[[Category:Analysis of variance]]
[[Category:Statistical deviation and dispersion]]