imported>Bazsola: /* Problem */

2025-06-14T01:47:10Z

Problem

New page

{{Short description|Asymptotic variances under heteroskedasticity}}
The topic of '''heteroskedasticity-consistent''' ('''HC''') '''standard errors''' arises in [[statistics]] and [[econometrics]] in the context of [[linear regression]] and [[time series analysis]]. These are also known as '''heteroskedasticity-robust standard errors''' (or simply '''robust standard errors'''), '''Eicker–Huber–White standard errors''' (also '''Huber–White standard errors''' or '''White standard errors'''),<ref>{{cite web|last1=Kleiber |first1=C. |last2=Zeileis |first2=A. |year=2006 |url=http://www.r-project.org/useR-2006/Slides/Kleiber+Zeileis.pdf |title=Applied Econometrics with R |work=UseR-2006 conference |archive-url=https://web.archive.org/web/20070422030316/http://www.r-project.org/useR-2006/Slides/Kleiber%2BZeileis.pdf |archive-date=April 22, 2007 |url-status=dead }}</ref> to recognize the contributions of [[Friedhelm Eicker]],<ref>{{Cite book |last=Eicker |first=Friedhelm |chapter=Limit Theorems for Regression with Unequal and Dependent Errors |title=Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability |year=1967 |volume=5 |issue=1 |pages=59–82 |chapter-url=http://projecteuclid.org/euclid.bsmsp/1200512981 |mr=0214223 |zbl=0217.51201 }}</ref> [[Peter J. Huber]],<ref>{{Cite book | last=Huber| first=Peter J.| chapter=The behavior of maximum likelihood estimates under nonstandard conditions| title=Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability| year=1967| volume=5| issue=1| pages=221–233| chapter-url=http://projecteuclid.org/euclid.bsmsp/1200512988| mr = 0216620| zbl=0212.21504}}</ref> and [[Halbert White]].<ref>{{Cite journal |last=White |first=Halbert |title=A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity |journal=[[Econometrica]] |volume=48 |pages=817–838 |year=1980 |doi=10.2307/1912934 |issue=4 |mr=575027 |jstor=1912934 |citeseerx=10.1.1.11.7646 }}</ref>

In regression and time-series modelling, basic forms of models make use of the assumption that the errors or disturbances ''u''<sub>''i''</sub> have the same variance across all observation points. When this is not the case, the errors are said to be heteroskedastic, or to have [[heteroskedasticity]], and this behaviour will be reflected in the residuals <math display="inline">\widehat{u}_i </math> estimated from a fitted model. Heteroskedasticity-consistent standard errors are used to allow the fitting of a model that does contain heteroskedastic residuals. The first such approach was proposed by Huber (1967), and further improved procedures have been produced since for cross-sectional data, [[time-series]] data and [[GARCH| GARCH estimation]].

Heteroskedasticity-consistent standard errors that differ from classical standard errors may indicate model misspecification. Substituting heteroskedasticity-consistent standard errors does not resolve this misspecification, which may lead to bias in the coefficients. In most situations, the problem should be found and fixed.<ref>{{Cite journal|last1=King|first1=Gary|last2=Roberts|first2=Margaret E.|date=2015|title=How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It|url=https://www.cambridge.org/core/product/identifier/S1047198700011670/type/journal_article|journal=Political Analysis|language=en|volume=23|issue=2|pages=159–179|doi=10.1093/pan/mpu015|issn=1047-1987}}</ref> Other types of standard error adjustments, such as [[clustered standard errors]] or [[Newey–West estimator|HAC standard errors]], may be considered as extensions to HC standard errors.

== History ==
Heteroskedasticity-consistent standard errors are introduced by [[Friedhelm Eicker]],<ref>{{Cite journal|title=Asymptotic Normality and Consistency of the Least Squares Estimators for Families of Linear Regressions|year=1963|doi=10.1214/aoms/1177704156|url=https://projecteuclid.org/euclid.aoms/1177704156|last1=Eicker|first1=F.|journal=The Annals of Mathematical Statistics|volume=34|issue=2|pages=447–456|doi-access=free}}</ref><ref>{{Cite journal|title=Limit theorems for regressions with unequal and dependent errors|journal=Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics|date=January 1967|volume=5|issue=1|pages=59–83|url=https://projecteuclid.org/euclid.bsmsp/1200512981|last1=Eicker|first1=Friedhelm}}</ref> and popularized in econometrics by [[Halbert White]].

==Problem==
Consider the linear regression model for the scalar <math>y</math>.

: <math>
y = \mathbf{x}^{\top} \boldsymbol{\beta} + \varepsilon, \,
</math>

where <math>\mathbf{x}</math> is a ''k'' × 1 column vector of explanatory variables (features), <math>\boldsymbol{\beta}</math> is a ''k'' × 1 column vector of parameters to be estimated, and <math>\varepsilon</math> is the [[Errors and residuals|residual error]].

The [[ordinary least squares]] (OLS) estimator is

: <math>
\widehat \boldsymbol{\beta}_\mathrm{OLS} = (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y}. \,
</math>

where <math>\mathbf{y}</math> is a vector of observations <math>y_i</math>, and <math>\mathbf{X}</math> denotes the matrix of stacked <math>\mathbf{x}_i</math> values observed in the data.

If the [[errors and residuals in statistics|sample errors]] have equal variance <math>\sigma^2</math> and are [[uncorrelated]], then the least-squares estimate of <math>\boldsymbol{\beta}</math> is [[BLUE]] (best linear unbiased estimator), and its variance is estimated with

: <math>\hat{\mathbb{V}}\left[\widehat\boldsymbol\beta_\mathrm{OLS}\right] = s^2 (\mathbf{X}^{\top}\mathbf{X})^{-1}, \quad s^2 = \frac{\sum_{i=0}^n \widehat \varepsilon_i^2}{n-k} </math>

where <math>\widehat \varepsilon_i = y_i - \mathbf{x}_i^{\top} \widehat \boldsymbol{\beta}_\mathrm{OLS}</math> are the regression residuals.

When the error terms do not have constant variance (i.e., the assumption of <math> \mathbb{E}[\varepsilon\varepsilon^{\top}] = \sigma^2 \mathbf{I}_n</math> is untrue), the OLS estimator loses its desirable properties. The formula for variance now cannot be simplified:

: <math> \mathbb{V}\left[\widehat\boldsymbol\beta_\mathrm{OLS}\right] = \mathbb{V}\big[ (\mathbf{X}^{\top}\mathbf{X})^{-1} \mathbf{X}^{\top}\mathbf{y} \big] = (\mathbf{X}^{\top}\mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{\Sigma} \mathbf{X} (\mathbf{X}^{\top}\mathbf{X})^{-1}</math>

where <math> \mathbf{\Sigma} = \mathbb{V}[\varepsilon].</math>

While the OLS point estimator remains unbiased, it is not "best" in the sense of having minimum mean square error, and the OLS variance estimator <math>\hat{\mathbb{V}} \left[ \widehat \boldsymbol{\beta}_\mathrm{OLS} \right]</math> does not provide a consistent estimate of the variance of the OLS estimates.

For any non-linear model (for instance [[logit]] and [[probit]] models), however, heteroskedasticity has more severe consequences: the [[maximum likelihood estimation|maximum likelihood estimates]] of the parameters will be biased (in an unknown direction), as well as inconsistent (unless the likelihood function is modified to correctly take into account the precise form of heteroskedasticity).<ref>{{cite web |first=Dave |last=Giles |title=Robust Standard Errors for Nonlinear Models |work=Econometrics Beat |date=May 8, 2013 |url=http://davegiles.blogspot.com/2013/05/robust-standard-errors-for-nonlinear.html }}</ref><ref>{{cite journal |first=Michael |last=Guggisberg |title=Misspecified Discrete Choice Models and Huber-White Standard Errors |journal=[[Journal of Econometric Methods]] |year=2019 |volume=8 |issue=1 |doi=10.1515/jem-2016-0002 }}</ref> As pointed out by [[William Greene (economist)|Greene]], “simply computing a robust covariance matrix for an otherwise inconsistent estimator does not give it redemption.”<ref>{{cite book |last=Greene |first=William H. |author-link=William Greene (economist) |title=Econometric Analysis |edition=Seventh |location=Boston |publisher=Pearson Education |year=2012 |isbn=978-0-273-75356-8 |pages=692–693 }}</ref>

==Solution==

If the regression errors <math>\varepsilon_i</math> are independent, but have distinct variances <math>\sigma^2_i</math>, then <math>\mathbf{\Sigma} = \operatorname{diag}(\sigma_1^2, \ldots, \sigma_n^2)</math> which can be estimated with <math>\widehat\sigma_i^2 = \widehat \varepsilon_i^2</math>. This provides White's (1980) estimator, often referred to as ''HCE'' (heteroskedasticity-consistent estimator):

: <math>
\begin{align}
\hat{\mathbb{V}}_\text{HCE} \big[ \widehat \boldsymbol{\beta}_\text{OLS} \big] &= \frac{1}{n} \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \bigg)^{-1} \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^\top \widehat{\varepsilon}_i^2 \bigg) \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \bigg)^{-1} \\
&= ( \mathbf{X}^{\top} \mathbf{X} )^{-1} ( \mathbf{X}^{\top} \operatorname{diag}(\widehat \varepsilon_1^2, \ldots, \widehat \varepsilon_n^2) \mathbf{X} ) ( \mathbf{X}^{\top} \mathbf{X})^{-1},
\end{align}
</math>

where as above <math>\mathbf{X}</math> denotes the matrix of stacked <math>\mathbf{x}_i^{\top}</math> values from the data. The estimator can be derived in terms of the [[generalized method of moments]] (GMM).

Also often discussed in the literature (including White's paper) is the covariance matrix <math>\widehat\mathbf{\Omega}_n</math> of the <math>\sqrt{n}</math>-consistent limiting distribution:

: <math>
\sqrt{n}(\widehat \boldsymbol{\beta}_n - \boldsymbol{\beta}) \, \xrightarrow{d} \, \mathcal{N}(\mathbf{0}, \mathbf{\Omega}),
</math>

where

: <math>
\mathbf{\Omega} = \mathbb{E}[\mathbf{X} \mathbf{X}^{\top}]^{-1} \mathbb{V}[\mathbf{X} \boldsymbol{\varepsilon}]\operatorname \mathbb{E}[\mathbf{X} \mathbf{X}^{\top}]^{-1},
</math>

and

: <math>
\begin{align}
\widehat\mathbf{\Omega}_n &= \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \bigg)^{-1} \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \widehat \varepsilon_i^2 \bigg) \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \bigg)^{-1} \\
&= n ( \mathbf{X}^{\top} \mathbf{X} )^{-1} ( \mathbf{X}^{\top} \operatorname{diag}(\widehat \varepsilon_1^2, \ldots, \widehat \varepsilon_n^2) \mathbf{X} ) ( \mathbf{X}^{\top} \mathbf{X})^{-1}
\end{align}
</math>

Thus,

:<math>
\widehat \mathbf{\Omega}_n = n \cdot \hat{\mathbb{V}}_\text{HCE}[\widehat \boldsymbol{\beta}_\text{OLS}]
</math>

and

:<math>
\widehat \mathbb{V}[\mathbf{X} \boldsymbol{\varepsilon}] = \frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \widehat \varepsilon_i^2 = \frac{1}{n} \mathbf{X}^{\top} \operatorname{diag}(\widehat \varepsilon_1^2, \ldots, \widehat \varepsilon_n^2) \mathbf{X}.
</math>

Precisely which covariance matrix is of concern is a matter of context.

Alternative estimators have been proposed in MacKinnon & White (1985) that correct for unequal variances of regression residuals due to different [[Leverage (statistics)|leverage]].<ref>{{Cite journal |last1=MacKinnon |first1=James G. |author-link=James G. MacKinnon |last2=White |first2=Halbert |author2-link=Halbert White |title=Some Heteroskedastic-Consistent Covariance Matrix Estimators with Improved Finite Sample Properties |journal=[[Journal of Econometrics]] |volume=29 |issue=3 |pages=305–325 |year=1985 |doi=10.1016/0304-4076(85)90158-7 |hdl=10419/189084 |hdl-access=free }}</ref> Unlike the asymptotic White's estimator, their estimators are unbiased when the data are homoscedastic.

Of the four widely available different options, often denoted as HC0-HC3, the HC3 specification appears to work best, with tests relying on the HC3 estimator featuring better power and closer proximity to the targeted [[Statistical hypothesis testing#Definition of terms|size]], especially in small samples. The larger the sample, the smaller the difference between the different estimators.<ref>{{Cite journal |last=Long |first=J. Scott |last2=Ervin |first2=Laurie H. |date=2000 |title=Using Heteroscedasticity Consistent Standard Errors in the Linear Regression Model |url=https://www.jstor.org/stable/2685594 |journal=The American Statistician |volume=54 |issue=3 |pages=217–224 |doi=10.2307/2685594 |issn=0003-1305|url-access=subscription }}</ref>

An alternative to explicitly modelling the heteroskedasticity is using a [[Resampling (statistics)|resampling method]] such as the [[Bootstrapping (statistics)#Wild bootstrap|wild bootstrap]]. Given that the [[Bootstrapping (statistics)#Methods for bootstrap confidence intervals|studentized bootstrap]], which standardizes the resampled statistic by its standard error, yields an asymptotic refinement,<ref>{{Cite book |last=C. |first=Davison, Anthony |url=http://worldcat.org/oclc/740960962 |title=Bootstrap methods and their application |date=2010 |publisher=Cambridge Univ. Press |isbn=978-0-521-57391-7 |oclc=740960962}}</ref> heteroskedasticity-robust standard errors remain nevertheless useful.

Instead of accounting for the heteroskedastic errors, most linear models can be transformed to feature homoskedastic error terms (unless the error term is heteroskedastic by construction, e.g. in a [[linear probability model]]). One way to do this is using [[weighted least squares]], which also features improved efficiency properties.

==See also==
{{Div col|colwidth=20em}}
*[[Delta method]]
*[[Generalized least squares]]
*[[Generalized estimating equation]]s
*[[Weighted least squares]], an alternative formulation
*[[White test]] — a test for whether heteroskedasticity is present.
*[[Newey–West estimator]]
*[[Quasi-maximum likelihood estimate]]
{{Div col end}}

==Software==
* [[EViews]]: EViews version 8 offers three different methods for robust least squares: M-estimation (Huber, 1973), S-estimation (Rousseeuw and Yohai, 1984), and MM-estimation (Yohai 1987).<ref>{{Cite web|url=http://www.eviews.com/EViews8/ev8ecrobust_n.html|title=EViews 8 Robust Regression}}</ref>
* [[Julia (programming language)|Julia]]: the <code>CovarianceMatrices</code> package offers several methods for heteroskedastic robust variance covariance matrices.<ref>[https://github.com/gragusa/CovarianceMatrices.jl CovarianceMatrices: Robust Covariance Matrix Estimators]</ref>
* [[MATLAB]]: See the <code>hac</code> function in the Econometrics toolbox.<ref>{{cite web |title=Heteroskedasticity and autocorrelation consistent covariance estimators |work=Econometrics Toolbox |url=https://www.mathworks.com/help/econ/hac.html}}</ref>
* [[Python (programming language)|Python]]: The Statsmodel package offers various robust standard error estimates, see [http://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.RegressionResults.html statsmodels.regression.linear_model.RegressionResults] for further descriptions
* [[R (programming language)|R]]: the <code>vcovHC()</code> command from the {{mono|sandwich}} package.<ref>[https://cran.r-project.org/web/packages/sandwich/index.html sandwich: Robust Covariance Matrix Estimators]</ref><ref>{{cite book |first1=Christian |last1=Kleiber |first2=Achim |last2=Zeileis |title=Applied Econometrics with R |location=New York |publisher=Springer |year=2008 |isbn=978-0-387-77316-2 |pages=106–110 |url=https://books.google.com/books?id=86rWI7WzFScC&pg=PA106 }}</ref>
* [[RATS (statistical package)|RATS]]: {{mono|robusterrors}} option is available in many of the regression and optimization commands ({{mono|linreg}}, {{mono|nlls}}, etc.).
* [[Stata]]: <code>robust</code> option applicable in many pseudo-likelihood based procedures.<ref>See online help for [https://www.stata.com/manuals15/p_robust.pdf <code>_robust</code>] option and [https://www.stata.com/manuals15/rregress.pdf <code>regress</code>] command.</ref>
* [[Gretl]]: the option <code>--robust</code> to several estimation commands (such as <code>ols</code>) in the context of a cross-sectional dataset produces robust standard errors.<ref>{{cite web |title=Robust covariance matrix estimation |work=Gretl User's Guide, chapter 22 |url=http://gretl.sourceforge.net/gretl-help/gretl-guide.pdf }}</ref>

==References==
{{Reflist}}

==Further reading==
* {{cite journal |first=David A. |last=Freedman |author-link=David A. Freedman |title=On The So-Called 'Huber Sandwich Estimator' and 'Robust Standard Errors' |journal=The American Statistician |volume=60 |year=2006 |issue=4 |pages=299–302 |doi=10.1198/000313006X152207 |s2cid=6222876 }}
* {{Cite book |first=James W. |last=Hardin |chapter=The Sandwich Estimate of Variance |pages=45–74 |title=Maximum Likelihood Estimation of Misspecified Models: Twenty Years Later |editor-first=Thomas B. |editor-last=Fomby |editor2-first=R. Carter |editor2-last=Hill |location=Amsterdam |publisher=Elsevier |year=2003 |isbn=0-7623-1075-8 }}
* {{Cite journal |last1=Hayes |first1=Andrew F. |last2=Cai |first2=Li |title=Using heteroskedasticity-consistent standard error estimators in OLS regression: An introduction and software implementation |journal=Behavior Research Methods |volume=39 |issue=4 |pages=709–722 |doi=10.3758/BF03192961 |pmid=18183883 |year=2007 |doi-access=free }}
* {{cite journal |first1=Gary |last1=King |author-link=Gary King (political scientist) |first2=Margaret E. |last2=Roberts |title=How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It |journal=[[Political Analysis (journal)|Political Analysis]] |volume=23 |issue=2 |year=2015 |pages=159–179 |doi=10.1093/pan/mpu015 |url=http://nrs.harvard.edu/urn-3:HUL.InstRepos:13572089 }}
* {{Cite book |last=Wooldridge |first=Jeffrey M. |author-link=Jeffrey Wooldridge |chapter=Heteroskedasticity-Robust Inference after OLS Estimation |title=Introductory Econometrics : A Modern Approach |edition=Fourth |location=Mason |publisher=South-Western |year=2009 |isbn=978-0-324-66054-8 |pages=265–271 }}
* Buja, Andreas, et al. "Models as approximations-a conspiracy of random regressors and model deviations against classical inference in regression." Statistical Science (2015): 1. [http://www-stat.wharton.upenn.edu/~buja/PAPERS/Buja_et_al_A_Conspiracy-rev1.pdf pdf]

[[Category:Regression analysis]]
[[Category:Simultaneous equation methods (econometrics)]]
[[Category:Estimation methods]]
[[Category:Regression with time series structure]]

Heteroskedasticity-consistent standard errors - Revision history

imported>Bazsola: /* Problem */