Reproducing kernel Hilbert space - Revision history

imported>Johaverm: /* growthexperiments-addlink-summary-summary:3|0|0 */

2025-12-27T20:26:10Z

growthexperiments-addlink-summary-summary:3|0|0

← Previous revision		Revision as of 20:26, 27 December 2025
Line 4:		Line 4:
	In [[functional analysis]], a '''reproducing kernel Hilbert space''' ('''RKHS''') is a [[Hilbert space]] of functions in which point evaluation is a continuous [[linear functional]]. Specifically, a Hilbert space <math>H</math> of functions from a set <math>X</math> (to <math>\mathbb{R}</math> or <math>\mathbb{C}</math>) is an RKHS if the point-evaluation functional <math>L_x:H\to\mathbb{C}</math>, <math>L_x(f)=f(x)</math>, is continuous for every <math>x\in X</math>. Equivalently, <math>H</math> is an RKHS if there exists a function <math>K_x \in H</math> such that, for all <math>f \in H</math>,<math display="block">\langle f, K_x \rangle = f(x).</math>The function <math>K_x</math> is then called the ''reproducing kernel'', and it reproduces the value of <math>f</math> at <math>x</math> via the inner product.		In [[functional analysis]], a '''reproducing kernel Hilbert space''' ('''RKHS''') is a [[Hilbert space]] of functions in which point evaluation is a continuous [[linear functional]]. Specifically, a Hilbert space <math>H</math> of functions from a set <math>X</math> (to <math>\mathbb{R}</math> or <math>\mathbb{C}</math>) is an RKHS if the point-evaluation functional <math>L_x:H\to\mathbb{C}</math>, <math>L_x(f)=f(x)</math>, is continuous for every <math>x\in X</math>. Equivalently, <math>H</math> is an RKHS if there exists a function <math>K_x \in H</math> such that, for all <math>f \in H</math>,<math display="block">\langle f, K_x \rangle = f(x).</math>The function <math>K_x</math> is then called the ''reproducing kernel'', and it reproduces the value of <math>f</math> at <math>x</math> via the inner product.

	An immediate consequence of this property is that convergence in norm implies uniform convergence on any subset of <math>X</math> on which <math>\\|K_x\\|</math> is bounded. However, the converse does not necessarily hold. Often the set <math>X</math> carries a topology, and <math>\\|K_x\\|</math> depends continuously on <math>x\in X</math>, in which case: convergence in norm implies uniform convergence on compact subsets of <math>X</math>.		An immediate consequence of this property is that convergence in norm implies [[uniform convergence]] on any subset of <math>X</math> on which <math>\\|K_x\\|</math> is bounded. However, the converse does not necessarily hold. Often the set <math>X</math> carries a topology, and <math>\\|K_x\\|</math> depends continuously on <math>x\in X</math>, in which case: convergence in norm implies uniform convergence on compact subsets of <math>X</math>.

	It is not entirely straightforward to construct natural examples of a Hilbert space which are not an RKHS in a non-trivial fashion.<ref>Alpay, D., and T. M. Mills. "A family of Hilbert spaces which are not reproducing kernel Hilbert spaces." J. Anal. Appl. 1.2 (2003): 107–111.</ref> Some examples, however, have been found.<ref> Z. Pasternak-Winiarski, "On weights which admit reproducing kernel of Bergman type", ''International Journal of Mathematics and Mathematical Sciences'', vol. 15, Issue 1, 1992. </ref><ref> T. Ł. Żynda, "On weights which admit reproducing kernel of Szegő type", ''Journal of Contemporary Mathematical Analysis'' (Armenian Academy of Sciences), 55, 2020. </ref>		It is not entirely straightforward to construct natural examples of a Hilbert space which are not an RKHS in a non-trivial fashion.<ref>Alpay, D., and T. M. Mills. "A family of Hilbert spaces which are not reproducing kernel Hilbert spaces." J. Anal. Appl. 1.2 (2003): 107–111.</ref> Some examples, however, have been found.<ref> Z. Pasternak-Winiarski, "On weights which admit reproducing kernel of Bergman type", ''International Journal of Mathematics and Mathematical Sciences'', vol. 15, Issue 1, 1992. </ref><ref> T. Ł. Żynda, "On weights which admit reproducing kernel of Szegő type", ''Journal of Contemporary Mathematical Analysis'' (Armenian Academy of Sciences), 55, 2020. </ref>
Line 10:		Line 10:
	While, formally, [[Square-integrable function\|''L''<sup>2</sup> spaces]] are defined as Hilbert spaces of equivalence classes of functions, this definition can trivially be extended to a Hilbert space of functions by choosing a (total) function as a representative for each equivalence class. However, no choice of representatives can make this space an RKHS (<math>K_0</math> would need to be the non-existent Dirac delta function). However, there are RKHSs in which the norm is an ''L''<sup>2</sup>-norm, such as the space of band-limited functions (see the example below).		While, formally, [[Square-integrable function\|''L''<sup>2</sup> spaces]] are defined as Hilbert spaces of equivalence classes of functions, this definition can trivially be extended to a Hilbert space of functions by choosing a (total) function as a representative for each equivalence class. However, no choice of representatives can make this space an RKHS (<math>K_0</math> would need to be the non-existent Dirac delta function). However, there are RKHSs in which the norm is an ''L''<sup>2</sup>-norm, such as the space of band-limited functions (see the example below).

	An RKHS is associated with a kernel that reproduces every function in the space in the sense that for every <math>x</math> in the set on which the functions are defined, "evaluation at <math>x</math>" can be performed by taking an inner product with a function determined by the kernel. Such a ''reproducing kernel'' exists if and only if every evaluation functional is continuous.		An RKHS is associated with a kernel that reproduces every function in the space in the sense that for every <math>x</math> in the set on which the functions are defined, "evaluation at <math>x</math>" can be performed by taking an inner product with a function determined by the kernel. Such a ''reproducing kernel'' exists [[if and only if]] every evaluation functional is continuous.

	The reproducing kernel was first introduced in the 1907 work of [[Stanisław Zaremba (mathematician)\|Stanisław Zaremba]]{{fact\|date=June 2025}} concerning [[boundary value problem]]s for [[Harmonic function\|harmonic]] and [[Biharmonic equation\|biharmonic functions]]. [[James Mercer (mathematician)\|James Mercer]] simultaneously examined [[Positive-definite kernel\|functions]] which satisfy the reproducing property in the theory of [[integral equation]]s. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of [[Gábor Szegő]], [[Stefan Bergman]], and [[Salomon Bochner]]. The subject was eventually systematically developed in the early 1950s by [[Nachman Aronszajn]] and Stefan Bergman.<ref>Okutmustur</ref>		The reproducing kernel was first introduced in the 1907 work of [[Stanisław Zaremba (mathematician)\|Stanisław Zaremba]]{{fact\|date=June 2025}} concerning [[boundary value problem]]s for [[Harmonic function\|harmonic]] and [[Biharmonic equation\|biharmonic functions]]. [[James Mercer (mathematician)\|James Mercer]] simultaneously examined [[Positive-definite kernel\|functions]] which satisfy the reproducing property in the theory of [[integral equation]]s. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of [[Gábor Szegő]], [[Stefan Bergman]], and [[Salomon Bochner]]. The subject was eventually systematically developed in the early 1950s by [[Nachman Aronszajn]] and Stefan Bergman.<ref>Okutmustur</ref>
Line 19:		Line 19:

	==Definition==		==Definition==
	Let <math>X</math> be an arbitrary [[Set (mathematics)\|set]] and <math>H</math> a [[Hilbert space]] of [[real-valued function]]s on <math>X</math>, equipped with pointwise addition and pointwise scalar multiplication. The [[Cartesian closed category#Evaluation\|evaluation]] functional over the Hilbert space of functions <math>H</math> is a linear functional that evaluates each function at a point <math>x</math>,		Let <math>X</math> be an arbitrary [[Set (mathematics)\|set]] and <math>H</math> a [[Hilbert space]] of [[real-valued function]]s on <math>X</math>, equipped with pointwise addition and pointwise [[scalar multiplication]]. The [[Cartesian closed category#Evaluation\|evaluation]] functional over the Hilbert space of functions <math>H</math> is a linear functional that evaluates each function at a point <math>x</math>,

	:<math> L_{x} : f \mapsto f(x) \text{ } \forall f \in H. </math>		:<math> L_{x} : f \mapsto f(x) \text{ } \forall f \in H. </math>
Line 103:		Line 103:
	:'''Theorem'''. Suppose ''K'' is a symmetric, [[positive definite kernel]] on a set ''X''. Then there is a unique Hilbert space of functions on ''X'' for which ''K'' is a reproducing kernel.		:'''Theorem'''. Suppose ''K'' is a symmetric, [[positive definite kernel]] on a set ''X''. Then there is a unique Hilbert space of functions on ''X'' for which ''K'' is a reproducing kernel.

	'''Proof'''. For all ''x'' in ''X'', define ''K<sub>x</sub>'' = ''K''(''x'', ⋅ ). Let ''H''<sub>0</sub> be the linear span of {''K<sub>x</sub>'' : ''x'' ∈ ''X''}. Define an inner product on ''H''<sub>0</sub> by		'''Proof'''. For all ''x'' in ''X'', define ''K<sub>x</sub>'' = ''K''(''x'', ⋅ ). Let ''H''<sub>0</sub> be the [[linear span]] of {''K<sub>x</sub>'' : ''x'' ∈ ''X''}. Define an inner product on ''H''<sub>0</sub> by

	:<math> \left\langle \sum_{j=1}^n b_j K_{y_j}, \sum_{i=1}^m a_i K_{x_i} \right \rangle_{H_0} = \sum_{i=1}^m \sum_{j=1}^n {a_i} b_j K(y_j, x_i),</math>		:<math> \left\langle \sum_{j=1}^n b_j K_{y_j}, \sum_{i=1}^m a_i K_{x_i} \right \rangle_{H_0} = \sum_{i=1}^m \sum_{j=1}^n {a_i} b_j K(y_j, x_i),</math>
Line 140:		Line 140:
	Mercer's theorem states that the spectral decomposition of the integral operator <math>T_K</math> of <math>K</math> yields a series representation of <math>K</math> in terms of the eigenvalues and eigenfunctions of <math> T_K </math>. This then implies that <math>K</math> is a reproducing kernel so that the corresponding RKHS can be defined in terms of these eigenvalues and eigenfunctions. We provide the details below.		Mercer's theorem states that the spectral decomposition of the integral operator <math>T_K</math> of <math>K</math> yields a series representation of <math>K</math> in terms of the eigenvalues and eigenfunctions of <math> T_K </math>. This then implies that <math>K</math> is a reproducing kernel so that the corresponding RKHS can be defined in terms of these eigenvalues and eigenfunctions. We provide the details below.

	Under these assumptions <math>T_K</math> is a compact, continuous, self-adjoint, and positive operator. The [[spectral theorem]] for self-adjoint operators implies that there is an at most countable decreasing sequence <math>(\sigma_i)_{i \geq 0} </math> such that <math display="inline">\lim_{i \to \infty}\sigma_i = 0</math> and		Under these assumptions <math>T_K</math> is a compact, continuous, self-adjoint, and [[positive operator]]. The [[spectral theorem]] for self-adjoint operators implies that there is an at most countable decreasing sequence <math>(\sigma_i)_{i \geq 0} </math> such that <math display="inline">\lim_{i \to \infty}\sigma_i = 0</math> and
	<math>T_K\varphi_i(x) = \sigma_i\varphi_i(x)</math>, where the <math>\{\varphi_i\}</math> form an orthonormal basis of <math>L_2(X)</math>. By the positivity of <math>T_K, \sigma_i > 0</math> for all <math>i.</math> One can also show that <math>T_K </math> maps continuously into the space of continuous functions <math>C(X)</math> and therefore we may choose continuous functions as the eigenvectors, that is, <math>\varphi_i \in C(X)</math> for all <math>i.</math> Then by Mercer's theorem <math> K </math> may be written in terms of the eigenvalues and continuous eigenfunctions as		<math>T_K\varphi_i(x) = \sigma_i\varphi_i(x)</math>, where the <math>\{\varphi_i\}</math> form an orthonormal basis of <math>L_2(X)</math>. By the positivity of <math>T_K, \sigma_i > 0</math> for all <math>i.</math> One can also show that <math>T_K </math> maps continuously into the space of continuous functions <math>C(X)</math> and therefore we may choose continuous functions as the eigenvectors, that is, <math>\varphi_i \in C(X)</math> for all <math>i.</math> Then by Mercer's theorem <math> K </math> may be written in terms of the eigenvalues and continuous eigenfunctions as

Line 328:		Line 328:
	\|mr=51437		\|mr=51437
	\|doi=10.1090/S0002-9947-1950-0051437-7		\|doi=10.1090/S0002-9947-1950-0051437-7
	\|doi-access=free}}		\|bibcode=1950TAMS...68..337A \|doi-access=free}}
	* Berlinet, Alain and Thomas, Christine. ''Reproducing kernel Hilbert spaces in Probability and Statistics'', Kluwer Academic Publishers, 2004.		* Berlinet, Alain and Thomas, Christine. ''Reproducing kernel Hilbert spaces in Probability and Statistics'', Kluwer Academic Publishers, 2004.
	* {{cite journal		* {{cite journal
Line 348:		Line 348:
	\|journal=Journal of Mathematical Analysis and Applications		\|journal=Journal of Mathematical Analysis and Applications
	\|volume=33 \|issue=1 \|year=1971 \|pages=82–95 \|doi=10.1016/0022-247X(71)90184-3		\|volume=33 \|issue=1 \|year=1971 \|pages=82–95 \|doi=10.1016/0022-247X(71)90184-3
			\|bibcode=1971JMAA...33...82K
	\|mr=290013		\|mr=290013
	\|doi-access=free}}		\|doi-access=free}}

imported>Roffaduft: redundant.

2025-06-15T05:39:46Z

redundant.

← Previous revision		Revision as of 05:39, 15 June 2025
Line 12:		Line 12:
	An RKHS is associated with a kernel that reproduces every function in the space in the sense that for every <math>x</math> in the set on which the functions are defined, "evaluation at <math>x</math>" can be performed by taking an inner product with a function determined by the kernel. Such a ''reproducing kernel'' exists if and only if every evaluation functional is continuous.		An RKHS is associated with a kernel that reproduces every function in the space in the sense that for every <math>x</math> in the set on which the functions are defined, "evaluation at <math>x</math>" can be performed by taking an inner product with a function determined by the kernel. Such a ''reproducing kernel'' exists if and only if every evaluation functional is continuous.

	The reproducing kernel was first introduced in the 1907 work of [[Stanisław Zaremba (mathematician)\|Stanisław Zaremba]] concerning [[boundary value problem]]s for [[Harmonic function\|harmonic]] and [[Biharmonic equation\|biharmonic functions]]. [[James Mercer (mathematician)\|James Mercer]] simultaneously examined [[Positive-definite kernel\|functions]] which satisfy the reproducing property in the theory of [[integral equation]]s. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of [[Gábor Szegő]], [[Stefan Bergman]], and [[Salomon Bochner]]. The subject was eventually systematically developed in the early 1950s by [[Nachman Aronszajn]] and Stefan Bergman.<ref>Okutmustur</ref>		The reproducing kernel was first introduced in the 1907 work of [[Stanisław Zaremba (mathematician)\|Stanisław Zaremba]]{{fact\|date=June 2025}} concerning [[boundary value problem]]s for [[Harmonic function\|harmonic]] and [[Biharmonic equation\|biharmonic functions]]. [[James Mercer (mathematician)\|James Mercer]] simultaneously examined [[Positive-definite kernel\|functions]] which satisfy the reproducing property in the theory of [[integral equation]]s. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of [[Gábor Szegő]], [[Stefan Bergman]], and [[Salomon Bochner]]. The subject was eventually systematically developed in the early 1950s by [[Nachman Aronszajn]] and Stefan Bergman.<ref>Okutmustur</ref>

	These spaces have wide applications, including [[complex analysis]], [[harmonic analysis]], and [[quantum mechanics]]. Reproducing kernel Hilbert spaces are particularly important in the field of [[statistical learning theory]] because of the celebrated [[representer theorem]] which states that every function in an RKHS that minimises an empirical risk functional can be written as a [[linear combination]] of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the [[empirical risk minimization]] problem from an infinite dimensional to a finite dimensional optimization problem.		These spaces have wide applications, including [[complex analysis]], [[harmonic analysis]], and [[quantum mechanics]]. Reproducing kernel Hilbert spaces are particularly important in the field of [[statistical learning theory]] because of the celebrated [[representer theorem]] which states that every function in an RKHS that minimises an empirical risk functional can be written as a [[linear combination]] of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the [[empirical risk minimization]] problem from an infinite dimensional to a finite dimensional optimization problem.
Line 134:		Line 134:
	We may characterize a symmetric positive definite kernel <math>K</math> via the integral operator using [[Mercer's theorem]] and obtain an additional view of the RKHS. Let <math>X</math> be a compact space equipped with a strictly positive finite [[Borel measure]] <math>\mu</math> and <math>K: X \times X \to \R</math> a continuous, symmetric, and positive definite function. Define the integral operator <math>T_K: L_2(X) \to L_2(X)</math> as		We may characterize a symmetric positive definite kernel <math>K</math> via the integral operator using [[Mercer's theorem]] and obtain an additional view of the RKHS. Let <math>X</math> be a compact space equipped with a strictly positive finite [[Borel measure]] <math>\mu</math> and <math>K: X \times X \to \R</math> a continuous, symmetric, and positive definite function. Define the integral operator <math>T_K: L_2(X) \to L_2(X)</math> as

	:<math> [T_K f](\cdot) =\int_X K(\cdot,t) f(t)\, d\mu(t) </math>		:<math> [T_K f](\cdot) =\int_X K({}\cdot{},t) f(t)\, d\mu(t) </math>

	where <math>L_2(X)</math> is the space of square integrable functions with respect to <math> \mu </math>.		where <math>L_2(X)</math> is the space of square integrable functions with respect to <math> \mu </math>.
Line 159:		Line 159:
	:<math> \left\langle f,g \right\rangle_H = \sum_{i=1}^\infty \frac{\left\langle f,\varphi_i \right\rangle_{L_2}\left\langle g,\varphi_i \right\rangle_{L_2}}{\sigma_i}. </math>		:<math> \left\langle f,g \right\rangle_H = \sum_{i=1}^\infty \frac{\left\langle f,\varphi_i \right\rangle_{L_2}\left\langle g,\varphi_i \right\rangle_{L_2}}{\sigma_i}. </math>

	This representation of the RKHS has application in probability and statistics, for example to the [[Karhunen–Loève theorem\|~~Karhunen-Loève~~ representation]] for stochastic processes and [[kernel PCA]].		This representation of the RKHS has application in probability and statistics, for example to the [[Karhunen–Loève theorem\|Karhunen–Loève representation]] for stochastic processes and [[kernel PCA]].

	==Feature maps==		==Feature maps==

139.80.239.160: /* Integral operators and Mercer's theorem */

2025-05-08T04:53:12Z

Integral operators and Mercer's theorem

Show changes