As we know, Gaussian processes modeling is often refer to as nonparametric modeling. But why? It has parameters in its covariance kernel:
$$\begin{align*}
K(x_i,x_j) = h^2\text{exp}\left(\frac{-(x_i-x_j)^2}{\lambda^2}\right)
\end{align*}$$
From the example of Gaussian kernel above, $h$ and $\lambda$ are the hyperparameters.
Some Points:
- “Parametric” versus “Nonparametric” are terms that do not apply to particular processes, they apply to the entire family of processes that could be fit to data.
- So if there’s no limit tothe number of parameters, we are estimating the parameter of infinite dimension, it turns out to be a nonparametric problem.
- Models can be nonparametric in one sense and parametric in another (e.g. semiparametric models).
- Nonparametric just means we cannot describe the model by a fixed set of parameters but they still can have parameters.
- Return back to the example above, $h$ and $\lambda$ are not parameters because they do not represent anything physical about the data itself. They just parametrize the covariance.
Relevant Materials:
[1] Anna Scaife: Machine Learning: Gaussian Process Modelling in Python [lecture]
[2] Yee Whye Teh: Bayesian Nonparametric [lecture]