Why are Gaussian Process Models called Nonparametric?

As we know, Gaussian processes modeling is often refer to as nonparametric modeling. But why? It has parameters in its covariance kernel:
$$\begin{align*} K(x_i,x_j) = h^2\text{exp}\left(\frac{-(x_i-x_j)^2}{\lambda^2}\right) \end{align*}$$
From the example of Gaussian kernel above, $h$ and $\lambda$ are the hyperparameters.

Some Points:

  • “Parametric” versus “Nonparametric” are terms that do not apply to particular processes, they apply to the entire family of processes that could be fit to data.
  • So if there’s no limit tothe number of parameters, we are estimating the parameter of infinite dimension, it turns out to be a nonparametric problem.
  • Models can be nonparametric in one sense and parametric in another (e.g. semiparametric models).
  • Nonparametric just means we cannot describe the model by a fixed set of parameters but they still can have parameters.
  • Return back to the example above, $h$ and $\lambda$ are not parameters because they do not represent anything physical about the data itself. They just parametrize the covariance.

Relevant Materials:

[1] Anna Scaife: Machine Learning: Gaussian Process Modelling in Python [lecture]
[2] Yee Whye Teh: Bayesian Nonparametric [lecture]