Flexible learning with adaptive predictive distributions

The fundamental step of several statistical procedures is the choice of the predictive distributions $\alpha_n(\cdot)=P(X_{n+1}\in\cdot\mid X_1,\ldots,X_n)$ for the data sequence $(X_1,X_2,\ldots)$. Two main examples are Bayesian predictive inference and machine learning, but the choice of $\alpha_n$ is crucial in many other frameworks, including species sampling sequences, the prequential approach, causal inference and predictive resampling. In this paper, a new sequence $\alpha_n$ of predictive distributions is introduced and investigated. The $\alpha_n$ have a mixture structure which makes them suitable to a wide variety of data. Each $\alpha_n$ depends on $(X_1,\ldots,X_n)$ only through the means, the covariance matrices and the relative frequencies of the mixture components. The updating rule of $\alpha_n$ is reminiscent of the P\'olya urns scheme. The main results are both theoretical and practical. Firstly, it is shown that $\norm{\alpha_n-\alpha}\overset{a.s.}\longrightarrow 0$, as $n\rightarrow\infty$, where $\norm{\cdot}$ is total variation distance and $\alpha$ a suitable random probability measure. An explicit formula for $\alpha$ is obtained as well. Secondly, the $\alpha_n$ are tested on simulated and real data. It turns out that, when used in predictive resampling, $\alpha_n$ performs well in posterior estimation of parameters such as moments, quantiles and regression coefficients.