On Bayesian Classification with Laplace Priors

Research output: Contribution to journalArticle

Authors

Colleges, School and Institutes

Abstract

We present a new classification approach, using a variational Bayesian estimation of probit regression with Laplace priors. Laplace priors have been previously used extensively as a sparsity-inducing mechanism to perform feature selection simultaneously with classification or regression. However, contrarily to the 'myth' of sparse Bayesian learning with Laplace priors, we find that the sparsity effect is due to a property of the maximum a posteriori (MAP) parameter estimates only. The Bayesian estimates, in turn, induce a posterior weighting rather than a hard selection of features, and has different advantageous properties: (1) it provides better estimates of the prediction uncertainty; (2) it is able to retain correlated features favouring generalisation; (3) it is more stable with respect to the hyperparameter choice and (4) it produces a weight-based ranking of the features, suited for interpretation.. We analyse the behaviour of the Bayesian estimate in comparison with its MAP counterpart, as well as other related models, (a) through a graphical interpretation of the associated shrinkage and (b) by controlled numerical simulations in a range of testing conditions. The results pinpoint the situations when the advantages of Bayesian estimates are feasible to exploit. Finally, we demonstrate the working of our method in a gene expression classification task. (C) 2007 Elsevier B.V. All rights reserved.

Details

Original languageEnglish
Pages (from-to)1271-1282
Number of pages12
JournalPattern Recognition Letters
Volume28
Issue number10
Publication statusPublished - 15 Jul 2007

Keywords

  • microarray gene expressions, Laplace prior, shrinkage effect, predictive features, variational Bayes, sparsity