Abstract

Instead of maximum-likelihood or MAP, Bayesian inference encourages the use of predictive densities and evidence scores. This is illustrated in the context of the multinomial distribution, where predictive estimates are often used but rarely described as Bayesian. By using an entropy approximation to the evidence, many Bayesian quantities can be expressed in information-theoretic terms. For example, testing whether two samples come from the same distribution or testing whether two variables are independent boils down to a mutual information score (with appropriate smoothing). The same analysis can be applied to discrete Markov chains to get a test for Markovianity.

Citations

4648	Neural Networks for Pattern Recognition - Bishop - 1995
388	Divergence Measures based on the Shannon Entropy - Lin - 1991
357	An essay towards solving a problem in the doctrine of chances - Bayes
226	Biometry: the principles and practice of statistics in biological research - Sokal, Rohlf - 1995
32	Agnostic classification of markovian sequences - El-Yaniv, Fine, et al. - 1997
6	Determining whether two data sets are from the same distribution - Wolpert - 1995
4	Mutual information as a Bayesian measure of independence - Wolf - 1994