Unification of field theory and maximum entropy methods for learning probability densities

Open Access

Unification of field theory and maximum entropy methods for learning probability densities

Justin B. Kinney

Phys. Rev. E 92, 032107 – Published 8 September 2015

Abstract

The need to estimate smooth probability distributions (a.k.a. probability densities) from finite sampled data is ubiquitous in science. Many approaches to this problem have been described, but none is yet regarded as providing a definitive solution. Maximum entropy estimation and Bayesian field theory are two such approaches. Both have origins in statistical physics, but the relationship between them has remained unclear. Here I unify these two methods by showing that every maximum entropy density estimate can be recovered in the infinite smoothness limit of an appropriate Bayesian field theory. I also show that Bayesian field theory estimation can be performed without imposing any boundary conditions on candidate densities, and that the infinite smoothness limit of these theories recovers the most common types of maximum entropy estimates. Bayesian field theory thus provides a natural test of the maximum entropy null hypothesis and, furthermore, returns an alternative (lower entropy) density estimate when the maximum entropy hypothesis is falsified. The computations necessary for this approach can be performed rapidly for one-dimensional data, and software for doing this is provided.

Received 21 March 2015

DOI:https://doi.org/10.1103/PhysRevE.92.032107

This article is available under the terms of the Creative Commons Attribution 3.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Authors & Affiliations

Justin B. Kinney^*

Simons Center for Quantitative Biology, Cold Spring Harbor Laboratory, Cold Spring Harbor, New York 11724, USA

^*jkinney@cshl.edu

Article Text

Click to Expand

References

Click to Expand

Issue

Vol. 92, Iss. 3 — September 2015

Reuse & Permissions

Author publication services for translation and copyediting assistance advertisement

Images

Figure 1
Illustration of the predictor-corrector homotopy algorithm. (a) The MAP curve (brown) is approximated using a finite set of densities $\{R, Q_{ℓ_{r}}, \dots, Q_{ℓ_{- 2}}, Q_{ℓ_{- 1}}, Q_{ℓ_{0}}, Q_{ℓ_{1}}, Q_{ℓ_{2}}, ..., Q_{ℓ_{m}}, Q_{\infty}\}$ . First the MAP density at an intermediate length scale $ℓ_{0} = L / \sqrt{G}$ is computed. A predictor-corrector algorithm is then used to extend the set of MAP densities outward to larger and to smaller values of $ℓ$ . These $ℓ$ values are chosen so that neighboring MAP densities lie within a geodesic distance of $≲ ε$ of each other. (b) Each step $Q_{ℓ_{t^{'}}} \to Q_{ℓ_{t}}$ has two parts. First, a predictor step (magenta) computes a new length scale $ℓ_{t}$ and an approximation $Q_{ℓ_{t}}^{(0)}$ of $Q_{ℓ_{t}}$ . A series of corrector steps $Q_{ℓ_{t}}^{(0)} \to Q_{ℓ_{t}}^{(1)} \to Q_{ℓ_{t}}^{(2)} \dots$ (orange) then converges to $Q_{ℓ_{t}}$ .
Reuse & Permissions
Figure 2
Density estimation without boundary conditions using the $α = 3$ field theory prior. (a) N = 100 data points were drawn from a simulated density $Q_{true}$ (black) and binned at $G = 100$ grid points. The resulting histogram (gray) is shown along with the field theory density estimate $Q_{ℓ^{*}}$ (orange) and the corresponding MaxEnt estimate $Q_{\infty}$ (blue). (b) The heat map shows the densities $Q_{ℓ}$ interpolating between the MaxEnt density at $ℓ = \infty$ and the data histogram at $ℓ = 0$ . (c) The log of the evidence ratio $E$ is shown as a function of $ℓ$ . (d) The differential entropy $H = - \int d x Q_{ℓ} ln Q_{ℓ}$ [24] is shown as a function of $ℓ$ ; the entropy at $ℓ = \infty$ is indicated by the dashed line. Dotted lines in (b–d) mark the optimal length scale $ℓ^{*}$ .
Reuse & Permissions
Figure 3
The optimal estimated density for any particular data set might or might not have maximum entropy. Panels (a–c) show three different choices for $Q_{true}$ (black), along with a histogram (gray) of $N = 100$ data points binned at $G = 100$ grid points. In each panel, $Q_{true}$ was chosen to be the sum of two equally weighted normal distributions separated by a distance of (a) 0, (b) 2.5, or (c) 5. Panels (d–f) show the evidence ratio curves computed for 100 data sets respectively drawn from the $Q_{true}$ densities in (a–c). Blue curves indicate $ℓ^{*} = \infty$ ; orange curves indicate finite $ℓ^{*}$ . Titles in (d–f) give the percentage of data sets for which $ℓ^{*} = \infty$ was found. The heat maps shown in panels (g–i) report the number of simulated data sets for which the $K$ coefficient was positive or negative and for which the MaxEnt density was or was not recovered.
Reuse & Permissions
Figure 4
Spectrum of the bilateral Laplacian of order $α = 3$ . (a) The first three Legendre polynomials provide an orthonormal basis for the kernel of $Δ^{3}$ . These choices for $ψ_{1}, ψ_{2}$ , and $ψ_{3}$ are plotted with decreasing opacity. (b) All other eigenfunctions are nontrivial linear combinations of factors of the form $exp (i κ x)$ . Their general behavior is illustrated by the basis functions $ψ_{4}, ψ_{5}, ψ_{6}$ , and $ψ_{7}$ , which are plotted with decreasing opacity. (c) The rank-ordered eigenvalues of $Δ^{3}$ lie at or below those of the standard Laplacian with any choice of boundary conditions. Shown are the eigenvalues of $Δ^{3}$ (black squares), together with the eigenvalues of $- \partial^{6}$ with periodic, Dirichlet, or Neumann boundary conditions (dark, medium, and light gray circles, respectively).
Reuse & Permissions

Physical Review E

covering statistical, nonlinear, biological, and soft matter physics