Abstract
We show how to bypass the integral calculation and express the Kullback-Leibler divergence between any two densities of an exponential family using a finite sum of logarithms of density ratio evaluated at sigma points. This result allows us to characterize the exact error of Monte Carlo estimators. We then extend the sigma point method to the calculations of q-divergences between densities of a deformed q-exponential family.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amari, S.I.: Information Geometry and Its Applications. Applied Mathematical Sciences. Springer, Japan (2016). https://doi.org/10.1007/978-4-431-55978-8
Amari, S.I., Ohara, A.: Geometry of \(q\)-exponential family of probability distributions. Entropy 13(6), 1170–1185 (2011)
Azoury, K.S., Warmuth, M.K.: Relative loss bounds for on-line density estimation with the exponential family of distributions. Mach. Learn. 43(3), 211–246 (2001)
Banerjee, A., Merugu, S., Dhillon, I.S., Ghosh, J.: Clustering with Bregman divergences. J. Mach. Learn. Res. 6, 1705–1749 (2005)
Barndorff-Nielsen, O.: Information and Exponential Families. John Wiley & Sons, Hoboken (2014)
Billingsley, P.: Probability and Measure, 3rd edn. Wiley, New York (1995)
Bregman, L.M.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys. 7(3), 200–217 (1967)
Carathéodory, C.: Über den Variabilitätsbereich der Koeffizienten von Potenzreihen, die gegebene Werte nicht annehmen. Mathematische Annalen 64(1), 95–115 (1907)
Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley & Sons, Hoboken (2012)
Csiszár, I.: Information-type measures of difference of probability distributions and indirect observation. Studia Scientiarum Mathematicarum Hungarica 2, 229–318 (1967)
Del Castillo, J.: The singly truncated normal distribution: a non-steep exponential family. Ann. Inst. Stat. Math. 46(1), 57–66 (1994)
Gayen, A., Kumar, M.A.: Projection theorems and estimating equations for power-law models. J. Multivariate Anal. 184, 104734 (2021)
Goldberger, J., Greenspan, H.K., Dreyfuss, J.: Simplifying mixture models using the unscented transform. IEEE Trans. Pattern Anal. Mach. Intell. 30(8), 1496–1502 (2008)
Julier, S., Uhlmann, J., Durrant-Whyte, H.F.: A new method for the nonlinear transformation of means and covariances in filters and estimators. IEEE Trans. Autom. Control 45(3), 477–482 (2000)
Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Stat. 22(1), 79–86 (1951)
Lemieux, C.: Monte Carlo and quasi-Monte Carlo sampling. Springer Science & Business Media, New York (2009). https://doi.org/10.1007/978-0-387-78165-5
Nielsen, F.: An elementary introduction to information geometry. Entropy 22(10), 1100 (2020)
Nielsen, F.: On Voronoi diagrams on the information-geometric Cauchy manifolds. Entropy 22(7), 713 (2020)
Nielsen, F., Garcia, V.: Statistical exponential families: A digest with flash cards. Technical Report. arXiv:0911.4863 (2009)
Nielsen, F., Nock, R.: Patch matching with polynomial exponential families and projective divergences. In: Amsaleg, L., Houle, M.E., Schubert, E. (eds.) SISAP 2016. LNCS, vol. 9939, pp. 109–116. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46759-7_8
Nielsen, F., Nock, R.: Cumulant-free closed-form formulas for some common (dis)similarities between densities of an exponential family. arXiv:2003.02469 (2020)
Peyré, G., Cuturi, M.: Computational optimal transport: with applications to data science. Found. Trends® Mach. Learn. 11(5–6), 355–607 (2019)
Risch, R.H.: The solution of the problem of integration in finite terms. Bull. Am. Math. Soc. 76(3), 605–608 (1970)
Tanaya, D., Tanaka, M., Matsuzoe, H.: Notes on geometry of \(q\)-normal distributions. In: Recent Progress in Differential Geometry and Its Related Fields, pp. 137–149. World Scientific (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Nielsen, F., Nock, R. (2021). Computing Statistical Divergences with Sigma Points. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2021. Lecture Notes in Computer Science(), vol 12829. Springer, Cham. https://doi.org/10.1007/978-3-030-80209-7_72
Download citation
DOI: https://doi.org/10.1007/978-3-030-80209-7_72
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-80208-0
Online ISBN: 978-3-030-80209-7
eBook Packages: Computer ScienceComputer Science (R0)