Abstract
In Krotov et al. (in: Lee (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., Red Hook, 2016) Krotov and Hopfield suggest a generalized version of the well-known Hopfield model of associative memory. In their version they consider a polynomial interaction function and claim that this increases the storage capacity of the model. We prove this claim and take the ”limit” as the degree of the polynomial becomes infinite, i.e. an exponential interaction function. With this interaction we prove that model has an exponential storage capacity in the number of neurons, yet the basins of attraction are almost as large as in the standard Hopfield model.
This is a preview of subscription content, log in via an institution to check access.
References
Amit, D.J., Gutfreund, H., Sompolinsky, H.: Spin-glass models of neural networks. Phys. Rev. A. 32(2), 1007–1018 (1985a). doi:10.1103/PhysRevA.32.1007
Amit, D.J., Gutfreund, H., Sompolinsky, H.: Storing infinite numbers of patterns in a spin-glass model of neural networks. Phys. Rev. Lett. 55, 1530–1533 (1985b). doi:10.1103/PhysRevLett.55.1530
Bovier, A.: Sharp upper bounds on perfect retrieval in the Hopfield model. J. Appl. Probab. 36(3), 941–950 (1999)
Bovier, A., Niederhauser, B.: The spin-glass phase-transition in the Hopfield model with \(p\)-spin interactions. Adv. Theor. Math. Phys. 5(6), 1001–1046 (2001). doi:10.4310/ATMP.2001.v5.n6.a2
Dembo, A., Zeitouni, O.: Large deviations techniques and applications. Stochastic Modelling and Applied Probability, vol. 38, p. 396. Springer, Berlin (2010) Corrected reprint of the second (1998) edition. ISBN 978-3-642-03310-0. doi:10.1007/978-3-642-03311-7
Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79(8), 2554–2558 (1982)
Krotov, D., Hopfield, J.J.: Dense associative memory for pattern recognition. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 1172–1180. Curran Associates, Inc., Red Hook (2016)
Loukianova, D.: Lower bounds on the restitution error in the Hopfield model. Probab. Theory Relat. Fields 107(2), 161–176 (1997). doi:10.1007/s004400050081
Löwe, M.: The storage capacity of generalized Hopfield models with semantically correlated patterns. Markov Process. Relat. Fields 5(1), 1–19 (1999)
Löwe, M.: On the storage capacity of Hopfield models with correlated patterns. Ann. Appl. Probab. 8(4), 1216–1250 (1998). doi:10.1214/aoap/1028903378
Löwe, M., Vermet, F.: The storage capacity of the Hopfield model and moderate deviations. Stat. Probab. Lett. 75(4), 237–248 (2005). doi:10.1016/j.spl.2005.06.001
Löwe, M., Vermet, F.: The capacity of \(q\)-state Potts neural networks with parallel retrieval dynamics. Stat. Probab. Lett. 77(14), 1505–1514 (2007). doi:10.1016/j.spl.2007.03.030
McEliece, R.J., Posner, E.C., Rodemich, E.R., Venkatesh, S.S.: The capacity of the Hopfield associative memory. IEEE Trans. Inform. Theory 33(4), 461–482 (1987). doi:10.1109/TIT.1987.1057328
Newman, C.M.: Memory capacity in neural network models: rigorous lower bounds. Neural Netw. 1(3), 223–238 (1988)
Talagrand, M.: Rigorous results for the Hopfield model with many patterns. Probab. Theory Relat. Fields 110(2), 177–276 (1998). doi:10.1007/s004400050148
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Demircigil, M., Heusel, J., Löwe, M. et al. On a Model of Associative Memory with Huge Storage Capacity. J Stat Phys 168, 288–299 (2017). https://doi.org/10.1007/s10955-017-1806-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10955-017-1806-y