Skip to main content
SpringerLink
Log in

Three Simple Approaches to Combining Neural Networks with Algorithms

  • Conference paper
  • First Online:
Data Analytics and Management in Data Intensive Domains (DAMDID/RCDL 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1223))

  • 334 Accesses

Abstract

Recently, deep neural networks have showed amazing results in many fields. To build such networks, we usually use layers from a relatively small dictionary of available modules (fully-connected, convolutional, recurrent, etc.). Being restricted with this set of modules complicates transferring technology to new tasks. On the other hand, many important applications already have a long history and successful algorithmic solutions. Is it possible to use existing methods to construct better networks? In this paper, we cover three approaches to combining neural networks with algorithms and discuss their pros and cons. Specifically, we will discuss three approaches: structured pooling, unrolling of algorithm iterations into network layers and explicit differentiation of the output w.r.t. the input.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://ai.stanford.edu/~btaskar/ocr/.

  2. 2.

    https://github.com/pytorch/examples/blob/master/mnist/main.py.

References

  1. Abadi, M., Agarwal, A., Barham, P., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). http://tensorflow.org/

  2. Agrawal, A., Amos, B., Barratt, S., Boyd, S., Diamond, S., Kolter, Z.: Differentiable convex optimization layers. In: Advances in Neural Information Processing Systems 32 (NeuIPS) (2019)

    Google Scholar 

  3. Andrychowicz, M., et al.: Learning to learn by gradient descent by gradient descent. In: Advances in Neural Information Processing Systems 29 (NIPS) (2016)

    Google Scholar 

  4. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Proceedings of the International Conference on Learning Representations (ICLR) (2015)

    Google Scholar 

  5. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006). https://www.microsoft.com/en-us/research/people/cmbishop/#!prml-book

    MATH  Google Scholar 

  6. Borenstein, E., Ullman, S.: Class-specific, top-down segmentation. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 109–122. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-47967-8_8

    Chapter  Google Scholar 

  7. Bottou, L., Le Cun, Y., Bengio, Y.: Global training of document processing systems using graph transformer networks. In: Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) (1997)

    Google Scholar 

  8. Chandra, S., Kokkinos, I.: Fast, exact and multi-scale inference for semantic image segmentation with deep Gaussian CRFs. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 402–418. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_25

    Chapter  Google Scholar 

  9. Chandra, S., Usunier, N., Kokkinos, I.: Dense and low-rank Gaussian CRFs using deep embeddings. In: Proceedings of International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  10. Chen, L., Schwing, A., Yuille, A., Urtasun, R.: Learning deep structured models. In: Proceedings of the International Conference on Machine Learning (ICML) (2015)

    Google Scholar 

  11. Chen, T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. In: Advances in Neural Information Processing Systems 31 (NeuIPS) (2018)

    Google Scholar 

  12. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12, 2493–2537 (2011)

    MATH  Google Scholar 

  13. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding (2018). arXiv:1810.04805

  14. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 580–587 (2014)

    Google Scholar 

  15. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). http://www.deeplearningbook.org

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  17. Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition. IEEE Sig. Process. Mag. 29, 82–97 (2012)

    Article  Google Scholar 

  18. Ionescu, C., Vantzos, O., Sminchisescu, C.: Matrix backpropagation for deep networks with structured layers. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2015)

    Google Scholar 

  19. Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Deep structured output learning for unconstrained text recognition (2014). arXiv:1412.5903v5

  20. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25 (NeuIPS) (2012)

    Google Scholar 

  21. Lafferty, J., McCallum, A., Pereira, F.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the International Conference on Machine Learning (ICML) (2001)

    Google Scholar 

  22. Le Cun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  23. Leblond, R., Alayrac, J.B., Osokin, A., Lacoste-Julien, S.: SEARNN: training RNNs with global-local losses. In: Proceedings of International Conference on Learning Representations (ICLR) (2018)

    Google Scholar 

  24. LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F.J.: Energy-based models. In: Bakir, G.H., Hofmann, T., Schölkopf, B., Smola, A.J., Taskar, B., Vishwanathan, S.V.N. (eds.) Predicting Structured Data (2006)

    Google Scholar 

  25. Lefkimmiatis, S.: Non-local color image denoising with convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  26. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  27. Maclaurin, D., Duvenaud, D., Adams, R.: Gradient-based hyperparameter optimization through reversible learning. In: Proceedings of the International Conference on Machine Learning (ICML) (2015)

    Google Scholar 

  28. Metz, L., Poole, B., Pfau, D., Sohl-Dickstein, J.: Unrolled generative adversarial networks. In: Proceedings of the International Conference on Learning Representations (ICLR) (2017)

    Google Scholar 

  29. Osokin, A.: How to put algorithms into neural networks. In: CEUR Workshop Proceedings: Proceedings of the International Conference on Data Analytics and Management in Data Intensive Domains (DAMDID/RCDL) (2019)

    Google Scholar 

  30. Paszke, A., Gross, S., Massa, F., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32 (NeuIPS) (2019)

    Google Scholar 

  31. Pérez-Cruz, F., Ghahramani, Z., Pontil, M.: Conditional graphical models. In: Bakir, G.H., Hofmann, T., Schölkopf, B., Smola, A.J., Taskar, B., Vishwanathan, S.V.N. (eds.) Predicting Structured Data. MIT Press, Cambridge (2007)

    Google Scholar 

  32. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 39(6), 1137–1149 (2017)

    Article  Google Scholar 

  33. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (2015)

    Google Scholar 

  34. Shevchenko, A., Osokin, A.: Scaling matters in deep structured-prediction models (2019). arXiv:1902.11088

  35. Taskar, B., Guestrin, C., Koller, D.: Max-margin Markov networks. In: Proceedings of Neural Information Processing Systems Conference (NIPS) (2003)

    Google Scholar 

  36. Tsochantaridis, I., Joachims, T., Hofmann, T., Altun, Y.: Large margin methods for structured and interdependent output variables. J. Mach. Learn. Res. 6, 1453–1484 (2005)

    MathSciNet  MATH  Google Scholar 

  37. van den Oord, A., Li, Y., Babuschkin, I., Simonyan, K., et al.: Parallel WaveNet: fast high-fidelity speech synthesis. In: Proceedings of the International Conference on Machine Learning (ICML) (2018)

    Google Scholar 

  38. Vu, T., Osokin, A., Laptev, I.: Context-aware CNNs for person head detection. In: Proceedings of International Conference on Computer Vision (ICCV) (2015)

    Google Scholar 

  39. Wu, Y., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation (2016). arXiv:1609.08144

  40. Zheng, S., et al.: Conditional random fields as recurrent neural networks. In: Proceedings of International Conference on Computer Vision (ICCV) (2015)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the Russian Science Foundation project 19-71-00082.

Author information

Authors and Affiliations

  1. National Research University Higher School of Economics, Moscow, Russia

    Anton Osokin

Authors
  1. Anton Osokin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anton Osokin .

Editor information

Editors and Affiliations

  1. Kazan Federal University, Kazan, Russia

    Alexander Elizarov

  2. National Research University Higher School of Economics, St. Petersburg, Russia

    Boris Novikov

  3. Federal Research Center “Computer Science and Control”, Moscow, Russia

    Sergey Stupnikov

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Osokin, A. (2020). Three Simple Approaches to Combining Neural Networks with Algorithms. In: Elizarov, A., Novikov, B., Stupnikov, S. (eds) Data Analytics and Management in Data Intensive Domains. DAMDID/RCDL 2019. Communications in Computer and Information Science, vol 1223. Springer, Cham. https://doi.org/10.1007/978-3-030-51913-1_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-51913-1_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-51912-4

  • Online ISBN: 978-3-030-51913-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation