Abstract
Convolutional neural networks (CNNs) have achieved breakthrough performance in image recognition and classification. However, it is still a challenge for CNNs to classify images where the classification results strongly depend on the recognition of globally correlated features, for example, the identification of biomolecular structure symmetry in cryo-electron microscopy images. In this work, we improved the traditional CNNs model to solve the problem of cryo-electron microscopy image symmetry recognition and developed a visualization system to help biological researchers to design the best model. Our system consists of the model overview view, the model structure view, and the case view. The system supports a comprehensive analysis of the model’s recognition accuracy, the number of calculations, and the convolutional layers between different models. It also supports structural analysis of models and visual analysis of specific input images. Through the real cases from the biological researchers, we verify that our system can effectively build a model to identify symmetry.
Graphic abstract
Similar content being viewed by others
References
Bairagi VK (2015) Symmetry-based biomedical image compression. J Digit Imaging 28(6):718
Celik T (2014) Spatial entropy-based global and local image contrast enhancement. IEEE Trans Image Process 23(12):5298
Chen X, Guan Q, Lo LT, Su S, Ren Z, Ahrens JP, Estrada T (2018) In situ TensorView: in situ visualization of convolutional neural networks. In: 2018 IEEE international conference on big data (big data). IEEE, pp 1899–1904
Dosovitskiy A, Brox T (2016) Inverting visual representations with convolutional networks. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 4829–4837. https://academic.microsoft.com/paper/2963989815
Erhan D, Bengio Y, Courville A, Vincent P (2009) Visualizing higher-layer features of a deep network. Univ Montreal 1341(3):1
He K, Zhang X, Ren S, Jian S (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR)
Jarrett K, Kavukcuoglu K, Ranzato M, LeCun Y (2009) What is the best multi-stage architecture for object recognition?. In: 2009 IEEE 12th international conference on computer vision, pp 2146–2153
Ji Y, Dan CM, Wei Z, Baker TS (2003) Orientation refinement of virus structures with unknown symmetry. In: 17th international parallel and distributed processing symposium (IPDPS 2003), 22–26 April 2003, Nice, France, CD-ROM/Abstracts Proceedings
Jing H, Qinqi W, Shinie C, Fan D, Hongtao G, Fan L (2015) Symmetry image symmetric axis detection positioning method. In: Symmetry image symmetric axis detection positioning method. https://academic.microsoft.com/paper/2839302056
Junpeng W, Liang G, Han-Wei S, Hao Y (2018) DQNViz: a visual analytics approach to understand deep Q-networks. IEEE Trans Vis Comput Graph, pp 205-220. https://doi.org/10.1109/TVCG.2018.2864504
Kinjo AR, Suzuki H, Yamashita R, Ikegawa Y, Kudou T, Igarashi R, Kengaku Y, Cho H, Standley DM, Nakagawa A, Nakamura H (2012) Protein data bank Japan (PDBj): maintaining a structural data archive and resource description framework format”. Nucleic Acids Res 40:453
Krizhevsky A, Sutskever I, Hinton G (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, vol 25, no 2
Laskowski RA, MacArthur MW, Moss DS, Thornton JM (1993) PROCHECK: a program to check the stereochemical quality of protein structures. J Appl Crystallogr 26(2):283
Lin M, Chen Q, Yan S (2014) Network in network. In: ICLR 2014: international conference on learning representations (ICLR) 2014
Liu M, Shi J, Li Z, Li C, Zhu J, Liu S (2016) Towards better analysis of deep convolutional neural networks. IEEE Trans Vis Comput Graph 23(1):91
Mahendran A, Vedaldi A (2015) Understanding deep image representations by inverting them. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 5188–5196. https://academic.microsoft.com/paper/1915485278
Rajasegaran J, Jayasundara V, Jayasekara S, Jayasekara H, Seneviratne S, Rodrigo R (2019) DeepCaps: going deeper with capsule networks. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 10,725–10,733
Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Advances in neural information processing systems. pp 3856–3866. https://academic.microsoft.com/paper/2963703618
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR 2015: international conference on learning representations 2015. https://academic.microsoft.com/paper/2962835968
Strobelt H, Gehrmann S, Behrisch M, Perer A, Pfister H, Rush AM (2018) SEQ2SEQ-VIS: a visual debugging tool for sequence-to-sequence models. IEEE Trans Vis Comput Graph, pp 1–1 https://doi.org/10.1109/TVCG.2018.2865044
Studholme C, Hill DLG, Hawkes DJ (1999) An overlap invariant entropy measure of 3D medical image alignment. Pattern Recognit 32(1):71
Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2016) Inception-v4, inception-ResNet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence, pp 4278–4284. https://academic.microsoft.com/paper/2964350391
Tagari M, Newman R, Chagoyen M, Carazo JM, Henrick K (2002) New electron microscopy database and deposition system. Trends Biochem Sci 27(11):589
Velankar S, Best C, Beuth B, Boutselakis CH, Cobley N, Silva AWSD, Dimitropoulos D, Golovin A, Hirshberg M, John M, Krissinel EB, Newman R, Oldfield T, Pajon A, Penkett CJ, Pineda-Castillo J, Sahni G, Sen S, Slowley R, Suarez-Uruena A, Swaminathan J, van Ginkel G, Vranken WF, Henrick K, Kleywegt GJ (2010) PDBe: protein data bank in Europe. Nucleic Acids Res 40:285
Wardman J (2012) A symmetry approach to virus architecture. https://academic.microsoft.com/paper/1608112681
Xie S, Girshick R, Dollar P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 5987–5995. https://academic.microsoft.com/paper/2549139847
Yosinski J, Clune J, Nguyen A, Fuchs T, Lipson H (2015) Understanding neural networks through deep visualization. arXiv:1506.06579
Zeiler MD, Krishnan D, Taylor GW, Fergus R (2010) Deconvolutional networks. In: 2010 IEEE computer society conference on computer vision and pattern recognition. https://academic.microsoft.com/paper/2293078015
Zeiler MD, Fergus R (2014) ‘Visualizing and understanding convolutional networks. In: 13th European conference on computer vision, ECCV 2014, pp 818–833
Zhou ZH, Zhang H, Jakana J, Lu XY, Zhang JQ (2003) Cytoplasmic Polyhedrosis Virus Structure at 8 Ã by electron cryomicroscopy: structural basis of capsid stability and mrna processing regulation. Structure 11(6):651
Acknowledgements
This work was supported by Key Research Program of Frontier Sciences, CAS, Grant No. QYZDB-SSW-SMC004-02.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Han, X., Shen, HW., Li, G. et al. IVDAS: an interactive visual design and analysis system for image data symmetry detection of CNN models. J Vis 24, 615–629 (2021). https://doi.org/10.1007/s12650-020-00721-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12650-020-00721-3