Abstract
Graph Convolutional Networks (GCN) can effectively extract rich information from non-structured data. However, in deep GCN models, the iterative propagation and updating of node information will lead to severe over-smoothing, which hampers the model’s performance. In our view, when the model suffers from over-smoothing, there will be little difference between the node features before and after updating. This paper proposes a more comprehensive smoothness metric according to nodes themselves, which considers both numerical and directional differences between nodes. Furthermore, (1) adding a similarity constraint between the initial features and the current layer features, which ensures the nodes’ representations avoid moving away from the initial features during the updating process; and (2) introducing a disparity constraint to the features of the nodes at each GCN layer to slow down the speed of node features becoming similar between before and after updating. We conduct extensive experiments on models with initial residual and achieve state-of-the-art results on several standard datasets.
Similar content being viewed by others
References
Lu M, Xiao Z, Li H, Zhang Y, Xiong NN (2022) Feature pyramid-based graph convolutional neural network for graph classification. J Syst Archit 128:102562
Ghnn (2022) Graph harmonic neural networks for semi-supervised graph-level classification. Neural Netw 151:70–79
Tang R, Jiang S, Chen X, Wang W, Wang W (2022) Network structural perturbation against interlayer link prediction. Knowl-Based Syst 250:109095
Maurya SK, Liu X, Murata T (2022) Simplifying approach to node classification in graph neural networks. J Comput Sci 62:101695
Abu-El-Haija S, Kapoor A, Perozzi B, Lee J (2019) N-GCN: multi-scale graph convolution for semi-supervised node classification. In: Amir Globerson and Ricardo Silva, editors, Proceedings of the thirty-fifth conference on uncertainty in artificial intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, vol 115 of Proceedings of machine learning research, pp 841–851. AUAI Press
Cao M, Yang M, Qin C, Zhu X, Chen Y, Wang J, Liu T (2021) Using deepgcn to identify the autism spectrum disorder from multi-site resting-state data. Biomed Signal Process. Control 70:103015
Chen D, Lin Y, Li W, Li P, Zhou J, Sun X (2020) Relieving the over-smoothing problem for graph neural networks from the topological view. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI Measuring New York, NY, USA, February 7-12, 2020, pp 3438–3445, AAAI Press, p 2020
Chen H, Huang Z, Xu Y, Deng Z, Huang F, He P, Li Z (2022) Neighbor enhanced graph convolutional networks for node classification and recommendation. Knowl Based Syst 246:108594
Chen H, Mao H, Li Y (2022) Elliptical convolution kernel: more real visual field. Neurocomputing 492:107–116
Chen J, Wang X, Xu X (2021) Gc-lstm: graph convolution embedded lstm for dynamic network link prediction. Applied Intelligence, pp 1–16
Chen M, Wei Z, Huang Z, Ding B, Li Y (2020) Simple and deep graph convolutional networks. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18, July 2020, Virtual event volume 119 of Proceedings of Machine Learning Research, pp 1725–1735, PMLR
Chen YL, Hsiao CH, Wu CC (2022) An ensemble model for link prediction based on graph embedding. Decis Support Syst 157:113753
Chong Y, Ding Y, Yan Q, Pan S (2020) Graph-based semi-supervised learning: a review. Neurocomputing 408:216–230
Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, volume 15 of JMLR Proceedings, pp 315–323, JMLR.org
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on computer vision and pattern recognition, CVPR 2016 Las Vegas, NV, USA, June 27-30, 2016, pp 770–778, IEEE Computer Society
Huang G, Liu Z, van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE Conference on computer vision and pattern recognition, CVPR 2017 Honolulu, HI, USA, July 21-26, 2017, pp 2261–2269, IEEE Computer Society
Huang Z, Tang Y, Chen Y (2022) A graph neural network-based node classification model on class-imbalanced graph data. Knowl Based Syst 244:108538
Kingma Diederik P, Adam JB (2015) A method for stochastic optimization. In: ICLR (Poster)
Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: 5th International conference on learning representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference track proceedings. OpenReview.net
Klicpera J, Bojchevski A, Günnemann S (2019) Predict then propagate: graph neural networks meet personalized pagerank. In: 7th International conference on learning representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net
Li B, Li Z, Yang Y (2021) Residual attention graph convolutional network for web services classification. Neurocomputing 440:45–57
Li G, Müller M, Qian G, Perez ICD, Abualshour A, Thabet AK, Ghanem B (2021) Deepgcns: making gcns go as deep as cnns. IEEE Transactions on Pattern Analysis and Machine Intelligence
Li K, Ye W (2022) Semi-supervised node classification via graph learning convolutional neural network. Applied Intelligence, pp 1–13
Li MW, Xu DY, Geng J, Hong WC (2022) A ship motion forecasting approach based on empirical mode decomposition method hybrid deep learning network and quantum butterfly optimization algorithm. Nonlinear Dyn 107(3):2447–2467
Li Q, Han Z, Wu XM (2018) Deeper insights into graph convolutional networks for semi-supervised learning. In: Sheila, A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp 3538–3545. AAAI Press
Liu M, Gao H, Ji S (2020) Towards deeper graph neural networks. In: Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp 338–348. ACM
Lu W, Zhan Y, Guan Z, Liu L, Yu B, Wei Z, Yang Y, Tao D (2021) Skipnode: on alleviating over-smoothing for deep graph convolutional networks. arXiv:2112.11628
Ma T, Pan Q, Wang H, Shao W, Tian Y, Al-Nabhan N (2020) Graph classification algorithm based on graph structure embedding. Expert Syst. Appl. 161:113715
Ma T, Wang H, Zhang L, Tian Y, Al-Nabhan N (2021) Graph classification based on structural features of significant nodes and spatial convolutional neural networks. Neurocomputing 423:639–650
Nasiri E, Berahmand K, Rostami M, Dabiri M (2021) A novel link prediction algorithm for protein-protein interaction networks by attributed graph embedding. Comput Biol Medicine 137:104772
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kȯpf A, Yang EZ, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: an imperative style, high-performance deep learning library. In: NeurIPS, pp 8024–8035
Qi C, Zhang J, Jia H, Mao Q, Wang L, Song H (2021) Deep face clustering using residual graph convolutional network. Knowl Based Syst 211:106561
Rong Y, Huang W, Xu T, Huang J (2020) Dropedge: towards deep graph convolutional networks on node classification. In: 8th International conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net
Sen P, Namata G, Bilgic M, Getoor L, Galligher B, Eliassi-Rad T (2008) Collective classification in network data. AI Mag 29(3):93–106
Wang Z, Li W, Su H (2021) Hierarchical attention link prediction neural network. Knowl Based Syst 232:107431
Wu Y, Song Y, Huang H, Ye F, Xie X, Jin H (2021) Enhancing graph neural networks via auxiliary training for semi-supervised node classification. Knowl Based Syst 220:106884
Yang Z, Cohen W, Salakhudinov R (2016) Revisiting semi-supervised learning with graph embeddings. In: ICML, vol 48 of JMLR Workshop and conference proceedings, pp 40–48. JMLR.org
Yin R, Li K, Zhang G, Lu J (2019) A deeper graph neural network for recommender systems. Knowl Based Syst, p 185
Yuan H, Ji S (2020) Structpool: structured graph pooling via conditional random fields. In: 8th International conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net
Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2021) Residual dense network for image restoration. IEEE Trans Pattern Anal Mach Intell 43(7):2480–2495
Zhao L, Akoglu L (2020) Pairnorm: tackling oversmoothing in gnns. In: 8th International conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net
Zhou K, Huang X, Li Y, Zha D, Chen R, Hu X (2020) Towards deeper graph neural networks with differentiable group normalization. In: Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in neural information processing systems 33: Annual conference on neural information processing systems 2020, NeurIPS 2020, December 6-12, 2020, virtual
Acknowledgements
This work was supported by the State Grid Corporation Headquarters Science and Technology Project under Grant 5700-202199539A-0-5-ZN.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix A: Hyper-parameters details
Appendix A: Hyper-parameters details
Our parameters mainly consist of the balance coefficients of the similarity constraint and disparity constraint. Other parameters remain the same as the original papers, and we use grid search method to select suitable parameters. We let \({\mathscr{L}}_{cons}, {\mathscr{L}}_{disp} \in \) [5e-6, 1e-6, 5e-5, 1e-5, 5e-4, 1e-4, 5e-3, 1e-3, 5e-2, 1e-2, 0.1], μ ∈[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. And in Table 4, we summarise the training configurations to reproduce the results of Table 2.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Chen, H., Li, Y. Multi-constraints in deep graph convolutional networks with initial residual. Appl Intell 53, 13608–13620 (2023). https://doi.org/10.1007/s10489-022-04222-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-022-04222-8