Skip to main content
SpringerLink
Log in

A deep reinforcement learning-based algorithm for reliability-aware multi-domain service deployment in smart ecosystems

  • S.I. : Deep Neuro-Fuzzy Analytics in Smart Ecosystems
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The transition towards full network virtualization will see services for smart ecosystems including smart metering, healthcare and transportation among others, being deployed as Service Function Chains (SFCs) comprised of an ordered set of virtual network functions. However, since such services are usually deployed in remote cloud networks, the SFCs may transcend multiple domains belonging to different Infrastructure Providers (InPs), possibly with differing policies regarding billing and Quality-of-service (QoS) guarantees. Therefore, efficiently allocating the exhaustible network resources to the different SFCs while meeting the stringent requirements of the services such as delay and QoS among others, remains a complex challenge, especially under limited information disclosure by the InPs. In this work, we formulate the SFC deployment problem across multiple domains focusing on delay constraints, and propose a framework for SFC orchestration which adheres to the privacy requirements of the InPs. Then, we propose a reinforcement learning (RL)-based algorithm for partitioning the SFC request across the different InPs while considering service reliability across the participating InPs. Such RL-based algorithms have the intelligence to infer undisclosed InP information from historical data obtained from past experiences. Simulation results, considering both online and offline scenarios, reveal that the proposed algorithm results in up to 10% improvement in terms of acceptance ratio and provisioning cost compared to the benchmark algorithms, with up to more than 90% saving in execution time for large networks. In addition, the paper proposes an enhancement to a state-of-the-art algorithm which results in up to 5% improvement in terms of provisioning cost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Sun G, Li Y, Liao D, Chang V (2018) Service function chain orchestration across multiple domains: a full mesh aggregation approach. IEEE Trans Netw Serv Manag 15(3):1175–1191. https://doi.org/10.1109/TNSM.2018.2861717

    Article  Google Scholar 

  2. Zhang P, Yao H, Liu Y (2018) Virtual network embedding based on computing, network, and storage resource constraints. IEEE Internet Things J 5(5):3298–3304. https://doi.org/10.1109/JIOT.2017.2726120

    Article  Google Scholar 

  3. Khebbache S, Hadji M, Zeghlache D (2017) Virtualized network functions chaining and routing algorithms. Comput Netw 114:95–110. https://doi.org/10.1016/j.comnet.2017.01.008

    Article  Google Scholar 

  4. Quang PTA, Bradai A, Singh KD, Picard G, Riggio R (2019) Single and multi-domain adaptive allocation algorithms for VNF forwarding graph embedding. IEEE Trans Netw Serv Manag 16(1):98–112. https://doi.org/10.1109/TNSM.2018.2876623

    Article  Google Scholar 

  5. Sun G, Yu H, Anand V, Li L (2013) A cost efficient framework and algorithm for embedding dynamic virtual network requests. Future Gener Comput Syst 29(5):1265–1277. https://doi.org/10.1016/j.future.2012.08.002

    Article  Google Scholar 

  6. Kaliyammal Thiruvasagam P, Kotagi VJ, Murthy CSR (2019) The more the merrier: enhancing reliability of 5G communication services with guaranteed delay. IEEE Netw Lett 1(2):52–55. https://doi.org/10.1109/lnet.2019.2902720

    Article  Google Scholar 

  7. Zhang P, Wang C, Qin Z, Cao H (2020) A multidomain virtual network embedding algorithm based on multiobjective optimization for Internet of Drones architecture in Industry 4.0. Software—Practice and Experience (October 2019), pp 1–19. https://doi.org/10.1002/spe.2815

  8. Kibalya G, Serrat J, Gorricho JL, Yao H, Zhang P (2020) A novel dynamic programming inspired algorithm for embedding of virtual networks in future networks. Comput Netw 179(May):107349. https://doi.org/10.1016/j.comnet.2020.107349

    Article  Google Scholar 

  9. Kibalya G, Serrat J, Gorricho JL, Pasquini R, Yao H, Zhang P (2019) A reinforcement learning based approach for 5G network slicing across multiple domains. In: 15th international conference on network and service management, CNSM 2019. https://doi.org/10.23919/CNSM46954.2019.9012674

  10. Leconte M, Paschos GS, Mertikopoulos P, Kozat UC (2018) A resource allocation framework for network slicing. In: Proceedings—IEEE INFOCOM 2018-April, pp 2177–2185. https://doi.org/10.1109/INFOCOM.2018.8486303

  11. Barakabitze AA, Ahmad A, Mijumbi R, Hines A (2020) 5G network slicing using SDN and NFV: a survey of taxonomy, architectures and future challenges. Comput Netw. https://doi.org/10.1016/j.comnet.2019.106984

    Article  Google Scholar 

  12. Kaur K, Garg S, Aujla GS, Kumar N, Rodrigues JJ, Guizani M (2018) Edge computing in the industrial internet of things environment: software-defined-networks-based edge-cloud interplay. IEEE Commun Mag 56(2):44–51. https://doi.org/10.1109/MCOM.2018.1700622

    Article  Google Scholar 

  13. Aujla GS, Kumar N, Zomaya AY, Ranjan R (2018) Optimal decision making for big data processing at edge-cloud environment: an SDN perspective. IEEE Trans Ind Inf 14(2):778–789. https://doi.org/10.1109/TII.2017.2738841

    Article  Google Scholar 

  14. Dietrich D, Abujoda A, Rizk A, Papadimitriou P (2017) Multi-provider service chain embedding with nestor. IEEE Trans Netw Serv Manag 14(1):91–105. https://doi.org/10.1109/TNSM.2017.2654681

    Article  Google Scholar 

  15. Dietrich D, Rizk A, Papadimitriou P (2015) Multi-provider virtual network embedding with limited information disclosure. IEEE Trans Netw Serv Manag 12(2):188–201. https://doi.org/10.1109/TNSM.2015.2417652

    Article  Google Scholar 

  16. Samuel F, Chowdhury M, Boutaba R (2013) PolyViNE: Policy-based virtual network embedding across multiple domains. J Internet Serv Appl 4(1):1–23. https://doi.org/10.1186/1869-0238-4-6

    Article  Google Scholar 

  17. Afolabi I, Bagaa M, Taleb T, Flinck H (2017) End-To-end network slicing enabled through network function virtualization. In: 2017 IEEE conference on standards for communications and networking, CSCN 2017, pp 30–35. https://doi.org/10.1109/CSCN.2017.8088594

  18. Shahriar N, Ahmed R, Chowdhury SR, Khan A, Boutaba R, Mitra J (2017) Generalized recovery from node failure in virtual network embedding. IEEE Trans Netw Serv Manag 14(2):261–274. https://doi.org/10.1109/TNSM.2017.2693404

    Article  Google Scholar 

  19. Ding W, Yu H, Luo S (2017) Enhancing the reliability of services in NFV with the cost-efficient redundancy scheme. IEEE Int Conf Commun 1:1–6. https://doi.org/10.1109/ICC.2017.7996840

    Article  Google Scholar 

  20. Beck MT, Botero JF, Samelin K (2017) Resilient allocation of service function chains. In: 2016 IEEE conference on network function virtualization and software defined networks, NFV-SDN 2016, pp 128–133. https://doi.org/10.1109/NFV-SDN.2016.7919487

  21. Cotroneo D, De Simone L, Iannillo AK, Lanzaro A, Natella R, Fan J, Ping W (2014) Network function virtualization: challenges and directions for reliability assurance. In: Proceedings—IEEE 25th international symposium on software reliability engineering workshops, ISSREW 2014, pp 37–42. https://doi.org/10.1109/ISSREW.2014.48

  22. Sun G, Xu Z, Yu H, Chen X, Chang V, Vasilakos AV (2019) Low-latency and resource-efficient service function chaining orchestration in network function virtualization. IEEE Internet Things J PP(c):1–1. https://doi.org/10.1109/jiot.2019.2937110

    Article  Google Scholar 

  23. Sun G, Zhu G, Liao D, Yu H, Du X, Guizani M (2019) Cost-efficient service function chain orchestration for low-latency applications in NFV networks. IEEE Syst J 13(4):3877–3888. https://doi.org/10.1109/JSYST.2018.2879883

    Article  Google Scholar 

  24. Zhang P, Yao H, Qiu C, Liu Y (2018) Virtual network embedding using node multiple metrics based on simplified electre method. IEEE Access 6:37314–37327. https://doi.org/10.1109/ACCESS.2018.2847910

    Article  Google Scholar 

  25. Di Mauro M, Longo M, Postiglione F (2018) Availability evaluation of multi-tenant service function chaining infrastructures by multidimensional universal generating function. IEEE Trans Serv Comput. https://doi.org/10.1109/TSC.2018.2885748

    Article  Google Scholar 

  26. Sun G, Liao D, Zhao D, Sun Z, Chang V (2018) Towards provisioning hybrid virtual networks in federated cloud data centers. Future Gener Comput Syst 87:457–469. https://doi.org/10.1016/j.future.2017.09.065

    Article  Google Scholar 

  27. Houidi I, Louati W, Ben Ameur W, Zeghlache D (2011) Virtual network provisioning across multiple substrate networks. Comput Netw 55(4):1011–1023. https://doi.org/10.1016/j.comnet.2010.12.011

    Article  MATH  Google Scholar 

  28. Zhang Q, Wang X, Kim I, Palacharla P, Ikeuchi T (2016) Vertex-centric computation of service function chains in multi-domain networks. In: IEEE NETSOFT 2016–2016 IEEE NetSoft conference and workshops: software-defined infrastructure for networks, clouds, IoT and services, pp 211–218. https://doi.org/10.1109/NETSOFT.2016.7502415

  29. Abujoda A, Papadimitriou P (2016) DistNSE: Distributed network service embedding across multiple providers. In: 2016 8th International conference on communication systems and networks, COMSNETS 2016 (i), pp 1–8. https://doi.org/10.1109/COMSNETS.2016.7439948

  30. Yao H, Chen X, Li M, Zhang P, Wang L (2018) A novel reinforcement learning algorithm for virtual network embedding. Neurocomputing 284:1–9. https://doi.org/10.1016/j.neucom.2018.01.025

    Article  Google Scholar 

  31. Mwanje SS, Schmelz LC, Mitschele-Thiel A (2016) Cognitive cellular networks: a Q-learning framework for self-organizing networks. IEEE Trans Netw Serv Manag 13(1):85–98. https://doi.org/10.1109/TNSM.2016.2522080

    Article  Google Scholar 

  32. Moysen J, Giupponi L (2018) From 4G to 5G: self-organized network management meets machine learning. Comput Commun 129(January):248–268. https://doi.org/10.1016/j.comcom.2018.07.015

    Article  Google Scholar 

  33. Zhang P, Wang C, Jiang C, Benslimane A (2020) Security-aware virtual network embedding algorithm based on reinforcement learning. IEEE Trans Netw Sci Eng 4697(c):1–11. https://doi.org/10.1109/TNSE.2020.2995863

    Article  Google Scholar 

  34. Singh M, Aujla GSS, Singh A, Kumar N, Garg S (2020) Deep learning based blockchain framework for secure software defined industrial networks. IEEE Trans Ind Inform 3203:1–1. https://doi.org/10.1109/tii.2020.2968946

    Article  Google Scholar 

  35. Houidi I, Louati W, Zeghlache D (2008) A distributed virtual network mapping algorithm. In: IEEE international conference on communications, pp 5634–5640. https://doi.org/10.1109/ICC.2008.1056

  36. Mahmoodi T, Van Helvoort H, Mansfield S (2017) Management and orchestration. IEEE Commun Stand Mag 1(4):60. https://doi.org/10.1109/MCOMSTD.2017.8258603

    Article  Google Scholar 

  37. Shen M, Xu K, Yang K, Chen HH (2014) Towards efficient virtual network embedding across multiple network domains. In: IEEE international workshop on quality of service, IWQoS, pp 61–70. https://doi.org/10.1109/IWQoS.2014.6914301

  38. Dietrich D, Rizk A, Papadimitriou P (2013) Multi-domain virtual network embedding with limited information disclosure. In: 2013 IFIP networking conference, IFIP networking 2013, 12(2), pp 188–201 (2013)

  39. Cao H, Zhu Y, Yang L, Zheng G (2017) A efficient mapping algorithm with novel node-ranking approach for embedding virtual networks. IEEE Access 5:22054–22066. https://doi.org/10.1109/ACCESS.2017.2761840

    Article  Google Scholar 

  40. Li S, Saidi MY, Chen K (2017) Multi-domainvirtualnetwork embedding with coordinated link mapping. Adv Sci Technol Eng Syst 2(3):545–552. https://doi.org/10.25046/aj020370

    Article  Google Scholar 

  41. Martin-Perez J, Bernardos CJ (2018) Multi-Domain VNF Mapping Algorithms. In: IEEE International symposium on broadband multimedia systems and broadcasting, BMSB 2018-June. https://doi.org/10.1109/BMSB.2018.8436765

  42. Xu Q, Gao D, Zhou H, Quan W, Shi W (2018) An energy-aware method for multi-domain service function chaining. J Internet Technol 19:1727–1739. https://doi.org/10.3966/160792642018111906010

    Article  Google Scholar 

  43. Boutigny F, Betgé-Brezetz S, Debar H, Blanc G, Lavignotte A, Popescu I (2018) Multi-provider secure virtual network embedding. In: 2018 9th IFIP international conference on new technologies, mobility and security, NTMS 2018—proceedings 2018-Janua, pp 1–5. https://doi.org/10.1109/NTMS.2018.8328706

  44. Sun G, Li Y, Zhu G, Liao D, Chang V (2018) Energy-efficient service function chain provisioning in multi-domain networks. In: IoTBDS 2018—Proceedings of the 3rd international conference on internet of things, big data and security 2018-March(January), pp 144–163. https://doi.org/10.5220/0006770301440152

  45. Sun G, Li Y, Yu H, Vasilakos AV, Du X, Guizani M (2019) Energy-efficient and traffic-aware service function chaining orchestration in multi-domain networks. Future Gener Comput Syst 91:347–360. https://doi.org/10.1016/j.future.2018.09.037

    Article  Google Scholar 

  46. Haeri S, Trajković L (2018) Virtual network embedding via Monte Carlo Tree search. IIEEE Trans Cyberneti 48(2):510–521. https://doi.org/10.1109/TCYB.2016.2645123

    Article  Google Scholar 

  47. Dolati M, Hassanpour SB, Ghaderi M, Khonsari A (2019) DeepViNE: Virtual network embedding with deep reinforcement learning. In: INFOCOM 2019—IEEE conference on computer communications workshops, INFOCOM WKSHPS 2019, pp 879–885. https://doi.org/10.1109/INFCOMW.2019.8845171

  48. Zhang Z, Su S, Lin Y, Cheng X, Shuang K, Xu P (2015) Adaptive multi-objective artificial immune system based virtual network embedding. J Netw Comput Appl 53:140–155. https://doi.org/10.1016/j.jnca.2015.03.007

    Article  Google Scholar 

  49. Pham TAQ, Hadjadj-aoul Y, Outtagarts A (2019) VNF-FG embedding: a deep reinforcement learning approach. IEEE Trans Netw Serv Manag 16(4):1318–1331. https://doi.org/10.1109/TNSM.2019.2947905

    Article  Google Scholar 

  50. Baggio G, Francescon A, Fedrizzi R (2017) Multi-domain service orchestration with X-MANO. In: 2017 IEEE conference on network softwarization: softwarization sustaining a hyper-connected world: en route to 5G, NetSoft 2017, pp 5–6. https://doi.org/10.1109/NETSOFT.2017.8004259

  51. Francescon A, Baggio G, Fedrizzi R, Ferrusy R, Ben Yahiaz I, Riggio R (2017) X–MANO: Cross–domain management and orchestration of network services, 2017. IEEE Conference on Network Softwarization (NetSoft), Bologna, 2017, pp 1–5. https://doi.org/10.1109/NETSOFT.2017.8004223

  52. Tusa F, Clayman S, Valocchi D, Galis A (2018) Multi-domain orchestration for the deployment and management of services on a slice enabled NFVI. In: 2018 IEEE conference on network function virtualization and software defined networks, NFV-SDN 2018, pp. 1–5. https://doi.org/10.1109/NFV-SDN.2018.8725769

  53. Cao H, Chen J, Guo Y, Zhu H, Yang L (2019) A novel and one-stage embedding algorithm for mapping virtual networks. In: 2018 24th Asia-Pacific conference on communications, APCC 2018, pp 156–161. https://doi.org/10.1109/APCC.2018.8633537

  54. Bianchi F, Presti FL (2017) A markov reward based resource-latency aware heuristic for the virtual network embedding problem. Perform Eval Rev 44(4):57–68. https://doi.org/10.1145/3092819.3092827

    Article  Google Scholar 

  55. Gong L, Wen Y, Zhu Z, Lee T (2014) Toward profit-seeking virtual network embedding algorithm via global resource capacity. In: Proceedings—IEEE INFOCOM, pp 1–9. https://doi.org/10.1109/INFOCOM.2014.6847918

  56. Amiri R, Mehrpouyan H, Fridman L, Mallik RK, Nallanathan A, Matolak D (2018) A machine learning approach for power allocation in HetNets considering QoS. In: IEEE international conference on communications 2018-May. https://doi.org/10.1109/ICC.2018.8422864

  57. Hejja K, Hesselbach X (2018) Online power aware coordinated virtual network embedding with 5G delay constraint. J Netw Comput Appl 124(February):121–136. https://doi.org/10.1016/j.jnca.2018.10.005

    Article  Google Scholar 

Download references

Acknowledgements

This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 777067 (NECOS project) and the national project TEC2015-71329-C2-2-R (MINECO/FEDER). This work is also supported by the “ Fundamental Research Funds for the Central Universities ” of China University of Petroleum (East China) under Grant 18CX02139A.

Author information

Authors and Affiliations

  1. Department of Network Engineering, Universitat Politecnica de Catalunya, C/ Jordi Girona, 1-3 - Edif.C3 - Campus Nord, 08034, Barcelona, Spain

    Godfrey Kibalya, Joan Serrat & Juan-Luis Gorricho

  2. Department of Electrical and Computer Engineering, Makerere University, Kampala, Uganda

    Dorothy Okello

  3. College of Computer Science and Technology, China University of Petroleum (East China), Qingdao, 266580, People’s Republic of China

    Peiying Zhang

Authors
  1. Godfrey Kibalya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joan Serrat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Juan-Luis Gorricho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dorothy Okello
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Peiying Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Godfrey Kibalya or Peiying Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kibalya, G., Serrat, J., Gorricho, JL. et al. A deep reinforcement learning-based algorithm for reliability-aware multi-domain service deployment in smart ecosystems. Neural Comput & Applic 35, 23795–23817 (2023). https://doi.org/10.1007/s00521-020-05372-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05372-x

Keywords

Search

Navigation