一种改进近端优化的多目标流QoS调度策略

doi:10.3969/j.issn.0255-8297.2024.03.011

Abstract

Abstract: Software-defined networking (SDN) can be equipped with flexible flow scheduling strategies to improve the quality of network service systems. However, as the complexity of business traffic increases, existing flow scheduling algorithms may suffer from performance degradation due to decreased scene matching. To address this problem, this paper proposes an intelligent routing strategy based on deep reinforcement learning. The strategy collects various link information through SDN, and implements feature extraction and state awareness based on long-short term memory networks and proximal policy optimization algorithms. The strategy generates dynamic flow scheduling strategies that meet quality of service (QoS) goals in business scenarios, thereby maximizing QoS. Experimental results show that the proposed scheme enhances the QoS index of the entire system by 7.06% compared to existing routing strategies, effectively improving the throughput of the business system.

Key words: deep reinforcement learning, quality of service, routing optimization, software defined network, online scheduling

CLC Number:

TP391.9

LIU Xingtong, ZHENG Hong, HUANG Jianhua. A Multi-objective Flow QoS Scheduling Strategy with Improved Proximal Optimization[J]. Journal of Applied Sciences, 2024, 42(3): 499-512.

References

[1] Agarwal S, Kodialam M, Lakshman T V. Traffic engineering in software defined networks [C]//2013 Proceedings IEEE INFOCOM, 2013: 2211-2219.
[2] Zhang Y, Wei Q S, Chen C, et al. Dynamic scheduling with service curve for QoS guarantee of large-scale cloud storage [J]. IEEE Transactions on Computers, 2018, 67(4): 457-468.
[3] Hartman T, Hassidim A, Kaplan H, et al. How to split a flow? [C]//2012 Proceedings IEEE INFOCOM, 2012: 828-836.
[4] 杨洋, 吕光宏, 赵会, 等. 深度学习在软件定义网络研究中的应用综述[J]. 软件学报, 2020, 31(7): 2184-2204. Yang Y, Lyu G H, Zhao H, et al. Survey on deep learning applications in software defined networking research [J]. Journal of Software, 2020, 31(7): 2184-2204. (in Chinese)
[5] Cui W Z, Qian C. Scalable and load-balanced data center multicast [C]//2015 IEEE Global Telecommunications Conference (GLOBECOM), 2015: 1-6.
[6] Chiesa M, Kindler G, Schapira M. Traffic engineering with equal-cost-multipath: an algorithmic perspective [J]. IEEE-ACM Transactions on Networking, 2017, 25(2): 779-792.
[7] Guo Z Y, Duan J, Yang Y Y. On-line multicast scheduling with bounded congestion in fattree data center networks [J]. IEEE Journal on Selected Areas in Communications, 2014, 32(1): 102-115.
[8] Lee M W, Li Y S, Huang X, et al. Robust multipath multicast routing algorithms for videos in software-defined networks [C]// IEEE International Symposium of Quality of Service (IWQoS), 2014: 218-227.
[9] 柯文龙, 王勇, 叶苗, 等. Ceph云存储网络中一种业务优先级区分的多播流调度方法[J]. 通信学报, 2020, 41(11): 40-51. Ke W L, Wang Y, Ye M, et al. Priority differentiated multicast flow scheduling method in Ceph cloud storage network [J]. Journal on Communications, 2020, 41(11): 40-51. (in Chinese)
[10] 孙茂鑫, 钱红燕, 陈兵, 等. SDN环境下基于MPTCP协议的切换管理[J]. 应用科学学报, 2017, 35(1): 117-127. Sun M X, Qian H Y, Chen B, et al. Handover management based on MPTCP in SDN environment [J]. Journal of Applied Sciences, 2017, 35(1): 117-127.(in Chinese)
[11] Li Y, Gao T, Yang J, et al. Phasic self-imitative reduction for sparse-reward goal-conditioned reinforcement learning [C]//International Conference on Machine Learning (ICML), 2022: 12765-12781.
[12] Su K, Lu Z. Divergence-regularized multi-agent actor-critic [C]//2022 International Conference on Machine Learning (ICML), 2022: 20580-20603.
[13] Lin S C, Akyildiz I F, Wang P, et al. QoS-aware adaptive routing in multi-layer hierarchical software defined networks: a reinforcement learning approach [C]//IEEE International Conference on Services Computing (SCC), 2016: 25-33.
[14] Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double Q-learning [J]. AAAI Conference on Artificial Intelligence, 2016, 30(1): 2094-2100.
[15] Yu C H, Lan J L, Guo Z H, et al. DROM: optimizing the routing in software-defined networks with deep reinforcement learning [J]. IEEE Access, 2018, 6: 64533-64539.
[16] Tan K, Bremner D, Le Kernec J, et al. Intelligent handover algorithm for vehicle-to-network communications with double-deep Q-learning [J]. IEEE Transactions on Vehicular Technology, 2022, 71(7): 7848-7862.
[17] 孙鹏浩, 兰巨龙, 申涓, 等. 基于牵引控制的深度强化学习路由策略生成[J]. 计算机研究与发展, 2021, 58(7): 1563-1572. Sun P H, Lan J L, Shen J, et al. Pinning control-based routing policy generation using deep reinforcement learning [J]. Journal of Computer Research and Development, 2021, 58(7): 1563-1572. (in Chinese)
[18] Saha N, Misra S, Bera S. Q-flag: QoS-aware flow-rule aggregation in software-defined IoT networks [J]. IEEE Internet of Things Journal, 2022, 9(7): 4899-4906.
[19] Kim G, Kim Y, Lim H. Deep reinforcement learning-based routing on software-defined networks [J]. IEEE Access, 2022, 10: 18121-18133.
[20] Mckeown N, Anderson T, Balakrishnan H, et al. OpenFlow: enabling innovation in campus networks [J]. ACM SIGCOMM Computer Communication Review, 2008, 38(2): 69- 74.
[21] Zhou W, Li L, Luo M, et al. REST API design patterns for SDN northbound API [C]//The 28th International Conference on Advanced Information Networking and Applications Workshops, 2014: 358-365.
[22] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms [DB/OL]. 2017[2023-03-07]. https://arxiv.org/abs/1707.06347.
[23] Xu Z Y, Tang J, Yin C X, et al. Experience-driven congestion control: when multi-path TCP meets deep reinforcement learning [J]. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1325-1336.
[24] 王帅, 董育宁, 李涛. 基于LSTM和特征生成的网络流量分类[J]. 应用科学学报, 2022, 40(5): 758-769. Wang S, Dong Y N, Li T. Network traffic classification based on LSTM and feature generation [J]. Journal of Applied Sciences, 2022, 40(5): 758-769. (in Chinese)
[25] Brandt S, Weil S. Ceph: reliable, scalable, and high-performance distributed storage [J]. Annals of Physics (Santa Cruz), 2007: 129-239.
[26] Macedo R, Paulo J, Pereira J, et al. A survey and classification of software-defined storage systems [J]. ACM Computing Surveys (CSUR), 2020, 53(3): 1-38.
[27] Knight S, Nguyen H X, Falkner N R, et al. The Internet topology zoo [J]. IEEE Journal on Selected Areas in Communications, 2011, 29(9): 1765-1775.

A Multi-objective Flow QoS Scheduling Strategy with Improved Proximal Optimization

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 8

Recommended Articles

Metrics

Comments

[1]	WANG Xin, ZHONG Weizhi, WANG Junzhi, XIAO Lijun, ZHU Qiuming. UAV Path Planning and Radio Mapping Based on Deep Reinforcement Learning [J]. Journal of Applied Sciences, 2024, 42(2): 200-210.
[2]	GE Xing, QIN Li, SHA Ying. Projected Reward for Multi-robot Formation and Obstacle Avoidance [J]. Journal of Applied Sciences, 2024, 42(1): 174-188.
[3]	PENG Kai, LIU Peichen, XU Xiaolong, ZHOU Xingyu. Computing Offloading of Multi-dependent Tasks in Smart Cities [J]. Journal of Applied Sciences, 2023, 41(3): 391-404.
[4]	ZHANG Kang, GAO Hong-hao, ZHU Yong-hua, XU Hua-hu. QoS Dynamic Web Services Composition Method Based on Improved Simulated Annealing Algorithm [J]. Journal of Applied Sciences, 2017, 35(5): 570-584.
[5]	LIAO Yong, TAN Xiao-heng, CHENG Jin-bo, ZHANG Yu-xiang, YANG Hong.etc. Modeling and Analysis of Communication Region of Primary Receiver in Cognitive Networks [J]. Journal of Applied Sciences, 2012, 30(2): 116-119.
[6]	XU Shuang, ZHAO Xiao-hui, YUAN Hao. Adaptive Resource Allocation Based on QoS in OFDM Cognitive Radio System [J]. Journal of Applied Sciences, 2011, 29(4): 381-384.
[7]	WU Yue, LI Jian-hua, BI Guang-guo. Call Admission Control Using Quality Degradation Strategy in Wireless Multimedia Networks [J]. Journal of Applied Sciences, 2006, 24(6): 656-660.
[8]	LI Jin-hui, SUN Xiao-han, ZHANG Ming-de, DING dong. The Implementation of Q Factor On-line Detection Based on DSP Technology in Optical Fiber LANs [J]. Journal of Applied Sciences, 2002, 20(1): 30-34.