一种改进近端优化的多目标流QoS调度策略

doi:10.3969/j.issn.0255-8297.2024.03.011

摘要/Abstract

摘要： 软件定义网络可以搭载灵活的流调度策略来提升网络服务系统的服务质量，但随着业务流量复杂度的提升，现有的流调度算法会因场景匹配度的下降而导致性能受到影响。为此提出一种基于深度强化学习的智能路由策略。该策略通过软件定义网络收集各链路信息，基于长短期记忆网络与近端策略优化算法实现特征提取与状态感知，最终决策生成符合业务场景下服务质量（quality of service,QoS）目标的动态流量调度策略，并实现QoS最大化。实验结果表明，所提的方案与现有的路由策略相比可以使整套系统QoS指标提升7.06%，有效地提升了业务系统的吞吐率。

关键词: 深度强化学习, 服务质量, 路由优化, 软件定义网络, 在线调度

Abstract: Software-defined networking (SDN) can be equipped with flexible flow scheduling strategies to improve the quality of network service systems. However, as the complexity of business traffic increases, existing flow scheduling algorithms may suffer from performance degradation due to decreased scene matching. To address this problem, this paper proposes an intelligent routing strategy based on deep reinforcement learning. The strategy collects various link information through SDN, and implements feature extraction and state awareness based on long-short term memory networks and proximal policy optimization algorithms. The strategy generates dynamic flow scheduling strategies that meet quality of service (QoS) goals in business scenarios, thereby maximizing QoS. Experimental results show that the proposed scheme enhances the QoS index of the entire system by 7.06% compared to existing routing strategies, effectively improving the throughput of the business system.

Key words: deep reinforcement learning, quality of service, routing optimization, software defined network, online scheduling

中图分类号:

TP391.9

刘星彤, 郑红, 黄建华. 一种改进近端优化的多目标流QoS调度策略[J]. 应用科学学报, 2024, 42(3): 499-512.

LIU Xingtong, ZHENG Hong, HUANG Jianhua. A Multi-objective Flow QoS Scheduling Strategy with Improved Proximal Optimization[J]. Journal of Applied Sciences, 2024, 42(3): 499-512.

参考文献

[1] Agarwal S, Kodialam M, Lakshman T V. Traffic engineering in software defined networks [C]//2013 Proceedings IEEE INFOCOM, 2013: 2211-2219.
[2] Zhang Y, Wei Q S, Chen C, et al. Dynamic scheduling with service curve for QoS guarantee of large-scale cloud storage [J]. IEEE Transactions on Computers, 2018, 67(4): 457-468.
[3] Hartman T, Hassidim A, Kaplan H, et al. How to split a flow? [C]//2012 Proceedings IEEE INFOCOM, 2012: 828-836.
[4] 杨洋, 吕光宏, 赵会, 等. 深度学习在软件定义网络研究中的应用综述[J]. 软件学报, 2020, 31(7): 2184-2204. Yang Y, Lyu G H, Zhao H, et al. Survey on deep learning applications in software defined networking research [J]. Journal of Software, 2020, 31(7): 2184-2204. (in Chinese)
[5] Cui W Z, Qian C. Scalable and load-balanced data center multicast [C]//2015 IEEE Global Telecommunications Conference (GLOBECOM), 2015: 1-6.
[6] Chiesa M, Kindler G, Schapira M. Traffic engineering with equal-cost-multipath: an algorithmic perspective [J]. IEEE-ACM Transactions on Networking, 2017, 25(2): 779-792.
[7] Guo Z Y, Duan J, Yang Y Y. On-line multicast scheduling with bounded congestion in fattree data center networks [J]. IEEE Journal on Selected Areas in Communications, 2014, 32(1): 102-115.
[8] Lee M W, Li Y S, Huang X, et al. Robust multipath multicast routing algorithms for videos in software-defined networks [C]// IEEE International Symposium of Quality of Service (IWQoS), 2014: 218-227.
[9] 柯文龙, 王勇, 叶苗, 等. Ceph云存储网络中一种业务优先级区分的多播流调度方法[J]. 通信学报, 2020, 41(11): 40-51. Ke W L, Wang Y, Ye M, et al. Priority differentiated multicast flow scheduling method in Ceph cloud storage network [J]. Journal on Communications, 2020, 41(11): 40-51. (in Chinese)
[10] 孙茂鑫, 钱红燕, 陈兵, 等. SDN环境下基于MPTCP协议的切换管理[J]. 应用科学学报, 2017, 35(1): 117-127. Sun M X, Qian H Y, Chen B, et al. Handover management based on MPTCP in SDN environment [J]. Journal of Applied Sciences, 2017, 35(1): 117-127.(in Chinese)
[11] Li Y, Gao T, Yang J, et al. Phasic self-imitative reduction for sparse-reward goal-conditioned reinforcement learning [C]//International Conference on Machine Learning (ICML), 2022: 12765-12781.
[12] Su K, Lu Z. Divergence-regularized multi-agent actor-critic [C]//2022 International Conference on Machine Learning (ICML), 2022: 20580-20603.
[13] Lin S C, Akyildiz I F, Wang P, et al. QoS-aware adaptive routing in multi-layer hierarchical software defined networks: a reinforcement learning approach [C]//IEEE International Conference on Services Computing (SCC), 2016: 25-33.
[14] Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double Q-learning [J]. AAAI Conference on Artificial Intelligence, 2016, 30(1): 2094-2100.
[15] Yu C H, Lan J L, Guo Z H, et al. DROM: optimizing the routing in software-defined networks with deep reinforcement learning [J]. IEEE Access, 2018, 6: 64533-64539.
[16] Tan K, Bremner D, Le Kernec J, et al. Intelligent handover algorithm for vehicle-to-network communications with double-deep Q-learning [J]. IEEE Transactions on Vehicular Technology, 2022, 71(7): 7848-7862.
[17] 孙鹏浩, 兰巨龙, 申涓, 等. 基于牵引控制的深度强化学习路由策略生成[J]. 计算机研究与发展, 2021, 58(7): 1563-1572. Sun P H, Lan J L, Shen J, et al. Pinning control-based routing policy generation using deep reinforcement learning [J]. Journal of Computer Research and Development, 2021, 58(7): 1563-1572. (in Chinese)
[18] Saha N, Misra S, Bera S. Q-flag: QoS-aware flow-rule aggregation in software-defined IoT networks [J]. IEEE Internet of Things Journal, 2022, 9(7): 4899-4906.
[19] Kim G, Kim Y, Lim H. Deep reinforcement learning-based routing on software-defined networks [J]. IEEE Access, 2022, 10: 18121-18133.
[20] Mckeown N, Anderson T, Balakrishnan H, et al. OpenFlow: enabling innovation in campus networks [J]. ACM SIGCOMM Computer Communication Review, 2008, 38(2): 69- 74.
[21] Zhou W, Li L, Luo M, et al. REST API design patterns for SDN northbound API [C]//The 28th International Conference on Advanced Information Networking and Applications Workshops, 2014: 358-365.
[22] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms [DB/OL]. 2017[2023-03-07]. https://arxiv.org/abs/1707.06347.
[23] Xu Z Y, Tang J, Yin C X, et al. Experience-driven congestion control: when multi-path TCP meets deep reinforcement learning [J]. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1325-1336.
[24] 王帅, 董育宁, 李涛. 基于LSTM和特征生成的网络流量分类[J]. 应用科学学报, 2022, 40(5): 758-769. Wang S, Dong Y N, Li T. Network traffic classification based on LSTM and feature generation [J]. Journal of Applied Sciences, 2022, 40(5): 758-769. (in Chinese)
[25] Brandt S, Weil S. Ceph: reliable, scalable, and high-performance distributed storage [J]. Annals of Physics (Santa Cruz), 2007: 129-239.
[26] Macedo R, Paulo J, Pereira J, et al. A survey and classification of software-defined storage systems [J]. ACM Computing Surveys (CSUR), 2020, 53(3): 1-38.
[27] Knight S, Nguyen H X, Falkner N R, et al. The Internet topology zoo [J]. IEEE Journal on Selected Areas in Communications, 2011, 29(9): 1765-1775.