Safety in DRL-Based Congestion Control: A Framework Empowered by Expert Refinement

doi:10.1109/TON.2025.3580436

	Safety in DRL-Based Congestion Control: A Framework Empowered by Expert Refinement
	Zhou, Jianer 1; Pan, Zhiyuan 2,3; Li, Zhenyu 4; Tyson, Gareth 5; Li, Weichao 1; Qiu, Xinyi 1; Pan, Heng 6; Zhang, Xinyi 6; Xie, Gaogang 6
	2025-06-25
发表期刊	IEEE TRANSACTIONS ON NETWORKING
页码	16
摘要	Deep reinforcement learning (DRL) has been used in congestion control algorithms (CCAs) for its ability to adapt to different network environments. However, its effectiveness is often hindered by the limited availability of training data and constrained training scales. While it has been proved that combining rule-based (expert) CCAs as a guide for DRL (namely hybrid CCAs) can address this limitation, we show through experimental measurements that rule-based CCAs potentially restrict action exploration of DRL models and may cause the DRL models to overly rely on them for higher reward gains. To address this gap, this paper proposes Marten, a framework that improves the effectiveness of rule-based CCAs for DRL. Marten's key innovations include an entropy-based dynamic exploration scheme that expands the exploration of DRL, and a reward adjustment scheme to prevent the DRL models' over-reliance on experts in hybrid CCAs. We have implemented Marten in both simulation platform OpenAI Gym and deployment platform QUIC. Experimental results in both emulated and production networks demonstrate Marten can improve throughput by 0.31% and reduce latency by 12.69% on average compared to the state-of-the-art hybrid CCAs. Compared to BBR, Marten achieves a 2.79% increase in throughput and an 11.73% reduction in latency on average.
关键词	Convergence Space exploration Safety Training Prediction algorithms Magnetosphere Ion radiation effects Adaptation models Throughput Technological innovation Congestion control machine learning deep reinforcement learning QUIC
DOI	10.1109/TON.2025.3580436
收录类别	SCI
语种	英语
资助项目	Major Key Project of PCL[PCL2025A02] ; Major Key Project of PCL[PCL2024Y02] ; PCL-CMCC Foundation for Science and Innovation[2024ZY1A0010] ; Young Scientists Fund of the National Natural Science Foundation of China[62202447]
WOS研究方向	Computer Science ; Engineering ; Telecommunications
WOS类目	Computer Science, Hardware & Architecture ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic ; Telecommunications
WOS记录号	WOS:001518836200001
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/42290
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Li, Zhenyu
作者单位	1.Pengcheng Lab, Shenzhen 518066, Peoples R China 2.Southern Univ Sci & Technol, Shenzhen 518055, Peoples R China 3.Tencent, Shenzhen, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Beijing 100045, Peoples R China 5.Hong Kong Univ Sci & Technol, Guangzhou, Peoples R China 6.Chinese Acad Sci, Comp Network Informat Ctr, Beijing 100045, Peoples R China
推荐引用方式 GB/T 7714	Zhou, Jianer,Pan, Zhiyuan,Li, Zhenyu,et al. Safety in DRL-Based Congestion Control: A Framework Empowered by Expert Refinement[J]. IEEE TRANSACTIONS ON NETWORKING,2025:16.
APA	Zhou, Jianer.,Pan, Zhiyuan.,Li, Zhenyu.,Tyson, Gareth.,Li, Weichao.,...&Xie, Gaogang.(2025).Safety in DRL-Based Congestion Control: A Framework Empowered by Expert Refinement.IEEE TRANSACTIONS ON NETWORKING,16.
MLA	Zhou, Jianer,et al."Safety in DRL-Based Congestion Control: A Framework Empowered by Expert Refinement".IEEE TRANSACTIONS ON NETWORKING (2025):16.