Generalizable Resource Scaling of 5G Slices using Constrained Reinforcement Learning (2306.09290v1)
Abstract: Network slicing is a key enabler for 5G to support various applications. Slices requested by service providers (SPs) have heterogeneous quality of service (QoS) requirements, such as latency, throughput, and jitter. It is imperative that the 5G infrastructure provider (InP) allocates the right amount of resources depending on the slice's traffic, such that the specified QoS levels are maintained during the slice's lifetime while maximizing resource efficiency. However, there is a non-trivial relationship between the QoS and resource allocation. In this paper, this relationship is learned using a regression-based model. We also leverage a risk-constrained reinforcement learning agent that is trained offline using this model and domain randomization for dynamically scaling slice resources while maintaining the desired QoS level. Our novel approach reduces the effects of network modeling errors since it is model-free and does not require QoS metrics to be mathematically formulated in terms of traffic. In addition, it provides robustness against uncertain network conditions, generalizes to different real-world traffic patterns, and caters to various QoS metrics. The results show that the state-of-the-art approaches can lead to QoS degradation as high as 44.5% when tested on previously unseen traffic. On the other hand, our approach maintains the QoS degradation below a preset 10% threshold on such traffic, while minimizing the allocated resources. Additionally, we demonstrate that the proposed approach is robust against varying network conditions and inaccurate traffic predictions.
- Telecom Italia, “Telecommunications - SMS, Call, Internet - MI,” 2015. [Online]. Available: https://doi.org/10.7910/DVN/EGZHFV
- Y. Liu, J. Ding, and X. Liu, “A constrained reinforcement learning based approach for network slicing,” in IEEE International Conference on Network Protocols (ICNP), 2020, pp. 1–6.
- R. Li, Z. Zhao, Q. Sun, C.-L. I, C. Yang, X. Chen, M. Zhao, and H. Zhang, “Deep reinforcement learning for resource management in network slicing,” IEEE Access, vol. 6, pp. 74 429–74 441, 2018.
- J. Li, J. Liu, T. Huang, and Y. Liu, “DRA-IG: the balance of performance isolation and resource utilization efficiency in network slicing,” in IEEE International Conference on Communications (ICC), 2020, pp. 1–6.
- J. X. Salvat, L. Zanzi, A. Garcia-Saavedra, V. Sciancalepore, and X. Costa-Perez, “Overbooking network slices through yield-driven end-to-end orchestration,” in ACM International Conference on emerging Networking EXperiments and Technologies (CoNEXT), 2018, pp. 353–365.
- F. Y. Yan, J. Ma, G. D. Hill, D. Raghavan, R. S. Wahby, P. Levis, and K. Winstein, “Pantheon: the training ground for internet congestion-control research,” in USENIX Annual Technical Conference (ATC), 2018, pp. 731–743.
- Q. Liu, N. Choi, and T. Han, “Constraint-aware deep reinforcement learning for end-to-end resource orchestration in mobile networks,” in IEEE International Conference on Network Protocols (ICNP), 2021, pp. 1–11.
- Y. Hua, R. Li, Z. Zhao, X. Chen, and H. Zhang, “GAN-powered deep distributional reinforcement learning for resource management in network slicing,” IEEE Journal on Selected Areas in Communications, vol. 38, no. 2, pp. 334–349, 2019.
- Q. Liu, N. Choi, and T. Han, “OnSlicing: online end-to-end network slicing with reinforcement learning,” in ACM International Conference on emerging Networking EXperiments and Technologies (CoNEXT), 2021, pp. 141–153.
- Q. Yang, T. D. Simão, S. H. Tindemans, and M. T. J. Spaan, “WCSAC: worst-case soft actor critic for safety-constrained reinforcement learning,” in AAAI Conference on Artificial Intelligence, vol. 35, no. 12, 2021, pp. 10 639–10 646.
- M. Xu, Z. Liu, P. Huang, W. Ding, Z. Cen, B. Li, and D. Zhao, “Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability,” 2022. [Online]. Available: https://arxiv.org/abs/2209.08025
- J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” 2017. [Online]. Available: https://arxiv.org/abs/1705.10528
- Y. Liu, A. Halev, and X. Liu, “Policy learning with constraints in model-free reinforcement learning: A survey,” in International Joint Conference on Artificial Intelligence (IJCAI), 2021.
- S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan, “Learning to walk in the real world with minimal human effort,” 2020. [Online]. Available: https://arxiv.org/abs/2002.08550
- C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,” 2018. [Online]. Available: https://arxiv.org/abs/1805.11074
- J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in International Conference on Machine Learning (ICML), 2015, pp. 1889–1897.
- S. Kullback and R. A. Leibler, “On information and sufficiency,” The Annals of Mathematical Statistics, vol. 22, no. 1, pp. 79–86, 1951.
- A. Papa, M. Klugel, L. Goratti, T. Rasheed, and W. Kellerer, “Optimizing dynamic RAN slicing in programmable 5G networks,” in IEEE International Conference on Communications (ICC), 2019, pp. 1–7.
- A. T. Z. Kasgari and W. Saad, “Stochastic optimization and control framework for 5G network slicing with effective isolation,” in IEEE Annual Conference on Information Sciences and Systems (CISS), 2018, pp. 1–6.
- O. Triebe, H. Hewamalage, P. Pilyugina, N. Laptev, C. Bergmeir, and R. Rajagopal, “NeuralProphet: Explainable forecasting at scale,” 2021. [Online]. Available: https://arxiv.org/abs/2111.15397
- F. D. Johansson, U. Shalit, and D. Sontag, “Learning representations for counterfactual inference,” 2016. [Online]. Available: https://arxiv.org/abs/1605.03661
- “Introduction — SD-RAN Docs 1.4.1-dev documentation,” https://docs.sd-ran.org/sdran-1.4/introduction.html, [Accessed 29-Sep-2022].
- J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
