Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use case (1910.07421v3)

Abstract: Deep Reinforcement Learning (DRL) has shown a dramatic improvement in decision-making and automated control problems. Consequently, DRL represents a promising technique to efficiently solve many relevant optimization problems (e.g., routing) in self-driving networks. However, existing DRL-based solutions applied to networking fail to generalize, which means that they are not able to operate properly when applied to network topologies not observed during training. This lack of generalization capability significantly hinders the deployment of DRL technologies in production networks. This is because state-of-the-art DRL-based networking solutions use standard neural networks (e.g., fully connected, convolutional), which are not suited to learn from information structured as graphs. In this paper, we integrate Graph Neural Networks (GNN) into DRL agents and we design a problem specific action space to enable generalization. GNNs are Deep Learning models inherently designed to generalize over graphs of different sizes and structures. This allows the proposed GNN-based DRL agent to learn and generalize over arbitrary network topologies. We test our DRL+GNN agent in a routing optimization use case in optical networks and evaluate it on 180 and 232 unseen synthetic and real-world network topologies respectively. The results show that the DRL+GNN agent is able to outperform state-of-the-art solutions in topologies never seen during training.

• The paper integrates Deep Reinforcement Learning with Graph Neural Networks to create a routing optimization agent that generalizes better across diverse network topologies and network states.
• Experimental results show the DRL+GNN agent outperforms conventional DRL techniques in generalizing to unseen network configurations and maintaining performance.
• The proposed system demonstrates robustness against network failures like link failures and exhibits scalability across various network sizes and structural characteristics.

Deep Reinforcement Learning and Graph Neural Networks for Routing Optimization

The paper presents an integration of Deep Reinforcement Learning (DRL) with Graph Neural Networks (GNN) to address the routing optimization problem in Optical Transport Networks (OTN). This combination aims to overcome the generalization limitation experienced by traditional neural network architectures when applied to unseen network topologies. The research investigates whether DRL agents can effectively learn and generalize optimal routing strategies over a diverse range of network configurations, without requiring specific tuning for each topology.

Summary of Contributions

1. Integration of GNN with DRL:
   • The research introduces GNN as the underlying architecture for DRL agents. This approach leverages the GNN's ability to work inherently with graph-structured data, making it suitable for networking scenarios where topologies are naturally represented as graphs.
2. Routing Optimization Use Case:
   • The paper specifically focuses on routing optimization within OTNs, where DRL is used for making real-time routing decisions based on incoming traffic demands. The proposed system demonstrates the ability to operate efficiently across various network states, demonstrating superior adaptability.
3. Experimental Evaluation and Results:
   • The DRL+GNN agent is evaluated against state-of-the-art DRL solutions trained and tested in multiple network topologies. Results indicate that the DRL+GNN agent can generalize better to network configurations unseen during the training phase and achieves performance improvements over conventional DRL techniques.
4. Robustness Against Network Failures:
   • A noteworthy use case explored in the paper is the DRL+GNN agent's resilience to link failures. The agent effectively adapts to topological changes and maintains performance, exhibiting robust operational characteristics.
5. Scalability and Generalization Capabilities:
   • The paper examines the scalability of the DRL+GNN architecture across synthetic and real-world network topologies, with varying sizes and structural characteristics. The findings emphasize that the proposed system scales gracefully, retaining computational efficiency and effective routing performance even in larger and more complex networks.

Implications and Future Research

The integration of GNNs into DRL agents for network optimization offers practical advantages in designing self-adaptive, self-driving networks capable of dynamic, scalable operation without extensive retraining for each unique network configuration. While the experiments primarily explore routing within OTNs, this approach may be extended to other domains of network optimization where graph structure representation is prevalent.

Future research can delve into enhancing GNN models to improve generalization further by training across a broader diversity of network topologies. Additionally, it could explore leveraging advanced DRL frameworks to bolster decision-making processes in networking scenarios with varied dynamic contexts, such as fluctuating traffic patterns and topology changes.

This research sets a foundation for developing DRL-based networking solutions that can be deployed as ready-to-operate products, simplifying network management and improving throughput and latency metrics with minimal manual oversight. It reflects a significant step toward autonomous network optimization, balancing the challenges of computational overhead and practical scalability across diverse network environments.