Using a Deep Reinforcement Learning Agent for Traffic Signal Control (1611.01142v1)

Abstract: Ensuring transportation systems are efficient is a priority for modern society. Technological advances have made it possible for transportation systems to collect large volumes of varied data on an unprecedented scale. We propose a traffic signal control system which takes advantage of this new, high quality data, with minimal abstraction compared to other proposed systems. We apply modern deep reinforcement learning methods to build a truly adaptive traffic signal control agent in the traffic microsimulator SUMO. We propose a new state space, the discrete traffic state encoding, which is information dense. The discrete traffic state encoding is used as input to a deep convolutional neural network, trained using Q-learning with experience replay. Our agent was compared against a one hidden layer neural network traffic signal control agent and reduces average cumulative delay by 82%, average queue length by 66% and average travel time by 20%.

• The paper introduces a deep reinforcement learning agent that employs a novel DTSE to reduce traffic delays and improve signal timing.
• It uses a deep convolutional neural network, trained with Q-learning and experience replay, to map detailed traffic states to optimal light configurations.
• Results show reductions of 82% in cumulative delay, 66% in queue length, and 20% in travel time compared to traditional control methods.

Using a Deep Reinforcement Learning Agent for Traffic Signal Control

Abstract

The paper addresses the enhancement of traffic signal control systems through the application of deep reinforcement learning (DRL). By leveraging modern advancements in DRL, an adaptive traffic signal control agent is proposed and implemented within the traffic microsimulator SUMO. A novel state space representation, the discrete traffic state encoding (DTSE), enables the system to incorporate detailed and relevant traffic information. This enhanced representation is fed into a deep convolutional neural network (CNN) which is trained using Q-learning with experience replay. The proposed DRL-based traffic signal control agent is demonstrated to significantly reduce average cumulative delay, queue length, and travel time compared to a conventional neural network-based agent.

Introduction

The challenge of optimizing urban traffic signal control is addressed by leveraging artificial intelligence techniques, specifically deep reinforcement learning. Due to the limitations of expanding road infrastructure, the focus is on improving the efficiency of existing traffic management systems. Reinforcement learning, with its agent-environment interaction framework, presents a suitable approach for traffic signal optimization. The proposed system seeks to better exploit the available data with minimal abstraction by introducing a dense state space representation and leveraging state-of-the-art deep learning architectures.

Reinforcement Learning Framework

The traffic signal control problem is effectively modeled as a Markov Decision Process, wherein the state of the environment is represented by the newly proposed DTSE. This representation captures detailed traffic dynamics including vehicle presence, speed, and the ongoing traffic signal phase. The action space is defined by the possible traffic light configurations, and the reward function is based on the change in cumulative vehicle delay, aiming to minimize traffic congestion.

Agent Design and Learning

The deep Q-network traffic signal control agent (DQTSCA) utilizes a deep CNN for approximating the action-value function, which maps observed states to action values, aiding in decision-making that maximizes the long-term cumulative reward. The network's architecture consists of convolutional layers followed by fully connected layers, designed to process the rich information contained in the DTSE. The agent's learning process incorporates experience replay to stabilize training, and an $\epsilon$-greedy strategy is employed for balancing exploration and exploitation during learning.

Experimental Setup

The experiments are conducted using SUMO, a traffic microsimulator, to validate the effectiveness of the proposed system. The DQTSCA is benchmarked against a traditional shallow neural network-based controller in a simulated traffic intersection. Various performance metrics, including vehicle throughput, queue length, travel time, and cumulative delay are evaluated to assess the system performance improvements brought by the deep learning approach.

Results

The DQTSCA dramatically reduces average cumulative delay by 82%, queue length by 66%, and travel time by 20% compared to the baseline model. The demonstrated improvements underscore the potential for deep RL to transform traffic management systems, providing valuable insights for real-world applications.

Conclusion

The paper showcases the feasibility and benefits of applying deep reinforcement learning for traffic signal control, yielding substantial improvements in traffic efficiency metrics. Future research directions include enhancing the agent's autonomy over yellow and red phases, scaling the approach to more complex traffic networks, and balancing fairness with optimality in traffic management policies.