Concurrent Transformer for Spatial-Temporal Graph Modeling

Abstract

Previous studies have shown that concurrently extracting spatial and temporal information is a better way to model spatial-temporal data. However, in these studies, the receptive field has been fixed to construct the carrier of concurrent extraction, resulting in the lack of flexibility in selecting receptive fields, and the loss of scalability for capturing long-range temporal dependencies. Furthermore, these studies will result in static weights that are insufficient to describe the complex spatial and temporal dependencies. In this paper, we propose the Concurrent Spatial-Temporal Transformer (CSTT), which ensures the denseness of the carrier of concurrent extraction, so that messages under different receptive fields can be passed more efficiently, thus making the selection of receptive field more flexible. Therefore, the scalability of capturing long-range temporal dependencies can also be promised. Additionally, a unified self-attention mechanism will be applied on the carrier of concurrent extraction to capture spatial and temporal information, so that the dependencies between both dimensions under different contextual information can be preserved. On these bases, we design an iterative strategy to further tackle long sequences. Experiments on four traffic real-world datasets illustrate that our algorithm can achieve significant improvement on the classical spatial-temporal modeling task.