Continual learning-based trajectory prediction with memory augmented networks

Abstract

Forecasting pedestrian trajectories is widely used in mobile agents such as self-driving vehicles and social robots. Deep neural network-based trajectory prediction models precisely predict pedestrian trajectories after training. However, the prediction models fail to avoid catastrophic forgetting when the data distribution shifts during continual learning, making it incredible to deploy the models on agents in real environments. A continual trajectory prediction method with memory augmented networks, CLTP-MAN, is proposed to handle the catastrophic forgetting issue by introducing a memory augmented network with sparse experience replay. CLTP-MAN comprises an external memory module, a memory extraction module, and a trajectory prediction module. The external memory module contains prior knowledge useful for trajectory prediction. The memory extraction module can read or write the key–value memories with a trainable controller. Last, the trajectory prediction module performs long-trajectory prediction by introducing a multi-hop attention mechanism to extract pivotal information from the external memory. Meanwhile, the catastrophic forgetting issue is handled through sparse experience replay. The two benchmarking datasets ETH/UCY and SDD are reintegrated according to the needs for continual learning to conduct quantitative and qualitative evaluations. The results verify that benefiting from external memory and the multi-hop attention mechanism, CLTP-MAN has better generalization than several mainstream methods. Sparse experience replay effectively reduces catastrophic forgetting, leading to reliable deployments on mobile agents.