Catastrophic Forgetting in Deep Graph Networks: A Graph Classification Benchmark.

Antonio Carta,Federico Errica,Andrea Cossu,Davide Bacciu

doi:10.3389/frai.2022.824655

Antonio Carta, Federico Errica + Show 2 more

Open Access

https://doi.org/10.3389/frai.2022.824655

Copy DOI

Abstract

In this work, we study the phenomenon of catastrophic forgetting in the graph representation learning scenario. The primary objective of the analysis is to understand whether classical continual learning techniques for flat and sequential data have a tangible impact on performances when applied to graph data. To do so, we experiment with a structure-agnostic model and a deep graph network in a robust and controlled environment on three different datasets. The benchmark is complemented by an investigation on the effect of structure-preserving regularization techniques on catastrophic forgetting. We find that replay is the most effective strategy in so far, which also benefits the most from the use of regularization. Our findings suggest interesting future research at the intersection of the continual and graph representation learning fields. Finally, we provide researchers with a flexible software framework to reproduce our results and carry out further experiments.

Highlights

Building a robust machine learning model that incrementally learns from different tasks without forgetting requires methodologies that account for drifts in the input distribution
Continual learning is often considered as a necessary condition for the development of artificial intelligent agents operating in the real world
Even though there exist continual learning strategies loosely inspired by neuroscience, most of them do not take into consideration that we, as humans, act in a world filled with structured and complex interactions between different objects

Summary

Introduction

Building a robust machine learning model that incrementally learns from different tasks without forgetting requires methodologies that account for drifts in the input distribution. The Continual Learning (CL) research field addresses the catastrophic forgetting problem (Grossberg, 1980; French, 1999) by devising learning algorithms that improve a model’s ability to retain previously gathered information while learning across multiple steps. Each step in a CL scenario constitutes a new learning experience providing new data to the model, whose distribution may be different with respect to the previously encountered ones. The literature on CL revolves around three main families of strategies aimed at tackling catastrophic forgetting (Parisi et al, 2019): regularization strategies, architectural strategies and replay strategies. This taxonomy includes most of the currently used CL strategies

Methods

Results

Conclusion