Are multi-layer backpropagation networks catastrophically amnesic?

Connectionist models with a backpropagation learning rule are known to have a serious problem. Such models exhibit catastrophic interference (or forgetting) with sequential training. Having learned a set of patterns, if the model is trained on another set of patterns, its performance on the first set can dramatically deteriorate very rapidly. The present study reconsiders this issue with four simulations. The model learned arithmetic facts sequentially, but the interference was only modest with random (hence approximately orthogonal) inputs. Essentially the same result was obtained when the inputs are made less orthogonal by adding irrelevant elements. Reducing the number of hidden units did not have major effects. This study suggests that the interference problem has been somewhat overstated.