Echo state networks in forecasting chaotic dynamics and emergent universalities

Abstract

Deep learning has become a popular way to accurately recognise, classify, generate and forecast data in various complex scenarios. It is commonly based on artificial neural networks, a type of machine learning architecture inspired by the human brain, that consist of multiple layers of neurons with non-linear activations. Their inside workings can be considered as a ‘black box’ because it is difficult, if not impossible, to truly understand “how” or “why” such networks work. Furthermore, the training relies on gradient decent methods which are inherently difficult and computationally expensive. Reservoir Computing has recently emerged as a new paradigm aimed to alleviate such difficulties. In this thesis, we study its particular realisation, known as Echo State Networks (ESNs), which show promise in many tasks, particularly in forecasting the dynamics of chaotic systems. The thesis will provide a detailed discussion of the ESN architecture, hyperparameters, and implementation. We introduce a new performance metric that highlights the networks maintaining small errors for the longest duration and use it in GridSearch to optimise ESN hyperparameters. We study the statistical properties of the correlation matrix that is formed during training, offering a novel approach not applied to ESNs or other types of recurrent networks. Our extensive numerical studies reveal universal results, consistent across tests and different chaotic training signals. Although analytical studies are limited, we introduce a simple ‘toy model’ to qualitatively describe some properties. We conclude our work with study of the statistics of trained output weights, which also exhibit universal characteristics. These universalities provide deeper insights into the inner workings and behaviour of ESNs, enhancing our understanding of how information spreads throughout the network.