Markovian Architectural Bias of Recurrent Neural Networks

Peter Tino, M Cernansky, L Benuskova

Research output: Contribution to journalArticle

154 Citations (Scopus)


In this paper, we elaborate upon the claim that clustering in the recurrent layer of recurrent neural networks (RNNs) reflects meaningful information processing states even prior to training [1], [2]. By concentrating on activation clusters in RNNs, while not throwing away the continuous state space network dynamics, we extract predictive models that we call neural prediction machines (NPMs). When RNNs with sigmoid activation functions are initialized with small weights (a common technique in the RNN community), the clusters of recurrent activations emerging prior to training are indeed meaningful and correspond to Markov prediction contexts. In this case, the extracted NPMs correspond to a class of Markov models, called variable memory length Markov models (VLMMs). In order to appreciate how much information has really been induced during the training, the RNN performance should always be compared with that of VLMMs and NPMs extracted before training as the "null" base models. Our arguments are supported by experiments on a chaotic symbolic sequence and a context-free language with a deep recursive structure. Index Terms-Complex symbolic sequences, information latching problem, iterative function systems, Markov models, recurrent neural networks (RNNs).
Original languageEnglish
Pages (from-to)6-15
Number of pages10
JournalIEEE Transactions on Neural Networks
Issue number1
Publication statusPublished - 1 Jan 2004


  • iterative function systems
  • Markov models
  • information latching problem
  • complex symbolic sequences
  • recurrent neural networks (RNNs)


Dive into the research topics of 'Markovian Architectural Bias of Recurrent Neural Networks'. Together they form a unique fingerprint.

Cite this