A long short-term memory architecture (LSTM) is a special type of recurrent neural network (RNN) designed to learn and remember information over long sequences of data.
Humans don’t restart their thoughts every second; those thoughts have persistence and build upon one another. That’s how people can read an explainer like this one and understand each word in sequence to create a longer coherent thought and build those into coherent knowledge. Neither traditional neural networks nor classic machine learning architectures do this; they can’t reason about previous events to inform later events. That makes working with linear data like a time series or text very difficult. RNNs address this issue by using a deep learning network architecture that contains loops which allow information to persist from one inference to the next.
Some of the most common applications of LSTM networks have been in natural language processing (NLP) and time series analysis. Other applications include speech recognition: many early voice assistants used LSTMs to transcribe spoken language into text, and in machine translation, where LSTM were once widely used for sentiment analysis and language modeling (now mostly replaced by transformer models). In time-series forecasting, LSTMs are still widely used for predicting future values in sequential data like stock prices and weather patterns. In healthcare, time series data can be analyzed to predict disease progression and treatment outcomes.
Since LSTMs are an extension of an RNN, it’s easiest to understand them through their predecessor.
Recurrent neural networks are a model architecture that focuses on learning from and predicting sequential data. They are different from other types of deep learning networks because they take information from prior inputs to influence how the current input is interpreted and what they should output.
Most deep learning networks assume that inputs and outputs are independent of each other and that information only flows forward as the network learns. An RNN on the other hand passes information about the previous state and the current input to each new node in the network. This architecture allows the model capture the information from previous steps and utilize it in the current step, enabling model to learn temporal dependencies and handle input of variable length.
Simple RNNs have long-term memory in the form of weights. Those weights change slowly during training, encoding general knowledge about the data. They also have short-term memory in the form of ephemeral activations, which pass from each node to successive nodes.