How do LSTM and GRU work?

Authors
How do LSTM and GRU work?

LSTM and GRU are special kinds of neural networks that read data one step at a time and keep a memory of what matters. They use small "gates" that decide what to remember, what to forget, and what to use, so they can remember important things for a long time.

In this blog, we will learn about how LSTM and GRU work. We will also see how a simple Recurrent Neural Network reads a sequence and why it forgets things, what gates are, how the three gates of an LSTM work together, how a GRU does the same job with fewer gates, how to use both in code, and when to use which one.

We will cover the following:

  • What is sequence data?
  • What is a Recurrent Neural Network?
  • The problem: an RNN forgets
  • What is a gate?
  • What is an LSTM?
  • The cell state
  • The forget gate
  • The input gate
  • The output gate
  • An LSTM step with numbers
  • Why does an LSTM remember for a long time?
  • What is a GRU?
  • The update gate and the reset gate
  • LSTM and GRU in code
  • LSTM vs GRU
  • Where are LSTM and GRU used today?

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is sequence data?

Before jumping into LSTM and GRU, we must know about sequence data, because LSTM and GRU are made to work on it.

Sequence data is data where the order matters.

Let's see a few examples:

  • A sentence: "The dog bit the man" and "The man bit the dog" have the same words, but a different meaning because of the order.
  • Daily temperatures over a month.
  • Stock prices over a year.
  • Sound, which is a long series of values over time.

To understand sequence data, a program must read it in order and remember what came before.

What is a Recurrent Neural Network?

A normal neural network looks at the whole input at once and has no memory. It cannot read step by step.

A Recurrent Neural Network (RNN) is a neural network that reads a sequence one step at a time and carries a small memory from one step to the next.

Recurrent means "happening again and again". The same network runs again and again, once for each step.

This small memory is called the hidden state. It is just a list of numbers that summarizes what the network has read so far.

 "The"      "cat"      "sat"
   |          |          |
   ↓          ↓          ↓
 [RNN] ---> [RNN] ---> [RNN] ---> final memory
       h1         h2         h3

Here, at each step, the RNN takes two things: the current word and the memory from the previous step. It mixes them and produces a new memory. h1, h2, and h3 are the hidden states after each word.

We have a detailed blog on Recurrent Neural Network that explains this in depth.

The problem: an RNN forgets

The simple RNN works well for short sequences. But for long sequences, it forgets the early parts.

Let's say we have this sentence:

"Riya grew up in France, she moved to India for her job ten years ago, she loves cricket now, but at home she still speaks fluent ___."

The answer is "French". To get it, the network must remember "France" from the very beginning.

In a simple RNN, the whole memory is rewritten at every step. The new memory is the old memory mixed with the new word. After many steps, the information about "France" gets mixed and diluted so many times that almost nothing remains.

There is a second reason too. During training, the network learns by sending a correction message backward through all the steps. This message is called the gradient. At every step back, it gets multiplied by a number that is often smaller than 1.

0.5 * 0.5 * 0.5 * ... (30 times) = almost 0

So, the early steps get almost no correction, and the network never learns to keep early information. This is called the vanishing gradient problem.

We needed a solution for that, and LSTM was introduced to solve this problem. Later, GRU came as a simpler version of the same idea.

What is a gate?

Before jumping into LSTM, we must understand one small idea: a gate.

A gate is a set of numbers between 0 and 1 that controls how much information passes through.

  • 0 means "let nothing pass". The gate is closed.
  • 1 means "let everything pass". The gate is fully open.
  • 0.5 means "let half pass".

The gate gets its numbers from a function called sigmoid. Sigmoid takes any number and squeezes it into the range 0 to 1. A big positive number becomes close to 1. A big negative number becomes close to 0.

We use the gate by multiplying:

information = [ 8,   4,   6 ]
gate        = [ 1,   0,   0.5 ]
result      = [ 8,   0,   3 ]

Here, the first value passes fully, the second value is blocked, and the third value is cut in half.

The gate values are not fixed. The network learns, from the current input and the memory, what gate values to use at each step. So, the gates learn when to open and when to close.

We will also see one more function called tanh. It squeezes any number into the range -1 to 1. It is used for the actual content, while sigmoid is used for the gates.

To learn Neural Networks, Activation Functions, and Backpropagation in depth, check out our AI and Machine Learning Program at Outcome School.

What is an LSTM?

LSTM stands for Long Short-Term Memory. It is a type of RNN that uses gates to decide what to forget, what to store, and what to output, so it can keep important information for a long time.

The name means it gives a long life to the short-term memory of an RNN. LSTM was introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997.

An LSTM carries two things from one step to the next:

  • Cell state (c): the long-term memory.
  • Hidden state (h): the short-term working memory, which is also the output at each step.

It has three gates:

  • Forget gate: what to erase from the long-term memory.
  • Input gate: what new information to write into the long-term memory.
  • Output gate: what part of the long-term memory to show as the output right now.

Let's draw the big picture:

                 cell state (long-term memory)
  c(prev) ----[ * forget gate ]----[ + new info ]-------------+----> c(new)
                                          ↑                   |
                                          |                   ↓
                               input gate * candidate    output gate
                                          ↑                   |
                                          |                   ↓
  h(prev) + x(now) -----------------------+                h(new) (output)

Here, the top line is the long-term memory. At each step, the forget gate removes some of it, the input gate adds some new information to it, and the output gate decides what to show from it as h(new). All three gates take their decision by looking at h(prev) and the current input x(now).

We will learn about each part one by one.

Think of an LSTM as a student who keeps a notebook while reading a long story. At each new page, the student does three things:

  • Erases old notes that are no longer useful.
  • Writes down new important facts.
  • Looks at the notebook and says what is relevant right now.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

The cell state

The cell state is the most important part of an LSTM.

The cell state is a list of numbers that flows from step to step and is changed only by small, controlled updates.

In a simple RNN, the memory is fully rewritten at every step. In an LSTM, the cell state is not rewritten. It only goes through two operations at each step: a multiplication by the forget gate and an addition of new information.

The forget gate

The forget gate decides how much of the old cell state to keep.

It looks at the previous hidden state h(prev) and the current input x(now), and produces a value between 0 and 1 for each number in the cell state.

f = sigmoid(W_f * [h(prev), x(now)] + b_f)

Here, W_f and b_f are the weights and bias of the forget gate, which are learned during training. [h(prev), x(now)] means we join the two lists together into one list.

  • f close to 1: keep this memory.
  • f close to 0: forget this memory.

Let's say the LSTM is reading a story and it remembers that the main person is "Riya". When the story says "Then Rahul entered the room...", the forget gate can lower the memory about who the main person is, to make room for the new person.

The input gate

The input gate decides how much new information to write into the cell state.

It works in two parts.

First, the LSTM creates a candidate, which is the new information that could be added. It uses tanh, so the values are between -1 and 1.

g = tanh(W_g * [h(prev), x(now)] + b_g)

Then, the input gate decides how much of this candidate to actually write:

i = sigmoid(W_i * [h(prev), x(now)] + b_i)

Now, the new cell state is calculated as below:

c(new) = f * c(prev) + i * g

Here, we can see that:

  • f * c(prev) is the old memory after forgetting some parts.
  • i * g is the new information that we decided to add.
  • We add both to get the updated memory.

The output gate

The output gate decides which part of the cell state to show as the hidden state, which is the output at this step.

o = sigmoid(W_o * [h(prev), x(now)] + b_o)

h(new) = o * tanh(c(new))

Here, we first pass the cell state through tanh to keep the values between -1 and 1. Then the output gate picks how much of each value to reveal.

The notebook can have many notes, but we do not need all of them right now. Let's say the next word to predict is a verb. The output gate can reveal the note "the subject is singular" so that the network picks "is" instead of "are", and keep the other notes hidden for now.

An LSTM step with numbers

Now, let's take one step with simple numbers for the sake of understanding. We will use only one number in the cell state.

Let's say the old cell state c(prev) = 5. It stores something important from earlier.

Step 1: The forget gate looks at the current input and decides to keep most of the memory.

f = 0.9
f * c(prev) = 0.9 * 5 = 4.5

Step 2: The candidate and input gate decide to add a little new information.

g = 1.0
i = 0.2
i * g = 0.2 * 1.0 = 0.2

Step 3: Update the cell state.

c(new) = 4.5 + 0.2 = 4.7

Step 4: The output gate decides how much to reveal.

o = 0.5
tanh(4.7) = 0.9998 (very close to 1)
h(new) = 0.5 * 0.9998 = 0.4999

Here, we can see that the old memory of 5 mostly survived as 4.7. If the forget gate stays near 1 for many steps, the memory survives for a very long time.

Now, let's see the opposite case. Suppose a new sentence starts and the old memory is no longer useful. The forget gate can become 0.0. Then 0.0 * 5 = 0, and the old memory is erased. Then the input gate can write fresh information.

This is how an LSTM controls its memory.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Why does an LSTM remember for a long time?

Now, the question is: why does this design fix the forgetting problem?

The answer is the plus sign in the cell state update:

c(new) = f * c(prev) + i * g

In a simple RNN, the memory passes through a full mixing step every time, and the backward correction message gets multiplied down at each step.

In an LSTM, the cell state moves forward mostly by being multiplied with the forget gate f and having something added. When f is close to 1, the old memory passes almost unchanged. In the backward direction, the correction message also flows back through the same path, multiplied by f at each step. So, when the network wants to remember, f stays near 1, and the message does not fade.

Let's compare:

Simple RNN, 30 steps:   0.5 * 0.5 * ... * 0.5   = almost 0
LSTM, 30 steps (f=0.99): 0.99 * 0.99 * ... * 0.99 = about 0.74

Here, we can see that the LSTM keeps a strong signal even after 30 steps. And the best part is that the network itself learns when to keep f near 1 and when to lower it.

So, the "France" from the beginning of our sentence can travel all the way to the end on the cell state.

What is a GRU?

Now that we have learned about LSTM, it's time to learn about GRU.

GRU stands for Gated Recurrent Unit. It is a simpler version of the LSTM that uses only two gates and keeps only one memory instead of two.

GRU was introduced by Kyunghyun Cho and his team in 2014.

The main differences from LSTM:

  • There is no separate cell state. The GRU keeps only the hidden state h, and it acts as both the long-term memory and the output.
  • There are two gates instead of three: the update gate and the reset gate.

Since it has fewer gates, it has fewer weights to learn and runs a little faster.

The update gate and the reset gate

The update gate decides how much of the old memory to keep and how much new memory to take.

z = sigmoid(W_z * [h(prev), x(now)] + b_z)

The update gate does the job of both the forget gate and the input gate of the LSTM, and that too in one go. It works like a slider between old and new:

h(new) = (1 - z) * h(prev) + z * h(candidate)

  • If z is close to 0, we keep the old memory almost as it is.
  • If z is close to 1, we replace it with the new candidate.
  • If z is 0.3, we keep 70% old and take 30% new.

Here, the forgetting and the adding are tied together. Whatever fraction we take from the new, we drop from the old.

The reset gate decides how much of the old memory to use while creating the new candidate.

r = sigmoid(W_r * [h(prev), x(now)] + b_r)

h(candidate) = tanh(W_h * [r * h(prev), x(now)] + b_h)

  • If r is close to 0, the candidate ignores the past and is built mostly from the current input. This is useful when a new topic starts.
  • If r is close to 1, the candidate uses the past fully.

Let's take a quick numeric example. Say h(prev) = 0.8, h(candidate) = -0.4, and the update gate z = 0.25.

h(new) = (1 - 0.25) * 0.8 + 0.25 * (-0.4)
       = 0.6 - 0.1
       = 0.5

Here, we kept 75% of the old memory and took 25% of the new candidate.

Think of the GRU as a person updating a single whiteboard instead of keeping a separate notebook. The reset gate decides whether to look at the old whiteboard while drafting the new note. The update gate decides how much of the whiteboard to wipe and replace with the new note.

Note: Some books and libraries write the update formula with z and (1 - z) swapped. The idea is exactly the same: one gate slides between old and new.

LSTM and GRU in code

Now, let's see the code for using both in PyTorch.

import torch
import torch.nn as nn

# 4 sentences, each 10 words long, each word is a list of 8 numbers
x = torch.randn(4, 10, 8)

lstm = nn.LSTM(input_size=8, hidden_size=16, batch_first=True)
gru = nn.GRU(input_size=8, hidden_size=16, batch_first=True)

lstm_out, (h_lstm, c_lstm) = lstm(x)
gru_out, h_gru = gru(x)

print(lstm_out.shape, h_lstm.shape, c_lstm.shape)
print(gru_out.shape, h_gru.shape)

Here, we have created some random input for the sake of example. In a real project, each word is first turned into a list of numbers that represents its meaning.

  • x has the shape (4, 10, 8), which means 4 sentences, 10 steps each, and 8 numbers per step.
  • input_size=8 tells the layer how many numbers come in at each step.
  • hidden_size=16 is the size of the memory, which means the hidden state has 16 numbers.
  • batch_first=True tells PyTorch that the first number in the shape is the number of sentences.
  • The LSTM returns the outputs at all steps, plus the final hidden state h_lstm and the final cell state c_lstm.
  • The GRU returns the outputs at all steps and the final hidden state h_gru. There is no cell state.

It will print the following:

torch.Size([4, 10, 16]) torch.Size([1, 4, 16]) torch.Size([1, 4, 16])
torch.Size([4, 10, 16]) torch.Size([1, 4, 16])

Here, we can notice that the LSTM gives back two memories and the GRU gives back only one. The 1 in the shape is the number of stacked layers.

Now, let's see the number of weights in each:

count = lambda m: sum(p.numel() for p in m.parameters())
print(count(lstm), count(gru))

It will print the following:

1664 1248

Here, we can see that the GRU has exactly 3/4 of the weights of the LSTM. This is because the LSTM has four sets of weights (forget gate, input gate, candidate, output gate), while the GRU has three sets (update gate, reset gate, candidate).

If we want to go deep into PyTorch and build a Neural Network from scratch, we have a complete program for it: our AI and Machine Learning Program at Outcome School.

LSTM vs GRU

Let me tabulate the differences between LSTM and GRU for your better understanding so that you can decide which one to use based on your use case.

PointLSTMGRU
Full nameLong Short-Term MemoryGated Recurrent Unit
Introduced19972014
Memories carriedTwo: cell state and hidden stateOne: hidden state
GatesThree: forget, input, outputTwo: update, reset
Sets of weightsFourThree
Forget and addControlled separatelyTied together by one update gate
SpeedA little slowerA little faster
Control over memoryMore fine controlSimpler
AccuracySimilar on many tasksSimilar on many tasks

On many tasks, both give similar results. A practical rule:

  • Start with GRU when we want a smaller and faster model, or when we have less data.
  • Use LSTM when the sequences are very long or the task needs fine control over memory.
  • If accuracy matters a lot, try both and keep the one that does better on our validation data.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where are LSTM and GRU used today?

LSTM and GRU powered many real products for years:

  • Speech recognition: Turning our voice into text.
  • Machine translation: Translating sentences from one language to another, before Transformers took over.
  • Text prediction: Suggesting the next word on phone keyboards.
  • Time series: Forecasting sales, energy use, or sensor readings.
  • Music and handwriting generation.

Note: Today, most LLMs (Large Language Models) use the Transformer architecture instead. A Transformer can look at all the tokens at once instead of one by one, which makes training on huge data much faster. But LSTM and GRU are still used a lot for small devices, real-time streaming data, and time series, because they are light and process one step at a time with a fixed amount of memory.

Now we must have understood how LSTM and GRU work.

Frequently Asked Questions

Who introduced LSTM and GRU?

LSTM was introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997. GRU was introduced by Kyunghyun Cho and his team in 2014. GRU came later as a simpler version of the same idea, with two gates instead of three and only one memory instead of two.

Is GRU faster than LSTM?

Yes, a little. A GRU has two gates instead of three and three sets of weights instead of four, so it has fewer weights to learn and runs a little faster. With the same sizes in PyTorch, a GRU has exactly 3/4 of the weights of an LSTM. On many tasks, both give similar results.

Why do LSTM and GRU use both sigmoid and tanh?

Sigmoid is used for the gates and tanh is used for the actual content. Sigmoid squeezes any number into the range 0 to 1, which is exactly what a gate needs to control how much information passes through. Tanh squeezes any number into the range -1 to 1, and it is used for the candidate and for the cell state before the output.

Why do LLMs use Transformers instead of LSTM?

Because a Transformer can look at all the tokens at once instead of one by one, which makes training on huge data much faster. LSTM and GRU read one step at a time. They are still used a lot for small devices, real-time streaming data, and time series, because they are light and use a fixed amount of memory.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.