Outcome School

LLM Inference Optimization

Authors
LLM Inference Optimization

LLM Inference Optimization is the set of techniques that make a Large Language Model generate its answers faster and serve more users at a lower cost.

Techniques like KV Cache, Paged Attention, Flash Attention, Speculative Decoding, and Continuous Batching are what make LLMs fast and scalable in production. Let’s learn all of these techniques one by one.

KV Cache in LLMs

In this blog, we will learn about KV Cache - where K stands for Key and V stands for Value - and why it is used in Large Language Models (LLMs) to speed up text generation.

We will start with how LLMs generate text one token at a time, understand the role of Key, Value, and Query inside the model, see the problem of repeated computation through an example, and then walk through how KV Cache solves this problem by storing and reusing past results.

Read here: KV Cache in LLMs

KV Cache Compression

In this blog, we will learn about KV Cache Compression, the set of techniques that shrink the memory an AI model uses to remember the conversation while it is writing its reply.

We will cover the following:

  • What is an LLM and how it writes text
  • What is Attention
  • What is the KV Cache
  • Why the KV Cache becomes huge
  • What is KV Cache Compression
  • Approach 1: Quantization
  • Approach 2: Token Eviction
  • Approach 3: Sharing Keys and Values across Heads
  • Approach 4: Low-Rank Compression
  • Comparison of the approaches
  • When to use which one

Read here: KV Cache Compression

Paged Attention in LLMs

In this blog, we will learn about Paged Attention, a technique that solves the memory waste problem of KV Cache, allowing LLMs to serve many more users at the same time.

We will start with a quick recap of KV Cache, understand the memory problem it creates, see how traditional memory allocation wastes space through an example, and then walk through how Paged Attention solves this problem by borrowing an idea from how computers manage memory.

Read here: Paged Attention in LLMs

Decoding Flash Attention in LLMs

In this blog, we will learn about Flash Attention by decoding it piece by piece - understanding why standard attention is slow, what makes Flash Attention fast, how it uses GPU memory cleverly, and why it is used in almost every modern Large Language Model (LLM).

We will cover the following:

  • A quick recap of standard attention
  • Why standard attention is slow
  • How GPU memory actually works (HBM vs SRAM)
  • The core idea behind Flash Attention
  • Tiling: breaking the work into small blocks
  • Online softmax: computing softmax without the full matrix
  • Recomputation in the backward pass
  • Flash Attention 2
  • Flash Attention 3
  • Advantages and impact of Flash Attention

Read here: Decoding Flash Attention in LLMs

Grouped Query Attention

In this blog, we will learn about Grouped-Query Attention (GQA) and how it differs from Multi-Head Attention (MHA).

Today, we will cover the following topics:

  • Quick Recap: Multi-Head Attention (MHA)
  • The Problem with Multi-Head Attention
  • What is Multi-Query Attention (MQA)?
  • What is Grouped-Query Attention (GQA)?
  • How Grouped-Query Attention Works
  • GQA is a Generalization of MHA and MQA
  • GQA vs MHA vs MQA
  • Real-World Use Cases
  • A Note on Terminology
  • Uptraining: Converting MHA to GQA
  • Quick Summary

Read here: Grouped Query Attention

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Speculative Decoding

In this blog, we will learn about Speculative Decoding - what it is, why LLM generation is slow without it, how a small draft model and a big target model work together to produce tokens faster, the rejection sampling math that guarantees no quality loss, real numbers showing the 2× to 3× speedup, where it is used in production, and the trade-offs to watch out for.

We will cover the following:

  • What problem does Speculative Decoding solve?
  • The Big Picture
  • Why is LLM generation slow?
  • The core idea behind Speculative Decoding
  • Step-by-step walkthrough
  • The verification step
  • Real numbers and speedup
  • Where it is used
  • Trade-offs
  • Quick Summary

Read here: Speculative Decoding

Decoding Medusa

In this blog, we will learn about Medusa, a simple way to make a language model generate text 2 to 3 times faster by giving it several extra heads that guess multiple future tokens at once.

We will cover the following:

  • What is Medusa
  • Why text generation is slow
  • A quick recap of speculative decoding
  • The problem with needing a draft model
  • The big idea: many heads on one model
  • How tree attention checks many guesses at once
  • The math behind the speedup with small numbers
  • The results
  • How Medusa lives on today
  • Quick Summary

Read here: Decoding Medusa

Decoding EAGLE

In this blog, we will learn about EAGLE, a state-of-the-art way to speed up language model generation by drafting tokens at the feature level instead of the token level.

We will cover the following:

  • What is EAGLE
  • A quick recap of speculative decoding
  • The problem with token-level drafting
  • The big idea: draft at the feature level
  • Resolving the uncertainty by feeding back the token
  • The math behind the speedup with small numbers
  • EAGLE-2 and dynamic draft trees
  • How EAGLE lives on today
  • Quick Summary

Read here: Decoding EAGLE

N-gram Speculation in LLMs

In this blog, we will learn about N-gram Speculation in LLMs, a simple trick that makes a language model write its answer faster by guessing the next few words from the text it has already seen.

We will cover the following:

  • How an LLM generates text
  • Why generating text is slow
  • What is Speculative Decoding
  • The cost of a draft model
  • What is an N-gram
  • What is N-gram Speculation
  • N-gram Speculation step by step
  • Why the output stays exactly the same
  • Where it works well and where it fails
  • N-gram Speculation vs Draft Model Speculative Decoding

Read here: N-gram Speculation in LLMs

Continuous Batching in LLMs

In this blog, we will learn about Continuous Batching, a technique that lets LLM servers handle many more users at the same time by keeping the GPU busy at every single step of generation.

We will cover the following:

  • Quick Recap: How an LLM Generates Tokens
  • Why Batching Matters for LLMs
  • The Old Way: Static Batching
  • The Problem with Static Batching
  • What is Continuous Batching?
  • The Ride-Share Analogy
  • How Continuous Batching Works Step by Step
  • A Numeric Example
  • Real Numbers and Speedup
  • Benefits of Continuous Batching
  • A Few Important Notes
  • Quick Summary

Read here: Continuous Batching in LLMs

Prompt Caching

In this blog, we will learn about how Prompt Caching works. We will also see why we need it, how it actually works inside a large language model, and where it is used in real systems like AI assistants and agents.

We will cover the following:

  • What is a prompt
  • A quick recap of how an LLM reads a prompt
  • What is Prompt Caching
  • Why we need Prompt Caching
  • The core idea behind Prompt Caching
  • The exact-prefix rule
  • Cache write vs cache read and TTL
  • What we should put in the cache
  • The benefits of Prompt Caching
  • Prompt Caching in the real world

Read here: How does Prompt Caching work?

Prefill vs Decode: LLM Inference Optimization

In this blog, we will learn about Prefill vs Decode, the two phases of LLM inference, and how understanding them helps us optimize the speed of an LLM. We will also see how the prefill and decode phases work, how the KV cache connects them, how they differ and when to use which one based on our use case, and how we optimize each phase to make an LLM faster.

We will cover the following:

  • The two phases: Prefill and Decode
  • Prefill explained in simple words
  • Decode explained in simple words
  • A diagram of the two phases and the KV cache flow
  • The KV cache as the bridge between the two phases
  • A step-by-step walkthrough of a few decode steps
  • Prefill vs Decode comparison table
  • Why this split matters: compute-bound vs memory-bound
  • The key metrics: TTFT, TPOT, throughput, and end-to-end latency
  • Optimization techniques mapped to each phase

Read here: Prefill vs Decode: LLM Inference Optimization

Prefill-Decode Disaggregation

In this blog, we will learn about Prefill-Decode Disaggregation, a way of running a large language model where the reading of the prompt and the writing of the answer happen on separate machines.

We will cover the following:

  • How an LLM answers a request
  • What is the KV Cache?
  • Prefill is compute-heavy, Decode is memory-heavy
  • The problem when both run on the same GPU
  • TTFT vs TPOT
  • The naive approaches and their issues
  • What is Prefill-Decode Disaggregation?
  • How Prefill-Decode Disaggregation works
  • Walkthrough of one request
  • Advantages of Prefill-Decode Disaggregation
  • Disadvantages of Prefill-Decode Disaggregation
  • Where it works well and where it is overkill
  • Co-located vs Disaggregated serving

Read here: Prefill-Decode Disaggregation

How does vLLM work?

In this blog, we will learn about how vLLM works. We will also see why we need it, how it manages memory so cleverly, and where it is used in the real world to serve large language models to many users at once.

We will cover the following:

  • What is serving an LLM
  • A quick recap of prefill, decode, and the KV cache
  • The problem: the KV cache eats GPU memory
  • Why naive serving wastes memory
  • What is vLLM
  • PagedAttention, the core idea
  • How PagedAttention shares memory
  • Continuous batching
  • The OpenAI-compatible API server
  • The benefits of vLLM
  • vLLM in the real world

Read here: How does vLLM work?

How does SGLang work?

In this blog, we will learn about how SGLang works. We will also see what problem it solves, how it makes serving large language models faster, and the clever ideas that make it special.

We will cover the following:

  • What is SGLang
  • RadixAttention: the heart of SGLang
  • How RadixAttention reuses past work
  • The frontend language of SGLang
  • Continuous batching in SGLang
  • Structured output and faster decoding
  • More powerful features of SGLang
  • How SGLang compares to vLLM

Read here: How does SGLang work?

How does TensorRT-LLM work?

In this blog, we will learn about how TensorRT-LLM works, NVIDIA's own engine that squeezes the highest possible speed out of an NVIDIA GPU when it runs a large language model.

We will cover the following:

  • What is inference
  • What is a GPU and what is a kernel
  • The problem: the GPU spends its time on the wrong things
  • What is TensorRT-LLM
  • The big idea: prepare the model ahead of time
  • The build step: from a model to an engine
  • Kernel fusion
  • Quantization
  • Custom attention kernels
  • The paged KV cache
  • In-flight batching
  • CUDA graphs
  • Speculative decoding
  • Running one model across many GPUs
  • How we actually serve the model
  • The PyTorch backend, the newer and easier path
  • The full journey of one request
  • TensorRT-LLM vs vLLM
  • Where it works well and where it fails

Read here: How does TensorRT-LLM work?

How does an LPU work?

In this blog, we will learn about how an LPU works, the chip that was built for one single job, running a large language model that is already trained and producing text as fast as possible.

We will cover the following:

  • What is an LPU
  • How an LLM writes text, one token at a time
  • The real bottleneck is memory, not math
  • Why a GPU struggles here
  • Keeping the model on the chip
  • Removing all the guesswork with a compiler
  • A network that never waits
  • What happens when we send a prompt
  • Where an LPU works well and where it does not
  • LPU vs GPU

Read here: How does an LPU work?

Frequently Asked Questions

Does Speculative Decoding reduce the quality of the output?

No. In Speculative Decoding, a small draft model and a big target model work together to produce tokens faster. The rejection sampling math guarantees no quality loss. It gives a speedup of around 2 to 3 times.

Can we speed up generation without a separate draft model?

Yes. Medusa gives the model several extra heads that guess multiple future tokens at once, so it does not need a draft model. N-gram Speculation guesses the next few words from the text the model has already seen.

What is the difference between the prefill and decode phases?

Prefill and decode are the two phases of LLM inference. Prefill reads the prompt and is compute-heavy, while decode writes the answer and is memory-heavy. The KV cache acts as the bridge between the two phases. Prefill-Decode Disaggregation goes one step further and runs these two phases on separate machines.

Which techniques help an LLM server handle more users at the same time?

Paged Attention and Continuous Batching both help here. Paged Attention solves the memory waste problem of the KV Cache, allowing LLMs to serve many more users at the same time. Continuous Batching keeps the GPU busy at every single step of generation, so the server can handle many more users.

Prepare yourself for AI Engineering Interview: AI Engineering Interview Questions

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.

Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.