How do you become an LLM Inference Engineer?

You become an LLM Inference Engineer by learning how a model generates text one token at a time, learning the techniques that make that generation fast and cheap, and proving it by serving one open model with a coding agent and measuring every improvement.

In this blog, we will learn about how to become an LLM Inference Engineer. We will also see what an LLM Inference Engineer does, what we must know before we start, the skills to learn in order from prefill and decode to custom kernels, what to build with a coding agent to prove our skills, how to show the work, and how to apply for the role.

We will cover the following:

• What is an LLM Inference Engineer?
• Prerequisites
• Skills to learn in order
• What to build with a coding agent
• How to show the work
• How to apply

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is an LLM Inference Engineer?

An LLM Inference Engineer is an engineer who makes model serving fast and cheap.

Inference is the step where a trained model is used to generate an answer. Training is done once. Inference happens every time a user sends a message. So, for a product with many users, inference is where most of the cost goes.

The LLM Inference Engineer owns this step. The job is to serve more users from the same GPU, to reduce the time a user waits for each token, and to do this without hurting the quality of the answers.

An LLM Engineer works on prompts, fine-tuning, and evaluation. An LLM Inference Engineer works on what happens after the model is ready, which means the serving system that runs it.

Prerequisites

Before jumping into the skills, we must have a few things in place.

First, we must be comfortable with Python. Most serving frameworks expose a Python interface, and most of our experiments will be Python scripts that we build with a coding agent and then review.

Second, we must have built some C++ with a coding agent. The fast parts of every serving system are written in C++ and CUDA. We do not need to be an expert, but we must be able to read a C++ file, ask the coding agent to change it, and verify the change.

Third, we must have run a model locally at least once. This teaches us how much memory the weights take and how slow generation is without any optimization.

Fourth, we must know what a GPU is. A GPU is a processor that runs thousands of small calculations at the same time, and it has its own memory, separate from the memory of the CPU.

Skills to learn in order

Each skill builds on the one before it.

Step 1: Learn how a transformer generates one token at a time.

A token is a small piece of text, usually a word or a part of a word. A transformer is the model architecture used by most modern LLMs. It takes all the tokens so far, runs them through many layers, and produces a probability for every possible next token. We pick one token, add it to the input, and repeat, so one full pass gives one token.

Every optimization we will learn is about making this loop cheaper or running more copies of it at once.

Step 2: Learn prefill versus decode, and why decode is memory bound.

Prefill is the first pass, where the model reads the whole prompt at once. Decode is every pass after that, where the model generates one new token.

Prefill processes many tokens in one pass, so the GPU has a lot of math to do and stays busy. Decode processes only one token per pass, but it still has to read every weight of the model from GPU memory to do it. So, in decode, the GPU spends most of its time waiting for weights to arrive from memory and very less time doing math.

We call this memory bound. The speed of decode is decided by how fast we can read memory, not by how fast we can multiply.

Step 3: Learn the KV cache and why memory decides batch size.

In every layer, the model computes a key and a value for every token. This key and value are the K and V in the name. Without a cache, every decode step would recompute K and V for every earlier token. The KV cache stores them once, so each new token only computes its own K and V and reuses the rest.

The KV cache lives in GPU memory, and it grows with every token of every user. The weights take a fixed amount of memory. Whatever is left is for the KV cache. The number of requests we can serve at the same time, which we call the batch size, is decided by how many KV caches fit in that leftover memory.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Step 4: Learn batching, continuous batching, and paged attention.

Batching means running many requests through the model in the same pass. Since decode is memory bound, reading the weights once and using them for 32 requests costs nearly the same as using them for 1 request. So, batching is the biggest win in inference.

Static batching runs a full batch and waits for the longest request to finish before starting a new batch, so short requests sit idle.

Continuous batching fixes this. At every decode step, finished requests leave the batch and new requests join. The GPU never waits for the slowest request.

Paged attention fixes a second problem. The KV cache of each request used to be stored in one contiguous block, reserved for the maximum possible length, so most of it sat empty. Paged attention splits the cache into small fixed-size pages and allocates a page only when it is needed. Less memory is wasted, so more requests fit, so the batch gets bigger.

Step 5: Learn quantization and what it trades.

Quantization stores the weights with fewer bits. A weight stored in 16 bits can be stored in 8 bits or 4 bits instead. The model becomes smaller, so it takes less memory, and decode reads less memory per token, so it gets faster.

The trade is accuracy. Fewer bits means each weight is stored less precisely, and the answers can get slightly worse. The job is to measure that loss on a real evaluation set and decide whether the speed is worth it. Common formats are INT8, FP8, and 4-bit.

Step 6: Learn speculative decoding.

Decode is memory bound, so the GPU has spare math capacity, and speculative decoding uses it. A small, fast draft model guesses the next few tokens. Then the big model checks all the guesses in one pass, which costs about the same as generating one token. The correct guesses are accepted, and the first wrong guess is replaced by the big model's own token. The output is exactly what the big model would have produced, only faster.

Step 7: Learn tensor and pipeline parallelism across GPUs.

A large model often does not fit in one GPU, so we must split it.

Tensor parallelism splits every layer across GPUs. Each GPU holds a slice of every weight matrix, computes its part, and the GPUs exchange results after each layer. It needs a very fast link between the GPUs.

Pipeline parallelism splits the layers themselves. GPU 1 holds the first few layers, GPU 2 holds the next few, and a request flows through them in order. It needs less communication but can leave some GPUs waiting.

A common setup is tensor parallelism inside one machine and pipeline parallelism across machines.

Step 8: Learn profiling with Nsight and reading a roofline.

A kernel is one function that runs on the GPU. Nsight is the profiler from NVIDIA that records every kernel the GPU ran, how long each one took, and how much memory it moved.

A roofline is a chart that shows the maximum possible speed of any kernel on a given GPU, with one line for the memory limit and one line for the math limit. If our kernel sits under the memory line, it is memory bound. If it sits under the math line, it is compute bound. The roofline tells us which one to fix.

Step 9: Learn serving frameworks such as vLLM and SGLang.

vLLM and SGLang are open-source serving frameworks that already implement continuous batching, paged attention, quantization, and speculative decoding. TensorRT-LLM is the framework from NVIDIA that compiles a model for a specific GPU.

We direct the coding agent to set one of these up, and we learn what each configuration flag changes in terms of the concepts above.

Step 10: Learn custom CUDA and Triton kernels as the last step.

CUDA is the language from NVIDIA for writing kernels. Triton is a Python-like language that makes kernel writing simpler. We reach this step only when the profiler shows that one specific kernel is the bottleneck and the framework does not fix it.

We describe the kernel to the coding agent, review the generated code, and then verify it in two ways. We check the output against the original kernel for correctness, and we check the profiler for speed.

Want to become an LLM Inference Engineer?

These are the skills this role needs, and we teach them in our AI and Machine Learning Program at Outcome School:

• KV Cache
• Paged Attention
• Continuous Batching
• Speculative Decoding
• Quantization and Optimizations
• vLLM

We take you from the fundamentals to building real AI systems, step by step.

What to build with a coding agent

Learning by example is the best way to learn. One project proves all of the skills above.

First, we pick one open model with around 7 billion parameters and serve it on one GPU with a plain generation loop and no serving framework. We ask the coding agent to write this script, and we review it.

Then, we measure. We ask the coding agent to write a load-testing script that sends many requests at once and records two numbers. The first is tokens per second, which is how many tokens the server produces each second. The second is latency, which we split into time to first token and time per output token. This is our baseline.

After that, we apply three optimizations one at a time, and we record what each one changed:

• Move the model to vLLM, which brings continuous batching and paged attention, and measure how many requests fit and how tokens per second changed.
• Quantize the model to 8 bits or 4 bits, measure the speed, and measure the accuracy on a small evaluation set.
• Add speculative decoding with a small draft model, and measure the latency per token.

We change one thing at a time. If we change three things and the number moves, we do not know which one did it.

Finally, we profile the final setup with Nsight and find the kernel that takes the most time.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

How to show the work

The project becomes a public repository with three things.

The first is a before-and-after table with one row for the baseline and one row per optimization. Each row has the speed and the accuracy after that change. A sample table looks like below:

Here, each row shows what one change cost in accuracy and what it gave in speed. The numbers come from our own GPU, and we state which GPU in the README.

The second is a profiler screenshot from Nsight, with the top kernels marked and one paragraph on what we learned.

The third is a README that explains the setup, the hardware, and how to reproduce every number. A reviewer must be able to run the same commands and get close to the same table.

How to apply

We apply for roles titled LLM Inference Engineer, Inference Engineer, ML Performance Engineer, or ML Systems Engineer. The role exists at model providers, at companies that serve open models as an API, and at product companies with a large inference bill.

In the application, we lead with the table. One line like "Served a 7B model on one GPU and took it from 410 to 2900 tokens per second with three measured optimizations" says more than a list of skills.

In the interview, we expect a practical build round where we work with a coding agent to tune a serving stack and explain every number, so the project above is also our preparation.

Now we must have understood how to become an LLM Inference Engineer, what to learn in order, what to build, and how to show it.

Frequently Asked Questions

Do we need to be a CUDA expert to become an LLM Inference Engineer?

No. Custom CUDA and Triton kernels are the last skill to learn, and we reach them only when the profiler shows one specific kernel is the bottleneck and the framework does not fix it. We do not need to be a C++ expert either, but we must be able to read a C++ file, ask a coding agent to change it, and verify the change.

Which model should we use for a first LLM inference project?

We pick one open model with around 7 billion parameters and serve it on one GPU. We start with a plain generation loop and no serving framework, measure tokens per second and latency as the baseline, and then apply one optimization at a time so we know what each one changed.

Why does batching make LLM inference cheaper?

Batching makes inference cheaper because decode is memory bound. The GPU has to read every weight of the model for each decode step, so reading the weights once and using them for 32 requests costs nearly the same as using them for 1 request. That is why batching is the biggest win in inference.

Does quantization make the model worse?

It can make the answers slightly worse. Quantization stores each weight with fewer bits, so the model gets smaller and faster, but each weight is stored less precisely. The job is to measure that accuracy loss on a real evaluation set and decide whether the speed is worth it.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.