All Blogs
What is Cross-Entropy Loss?
Cross-Entropy Loss is a number that tells us how wrong our predicted probabilities are, compared to the true answer. We will learn its math with an example.
How Does Gradient Descent Work?
Gradient Descent is an optimization algorithm that reduces a model's loss by moving its weights step by step in the direction of the steepest downward slope.
What is a Vision Transformer (ViT) and How Does It Work?
A Vision Transformer (ViT) splits an image into small patches, treats each patch like a word, and processes them with a transformer to classify the image.
What is the Feed-Forward Network in LLMs and What Does It Do?
The Feed-Forward Network is the part of every Transformer layer where each token is processed on its own, after attention. It holds most of the model's parameters.
What is Flash Attention and Why Is It So Fast?
Flash Attention computes the same attention that Transformers use, but in a much faster and more memory-efficient way on the GPU. The math and result stay the same.
What is Mixture of Experts (MoE) and How Does It Work?
Mixture of Experts (MoE) uses many small expert networks and a router that picks only a few of them for each input. This makes large models faster and cheaper.
How Does the Transformer Architecture Work?
A Transformer is a tokens-in, tokens-out machine that uses attention to understand how every word relates to every other word. It powers every modern LLM.
How Does Backpropagation Work? The Math Explained Step by Step
Backpropagation calculates how much each weight in a neural network contributed to the error, by sending the error backward, so that we can adjust the weights.
Why Do We Scale Attention by √dₖ? The Math Behind the Scaling Factor
We scale attention by √dₖ because dot products grow bigger as dₖ grows, which pushes softmax to extreme values. Dividing by √dₖ keeps them in a healthy range.