Endpointing is the part of a voice assistant that decides when the user has finished speaking, so that the assistant knows when to stop listening and start replying. It does this by watching for silence, listening to how the user speaks, and reading what the user has said so far.

How does endpointing decide when the user has finished speaking?

A Multilayer Perceptron (MLP) is a neural network made of layers of simple units called perceptrons, where every unit in one layer is connected to every unit in the next layer. It works by passing numbers forward through these layers, where each unit multiplies, adds, and applies a simple rule, until the last layer gives the answer.

What is a Multilayer Perceptron (MLP) and How Does It Work?

A voice agent hides the delay of a slow tool call by never letting the line go silent. It speaks a short line while the tool runs, gives progress updates during a long wait, keeps the conversation going while the tool works in the background, and starts the tool early so that there is less delay to hide.

How does a voice agent hide the delay of a slow tool call?

Filler phrases hide latency in a voice agent by filling the silence with a short, natural line like "Let me check that for you" while the agent is still working on the real answer, so the user hears a reply right away instead of waiting in silence.

How do filler phrases hide latency in a voice agent?

Speculative LLM generation is a technique where a voice agent starts preparing its reply while the user is still finishing their sentence, keeps that reply hidden, and plays it only if the user's final words match what the reply was based on.

How does speculative LLM generation start replying before the user finishes?

Sentence chunking feeds LLM output into TTS faster by cutting the reply into sentences while the LLM is still writing it, and sending each finished sentence to the TTS right away, so the voice starts speaking after the first sentence instead of after the full reply.

How does sentence chunking feed LLM output into TTS faster?

A TTS model starts speaking before the full sentence arrives by reading the text in small pieces as it comes in, staying only a few words behind the text, and turning each piece into sound while the rest of the sentence is still being written.

How does a TTS model start speaking before the full sentence arrives?

Matrix multiplication in neural networks is the operation that takes the input numbers, multiplies them with the weights of all the neurons in a layer, and adds up the results, all in one step. Most of the work a neural network does is this one operation, repeated layer after layer.

How Does Matrix Multiplication Work in Neural Networks?

Time-to-first-audio matters more than total response time because a person on a voice call feels only the silence before the agent starts speaking. Once the agent starts speaking, the person is busy listening, and the rest of the reply can keep being prepared in the background.

Why does the time-to-first-audio matter more than total response time?

Voice-to-voice latency of a voice agent gets measured as the time from the moment the user stops speaking to the moment the user hears the first sound of the agent's reply. We measure it by recording both voices on one timeline, finding the end of the user's speech and the start of the agent's speech, and subtracting the two.

How does the voice-to-voice latency of a voice agent get measured?

Cascaded vs Speech-to-Speech is the choice between building a voice agent as a chain of three models (speech-to-text, then an LLM, then text-to-speech) or as one model that hears audio and speaks audio directly. We choose based on what matters most for our use case: control and accuracy, or speed and natural sound.

Cascaded vs Speech-to-Speech: How to choose the right voice agent architecture?

PHP2Shell is the name of two critical security bugs found inside PHP itself, which let an attacker take full control of a website's server just by sending it a few web requests, without any username or password.

What is PHP2Shell? The PHP Vulnerability Explained Simply

A latency budget for a voice AI agent is a plan that splits the total waiting time we allow for a reply into smaller pieces, one for each step of the agent, so that the agent always starts replying fast enough to feel natural.

How do you build a latency budget for a voice AI agent?

A Neural Network is a model made of many small units called neurons, arranged in layers, that turn input numbers into a prediction. It learns by adjusting its numbers again and again until its predictions become close to the correct answers.

What is a Neural Network and How Does It Work?

Barge-in in a voice AI agent is the ability of the user to interrupt the agent while it is speaking, and the ability of the agent to stop talking right away and listen.

What is barge-in in a voice AI agent?

Turn-taking in a voice AI agent is the skill of knowing when the user has finished speaking, when the agent should start speaking, and when the agent should stop because the user wants to talk again.

What is turn-taking in a voice AI agent?

LLMs hallucinate because they write an answer one token, a small piece of text, at a time by picking a likely next token, and when they do not know the right token, they still have to pick one, so a confident-looking guess comes out.

Why Do LLMs Hallucinate? The Token-Level Explanation

ChatGPT remembers you by saving small notes about you outside the AI model and adding those notes back into the conversation whenever you chat. The model itself does not change when you chat with it, it reads your notes before it replies.

How does ChatGPT remember you?

Browser Agents are AI agents that use a web browser like a human does. They look at a web page, decide what to click or type, do it, look at the new page, and repeat until the task is finished.

How do Browser Agents work?

Query Rewriting and Multi-Query Retrieval are two ways of improving the search step of an AI system. Query Rewriting uses an LLM to turn the user's messy question into one clear search query, and Multi-Query Retrieval uses an LLM to turn the question into several different search queries, searches with all of them, and merges the results.

How do Query Rewriting and Multi-Query Retrieval work?

Tree of Thoughts is a way of making an LLM solve a hard problem by trying many possible next steps, checking which ones look good, and going deeper only on the good ones, just like exploring the branches of a tree.

How does Tree of Thoughts work?

Contextual Retrieval is a way to improve RAG by adding a short note to every chunk of a document before we store it. The note explains where the chunk comes from, so the system can find the right chunk even when the chunk alone does not say enough.

How does Contextual Retrieval work?

Ollama is a free tool that lets us download and run LLMs on our own computer with one simple command. It works by running a small server in the background that downloads compressed model files, loads them into memory, and uses a fast engine to generate text when we send it a prompt.

How does Ollama work?

A Stop Token is a special token in the vocabulary of an LLM that means the text ends here. When the model produces it, the generation loop stops writing.

Stop Tokens in LLMs

Tokenization in LLMs is the process of breaking text into small pieces called tokens and then converting each token into a number that the model can work with.

Tokenization in LLMs

A CUDA Kernel is a function that runs on the GPU and is executed by many threads at the same time, where each thread finds and does its own small part of the work.

How do CUDA Kernels work?

The Future of Software Development is a world where AI models become so smart, so fast, and so cheap that we simply tell the computer what we want, and AI agents build, test, and ship it for us. The work of software engineers moves from writing every line of code to building, checking, and improving the systems around these AI models.

The Future of Software Development

LLM architecture evolved from RNNs that read one word at a time, to Attention, to the Transformer, to scaling, and to Mixture of Experts (MoE) that we use today.

Evolution of LLM Architecture

Top-k Sampling keeps a fixed number of the most likely tokens, and Top-p Sampling keeps the top tokens whose total probability reaches p. The LLM picks from them.

How do Top-k and Top-p Sampling work?

Jev is a System One Model, a new kind of AI model that does not write text. It only makes fast decisions, with a confidence score, that our software can use directly.

Jev and System One Models Explained

A Real-Time Voice AI Agent listens to us, understands, thinks, takes actions, and talks back in a natural voice within a fraction of a second. Let's design one.

Design a Real-Time Voice AI Agent

Temperature is a single number that controls how an LLM picks the next token. Low Temperature gives safe, predictable answers, and high gives creative ones.

How does Temperature control LLM output?

Recursive Self-Improvement (RSI) is when an AI system improves its own abilities, and then the improved version improves itself further, again and again.

What is Recursive Self-Improvement (RSI)?

N-gram Speculation makes an LLM write faster by guessing the next few tokens from matching text it has already seen, and then verifying them in one run.

What is N-gram Speculation in LLMs and How Does It Speed Up Generation?

Prefill-Decode Disaggregation runs an LLM so that reading the prompt (prefill) and writing the answer (decode) happen on separate machines, each tuned for its job.

What is Prefill-Decode Disaggregation in LLM Inference?

KV Cache Compression is a set of techniques that shrink the memory an LLM uses to remember the conversation, using quantization, token eviction, and more.

What is KV Cache Compression?

Chunking for RAG means cutting a big document into smaller pieces so that the AI can find the right piece at the right time. We will learn the best strategies.

How to Chunk Documents for RAG? Chunking Strategies Explained

Vectorless RAG answers questions from our documents without embeddings or a vector database. The LLM reads the document structure and decides which part to open.

What is Vectorless RAG? RAG Without Embeddings or a Vector Database

Attention sinks are the first few tokens that get a huge share of attention in an LLM. Keeping them lets StreamingLLM handle very long conversations without breaking.

How do Attention Sinks work?

In Sliding Window Attention, each word looks only at a fixed number of nearby words instead of all the words, which makes attention fast for long text.

How does Sliding Window Attention work?

TensorRT-LLM is an open-source library from NVIDIA that makes LLMs run as fast as possible on NVIDIA GPUs by preparing the model ahead of time.

How does TensorRT-LLM work?

An LPU (Language Processing Unit) is a chip built for one job, running a trained LLM and producing text as fast as possible, by keeping the model on the chip.

How does an LPU work?

EAGLE is a speculative decoding method that makes an LLM generate text faster, without changing its output, by guessing future tokens at the feature level.

What is EAGLE? Feature-Level Speculative Decoding Explained

Medusa makes an LLM generate text 2 to 3 times faster by adding extra heads that guess several future tokens at once, and then checking them together.

What is Medusa? Multi-Head Speculative Decoding Explained

An RNN reads a sentence one word at a time, carrying a memory, whereas a Transformer reads all the words at once and uses attention to connect them directly.

How do RNNs and Transformers differ?

AI does its best work when we can clearly check if a task is done, and it struggles when only a human can judge it. That check is our definition of done.

AI Is Only as Good as Our Definition of Done

Prefix Tuning keeps a large language model frozen and trains only a small set of extra numbers, called the prefix, to adapt the model to a new task cheaply.

How does Prefix Tuning work?

Graph Engineering is designing an AI system as a graph, where every step is a node and every path between steps is an edge, instead of one giant prompt.

What is Graph Engineering?

Loop Engineering is designing the repeating cycle that an AI agent runs, so that it keeps making real progress on a task and stops at the right moment.

What is Loop Engineering?

The Lost in the Middle problem is when an LLM uses the beginning and the end of a long input well, but pays very less attention to what is in the middle.

What is the Lost in the Middle Problem in LLMs and How to Fix It?

LLM watermarking hides a signal inside the text a model writes. A secret key slightly nudges the word choices, and a detector with the same key finds it later.

How Does LLM Watermarking Work?

Deep RL from Human Preferences is the 2017 paper that taught machines what we want by asking humans to pick the better of two clips. It is the origin of RLHF.

What is Deep RL from Human Preferences? The Paper That Started RLHF

InstructGPT is the model that taught GPT-3 to follow our instructions by learning from human feedback (RLHF). This work led directly to ChatGPT.

What is InstructGPT? How GPT-3 Learned to Follow Instructions

ColBERT is a retrieval method that keeps the word-by-word matching of a slow BERT reranker but makes it fast enough to search millions of passages.

What is ColBERT? Late Interaction Retrieval Explained

LLM guardrails are safety checks that sit around an LLM to control what goes in and what comes out, and they stop dangerous or unwanted answers.

How do LLM guardrails work?

In Cloud Deployment, the AI model runs on a server and the device asks it over the internet. In On-device Deployment, the model runs inside the device itself.

Cloud vs On-device Model Deployment

In Transformers, the Encoder reads the whole input in both directions to understand it, whereas the Decoder generates the output one token at a time.

Encoder vs Decoder in Transformers

Generative AI is a type of AI that can create new things, like text, images, audio, video, and code. We will learn how it learns and creates something new.

What is Generative AI?

Prompt Injection is an attack where someone slips their own instructions into the text sent to an LLM, so that it follows the attacker instead of the developer.

What is Prompt Injection in LLMs and How Do We Defend Against It?

Precision tells us how many of the items we flagged were actually correct, whereas Recall tells us how many of all the real items we actually caught.

Precision vs Recall

An embedding is a list of numbers that represents the meaning of something, so that things with similar meaning get similar numbers. It powers AI search.

What are Embeddings?

An Agent Skill is a folder of instructions, and optionally scripts, that an AI agent loads by itself only when the task needs it. Let's see how Skills work.

What are Agent Skills?

MCP (Model Context Protocol) is an open standard that defines one common way for AI applications to connect to outside tools and data. It was created by Anthropic.

What is MCP (Model Context Protocol)?

Fine-tuning is taking a model that is already trained and training it a little more on our own data, so that it becomes good at our specific task.

How does fine-tuning work?

OKF (Open Knowledge Format) is an open standard for writing what an organization knows about its data as plain markdown files that any AI agent can read.

What is OKF (Open Knowledge Format)?

Semantic Search finds results based on meaning, not exact words. It turns text into embeddings and returns the items whose meaning is closest to the query.

How does Semantic Search work?

Cursor is an AI code editor. It indexes our codebase, understands it by meaning, and helps us with Tab autocomplete, Chat, and Agent mode to write code faster.

How does Cursor work?

Context compaction shrinks the old part of a long conversation into a short summary, so that the important facts stay and the context window gets free space again.

How does context compaction work?

Claude Code is a coding agent from Anthropic that runs in the terminal. It completes tasks by reading our code, editing files, running commands, and checking its work.

How does Claude Code work?

llama.cpp runs LLMs on everyday hardware by shrinking the model with quantization, loading it fast with memory mapping, and sharing work between CPU and GPU.

How does llama.cpp run LLMs on everyday hardware?

Model Quantization stores and computes a model's numbers at lower precision, so the model takes less memory and runs faster, even on a laptop or a phone.

How does Model Quantization work?

Chain-of-Thought (CoT) Prompting is a technique where we ask the model to write out its reasoning steps before the final answer, which makes it more accurate.

How does Chain-of-Thought (CoT) Prompting work?

Prompt Chaining breaks one big task into smaller prompts, where the output of one prompt becomes the input of the next. It solves bigger tasks reliably.

How does Prompt Chaining work?

PyTorch is an open-source library to build and train machine learning models. It uses tensors, a computation graph, and autograd, and runs fast on the GPU.

How does PyTorch work?

Semantic Caching is a cache that matches questions by their meaning instead of their exact words, so an AI app can reuse past answers for similar questions.

How does Semantic Caching work?

Hybrid Search combines keyword search and semantic search, and merges their results into one final ranked list, so that we get the best of both.

How does Hybrid Search work?

HyDE (Hypothetical Document Embeddings) is a RAG technique where the AI first writes a fake answer to the question, and then we search using that fake answer.

How does HyDE work in RAG?

Prefill reads the whole prompt in one pass and produces the first token, whereas Decode generates the rest of the tokens one at a time using the KV cache.

Prefill vs Decode: LLM Inference Optimization

A GPU is a chip that does a huge number of simple calculations at the same time. Deep learning is mostly matrix multiplication, so GPUs make it very fast.

How does a GPU work for Deep Learning?

LangGraph is a framework to build LLM applications where the work is organized as a graph of steps, with nodes, edges, and a shared state.

How does LangGraph work?

LangChain is a framework that helps us build LLM applications by connecting the LLM with our data, tools, and logic through chains, memory, and agents.

How does LangChain work?

SGLang is a high-performance framework that serves LLMs to many users at the same time, as fast as possible, using ideas like RadixAttention to reuse past work.

How does SGLang work?

Approximate Nearest Neighbor (ANN) Search finds items that are very close to the most similar one, much faster, by allowing a tiny chance of not being exact.

How does Approximate Nearest Neighbor (ANN) search work?

A TPU (Tensor Processing Unit) is a chip made by Google for machine learning. It uses a systolic array to do matrix multiplication very fast and efficiently.

How does a Google TPU work?

An image embedding is a list of numbers that represents the meaning of an image. Similar images get similar numbers, so we can compare and search images easily.

How do Image Embeddings work?

A computer-use agent is an AI program that operates a computer like a human. It looks at the screen, moves the mouse, clicks, and types to finish our goal.

How do Computer-Use Agents work?

Sakana Fugu is a family of AI models that work like a conductor. They decide which other AI models should work on our question and combine their answers.

What is Sakana Fugu? The Technical Report Explained

A Diffusion Language Model writes text by starting from pure gibberish and cleaning it up again and again, instead of writing one word at a time from left to right.

How do Diffusion Language Models (DLMs) work?

An Embedding Cache stores the embeddings we have already computed, so that we reuse them for the same text instead of computing them again. It saves money and time.

How does an Embedding Cache work?

A World Model is an AI that learns an internal copy of how an environment behaves, so that it can predict what happens next when an action is taken.

How do World Models work?

vLLM is a high-throughput engine for serving LLMs. It uses PagedAttention to manage the KV cache memory efficiently, so it can serve many more users at once.

How does vLLM work?

LLM Inference Optimization is the set of techniques, like KV Cache, Flash Attention, and Continuous Batching, that make LLMs fast and scalable in production.

LLM Inference Optimization

Function Calling lets an LLM use external tools and APIs. The model does not run the function itself. It tells our code which function to call and with what inputs.

How does Function Calling work in LLMs?

GGUF is a single file format that stores everything needed to run an LLM locally, like the weights, the tokenizer, and the settings, all in one file.

How does GGUF work?

Knowledge Distillation is a technique where we train a small model to copy the behavior of a big model, so that we can run powerful AI at a low cost.

How does Knowledge Distillation work?

Token Streaming sends the LLM's reply piece by piece as each piece is produced, using SSE over one open connection, instead of waiting for the whole reply.

How does Token Streaming work?

Prompt Caching saves the work an LLM already did for a repeated part of a prompt, so that it can reuse that work next time. It makes responses faster and cheaper.

How does Prompt Caching work?

A Reranker is a model that takes a list of documents and reorders them, putting the most relevant ones at the top for a given question. It makes RAG more accurate.

How does a Reranker work?

JEPA (Joint Embedding Predictive Architecture) is an idea from Yann LeCun where a model predicts the hidden part of the data in the embedding space, not in pixels.

What is Joint Embedding Predictive Architecture (JEPA) and How Does It Work?

A Vector Database stores data as lists of numbers and helps us find items that are similar in meaning, not just items that match exactly. It powers AI search.

All Blogs