All Blogs
How does vLLM work?
vLLM is a high-throughput engine for serving LLMs. It uses PagedAttention to manage the KV cache memory efficiently, so it can serve many more users at once.
LLM Inference Optimization
LLM Inference Optimization is the set of techniques, like KV Cache, Flash Attention, and Continuous Batching, that make LLMs fast and scalable in production.
How does Function Calling work in LLMs?
Function Calling lets an LLM use external tools and APIs. The model does not run the function itself. It tells our code which function to call and with what inputs.
How does GGUF work?
GGUF is a single file format that stores everything needed to run an LLM locally, like the weights, the tokenizer, and the settings, all in one file.
How does Knowledge Distillation work?
Knowledge Distillation is a technique where we train a small model to copy the behavior of a big model, so that we can run powerful AI at a low cost.
How does Token Streaming work?
Token Streaming sends the LLM's reply piece by piece as each piece is produced, using SSE over one open connection, instead of waiting for the whole reply.
How does Prompt Caching work?
Prompt Caching saves the work an LLM already did for a repeated part of a prompt, so that it can reuse that work next time. It makes responses faster and cheaper.
How does a Reranker work?
A Reranker is a model that takes a list of documents and reorders them, putting the most relevant ones at the top for a given question. It makes RAG more accurate.
What is Joint Embedding Predictive Architecture (JEPA) and How Does It Work?
JEPA (Joint Embedding Predictive Architecture) is an idea from Yann LeCun where a model predicts the hidden part of the data in the embedding space, not in pixels.