Skip to content

LLM · Inference and Serving

This module covers decoding, sampling, KV caching, quantization, MoE, and inference serving frameworks.

Keep three questions separate as you read: Does output selection change the target distribution? How do architecture and precision affect resource requirements? Can serving schedules meet latency and throughput constraints? Parameter count, bit width, or a framework name alone cannot determine end-to-end performance.

Chapters

  1. Chapter 12: Decoding Strategies
  2. Chapter 13: Tuning Temperature, Top-P, and Top-K
  3. Chapter 14: KV Cache and Prompt Caching
  4. Chapter 15: Model Quantization
  5. Chapter 19: Mixture-of-Experts Models
  6. Chapter 20: Choosing a Serving Framework

Return to LLM Topics.

Architecture examples retain explicit model versions. API defaults, cache prices, and framework support matrices depend on the official sources cited in each chapter and the deployed version; historical experimental results are not guarantees of current performance.