Inference

Inference is the use of a trained model to compute an output from new input without performing the model’s training update process.

For a language model, inference commonly means producing probability distributions and generated tokens from the current context.

Inference cost depends on model size, input length, output length, hardware, and serving strategy.

Related knowledge

Context engineering for LLM systemsDeliberately select and structure the information a model receives instead of treating the context window as an unbounded memory store.