Latency

For language models, a distinction is made between time to first token and output speed in tokens per second. Short latency is crucial for voice assistants and real-time applications.

Latency depends on model size, input length, server load, network path and hardware. Smaller models and local execution are often much faster.