The phase of inference where the model processes all input tokens in parallel to build the KV cache before generating the first output token — its duration determines time-to-first-token.
The phase where the model processes all input tokens in parallel to build the KV cache before generating output.
The phase of inference where the model processes all input tokens in parallel to build the KV cache before generating the first output token — its duration determines time-to-first-token.