Converting model weights and/or activations from higher precision (FP16/BF16) to lower precision (INT8, INT4, FP8) at inference time, reducing memory footprint and increasing throughput at a small accuracy cost.
Converting model weights to lower precision at inference time to reduce memory and increase throughput.
Converting model weights and/or activations from higher precision (FP16/BF16) to lower precision (INT8, INT4, FP8) at inference time, reducing memory footprint and increasing throughput at a small accuracy cost.