Add BF16 support for int8_vectorwise_quant / LLM.int8 activation quant#1985
Add BF16 support for int8_vectorwise_quant / LLM.int8 activation quant#1985kru2710shna wants to merge 2 commits into
Conversation
| // One block per row. | ||
| // Threads load column values in a striped arrangement. | ||
| // e.g. t0 reads row[0], row[0+nthreads], .. | ||
| // and t1 reads row[1], row[1+nthreads], .. | ||
| // Each thread will determine its local absmax. | ||
| // We then do a blockwise reduction to determine the row's absmax. |
There was a problem hiding this comment.
Why was this comment removed?
There was a problem hiding this comment.
Not intentional. These got dropped when I rewrote the kernel body for float accumulation. They're accurate and unrelated to the change, so I will restored them verbatim in the next commit
| // For sparse decomposition, we do not want to quantize the outliers. | ||
| // Instead they're zeroed out. |
There was a problem hiding this comment.
Same cause as above, accidental during the float-reduction rewrite. Restored. Thanks for catching both.
There was a problem hiding this comment.
This warning would likely no longer apply.
|
I think it would be worthwhile to update some of the other relevant tests to exercise the new bf16 path. E.g. in |
|
Will do |
Templates int8VectorQuant on T and adds bf16 kernel instantiations plus a cint8_vector_quant_bf16 C ABI entry point, mirroring the existing gemm_4bit_inference_naive fp16/bf16/fp32 pattern. The blockwise absmax reduction now accumulates in float rather than T: required for bf16 to compile cleanly and slightly improves fp16 accuracy (rowStats was already float, so downstream is unaffected). Removes the forced A.to(torch.float16) casts in MatMul8bitLt so bf16 activations quantize natively. Closes bitsandbytes-foundation#1868.
…rning - Restore explanatory comments in kInt8VectorQuant that were dropped during the float-accumulation rewrite (striped-load pattern, sparse-decomp absmax, outlier zeroing). - Remove the 'inputs will be cast to float16' warning in MatMul8bitLt, which no longer applies now that the forced fp16 cast is gone.
kru2710shna
force-pushed
the
feature/int8-bf16-vectorwise-quant
branch
from
e30d08c to
cdd9e23
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
2 participants
Templates int8VectorQuant on T and adds bf16 kernel instantiations plus a cint8_vector_quant_bf16 C ABI entry point, mirroring the existing gemm_4bit_inference_naive fp16/bf16/fp32 pattern. The blockwise absmax reduction now accumulates in float rather than T: required for bf16 to compile cleanly and slightly improves fp16 accuracy (rowStats was already float, so downstream is unaffected). Removes the forced A.to(torch.float16) casts in MatMul8bitLt so bf16 activations quantize natively.
Closes #1868.