[BAD VERSION2] Make module swap the main QAT flow again #1019
Conversation
Summary: Following #987, this commit makes module swap the main QAT flow today. We remove all tensor subclass fake quantize injection logic since this is not needed in both the long term and the short term plans for QAT. In the short term, we will continue to use a full module swap flow, and only migrate to the long term flow once there is general distributed support for tensor subclasses and when tensor subclass composability provides meaningful benefits. Test Plan: python test/quantization/test_qat.py [ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1019
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 0756f39 with merge base 5a4857e ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
| # | 8da4w QAT | | ||
| # ================= | ||
|
|
||
| def int8_dynamic_activation_int4_weight_fake_quantize(group_size=32): |
There was a problem hiding this comment.
so we are not planning to use fake quant tensor subclass? I remember there was some benefits, can you write down the plan for this? maybe either in summary or code comments
There was a problem hiding this comment.
I thought we decided on using fake quantize tensor subclass for the long term: #987?
There was a problem hiding this comment.
This path is doing tensor subclass injection. I think we don't want to do that in the long term. E.g. what fp8 training does is use module swaps to insert the subclasses, like:
ao/torchao/float8/float8_linear.py
Line 500 in dec0313
There was a problem hiding this comment.
oh OK I thought AffineFakeQuantizedTensor was removed. then sounds good
|
oh no, this was merged into the wrong branch, need to reopen again |
andrewor14
changed the title
|
Reopened here: #1037 |
Stack from ghstack (oldest at bottom):
Summary: Following #987, this
commit makes module swap the main QAT flow today. We remove all
tensor subclass fake quantize injection logic since this is not
needed in both the long term and the short term plans for QAT.
In the short term, we will continue to use a full module swap
flow, and only migrate to the long term flow once there is
general distributed support for tensor subclasses and when
tensor subclass composability provides meaningful benefits.
Test Plan:
python test/quantization/test_qat.py