In what scenarios should QuantManagedCollisionEmbeddingCollection be used?

[tiankongdeguiji]

I believe that when we need to use a quantized ManagedCollisionEmbeddingCollection, we can configure the mapping from EmbeddingCollection to QuantEmbeddingCollection within quantize_dynamic. This makes the existence of a QuantManagedCollisionEmbeddingCollection class unnecessary. In what scenarios should QuantManagedCollisionEmbeddingCollection be used?

[tiankongdeguiji]

hi, @iamzainhuda @kausv @dstaay-fb @henrylhtsang @PaulZhang12 can you see this problem?

[kausv]

@tiankongdeguiji QuantManagedEmbeddingCollection is a stronger indication that the underlying EC is quantized. It is also to be used for inference as it implicitly calls reset_inference_mode().
The functionality can be achieved without using the class too if needed.

[tiankongdeguiji]

@tiankongdeguiji QuantManagedEmbeddingCollection is a stronger indication that the underlying EC is quantized. It is also to be used for inference as it implicitly calls reset_inference_mode(). The functionality can be achieved without using the class too if needed.

thx! I believe that during the inference phase, the MC module will not do profile. Could you please explain the rationale behind setting self._evicted = False and self._last_eviction_iter = -1 in the reset_inference_mode() method? It seems to me that self._evicted and self._last_eviction_iter are only utilized within the profile function.

[kausv]

I think reset_inference_mode() should also set self.training to False which we have missed.
Since profile() is called in forward we want to skip that. We also don't want to perform any evictions.
This function does a lot more in another instance of ManagedCollision that I am working on open sourcing.

[tiankongdeguiji]

I think reset_inference_mode() should also set self.training to False which we have missed. Since profile() is called in forward we want to skip that. We also don't want to perform any evictions. This function does a lot more in another instance of ManagedCollision that I am working on open sourcing.

thx! Could you see this problem #2669 ?