You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
init_size=1e-3# in config.py also "init_size": 1e-3max_scale=1.-1e-3# in config.py also "max_scale": 1 - 1e-3self.scale=nn.Parameter(torch.Tensor([init_size]), requires_grad=True)
min_scale=1e-2#self.init_sizemax_scale=self.max_scalereturnF.normalize(embeddings, p=2, dim=1) *self.scale.clamp_min(min_scale).clamp_max(max_scale)
So self.scale (initialized always to init_size = 1e-3) is always outside the clamp range (min_scale = 1e-2 and max_scale = 1 - 1e-3), and so always receives zero gradient.
Is it expected / by design or was it some debug setting min_scale = 1e-2 which by mistake was not removed?
The text was updated successfully, but these errors were encountered:
Hi @ines-chami!
At https://github.com/HazyResearch/HypHC/blob/master/model/hyphc.py#L42 :
So
self.scale
(initialized always toinit_size = 1e-3
) is always outside the clamp range (min_scale = 1e-2
andmax_scale = 1 - 1e-3
), and so always receives zero gradient.Is it expected / by design or was it some debug setting
min_scale = 1e-2
which by mistake was not removed?The text was updated successfully, but these errors were encountered: