v gates

[evanatyourservice]

fwiw i've found v gates to cause unstable training, with some weird gradient explosions in early layers and NaNs, when the networks are very large. It helps with smaller networks, but doesn't train with large networks.

[lucidrains]

oh interesting, thanks for sharing this finding

what happens if you replace the SiLU with a nn.Sigmoid()?

[evanatyourservice]

Ah yeah let me try that, I was actually wondering if the problem is the silu or not bounding activation but didn’t try any replacements.

[lucidrains]

how did it go?