New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decoder Mask #62

Open

Muennighoff opened this issue Mar 5, 2021 · 0 comments

Muennighoff commented Mar 5, 2021

2 quick q's on the decoder:

Because of the neat cumsum, are we treating the decoder masks the same as the encoder masks (i can't find any different treatment in thecode except for the context route map) ?
the cumsum at the same time seems to be memory intensive; for an enc-dec / dec-only ~4096 toks text summarizer would you go with the performer (if yes enc-dec or discard the enc) ?

& awesome repo as always!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment