Transformer number of heads

[bitfort]

Currently number of heads in attention if 16, proposal is to move to 8 heads with the understanding that it achieves the same quality with better performance.

[bitfort]

SWG Notes:

This reduces computation and Google has data to show this does not reduce quality. Also, this is commonly used in production.

AI(Google) spread sheet showing difference in runs.

[danielcdh]

We drafted a documents to analyze the computation requirements as well as convergence experiments with this change: https://docs.google.com/a/google.com/document/d/e/2PACX-1vR3qcsQSL6r4xvHQP9-R40Rq33qSF5yqm47esWRTbRPzremPYs6-ZNqpSypiyYXyRdE-D7VLayEUY_c/pub

[David-Levinthal]

there seem to be permission access issues getting to the doc??? You need permission to access this published document. You are signed in as *[email protected] <[email protected]>*, but you don't have permission to access this published document. You may need to sign in as a different user <https://docs.google.com/document/d/e/2PACX-1vR3qcsQSL6r4xvHQP9-R40Rq33qSF5yqm47esWRTbRPzremPYs6-ZNqpSypiyYXyRdE-D7VLayEUY_c/logout> .

…

On Thu, Feb 21, 2019 at 9:53 AM Dehao Chen ***@***.***> wrote: We drafted a documents to analyze the computation requirements as well as convergence experiments with this change: https://docs.google.com/a/google.com/document/d/e/2PACX-1vR3qcsQSL6r4xvHQP9-R40Rq33qSF5yqm47esWRTbRPzremPYs6-ZNqpSypiyYXyRdE-D7VLayEUY_c/pub — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <mlcommons/policies#172 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIUuT2xM6xE_VWs9hi0yFeU8_bYKHVLSks5vPt0VgaJpZM4aGi8A> .

[petermattson]

SWG: Seems tentatively OK but want to check with customers and will finalize next week.