Transformer appears to be outdated, lower BLEU

[ableiweisss]

The Transfomer code in MLPerf is a bit outdated, and has a BLEU score lower than the official TensorFlow version:

https://github.com/tensorflow/models/tree/master/official/transformer

[tremblerz]

Hi @ableiweisss
Would you like to point out what aspect of Transformer appears to be outdated? Is it the preprocessing module, main model or the code performing evaluation?

Regarding the BLEU score, there was some discussion before also whether it should be kept the same as the score reported in the original paper. My understanding is that, once the BLEU score has crossed value of 22~23, further improvement in the BLEU score is marginal with respect to the number of training epochs and can bring certain inconsistency in training time due to random seed ( see mlcommons/training#125 ).

[ableiweisss]

Hi @tremblerz
The official TensorFlow repo reports a BLEU score of 27.7 for base and 28.9 for big. MLPerf reports 25, which is significantly lower. Is this due to code differences or rather as a target simply related to minimizing the number of epochs?

[ddutta]

Maybe we will take a deeper look for 1.0 .... thanks for pointing that out. At this point, for 0.5, maybe we could just let this version be.

…

On Thu, Oct 11, 2018 at 9:13 AM ableiweisss ***@***.***> wrote: Hi @tremblerz <https://github.com/tremblerz> The official TensorFlow repo reports a BLEU score of 27.7 for base and 28.9 for big. MLPerf reports 25, which is significantly lower. Is this due to code differences or rather as a target simply related to minimizing the number of epochs? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <mlcommons/policies#112 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA1h-ubTh2v13HmoKGvE8p1eIbd8NnzPks5uj24YgaJpZM4XXRv4> .

-- -Debo~

[petermattson]

SWG:
We plan to raise for 5.1.