Runing T2T Teachers

CUNI has trained models with Tensor-to-Tensor. This page describes how to run them.

1. Set up a python virtual environment with python 3.5 as the default binary and install tensorflow-gpu==1.15.3, tensor2tensor==1.15.5, tf-datasets==3.1.0.

   virtualenv ~/venv/t2t -p python3.5
   . ~/venv/t2t/bin/activate
   pip install tensor2tensor==1.15.5 tensorflow-gpu==1.15.3 tf-datasets==3.1.0 --use-feature=2020-resolver
   

   Do NOT install the cpu-version of tensorflow! In that case tensorflow will run on CPU even if you specify a GPU. If the above doesn't suffice, find a full requirements.txt here.
2. Split the data you want to translate into chunks of about 10K sentences. I found Tensor2Tensor very slow, and it won't start writing output until it's finished translating everything. My first batch (1M sentences) took well over a week to translate.
3. I have put a wrapper script in train-students/t2t/ that I used on CSD3. It's pretty unpolished and rough around the edges, as it was meant to be a one-off thing. I found that batch sizes of 16 or even 8 are the safest with respect to t2t not crashing with OOM errors.
4. T2T will put the n-best translation in tab-separated fields in a single line (unlike Marian). Run train-student/data/bestbleu.py with -t t2t to make things work.
5. You will need to train a new SentencePiece model for the students, as T2T incorporates everything in its models and it's hard to get them out. For training the SentencePiece model, I recommend using a vocabulary coverage of 100%, as I've noticed that using the default leads to some characters (including '=') being declared UNK by a SentencePiece model trained with the default coverage.