[KeyError] during train_model()

[TonyZhang03]

I am trying to follow this post to train a convai model, but hit KeyError. Thanks for advice.

import simpletransformers
from simpletransformers.conv_ai import ConvAIModel, ConvAIArgs
import torch
cuda_available = torch.cuda.is_available()
model_args = ConvAIArgs()
model_args.max_history = 5
model_args.max_length = 30
model_args.min_length = 3
model_args.num_train_epochs = 50
model_args.save_model_every_epoch = False
model_args.overwrite_output_dir = True
model_args.reprocess_input_data = True

model = ConvAIModel(
    "gpt",
    "gpt_personachat_cache",
    use_cuda=cuda_available,
    args=model_args
    
)
model.train_model("personachat_self_original.json")  

training took a while, at the end, it yield a KeyError. Below is error message:

KeyError                                  Traceback (most recent call last)
<ipython-input-4-39e59a5e3218> in <module>
----> 1 model.train_model("personachat_self_original.json")

~\anaconda3\envs\bot_env\lib\site-packages\simpletransformers\conv_ai\conv_ai_model.py in train_model(self, train_file, output_dir, show_running_loss, args, eval_file, verbose, **kwargs)
    223         self._move_model_to_device()
    224 
--> 225         train_dataloader, train_sampler = self.load_and_cache_examples(
    226             dataset_path=train_file, verbose=verbose, no_cache=self.args.no_cache or self.args.reprocess_input_data,
    227         )

~\anaconda3\envs\bot_env\lib\site-packages\simpletransformers\conv_ai\conv_ai_model.py in load_and_cache_examples(self, dataset_path, evaluate, no_cache, verbose, silent)
    762         # for dataset_name, dataset in personachat.items():
    763         datasets = defaultdict(list)
--> 764         num_candidates = len(dataset[0]["utterances"][0]["candidates"])
    765         if args.num_candidates > 0 and not evaluate:
    766             num_candidates = min(args.num_candidates, num_candidates)

KeyError: 0

[TonyZhang03]

BTW, the error yielded only after training took place quite a while, I am not sure the training finished or not, but from progress bar, it seems finished...I tried interact(), it worked. not sure how critical this error is.
if this error can be neglected, what is load model method for me to load the trained model into my python code? what is method to use the model as chatbot engine to generate "response" and talk to human?

Update: it seems training not finished, I can't find "outputs/" directory and no model saved.

Update again, it seems my bad, when I let model download training json file by itself, no this KeyError, though another error....it looks like something wrong with the json training file I downloaded. The new error is not enough memory, I will try it in Colab. Thanks

[TonyZhang03]

it is weird that training in Google colab got same "out of memory" error as in local machine.

RuntimeError: CUDA out of memory. Tried to allocate 518.00 MiB (GPU 0; 11.17 GiB total capacity; 10.36 GiB already allocated; 182.81 MiB free; 10.57 GiB reserved in total by PyTorch)

Does this model training require huge memory or is it somewhere I did wrong? can anyone help?

Update: I tried None (GPU), also run out of memory and cause kernel restart at first epoch. I checked that Colab give 12.8GB RAM for both GPU and None run time. So where is problem? This is my first time to use SimpleTransformers.

[TonyZhang03]

below is GPU run error:

/usr/local/lib/python3.7/dist-packages/transformers/models/openai/modeling_openai.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, mc_token_ids, labels, mc_labels, output_attentions, output_hidden_states, return_dict)
    713             mc_loss = loss_fct(mc_logits.view(-1, mc_logits.size(-1)), mc_labels.view(-1))
    714         if labels is not None:
--> 715             shift_logits = lm_logits[..., :-1, :].contiguous()
    716             shift_labels = labels[..., 1:].contiguous()
    717             loss_fct = CrossEntropyLoss()

RuntimeError: CUDA out of memory. Tried to allocate 518.00 MiB (GPU 0; 11.17 GiB total capacity; 10.36 GiB already allocated; 182.81 MiB free; 10.57 GiB reserved in total by PyTorch)

[ThilinaRajapakse]

CUDA memory error means that the GPU does not have enough memory to train the model with the specified hyperparameters. You can try to reduce the train_batch_size and max_seq_length. This will result in less memory being used.