Using own retrieval results

[AndresFRJ98]

Hello. I am attempting to experiments with using my own retrieval results and assessing the impact on the performance of the model (for the full-wiki setting)

If I understood correctly, the process would be as follows.

Generate my own version of hotpot_dev_fullwiki_v1.json in which the context entry of the JSON has been modified with my own paragraphs. All the other keys should be left identical. I shall refer to this json file as custom_dev_fullwiki.json.

Likewise for the test set, to create custom_test_fullwiki.json

Then, I would run the preprocessing , training and evaluation commands as specified in the README.

To make meaningful comparison of metrics, once I change my retrieval method and generate a 'v2.json' set of files, I should start again from scratch on preprocessing, then train etc.

Is this correct?

[ManuelRios18]

Yeah, I think you are right :)