Corpus improvements

[Muennighoff]

This is an issue to collect all corpus improvements. It costs a few thousand USD to update all corpora & indices, so it does not make sense to do it for every issue we find, but collect a bunch of issues and then update them together with also including all new data produced on the respective website at that date! Feel free to add things directly / link to issues.

Wikipedia:

• Wikipedia corpus tables ignored #15
• Switch to MegaWika v2 with better processing - Final launch TODOs #12 (comment)

Arxiv:

• Nothing thus far from me; feel free to add

StackExchange

• Stackexchange should be StackExchange in the text
• Maybe add some more exchanges, e.g. AI
• Remove samples without answers e.g. Why must Voldemort's body be killed last, see screen