Can I implement online learning on a big CSV Dask dataset?

[jorgesisco]

I have 80 csv files that I loaded them all in one dataframe using Dask, after transforming the data using onehotencoding, I get a really huge dataset impossible to train even with large instances on amazon sagemaker.

I am trying to find an alternative by training the model with partial data but using the whole dataset at the end, is this possible?

Any advice would be welcome 🙏

[MaxHalford]

Hello. Yes, I would say it's possible with River. What is blocking you? Have you tried using River?

[jorgesisco]

Yes I am using it right now for the 1st time. As far as I understand I have to provide the data to the model in a dict format, my theory is I need to figure it our how to iterate each data point (row), convert it to a dict and apply .learn_one, update. and so on.

The only thing that confuses me on river, is why would I predict and update that value if I have not trained the model yet?

Ps. Looking for an online learning approach I found River, looks promising.

Best Regards.

[MaxHalford]

The only thing that confuses me on river, is why would I predict and update that value if I have not trained the model yet?

So usually River is meant for pure online learning. The data is expected to come from a stream. But in your case you just want to train on a big dataset. Usually I would recommend trying scikit-learn's models that support partial_fit, which will do the job more than fine. With River, you can disable the need for the predict_one method call, see here. I would also recommend reading this notebook, which might be relevant to you.

Good luck!

[jorgesisco]

Thanks a lot for your help, actually I am interested on that stream data approach, because this is an initial training, I intent to deploy this model to make predictions. But in the future we are going to need to train the model with new data, I am assuming it is possible right?

Edit: This is my last question to you, I promise!

[MaxHalford]

I am assuming it is possible right?

Yes of course! That's the main selling point of online learning. We're just lacking a bit in terms of "warming up", as in pre-training your model on an existing big dataset.

Edit: This is my last question to you, I promise!

It's never a problem :)