Stream clustering for text-based data

[Dennis1989]

Hey,

I was wondering if you are interested in supporting stream clustering algorithms working on textual inputs. There are some recent developments in the field, and we would be willing to work on a wrapper for our textClust algorithm.

Best regards,

Dennis

[MaxHalford]

Hey Dennis 👋

In theory, nothing is stopping you from building a TFIDF + clustering_model pipeline in River. But maybe there's some more specific things to put in place. Anyway, I'm not opposed to a contribution in this domain. I would however prefer if that contribution would make use of existing River building blocks, instead of being a wrapper on top of an external library. I could reconsider if the textClust is doing something River does not provide.

I hope that's clear enough! Let me know if not.

[hoanganhngo610]

@MaxHalford In fact, this is indeed what we have been looking for in a while, with a lot of requests about text clustering. As said, we can easily have a TFIDF + clustering pipeline, but it is still much cooler having a native text clustering algorithm in hand. And textClustPy seems to do that in a perfect fashion.

@Dennis1989 You have done great work on the package, and I extremely appreciate it. I know that the request of @MaxHalford is huge, but if you want to extend the base Clusterer and add your algorithm into River, I would certainly be extremely happy to help and try my best to make sure that it reaches the greatest audience possible out there.

The work flow is also quite simple and straightforward, as Max said. I have also recently delivered a speech on online clustering with River at KDD 2022, so an example notebook with a running algo on River would be a perfect addition to this.

[Dennis1989]

That sounds excellent @MaxHalford and @hoanganhngo610 !!

I think it shouldn't be too hard to port the algorithm to River, and I agree that there is a high chance that our algorithm will get a significant popularity boost. So for the sake of science (and application), this is the right thing to do :)

I will spend some time preparing everything. Providing an example via a notebook indeed is probably super helpful for users.

I would like to start with the core algorithm. To evaluate existing benchmark datasets, this should be sufficient. However, I think one advantage of our implementation is that we offer the capability to directly hook on a live Twitter Stream (which is probably the most prominent use-case for a textual stream cluster setting, at least from an application point-of-view). Maybe, in the long run, this could also be something that could be included.

[MaxHalford]

I think it shouldn't be too hard to port the algorithm to River, and I agree that there is a high chance that our algorithm will get a significant popularity boost. So for the sake of science (and application), this is the right thing to do :)

Great, thanks for having a positive mental attitude :)

I will spend some time preparing everything. Providing an example via a notebook indeed is probably super helpful for users.

Cool, feel free to ask questions if you're unsure.

However, I think one advantage of our implementation is that we offer the capability to directly hook on a live Twitter Stream (which is probably the most prominent use-case for a textual stream cluster setting, at least from an application point-of-view). Maybe, in the long run, this could also be something that could be included.

I'll take care of this! I'll even add a hook to Twitch chat streams, if there's an API for it. I'll ping you in a pull request of two :)

[Dennis1989]

Hey,

I have two architectural questions that I currently face while porting the algorithm.

1. I use your BagOfWord implementation to get my list of ngrams, which is then used as the basis for micro-cluster creation. Here I have two options: Store the BoW object as a member in my textclust instance or go the pipeline route. However, as textclust requires BoW input as it is fundamentally built upon TF-IDF, it would always be the same pipeline, which means unnecessary programming lines for users.

2. Although we use TF-IDF which already prevents frequent words from having great influence, we additionally remove stop words directly each time a new observation arrives at the stream. Does it make sense to override the preprocessor step?

Best,

Dennis

[MaxHalford]

Hey hey.

I use your BagOfWord implementation to get my list of ngrams, which is then used as the basis for micro-cluster creation. Here I have two options: Store the BoW object as a member in my textclust instance or go the pipeline route. However, as textclust requires BoW input as it is fundamentally built upon TF-IDF, it would always be the same pipeline, which means unnecessary programming lines for users.

I would suggest going with the pipeline route. I get it that you would always use a BoW before textClust, but you really never know how creative users want to be. For instance, what if users only have token counts and not texts? The pipeline route is more flexible that way, as it allows for more creative use. It also makes apparent what is going on. You can still make sure users use the BoW by providing a good example in the docstring. As an existing example in River, the linear regression should almost always be preceded by a standard scaler, but we make this happen with a pipeline too.

Although we use TF-IDF which already prevents frequent words from having great influence, we additionally remove stop words directly each time a new observation arrives at the stream. Does it make sense to override the preprocessor step?

Good question. The short answer is that you can indeed use the preprocessor function to remove stop words with a function. But we could also make this explicit by adding a stop_words parameter. From this we can build the function ourselves in VectorizerMixin, which BagOfWords and TFIDF inherit from. If it's ok with you, I can take care of this :)