Standardize spacy #347
Standardize spacy #347
Conversation
…o standardize_spacy
| @@ -13,15 +13,39 @@ | |||
| glove = None | |||
|
|
|||
|
|
|||
| def initialize_models(): | |||
| def initialize_models(model: str = "spacy", lang: str = "en"): | |||
There was a problem hiding this comment.
If we are making it generic, it would be great if we can create an enum for all heavy models which we want to load.
Because in future it may increase.
Something like:
LoadOnceModel.SPACY,
LoadOnceModel.GLOVE
| spacy_nlp = spacy.load("fr_core_news_sm") | ||
| elif model == "glove": | ||
| # load glove | ||
| glove = vocab.GloVe(name="6B", dim="100") | ||
|
|
There was a problem hiding this comment.
We should have an 'else' block also where we can throw an exception with an unsupported message. (if it doesn't match any model name)
| # load glove | ||
| glove = vocab.GloVe(name = "6B", dim = "100") | ||
| if model == "spacy": | ||
| if lang == "en": |
There was a problem hiding this comment.
Cosmetic Change (line 36-45): Better to create a map of 'lang' vs 'spacy model name' which will eliminate multiple lines of code.
| spacy_nlp = spacy.load("en_core_web_sm") | ||
| elif lang == "es": | ||
| spacy_nlp = spacy.load("es_core_news_sm") | ||
| elif lang == "zh": |
There was a problem hiding this comment.
To make it more informative, can we add a log message of whatever model we are loading as there are multiple models?
Something like:
"Loading zh_core_web_sm model of spacy......."
|
Sure @aadesh11. Will address the comments. This PR might take a while to complete :) |
|
@AbinayaM02 should we close this PR? |
Hi @kaustubhdhole: I wanted to standardize the loading of all spacy models in a single place. Not finding time to finish it. Let it be on draft and I'll try to close it once the first release is done (hopefully soon)! :) |
Fixes #339.