Initial release of grpc remote procedure call module. This is an experimental module that allows fast access to some functions of the project from other programming languages. We provide initial Python access codes for experimentation. Remote API is also subject to change until Version 1.0.0. Refer to the documentation for more information.
We now include a sentence normalization functionality. Before this, normalization module only provided simple word based spell check suggestion mechanism. Now. system offers a best effort text normalization functionality. We used several heuristics and applied language model based Viterbi search on candidate words collected with an offline contextual graph random walk algorithm.
We introduce a mechanism for analyzing Turkish informal words. For example, word okuycam, analysis
may be:
[okumak:Verb] oku:Verb+yca:FutInformal+m:A1sg
Informal morpheme names (like FutInformal) have Informal suffix.
For enabling informal morphological analysis, TurkishMorphology class should be initialized like this:
Builder b = TurkishMorphology.builder();
RootLexicon lexicon = DictionarySerializer.loadFromResources("/tr/lexicon.bin");
b.morphotactics(new InformalTurkishMorphotactics(lexicon));
b.useLexicon(lexicon);
TurkishMorphology morphology = b.build();
morphology.analyze("okumuycam");
In next releases probably there will be an easier way of enabling this mechanism. Note that ambiguity resolution mechanism may not work well if sentence contains informal morphemes. There is also a simple informal to formal conversion mechanism. For example:
TODO
Initial release of classification module. Classification module is based on our Java port of fastText project. Refer to the documentation for more information. There are two usage examples in examples module. There is also a Turkish wiki page on generating and evaluating classifier for Turkish news headline categories.
There is now a single jar containing all dependencies named zemberek-full.jar.
Initial release of apps module. All console applications will be in this module. When zemberek-full.jar is run directly, a catalogue of available applications are listed. They can be run directly bt the class name without the package name. For example:
java -jar zemberek-full.jar TrainClassifier
will run the TrainFastTextClassifier application and list available parameters.
OpenNlp NER annotation style is added.
toAscii method is added to TurkishAlphabet
Common foreign words with diacritic letters (like José) will not generate an exception warning.
Tokenizer will handle ellipsis … character correctly.
createWithTextDictionaries() method in TurkishMorphology and addDefaultDictionaries() in
TurkishMorphology.Builder are deprecated.
parseEntries() in SentenceAnalysis is deprecated.
Deprecated analyzeAndResolveAmbiguity(String sentence) in TurkishMorphology is removed.
Dictionary Rules and Morphemes wiki pages are added.
There is also word vector generation functionality but it is not yet documented.
There are more grpc services available but module is not yet ready for release.
This is a major release with improved Morphological disambiguation and initial release of NER module.
-
Morphological Disambiguation is improved and now became more usable.
-
There is now a Named Entity Recognition (NER) module for Turkish. This is our initial implementation and no pretrained NER model is provided yet. Users can train their own models. Please refer to the documentation for details.
-
Around 12 thousands lines of code is removed from project. Previous morphology and disambiguation code is gone.
-
Behavior for inputs proper nouns with apostrophe is changed. For example there will no longer be an analysis with root
obafor inputObama'ydı. However handling proper nouns with suffixes and apostrophes is still under development. -
After some cleaning, person names dictionary is added.
-
"…" is now considered a sentence boundary character.
-
Morphotactics improvements.
-
A Grpc server module is under development. Remote server functions can be accessed with multiple client libraries (Python, C#, Javascript etc.). Initial version will probably be available in the next release.
Breaking Changes:
analysisparameter in SentenceWordAnalysis class is nowbestAnalysis.- For consistency,
analyzeAndResolveAmbiguitymethod inTurkishMorphologyis changed toanalyzeAndDisambiguate. Old method still works but deprecated. addDefaultDictionaries()inTurkishNMorphology.Builderis deprecated. UseaddDefaultBinaryDictionaryinstead.- Maven distibutions now includes sources (via Ali Ok)
This is a bug fix and small changes release.
-
A problem in ambiguity resolution is fixed. It was throwing NPE for unknown words. (Issue #157)
-
Abbreviations are not allowed to have possessive suffixes.
-
Morphotactics improvements.
-
We started experimenting with server side modules.
This release is the result of some major refactoring of the Morphology module. There are many breaking changes.
Maven artifact id names have now zemberek- prefix.
Morphology module is re-written almost from scratch. Turkish morphotactics are now expressed in a simpler and more readable way in the code. New analyzer handles pronouns better and probably it generates more accurate results. But because this is a complete re-write, there might be new bugs and regressions.
Ambiguity resolution mechanism is changed. It now uses the old but popular Averaged Perceptron algorithm. For now, it is trained with the data generated from some corpora using simple rules. Therefore in this version disambiguation may not work so accurately. But it will improve in the upcoming releases quickly. Nevertheless, new module is probably working better than previous releases. Previous language model based algorithm is retired for now, but in future we may use a hybrid approach.
Default analysis representation is changed. Some examples:
kitap ("book", Noun, Singular.)
[kitap:Noun] kitap:Noun+A3sg
kitabımda ("in my book" Noun, Singular, First person possession, Locative)
[kitap:Noun] kitab:Noun+A3sg+ım:P1sg+da:Loc
dedim ("I told" Verb, past tense, first person singular)
[demek:Verb] de:Verb+di:Past+m:P1sg
diyerek ("By telling" Verb, derived to an adverb)
[demek:Verb] di:Verb|yerek:ByDoingSo→Adv
We decided to omit displaying implicit Pnon and Nom suffixes from nouns to make it more readable.
This format is probably not final. We consider changing some morpheme names and refine the representation.
We now use Caffeine for caching analysis results. There are static and dynamic caches for speeding up the word analysis.
Word generation mechanism is also re-written.
Dictionary serialization mechanism is written using protocol-buffers.
Now initialization of TurkishMorphology class is faster.
There are Email, Url, Mention, HashTag, Emoticon, RomanNumeral, RegularAbbreviation, Abbreviation secondary POS information.
We added examples module. This is like a copy of turkis-nlp-examples project. Users can see
high level usage examples there.
FAQ and front page Readme files are updated.
Z3AbstractDisambiguator, TurkishMorphDisambiguator, Z3AbstractDisambiguator, Z3MarkovModelDisambiguator, Z3ModelA removed from morphology modue.
TurkishSuffixes, TurkishSentenceAnalyzer, WordAnalyzer, SimpleGenerator, DynamicLexiconGraph, DynamicSuffixProvider, SuffixData, SuffixSurfaceNode, StemNode, StemNodeGenerator, Suffix, SuffixForm, SuffixProvider, SuffixSurfaceNodeGenerator are removed from morphology modue.
TurkishMorphology analysis methods now return WordAnalysis object instead of List<WordAnalysis>
. WordAnalysis contains a List<SingleAnalysis> where analysis details can be reached. Methods like
getStem() or getLemmas() are moved from WordAnalysis to SingleAnalysis.
Generation is now handled by WordGenerator class. generation rules are changed so that if user does not
provide empty surface morphemes, system search through them anyway. Check GenerateWords example class.
System memory footprint is reduced. Analysis performance may be a bit slower but with cache, impact should be small. We will provide measurements later.
We wrote a port of Facebook's FastText library in Java. It can be used for word embeddings and classification tasks. However it is not yet ready for release.
There is an experimental Named Entity Recognition module. But it is not yet ready for release.
TurkishSpellChecker now can load internal resources correctly.
We made a lot of changes in Tokenization module. Some of them are breaking changes.
Package name is changed from zemberek.tokenizer to zemberek.tokenization for consistency.
Now there is a better sentence extraction class called TurkishSentenceExtractor. This class can split documents and paragraphs into sentences. It uses rules and a simple binary averaged perceptron algorithm for finding sentence boundaries. LexerSentenceExtractor is removed.
Tokenization is also improved. More token types are introduced. Including Date, Hashtag, Mention, URL, Email and Emoticon. However these tokens are not included as a POS type in morphological analysis yet. This will be done in upcoming versions. We changed the ZemberekTokenizer name to TurkishTokenizer. There are some low-level breaking changes. Token TurkishWord is now Word and TurkishWordWithApos is WordWithApostrophe.
Please refer to documentation for both sentence extraction and tokenization usage examples and test results.
Zemberek now includes an alpha level spell-checker. This spell checker is intended for well formed documents. It is not for automatic spell correction / normalization. Basically it can check if an individual word is written correctly and give suggestions for a word. Suggestions are ranked against a small unigram language model for now. We will improve the spell checker in later versions.
Plase refer to documentation for usage examples.
There are no big changes in morphology in this version. However several addition and fixes were made in dictionaries. Including around 100 more Adverbs and several Question type fixes. Suffix morphotactics for Pronouns are still mostly broken. Please refer to these commits for changes: 1 2 3 4 5
There was yet another memory leak fixed on use of HashMaps in RootLexicon.
We have started a new morphotactics work in experiment module. But it is not yet usable.
This module is removed and content is moved to core module for now. It was not holding much weight and code requires an overhaul. Before 1.0.0 we may move it to normalization module.
- Fixed a memory leak. Previous versions may suffer from this under heavy load. 87
- Re-introduce lang-id module. This provides a simple language identification API.
- Added normalization module. So far it only provides fast Levenshtein distance dictionary matching.
- Added an experiment module. This module will be used for experimental features.
- Dictionary fixes.
- Added city, village and district names.
- System can generate full jar containing all zemberek modules.
- Speed up Antlr based tokenizer. Now it is three times faster again. 89
- TurkishMorphology can be configured for not using cache and UnidentifiedTokenAnaysis.
- Eliminate static cache from TurkishMorphology 86
- Fix: Some inputs may cause excessive hypothesis generation during analysis 88
- Fix: Proper Nouns ending -nk or -og should not have Voicing attribute automatically.83
- Fix: "foo \nabc" should be tokenized as "foo \n abc" 69
- There are some name changes. TurkishMorphology.TurkishMorphParserBuilder -> TurkishMorphology.Builder UnidentifiedTokenAnalyzer parse -> analyze
- Improved morphological analysis coverage by cross checking with Oflazer-Analyzer. For this, a list of more than 7 million words are extracted from a 2 billion word corpora. Then a list of words that can be analyzed only by Oflazer-Analyzer is generated and Zemberek is fixed as much possible.
- Breaking change: zemberek.morphology.parse package is now zemberek.morphology.analysis 67
- Breaking change: Several classes are renamed.
TurkishWordParserGenerator -> TurkishMorphology
SentenceMorphParse -> SentenceAnalysis
MorphParse -> WordAnalysis
WordParser -> WordAnalyzer 67 - Breaking change: Methods with name "parse" are renamed to "analyze". 67
- Custom Antlr dependency is removed. We now use latest stable Antlr version. We decided this because maintaining a patched version of Antlr was time consuming. We were using such a fork because original version was around 3 times slower. But current speed is good enough. We may remove Antlr dependency altogether in future releases because tokenization should be less strict and it should not do detailed classification. [68] (ahmetaa#68)
- Add Oflazer compatible secondary POS information for Postp. 65
- Tokenization problem after capital letters after apostrophe. 64
- Cannot parse diyebil-, diyecek-, diyen-. 61
- Proper nouns should not have Voicing attribute automatically. 57
- Can parse reçelsi but not reçelimsi. 54
- Cannot parse maviceydi, yeşilcedir. 53
- Cannot parse "soyadları" 55
- Wrong start and stop indexes for abbreviation words on tokenization. 51
- Fixes in caching mechanism in TurkishMorphology.
- Added a dependency module. It does not perform parsing yet.
A lot of internal code changes. Added static and dynamic cache mechanisms for word parsing.
- Can parse [abdye ABDye] but not [abd'ye] [ABD'ye] #44
- Cannot parse words : [ cevaplandırmak çeşitlendirmek ] #42
- System can parse [ankaraya] but not [ankara'ya] #40
- Add ability to add a new Dictionary Item in run-time. #37
- resource test-lexicon-nouns.txt not found #36 (elifkus)
- Garip bir tokenization and stem problemi #30
- Cannot parse the word: yiyen #25 (volkanagun)
- First unstable public release.
- Removed language identification and spelling modules. They are different applications now.