Lindera ko-dic Builder

ko-dic dictionary builder for Lindera. This project fork from fulmicoton's kuromoji-rs.

Install

% cargo install lindera-ko-dic-builder

Build

The following products are required to build:

• Rust >= 1.46.0

% cargo build --release

Build small binary

You can reduce the size of the dictionary by using the "compress" feature flag.
Instead, it can only be used with Lindera, which supports compression.

This repo example is this.

% cargo build --release --features compress

It also depends on liblzma to compress the dictionary. Please install the dependent packages as follows:

% sudo apt install liblzma-dev

Dictionary version

This repository contains mecab-ko-dic-2.1.1-20180720.

Building a dictionary

Building a dictionary with lindera-ko-dic command:

% KO_DIC_VERSION=2.1.1-20180720
% curl -L -O "https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-${KO_DIC_VERSION}.tar.gz"
% tar zxvf ./mecab-ko-dic-${KO_DIC_VERSION}.tar.gz
% lindera-ko-dic-builder -s ./mecab-ko-dic-${KO_DIC_VERSION} -d ./lindera-ko-dic-${KO_DIC_VERSION}

Dictionary format

Information about the dictionary format and part-of-speech tags used by mecab-ko-dic id documented in this Google Spreadsheet, linked to from mecab-ko-dic's repository readme.

Note how ko-dic has one less feature column than NAIST JDIC, and has an altogether different set of information (e.g. doesn't provide the "original form" of the word).

The tags are a slight modification of those specified by 세종 (Sejong), whatever that is. The mappings from Sejong to mecab-ko-dic's tag names are given in tab 태그 v2.0 on the above-linked spreadsheet.

The dictionary format is specified fully (in Korean) in tab 사전 형식 v2.0 of the spreadsheet. Any blank values default to *.

Index	Name (Korean)	Name (English)	Notes
0	품사 태그	part-of-speech tag	See 태그 v2.0 tab on spreadsheet
1	의미 부류	meaning	(too few examples for me to be sure)
2	종성 유무	presence or absence	T for true; F for false; else *
3	읽기	reading	usually matches surface, but may differ for foreign words e.g. Chinese character words
4	타입	type	One of: Inflect (활용); Compound (복합명사); or Preanalysis (기분석)
5	첫번째 품사	first part-of-speech	e.g. given a part-of-speech tag of "VV+EM+VX+EP", would return VV
6	마지막 품사	last part-of-speech	e.g. given a part-of-speech tag of "VV+EM+VX+EP", would return EP
7	표현	expression	활용, 복합명사, 기분석이 어떻게 구성되는지 알려주는 필드 – Fields that tell how usage, compound nouns, and key analysis are organized

Tokenizing text using produced dictionary

You can tokenize text using produced dictionary with lindera command:

% echo "하네다공항한정토트백" | lindera -d ./lindera-ko-dic-2.1.1-20180720

하네다  NNP,인명,F,하네다,*,*,*,*
공항    NNG,장소,T,공항,*,*,*,*
한정    NNG,*,T,한정,*,*,*,*
토트백  NNG,*,T,토트백,Compound,*,*,토트/NNP/인명+백/NNG/*
EOS

For more details about lindera command, please refer to the following URL:

• Lindera CLI

API reference

The API reference is available. Please see following URL:

• lindera-ko-dic-builder