Struct NoriTokenizer 

pub struct NoriTokenizer { /* private fields */ }

Nori 토크나이저

Lucene Nori의 KoreanTokenizer와 호환되는 인터페이스

Implementations

impl NoriTokenizer

pub fn new( decompound_mode: DecompoundMode, output_unknown_unigrams: bool, ) -> Result<Self>

새 Nori 토크나이저 생성

Arguments

• decompound_mode - 복합명사 분해 모드
• output_unknown_unigrams - 미등록어 유니그램 출력 여부

Example

use mecab_ko_core::nori_compat::{NoriTokenizer, DecompoundMode};

let tokenizer = NoriTokenizer::new(DecompoundMode::Mixed, true).unwrap();

Errors

Returns an error if the internal tokenizer fails to initialize.

pub fn with_dict( dict_path: &str, decompound_mode: DecompoundMode, output_unknown_unigrams: bool, ) -> Result<Self>

사전 경로를 지정하여 생성

Errors

Returns an error if the tokenizer fails to load the dictionary.

pub fn tokenize(&mut self, text: &str) -> Result<Vec<NoriToken>>

텍스트를 Nori 스타일로 토큰화

Example

let tokens = tokenizer.tokenize("형태소분석").unwrap();
for token in tokens {
    println!("{}: {}", token.surface, token.pos_tag);
}

Errors

Returns an error if tokenization fails.