konlp.tokenize package

Subpackages

• konlp.tokenize.klt package
  • Submodules
  • konlp.tokenize.klt.autospace module
  • Module contents

Submodules

konlp.tokenize.api module

Korean Natural Language Toolkit tonkenizer interface

class konlp.tokenize.api.SimpleTokenizer

Bases: konlp.tokenize.api.TokenizerI

For an example about how to inherit the class above

tokenize(string)

Simple string tokenizer by white-space character

Parameters:string (str) – String to tokenize Returns:Tokenized tokens Return type:str

class konlp.tokenize.api.TokenizerI

Bases: object

Tokenizer Interface

tokenize(string)

Return a tokenized copy of string.

Parameters:string (str) – String to tokenize Returns:Tokenized tokens Return type:list(str) Raises:NotImplementedError – If not implement this method on a class that extends this class

Module contents

we are preparing this module