This paper presents a mechanism of new word identification in Chinese text where
probabilities are used to filter candidate character strings and to assign POS to the selected …

One of the most prominent problems in computer processing of the Chinese language is
identification of the words in a sentence. Since there are no blanks to mark word boundaries …

We present a novel new word extraction method from Japanese texts based on expected
word frequencies. First, we compute expected word frequencies from Japanese texts using a …

Statistical methods for extracting Chinese unknown words usually suffer a problem that
superfluous character strings with strong statistical associations are extracted as well. To …

There is no blank to mark word boundaries in Chinese text. As a result, identifying words is
difficult, because of segmentation ambiguities and occurrences of unknown words …

We present a study of new word identification (NWI) to improve the performance of a
Chinese word segmenter. In this paper the distribution and types of new words are …

We introduce a character-based chunking for unknown word identification in Japanese text.
A major advantage of our method is an ability to detect low frequency unknown words of …

Chinese new words are particularly problematic in Chinese natural language processing.
With the fast development of Internet and information explosion, it is impossible to get a …

Word Identification has been an important and active issue in Chinese Natural Language
Processing. In this paper, a new mechanism, based on the concept of sublanguage, is …

An unsupervised iterative approach for extracting a new lexicon (or unknown words) from a
Chinese text corpus is proposed in this paper. Instead of using a non-iterative segmentation …