This is a brief summary of paper for me to study and simply arrange it, Character n-gram Embeddings to Improve RNN Language Models (Takase et al., AAAI 2019) I read and studied.

They propose character n-gram embedding with simplified Multi-dimensional self embedding as follows:

Takase et al., 2019 AAAI

The Multi-dimensional self embedding used the Mutli-dimensional self attention.

The Multi-dimensional self attention compute weights for each dimension of given embeddings and sumps up the weighted embeddings(i.e., element-wise weighted sum).

They used anothe way to improve perplexity. it is word tyiing which shared the embedidng matrix with weigth matrix to evaluate the probablity distribution.

Aslo They showed the their model is superior on the application task such as machine translation and headline generation tasks.

Note(Abstract): This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. They focus on character n-grams based on research in the field of word embedding construction. Their proposed method constructs word embeddings from character ngram embeddings and combines them with ordinary word embeddings. We demonstrate that the proposed method achieves the best perplexities on the language modeling datasets: Penn Treebank, WikiText-2, and WikiText-103. Moreover, their conduct experiments on application tasks: machine translation and headline generation. The experimental results indicate that their proposed method also positively affects these tasks.

Download URL:
The paper: Character n-gram Embeddings to Improve RNN Language Models (Takase et al., AAAI 2019)

Reference

Paper
- arXiv Version: Character n-gram Embeddings to Improve RNN Language Models (Takase et al. arXiv 2019)
- AAAI Version: Character n-gram Embeddings to Improve RNN Language Models (Takase et al., AAAI 2019)
How to use html for alert
- how to use icon
For your information
- How to use MathJax on stackoverflow

Character n-gram Embeddings to Improve RNN Language Models

Title of paper - Character n-gram Embeddings to Improve RNN Language Models

Character n-gram Embeddings to Improve RNN Language Models

Title of paper - Character n-gram Embeddings to Improve RNN Language Models

Reference