Reading notes from Wednesday 10/16.

Incorporating Latent Meanings of Morphological Compositions to Enhance Word Embeddings

Abstract

What is the problem the paper identifies?

Current methods of word embedding process language on the level of words rather than on the level of morphemes. Even when a word embedding process takes morphemes into account, they are directly incorporated, and ignore their latent meanings. The paper seeks to identify and train word embedding processes on these latent meanings.

What is the contribution the paper presents?

The authors propose three Latent Meaning Models (LMMs):

1. LMM-A
2. LMM-S
3. LMM-M

and run experiments on word similarity, syntactic analogy and text classification. This experimentation validates the increased performance of the three LMMs versus the traditional word embedding models.

Introduction

Give more detail about the specific problem the paper is addressing.

Word embedding is being frequently used as a sub-section of NLP, and is used in various tasks (text classification, information retrieval, sentiment analysis, machine translation). However, the current state-of-the-art word embedding methods typically capture semantic information based on words. The newest models go one step deeper, to morphemes; yet the authors propose that even this is not deep enough. Instead, word embeddings should be trained on the latent meanings of morphemes.

ex. unbelievable → (the word)

un - believe - able → (the morphemes)

not - believe - able - capable (the latent meanings of each morpheme)

Using these latent meanings will increase the similarity between appropriate cross pairs by reducing and quantifying the distance their word embeddings have in the resultant vector space.

The authors use three models for this task (see LMMs above), and verify their efficacies by comparing their test results to the current baseline of word embedding accuracy. Each LMM model outperforms the baseline word embedding models in areas of word similarity, syntactic analogy, and text classification.

What terms or concepts are you confused about, and how important do you think each will be to your understanding of the paper?

In the introduction, the paper mentions different practices for computing word embeddings: direct and statistical (regarded as Explicit). They then state that they use a different method entirely.

Direct computations find word embeddings by "adding the representations of morphemes/characters to context words or optimizing a joint objective over distributional statistics and morphological properties."

Statistical computations "introduce some probabilistic graphical models to build relationships between words and their internal compositions."

I do not understand either of these ways, nor do I understand the third direction the paper chooses to go.

Experiments / Results

How does the author show that their approach is successful? What evidence does the author provide? Is it compelling?

The authors test their LMMs against currently accepted word embedding models (CBOW, Skip-gram, and GloVe; see below in Background / Related Work) as well as their own Explicitly Morpheme-related Model (EMM). Use of the EMM allows for direct comparison between their LMMs and the explicit model.

The evaluation benchmarks used are

1. word similarity: used to measure the ability of word embeddings to capture semantic information. Performed 5 - 7% better than the generally accepted CBOW framework, and still better than EMM (which still was morpheme-based).
2. syntactic analogy: used to find d in a statement that follows the form "a is to b as c is to d," where d is unknown. Again, LMMs performed better than CBOW and EMM. The advantage lies in how similar words gather near to the latent meanings of their morphemes.
3. text classification: used to divide words into categories (e.g., guns, motorcycles, electronics, etc.). This has the greatest use in NLP. Again, LMMs performed better than CBOW and EMM.

Background / Related Work

How does the work in this paper build on what has been done before?

The paper summarizes existing technologies, and explains the benefits versus drawbacks of each one.

Word-level word embedding:

• based on neural networks: CBOW (Continuous Bag of Words) and Skip-gram both use neural networks to predict context with target words.
• based on matrix factorization: another method is to measure the correlation between the word information matrix and the context information matrix (for example, GloVe).

These "capture word-level semantic information while neglecting inner structures of words," which the authors aim to discourage in their own use of LMMs.

*note that due to its track record of high efficiency, the authors use CBOW to construct their own word embeddings.

Morphology-based word embedding:

• based on directly adding representations: splits words into morphemes and then directly adding those to the input layer.
• based on probabilistic graphical models: connects words with their morphological compositions

These are different from the work of this paper because they directly exploit internal compositions, and thus introduce unintended by-products. Using the latent meanings provides deeper insights for training word embeddings, and there are no by-products.

How is this work relevant to our own research?

The paper notes that the current baseline of word- and morpheme- level methods often ignore the meaningful inner structures of language. Therefore, their algorithms do not translate well to other languages. Breaking down language into smaller sub-meanings will help word embedding algorithms become universal.

"These methods can usually capture word-level semantic information but ignore the meaningful inner structures of words like English morphemes or Chinese characters."

Body