Link to the assignment here.
Incorporating Latent Meanings of Morphological Compositions to Enhance Word Embeddings
Yang Xu, Jiawei Liu, Wei Yang, and Liusheng Huang. July 2018. Incorporating Latent Meanings of Morphological Compositions to Enhance Word Embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://www.aclweb.org/anthology/P18-1114.pdf.
Incorporating Latent Meanings of Morphological Compositions to Enhance Word Embeddings
Word embedding is being frequently used as a sub-section of NLP, and is used in various tasks (text classification, information retrieval, sentiment analysis, machine translation). However, the current state-of-the-art word embedding methods typically capture semantic information based on words. The newest models go one step deeper, to morphemes; yet the authors propose that even this is not deep enough. Instead, word embeddings should be trained on the latent meanings of morphemes. Using these latent meanings will increase the similarity between appropriate cross pairs by reducing and quantifying the distance their word embeddings have in the resultant vector space. The authors use three models for this task and verify their efficacies by comparing their test results to the current baseline of word embedding accuracy. Each LMM model outperforms the baseline word embedding models in areas of word similarity, syntactic analogy, and text classification.
This paper is very relevant to our research topic, as it provides a good background for understanding word embeddings while improving on the state-of-the-art methods used at the time of its publication. I think it is important for us to understand the mechanics of language modeling as well as concepts of fairness, which this work provides.
What Kind of Language is Hard to Language-Model?
Sebastian J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. July 2019. What Kind of Language is Hard to Language-Model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. https://www.aclweb.org/anthology/P19-1491.pdf.
What Kind of Language Is Hard to Language-Model?
Current NLP models can be used on any language but do not always perform equally well on all languages. There is not a clear body of work providing empirical studies on this phenomenon, as our high-resource languages are typically European (and thus not very typologically diverse) and we do not have significant corpora on others. The previous work of the authors used Europarl to control for differences in content to use a fully parallel text corpus. In this paper, they introduce a method of comparison across corpora that are not wholly parallel, and thus are able to expand the extent of their empirical study (with the use of 62 languages from 13 language families, along with the original 21 languages from Europarl). In terms of defining the difficulty of a language for modeling, the authors also fail to reproduce their previous findings concerning morphological complexity as an indication of such difficulty. Instead, they introduce a difficulty parameter that accounts for variance in datasets and can deal with missing translations. They also propose that the difficulty of a language's ability to model is based on simpler terms: word inventory and the raw character sequence length. Lastly, the paper questions whether the use of "translationese" is appropriate for NLP, as their metric for language difficulty shows that it is different from other languages but not any easier.
This work raises questions that are very similar to our own research topic. Since we are aiming to collect empirical evidence that not all languages perform equally well across today's NLP models, this paper's conclusions about what features of a language make it difficult to model are highly relevant. Ultimately, however, their paper is inconclusive.
Low-resource Cross-lingual Event Type Detection in Documents via Distant Supervision with Minimal Effort
Aldrian Obaja Muis, Naoki Otani, Nidhi Vyas, Ruochen Xu, Yiming Yang, Teruko Mitamura, and Eduard Hovy. August 2018. Low-resource Cross-lingual Event Type Detection in Documents via Distance Supervision with Minimal Effort. In Proceedings of the 27th International Conference on Computational Linguistics. https://www.aclweb.org/anthology/C18-1007v2.pdf.
Low-resource Cross-lingual Event Type Detection via Distant Supervision with Minimal Effort
The current established way of applying NLP methods to low-resource languages its to transfer the learning process from a high-resource language using a shared word representation (bilingual dictionary), which models are then trained on. But this approach assumes that there is a large amount of labeled data in the high-resource language. The paper introduces two methods of solution: first, a simple keyword-matching system that uses a bilingual dictionary and minimal external verification, and second, a neural adversarial network that learns bilingual word representations for information transfer. These two are found to work equally well in the experiment the paper utilizes: a situation frame (SF) detection task. A SF is either an issue or a need from a news article; situation frame detection is the task of classifying SFs (e.g., in an article about a food shortage, classification as a food need). Lastly, the paper proposes methods for combining the two methods outlined above to maximize performance, and analyzes their results.
Another topic we hope to touch on in our literature review is the issue of utilizing low-resource languages. Because previous about the disparity between NLP models across languages often does not address the different sizes of test corpora, we are aiming to analyze how a language's lack of quality data may be a factor in its ability to model well. Thus, this paper is relevant.