Reading notes from Wednesday 10/23.
What Kind of Language Is Hard to Language-Model?
Previous paper referenced:
Are All Languages Equally Hard to Language-Model?
What is the problem the paper identifies?
Currently accepted NLP models do not work equally well across languages. In their previous work, the authors found that neural network language models do not perform equally well over all high-resource European languages.
What is the contribution the paper presents?
The authors introduce a paired-sample multiplicative mixed-effects model to determine language difficulty coefficients from pairwise parallel corpora (their model can handle missing data). They show that "translationese" is not any easier to model than the difficult languages themselves, and disprove a conclusion from their previous work (that difficult languages are categorized by inflectional morphology).
Give more detail about the specific problem the paper is addressing.
Current NLP models can be used on any language but do not always perform equally well on all languages. There is not a clear body of work providing empirical studies on this phenomenon, as our high-resource languages are typically European (and thus not very typologically diverse) and we do not have significant corpora on others.
The previous work of the authors used Europarl to control for differences in content to use a fully parallel text corpus. In this paper, they introduce a method of comparison across corpora that are not wholly parallel, and thus are able to expand the extent of their empirical study (with the use of 62 languages from 13 language families, along with the original 21 languages from Europarl).
In terms of defining the difficulty of a language for modeling, the authors also fail to reproduce their previous findings concerning morphological complexity as an indication of such difficulty. Instead, they introduce a difficulty parameter that accounts for variance in datasets and can deal with missing translations. They also propose that the difficulty of a language's ability to model is based on simpler terms: word inventory and the raw character sequence length.
Lastly, the paper questions whether the use of "translationese" is appropriate for NLP, as their metric for language difficulty shows that it is different from other languages but not any easier.
What contributions does the work provide?
How does the author show that their approach is successful? What evidence does the author provide? Is it compelling?
(2) The Surprisal of a Sentence
The authors introduce methods of fair comparison by controlling for variables in language other than intent, such as topic, meaning, style, and information density.
To do this, the paper uses a multi-text, which has direct translations of the same content. (These datasets are Europarl and the Bible.)
The authors then define surprisal, which is a metric used to measure the difficulty of representing a word (by the number of bits needed in a compression scheme). Surprisal reflects both word complexity and the model's ability to anticipate the word. In other words, having high surprisal (which is bad) can mean that the sentence is difficult and / or the model is bad.
The paper uses both Char-RNNLM and BPE-RNNLM.
Lastly, the authors note that in their last work, they measured models by language based on their total surprisal
$$ \sum _i NLL(s{_i}_j) $$
To account for missing data, they now use a probabilistic regression model that uses an "information content" factor n_j (for intent) and a language-specific difficulty factor d_j.
(4) The Datasets and Baseline Results
Europarl consists of discussions from the European Parliament. It contains 21 languages. This corpus is the most parallel because the original intent is always known.
The Bible consists of far more data for far many more languages, and each language had the option of several translations. The authors selected the versions that seemed most similar in intent using an integer linear program (ILP). The paper notes that standard deviation for difficulty factors across different translations within the same language are small, and thus language difficulty depends primarily upon intent (and not stylistic differences).
BPE-RNNLM had less variance than Char-RNNLM, and modifying the number of merges for BPE-RNNLM also did not drastically change the difficulty of a language (which validates this approach).
(5) What correlates with difficulty?
The authors measured correlation between difficulty and a number of qualities. The variables that had no significance were morphological counting complexity (disproving their previous work), average dependency length, and World Atlas of Language Structures (WALS) features.
The paper does find a correlation between language difficulty and raw character sequence length (for Char-RNNLM) and raw word inventory (literally, the number of distinct tokens in a training set). The authors note that because raw word inventory does correlate with morphological counting complexity, their previous conclusion may still hold true, but because MCC is immeasurable for the Bible, they cannot prove this.
(6) Translationese
The ability to measure whether or not translationese is any easier to model than its linguistic counterparts is dependent on an ability to handle missing data. Since this paper proposes a probabilistic approach (using a regression model), they are able to answer this question.
It is widely accepted that translationese is easier. But the authors ask whether this is due to the inherent difficulty of translationese, or because RNN models are primarily trained on translationese text (since even most English datasets are translationese). (For summary: "... translationese might merely be different ... not necessarily easier to model, but over-represented when training the model, making the translationese text sentences more predictable."
To test this hypothesis, the paper trains language models on equal parts translationese and native text. The previously observed lower difficulty score of translationese disappears.