Personal work on COLING Submission.
Starting 3.9.20, 3:30pm.
The current state of NLP research is predominantly based on English and languages with English-similar syntactic structures; consequently, contemporary NLP technologies do not account for linguistic properties unrepresented in English. This causes NLP tools to perform unequally across languages of different origins. We have conducted an empirical study using multi-class text classification on eight languages and seven models, demonstrating a severe disparity between the performance of models trained on typically versus atypically modeled languages.
Historically, NLP research has primarily used English text corpora for reasons involving convenience: larger availability of data, inherent word tokenization, and the tendency of researchers to speak English. However, the increasing prevalence of NLP technologies has made it imperative that models perform equally on English and other European as well as non-European languages, regardless of their presence on the world stage.
Current NLP tools are cross-lingual in the sense that there is nothing explicitly preventing them from being utilized for LRLs given that there exist annotated data (Mielke et. al., 2019). However, this does not mean that NLP tools perform fairly across all languages. While it is generally recognized that linguistic properties should be taken into account during the creation of NLP models (Bender et. al., 2009), there is little consensus on which characteristics most influence a model's ability to be language-agnostic. Therefore, we seek to identify the attributes of languages that give rise to this difference in the performance of NLP models. We will do this through a comprehensive analysis of typological and morphological differences across our example languages, and a technical examination of which aspects of language models they may affect (word embeddings and training corpora efficacy) with the goal of pinpointing specific linguistic properties that cause this phenomenon.
Towards this end, we conduct an empirical study demonstrating the performance of seven text-classification models, each trained on eight languages: Chinese, English, French, German, Italian, Japanese, Russian, and Spanish. In these experiments, we have introduced a control by training each model on equally sized datasets across languages. We do this to compose a performance baseline and demonstrate that additional developments are necessary to optimize the performance of models on languages that are not syntactically similar to English.
Because NLP technologies are typically created with English typological structures in mind, we expect the European languages (English, German, French, Italian, Spanish, and Russian) to consistently outperform the other languages in our dataset (Chinese and Japanese) since they are the most syntactically similar to English. In particular, we expect English to score the highest across all models, followed by German (which is also a Germanic language) and then French, Italian, and Spanish.
Additionally, through a discussion of relevant literature and an analysis of our results, we will investigate which aspects of current state-of-the-art NLP models are not language-agnostic. We propose three primary areas of possible remedy: word embeddings, typological characteristics, and the size of models’ training corpora.
Thus, our contributions include:
There are three factors that influence a model's ability to be language-agnostic: word embeddings, size of the training dataset used, and typological structure.
Word embeddings often capture semantic information based on words. This is inherently problematic because not all languages have easily distinguishable words as English does; for example, words in Chinese are not separated by spaces. Moreover, certain methods discern semantic information from the internal structures of words by using morphemes, the smallest semantic unit in language (for example, "unbelievable" has the morphemes "un," "believe," and "able") (Thang et al., 2013; Chen et al., 2015; Cotterell et al., 2016). Additional work in this area proposes that word embeddings be trained on the latent meanings of morphemes (meaning our previous example would be represented by "not," "believe," "able," "capable") (Xu et. al. 2018). Because morphological structure differs significantly among languages, the current practice of using word or morpheme representations for word embeddings is a potential barrier to language fairness in NLP.