Reading notes for Wednesday 11/6.
Choosing Transfer Languages for Cross-Lingual Learning
What is the problem the paper identifies?
To model low-resource (task) languages, high-resource (transfer) languages are often utilized through cross-lingual transfer. This significantly improves the performance of NLP tools on low-resource languages. However, there is no established criteria for choosing a high-resource language for use as a transfer language. This is usually chosen through a mix of intuition and ad hoc methods. Since specific characteristics of a language do contribute to that language's success as a transfer agent, this paper seeks to dissect this task.
What is the contribution the paper presents?
This paper presents a model that determines the value of a particular language as a transfer agent (as a ranking problem).
Give more detail about the specific problem the paper is addressing.
Generally, a common approach to training NLP models on low-resource languages is to first obtain a high-resource language and then use cross-lingual transfer to improve the performance of the former. There are many applications and methods towards this end. However, researchers currently using this strategy often pick their high-resource (transfer) language based on heuristic reasoning; for example, choosing a transfer language based on a shared family with the target or typological properties relevant for the intended NLP task. Other factors, unconsidered from the relevant works, may also be relevant: lexical overlap, size of the target or transfer data.
The authors propose a framework LANGRANK that approaches the stated problem as a ranking task. LANGRANK represents each potential transfer language by characteristics pertinent to cross-lingual transfer, such as word overlap and dataset size. The model ranks candidate transfer languages according to performance of each input transfer agent with a specific target language. This is very resource-intensive, but is efficient later, during test time.
This work implements LANGRANK through gradient boosted decision trees (GBDTs) and evaluates it through applications in
Performance of languages chosen through LANGRANK are compared by counterparts chosen by heuristic criteria.
What contributions does the work provide?
Problem Formulation
Without the use of a model, an exhaustive way to accomplish our goal would be to gather datasets from every possible transfer language, and measure the performance of our task language over all our options. Obviously this is very resource intensive, and often not even possible because we may not have access to adequate text corpora from multiple sources.
Instead, the paper seeks to predict performance from numerous candidate transfer languages. However, this does require a large dataset of training task languages, which result in corresponding scores, which are then used in a standard ranking model.
"...we turn to formulating our goal as a ranking task: given an NLP task, a low-resource task language t, and a list of J available high-resource transfer languages a_1, a2_, . . . , a_J , attempt to predict their ranking according to their expected scores
$$ c_{t, a_1}, c_{t, a_2},...c_{t, a_J} $$
without actually calculating the scores themselves."
Ranking Features / Criteria
Dataset-dependent features: statistical characteristics of a dataset. To learn dataset-dependent features, we must first have the dataset itself, which makes these criteria less practical when a transfer language is being evaluated without yet having been acquired.
Dataset-independent features: typological characteristics of a language.
Ranking Model
The authors choose the GBDT model with LambdaRank. They note that an important factor in their choice was the easy visualization of decision trees, as this will help identify important features for future heuristic reasoning.
Experiments
The performance of LANGRANK was tested on four common NLP tasks:
Ultimately, the use of all LANGRANK features (or even a subset of them) resulted in better performance for all four NLP tasks than the use of a heuristically-chosen baseline feature.
Additionally, the use of dataset-dependent features also enhances the quality of LANGRANK predictions (although the authors note that of course, this is not always practical in application).
To provide a logical framework for ad hoc decisions (when the use of LANGRANK is not practical), the authors also display feature importance for each NLP task. (Note that this is easy to do due to the use of a decision tree.)