Reading notes from Wednesday 10/16.

With questions from William Griswold's "How to Read an Engineering Research Paper"

Link to the paper here.

How to Read an Engineering Research Paper

1. What are motivations for this work?

   Because machine learning algorithms are being used increasingly in areas of large consequence, discriminatory biases present in their algorithms have come into the spotlight. The antithesis for such biases is fairness, which is introduced to learning frameworks in the form of regularizers. Yet fairness has been shown to come at the price of accuracy, and as such the fairness-accuracy tradeoff is an important question in related works.

   Previous work has focused on binary classification settings, and when regression settings have been used, they have been primarily linear. More importantly, there is a common misconception that there should be a universal definition for fairness, and that future research has yet to find it.

2. What is the proposed solution?

   The paper introduces a "flexible family of fairness regularizers" with varying weights, and uses regression (both linear and logistic) to model predictions across six different datasets. To achieve this flexibility, they use three fairness penalties:

   1. individual fairness: for every cross pair, the model is penalized for how differently it treats x and x(prime). Does not allow for compensation. Considers each cross pair separately.
   2. group fairness: the penalty corresponds to the difference in labels between pairs of cross pairs. Does allow for compensation (can over-value instances of similar cross-pairs). Considers all the cross pairs together.
   3. hybrid notions of fairness: generally, takes elements from both individual and group fairness. One proposed method is to group cross pairs (for binary labeled data) based on their labels.

3. What is the work's evaluation of the proposed solution?

   Ultimately, the paper concludes that it is impossible to search for a single definition of fairness. Rather, the practitioners of each individual model and dataset must determine the most effective method for utilizing fairness regularizers, choosing a specific fairness penalty, accuracy-fairness tradeoff, and regression setting. Doing so will optimize the efficient frontier for each individual circumstance.

   The authors introduce a concept to quantify this decision: Price of Fairness (PoF). This is proposed as a metric to understand fairness over the accuracy optimum.

4. What is your analysis of the identified problem, idea and evaluation?

   The most controversial point made in this paper is that all previous work in the field of fairness makes a fundamentally incorrect assumption: that it is possible to define a concept of universal fairness. Because accuracy and fairness are mutually exclusive, the authors argue that depending on the priorities of the practitioners and model at hand, the method of measuring fairness varies.

   This point is well-demonstrated through the use of varying methods on the six different datasets. Because each dataset had widely diverse assumptions, protected attributes, and fairness-accuracy costs, the authors succeed in showing that it is misguided to assume a single definition of fairness.

5. What are the contributions?

   • the use of a flexible but convex family of fairness regularizers, which demonstrates various methods for future practitioners;
   • an empirical and data-dependent measure of measuring the fairness-accuracy tradeoff, defined as the PoF; and
   • an extensive set of resultant data from the six different datasets.

6. What are future directions for this research?

   The paper does not explicitly mention thoughts of future research on this topic. But given that they say the theory has already been established ("Combined with recent theoretical results... that also highlight the incompatibility of various fairness measures..."), the implication is that future work will focus on cataloging the various methods' efficacy in various situations.

7. What questions are you left with?

   Some of the methods of experimentation do not seem to be properly elaborated on. For instance, I am confused on how they controlled for the protected attributes they identified for the six datasets, as it is not possible to simply remove the corresponding variable from the model without still experiencing related bias.

8. What is your take-away message from this paper?

   • the concept of fairness is multi-faceted and a point of contention among machine learning researchers;
   • practitioners (me!) must analyze the specific intentions of their uses of predictive models, so as to accurately define them;
   • the terms I need to better understand are fairness, bias, regularizer, efficient frontier, regression, classification, convex, weights, and cross pairs.