Reading notes from Tuesday 1/29.

Evaluating Fairness Metrics in the Presence of Dataset Bias

Introduction

Discriminatory behavior is not a deliberate feature of these systems, but rather the result of biases present in the input data used to train the systems.

Contributions

• frame the problem of bias detection as a casual inference problem with observational data: they approach bias detection as something to be done manually by an observant eye, as opposed to a single, hard-cut process. This mirrors the nature of analyzing fairness in ML, which is often done in similar methods.
• investigate the performance of six different fairness metrics under conditions of varying dataset bias: given that the bias is a known quantity and purposefully introduced into the datasets.
• based on these potential consequences we present a set of recommended best practices to guide fairness metric selection: again, using human discernibility as opposed to a procedural process.

Related Work

The literature surrounding fairness in ML focuses on three main points:

1. to quantify the degree of unfair bias present (in data or model predictions)
2. to remove the unfair bias present (in data of model predictions)
3. to develop ML algorithms that have fairness constraints

This paper focuses on 6 metrics of measuring fairness.

...the proliferation of these metrics reflects the many and sometimes mutually exclusive interpretations of ‘fairness’ in machine learning contexts.

1. difference in means
2. difference in residuals
3. equal opportunity
4. equal mis-opportunity