Using sensitive personal data may be necessary for avoiding discrimination in data-driven decision models

Abstract

Increasing numbers of decisions about everyday life are made using algorithms. By algorithms we mean predictive models (decision rules) captured from historical data using data mining. Such models often decide prices we pay, select ads we see and news we read online, match job descriptions and candidate CVs, decide who gets a loan, who goes through an extra airport security check, or who gets released on parole. Yet growing evidence suggests that decision making by algorithms may discriminate people, even if the computing process is fair and well-intentioned. This happens due to biased or non-representative learning data in combination with inadvertent modeling procedures. From the regulatory perspective there are two tendencies in relation to this issue: (1) to ensure that data-driven decision making is not discriminatory, and (2) to restrict overall collecting and storing of private data to a necessary minimum. This paper shows that from the computing perspective these two goals are contradictory. We demonstrate empirically and theoretically with standard regression models that in order to make sure that decision models are non-discriminatory, for instance, with respect to race, the sensitive racial information needs to be used in the model building process. Of course, after the model is ready, race should not be required as an input variable for decision making. From the regulatory perspective this has an important implication: collecting sensitive personal data is necessary in order to guarantee fairness of algorithms, and law making needs to find sensible ways to allow using such data in the modeling process.

Appendix: Omitted variable bias

We provide a theoretical expectation for the omitted variable bias in the ordinary least squares (OLS) estimation of linear regression coefficients. The theory is known in multiple statistical textbooks, we adapt the reasoning for discrimination prevention. For better interpretability we focus on a simple case with one legitimate variable, extension to more variables is straightforward.

Let the true underlying model behind data be

$$\begin{aligned} y = b_0 + b_1x + \beta s + e, \end{aligned}$$

(10)

where x is a legitimate variable (such as education), s is a sensitive variable (such as ethnicity), y is the target variable (such as salary), e is random noise with the expected value of zero, and $\beta$, $b_1$, and $b_0$ are non-zero coefficients.

Assume a data scientist decides to fit model $y = \hat{b}_0 + \hat{b}_1x$.

Following the standard (OLS) procedure for estimating regression parameters the data scientist gets:

$$\begin{aligned} \hat{b}_1= \frac{\hat{ Cov }(x,y)}{\hat{ Var }(x)},\end{aligned}$$

(11)

$$\begin{aligned} \hat{b}_0= \bar{y} - \hat{b}_1\bar{x}, \end{aligned}$$

(12)

where bar denotes the mean, and hat denotes that it is estimated from data.

Next we plug-in the true underlying model from Eq. (10)

$$\begin{aligned} \hat{b}_1&= \frac{\hat{ Cov }(x,b_0 + b_1x + \beta s + e)}{\hat{ Var }(x)}\\ &= \frac{\hat{ Cov }(x,b_0)}{\hat{ Var }(x)} + \frac{b_1\hat{ Cov }(x,x)}{\hat{ Var }(x)} + \frac{b_2\hat{ Cov }(x,s)}{\hat{ Var }(x)} + \frac{\hat{ Cov }(x,e)}{\hat{ Var }(x)} \nonumber \\ &= b_1 + \beta \frac{\hat{ Cov }(x,s)}{\hat{ Var }(x)},\end{aligned}$$

(13)

$$\hat{b}_0= \bar{y} - \hat{b}_1\bar{x} = \bar{y} - b_1\bar{x} - \beta \frac{\hat{ Cov }(x,s)}{\hat{ Var }(x)}\bar{x}$$

(14)

$$\begin{aligned}= b_0 - \beta \frac{\hat{ Cov }(x,s)}{\hat{ Var }(x)}\bar{x}. \end{aligned}$$

(15)

This demonstrates that unless $Cov (x,s)$ is zero, or $\beta$ is zero, the estimates $\hat{b}_1$ and $\hat{b}_0$ will be biased by a component that carries forward discrimination.