pMSE Mechanism: Differentially Private Synthetic Data with Maximal Distributional Similarity

Abstract

We propose a method for the release of differentially private synthetic datasets. In many contexts, data contain sensitive values which cannot be released in their original form in order to protect individuals’ privacy. Synthetic data is a protection method that releases alternative values in place of the original ones, and differential privacy (DP) is a formal guarantee for quantifying the privacy loss. We propose a method that maximizes the distributional similarity of the synthetic data relative to the original data using a measure known as the pMSE, while guaranteeing $\epsilon$-DP. We relax common DP assumptions concerning the distribution and boundedness of the original data. We prove theoretical results for the privacy guarantee and provide simulations for the empirical failure rate of the theoretical results under typical computational limitations. We give simulations for the accuracy of linear regression coefficients generated from the synthetic data compared with the accuracy of non-DP synthetic data and other DP methods. Additionally, our theoretical results extend a prior result for the sensitivity of the Gini Index to include continuous predictors.

Appendices

10 Appendix: Proof of Theorem 4.1

Proof

We first show that using the expected value, and approximating it, can be bounded above by the supremum across all possible datasets $X^s$ generated using $\theta$.

$$\begin{aligned} \varDelta u = \underset{\theta }{sup} \; \underset{\delta (X,X^\prime )=1}{sup} \; |u(X, \theta ) - u(X^\prime , \theta )| \end{aligned}$$

(5)

can be rewritten as

$$\begin{aligned} \varDelta u = \underset{\theta }{sup} \; \underset{\delta (X,X^\prime )=1}{sup} \; |E_\theta [pMSE(X, X^s_\theta )|X,\theta ] - E_\theta [pMSE(X^\prime , X^s_\theta )|X,\theta ]| \end{aligned}$$

(6)

where $u(X, \theta ) = E_\theta [pMSE(X, X^s_\theta )|X, \theta ]$. Since the absolute value is a convex function, we can apply Jensen’s inequality and get

$$\begin{aligned} \le \underset{\theta }{sup} \; \underset{\delta (X,X^\prime )=1}{sup} \; E_\theta [|pMSE(X, X^s_\theta ) - pMSE(X^\prime , X^s_\theta )||X,\theta ]. \end{aligned}$$

(7)

Then by taking the supremum over any data set $X^s_\theta$, we obtain

$$\begin{aligned} \le \underset{X^s_\theta }{sup} \; \underset{\delta (X,X^\prime )=1}{sup} \; |pMSE(X,X^s_\theta ) - pMSE(X^\prime , X^s_\theta )|. \end{aligned}$$

(8)

This also bounds our approximation of the expected value that we propose to use in practice, since the supremum is also greater than or equal to the sample mean.

Now writing this explicitly in terms of the CART model, we get

$$\begin{aligned} \underset{a_i, \; m_i, \; a_i^\prime , \; m_i^\prime }{sup} \; \frac{1}{2n}\Bigg |\varSigma _{i = 1}^{D + 1} m_i\Big (\frac{a_i}{m_i} - 0.5\Big )^2 - m_i^\prime \Big (\frac{a_i^\prime }{m_i^\prime } - 0.5\Big )^2\Bigg | \end{aligned}$$

(9)

where $a_i$, $m_i$, and D are defined as before, and $a_i^\prime$ and $m_i^\prime$ are the corresponding values for the model fit using $X^\prime$. Expanding this we get

$$\begin{aligned} \underset{a_i, \; m_i, \; a_i^\prime , \; m_i^\prime }{sup} \; \frac{1}{2n}\Bigg |\varSigma _{i = 1}^{D + 1} \Big (\frac{a_i^2}{m_i} - a_i - 0.25m_i\Big ) - \Big (\frac{a_i^{\prime 2}}{m_i^\prime } - a_i^\prime - 0.25m_i^\prime \Big )\Bigg | \end{aligned}$$

(10)

and we can cancel the third terms because $\varSigma _{i=1}^{D+1}m_i = \varSigma _{i=1}^{D+1}m_i^\prime$. When we multiple by 2n, the remaining inside term is equivalent to the sensitivity of the impurity, i.e.,

$$\begin{aligned} \underset{a_i, \; m_i, \; a_i^\prime , \; m_i^\prime }{sup} \; \Bigg |GI(X, X^s, D) - GI(X^\prime , X^s, D) \Bigg | = \varDelta GI \end{aligned}$$

(11)

By bounding the impurity, we bound the pMSE. We can rewrite the above as

$$\begin{aligned} \Bigg |\underset{D}{min} \; GI(X, X^s, D) - \underset{D}{min} \; GI(X^\prime , X^s, D) \Bigg | \end{aligned}$$

(12)

since the optimal CART model finds the minimum impurity across any D. The greatest possible difference then is the difference between these two optimums. And we can bound this above by

$$\begin{aligned} \le \underset{D}{sup} \; \Bigg |GI(X, X^s, D) - GI(X^\prime , X^s, D) \Bigg |. \end{aligned}$$

(13)

Let $X^{comb}$ and $X^{\prime comb}$ be the combined data matrices as described in Algorithm 1, including the 0, 1 outcome variable. Recall that only one record has changed between $X^{comb}$ and $X^{\prime comb}$ (total number of records staying fixed), and it is labeled 0. We know that for a given D optimal split points producing $D + 1$ nodes on $X^{comb}$, there are $a_i$ records labeled 1 and $\tilde{m}_i$ total records in each bin, such that $\exists \; j \ne k \ne l_1 \ne ... \ne l_{D-1} \; s.t. \; \tilde{m}_j - m_j = m_k - \tilde{m}_k = 1, \; \tilde{m}_{l_v} = m_{l_v}$ for $v = \{1,..., D - 1\}$. In the same way, for a given D optimal split points producing $D + 1$ nodes on $X^{\prime comb}$, there are $a^\prime _i$ records labeled 1 and $\tilde{m}^\prime _i$ total records in each bin, such that $\exists \; j^\prime \ne k^\prime \ne l^\prime _1 \ne ... \ne l^\prime _{D-1} \; s.t. \; \tilde{m}^\prime _{j^\prime } - m^\prime _{j^\prime } = m^\prime _{k^\prime } - \tilde{m}^\prime _{k^\prime } = 1, \; \tilde{m}^\prime _{l^\prime _v} = m^\prime _{l^\prime _v}$ for $v = \{1,..., D - 1\}$. What this simply means is that after changing one record, the discrete counts in the nodes change by at most one in two of the nodes and does not change in the other $D - 1$ nodes.

Due to the fact that the CART model produces the D splits that minimize the impurity, we know both that

$$\begin{aligned} \varSigma _{i = 1}^{D + 1}{a^\prime _i\Big (1-\frac{a^\prime _i}{m^\prime _i}\Big )} \le \varSigma _{i = 1}^{D + 1}{a_i\Big (1-\frac{a_i}{\tilde{m}_i}\Big )} \end{aligned}$$

(14)

and

$$\begin{aligned} \varSigma _{i = 1}^{D + 1}{a_i\Big (1-\frac{a_i}{m_i}\Big )} \le \varSigma _{i = 1}^{D + 1}{a^\prime _i\Big (1-\frac{a^\prime _i}{\tilde{m}^\prime _i}\Big )}. \end{aligned}$$

(15)

The inequality (14) implies that after changing one record, if new split points are chosen, the impurity must be equivalent or better than simply keeping the previous splits and changing the counts. The inequality (15) implies that the first split points chosen must be equivalent or better than using the new splits with the changed counts. If this were not the case, the first split points would have never been made in the first place. These lead to the final step.

Because we have an absolute value, we consider two cases.

(16)

The last step we know because $a_i \le m_i$, and $\frac{n^2}{n(n-1)} \le 2$.

(17)

Finally, this gives us $\varDelta GI \le 2 \implies \frac{\varDelta GI}{2n} = \varDelta u \le \frac{1}{n}$.

11 Appendix: Full Simulation Results

Fig. 4.

Boxplots showing simulation results. The rows indicate the different coefficients, and the columns indicate different values of $\epsilon$. Boxplots are also subdivided within methods by the tree depth (for the pMSE mechanism method) and the bound (for others).