Disparate privacy risks from medical AI

1Send a request

A hospital deploys a diagnostic model, the target model, behind a prediction interface. Any user with access can submit a patient’s record, here a chest radiograph that serves as the target record, and the target model computes a prediction from it.

2Receive a prediction

The interface returns the target model’s output: a 93% probability of pneumonia. The target model and the data it was trained on stay hidden. For an ordinary user, the interaction ends here.

3Membership inference

An untrusted user who holds the target record can put the same output to another use and ask: was the target record part of the training data? Put differently, is the target model behind the interface model A, trained with the target record, or model B, trained without it? The answer alone can be sensitive. If a target model was trained only on patients with cancer, membership reveals a diagnosis.

4Build a reference

Models tend to output slightly higher confidence scores for records they were trained on, and the difference is often tiny, here about one percentage point. What counts as “higher” differs from record to record, so the attacker trains reference models, half with the target record and half without. Split into these two groups, each model outputs a score for the target record. The two sets of scores form two distributions: one for members, one for non-members.

5Make the guess

The attacker places the target model’s output, 93%, against both distributions. It is about 7 times more likely under the member distribution, so the attack guesses member. This comparison is a likelihood-ratio test, the idea behind LiRA, the state-of-the-art attack used in the study. One query is enough, and federated learning offers no protection because the attack only queries the finished target model.

6Measuring attack success in aggregate

Attack success is usually reported in aggregate, as one number for the whole dataset. One target model is trained on a random 50% subset, and attack scores are computed for every record. Higher scores point to membership. Each threshold on these scores flags some records as members and gives one point on the ROC curve: the true positive rate (TPR) against the false positive rate (FPR).

Sweeping the threshold traces the curve, and the area under it is the aggregate AUC. 50% is random guessing and 100% is perfect. Here it is only 58%, so the target model looks safe, even though a few highly exposed members are flagged without a single false positive.

7Measuring attack success for individual records

Instead, the study measured attack success for each target record separately. It trained 200 target models per dataset, each on a random half of the data, so that exactly 100 were trained with any given target record and 100 without it. The confidence scores these target models output for the target record form two distributions. The record-level MIA AUC measures how well the two are separated: it is the chance that a target model trained with the target record outputs a higher score than one trained without it. It is 50% if the two distributions overlap completely and 100% if they are fully separated.

State-of-the-art attacks such as LiRA (Carlini et al., 2022) and RMIA (Zarifzadeh et al., 2024) assume these distributions are Gaussian, so the record-level AUC follows from their means μ and standard deviations σ: $$\text{AUC}=\mathrm{\Phi }\left(\frac{{\mu }_{\text{with}}-{\mu }_{\text{without}}}{\sqrt{{\sigma }_{\text{with}}^2+{\sigma }_{\text{without}}^2}}\right)$$ where Φ is the cumulative distribution function of the standard normal distribution.

Each grey line is one target record’s ROC curve. Most sit near the diagonal, which means random guessing, but a few bow sharply towards the top left: for these records, the attack works almost perfectly. Select a curve to see the two distributions behind it.

8From record to patient-level risk

A patient often contributes several records, such as radiographs from different visits. Exposing any one of them reveals that the patient was in the training data, so a patient’s score is the highest score among their records. This patient faces an attack with an AUC of 97%, while the whole-dataset number stays at 58%. The findings below show how many patients are in this position.