Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model

Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model

Authors: Tudor Cebere, Aurélien Bellet, Nicolas Papernot

The paper explore the privacy guarantees of machine learning models trained using Differentially Private Stochastic Gradient Descent (DP-SGD) within the confines of the hidden state threat model. This threat model, distinct from the standard model, restricts adversaries to only the final model without access to intermediate updates, challenging prevalent assumptions and seeking to tighten privacy auditing in this constrained environment.

Core Contributions

The researchers introduce a new set of gradient-crafting adversaries that escalate privacy loss for the final model in the hidden state threat model. Their proposed adversaries ingeniously craft a gradient sequence without intermediate model knowledge to yield the highest possible privacy loss for the final model. This approach fundamentally contrasts existing methods that generally release intermediate models, helping bridge the notable gap between empirical privacy auditing lower bounds and privacy accounting theoretical upper bounds.

Key contributions include:

• Adversarial Gradient-Crafting: The research posits adversaries that craft gradients before the training begins, incorporating mechanisms to intensify privacy loss without intermediate model updates.
• Empirical Validation: Through experimental results, the authors assert their adversaries consistently outperform previous attempts in the hidden state model. They demonstrate that inserting crafted gradients at every optimization step reveals no privacy amplification, challenging prevailing beliefs.
• Performance in Non-Convex Settings: By not inserting crafted gradients at every step, the researchers show evidence of privacy amplification in general non-convex settings. The paper meticulously outlines scenarios where existing privacy upper bounds can be refined, suggesting privacy amplification phenomena despite the weaker outcomes in non-convex regimes versus convex scenarios.

Strong Numerical Results and Claims

The paper presents strong numerical results underpinning their claims:

• Non-Amplification with Frequent Gradient Insertion: Experimental evidence shows that when crafted gradients are inserted at each training step, privacy does not amplify, offering critical insights into the robustness of DP-SGD under certain adversarial conditions.
• Sigificant Outperformance: The newly proposed gradient-crafting adversaries significantly outperform canary-crafting adversaries from previous research, delivering tighter lower bounds on the privacy parameters.
• Privacy Amplification Trends: In scenarios where gradients are not inserted at every step, the emerging privacy amplification phenomenon—although weaker in non-convex regimes—indicates room for improvement in existing privacy upper bounds.

Implications and Future Developments

Practical Implications:

1. Model Deployment: For practitioners open-sourcing trained models, these findings emphasize that withholding intermediate models alone may not suffice to ensure privacy, particularly if adversarially crafted gradients are inserted at each step.
2. Privacy Accounting Enhancement: The insights from this research could be employed to refine privacy accounting mechanisms, making privacy guarantees more robust in non-convex regimes.

Theoretical Implications:

1. New Tight Auditing Techniques: By providing tighter auditing methods for the hidden state model, the paper sets the stage for future audits that enhance the understanding and effectiveness of DP mechanisms.
2. Understanding Privacy Amplification: The evidence supporting weaker yet existing privacy amplification in non-convex settings opens avenues for further research into privacy dynamics beyond convex regimes.

Future Research Directions:

• Adversarial Crafting Feasibility: Investigating practical scenarios where adversaries can craft optimal gradient sequences and exploring the feasibility and limitations of such crafting in real-world deployments.
• Extended Threat Models: Broadening the study to more complex threat models, including federated learning environments with partial participation, where intermediate models remain hidden to a subset of clients.
• Improved Privacy Accountants: Developing new privacy accountants that integrate findings on privacy amplification within non-convex regimes, thereby enhancing both theoretical models and practical implementations.

Conclusion

By challenging and refining the existing understanding of privacy guarantees in DP-SGD under the hidden state threat model, this paper lays a substantial groundwork for more precise and robust privacy auditing methods. The implications of their findings resonate through both practical deployments and theoretical advances, marking critical steps toward strengthening the privacy foundations of differentially private learning mechanisms.