Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Training Data Protection with Compositional Diffusion Models (2308.01937v4)

Published 2 Aug 2023 in cs.LG, cs.AI, cs.CR, and cs.CV

Abstract: We introduce Compartmentalized Diffusion Models (CDM), a method to train different diffusion models (or prompts) on distinct data sources and arbitrarily compose them at inference time. The individual models can be trained in isolation, at different times, and on different distributions and domains and can be later composed to achieve performance comparable to a paragon model trained on all data simultaneously. Furthermore, each model only contains information about the subset of the data it was exposed to during training, enabling several forms of training data protection. In particular, CDMs enable perfect selective forgetting and continual learning for large-scale diffusion models, allow serving customized models based on the user's access rights. Empirically the quality (FID) of the class-conditional CDMs (8-splits) is within 10% (on fine-grained vision datasets) of a monolithic model (no splits), and allows (8x) faster forgetting compared monolithic model with a maximum FID increase of 1%. When applied to text-to-image generation, CDMs improve alignment (TIFA) by 14.33% over a monolithic model trained on MSCOCO. CDMs also allow determining the importance of a subset of the data (attribution) in generating particular samples, and reduce memorization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Lqf: Linear quadratic fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15729–15739, 2021.
  2. Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313–326, 1982.
  3. All are worth words: a vit backbone for score-based diffusion models. arXiv preprint arXiv:2209.12152, 2022.
  4. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE, 2021.
  5. \\\backslash\a-la-carte prompt tuning (apt): Combining distinct data via composable prompting. arXiv preprint arXiv:2302.07994, 2023.
  6. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188, 2023.
  7. Forget unlearning: Towards true data-deletion in machine learning. arXiv preprint arXiv:2210.08911, 2022.
  8. Technical report cns-tr-2011-001, california institute of technology. Caltech, 2011.
  9. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023.
  10. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  11. Differentially private diffusion models. arXiv preprint arXiv:2210.09929, 2022.
  12. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  13. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. In International Conference on Machine Learning, pages 8489–8510. PMLR, 2023.
  14. Safe: Machine unlearning with shard graphs. arXiv preprint arXiv:2304.13169, 2023.
  15. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
  16. Erasing concepts from diffusion models. arXiv preprint arXiv:2303.07345, 2023.
  17. Differentially private diffusion models generate useful synthetic images. arXiv preprint arXiv:2302.13861, 2023.
  18. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
  19. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
  20. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In European Conference on Computer Vision, pages 383–398. Springer, 2020b.
  21. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 792–801, 2021.
  22. Mixed differential privacy in computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8376–8386, 2022.
  23. Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030, 2019.
  24. Adaptive machine unlearning. Advances in Neural Information Processing Systems, 34:16319–16330, 2021.
  25. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  26. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
  27. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  28. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897, 2023.
  29. Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. arXiv preprint arXiv:2311.17717, 2023.
  30. Visual prompt tuning. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 709–727. Springer, 2022.
  31. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.
  32. No matter how you slice it: Machine unlearning with sisa comes at the expense of minority classes. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 622–637. IEEE, 2023a.
  33. No matter how you slice it: Machine unlearning with sisa comes at the expense of minority classes. In First IEEE Conference on Secure and Trustworthy Machine Learning, 2023b.
  34. Information-theoretic diffusion. arXiv preprint arXiv:2302.03792, 2023.
  35. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  36. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
  37. Privacy adhering machine un-learning in nlp. arXiv preprint arXiv:2212.09573, 2022.
  38. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22691–22702, 2023.
  39. Mnist handwritten digit database. Dataset, 2010.
  40. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  41. On the stochastic realization problem. SIAM Journal on Control and Optimization, 17(3):365–389, 1979.
  42. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
  43. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423–439. Springer, 2022.
  44. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022a.
  45. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022b.
  46. Dreamix: Video diffusion models are general video editors. arXiv preprint arXiv:2302.01329, 2023.
  47. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pages 931–962. PMLR, 2021.
  48. Edward Nelson. Dynamical theories of Brownian motion. Princeton university press, 1967.
  49. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012.
  50. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  51. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International Conference on Machine Learning, pages 18332–18346. PMLR, 2022.
  52. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  53. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  54. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075–18086, 2021.
  55. Visual prompt tuning for generative transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19840–19851, 2023.
  56. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
  57. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
  58. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574–584. PMLR, 2020b.
  59. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020c.
  60. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pages 4126–4142. PMLR, 2021.
  61. Provable copyright protection for generative models. arXiv preprint arXiv:2302.10870, 2023.
  62. Compositional text-to-image synthesis with attention map control of diffusion models. arXiv preprint arXiv:2305.13921, 2023.
  63. Erasediff: Erasing data influence in diffusion models. arXiv preprint arXiv:2401.05779, 2024.
  64. Raphael: Text-to-image generation via large mixture of diffusion paths. arXiv preprint arXiv:2305.18295, 2023.
  65. Arcane: An efficient architecture for exact machine unlearning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4006–4013, 2022.
  66. Legonet: A fast and exact unlearning architecture. arXiv preprint arXiv:2210.16023, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Aditya Golatkar (19 papers)
  2. Alessandro Achille (60 papers)
  3. Ashwin Swaminathan (18 papers)
  4. Stefano Soatto (179 papers)
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now