Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Batch-oriented Element-wise Approximate Activation for Privacy-Preserving Neural Networks (2403.10920v1)

Published 16 Mar 2024 in cs.CR

Abstract: Privacy-Preserving Neural Networks (PPNN) are advanced to perform inference without breaching user privacy, which can serve as an essential tool for medical diagnosis to simultaneously achieve big data utility and privacy protection. As one of the key techniques to enable PPNN, Fully Homomorphic Encryption (FHE) is facing a great challenge that homomorphic operations cannot be easily adapted for non-linear activation calculations. In this paper, batch-oriented element-wise data packing and approximate activation are proposed, which train linear low-degree polynomials to approximate the non-linear activation function - ReLU. Compared with other approximate activation methods, the proposed fine-grained, trainable approximation scheme can effectively reduce the accuracy loss caused by approximation errors. Meanwhile, due to element-wise data packing, a large batch of images can be packed and inferred concurrently, leading to a much higher utility ratio of ciphertext slots. Therefore, although the total inference time increases sharply, the amortized time for each image actually decreases, especially when the batch size increases. Furthermore, knowledge distillation is adopted in the training process to further enhance the inference accuracy. Experiment results show that when ciphertext inference is performed on 4096 input images, compared with the current most efficient channel-wise method, the inference accuracy is improved by 1.65%, and the amortized inference time is reduced by 99.5%.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
  2. Privacy-preserving decision trees training and prediction. ACM Transactions on Privacy and Security, 25(3):1–30, 2022.
  3. Energy efficient and secure neural network–based disease detection framework for mobile healthcare network. ACM Transactions on Privacy and Security, 26(3):1–27, 2023.
  4. A methodology for training homomorphic encryption friendly neural networks. In Applied Cryptography and Network Security Workshops: ACNS 2022 Satellite Workshops, AIBlock, AIHWS, AIoTS, CIMSS, Cloud S&P, SCI, SecMT, SiMLA, Rome, Italy, June 20–23, 2022, Proceedings, pages 536–553. Springer, 2022.
  5. ngraph-he: a graph compiler for deep learning on homomorphically encrypted data. In Proceedings of the 16th ACM International Conference on Computing Frontiers, pages 3–13, 2019.
  6. (leveled) fully homomorphic encryption without bootstrapping. ACM Transactions on Computation Theory (TOCT), 6(3):1–36, 2014.
  7. Low latency privacy preserving inference. In International Conference on Machine Learning, pages 812–821. PMLR, 2019.
  8. Privacy-preserving classification on deep neural network. Cryptology ePrint Archive, 2017.
  9. A full rns variant of approximate homomorphic encryption. In Selected Areas in Cryptography–SAC 2018: 25th International Conference, Calgary, AB, Canada, August 15–17, 2018, Revised Selected Papers 25, pages 347–368. Springer, 2019.
  10. Homomorphic encryption for arithmetic of approximate numbers. In Advances in Cryptology–ASIACRYPT 2017: 23rd International Conference on the Theory and Applications of Cryptology and Information Security, Hong Kong, China, December 3-7, 2017, Proceedings, Part I 23, pages 409–437. Springer, 2017.
  11. Faster cryptonets: Leveraging sparsity for real-world encrypted inference. arXiv preprint arXiv:1811.09953, 2018.
  12. L2 regularization for learning kernels. arXiv preprint arXiv:1205.2653, 2012.
  13. Chet: an optimizing compiler for fully-homomorphic neural-network inferencing. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 142–156, 2019.
  14. Somewhat practical fully homomorphic encryption. Cryptology ePrint Archive, 2012.
  15. Octid: Optical coherence tomography image database. Computers & Electrical Engineering, 81:106532, 2020.
  16. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International conference on machine learning, pages 201–210. PMLR, 2016.
  17. Better bootstrapping for approximate homomorphic encryption. In Topics in Cryptology–CT-RSA 2020: The Cryptographers’ Track at the RSA Conference 2020, San Francisco, CA, USA, February 24–28, 2020, Proceedings, pages 364–390. Springer, 2020.
  18. Cryptodl: Deep neural networks over encrypted data. arXiv preprint arXiv:1711.05189, 2017.
  19. Privacy-preserving deep sequential model with matrix homomorphic encryption. In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security, pages 377–391, 2022.
  20. End-to-end privacy preserving deep learning on multi-institutional medical imaging. Nature Machine Intelligence, 3(6):473–484, 2021.
  21. Optimized privacy-preserving cnn inference with fully homomorphic encryption. IEEE Transactions on Information Forensics and Security, 18:2175–2187, 2023.
  22. Precise approximation of convolutional neural networks for homomorphically encrypted data. arXiv preprint arXiv:2105.10879, 2021.
  23. Hemet: a homomorphic-encryption-friendly privacy-preserving mobile neural network architecture. In International conference on machine learning, pages 7102–7110. PMLR, 2021.
  24. Safenet: A secure, accurate and fast neural network inference. In International Conference on Learning Representations, 2021.
  25. Delphi: a cryptographic inference system for neural networks. In Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, pages 27–30, 2020.
  26. Classification of encrypted word embeddings using recurrent neural networks. In PrivateNLP@ WSDM, pages 27–31, 2020.
  27. Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.
  28. Microsoft SEAL (release 3.5). https://github.com/Microsoft/SEAL, April 2020. Microsoft Research, Redmond, WA.
  29. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(11):1958–1970, 2008.
  30. Adversarial learning of portable student networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  31. Ppolynets: Achieving high prediction accuracy and efficiency with parametric polynomial activations. IEEE Access, 6:72814–72823, 2018.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now