Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Practical Region-level Attack against Segment Anything Models (2404.08255v2)

Published 12 Apr 2024 in cs.CV and cs.CR

Abstract: Segment Anything Models (SAM) have made significant advancements in image segmentation, allowing users to segment target portions of an image with a single click (i.e., user prompt). Given its broad applications, the robustness of SAM against adversarial attacks is a critical concern. While recent works have explored adversarial attacks against a pre-defined prompt/click, their threat model is not yet realistic: (1) they often assume the user-click position is known to the attacker (point-based attack), and (2) they often operate under a white-box setting with limited transferability. In this paper, we propose a more practical region-level attack where attackers do not need to know the precise user prompt. The attack remains effective as the user clicks on any point on the target object in the image, hiding the object from SAM. Also, by adapting a spectrum transformation method, we make the attack more transferable under a black-box setting. Both control experiments and testing against real-world SAM services confirm its effectiveness.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
  2. Security evaluation of pattern classifiers under attack. IEEE transactions on knowledge and data engineering, 26(4):984–996, 2013.
  3. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017.
  4. Semantic segment anything. https://github.com/fudan-zvg/Semantic-Segment-Anything, 2023a.
  5. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.
  6. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023b.
  7. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185–9193, 2018.
  8. Evading defenses to transferable adversarial examples by translation-invariant attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4312–4321, 2019.
  9. Adversarial examples for semantic image segmentation. arXiv preprint arXiv:1703.01101, 2017.
  10. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  11. Countering adversarial images using input transformations. arXiv preprint arXiv:1711.00117, 2017.
  12. Backpropagating linearly improves transferability of adversarial examples. Advances in neural information processing systems, 33:85–95, 2020.
  13. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022.
  14. Adversarial attacks for image segmentation on multiple lightweight models. IEEE Access, 8:31359–31370, 2020.
  15. Segment anything in high quality. In NeurIPS, 2023.
  16. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  17. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04767, 2023.
  18. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1778–1787, 2018.
  19. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv:1908.06281, 2019.
  20. Frequency domain model augmentation for adversarial attack. In European Conference on Computer Vision, pages 549–566. Springer, 2022.
  21. Segment anything in medical images. arXiv preprint arXiv:2304.12306, 2023.
  22. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
  23. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  24. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 89:102918, 2023.
  25. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9078–9086, 2019.
  26. Impact of adversarial examples on deep learning models for biomedical image segmentation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part II 22, pages 300–308. Springer, 2019.
  27. Robustness of sam: Segment anything under corruptions and beyond. arXiv preprint arXiv:2306.07713, 2023.
  28. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  29. Proximal splitting adversarial attack for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20524–20533, 2023.
  30. Anything-3d: Towards single-view anything reconstruction in the wild, 2023.
  31. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
  32. Can sam segment anything? when sam meets camouflaged object detection. arXiv preprint arXiv:2304.04709, 2023.
  33. Segment anything meets semantic communication. arXiv preprint arXiv:2306.02094, 2023.
  34. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.07204, 2017.
  35. Global feature attention network: Addressing the threat of adversarial attack for aerial image semantic segmentation. Remote Sensing, 15(5):1325, 2023.
  36. Skip connections matter: On the transferability of adversarial examples generated with resnets. arXiv preprint arXiv:2002.05990, 2020.
  37. Mitigating adversarial effects through randomization. arXiv preprint arXiv:1711.01991, 2017a.
  38. Adversarial examples for semantic segmentation and object detection. In Proceedings of the IEEE international conference on computer vision, pages 1369–1378, 2017b.
  39. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2730–2739, 2019.
  40. Efficientsam: Leveraged masked image pretraining for efficient segment anything. arXiv preprint arXiv:2312.00863, 2023.
  41. Track anything: Segment anything meets videos, 2023.
  42. Investigating top-k white-box and transferable black-box attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15085–15094, 2022.
  43. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289, 2023a.
  44. Attack-sam: Towards evaluating adversarial robustness of segment anything model. arXiv preprint arXiv:2305.00866, 2023b.
  45. Uvosam: A mask-free paradigm for unsupervised video object segmentation via segment anything model. arXiv preprint arXiv:2305.12659, 2023c.
  46. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  47. Fast segment anything, 2023.
  48. Zihao Zhao. Enhancing autonomous driving with grounded-segment anything model: Limitations and mitigations. In 2023 IEEE 3rd International Conference on Data Science and Computer Application (ICDSCA), pages 1258–1265. IEEE, 2023.
  49. On success and simplicity: A second look at transferable targeted attacks. Advances in Neural Information Processing Systems, 34:6115–6128, 2021.
  50. Black-box targeted adversarial attack on segment anything (sam). arXiv preprint arXiv:2310.10010, 2023.
  51. A pilot study of query-free adversarial attack against stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2384–2391, 2023.
  52. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now