Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CCC++: Optimized Color Classified Colorization with Segment Anything Model (SAM) Empowered Object Selective Color Harmonization (2403.11494v2)

Published 18 Mar 2024 in cs.CV

Abstract: In this paper, we formulate the colorization problem into a multinomial classification problem and then apply a weighted function to classes. We propose a set of formulas to transform color values into color classes and vice versa. To optimize the classes, we experiment with different bin sizes for color class transformation. Observing class appearance, standard deviation, and model parameters on various extremely large-scale real-time images in practice we propose 532 color classes for our classification task. During training, we propose a class-weighted function based on true class appearance in each batch to ensure proper saturation of individual objects. We adjust the weights of the major classes, which are more frequently observed, by lowering them, while escalating the weights of the minor classes, which are less commonly observed. In our class re-weight formula, we propose a hyper-parameter for finding the optimal trade-off between the major and minor appeared classes. As we apply regularization to enhance the stability of the minor class, occasional minor noise may appear at the object's edges. We propose a novel object-selective color harmonization method empowered by the Segment Anything Model (SAM) to refine and enhance these edges. We propose two new color image evaluation metrics, the Color Class Activation Ratio (CCAR), and the True Activation Ratio (TAR), to quantify the richness of color components. We compare our proposed model with state-of-the-art models using six different dataset: Place, ADE, Celeba, COCO, Oxford 102 Flower, and ImageNet, in qualitative and quantitative approaches. The experimental results show that our proposed model outstrips other models in visualization, CNR and in our proposed CCAR and TAR measurement criteria while maintaining satisfactory performance in regression (MSE, PSNR), similarity (SSIM, LPIPS, UIUI), and generative criteria (FID).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Y. C. Huang, Y. S. Tung, J. C. Chen, S. W. Wang and J. L. Wu, “An adaptive edge detection based colorization algorithm and its applications”, in Proc. 13th annual ACM international conference on Multimedia, Hilton, Singapore, pp. 351-354, 2005.
  2. A. Levin, D. Lischinski and Y. Weiss, “Colorization using optimization,” in ACM SIGGRAPH 2004 Papers, Rochester, NY, US: ACM, pp.689–694, 2004.
  3. L. Yatziv and G. Sapiro, “Fast image and video colorization using chrominance blending,” IEEE Transactions on Image Processing, vol. 15, no. 5, pp. 1120–1129, 2006.
  4. Y. Qu, T.T Wong, and P.A. Heng, “Manga colorization”, ACM Transactions on Graphics (ToG), vol. 25, no. 3, pp.1214-1220, 2006.
  5. Q. Luan, F. Wen, D. Cohen-Or, L. Liang, Y.-Q. Xu, and H.-Y. Shum, “Natural image colorization,” In Proceedings of the 18th Eurographics conference on Rendering Techniques, pp. 309-320. 2007.
  6. T. Welsh, M. Ashikhmin and K. Mueller, “Transferring color to greyscale images,” in Proc. 29th Annual Conference on Computer Graphics and Interactive Techniques, Texas, San Antonio, USA, pp. 277–280, 2002.
  7. R. Ironi, D. Cohen-Or, D. Lischinski, “Colorization by Example.” Rendering techniques 29, pp. 201-210, 2005.
  8. Y.-W. Tai, J. Jia and C.-K. Tang, “Local color transfer via probabilistic segmentation by expectation-maximization,” 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 2005, pp. 747-754 vol. 1, doi: 10.1109/CVPR.2005.215.
  9. A. Y.-S. Chia, S. Zhuo, R. K. Gupta, Y.-W. Tai, S.-Y. Cho, P. Tan, S. Lin, “Semantic colorization with internet images.” ACM Transactions on Graphics (ToG) vol. 30, no. 6 pp. 1-8, 2011.
  10. X. Liu, L. Wan, Y. Qu, T.-T. Wong, S. Lin, C.-S. Leung, P.-A. Heng, “Intrinsic colorization,” In ACM SIGGRAPH Asia 2008 papers, pp. 1-9. 2008.
  11. U. Sousa, R. Kabirzadeh and P. Blaes, “Automatic colorization of grayscale images,” Department of Electrical Engineering, Stanford University, 2013.
  12. Y. Wu, X. Wang, Y. Li, H. Zhang, X. Zhao, and Y. Shan, “Towards vivid and diverse image colorization with generative color prior,” in Proc. IEEE/CVF International Conference on Computer Vision, Montreal, Canada, pp. 14377-14386, 2021.
  13. H. Bahng, S. Yoo, W. Cho, D. K. Park, Z. Wu, X. Ma, and J. Choo, “Coloring with words: guiding image colorization through text-based palette generation,” in Proc. European Conference on Computer Vision (ECCV), Munich, Germany, pp. 431-447, 2018.
  14. Y. Liang, D. Lee, Y. Li and B.-S. Shin, “Unpaired medical image colorization using generative adversarial network,” Multimedia Tools and Applications, vol. 81, no. 19, pp. 26669-26683, 2022.
  15. S. Zang, M. Chen, Z. Ai, J. Chi, G. Yang, C. Chen, and T. Yu,“Texture-aware gray-scale image colorization using a bistream generative adversarial network with multi scale attention structure,” Engineering Applications of Artificial Intelligence, vol. 122, pp. 106094, 2023.
  16. G. Larsson, M. Maire, and G. Shakhnarovich, “Learning representations for automatic colorization,” in European Conference on Computer Vision (ECCV), Midtown Manhattan, NY, US: Springer, pp. 577-593, 2016.
  17. M. Gain, A. Rahman, R. Debnath, M. M. Alnfiai, A. Sheikh, M. Masud and A. K. Bairagi, “An Improved Encoder-Decoder CNN with Region-Based Filtering for Vibrant Colorization,” Computer Systems Science and Engineering, vol. 46, no. 1, pp. 1059-1077, 2023.
  18. J. An, K. K. Gagnon, Q. Shi, H. Xie and R. Cao, “Image colorization with convolutional neural networks,” in 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Manhattan, NY, US: IEEE, pp. 1-4, 2019.
  19. S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification,” ACM Transactions on Graphics(ToG), vol. 35, no. 4, pp. 1–11, 2016.
  20. J. W. Su, H. K. Chu and J. B. Huang, “Instance-aware image colorization,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, pp. 7968-7977, 2020
  21. J. Dai, B. Jiang, C. Yang, L. Sun and B. Zhang, “Local pyramid attention and spatial semantic modulation for automatic image colorization,” in CCF Conference on Big Data, Midtown Manhattan, NY, US: Springer, pp. 165-181, 2022.
  22. K. Ruwani M. Fernando, and Chris P. Tsokos, “Dynamically Weighted Balanced Loss: Class Imbalanced Learning and Confidence Calibration of Deep Neural Networks,” In IEEE Trans. Neural Networks and Learning Systems, vol. 33, no. 7, pp. 2940-2951, 2021. doi: 10.1109/TNNLS.2020.3047335.
  23. D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” ICLR, 2017.
  24. B. C. Wallace and I. J. Dahabreh, “Improving class probability estimates for imbalanced data,” Knowledge and information systems, vol. 41, no. 1, pp. 33–52, 2014.
  25. R. Zhang, P. Isola and A. A. Efros, “Colorful image colorization,” in European Conference on Computer Vision, Midtown Manhattan, NY, US: Springer, pp. 649–666, 2016.
  26. M. Gain, A. D. Raha, R. Debnath, “CCC: Color Classified Colorization,” arXiv preprint arXiv:2403.01476 (2023).
  27. B. Zhou, A. Lapedriza, A. Khosla, A. Oliva and A. Torralba, “Places: a 10 million image database for scene recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 6, pp. 1452-1464, 2018.
  28. R. Zhang, J. Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Transactions on Graphics (TOG), vol. 36, no. 4, pp. 1- 11, 2017.
  29. E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, “Color transfer between images,” IEEE Computer graphics and applications, vol. 21, no. 5 pp. 34–41, 2001.
  30. G. Charpiat, H. Matthias, and S. Bernhard, “Automatic image colorization via multimodal predictions.” In Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, pp. 126-139, 2008.
  31. R. K. Gupta, A. Y.-S. Chia, D. Rajan, E. S. Ng, and H. Zhiyong, “Image colorization using similar images.” In Proceedings of the 20th ACM international conference on Multimedia, pp. 369-378. 2012.
  32. A. Bugeau, V. -T. Ta and N. Papadakis, “Variational Exemplar-Based Image Colorization,” in IEEE Transactions on Image Processing, vol. 23, no. 1, pp. 298-307, Jan. 2014, doi: 10.1109/TIP.2013.2288929.
  33. R. Dahl, “Automatic colorization,” 2016. [Online]. Available: https://tinyclouds.org/colorize.
  34. J. Hwang and Y. Zhou, “Image colorization with deep convolutional neural networks,” in Stanford University, Tech. Rep., vol. 219, Stanford, CA 94305, United States, pp. 1-7, 2016.
  35. F. Baldassarre, D. G. Morın, and L. Rodes-Guirao, “Deep koalarization: image colorization using cnns and inception-resnet-v2,” arXiv preprint arXiv:1712.03400, 2017.
  36. T. -T. Nguyen-Quynh, S. -H. Kim and N. -T. Do, “Image colorization using the global scene-context style and pixel-wise semantic segmentation,” IEEE Access, vol. 8, pp. 214098-214114, 2020.
  37. J. Yun, S. Lee, M. Park, and J. Choo, “iColoriT: Towards Propagating Local Hints to the Right Region in Interactive Colorization by Leveraging Vision Transformer,” In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1787-1796. 2023.
  38. G. Kong, H. Tian, X. Duan, and H. Long, “Adversarial edge-aware image colorization with semantic segmentation,” IEEE Access, vol. 9, pp. 28194-28203, 2021.
  39. G. Ozbulak, “Image colorization by capsule networks,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, pp. 0-0, 2019.
  40. S, Treneska, E. Zdravevski, I. M. Pires, P. Lameski, and S. Gievska, “GAN-based image colorization for self-supervised visual feature learning,” Sensors, vol. 22, no. 4, pp. 1599, 2022.
  41. J. Antic, “A deep learning based project for colorizing and restoring old images and video!,” 2018. [Online]. Available: https://github.com/jantic/DeOldify.
  42. A. R. Robertson, “The CIE 1976 color-difference formulae”, Color Research & Application, vol. 2, no. 1, pp.7–11, 1977.
  43. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database”, IEEE Computer Vision and Pattern Recognition (CVPR), 2009.
  44. M. Nilsback, and A. Zisserman, “Automated Flower Classification over a Large Number of Classes”, In Proc. Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
  45. Z. Liu, P. Luo, X. Wang, and X. Tang, “Large-scale CelebFaces Attributes (CelebA) Dataset”, In Proc. International Conference on Computer Vision (ICCV), 2015.
  46. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, “Microsoft coco: Common objects in context”, In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, pages 740-755, 2014.
  47. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, and A. Desmaison, “Pytorch: an imperative style, high-performance deep learning library,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  48. Z. Wang and A.C. Bovik, “Mean squared error: Love it or leave it? A new look at signal fidelity measures,” IEEE Signal Processing Magazine, vol. 26, no. 1, pp.98-117, 2009.
  49. A. Hore´ and D. Ziou, “Image quality metrics: PSNR vs. SSIM,” in 20th International Conference on Pattern Recognition, Manhattan, NY, US: IEEE, pp. 2366–2369, 2010.
  50. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric”, in Proc. CVPR, 2018.
  51. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local nash equilibrium”, in Proc. Advances in Neural Information Processing Systems, vol. 30, 2017.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now