Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
157 tokens/sec
GPT-4o
43 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Perception-Distortion Balanced Super-Resolution: A Multi-Objective Optimization Perspective (2312.15408v2)

Published 24 Dec 2023 in eess.IV and cs.CV

Abstract: High perceptual quality and low distortion degree are two important goals in image restoration tasks such as super-resolution (SR). Most of the existing SR methods aim to achieve these goals by minimizing the corresponding yet conflicting losses, such as the $\ell_1$ loss and the adversarial loss. Unfortunately, the commonly used gradient-based optimizers, such as Adam, are hard to balance these objectives due to the opposite gradient decent directions of the contradictory losses. In this paper, we formulate the perception-distortion trade-off in SR as a multi-objective optimization problem and develop a new optimizer by integrating the gradient-free evolutionary algorithm (EA) with gradient-based Adam, where EA and Adam focus on the divergence and convergence of the optimization directions respectively. As a result, a population of optimal models with different perception-distortion preferences is obtained. We then design a fusion network to merge these models into a single stronger one for an effective perception-distortion trade-off. Experiments demonstrate that with the same backbone network, the perception-distortion balanced SR model trained by our method can achieve better perceptual quality than its competitors while attaining better reconstruction fidelity. Codes and models can be found at https://github.com/csslc/EA-Adam}{https://github.com/csslc/EA-Adam.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Z. Wang, J. Chen, and S. C. H. Hoi, “Deep learning for image super-resolution: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 10, pp. 3365–3387, 2021.
  2. D. Chao, L. C. Change, H. Kaiming, and T. Xiaoou, “Learning a deep convolutional network for image super-resolution,” in European conference on computer vision, 2014, pp. 184–199.
  3. C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in European conference on computer vision.   Springer, 2016, pp. 391–407.
  4. M. S. Sajjadi, B. Scholkopf, and M. Hirsch, “EnhanceNet: Single image super-resolution through automated texture synthesis,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 4491–4500.
  5. M. Haris, G. Shakhnarovich, and N. Ukita, “Deep back-projection networks for super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1664–1673.
  6. J. Kim, J. K. Lee, and K. M. Lee, “Deeply-recursive convolutional network for image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1637–1645.
  7. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4681–4690.
  8. X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “ESRGAN: Enhanced super-resolution generative adversarial networks,” in Proceedings of the European conference on computer vision (ECCV) workshops, 2018, pp. 0–0.
  9. Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 286–301.
  10. S. Anwar and N. Barnes, “Densely residual laplacian super-resolution,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 3, pp. 1192–1204, 2020.
  11. T. Tong, G. Li, X. Liu, and Q. Gao, “Image super-resolution using dense skip connections,” in Proceedings of the IEEE International Conference on computer vision, 2017, pp. 4799–4807.
  12. J. Liang, H. Zeng, and L. Zhang, “Details or Artifacts: A locally discriminative learning approach to realistic image super-resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5657–5666.
  13. T. Xu, P. Mi, X. Zheng, L. Li, F. Chao, G. Jiang, W. Zhang, Y. Zhou, and R. Ji, “What hinders perceptual quality of PSNR-oriented methods?” arXiv preprint arXiv:2201.01034, 2022.
  14. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  15. J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in European conference on computer vision.   Springer, 2016, pp. 694–711.
  16. Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, vol. 2.   Ieee, 2003, pp. 1398–1402.
  17. Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6228–6237.
  18. J. W. Soh, G. Y. Park, J. Jo, and N. I. Cho, “Natural and realistic single image super-resolution with explicit natural manifold discrimination,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 8122–8131.
  19. Y. Blau, R. Mechrez, R. Timofte, T. Michaeli, and L. Zelnik-Manor, “The 2018 PIRM challenge on perceptual image super-resolution,” in Computer Vision – ECCV 2018 Workshops.   Cham: Springer International Publishing, 2019, pp. 334–355.
  20. K. Zhang, L. V. Gool, and R. Timofte, “Deep unfolding network for image super-resolution,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 3217–3226.
  21. W. Zhang, Y. Liu, C. Dong, and Y. Qiao, “RankSRGAN: Generative adversarial networks with ranker for image super-resolution,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3096–3105.
  22. Q. Cai, J. Li, H. Li, Y.-H. Yang, F. Wu, and D. Zhang, “Tdpn: Texture and detail-preserving network for single image super-resolution,” IEEE Transactions on Image Processing, vol. 31, pp. 2375–2389, 2022.
  23. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020.
  24. E. Prashnani, H. Cai, Y. Mostofi, and P. Sen, “PieAPP: Perceptual image-error assessment through pairwise preference,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1808–1817.
  25. D. Fuoli, L. Van Gool, and R. Timofte, “Fourier space losses for efficient perceptual image super-resolution,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2360–2369.
  26. W. Li, K. Zhou, L. Qi, L. Lu, and J. Lu, “Best-Buddy GANs for highly detailed image super-resolution,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 2, 2022, pp. 1412–1420.
  27. Y. Zhang, B. Ji, J. Hao, and A. Yao, “Perception-distortion balanced ADMM optimization for single-image super-resolution,” in European Conference on Computer Vision.   Springer, 2022, pp. 108–125.
  28. L. Xie, X. Wang, X. Chen, G. Li, Y. Shan, J. Zhou, and C. Dong, “Desra: detect and delete the artifacts of gan-based real-world super-resolution models,” arXiv preprint arXiv:2307.02457, 2023.
  29. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  30. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586–595.
  31. N. Srinivas and K. Deb, “Muiltiobjective optimization using nondominated sorting in genetic algorithms,” Evolutionary computation, vol. 2, no. 3, pp. 221–248, 1994.
  32. K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: NSGA-II,” IEEE transactions on evolutionary computation, vol. 6, no. 2, pp. 182–197, 2002.
  33. Q. Zhang and H. Li, “MOEA/D: A multiobjective evolutionary algorithm based on decomposition,” IEEE Transactions on evolutionary computation, vol. 11, no. 6, pp. 712–731, 2007.
  34. Y. Wang, L. Wang, H. Wang, P. Li, and H. Lu, “Blind single image super-resolution with a mixture of deep networks,” Pattern Recognition, vol. 102, p. 107169, 2020.
  35. J. Liang, H. Zeng, and L. Zhang, “Efficient and degradation-adaptive network for real-world image super-resolution,” in European Conference on Computer Vision, 2022.
  36. X. Wang, K. Yu, C. Dong, X. Tang, and C. C. Loy, “Deep network interpolation for continuous imagery effect transition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1692–1701.
  37. M. Liu, J. Pan, Z. Yan, W. Zuo, and L. Zhang, “Adaptive network combination for single-image reflection removal: A domain generalization perspective,” arXiv preprint arXiv:2204.01505, 2022.
  38. B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 136–144.
  39. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2472–2481.
  40. J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “SwinIR: Image restoration using swin transformer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1833–1844.
  41. X. Zhang, H. Zeng, S. Guo, and L. Zhang, “Efficient long-range attention network for image super-resolution,” arXiv preprint arXiv:2203.06697, 2022.
  42. X. Chen, X. Wang, J. Zhou, Y. Qiao, and C. Dong, “Activating more pixels in image super-resolution transformer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 367–22 377.
  43. Q. Cai, Y. Qian, J. Li, J. Lyu, Y.-H. Yang, F. Wu, and D. Zhang, “Hipa: Hierarchical patch transformer for single image super resolution,” IEEE Transactions on Image Processing, vol. 32, pp. 3226–3237, 2023.
  44. H. Huang, R. He, Z. Sun, and T. Tan, “Wavelet-SRNET: A wavelet-based CNN for multi-scale face super resolution,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1689–1697.
  45. A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” IEEE Signal processing letters, vol. 20, no. 3, pp. 209–212, 2012.
  46. C. Ma, C.-Y. Yang, X. Yang, and M.-H. Yang, “Learning a no-reference quality metric for single-image super-resolution,” Computer Vision and Image Understanding, vol. 158, pp. 1–16, 2017.
  47. T. Higuchi, S. Tsutsui, and M. Yamamura, “Theoretical analysis of simplex crossover for real-coded genetic algorithms,” in International Conference on Parallel Problem Solving from Nature.   Springer, 2000, pp. 365–374.
  48. X. Zhou, A. K. Qin, M. Gong, and K. C. Tan, “A survey on evolutionary construction of deep neural networks,” IEEE Transactions on Evolutionary Computation, vol. 25, no. 5, pp. 894–912, 2021.
  49. R. Saravanan, S. Ramabalan, N. G. R. Ebenezer, and C. Dharmaraja, “Evolutionary multi criteria design optimization of robot grippers,” Applied Soft Computing, vol. 9, no. 1, pp. 159–172, 2009.
  50. H. Mala-Jetmarova, N. Sultanova, and D. Savic, “Lost in optimisation of water distribution systems? A literature review of system design,” Water, vol. 10, no. 3, p. 307, 2018.
  51. Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun, “Single path one-shot neural architecture search with uniform sampling,” in European conference on computer vision.   Springer, 2020, pp. 544–560.
  52. J. Yu, P. Jin, H. Liu, G. Bender, P.-J. Kindermans, M. Tan, T. Huang, X. Song, R. Pang, and Q. Le, “BigNAS: Scaling up neural architecture search with big single-stage models,” in European Conference on Computer Vision.   Springer, 2020, pp. 702–717.
  53. G. Ying, X. He, B. Gao, B. Han, and X. Chu, “EAGAN: Efficient two-stage evolutionary architecture search for GANs,” in European Conference on Computer Vision.   Springer, 2022, pp. 37–53.
  54. Y. Zhou, G. G. Yen, and Z. Yi, “A knee-guided evolutionary algorithm for compressing deep neural networks,” IEEE transactions on cybernetics, vol. 51, no. 3, pp. 1626–1638, 2019.
  55. X. Cui, W. Zhang, Z. Tüske, and M. Picheny, “Evolutionary stochastic gradient descent for optimization of deep neural networks,” Advances in neural information processing systems, vol. 31, 2018.
  56. H. Zhang, K. Hao, L. Gao, B. Wei, and X. Tang, “Optimizing deep neural networks through neuroevolution with stochastic gradient descent,” IEEE Transactions on Cognitive and Developmental Systems, 2022.
  57. S. Yang, Y. Tian, C. He, X. Zhang, K. C. Tan, and Y. Jin, “A gradient-guided evolutionary approach to training deep neural networks,” IEEE Transactions on Neural Networks and Learning Systems, 2021.
  58. Y. Tian, H. Chen, H. Ma, X. Zhang, K. C. Tan, and Y. Jin, “Integrating conjugate gradients into evolutionary algorithms for large-scale continuous multi-objective optimization,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 10, pp. 1801–1817, 2022.
  59. Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, “GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in International conference on machine learning.   PMLR, 2018, pp. 794–803.
  60. Y. Chen, X. Dai, M. Liu, D. Chen, L. Yuan, and Z. Liu, “Dynamic convolution: Attention over convolution kernels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 030–11 039.
  61. E. Agustsson and R. Timofte, “NTIRE 2017 challenge on single image super-resolution: Dataset and study,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 126–135.
  62. M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” 2012.
  63. R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in International conference on curves and surfaces.   Springer, 2010, pp. 711–730.
  64. D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2.   IEEE, 2001, pp. 416–423.
  65. J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 5197–5206.
  66. Y. Matsui, K. Ito, Y. Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, and K. Aizawa, “Sketch-based manga retrieval using Manga109 dataset,” Multimedia Tools and Applications, vol. 76, no. 20, pp. 21 811–21 838, 2017.
  67. C. Ma, Y. Rao, Y. Cheng, C. Chen, J. Lu, and J. Zhou, “Structure-preserving super resolution with gradient guidance,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 7769–7778.
  68. J. Park, S. Son, and K. M. Lee, “Content-aware local gan for photo-realistic super-resolution,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 10 585–10 594.
  69. W. Wang, R. Guo, Y. Tian, and W. Yang, “Cfsnet: Toward a controllable feature space for image restoration,” in Proceedings of the IEEE/CVF International Conference on Computer Vision ICCV, October 2019.
  70. S. Vasu, N. Thekke Madam, and A. Rajagopalan, “Analyzing perception-distortion tradeoff using enhanced perceptual super-resolution network,” in Proceedings of the European Conference on Computer Vision ECCV Workshops, 2018, pp. 0–0.
  71. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2019.
  72. J. Shi, J. Sun, Q. Zhang, H. Zhang, and Y. Fan, “Improving pareto local search using cooperative parallelism strategies for multiobjective combinatorial optimization,” IEEE Transactions on Cybernetics, 2022.
  73. J. Shi, Q. Zhang, and J. Sun, “PPLS/D: Parallel pareto local search based on decomposition,” IEEE transactions on cybernetics, vol. 50, no. 3, pp. 1060–1071, 2018.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - csslc/EA-Adam (7 stars)