SSIM-Based CTU-Level Joint Optimal Bit Allocation and Rate Distortion Optimization

Structural similarity (SSIM)-based distortion $D\text{SSIM}$ is more consistent with human perception than the traditional mean squared error $D\text{MSE}$. To achieve better video quality, many studies on optimal bit allocation (OBA) and rate-distortion optimization (RDO) used $D\text{SSIM}$ as the distortion metric. However, many of them failed to optimize OBA and RDO jointly based on SSIM, thus causing a non-optimal R-$D\text{SSIM}$ performance. This problem is due to the lack of an accurate R-$D\text{SSIM}$ model that can be used uniformly in both OBA and RDO. To solve this problem, we propose a $D\text{SSIM}$-$D\text{MSE}$ model first. Based on this model, the complex R-$D\text{SSIM}$ cost in RDO can be calculated as simpler R-$D\text{MSE}$ cost with a new SSIM-related Lagrange multiplier. This not only reduces the computation burden of SSIM-based RDO, but also enables the R-$D\text{SSIM}$ model to be uniformly used in OBA and RDO. Moreover, with the new SSIM-related Lagrange multiplier in hand, the joint relationship of R-$D\text{SSIM}$-$\lambda\text{SSIM}$ (the negative derivative of R-$D\text{SSIM}$) can be built, based on which the R-$D\text{SSIM}$ model parameters can be calculated accurately. With accurate and unified R-$D_\text{SSIM}$ model, SSIM-based OBA and SSIM-based RDO are unified together in our scheme, called SOSR. Compared with the HEVC reference encoder HM16.20, SOSR saves 4%, 10%, and 14% bitrate under the same SSIM in all-intra, hierarchical and non-hierarchical low-delay-B configurations, which is superior to other state-of-the-art schemes.