Iteration Complexity of Randomized Block-Coordinate Descent Methods for Minimizing a Composite Function (1107.2848v1)

Abstract: In this paper we develop a randomized block-coordinate descent method for minimizing the sum of a smooth and a simple nonsmooth block-separable convex function and prove that it obtains an $\epsilon$-accurate solution with probability at least $1-\rho$ in at most $O(\tfrac{n}{\epsilon} \log \tfrac{1}{\rho})$ iterations, where $n$ is the number of blocks. For strongly convex functions the method converges linearly. This extends recent results of Nesterov [Efficiency of coordinate descent methods on huge-scale optimization problems, CORE Discussion Paper #2010/2], which cover the smooth case, to composite minimization, while at the same time improving the complexity by the factor of 4 and removing $\epsilon$ from the logarithmic term. More importantly, in contrast with the aforementioned work in which the author achieves the results by applying the method to a regularized version of the objective function with an unknown scaling factor, we show that this is not necessary, thus achieving true iteration complexity bounds. In the smooth case we also allow for arbitrary probability vectors and non-Euclidean norms. Finally, we demonstrate numerically that the algorithm is able to solve huge-scale $\ell_1$-regularized least squares and support vector machine problems with a billion variables.

Iteration Complexity of Randomized Block-Coordinate Descent Methods for Minimizing a Composite Function: An Overview

The paper "Iteration Complexity of Randomized Block-Coordinate Descent Methods for Minimizing a Composite Function" focuses on developing and analyzing a method to efficiently solve large-scale structured convex optimization problems. Specifically, it introduces a randomized block-coordinate descent (RBCD) method for minimizing the sum of a smooth function and a simple nonsmooth block-separable convex function.

Key Contributions

The paper presents a comprehensive analysis of the iteration complexity of the RBCD method for both general convex and strongly convex composite functions. Unlike the work of Nesterov, this method achieves improved complexity bounds and removes the need for regularization with unknown scaling factors.

1. Uniform Block-Coordinate Descent for Composite Functions (UCDC): The paper analyzes the UCDC method where the block to be updated is chosen uniformly at random. For convex objective functions, UCDC achieves an $\epsilon$-accurate solution with high probability in at most $O(\tfrac{n\max\{#1{L}{x_0}, F(x_0)-F^*\}}{\epsilon} \log \tfrac{1}{\rho})$ iterations, where $n$ is the number of blocks. For strongly convex functions, UCDC exhibits linear convergence with complexity $O(n \log\tfrac{F(x_0)-F^*}{\rho\epsilon})$.
2. Randomized Block-Coordinate Descent for Smooth Functions (RCDS): This method extends the analysis to smooth functions allowing for arbitrary probability vectors and non-Euclidean norms. It achieves an $\epsilon$-accurate solution with high probability using $O(\tfrac{#1{LP^{-1}{x_0}}{\epsilon} (1 + \log \tfrac{1}{\rho}) - 2)$ iterations in the general convex case, where $L$ encodes the block coordinate Lipschitz constants of the gradient of $f$. For strongly convex functions, the complexity improves to $$O(\tfrac{1}{\mu}\log \tfrac{f(x_0)-f^*}{\epsilon\rho})$$.

Numerical Results and Practical Implications

The numerical experiments demonstrate the practical efficacy of the RBCD method. The algorithm successfully addresses large-scale optimization problems such as $\ell_1$-regularized least squares and support vector machine problems with up to a billion variables. Notably, the experiments highlight the following practical aspects:

1. Scalability: The method handles enormous problem sizes efficiently, indicating its suitability for real-world large-scale applications.
2. Adaptivity: By allowing arbitrary probability vectors, the method can be fine-tuned to achieve better performance on different types of problem instances. Introduced heuristics, such as adaptive changing of probabilities, showcase potential speed-ups.
3. Comparative Performance: RBCD demonstrates improvements over existing methods, especially in the context of minimizing composite functions with smooth and nonsmooth components.

Theoretical and Practical Significance

The implications of the research are significant for both theoretical and practical advancements in convex optimization. The improved iteration complexity bounds confirm that RBCD methods can achieve faster convergence compared to traditional approaches, which is critical for managing large-scale data efficiently. On the theoretical front, the analysis extends the understanding of RBCD methods in composite optimization settings, contributing to the broader literature on convex optimization techniques.

Future Developments in AI

Looking forward, RBCD methods could be further optimized by exploring adaptive and accelerated variants, potentially leveraging more sophisticated probabilistic models to guide the coordinate selection process. Additionally, integrating these methods into machine learning frameworks could substantially improve the training times of models on massive datasets. As AIs continue to process increasing amounts of data, the development of efficient and scalable optimization algorithms like RBCD will be pivotal in advancing their capabilities.