Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Moreau Envelope for Nonconvex Bi-Level Optimization: A Single-loop and Hessian-free Solution Strategy (2405.09927v1)

Published 16 May 2024 in math.OC and cs.LG

Abstract: This work focuses on addressing two major challenges in the context of large-scale nonconvex Bi-Level Optimization (BLO) problems, which are increasingly applied in machine learning due to their ability to model nested structures. These challenges involve ensuring computational efficiency and providing theoretical guarantees. While recent advances in scalable BLO algorithms have primarily relied on lower-level convexity simplification, our work specifically tackles large-scale BLO problems involving nonconvexity in both the upper and lower levels. We simultaneously address computational and theoretical challenges by introducing an innovative single-loop gradient-based algorithm, utilizing the Moreau envelope-based reformulation, and providing non-asymptotic convergence analysis for general nonconvex BLO problems. Notably, our algorithm relies solely on first-order gradient information, enhancing its practicality and efficiency, especially for large-scale BLO learning tasks. We validate our approach's effectiveness through experiments on various synthetic problems, two typical hyper-parameter learning tasks, and a real-world neural architecture search application, collectively demonstrating its superior performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Amortized implicit differentiation for stochastic bilevel optimization. In ICLR, 2022a.
  2. Non-convex bilevel games with critical point selection maps. In NeurIPS, volume 35, pp.  8013–8026, 2022b.
  3. Optimality conditions for bilevel programs via Moreau envelope reformulation. arXiv preprint arXiv:2311.14857, 2023.
  4. Beck, A. First-order methods in optimization. SIAM, 2017.
  5. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In ICML, pp.  115–123, 2013.
  6. Implicit differentiation of lasso-type models for hyperparameter optimization. In ICML, pp.  810–821, 2020.
  7. Implicit differentiation for fast hyperparameter selection in non-smooth convex learning. JMLR, 23(1):6680–6722, 2022.
  8. Variable smoothing for weakly convex composite functions. JOTA, 188:628–649, 2021.
  9. Near-optimal fully first-order algorithms for finding stationary points in bilevel optimization. arXiv preprint arXiv:2306.14853, 2023.
  10. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. In NeurIPS, volume 34, pp.  25294–25307, 2021.
  11. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, pp.  1294–1303, 2019.
  12. A fast and convergent proximal algorithm for regularized nonconvex and nonsmooth bi-level optimization. arXiv preprint arXiv:2203.16615, 2022.
  13. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. In NeurIPS, volume 35, pp.  26698–26710, 2022.
  14. Meta-learning of neural architectures for few-shot learning. In CVPR, pp.  12365–12375, 2020.
  15. Gradient-based regularization parameter selection for problems with nonsmooth penalty functions. JCGS, 27(2):426–435, 2018.
  16. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pp.  1126–1135, 2017.
  17. Forward and reverse gradient-based hyperparameter optimization. In ICML, pp.  1165–1173, 2017.
  18. Bilevel programming for hyperparameter optimization and meta-learning. In ICML, pp.  1568–1577, 2018.
  19. Value function based difference-of-convex algorithm for bilevel hyperparameter selection problems. In ICML, pp.  7164–7182, 2022.
  20. Moreau envelope based difference-of-weakly-convex reformulation and algorithm for bilevel programs. arXiv preprint arXiv:2306.16761, 2023.
  21. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246, 2018.
  22. On the iteration complexity of hypergradient computation. In ICML, pp.  3748–3758, 2020.
  23. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. SIOPT, 33(1):147–180, 2023.
  24. Huang, F. Adaptive mirror descent bilevel optimization. arXiv preprint arXiv:2311.04520, 2023a.
  25. Huang, F. On momentum-based gradient methods for bilevel optimization with nonconvex lower-level. arXiv preprint arXiv:2303.03944, 2023b.
  26. Enhanced bilevel optimization via Bregman distance. In NeurIPS, volume 35, pp.  28928–28939, 2022.
  27. Lower bounds and accelerated algorithms for bilevel optimization. JMLR, 23:1–56, 2022.
  28. Convergence of meta-learning with task-specific adaptation over partial parameters. In NeurIPS, volume 33, pp.  11490–11500, 2020a.
  29. Bilevel optimization: Nonasymptotic analysis and faster algorithms. arXiv preprint arXiv:2010.07962, 2020b.
  30. Will bilevel optimizers benefit from loops. In NeurIPS, volume 35, pp.  3011–3023, 2022.
  31. On penalty methods for nonconvex bilevel optimization and first-order stochastic approximation. arXiv preprint arXiv:2309.01753, 2023a.
  32. A fully first-order method for stochastic bilevel optimization. In ICML, pp.  18083–18113, 2023b.
  33. Improved bilevel model: Fast and optimal algorithm with theoretical guarantee. arXiv preprint arXiv:2009.00690, 2020.
  34. A fully single loop algorithm for bilevel optimization without hessian inverse. In AAAI, volume 36, pp.  7426–7434, 2022.
  35. DARTS: Differentiable architecture search. In ICLR, 2018.
  36. A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In ICML, pp.  6305–6315, 2020.
  37. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE TPAMI, 44(12):10045–10067, 2021a.
  38. A value-function-based interior-point method for non-convex bi-level optimization. In ICML, pp.  6882–6892, 2021b.
  39. Towards gradient-based bilevel optimization with non-convex followers and beyond. In NeurIPS, volume 34, pp.  8662–8675, 2021c.
  40. A general descent aggregation framework for gradient-based bi-level optimization. IEEE TPAMI, 45(1):38–57, 2022.
  41. Learning with constraint learning: New perspective, solution strategy and various applications. arXiv preprint arXiv:2307.15257, 2023a.
  42. Averaged method of multipliers for bi-level optimization without lower-level strong convexity. In ICML, pp.  21839–21866, 2023b.
  43. Augmenting iterative trajectory for bilevel optimization: Methodology, analysis and extensions. arXiv preprint arXiv:2303.16397, 2023c.
  44. Lu, S. SLM: A smoothed first-order Lagrangian method for structured constrained nonconvex optimization. In NeurIPS, 2023.
  45. First-order penalty methods for bilevel optimization. arXiv preprint arXiv:2301.01716, 2023.
  46. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. In ICLR, 2019.
  47. Task-driven dictionary learning. IEEE TPAMI, 34(4):791–804, 2011.
  48. Mordukhovich, B. S. Variational analysis and applications, volume 30. 2018.
  49. Hyperparameter learning via bilevel nonsmooth optimization. arXiv preprint arXiv:1806.01520, 2018.
  50. Pedregosa, F. Hyperparameter optimization with approximate gradient. In ICML, pp.  737–746, 2016.
  51. Meta-learning with implicit gradients. In NeurIPS, volume 32, 2019.
  52. Rockafellar, R. T. Conjugate duality and optimization. SIAM, 1974.
  53. On penalty-based bilevel gradient descent method. In ICML, pp.  30992–31015, 2023.
  54. A constrained optimization approach to bilevel optimization with multiple inner minima. arXiv preprint arXiv:2203.01123, 2022a.
  55. On the convergence theory for hessian-free bilevel algorithms. In NeurIPS, volume 35, pp.  4136–4149, 2022b.
  56. An alternating optimization method for bilevel problems under the Polyak-Łojasiewicz condition. In NeurIPS, 2023.
  57. Pc-darts: Partial channel connections for memory-efficient architecture search. In ICLR, 2019.
  58. Accelerating inexact hypergradient descent for bilevel optimization. arXiv preprint arXiv:2307.00126, 2023a.
  59. Achieving O⁢(ϵ−1.5)𝑂superscriptitalic-ϵ1.5{O}(\epsilon^{-1.5})italic_O ( italic_ϵ start_POSTSUPERSCRIPT - 1.5 end_POSTSUPERSCRIPT ) complexity in Hessian/Jacobian-free stochastic bilevel optimization. In NeurIPS, 2023b.
  60. Constrained bi-level optimization: Proximal lagrangian value function approach and hessian-free algorithm. In ICLR, 2024.
  61. Optimality conditions for bilevel programming problems. Optimization, 33(1):9–27, 1995.
  62. Difference of convex algorithms for bilevel programs with applications in hyperparameter selection. MP, 198(2):1583–1616, 2023.
  63. Bome! bilevel optimization made easy: A simple first-order approach. In NeurIPS, volume 35, pp.  17248–17262, 2022.
  64. An introduction to bi-level optimization: Foundations and applications in signal processing and machine learning. arXiv preprint arXiv:2308.00788, 2023.
  65. Adversarial attacks on graph neural networks via meta learning. In ICLR, 2018.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets