Multi-objective Differentiable Neural Architecture Search (2402.18213v3)

Abstract: Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that require training a neural network. Typically, in MOO for neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences to trade-off performance and hardware metrics, yielding representative and diverse architectures across multiple devices in just a single search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments involving up to 19 hardware devices and 3 different objectives demonstrate the effectiveness and scalability of our method. Finally, we show that, without any additional costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation and a decoder-only space for language modelling.

• The paper introduces a unified hardware-aware and gradient-based NAS method that generates diverse Pareto-optimal architectures for multiple devices in one search.
• It employs a MetaHypernetwork conditioned on user preferences and device embeddings to convert continuous architecture distributions into differentiable discrete designs.
• Empirical results demonstrate superior Pareto front quality and search efficiency, outperforming baseline methods across various tasks and objectives.

Multi-objective Differentiable Neural Architecture Search: MODNAS

Introduction and Motivation

Multi-objective optimization (MOO) for neural architecture search (NAS) is essential for navigating tradeoffs between predictive performance and hardware efficiency metrics (e.g., latency, energy consumption, memory footprint) as modern neural architectures are increasingly deployed across heterogeneous computing devices. However, mainstream NAS approaches commonly address this challenge by integrating hardware constraints directly into the search objective, yielding only a single solution per constraint and per device—requiring repeated, costly searches for profiling the entire Pareto front across devices or tradeoff constraints.

"Multi-objective Differentiable Neural Architecture Search" (2402.18213) proposes a unified, hardware-aware, and gradient-based NAS formulation that produces diverse, distributed Pareto-optimal architecture sets for varying user preference scalarizations and different devices in a single search, thereby achieving substantial search efficiency improvements and state-of-the-art Pareto set quality.

Methodological Overview: MODNAS Formulation

The proposed method, MODNAS, frames hardware-aware multi-objective NAS as a multi-task, multi-objective bi-level optimization problem. Each device is treated as a separate optimization task with $M$ potentially conflicting objectives (e.g., accuracy, latency, energy). The Pareto front sampling is controlled by a user-specified preference vector (scalarization) $r$ in the $M$-dimensional simplex, enabling convex combinations of objectives and flexible navigation of tradeoffs.

A key innovation is the use of a MetaHypernetwork $H_\Phi$ conditioned on both a preference vector $r$ and learnable device embeddings $d_t$. This hypernetwork outputs a continuous, unnormalized architectural distribution $\Tilde{\alpha}$, which the Architect module transforms into differentiable discrete architectures via the ReinMax estimator, eliminating the need for expensive search restarts for every device or constraint.

Figure 1: MODNAS system diagram, featuring the MetaHypernetwork $H_\Phi(r, d_t)$ to produce architecture distributions conditioned on user preferences and device characteristics, facilitating efficient multi-objective optimization across devices.

The Supernetwork ties this mechanism together via parameter sharing, supporting memory efficiency and scalable architecture evaluation. For non-differentiable or costly hardware metrics, a pre-trained MetaPredictor regresses device-specific objective values from architecture and device embeddings, enabling gradient-based updates for all objectives.

Optimization employs Multiple Gradient Descent (MGD)—using Frank-Wolfe–driven convex combinations of per-task gradients—to find updates for $H_\Phi$ that yield Pareto improvements on all devices and objectives. The algorithm performs bi-level, stochastic updates: outer-level updates $H_\Phi$ (architecture distribution), inner-level updates the Supernetwork weights, and both exploit scalarizations sampled from the Dirichlet prior over the simplex.

Experimental Evaluation

Large-scale Multi-device and Multi-objective Benchmarks

MODNAS is evaluated over NAS-Bench-201 (19 devices, up to 3 objectives), MobileNetV3/OFA (12 devices, ImageNet-1k), and a hardware-aware Transformer (HAT) search space (WMT'14 En-De, 3 devices). Each experiment assesses Pareto front quality using hypervolume (HV), GD, IGD, and variants, profiling Pareto fronts on test devices (zero-shot) by passing preference vectors and device embeddings through the trained MetaHypernetwork.

Figure 2: MODNAS achieves broad Pareto front coverage (radar plot hypervolume) across 19 devices—significantly outperforming random and multi-run constraint-based baselines on NAS-Bench-201.

The results indicate that MODNAS outperforms all tested baselines (e.g., random search, random hypernetwork, MetaD2A+HELP) in hypervolume, especially on unseen/test devices, and achieves better front diversity versus prior approaches focused primarily on accuracy. Notably, MODNAS yields these results via a single search, irrespective of the number of objectives ($M$) or devices ($T$).

Gradient Aggregation Schemes and Robustness

MODNAS systematically compares mean gradient, sequential updates, MC-sampled updates, and MGD. The MGD approach exhibits superior convergence speed and final HV under all tested scenarios.

Figure 3: Comparison of search hypervolume progression across gradient schemes—MGD delivers faster and higher HV convergence than baseline aggregation methods.

Three-objective Scalability

MODNAS is further validated on a tri-objective scenario (accuracy, latency, energy) on NAS-Bench-201 (FPGA and Eyeriss). The method demonstrates near-optimal HV and high-quality Pareto fronts without added search complexity.

Figure 4: 3-objective window—MODNAS matches or surpasses baselines on hypervolume and Pareto coverage when scaling to tri-objective MOO.

Modulation via User Priors

By introducing explicit scalarized or hard constraints (e.g., latency caps) into the MetaHypernetwork inputs or training procedure, MODNAS flexibly adapts the sampled Pareto front shape and focus—e.g., producing more performant architectures under tighter hardware requirements—demonstrating both flexibility and practical deployment relevance.

Figure 5: Pareto and HV sensitivity—MODNAS dynamically modulates fronts with latency constraints, highlighting front concentration and accuracy tradeoff.

Transferability and Generalization

Across tasks and predictors (including large vision and sequence datasets), MODNAS zero-shot transfers to new devices with strong Pareto coverage and accuracy, supported by reliable MetaPredictor performance for hardware metrics.

Figure 6: HAT Transformer search space—HV plots show MODNAS maintaining dominant coverage across multiple hardware domains.

ImageNet/OFA Validation

On MobileNetV3/OFA search space with 12 devices, MODNAS demonstrates efficient search cost and higher Pareto set HV across all devices, outperforming OFA+HELP, with MODNAS requiring substantially less GPU time per device/constraint coverage.

Figure 7: MODNAS hypervolume (radar) for MobileNetV3—across accuracy-latency axes and target devices, MODNAS consistently dominates baseline methods.

Implications and Future Applications

MODNAS unifies multi-objective, hardware-aware NAS using a differentiable, meta-learned approach that, for the first time, delivers full Pareto front approximation, device transfer, and constraint flexibility in a single search. The method's scalability to many objectives and devices, strong zero-shot transfer, and search efficiency directly address long-standing obstacles in deploying deep architectures under real-world compute and energy constraints.

MODNAS's framework is immediately relevant for automated deployment pipelines, where variable device constraints and user priorities are non-stationary and must be navigated with limited search budgets. The approach's extension to fairness, robustness, and other multi-objective domains is straightforward, and future work can exploit learned device/embedding spaces for rapid adaptation in dynamic or edge environments.

The methodology offers theoretical connections to meta-learning, multi-task optimization, and hypernetwork generalization. Its plugin architecture supports integration with arbitrary gradient-based NAS spaces, search spaces with complex architectural topologies, and meta-predictors for further hardware, performance, or fairness objectives.

Conclusion

MODNAS (2402.18213) introduces a principled, scalable, and hardware-aware differentiable NAS paradigm to simultaneously profile the global multi-objective Pareto front across arbitrary devices and user priorities, all in a single search. Empirical evidence on benchmarks with up to 19 devices and three objectives demonstrates robust generalizability, superior Pareto set quality, and search cost reduction, establishing a new state of the art for practical multi-objective NAS. The design is compatible with diverse objectives and offers strong potential for extension into fairness and deployment-centric auto-ML.