TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Search (2404.00271v2)

Abstract: Neural Architecture Search (NAS) is a powerful technique for discovering high-performing CNN architectures, but most existing methods rely on costly training or extensive sampling. Zero-shot NAS offers a training-free alternative by using proxies to predict architecture performance. However, existing proxies are often suboptimal -- frequently outperformed by simple metrics like parameter count or FLOPs -- and they generalize poorly across different search spaces. Moreover, current model-based proxies struggle to adapt to new operators without access to ground-truth accuracy, limiting their transferability. We propose TG-NAS, a universal, model-based zero-cost (ZC) proxy that combines a Transformer-based operator embedding generator with a Graph Convolutional Network (GCN) to predict architecture performance. Unlike prior model-based predictors, TG-NAS requires no retraining and generalizes across arbitrary search spaces. It serves as a standalone ZC proxy with strong data efficiency, robustness, and cross-space consistency. Extensive evaluations across diverse NAS benchmarks demonstrate TG-NAS's superior rank correlation and generalizability compared to existing proxies. Additionally, it improves search efficiency by up to 300x and discovers architectures achieving 93.75% CIFAR-10 accuracy on NAS-Bench-201 and 74.9% ImageNet top-1 accuracy on the DARTS space, establishing TG-NAS as a promising foundation for efficient, generalizable NAS.