Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search (2210.06015v4)

Published 12 Oct 2022 in cs.LG and stat.ML

Abstract: Energy consumption from the selection, training, and deployment of deep learning models has seen a significant uptick recently. This work aims to facilitate the design of energy-efficient deep learning models that require less computational resources and prioritize environmental sustainability by focusing on the energy consumption. Neural architecture search (NAS) benefits from tabular benchmarks, which evaluate NAS strategies cost-effectively through precomputed performance statistics. We advocate for including energy efficiency as an additional performance criterion in NAS. To this end, we introduce an enhanced tabular benchmark encompassing data on energy consumption for varied architectures. The benchmark, designated as EC-NAS, has been made available in an open-source format to advance research in energy-conscious NAS. EC-NAS incorporates a surrogate model to predict energy consumption, aiding in diminishing the energy expenditure of the dataset creation. Our findings emphasize the potential of EC-NAS by leveraging multi-objective optimization algorithms, revealing a balance between energy usage and accuracy. This suggests the feasibility of identifying energy-lean architectures with little or no compromise in performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. “A comprehensive survey of neural architecture search: Challenges and solutions,” ACM Computing Surveys, vol. 54, no. 4, pp. 1–34, 2021.
  2. “Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition,” in International Conference on Computer Vision (ICCV), 2021.
  3. “Accelerating neural architecture search using performance prediction,” in International Conference on Learning Representations (ICLR) - Workshop Track, 2017.
  4. “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International Conference on Machine Learning (ICML), 2019.
  5. “Green AI,” Communications of the ACM, vol. 63, no. 12, pp. 54–63, 2020.
  6. “Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models,” ICML Workshop on Challenges in Deploying and monitoring Machine Learning Systems, 2020.
  7. “Compute trends across three eras of machine learning,” in International Joint Conference on Neural Networks (IJCNN), 2022.
  8. “Tabular benchmarks for joint architecture and hyperparameter optimization,” Arxiv, 2019.
  9. “Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks,” in International Conference on Learning Representations (ICLR), 2022.
  10. “NAS-Bench-101: Towards reproducible neural architecture search,” in International Conference on Machine Learning (ICML), 2019.
  11. “NAS-Bench-1Shot1: Benchmarking and dissecting one-shot neural architecture search,” in International Conference on Learning Representations (ICLR), 2020.
  12. “EA-HAS-bench: Energy-aware hyperparameter and architecture search benchmark,” in International Conference on Learning Representations (ICLR), 2023.
  13. “Neural architecture search for speech emotion recognition,” Arxiv, 2022.
  14. “Search for efficient deep visual-inertial odometry through neural architecture search,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
  15. Alex Krizhevsky, “Learning multiple layers of features from tiny images,” Tech. Rep., Univeristy of Toronto, 2009.
  16. “Measuring the carbon intensity of ai in cloud instances,” in Conference on Fairness, Accountability, and Transparency (FAccT), 2022.
  17. “Multi-objective optimization with unbounded solution sets,” in NeurIPS Workshop on Bayesian Optimization (BayesOpt 2016), 2016.
  18. “Bag of baselines for multi-objective joint neural architecture search and hyperparameter optimization,” in ICML Workshop on Automated Machine Learning (AutoML), 2021.
  19. “Nas-bench-x11 and the power of learning curves,” Advances in Neural Information Processing Systems (NeurIPS), 2021.
  20. “KNAS: green neural architecture search,” in International Conference on Machine Learning (ICML), 2021.
  21. “Carbon footprint of selecting and training deep learning models for medical image analysis,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2022.
  22. “Few-shot neural architecture search,” in International Conference on Machine Learning (ICML), 2021.
  23. “Uptime Institute global data center survey 2020,” Tech. Rep., Uptime Institute, 07 2020.
  24. “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems (NeurIPS)). 2019.
  25. “Adam: A method for stochastic optimization,” in International Conference on Learning Representations, 2015.
  26. “Towards the systematic reporting of the energy and carbon footprints of machine learning,” Journal of Machine Learning Research, vol. 21, no. 248, pp. 1–43, 2020.
  27. “MobileNets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.
  28. “Constructing fast network through deconstruction of convolution,” Advances in Neural Information Processing Systems, 2018.
  29. “Hyperparameter power impact in transformer language model training,” in Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, Virtual, Nov. 2021, pp. 96–118, Association for Computational Linguistics.
  30. “Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach,” IEEE Transactions on Evolutionary Computation, vol. 3, no. 4, pp. 257–271, 1999.
  31. “Performance assessment of multiobjective optimizers: An analysis and review,” IEEE Transactions on Evolutionary Computation, vol. 7, no. 2, pp. 117–132, 2003.
  32. “Speeding up many-objective optimization by Monte Carlo approximations,” Artificial Intelligence, vol. 204, pp. 22–29, 2013.
  33. “Approximation quality of the hypervolume indicator,” Artificial Intelligence, vol. 195, pp. 265–290, 2013.
  34. “SMS-EMOA: Multiobjective selection based on dominated hypervolume,” European Journal of Operational Research, vol. 181, no. 3, pp. 1653–1669, 2007.
  35. “Covariance matrix adaptation for multi-objective optimization,” Evolutionary Computation, vol. 15, no. 1, pp. 1–28, 2007.
  36. “HypE: An algorithm for fast hypervolume-based many-objective optimization,” Evolutionary computation, vol. 19, no. 1, pp. 45–76, 2011.
  37. James Edward Baker, “Adaptive selection methods for genetic algorithms,” in International Conference on Genetic Algorithms and their Applications, 1985, vol. 1.
  38. “How genetic algorithms work: A critical look at implicit parallelism,” in International Conference on Genetic Algorithms, 1989, pp. 20–27.
Citations (2)
View on Semantic Scholar

Summary

  • The paper introduces energy consumption as a critical evaluation metric in NAS, proposing the EC-NAS benchmark for sustainable deep learning model design.
  • It designs an open-source tabular benchmark with precomputed energy and performance data, enabling multi-objective optimization for both energy use and accuracy.
  • The paper also presents a surrogate model to predict energy consumption, significantly reducing the computational expense of searching for efficient neural architectures.

The paper "EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search" addresses the growing concern of energy consumption in deep learning model selection, training, and deployment. The authors focus on enhancing the design of energy-efficient models by emphasizing the importance of energy usage as a critical performance criterion in Neural Architecture Search (NAS).

Key Contributions and Methodology

  • Energy Consumption as a Criterion: Traditionally, NAS benchmarks focus on model accuracy and computational cost. This work introduces energy efficiency as an additional dimension to evaluate NAS strategies, aligning with the need for environmentally sustainable AI.
  • EC-NAS Benchmark: The authors propose EC-NAS, an enhanced tabular benchmark that contains precomputed performance statistics, including energy consumption data for various architectures. This benchmark is designed to facilitate research focused on reducing the energy footprint of deep learning.
  • Surrogate Model for Energy Prediction: To address the high energy cost of dataset creation in NAS, the paper presents a surrogate model capable of predicting the energy consumption of different architectures. This approach helps mitigate the computational and energy expenses involved.
  • Open-Source Accessibility: EC-NAS is released as an open-source resource, highlighting the authors' commitment to promoting research into energy-efficient NAS.

Findings

The research demonstrates the efficacy of EC-NAS through the application of multi-objective optimization algorithms, balancing energy usage and model accuracy. Key findings include:

  • Energy-Accuracy Trade-off: The paper reveals that it is possible to identify architectures that are energy-efficient without significantly sacrificing accuracy. This underscores the potential for more sustainable model designs in AI.
  • Importance of Energy-Sensitive Benchmarks: By incorporating energy consumption into NAS evaluations, researchers can develop models that are both performance-effective and environmentally responsible.

Overall, this paper advocates for an energy-conscious approach in NAS research, providing tools and insights to propel advancements in sustainable deep learning architectures. It represents a significant step towards reducing the ecological impact of AI technologies.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - saintslab/EC-NAS-Bench: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search (15 stars)