Papers
Topics
Authors
Recent
Search
2000 character limit reached

On the Measure of Intelligence

Published 5 Nov 2019 in cs.AI | (1911.01547v2)

Abstract: To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.

Authors (1)
Citations (210)

Summary

  • The paper presents a novel framework that redefines intelligence as the efficiency of acquiring and applying skills across diverse tasks.
  • It leverages Algorithmic Information Theory and establishes the Abstraction and Reasoning Corpus (ARC) to rigorously test AI generalization.
  • The research paves the way for developing AI systems with human-like flexibility and practical learning efficiency for real-world challenges.

Unveiling the Complexity of Measuring Intelligence in AI Systems: A Deep Dive into the "On the Measure of Intelligence" Paper

Understanding the Need for a New Definition of Intelligence

The field of AI has long been driven by the goal of replicating or surpassing human intelligence. However, achieving this overarching goal requires a clear understanding and definition of intelligence that not only caters to the human perspective but also applies to AI systems. Historically, the AI community has primarily focused on enhancing task-specific skills, underestimating the importance of broad abilities and general intelligence. The paper presents a critical examination of existing definitions and approaches to measuring intelligence, highlighting the limitations of skill-based evaluations and the necessity for a measure that embodies the essence of true intelligence – the ability to efficiently acquire and apply knowledge and skills across diverse sets of challenges.

A Fresh Perspective on Intelligence

Drawing upon Algorithmic Information Theory, the paper introduces a comprehensive framework to redefine intelligence in AI systems. Intelligence is conceptualized as skill-acquisition efficiency across a scope of tasks, emphasizing the ability to generalize from limited data and prior experience. This definition shifts the focus from mere skill proficiency towards a more holistic view of intelligence that includes adaptability, problem-solving, and learning efficiency. The framework meticulously outlines the parameters of this new definition, including the critical roles of priors (innate knowledge), experience, and generalization difficulty, and its implications for AI research and evaluation.

The Abstraction and Reasoning Corpus (ARC) as a Benchmark

To operationalize this refined understanding of intelligence, the paper introduces the Abstraction and Reasoning Corpus (ARC) - a benchmark designed to rigorously test the general intelligence of AI systems. The ARC challenges AI systems with a range of abstract tasks that require understanding and applying core knowledge principles, mirroring the foundation of human cognitive abilities. This novel benchmark adheres to the stringent criteria outlined for a fair and effective intelligence evaluation tool; it emphasizes novelty, generalization, and human-like prior knowledge, aiming to facilitate a direct comparison between human and machine intelligence.

Implications and the Road Ahead

The introduction of this new definition of intelligence and the ARC benchmark represents a paradigm shift in AI research and evaluation. It encourages the development of AI systems not just with specialized abilities but with the genuine capacity for learning, adaptability, and problem-solving. The framework and ARC set the stage for future investigations into AI’s potential to achieve a form of intelligence that is both broad and deep, fostering advancements toward truly intelligent systems.

As AI continues to evolve, this paper's insights challenge researchers to think beyond conventional metrics of success and skill acquisition. By redefining intelligence in a way that captures the essence of cognitive flexibility and efficiency, the paper lays the groundwork for the next generation of AI systems. These systems would not only excel at specific tasks but also demonstrate a human-like ability to navigate the complexities of the real world. The journey toward achieving general AI is complex and fraught with challenges, but with comprehensive frameworks like the one proposed, the field is poised to make significant strides.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 217 tweets with 12473 likes about this paper.

YouTube

It's Not About Scale, It's About Abstraction 126K views
CHATGPT DOESN'T REASON! (Top scientist bombshell) 78K views
ChatGPT und die Mathematik 2025 42K views
How Do AI Models Actually Think? 41K views
The Mathematical Foundations of Intelligence [Professor Yi Ma] 37K views
Solving Chollet's ARC-AGI with GPT4o 35K views
How Machines Learn to Ignore the Noise (Kevin Ellis + Zenna Tavares) 23K views
ARC Prize Version 2 Launch Video! 22K views
Transformers Need Glasses! 20K views
29.4% ARC-AGI-2 🤯 (TOP SCORE!) - Jeremy Berman 19K views
Artificial Intelligence in 2020 19K views
Exploring Program Synthesis: Francois Chollet, Kevin Ellis, Zenna Tavares 19K views
Decompiling Dreams: A New Approach to ARC? 18K views
Test-Time Adaptation: the key to reasoning with DL 14K views
New ARC-AGI high score! 55.5% (MindsAI) 12K views
Moving Beyond Surface Statistics (Apple researcher) 12K views
This is what happens when you let AIs debate 11K views
#68 DR. WALID SABA 2.0 - Natural Language Understanding [UNPLUGGED] 9.3K views
#70 - LETITIA PARCALABESCU - Symbolics, Linguistics [UNPLUGGED] 8.1K views
Contrastive Learning for Unpaired Image-to-Image Translation 7K views
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer 6.1K views
#97 SREEJAN KUMAR - Human Inductive Biases in Machines from Language 4.7K views
On the Measure of Intelligence (Introduction) 2.9K views
Are Neural Networks Intelligent? 2.5K views
Data Science Myths & Tips! 2.2K views
LLMs as a System of Multiple Expert Agents to solve the ARC Challenge (Detailed Walkthrough) 1.7K views
Artificial Intelligence Is Not Real Ep.0: Defining Generalization [Series on AI/Machine Learning] 1.6K views
CoDA: Contrast-Enhancing and Diversity-Promoting Data Augmentation for NLU 1.1K views
AI Agents That Matter with Sayash Kapoor and Benedikt Stroebl - Weaviate Podcast #104! 960 views
The Gradient Podcast - François Chollet: Keras and Measures of Intelligence 946 views
Pourquoi faire confiance à la science ? 825 views
OpenAI o3 Duyuruldu! AGI'a Erişildi mi? ARC-AGI Nedir? 526 views
o3, grok2, pro 2. Intro to the new set of words and ideas they came up with 438 views
An AI cookery dumpster fire, the ARC prize, and a creepy new AI friend 385 views
Como e por que Medir a Inteligência Artificial? A Perspectiva de François Chollet da Google 166 views
Session 40 - On the measure of Intelligence - "fair" AI model benchmarks 163 views
The Progress of Artificial Intelligence and Why Humans Will Not Be Made Obsolete Anytime Soon 151 views
How and Why Should We Measure Artificial Intelligence? François Chollet's Perspective from Google 118 views
Arno.mk Podcast Episode 04: "On the Measure of Intelligence" by François Chollet 96 views
Measuring AGI: AI vs Humans 60 views
INTERVIEW: Scaling Democracy w/ (Dr.) Igor Krawczuk 58 views
EP 1 Intelligence as Efficiency 47 views
Intelligence, The Bitter Lesson and the Better Lesson 45 views
On the Measure of Intelligence Paper by François Chollet - 31 views
Episode 6: Reaction Video to Open AI announcement of O3 & O3 mini 29 views
The Progress of Artificial Intelligence and Why Humans Will Not Be Made Obsolete Anytime Soon 29 views
Abstract Reasoning Corpus (ARC) Benchmark - Defining and Measuring Intelligence 1/2 28 views
Grad School and AGI - First Spring 2023 Meeting (Extra Bits) 27 views
Abstract Reasoning Corpus (ARC) Benchmark - Defining and Measuring Intelligence 2/2 19 views
On the Measure of Intelligence 8 views

HackerNews

  1. The Measure of Intelligence (16 points, 1 comment) 
  2. The Measure of Intelligence (2019) (4 points, 0 comments) 
  3. On the measure of intelligence (2019) (2 points, 1 comment) 
  4. On the Measure of Intelligence (2 points, 0 comments) 
  5. The Measure of Intelligence (1 point, 0 comments)