The reported $100 billion revenue threshold we talked about earlier conflates industrial success with cognitive functionality, as if a system’s skill to generate income says something significant about whether or not it may possibly “suppose,” “cause,” or “perceive” the world like a human.
Relying in your definition, we could have already got AGI, or it might be bodily inconceivable to realize. Should you outline AGI as “AI that performs higher than most people at most duties,” then present language fashions doubtlessly meet that bar for sure forms of work (which duties, which people, what’s “higher”?), however settlement on whether or not that’s true is way from common. This says nothing of the even murkier idea of “superintelligence”—one other nebulous time period for a hypothetical, god-like mind thus far past human cognition that, like AGI, it defies any stable definition or benchmark.
Given this definitional chaos, researchers have tried to create goal benchmarks to measure progress towards AGI, however these makes an attempt have revealed their very own set of issues.
Why benchmarks preserve failing us
The seek for higher AGI benchmarks has produced some fascinating options to the Turing Take a look at. The Abstraction and Reasoning Corpus (ARC-AGI), launched in 2019 by François Chollet, checks whether or not AI programs can remedy novel visible puzzles that require deep and novel analytical reasoning.
“Virtually all present AI benchmarks might be solved purely through memorization,” Chollet instructed Freethink in August 2024. A serious drawback with AI benchmarks presently stems from information contamination—when take a look at questions find yourself in coaching information, fashions can seem to carry out nicely with out actually “understanding” the underlying ideas. Massive language fashions function grasp imitators, mimicking patterns present in coaching information, however not all the time originating novel options to issues.
However even subtle benchmarks like ARC-AGI face a elementary drawback: They’re nonetheless attempting to cut back intelligence to a rating. And whereas improved benchmarks are important for measuring empirical progress in a scientific framework, intelligence is not a single factor you possibly can measure, like peak or weight—it is a complicated constellation of skills that manifest in a different way in several contexts. Certainly, we do not even have a whole useful definition of human intelligence, so defining synthetic intelligence by any single benchmark rating is prone to seize solely a small a part of the entire image.
