For years, the AI industry has respected a set of principles called “Land laws”. Openai researchers described them in the founding article of 2020, “failure of laws for neural language models”.
“The model performance depends most strongly on the scale, which consists of three factors: the number of parameters of the N model (excluding interest), the size of the D data D and the quantity of calculation C used for training,” wrote the authors.
Essentially, the more it is more when it comes to building a very intelligent AI. This idea has fueled enormous investments in data centers that allow AI models to process and learn huge amounts of existing information.
But recently, Silicon Valley AI experts began to challenge this doctrine.
“The most interesting problems are evolving extremely badly,” said Meta’s National University of Singapore on Sunday at the National University of Singapore. “You cannot just assume that more data and more calculation means a smarter AI.”
The point of Lecun depends on the idea that the formation of AI on large quantities of basic subjects, such as Internet data, will not lead to a kind of superintendent. Intelligent AI is a different breed.
“The error is that very simple systems, when working for simple problems, people extrapolate them to think that they will work for complex problems,” he said. “They do incredible things, but it creates a religion of scaling that you just need to further extend systems and they will naturally become smarter.”
Currently, the impact of scaling is amplified because many of the latest breakthroughs in AI are actually “really easy,” said Lecun. The largest models of languages are now trained in the amount of information in the visual cortex of a four-year-old child, he said.
“When you deal with real problems with ambiguity and uncertainty, it is no longer just a question of setting up,” he added.
The progress of AI has slowed down lately. This is due in part to a corpus declining usable public data.
Lecun is not the only eminent researcher to question the power of the scale. AI CEO Alexandr Wang said that the industry’s greatest question in the cerebral valley is “the largest question in the brain conference. The CEO of Cohere, Aidan Gomez, qualified it in a “most stupid” way to improve the models of AI.
Lecun pleads for a training approach more based in the world.
“We need AI systems that can learn new tasks very quickly. They must understand the physical world – not just text and language, but the real world – have a certain level of common sense and capacities to reason and plan, have a persistent memory – everything we expect from intelligent entities,” he said during his speech on Sunday.
Last year, in an episode of Lex Fridman’s podcast, Lecun said that unlike important languages, which can only predict their next steps according to models, the world models have a higher level of cognition. “The additional component of a world model is something that can predict how the world will evolve due to an action you could take.”
businessinsider
