In February, Demis Hassabis, CEO of Google’s DeepMind AI research lab, warned that devoting increasing amounts of computation to the types of AI algorithms widely used today could lead to diminishing returns. Moving to the “next level” of AI, so to speak, Hassabis said, will instead require breakthroughs in basic research that offer viable alternatives to today’s entrenched approaches.
Former Tesla engineer George Morgan agrees. So he founded a startup, Symbolica AI, for this specific purpose.
“Traditional deep learning and generative language models require unimaginable scale, time and energy to produce useful results,” Morgan told TechCrunch. “By creating (new) models, Symbolica can achieve greater accuracy with reduced data requirements, reduced training time, lower cost and with structured outputs that are provably correct. »
Morgan dropped out of Rochester to join Tesla, where he worked on the team developing Autopilot, Tesla’s suite of advanced driver assistance features.
At Tesla, Morgan says he realized that current AI methods — most of which revolved around increasing computation — would not be sustainable in the long term.
“Current methods only have one knob to turn: increase the scale and hope for emergent behavior,” Morgan said. “However, scaling requires more compute, more memory, more money for training, and more data. But ultimately this doesn’t get you significantly better performance.
Morgan is not the only one to reach this conclusion.
In a memo this year, two executives from TSMC, the semiconductor maker, said that if the AI trend continues at its current pace, the industry will need a chip with 1 trillion transistors , a chip containing 10 times more transistors than average. chip today – within a decade.
It is unclear whether this is technologically feasible.
Elsewhere, an (unpublished) report co-authored by Stanford and Epoch AI, an independent AI research institute, reveals that the cost of training cutting-edge AI models has increased significantly over the year elapsed and has changed. The report’s authors estimate that OpenAI and Google spent approximately $78 million and $191 million, respectively, to train GPT-4 and Gemini Ultra.
With costs poised to rise further — see OpenAI and Microsoft’s announced plans for a $100 billion AI data center — Morgan has begun studying what he calls AI models “structured”. These structured models encode the underlying structure of the data – hence their name – instead of trying to approximate information from huge datasets, like conventional models, allowing them to achieve what Morgan calls better performance using less overall computation.
“It is possible to produce domain-specific structured reasoning capabilities in much smaller models,” he said, “by combining a deep mathematical toolkit with advances in deep learning.”
Symbolic AI is not exactly a new concept. They date back decades and are rooted in the idea that AI can be built on symbols that represent knowledge using a set of rules.
Traditional symbolic AI solves tasks by defining sets of symbol manipulation rules dedicated to particular tasks, such as editing lines of text in word processing software. This is in contrast to neural networks, which attempt to solve tasks through statistical approximation and learning from examples. Symbolica aims to take advantage of the best of both worlds.
Neural networks are the cornerstone of powerful AI systems such as OpenAI’s DALL-E 3 and GPT-4. But, Morgan says, scale is not the be-all and end-all; Combining mathematical abstractions with neural networks might actually be better positioned to efficiently encode knowledge of the world, reason through complex scenarios, and “explain” how they arrive at an answer, Morgan claims.
“Our models are more reliable, more transparent and more accountable,” Morgan said. “There are immense commercial applications of structured reasoning capabilities, particularly for code generation – that is, reasoning across large code bases and generating useful code – where existing offerings fall short. “
Symbolica’s product, designed by its team of 16, is a toolkit for creating symbolic AI models and pre-trained models for specific tasks, including code generation and proving mathematical theorems. The exact business model is evolving. But Symbolica could provide consulting and support services to companies that want to create custom designs using its technologies, Morgan said.
“The company will work closely with large enterprise partners and customers, creating custom structured models with significantly enhanced reasoning capabilities, tailored to individual customer needs,” Morgan said. “They will also develop and sell cutting-edge code summarization models to large enterprises.”
Today marks Symbolica’s stealth launch, so the company has no customers — at least none it’s willing to talk about publicly. Morgan did, however, reveal that Symbolica landed a $33 million investment earlier this year, led by Khosla Ventures. Other investors included Abstract Ventures, Buckley Ventures, Day One Ventures and General Catalyst.
Indeed, $33 million is not a small number; Symbolica’s backers obviously have confidence in the startup’s science and roadmap. Vinod Khosla, the founder of Khosla Ventures, told me via email that he believes Symbolica “tackles some of the most important challenges facing the AI industry today.”
“To enable large-scale commercial adoption of AI and regulatory compliance, we need models with structured results that can achieve greater accuracy with fewer resources,” Khosla said. “George has assembled one of the best teams in the industry to make this happen.”
But others are less convinced that symbolic AI is the right path forward.
Os Keyes, a doctoral student at the University of Washington specializing in data law and ethics, notes that symbolic AI models depend on highly structured data, making them both “extremely fragile” and context-dependent and specificity. In other words, symbolic AI needs well-defined knowledge to work – and defining that knowledge can be a lot of work.
“It could still be interesting if it combines the benefits of deep learning and symbolic approaches,” Keyes said, referring to DeepMind’s recently released AlphaGeometry, which combined neural networks with an AI-inspired symbolic algorithm. to solve difficult geometry problems. “But time will tell.”
Morgan countered by saying that current training methods will soon no longer be able to meet the needs of companies that want to harness AI for their purposes, making any promising alternative worth investing in. And, he said, Symbolica is strategically well positioned to do just that. the future, given that it has “several years” of runway with its latest round of funding and its models are relatively small (and therefore cheap) to train and operate.
“Tasks such as automating software development, for example, at scale will require models with formal reasoning capabilities and lower operating costs, to analyze large databases of code and produce and iterate useful code,” he said. “The public perception of AI models remains largely that of ‘scale is all you need.’ Thinking symbolically is absolutely necessary to progress in the field: structured and explainable results with formal reasoning capabilities will be necessary to meet demands.
There’s nothing stopping a large AI lab like DeepMind from building its own symbolic AI or hybrid models and, aside from Symbolica’s points of differentiation, Symbolica is entering an extremely crowded and well-capitalized field of AI. But Morgan still anticipates growth and expects San Francisco-based Symbolica’s workforce to double by 2025.
techcrunch
