Since its original launch at Google I / O 2024, Project Astra has become a test field for the ambitions of the AI assistant from Google. The multimodal bot and which sees everything is a consumer product, and it will not be available for anyone outside a small group of testers. What Astra represents instead is a collection of the biggest, wildest and most ambitious dreams of Google on what AI could do for people in the future. Greg Wayne, research director of Google Deepmind, says that he considers Astra as “the kind of concept car of a universal AI assistant”.
Finally, the things that operate in Astra are shipped to Gemini and other applications. This already included some of the team’s work on the vocal output, memory and certain features for using the basic computer. As these features become general public, the Astra team finds something new to work.
This year, at his conference on I / O developers, Google announced new Astra features which report how the company came to see its assistant – and how intelligent the assistant can be. In addition to answering questions and using your phone’s camera to remind you where you have left your glasses, Astra can now perform tasks on your behalf. And he can do it without even asking.
Astra’s new most impressive feature is its new proactivity. “Astra can choose when to speak according to the events he sees,” explains Wayne. “It is actually, in a continuous sense, observation, then he can comment.” This is a big change: instead of pointing your phone on something and asking your AI assistant on this subject, Astra’s plan is to have this assistant to watch, to listen and to wait for his moment.
Astra’s plan is to have his assistant to watch, listen and wait for his moment to intervene
If Astra looks while you do your homework, Wayne offers for example, he could notice that you made a mistake and you report where you were wrong, rather than waiting for you to finish and specifically ask the bot to check your work. If you are on an intermittent fast, Astra could remember to eat just before your designated time is elapsed – or you ask slowly if you should really eat now, given your diet.
The teaching of Astra to act on his own will has been part of the plan from the start, explains Demis Hassabis, CEO of Deepmind. He calls it “read the room” and says that, as difficult as you think it is to learn to make a computer, it is actually much more difficult than that. Knowing when to make bursting, what tone to take, how to help and when to be silent, is something that humans do relatively well, but it is difficult to quantify or study. What if the product does not work well and starts to kill without being not having come and unwanted? “Well, no one would use it if it did,” said Hassabis. These are the issues.
A really great and proactive assistant is still far, but something of which will certainly need a huge amount of information about you. This is another new thing that happens to Astra: the assistant can now access information on the web and from other Google products. He can see what’s on your calendar, to tell you when to leave; He can see what’s in your email to determine your confirmation number while you are heading for reception to register. At least it is the idea. Having it operated at all – then consistently and reliably – will take some time.
The last piece of the puzzle, however, really gets together: Astra learns to use your Android phone. BIBO XIU, product manager of the Deepmind team, showed me a demo in which she pointed her phone camera on a pair of Sony headphones, and asked which ones they were. Astra said it was the WH-1000XM4 or the WH-1000XM3 (and honestly, how could we expect someone or anything could make the difference), and Xiu asked Astra to find the manual, then explain how to associate them with his phone. After Astra explained, Xiu interrupted: “Can you go ahead and open parameters and twin the headphones for me, please?” All alone, Astra did exactly that.
The process was not perfectly transparent – Xiu had to manually activate a feature that allowed Astra to see the screen of his phone. The team is still working on the realization of this automatically, it says: “But that’s the goal, that it can understand what it can and cannot see for the moment.” This type of use of the automated device is the same thing as Apple works with its new generation Siri, and the two companies imagine an assistant who can navigate applications, adjust the settings, respond to messages and even play games without you needing to touch the screen. It is an incredibly difficult thing to build, of course: Xiu’s demo was impressive and was almost a task as simple as you can imagine. But Astra has made progress.
Currently, most of the so-called “agentics” are not working very well, or not at all. Even in the best of cases, it always forces you to make a lot of lifting: you must encourage the system to each turn, provide the entire context and the additional information that the application needs and make sure everything is going well. Google’s goal is to start deleting all this operation, step by step. He wants astra to know when he is necessary, know what to do, know how to do it and know where to find what he needs to do it. Each part of this will require technological breakthroughs, most of which have not yet done. Then there will be complicated user interface problems, privacy issues and more problems.
If Google or someone will build a really universal AI assistant, however, they will have to do these things correctly. “This is another level of intelligence necessary to be able to achieve it,” explains Hassabis. “But if you can, it will categorically feel different from today’s systems. I think that a universal assistant must have it to be really useful.”
