Google just gave us a tantalizing glimpse into the future of AI agents

This is something Google CEO Sundar Pichai is thinking about a lot right now.

“I think of AI agents as intelligent systems that demonstrate planning reasoning and memory,” Pichai said during a roundtable with journalists ahead of the developer show this week.

“They are able to think ahead multiple steps and work on software and systems to accomplish something on your behalf and, more importantly, under your supervision.”

Bottom line: AI agents have the best chance of moving this technology from “nice to have” to “necessary.”

Astra Project

Pichai said Google is in the “very early days” of developing this system, but promised we’ll see a glimpse of the “agentic direction” of Google products at IO. And there have been several, but the most important one that we will talk about is the Astra project.

Astra is a vision of what Google Assistant should have been all along. You can also think of it as a smarter version of Google Lens, which uses real-time computer vision capabilities to let you ask it questions about what you can see and hear around you.

“We always wanted to create a universal agent that would be useful in everyday life,” said Demis Hassabis, director of Google DeepMind. “Imagine agents that can better see and hear what we do, understand the context we are in, and respond quickly to conversations, making the pace and quality of interaction much more natural.”

Google showed a demo of someone holding their phone with the camera on and asking the AI ​​voice assistant questions about what they were seeing. For example, they pointed to him through the window and asked: “Which neighborhood do you think I am in?” Fittingly, this was Google’s office in King’s Cross, London.

Hassabis stressed that this demonstration video was recorded “in real time.” Google faced widespread criticism in December after a demo of the Gemini AI model was altered. So Google had to emphasize that it really can do this, especially after OpenAI showed off a similar demo on Monday.

While there really isn’t any manipulation here, Astra is certainly impressive and will probably be the big takeaway from the show. But there are other ways for these AI agents to emerge in the shorter term.

Google has revealed a combination of updates that will soon make its Gemini AI chatbot more powerful and proactive. Some of this is unlocked as Google continues to increase popup, which is the amount of information a large language model can ingest at once.

Let’s say you want to know something buried deep in a series of very long documents that you don’t want to spend hours going through. With a large pop-up window, you can share all documents with Gemini and then ask questions. The model can respond quickly, based on all the information it has just ingested.

Tentacles and sewing

But it’s in its existing products that Google really has an advantage when it comes to enabling agent-like qualities.

With its tentacles already present in many aspects of our lives, from email to search to maps, Google can synthesize all of its knowledge about users and the world around them to not only answer queries, but also adapt the answers.

“My Gemini should really be different from your Gemini,” said Sissie Hsiao, head of Gemini and Google Assistant.

Later this year, Gemini will be able to plan your vacation with a much finer level of detail, according to Hsiao. The idea is that you’ll be able to plug in all your specific requests (you like hiking, you hate it when it’s too hot, and you’re allergic to shellfish) and Gemini will send you back a detailed itinerary. Chatbots can already do this sort of thing, but, if allowed, Gemini will have access to your flight information, travel confirmations in Gmail and perhaps your hotel, and can use that to inform its responses.

“An AI assistant should be able to solve complex problems, take actions for you, and feel very natural and fluid when you use it,” Hsiao said.

Much of what agents will do will be cutting out steps and shortening tasks. Google is also thinking about it when it comes to search, as the company evolves its most valuable product to spit out answers obtained using generative AI.

It deploys a custom version of Gemini designed specifically for search, combining its knowledge of the web with the AI ​​model’s multimodal capabilities and its giant pop-up window.

Liz Reid, head of search at Google, showed an example of using Google Lens to take video from a record player. The user asks Gemini why the arm won’t stay in place. Google responds with exact instructions for that specific turntable.

As always, Google has some confusing branding issues to resolve: Lens, Assistant, Gemini, Astra. Ultimately, though, a lot of this ends up coalescing. On Tuesday, Google gave us some clues about what this logical conclusion will look like.

Or, as Reid puts it: “It’s a way for Google to do the searching for you.” »