Google I/O: hands-on with the Astra project

As per usual, Google I/O 2024 is an absolute whirlwind of news and announcements. This year, rather than focusing on hardware, Android or Chrome, Google spent most of the developer conference convincing us that its AI Features deserve to be prioritized. One such project is Project Astra, a multimodal AI assistant that you can semi-converse with and that can simultaneously use the camera to identify objects and people.

I say “semi” because it’s obvious after the demo that this part of Gemini is in its infancy. I spent a few brief minutes with Project Astra on the Pixel 8 Pro to see how it works in real time. I didn’t have enough time to test it in its entirety or try to trick it, but I got an idea of ​​what the future might look like as an Android user.

Ask him almost anything

The goal of Project Astra is to be like an assistant that also guides you in the real world. It can answer questions about the environment around you by identifying objects, faces, moods and textiles. It can even help you remember where you last placed something.

There were four different demos to choose from for Project Astra. They included Storyteller mode, which tasks Gemini with concocting a story based on various inputs, and Pictionary, essentially a guessing game with the computer. There was also an alliteration mode, in which the AI ​​showed off its prowess for finding words with the same starting letter, and Free-Form let you chat back and forth.

The demo I received was a version of Free-Form on the Pixel 8 Pro. Another journalist in my group had asked for it upfront, so most of our demonstration focused on using the device and this assistant-like mode together.

With the camera pointed at another reporter, the Pixel 8 Pro, Gemini was able to identify that the subject was a person – we explicitly told it that the person identified as a man. Then it was correctly identified that he was carrying his phone. In a follow-up question, our group asked about his clothes. He gave a generalized response that “he appears to be wearing casual clothes.” Then we asked him what he was doing, to which Project Astra replied that it looked like he was putting on a pair of sunglasses (he was) and striking a casual pose.

I grabbed the Pixel 8 Pro for a quick minute. I asked Gemini to correctly identify a pot of fake flowers. They were tulips. Gemini noticed that they were also colorful. From there, I didn’t really know what else to do, and my time was up. I left with more questions than when I arrived.

With Google’s AI, it feels like a leap of faith. I can see how identifying a person and their actions could be an accessibility tool to help a blind or visually impaired person navigate the world around them. But that is not the aim of this demonstration. This was to present the capabilities of the Astra project and how we will interact with it.

My biggest question is: will something like Project Astra replace Google Assistant on Android devices? After all, this AI can remember where you place your stuff and pick up on nuances – at least that’s what the demo conveyed. I couldn’t get an answer from the few people at Google I asked. But I have a strong feeling that the future of Android will be less about tapping to interact with the phone and more about talking to it.