What Gemini and Google AI features are we expecting?

Over the past year, Google has previewed a number of Gemini-branded and other AI features in its consumer apps. Here’s everything that was announced and when they might be available.

Pixels

At the end of the Made by Google 2023 conference, a Zoom Enhance feature that “intelligently fills gaps between pixels and predicts the finest details” was announced for the Pixel 8 Pro. Relying on a “custom AI-driven generative image model” built into the device, Google touted the feature as being useful when you forget to zoom.

This is an incredible application of generative AI, which opens up a whole new world of possibilities for framing and editing your images. So the kind of zoom enhancement you’re used to seeing in science fiction is right there on the phone you’re holding in your hands.

In October, Google said it would “come later.” After three Pixel Feature Drops, it hasn’t arrived yet. It’s unclear if the model Google is referring to is Gemini Nano with multimodality. At this point, it might as well debut with the Pixel 9 Pro as that phone’s flagship photo feature.

Google Home

In the Google Home app, generative AI will be used to summarize events into a “simplified view of what happened recently.” This “quick and easy summary” will use bullet points, while you’ll also be able to ask for information about your home conversationally to find historical video clips and get automations. The “experimental features” will be available to Nest Aware subscribers in 2024.

Fitbit

Fitbit Labs will allow Fitbit Premium users to test and provide feedback on experimental AI capabilities.

One such feature is a chatbot that lets you ask questions about your Fitbit data in a natural, conversational way. This “personalized coaching” that takes fitness goals into account aims to generate “actionable messages and advice,” with responses that can include personalized graphics.

• “For example, you can dig deeper into how many Active Zone Minutes (AZM) you have and how that correlates with how restorative your sleep is.”
• “…this model may be able to analyze variations in your sleep patterns and sleep quality, and then suggest recommendations for how you might modify the intensity of your training based on that information.”

Behind the scenes, this app is powered by a new personal health master’s program from Fitbit and Google Research based on Gemini. Since March, it has been available “later this year” to a “limited number of Android users enrolled in the Fitbit Labs program in the Fitbit mobile app.”

Google Photos

Ask Photos will let you ask questions about images and videos in your library. Beyond image search, it can pull information and give you a text answer. Powered by Gemini, example queries include “Show me the best photo of every national park I’ve visited” and “What themes did we choose for Lena’s birthday party?” It can be used to “suggest the best photos” and create captions for them. Ask Photos is an “experimental feature” that will be rolling out soon, with Google already announcing more features in the future.

Gmail + Google Workspace

In Gmail for Android and iOS, you’ll find a Gemini button in the top-right corner that lets you bring up the mobile equivalent of a sidebar for entering full prompts. Gmail is also getting contextual smart replies that offer more personalized, detailed, and nuanced suggestions. This will roll out to Workspace Labs in July.

At the Cloud Next 2024 conference in April, Google also previewed a voice prompt feature for Help me write in Gmail mobile. Meanwhile, an “instant polish” feature will “convert raw notes into a full email with just one click.”

On desktop web, the side panel is available in Gmail, Google Drive, and Docs/Sheets/Slide. Gemini will then be available on Google Chat to summarize conversations and answer questions.

Google Maps

Last February, Google announced that Maps would use LLMs to power an “Ask about” chatbot. You can use it to find places matching your query with support for follow-up questions. It draws on information on 250 million places and user-submitted photos, videos and reviews.

Chromium

Gemini Nano comes to Chrome for desktop and lets you use browser features like Help Me Write. It should be available on most modern laptops and desktops.

Google Search

In addition to launching AI Previews, Google previewed a number of upcoming features that will first arrive in Search Labs:

• You will be able to take the original AI overview and make it “simpler” (in just a few sentences) or “break it down” (longer answer).
• Multi-step reasoning skills will allow you to ask a complex question in one go rather than breaking it into multiple queries.
• Meal and Travel Planning
• AI-curated search results page
• Video Searches: Record a video and ask a question about it

Android

Gemini Nano with multimodality will launch on Pixel “later this year” and will offer features like on-device/offline TalkBack descriptions and real-time scam alerts that listen to a call for telltale patterns. Google will share more details later this year.

At I/O 2024, Google also showcased how Gemini on Android will soon be an overlay panel instead of opening a full-screen UI to view results. In addition to preserving context, this will allow you to drag and drop a generated image into a conversation. For Gemini Advanced subscribers, the “Request this video” and “Request this PDF” buttons will see Gemini videos and summary documents, respectively. This will take place “over the coming months”. Additionally, dynamic suggestions will use Gemini Nano with multimodality to understand what’s on your screen:

For example, if you enable Gemini in a conversation about pickleball, suggestions might include “Find pickleball clubs near me” and “Rules of pickleball for beginners.”

Another addition that will be particularly useful on mobile are Gemini extensions for Google Calendar, Tasks, and Keep. These will let you take a photo of a page with several upcoming dates that Gemini can turn into calendar events. In the coming months, a “Utilities” feature will allow Gemini mobiles to access Android’s Clock app.

We also expect the Gemini mobile to arrive on the Pixel tablet this summer.

Gemini

Live will let you have a two-way conversation with Gemini. To make the experience more natural, Gemini will return concise responses that you can interrupt to add new information or ask for clarification. You can choose from 10 different voices, with Google imagining Gemini Live as useful for preparing for an interview or rehearsing a speech. It will be available in the “coming months” for Gemini Advanced members.

“Later this year,” Gemini Live will let you launch a live camera mode. You just point to something in the real world and ask a question about it. This feature is powered by Project Astra.

Gems are personalized versions of Gemini that allow you to have a “gym partner, sous chef, coding partner, or creative writing guide.” Gemini Advanced members will be able to create custom Gems, while all users will have access to pre-made Gems, like Learning Coach.

Simply describe what you want your Gem to do and how you want it to react, for example “you are my running coach, give me a daily running plan and be positive, upbeat and motivating”. Gemini will take these instructions and, with one click, enhance them to create a Gem that meets your specific needs.

Gemini Advanced users will also get an “immersive planner” that goes beyond simply suggesting activities, taking into account travel times and stops, as well as people’s interests, to create a detailed itinerary. Gemini will use flight/trip details in Gmail, Google Maps recommendations for restaurants and museums near your hotel, and search for other activities.