Google Photos introduces AI search feature, “Ask Photos”

Google Photos is getting an AI infusion with the launch of an experimental feature, Ask Photos, powered by Google’s Gemini AI model. The new addition, rolling out later this summer, will allow users to search their Google Photos collection using natural language queries that leverage AI’s understanding of their photo’s content and other metadata.

While before users could search for specific people, places or objects in their photos, thanks to natural language processing, the AI ​​upgrade will make finding the right content more intuitive and less of a process. manual search, Google announced Tuesday during its annual Google I. /O Developer Conference 2024.

For example, instead of searching for something specific in your photos, like “Eiffel Tower,” you can now ask the AI ​​to do something much more complex, like finding “the best photo of each of the national parks that I visited.” The AI ​​uses a variety of signals to determine what makes the photo the “best” in a given set, including things like lighting, blur, lack of background distortion, etc. It can then combine this with its understanding of the geolocation of a set of photos or dates to retrieve only images taken in US national parks.

This feature builds on the recent launch of Photo Stacks in Google Photos, which aggregates near-duplicate photos and uses AI to highlight the best photos in the group. As with Photo Stacks, the goal is to help people find the photos they want as their digital collections grow. More than 6 billion images are uploaded to Google Photos daily, according to Google, to give you an idea of ​​the scale.

Additionally, the “Request Photos” feature will allow users to ask questions to get other types of helpful answers. In addition to asking for the best photos from a vacation or another group, users can ask questions that require an almost human understanding of what’s in their photos.

For example, a parent could ask Google Photos what themes they used for their child’s last four birthday parties, and it could return a simple answer accompanied by photos and videos on the themes of mermaid, princess, and unicorn previously used and when.

This type of query is made possible because Google Photos not only understands the keywords you enter, but also natural language concepts, like “themed birthday party.” It can also leverage AI’s multimodal capabilities to understand whether text in a photo might be relevant to the query.

Another example shown to the press by CEO Sundar Pichai ahead of today’s Google I/O developer conference shows a user asking the AI ​​to show them their child’s swimming progress. The AI ​​aggregated highlights from photos and videos of the child swimming over time.

Another new feature leverages the use of search to find answers from photo text. This way, you can take a photo of something you’d like to remember later, like your license plate or passport number, and then have the AI ​​retrieve that information when you need it.

If the AI ​​ever makes a mistake and you correct it – perhaps by flagging a photo that isn’t from a birthday party or a photo you wouldn’t feature on vacation – it will remember that response to improve over time. This also means that the AI ​​becomes more personalized the more you interact with it.

When you find photos you’re ready to share, AI can help you write a caption that summarizes the content of the photos. For now, this is a basic summary, but does not offer the possibility of choosing between different styles. (But given that it uses Gemini under the hood, a cleverly written prompt might work to return a certain style if you try it.)

Google says it will have safeguards in place to not respond in certain cases (perhaps not asking the AI ​​for the “best nudes”?). It also did not include potentially offensive content when training the model. But the feature is being launched as an experiment, so it may require additional controls to be added over time as Google responds to how people use it.

The Ask Photos feature will initially be supported in the United States in English before rolling out to more markets. For now it will only be a text-based feature, similar to asking questions of an AI chatbot. Over time, however, it could integrate more deeply with Gemini running on the device, like on Android.

The company claims that users’ personal data in Google Photos is not used for advertising purposes. Humans also won’t review AI conversations and personal data in Ask Photos, except “in rare cases to address abuse or harm,” Google says. People’s personal data in Google Photos is also not used to train other generative AI products, like Gemini.