Google is said to be rolling out new Gemini AI assistant features on Android smartphones that enable users to ask the AI chatbot questions related to what’s on their phone’s display. Moreover, Gemini has the ability to offer YouTube video summaries and respond to queries to specific YouTube videos.
These features have been available on the latest Pixel 9 series smartphones; however, it looks like they are rolling out to other Android smartphones as well. Once the feature is live on your device, open the floating window for the Gemini AI assistant by long pressing the power button.
Users will see a new “Ask about this screen” floating button above the Gemini window. Tapping on the button takes a screenshot of what is on the screen and shares it with Gemini AI.
Also Read: Airtel Wynk Shutting Down; To Offer Apple Music On Discount
How Does The New Gemini AI Feature Work
Once a user takes a screenshot, they can ask Google's Gemini questions about objects on the screen. However, it should be noted that since Gemini uses the screenshot to gather display information, it won’t be able to answer questions about the entire page.
For instance, if users open a news article in a browser and ask the AI chatbot about what is on the display. It will only have access to the information captured in the screenshot, not the entire web page. The feature also does not allow users to capture an extended screenshot.
Similar to the “Ask about this screen” option, Gemini has an “Ask about this video” option playing select YouTube videos. The option shows up on the YouTube videos that have captions, although it works even if captions are not enabled for the video.
Once users select the “Ask about this video” option, Gemini AI can summarise the video for them or answer specific questions about its content. Since the feature works with captioned videos, the visuals in the video are not analyzed.
Also Read: YouTube Boosts Premium Service Subscription Prices In India
Gemini Live To Answer Queries About Things In-Camera Viewfinder
Google gave everyone a sneak peek into Project Astra at I/O this May with demos of an AI chatbot answering questions about things in the camera viewfinder. At the time, the company said that some of its features would make it to the Gemini app later this year, and it has now launched a few more details.
Towards the end of the company’s Made by Google event, Google stated that it would soon integrate Project Astra into Gemini Live -- the AI chatbot’s new conversational mode. The integration will enable users to have a conversation with Gemini about things in their camera viewfinder and get answers in real-time.