I Tried Pixel 11's New Gemini Features. One Made Me Sound Fluent in Spanish - CNET

Exclusive: I stopped by Google's HQ to demo the Pixel 11's new AI features and hear from the people who brought them to life.

In real life, my Spanish is a little rusty. I can understand most conversations, but speaking is another story, as if the words and verb tenses are sitting on some dusty shelf I haven’t touched since high school. Right now, though, I’m hearing myself speak Spanish so rapidly and smoothly that I sound fluent.

Except I’m not. It’s Google’s Live Translate feature, dubbing one of my YouTube reviews in real time.

With the brand-new Pixel 11 Pro XL in my hand, I can tap on virtually any YouTube video and hear it translated on the fly, opening up a world of foreign-language content that I can have dubbed into English – or Spanish or Arabic, if I really want to put my language skills to the test.

Live Translate for media is just one of a handful of new Gemini Intelligence features launching on Google’s new Pixel 11 series, announced Wednesday. Along with instant translations, this suite of AI capabilities is designed to handle mundane tasks like adding something to your calendar, automatically filling out a form and sharing your flight details with loved ones. That way, you don’t have to hop between apps to track down that information yourself.

Some Gemini Intelligence features launched on Samsung’s latest foldable phones, the Galaxy Z Fold 8, Z Fold 8 Ultra and Z Flip 8. But several new features I had the opportunity to try out last week at Google’s Mountain View, California, headquarters are arriving on Pixel 11 phones first, and will later become available on additional premium Android phones.

“We want to make sure that if you’re getting a great Android device, you should get all the great AI experiences that we’re building as well. It’s not a Pixel-only thing,” Seang Chau, vice president and general manager of Android Platform, told me. “Gemini Intelligence then becomes a brand statement that’s essentially: ‘What you have in your hand is the best that we have to offer.'”

It wouldn’t be a Pixel launch without some AI camera updates. A new feature called Magic Capture lives within the camera app, next to the photo and video controls. When tapped, Magic Capture will analyze the scene and then automatically find the best shots, so you don’t have to tap the shutter button for each individual snapshot. Just hold your phone up, and your Pixel will record the scene and then pinpoint the best moments for a photo.

I tested this out while kicking a soccer ball back and forth. I pointed the Pixel’s camera at my passing partner and could focus more on playing rather than checking my framing or taking photos at the right moment. Magic Capture ended up choosing some pretty epic shots of him doing tricks, without me looking out for when to hit the shutter button.

This can be helpful if you’re trying to decide between taking a video or snapping photos, or if you just prefer to be in the moment as much as possible, rather than seeing the world through your smartphone’s viewfinder.

Rambler takes voice-to-text to the next level by removing filler words like “um” and “uh.” You can also reverse course by saying something like, “actually, never mind,” and I’ll only take into account what it thinks you want to say.

I tested out Rambler by arranging dinner plans and stumbling through whether I wanted sushi or pizza, and if I wanted to meet at 5 or 5:30. The final text was much more cohesive than what actually came out of my mouth: “Hey, let’s get dinner today. Maybe we should get sushi. Should we meet up at 5:00?” I don’t typically use voice-to-text because I don’t like being called out for rambling, but it looks like Rambler is here to fix just that.

Choosing where to eat is one of my least favorite activities because, as my Rambler example demonstrates, I am wildly indecisive.

Now, with detailed visual cards through Gemini Intelligence, when someone texts you a restaurant suggestion, you’ll see a prompt above the text bar to open an AI overview right in messages. That way, you can get more information about that spot without opening a new window. You can also tap to open Google Maps to determine if it’s worth the trek.

You can also use this feature to quickly share your flight information. If someone texts, “What is your flight number to New York?” Gemini will pull those details from Gmail, so that person can track your arrival, and you don’t have to dig.

Circle to Search is one of my favorite features because it lets you quickly look up anything on your phone’s screen by just circling an image, such as an outfit or accessory. I typically use this if I want to buy a product I spot online, like when I’m scrolling through Instagram.

Now, you can use Circle to Search by just launching your Pixel 11’s camera, holding down the navigation bar and circling what’s in your viewfinder – no need to actually snap the photo. That means you can eliminate one step without cluttering your Photos app with random images you don’t actually want to save.

I tried this out by holding the Pixel 11 Pro XL’s camera up to some fresh white kicks someone was wearing, and it showed me a handful of links to buy my own pair of Nike Air Force 1s. I also used Circle to Search to identify a plant, which is apparently called a California Glory Flannel Bush. I typically use Google Lens to do that, but I suppose I now have another option for quickly searching an image.

One of AI’s most important contributions is arguably its ability to make certain features more accessible. That’s the case with Google’s new sign-language-to-text (SL2T) translation model, which allows people who are deaf to sign to their phone instead of typing. Starting with Pixel 11, people can use Gboard and Live Transcribe to translate American Sign Language to English. More devices will get this capability, and additional languages will be added soon, Google says.

People who are deaf or hard of hearing can now sign to search the web, compose messages or draft documents, or ask Gemini to execute a task using ASL. They can also use Live Transcribe to sign responses in conversations instead of typing back and forth. That can make for a faster and more natural exchange.

“Sign languages convey meaning through simultaneous movements of the hands, arms, torso, head and face,” a Google blog post reads. “Accurately tracking these at high frame rates is a difficult and computationally demanding computer vision task.”

The SL2T model is trained on over 100,000 hours of sign language data across more than 50 languages, according to Google, with around a quarter of the data in ASL. To protect people’s privacy, the company says an “on-device model tracks the location of points on the signer, and only these geometric coordinates are sent to the server for translation, allowing the original video to be discarded immediately.” That sequence is then directly translated into text.

“We spend a lot of time making sure our phones work for everybody,” said Shenaz Zack, senior director of product management at Google. “We have to solve for all our users, because this phone is not for only one user.”

Zack says practicality is at the center of all the new Gemini Intelligence features launching on the Pixel 11 series.

“How do we help our users get back to doing what they love and not worry about the cognitive load? That’s what guides us across all of these areas,” she said. “We want to make sure that we’re solving the problems that users need solving.”

Even when those problems are relatively minor, like switching the language of a YouTube video. At least now I know what to work toward as I brush up on my Spanish.