The hidden cost of Google's AI defaults and the illusion of choice

So Google doesn’t scan your inbox or documents to train Gemini. Great. But Gemini can use tools to connect to Workspace and other Google products when appropriate, based on your prompt. Google says its AI models can be trained on Gemini inputs and outputs. And guess what those outputs might include—yes, your data. Gemini outputs can include summaries and snippets of email or files, and that data can then become fodder for AI training. Google says the goal is to train Gemini to be a better assistant, and the way people interact with the bot is a key element, but it tries to “filter and reduce” personal information going into AI training datasets. There’s no way for us to know how well this automated process works, though. Google is quick to point out that users have control over these features. If you want to keep your private data truly private, you can avoid ever letting Gemini interact with your files or opt out of sharing any data for AI training. Unfortunately, Google doesn’t make it easy to leave Gemini behind. Companies generally won’t admit to designing an interface to manipulate users, but the intent doesn’t determine whether a UI design is a dark pattern. “It doesn’t matter whether it’s intentional or not,” said Marie Potel of Fair Patterns, a startup building AI models that detect dark patterns and predatory design. “What matters is whether the autonomy—the agency—of users is respected and whether the design goes against what users want to do.” If the only way to opt out of AI training is to permanently disable your chat history, that arrangement doesn’t seem to respect the user’s agency—it’s a forced action. Even finding the right menu to opt out of training can be a chore. There’s a link hiding in the Gemini app settings, but it’s labeled only as “Activity,” and there are also direct links if you search through Google’s support articles. Interestingly, the Gemini controls are absent from Google’s account privacy settings, where you’d expect to find them. A company representative said there should be a link in the Activity Controls, along with submenus for Android, Maps, Search, Assistant, and more. After checking multiple accounts, we have yet to see a link to the Gemini privacy menu. In Gmail, Gemini can craft and tweak emails, summarize email chains, organize your inbox, and create AI Overviews of your email. The amount of AI you get depends on whether you’re paying for higher AI limits, but everyone gets some, and these features are constantly expanding. But maybe you don’t want the AI generating hallucinatory summaries in your email. Turning that off is an exercise in frustration because there is no granularity—plenty of features have simple toggles in the Gmail settings but not Gemini. Default settings are powerful even without predatory design, but the way defaults are managed can easily veer into a dark place. According to Dr. Harry Brignull, who coined the term “dark pattern” in 2010, companies have long relied on defaults to bring users along for the ride. Speaking about the tech industry generally, Brignull said that companies are acutely aware of this effect and the advantages it can bestow. “We could talk about this as the pre-selection dark pattern,” he said. “If you want to get people to opt in to something, instead of asking them in a sign-up step explicitly with a link to learn more, you can take that and make it pre-selected. Put that in the settings three to four clicks deep, and that way you may feel you have the means to argue that users are given a choice when, in fact, you know from your own stats that by designing it this way, very few people are even getting to that page.” Google has billions of users hooked on its products, and that gives it a lot of power to get new features in front of users, since people rarely change the defaults. Google has paid handsomely to set defaults on devices like the iPhone for this reason. The issue came up repeatedly in Google’s recent antitrust cases, and government lawyers used it to paint the company as anticompetitive. What we’re seeing in both free and paid Google accounts is the power of defaults in the AI era. The default is sharing data for AI training. The default is AI summaries in your email. The default is AI-powered document creation. You can change these settings, but Google has to know most people won’t do that, because the options are hard to find and don’t work as they should.