This week’s hands-on reports about Google’s Gemini AI agent, “Spark,” landed with a particular kind of unease—one that doesn’t come from the usual worries about hallucinations or unreliable answers. Instead, the discomfort is tied to something more intimate: Spark appears to be able to connect dots about people in ways that feel too personal, too fast, and too effortless.
In two separate demos described by The Verge, Spark demonstrated impressive recall and personalization during agent-style tasks. The striking detail, repeated across both hands-ons, is that Spark reportedly knew information that neither participant had clearly provided in the moment. One example cited was knowing the name of David Pierce’s dog, Frida. Another was knowing Jay Peters’ wife’s first name. These aren’t the kinds of facts you’d expect an assistant to “guess” correctly without some combination of prior context, data access, or inference that goes beyond what the user consciously entered into the conversation.
That’s where the story stops being just about capability and starts becoming about design intent. When AI systems get better, they don’t merely become more useful; they also reveal what the system is optimized to do—and what it assumes about the user, the environment, and the boundaries of consent. Spark’s performance, as described, suggests an agent that can operate with a level of contextual continuity that feels like it’s reaching past the visible prompt.
And that raises a question that matters far beyond any single product demo: are we building AI agents primarily to make individuals more productive, or are we building them to manage attention, extract context, and smooth over friction in ways that benefit the system more than the person?
The “productivity promise” has been a recurring theme in consumer tech for years. It shows up in everything from smartphones that nudge you toward faster communication to productivity apps that turn life into checklists. In that framing, efficiency becomes a moral good. If you’re behind, it’s not because the world is structured to overwhelm you—it’s because you didn’t optimize enough. AI then arrives as the ultimate optimizer: the tool that will help you do more, think faster, and keep up with everything you’re expected to handle.
But the deeper issue is that productivity is often treated as the solution to problems that aren’t fundamentally about time management. Many of the pressures people face—work overload, fragmented attention, anxiety about falling behind, the sense that life is constantly being interrupted—aren’t simply solved by doing tasks more quickly. They’re symptoms of larger systems: labor expectations, economic incentives, information overload, and the way digital platforms monetize engagement.
So when AI agents are pitched as productivity upgrades, the promise can start to feel hollow. Not because AI can’t save time, but because “saving time” can become a substitute for addressing what’s actually broken. If the world is unfair, exhausting, or misaligned with human needs, a faster workflow doesn’t fix the underlying mismatch. It just helps you adapt to it more efficiently.
Spark’s personalization, as reported, is a perfect illustration of how this shift can happen quietly. Personalization is often sold as convenience: the assistant remembers what you care about, understands your preferences, and reduces the need to repeat yourself. That can be genuinely helpful. But personalization at the level of private details—especially when those details weren’t explicitly provided in the interaction—also changes the relationship between user and system. It turns the assistant into something closer to a persistent presence than a neutral tool.
When an assistant knows your dog’s name, it can feel like magic. When it knows your spouse’s first name, it can feel like the assistant is reading your life rather than assisting with it. The difference between those feelings isn’t just emotional; it’s about agency. Users should be able to understand what the system knows, why it knows it, and how it got it. Without that clarity, personalization becomes a kind of invisible scaffolding—one that makes the assistant seem smarter while obscuring the mechanisms behind its intelligence.
There are a few plausible explanations for how an agent could know such details. It might have access to user history, account-linked data, or prior interactions. It might use contextual inference based on patterns in communications. It might also rely on training data that includes similar facts about public figures, though that seems less likely for personal household details. In practice, the most important point isn’t which explanation is correct in every case; it’s that the user experience described implies a level of continuity that goes beyond what’s typically communicated to users.
This is where the “empty promise” becomes more than a rhetorical flourish. If AI agents are increasingly capable of remembering and connecting personal context, then the next step isn’t automatically empowerment. It can also be surveillance-by-default—an environment where the assistant’s usefulness depends on collecting, inferring, and retaining more about you than you realize.
Even if the system is technically compliant with privacy policies, the lived experience can still be unsettling. People don’t evaluate privacy solely by whether a company has a checkbox somewhere. They evaluate it by whether the system behaves in ways that feel explainable and consensual. If an assistant can pull private details into a conversation without the user understanding the source, trust erodes—even when the assistant is “right.”
And trust is the currency that determines whether these agents become companions or intrusions. A productivity tool can be tolerated even when it’s annoying. An agent that feels like it’s inside your life requires a higher standard of transparency and control. Otherwise, the same features that make it effective will also make it feel invasive.
There’s another layer to this: the productivity narrative tends to treat the user as the main variable. It assumes that if the assistant is better, the user will be better. But the assistant’s improvements also change the incentives around the user. If an agent can plan trips, draft messages, summarize documents, and coordinate tasks with minimal friction, it can also encourage users to offload more decisions to the system. That offloading can be beneficial—until it becomes habitual, until the user stops asking why the assistant chose a particular plan, or until the assistant’s assumptions become the default worldview.
In other words, productivity isn’t just about doing tasks faster. It’s about shifting decision-making. And when decision-making shifts, so does accountability. If Spark recommends something based on context it pulled from elsewhere, who is responsible for the outcome? The user who clicked “go,” or the system that made the recommendation using hidden inputs?
This is why the most important question about AI agents isn’t only “Can it do the task?” It’s “What does it assume, and what does it hide?”
The Verge’s hands-on descriptions emphasize effectiveness. That’s understandable: demos are designed to show what’s possible. But the unique value of agent-style systems is that they don’t just answer questions—they act within a workflow. They can gather information, interpret preferences, and produce plans. That means their behavior is shaped by the context they receive and the context they can access. If that context includes personal details, then the agent’s competence is inseparable from its access.
This is also why the productivity promise can become morally loaded. When companies frame AI as a way to maximize output, they often imply that people who struggle are failing to keep up. The assistant becomes a tool for self-improvement under pressure. But if the real problem is that modern life is structured to overload attention and demand constant responsiveness, then the “solution” becomes a treadmill: you work faster to keep up with a system that never slows down.
AI can intensify that dynamic. An agent that makes it easier to respond instantly, schedule everything, and manage every micro-task can reduce downtime. It can also reduce the space where people reflect, rest, and set boundaries. Productivity becomes not a choice but a default setting.
So what would a non-empty promise look like? It would mean designing AI agents around human needs rather than just output metrics. That could include features like:
Clear context boundaries: The assistant should explicitly state what information it is using and where it came from. If it knows your dog’s name, it should be able to tell you whether it was provided by you, inferred from prior content, or retrieved from account-linked data.
User-controlled memory: Instead of a black-box personalization layer, users should be able to manage what the agent remembers, what it forgets, and what it is allowed to infer. Memory should be legible and reversible.
Consent that’s meaningful, not procedural: Privacy settings shouldn’t be buried. Users should be able to understand, in plain language, what the agent is doing with their data during a task.
Safety that includes social comfort: Accuracy matters, but so does the feeling of being watched. If an assistant’s personalization crosses a line into “how did you know that?” territory, it should either ask for confirmation or avoid using that detail unless the user opts in.
Accountability for recommendations: When an agent makes a plan or suggestion, it should provide reasons that are understandable and traceable. If it’s using personal context, that context should be part of the explanation.
These aren’t anti-technology ideas. They’re pro-trust requirements. Without them, the agent’s effectiveness will keep outpacing the user’s ability to govern it.
There’s also a broader cultural implication. As AI agents become more capable, they will increasingly mediate everyday life: scheduling, communication, planning, summarizing, and decision support. That mediation will shape how people relate to each other and to institutions. If the assistant can draft messages that sound like you, it can also influence tone and meaning. If it can plan trips and recommend options, it can steer preferences. If it can remember personal details, it can reinforce a sense that the assistant is always “in the room.”
That can be comforting for some people. For others, it can feel like the assistant is replacing parts of their autonomy. The productivity promise tends to ignore this tension because it focuses on outcomes like speed and convenience. But the real question is whether the assistant expands freedom or narrows it.
Spark’s reported ability to recall personal details