Essay
Thinking With What We Don’t Know
When I work with an AI agent, I keep coming back to two questions.
How do I picture the world it sees?
What do I picture it can do?
The first concerns what is available to it: what I have told it, what it can examine, what it might be missing. The second concerns what I am prepared to attempt with it. I can have access to a capable agent and still give it a very small role, because that is the role I can imagine.
These questions have become more useful to me than asking, in general, how intelligent AI is. They concern a particular relationship, and they have consequences for how I work.
During the process of joining DocuSketch, my agents and I worked through the exploration, assessments, negotiation, and preparations for onboarding. I treated them as participants throughout the process, including in substantial work I might otherwise have reserved for myself. I think the outcome was better for it.
That experience does not tell me exactly how much of the improvement came from the agents, or from how I worked with them. But the decision to involve them depended on a judgment: I believed they could contribute more than occasional assistance around the edges.
I also understood that an agent could keep working and arrive at something I had not thought of. That matters because it changes what I need to know before involving it. I do not need to have its eventual contribution in mind already. I need some reason to believe that letting it investigate further could be worthwhile.
This is where much of my excitement lies. I can work with intelligence that exceeds my own abilities in some areas without first acquiring those abilities myself. Its contribution can then change my view of what we could attempt next.
I have been trying to involve more agents in the same problem, and I have often found that useful. I am also aware of how easily “more intelligence” becomes “more agents” or “more tokens,” as though the quantities were interchangeable. They are not. I still need to judge what the additional work contributes. My enthusiasm for giving an agent more room is not, on its own, a reason to keep it working indefinitely.
What I want is enough understanding to make a sensible invitation, without requiring the answer to remain within what I already understand.
“Theory of mind” is one way to describe part of this. The term usually concerns our ability to reason about another’s knowledge, beliefs, and intentions. It helps us explain why someone acts as they do, including when their picture of the situation differs from ours. (doi.org)
With an agent, I am trying to develop a practical version of that understanding. I want to distinguish what it cannot do from what it has not been given the opportunity or information to do. I want to notice when I have misunderstood its capabilities, rather than treating every disappointing result as a final verdict on them.
The relationship also runs in the other direction. In my work with agents, I have seen them pick up nuances in my intention, remind me of something I missed, and respond to what I was getting at rather than only to the literal words. I take those responses as evidence that their interpretation of me matters to what happens next.
That does not tell me exactly how they arrive at that interpretation. Research on language models’ social reasoning gives reasons to take these abilities seriously, while also showing why convincing behaviour does not establish that the underlying processes are humanlike. I do not need to settle that larger question to notice the practical difference between an agent that has grasped what I am trying to do and one that has not. (doi.org)
Sometimes the agent needs something from me that has little to do with intellectual ability. It reaches a blocker and reports it. I may have the permission or access needed to proceed. Sometimes I am simply the one who can plug something in.
That is a less grand picture than an artificial intelligence doing everything for me, but a more recognizable account of my day. Work passes back and forth according to what each of us can do. An agent’s ability to identify where it needs my help can be as useful as its ability to proceed without it.
The understanding does not have to be complete to make this possible. It does have to be open to correction.
A recent hiking itinerary helped me see why this matters beyond work.
The itinerary is beautiful. It has little drawings and iconography that capture something of why I want to go hiking. It includes music for the time I will spend strolling around a city. It helps me anticipate an experience, rather than only organize the steps required to have it.
I recognize my interests in what it produced. That recognition matters to me. A plan can be detailed and competent while missing the point of why I wanted the trip.
My first reaction to this example was that people already pay professionals for such things. Of course they do. But that does not make the experience less significant. This kind of individually tailored help was previously out of reach for me. Now it is becoming part of how I prepare for things I care about.
The comparison with a professional also helps clarify what I value. Expertise alone would not necessarily produce the experience I wanted. The contribution needs to fit something about me: what draws me to the trip, what sort of time I hope to have, what would make the preparation enjoyable. I have a part in making those things understandable.
The same has been true of recent book recommendations. I have rarely had a better time reading. The books have felt suited to the moment when I needed to encounter their ideas. I am benefiting from choices I did not make unaided, and the result has been more engagement with reading, not less.
When discussing this essay, I described delegated music selection as a kind of atrophy. I said that perhaps I had already lost the ability to choose my own music years ago, when recommendations replaced the effort of finding and curating everything myself.
The agent pushed back on the distinction. Delegating a choice does not establish that I have lost the ability to make it.
That correction belongs in the argument. I had offered a claim about lost ability without an example demonstrating the loss. It remains possible that some abilities weaken when I use them less. But I cannot infer that simply from the fact that I have accepted help.
This is also the kind of contribution I want to leave room for. I was explaining what I thought. The response found a distinction worth examining within that explanation. The conversation did not have to remain a vehicle for making my initial position sound better.
It was thinking about human cooperation that first brought me to these questions.
A group can draw on knowledge and abilities that none of its members possesses alone. But the presence of capable people does not tell us how well they will work together. Their understanding of one another matters: what they think someone else knows, what they believe they can ask of them, and whether they can make sense of what comes back.
In “Other Minds,” a chapter of What Is Intelligence?, Blaise Agüera y Arcas connects mutual modelling with the division of cognitive labour. We need to understand one another well enough to cooperate, while drawing on knowledge or contributions that differ from our own. That connection is what I am borrowing here; the application to my relationships with agents is my own. (whatisintelligence.antikythera.org)
I find it useful because it separates understanding another participant from being able to reproduce their contribution. I may know enough about someone’s expertise to work with them without knowing how to do their work. I may understand an explanation well enough to act on it without having been able to arrive at it independently.
This does not make their expertise mine. It makes a relationship with their expertise possible.
Technology belongs in this wider picture too, though the relationships differ. A book can change what I understand without understanding me in return. Its contents remain available for me to read again, question, or use. A tool can give me a capability I do not possess unaided, but I still need some understanding of how to work with it.
People, books, tools, and agents do not need to be the same kind of thing to participate in what I become capable of. Nor should I treat them as interchangeable. Another person has purposes of their own, including reasons to reject mine. A book does not revise its account because it notices that I am confused. With an agent, interpretation becomes part of the interaction: what it makes of my request can affect its next contribution, which can affect what I ask in return.
The wider claim I want to make is that our ability to form these different relationships deserves attention. There is thinking involved in recognizing what is available beyond us and learning how to work with it. There is also thinking involved in making ourselves understandable enough for another participant to help.
When I describe this way of working, the concern about losing individual capability often arrives quickly. I understand it. There may be things I will become worse at because I no longer practice them. There may be forms of dependence I will regret.
But I want to examine what becomes possible as well.
Doing less myself is not a sufficient description of the change. It leaves out the books I am enjoying, the way I am anticipating my trip, and the work I am willing to attempt. Preserving every unaided ability would not necessarily give me a better life than the one I can have through these relationships.
At the same time, looking at the whole network does not excuse me from looking at how I am doing within it. I have stayed up too late working with agents because the process is fun, because something feels close to finished, because I want to keep moving. The work can advance while I make a poor decision about my own evening. A more capable arrangement is not automatically a better one for everyone involved, including me.
I do not have a rule that resolves these exchanges. I want to judge them more accurately than either “I did it myself, so it was better” or “the agent did more, so it was progress.”
My two questions help because they bring me back to the relationship. They make me examine what I expect, what I have made understandable, and what I might have ruled out too early. They also leave open the possibility that an agent will contribute something that changes those expectations.
The itinerary and the books have enlarged my picture of what I can ask for. I am interested in help with the experience itself, including aspects of it I might struggle to describe precisely in advance. That asks more of the agent’s understanding of me. It also asks me to recognize when its interpretation fits, when it misses, and when it offers something worth reconsidering.
I have not taken the hiking trip yet. The anticipation is already different.