Thinking With What We Don’t Know — Earlier version

Sometimes, when an AI agent gives me the wrong result, I discover that part of the task never left my head. I know which file matters, or which part of an example was incidental. From my side, the request feels complete. The agent has whatever context I have supplied, whatever it has actually read, and the assumptions it makes from there.

The agents I work with can inspect files and use software tools, choosing intermediate steps toward a goal. I don’t have to specify every action. In my better interactions, I keep revising my sense of what the agent knows about the task and how it is interpreting me. I give it room to investigate, then inspect what it actually did. When the work exposes a mistaken expectation, I may need to supply more context or change how I check the result. I can be clear about what I want while remaining wrong about what the agent needs from me.

This has made me reconsider what I count as intelligence. The capability I draw on may remain outside me. Yet making it useful can require me to understand something I was missing and change how I approach the problem. I think that work belongs in our understanding of intelligence, alongside the ability to find an answer unaided.

Reading Blaise Agüera y Arcas’s What Is Intelligence? helped me recognize the connection to other people. In his discussion of other minds, he relates mutual modeling to cooperation among people with different knowledge. We need enough understanding of one another to coordinate, while each brings something the others do not already have.

When I ask someone for help, I am making judgments about what they know and what they will understand me to be asking. They are also making judgments about me. An explanation that assumes too much leaves me unable to use it; one that assumes too little can bury the part I need. A question from me can reveal an assumption the speaker had not realized needed explaining. They adjust the explanation, and I may have to adjust the question. Making knowledge usable is work on both sides of the conversation.

There also has to be room for them to surprise me. I seek their help partly because they may notice what I cannot. If I had to predict everything they could contribute, much of the reason for asking would disappear. The understanding I need is enough to follow an unexpected answer and recognize that it may require me to change the question.

A hammer offers a simpler way into this problem. Imagine a nail standing partly driven into a piece of wood. A practiced user can anticipate the hammer’s path toward the nail and bring its flat face down squarely on the head. They can feel the difference between a clean hit and one that glances off the edge. As the nail sinks, they adjust the next swing. If it starts to bend, they stop to straighten or replace it instead of repeating the same motion harder.

Much of that understanding is practiced movement. It need not come with a spoken account of angles or force. Learning to use the hammer changes what the person can imagine doing with the nail. A motion that would once have been an uncertain swing becomes something they can anticipate and correct. The hammer itself has acquired no new capacity. The person has learned to act with a capacity that was already there, through an understanding of how hand, tool, and material work together.

The model I need for the hammer concerns movement and contact. With another person, I also need some sense of what they believe I mean; the hammer has no such interpretation. But people are present in the scene in another way. Someone made the head to withstand repeated blows and fitted it to a handle I can grip. Their decisions spare me having to make a suitable tool before I can begin.

Teaching continues that work. An experienced person can watch a learner’s swing and point out that the head is approaching at an angle. They make something visible that the learner wasn’t noticing. The learner still has to practice; neither a well-made hammer nor a good explanation delivers the whole craft. But the learning begins with a usable tool and help in understanding it. The person driving the nail benefits from work they did not have to recreate. The maker and teacher may never know what the learner goes on to build. Their contributions can make that building possible without determining its purpose.

An agent can investigate the task itself, including assumptions hidden in my request. Suppose I ask an agent to fix a bug: people sometimes save a form and find two copies of the same entry. I suspect they are clicking twice, so I ask it to disable the Save button after the first click. Given access to the repository—the project’s code—and a way to run tests, it could make the change I requested. The button becomes inactive. A test confirms that a second click does nothing.

A patch is not yet a repair. That test checks the behavior I specified, which might rest on a wrong diagnosis. A second agent, given my explanation and the passing test, could approve the change while sharing the same mistake. Two approvals would leave the original assumption unexamined.

Given the sequence that produced a duplicate and room to look beyond the button, the agent might find that the application repeats a request because it has not yet received confirmation that the entry was saved. One click can therefore produce two entries. Disabling the button leaves the cause untouched. The next change would have to deal with those repeated requests. Now the investigation has improved my account of the problem, and I have to let the assignment change with it. The agent needed context in which to investigate and enough freedom to follow what it found. Insisting on my original solution would undo the benefit of asking for help.

To test a revised fix, we would need a check that reproduces the duplicate entry with the old code and no longer does with the new. We could inspect the change and try the behavior through the form itself. The agent’s confident account of having done those things would not substitute for evidence that it had. Even then, a reviewer might find something our checks missed.

The repair alone, though, would tell us little about my own intelligence. A wealthy person could pay an excellent team to solve the entire problem. The team might supply the diagnosis and the checks without needing useful guidance from the person paying. Their software could be repaired without their ever understanding the problem. They might have exercised good judgment, or benefited from others compensating for poor judgment. Success alone would not distinguish the cases. The thinking I am interested in could be happening among the people hired, with hardly any contribution from their customer.

A project may already have clear documentation and useful tests because someone else made it easier to work on. If a reviewer notices what I missed, their explanation might improve my understanding; it would not make me the person who found the mistake. Someone with sound judgment might also lack the money or permission to obtain the help the task needs.

I see intelligence in the reasoning required to make the help usable. To recognize that a passing test answers too narrow a question, I have to understand the gap. Learning that my diagnosis was wrong changes which investigation makes sense next. Those acts do not become incidental because someone else is doing the implementation. There is room for invention in making a contribution testable, or finding an approach to the problem that was previously out of reach. Those are ways of thinking through a problem whose solution I cannot produce alone.

I don’t need to reproduce every specialist’s work to participate responsibly. My understanding of why a check matters can be much smaller than the understanding needed to implement the repair. A reviewer can make their judgment usable by explaining what they checked and what remains uncertain. That can give me grounds for relying on work I couldn’t have produced. It can also reveal a consequential gap, in which case we need more help or should stop. Dependence can make action possible without pretending that the supported ability is all my own. Authorizing a change still leaves me answerable for that decision.

I have become more interested in the relationship that makes a resource useful than in simply having more resources. In the bug example, access to the code gives the agent something to investigate. Talking with the person experiencing the failure gives us a chance to discover whether we have described the problem adequately. They need to understand what we are investigating, too, so they can tell us when it misses their experience. Neither contribution becomes useful just by being nearby. Someone has to make the connection workable.

That connection can produce an understanding nobody brought to the task ready-made. Imagine a developer and someone who uses the form trying to agree on what a duplicate is. The developer starts with matching content. The user explains that they sometimes need to make two identical entries deliberately. The developer sketches what happens between clicking Save and storing an entry. Looking at that sequence together, they begin distinguishing an intended submission from the software’s attempts to save it. They can now investigate whether the application can tell the difference.

The task becomes preserving each intended submission once, rather than removing entries that happen to look alike. That description can become a test they both understand. The developer can check how the program handles repeated attempts, and the user can check that separate intended submissions are kept. They have learned enough about each other’s problem to investigate something their original descriptions obscured. Neither person needed to acquire the other’s entire expertise for this new way of working to become possible.

Understanding each other does not guarantee that we will want the same thing. The person using the software may reject a technically convenient fix because it interferes with what they need to do. Their purposes matter independently of how helpful their objection is to me. They can decline a proposal or ask that the project change direction. Clever coordination can serve an unfair arrangement too; its intelligence does not settle whether the arrangement is worth participating in.

The more of the underlying work I understand, the more an explanation can do for me. I can follow it far enough to challenge an assumption, or recognize a question I hadn’t known to ask. When someone shows me a distinction I had missed, I can take it into the next problem. Relying on their knowledge and developing my own can happen in the same exchange. I want my knowledge and my resourcefulness to grow together.

I can also become able to take part in work that remains beyond what I could do alone. That asks something different of me from mastering every part of it. I have to learn how another person’s contribution connects with mine, and how to proceed when neither of us understands every part of the work. Becoming better at that is a form of growth even when the gaps in my knowledge remain substantial. Some things I will need to learn myself; for others, I need to learn how to participate.

That growth also includes making my own understanding easier to work with. I can leave an explanation that makes sense to someone who does not share my starting point, or enough of the reasoning behind a decision for them to question it. When they do, I need to consider that the difficulty may be in my account of the problem. Their question deserves a chance to change it. Something I understand may become useful to them in a way I had not anticipated. They may also change what seems worth doing. I want my own understanding to give someone else room to think. Where that leads us is something we will have to decide together.

Source note

Blaise Agüera y Arcas, What Is Intelligence?, chapter 5, “Other Minds”, especially “Sphexish.” His discussion connects mutual modeling with complementary knowledge in cooperation. The extension to tools and agent use here is my own proposal.