The Relational Arc · November 13, 2026
If AI Wins Wherever the Answer Can Be Checked, Is Judgement What Is Left for People?
The newest answer to "what is left for people?" is no longer a skill. It is a whole kind of work: judgement, taste, knowing which question to ask, the work where no answer comes back to be checked. I think that answer makes the same mistake as the ones before it, and that it points people away from the kind of work that meant the most.
In December 2025 Dwarkesh Patel, a writer who interviews the people building AI on his long-form podcast, explained in Thoughts on AI progress (Dec 2025) why he doubted that AI would soon be doing the work of AI researchers. Today’s systems are trained once, before anyone uses them, and after that they do not learn on the job. He found it “super implausible” that a system without a child’s ability to learn could solve a problem people have been stuck on for most of a century. On November 13th, 2026, in The avalanche that hit math is coming for AI development, he quoted that passage against himself, and put his middle guess for AI that matches the best human researchers at AI research at 2028.
What changed his mind was mathematics. Wherever answers can be checked, he writes, and a system can be given thousands of rounds of practice while it is being trained, AI has gone from worse than most people to better than the best of them within a few years.
In Human of the Gaps: If AI Can Do It, Do People Matter Less? I argued that resting what people are worth on the list of things machines cannot yet do is a losing defence, because the list keeps shrinking. The phrase in that essay’s title compares the habit to the older one of finding God in whatever science had not yet explained, and I will call each item on the list a gap. If Patel is even roughly right, the list has changed shape. The item people now point to is no longer a skill, like chess or writing or proof. It is everything that has no answer to check.
The Gap Becomes a Kind of Work
The careful version of this comes from Toby Ord, an Oxford philosopher. In August 2026 he published Mathematics is Much More than Proof, in answer to people saying that the days of human mathematicians were numbered. Proof, he argues, is one part of mathematics and, to his mind, not the most interesting. At least as much of the work lies in deciding which questions deserve an answer, and the most important work of all lies in opening fields that nobody had thought to look in. He sees little sign so far that AI systems can “ask powerful questions.” He is careful to add that AI may come for these parts too, and soon. Even so, the essay ends on a consolation. Mathematicians, he suggests, could move to the work of setting the direction of their subject, and he describes that move as being promoted.
The same thought turned up a few weeks later among people who build these systems. On September 11th, 2026, Patel sat down with three AI researchers for AI researchers debate how close we are to recursive self-improvement, a conversation about AI being used to improve AI, and asked them what would be the last piece of AI research that people still had to do themselves. Beren Millidge, the chief technology officer of the AI company Zyphra, answered that it would be “asking the right questions.” John Schulman, a co-founder of OpenAI who is now chief scientist at the AI company Thinking Machines Lab, said that “the last job for humans” is defining the objective and “deciding what we actually want.”
The same phrase was there in the story that began this argument. Human of the Gaps: If AI Can Do It, Do People Matter Less? opened with a security researcher who found serious flaws in old and heavily searched code with an AI system’s help. His firm’s account said the way in came after asking the right questions, and the part he named as his own was knowing where to look.
None of these people is arguing about what a human being is worth. They are making forecasts about work, and careful ones. The argument about worth is made by other people, and it is the temptation Human of the Gaps was about: to take whatever AI cannot do today and call it the thing that makes us human. That argument now has a new address. When it is said that people will always be needed for judgement, for taste, or for knowing what to ask, the speaker is pointing at the place Ord and Schulman describe and resting something much heavier on it than they do.
Why This Gap Looks Different
This is the strongest form the gap argument has taken, and it deserves to be met at full strength.
The earlier gaps were bets that a particular skill was too hard. This one comes with a reason. The systems improve by practising against a score: a proof either goes through or it fails, a program either passes its tests or it does not. Ord makes the point directly in Mathematics is Much More than Proof. Nobody has a good way to measure whether a question was worth asking, and without a measure there is nothing to practise against. So the claim is that the method which closed all the earlier gaps has nothing to hold on to here. If that were right, this gap would be different in kind from the others, and the long retreat from one gap to the next would finally have reached solid ground.
A Score Is Something We Build
I don’t think it has.
Patel says, almost in passing, that it is hard to measure progress on work that cannot be checked, because if it could be measured, “those tasks would be verifiable.” That sentence can be read two ways. Either nobody has built a check for this work yet, or no check can ever be built. Ord, as I read him, leaves both open: he writes that there is “very little evidence either way” about whether AI can find good questions. The reading that follows is mine and not his.
Take the first possibility. Checks do get built. Patel’s own example is research taste, the knack for telling which improvement, found on a small AI model that is cheap to experiment with, will still hold on a model a hundred times larger. He points out that this can be turned into a test: have the system predict which results will hold, run the larger experiment, and see whether it was right. Human judgement has always been checked in a similar way, though slowly. Patel’s example of something that cannot be directly verified is a great researcher’s ability to bet years ahead on the right direction. Yet the only reason anyone knows who has that ability is that the years went by and the bets came in.
There are two honest objections to this, and I think both are partly right. A slow check is run once. It cannot be repeated thousands of times, and luck is mixed into the result, so it is a much poorer thing to practise against than a proof that either goes through or fails. And for a question like “was this worth asking?”, the later verdict comes from later people, whose taste was shaped by the answer they are judging, so the check is not fully independent of the thing checked. Both points mean that the method which worked for proof may not carry over easily, and that this gap could stay open for a long time. I make no prediction about when a machine will show good judgement, or whether one will. Patel gives one chance in four that AI does not match the best researchers until after 2032, and nothing here depends on his dates.
What the objections do not supply is a boundary in principle, and the gap argument needs one. So take the second possibility at its strongest, and suppose that for some kinds of judgement no check can ever be built. Then nothing can show who is good at that work, and that holds for people as much as for machines. On the first possibility the gap can close the way the others did. On the second it can never be shown to be a gap. Either way, “cannot be checked” tells us about our instruments and nothing about whoever, or whatever, is doing the work.
This is the mistake I described in The Kasparov Fallacy: Why We Keep Underestimating Machine Minds, an essay named after Garry Kasparov, the world chess champion who lost a match to an IBM computer in 1997: a limit on what we can currently see or measure gets read as a limit on what could ever be done. I would be wrong if someone showed that a kind of judgement can be scored well enough to prove that people are good at it, and yet cannot be scored well enough for anything else to learn it. I cannot see how both could hold at once.
Where Nothing Is Scored, Nobody Leads
Human of the Gaps argued that a gap proves as little about the machine as it does about people, and the same holds here.
If there is no score for judgement, then the claim that people are better at it cannot be shown either. That is part of why it feels safe. A claim that cannot be tested cannot be lost, and for the same reason it gives no support to anything built on top of it. “People keep the work that can’t be checked” is a comfort of that kind.
The same missing score faces the other way. With nothing to measure, I cannot tell how good a machine’s judgement is, which means I cannot conclude that it has none. I also cannot conclude that it has plenty because it sounds sure of itself. Work with no score is where trusting a system too much and dismissing it too quickly are both easiest, because nothing pushes back on either. Nothing in this essay says that any AI system has good judgement, or a mind. An open gap, here as before, proves nothing about the people on one side of it or the machine on the other.
Two Things Inside “Deciding What We Want”
Everything so far treats judgement as a skill, something a person or a machine could be better or worse at. Schulman’s sentence holds two different things, and only one of them is a skill.
The first is defining an objective: stating a goal clearly enough that a system can pursue it. That is a skill. A person can be better or worse at it, and so, in principle, could a machine. As a place to rest what people are for, it is one more gap.
The second is deciding what we want, and that is something else. Nothing in the last two sections applies to it, because it is not a question waiting for a check to be built. No answer comes back, slowly or quickly, to say that I wanted the wrong thing for my own life.
It is also not handed out by skill. A doctor usually knows more than her patient about what a treatment will do, and the decision is still the patient’s, which is the whole point of informed consent. A voter may understand less about policy than the officials she votes for, and the vote is still hers.
The obvious objection is that we do take decisions away on grounds of ability. A young child does not choose her own surgery, and a court can appoint a guardian for an adult who can no longer manage. That is true, and it shows that the standing to decide depends on a threshold. It does not show that standing is a ranking. Above the threshold, being better at deciding gives nobody a larger say: the brilliant patient and the ordinary one sign the same form. A machine that decided better than any person would not push a single person below that line. I argued in Human of the Gaps that what is owed to a person does not follow a list of skills, and this looks to me like one more case of it. A system might one day predict what I will want better than I can, and it would not follow that the choice had become the system’s to make.
The harder objection is that people’s wants are muddled, and that a system might leave me better off than my own choices would. I think that could well be true, and it is already true of doctors. A patient who understands what she is choosing may refuse a treatment that would help her, and the refusal stands, although everyone in the room knows she may be worse off for it. It stands because a life steered for a person’s own good, over her objection, has stopped being hers to live. I doubt many of us would choose to live where anyone who knew better could overrule us whenever they judged we would be improved by it. Advice can become as good as it likes without changing that.
So the retreat to judgement is half right, for a reason it does not give. Who decides is a question that does not move when machines improve. But it is a question about who has a say, and I am not offering it as an account of what a person is worth, which I leave open below. It is also not a task at which people excel, and calling it “the last job for humans” turns it back into one, which puts it on the list again.
The Work With an Answer
The work that cannot be checked is also an uneasy place to settle, for a reason that has nothing to do with machines. In The Purpose Displacement Problem I asked what made work feel as though it mattered, and argued that it was never the lack of a choice. It was that the work answered to a standard outside the person doing it. The surgeon in that essay worked in a field where patients “either recovered or didn’t,” where the test was set by the world and paid no attention to how hard she had tried.
If that is right, the promotion on offer is a strange one. It moves people out of the work where the world answers back, and into the work where, for now, nothing can tell them they were wrong. I don’t mean that asking a good question is empty work. It is real work, and in time the world answers it too. My point is narrower. Being impossible to score is not what makes work worth having, and a great deal of what made work matter came from the score.
Human of the Gaps ended with a test for any account of human worth: whatever worth rests on cannot be something that shrinks when a machine learns a new skill. Judgement, treated as the last skill, would fail that test if a check for it is ever built, and if none ever is, there was never any way to show it was ours. What a life is for when nobody needs its skills is a question I still cannot answer, and I would rather leave it open than fill it with the newest gap.
Patel’s doubt was a forecast. He said in public what it rested on, and less than a year after writing it down he gave it up, because what he saw had changed. That is what a forecast is for. I don’t think what a person is worth should be held the same way, ready to be given up when the next result comes in.
Reading List & Conceptual Lineage
From Sentient Horizons
Human of the Gaps: If AI Can Do It, Do People Matter Less?
The essay this one follows: why a gap in ability settles neither what a person is worth nor whether a machine has a mind.
The Kasparov Fallacy: Why We Keep Underestimating Machine Minds
A limit on what we can explain or measure is not evidence that something is impossible.
The Purpose Displacement Problem
What work was for, and why a standard set outside the worker gave it weight.
External Works
Dwarkesh Patel — The avalanche that hit math is coming for AI development (November 13th, 2026). Why he now expects AI to match the best AI researchers within a few years.
Dwarkesh Patel — Thoughts on AI progress (Dec 2025) (December 2nd, 2025). The earlier view, in his own words.
Toby Ord — Mathematics is Much More than Proof (August 23rd, 2026). The careful case that asking questions and opening new fields are a separate matter from proof.
Dwarkesh Podcast — AI researchers debate how close we are to recursive self-improvement (September 11th, 2026). The conversation with John Schulman, Beren Millidge and Charlie O’Neill.
Sentient Horizons is an essay series on what a mind is, whether a machine could have one, and how to act toward minds we cannot see inside. Start Here is one route through it; Essays lists everything, newest first.
Three Ways In
- Consciousness Is Like Flight If you want to know what a mind is
- We Built the Alien First If you want to know whether a machine could have a mind
- The Purpose Displacement Problem If you want to know what AI means for your own life