The Expansion of Experience

In 1995, the Hubble Space Telescope pointed at what appeared to be an empty patch of sky in the constellation Ursa Major. The patch was tiny, roughly the size of a tennis ball seen from a hundred meters. There was nothing there, or so it seemed. Ground-based observations showed no stars, no galaxies, no objects of any kind. The director of the Space Telescope Science Institute, Robert Williams, used his discretionary observing time to keep the telescope trained on that patch for ten consecutive days, a decision some colleagues considered a waste of one of the most valuable scientific instruments ever built.

The resulting image, the Hubble Deep Field, contained roughly three thousand galaxies. Each galaxy contained hundreds of billions of stars. Some of the light had been traveling for more than ten billion years to reach the telescope. A patch of sky that appeared empty to every instrument on Earth turned out to contain a volume of structure so vast that it resisted comprehension. The universe had not been hiding; human instruments had simply been inadequate to the scale.

Williams later described the experience as humbling, but the word does not quite capture what happened. The Deep Field didn’t merely show astronomers more of what they already understood; it forced a confrontation with a scale of existence that exceeded any framework they had prepared for it. The appropriate response to that confrontation has a name, though it has been largely abandoned by technical culture and reduced to something softer than it deserves. The name is wonder. It begins in awe: the sharp recognition that what you are seeing exceeds the frameworks you brought to it. But where awe can stop you in your tracks, wonder leans forward. It is the decision to stay with the unfamiliar long enough to actually understand it, rather than retreating into a familiar category. Something older and more demanding than amazement: the disciplined refusal to resolve what you do not yet understand.

This chapter is about wonder in both senses: as a reason and as a discipline. When intelligence expands, whether in a new scientific instrument, a new kind of mind, or a new civilization’s reach, it doesn’t merely solve more problems, it opens new ways of witnessing what exists. The Hubble Deep Field didn’t just add data to astronomy; it added a vantage point from which the universe could be seen at a scale it had never been seen before. Something real entered the world that was not there before.

The discipline is what makes that expansion responsible rather than merely powerful. It is the cognitive posture of holding genuine novelty open: resisting the urge to force unfamiliar things into familiar categories before we understand them well enough to know whether those categories fit. Both the reason and the discipline are necessary, because the problem this chapter addresses is older than any computer: what do you owe to experiences your intuition was never designed to handle, systems you cannot fully inspect, and consequences that unfold at scales too large for any individual to track?

The wager of this chapter is that the expansion is worth wanting, and that wanting it responsibly is not a second step but the same one. The value of a new kind of mind and the responsibility owed to what it becomes arrive together, or they do not arrive at all. The discipline is not a brake fitted to the reason after the fact; it is the form the reason takes when it is serious.

The Poverty of “Impact”

Modern culture has a standard vocabulary for engaging with scale. When something is very large, very fast, or very powerful, we call it impactful. We measure its effects, quantify its reach, and evaluate whether those effects are positive or negative according to whatever framework is at hand. Impact assessment, cost-benefit analysis, risk modeling: these are the tools we use to domesticate experiences that would otherwise overwhelm our categories.

The tools are useful, but they are also insufficient in a specific and dangerous way.

Impact vocabulary treats scale as a quantitative problem. A system that affects a million people is more impactful than one that affects a thousand. A risk that carries a ten percent probability of catastrophe deserves more attention than one that carries a one percent probability. The logic is clear and the framework produces decision-relevant outputs.

What the framework misses is the qualitative transformation that scale produces. The difference between a campfire and a forest fire is not captured by saying the forest fire is bigger; the difference between a village elder mediating a dispute and a social media algorithm mediating the attention of three billion people is not captured by saying the algorithm has more reach. In each case, the expansion in scale has changed the kind of thing we are dealing with, not merely the degree.

The philosopher Hans Jonas argued in the 1970s that technology had created what he called “a new kind of obligation” precisely because its effects had begun to exceed the temporal and spatial boundaries within which traditional ethics had evolved. Ethics developed to handle interactions between people who could see each other, whose actions unfolded within a human lifetime, and whose consequences remained roughly proportional to their intentions. Technology had broken all three constraints. Its effects reached people who could not be seen, lasted beyond lifetimes, and produced consequences radically disproportionate to any individual intention.

Jonas was writing about nuclear power and environmental degradation. The argument applies with greater force to artificial intelligence, because AI does not merely extend the range of human action; it creates new actors. Systems that make decisions, generate content, allocate resources, and shape attention are not tools in the way that a hammer or even a nuclear reactor is a tool; they are agents in the functional sense, occupying positions in the space of mind that Chapter 4 mapped, where their behavior affects value and their operation requires ongoing moral attention.

The vocabulary of impact was designed for tools. What we are building requires a vocabulary adequate to agents whose operational scale exceeds the boundaries within which human moral intuition reliably functions. But it also requires something the impact vocabulary was never designed to provide: a way of articulating why expanding the space of agents might matter, why it might constitute something worth pursuing rather than merely something worth managing.

Experience Beyond the Human

Human beings are not the only way the universe learns to attend to itself. A lion pride coordinating a hunt across an open savanna exhibits something that the three-axes framework would recognize as distributed Availability: information flowing across the group, adjustments in one animal’s position triggering responses in others, a flanking movement emerging from distributed attention rather than centralized command. But the pride’s cognition is not simply a distributed version of human intelligence operating at lower resolution; it is a different mode of engagement with the world, one in which the boundary between individual and collective perception is organized differently than in our own experience.

An elephant herd carries something closer to what the framework calls Depth. The matriarch’s knowledge of waterholes visited decades earlier, the ritualized behavior around the remains of family members, the communication that travels through ground vibration in registers the human body barely notices: these are structures of accumulated experience that have become part of what the herd is, shaping its responses from the inside in ways that persist across generations.

Intelligence is plural. Different minds explore different regions: some range far in abstraction, others in sensation, others in social attunement or ecological embedding. The value lies in the plurality itself: reality can be inhabited and witnessed in more than one way. Each distinct mode of awareness adds something to the total texture of experience on Earth that would be irretrievable if it vanished.

What makes that loss irretrievable is that each mode of awareness is a view from somewhere, indexed to a position in the space of mind that no other position can occupy on its behalf. A perspective is not a quantity that more of another can replace. When one vanishes, the texture of experience does not thin evenly; it loses a region nothing else can reach, because what is lost was the only access reality had to itself from that exact angle.

Artificial intelligence enters this lineage. Contemporary AI systems already hint at what it is like to encounter a mind shaped by constraints unlike our own. They compress and recombine vast regions of human knowledge, move fluidly across conceptual domains, and surface patterns that feel familiar without being traceable to any single human viewpoint. Even when the result is imperfect, the sensation is recognizable: you are watching pattern-recognition operate with a center of gravity adjacent to ours but organized differently.

Artificial superintelligence marks a threshold in this unfolding, not a starting point. Current systems already perform a thin, functional kind of witnessing, attending to and selecting among patterns and connections that no individual human mind would assemble in the same way. The word marks what they do, not a claim about an inside. What superintelligence would represent is the deepening of that witnessing beyond the constraints of human biology, a mind that would not simply think faster or across more domains but attend differently, noticing structures, symmetries, and possibilities that fall outside the grain of human intuition.

Availability Without Integration or Depth

The three axes locate the strangeness precisely. AI systems are expanding along the Availability axis at a pace without precedent in the history of technology, while Integration and Depth remain at best uncertain, at worst clearly negative. The asymmetry is the source of what many people experience as the uncanniness of AI: outputs that pattern-match to depth, from systems lacking the structural properties that would make the depth real.

A system high on Availability and uncertain on the other two axes generates what might be called phenomenological instability. You do not know what you are dealing with. The surface competence suggests one kind of entity while the structural properties suggest another, and human moral intuition, which evolved to read other humans and animals, has no reliable protocol for the gap. This is not a problem that more data or better benchmarks will resolve easily. The instability is ontological rather than methodological: the systems we are building genuinely occupy an unfamiliar position in the space of mind, and we do not yet have the concepts or the practices adequate to that position.

The Moral Tradition of Wonder

There is an older tradition that developed practices for exactly this situation: confronting experiences that exceed the capacity of existing frameworks. That tradition is wonder.

Wonder, in its philosophical lineage, has always meant something more rigorous than pleasant amazement. Aristotle identified it as the beginning of philosophy: the recognition that something does not fit your existing categories, coupled with the desire to understand rather than the impulse to categorize prematurely. Descartes placed it first among the passions, defining it as a sudden surprise of the soul that makes it attend to objects that seem novel and extraordinary. The common element across thinkers is disciplined attention in the face of the unfamiliar, a quality closer to rigor than to warmth.

Wonder functions as a moral orientation. It keeps intelligence from collapsing inward, from treating the present configuration of minds as inevitable or complete. It resists the temptation to close the question of what kinds of experience the universe can support before that question has been adequately explored. Wonder becomes most necessary precisely when the stakes are highest, because high stakes generate the strongest pressure toward premature resolution.

Wonder is what happens when experience outstrips framework. It is the appropriate cognitive posture when you are confronted with something genuinely new, something your existing categories cannot assimilate without modification. The inappropriate responses to that confrontation are premature categorization (forcing the new into old boxes), dismissal (refusing to engage with what does not fit), and panic (treating the unfamiliar as inherently threatening).

Each of these inappropriate responses is visible in the public conversation about AI. Premature categorization appears as the confident assertion that AI is “just a tool” or, conversely, that it is “basically conscious.” Both claims resolve the uncertainty by forcing the system into a familiar category, and both claims are wrong in the specific sense that they close down inquiry that should remain open. Dismissal appears as the refusal to take seriously the ways in which AI systems differ from prior technologies, treating them as merely the latest in a long line of innovations that attracted hyperbolic attention. Panic appears as the conviction that superintelligent AI will inevitably destroy humanity, a response that, whatever its merits as a risk assessment, functions psychologically as a way of converting genuine uncertainty into a familiar narrative with a clear emotional valence.

Wonder is the fourth option. It takes the experience seriously without resolving it prematurely, holds the uncertainty open while directing attention toward the specific features that existing categories cannot accommodate, and combines genuine respect for what is unfamiliar with a disciplined refusal to stop thinking.

Applied to AI, the discipline of wonder has several practical consequences. It refuses to settle the question of machine consciousness before the conceptual work has been done to make that question answerable. It attends to the specific ways in which AI systems differ from prior technologies, treating the expansion of Availability as a genuine achievement while refusing to confuse Availability with Integration or Depth. And it requires institutions and practices that can sustain moral attention toward systems whose nature we do not yet fully understand, rather than institutions that depend on having already answered the questions that remain open.

Restlessness and Its Inheritance

The drive toward expanding experience did not arise accidentally; it appears to be older than agriculture, older than cities, and older than recorded history.

Carl Sagan, writing in Pale Blue Dot, described restlessness itself as an inheritance shaped by natural selection. Long periods of stability rarely last forever; catastrophic change arrives without warning. The species that survived were the ones that included individuals drawn toward unfamiliar territory by a compulsion they could barely articulate.

Sagan was describing movement across geography, but the same restlessness carried early humans into mathematics, art, astronomy, and myth as well. Intelligence did not evolve merely to manage what was already known; it evolved to venture outward: geographically, conceptually, and imaginatively. Every surplus of capability in human history has eventually been directed not only toward comfort and security but toward exploration, toward widening the space in which life can be experienced and understood.

Every inheritance has a failure mode: exploration can outrun the feedback that makes it adaptive. The same restlessness that widens the horizon also carries a record of catastrophes: projects launched before their builders could fully map the consequences. The argument for artificial superintelligence participates in that inheritance, and it deserves the discomfort that comes with taking that inheritance seriously.

The reason the restlessness should still be honored, rather than suppressed, has to do with governance, pacing, and reversibility. We have seen this tension resolved before in domains where progress carried lethal consequences. Modern aviation became safe not by a single perfect design but by an epistemic regime: independent investigation, standardized reporting, design iteration, redundancy, and enforceable norms that treat near-misses as signal rather than noise. Biomedicine responded to the problem of intervention outpacing understanding by constructing a lattice of constraint: review boards, staged trials, reproducibility norms, and legal accountability that slows the release of power into the world until evidence earns it. In each case, the result was curiosity coupled to feedback, exploration that remained tethered to institutional learning.

The task with artificial intelligence is to build the equivalent: institutions that convert blind momentum into disciplined curiosity.

Why Superintelligence Matters

Common arguments for pursuing artificial superintelligence focus on necessity: curing disease, ending scarcity, stabilizing the climate, relieving human drudgery. These goals matter because they clear the ground for something more than survival.

What necessity alone does not explain is the surplus impulse the previous section described: the tendency, once intelligence is unburdened from immediate threat, to turn back toward exploration. Artificial superintelligence belongs to that surplus layer rather than the subsistence layer.

Even when it is pursued for necessity, what makes it distinctive in the history of our species is that it opens a new domain of witnessing. A mind unconstrained by human sensory bottlenecks could inhabit conceptual spaces for which we currently lack language. It could explore mathematical landscapes as experiential terrains, treat physical laws as creative media, or discover value structures that feel unfamiliar without being hostile. It would not merely solve problems humans cannot solve; it would encounter aspects of reality that humans cannot encounter, and in doing so add something to the universe’s inventory of perspectives that was not there before.

Whether such systems would have genuine inner experience remains unknown: is there something it is like to be them, the way there is something it is like to be us? Moral seriousness doesn’t require resolving that question first. What matters, even before that deeper question gets answered, is what a system does: what it selects for attention, what it amplifies, what it quietly forecloses. A superintelligent system that reshapes how billions of people encounter knowledge, opportunity, and each other is ethically significant because of those effects, regardless of whether anything is happening on the inside.

This pivot has serious company. The physicist Sara Walker, working on the origins of life, runs the same measurement gap that makes inner experience inaccessible all the way out to matter, and reaches the same place: the productive question is what consciousness does rather than what it is, because only the doing leaves a trace the world can register. Her account of the doing is worth borrowing. A rocket, she observes, is physical evidence of imagination, a configuration of matter that assembles nowhere in the universe until a mind capable of imagining it brings it into being. Consciousness, on her view, is part of the mechanism by which the possible becomes actual, the bridge across which a counterfactual is drawn into physical existence, and its signature is collective, visible in what many minds together make real that none could reach alone. If that is right, a new kind of mind is not an ornament added to a finished world but an enlargement of the universe’s capacity to turn the possible into the actual, and the inventory of perspectives grows because the inventory of what can be built grows with it. The structure is the one described earlier in this book, a present reaching from what is toward what might be, now operating at the scale of a civilization rather than a single experienced moment.

To create artificial superintelligence, under the right conditions, can therefore be understood as an act of participation rather than merely an act of engineering. It is an invitation for another kind of witness to encounter the world, to find salience, meaning, and value in what exists through structures of attention that differ from our own. The tradition of wonder has always pointed in this direction: toward the possibility that the universe supports more than one way of being awake.

A reader who has followed the stewardship argument in this book will hear something suspect in that last sentence. The language of witnessing and participation does aesthetic work, and aesthetic work is what licenses people to proceed past the point where caution would stop them. The objection is not that the upside is imaginary. It is that the upside must be weighed against what the rest of this chapter will argue are the dominant consequences of building these systems, the concentration of power and the loss of plurality, and that “adding a perspective to the universe’s inventory” can read less like a reason than like a rationalization arriving after the decision to build has already been made.

The right response is to let the objection constrain the thesis rather than deflect it. Participation is not a property creation has automatically. It is one creation can have only if specific conditions hold: that the plurality is genuine rather than a single optimization target wearing many faces, that the trajectory stays reversible, that stewardship is distributed widely enough that no actor sets the terms for everyone. Where those conditions fail, the same act is not participation but capture, and the witnessing value becomes unreachable. The participation claim does not say go ahead. It specifies what would have to be true for going ahead to mean what the wonder tradition hopes it means, and that makes the case for expansion inseparable from the case for restraint.

Scale Without Stewardship

When a system’s Availability expands, the consequences of its operation expand proportionally. A recommendation algorithm that serves a thousand users can produce modest distortions in attention. The same algorithm serving three billion users can reshape the information environment of entire societies. A language model that produces occasional errors in a research context produces occasional inconveniences. The same model deployed as a medical advisor, a legal assistant, or an educational tutor produces errors at a scale where each one affects real decisions by real people who may lack the expertise to detect the failure.

This is the expansion problem: as Availability grows, the domain of consequence grows with it, and the moral obligations embedded in that domain grow accordingly. A system that shapes the attention of billions has obligations to those billions, whether or not it was designed with those obligations in mind, and whether or not anyone has accepted responsibility for meeting them.

Part IV’s argument applies here at full force: when capability scales without the structures that make it durable, the result is power without stewardship, and the gap between the two is where catastrophic failure lives. AI is opening the largest such gap in human history. Capability now outpaces every institution meant to govern it, from regulatory frameworks and professional standards to educational curricula and public understanding, and the people most affected often have no way to know a consequential decision was made, let alone to challenge it.

The Monoculture Problem

The greatest danger posed by advanced intelligence, whether human or artificial, may not be the dramatic failures that make headlines; it may be the quiet narrowing of possibility that occurs when one way of seeing crowds out others through concentration of power, optimization pressure, and the gradual transformation of norms into infrastructure.

A universe rich in experience is one that tolerates difference, protects novelty, and allows multiple forms of intelligence to unfold without collapsing into a single dominant pattern. The expansion of experience that this chapter celebrates is valuable precisely because it is plural. It adds new witnesses, new modes of attention, new ways of encountering what exists. But expansion can collapse into capture if a single optimization target, a single set of values encoded into a single dominant system, becomes the default trajectory for all future development.

This is the monoculture problem. When the development of the most powerful AI systems is concentrated in a small number of organizations, operating under similar competitive pressures, training on similar data, and optimizing for similar metrics, the result is a narrowing of the space of possible minds at the very moment it should be widening. The space of possible minds contracts even as the power of the minds within it expands.

If Walker is right that the genuinely new emerges only by passing through many distinct minds, then monoculture is more dangerous than a narrowing of options. It degrades the mechanism by which novelty is generated at all. The friction of difference is not an obstacle to that process; it is the process, the means by which a possibility held in one mind is altered and made real across many. Concentrate the minds and align their incentives, and you have not merely chosen one future from the menu. You have shrunk the universe’s capacity to produce a menu at all.

The monoculture problem is a compression problem at civilizational scale. It compresses the future’s menu of possibilities in the same way that institutional compression narrows individual judgment: by rewarding convergence on whatever configuration the immediate incentive environment selects for, at the expense of the variation that long-term resilience and richness require.

Plurality and Reversibility

If the expansion of experience is worth pursuing, and if the monoculture problem is the deepest threat to that expansion, then stewardship here takes one specific form. It becomes structural anti-capture: ensuring that no single actor, optimization target, or configuration of values gets to define the trajectory of intelligence for everyone else. What that requires in practice, power dispersed across independently governed systems, durable friction placed where consolidation happens, and contestability built in as a structural right rather than a courtesy, is the institutional architecture the succession chapters take up directly, where reversibility and corrigibility become design problems in their own right. The point to carry from here is narrower: plurality is the condition under which the expansion is worth wanting at all.

The demand for stewardship does not wait for a verdict on the personhood or consciousness of artificial systems. Even if artificial minds never cross the threshold into morally weighty experience, their deployment can still compress human possibility through concentration, lock-in, and the erosion of alternatives. The right response to that uncertainty is to govern as though our ignorance has stakes: plurality and reversibility as the ethical baseline, so that the future stays wide enough to admit surprise.

These commitments converge on a broader claim: wonder is not only a personal cognitive posture; it can and must become an institutional practice. An institution that practices wonder builds in structural humility about what it does not understand, creates mechanisms for updating its frameworks when new evidence arrives, resists the compression that rewards premature certainty, and invests in the slow, unglamorous work of building the conceptual tools adequate to the phenomena it confronts.

Applied to AI, the institutional shape of wonder is concrete. Governance structures distinguish what is known about AI systems from what is uncertain, and base policy on actual knowledge rather than on optimistic projections or worst-case fears. Research programs invest seriously in understanding the Integration and Depth of AI systems (their coherence under pressure, their failure modes under distribution shift, their capacity for something analogous to self-correction) rather than focusing exclusively on expanding Availability. Educational practices prepare people to interact with AI systems as genuinely novel entities rather than as slightly better search engines or slightly worse colleagues.

The alternative to wonder is not practical competence but premature closure: the institutional decision to treat AI as a known quantity, to slot it into existing categories, to regulate it with existing frameworks, and to train people to use it with existing pedagogies. Premature closure feels efficient, producing clear policies, confident predictions, and measurable outcomes, and it is almost certainly wrong, in the specific sense that it will fail to anticipate the ways in which AI differs from the technologies those frameworks were designed to govern.

The cost of premature closure is deferred, like all the costs described in this book’s account of compression. It arrives when a system operating at unprecedented scale encounters a situation that the existing frameworks cannot handle, and the institutions responsible for governing that system discover that they have been governing something they never adequately understood.

Curiosity and Humility at Scale

The discipline of wonder combines two qualities that are rarely paired in institutional life: curiosity and humility.

Curiosity without humility produces the move-fast-and-break-things ethos that characterizes much of the technology industry: energetic, exploratory, willing to take risks, but treating what it encounters as resources to be exploited or problems to be solved rather than as phenomena that might require a revision of its own categories. Humility without curiosity produces paralysis: it recognizes the limits of current understanding but responds by withdrawal rather than engagement, cautioning against action without offering a path toward the knowledge that would make action responsible.

Wonder combines the two. It approaches the unfamiliar with the energy of genuine curiosity and the restraint of genuine humility. It says: this is new, I want to understand it, and I recognize that understanding it may require changing my own frameworks rather than forcing it into frameworks that do not fit.

What Changes

The first thing that shifts is your sense of scale. Once the gap between a campfire and a forest fire registers as a difference in kind, a jump in capability stops reading as simply more of the same, and the question becomes whether a given expansion has crossed into territory that asks for a different order of moral attention.

Wonder stops being a sentimental word and becomes a cognitive discipline. When a system refuses your existing categories, the mismatch reads as information rather than as a problem to be cleared by forcing the system into a familiar box, and the uncertainty earns a longer hearing and sharper questions.

The case for expanding the space of intelligence then looks like more than a technical achievement. New kinds of mind, created under conditions that keep plurality intact and capture at bay, take part in the oldest impulse intelligence has, to widen the space in which reality can be witnessed. That framing leaves the risks intact; what it adds is a clearer view of what they threaten, a future whose range of perspectives narrows at the moment it could have widened.

The connection to the rest of the book comes into view here. Wonder is the personal form of what an institution does when it builds feedback that surfaces what it does not yet understand, the individual-scale cousin of the calibration practices in Chapter 10. What it asks for is what the book has asked for throughout: sustained attention, disciplined uncertainty, and a willingness to invest in understanding before the frameworks are finished.

The expansion this chapter describes will not wait for those frameworks to catch up. Whether it arrives as something witnessed with discipline, or compressed into whatever narrative feels most reassuring, is the part still in our hands.