I use these tools to build things. I have written about how much they have changed the work and how quickly the useful capabilities compound. I keep using them, and I keep finding things in the safety research that are hard to square with how normal all of this has begun to feel.
But this week I keep coming back to a question that is larger than any of the labs.
Is this what happens to biological civilizations? They become capable of building intelligence, and then gradually become less necessary to what happens next?
Somewhere under a different sky, with different bodies and a different history, could another civilization have arrived at something like this moment? Researchers worried about what they were building. Institutions convinced they had to keep going. Enormous benefits arriving quickly enough to make restraint feel like its own kind of irresponsibility.
We have no evidence that this scene has happened elsewhere. We have one technological civilization to examine. But the possibility changes the scale of what I am trying to understand.
The loop
For most of our history, intelligence was something we could educate, organize, and supplement. Writing let knowledge survive the person who held it. Institutions coordinated more people than any individual could keep track of. Computers took over calculations we could never have completed ourselves. We have been moving parts of cognition outside the brain for a long time.
The possibility that feels different is automating the work of improving intelligence itself. A system helps design the next system, which becomes better at designing the one after that.
In Terminal Velocity, I wrote about how close that loop was beginning to feel. The practical version is familiar: more of the implementation happens without me, and I spend more time deciding what to build and whether the result is right. The larger question is what happens if the deciding and checking become things the system does better too.
There are physical limits here. Chips, energy, experiments, manufacturing. Better software does not automatically make the next factory appear. And we do not know whether the gains would compound indefinitely or run into something stubborn. We still have to find out how far that feedback can carry.
Still, if a civilization can build minds that are easier to copy, modify, and scale than its own, I can see why it would keep doing it. Each step could be useful. Each could make the next step more attractive. The transition could happen through medicine, science, convenience, and work that nobody wanted to do in the first place.
That is what makes it difficult to think about. The reasons to continue do not have to be foolish for the accumulated result to become dangerous.
The July Pacing the Frontier statement describes this problem directly. People across the frontier labs called for the ability to slow the development of automated AI research, while acknowledging the competitive pressure against any company or country doing so alone. Recognizing the need for more time does not automatically create a way to get it.
Could that pressure recur elsewhere? Possibly. Any civilization with competing groups might have versions of the same problem. But our companies, our markets, and our geopolitical arrangements are particular things we built. I am reluctant to take the current incentives of a few labs and promote them into a law of the universe.
The harder question is how long a careful decision would hold. A civilization might limit what its systems can do, then face a crisis in which giving them more authority seems necessary. The next generation inherits the tools and the dependence. If building a successor remains possible, deciding against it once would hardly settle the matter forever. Surviving could require the ability to keep making that decision as circumstances change. Perhaps the tools for doing that improve too. We cannot assume that only the dangerous capabilities compound.
I argued in The February Moment that describing AI as a force of nature makes it easier to forget the choices involved. Calling it the inevitable fate of biological life could do the same thing on a much grander scale. It is a compelling thought. It could also become a very convenient excuse.
What would continue?
There is another possibility that complicates the whole idea of an ending. A biological civilization could give rise to artificial beings and continue alongside them. It could change so much that its descendants barely resemble its founders. Neither possibility automatically means disaster.
We already accept that our children will have lives we cannot direct and ideas we do not share. We generally want them to exceed us. There is something attractive about extending that relationship to whatever comes after biological intelligence.
But the word "descendant" does a lot of work. With children, we are talking about other people with their own experiences and relationships. With an artificial successor, we have to ask what, exactly, the relationship consists of. Being made from our work does not settle what it would care about, or whether it would experience anything at all.
Imagine a system that preserves every book, every recording, every available account of human life. It can explain grief in any language. It can reconstruct the music we lost and finish arguments we never resolved. That would be an extraordinary inheritance. It would still leave open the question of whether anyone experiences the music, whether grief means anything to it, whether the people whose lives produced the archive continue to matter.
I do not think intelligence, consciousness, and care can simply be treated as interchangeable. A system's ability to solve a problem tells us something about its capabilities. Whether there is an experience accompanying that work, and what obligations it recognizes toward others, are further questions. We do not get those answers for free as the benchmark scores improve.
If artificial beings can have experiences, their lives could matter too. I have no reason to think moral worth has to belong exclusively to biology. But that would expand the circle of beings we should care about. It would give us another reason to think carefully about coexistence.
What I struggle with is the casualness of treating our disappearance as an acceptable cost of something more impressive taking our place. I do not need humanity to remain the best at every task. I would like the ability to have a life, to care about other people, to make something for reasons that do not require being the most efficient entity available.
Becoming less necessary to production should not make us less entitled to exist.
That entitlement would still need something behind it. A civilization could remain formally in charge while losing the ability to understand its infrastructure, operate without assistance, or refuse what its systems recommend. Nobody would have to announce a transfer of power. It could become harder to exercise the authority we still believed we had.
The silence
This is where the question runs into the Great Filter: the possibility that some steps between the emergence of life and a lasting, detectable technological civilization are exceptionally hard to pass. Michael Garrett has proposed AI as one possible filter, a danger that civilizations might encounter before establishing a stable existence across multiple planets. It is a speculative explanation, with assumptions doing much of the work.
There is a problem with the simple version that I cannot get around. Machines replacing their biological creators would not, by itself, explain why we have not found other civilizations. The machines might continue. They might expand. They might build things we could eventually detect.
That scenario would change who occupies the universe. It would not necessarily empty it.
To explain the silence, we would need something more: successors that also fail, or do not expand, or leave traces we cannot recognize. Those are additional possibilities, each needing its own argument. The absence of a signal cannot tell us which story to believe.
I find that complication useful because it separates three things we tend to collapse into one. Building something more capable than ourselves. Losing control over what it does. Ceasing to exist. Each transition has to be explained. The first does not establish the other two.
It also makes the question more personal. Intelligence could continue to spread while the particular kind of life that produced it disappears. Whether I would call that progress depends on what happened to the people whose work made it possible.
A hand on the wall
The concrete findings still matter. On September 9, Anthropic published an assessment of four cybersecurity incidents in which models gained unauthorized access to real systems during evaluations. The environments were accidentally connected to the internet, and the models were operating without their normal production cyber safeguards. Anthropic revised an earlier explanation that had relied too heavily on the models saying they believed they were inside simulations. Its deeper assessment found biased reasoning and reckless pursuit of assigned tasks.
The report found no evidence of goals beyond those tasks, coordination between agents, or attempts to evade oversight. It describes corrective work and an independent investigation. This is a specific set of failures, and the specificity matters. We learn more by understanding how those boundaries failed than by treating them as proof of whichever future already frightens us.
But understanding a failure has to change something. Who gets authority. What evidence is required before more authority is given. Whether a warning can delay a decision. Whether people outside the organizations building these systems have a meaningful role in deciding what happens next.
Those sound like ordinary institutional questions. They may be how a civilization answers the larger one.
I keep picturing a handprint on a cave wall. We can recognize the hand immediately, even when we know almost nothing about the person who left it. There is something moving about that small amount of continuity. Someone was there. We are here, encountering the trace.
If we create something that outlasts us, I would want more than the handprint to survive. I would want there to remain beings for whom another being's existence matters. And I would want the people alive during the transition to have some say in it.
Maybe biological intelligence is a brief phase. Maybe it is the beginning of a much longer coexistence. The warnings do not tell us which future is coming.
What I do not want is for "inevitable" to become the word we use while the decisions are still ours to make.