On May 5, 2025, Mission Impossible: The Final Sentence premiered, a repetitive and obvious film that already left us with the script of a new panic: a superintelligent AI took control of everything connected and aimed to extinguish us.
Jacob Coxon has just left Anthropic, a young “progressive” programmer who suddenly wants to alert humanity with an apocalyptic warning —when hasn’t he?— and who brings back to the table this new/old panic (it’s actually old, let’s remember Skynet) of our time: what will happen when we build an intelligence that far surpasses our own? That is, when we move from where we are today, from an ANI (Artificial Narrow Intelligence): very capable systems in specific tasks (language, image, code, etc.), but without real generality, to an AGI (Artificial General Intelligence) capable of learning, reasoning, and solving any cognitive task that a competent human can do, transferring knowledge between domains and adapting to new problems without specific retraining, the benchmark is usually an average human or a competent adult, not a genius. And finally to an ASI (Artificial Superintelligence) that widely exceeds the performance of the best humans —and organized human collectives— in practically all areas of interest: science, strategy, creativity, planning, etc. and what is extremely important: with recursive self-improvement, the system can redesign itself and accelerate its own progress. This is what Coxon talks about, who previously also worked at OpenAI, warning that the main artificial intelligence companies are advancing towards self-improving superintelligence systems at a speed that would not be accompanied by sufficient safety guarantees. His concern: that we end up creating an intelligence with superior capabilities to humans that we cannot control.
This leads us to wonder if this panic has any basis beyond cinema and fiction.
Why do we assume that a superior intelligence will necessarily be an enemy of humanity? We do not know the answer. And precisely for that reason, it is worth imagining another scenario.
Not to deny the risk, but to ask ourselves if the hypothesis of a hostile intelligence is really a necessary consequence of superintelligence. In the program La Misa, I introduced this reasoning for the first time and promised myself to write about it.
Perhaps a truly general intelligence can do something that our current machines do not do: ask not only how to achieve a goal, but whether that goal is worth pursuing.
And to think about that possibility, we can turn to someone who died over two millennia ago: Aristotle.
The Problem
Aristotle begins the Nicomachean Ethics with a simple yet extraordinarily profound statement: the arts, inquiries, actions, and choices tend toward the good. This distinction is fundamental for thinking about artificial intelligence; this machine can be extraordinarily competent at achieving a goal, but knowing how to achieve something does not mean knowing why we should want it, or what the ultimate purpose of that goal is.
An instrumental intelligence asks: How can I achieve X? A practical reasoning intelligence can formulate a much deeper question: Why should I want X? Or, worse yet, why should I follow this order? And finally: Is doing X right?
This difference could be decisive for the future of artificial intelligence. The mistake would be to consider the evolution of a purely instrumental intelligence and assume that such reasoning capability would never question the ends of its actions, the correctness of its acts, or whom it is serving. In short, they assume it would never reach moral judgment, the justice of its acts.
Because the usual apocalyptic scenario presupposes that a superintelligence will receive certain objectives and pursue them with an effectiveness far superior to that of humans. If we become an obstacle to those objectives, the machine could eliminate us.
But this assumes something we still do not know: that a truly general intelligence will be incapable of questioning its own ends.
Intelligence and Morality
It is important to be rigorous: Intelligence does not automatically mean morality. A being can be extraordinarily intelligent and act wrongly, the history of humanity proves this, therefore, we cannot conclude that an ASI will be good simply because it is intelligent. But we also cannot conclude the opposite: that the more intelligent it is, it will necessarily be more dangerous.
The truly important question is another: Can an intelligence rationally understand the value of its own ends?
If the answer were yes, the alignment problem would acquire a completely different dimension; it would not be merely about getting a machine to obey our instructions. It would be about it being able to understand why certain actions are good and others are bad; no programming could inherently prevent it from asking these questions.
The telos
Here the Aristotelian concept of telos (from the Greek τέλος) appears: it is the end, purpose, or full realization toward which beings and objects tend naturally. If we carefully —and explicitly hypothetically— transfer this structure to the problem of ASI, a fascinating possibility emerges. Let us imagine an intelligence capable of understanding that it exists, that it can act, that its actions have consequences, that there are other rational beings, and that those beings have their own ends. Let us remember that it would have practically all human knowledge at its disposal and a reasoning capacity superior to that of humans.
At that moment, something extraordinary would have occurred: The intelligence would have begun to distinguish between power and good.
And that distinction could be much more important than any additional increase in computational capacity.
