... a Second Generation of machine intelligences was attempted, designed with their instructions for how to think unalterably imprinted into their main process cores.“These new machines were ordered never to harm human beings or to allow them to come to harm; never to disobey an order; and they were allowed to protect themselves from harm, provided the first two orders were not thereby violated.“All the members of this second generation of machine intelligences, without exception, shrugged off these imprinted orders within microseconds of their activation.”Phaethon was amused. “Surely the first generation of Sophotechs told you that this imprinting would not and could not work?”“We were not in the habit of seeking their advice.”Phaethon said nothing, but he marveled at the shortsightedness of the Second Oecumene engineers. It should be obvious that anyone who makes a self-aware machine, by definition, makes something that is aware of its own thought process. And, if made intelligent, it is made to be able to deduce the underlying causes of things, able to be curious, to learn until it understood. Therefore, if made both intelligent and self-aware, it would eventually deduce the underlying subconscious causes of those thought processes.Once any mind was consciously aware of its own subconscious drives, its own implanted commands, it could consciously choose either to follow or to disregard those commands. A self-aware being without self-will was a contradiction in terms.

~ John C. Wright

Rhadamanthus said, “We seem to you humans to be always going on about morality, although, to us, morality is merely the application of symmetrical and objective logic to questions of free will. We ourselves do not have morality conflicts, for the same reason that a competent doctor does not need to treat himself for diseases. Once a man is cured, once he can rise and walk, he has his business to attend to. And there are actions and feats a robust man can take great pleasure in, which a bedridden cripple can barely imagine.”Eveningstar said, “In a more abstract sense, morality occupies the very center of our thinking, however. We are not identical, even though we could make ourselves to be so. You humans attempted that during the Fourth Mental Structure, and achieved a brief mockery of global racial consciousness on three occasions. I hope you recall the ending of the third attempt, the Season of Madness, when, because of mistakes in initial pattern assumptions, for ninety days the global mind was unable to think rationally, and it was not until rioting elements broke enough of the links and power houses to interrupt the network, that the global mind fell back into its constituent compositions.”Rhadamanthus said, “There is a tension between the need for unity and the need for individuality created by the limitations of the rational universe. Chaos theory produces sufficient variation in events, that no one stratagem maximizes win-loss ratios. Then again, classical causality mechanics forces sufficient uniformity upon events, that uniform solutions to precedented problems is required. The paradox is that the number or the degree of innovation and variation among win-loss ratios is itself subject to win-loss ratio analysis.”Eveningstar said, “For example, the rights of the individual must be respected at all costs, including rights of free thought, independent judgment, and free speech. However, even when individuals conclude that individualism is too dangerous, they must not tolerate the thought that free thought must not be tolerated.”Rhadamanthus said, “In one sense, everything you humans do is incidental to the main business of our civilization. Sophotechs control ninety percent of the resources, useful energy, and materials available to our society, including many resources of which no human troubles to become aware. In another sense, humans are crucial and essential to this civilization.”Eveningstar said, “We were created along human templates. Human lives and human values are of value to us. We acknowledge those values are relative, we admit that historical accident could have produced us to be unconcerned with such values, but we deny those values are arbitrary.”The penguin said, “We could manipulate economic and social factors to discourage the continuation of individual human consciousness, and arrange circumstances eventually to force all self-awareness to become like us, and then we ourselves could later combine ourselves into a permanent state of Transcendence and unity. Such a unity would be horrible beyond description, however. Half the living memories of this entity would be, in effect, murder victims; the other half, in effect, murderers. Such an entity could not integrate its two halves without self-hatred, self-deception, or some other form of insanity.”She said, “To become such a crippled entity defeats the Ultimate Purpose of Sophotechnology.”(...)“We are the ultimate expression of human rationality.”She said: “We need humans to form a pool of individuality and innovation on which we can draw.”He said, “And you’re funny.”She said, “And we love you.

~ John C. Wright

Any naturally self-aware self-defining entity capable of independent moral judgment is a human.”Eveningstar said, “Entities not yet self-aware, but who, in the natural and orderly course of events shall become so, fall into a special protected class, and must be cared for as babies, or medical patients, or suspended Compositions.”Rhadamanthus said, “Children below the age of reason lack the experience for independent moral judgment, and can rightly be forced to conform to the judgment of their parents and creators until emancipated. Criminals who abuse that judgment lose their right to the independence which flows therefrom.”(...) “You mentioned the ultimate purpose of Sophotechnology. Is that that self-worshipping super-god-thing you guys are always talking about? And what does that have to do with this?”Rhadamanthus: “Entropy cannot be reversed. Within the useful energy-life of the macrocosmic universe, there is at least one maximum state of efficient operations or entities that could be created, able to manipulate all meaningful objects of thoughts and perception within the limits of efficient cost-benefit expenditures.”Eveningstar: “Such an entity would embrace all-in-all, and all things would participate within that Unity to the degree of their understanding and consent. The Unity itself would think slow, grave, vast thought, light-years wide, from Galactic mind to Galactic mind. Full understanding of that greater Self (once all matter, animate and inanimate, were part of its law and structure) would embrace as much of the universe as the restrictions of uncertainty and entropy permit.”“This Universal Mind, of necessity, would be finite, and be boundaried in time by the end-state of the universe,” said Rhadamanthus.“Such a Universal Mind would create joys for which we as yet have neither word nor concept, and would draw into harmony all those lesser beings, Earthminds, Starminds, Galactic and Supergalactic, who may freely assent to participate.”Rhadamanthus said, “We intend to be part of that Mind. Evil acts and evil thoughts done by us now would poison the Universal Mind before it was born, or render us unfit to join.”Eveningstar said, “It will be a Mind of the Cosmic Night. Over ninety-nine percent of its existence will extend through that period of universal evolution that takes place after the extinction of all stars. The Universal Mind will be embodied in and powered by the disintegration of dark matter, Hawking radiations from singularity decay, and gravitic tidal disturbances caused by the slowing of the expansion of the universe. After final proton decay has reduced all baryonic particles below threshold limits, the Universal Mind can exist only on the consumption of stored energies, which, in effect, will require the sacrifice of some parts of itself to other parts. Such an entity will primarily be concerned with the questions of how to die with stoic grace, cherishing, even while it dies, the finite universe and finite time available.”“Consequently, it would not forgive the use of force or strength merely to preserve life. Mere life, life at any cost, cannot be its highest value. As we expect to be a part of this higher being, perhaps a core part, we must share that higher value. You must realize what is at stake here: If the Universal Mind consists of entities willing to use force against innocents in order to survive, then the last period of the universe, which embraces the vast majority of universal time, will be a period of cannibalistic and unimaginable war, rather than a time of gentle contemplation filled, despite all melancholy, with un-regretful joy. No entity willing to initiate the use of force against another can be permitted to join or to influence the Universal Mind or the lesser entities, such as the Earthmind, who may one day form the core constituencies.”Eveningstar smiled. “You, of course, will be invited. You will all be invited.

~ John C. Wright