Preprint from Nanotechnology Perceptions March 27, 2006. Published on KurzweilAI.net March 24, 2006.

The Center for Responsible Nanotechnology (CRN) has created a series of new research papers in which industry experts predict profound impacts of nanotechnology on society. The first set of 11 of these original essays by members of CRN’s Global Task Force will appear in the March 27 issue of the journal Nanotechnology Perceptions. KurzweilAI.net will syndicate these essays over that week. In this preview, Chris Phoenix, CRN’s director of research, presents the challenge of how to deal with possible unintended consequences of molecular manufacturing.

Humans are good at pushing limits. We can survive in scorching deserts and in the frozen Arctic. We have flown faster than sound and sent robots to other planets. We have managed, with help from fossil fuels, to feed six billion people. Even before we had motors and technological navigation equipment, some of us were able to find and colonize islands in the middle of the vast Pacific Ocean.

Pushing limits has its darker side as well. Humans are not good at respecting each other’s rights; the ferocity of the Mongol hordes remains legendary, and the 20^th century provides multiple examples of state-sponsored mass murder. Natural limits frequently are pushed too far, and whole civilizations have been wiped out by environmental backlash. We are too good at justifying our disrespect of limits, and then we often become increasingly destructive as the problem becomes more acute. More than a century ago, Lord Acton warned that "absolute power corrupts absolutely." This can be restated as, "Complete lack of limits leads to unlimited destruction."

Molecular manufacturing has the potential to remove or bypass many of today’s limits. It is not far wrong to say that the most significant remaining limits will be human, and that we will be trying our hardest to bypass even those. To people with faith in humanity’s good nature and high potential, this will come as welcome news. For many who have studied history, it will be rather frightening. A near-total lack of limits could lead straight to a planet-wide dictatorship, or to any of several forms of irreversible destruction.

Many of the plans that have been proposed to deal with molecular manufacturing, by CRN and others, assume (usually implicitly) that the plan will be implemented within some bigger system, such as the rule of law. This will be problematic if molecular manufacturing is powerful enough that its users can make their own law. We cannot assume that existing world systems will continue to provide a framework in which molecular manufacturing will play out. Those systems that adopt the new technology will be transformed; those that do not will be comparatively impotent. We will have to find ways for multiple actors empowered by molecular manufacturing to coexist constructively, without reliance on the stabilizing forces provided by today’s global institutions.

Any active system without limits will run off the rails. The simplest example is a reproducing population, which will indulge in exponential growth until it exhausts its resources and crashes. Another example can be found in the "excesses" of behavior that are seen in political revolutions. Humans systems need limits as much as any other system, for all that we try to overcome them.

Through all of history, the presence of limits has been a reasonable assumption. Nations were limited by other nations; populations were limited by geography, climate, or disease; and societies would sometimes be stable long enough to develop and agree on a morality that provided additional useful limits. A society that overstepped its bounds could expect to collapse or be out-competed by other societies.

It’s tempting to think that humanity has developed a new worldview—the Enlightenment—that will provide internal moral limits. However, the Enlightenment may be fading. It was supported by, and synergistic with, the brief period when people could be several times more productive using machines than by manual labor. During that period, individual people were quite valuable. However, now that we’re developing automation, people can be many times as productive (not just several times), and we don’t need all that productivity. And indeed, as abundance develops into glut, Enlightenment values and practices may be fading.

It’s tempting to think that, left to themselves, people will be generally good. History, in both microcosm and macrocosm, shows that this doesn’t work any better than Communism did. Without sufficient external limits, some people will start cheating, or choosing to violate the moral code of their society. Not only will this reduce benefits for everyone, but the ingrained human aversion to being taken advantage of will cause others to join the cheaters if they can’t prevent them. This leads to a vicious cycle, and the occasional saint won’t be enough to stop the degeneration.

It’s tempting to think that, now that we have digital computers, everything has changed and the old rules of scarcity and competition needn’t apply. As explored in CRN’s paper "Three Systems of Action," ^[i] digital data transfer can be unlimited-sum, with benefit unrelated to and far larger than the cost. But digital information does not replace existing systems or issues wholesale. And increasing Internet problems such as spam, phishing, and viruses demonstrate that domains of digital abundance and freedom cannot moderate their own behavior very well.

It’s tempting to think that an ongoing power struggle between human leaders would provide limits. But in an age of molecular manufacturing, this seems unlikely for two reasons. First, such a competition almost certainly would be unstable, winner-take-all, and end up in massive oppression: no better than simply starting out with a dictatorship. Second, the contest probably would shift quickly to computer-assisted design and attack, and that would be even worse than all-out war between mere humans, even humans assisted by molecular manufactured weapons. Civilians would probably be a major liability in such conflicts: easy to kill and requiring major resources (not to mention oppressive lifestyle changes) to defend.

Molecular manufacturing will give its wielders extreme power—certainly enough power to overcome all significant non-human limits (at least within the context of the planet; in space, there will be other limits such as scarcity of materials and speed of light). Even if the problem of cheaters could be overcome, we do not have many internal limits these days; the current trend in capitalism is to deny the desirability of all limits except those that arise from competition. What’s left?

Somehow, we have to establish a most-powerful system that limits itself and provides limits for the rest of our activities. Long ago, Eric Drexler proposed an Active Shield.^[ii] Others have proposed building an AI to govern us—though they have not explained how to build internal limits into the AI. I have proposed creating a government of people who have accepted modifications to their biochemistry to limit some of their human impulses. All of these suggestions have problems.

Open communication and accountability may supply part of the answer. David Brin has proposed "reciprocal accountability."^[iii] It’s been noted that democracies, which embody transparency and accountability, rarely have famines or go to war with each other. Communication and accountability may be able to overcome the race to the bottom that happens when humans are left to their own devices. But communication and accountability depend on creation and maintenance of the infrastructure; on continued widespread attention; and on forensic ability (being able to connect effect back to cause in order to identify perpetrators). Recent trends in US media and democracy are not encouraging; it seems people would rather see into bedrooms than boardrooms. And it’s not clear whether people’s voices will still matter to those in power once production becomes sufficiently automated that nation-scale productivity can be maintained with near-zero labor.

If we can somehow find meta-limits, then within those limits a variety of administration methods may work to optimize day-to-day life. In other words, the problem with administrative suggestions is not inherent in the suggestions themselves; it is that the suggestions rely on something else to provide limits. Without limits, nothing can be stable; with limits, wise administration will still be needed, and best practices should be researched. But perhaps the biggest problem of all will be how to develop a system of near-absolute power that will not become corrupt.

[i] http://crnano.org/systems.htm

[ii] http://www.foresight.org/EOC/EOC_Chapter_11.html#section04of05

[iii] http://davidbrin.blogspot.com/2005/09/another-pause-this-time-for-soa.html

© 2006 Chris Phoenix. Reprinted with permission.