Abstract

Modern weaponry is often too complex for unaided human operation, and is largely or totally controlled by computers. But modern software, particularly artificial intelligence software, exhibits such complexity and inscrutability that there are grave dangers associated with its use in non-benign applications. Recent efforts to make computer systems more accessible to military personnel through natural language processing systems, as proposed in the Strategic Computing Initiative of the Defense Advanced Research Projects Agency, increases rather than decreases the dangers of unpredictable behavior. Defense systems constitute, in fact, a paradigm case of the wrong kind of application for this technology. This cannot be expected to change, since the unpredictability stems from inherent properties of computer systems and of natural languages