Abstract
One recent priority of the U.S. government is developing autonomous robotic systems. The U.S. Army has funded research to design a metric of evil to support military commanders with ethical decision-making and, in the future, allow robotic military systems to make autonomous ethical judgments. We use this particular project as a case study for efforts that seek to frame morality in quantitative terms. We report preliminary results from this research, describing the assumptions and limitations of a program that assesses the relative evil of two courses of action. We compare this program to other attempts to simulate ethical decision-making, assess possibilities for overcoming the trade-off between input simplification and output reliability, and discuss the responsibilities of users and designers in implementing such programs. We conclude by discussing the implications that this project highlights for the successes and challenges of developing automated mechanisms for ethical decision making.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anderson SL, Anderson M (2009) How machines can advance ethics. Philosophy Now 72:17–19
Anderson EF, Frith KH, Caspers B (2011) Linking economics and quality: developing an evidence-based nurse staffing tool. Nurs Admin Q 35:53–60
Arkin RC (2007) Governing lethal behavior: embedding ethics in a hybrid deliberative/reactive robot architecture. U.S. Army Research Office Technical Report GIT-GVU-07-11
Baron-Cohen S (2003) The science of evil: on empathy and the origins of cruelty. Basic Books, New York
Beauchamp TL, Childress JF (1979) Principles of biomedical ethics. Oxford University Press, New York
Bell MZ (1985) Why expert systems fail. J Oper Res Soc 36:613–619
Belmont Report (1979) Belmont report—ethical principles and guidelines for the protection of human subjects of research. Available online at http://ohsr.od.nih.gov/guidelines/belmont.html
Center for Modeling, Simulation, and Analysis, and Center for the Management of Science and Technology [CMSA/CMOST] (2010) Developing and calibrating a quantitative metric of evil for use in course of action analysis. Final technical report. AMREDC, SSDD
Dawes RM (1971) A case study of graduate admissions: applications of three principles of human decision making. Am Psychol 26:180–188
Dawes RM (1979) The robust beauty of improper linear models in decision making. Am Psychol 34:571–582
Dawes RM, Corrigan B (1974) Linear models in decision making. Psychol Bull 81:93–106
Dawes RM, Faust D, Meehl PE (1989) Clinical versus actuarial judgment. Science 243:1668–1674
DeMarco JP (2000) Principalism and moral dilemmas: a new principle. J Med Ethics 31:101–105
Dixit AK, Skeath S (2004) Games of strategy, 2nd edn. W.W. Norton & Company, Inc, New York
Fan R (1997) Self-determination vs. family-determination: two incommensurable principles of autonomy: a report from East Asia. Bioethics 11:309–322
Finn P (2011) A future for drones: automated killing. The Washington Post. Retrieved 26 June 2012, from http://www.washingtonpost.com/national/national-security/a-future-for-drones-automated-killing/2011/09/15/gIQAVy9mgK_story.html
Gips J (1995) Towards the ethical robot. In: Ford K, Glymour C, Hayes P (eds) Android epistemology. MIT Press, Cambridge, pp 243–252
Goodwin P, Wright G (2004) Decision analysis for management judgment, 3rd edn. Wiley, West Sussex
Holm S (1995) Not just autonomy—the principles of American biomedical ethics. J Med Ethics 21:332–338
McLaren BM (2003) Extensionally defining principles and cases: an AI model. Artif Intell J 150:145–181
McLaren BM (2005) Lessons in machine ethics from the perspective of two computational models of ethical reasoning. Papers from the AAAI Fall symposium, technical report FS-05-06, pp 70–77
McLaren BM (2006) Computational models of ethical reasoning: challenges, initial steps, and future directions. IEEE Intell Syst 6:2–10
Mental Health Advisory Team (MHAT) IV, Office of the Surgeon General (2006) Final report: operation Iraqi freedom 05-07
Mitchell TO (1969) Observer’s hostility as a factor in judgments of behavior in hostility-provoking situations. Ph.D. Dissertation
Morrow L (2003) Evil: an investigation. Basic Books, New York
Ornstein SM (1987) Computers in battle: a human overview. In: Bellin DB, Chapman G (eds) Computers in battle—will they work?. Harcourt Brace Jovanovich, Inc., Orlando, pp 1–43
Rogerson MD, Gottlieb MC, Handelsman MH, Knapp S, Younggren J (2011) Nonrational processes in ethical decision making. Am Psychol 66:614–623
Rozoff R (2010) Decade of the drone: America’s aerial assassins. Global Res. Retrieved 26 June 2012, from http://globalresearch.ca/index.php?context=va&aid=18027
Sharkey N (2008a) The ethical frontiers of robotics. Science 322:1800–1801
Sharkey N (2008b) Grounds for discrimination: autonomous robot weapons. Rusi Defense Syst 11:86–89
Sullins JP (2010) RoboWarfare: can robots be more ethical than humans on the battlefield? Ethics Inf Technol 12:263–275
Tackett GB (2009) Framework for quantification of evil as a metric for course of action (CoA) analysis. Draft technical report. AMRDEC, RDECOM
U.S. Department of Defense (2007) Unmanned systems roadmap 2007–2032
Welner M (2007) Classifying crimes by severity: from aggravators to depravity. In: Douglass J, Ressler R, Burgess A (eds) A crime classification manual. Jossey-Bass, San Francisco, pp 55–72
Zimbardo PG (2004) A situationist perspective on the psychology of evil: understanding how good people are transformed into perpetrators. In: Miller AG (ed) The social psychology of good and evil. Guilford Press, New York, pp 21–50
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Reed, G.S., Jones, N. Toward Modeling and Automating Ethical Decision Making: Design, Implementation, Limitations, and Responsibilities. Topoi 32, 237–250 (2013). https://doi.org/10.1007/s11245-012-9127-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11245-012-9127-x