Abstract
In this article we evaluate the BankXX program from several perspectives. BankXX is a case-based legal argument program that retrieves cases and other legal knowledge pertinent to a legal argument through a combination of heuristic search and knowledge-based indexing. The program is described in detail in a companion article in Artificial Intelligence and Law 4: 1--71, 1996. Three perspectives are used to evaluate BankXX:(1) classical information retrieval measures of precision and recall applied against a hand-coded baseline; (2) knowledge-representation and case-based reasoning, where the baseline is provided by the functionality of a well-known case-based argument program, HYPO (Ashley, 1990); and (3) search, in which the performance of BankXX run with various parameter settings, for instance, resource limits, is compared. In this article we report on an extensive series of experiments performed to evaluate the program. We also describe two additional experiments concerning(1) the program's search behavior; and (2) the use of a modified form of precision and recall based on case similarity. Finally we offer some general conclusions that might be drawn from these particular experiments.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ashley, K. D. (1990). Modeling Legal Argument: Reasoning with Cases and Hypotheticals. MIT Press: Cambridge, MA.
Birkhoff, G. (1967). Lattice Theory. American Mathematical Society: Providence, RI.
Blair, D. C. & Maron, M. E. (1985). An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System. Communications of the ACM 28(3): 289–299. March 1985.
Branting, L. K. (1991). Building Explanations from Rules and Structured Cases. International Journal of Man-Machine Studies 34: 797–837.
Eisenberg, M. A. (1982). Gilbert Law Summaries: Contracts. Harcourt Brace Jovanovich Legal and Professional Publications: New York.
Fox, J. & Clarke, M. (1991). Towards a Formalisation of Arguments in Decision Making. In AAAI Spring Symposium Series, 1991, Argument and Belief, 92–99. Palo Alto, CA.
Gardner, A. vdl. (1987). An Artificial Intelligence Approach to Legal Reasoning. MIT Press: Cambridge, MA.
Gentner, D. & Forbus, K. D. (1991). MAC/FAC: A Model of Similarity-based Retrieval. In Proceedings of The Thirteenth Annual Conference of the Cognitive Science Society, 504–509. Chicago, IL. Lawrence Erlbaum: Hillsdale, NJ.
Harmon, D. K. (1995). Overview of the Third Text REtrieval Conference (TREC-3). In Harmon, D. K. (ed.) Proceedings of the Third Text REtrieval Conference (TREC-3) National Institute of Standards and Technology Special Publication 500-225, Washington, D.C.
Perelman, C. & Olbrechts-Tyteca, L. (1969). The New Rhetoric: A Treatise on Argumentation. University of Notre Dame Press: Notre Dame, Indiana.
Rissland, E. L. (1990). Dimension-based Analysis of Hypotheticals from Supreme Court Oral Argument. In Proceedings of The Second International Conference on AI and Law, 111–120. Vancouver, BC. ACM Press: New York.
Rissland, E. L. & Ashley, K. D. (1987). A Case-Based System for Trade Secrets Law. In Proceedings of The First International Conference on Artificial Intelligence and Law, 60–66. Boston, MA. ACM Press: New York.
Rissland, E. L., Daniels, J. J., Rubinstein, Z. B. & Skalak, D. B. (1993). Case-Based Diagnostic Analysis in a Blackboard Architecture. Proceedings of the Eleventh National Conference on Artificial Intelligence, 66–72. Washington, DC. AAAI Press/MIT Press.
Rissland, E. L., Skalak, D. B. & Friedman, M. T. (1996). BankXX: Supporting Legal Arguments through Heurisitic Retrieval. Artificial Intelligence and Law: An International Journal 4: 1–71.
Rissland, E. L., Valcarce, E. M. & Ashley, K. D. (1984). Explaining and Arguing with Examples. Proceedings of the Fourth National Conference on Artificial Intelligence. Austin, TX. AAAI Press.
Salton, G. (1989). Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer. Addison-Wesley: Reading, MA.
Salton, G. & McGill, M. J. (1983). Introduction to Modern Information Retrieval. McGraw-Hill: New York.
Skalak, D. B. & Rissland, E. L. (1992). Arguments and Cases: An Inevitable Intertwining. Artificial Intelligence and Law: An International Journal 1(1), 3–48.
Toulmin, S. E. (1958). The Uses of Argument. Cambridge University Press.
van Rijsbergen, C. J. (1979). Information Retrieval, Second Edition. Butterworths: London.
Veloso, M. M. & Carbonell, J. G. (1991). Variable-Precision Case Retrieval in Analogical Problem Solving. In Proceedings of The Third Case-Based Reasoning Workshop, May 1991. Washington, D.C. Morgan Kaufmann: San Mateo, CA.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Rissland, E.L., Skalak, D.B. & Friedman, M.T. Evaluating a Legal Argument Program: The BankXX Experiments. Artificial Intelligence and Law 5, 1–74 (1997). https://doi.org/10.1023/A:1008215000938
Issue Date:
DOI: https://doi.org/10.1023/A:1008215000938