Automated legal reasoning and its application in smart contracts and automated decisions are increasingly attracting interest. In this context, ethical and legal concerns make it necessary for automated reasoners to justify in human-understandable terms the advice given. Logic Programming, specially Answer Set Programming, has a rich semantics and has been used to very concisely express complex knowledge. However, modelling discretionality to act and other vague concepts such as ambiguity cannot be expressed in top-down execution models based on Prolog, and in (...) bottom-up execution models based on ASP the justifications are incomplete and/or not scalable. We propose to use s(CASP), a top-down execution model for predicate ASP, to model vague concepts following a set of patterns. We have implemented a framework, called s(LAW), to model, reason, and justify the applicable legislation and validate it by translating (and benchmarking) a representative use case, the criteria for the admission of students in the “Comunidad de Madrid”. (shrink)

Transformer-based language models (TLMs) have widely been recognized to be a cutting-edge technology for the successful development of deep-learning-based solutions to problems and applications that require natural language processing and understanding. Like for other textual domains, TLMs have indeed pushed the state-of-the-art of AI approaches for many tasks of interest in the legal domain. Despite the first Transformer model being proposed about six years ago, there has been a rapid progress of this technology at an unprecedented rate, whereby BERT and (...) related models represent a major reference, also in the legal domain. This article provides the first systematic overview of TLM-based methods for AI-driven problems and tasks in the legal sphere. A major goal is to highlight research advances in this field so as to understand, on the one hand, how the Transformers have contributed to the success of AI in supporting legal processes, and on the other hand, what are the current limitations and opportunities for further research development. (shrink)

Accurate data annotation is essential to successfully implementing machine learning (ML) for regulatory compliance. Annotations allow organizations to train supervised ML algorithms and to adapt and audit the software they buy. The lack of annotation tools focused on regulatory data is slowing the adoption of established ML methodologies and process models, such as CRISP-DM, in various legal domains, including in regulatory compliance. This article introduces Ant, an open-source annotation software for regulatory compliance. Ant is designed to adapt to complex organizational (...) processes and enable compliance experts to be in control of ML projects. By drawing on Business Process Modeling (BPM), we show that Ant can contribute to lift major technical bottlenecks to effectively implement regulatory compliance through software, such as the access to multiple sources of heterogeneous data and the integration of process complexities in the ML pipeline. We provide empirical data to validate the performance of Ant, illustrate its potential to speed up the adoption of ML in regulatory compliance, and highlight its limitations. (shrink)

Analyzing and evaluating legal case reports are labor-intensive tasks for judges and lawyers, who usually base their decisions on report abstracts, legal principles, and commonsense reasoning. Thus, summarizing legal documents is time-consuming and requires excellent human expertise. Moreover, public legal corpora of specific languages are almost unavailable. This paper proposes a transfer learning approach with extractive and abstractive techniques to cope with the lack of labeled legal summarization datasets, namely a low-resource scenario. In particular, we conducted extensive multi- and cross-language (...) experiments. The proposed work outperforms the state-of-the-art results of extractive summarization on the Australian Legal Case Reports dataset and sets a new baseline for abstractive summarization. Finally, syntactic and semantic metrics assessments have been carried out to evaluate the accuracy and the factual consistency of the machine-generated legal summaries. (shrink)

Legal inference is fundamental for building and verifying hypotheses in police investigations. In this study, we build a Natural Language Inference dataset in Korean for the legal domain, focusing on criminal court verdicts. We developed an adversarial hypothesis collection tool that can challenge the annotators and give us a deep understanding of the data, and a hypothesis network construction tool with visualized graphs to show a use case scenario of the developed model. The data is augmented using a combination of (...) Easy Data Augmentation approaches and round-trip translation, as crowd-sourcing might not be an option for datasets with sensible data. We extensively discuss challenges we have encountered, such as the annotator’s limited domain knowledge, issues in the data augmentation process, problems with handling long contexts and suggest possible solutions to the issues. Our work shows that creating legal inference datasets with limited resources is feasible and proposes further research in this area. (shrink)

Technology has substantially transformed the way legal services operate in many different countries. With a large and complex collection of digitized legal documents, the judiciary system worldwide presents a promising scenario for the development of intelligent tools. In this work, we tackle the challenging task of organizing and summarizing the constantly growing collection of legal documents, uncovering hidden topics, or themes that later can support tasks such as legal case retrieval and legal judgment prediction. Our approach to this problem relies (...) on topic discovery techniques combined with a variety of preprocessing techniques and learning-based vector representations of words, such as Doc2Vec and BERT-like models. The proposed method was validated using four different datasets composed of short and long legal documents in Brazilian Portuguese, from legal decisions to chapters in legal books. Analysis conducted by a team of legal specialists revealed the effectiveness of the proposed approach to uncover unique and relevant topics from large collections of legal documents, serving many purposes, such as giving support to legal case retrieval tools and also providing the team of legal specialists with a tool that can accelerate their work of labeling/tagging legal documents. (shrink)

Financial risks are among the most important risks in the construction industry projects, which significantly impact project objectives, including project cost. Besides, financial risks have many interactions with each other and project parameters, which must be taken into account to analyze risks correctly. In addition, a source of financial risks in a project is the contract, which is the most important project document. Identifying terms related to financial risks in a contract and considering their effects on the risk management process (...) is an essential issue that has been neglected. Hence, an integrated model for evaluating financial risks and their related contractual clauses were presented. To this end, the effect of financial risks on the project cost was simulated using a system dynamics model. Moreover, terms related to financial risks in a contract text were identified and extracted using text mining, and their effect was included in the system dynamics model. The model was implemented in a hospital construction project in Tehran as a case study, and its results were analyzed. The innovation of the research is integrating text mining and the system dynamics model to investigate the effect of financial risks and related contractual clauses on the project cost. (shrink)

Legal documents, like contracts or laws, are subject to interpretation. Different people can have different interpretations of the very same document. Large parts of judicial branches all over the world are concerned with settling disagreements that arise, in part, from these different interpretations. In this context, it only seems natural that during the annotation of legal machine learning data sets, disagreement, how to report it, and how to handle it should play an important role. This article presents an analysis of (...) the current state-of-the-art in the annotation of legal machine learning data sets. The results of the analysis show that all of the analysed data sets remove all traces of disagreement, instead of trying to utilise the information that might be contained in conflicting annotations. Additionally, the publications introducing the data sets often do provide little information about the process that derives the “gold standard” from the initial annotations, often making it difficult to judge the reliability of the annotation process. Based on the state-of-the-art, the article provides easily implementable suggestions on how to improve the handling and reporting of disagreement in the annotation of legal machine learning data sets. (shrink)

Identifying, classifying, and analyzing arguments in legal discourse has been a prominent area of research since the inception of the argument mining field. However, there has been a major discrepancy between the way natural language processing (NLP) researchers model and annotate arguments in court decisions and the way legal experts understand and analyze legal argumentation. While computational approaches typically simplify arguments into generic premises and claims, arguments in legal research usually exhibit a rich typology that is important for gaining insights (...) into the particular case and applications of law in general. We address this problem and make several substantial contributions to move the field forward. First, we design a new annotation scheme for legal arguments in proceedings of the European Court of Human Rights (ECHR) that is deeply rooted in the theory and practice of legal argumentation research. Second, we compile and annotate a large corpus of 373 court decisions (2.3M tokens and 15k annotated argument spans). Finally, we train an argument mining model that outperforms state-of-the-art models in the legal NLP domain and provide a thorough expert-based evaluation. All datasets and source codes are available under open lincenses athttps://github.com/trusthlt/mining-legal-arguments. (shrink)

As lawmakers produce norms, the underlying normative system is affected showing the intrinsic dynamism of law. Through undertaken actions of legal change, the normative system is continuously modified. In a usual legislative practice, the time for an enacted legal provision to be in force may differ from that of its inclusion to the legal system, or from that in which it produces legal effects. Even more, some provisions can produce effects retroactively in time. In this article we study a simulation (...) of such process through the formalisation of a temporalised logical framework upon which a novel belief revision model tackles the dynamic nature of law. Represented through intervals, the temporalisation of sentences allows differentiating the temporal parameters of norms. In addition, a proposed revision operator allows assessing change to the legal system by including a new temporalised literal while preserving the time-based consistency. This can be achieved either by pushing out conflictive pieces of pre-existing norms or through the modification of intervals in which such norms can be either in force, or produce effects. Finally, the construction of the temporalised revision operator is axiomatically characterised and its rational behavior proved through a corresponding representation theorem. (shrink)

Despite the public availability of legal documents, there is a need for finding specific information contained in them, such as paragraphs, clauses, items and so on. With such support, users could find more specific information than only finding whole legal documents. Some research efforts have been made in this area, but there is still a lot to be done to have legal information available more easily to be found. Thus, due to the large number of published legal documents and the (...) high degree of connectivity, simple access to the document is not enough. It is necessary to recover the related legal framework for a specific need. In other words, the retrieval of the set of legal documents and their parts related to a specific subject is necessary. Therefore, in this work, we present a proposal of a RDF-based graph to represent and search parts of legal documents, as the output of a set of terms that represents the pursued legal information. Such a proposal is well-grounded on an ontological view, which makes possible to describe the general structure of a legal system and the structure of legal documents, providing this way the grounds for the implementation of the proposed RDF graph in terms of the meaning of their parts and relationships. We posed several queries to retrieve parts of legal documents related to sets of words and the results were significant. (shrink)

The general mitigating and aggravating circumstances of criminal liability are elements attached to the crime that, when they occur, affect the punishment quantum. Cuban criminal legislation provides a catalog of such circumstances and some general conditions for their application. Such norms give judges broad discretion in assessing circumstances and adjusting punishment based on the intensity of those circumstances. In the interest of broad judicial discretion, the law does not establish specific ways for measuring circumstances’ intensity. This gives judges more freedom (...) and autonomy, but it also imposes on them more social responsibility and challenges them to manage the uncertainty and subjectivity inherent in this complex activity. This paper proposes a model to aid the linguistic assessment of circumstances’ intensity and to provide linguistic and numerical recommendations to determine an appropriate punishment interval. M-LAMAC determines the collective evaluation of circumstances of the same type, determines the prevalence of a type of circumstance by means of a compensation function, recommends the required modification in the input interval, and finally recommends a numerical interval adjusted to the judges’ initially expressed preferences. The model’s applicability is demonstrated by means of several experiments on a fictitious case of bank document forgery. (shrink)

With the ever-growing accessibility of case law online, it has become challenging to manually identify case law relevant to one’s legal issue. In the Netherlands, the planned increase in the online publication of case law is expected to exacerbate this challenge. In this paper, we tried to predict whether court decisions are cited by other courts or not after being published, thus in a way distinguishing between more and less authoritative cases. This type of system may be used to process (...) the large amounts of available data by filtering out large quantities of non-authoritative decisions, thus helping legal practitioners and scholars to find relevant decisions more easily, and drastically reducing the time spent on preparation and analysis. For the Dutch Supreme Court, the match between our prediction and the actual data was relatively strong (with a Matthews Correlation Coefficient of 0.60). Our results were less successful for the Council of State and the district courts (MCC scores of 0.26 and 0.17, relatively). We also attempted to identify the most informative characteristics of a decision. We found that a completely explainable model, consisting only of handcrafted metadata features, performs almost as well as a less well-explainable system based on all text of the decision. (shrink)

Access to legal information is fundamental to access to justice. Yet accessibility refers not only to making legal documents available to the public, but also rendering legal information comprehensible to them. A vexing problem in bringing legal information to the public is how to turn formal legal documents such as legislation and judgments, which are often highly technical, to easily navigable and comprehensible knowledge to those without legal education. In this study, we formulate a three-step approach for bringing legal knowledge (...) to laypersons, tackling the issues of navigability and comprehensibility. First, we translate selected sections of the law into snippets (called CLIC-pages), each being a small piece of article that focuses on explaining certain technical legal concept in layperson’s terms. Second, we construct a _Legal Question Bank_, which is a collection of legal questions whose answers can be found in the CLIC-pages. Third, we design an interactive _CLIC Recommender_. Given a user’s verbal description of a legal situation that requires a legal solution, CRec interprets the user’s input and shortlists questions from the question bank that are most likely relevant to the given legal situation and recommends their corresponding CLIC pages where relevant legal knowledge can be found. In this paper we focus on the technical aspects of creating an LQB. We show how large-scale pre-trained language models, such as GPT-3, can be used to generate legal questions. We compare machine-generated questions against human-composed questions and find that MGQs are more scalable, cost-effective, and more diversified, while HCQs are more precise. We also show a prototype of CRec and illustrate through an example how our 3-step approach effectively brings relevant legal knowledge to the public. (shrink)

The interpretability of AI is just as important as its performance. In the LegalAI field, there have been efforts to enhance the interpretability of models, but a trade-off between interpretability and prediction accuracy remains inevitable. In this paper, we introduce a novel framework called LK-IB for compulsory measure prediction (CMP), one of the critical tasks in LegalAI. LK-IB leverages Legal Knowledge and combines an Interpretable model and a Black-box model to balance interpretability and prediction performance. Specifically, LK-IB involves three steps: (...) (1) inputting cases into the first module, where first-order logic (FOL) rules are used to make predictions and output them directly if possible; (2) sending cases to the second module if FOL rules are not applicable, where a case distributor categorizes them as either “simple” or “complex“; and (3) sending simple cases to an interpretable model with strong interpretability and complex cases to a black-box model with outstanding performance. Experimental results demonstrate that the LK-IB framework provides more interpretable and accurate predictions than other state-of-the-art models. Given that the majority of cases in LegalAI are simple, the idea of model combination has significant potential for practical applications. (shrink)

This paper addresses the black-box problem in artificial intelligence (AI), and the related problem of explainability of AI in the legal context. We argue, first, that the black box problem is, in fact, a superficial one as it results from an overlap of four different – albeit interconnected – issues: the opacity problem, the strangeness problem, the unpredictability problem, and the justification problem. Thus, we propose a framework for discussing both the black box problem and the explainability of AI. We (...) argue further that contrary to often defended claims the opacity issue is not a genuine problem. We also dismiss the justification problem. Further, we describe the tensions involved in the strangeness and unpredictability problems and suggest some ways to alleviate them. (shrink)

Socialist courts are supposed to apply the law, not make it, and socialist legality denies judicial decisions any precedential status. In 2011, the Chinese Supreme People’s Court designated selected decisions as Guiding Cases to be referred to by all judges when adjudicating similar disputes. One decade on, the paucity of citations to Guiding Cases has been taken as demonstrating the incongruity of case-based adjudication and the socialist legal tradition. Citations are, however, an imperfect measure of influence. Reproduction of language uniquely (...) traceable to Guiding Cases can also be evidence of their impact on judicial decision-making. We employ a local alignment tool to detect unattributed text reuse of Guiding Cases in local court decisions. Our findings suggest that Guiding Cases are more consequential than commonly assumed, thereby complicating prevailing narratives about the antagonism of socialist legality to case law. (shrink)

Bayesian approaches to legal reasoning propose causal models of the relation between evidence, the credibility of evidence, and ultimate hypotheses, or verdicts. They assume that legal reasoning is the process whereby one infers the posterior probability of a verdict based on observed evidence, or facts. In practice, legal reasoning does not operate quite that way. Legal reasoning is also an attempt at inferring applicable rules derived from legal precedents or statutes based on the facts at hand. To make such an (...) inference, legal reasoning follows syllogistic logic and first order transitivity. This paper proposes a Bayesian model of legal syllogistic reasoning that complements existing Bayesian models of legal reasoning using a Bayesian network whose variables correspond to legal precedents, statutes, and facts. We suggest that legal reasoning should be modelled as a process of finding the posterior probability of precedents and statutes based on available facts. (shrink)

We combine linear temporal logic (with both past and future modalities) with a deontic version of justification logic to provide a framework for reasoning about time and epistemic and normative reasons. In addition to temporal modalities, the resulting logic contains two kinds of justification assertions: epistemic justification assertions and deontic justification assertions. The former presents justification for the agent’s knowledge and the latter gives reasons for why a proposition is obligatory. We present two kinds of semantics for the logic: one (...) based on Fitting models and the other based on neighborhood models. The use of neighborhood semantics enables us to define the dual of deontic justification assertions properly, which corresponds to a notion of permission in deontic logic. We then establish the soundness and completeness of an axiom system of the logic with respect to these semantics. Further, we formalize the Protagoras versus Euathlus paradox in this logic and present a precise analysis of the paradox, and also briefly discuss Leibniz’s solution. (shrink)

Law article prediction is a task of predicting the relevant laws and regulations involved in a case according to the description text of the case, and it has broad application prospects in improving judicial efficiency. In the existing research work, researchers often only consider a single case, employing the neural network method to extract features for prediction, which lack the mining of related and common element information between different data. In order to solve this problem, we propose a law article (...) prediction method that integrates the characteristics of common elements. It can effectively utilize the co-occurrence information of the training data, fully mine the relevant common elements between cases, and fuse local features. Experiments show that our method performs well. (shrink)

Internal misconduct is a universal problem in prisons and affects the maintenance of social order. Consequently, correctional institutions often develop rehabilitation programs to reduce the likelihood of inmates committing internal offenses and criminal recidivism after release. Therefore, it is necessary to identify the profile of each offender, both for the appropriate indication of a rehabilitation program and the level of internal security to which he must be submitted. In this context, this work aims to discover the most significant characteristics in (...) predicting inmate misconduct from ML methods and the SHAP approach. A database produced in 2004 through the Survey of Inmates in State and Federal Correctional Facilities in the United States of America was used, which provides nationally representative data on prisoners from state and federal facilities. The predictive model based on Random Forest performed the best, thus, we applied the SHAP to it. Overall, the results showed that features related to victimization, type of crime committed, age and age at first arrest, history of association with criminal groups, education, and drug and alcohol use are most relevant in predicting internal misconduct. Thus, it is expected to contribute to the prior classification of an inmate on time, to use programs and practices that aim to improve the lives of offenders, their reintegration into society, and consequently, the reduction of criminal recidivism. (shrink)

This paper analyses and compares some of the automated reasoners that have been used in recent research for compliance checking. Although the list of the considered reasoners is not exhaustive, we believe that our analysis is representative enough to take stock of the current state of the art in the topic. We are interested here in formalizations at the _first-order_ level. Past literature on normative reasoning mostly focuses on the _propositional_ level. However, the propositional level is of little usefulness for (...) concrete LegalTech applications, in which compliance checking must be enforced on (large) sets of individuals. Furthermore, we are interested in technologies that are _freely available_ and that can be further investigated and compared by the scientific community. In other words, this paper does not consider technologies only employed in industry and/or whose source code is non-accessible. This paper formalizes a selected use case in the considered reasoners and compares the implementations, also in terms of simulations with respect to shared synthetic datasets. The comparison will highlight that lot of further research still needs to be done to integrate the benefits featured by the different reasoners into a single standardized first-order framework, suitable for LegalTech applications. All source codes are freely available at https://github.com/liviorobaldo/compliancecheckers, together with instructions to locally reproduce the simulations. (shrink)

This article proposes an innovative methodology for enhancing the technical validation, legal alignment and interdisciplinarity of attempts to encode legislation. In the context of an experiment that examines how different legally trained participants convert select provisions of the Australian Copyright Act 1968 (Cth) into machine-executable code, we find that a combination of manual and automated methods for coding validation, which focus on formal adherence to programming languages and conventions, can significantly increase the similarity of encoded rules between coders. Participants nonetheless (...) encountered various interpretive difficulties, including syntactic ambiguity, and intra- and intertextuality, which necessitated legal evaluation, as distinct from and in addition to coding validation. Many of these difficulties can be resolved through what we call a process of ‘legal alignment’ that aims to enhance the congruence between encoded provisions and the true meaning of a statute as determined by the courts. However, some difficulties cannot be overcome in advance, such as factual indeterminacy. Given the inherently interdisciplinary nature of encoding legislation, we argue that it is desirable for ‘rules as code’ (‘RaC’) initiatives to have, at a minimum, legal subject matter, statutory interpretation and technical programming expertise. Overall, we contend that technical validation, legal alignment and interdisciplinary teamwork are integral to the success of attempts to encode legislation. While legal alignment processes will vary depending on jurisdictionally-specific principles and practices of statutory interpretation, the technical and interdisciplinary components of our methodology are transferable across regulatory contexts, bodies of law and Commonwealth and other jurisdictions. (shrink)

Recently, the pandemic caused by COVID-19 is severe in the entire world. The prevention and control of crimes associated with COVID-19 are critical for controlling the pandemic. Therefore, to provide efficient and convenient intelligent legal knowledge services during the pandemic, we develop an intelligent system for legal information retrieval on the WeChat platform in this paper. The data source we used for training our system is “The typical cases of national procuratorial authorities handling crimes against the prevention and control of (...) the new coronary pneumonia pandemic following the law”, which is published online by the Supreme People’s Procuratorate of the People’s Republic of China. We base our system on convolutional neural network and use the semantic matching mechanism to capture inter-sentence relationship information and make a prediction. Moreover, we introduce an auxiliary learning process to help the network better distinguish the relation between two sentences. Finally, the system uses the trained model to identify the information entered by a user and responds to the user with a reference case similar to the query case and gives the reference legal gist applicable to the query case. (shrink)

Artificial intelligence plays an increasingly important role in legal disputes, influencing not only the reality outside the court but also the judicial decision-making process itself. While it is clear why judges may generally benefit from technology as a tool for reducing effort costs or increasing accuracy, the presence of technology in the judicial process may also affect the public perception of the courts. In particular, if individuals are averse to adjudication that involves a high degree of automation, particularly given fairness (...) concerns, then judicial technology may yield lower benefits than expected. However, the degree of aversion may well depend on how technology is used, i.e., on the timing and strength of judicial reliance on algorithms. Using an exploratory survey, we investigate whether the stage in which judges turn to algorithms for assistance matters for individual beliefs about the fairness of case outcomes. Specifically, we elicit beliefs about the use of algorithms in four different stages of adjudication: (i) information acquisition, (ii) information analysis, (iii) decision selection, and (iv) decision implementation. Our analysis indicates that individuals generally perceive the use of algorithms as fairer in the information acquisition stage than in other stages. However, individuals with a legal profession also perceive automation in the decision implementation stage as less fair compared to other individuals. Our findings, hence, suggest that individuals do care about how and when algorithms are used in the courts. (shrink)

Automated Vehicles (AVs) are designed and programmed to follow traffic rules. However, there is no separate and comprehensive regulatory framework dedicated to AVs. The current Queensland traffic rules were designed for humans. These rules often contain open texture expressions, exceptions, and potential conflicts (conflict arises when exceptions cannot be handled in rules), which makes it hard for AVs to follow. This paper presents an automatic compliance checking framework to assess AVs behaviour against current traffic rules by addressing these issues. Specifically, (...) it proposes a framework to determine which traffic rules and open texture expressions need some additional interpretation. Essentially this enables AVs to have a suitable and executable formalization of the traffic rules. Defeasible Deontic Logic (DDL) is used to formalize traffic rules and reasoning with AV information (behaviour and environment). The representation of rules in DDL helps effectively in handling and resolving exceptions, potential conflicts, and open textures in rules. 40 experiments were conducted on eight realistic traffic scenarios to evaluate the framework. The evaluation was undertaken both quantitatively and qualitatively. The evaluation result shows that the proposed framework is a promising system for checking Automated Vehicle interpretation and compliance with current traffic rules. (shrink)

The appropriate understanding and fast processing of lengthy legal documents are computationally challenging problems. Designing efficient automatic summarization techniques can potentially be the key to deal with such issues. Extractive summarization is one of the most popular approaches for forming summaries out of such lengthy documents, via the process of summary-relevant sentence selection. An efficient application of this approach involves appropriate scoring of sentences, which helps in the identification of more informative and essential sentences from the document. In this work, (...) a novel sentence scoring approach DCESumm is proposed which consists of supervised sentence-level summary relevance prediction, as well as unsupervised clustering-based document-level score enhancement. Experimental results on two legal document summarization datasets, BillSum and Forum of Information Retrieval Evaluation (FIRE), reveal that the proposed approach can achieve significant improvements over the current state-of-the-art approaches. More specifically it achieves ROUGE metric F1-score improvements of (1−6)% and (6−12)% for the BillSum and FIRE test sets respectively. Such impressive summarization results suggest the usefulness of the proposed approach in finding the gist of a lengthy legal document, thereby providing crucial assistance to legal practitioners. (shrink)

Evidences are important proofs to support judicial trials. Automatically extracting evidences from judgement documents can be used to assess the trial quality and support “Intelligent Court”. Current evidence extraction is primarily depended on sequence labelling models. Despite their success, they can only assign a label to a token, which is difficult to recognize nested evidence entities in judgment documents, where a token may belong to several evidences at the same time. In this paper, we present a novel evidence extraction architecture (...) called ATT-MRC, in which extracting evidence entities is formalized as a question answer problem, where all evidence spans are screened out as possible correct answers. Furthermore, to address the data imbalance problem in the judgement documents, we revised the loss function and combined it with a data enhancement technique. Experimental results demonstrate that our model has better performance than related works in evidence extraction. (shrink)