Abstract
Automated classification of legal documents has been the subject of extensive research in recent years. However, this is still a challenging task for long documents, since it is difficult for a model to identify the most relevant information for classification. In this paper, we propose a two-stage supervised learning approach for the classification of petitions, a type of legal document that requests a court order. The proposed approach is based on a word-level encoder–decoder Seq2Seq deep neural network, such as a Bidirectional Long Short-Term Memory (BiLSTM) or a Bidirectional Encoder Representations from Transformers (BERT) model, and a document-level Support Vector Machine classifier. To address the challenges posed by the lengthy legal documents, the approach introduces a human-in-the-loop approach, whose task is to localize and tag relevant segments of text in the word-level training part, which dramatically reduces the dimension of the document classifier input vector. We performed experiments to validate our approach using a real-world dataset comprised of 270 intermediate petitions, which were carefully annotated by specialists from the 15th civil unit of the State of Alagoas, Brazil. Our results revealed that both BiLSTM and BERT-Convolutional Neural Networks variants achieved an accuracy of up to 95.49%, and also outperformed baseline classifiers based on the Term Frequency–Inverse Document Frequency test vectorizer. The proposed approach is currently being utilized to automate the aforementioned justice unit, thereby increasing its efficiency in handling repetitive tasks.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aggarwal CC, Zhai C (2012) A survey of text classification algorithms. In: Mining text data. Springer, pp 163–222
Aguiar A, Silveira R, Pinheiro V, Furtado V, Neto JA (2021) Text classification in legal documents extracted from lawsuits in Brazilian courts. In: Brazilian conference on intelligent systems. Springer, pp 586–600
Altınel B, Ganiz MC (2018) Semantic text classification: a survey of past and recent advances. Inf Process Manag 54(6):1129–1153
Araújo DC, Lima A, Lima JP, Costa JA (2021) A comparison of classification methods applied to legal text data. In: Progress in artificial intelligence: 20th EPIA conference on artificial intelligence, EPIA 2021, Virtual Event, September 7–9, 2021, Proceedings 20. Springer, pp 68–80
Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv:1409.0473
Bambroo P, Awasthi A (2021) Legaldb: long distilbert for legal document classification. In: 2021 International conference on advances in electrical, computing, communication and sustainable technologies (ICAECT). IEEE, pp 1–4
CNJ (2020) Justiça em números 2020. Technical report, CNJ, Brasília, 2020. Also available as https://www.cnj.jus.br/pesquisas-judiciarias/justica-em-numeros/
CNJ and GovRisk (2017) UK–Brazil cooperation: improving efficiency and performance in Brazil’s judiciary, 2016/2017. Technical report, CNJ, Brasília, 2017. Also available as https://bibliotecadigital.cnj.jus.br/jspui/handle/123456789/345
Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P (2011) Natural language processing (almost) from scratch. J Mach Learn Res 12(ARTICLE):2493–2537
Da Silva NC, Braz F, de Campos T, Gusmao D, Chaves F, Mendes D, Bezerra D, Ziegler G, Horinouchi L, Ferreira M, et al (2018) Document type classification for Brazil’s supreme court using a convolutional neural network. In: 10th International conference on forensic computer science and cyber law (ICoFCS), Sao Paulo, Brazil, pp 29–30
Dal Pont TR, Sabo IC, Wilton PEV, de Menezes VA, Copetti R, Zambrota L, Martins PP, Costa EC, Schnitzler EL, Santos PM, et al (2021) Classification and association rules in Brazilian supreme court judgments on pre-trial detention. In: International conference on electronic government and the information systems perspective. Springer, pp 131–142
de Araujo PHL, de Campos TE, Braz FA, da Silva NC (2020) Victor: a dataset for Brazilian legal documents classification. In: Proceedings of the 12th language resources and evaluation conference, pp 1449–1458
Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805
Du J, Gui L, Xu R, He Y (2017) A convolutional attention model for text classification. In: National CCF conference on natural language processing and Chinese computing. Springer, pp 183–195
Enamoto L, Santos AR, Maia R, Weigang L, Filho GPR (2022) Multi-label legal text classification with BiLSTM and attention. Int J Comput Appl Technol 68(4):369–378
Fernandes WPD, Silva LJS, Frajhof IZ, de Almeida GdFCF, Konder CN, Nasser RB, de Carvalho GR, Barbosa SDJ, Lopes HCV et al (2020) Appellate court modifications extraction for Portuguese. Artif Intell Law 28(3):327–360
Goyal A, Gupta V, Kumar M (2018) Recent named entity recognition and classification techniques: a systematic review. Comput Sci Rev 29:21–43
Guille A, Attali H (2022) Document classification with hierarchical graph neural networks. In: 18th International workshop on mining and learning with graphs
Guzella TS, Caminhas WM (2009) A review of machine learning approaches to spam filtering. Expert Syst Appl 36(7):10206–10222
Hu D (2019) An introductory survey on attention mechanisms in NLP problems. In: Proceedings of SAI intelligent systems conference. Springer, pp 432–448
Huang Z, Xu W, Yu K (2015) Bidirectional LSTM-CRF models for sequence tagging. arXiv:1508.01991
Joachims T (1998) Text categorization with support vector machines: learning with many relevant features. In: European conference on machine learning. Springer, pp 137–142
Jurafsky D, Martin JH (2009) Speech and Language Processing, 2nd edn. Prentice-Hall Inc., Upper Saddle River. ISBN 0131873210
Kanakaraddi SG, Nandyal SS (2018) Survey on parts of speech tagger techniques. In: 2018 International conference on current trends towards converging technologies (ICCTCT). IEEE, pp 1–6
Kim Y (Oct. 2014) Convolutional neural networks for sentence classification. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), Doha, Qatar. Association for Computational Linguistics, pp 1746–1751. https://doi.org/10.3115/v1/D14-1181. https://aclanthology.org/D14-1181
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
Kowsari K, Jafari Meimandi K, Heidarysafa M, Mendu S, Barnes L, Brown D (2019) Text classification algorithms: a survey. Information 10(4):150
Kuru O, Can OA, Yuret D. (Dec. 2016) CharNER: Character-level named entity recognition. In: Proceedings of COLING 2016, the 26th international conference on computational linguistics: technical papers. The COLING 2016 Organizing Committee, Osaka, pp 911–921. https://aclanthology.org/C16-1087
Lai S, Xu L, Liu K, Zhao J (2015) Recurrent convolutional neural networks for text classification. In: Twenty-ninth AAAI conference on artificial intelligence
Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C (2016) Neural architectures for named entity recognition. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies. Association for Computational Linguistics, San Diego, pp 260–270. https://doi.org/10.18653/v1/N16-1030. https://aclanthology.org/N16-1030
Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International conference on machine learning. PMLR, pp 1188–1196
Li J, Sun A, Han J, Li C (2020) A survey on deep learning for named entity recognition. IEEE Trans Knowl Data Eng 34:50–70
Ling W, Dyer C, Black AW, Trancoso I, Fermandez R, Amir S, Marujo L, Luís T (Sept. 2015) Finding function in form: compositional character models for open vocabulary word representation. In: Proceedings of the 2015 conference on empirical methods in natural language processing. Association for Computational Linguistics, Lisbon, pp 1520–1530. https://doi.org/10.18653/v1/D15-1176. https://aclanthology.org/D15-1176
Liu G, Guo J (2019) Bidirectional LSTM with attention mechanism and convolutional layer for text classification. Neurocomputing 337:325–338
Liu L, Liu K, Cong Z, Zhao J, Ji Y, He J (2018) Long length document classification by local convolutional feature aggregation. Algorithms 11(8):109
Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized BERT pretraining approach. arXiv:1907.11692
Luong T, Pham H, Manning CD (Sept. 2015) Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 conference on empirical methods in natural language processing. Association for Computational Linguistics, Lisbon, pp 1412–1421. https://doi.org/10.18653/v1/D15-1166. https://aclanthology.org/D15-1166
Ma X, Hovy E (2016) End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In: Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers). Association for Computational Linguistics, Berlin, pp 1064–1074. https://doi.org/10.18653/v1/P16-1101. https://aclanthology.org/P16-1101
Mee A, Homapour E, Chiclana F, Engel O (2021) Sentiment analysis using TF–IDF weighting of UK MPs’ tweets on Brexit. Knowl Based Syst 229:107238
Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119
Mirończuk MM, Protasiewicz J (2018) A recent overview of the state-of-the-art elements of text classification. Expert Syst Appl 106:36–54
Otter DW, Medina JR, Kalita JK (2020) A survey of the usages of deep learning for natural language processing. IEEE Trans Neural Netw Learn Syst 32(2):604–624
Pennington J, Socher R, Manning CD (2014) Glove: global vectors for word representation. In: Empirical methods in natural language processing (EMNLP), pp 1532–1543. http://www.aclweb.org/anthology/D14-1162
Pérez-Rosas V, Kleinberg B, Lefevre A, Mihalcea R (2017) Automatic detection of fake news. arXiv:1708.07104
Plank B, Søgaard A, Goldberg Y (2016) Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. arXiv:1604.05529
Qaiser S, Ali R (2018) Text mining: use of TF–IDF to examine the relevance of words to documents. Int J Comput Appl 181(1):25–29
Resck LE, Ponciano JR, Nonato LG, Poco J (2022) Legalvis: exploring and inferring precedent citations in legal documents. IEEE Trans Vis Comput Graph
Scholkopf B, Smola AJ (2001) Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT Press, Cambridge
Schuster M, Nakajima K (2012) Japanese and Korean voice search. In: 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 5149–5152
Song D, Vold A, Madan K, Schilder F (2021) Multi-label legal document classification: a deep learning-based approach with label-attention and domain-specific pre-training. Inf Syst 106:101718
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008
Wan L, Papageorgiou G, Seddon M, Bernardoni M (2019) Long-length legal document classification. arXiv:1912.06905
Wang Q, Zhao K, Amor R, Liu B, Wang R (2022) D2GCLF: Document-to-graph classifier for legal document classification. In: Findings of the association for computational linguistics: NAACL, vol 2022, pp 2208–2221
Wang X, Liu Y, Sun C-J, Wang B, Wang X (2015) Predicting polarities of tweets by composing word embeddings with long short-term memory. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pp 1343–1353
Wei F, Qin H, Ye S, Zhao H (2018) Empirical study of deep learning for text classification in legal document review. In: 2018 IEEE international conference on big data (big data). IEEE, pp 3317–3320
Yadav V, Bethard S (2018) A survey on recent advances in named entity recognition from deep learning models. In: Bender EM, Derczynski L, Isabelle P (eds) Proceedings of the 27th international conference on computational linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20–26, 2018. Association for Computational Linguistics, pp 2145–2158. https://aclanthology.org/C18-1182/
Yang J, Ma W, Zhang M, Zhou X, Liu Y, Ma S (2021) Legalgnn: legal information enhanced graph neural network for recommendation. ACM Trans Inf Syst (TOIS) 40(2):1–29
Young T, Hazarika D, Poria S, Cambria E (2018) Recent trends in deep learning based natural language processing [review article]. IEEE Comput Intell Mag 13(3):55–75. https://doi.org/10.1109/MCI.2018.2840738
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Costa, Y.D.R., Oliveira, H., Nogueira, V. et al. Automating petition classification in Brazil’s legal system: a two-step deep learning approach. Artif Intell Law (2023). https://doi.org/10.1007/s10506-023-09385-4
Accepted:
Published:
DOI: https://doi.org/10.1007/s10506-023-09385-4