Evolution of the Capabilities of Large Language Models in the Legal Field: Meta-analysis of Four Experimental Studies
( Pp. 209-220)
More about authors
Roman V. Dushkin
senior lecturer at Department 22 “Cybernetics”; Nuclear Research University MEPhI
Scientific Research Nuclear University of MEPhI
Moscow, Russian Federation Vladimir N. Podoprigora Cand. Sci. (Econ.), head of the laboratory; Russian Economic University. G.V. Plekhanov
Plekhanov Russian University of Economics
Moscow, Russian Federation Alexey A. Kuzmin General Director
Ecosystem Digital Solutions LLC
Moscow, Russian Federation Kirill R. Dushkin analyst; A-Z Expert LLC
LLC "A-Ya expert"
Moscow, Russian Federation
Scientific Research Nuclear University of MEPhI
Moscow, Russian Federation Vladimir N. Podoprigora Cand. Sci. (Econ.), head of the laboratory; Russian Economic University. G.V. Plekhanov
Plekhanov Russian University of Economics
Moscow, Russian Federation Alexey A. Kuzmin General Director
Ecosystem Digital Solutions LLC
Moscow, Russian Federation Kirill R. Dushkin analyst; A-Z Expert LLC
LLC "A-Ya expert"
Moscow, Russian Federation
Abstract:
This paper presents a meta-analysis of four experimental studies from the Norm! project, aimed at systematically studying the effectiveness of large language models in the legal field. The study includes a comparative analysis of junior and senior models, optimization of system prompts, and testing of multi-agent architectures on tasks in Russian family and civil law. A key discovery was the identification of a nonlinear relationship between architectural complexity and the quality of results: the transition from simple to complex systems provides a slight increase in quality (15–40%) with an exponential increase in resource costs (by a factor of 10–15). The flagship models GPT-4.1 and Gemini 2.5 Pro demonstrate superior quality (9.04 and 8.52 points), but junior LLMs with efficiency coefficients up to 130.3 remain cost-effective. A universal problem area for all architectures is tasks requiring an integrative analysis of multiple legal norms. The results form scientifically sound recommendations for various implementation scenarios: from mass consulting services to specialized legal applications, defining the prospects for the development of hybrid architectures in legal practice.
How to Cite:
Dushkin R.V., Podoprigora V.N., Kuzmin A.A., and Dushkin K.R. Evolution of the capabilities of large language models in the legal field: Meta-analysis of four experimental studies. Computational Nanotechnology. 12, 3 (2025), 209–220. DOI: 10.33693/2313-223X-2025-12-3-209-220. EDN: CBJQVM
Reference list:
Dushkin R.V. Artificial Intelligence. Moscow: DMK-Press, 2019. 280 p. ISBN: 978-5-97060-787-9.
Lai J., Gan W., Wu J. et al. Large language models in law: A survey. AI Open. 2024. URL: https://www.sciencedirect.com/science/article/pii/S2666651024000172 (data of accesses: 13.10.2023).
Ma S., Chen C., Chu Q. et al. Leveraging large language models for relevance judgments in legal case retrieval. arXiv preprint arXiv:2403.18405. 2024. URL: https://arxiv.org/abs/2403.18405 (data of accesses: 13.10.2023).
Paul V. Automation in legal: The increasing role of AI. Medium. 2024. URL: https://medium.com/@vincentpaulai/automation-in-legal-the-increasing-role-of-ai-70724ef0b225 (data of accesses: 13.10.2023).
Magesh V., Surani F., Dahl M. et al. Hallucination-free? Assessing the reliability of leading AI legal research tools. arXiv preprint arXiv:2405.20362. 2024. URL: https://arxiv.org/abs/2405.20362 (data of accesses: 13.10.2023).
The future of artificial intelligence in the legal industry: Opportunities, challenges, and ethical considerations. Legal Stuff. URL: https://medium.com/@legal.stuff.notion/the-future-of-artificial-intelligence-in-the-legal-industry-opportunities-challenges-and-ethical-61c3198b425a (data of accesses: 13.10.2023).
Korneenkov A.A., Yanov Yu.K., Ryazantsev S.V. et al. Meta-analysis of clinical studies in otorhinolaryngology. Bulletin of Otorhinolaryngology. 2020. Vol. 85. No. 2. Pp. 26–30. (In Rus.). DOI: 10.17116/otorino20208502126.
Dushkin R.V. Overview of approaches and methods of artificial intelligence. Radio Electronic Technologies. 2018. No. 3. Pp. 85–89. (In Rus.)
Rumyantsev P.O., Saenko U.V., Rumyantseva U.V. Statistical methods of analysis in clinical practice. Part 1. Univariate statistical analysis. Problems of Endocrinology. 2009. Vol. 55. No. 5. Pp. 48–55. (In Rus.). DOI: 10.14341/probl200955548-55.
Dushkin R.V. Why hybrid AI systems hold the future. Economic Strategies. 2018. No. 6 (156). Pp. 84–93. (In Rus.)
Lai J., Gan W., Wu J. et al. Large language models in law: A survey. AI Open. 2024. URL: https://www.sciencedirect.com/science/article/pii/S2666651024000172 (data of accesses: 13.10.2023).
Ma S., Chen C., Chu Q. et al. Leveraging large language models for relevance judgments in legal case retrieval. arXiv preprint arXiv:2403.18405. 2024. URL: https://arxiv.org/abs/2403.18405 (data of accesses: 13.10.2023).
Paul V. Automation in legal: The increasing role of AI. Medium. 2024. URL: https://medium.com/@vincentpaulai/automation-in-legal-the-increasing-role-of-ai-70724ef0b225 (data of accesses: 13.10.2023).
Magesh V., Surani F., Dahl M. et al. Hallucination-free? Assessing the reliability of leading AI legal research tools. arXiv preprint arXiv:2405.20362. 2024. URL: https://arxiv.org/abs/2405.20362 (data of accesses: 13.10.2023).
The future of artificial intelligence in the legal industry: Opportunities, challenges, and ethical considerations. Legal Stuff. URL: https://medium.com/@legal.stuff.notion/the-future-of-artificial-intelligence-in-the-legal-industry-opportunities-challenges-and-ethical-61c3198b425a (data of accesses: 13.10.2023).
Korneenkov A.A., Yanov Yu.K., Ryazantsev S.V. et al. Meta-analysis of clinical studies in otorhinolaryngology. Bulletin of Otorhinolaryngology. 2020. Vol. 85. No. 2. Pp. 26–30. (In Rus.). DOI: 10.17116/otorino20208502126.
Dushkin R.V. Overview of approaches and methods of artificial intelligence. Radio Electronic Technologies. 2018. No. 3. Pp. 85–89. (In Rus.)
Rumyantsev P.O., Saenko U.V., Rumyantseva U.V. Statistical methods of analysis in clinical practice. Part 1. Univariate statistical analysis. Problems of Endocrinology. 2009. Vol. 55. No. 5. Pp. 48–55. (In Rus.). DOI: 10.14341/probl200955548-55.
Dushkin R.V. Why hybrid AI systems hold the future. Economic Strategies. 2018. No. 6 (156). Pp. 84–93. (In Rus.)
Keywords:
large language models, legal artificial intelligence, meta-analysis, multi-agent systems, system prompts, cost-effectiveness, legal consulting, RAG systems, family law, artificial intelligence system architecture.
