Using Large Language Models to Analyse Free Indirect Discourse. A Comparative Study of Italian Fiction, 1830–1930
DOI:
https://doi.org/10.60923/issn.2532-8816/25846Keywords:
computational narratology, Large Language Models, prompt engineering, automatic annotation, free indirect discourse, Italian novel 1830-1930, AIUCD2025Abstract
This paper presents a systematic evaluation of the use of large language models (LLMs) for automatically annotating free indirect speech (FIS) in Italian fiction published between 1830 and 1930. Six commercial and open-source LLMs were tested using three prompting strategies on a sample of 1,000 text passages from the PRIN RIND project corpus, which were annotated by experts and used as a benchmark to evaluate the models. A series of statistical analyses measured the models’ agreement with this benchmark, as well as the stability of this agreement when prompts, texts and time periods varied. The commercial models achieved moderate agreement with the expert annotations and were found to be equivalent to one another. An open-source model run locally achieved results similar to those of the commercial models. The most effective prompting strategy depended on the individual model. Furthermore, agreement varied across the chronology of the texts, with the models flagging the phenomenon more frequently in more recent novels. Our analysis shows that LLMs can be used to annotate complex literary phenomena such as FIS. Due to their interpretative nature, such phenomena have so far remained beyond the scope of traditional quantitative methods.
References
Anthropic. 2026. “Introducing Sonnet 4.6.” Anthropic, 17 febbraio 2026. https://www.anthropic.com/news/claude-sonnet-4-6.
Aroyo, Lora, e Chris Welty. 2015. “Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation.” AI Magazine 36 (1): 15–24.
Bach, Stephen H., Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, et al. 2022. “PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts.” arXiv:2202.01279.
Bally, Charles. 1912. “Le style indirect libre en français moderne.” Germanisch-Romanische Monatsschrift 4: 549–556, 597–606.
Bamman, David, Kent K. Chang, Li Lucy, e Naitian Zhou. 2024. “On Classification with Large Language Models in Cultural Analytics.” In Proceedings of the Computational Humanities Research Conference (CHR 2024), 494–527. CEUR Workshop Proceedings 3834. https://ceur-ws.org/Vol-3834/paper119.pdf.
Barthelmebs-Raguin, Hélène, Greta Komur-Thilloy, Juan Manuel López Muñoz, Sophie Marnette, e Laurence Rosier, dir. 2018. Le discours rapporté: temporalité, histoire, mémoire et patrimoine discursif. Paris: Classiques Garnier.
Bode, Katherine. 2018. A World of Fiction: Digital Collections and the Future of Literary History. Ann Arbor: University of Michigan Press.
Brooke, Julian, Adam Hammond, e Graeme Hirst. 2017. “Using Models of Lexical Style to Quantify Free Indirect Discourse in Modernist Fiction.” Digital Scholarship in the Humanities 32 (2): 234–250.
Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33: 1877–1901. arXiv:2005.14165.
Brunner, Annelen. 2013. “Automatic Recognition of Speech, Thought, and Writing Representation in German Narrative Texts.” Literary and Linguistic Computing 28 (4): 563–575.
Brunner, Annelen, Ngoc Duyen Tanja Tu, Lukas Weimer, e Fotis Jannidis. 2020. “To BERT or Not to BERT. Comparing Contextual Embeddings in a Deep Learning Architecture for the Automatic Recognition of Four Types of Speech, Thought and Writing Representation.” In Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing (SwissText/KONVENS 2020), a cura di Sarah Ebling, Don Tuggener, Manuela Hürlimann, Mark Cieliebak e Martin Volk. CEUR Workshop Proceedings 2624, paper 5. https://ceur-ws.org/Vol-2624/paper5.pdf.
Brunner, Annelen, Ngoc Duyen Tanja Tu, Lukas Weimer, e Fotis Jannidis. 2019. “Deep Learning for Free Indirect Representation.” In Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Short Papers, 241–245. Erlangen.
Brunner, Annelen, Ngoc Duyen Tanja Tu, Lukas Weimer, e Fotis Jannidis. 2020. “To BERT or Not to BERT. Comparing Contextual Embeddings in a Deep Learning Architecture for the Automatic Recognition of Four Types of Speech, Thought and Writing Representation.” In Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing (SwissText/KONVENS 2020), a cura di Sarah Ebling, Don Tuggener, Manuela Hürlimann, Mark Cieliebak e Martin Volk, 29–40. CEUR Workshop Proceedings 2624. https://ceur-ws.org/Vol-2624/paper5.pdf.
Calaresu, Emilia. 2000. Il discorso riportato. Una prospettiva testuale. Modena: Il Fiorino.
Calaresu, Emilia. 2004. Testuali parole. La dimensione pragmatica e testuale del discorso riportato. Milano: FrancoAngeli.
Cangiano, Mimmo. 2018. La nascita del modernismo italiano. Filosofie della crisi, storia e letteratura (1903–1922). Macerata: Quodlibet.
Castellana, Riccardo. 2010. “Realismo modernista. Un’idea del romanzo italiano (1915–1925).” Italianistica 39 (1): 23–45.
Ciotti, Fabio. 2025. “Lettori artificiali e macchine pigre: cosa i Large Language Model possono farci capire sulla cooperazione interpretativa.” In Letteratura e intelligenza artificiale. Un dialogo interdisciplinare, a cura di Tiziana Catarci, Agnese Macori e Daniel Raffini, 109–146. Roma: Lithos.
Ciotti, Fabio. 2026. “An Agenda on Generative AI for the Digital Humanities.” Umanistica Digitale 10 (24): 1–14. https://doi.org/10.60923/issn.2532-8816/25557.
Ciotti, Fabio, Aurora Argenzio, e Anna Chiara Corradino. 2025. “Usare i Large Language Model per l’analisi del testo narrativo: strategie di prompt engineering per il riconoscimento del discorso indiretto libero nella narrativa italiana 1830–1930.” In Diversità, Equità e Inclusione: Sfide e Opportunità per l’Informatica Umanistica nell’Era dell’Intelligenza Artificiale. Proceedings del XIV Convegno Annuale AIUCD2025, a cura di Simone Rebora, Marco Rospocher e Stefano Bazzaco, 349–356. Verona: AIUCD. https://doi.org/10.6092/unibo/amsacta/8380.
Cohn, Dorrit. 1978. Transparent Minds: Narrative Modes for Presenting Consciousness in Fiction. Princeton: Princeton University Press.
Dahl, Liisa. 1970. Linguistic Features of the Stream-of-Consciousness Techniques of James Joyce, Virginia Woolf and Eugene O’Neill. Annales Universitatis Turkuensis, Series B, 116. Turku: Turun Yliopisto.
Daneman, Meredyth, e Patricia A. Carpenter. 1980. “Individual Differences in Working Memory and Reading.” Journal of Verbal Learning and Verbal Behavior 19 (4): 450–466. https://doi.org/10.1016/S0022-5371(80)90312-6.
Davis, Jesse, e Mark Goadrich. 2006. “The Relationship between Precision-Recall and ROC Curves.” In Proceedings of the 23rd International Conference on Machine Learning (ICML 2006), 233–240. New York: ACM.
Donnarumma, Raffaele. 2012. “Tracciato del modernismo italiano.” In Sul modernismo italiano, a cura di Romano Luperini e Massimiliano Tortora, 13–38. Napoli: Liguori.
Fludernik, Monika. 1993. The Fictions of Language and the Languages of Fiction: The Linguistic Representation of Speech and Consciousness. London: Routledge.
Genette, Gérard. 1972. Figures III. Paris: Seuil.
Google DeepMind. 2026. “Gemini 3.5 Flash: Model Card.” Google DeepMind, 19 maggio 2026. https://deepmind.google/models/model-cards/gemini-3-5-flash/.
Landis, J. Richard, e Gary G. Koch. 1977. “The Measurement of Observer Agreement for Categorical Data.” Biometrics 33 (1): 159–174.
Long, Hoyt, e Richard Jean So. 2016. “Turbulent Flow: A Computational Model of World Literature.” Modern Language Quarterly 77 (3): 345–367. https://doi.org/10.1215/00267929-3570656.
Luperini, Romano. 2018. Dal modernismo a oggi. Storicizzare la contemporaneità. Roma: Carocci.
Moretti, Franco. 2013. “‘Operationalizing’: Or, the Function of Measurement in Literary Theory.” New Left Review 84: 103–119.
Moretti, Franco, a cura di. 2017. Canon/Archive: Studies in Quantitative Formalism from the Stanford Literary Lab. New York: n+1 Books.
Mortara Garavelli, Bice. 1985. La parola d’altri. Prospettive di analisi del discorso. Palermo: Sellerio.
Mortara Garavelli, Bice. 1995. “Il discorso riportato.” In Grande grammatica italiana di consultazione, vol. III, Tipi di frase, deissi, formazione delle parole, a cura di Lorenzo Renzi, Giampaolo Salvi e Anna Cardinaletti, 427–468. Bologna: il Mulino.
Ondelli, Stefano, e Pietro Mazzarisi. 2025. “Il corpus di prosa letteraria del progetto RIND (1830–1930): assunti teorici e vincoli pratici.” In Diversità, Equità e Inclusione: Sfide e Opportunità per l’Informatica Umanistica nell’Era dell’Intelligenza Artificiale. Proceedings del XIV Convegno Annuale AIUCD2025, a cura di Simone Rebora, Marco Rospocher e Stefano Bazzaco, 289–296. Verona: AIUCD. https://doi.org/10.6092/unibo/amsacta/8380.
OpenAI. 2025. “gpt-oss-120b & gpt-oss-20b Model Card.” arXiv:2508.10925.
OpenAI. 2026. “Introducing GPT-5.5.” OpenAI, 23 aprile 2026. https://openai.com/index/introducing-gpt-5-5/.
Orlando, Riccardo, Luca Moroni, Pere-Lluís Huguet Cabot, Simone Conia, Edoardo Barba, Sergio Orlandini, Giuseppe Fiameni, e Roberto Navigli. 2024. “Minerva LLMs: The First Family of Large Language Models Trained from Scratch on Italian Data.” In Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024), 707–719. Pisa: CEUR Workshop Proceedings. https://aclanthology.org/2024.clicit-1.77/.
Piper, Andrew, e Sunyam Bagga. 2024. “Using Large Language Models for Understanding Narrative Discourse.” In Proceedings of the 6th Workshop on Narrative Understanding, a cura di Yash Kumar Lal, Elizabeth Clark, Mohit Iyyer, Snigdha Chaturvedi, Anneliese Brei, Faeze Brahman e Khyathi Raghavi Chandu, 37–46. Miami: Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.wnu-1.4.
Qwen Team. 2025. “Qwen3 Technical Report.” arXiv:2505.09388.
Schöch, Christof, Daniel Schlör, Stefanie Popp, Annelen Brunner, Ulrike Henny, e José Calvo Tello. 2016. “Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels.” In Digital Humanities 2016: Conference Abstracts, 346–353. Kraków: Jagiellonian University & Pedagogical University.
Sullet-Nylander, Françoise, Malin Roitman, Juan Manuel López Muñoz, Sophie Marnette, e Laurence Rosier, dir. 2014. Discours rapporté, genre(s) et médias. Stockholm: Stockholm University Press.
Tortora, Massimiliano, a cura di. 2018. Il modernismo italiano. Roma: Carocci.
Underwood, Ted, Ziliang Qiu, Sarah Griebel, Laura K. Nelson, Edwin Roland, Wenyi Shang, e Matthew Wilkens. 2026. “Chronologic: Measuring Language Models’ Ability to Represent the Past.” arXiv:2609.23178. https://arxiv.org/abs/2609.23178.
Vološinov, Valentin N. 1929. Marksizm i filosofija jazyka. Leningrad: Priboj. Trad. it. Marxismo e filosofia del linguaggio. Bari: Dedalo, 1976.
Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, e Denny Zhou. 2022. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” Advances in Neural Information Processing Systems 35: 24824–24837.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Fabio Ciotti, Anna Chiara Corradino, Aurora Argenzio
This work is licensed under a Creative Commons Attribution 4.0 International License.
