Between the Human Gaze and AI: a Human-in-the-Loop Approach to the Study of Late Medieval Illuminated Manuscripts
DOI:
https://doi.org/10.60923/issn.2532-8816/25623Keywords:
deep learning, human-in-the-loop, classification, illuminated manuscriptsAbstract
The application of artificial intelligence to cultural heritage is a multidisciplinary research field that brings together experts from different backgrounds, merging computer science and historical expertise. Recently, the systematic digitization of collections has opened up new possibilities for experimentation, but human-centered AI and qualitative approaches continue to be preferred over unsupervised quantitative ones. In the case of manuscript codices, a variety of deep neural networks have been developed for the automatic analysis of pages, the extraction of layout components, or the transcription of text. Multiple attempts have been also made in the illuminated specimens to detect miniatures and other decorative elements. Such models are usually intended for a specialist audience and developed to speed up the consultation process, but they can also be integrated into visualization interfaces and become valid tools for enhancing volumes. The paper presents the deep neural network developed in the “IlluminAI Project” of the CNR-ISPC and the incremental learning cycle used, at the same time, for its training and the semi-automatic labeling of an original dataset. After a brief introduction to contextualize the project, we will describe and discuss the cycle implemented to optimize human involvement to verify and correct the predictions made by the model.
References
Albritton, Benjamin, Georgia Henley, and Elaine Treharne. 2020. Medieval Manuscripts in the Digital Age. London: Routledge.
Alexander, Jonathan J. G. 1994. The Painted Page: Italian Renaissance Book Illumination, 1450-1550. New York: Prestel Pub.
Ali, Sajid, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M. Alonso-Moral, Roberto Confalonieri, Riccardo Guidotti, Javier Del Ser, Natalia Díaz-Rodríguez, and Francisco Herrera. 2023. “Explainable Artificial Intelligence (XAI): What We Know and What Is Left to Attain Trustworthy Artificial Intelligence.” Information Fusion 99 (2023): 101805. https://doi.org/10.1016/j.inffus.2023.101805.
Aouinti, Fouad, Victoria Eyharabide, Xavier Fresquet, and Frédéric Billiet. 2022. “Illumination Detection in IIIF Medieval Manuscripts Using Deep Learning.” Digital Medievalist 15 (1): 1–18. https://doi.org/10.16995/dm.8073.
Borghesani, Daniele, Costantino Grana, and Rita Cucchiara. 2014. “Miniature Illustrations Retrieval and Innovative Interaction for Digital Illuminated Manuscripts.” Multimedia Systems 20 (2014): 65–79. https://doi.org/10.1007/s00530-013-0315-3.
Campagnolo, Alberto. 2024. “Codificare Il Codice. La Materialità del Libro e la Sua Digitalizzazione Verso una Codicologia Digitale.” Umanistica Digitale 8 (17): 1–23. https://doi.org/10.6092/issn.2532-8816/16694.
Coustaty, Mickaël, Rudolf Pareti, Nicole Vincent, and Jean-Marc Ogier. 2011. “Towards Historical Document Indexing: Extraction of Drop Cap Letters.” International Journal on Document Analysis and Recognition (IJDAR) 14 (3): 243–54. https://doi.org/10.1007/s10032-011-0152-x.
Deng, Jia, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. “ImageNet: A Large-Scale Hierarchical Image Database.” 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Miami, FL, USA), 2009, 248–55. https://doi.org/10.1109/CVPR.2009.5206848.
Esposito, Floriana, Stefano Ferilli, Teresa M. A. Basile, and Nicola Di Mauro. 2008. “Machine Learning for Digital Document Processing: From Layout Analysis to Metadata Extraction.” In Machine Learning in Document Analysis and Recognition. Studies in Computational Intelligence, vol. 90. Berlin, Heidelberg: Springer, 105–538. https://doi.org/10.1007/978-3-540-76280-5_5.
Fanini, Bruno, Daniele Ferdani, Emanuel Demetrescu, Simone Berto and Enzo d’Annibale. 2021. “ATON: An Open-Source Framework for Creating Immersive, Collaborative and Liquid Web-Apps for Cultural Heritage.” Applied Sciences 11 (22): 11062. https://doi.org/10.3390/app112211062.
Fiorucci, Marco, Marina Khoroshiltseva, Massimiliano Pontil, Arianna Traviglia, Alessio Del Bue, and Stuart James. 2020. “Machine Learning for Cultural Heritage: A Survey.” Pattern Recognition Letters 133 (2020): 102–8. https://doi.org/10.1016/j.patrec.2020.02.017.
He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. “Deep Residual Learning for Image Recognition.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Las Vegas, NV, USA), 2016, 770–78. https://doi.org/10.1109/CVPR.2016.90.
Islam, Mohammad Anwarul, and Ionut E. Iacob. 2023. “Manuscripts Character Recognition Using Machine Learning and Deep Learning.” Modelling 4 (2): 168–88. https://doi.org/10.3390/modelling4020010.
Lee, Benjamin Charles Germain, Jaime Mears, Eileen Jakeway, Meghan Ferriter, Chris Adams, Nathan Yarasavage, Deborah Thomas, Kate Zwaard, and Daniel S. Weld. 2020. “The Newspaper Navigator Dataset: Extracting Headlines and Visual Content from 16 Million Historic Newspaper Pages in Chronicling America.” CIKM ’20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management (New York, NY, USA), 2020, 3055–62. https://doi.org/10.1145/3340531.3412767.
Lombardi, Francesco, and Simone Marinai. 2020. “Deep Learning for Historical Document Analysis and Recognition—A Survey.” Journal of Imaging 6 (10): 110. https://doi.org/10.3390/jimaging6100110.
Mehri, Maroua, Pierre Héroux, Rémy Mullot, Jean-Philippe Moreux, Bertrand Coüasnon, and Bill Barrett. 2017. “HBA 1.0: A Pixel-Based Annotated Dataset for Historical Book Analysis.” HIP ’17: Proceedings of the 4th International Workshop on Historical Document Imaging and Processing (New York, NY, USA), 2017, 107–12. https://doi.org/10.1145/3151509.3151528.
Milani, Federico, and Piero Fraternali. 2021. “A Dataset and a Convolutional Model for Iconography Classification in Paintings.” Journal on Computing and Cultural Heritage 14 (4): article 46, 1–18. https://doi.org/10.1145/3458885.
Minisini, Valeria, Bruno Fanini, and Giorgio Gosti. 2025. “The IlluminAI Project: A Deep Neural Network and Immersive Visualization System to Enhance Illuminated Manuscripts.” In Digital Heritage, edited by Stefano Campana, Daniele Ferdani, Holger Graf, Gabriele Guidi, Zackary Hegarty, Sofia Pescarin, and Fabio Remondino. The Eurographics Association. https://doi.org/10.2312/dh.20253119.
Mosqueira‑Rey, Eduardo, Elena Hernández‑Pereira, David Alonso‑Ríos, José Bobes‑Bascarán, and Ángel Fernández‑Leal. 2023. “Human‑in‑the‑loop Machine Learning: A State of the Art.” Artificial Intelligence Review 56 (2023): 3005–54. https://doi.org/10.1007/s10462-022-10246-w.
Nikolaidou, Konstantina, Mathias Seuret, Hamam Mokayed, and Marcus Liwicki. 2022. “A Survey of Historical Document Image Datasets.” International Journal on Document Analysis and Recognition (IJDAR) 25 (2022): 305–38. https://doi.org/10.1007/s10032-022-00405-8.
Pächt, Otto. 1984. Buchmalerei Des Mittelalters: Eine Einführung. München: Prestel.
Pietroni, Eva, Alessandra Chirivì, Bruno Fanini, and Alberto Bucciero. 2023. “An Innovative Approach to Shape Information Architecture Related to Ancient Manuscripts, Through Multi-layered Virtual Ecosystems. From Codex4D to DataSpace Project.” Extended Reality. XR Salento 2023. Lecture Notes in Computer Science, vol. 14219. Cham: Springer, 247–67. https://doi.org/10.1007/978-3-031-43404-4_16.
Porter, Reid, James Theiler, and Don Hush. 2013. “Interactive Machine Learning in Data Exploitation.” Computing in Science & Engineering 15 (5): 12–20. https://doi.org/10.1109/MCSE.2013.74.
Rahal, Najoua, Lars Vögtlin, and Rolf Ingold. 2024. “Approximate Ground Truth Generation for Semantic Labeling of Historical Documents with Minimal Human Effort.” International Journal on Document Analysis and Recognition (IJDAR) 27 (2024): 335–47. https://doi.org/10.1007/s10032-024-00475-w.
Schlecht, Joseph, Bernd Carqué, and Björn Ommer. 2011. “Detecting Gestures in Medieval Images.” Proceedings of the 18th IEEE International Conference on Image Processing (Brussels, Belgium), 2011, 1285–88. http://doi.org/10.1109/ICIP.2011.6115669.
Selvaraju, Ramprasaath R., Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.” Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV) (Venice, Italy), 2017, 618–26. https://doi.org/10.1109/ICCV.2017.74.
Tkachenko, Maxim, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2025. Label Studio: Data Labeling Software. Released 2020. https://github.com/HumanSignal/label-studio.
Zhao, Wentao, Dalin Zhou, Xinguo Qiu, and Wei Jiang. 2021. “Compare the Performance of the Models in Art Classification.” PLoS ONE 16 (3): e0248414. https://doi.org/10.1371/journal.pone.0248414.
Zhou, Bolei, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. “Learning Deep Features for Discriminative Localization.” Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Las Vegas, NV, USA), 2016, 2921–29. https://doi.org/10.1109/CVPR.2016.319.
Zhuang, Fuzhen, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. 2021. “A Comprehensive Survey on Transfer Learning.” Proceedings of the IEEE 109 (1): 43–76. https://doi.org/10.1109/JPROC.2020.3004555.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Valeria Minisini, Giorgio Gosti, Bruno Fanini

This work is licensed under a Creative Commons Attribution 4.0 International License.