Reading the Unreadable: Artificial Intelligence for Text Interpretation in Degraded Manuscripts
DOI:
https://doi.org/10.60923/issn.2532-8816/25619Keywords:
Multispectral imaging, damaged manuscripts, semantic segmentation, deep learning, human-in-the-loop, interepistemic paradigm, AIUCD2025Abstract
“Manuscripts do not burn”, goes the famous maxim pronounced by the Devil-Woland in M.A. Bulgakov’s masterpiece The Master and Margarita. Alluding to the eternal value of the Word, this formula also appears to fit the material reality of manuscripts as concrete historical documents. This study addresses the methodological and epistemological implications of employing artificial intelligence in the study of severely damaged manuscripts, proposing a paradigm shift from traditional Handwritten Text Recognition (HTR) systems that fail under regimes of severe material degradation. Three reference computer vision architectures are compared on the multispectral MADAM dataset: object detection (Faster R-CNN), semantic segmentation (U-Net), and instance segmentation (Mask R-CNN), evaluated on incremental training subsets ranging from 6 to 84 patches. Semantic segmentation outperforms both object detection and instance segmentation across all metrics, achieving stable performance from 24 annotated patches; conversely, Faster R-CNN and Mask R-CNN architectures require larger datasets to avoid divergence during training (Carcagnì et al. 2026). However, U-Net encounters a structural limitation in case of thin or nearly transparent strokes where residual ink is insufficient to support segmentation; under such conditions, textual completion requires the expert’s judgment. This work, therefore, justifies a human-in-the-loop (HITL) approach in which algorithmic analytical capabilities and philological, linguistic, and semiotic expertise operate in a complementary manner throughout the entire recognition pipeline. A new hermeneutic paradigm, where the acronym AI alludes to a necessary collaboration between Artificial Intelligence and Artisan Intelligence – for the safeguarding of Artistic Intelligence of mankind.
References
Assael, Yannis, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita Chatzipanagiotou, Ion Androutsopoulos et al. 2022. “Restoring and Attributing Ancient Texts Using Deep Neural Networks.” Nature 603 (7900): 280–283. https://doi.org/10.1038/s41586-022-04448-z.
Assael, Yannis, Thea Sommerschield, Alison Cooley, Brendan Shillingford, John Pavlopoulos, Priyanka Suresh, Bailey Herms et al. 2025. “Contextualizing Ancient Texts with Generative Neural Networks.” Nature 645 (8079): 141–147. https://doi.org/10.1038/s41586-025-09292-5.
Baena, Raphael, Syrine Kalleli, and Mathieu Aubry. 2024. “General Detection-based Text Line Recognition.” In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), edited by Amir Globerson, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub Tomczak, and Cheng Zhang, 42388–42404. San Diego: Neural Information Processing Systems Foundation, Inc. (NeurIPS). https://doi.org/10.52202/079017-1342.
Bakhtin, Mikhail. 1999. Problems of Dostoevsky’s Poetics. Translated by Caryl Emerson. 8th ed. Minneapolis: University of Minnesota Press.
Bulgakov, Mikhail. 1967. The Master and Margarita. Translated by Michael Glenny. London: The Harvill Press.
Cao, Jianfang, Zibang Zhang, Aidi Zhao, Hongyan Cui, and Qi Zhang. 2020. “Ancient Mural Restoration Based on a Modified Generative Adversarial Network.” Heritage Science (8): 7. https://doi.org/10.1186/s40494-020-0355-x.
Carcagnì, Pierluigi, Marco Del Coco, Marco Leo, Laura Gazzani, Marco Malagodi, Melania Paturzo, and Claudia Daffara. 2026. “MADAM: Manuscript Annotated Dataset Based on Multispectral Imaging for Handwritten Text Enhancement and Restoration.” In The Future of Heritage Science and Technologies II. Florence Heri-Tech 2024: Lecture Notes in Mechanical Engineering, edited by Rocco Furferi, Lapo Governi, Yary Volpe, Francesco Gherardini, and Kate Seymour, 572–584. Cham: Springer. https://doi.org/10.1007/978-3-031-98379-5_43.
Chen, Liang-Chieh, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.” In Computer Vision – ECCV 2018. Lecture Notes in Computer Science (vol 11211), edited by Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, 833–851. Cham: Springer. https://doi.org/10.1007/978-3-030-01234-2_49.
Chen, Kai, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng et al. 2019. “Hybrid Task Cascade for Instance Segmentation.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): 4969–4978. Long Beach: IEEE. https://doi.org/10.1109/CVPR.2019.00511.
Cimino, Dafne, Giacomo Marchioro, Pietro De Paolis, and Claudia Daffara. 2023. “Evaluating the Integration of Thermal Quasi-reflectography in Manuscript Imaging Diagnostic Protocols to Improve Non-invasive Materials Investigation.” Journal of Cultural Heritage (62): 72–77. https://doi.org/10.1016/j.culher.2023.04.009.
Daffara, Claudia, Nicole de Manincor, Laura Gazzani, Sara Mazzocato, Dumitru Scutelnic, Anna Trovati, Paolo Pellegrini et al. 2025. “Artificial Intelligence and Physics for Art Diagnostics: First Results from ‘AIPAD’ Project.” In Proc. SPIE 13569, Optics for Arts, Architecture, and Archaeology (O3A) X, edited by Haida Liang, and Roger Groves, 135691A. Bellingham: SPIE. https://doi.org/10.1117/12.3066653.
Dhali, Maruf A., Jan Willem de Wit, and Lambert Schomaker. 2026. “BiNet: Degraded-Manuscript Binarization in Diverse Document Textures and Layouts Using Deep Encoder-Decoder Networks.” In Document Analysis and Recognition – ICDAR 2025 Workshops. Lecture Notes in Computer Science (vol 16226), edited by Lianwen Jin, Richard Zanibbi, and Veronique Eglin, 166–193. Cham: Springer. https://doi.org/10.1007/978-3-032-09371-4_11.
Graves, Alex, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. “Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks.” In ICML ‘06: Proceedings of the 23rd international conference on Machine learning, edited by William W. Cohen, and Andrew Moore, 369–376. New York: Association for Computing Machinery. https://doi.org/10.1145/1143844.1143891.
Hassner, Tal, Malte Rehbein, Peter A. Stokes, and Lior Wolf. 2013. “Computation and Palaeography: Potentials and Limits.” Dagstuhl Manifestos 2 (1): 14–35. https://doi.org/10.4230/DagMan.2.1.14.
He, Kaiming, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. “Mask R-CNN.” In 2017 IEEE International Conference on Computer Vision (ICCV): 2980–2988. Long Beach: IEEE. https://doi.org/10.1109/ICCV.2017.322.
He, Sheng, and Lambert Schomaker. 2019. “DeepOtsu: Document Enhancement and Binarization Using Iterative Deep Learning.” Pattern Recognition (91): 379–390. https://doi.org/10.1016/j.patcog.2019.01.025.
Hollaus, Fabian, Simon Brenner, and Robert Sablatnig. 2019. “CNN Based Binarization of MultiSpectral Document Images.” In 2019 International Conference on Document Analysis and Recognition (ICDAR): 533–538. Long Beach: IEEE. https://doi.org/10.1109/ICDAR.2019.00091.
Holzinger, Andreas. 2016. “Interactive Machine Learning for Health Informatics: When Do We Need the Human-in-the-loop?.” Brain Informatics 3 (2): 119–131. https://doi.org/10.1007/s40708-016-0042-6.
Kahle, Philip, Sebastian Colutto, Günter Hackl, and Günter Mühlberger. 2017. “Transkribus – A Service Platform for Transcription, Recognition and Retrieval of Historical Documents.” In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR): 19–24. Long Beach: IEEE. https://doi.org/10.1109/ICDAR.2017.307.
Kiessling, Benjamin, Robin Tissot, Peter Stokes, and Daniel Stökl Ben Ezra. 2019. “eScriptorium: An Open Source Platform for Historical Document Analysis.” In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW): 19–24. Long Beach: IEEE. https://doi.org/10.1109/ICDARW.2019.10032.
Kirillov, Alexander, Yuxin Wu, Kaiming He, and Ross Girshick. 2020. “PointRend: Image Segmentation as Rendering.” In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): 9796–9805. Long Beach: IEEE. https://doi.org/10.1109/CVPR42600.2020.00982.
Li, Minghao, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang et. al. 2023. “TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models.” In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, edited by Brian Williams, Yiling Chen, and Jennifer Neville, 13094–13102. Washington: AAAI Press. https://doi.org/10.1609/aaai.v37i11.26538.
Li, Yuting, Dexiong Chen, Tinglong Tang, and Xi Shen. 2024. “HTR-VT: Handwritten Text Recognition with Vision Transformer.” Pattern Recognition (158): 110967. https://doi.org/10.1016/j.patcog.2024.110967.
Lin, Tsung-Yi, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. “Focal Loss for Dense Object Detection.” In 2017 IEEE International Conference on Computer Vision (ICCV): 2999–3007. Long Beach: IEEE. https://doi.org/10.1109/ICCV.2017.324.
Liu, Wei, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. 2016. “SSD: Single Shot MultiBox Detector.” In Computer Vision – ECCV 2016. Lecture Notes in Computer Science (vol 9905), edited by Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, 21–37. Cham: Springer. https://doi.org/10.1007/978-3-319-46448-0_2.
Lombardi, Francesco, and Simone Marinai. 2020. “Deep Learning for Historical Document Analysis and Recognition – A Survey.” Journal of Imaging 6 (10): 110. https://doi.org/10.3390/jimaging6100110.
Long, Jonathan, Evan Shelhamer, and Trevor Darrell. 2015. “Fully Convolutional Networks for Semantic Segmentation.” In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 3431–3440. Long Beach: IEEE. https://doi.org/10.1109/CVPR.2015.7298965.
Maheswari, S. Uma, P. Uma Maheswari, and G. R. S. Aakaash. 2024. “An Intelligent Character Segmentation System Coupled with Deep Learning Based Recognition for the Digitization of Ancient Tamil Palm Leaf Manuscripts.” Heritage Science (12): 342. https://doi.org/10.1186/s40494-024-01438-4.
Mazzocato, Sara, Dafne Cimino, and Claudia Daffara. 2024. “Integrated Microprofilometry and Multispectral Imaging for Full-field Analysis of Ancient Manuscripts.” Journal of Cultural Heritage (66): 110–116. https://doi.org/10.1016/j.culher.2023.11.014.
McClelland, James L., and David E. Rumelhart. 1981. “An Interactive Activation Model of Context Effects in Letter Perception: Part 1. An Account of Basic Findings.” Psychological Review 88 (5): 375–407. https://doi.org/10.1037/0033-295X.88.5.375.
Ntirogiannis, Konstantinos, Basilis Gatos, and Ioannis Pratikakis. 2014. “ICFHR2014 Competition on Handwritten Document Image Binarization (H-DIBCO 2014).” In 2014 14th International Conference on Frontiers in Handwriting Recognition: 809–813. Long Beach: IEEE. https://doi.org/10.1109/ICFHR.2014.141.
Nikolaidou, Konstantina, Mathias Seuret, Hamam Mokayed, and Marcus Liwicki. 2022. “A Survey of Historical Document Image Datasets.” International Journal on Document Analysis and Recognition (IJDAR) 25 (4): 305–338. https://doi.org/10.1007/s10032-022-00405-8.
Otsu, Nobuyuki. 1979. “A Threshold Selection Method from Gray-Level Histograms.” IEEE Transactions on Systems, Man, and Cybernetics 9 (1): 62–66. https://doi.org/10.1109/TSMC.1979.4310076.
Perino, Michela, Lucilla Pronti, Candida Moffa, Michela Rosellini, and Anna Candida Felici. 2024. “New Frontiers in the Digital Restoration of Hidden Texts in Manuscripts: A Review of the Technical Approaches.” Heritage 7 (2): 683–696. https://doi.org/10.3390/heritage7020034.
Pratikakis, Ioannis, Konstantinos Zagori, Panagiotis Kaddas, and Basilis Gatos. 2018. “ICFHR 2018 Competition on Handwritten Document Image Binarization (H-DIBCO 2018).” In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR): 489–493. Long Beach: IEEE. https://doi.org/10.1109/ICFHR-2018.2018.00091.
Rayner, Keith. 1998. “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 124 (3): 372–422. https://doi.org/10.1037/0033-2909.124.3.372.
Redmon, Joseph, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. “You Only Look Once: Unified, Real-Time Object Detection.” In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 779–788. Long Beach: IEEE. https://doi.org/10.1109/CVPR.2016.91.
Ren, Shaoqing, Kaiming He, Ross Girshick, and Jian Sun. 2017. “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.” IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (6): 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Reul, Christian, Dennis Christ, Alexander Hartelt, Nico Balbach, Maximilian Wehner, Uwe Springmann, Christoph Wick et al. 2019. “OCR4all–An Open-Source Tool Providing a (Semi-)Automatic OCR Workflow for Historical Printings.” Applied Sciences 9 (22): 4853. https://doi.org/10.3390/app9224853.
Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. 2015. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science (vol 9351), edited by Nassir Navab, Joachim Hornegger, William Wells, and Alejandro Frangi, 234–241. Cham: Springer. https://doi.org/10.1007/978-3-319-24574-4_28.
Sauvola, Jaakko, and Matti Pietikäinen. 2000. “Adaptive Document Image Binarization.” Pattern Recognition 33 (2): 225–236. https://doi.org/10.1016/S0031-3203(99)00055-2.
Singh, Sukhdeep, Sudhir Rohilla, and Anuj Sharma. 2025. “An Inclusive Review on Deep Learning Techniques and Their Scope in Handwriting Recognition.” Electronic Letters on Computer Vision and Image Analysis 24 (1): 51–80. https://doi.org/10.5565/rev/elcvia.1986.
Srivastava, Shrey, Amit Vishvas Divekar, Chandu Anilkumar, Ishika Naik, Ved Kulkarni, and V. Pattabiraman. 2021. “Comparative Analysis of Deep Learning Image Detection Algorithms.” Journal of Big Data (8): 66. https://doi.org/10.1186/s40537-021-00434-w.
Stevens, Michael E., and Steven B. Burg. 1997. Editing Historical Documents: A Handbook of Practice. Lanham: Rowman Altamira.
Stokes, Peter A. 2009. “Computer-Aided Palaeography, Present and Future.” In Kodikologie und Paläographie im Digitalen Zeitalter / Codicology and Palaeography in the Digital Age, edited by Malte Rehbein, Patrick Sahle, and Torsten Schassan, 309–338. Norderstedt: BoD.
Tensmeyer, Chris, and Tony Martinez. 2017. “Document Image Binarization with Fully Convolutional Neural Networks.” In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR): 99–104. Long Beach: IEEE. https://doi.org/10.1109/ICDAR.2017.25.
Tonazzini, Anna, Emanuele Salerno, Zienab A. Abdel-Salam, Mohamed Abdel Harith, Luciano Marras, Asia Botto, Beatrice Campanella et al. 2019. “Analytical and Mathematical Methods for Revealing Hidden Details in Ancient Manuscripts and Paintings: A Review.” Journal of Advanced Research (17): 31–42. https://doi.org/10.1016/j.jare.2019.01.003.
van Lit, L.W. Cornelis. 2019. Among Digitized Manuscripts: Philology, Codicology, Paleography in a Digital World. Leiden: Brill. https://doi.org/10.1163/9789004400351.
Wang, Yintong, Wenjie Xiao, and Shuo Li. 2021. “Offline Handwritten Text Recognition Using Deep Learning: A Review.” Journal of Physics: Conference Series (1848): 012015. https://doi.org/10.1088/1742-6596/1848/1/012015.
Wu, Xingjiao, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. 2022. “A Survey of Human-in-the-loop for Machine Learning.” Future Generation Computer Systems (135): 364–381. https://doi.org/10.1016/j.future.2022.05.014.
Zhao, Fanhuna, Hui Ren, Zhibin Su, Xian Zhu, and Chengya Zhang. 2025. “Diffusion-based Heterogeneous Network for Ancient Mural Restoration.” Heritage Science (13): 206. https://doi.org/10.1038/s40494-025-01719-6.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Dumitru Scutelnic, Anna Trovati, Margherita De Michiel, Claudia Daffara
This work is licensed under a Creative Commons Attribution 4.0 International License.
