A Manual for Web Corpus Crawling of Low Resource Languages

Authors

  • Armin Hoenen Institute for Empirical Linguistics Goethe University Frankfurt
  • Cemre Koc Institute for Empirical Linguistics Goethe University Frankfurt
  • Marc Daniel Rahn Institute for Empirical Linguistics Goethe University Frankfurt

DOI:

https://doi.org/10.6092/issn.2532-8816/9931

Keywords:

web information retrieval, low resource languages, corpus generation

Abstract

Since the seminal publication of “Web as Corpus” [1], the potential of creating corpora from the web has been realized for good for the creation of both online and offline corpora: noisy vs. clean, balanced vs. convenient, annotated vs. raw, small vs. big are only some antonyms that can be used to describe the range of possible corpora that can be and have been created. In our case, in the wake of the project Under Resourced Language Content Finder (URLCoFi), we describe a systematic approach to the compilation of corpora for low (or under) resource(d) languages (LRL) from the web in connection with a free eLearning course funded by studiumdigitale at Goethe University, Frankfurt. Despite the ease of retrieval of documents from the web, some characteristics of the digital medium introduce certain difficulties. For instance, if someone was to collect all documents on the web in a certain language, firstly, the collection could only be a snapshot since the web constantly changes content and secondly, there would be no way to ascertain completeness. In this paper, we show ways to deal with such difficulties in search scenarios for LRLs presenting experiences springing from a course about this topic.

[1] A. Kilgarriff and G. Grefenstette, “Web as corpus,” in Proceedings of Corpus Linguistics 2001, 2001, pp. 342–344.

Downloads

Published

2020-05-18

How to Cite

Hoenen, A., Koc, C., & Rahn, M. D. (2020). A Manual for Web Corpus Crawling of Low Resource Languages. Umanistica Digitale, 4(8). https://doi.org/10.6092/issn.2532-8816/9931