Can large language models support critical discourse analysis? A pilot experiment
DOI:
https://doi.org/10.60923/issn.2532-8816/23143Keywords:
critical discourse analysis, Large Language Models, artificial intelligence, critical discourse studies, quantitative analysisAbstract
This article presents a pilot experiment that explores the use of Large Language Models (LLMs) in the context of Critical Discourse Analysis (CDA). The study investigates the extent to which an LLM can reproduce ideologically oriented discourse analyses. The proposed approach involves constructing a consensus-based gold standard from the annotations of three human raters, which was then used to evaluate the accuracy of an automated analysis performed by the LLM. The case study examines a corpus of thirty opinion articles from ideologically diverse newspapers to investigate how the October 7 attack was portrayed in the media. The results indicate that LLMs perform well, particularly with respect to macro- and superstructural features, but may struggle with microstructural phenomena such as euphemism detection, underscoring their potential role as supporting tools rather than substitutes for human analysis.
References
Artstein, Ron, and Massimo Poesio. 2008. “Inter-Coder Agreement for Computational Linguistics.” Computational Linguistics 34 (4): 555–596. https://doi.org/10.1162/coli.07-034-R2.
Augenstein, Isabelle, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Filippo Menczer, Ruben Miguez, Preslav Nakov, Dietram Scheufele, Shivam Sharma, and Giovanni Zagni. 2024. “Factuality challenges in the era of large language models and opportunities for fact-checking.” Nature Machine Intelligence, 6(8), 852–863. https://doi.org/10.1038/s42256-024-00881-z
Chew, Ryan, Jonathan Bollenbacher, Michael Wenger, Jacob Speer, and Alice Kim. 2023. “LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding (Version 1).” arXiv. https://doi.org/10.48550/ARXIV.2306.14924.
Cohen, Jacob. 1960. “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement 20 (1): 37–46. https://doi.org/10.1177/001316446002000104.
Curry, Neil, Paul Baker, and Gavin Brookes. 2024. “Generative AI for Corpus Approaches to Discourse Studies: A Critical Evaluation of ChatGPT.” Applied Corpus Linguistics 4 (1): 100082. https://doi.org/10.1016/j.acorp.2023.100082.
DeJeu, Elizabeth B. 2025. “Can (and Should) LLMs Perform Critical Discourse Analysis?” Journal of Multicultural Discourses 19 (3): 188–195. https://doi.org/10.1080/17447143.2025.2492145.
Fairclough, Norman. 1995. Critical Discourse Analysis. London: Longman.
Fairclough, Norman, Jane Mulderrig, and Ruth Wodak. 2011. “Critical Discourse Analysis.” In Discourse Studies: A Multidisciplinary Introduction, edited by Teun A. van Dijk, 357–378. London: SAGE Publications Ltd. https://doi.org/10.4135/9781446289068.n17.
Fleiss, Joseph L. 1971. “Measuring Nominal Scale Agreement among Many Raters.” Psychological Bulletin 76 (5): 378–382. https://doi.org/10.1037/h0031619.
Gillings, Michael, Thomas Kohn, and Gerlinde Mautner. 2024. “The Rise of Large Language Models: Challenges for Critical Discourse Studies.” Critical Discourse Studies, 1–17. https://doi.org/10.1080/17405904.2024.2373733.
Holsti, Ole R. 1969. Content Analysis for the Social Sciences and Humanities. Reading, MA: Addison-Wesley.
Kaneva, Nadia, and Stewart M. Hoover. 2009. Fundamentalisms and the Media. New York: Continuum.
Kilgarriff, Adam, Vít Baisa, Jan Bušta, Miloš Jakubíček, Vojtěch Kovář, Jan Michelfeit, Pavel Rychlý, and Vít Suchomel. 2014. “The Sketch Engine.” Lexicography 1 (1): 7–36. https://doi.org/10.1007/s40607-014-0009-9.
Krippendorff, Klaus. 2013. Content Analysis: An Introduction to Its Methodology. 3rd ed. Thousand Oaks, CA: SAGE Publications, Inc.
McCombs, Maxwell E., and Donald L. Shaw. 1972. “The Agenda-Setting Function of Mass Media.” Public Opinion Quarterly 36 (2): 176–187. https://doi.org/10.1086/267990.
Missier, Chiara A. 2021. “Framing Fundamentalism in the Digital Media Space.” International Communication Studies 30 (1): 20–41.
Van Dijk, Teun A. 1983. “Discourse Analysis: Its Development and Application to the Structure of News.” Journal of Communication 33 (2): 20–43. https://doi.org/10.1111/j.1460-2466.1983.tb02386.x.
Van Dijk, Teun A. 1985. “Structures of News in the Press.” In Discourse and Communication, edited by Teun A. van Dijk, 69–93. Berlin: De Gruyter. https://doi.org/10.1515/9783110852141.69.
Van Dijk, Teun A. 1988. News as Discourse. Hillsdale, NJ: L. Erlbaum Associates.
Van Dijk, Teun A. 1993a. Elite Discourse and Racism. Thousand Oaks, CA: SAGE Publications, Inc.
Van Dijk, Teun A. 1993b. “Principles of Critical Discourse Analysis.” Discourse & Society 4 (2): 249–283. https://doi.org/10.1177/0957926593004002006.
Van Dijk, Teun A. 1995a. “Discourse Analysis as Ideology Analysis.” In Language and Peace, edited by Christina Schäffner e Anita Wenden, 17–33. Aldershot: Dartmouth.
Van Dijk, Teun A. 1995b. “Discourse Semantics and Ideology.” Discourse & Society 6 (2): 243–289. https://doi.org/10.1177/0957926595006002006.
Van Dijk, Teun A. 1998. “Opinions and Ideologies in the Press.” In Approaches to Media Discourse, edited by Allan Bell e Peter Garrett, 21–63. Oxford: Blackwell.
Zhu, Jing, and Yang Liu. 2024. “The Representation of Israel–Hamas Conflict in the American and Chinese News Coverage: A Corpus-Assisted Comparative Critical Discourse Analysis.” International Journal of Social Sciences and Public Administration 4 (2): 360–378. https://doi.org/10.62051/ijsspa.v4n2.49.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Francesca Cristiano, Emiliano Giovannetti
This work is licensed under a Creative Commons Attribution 4.0 International License.
