ChatGPT-4 and Italian Dialects: Assessing Linguistic Competence

Abstract

The purpose of this study is to evaluate ChatGPT-4’s language proficiency in Italian dialects. At the outset, it is clarified what is meant by ‘language ability’ within the context of Large Language Models. This involves identifying the tasks that ChatGPT might face in real-world scenarios, from which we can derive inferential assumptions regarding its linguistic ability. The skills identified, which served as foundation for test design, include comprehension and translation, dialect recognition, analysis of the distinctive features, error detection, text production, interaction, theoretical background, and self-assessment. The tests were crafted to mimic situations requiring these competencies, trying to emulate authentic ChatGPT-User interactions. The results highlight ChatGPT’s excellent prowess in understanding and recognizing Italian dialects and their subvarieties, and a robust background and awareness of its own knowledge. However, the model exhibits significant gaps in analytical skills and struggles with text production and interactive tasks, suggesting superior passive linguistic capabilities compared to active ones.