Can artificial intelligence truly understand the subtleties of human politics? A new study from the University of Copenhagen, led by Ilias Chalkidis, Stephanie Brandl, and Paris Aslanidis, throws cold water on that idea. Their research delves into the surprisingly complex task of using AI to identify populism in political speech, revealing unexpected limitations in even the most advanced language models.
The Elusive Nature of Populism
Populism, that potent cocktail of anti-establishment sentiment and appeals to “the people,” isn’t easily defined. Scholars have wrestled with its multifaceted nature for decades, with three main schools of thought emerging: top-down manipulation theories, ideational approaches, and discursive perspectives. Chalkidis, Brandl, and Aslanidis chose a discursive approach, focusing on the specific language used to construct a binary opposition between “the people” and “the elites.” This focus on language made the task more amenable to AI analysis.
Testing AI’s Political Acumen
The researchers evaluated a range of pre-trained language models, both open-source and proprietary, including giants like Meta’s Llama, Google’s Gemini, and OpenAI’s GPT-4. They fed these models datasets of Donald Trump’s speeches, meticulously labeled for instances of people-centrism (positive references to “the people”) and anti-elitism (negative references to “the elites”). The co-occurrence of both was considered “fully populist.”
The results were, to put it mildly, underwhelming. Even the most advanced models struggled to reliably identify populist discourse, often misinterpreting common phrases or failing to grasp the context necessary for accurate interpretation. A fine-tuned RoBERTa classifier performed best, but even its accuracy was far from perfect, highlighting the challenge of teaching AI to understand nuanced human concepts.
Why This Matters
The inability of cutting-edge AI to accurately detect populism has significant implications. Accurate identification of populist rhetoric is crucial for researchers studying political discourse, for journalists seeking to understand political narratives, and even for social media platforms battling the spread of misinformation. If the most sophisticated AI systems fail at this task, it points to the immense complexity involved and the limitations of current AI capabilities.
Beyond Trump: Generalizability and Cross-Cultural Challenges
The researchers extended their analysis beyond Trump, testing their models on speeches by European politicians. Here, they observed a surprising shift: instruction-tuned LLMs, those trained to follow instructions, outperformed the fine-tuned RoBERTa, suggesting that these models may be more robust when dealing with diverse data. This underscores the complex interplay between training data and model architecture in AI performance.
However, the study also revealed the limits of cross-cultural transfer. The models were less accurate when applied to speeches in other languages, indicating that cultural context plays a significant role in interpreting political rhetoric. This highlights the need for more diverse and representative training datasets to improve the generalizability of AI in analyzing political discourse.
A New Lens on Populism
This study isn’t just about the limitations of AI; it offers a new perspective on populism itself. The researchers’ findings suggest that populism isn’t a static ideology, but rather a dynamic rhetorical strategy, constantly shifting and adapting. Their fine-grained analysis, breaking down speeches into individual sentences and measuring the intensity of populist rhetoric over time, sheds light on the strategic deployment of these appeals.
The Future of AI and Political Analysis
The work of Chalkidis, Brandl, and Aslanidis serves as a cautionary tale. While AI holds immense potential for analyzing large datasets of political speech, it’s crucial to acknowledge its limitations. These models are not yet capable of fully understanding the subtle nuances of human language and the complex sociopolitical contexts that shape political rhetoric. Future research must focus on developing more robust methods, incorporating contextual information effectively, and creating more diverse and representative training datasets. Only then can AI truly contribute to our understanding of the complexities of populism and the ever-evolving landscape of political discourse.
