Putting the AI in the ApprAIsal of Dialogue: AI-Based Analysis of Classroom and Online Discussions

Putting the AI in the ApprAIsal of Dialogue: AI-Based Analysis of Classroom and Online Discussions

📅 Tuesday 19/05 16:00-17:30h

📍 Main Hall

🔎 Needs Analysis

Manual analysis of dialogue has always challenged researchers, being time-consuming and often resulting in inconsistencies and low reproducibility. Some of these barriers can now be mitigated via the use of AI-driven analysis tools. This creates an urgent need for researchers to understand how AI can meaningfully evaluate educational dialogue. Since PhD candidates in TEL increasingly collect discussion data – whether face-to-face, video-based, or CSCL-written – scalable methods to analyze dialogue across epistemological, social, thematic, and task-related dimensions become an essential part of their research toolkit. As AI-supported analytics and AI-generated evaluation reports offer new methodological possibilities, they raise issues of reliability, transparency, and validity. This workshop addresses these gaps, providing hands-on experience with real discussion data and demonstrating how AI can enhance, rather than replace analytical judgment, while also highlighting its limitations.

📒 Session Description

We will first introduce the AI tools suited for producing evaluation reports of educational dialogue, comparing their strengths and weaknesses. We will then present criteria for effective prompts that generate the analyses under study, and demonstrate outputs produced by different prompts (e.g., participation patterns, thematic clustering, and keyword alignment). Participants will then work hands-on with either their own datasets or a provided transcript. Activities will begin with guided anonymization of discussant names and embedding anonymized identifiers into the transcripts. Participants will subsequently run AI-based analyses using tools of their choice, compare outputs to expectations, revise prompts, and interpret the results. Guided prompts will help identify misclassifications, biases and missing elements. We will conclude with a reflection on reliability, validity, and ethical concerns, and discuss the application of methodological insights to participants’ current research.

💡 Learning Objectives

By the end of the workshop, participants will be able to:

• Formulate the prompts necessary to generate AI-based analyses of participation patterns, main themes, keyword usage, and task alignment.
• Anonymize discussion data before uploading materials to AI tools.
• Generate AI-supported analyses using their own datasets or a provided sample dataset.
• Validate and critically assess AI-generated analytics, both quantitative and qualitative.
• Compile validated analyses into a coherent evaluation report (as a downloadable document) in the language of their choice, including discussion of adjustments required for languages with idiosyncratic features such as right-to-left writing systems or special symbols.
• Compare manual and AI-supported coding approaches, identifying strengths, weaknesses, and potential biases.
• Address ethical considerations such as privacy, data handling, and accuracy.
• Transfer the methods and workflows introduced in the workshop to their ongoing dissertation projects.