Putting the AI in the ApprAIsal of Dialogue: AI-Based Analysis of Classroom and Online Discussions

JTELSS logo
Putting the AI in the ApprAIsal of Dialogue: AI-Based Analysis of Classroom and Online Discussions 📅 Tuesday 19/05 16:00-17:30h 📍 Main Hall 🔎 Needs Analysis Manual analysis of dialogue has always challenged researchers, being time-consuming and often resulting in inconsistencies and low reproducibility. Some of these barriers can now be

Speakers

Noa Brandel
Noa Brandel
The Open University of Israel; Hebrew University of Jerusalem, Israel

Start

End

Putting the AI in the ApprAIsal of Dialogue: AI-Based Analysis of Classroom and Online Discussions

📅 Tuesday 19/05 16:00-17:30h
📍 Main Hall
🔎 Needs Analysis

Manual analysis of dialogue has always challenged researchers, being time-consuming and often resulting in inconsistencies and low reproducibility. Some of these barriers can now be mitigated via the use of AI-driven analysis tools. This creates an urgent need for researchers to understand how AI can meaningfully evaluate educational dialogue. Since PhD candidates in TEL increasingly collect discussion data – whether face-to-face, video-based, or CSCL-written – scalable methods to analyze dialogue across epistemological, social, thematic, and task-related dimensions become an essential part of their research toolkit. As AI-supported analytics and AI-generated evaluation reports offer new methodological possibilities, they raise issues of reliability, transparency, and validity. This workshop addresses these gaps, providing hands-on experience with real discussion data and demonstrating how AI can enhance, rather than replace analytical judgment, while also highlighting its limitations.

📒 Session Description

We will first introduce the AI tools suited for producing evaluation reports of educational dialogue, comparing their strengths and weaknesses. We will then present criteria for effective prompts that generate the analyses under study, and demonstrate outputs produced by different prompts (e.g., participation patterns, thematic clustering, and keyword alignment). Participants will then work hands-on with either their own datasets or a provided transcript. Activities will begin with guided anonymization of discussant names and embedding anonymized identifiers into the transcripts. Participants will subsequently run AI-based analyses using tools of their choice, compare outputs to expectations, revise prompts, and interpret the results. Guided prompts will help identify misclassifications, biases and missing elements. We will conclude with a reflection on reliability, validity, and ethical concerns, and discuss the application of methodological insights to participants’ current research.

💡 Learning Objectives

By the end of the workshop, participants will be able to:

  • Formulate the prompts necessary to generate AI-based analyses of participation patterns, main themes, keyword usage, and task alignment.
  • Anonymize discussion data before uploading materials to AI tools.
  • Generate AI-supported analyses using their own datasets or a provided sample dataset.
  • Validate and critically assess AI-generated analytics, both quantitative and qualitative.
  • Compile validated analyses into a coherent evaluation report (as a downloadable document) in the language of their choice, including discussion of adjustments required for languages with idiosyncratic features such as right-to-left writing systems or special symbols.
  • Compare manual and AI-supported coding approaches, identifying strengths, weaknesses, and potential biases.
  • Address ethical considerations such as privacy, data handling, and accuracy.
  • Transfer the methods and workflows introduced in the workshop to their ongoing dissertation projects.
🧩 Pre-activities and prerequisites

Participants are asked to bring their laptops in order to work with data during the workshop. They should have two files prepared:*

  1. Their own dataset, i.e. at least one transcript of an educational discussion. Discussions may be face-to-face (in-class discussions or interviews), video-based, or written CSCL discussions; transcripts may be manually produced or automatically generated.
  2. A separate file containing the participants’ true names, to be used for anonymization practice.

* If participants are unable to bring their own data, a sample transcript and name list will be provided.
Participants should also be registered with at least one AI tool available to them, preferably under a paid plan, as unpaid versions often do not support all functionalities required for dialogue analysis.
Finally, in preparation for the workshop, participants should briefly reflect on the analytical dimensions and evaluation criteria central to their research (e.g., participation, argumentation, themes).



+ GOOGLE CALENDAR  + ICAL IMPORT