1. Learning Analytics Beyond Traditional Classrooms: Addressing the Tensions of Cognitive and Meta-Cognitive Goals in Exercise Sessions
Zhenyu Cai, Richard Davis, Roland Tormey and Pierre Dillenbourg
Abstract: A range of learning analytics (LA) tools have been designed and integrated into university classes to facilitate teachers’ reflection and orchestration. However, exercise sessions, the educational setting that complements lectures with practical activities, are commonly overlooked by LA researchers and designers. Little work has focused on involving the key stakeholders, teaching assistants (TAs), and incorporating human-centered design approaches in this context. To address this gap, we conducted a qualitative, task-based study to understand the common approaches and challenges for teaching and learning in exercise sessions, and to explore TAs’ visions for LA dashboards that could be adapted into their current practices. Our results indicated that TAs in exercise sessions had markedly different needs compared to instructors in lectures. Specifically, TAs held two levels of goals in supporting students’ cognitive and meta-cognitive activities, and while LA tools were seen as offering numerous potential benefits, they were also seen as introducing tensions threatened to disrupt the delicate balance of goals at both levels. We propose ways of designing dashboards to respect students’ privacy, autonomy, and meta-cognitive development while also helping to meet their learning needs.
2. Design Framework for Multimodal Learning Analytics Leveraging Human Observations
Viktor Holm-Janas, Oriel Caro Miya Marshall, Zaibei Li, Jesper Bruun and Daniel Spikol
Abstract: Collecting and processing data from learning-teaching settings like classrooms is costly and time-consuming for human observers, which limits the types of studies that can be performed. Multimodal Learning Analytics (MMLA) is an avenue to approach in-depth data from multiple streams of data and information. MMLA researchers are working towards more theory-driven development of these systems, emphasizing transparent and explainable data and the availability of these systems. The need for broader and more transparent data in learning science allows researchers to investigate how to leverage the strengths of MMLA and human observers to capture meaningful data from learning activities. This article presents a design framework for leveraging human observations when designing an MMLA system. Supported by a case study using the proposed framework, an MMLA system is developed to measure participation in group work during a classroom physics experiment. This study shows promise in human-understandable measures and analysis in MMLA to make connections between sensor data and human observations. However, it also shows challenges in the rigid nature of automatic data analysis from learning activities and the types of data and observations accessible by sensors.
3. An experimental study into the effects of an advisory dashboard on students’ online and offline learning
Arjen Vetten
Abstract: Learning analytics dashboards can provide students with personalized actionable feedback to enhance students’ learning. Previous studies have shown positive effects of such advisory dashboards on course engagement and self-regulatory activities. To date, no studies have investigated the effects of advisory dashboards on students’ offline learning activities, such as taking and reviewing written notes, while previous research suggests that many students also employ offline learning activities. Using a randomized control trial, the current study investigated the effects of an advisory dashboard on students’ online and offline remediation activities. Before the start of a second-year course, 65 Bachelor of Law students completed a prior knowledge test concerning the topics of a preceding first-year course. The experimental group (n = 30) received their test results and personalized feedback to review particular knowledge clips and quizzes to remediate their prior knowledge. The control group (n = 35) received their test results and a generic, non-personalized remediation advice. A combination of digital trace and questionnaire data measured students’ remediation activities, including the review of knowledge clips, online quizzes, readings and written notes. The findings did not reveal significant effects of the personalized actionable feedback on students’ remediation activities. However, overall students showed a strong preference for offline activities to remediate their prior knowledge. This calls for further studies on students’ offline learning activities in response to personalized actionable feedback.
4. Tracking students’ progress in educational escape rooms
Sonsoles López-Pernas, Aldo Gordillo, Enrique Barra Arias
Abstract: Learning analytics dashboards serve as the primary tool for educators to visually access data and insights concerning teaching and learning. Recent studies have revealed that these dashboards often offer simplistic data visualizations, failing to leverage the latest research advancements in analyzing and portraying the learning process effectively. In this article, we showcase a successful case where we adapted a visualization originally utilized in research for practical implementation by teachers. Specifically, we outline the processes involved in converting and integrating a static sequence analysis visualization into an interactive web format within a learning analytics dashboard. This dashboard aims to monitor students’ temporal trajectories in educational escape rooms in real-time. Through interviews with teachers, we explore how they utilize the dashboard and present a qualitative analysis of their responses.
5. Learning Swedish with AI: Exploring Multimodal Learning Analytics in Language Practice
Hamza Ouhaichi, Daniel Spikol, Bahtijar Vogel and Zaibei Li
Abstract: This study investigates the application of Multimodal Learning Analytics (MMLA) in language practice, specifically within the authentic and dynamic environment of language café settings. We use the MMLA Model for Design and Analysis (MAMDA), which is a design science approach, to systematically explore the requirements for designing the MMLA system. We identify and map three elements: 1) Learning indicators, referring to spoken language learning signs, such as tone, amount and frequency of speech, and pronunciation. 2) Respective modalities and sensors, referring to the format of data to be collected and 3) Analytics models, including NLP models, that can be employed to identify and process the modalities. As a result, we propose a conceptual system that utilizes AI voice assistant as conversational aid, while concurrently collecting audio data for MMLA to enhance language learning experiences. The system is meant for providing insights into learning patterns, participant engagement, and the overall effectiveness of language practice strategies. While presenting a novel system showcaseing the use of AI and data analytics in a unique educational setting, the study’s central focus is to test and critically reflect on MAMDA as a framework for designing and analysing MMLA systems. Therefore we provide a discussion and analysis of how MAMDA was adopted and how it can be a standard guideline for building MMLA solutions.
6. Enhancing Student Motivation through LLM-Powered Learning Environments: A Comparative Study
Kathrin Seßler, Ozan Kepir and Enkelejda Kasneci
Abstract: The integration of ChatGPT and other large language models (LLMs) into educational environments has raised widespread discussions about the potential positive and negative effects. This emphasizes the importance of a solid foundation for the debate based on empirical evidence. To fully understand the impact of LLMs on learning, it is crucial to consider the different aspects of the learning process, including both cognitive and motivational factors, as both play important roles in creating a positive learning experience and outcome. This study investigates the different motivational influences on the learning of university students in a traditional, static environment versus an LLM-supported learning platform. Our study includes 50 participants and focuses on their engagement in learning about a mathematical topic. The study demonstrates a statistically significant (p < 0.001) increase in motivation among participants who use an LLM-powered platform for learning, compared to those who access a static website, with a large effect size (Cohen’s d = -1.387). This suggests that interactive, LLM-driven learning tools can substantially enhance learner engagement and motivation. These findings shed a positive light on the future developments of LLM-based educational platforms.
7. Comparison of Large Language Models for Generating Contextually Relevant Questions
Ivo Lodovico Molina, Valdemar Švábenský, Tsubasa Minematsu, Li Chen, Fumiya Okubo and Atsushi Shimada
Abstract: This study explores the effectiveness of Large Language Models (LLMs) for Automatic Question Generation in educational settings. Three LLMs — GPT-3.5 Turbo, Flan T5 XXL, and Llama 2-Chat 13B — are compared in their ability to create questions from university slide text without fine-tuning. Questions were obtained in a two-step pipeline: first, answer phrases were extracted from slides using Llama 2-Chat 13B; then, the three models generated questions for each answer. To analyze whether the questions would be suitable in educational applications for students, a survey was conducted with 46 students who evaluated a total of 246 questions across five metrics: clarity, relevance, difficulty, slide relation, and question-answer alignment. Results indicate that GPT-3.5 and Llama 2-Chat 13B outperform Flan T5 XXL by a small margin, particularly in terms of clarity and question-answer alignment. GPT-3.5 especially excels at tailoring questions to match the input answers. The contribution of this research is the analysis of the capacity of LLMs for Automatic Question Generation in education. Further work can focus on fine-tuning the models and prompt engineering to continually improve question alignment and other qualities.
8. BloomLLM: Large Language Models Based Question Generation Combining Supervised Fine-tuning and Bloom’s Taxonomy
Nghia Duong-Trung, Xia Wang and Milos Kravcik
Abstract: Adaptive assessment is challenging, and considering various competence levels and their relations makes it even more complex. Nevertheless, recent developments in artificial intelligence (AI) provide new means of addressing these relevant issues. According to our investigation in the educational domain of AI, foundational LLMs such as ChatGPT can effectively generate simple-level questions in Bloom’s Taxonomy but are unsuccessful with tasks related to higher-level competencies. In this paper, we introduce BloomLLM, a novel adaptation of Large Language Models (LLMs) specifically designed to enhance the generation of educational content in alignment with Bloom’s Revised Taxonomy. BloomLLM performs well across all levels of competencies by providing meaningful, semantically connected questions. It is achieved by addressing the challenges of foundational LLMs, such as lack of semantic interdependence of levels and increased hallucination, which often result in unrealistic and impractical questions. BloomLLM, fine-tuned on ChatGPT-3.5-turbo, was developed by fine-tuning 1026 questions spanning 29 topics in two master courses during the winter semester 2023. The model’s performance, outpacing ChatGPT-4, even with varied prompting strategies, marks a significant advancement in applying generative AI in education. We have publicly made the BloomLLM codes and training datasets available to promote transparency and reproducibility.
9. Evaluation of an LLM-powered student agent for teacher training
Saptarshi Bhowmik, Luke West, Alex Barett, Noudi Zhang, Chihpu Dai, Zlatko Sokolikj, Sherry Southerland, Xin Yuan and Fengfeng Ke
Abstract: As technology continues to advance, there is a growing interest in exploring the potential of generative agents and large language model (LLM)- powered virtual students to revolutionize the field of education. In this work, we present Evelyn AI, a LLM-powered virtual student conversation agent that we developed for pre-service teacher training in a virtual environment. By leveraging and enhancing LLM’s text completion features, Evelyn AI is not only able to generate realistic conversation that adapts to the evolving classroom discussion, but also able to authentically simulate students with a wide-range of characteristics including different baseline conceptual understanding levels and different cognitive-affective states. These features enable personalized and adaptive training, and promote a more engaging and immersive learning experience for pre-service teachers. We describe the design and implementation of Evelyn AI, and report results of our three investigations assessing the utility of Evelyn AI for pre-service teacher training.
10. Integrating generative artificial intelligence tools to develop digital competences in secondary schools
Liron Levy-Nadav, Tamar Shamir-Inbal and Ina Blau
Abstract: Artificial intelligence (AI) technologies can promote significant changes in various areas of society, and in particular, in the field of education. The purpose of this study is to examine the digital competences students need to develop in order to use generative artificial intelligence (GAI) tools for learning purposes. 34 semi-structured interviews were conducted with 17 secondary school teachers in an interval of half a year, and triangulated with analysis of actual learning activities. During the interviews, the participants reported the digital competences their students develop using GAI tools, and the skills they should practice. The digital competences that emerged were analyzed based on the Digital Competence Framework for Citizens 2.2 (DigComp 2.2; Vuorikari et al., 2022) that includes five main areas: Information and data literacy; Communication and collaboration; Digital content creation; Safety; and Problem solving. The Framework also highlights knowledge, skills, and attitudes needed to deal with new digital developments, such as AI. Despite DigComp 2.2’s comprehensiveness, its pre-GAI era foundation necessitates updates to fully address competences needed for GAI use. The study findings indicate that the utilization of GAI tools requires proficiency in all DigComp 2.2 domains, emphasizing critical thinking, information privacy, creating new content with GAI, and ethical tool application. Importantly, employing GAI tools calls for the expansion of the framework regarding the ability to analyze the pros and cons of AI tools, and the user’s ability to choose GAI tools that are appropriate for a given task. The findings suggest refining the framework’s explanation regarding prompt-writing skills and adding the skill of managing ongoing dialogue with GAI tools. Expanding the DigComp 2.2 framework would offer a better reflection on the current integration of GAI tools in education. This expansion would emphasize the necessity of developing competences and practicing informed and beneficial use of GAI.
11. Seeing the Forest from the Trees: Unveiling the Landscape of Generative AI for Education through Six Evaluation Dimensions
Yael Feldman-Maggor, Teresa Cerratto-Pargman and Olga Viberg
Abstract: Artificial intelligence (AI) holds significant promise as a technology that may improve the quality of educational practices. This includes specialized AI-powered technologies tailored for education and general AI-based technologies, including recently popular generative AI tools that stakeholders are increasingly adapting for teaching and learning. Integrating AI tools into educational settings holds numerous potential pedagogical benefits, such as assisting teachers in planning lessons, promoting personalization and student autonomy, or helping to create student collaboration. However, concerns about bias and discrimination linked to the use of these technologies have rapidly emerged. Today, standardized evaluation criteria to assess the potential contribution of such tools to education and their reliability within the learning and teaching context are lacking. To address this gap, we build on an existing taxonomy for the evaluation of open educational resources (OER) to better suit the unique features of generative AI. The result is a six-dimensional evaluation approach that includes descriptive, pedagogical, representational, communication, scientific content, as well as the ethical and transparency dimension. We then apply this approach to examine the educational potential and ethical concerns around 30 AI tools. The analysis facilitates a critical map-ping of the potential and risks of AI-powered technologies in education settings.
12. Rhetor: Providing LLM-based Feedback for Students’ Argumentative Essays
Kexin Yang, Sungjin Nam, Yuchi Huang and Scott Wood
Abstract: This study introduces the design and evaluation of an essay tutoring application based on a large language model (LLM). We created an LLM-based prototype tool to provide personalized feedback to improve students’ argumentative writing and critical thinking skills, based on learning sciences principles (i.e., scaffolding and worked examples). We conducted user testing and interviews with participants from a large U.S.-based educational testing company and writing education experts. Overall, the participants found the system easy to use and can augment human tutors. Based on the results, we presented nine design principles for LLM-based essay-writing tutors.
13. Singular Action, Complex Cognition: An Intelligent Tutoring System in Riichi Mahjong
Marshall An, Mufei He, Wei Yao and John Stamper
Abstract: This paper investigates the application of Intelligent Tutoring Systems (ITS) in facilitating the acquisition of complex cognitive skills within the context of Riichi Mahjong. Riichi Mahjong, also known as Japanese Mahjong, is a domain where singular actions are the result of complex cognitive processes. Utilizing Cognitive Task Analysis (CTA), we have constructed an expert cognitive model and developed an ITS that deconstructs these singular actions into subgoals and subsequently into Knowledge Components (KCs). A pilot observational study provided preliminary evidence of the ITS’s effectiveness in improving skill acquisition. By analyzing learning curves with DataShop and applying the Multi-method Approach to Data-Driven Redesign (MADDRED), we present the proposed ITS redesign to address over-practice. Our future research agenda is to conduct a randomized controlled trial to assess the efficacy of the original versus the redesigned ITS. This study contributes to the broader discourse on the efficacy of ITS in teaching skills where complex cognition manifests in singular actions, paving the way for future advancements in intelligent tutoring across various domains.
14. Leveraging Intelligent Tutoring Systems to Enhance Project-Based Learning in Workforce Training at Community Colleges
Marshall An, Leah Teffera, Mahboobeh Mehrvarz, Bruce Li, Christopher Bogart, Majd Sakr and Bruce McLaren
Abstract: This paper investigates the application of Intelligent Tutoring Systems (ITS) in facilitating the acquisition of complex cognitive skills within the context of Riichi Mahjong. Riichi Mahjong, also known as Japanese Mahjong, is a domain where singular actions are the result of complex cognitive processes. Utilizing Cognitive Task Analysis (CTA), we have constructed an expert cognitive model and developed an ITS that deconstructs these singular actions into subgoals and subsequently into Knowledge Components (KCs). A pilot observational study provided preliminary evidence of the ITS’s effectiveness in improving skill acquisition. By analyzing learning curves with DataShop and applying the Multi-method Approach to Data-Driven Redesign (MADDRED), we present the proposed ITS redesign to address over-practice. Our future research agenda is to conduct a randomized controlled trial to assess the efficacy of the original versus the redesigned ITS. This study contributes to the broader discourse on the efficacy of ITS in teaching skills where complex cognition manifests in singular actions, paving the way for future advancements in intelligent tutoring across various domains.
15. Making Diagnostic Decisions Count: Design and Development of a Virtual Patient Environment for Fostering Medical Education
Saroj Sharma, Carolin Thiel, Daniela Yildiz and Armin Weinberger
Abstract: Misdiagnosis and inadequate clinical reasoning are two major causes of medical errors. Virtual patients (VPs) are found to be efficient tools for building diagnostic competence and clinical reasoning in medical students. However, learning with VPs often suffers from unrealistic and uncommon diagnostic decisions, e.g., prematurely taking bone marrow samples. Another problem is “gaming the system”, i.e., jumping to the correct diagnosis, rather than engaging in practices of clinical reasoning. Here, we present an approach that includes additional game elements, namely counters that keep track of costs of the diagnostic decisions, economic efficiency, patient safety, patient satisfaction, and total diagnosis time. By using these novel counters, the PaFaSi VP platform aims to prevent students from gaming the system, while building critical skills for real-life clinical scenarios. The platform is supplemented by tutorial sessions with debriefings and collaborative activities, which allow the students to first try to solve the problems in risk free environments and later compare, justify, and if needed modify their initial diagnosis. The pilot studies show unanimous acceptance of the platform as a learning tool among medical students with varying degrees of prior knowledge. In qualitative interviews, students refer to the counters as “good reflection”, “increasing the feeling of responsibility”, and evaluated it as an incentive for intense group discussion. PaFaSi has shown to be helpful in designing instructional tools to foster medical education, allowing also medicine instructors to create their own VP environments and patient cases, tune the respective instructional design, and launch VPs for specific learning purposes.
16. Design and Development of an AI-Enhanced Collaborative Chat Platform for Medical Education
Tarkan Üsküdar, Carolin Thiel, Daniela Yildiz, Albulene Grajcevci, Anish Singh, Saroj Sharma and Armin Weinberger
Abstract: Computer-supported collaborative learning (CSCL) can greatly benefit from adaptive scaffolding, which requires analyzing the contributions of each learner and taking actions to facilitate discussion. The advancements in natural language processing and the publication of Generative Pre-trained Transformers (GPT) offer both opportunities and challenges to address these requirements and offer real-time, dynamic support within a collaborative learning chat. In this paper, we present a platform capable of discourse analysis and real-time support with GPT-based Conversational Agents (CA), providing an architecture supporting CA design. Along with the proposed architecture, we examine the limitations of GPT-based CAs and suggest solutions to mitigate them. A case study within medical education fostering collaborative clinical reasoning about simulated patient cases in the PaFaSi environment is presented to demonstrate how an iterative design process involving subject matter experts can improve the performance of GPT-based CAs. Preliminary results of the pilot study show that the PaFaSi CAs provided learners with adequate feedback as well as scaffolding for their collaborative clinical reasoning in most cases. However, one problematic phenomenon emerged with the CA also reassuring suboptimal reasoning strategies of learners. Further soft spots of the CA were prioritizing discussion topics as well as being aware of social aspects like insecurities or discontent of learners. Thus, potential improvement would include enlarging the context data with relevant medical knowledge, fine-tuning a CA for collaboration purposes, and dividing the responsibilities over multiple agents. This positions GPT-based CAs as a promising option for real-time assessment and dynamic support to enhance CSCL.
17. Recommending Is Reflecting: A Surprising Benefit of Social Recommender Systems for Teachers
Elad Yacobson and Giora Alexandron
Abstract: Socially-based recommender systems for teachers utilize the collective wisdom of online teacher communities to assist teachers in locating educational content. Such systems rely on evaluative feedback that teachers provide about learning resources that they have used in order to rank these resources. Thus, the feedback that teachers contribute is the data that ‘fuel’ these recommendation engines. Previous research examined incentive mechanisms that would motivate teachers to provide evaluative feedback, and were based on the view that the act of providing feedback has no direct merit for the recommending teacher. We challenge this viewpoint, hypothesizing that writing a review about a learning resource would necessitate teachers to revisit and critically evaluate their teaching experiences with it, thus promoting reflective thinking. Reflective thinking is often praised for contributing to teachers’ professional development. If this hypothesis is deemed true, it underscores a surprising self-benefit that teachers gain from providing feedback. To investigate this hypothesis, we performed qualitative research that included two analyses. In the first, we applied a verbal analysis protocol to measure the level of reflective thinking within a few hundred reviews of learning resources provided by teachers. To triangulate these findings, we qualitatively analyzed think-aloud protocols from eight teachers as they completed evaluative reviews of learning resources they had used in class. Our findings reveal that writing feedback on learning resources indeed promotes reflective thinking, although the depth and content of the reflection are somewhat limited. Implications of these findings to the design of socially-based educational recommender systems are discussed.
18. Achieving tailored feedback by means of a teacher dashboard? Insights into teachers’ feedback practices
Lena Borgards, Onur Karademir, Sebastian Strauß, Daniele Di Mitri, Marcus Kubsch, Markus Brobeil, Adrian Grimm, Sebastian Gombert, Knut Neumann, Hendrik Drachsler, Maren Scheffel and Nikol Rummel
Abstract: Providing feedback is a crucial element of teaching practice and powerful for improving student learning. Yet, monitoring and assessing individual students’ performance to provide them with feedback tailored to their needs can be challenging, especially due to large class sizes. Teacher dashboards can provide teachers with valuable information about the individual students, thus supporting them in identifying student needs and enabling them to provide feedback aligned with students’ performance. We conducted a field study in physics classrooms to investigate which information displayed on the dashboard teachers based their feedback on, and to what extent teachers used the information to tailor their feedback to students’ needs. To answer our research questions, we analysed data from N = 7 teachers in German secondary schools who used a teacher dashboard that allowed teachers to provide feedback to their n = 225 students. Our findings imply that teachers tailored their feedback to students’ performance predominantly based on visualisations of students’ task completion. In particular, teachers used the dashboard to provide feedback to lower-performing students with the aim to support these students in taking actions to progress in their learning. Thus our study adds empirical evidence that teachers not only make use of teacher dashboards to provide more even feedback and feedback of different types, but also that teachers provide students on different performance levels with different types of feedback.
19. Investigating teachers’ perceptions and needs in whole-school level technology integration
Edna Milena Sarmiento-Márquez, Linda Helene Sillat and Kairit Tammets
Abstract: This study monitored the whole-school implementation of Chromebooks at an Estonian school, where each student and teacher received a personal device. Our research began with a systematic literature review to construct a framework for understanding the whole-school level implementation of 1:1 devices. Following this theoretical groundwork, we closely monitored the implementation of Chromebooks. In this study, we investigate teachers’ experiences with the integration of Chromebooks, focusing on three key aspects: their pedagogical practices, their motivation to engage with Chromebooks through the lens of the Expectancy-Value-Cost (EVC) framework, and their perceptions of support in terms of autonomy, competence, and overall support to incorporate Chromebooks into their teaching. Our analysis is based on the mixed-methods approach, combining descriptive statistics with qualitative data to demonstrate the changes in teachers’ pedagogical practices and perceived motivation and support. Correlation analysis was conducted to explore the relationships between pedagogical practices, motivational factors and perceived support, offering insights into the interconnectedness of these variables in the context of Chromebook integration. Our approach intertwines school support, teacher motivation and pedagogical practices within the TEL context, applying the EVC framework to holistically examine technology adoption. By considering these interconnected dimensions, we shed new light on the complex interplay between institutional support, individual motivation and teaching strategies.
20. Design and orchestration in the age of GenAI: an activity theory perspective
Konstantinos Michos and Ishari Amarasinghe
Abstract: The introduction of GenAI in teaching and learning presents both challenges and opportunities and research shows initial evidence of student and teacher perspectives. However, limited work employs theoretical frameworks to understand the use of GenAI for common educational tasks such as the design and orchestration of student learning experiences. In this article, we use the third generation of cultural-historical activity theory to explain how these educational tasks (design and orchestration) are embedded in activity systems that include common outcomes but also contradictions. The study provides cases of ten pre-service teachers who designed chatbot-based learning activities and potential dialogues of students with chatbots in different teaching subjects in high schools. Activity theory provided a comprehensive theoretical lens to elaborate elements that require attention when designing and orchestrating learning tasks that involve GenAI.
21. Beyond Search Engines: Can Large Language Models Improve Curriculum Development?
Mohammad Moein, Mohammadreza Molavi, Abdolali Faraji, Mohammadreza Tavakoli and Gábor Kismihók
Abstract: While Online Learning is growing and becoming widespread, the associated curricula often suffer from a lack of coverage and outdated content. In this regard, a key question is how to dynamically define the topics that must be covered to thoroughly learn a subject (e.g., a course). Large Language Models (LLMs) are considered candidates that can be used to address curriculum development challenges. Therefore, we developed a framework and a novel dataset, built on YouTube, to evaluate LLMs’ performance when it comes to generating learning topics for specific courses. The experiment was conducted across over 100 courses and nearly 7,000 YouTube playlists in various subject areas. Our results indicate that GPT-4 can produce more accurate topics for the given courses than extracted topics from YouTube video playlists in terms of BERTScore.
22. Learning Analytics-Supported Learning Design for a Dutch Distance Learning University
Seyyed Kazem Banihashem, Maryam Alqassab, Konstantinos Georgiadis, Marcel Schmitz and Hendrik Drachsler
Abstract: The existing literature acknowledges the potential of learning analytics in enhancing effective learning design, but only when the learning analytics elements are integrated at the learning design stage. Previous research showed successful attempts to develop learning analytics-supported learning designs using the learning design tool ” FoLA2″ (Fellowship of the Learning Activity and Analytics), which was created for this purpose. However, the learning context plays a major role in successful implementation as different instructional settings require customized approaches for leveraging learning analytics indicating one-size-fits-all solution is not practical. Similarly, ‘inclusive learning’ requires contextualized learning analytics-supported learning designs to be effectively adopted by teachers and educational institutions. Addressing this research gap, this case study reports on the customization of the FoLA2 tool for the context of distance university education. This research was conducted at the Open University of the Netherlands (OUNL) which is known for its diverse student population. Building on the activating distance education model of the OUNL, we conducted interviews with teachers and analyzed different education documents of the OUNL to guide the customization of the FoLA2 tool for the specific needs and the instructional conditions of the OUNL. The results of this study can provide insights into the design of learning analytics-supported learning design within a distance education institution that caters for learners with varying individual characteristics and learning conditions.
23. Multimodal Sensing of Goals and Activities During Interactions With a Co-created Robot
Paras Sharma, Veronica Bella, Angela E.B. Stewart and Erin Walker
Abstract: Culturally responsive computing (CRC) curricula engage learners in reflections on power and identity as they build technologies. Open-design tasks, with open goals and pathways, that connect to learner experiences are common in CRC and could be enhanced using adaptive technologies. Current adaptive technologies function best in well-defined learning trajectories. However, it is unclear how to design these technologies to respond to individual learners’ ideas in open-design settings, particularly when learners have a high degree of agency in defining their goals. In this paper, we prototype a learning system that uses multimodal sensing, log data, and reflective dialogues to build explanatory learner models in open-design settings. We implement and deploy our system in a 2-week summer camp with middle school girls and analyze the collected data to evaluate the effectiveness of our system to understand learner goals and their processes for achieving them. We show the importance of multimodal interaction pathways in open-ended tasks to accommodate diverse learner preferences and suggest interaction strategies for future adaptive systems supporting open-ended learning activities.
24. Are you up for DigiTech? The role of internal and external drivers in the adoption of digital technology in education
Adriaan Vervoort, Lisa Koutsoviti Koumeri, Nuria González Castellano and Katrien Struyven
Abstract: Integration of digital technology in the learning environment is met with a multitude of barriers that need to be overcome prior to harnessing the complete array of potential advantages for teaching and learning. While many studies have explored the extrinsic (institutional context) and intrinsic barriers (beliefs and attitudes) from a teachers’ perspective respectively, the relationship between both is underexposed in research. In this study, data collected using an adapted version of the SELFIE questionnaire was used to cluster teachers’ responses (n = 269) and identify different profiles. K-means clustering was applied followed by a multinomial regression analysis to explore the relation between the clusters and demographic predictors. The results reveal three meaning- ful clusters that can be explained by internal drivers (engagement based on attitudes) and external drivers (perceived support). Cluster 1 shows high engagement towards using technology and is positive about support. Cluster 2 shows low engagement towards technology use and expresses mediocre support. Cluster 3 also expresses mediocre support like the sec- ond group, however, their engagement is as high as the first. The cluster groups are associated with teachers’ innovation profile, confidence, and the school they work at. Differentiating between teachers based on inter- nal and external drivers appears relevant and is recommended as a good practice to address teachers’ needs within the digital transformation of a school organization.
25. The Impact of Robotic Programming Environments on Computational Thinking with an Effect on Word Reading Fluency and Decoding
Shelley Van Bergen and Nardie Fanchamps
Abstract: Word reading fluency and decoding can be important determinants when primary school students learn programming in order to develop computational thinking (CT). The question that arises is whether an interaction can be established regarding a growth on CT and the level of reading ability of students. Therefore, our experimental research aims to investigate whether different robotic programming environments can be applied to improve students’ level of CT and thus reading skills. The study was conducted among Dutch primary school students grade 3 and 4, aged 6 to 8. For both experimental groups, either BeeBot or Ozobot were applied as robotics programming interventions. The control group followed regular reading lessons. Reading exercises to be solved through programming applications were designed for both experimental groups. Via pre-test-posttest design, proficiency on CT was determined by administering the Beginners Computational Thinking Test (BCTt). To establish reading proficiency, the “three-minute test” (DMT) was administered. Our research indicates a correlation between a development on CT and the level of reading fluency and decoding. Although not all subcategories of CT are a predictor on reading skills, programming tangible or code-based robotics did prove to be an explanatory model regarding the results of the reading test taken. The findings obtained imply that the application of programming and related CT development may be beneficial for the development of reading ability. Follow-up research is recommended to be able to make more specific statements on the effect of development on CT related to the reading level of students in primary education.