Design Science Research: How to set-up TEL-related field studies in practice

Design Science Research: How to set-up TEL-related field studies in practice

Monday 23/05 10:30-12:00h

Main Hall

—

Monday 23/05 14:00-15:30h

Outdoor Area A

This session is organized by the DETEL Erasmus+ project.  
DETEL is a strategic partnership among nine European universities and EATEL to reflect their expertise in doctoral education into a new study program in TEL, extended with rich and professionally produced OERs.

Abstract

Much research in TEL is design work: by understanding the addressed domain and collaborating with respective stakeholders, the research team designs an intervention that is intended to support the learning needs of a certain group of learners. Usually, this learning intervention is first designed in multiple iterations and afterwards evaluated in the field. This allows to study how and why a design can address the learning needs, and to test underlying domain as well as theory assumptions at the same time.
Design Science Research aims to add to domain, scientific and design knowledge of how things can and should be designed to achieve a desired set of outcomes. Taking a socio-technical perspective on technology, DSR aims to understand the developed technology in relation to people (users of the technology) and their social context, activities, and working/learning environments. Field studies or “studies in-the-wild” can be seen as evaluation methods in DSR. Within field studies we define how and based on which indicators to measure the outcomes of the developed and designed learning intervention(s).
The goal of this workshop is to provide theoretical knowledge and related “hands-on” activities on how to set-up field studies.

Needs Analysis

Technology enhanced learning typically deals with a specific type of technology or interventions that is somehow referred to learning. However, developing a technology or intervention is not enough. To find out if a technology or interventions does or supports what it is intended to do in a specific context or in the field, a scientific evaluation and corresponding scientific results are of crucial relevance, for both, to proceed with the own work and to publish in science.
Our goal in this workshop is to give PhD candidates a basic scaffolding on how to thoroughly implement DSR and set up respective studies in-the-wild. From our perspective, conducting good field studies to measure the impact of a technology or interventions belongs to the basic trainings for PhD candidates in TEL. With this workshop, we want to give PhD candidates some room for discussions, try outs and exchange with other PhD candidates and experienced researchers to be able to set up field studies using hierarchical principle of evaluating learning interventions and the technology acceptance model. We are aware, that setting up field studies needs some experience and expertise; however, we see especially the JTEL SS as perfect environment for practicing the art of conducting studies “in the wild”.

Learning Objectives

After the workshop, participants will be able to:

• Describe what design science research is.
• Decide when a field study as evaluation method is applicable in relation to their developed technology enhanced learning intervention.
• Apply the basic principles of a field study.
• Apply the hierarchy of evaluation dimensions by Kirkpatrick’s model.
• Apply the levels of the technology acceptance model.
• Set-up a basic study “in the wild”.

Pre-activities

We welcome all PhD candidates, senior researchers, or instructors to join our workshop.
As pre-activity and to get the best out of the workshop, we ask the PhD candidates to prepare the following. Note down …

• A technology or intervention they would like to evaluate in a field study.
• The context or environment where the field study is planned to take place.
• Your desired target groups (who should participate in your study).
• A detailed description of WHAT you want to (or try to) MEASURE in the field study e.g. usability, usefulness, learning outcome, impact…
• Your desired outcomes.

Additionally, we will share relevant papers that we ask all participants to read before the workshop takes place, but, voluntarily.

Session Description

The goal of this workshop is to provide theoretical knowledge about:

• Design Science Research methodology
• Field studies using a hierarchical principle of evaluating learning interventions based on Kirkpatrick & Kirkpatrick
• Technology Acceptance model as baseline for assessing technology acceptance

As “hands-on” activity we will

• elaborate on learning needs and current research activities of the participants
• provide a blueprint for field studies in technology enhanced learning
• ask students to plan a field study as part of design iterations for their own PhD in rough lines individually
• discuss of the participants’ study plans with their peers and the workshop organizers

In more detail, in the first hour, we will present the theoretical knowledge about the design science research and its cycles, the method of field studies in general, the model of Kirkpatrick & Kirkpatrick and the technology acceptance model. All these theoretical information will be presented including striking examples and allow to raise questions. In the second hour, elaborate on the participants’ learning needs, thus which skills and competences are needed and align with the participants’ ongoing research activities.
Furthermore, all attending participants should prepare a rough plan for a field study related to their PhD topic and the technology or intervention they have already developed or will plan to develop including also the context where the field study should be located in. In the last hour, depending on the number of participants, we will initiate a discussion session, either in-peer groups, with instructors or experience researchers and with us.

Post-activities

As we will be available during the whole week of the summer school, we also offer to further discuss with the participants their field study planning after the workshops in detail, if wished.