Field Studies in TEL

Methodology Workshop

Much research in TEL is design work – i.e., together with the relevant stakeholders, the research team designs an intervention that is intended to support learning. This intervention needs to be evaluated alongside the design process to show the extent to which this goal has been reached for the prototypical iteration; and to gain additional insights that are sought for.

Field studies are one main type of evaluations. They are challenging to set up and in case of a bad study design cannot be easily repeated due to efforts and costs of running a field study. As well, field studies need to be meaningfully positioned in the design projects to inform next steps such as important design decisions.

The goal of this lecture and workshop is:

• To provide a blueprint for field studies in technology enhanced learning following the hierarchical evaluation of learning interventions from Kirkpatrick & Kirkpatrick (see below)
• To have students plan a field study for their own PhD in rough lines individually
• To discuss their plans with peers and the lecturer and other senior researchers who may be present – i.e., students will get feedback on their own plan
• To explore criteria for properly embedding field studies in design work: e.g. clarifying questions that are difficult to be answered in the design process and gaining evidence for important assumptions

The blueprint for field studies is to evaluate in a hierarchy of research questions/evaluation level: First, one assesses the observable (learning) activities that are carried out – this helps understand the success of the intervention and it is possible to identify problems. Second, one assesses concrete learning insights that are generated. Thirdly, one assesses a change in task performance, preferably with a mix methods approach. Sometimes, also the impact on a wider social entity (e.g., an organization) is relevant.