The role of specialist evaluation

How are we to measure the effect of iconomix, when it consists of a multitude of elements ranging from individual tasks to educational games and entire teaching units?

It would be difficult to assess the overall impact of iconomix’s diverse offering on its entire, heterogeneous target group (upper secondary schools). Also, this kind of comprehensive assessment would stand in contradiction to the flexible way in which the tool is deployed in practice.

For this reason, instead of performing a comprehensive evaluation of iconomix as a whole, we concentrate on the quality of its individual teaching units. Each teaching unit is subject to evaluation, either in the course of its elaboration or as it is developed further.

This allows us to limit the evaluation to two dimensions: first, in terms of the impact of individual teaching materials (the teaching units), and second, as regards a rather more homogeneous component of upper secondary school instruction, since the individual teaching units tend to be oriented to a particular type of school and subject.

We impose a third restriction as well. We refrain from undertaking a summative evaluation, for the following reason: iconomix provides instructors with well-founded, high-quality teaching and learning resources. The instructors themselves are, however, responsible for the actual instruction, for which they have been specially trained. While iconomix supports instruction with didactic notes, tutorials, peer-to-peer workshops, etc., what happens in the classroom is up to the teacher. A meticulous assessment of the effectiveness of an ideal teaching unit is therefore of only limited practical relevance.

This is why we focus not on the success of such a unit as measured by academic means, but rather use formative evaluation to gradually improve the professional quality of our educational resources in an iterative process with multiple loops. Every year, we invest almost 10 percent of our resources in this endeavour.

The scientific evaluation of iconomix involves the following four methods:

• The first method deployed is known as the DBR (design-based research) approach. DBR comprises field tests in the classroom, including for example video work, participant observation and qualitative interviews. Our evaluation partners are educational experts from Swiss universities experienced in design-based research. When possible, their findings are published as articles in academic journals.

• Secondly, digital educational resources, in particular, are reviewed and certified by independent professional evaluators such as the Education Alliance Finland (EAF). For instance, the EAF assesses student activities in relation to their learning objectives and their commitment to learning, as well as the quality of the instructional material. This process leads to a report and a certificate; the report’s core findings are published online.

• A third approach involves our working closely with renowned educational experts from Swiss academic institutions on the development of assignments, the true centrepiece of every teaching unit. These experts provide us with practical recommendations on the draft assignments we present them for review, which we then implement as faithfully as possible.

• Finally, we perform trial runs of more complex teaching units before scaling them and presenting them for general use. This allows us to identify and address problems that may go unnoticed during the conception phase. A teaching unit is labelled ‘beta version’ for the duration of the pilot phase, which may last a year or more.