USAGE
ANALYSIS IN LEARNING SYSTEMS
Edited by Christophe Choquet, Vanda Luengo, and Kalina Yacef
USAGE ANALYSIS IN LEARNING SYSTEMS Edited by Christophe Choquet, Vanda Luengo, and Kalina Yacef
Association for the Advancement of Computing in Education www.aace.org
EDUCATION & INFORMATION TECHNOLOGY DIGITAL LIBRARY he EdITLib is your source for 15+ years (with your 1 year T subscription) of peer-reviewed, published articles (20,000+) and papers on the latest research, developments, and applications related to all aspects of Educational Technology and E-Learning.
• Conduct scholarly research • Receive Table of Contents alerts • Keep current on the latest research and publications in your field
Journals – www.aace.org/pubs • International Journal on E-Learning (Corporate, Government, Healthcare, & Higher Education) • Journal of Computers in Mathematics and Science Teaching • Journal of Educational Multimedia and Hypermedia • Journal of Interactive Learning Research • Journal of Technology and Teacher Education • AACE Journal (electronic) • Contemporary Issues in Technology & Teacher Education (electronic)
Conference Papers/Presentations – www.aace.org/conf ED-MEDIA – World Conference on Educational Multimedia, Hypermedia & Telecommunications E-Learn – World Conference on E-Learning in Corporate, Healthcare, Government, and Higher Education SITE – Society for Information Technology and Teacher Education International Conference and others...
Please Recommend Your Library Subscribe!
www.EdITLib.org Sponsored by AACE:
[email protected] 757-366-5606
USAGE ANALYSIS IN LEARNING SYSTEMS
Articles Preface: Usage Analysis in Learning Systems Christophe Choquet, Vanda Luengo, and Kalina Yacef . . . . . . . . . . . . . . . . . . . . . .7 Modeling Tracks for the Model Driven Re-engineering of a TEL System Christophe Choquet and Sebastien Iksal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Results from Action Analysis in an Interactive Learning Environment Oliver Scheuer, Martin Mühlenbrock, and Erica Melis . . . . . . . . . . . . . . . . . . . . . .33 Towards Live Informing and Automatic Analyzing of Student Learning: Reporting in ASSISTment System Mingyu Feng and Neil T. Heffernan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55 Beyond Logging of Fingertip Actions: Analysis of Collaborative Learning Using Multiple Sources of Data Nikolaos Avouris, Georgios Fiotakis, Georgios Kahrimanis, Meletis Margaritis, and Vassilis Komis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79 Monitoring an Online Course with the GISMO Tool: A Case Study Riccardo Mazza and Luca Botturi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99 Matching the Performed Activity on an Educational Platform with a Recommended Pedagogical Scenario: A Multi-Source Approach Jean-Charles Marty, Jean-Mathias Heraud, Thibault Carron, and Laure France . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115 From Data Analysis to Design Patterns in a Collaborative Context Vincent Barre, Christophe Choquet, and Hassina El-Kechaï . . . . . . . . . . . . . . . .133 A Structured Set of Design Patterns for Learners’ Assessment Élisabeth Delozanne, Françoise Le Calvez, Agathe Merceron, and Jean-Marc Labat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .157
Usage Analysis in Learning Systems (ISBN 1-880094-69-X) is published by the Association for the Advancement of Computing in Education (AACE), an international, educational, nonprofit organization. Published by: AACE, PO Box 1545, Chesapeake, VA 23327-1545, USA 757-366-5606; Fax: 703-997-8760; E-mail:
[email protected] © Copyright 2009 by AACE. Website: http://www.aace.org
Usage Analysis in Learning Systems, 7-8
PREFACE
Usage Analysis in Learning Systems CHRISTOPHE CHOQUET LIUM/IUT de Laval, France
[email protected] VANDA LUENGO LIG/University Joseph Fourier, Grenoble, France
[email protected] KALINA YACEF University of Sydney, Australia
[email protected] This book explores different approaches for analyzing student data collected electronically in computer-based learning activities in order to support teaching and learning. The articles selected for this special issue are extended versions of the papers presented at the workshop on Usage Analysis in Learning Systems, organized in conjunction with the 12th International Conference on Artificial Intelligence (Amsterdam, The Netherlands, July 2005). Learning systems track student usage in order to dynamically adapt some aspects of the contents, format and order of the learning material to the individual student. These large amounts of student data can also offer material for further analysis using statistical and data mining techniques. In this emerging field of research, approaches for such analysis vary greatly depending on the targeted user (e.g., teacher, learner, instructional designer, education researcher, parent, software engineer) and what the analysis is intended for. The articles presented in this special issue cover various stages of the data’s "cycle of life:" Usage tracking modeling: Choquet and Iksal recommend that the software development process should explicitly integrate a usage analysis phase, as it can provide designers with significant information on how their systems are used for reengineering purposes. They present a generic usage tracking language (UTL) and describe an instantiation with IMS-Learning Design. Usage data analysis: Scheuer, Mühlenbrock and Melis propose a system,
8
Choquet, Luengo, and Yacef
SIAM, (System for Interaction Analysis by Machine learning), working towards an automatic analysis of interactive data based on standard database server and machine learning techniques. Feng and Heffernan present their web-based system (ASSISTment), which was used to support teachers in middle school mathematics classes and which provides a range of reports providing real time reporting to teachers in their classroom. Usage data visualization: Avouris, Fiotakis, Kahrimanis, Margaritis and Komis propose a software environment, Collaborative Analysis Tool, (ColAT) that supports inter-relation of resources in order to analyse the collected evidence and produce interpretative views of the activity. Mazza and Botturi present GISMO, an open source, graphic student-tracking tool integrated into Moodle. GISMO provides visualizations of behavioral, cognitive and social data from the course, allowing constant monitoring of students’ activities, engagement and learning outcomes. Marty, Heraud, Carron and France observe learners’ behavior within a web-based learning environment to understand it. The observations are gathered from various sources and form a trace, which is finally visualized. Usability of data: Barre, Choquet, and El-Kechaï propose design patterns for recording and analyzing usage in learning systems taking into account multiple perspectives – the role of the user, the purpose of the data analysis, and the type of data. Delozanne, Le Calvez, Merceron, and Labat present design patterns which objective is to provide support for designers to track and analyze the use of a learning system by its different actors. We hope this special issue will stimulate ideas to grow this exciting field of research! We would like to thank all the reviewers of this special issue for their constructive feedback. Christophe Choquet, Vanda Luengo and Kalina Yacef
Usage Analysis in Learning Systems, 9-32
Modeling Tracks for the Model Driven Re-engineering of a TEL System CHRISTOPHE CHOQUET LIUM/IUT de Laval, France
[email protected] SEBASTIEN IKSAL LIUM/IUT de Laval, France Sé
[email protected] In the context of distance learning and teaching, the re-engineering process needs a feedback on the learners' usage of the learning system. The feedback is given by numerous vectors, such as interviews, questionnaires, videos or log files. We consider that it is important to interpret tracks in order to compare the designer’s intentions with the learners’ activities during a session. In this article, we present the usage tracking language (UTL). This language was designed to be generic and we present an instantiation of a part of it with IMS-Learning Design, the representation model we chose for our three years of experiments. At the end of the article, we describe several use cases of this language, based on our experimentations.
Introduction Nowadays, numerous interactive systems are available on the Web. Most of these systems need some kind of feedback on the usage in order to improve them. In the specific context of distance learning and teaching, the desynchronization between teachers’ two major roles – instructional designer and tutor – brings about a lack of uses feedback. The software development process should explicitly integrate a usage analysis phase, which can provide designers with significant information on their systems’ uses for a re-engineering purpose (Corbière & Choquet, 2004a). Semantic Web aims at facilitating data management on the Web. It brings languages, standards and corresponding tools that make the sharing and building of automatic and semi-automatic programs easier (Berners-Lee, 1999). Automatic usage
10
Choquet and Iksal
analysis is often made by mathematicians or computer engineers. In order to facilitate the appropriation, the comprehension and the interpretation of results by instructional designers, who are the main actors of an e-learning system development process, we think they should be fully integrated in this analysis phase. The research contribution we present in this article is fully in line with our approach to the engineering and re-engineering of e-learning systems, where we particularly stress the need for a formal description of the design view, in terms of scenarios and learning resources, to help the analysis of observed uses (i.e., descriptive scenarios) and to compare them with the designer's intention (i.e., predictive scenario) (Corbière & Choquet, 2004b; Lejeune & Pernin, 2004), in order to enhance the quality of the learning. When designers use an Educational Modeling Language (EML) such as Learning Design (Koper, Olivier, & Anderson, 2003) proposed by IMS Global Learning Consortium (IMS, 2006) to explicit their intention regarding the learners’ activities during a session, a set of observation needs are implicitly defined. Thus, one of the student data analysis difficulties resides in the correlation between these needs and the tracking means provided by the educational environment (not only the computer-based system, but also the whole learning organization, including humans and data collection vectors such as video recorders, questionnaires, etc.). Our aim is to provide the actors of a Technology Enhanced Learning (TEL) System with a language dedicated to the description of the tracks and their semantics, including the definition of the observation needs and the means required for data acquisition. This language, the Usage Tracking Language (UTL), aims to be neutral regarding technologies and EMLs, and has to be instantiated on the EML used to define the pedagogical scenario and on the tracks format. Moreover, this language allows the structuring of tracks, from raw data – those acquired and provided by the educational environment during the learning session – to indicators (ICALTS, 2004) which mean something significant for its user. They usually denote a significant fact or event that happened during the learning session, on which users (designers, tutors, learners, and analysts) could base some conclusions concerning the quality of the learning, the interaction or the learning environment itself. A first part of our proposal has been developed today; it focuses only on the transformation of tracks by adding semantic. All the systems which need to analyse the user behaviour work with data-mining techniques (Mostow, 2004) or by hand. These techniques are often used to build user models or to adapt the content or the layout to the user (Zheng, Fan, Huan, Yin, Wei-Ying, & Liu, 2002). They are based on statistical or mathematical analyses (Bazsaliscza & Naim, 2001). Very often, these kinds of analysis have the particularity to work on raw data in order to bring out some indicators such as
Modeling Tracks for the Model Driven Re-engineering of a TEL System
11
the most frequently used keyword. In our case, we are interested in analysing the user behaviour, to improve the pedagogical scenario and the learning materials. Our proposal consists of an analysis driven by models. We want to guide the data analysis by means of the instantiated learning scenario, so we focus on the representation model of a scenario and the tracks’ format. The next section deals with the different viewpoints one could have on tracks depending on his role: designer, tutor, learner, or analyst. We introduce here the DGU model, for Defining, Getting and Using tracks, which allows the focus on three different facets of a track. Then, after having identified several kinds of data which are relevant and used for analyzing a learning session, we present their information models. The third section of this article presents the operational part of the Usage Tracking Language, namely UTL.1, which allows the description of the track semantics recorded by an LMS and to link them to observation needs defined in the predictive scenario. This language could be instantiated both on the formal language used to describe the pedagogical scenario and in the track file format implemented by the LMS. It could be considered as a first operational version (i.e., a subset) of the language presented by the previous section, which would aim to cover a wider scope of tracks and situations. In a fourth part, we provide several use cases which highlight the possibilities of this language. Finally, we conclude this article with some ideas concerning the next version of UTL, namely UTL.2. All the examples cited in this article are taken from a number of tests we have made with our students over the last three years. They all concern a learning system which is composed of six activities designed for teaching network services programming skills. We used the Free Style Learning system (Brocke, 2001), based on Open-USS LMS (Grob, Bensberg, & Dewanto, 2004), in which students can navigate as they choose to between all the activities. Our designers have defined a predictive scenario and, each year, we have compared this scenario with descriptive ones, by hand, for a reengineering purpose. TRACKS MODEL Three Viewpoints on Tracks: The DGU Model The practice of tracking is common (Hanson, Kraut, & Farber, 1984). The main problem of this method is the significant amount of data recorded. Therefore, one has to formulate hypotheses in order to conduct analysis, extract relevant information and link them. The relevance of this method was studied by Teubner and Vaske (1988). In our re-engineering framework, we stress the need for track modeling before the learning session, and to consider tracks as a pedagogical object, like any other learning object, such as resources or scenarios for instance. If this is
12
Choquet and Iksal
frequently the case in existing systems, when the tracking purpose is to provide learners and/or the system with useful information, it is more unusual for supporting the tutor, and it's rare for providing feedback to designers. With this in mind we could say that, as far as it could be possible for him, the designer who is engaged in the engineering process of a TEL system, should model the tracking needs of the learning session (Barré, El Kechaï, & Choquet, 2005). Then, the tracking needs should be instantiated on the educational environment in order to model the effective tracking means. Finally, one should also model the expected uses of these tracks, in terms of building the descriptive scenario for analyzing the usage. This is the way we have defined the three facets for the tracks modeling: • the Defining (D) facet which models the tracks needs; • the Getting (G) facet which models the tracks means; • the Using (U) facet, which models the tracks uses. In some cases, data acquired during a session are unexpected. If they have some utility, they also need to be modeled among these three facets in order to be incorporated in the descriptive scenario of the learning session and to probably bring up a new re-engineering cycle of the TEL system. Figure 1 shows the general overview of the DGU model. Track Conceptual Model Some recent European works focus on the tracking problematic and have provided outcomes on track representation, acquisition and analysis. Most of these works (DPULS, 2005; ICALTS, 2004; IA, 2005; TRAILS, 2004) have taken place in the Kaleidoscope European Network of Excellence (Kaleidoscope, 2004) and each of these projects have contributed to model tracks.
Figure 1. The DGU model
Modeling Tracks for the Model Driven Re-engineering of a TEL System
13
The TRAILS project (Personalized and Collaborative Trails of Digital and Non-Digital Learning Objects) investigates the trails that learners follow and create as they navigate through a space of learning objects. Learners interact with learning objects in the form of trails – time ordered sequences of learning objects. TRAILS is focused on the tracking of individual routes through learning objects, in order to identify the cognitive trail by analyzing the sequence of the learning objects the learner has consulted. This approach is close to (Champin, Prié, & Mille, 2003) and (Egyed-Zsigmond, Mille, & Prié, 2003), where authors consider user tracks in a hypermedia system as a sequence of actions, and use this for identifying the general goal of the user. Although, in this article, we don't restrict the meaning of a track as a sequence of actions, but in a more general way, as a datum which provides information on the learning session, we think, as the TRAILS project, that tracks have to be abstracted in a way they could be useful for understanding the cognitive trail of the learner. Moreover, we also think that a track has to be modeled as a descriptive scenario (emergent trail for TRAILS) and to be linked to, and compared with the predictive scenario (planned trails for TRAILS). The ICALTS project (Interaction & Collaboration Analysis' supporting Teachers & Students' Self-regulation) and the IA project (Interaction Analysis - supporting participants in technology based learning activities), which is a follow-up of ICALTS, …Propose that the design of technology based learning environments must not be limited to the initial means of action and communication, but should be extended by providing means of analysis of the very complex interactions that occur, when the participants of the learning activities work in individual or collaborative mode (presentation of the Interaction Analysis project, at http://www.noe-kaleidoscope.org/pub/ researcher/activities/jeirp/).
This position leads to consider, as ourselves, the definition of usage analysis means as a pedagogical design task, and the tracks themselves, as pedagogical objects. These projects have introduced and defined the concept of Interaction Analysis Indicator, as …Variables that describe something related to: (a) the mode or the process or the quality of the considered cognitive system learning activity (task related process or quality), (b) the features or the quality of the interaction product and/or (c) the mode, the process or the quality of the collaboration, when acting in the frame of a social context forming via the technology based learning environment (Kaleidoscope, n.d., p.12).
14
Choquet and Iksal
The DPULS project (Design Patterns for recording and analyzing Usage of Learning Systems) wanted to deal especially with analyzing the usage of elearning systems in real training or academic contexts, in order to help teachers to re-design alternatives to cope with observed difficulties in the learning scenario. This project has defined a structured set of Design Patterns (this structured set is accessible at: http://lucke.univ-lemans.fr:8080/dpuls/login.faces) thus providing instructional designers, teachers and tutors with experimented and possibly reusable patterns to support them in analyzing recurrent problems and tracking students’ activity. This project has enlarged the definition of the concept of Indicator as a feature of a datum (usually of a derived datum). It highlights a relation between the datum and an envisaged event which has a pedagogical significance. It has also proposed a definition for some concepts related to the tracks: raw-datum (recorded by the system), additional-datum (a datum which is not calculated or recorded, but linked to the learning situation, such as the predictive scenario, or a domain taxonomy), and derived-datum (calculated from other data). All of these projects have influenced our proposal. We have identified two main data types for tracks: the derived-datum type and the primary-datum type. The primary data are not calculated or elaborated with the help of other data or knowledge. They could be recorded before, during or after the learning session by the learning environment, for instance, a log file recorded by the system, a video tape of the learner during the session, a questionnaire acquired before or after the session, or the sets of posts in a forum. This kind of data is classified as raw-datum. The content-datum type concerns the outcomes provided by the learning session actors (learners, tutors and/or teachers). These data are mainly the productions of the learners, intended to be assessed, but they also could be, for instance, a tutor report on the activity of a learner, or on the use of a resource. Both of these data have to be identified in the collection of tracks provided by the learning environment, in terms of location and format. We introduce here the keyword and the value elements for this purpose. These elements will be discussed in the next section. The additional-datum type qualifies, as DPULS did, a datum which is linked to the learning situation and could be involved in the usage analysis. The derived data are calculated or inferred from primary data or other derived data. The indicator type qualifies derived data which have a pedagogical significance. Thus, an indicator is always relevant to a pedagogical context: it is always defined for at least one exploitation purpose, and linked to at least one concept of the scenario. We will detail this specific aspect further in the article. A derived datum which has to be calculated but which has no pedagogical significance is qualified as an intermediate-datum. We will now detail the information model of each data type of this model. The formalism used in the following schemas has the IMS Learning Design Information Model (IMS/LD, 2003) notation.
Modeling Tracks for the Model Driven Re-engineering of a TEL System
15
Figure 2. The conceptual model of a track • Only elements are shown (no attributes). • The diagrams are tree structures, to be read from left to right. An element on the left contains the elements on the right side. The element most to the left is the top of the tree. • An OR relationship in the diagrams is represented with <xsd:element name="Role" type="TraceableConceptType" substitutionGroup="TraceableConcept"/> <xsd:element name="LearningObject" type="TraceableConceptType" substitutionGroup="TraceableConcept"/> <xsd:element name="Resource" type="TraceableConceptType" substitutionGroup="TraceableConcept"/>
The next stage consists in instantiating the UTL-LD file with a specific scenario which we decided to analyse. This step is necessary to associate semantic to tracks, that is to say, to link each track with the relevant object of the learning scenario. The following piece of code represents relationships between all activities and resources of our experiment.
Instantiation of UTL in FSL Log Format After having prepared data about the learning scenario, we have to describe the tracks’ format according to the deployment platform: Free Style Learning. Our scenario was deployed on FSL, so we describe the format of FSL tracks. The following piece of code is the representation of tracks con-
24
Choquet and Iksal
cerning the management of the resource VideoIntro which is an introduction of the course based on a video. Operations are start, stop, pause and play. We describe here keywords that are necessary to identify the track, for instance “Intro gestartet” for the beginning of the video, and also values that have to be extracted, for instance the date of the track. FreeApp Intro gestartet FreeVideoPlayer start FreeVideoPlayer pause FreeVideoPlayer stop
Modeling Tracks for the Model Driven Re-engineering of a TEL System
25
Use Scenarios Our first need on usage analysis is about track analysis. We have three years of logs on two different experiments. For each of these case studies, we have a prescribed scenario described in IMS Learning Design. All resources used by this scenario are indexed with LOM (LOM, 2002). We use our Usage Tracking Language to bring semantics to each track. The first step consists in the interpretation of tracks according to the designer model and the corresponding track semantic description. Next, the observed usage of the learning system is available for the analysis. Usage Analysis: Track Interpretation At the beginning, automatic track analysis needs an automatic interpretation of these tracks. UTL is designed to add semantics to the content of the log files. We use it to filter the content of the log files, that is to say, to keep only tracks that are considered relevant by the designer. A track is relevant if a description is given inside the UTL file. The second use of UTL consists of associating a specific type to each track and in extracting values that are representative of the learner’s activity. The result of this stage is a data structure which contains the interpretable tracks and which is shareable between analysis services. The data structure is available also for each researcher who wishes to propose new services. Usage Analysis: Derived Data Processing There are various ways to use the interpreted tracks. For instance, we may evaluate a resource use, or compare a learner scenario with the predictive scenario. Moreover, with the same raw data, different interpretations could be made. For instance, mails sent between the actors of a learning session could be parsed in order to find semantic markers which could reveal the social and affective roles in the group (Cottier, & Schmidt, 2005), or could be visualized as an oriented graph between actors in order to measure the cohesion in the group (Reffay, & Chanier, 2003). Using the semantic description of tracks allows us to define services in a declarative way, that is to say, providing analysis services independent from a platform track format. We currently are developing a first set of these services. Examples of analysis results are the following: rate of use of a resource, performance of a student, emergence of a role (e.g., leader), extraction of an observed learning scenario, and detection of a sequence of resource uses which have not been prescribed. To present some ideas in the usage analysis, we will focus on three cases: a statistical datum, a result which has to be re-transcribed in the designer’s model, and intelligent information detection.
26
Choquet and Iksal
A statistical datum. These data are, for instance, the rate of use of a resource, the average mark concerning the evaluation exercise, or the time spent on a particular activity (the shortest, the average, and the longest). We filter the tracks according to their semantics and to make a small calculation on them. As an example, for the rate of use, a first solution is to count students for whom we find at least one track about the use of the resource. In our experiments, we had a first phase of engineering when the designer has declared a need of observation concerning the use of resources. We highlighted that numerous students spent less than 15s on resources at the beginning of the scenario. In a second phase, called re-engineering phase, we modified the observation tools in order to filter resources’ use fewer than 15s, in order to detect the exploration period when students clicked everywhere.
Re-transcription in the designer’s model. One of the main goals of the reengineering driven by models is to use the same representation model for the description of the predictive scenario by the designer as for the observed scenario build with tracks generated by the learning system. In our first experiments, we worked with IMS-LD as a representation model. The interest in the use of a common model is the possible comparison between the different scenarios that leads us to identify non-predicted usage of resources or incoherence in the sequence of activities. In one of our experiments (the one based on FSL), we observed that some students have used the evaluation exercise as a quiz at the beginning of the experiment, they have only navigated inside the list of questions in order to self-evaluate their knowledge (before the first activity of the learning session). This observation leads the designer to propose two facets to the exercise, one for evaluation and another for a quiz. We consider two kinds of re-transcription of the observed scenario: the one generated from single student tracks, and a stereotypical scenario that represent a combination of all student scenarios. (a) Re-transcription of one student’s observed scenario. First, we have to read the representation model in order to identify the core concept, such as the activity for IMS-LD. Next, we filter tracks in order to represent this concept and all its components. The last step consists of organizing all instances of the core concept in a sequence which corresponds to the observed scenario. (b) Re-transcription of a stereotypical observed scenario. A stereotypical observed scenario corresponds to the combination of all student scenarios. To build this scenario, we must have all the students’ observed scenarios. Next, we compare the sequence of core elements (e.g., activities), and we compare in depth each element. We observe the percentages about the use or the position in the sequence of each element. A stereotypical scenario is a graph where each relation is qual-
Modeling Tracks for the Model Driven Re-engineering of a TEL System
27
ified with the percentage of students who have chosen the corresponding direction. Usage Analysis: From Raw-data to Indicators If we consider all information models of the first part of this article, in this section, we propose a scenario which uses interpreted tracks to compute indicators. This scenario is based on a design pattern taken from the framework of the DPULS Project (DPULS, 2005) which is called: “Playing Around with Learning Resources” (this Design Pattern is accessible at: http://lucke.univlemans.fr:8080/dpuls/login.faces) This pattern provides an approach to detect learner playing around with resources at the beginning of an activity. In this pattern, we propose a solution based on the computation of two indicators: The characterization of the sequence of resources and the characterization of the time of an activity. The mean of the first indicator is the following: the sequence of resources attempted by a learner is valued as non-significant if the duration of each resource is less than a fraction (for us 10%) of the Typical Learning Time defined for the relative resource. The second indicator says: the time of an activity is qualified as the beginning if the effective duration of the activity is less than a fraction (for us 10%) of the Typical Learning Time of an activity. In Figure 10, the graph which represents the use of data is presented. We can find in this graph the indicators but also some intermediate data, additional data and raw data. UTL.1, as presented before is able to identify and extract these raw data; the example can be found in the previous section. UTL.2 which includes the first section of this article has to take into account the use of these raw data for the generation of indicators for the
Figure 10. Map of indicators and data used
28
Choquet and Iksal
pedagogical designer. The started time and the stop time of a resource are used to evaluate the duration of the resource. The additional data such as the typical learning time can be extracted from the prescribed scenario (for instance the field 5.9 of the LOM). It can be also given by the designer, such as the Playing Around Typical Learning Time of a resource. This is a percentage of the use time of a resource considered as a minimum time, under which limit the use cannot be taken into account. Now, we will present the description of these data within our proposal. Table 1 presents the information table for the raw-datum called Started time of a resource, Table 2 presents the information table for the additional-datum called Typical learning time of a resource, Table 3 describes the information table for the Intermediate-datum called Sequence of resources and finally, Table 4, where Pc means Pedagogical-context, Tc, Traceable-concept and Ep, Exploitation-purpose, presents the information table for the indicator called The characterization of the sequence of resources. All these descriptions expressed in an XML file can be processed by a system in order to pre-calculate some data and to interact with an analyst or a designer for data that need a semi-automatic or manual method. The sysTable 1 Information Table for a Raw-Datum D G
U
Title Description Acquisition-time Record-type Record-tool.Title Record-tool.Location Content.Keyword Content.Value Used-by
Started time of the video intro These datum stores the time of the beginning of a video’s use. During the session Log file FSL methods for the generation of tracks ~exp/StudentID/file.FSL “FreeApp” from character “intro gestartet” from character 33 to 40 42 to 57 Date from character in position 1 to 26 "Sequence of resources"
Table 2 Information Table for an Additional-Datum D
Title Type Description
G U
Location Content Format Used-by
Typical learning time of the video intro Number These datum stores the estimated time for the resource of our scenario which is an introduction by mean of a video In the metadata section of the scenario 182 Integer, time in seconds "Sequence of resources"
Modeling Tracks for the Model Driven Re-engineering of a TEL System
29
Table 3 Information Table for an Intermediate-Datum D G
U
Title Title Component.Primary-Datum … Method.Type Method.Tool.Title Method.Tool.Location Content Format Used-by
Sequence of resources Retrieve and organize resources’ starting tracks "Started time of the video intro" Automatic Calculation_Resources_Sequence Method of a java library for the project Too long for this article List of couple (resource-name, started-time) "The characterization of the sequence of resources"
Table 4 Information Table for an Indicator D G
U
Title Title Component.Derived-Datum … Description
Method.Type Method.Tool.Title Method.Tool.Location Content Format Pc.Tc.Title Pc.Tc.Type Pc.Tc.Value … Ep.Type Ep.Recipient-role
The characterization of the sequence of resources Evaluation of the relevance of a sequence of resources "Sequence of resources" The sequence of resources attempted by a learner is valued as "non significant" if the duration of each resource is less than a fraction (for instance 10%) of the Typical Learning Time defined for the relative resource Automatic Calculation_Relevance_Resources_Sequence Method of a java library for the project "Significant" or "Non significant" String Scenario Abstract Scenario "Learning Web Server Programming" Reengineering Designer
tem which uses this extension is not currently developed. Only the UTL version presented in the third section is useable. Perspectives The DGU (Defining, Getting and Using) model presented in this article is well suited for defining what the system has to track, based on the predictive scenario designed for a learning activity. For each data, the designer
30
Choquet and Iksal
can define what to track, how to track (i.e., the tracking means) and why it should be tracked (i.e., the semantics of the track chunk). Each data can be combined with others in order to provide high level indicators for the analyst or the designer. At the moment, a part of this model is developed; we called it UTL.1 for Usage Tracking Language. We have now to extend UTL.1 in UTL.2 by including all the elements of DGU, and also all services needed to calculate indicators and intermediate-data. Because of its metalevel, this language could be also used, after usage analysis, to define and highlight semantic links between predictive and descriptive scenarios. Works such as (Seel, & Dijkstra, 1997) have shown that teachers and trainers – who are the main potential designers of educational systems – have some difficulties in instructional design, especially regarding the explicitation and technical reification of their pedagogical intentions. We are defining rules which can be inferred on the meta-model (e.g. the XML-Schema) of the instructional language used by a designer (for instance, Learning Design) in order to identify opportunities and observation possibilities (Barré, & Choquet, 2005). They reason on the structure of the instructional language (datatype, relations, and so on) and provide the designer with information on the needs of observation. These needs are relative to the concepts of the language and thus, define the traceable concepts. Using these rules with UTL.2 could be a way to provide designers with a semi-automatic tool for decision helping purposes. Our approach of student data capture is focused on automatic techniques driven by designer prescriptions. UTL is presently without the spectrum of both existing non-automatic techniques, such as interviews for instance, and data-mining or machine learning ones. We think all these techniques, including ours, are complementary. Now, we have to operationalize and validate UTL.2 with all types of data (e.g., electronic, interviews, video). We have started a study with researchers specialized in usage analysis (Communication Science background) of which the objective is to define when, why and how a designer has to explicit the requirements to these techniques. Actually, we have two new experiments which will be used on UTL.2. These experiments lead us to consider new tracks’ formats and also new EMLs. In the Symba experiment, students have to organize themselves the project management concerning a website development by using the Symba tool (Betbeder, & Tchounikine, 2003), the scenario is in IMS-LD. In the AEB experiment, we mobilize various actors of apprenticeship training for the collaborative and iterative development of an Apprenticeship Electronic Booklet (AEB). These actors are also the future users of the AEB. They are trainers, training managers, employers and apprentices. The AEB is a learning system where information concerning the apprentice’s training progression is consigned. Its goal is to help them in the appropriation of their training and to give the trainers and the employers the possibility to evaluate their apprentice’s knowledge acquisition, to per-
Modeling Tracks for the Model Driven Re-engineering of a TEL System
31
ceive their progression in the training and to regulate it. In this experiment, the representation model used is a set of use cases. References Barré, V., & Choquet, C. (2005). Language independent rules for suggesting and formalizing observed uses in a pedagogical re-engineering context. IEEE International Conference on Advanced Learning Technologies (ICALT'2005), 550-554. Barré, V., El Kechaï, H., & Choquet, C. (2005). Re-engineering of collaborative e-learning systems: Evaluation of system, collaboration and acquired knowledge qualities. AIED'05 Workshop: Usage Analysis in Learning Systems, 9-16. Bazsaliscza, M., & Naim, P. (2001). Data mining pour le web. Paris: Eyrolles Eds. Berners-lee, T. (1999). Weaving the web. San Francisco: Harper Eds. Betbeder, M.-L., & Tchounikine, P. (2003). Symba a framework to support collective activities in an educational context. ICCE’2003, 188-196. Brocke, J. V. (2001). Freestyle learning – Concept, platforms, and applications for individual learning scenarios. 46th International Scientific Colloquium, 149-151. Champin, P.-A., Prié, Y., & Mille, A. (2003). MUSETTE: Modeling USEs and tasks for tracing experience. Workshop 'From Structured Cases to Unstructured Problem Solving Episodes For Experience-Based Assistance', ICCBR'03, 279-286. Corbière, A., & Choquet, C. (2004a). Re-engineering method for multimedia system in education. IEEE Sixth International Symposium on Multimedia Software Engineering (MSE), 80-87. Corbiere, A., & Choquet, C. (2004b). A model driven analysis approach for the re-engineering of e-learning systems. ICICTE'04, 242-247. Cottier, P., & Schmidt, C. (2005). Le dialogue en contexte: Pour une approche dialogique des environnements d'apprentissage collectif. Revue d'intelligence artificielle, 19(1-2), 235-252. DPULS (2005). Design patterns for recording and analysing usage of learning systems. Consulted May, 2006, from http://www.noe-kaleidoscope.org Egyed-Zsigmond, E., Mille, A., & Prié, Y. (2003). Club (Trèfle): A use trace model. 5th International Conference on Case-Based Reasoning, 146-160. El Kechaï, H., & Choquet, C. (2006). Understanding the collective design process by analyzing intermediary objects. The 6th IEEE International Conference on Advanced Learning Technologies (ICALT'2006). Submitted for publication. Grob, H. L., Bensberg, F., & Dewanto, B. L. (2004). Developing, deploying, using and evaluating an open source learning management system. Journal of Computing and Information Technology, 12(2), 127-134. Hanson S.-J., Kraut R.-E., & Farber J.-M. (1984). Interface design and multivariate analysis of UNIX command use. ACM Transactions on Information Systems (TOIS), 2(1), 42-57. IA (2005). Interaction analysis. Consulted May, 2006, from http://www.noe-kaleidoscope.org (ICALTS, 2004). Interaction & collaboration analysis' supporting teachers & students' self-regulation. Consulted May, 2006, from http://www.noe-kaleidoscope.org IMS (2006). IMS Global Learning Consortium. Consulted May, 2006, from http://www.imsglobal.org/ IMS/LD (2003). IMS Learning Design. Consulted May, 2006, from http://www.imsglobal.org/ learningdesign/index.html
32
Choquet and Iksal
Kaleidoscope (2004). Consulted May, 2006, from http://www.noe-kaleidoscope.org Kaleidoscope (n.d.). State of the art: Interaction analysis indicators. Available at http://www.rhodes.aegean.gr/LTEE/Kaleidoscope-Icalts/ Koper, R., Olivier, B., & Anderson, T. (2003). IMS learning design information model (version 1.0). IMS Global Learning Consortium, Inc. Laforcade, P., & Choquet, C. (2006). Next step for educational modeling languages: The model driven engineering and re-engineering approach. The 6th IEEE International Conference on Advanced Learning Technologies (ICALT'2006) Submitted for publication. Lejeune, A., & Pernin, J-P. (2004). A taxonomy for scenario-based engineering. Cognition and Exploratory Learning in Digital Age, (CELDA 2004), 249-256. LOM (2002). Draft standard for learning object metadata. (IEEE). Mostow, J. (2004). Some useful design tactics for mining ITS data. Proceedings of the ITS2004 Workshop on Analyzing Student-Tutor Interaction Logs to Improve Educational Outcomes, 20-28. Reffay, C., & Chanier, T. (2003). How social network analysis can help to measure cohesion in collaborative distance-learning. Proceedings of Computer Supported Collaborative Learning Conference (CSCL'2003), 343-352. Seel, N., & Dijkstra, S. (1997). General introduction. Instructional design: International perspectives, 2, 1-13. Hillsdale, NJ, Lawrence Erlbaum Associates. Teubner, A., & Vaske, J. (1988). Monitoring computer users' behaviour in office environments. Behaviour and Information Technology, 7, 67-78. TRAILS (2004). Personalised and collaborative trails of digital and non-digital learning objects. Consulted May, 2006, from http://www.noe-kaleidoscope.org Zheng, C., Fan, L., Huan, L., Yin, L., Wei-Ying, M., & Liu, W. (2002). User intention modeling in web applications using data mining. World Wide Web: Internet and Web Information Systems, 181–191.
Usage Analysis in Learning Systems, 33-53
Results from Action Analysis in an Interactive Learning Environment OLIVER SCHEUER German Research Center for Artificial Intelligence DFKI, Germany
[email protected] MARTIN MÜHLENBROCK European Patent Office, The Netherlands
[email protected] ERICA MELIS German Research Center for Artificial Intelligence DFKI, Germany
[email protected] Recently, there is a growing interest in the automatic analysis of learner activity in web-based learning environments. The approach and system SIAM (System for Interaction Analysis by Machine learning) presented in this article aims at helping to establish a basis for the automatic analysis of interaction data by developing a data logging and analysis system based on a standard database server and standard machine learning techniques. The contribution is the integration of components which are appropriate for large amount of data. The analysis system has been connected to the web-based interactive learning environment for mathematics, ActiveMath, but is designed to allow for interfacing to other web-based learning environments, too. The results of several usages of this action analysis tool are presented and discussed. They indicate potentials for further development and usages.
Introduction Recently, there is a growing interest in the automatic analysis of learner interaction data with web-based learning environments. This is largely due to the increasing availability of log data from learning environments and in particular from web-based ones. The potential outputs include the detection of regularities and deviations in the learners’ or teachers’ actions as well as
34
Scheuer, Mühlenbrock, and Melis
building of models which can predict learner features from log data. The objective is to use those outputs to support teachers and learners by providing them with information that helps to manage their learning and teaching or to use the information for adaptation actions of eLearning systems. Commercial systems such as WebCT, Blackboard, and LearningSpace already give access to some information related to the activity of the learners including some statistical analyses, and provide teachers with information on course attendance and exam results. With this information already being useful, it only represents the tip of the iceberg of what might be possible by using advanced technologies. This upcoming research area, (i.e., addressing the automatic analysis of learner interaction data), is related to several well-established areas of research including intelligent tutoring systems, web mining, statistics, and machine learning, and can build upon results form these fields for achieving its objectives. In contrast to intelligent tutoring systems, learner interaction analysis does not rely on models of the learner or of the domain knowledge since these are heavy to build and maintain. In this regards, learner interaction analysis is comparable to website data mining but with a specific perspective on learning settings and with the availability of pedagogical data that usually are not available in web mining applications that are mostly based on click-through data only. Click-through data streams only allow for a rather shallow analysis, but with the inclusion of pedagogical data, more advanced techniques can be adopted from the field of machine learning, for example, in order to learn models of individual behavioural and non-observable variables that can predict the nonobservable variables from behaviour of future students . Although a number of open questions have already been tackled (Arroyo, Murray, & Woolf., 2004; Heiner, Beck, & Mostow, 2004; Merceron & Yacef , 2003; Merceron, Oliveira, Scholl, & Ullrich 2004; Mostow, 2004; Oliveira & Domingues, 2003; Zhang & Lu, 2001), there is not yet a systematic approach in analysis interaction data from huge learner action logs nor are there common architectures. The approach presented in this article aims at helping to establish a basis for the automatic analysis of interaction data by developing a data logging and analysis system based on a standard database server and standard machine learning techniques. This work was conducted in context of the iClass project which aimed at developing an intelligent, cognitive-based e-learning system, to enhance the iClass system with profiling capabilities. Because the iClass system was still under development when this research was carried out, our analysis system has been connected to the web-based interactive learning environment for mathematics ActiveMath which provides the log files. However, the system is designed for interfacing to other webbased learning environments, too, which will enable us to connect it with the upcoming iClass system. It has been tested with a medium scale experiment
Results from Action Analysis in an Interactive Learning Environment
35
in which four classes of a secondary school participated throughout a school term of five months on a weekly basis as well as on a large scale experiments in context of an introductory mathematics course at a UK-based University. This article is organized as follows. In the following section, the SIAM system will be described, which is comprised of a learning environment, a logging component, and an analysis component. Subsequently, two studies will be presented: In the first one, the system's capabilities to estimate students’ performance and gender, and in the second one the relationship between students' behaviour and cognitive style was investigated. The Action Analysis System SIAM The SIAM system is comprised of three major parts, that is, a learning environment, an action logging component, and an action analysis component (see Figure 1). These system parts will be described in more detail in this section. In addition to these components, there are also three data repositories involved in storing and providing information: a learning material database, a set of user log files, and a database containing logs and possibly additional user data and context data. The analysis subsystem has been implemented by using standard technology such as Java and mySQL, which are available for a number of platforms and operating systems, together with the suitable drivers for database
Figure 1. System architecture
36
Scheuer, Mühlenbrock, and Melis
connectivity. In addition, the Analyzer is based on the Weka (Witten & Frank, 1999) and YALE (Fischer, Klinkenberg, Mierswa, & Ritthoff, 2002) toolkits, which provide tools for visualizing and exploring data as well as means for integrating machine learning functionality into applications. As a starting point, the web-based learning environment ActiveMath has been used to provide a testbed for developing and testing the action logging and analysis components. However, these components have been designed to be mostly independent of a specific learning environment, allowing for providing the same logging and analysis functionality to other learning environments.
ActiveMath Learning Environment ActiveMath is a web-based learning environment that dynamically generates interactive courses adapted to the student's goals, preferences, capabilities, and prior knowledge (Melis et al., 2001; Melis et al., 2006). The content is represented in a XML-knowledge language for an educational context which greatly supports reusability and interoperability of the encoded content. ActiveMath supports individualized learning material in a user-adaptive environment, active and exploratory learning by using (mathematics) service tools and with feedback (see Figure 2). For different purposes and for different users, the learning material and its presentation can be adapted: the selection of the content, its organization, and the means for supporting the user have to be different for a novice and an expert user, for an engineer and a mathematician, for different learning situations such as a quick review and a full study. Since there is no way of knowing in advance the goals, the profile, and the preferences of any user when designing the system, ActiveMath builds on adaptive course generation. One component of ActiveMath – its event framework – is especially relevant to the SIAM system since it provides the information to be logged. The event framework (Melis et al., 2006) realizes a publish-subscribe scenario and governs the asynchronous communication between system components and even communication of remote services. Events are a mechanism for a powerful and flexible, yet rather loose integration of components. An example for event publication is the following: when the learner finished working on an exercise, the exercise subsystem issues an event. The event carries information describing the learner, the identifier of the exercise, the success rate, and the time stamp of the event. Listeners that can subscribe to such an event can be the learner model as well as the suggestor of the tutorial component. A component that publishes events is called an event source. A component that subscribes to the events published by an event source is called a listener which receives event messages from the event source. In contrast to a full-fledged messaging model, events remain anonymous rather than being sent from a specific sender to a specific recipient: when publishing an event, the event source is usually not aware who is listening
Results from Action Analysis in an Interactive Learning Environment
37
to the events (only the module managing the subscriptions is). Moreover, usually the listener does not care which component or module created the event, it only knows where to subscribe to the events it is interested in. The Action Logging Component receives the trace of user actions by subscribing to the event framework for relevant events. To return to the example above, when for example, the learner finished working on an exercise, the generated event will be received by the Action Logging Component. Figure 2 shows the ActiveMath user interface. The left panel shows the table of contents (TOC) of the currently selected book (in ActiveMath, the term book is used as a metaphor for a course). The contents are organized in a hierarchy with expandable and collapsible nodes. The lowest level is constituted by single book pages which can be selected. A colored bullet lefthand side of each book page item indicates the system’s belief in the student’s mastery for the respective contents. Right of the TOC the currently selected book page is presented. Each page consists of a sequence of learning items, which may be reading material, exercises, or interactive examples. Exercises are displayed in a separate window, when the Start exercise link is clicked. Relevant concepts occurring in the texts are hyperlinked; their selection opens a window containing a concept definition or other additional information. In the upper-right of each
Figure 2. ActiveMath user interface
38
Scheuer, Mühlenbrock, and Melis
displayed learning item is a Note icon, where students can access the notes functionality. Students can use it to annotate learning items with private or public comments, or to read already existing notes. In the lower part of the page a previous- and a next-button are located. As alternative to the TOC, from which pages can be accessed based on a hierarchical overview, these buttons allow movement through the contents in a predefined linear sequence. Finally, in the upper-part of the screen, the menu bar is located which offers functionalities that are independent of specific contents and which are generally available. This includes a Menu link to return to the main page were books can be selected, a Dictionary link which opens a window where search queries can be submitted and a Logout button to finish the current session. Action Logging Component For each learner, the ActiveMath environment generates an online log that lists user actions in the learning environment in terms of general information such as time, type of action, user name, and session number, as well as specific information including which page has been presented to the user, which item has been seen by the user, which exercise has been tackled and solved or not solved. The Updater (see Figure 1) receives event information on the users’ actions from the learning environment, and transforms every user event into one or more corresponding database tables. Usually, the Updater receives the information online from the event queue, but it can also read in files with log data that have been generated in an offline mode. The Log Database (see Figure 1) is at the center of the SIAM system. It contains not only representations of the raw data in the user logs (see Table 1), but also has tables that hold the results of the analysis. Moreover, it contains tables for additional background knowledge concerning the users, context, or courses among others (see Table 2). The basic level of the database, which corresponds to the raw log data, is organized in tables that represent generic event information as well as eventspecific data. The structure of these tables has been designed closely to the events specification, since this allows for simpler updating operations when the event subsystem is changed or replaced by another system. Table 1 lists the basic event tables together with their fields and a short description. In addition, as shown in Table 2, the database includes tables that hold additional information on the users and sessions such as gender and holiday periods, respectively, as well as tables that are derived from these by means of database queries. In addition, the Updater provides some data completion functionality. Every now and then for some tables the information is not complete. For instance, most users do not log out of the learning system explicitly by using the button provided in the user interface, but simply close the browser or
Results from Action Analysis in an Interactive Learning Environment
39
Table 1 Log Database Schema for Basic Level Table Event
Attributes
eventId timestamp source session userId type eventUserCreated eventId userName eventUserLogged-In eventId ip userAgent eventUserLogged-Out eventide eventPage-Presented eventId book page eventExercise-Started eventId exercise eventExerciseStep eventId userInput eventExercise-Finished eventId exercise successRate eventMastery-Changed eventId item masteryDimension oldValue newValue eventItemPresented eventId item itemType eventUserProperty-Changed eventId propertyNam oldValue newValue eventItemSeen eventId duration item
Description Generic information for all events
Registration of a new user to the system
User logs into the system; start of a new session User logs out of the system; end of a session System presents a requested book page to the user
User starts an exercise User submits an input for an exercise Exercise is finished
System changes the mastery level of an user
Presentation of a learning item to the user
Indicates changes of some user meta data
Gives a more fine-grained resolution (on item level) of the user's focus (uses information of an eye-tracker)
shut down their computer. In this case usually no event is generated concerning the logout. The corresponding logout table is enhanced by information that is derived from the other events the user created and on heuristics concerning pauses and open hours among others. This information is automatically added to the login table, but is marked as derived information to be distinguishable from the original log data.
40
Scheuer, Mühlenbrock, and Melis
Since log files can grow very large, a tool for a realistic usage (rather than for small-scale academic purposes only) needs to built on a database. For us this was a major design decision but only few research prototypes developed so far take this need into consideration.
Action Analysis Component The Analyzer (see Figure 1) performs data aggregation and evaluation in terms of the queries to the log database. It also incorporates a number of machine learning methods for automatically analyzing the data and takes the data from the Log Database as an input. If needed, adjustments and preferences can be input by the engineer who is running the analysis. In addition of getting a better insight into the underlying relationships in the data, the results of the analysis can be used for the prediction and classification of future sessions. Up to now, the Analyzer is not an integrated, fully automated component but consists of a set of SQL-query scripts that preprocess the raw action data, and the machine learning tool box YALE (Fischer et al., 2002) to execute the actual machine learning analysis. YALE offers a wide range of operators for performing data pre-processing (discretisation, filtering, normalisation and others), learning (more than 100 learning algorithms), validation (e.g., cross-validation, several performance evaluation criteria) and other analysis-relevant tasks. In the following, a selection of learning schemes used in the Action Analysis Component is presented: Decision tree learner: Many machine learning methods provide their output in an intelligible, human readable form. For instance, methods for generating decision tress from data, such as C4.5 (Quinlan, 1993), allow for a treeshaped representation of the learning results. A decision tree is constructed by the algorithm first selecting an attribute to place at the root node of the tree and make one branch for each possible value. This splits up the example set into subsets, one for every value of the attribute. The attribute is selected in a way that maximizes the information gain by the chosen attribute. This process is repeated recursively for each branch, using only those instances that actually reach the branch. If at any time all instances at a node have the same classification, the developing of that part of the tree is stopped. Rule Learner: Rule learning algorithms generate a set of prediction rules each consisting of a class values and its condition. The PRISM rule learner, for instance, constructs rules in the following way: After choosing a class value the algorithm starts with an empty rule and adds iteratively new conditions of the form ‘attribute X has value Y.’ Each added condition narrows the scope of the rule because less trainings instances will match the extended condition. On the other hand, the attribute-value pairs are chosen in a way which increases the accuracy of the rule (a higher share of the matched instances will be correctly classified). If the rule is perfect (100% accuracy) the algorithms goes over to construct the next rule. To cover all trainings
Results from Action Analysis in an Interactive Learning Environment
41
instances, several rules for one class value might be necessary. This procedure is repeated for all class values. Naïve Bayes: Naïve Bayes is a widely-used probabilistic classifier, based on Bayes’ Theorem. It assumes that all attributes are independent (therefore naive). Dependencies between attributes might comprise the quality of computed models. Support Vector Machine (SVM): Support Vector Machine algorithms try to find hyperplanes in the attribute space which optimally separate instances of different classes. Optimality is achieved by maximising the margins between classes (the margin is the minimum distance between class instances and the separating hyperplane). Because more complex patterns can not be captured with a linear separation, SVM apply the kernel trick: instead of changing the SVM core algorithm, the attribute space is transformed by defining a non-linear kernel. Boosting algorithms: Boosting algorithms (e.g., AdaBoost) are metaalgorithms which build on so-called weak learners that are rather simple learning methods, for instance a Decision Stump algorithm (computes a 1level decision tree). The boosting algorithm iteratively applies the weak learner to the training instances to compute a set of predictive models. Initially, all instances are equally weighted. In each iteration, weights of the falsely classified instances gets increased and of correctly classified ones decreased. In the subsequent iteration the learner will try to find a model which takes the falsely classified ones more into account. The complete model of the whole boosting procedure combines all the computed simple models, weighted by their accuracy. Problems and Solutions Although the SIAM system presented here considerably facilitates the process of action log analysis there are still some open questions and problem to be tackled which will have to be addressed in future. Up to now, pre-processing is executed in a semi-automated way: before starting the analysis the existing SQL scripts are adjusted according to the purpose of the current investigation and then applied by the Analyzer component to the raw data. This indicates further potential for automation: it has to be found out which usage aspects are generally relevant to the analysis process, independent of specific purposes of individual investigations. For example, students’ success rates, the number of pages read or their overall online time will in most investigations play a role. The computation of such aspects could be completely automated. The computation of aspects which are specific to individual investigations will still need manual adjustments. A further candidate for automation is the inclusion of external, nonbehavioural data which are not contained in the action logs (e.g., gender, questionnaire data, pre- and post test results). Currently, this essential infor-
42
Scheuer, Mühlenbrock, and Melis
mation is manually added to the database, from spreadsheets provided by teachers. It has to be investigated which interfaces can be offered to teachers to automatically feed the database. There is ongoing work concerning the inclusion of pedagogical and domain data. State-of-the art e-learning systems use learning objects which are annotated with pedagogical or domain-specific metadata. Including this information will allow more sophisticated analyses. For example, performances can be analyses for different topics or different levels of difficulty.
Related Work Merceron and Yacef (2005) present a case study how educational data can be mined to support teachers to understand their students' learning and students to promote self-reflection. The analysis was based on data of the Logic-ITA system, a web-based tutoring tool to practise logical proofs. The conducted analyses included association rule algorithm that can be used to identify sets of mistakes that often occur together, or a decision tree algorithm to predict final exam marks given students prior usage behaviour. They used their tool for advanced data analysis in education (TADA-Ed) (Benchaffai, Debord, Merceron, & Yacef, 2004) to carry out the analyses. There are a number of tools which are more concerned with an appropriate presentation of student action data than with an automated analysis. For instance, The LISTEN Reading Tutor is intended to help children learning to read. The system displays stories, sentence by sentence, which children then have to read aloud. Children's utterances as well as other student-tutor interaction are logged into a relational database. Mostow et al. (2004) present a tool to browse the student-tutor interaction data of the LISTEN Reading Tutor which are presented in a hierarchical tree structure. The Data Recording and Usage Interaction Analysis in Asynchronous Discussions System (D.I.A.S.) presented by Bratitsis and Dimitracopoulou (2005) is intended to improve asynchronous discussions. The usage data is stored in a database system. From the raw data, meaningful statistics (so-called interaction analysis indicators) which give account of passive and active participation, and thread initiations, are extracted by means of SQL queries. These indicators can be presented to discussion participants, but also to monitoring persons as teachers by a visualisation module. Discussion participants may benefit from an increased awareness of their own actions (metacognition) as well as those of other participants. Teachers can identify problems and, if necessary, intervene. Using Action Analysis The Action Analysis System has been investigated in a mathematics courses in a secondary school and at a UK-based University. The objectives of those tests have been (1) to collect experience with the SIAM system and to find possible ways to improve it, (2) to conduct the actual analyses to get some insight
Results from Action Analysis in an Interactive Learning Environment
43
how students use the ActiveMath and to uncover relationships between individual (e.g., gender and cognitive style), behavioural and external factors (teacher).
Estimating Performance and Gender The SIAM system has been tested in a medium sized experiment in a mathematics course in a secondary school. About 70 students from three different courses used the learning environment for a period of five months. A further course of about 20 students were taught the same subject but in the traditional classroom manner. The other three courses used the ActiveMath learning environment on a weekly basis in two-hour lessons. During the online course, each class was split into two subgroups using different computer rooms. Many students were already familiar with computers, but a considerable number needed further instruction even for basic operations such as login. A preliminary evaluation of the logged data after a first couple of sessions showed some problems in the quality of the data. For instance, instead of registering with the ActiveMath system only in the very first session and using the created user account in the sequel, a large number of students created a new account including a new user name for each session, which makes difficult the longitudinal analysis of the data. The problem was resolved by having the students create only one account and making the registration procedure inaccessible for them after that. Figure 3 provides an overview on the data that shows the number of events related to the hours of the day. Clearly, the major amount of events was created during lesson hours between 9 am and 2 pm (14 h), but some events were
Figure 3. Number of user actions in relation to hours of the day
44
Scheuer, Mühlenbrock, and Melis
generated earlier or later in the day. Some of them are due to a small number of students using the system in off-time, and most of them are due to teachers and system administrators preparing or evaluating the system. At the end of the term, a written post test has been done with the students to assess what they had learnt. The results were added to the database manually as well as some further information on gender, teacher, and so forth. Further information was semi-automatically generated by the Analyzer from the log data and added to the database. For each student the information in Table 2 has been gathered for further analysis. For anonymity reasons the students used arbitrary user names in the learning environments, and they were to give these user names also in the post test. However, in one course the students put down their real names on the test sheets, a fact which makes the linking to their log data impossible. Finally, 25 student records were complete and clean enough for being used in the further analysis. Figures 4 and 5 show the decision trees that were generated for characterizing the attributes post-test (with values low, medium, and high) and gender (with values male and female), respectively. The decision tree for post test (see figure 4) shows that for predicting the result the teacher is most influential, that is, with teachers Mr. E. and Mr. G. the post test result is expected to be low. However, with Mr. K. the test result is high if the student is in class 6a, would be low if he was in class 6b and male, and medium if she was in class 6b and female. Similarly, the decision tree for gender (see Figure 5) depicts that when the post test result is bad or medium the user would be male Table 2 Data From Manual Input and Gathered Automatically Attribute
Input
Gender Class Teacher
manual manual manual
post test Exercises started Exercises finished successful exercises reading actions solving actions Dictionary Integration off time
manually semi-automatic semi-automatic semi-automatic semi-automatic semi-automatic semi-automatic manual semi-automatic
Comment Course, each comprised of about 20 students Each class has been split into two subgroups with each being taught by another teacher Results in the post test done in writing
Whether the student used the dictionary for lookup Whether the student is handicapped Whether the student accessed the learning environment beyond lesson hours
Results from Action Analysis in an Interactive Learning Environment
Low Medium High
Low
Medium
High
11 2
1 3 2
1 5
45
Teacher
Mr. E. low
Mr. G.
Mr. K. low
Class 6a
6b
high
Gender
male low
female medium
Figure 4. Decision tree for post test and corresponding confusion matrix
or female, respectively, and if it was high the reading activity would indicate a male use if it was low and a female user if it was medium or high. This is an interesting results that can be interpreted as indicating that female users more successful in tests and a harder working concerning reading material. Importantly, these attributes have solely been selected by the learning algorithm from the ones in Table 2, which means that other attributes such as successful exercise solving and off time system usage were of minor relevance. Figures 4 and 5 also give the quality of the decision trees in terms of confusion matrices. A confusion matrix displays the result of testing the decision tree with data as a two-dimensional matrix with a row and a column for each class. Each matrix element shows the number of test examples for which the actual class is the row and the predicted class is the column. Good results correspond to large numbers down the main diagonal elements and small, ideally zero, off-diagonal elements. Hence the confusion matrices in Figures 4 and 5 indicate not ideal, but quite good decision trees.
46
Scheuer, Mühlenbrock, and Melis
Male Female
Male
Female
11 2
1 3
Post test
low male
medium female
low
high
Reading
medium
male
female
high female
Figure 5. Decision tree for gender and corresponding confusion matrix CONCLUSION
In this study, a decision trees were computed capable of predicting gender and performance. These analyses indicate possible usages of the SIAM system albeit, due to the small sample size, the expressiveness and generality of the presented results are limited. One reason for this fact is the high number of instances which had to be excluded from the analysis. This underlines the importance of a careful preparation. Estimating Cognitive Styles A further study was conducted in the context of an introductory mathematics course for computer science students at a UK-based University with around 300 participating students. Its main objective was to investigate the relationship between students’ cognitive styles and their observable behaviour in the ActiveMath system by means of machine learning.
Introduction: Cognitive styles Cognitive styles describe an “individual's preferred and habitual approach to organising and representing information” (Riding & Rayner, 1998). Among the considerable number of existing style theories, especial-
Results from Action Analysis in an Interactive Learning Environment
47
ly Witten’s idea of field-dependency (Witten & Frank, 1999) attracted attention in the scientific community and was subject to numerous studies. Fielddependent individuals are attributed with the tendency to rely on an external frame of reference (the field) when tackling a task whereas field-independent ones make more use of an internal mental model. As an implication for instruction, field-dependent students may benefit to a larger extent from external help and guidance whereas field-independent ones can take advantage from instructional settings which allow a higher degree of freedom. The advent of hypertext gave rise to a new strand of research which investigated how subjects with different cognitive styles can cope with differently structured environments. Hypertext environments provide a nearly ideal testbed for examining these relationships: field-dependent and fieldindependent subjects may show differences in navigation behaviour and tool use, and their performance may differ depending on the characteristics of the underlying hypertext system (linear vs. non-linear structure, offered tools, offered navigational support). Chen and Macredie (2002) give a comprehensive overview of empirical studies concerning the relationship between field-dependency and hypermedia navigation and their findings. What can be the benefit of knowing students’ cognitive style? There are a number of studies which address this question by comparing scores of subjects performing a task in an environment which corresponds to their cognitive style (matching condition) with scores of subjects in a non-matching environment (Ford & Chen, 2001; Witten, 1999). The results essentially support the hypothesis that matching students’ style does have a positive effect on performances. Therefore, cognitive styles gained some attraction in the adaptive hypermedia community: tailoring content structuring, presentation style and navigation support to individuals’ cognitive style may be a promising approach to improve usability, and in the area of hypertext learning, to increase students’ learning gains. Traditionally, cognitive styles are determined by means of questionnaires and psychological tests. A machine learning model could directly derive styles from behavioural data and would make the application of additional instruments dispensable. Pre-study: Hypotheses Building The quality of a machine learning model is strongly dependent on the set of attributes it is generated from. For the purposes of this study, attributes should reflect behavioural aspects which are relevant for the decision between field-independence and field-dependence, that is, attributes in which field-independent subjects significantly differ from field-dependent ones. Therefore, a pre-study was conducted to review empirical findings reported in the relevant literature. We found evidence that the following aspects are linked with field-dependence: Linearity of navigation, revisita-
48
Scheuer, Mühlenbrock, and Melis
tion behaviour, material coverage, pace, home page visiting behaviour, use of hyperlinks, exercise behaviour and tool usage. The analysis will show if these results can be replicated or not. Experimental Setting Around 300 students of a UK-based University participated in an introductory course in mathematics for computer scientists. For one part of the course, the instruction was based on the ActiveMath system. The computermediated contents covered the topics functions, matrices and graphs. ActiveMath was partly used in supervised tutorial sessions, partly on students’ own discretion. The supervised sessions covered four tutorial weeks: week 1 was mainly devoted to get to know the system, week 2 and 3 to learn the material, and week 4 to revise for exam preparation. Additionally, students were asked to take part in the Cognitive Style Analysis (CSA), a computerised test to determine their cognitive styles. Low test scores correspond to a field-dependent (FD) cognitive style, high scores to a field-independent (FI) cognitive style. ANALYSIS AND RESULTS
Based on the results of the pre-study, a set of attributes, describing students’ behaviour in the ActiveMath system, was computed for each supervised session (see Table 3, in parentheses the hypotheses they refer to). These data, combined with the CSA results of the respective students, constituted the input for the machine learning algorithms. In a first processing step, inappropriate data was filtered out (students not taking part in the CSA test, outliers in terms of extremely short online time (< 20 minutes), small number of pages (< 3), small number of exercises finished (AOPe_pc-src-84 2006-03-1516:10:33 <millisecond>003 NotreKLjava.exe <eventType>keyKeyboard<eventValue>"Coffee Room","de[SPACE]trouver[SPACE]" <producer>AOPe_pc-src-84 2006-03-1516:10:37 <millisecond>000
Matching Performed and Recommended Pedagogical Activities
121
We can note that the use of keyloggers may not respect anonymity which constitutes one of the aspects of privacy. This assertion has to be moderated according to the legislations of the various countries where such systems are used. Within the framework of our experiments, we have chosen the following four principles inspired by the Information and Telecommunication Access Principles Position Statement (http://www.cla.ca/about/access.htm): • No data should be collected from people not aware of it; • Individuals should have the choice to decide how their information is used and distributed; • Individuals should have the right to view any data files about themselves and the right to contest the completeness and accuracy of those files; • Organisations must ensure that the data is secure from unauthorized access. Other Observations In spite of the quantitative richness of the traces resulting from the learners’ stations, some crucial interactions to model the learners’ behaviour might be lacking. Thus, if the observed lesson proceeds in a classroom where a teacher is present, none of the previously observed sources would mention an oral explanation from the teacher or an oral dialogue between students. In the case of distance learning, the problem is identical: it is impossible to know whether or not the learner is in front of his/her machine. Complementary sources of observations could consist in using video (Adam, Meillon, Dubois, Tcherkassof, 2002; Avouris, Komis, Fiotakis, Margaritis, & Voyiatzaki, 2005) or human observers during the lesson. In our experiment, we decided to have human observers to validate possible assumptions. Obtaining an Interpretable Trace In this section, we will show how a person will be able to compose an interpretable trace by annotating the raw observations. We call this person the trace composer. This role could be played either by a student who wishes to keep a trace of his/her path in the session for metacognition purposes, or by a teacher who wishes to compare several traces in order to improve his/her scenario. In the case of our experiment, the trace composer is a teacher, an associate professor at Ecole Supérieure de Commerce (ESC), who conducted the learning session. Raw Observations As stated above, we chose four sources of observation dealing with: • the recommended process (i.e., the teaching scenario imagined by the teacher),
122
Marty, Heraud, Carron, and France
• the actions passing through the intranet server, • the local activities (students’ computers) and • the external observation (two human observers present in each room in order to collect information concerning the exchanges carried out between users, by specifying on a grid their contents and their type). Each of these four sources generates different information. The digital traces (obtained through the first three sources) are presented in Figure 2 and correspond to increasing levels of granularity. We can notice that the traces are linked to each other. For instance, the learning scenario activities observed correspond to several observed elements on the server. For example, three elements observed on the server correspond to exercise 1 failed: (a) the retrieval of the statement, (b) the recording of the file containing the proposed exercise solution, (c) the consultation of the negative validation from the teacher. There are observed elements on the server which do not match any activity envisaged in the learning scenario. For example, the learner posted a message in a forum (d). The observations in the learner’s station can be divided into two categories: • On the one hand, local interactions with software on the learner’s station. For example, the learner can use Microsoft Word (w) to compose a text. • On the other hand, interactions with other computers on the network. For example, communications using an Instant Messenger (f) and (g). Among the latter interactions, those with the server can be easily identified and removed because they already appear in the server log. Comparison with the Recommended Learning Scenario In this section, we compare the activity carried out with the recommended activity in order to measure the differences between the achieved scenario and the recommended learning scenario (cf. 3.2.1). This comparison enables us to
Figure 2. Raw observations
Matching Performed and Recommended Pedagogical Activities
123
estimate the comprehensibility of our observation (cf. 3.2.2). In order to help the trace composer, we propose a graphic representation of this estimate (cf. 3.2.3). Comparison Between Achieved Activity and Recommended Activity This comparison is based on the description of the scenario and the traces obtained through the first source (at the same level of granularity). At this step, there is no need to access the other sources. There are many dimensions to measure the variations with the recommended learning scenario: for instance, the quantity of documents used and produced by the learner or the wealth of exchange with others. Within the framework of our experimentation, it was important for the teacher that the learner ends the learning scenario in a given time. We thus chose to compare the duration of the activity carried out with the duration recommended by the designer of the activity.
Intuitive Estimate of the Clarity of the Observation We define the comprehensibility of a zone as the probability that no ignored activity took place for this duration. The comparison system enables us to represent the observation of an activity with a duration close to the one recommended by the teacher with a strong comprehensibility. On the contrary, if we observe an activity with a duration definitely higher than what was recommended, then the probability that another activity (even one not related to the training) was missed is high. We thus consider that our observation was incomplete in this shaded area. We therefore propose observations available to the trace composer from other sources to let him/her complete the trace. Graphic Representation of the Comprehensibility: The Shadow Bar The shadow bar is a graphic representation of the comprehensibility of an observation. The colour of each area is determined by the estimate of comprehensibility: clear if the observed activities are in line with their recommended durations; dark if the observed activities exceed the time recommended for their permformance; and even completely black if no observation explains the elapsed time. Figure 3 presents the shadow bar corresponding to an observation session. In this example, only observations of the activities of the learning scenario appear in the trace. (a) is the time recommended for exercise 1. (b) is the time exceeding the recommended duration. In the black zone (c), no activity linked with the scenario was observed. Prototype We have designed a user interface prototype that implements a multisource visualisation system coupled with a shadow bar. The prototype was done in Java 2D. The visual representation of the sources and the shadow bar are identical to the previous figures except on the following point: there is
124
Marty, Heraud, Carron, and France
Figure 3. The shadow bar
less visual information, to decrease the cognitive overload of the trace composer, but all the previous data are still available as contextual pop-up menu. For instance, the status (failure/success) of the exercise 1 box is not permanently displayed because it is considered as an attribute of exercise 1. All the attributes of a log box can be locally displayed in a popup. For an example of such a display, see the sixth box of the server source in Figure 4. The information in the popup is displayed with a fish-eye which allows one to maintain readable a single line inside a large amount of data. This prototype was implemented to help the trace composer. The interactions with the prototype are described in the next section. Interactive Construction of the Interpretable Trace In this section, we show how the shadow bar helps the trace composer to choose and annotate the observations among the various sources.
Figure 4. The prototype of the trace composition tool
Matching Performed and Recommended Pedagogical Activities
125
Figure 3 shows the initial situation presented to the trace composer. Only the observations of the activities of the learning scenario are presented. The shadow bar is black during times when no activity has been identified and darkens when an activity exceeds the duration recommended by the teacher. When the trace composer selects shaded zones, the observations on the server corresponding to these zones are posted in the lower part of the chart. For instance, Figure 5 shows the state of the visualisation tool when the trace composer has selected the first totally black zone (between exercise 1 and document 1 read). The trace composer can retain observed elements that s/he considers significant within the framework of the observed learning task. For instance, in Figure 6, four elements (highlighted in black) are of a comparable nature: forum consultation. The trace composer selects these four elements and annotates them as “forum”.
Figure 5. Using the shadow bar to show relevant boxes
Figure 6. Selecting and annotating boxes
126
Marty, Heraud, Carron, and France
The result obtained is presented in Figure 7: a new observation was added to the trace that makes the shadow bar clearer in the corresponding zones. If the trace composer considers observations not to be significant for the learning task, s/he can indicate them as explanatory elements of the time elapsed without including them in the trace. For example, in Figure 8, if s/he designates the two remaining boxes as not significant, the shadow bar on the corresponding zone in Figure 9 is
Figure 7. Adding observations from the server
(a)
(c)
(b)
Figure 8. Choosing observation on the learner’s station
Matching Performed and Recommended Pedagogical Activities
127
cleared with no observation being added. If shaded zones persist in the bar, observations on the learner’s station can be integrated in a similar way. Information in pop-up menus (a) and (b) indicates the direction of the communication which depends on who the initiator of this communication is. The idea is to distinguish communication where the student is the initiator of the exchange, which is probably related to the current activity, from communication where the initiator is someone else, which may not be related to the current activity. For example, in Figure 8, the dialogues of zone (a) were initiated by an external person and were not taken into account by the trace composer. On the other hand, the communications with another student of the session in (b) were annotated as “dialog” in (c). Result Figure 9 presents the trace obtained at the end of the explanation process of the shadow bar. This trace contains only observations of activities related to the observed learning task, according to the trace composer. The clearness of the shadow bar at the end guarantees the low probability of having missed the observation of a learning activity, according to the time metric in this case. DISCUSSION Discussion of Preliminary Results The amount of data recovered during our experimentation is significant and requires the development of specific tools to help the analysis, in particular to automate the composition of elements resulting from different sources. The complete analysis of the results of this experimentation is a work in progress. We want to report here on a first analysis of this experiment. The aim is to take the different traces obtained through the trace composer for a student who succeeded and to compare them with those obtained for one who failed.
Figure 9. Interpretable trace
128
Marty, Heraud, Carron, and France
The learning scenario comprised two exercises and only 7 out of the 36 students finished the scenario before the time limit. We thus isolated the observations of these 7 students in order to compare them with the others. The scenarios carried out by these 7 students are rather similar. If we consider a sufficiently large unit of time (10 minutes) to take into account the various speeds of execution of these students, then the trace of the scenario activities presented on the right of Figure 10 summarises their activity. As a primary conclusion, we supposed that the 7 students that had finished were faster than the others. A more detailed investigation, illustrated in Figure 11, showed us that the true common point between these students was an additional activity, not defined in the learning scenario: all these students requested the validation of exercise 1 by the teacher present in their room. Among the students who had not finished, some read the session documents again; others restarted exercises of the preceding session or compared their results with other students before starting exercise 2. Their traces are too different to be represented in a standard trace. Consequently, it is not possible to compare them with the trace presented in Figure 11. We note the emergence of a new task carried out by all learners that successfully completed the two exercises. It is then legitimate to revise the learning scenario and to wonder whether the insertion of this new task in the scenario is necessary or not. Obviously, the system will not make such a decision. It is in general the designer of the scenario who will deliver his/her
Figure 10. Standard traces of students according to their final results
Figure 11. Standard trace of a student who finished the scenario
Matching Performed and Recommended Pedagogical Activities
129
opinion on the usefulness of this modification. We think that this public needed to be reassured about the quality of their work before starting the second exercise. If we had to redo this lesson with this learning scenario, we would add to the scenario an activity of validation by the teacher between the two exercises. Discussion of the Hypothesis of Our Experiment Our methodology is mainly pragmatic and empirical. We choose to develop a tool allowing us to obtain and keep relevant information about the tasks done by students during a practical session on a technology-enhanced learning system. We validate the results through real experiments: we choose to interfere as little as possible with a classical session in order to get real and valid data. Students are warned that their usual tools are equipped with observation functionalities. The pedagogical sessions took place as usual. The experiment gave us results concerning the improvement of the learning scenario. In order to reach this objective, our approach involves observing the users in a usual working session and tracing the overall activity on the educational platform. We want to point out that multiple sources are fundamental to understand the activity in detail, since they provide valuable additional information. Within its particular scope, the experiment is obviously a success. However, we are aware of the fact that the experiment presented several biases. Some additional experiments should be scheduled to generalise our first results. Thus it would be valuable to set up a new experiment in which the trace composer and the teacher are not the same person; in which the teaching domain is not related to computer science; with a higher number of students; with a more complex pedagogical scenario. The external source was not used in this experiment. We believe that in some cases, it could be very useful, especially to explain dark areas (the student went out of the classroom). CONCLUSION
The work presented in this article proposes a multi-source approach to help an analyst to understand an activity better. We focussed on the match between a recommended pedagogical scenario and the activity performed. This work was illustrated throughout the article by an experiment carried out with two groups of students. The discussion section of this article gives an idea of the benefits which can be drawn from such an approach. The prospects at the end of this experiment are numerous. First, we used the factor of comprehensibility based on the time taken to carry out certain activities. This is only one of the numerous examples of possible metrics. We are going to consider other metrics such as the degree of communication. We noticed from a first general analysis that we can detect a panic state
130
Marty, Heraud, Carron, and France
when the exchanges between participants increase without reason. In addition, the external source of observation, although presented, has not yet been explored. In this experiment, it was simply used as validation for a certain number of assumptions. It would be necessary to consider for the trace composer a means to use this source of information as well as the others when s/he wishes to eliminate shaded areas. Other directions based on this work are currently being explored. We decided to investigate a more precise observation of what is going on throughout the session in the classroom as in (Feng & Heffernan, 2005). This observation is oriented by user classes (with respect to their skills) and a visualization dashboard has been proposed in (France, Heraud, Marty, Carron, & Heili, 2006). This work then led us to consider a more general architecture based on specialized agents (Carron, Marty, Heraud, & France, 2006). In a more general way, increasing the wealth of the observation level and finely clarifying the sequencing of a lesson are paramount stages. They will make it possible to improve the quality of the learning scenarios implemented and, in a second phase, to allow us to evaluate their level of maturity (Marty et al., 2004). References Adam, J. M., Meillon, B., Dubois, M. & Tcherkassof, A. (2002). Methodology and tools based on video observation and eye tracking to analyze facial expressions recognition. 4th Int. Conference on Methods and Techniques in Behavioral Research. Apache (n.d.). The Apache Software Foundation. "Apache HTTP Server Project", http://httpd.apache.org/. Avouris, N., Komis, V., Fiotakis, G., Margaritis, M. & Voyiatzaki, E. (2005). Logging of fingertip actions is not enough for analysis of learning activities. AIED’05 workshop on Usage analysis in learning systems, 1-8. Carron, T., Marty, J. C., Heraud, J. M. & France, L. (2006). Helping the teacher to re-organize tasks in a collaborative learning activity: An agent-based approach. The 6th IEEE International Conference on Advanced Learning Technologies, 552-554. Cooley, R., Mobasher, B., & Srivastava J. (1999). Data preparation for mining world wide web browsing patterns. Knowledge and Information Systems (KAIS), 1(1), 5-32. Egyed-Zsigmond, E., Mille, A. & Prié Y. (2003). Club [clubsuit] (Trèfle): A use trace model. 5th International Conference on Case-Based Reasoning, 146-160. Feng, M. & Heffernan, N. T. (2005). Informing teachers live about student learning: Reporting in assistment system. AIED workshop on Usage analysis in learning systems, 25-32. France, L., Heraud, J. M., Marty, J.C., Carron, T. & Heili J. (2006). Monitoring virtual classroom: Visualization techniques to observe student activities in an e-learning system. The 6th IEEE International Conference on Advanced Learning Technologies, 716-720. Iksal, S., & Choquet, C. (2005). Usage analysis driven by models in a pedagogical context. Workshop Usage analysis in learning systems at AIED2005: 12th International Conference on Artificial Intelligence in Education, 49-56.
Matching Performed and Recommended Pedagogical Activities
131
IMS (n.d.). Global learning consortium: Learning design Specification, http://www.imsglob al.org/learningdesign/. Martel, C., Vignollet, L., Ferraris, C., David, J.-P., & Lejeune, A. (2006). Modeling collaborative learning activities on e-learning platforms. The 6th IEEE International Conference on Advanced Learning Technologies, 707-709. Marty, J. C., Heraud, J. M., Carron, T., & France, L. (2004). A quality approach for collaborative learning scenarios. Learning Technology newsletter of IEEE Computer Society, 6(4), 46-48. Mazza, R., & Milani, C. (2005). Exploring usage analysis in learning systems: Gaining insights from visualisations. AIED’05 workshop on Usage analysis in learning systems, 65-72. Paulk, M. C., Curtis, B., Chrissis, M. B., & Weber, C.V. (1993). Capability maturity model Version 1.1. IEEE Software, 18-27. Pscenario (n.d.). TECFA’s pedagogical scenario tool for PostNuke, http://tecfaseed.unige.ch/door/ (download: http://tecfaseed.unige.ch/door/index.php? name=Downloads&req=viewsdownload &sid=12). Spärk-Jones, K., & Galliers, J. R. (1996). Evaluating natural language processing systems: An analysis and review (Lecture notes in artificial intelligence). Springer. Zaïane, O.R. (2001). Web usage mining for a better web-based learning environment. Conference on Advanced Technology for Education, 60-64.
Acknowledgements This work has been performed in the ACTEURS project (2005-2007), founded by the French Ministry of Education, (ACI “Terrains, techniques, théories”) and in the Computer Science Cluster (project EIAH) founded by the Rhône-Alpes Region.
Usage Analysis in Learning Systems, 133-156
From Data Analysis to Design Patterns in a Collaborative Context VINCENT BARRÉ LIUM/IUT de Laval, France
[email protected] CHRISTOPHE CHOQUET LIUM/IUT de Laval, France
[email protected] HASSINA EL-KECHAÏ LIUM/IUT de Laval, France
[email protected] The underlying aim of the work related in this article, was to define Design Patterns for recording and analyzing usage in learning systems. The implied “bottom-up” approach when defining a Design Pattern brought us to examine data collected in our learning system through different lights: (1) the data type, (2) the human roles involved in the production of the data, or interested by their uses, and (3) the nature of the data analysis. This method has allowed us to have a global view on the data, which can be easily generalized and formalized.
Introduction The desynchronization between the design and the uses in distance education penalizes the iterative optimization of the system’s quality by not taking into account uses with a reengineering objective. That’s why in (Corbière, & Choquet, 2004a) we have proposed a meta-architecture model which explicitly integrates a step dealing with the observation and the comportment analysis of distance learning systems and learning process actors in an iterative process, guided by design intentions. We underline, in particular, the need for a formal description of the design point of view of the scenario, called the prescriptive scenario, as well as the assistance in use analy-
134
Barré, Choquet, and El-Kechaï
sis by allowing the comparison of descriptive scenarios (an a posteriori scenario that effectively describes the learning situation’s sequence (Corbière, & Choquet, 2004b; Lejeune, & Pernin, 2004)) with the predictive scenario. This produces information, significant for designers from a pedagogical point of view, when they perform a retro-conception or a reengineering (Chikofsky, & Cross, 1990) of their systems. In the framework of REDiM (French acronym for Reengineering Driven by Models of e-Learning Systems) project, we are particularly interested in supporting the implementation of designers’ two main roles: (i) to establish the predictive scenario of a given learning situation, and (ii) to anticipate descriptive scenario construction by defining situation observation needs allowing the effective evaluation of the learners’ activity. Thus, describing in details and formalizing the data recorded during a learning session, in order to build significant indicators which qualify the activity, is a crucial step for us. Moreover, such an analysis expertise could be capitalized in a pattern, which could be used in the design of an another TEL system. In this article, we will focus on a particular collaborative e-learning system named Symba. More precisely, we will observe the effective use of a pedagogical scenario in the context of a collective activity supported by collaborative tools. Our experiment thus consists of a project management collective activity, and more specifically, of a web-project management activity (specification and implementation of a website). From our pedagogical reengineering viewpoint, considerable interesting information can arise from this experiment. In particular, we are interested in comparing descriptive scenarios with predictive ones. Moreover, in a collaborative context, another interesting advisability is to compare roles emerging from the activity to those anticipated by designers. In our experiment, and in accordance with a normalization context, we have used the pedagogical model arising from IMS consortium Learning Design (Koper, Olivier, & Anderson, 2003) in order to describe learning activities and to explicit pedagogical scenarios. Nevertheless, we only use IMS LD as a mean for designers to express their intentions, and not in an implementation perspective. The underlying aim of the work related in this article, was to define Design Patterns for recording and analyzing usage in learning systems, in the framework of the DPULS project (DPULS, 2005). The implied bottomup approach when defining a Design Pattern brought us to examine our experience’s data through different lights: (1) the data type, (2) the human roles involved in the production of the data, or interested by their uses, and (3) the nature of the data analysis. This method has allowed us to have a global view on the data, which can be easily generalized and formalized. We are thus within the framework depicted in Figure 1. We have set up an experiment (Symba) to produce data that will be described and analyzed so as to produce indicators (feature of a data highlighting its connection with
From Data Analysis to Design Patterns in a Collaborative Context
135
an envisaged event having a pedagogical significance (DPULS, 2005)). Once formalized, the analyses will then lead to Design Patterns. Thus, according to the framework depicted in Figure 1 and after a short presentation of Symba experiment, we will focus on the three main viewpoints leading towards design pattern elaboration. More precisely, we will first have to ask ourselves about the formal definition of roles endorsed by actors of the system and to specify the motivation that each role has towards data analysis. In (DPULS, 2005) we define a role as a set of tasks performed by one or more human or artificial agent according to specific needs and competencies, e.g. designer, learner, teacher… The second viewpoint will allow us to clarify what kind of data will be analyzed, that is, to formalize and classify each data that will be manipulated by actors in many roles in our experiment. The last viewpoint, which will lead to the formalization of design pattern, will focus on data analysis to distinguish who analyzes data, from who uses the analysis results. Then, those three viewpoints will lead to the formalization of two kinds of design patterns. One kind will be used by analysts in order to produce new data (that is, will represent a formalization of their analysis knowledge), whereas the other kind will be used for high level purposes, that is for engineering, reengineering or regulation purposes, and those patterns will lead to the production of indicators. Presentation of Symba Experiment We have used an experimental CSCL support environment called SYMBA (Betbeder, & Tchounikine, 2003). This environment is a web-based system, developed by the LIUM laboratory in the framework of a Ph.D. study, in order to support Collective Activities in a Learning Context. It was designed following a double objective: (i) allowing students to explicitly work on their
Figure 1. Focus on some aspects of our methodology
136
Barré, Choquet, and El-Kechaï
organization and (ii) providing tailorability (Morch, & Mehandjiev, 2000) features which let students decide which tools and resources they want to be accessible in order to achieve tasks they have defined. With this system, students have to develop a dynamic website using previously taught web project management methodology. A predictive scenario of our experiment is depicted in Figure 2, we will now detail this scenario. According to our theoretical framework, students first have to work collectively (and agree) on the project organization (such as what to be done, who does what, when tasks have to be finished, which tools are necessary for a particular task) before beginning the second part, which consists of collectively performing the tasks they have defined, according to their organization. During all their activities, learners are self-managed and have to define (and collectively agree on) roles they will endorse and act in consequence. From this viewpoint, it will be interesting to take into account that roles have many meanings, and particularly a functional meaning (that is, related to an action, linked to people’s status in an organization) and a socio-affective meaning (that is, related to the way people slip their personality into a functional role). In concrete terms, the learners’ activity was organized in five steps (see Figure 2), and for each one instructional designers set up a task model (see Figure 3) which was communicated to learners for the first three steps. The formal description of what is a correct project organization (that is, task model for each step) is formalized using IMS Learning Design (see data S4.1). In order to explicit this task model, we present in Figure 3 what has to be done for the second step. Many Roles, Different Motivations in Data Analysis In our system, one actor can endorse many roles (for example, the teacher can be either instructional designer, assessor tutor or observed uses analyst). We think that it is very important to focus on role definition rather than on actor definition. We will, thus, began by presenting the different roles endorsed by actors of our system (see Figure 4), and, then, we will detail motivation that those roles have in data analysis.
Presentation of Roles This experimental system is used by actors endorsing four distinct categories of roles (one actor can play more than one role in our experimental system). The first category is made of 56 learners in higher education, from the Laval Institute of Technology, University of Maine (France). They were associated in small groups of five students and they worked either at the University or at home using tools offered by Symba. Those proposed tools are centered around the description, organization and perception of the activity, but learners must also use the environment in order to explicit the organization of their work, with a sharable plan and task editors. The activity
From Data Analysis to Design Patterns in a Collaborative Context
137
Figure 2. Predictive scenario of the web-project management proposed to the learners lasts for four weeks (35 working hours per week) and a predictive pedagogical scenario implying a collaborative learning was proposed, even if students are free to adopt or modify it. One can notice that this predictive scenario may involve concepts that have not yet been taught. The second category is made up of three kinds of tutors. We have moderator tutors whose role is to monitor activity within the learning session and to fill in reports for evaluating tutors (i.e., assessor tutors) in charge of
138
Barré, Choquet, and El-Kechaï
Figure 3. One step of students’ predicted organization
Figure 4. Four kinds of roles (with sub-roles) in our experiment
From Data Analysis to Design Patterns in a Collaborative Context
139
evaluating learners’ activity. This measures knowledge they have acquired. Lastly, domain experts are in charge of assisting learners in their tasks by helping them to solve specific problems connected to their expertise domain. A third category is made of instructional designers. They specify the predictive pedagogical scenario and the uses of the learning system to be observed, they also use results of the effective use of the Learning System analysis in order to improve it (reengineering process). The last category is made of two kinds of analysts. Observed uses modelers are building tracks with collected raw data, either from the Learning system or not (that is, with collected observed uses), whereas observed uses analysts are analyzing the observed uses in order to synthesize information.
Different Motivations in Data Analysis In our experiment, people in many roles want to (and are interested in) analyze data. Instructional designers want to verify if the roles they have predicted are adopted by learners and to detect unforeseen new roles. They are also interested in understanding the effective progress of a session in order to discover inconsistencies in it for reengineering purposes. Observed uses modelers are interested in finding new techniques to improve their analysis abilities, whereas observed uses analysts are interested in finding new patterns to improve their analysis abilities. A part of a moderator tutor’s job is to make reports for assessor tutors on learners’ abilities to collaborate and to work in a group. Assessor tutors want to evaluate knowledge acquired by learners in web project management by verifying if the produced organization is coherent with the method taught during web project management courses. Lastly, domain experts are also involved in analyzing data. Whilst they do not currently analyze data since this analysis cannot be done during the learning session (exploratory manual analysis in this first stage), they nevertheless would be interested in analyzing data so as to understand what learners have done previously when they ask them for help.
What Kind of Data is Being Analyzed? In this section, we will distinguish primary data (data that have not been processed) from derived ones (data obtained from other data). In our experimental system, we have three kinds of primary data (see Figure 5): raw data (data recorded by the learning system), additional data (data linked to an activity but not recorded by the system during sessions) and content data (data produced by the system actors). We also have derived data, some of them being indicators (that is, they have to be interpreted, taking into account the learning activity, the profile and roles of the actors, as well as interaction context), others being intermediate data. From a reengineering perspective, we will use some raw data (either recorded by the learning system or not) in order to derive some new data that
140
Barré, Choquet, and El-Kechaï
will be useful for system actors. We will also need some additional data, such as the predictive scenario for the activity, and content data, that is, outcomes produced by actors during their activities. From a pedagogical perspective, a learning assessment includes both cognitive achievement and collaboration capabilities aspects. Whereas the first aspect can be evaluated by learners’ production analysis (Jonassen, Davidson, Collins, Campbell, & Bannan Haag, 1995) and mainly concerns data labeled as S-3.1, S-3.2, S-5.1 and S-5.2 in Figure 5, the second one can be evaluated using comportment observation and perception analysis (Henri, & Lundgren-Cayrol, 2001). In this article, we will focus on this second aspect and more precisely on role emergence and we will now detail the most important data that helps us to formalize emerging roles arising from learners’ activity. We will first detail the raw data, either recorded by the learning system or not. Please note that many of our raw data deal with communication tools’ tracks and that, in the original tracks, messages are written in French. All those messages have been translated into English. Data S-1.2 (data arising from chat) corresponds to the transcription of all communications exchanged between learners via the chat service. A partial transcription of such messages can be found in Figure 6. Data S-1.3 (data arising from newsgroups) corresponds to the transcription of the entire set of messages posted on newsgroup services. An example of such a message can be found in Figure 7.
Figure 5. Dependencies between data
From Data Analysis to Design Patterns in a Collaborative Context
141
Figure 6. Excerpt of messages exchanged on the chat service
Figure 7. Excerpt of a message exchanged on the newsgroup tool Data S-2.1 (data arising from questionnaires) consists of questionnaires, whose main goal is to evaluate the group functioning by measuring parameters such as participation, collaboration and organization. Student answers to questionnaires are measured with a Likert scale (Babbie, 1992), graduated from 1 to 5 (strictly disagree, disagree, neither agree nor disagree, agree, completely agree). Learners can also give some detailed explanations about their answers. An example of such a questionnaire can be found in Figure 8.
Figure 8. Excerpt of a completed questionnaire
142
Barré, Choquet, and El-Kechaï
We will now detail data obtained by combining it with other data (either primary data or already synthesized ones). Data S-3.2 (data related to collaborative communication tools, i.e., role emergence from chat) is derived from the transcription of all communications exchanged with a chat service. Emerging roles are extracted from this transcription using pragmatic markers (Cottier, & Schmidt, 2004) that need to be defined. Notice that a particular learner can assume many roles throughout a given learning session. All these roles will be reported. So, you need to define pragmatic markers associated to each role you want to find within the gathered sequences. In order to do that, you need to detect both the lexical field of pertinent terms linked to a particular role and the dialog topics. Detecting lexical fields mainly consists in finding words that can be termed as activity warning, such as: work, organization, design, to work, to organize, and to design. You must often base yourself on the fact that those terms are the most used throughout dialog in order to consider them as pertinent. To detect a dialog topic, you can use their proximity with lexical fields previously detected. In order to correctly define pragmatic markers, you also need to identify who is the originator, who are the recipients of the gathered dialog and to interpret the exchange meaning (see Figure 9 for an example). Pragmatic markers linked to our experiment have been manually extracted and organized into an ontology. We are currently working on an automated service that will extract emerging roles from communication tools tracks. This data therefore consists of a list of roles arising from observed communications. This list is annotated with information about which student(s) takes which role(s) and consists in a structured file (see Figure 10). One can notice that these roles and assignments can be identical to those arising from other communication tools (e.g., newsgroups, see data S-3.3). Data S-3.3 (data related to collaborative communication tools, i.e., role emergence from newsgroups) is derived from the transcription of all com-
Figure 9. Pragmatic makers example: lexical fields are highlighted, dialog topic are underlined
From Data Analysis to Design Patterns in a Collaborative Context
143
Figure 10. Annotated list of roles arising from chat (or newsgroup) analysis munications exchanged on newsgroups. Emerging roles are extracted from this transcription using pragmatic markers that need to be defined (see Figure 11). This data therefore consists of a list of roles arising from observed communications (see Figure 10). This list is annotated with information about which student(s) takes which role(s) and consists in a structured file (same structure as for data S-3.2). One can notice that these roles and assignments can be identical to those arising from other communication tools (e.g., chat service, see data S-3.2). Data S-3.4 (data related to questionnaire synthesis) is made of answers to questionnaires (data S-2.1) synthesized in percentages and reported within an evaluation grid summarizing this information for each question.
Figure 11. Pragmatic markers identifying a ‘functional leader' role in newsgroups
144
Barré, Choquet, and El-Kechaï
Data S-3.5 (data related to new roles arising from learners’ activity). The study of interactions made with Symba communication tools (data S-3.2 and data S-3.3), as well as detailed answers made to questionnaires (data S-2.1), allow for the evaluation of the collaborative process from a cognitive and socio-affective viewpoint. This manually derived data consists of an ad-hoc free text report (whose writing is guided by some questions) and allows the evaluation of the coherence between the different facets of each role. In order to highlight this data, we can take an example from our experiment. We have observed the following key points: (i) in the transcription of chat messages (data S-1.2), one student has been unanimously appointed project leader and, therefore, one can expect that this student will play his leader role and that he was chosen by his colleagues because they think he is qualified in group management. (ii) Analysis of data S-1.2 with help of pragmatic markers (in order to produce data S-3.2 – roles emerging from chat messages) indicates that most of the interactions are organized around another student. (iii) In detailed answers to questionnaire (data S-2.1), everyone acknowledges that the initially designed project leader was rejected by all other team members, even if he tried to fulfill his (task based) functional role. This rejection was based on a lack of communication skills of this project leader. To synthesize this situation, one can say that instructional designers have defined a predictive role of project leader and are expecting that this project leader act as a leader on their own. Although he was effectively a leader with respect to tasks they have to do, he was not completely accepted as a leader with respect to his communication skills. Consequently, one can suggest that this project leader role can be split in two facets: a functional one and a socio-affective one (see Figure 12).
Figure 12. Functional and socio-affective leadership (Hotte, 1998)
From Data Analysis to Design Patterns in a Collaborative Context
145
Moreover, in our experiment we are in the context of a self-managed group. In such a context, the two facets must be simultaneously present in the student assuming the project leader role (in order to coordinate interactions inside the group and to facilitate communication between group members). Otherwise, if they are not simultaneously present in the same people, this leads to a group dispersal and reduces collaboration quality. Thus, in our experiment, the data S-3.5 allows us to verify the coherence of the different facets of the roles arising from activity. Data S-3.6 (collaboration quality) corresponds to an indicator allowing the evaluation of the collaboration quality between learners. This data consists of an ad-hoc free text report (whose writing is guided by some questions) and is manually made using reports showing role coherence arising from learners’ activity (data S-3.5) combined with the task model produced by designers (data S-4.1). This verifies the parallels between predicted roles and observed ones (at a per learner level) and requires information on whether collaboration takes place between learners or not. This is derived from the questionnaires, both synthesized form and detailed answers (that is, data S-2.1 and data S-3.4). We will lastly describe one additional data which is used to highlight synthesized data. Data S-4.1 (task model specified by instructional designers) corresponds to the task model as anticipated by designers (see Figure 3). That is, an indication of the activity sequence that learners are supposed to produce using the workplace organization from Symba. This task model is expressed using IMS Learning Design (and, technically, it is an XML file conforming to IMS/LD specification, see Figure 13).
Figure 13. Predictive task organization (excerpt)
146
Barré, Choquet, and El-Kechaï
DATA ANALYSIS
Analysts, and sometimes tutors, analyze data in order to synthesize the information they contain. Results of these different analyses are then used by many actors of our e-learning system. Analysts use them in order to produce new analyses, tutors use them to evaluate learners and designers use them to improve their predictive scenario (following a reengineering cycle) and to capitalize knowledge so as to produce new scenarios (engineering purpose). In the previous section, we have described data (either primary or derived ones) necessary to produce ‘collaboration quality’ indicator. We will now detail how those data are analyzed to produce this indicator. We recall that all dependencies between data are depicted in Figure 5.
Who Analyses Data, How and When? Presently, all analyses are made by the observed uses modelers (analyzing raw data) and observed uses analysts (making analysis from analysis reports made by observed uses modelers). In this first step of our experiment, most of our analyses are done manually, at the end of a learning session. We will first detail the analysis made by observed uses modelers. The analysis of data S-1.2 (data arising from chat), that is, tracks produced by learners via their interactions through the chat system, is done using pragmatic markers (Cottier, & Schmidt, 2004) in order to identify emerging roles. Analysis of data S-1.3 (data arising from newsgroups) is very similar: tracks are produced by learners, by their interactions through newsgroups, and are analyzed with pragmatic markers (Cottier, & Schmidt, 2004) at the end of the session. Data S-3.2 (role emergence from chat) and data S-3.3 (role emergence from newsgroups) are then analyzed together. Roles lists arising from both data are merged into one list which is then enriched with annotations (learners in role) that they contain. Analysis of data S-2.1 (data arising from questionnaires) is made by observed uses modelers and consists of synthesizing answers to questionnaires in percentages and to report them with an evaluation grid. We will now evoke analysis made by observed uses analysts. The analysis they have to do mainly consists of synthesizing information from data S2.1, S-3.4, S-3.5 and S-4.1 in order to produce collaboration quality indicator (that is, data S-3.6). Since data S-2.1 (data arising from questionnaires) also contains detailed answers to questionnaires, it can be used in order to make a synthesis concerning collaboration inside the group. The analysis of data S-3.4 (questionnaires synthesis) is carried out by the human analyst to highlight whether collaboration takes place or not (focusing on learners’ abilities to collaborate
From Data Analysis to Design Patterns in a Collaborative Context
147
and to work in a group). For this purpose, questionnaires were built allowing the evaluation of variables such as: participation, collaboration, communication, work atmosphere, and leadership (see Figure 14). They also need to analyze data S-3.5 (data related to new roles arising from learners’ activity) since they need information regarding similarities between effective roles and their socio-affective facets. For example, a learner having a functional role of project manager would ideally have an organizer or a leader socio-affective role and be rather active (whereas if he has a follower socio-affective role, he would be less successful in his task). Finally, they need to analyze data S-4.1 in order to compare roles arising from the activity to those that were predicted by instructional designers. For the moment, this analysis is done at the end of a learning session. Nevertheless, obtained results suggest that it would be judicious to detect functional and socio-affective role mismatching during the session in order to react as soon as it is detected. This would imply adopting a formative evaluation rather than a summative one (especially for questionnaires). This would also imply automating role extraction with pragmatic markers. We are presently working on it, building an ontology of pragmatic markers. We have already extracted a first subset of lexical fields and dialog topics and we are currently working on extending them. Who Uses the Results of the Analysis, How and for Which Kind of Action? The results of the different analyses are used by many actors of our elearning system. Analysts use them in order to produce new analyses, tutors use them to evaluate learners and designers use them to improve their predictive scenario (following a reengineering cycle) and to capitalize knowledge so as to produce new scenarios (engineering purpose). Therefore, motivations in data analysis can be viewed from one of the following viewpoints: engineering, reengineering, or learners regulation. Moreover, all of our
Figure 14. “Leadership” evaluation with questionnaire
148
Barré, Choquet, and El-Kechaï
analyses are not carried out with the same goal. Some of them are only intermediary results used to produce new data, whilst others are indicators having a meaning by themselves. We will first detail how analysts (both observed uses modelers and observed uses analysts) use the results of previous analysis in order to build new data. First, observed uses modelers are in charge of formatting roles identified by analysts using pragmatic markers on chat messages (data S1.2) and newsgroups messages (data S-1.3). They produce, respectively, data S-3.2 and data S-3.3. Observed uses modelers also need to format percentages calculated by analyst arising from analysis of questionnaires (data S-2.1) using a synthesis grid. Results of analysis of data S-3.2 and data S3.3 are manually formatted in order to constitute a basis for data S-3.5. Then the results of the detailed answers analysis in data S-2.1 is used by observed uses analysts to confirm roles arising from data S-3.2 and data S-3.3. For example, in our questionnaire we have asked students about leadership. The students have to explain if this role has been assumed by one particular student and to express their opinion about commitment of their leaders. If most student answers are similar, pointing out the same student, we can consider that this student has a leader role and therefore this confirms that this role emerged from the activity. We will now detail how indicators are (or can be) used from the following viewpoints: engineering, reengineering, and learners regulation. In our experiment, we have identified three indicators: acquired knowledge quality (data S-3.7), system quality (data S-3.9) and collaboration quality (data S3.6). We will now focus on this last one which comes from the data S-2.1, data S-3.4, data S-3.5 and data S-4.1 joint analyses. From an engineering viewpoint, this indicator will be useful for instructional designers to capitalize knowledge and produce new collaborative scenarios. From a reengineering viewpoint, this indicator will also be useful to instructional designers as it allows them to improve their predictive scenario, taking into consideration effective collaboration that has been observed. Lastly, from a regulation viewpoint, this indicator could be used by moderator tutors in order to correct collaboration problems that can emerge during the activity. Nevertheless, this use implies that these indicators must be computed during the learning session, which is not yet the case. It is also to be expected that these indicators could also be used by assessor tutors to attribute (at least partially) a grade to learners, but, in our experiment, assessor tutors have rejected this use as it seemed too subjective for them. More precisely, they pointed out that they were afraid of penalizing students since this indicator corresponds to an evaluation of the collaboration within the whole group (rather than an individual assessment). They nevertheless point out that such an indicator could be a great help from a regulation viewpoint (if it can be computed during the session rather than at the end of a learning session).
From Data Analysis to Design Patterns in a Collaborative Context
149
From Data Analysis to Design Patterns Design Patterns were originally developed in architecture, then in software engineering and now one finds design patterns in communication interactions and e-learning issues: human computer interaction, web design, pedagogical patterns, and patterns for implementing an institutional e-learning centre1. Design Patterns embody the design experience a community has developed and learned. They describe recurrent problems, the rationale for their solution, how to apply the solution and some of the trade-offs in applying the solution. In this article, we have already shown a way to produce a collaboration quality indicator in order to evaluate whether collaboration takes place or not in a group and to detect functional vs. socio-affective role mismatching among learners. Thus, we will now use design patterns in order to formalize both our observation problem and how to use the ‘collaboration quality’ indicator to tackle this problem. We will use the design pattern language (Celorrio, & Verdejo, 2005) finalized by DPULS project (DPULS, 2005). We will nevertheless need to slightly modify this language so that it fits our needs better. Firstly, the DPULS language defines indicators used by described solution. We need a little more information on indicators, and, particularly, to distinguish if they are used as an input to achieve solution, or built during solution achievement, and, thus, are part of the solution. The second point is related to different viewpoints from which indicators can be used. In previous section, we have seen that indicators can be used from the following viewpoints: engineering, reengineering, and learners regulation. Thus, we need to incorporate these viewpoints in our design pattern language. Emergence of Roles by Tutor Work done in order to produce data S-3.2 and data S-3.3, that is extracting emerging roles from the transcription of communications between learners using pragmatic markers (Cottier, & Schmidt, 2004) is not really dependant on the communication tool used. Indeed, work done with data arising from the newsgroups is very similar to work done with data arising from chat. The key point of this analysis is the method, that is, the use of pragmatic markers. This idea can be abstracted (or captured) with the design pattern shown in Figures 15 – 19. This pattern is thus an abstraction of the process used to produce data S3.2 and S-3.3. But one can also use this DP in order to define an automated service that extracts emerging roles from communication tools tracks. In order to define such a service, you must provide two different elements: (i) communication tool tracks, and (ii) pragmatic markers (for your domain and tailored for the specific roles you want to observe) organized in an ontology (obtained using our “C2.1 - Emergence of roles by tutor” Design Pattern).
150
Figure 15. C2.1 DP > General section
Figure 16. C2.1 DP > Problem section
Barré, Choquet, and El-Kechaï
From Data Analysis to Design Patterns in a Collaborative Context
Figure 17. C2.1 DP > Solution section
Figure 18. C2.1 DP > Related Patterns section
Figure 19. C2.1 DP > Pattern identification section
151
152
Barré, Choquet, and El-Kechaï
Collective Organization of a Synchronous Activity Another key point of our data analysis can be abstracted in a similar way: data S-3.6 (collaboration quality). First of all, the building of data S-3.6 uses data S-3.2 and data S-3.3 which are concerned by our first design pattern. Moreover, we use in this process two indicators formalized in (Manca, Persico, Pozzi, & Sarti, 2005a, 2005b). These two indicators can be used as a substitute (and as an enhancement) for data S-3.4 (questionnaire synthesis). The first one is active participation and aims at detecting the number of performed actions showing active participation by students in a given area / course. It takes into account three main activities: sending a message, uploading a document, and attending a chat. The second one is passive participation and aims at detecting the number of passive acts performed by students (e.g., a sending a message, downloading a document). This process thus leads to the design pattern shown in Figures 20 – 24:
Figure 20. C2 DP > General section
From Data Analysis to Design Patterns in a Collaborative Context
Figure 21. C2 DP > Problem section
Figure 22. C2 DP > Solution section
153
154
Barré, Choquet, and El-Kechaï
Figure 23. C2 DP > Related Patterns section2
Figure 24. C2 DP > Pattern identification section CONCLUSION
In a collaborative e-learning system, tracks arising from communication tools allow us to build useful indicators for all system actors. Indeed, some indicators like collaboration quality (data S-3.6) can, at once, be used by tutors to evaluate learners, by analysts to build other indicators and by designers to evaluate the relevance of their pedagogical scenarios. From this last point of view, we have shown in this article that considering emerging roles arising from communication tools tracks can be useful for reengineering purposes. For example, in our experiment, we have clarified a first reengineering cycle, and this first cycle has allowed us to enrich the predictive scenario made by designers by adding socio-affective roles arising from learning session tracks analysis. Role emergence was one key point of our reengineering process, and was in keeping with comparison of predictive
From Data Analysis to Design Patterns in a Collaborative Context
155
scenarios and descriptive ones enriched with emerging roles. Another interesting point is that proposed indicators can be used in a more general framework than that of our experiment. Indeed, role mining from communication tools tracks can help to enlighten the effective use of the collaborative system and to push the collaboration quality indicator forward, whatever the collaborative experiment may be. These general frameworks then lead to the definition of design patterns. Moreover, in order to support the production of such generic indicators, we have defined software tools (Iksal, Barré, Choquet, & Corbière, 2004) that, once fully developed, will allow the analysis of the collected data based both on the predictive scenario and the formal description of elements to be observed. They will produce formal representations of user comportment, based on observation needs, and thus form a useful guide to implement the reengineering process. Finally, although we have not formalized our methodology for defining Design Patterns, we think our approach could be generalized and applied to another experiences. Indeed, the participants of the DPULS project (ten research teams) have more or less employed the same methodology on their own experiments and the result was the definition of a structured set of forty Design Patterns3. We hope this result could be the first step of a process for capitalizing and sharing through the Technology Enhanced Learning community knowledge on usage analysis. References Babbie, E. (1992). The practice of social research. Belmont, California, Wadsworth. Betbeder, M.-L., & Tchounikine, P. (2003). Symba: A framework to support collective activities in an educational context. International Conference on Computers in Education (ICCE 2003). December 2-5, 2003. Hong-Kong (China). 188-196. Celorrio, C., & Verdejo, F. (2005). Deliverable 32.5.01: The design pattern language. DPULS Project, Kaleidoscope NoE, 2005. [On-line]. Available : http://www.noe-kaleidoscope.org/ Chikofsky, E. J., & Cross II, J. H. (1990). Reverse engineering and design recovery: A taxonomy. IEEE Software, 7(1), 13-17. Corbière, A., & Choquet, C. (2004a). Designer integration in training cycles: IEEE LTSA model adaptation. International Conference on Computer Aided Learning in Engineering Education (CALIE’04), February 16-18, 2004. Grenoble (France). 51-62. Corbière, A., & Choquet, C. (2004b). A model driven analysis approach for the re-engineering of e-learning systems. ICICTE'04. July 1-3, 2004. Samos (Greece). 242-247. Cottier, P., & Schmidt, C.T. (2004). Le dialogue en contexte: Pour une approche dialogique des environnements d'apprentissage collectif. Colloque ARCo 2004. December 8-10, 2004. Compiègne (France). DPULS (2005). Design patterns for recording and analyzing usage in learning systems. Workpackage 32, Kaleidoscope Network of Exellence, supported by the European Community under the Information Society and Media Directorate-General, Content Directorate, Learning and Cultural Heritage Unit. Contract 507838. Consulted June, 2006, at http://www.noe-kaleidoscope.org/
156
Barré, Choquet, and El-Kechaï
Henri, F., & Landgren-Cayrol, K. (2001). Apprentissage collaboratif à distance: pour comprendre et concevoir les environnements d’apprentissage virtuels. Sainte-Foy (Québec, Canada): Presses de l'Université du Québec, ISBN 2-7605-1094-8. Hotte, R. (1998). Modlisation d’un système d’aide multiexpert pour l’apprentissage coopératif à distance. Unpublished doctoral dissertation, Université Denis Diderot/Paris 7. Iksal, S., Barré, V., Choquet, C., & Corbière, A. (2004). Comparing prescribed and observed for the re-engineering of e-learning systems. IEEE Sixth International Symposium on Multimedia Software Engineering, December 13-15, 2004. Miami (USA). Koper, R., Olivier, B., & Anderson, T. (2003). IMS Learning Design v1.0 Final Specification [on-line]. Available: http://www.imsglobal.org/learningdesign/index.html. Jonassen, D. H., Davidson, M., Collins, M., Campbell, J., & Bannan Haag, B. (1995). Constructivism and computer-mediated communication in distance education. Journal of Distance Education. 9(2), 7-27. Lejeune, A., & Pernin, J-P. (2004). A taxonomy for scenario-based engineering. Cognition and Exploratory Learning in Digital Age (CELDA 2004), December 2004. Lisboa (Portugal). 249-256. Manca, S., Persico, D., Pozzi, F., & Sarti, L. (2005a). An approach to tracking and analyzing interactions in CSCL environments. Proceedings of the E-learning Conference, 2005. Berlin (Germany). Manca, S., Persico, D., Pozzi, F., & Sarti, L. (2005b). Deliverable 32.4.01: Know-how list. DPULS Project, Kaleidoscope NoE, 2005. [On-line]. Available : http://www.noe-kaleidoscope.org/ Morch, A., & Mehandjiev, N. D. (2000). Tailoring as collaboration: The mediating role of multiple representation and application units. CSCW'2000, December 2-6, 2000. Philadelphia, Pennsylvania (USA). 75-100.
Acknowledgments This work has been done within the framework of the DPULS project (DPULS, 2005), funded by Kaleidoscope Network of Excellence supported by the European Community. Notes e-LEN project, see http://www2.tisip.no/E-LEN/patterns_info.php (last consulted, April 2006). Pattern C1 consists in evaluating if and to extent collaboration is taking place using quantitative and qualitative analysis of interactions, Pattern C2.2 differs from pattern C2.1 by using an automated service in order to analyze communication tool tracks. 3 These Design Patterns are accessible at: http://lucke.univ-lemans.fr:8080/dpuls/login.faces 1 2
Usage Analysis in Learning Systems, 157-181
A Structured Set of Design Patterns for Learners’ Assessment ÉLISABETH DELOZANNE Université Paris-Descartes, France
[email protected] FRANÇOISE LE CALVEZ Université Paris 6, France
[email protected] AGATHE MERCERON Technische Fachhochschule Berlin, Germany
[email protected] JEAN-MARC LABAT Université Paris 6, France
[email protected] In this article we present a structured set of Design Patterns (DPs) that deals with tracking students while they solve problem in specific domains such as programming or mathematics. Our collection of 17 DPs yields a three step approach: First step: to collect and analyze information on each student for each exercise; Second step: to build a higher level view of one student's activity on a set of exercises; Third step: to build an overview of the whole class activity. To evaluate our DP set we investigate whether our DPs account for experiences from the literature as a first step toward a pattern language for students’ assessment.
Introduction The usage of learning systems is a large research field and there is a lot of scattered work on this issue. In our work we assume that a Design Pattern approach is a way to collect and to share experiences, to have a meta reflection and to capitalize on context specific research results. The first set of Design Patterns was suggested by Alexander (Alexander et al., 1977) in the architecture domain. Alexander’s definition of a Pattern is still a reference:
158
Delozanne, Le Calvez, Merceron, and Labat Each pattern describes a problem which occurs over and over again in our environment and then describes the core of the solution of that problem, in such a way that you can use this solution a million times over, without ever doing the same way twice (1977, p. x).
In Alexander’s perspective a network of related patterns creates a language to be used by every one involved in a design process whether one designs a house for oneself, or works with others to design offices or public spaces. In the last three decades, the Pattern approach has found its way into many disciplines such as architecture, software engineering, human computer interaction, website design, and e-learning (Alexander et al., 1977; Buschmann, Meunier, Rohnert, Sommerlad, & Stal 1996; Schmidt, Stal, Rohnert, & Buschmann, 2000; Avgeriou, Papasalouros, Retalis, & Skordalakis, 2003; van Duyne, Landay, & Hong, 2003; Chung, Hong, Lin, Prabaker, Landay & Liu 2004; van Welie, 2004; Deng, Kemp, & Todd, 2005). In the e-learning research field, collections of pedagogical patterns are now available. The Pedagogical Pattern Project (PPP) (PPP, n.d.) provides three collections of patterns for educational scenarios. They aim to capture experts’ practice, in that case, experienced teachers. These high-level patterns use an Alexandrian format and are narrative expressed in a you-form to address academic teachers’ or industry instructors’ problems. One collection of fourteen Patterns is presented as a step towards a pattern language for computer science course development: teaching from different perspectives, active learning, feedback patterns, patterns for experiential learning and patterns for gaining different perspectives. A second set of forty-eight patterns forms a pattern language to teach seminars effectively. A last set of five patterns about running a course is suggested. PPP patterns do not address explicitly the use of technology. The E-LEN project (Avgeriou et al., 2003; Avgeriou P., Vogiatzis D., Tzanavari A., Retalis S., 2004; Goodyear et al., 2004; E-LEN, n.d.) provides a booklet with guidelines to develop Design Patterns for e-learning. It also provides a repository of forty patterns classified into four different special interest groups (SIG): Learning resources and learning management systems (LMS), lifelong learning, collaborative learning and adaptive learning. They aim to construct a knowledge base for educational designers and they promote a Design Pattern approach to collect and disseminate re-usable design knowledge, methods of sharing design experience and supporting the work of multidisciplinary teams. Their DP users are e-learning designers. Their DPs are high-level patterns. For instance, the DP student tracking suggests the functionalities to be implemented in the system but it is not clear how to implement these functionalities. It is the designer’s responsibility to generate a solution adapted to his/her context and eventually, to create a more spe-
A Structured Set of Design Patterns for Learners’ Assessment
159
cialized Design Pattern if he/she detects invariant in his/her solution. “A pattern suggests rather than prescribes a solution” (E-LEN, n.d.). The “Design Patterns for Recording and Analysing Usage of Learning Systems” (Choquet, 2004) (DPULS ) project is part of the Kaleidoscope European Network of Excellence. Its objective was to come up with a set of Design Patterns (DPs) that allows the tracking of actors’ activity. As in the E-LEN project, the goal is to support good design decision-making but also to share experiences and to build a common language between a European set of research teams. The DPULS DPs focus on data collection and analysis to investigate more deeply the student tracking problem and to complement PPP and E-LEN projects. In this article we present the subset of the DPULS DPs that deals with tracking students' know-how and strategies while they solve problems in specific domains such as programming or mathematics. This subset is called Learners' Assessment DPs (LA DPs for short). We first specify our research goals and methodology and compare them to related works. After introducing three different successful practices that grounded our DP design, we present the structured set of DPs on Learners' Assessment. We end with a discussion comparing our approach to others and draw some perspectives. Research Goals and Methodology In communities that have adopted the Design Pattern approach, there is a large agreement on the definition of a pattern as a solution to a recurrent problem in a context (Goodyear et al., 2004). However many issues are debated such as for example: • Who are DP users? DPs are written to support experts, researchers, members of a community of practice, multidisciplinary teams, end-users of the designed product or even a machine to implement the solution. • Why create a DP set? The purposes can vary, for example, to create a common language in a multidisciplinary or multicultural design or research team, to capture/disseminate expertise, to capitalize on previous experiences, to ensure quality, or to teach. • What is the format to express DPs? Many efforts are devoted to define DP form (narrative or UML or XML schemes), structure, naming and referencing. • How to create a DP? The bottom-up approach is the most common approach but some authors suggest a top-down approach or a “bottomup approach inform by theory” (E-LEN, n.d.). • How to create a DP language? Structure, granularity, DP combination, and specificity are key points to be discussed. • How to validate or evaluate a DP language? Several ways of validation are presented in the literature: the “rule of three” (three examples of
160
Delozanne, Le Calvez, Merceron, and Labat
successful experiences using the pattern), peer review, feedback from practitioners and usability testing. Design Patterns are usually drafted, shared, criticized and refined through an extended process of collaboration. To this end, a five-step methodology was adopted in DPULS. First, we studied a number of practices in our context of interest (DPULS, 2005, 2-3). Second, from these practices, we selected a set of common and recurrent problems and we set up a know-how list (DPULS, 2005, 4). Third, we worked on descriptions of problem statements and solutions in a way general enough to cover each individual experience, and it was a major task. From this step we negotiated a common format to express the set of DPs (DPULS, 2005, 5) and to design a browser to support the navigation in this set of DPs (DPULS, 2005, 7). Fourth, at the same time we worked to reach an agreement on whether and how the different problems were linked. Many iterations were necessary between the third and the fourth step. Fifth, we came up with a set of DPs stable enough to account for other similar practices (DPULS, 2005, 6) and we entered the patterns in the browser. Every step involved interactions between partners. The purpose of the DP set presented here was triple: • To express invariance in solutions experimented by DPULS teams to solve assessment problems, • To account for others’ experiences on the same class of problems using our Pattern language, • To support designers who deal with a learner’s assessment in individual learning systems. Here is a scenario that illustrates the sort of problem practitioners might face when designing a system for understanding a student's actual learning activity. Sarah is a teacher who organizes lab work for a group of students. She wants a report on the lab work session to check whether some knowhow has been mastered and to get an insight on the kind of strategies the students have used. Her aim is to plan the next session and to adapt her teaching to her students' understanding. She needs an overview on the students' activity during the lab session.
How should the system be designed to make this scenario possible? Our DPs yield a three step approach: • First step: to collect and analyze information on each student for each exercise; • Second step: to build a higher level view of one student's activity on a set of exercises;
A Structured Set of Design Patterns for Learners’ Assessment
161
• Third step: to build an overview of the whole class activity. In e-learning systems, there are different assessment approaches. Our approach goes beyond most current assessment practices where assessment allocates grades and is made through quizzes, multiple choice questions or numerical answers, e.g. “management of on-line questionnaire” (Avgeriou et al., 2003). We also aim to assess students' productions when they are asked to perform tasks specially designed to make the taught knowledge personally meaningful. We deal with problems where students have to perform complex cognitive operations. This type of assessment requires more than a right/wrong assessment. It requires a multidimensional analysis based on accurate pedagogical, didactical or cognitive studies of the student’s activity. In our scenario, this point can be illustrated by: Sarah is an experienced teacher. She is aware of students' personal interpretations and, from her experience or from cognitive research results, she derived a typology of students' answers to a class of problem. She wants to group her students using this typology.
The DPULS set of DPs captures experiences from a multidisciplinary and multicultural team of researchers2. It was built with a bottom-up approach to capture the participants’ expertise and to integrate European research works. In this article we present DPs in a narrative format for human readability. A machine readable version exists that is processed by a DP Browser (DPULS, 2005, 7). They are high level patterns validated by peer review and by at least the “rule of three”. Background Experiences What characterizes a set of DPs is not the originality of its content – usually most of the ideas are known since they are proven solutions to wellidentified problems. Rather, the merit of a set of DPs is to bring together and to structure existing best practices. In our work, we first generalized from six very different e-learning assessment experiences in the AIDA team3. Six projects deal with students' assessments: Combien? (Combien, 2000), Diane (Hakem, Sander, & Labat, 2005), Java Course (Duval, Merceron, Scholl, & Wargon, 2005), Logic-ITA (Merceron, & Yacef, 2004), Pépite (Pepite, 2000), Math Logs (Vandebrouck, Cazes, Gueudet, & Hersant, 2005). In this section, we present three of these experiences so that readers can give a concrete content to the DPs presented in the next section. We selected them to show a large range of assessment contexts. Combien? is a learning environment to train undergraduate students in combinatorics. Pépite is a diagnosis system that analyzes students' productions to diagnose their algebraic competence in secondary school algebra. Logic-ITA is a web based tutoring system for training second year university students in propositional Logic.
162
Delozanne, Le Calvez, Merceron, and Labat
Experience 1: The Combien? Project (Combien, 2000) Combien? trains students to solve combinatorics exercises and to justify their solution. In combinatorics (counting the number of ways of arranging given objects in a prescribed way), the main part of the solving process does not come from a clever chaining of inferences or calculations, but from the elaboration of a suitable representation and from the transformation of one representation into an equivalent one. This work is based on research that emphasizes the importance of representations in problem solving. Combien? is grounded in the constructive method (Le Calvez, Giroire, Duma, Tisseau, & Urtasun, 2003), a problem solving method set up by a group of teachers, adapted to the usual students' conceptions in order to give them access to the mathematical theory of the domain. Students are asked to build a generic element (called configuration) of the Solution Set by describing their construction as a set and a set of constraints. Then, they have to reason about this construction to find the numerical solution. Combien? offers one interface for each class of exercises. At each step of the construction the system automatically determines whether the students’ ongoing solution leads to a right construction or not. In the latter case it gives hints to help students to understand their errors. Each year, one hundred second-year university students use the system in lab sessions. Students can also work with Combien? at home for personal training. All the students’ actions (data input and validation) and their time stamping are recorded so that the session can be played again by the system. Combien? detects students’ errors, classifies them, and records their type and their characterization. All this information is recorded in XML files analyzed a posteriori. Combien? offers two types of analysis. For each student, Combien? presents a detailed account of her session. For each exercise, it reports the total solving duration, the number of errors and their type, the number of hesitations, the exercise achievement, and the number of course consultations. This analysis is available for both students and teachers. In addition, Combien? produces a classification of the exercises according to their level of difficulty and also groups students according to the type of errors they made, and their success rates to the exercises. Teachers use these classifications to define learning activities adapted to each group. Experience 2: The Pépite Project (Pepite, 2000) Pépite is an application that collects students’ answers to a set of exercises and builds a cognitive profile of their competence in algebra. It is based on an educational research that identified learning levers and obstacles in students’ algebra learning (Delozanne, Prévit, Grugeon, & Jacoboni 2003; Delozanne, Vincent, Grugeon, Gélis, Rogalski, & Coulange, 2005). A teacher gives a test to her students and Pépite provides her with three outcomes: first an overview on the whole class by grouping her students
A Structured Set of Design Patterns for Learners’ Assessment
163
Figure 1. One problem-solving step with Combien? according to identified strong and weak points, second with a detailed report of each student’s cognitive profile and third with a list of learning activities tailored to each group. Four hundred students took the Pépite test. Several classes of users used the Pépite diagnosis. Math teachers used Pépite to monitor learning activities in the classroom but also as a basis for a dialog to give a personalized feedback to a single student and as an entry to a meta-reflection on her algebraic competence. Educational researchers used Pépite to identify stereotypes of students and define appropriate teaching strategies to each stereotype. Designers used these experiments to improve the software design: collecting more students’ answers helps to strengthen the automatic diagnosis; analysing teachers’ uses helps to better understand teachers’ needs and to offer a better support for teachers’ activity. Pépite automatically collects students’ answers to the exercises. Answers are expressed by algebraic expressions, by a whole algebraic reasoning, by using students’ own words, by multiple choices or by clickable areas. The students’ assessment is a three-step process. First, each student’s answer is
164
Delozanne, Le Calvez, Merceron, and Labat
coded according to a set of 36 criteria on 6 dimensions (Figure 2): treatments (correct, incorrect, partially correct, not attempted, not coded), meaning of letters (unknown, variable, generalized number, abbreviation or label), algebraic calculation (e.g., correct usage of parenthesis, incorrect usage of parenthesis, incorrect identification of + or x, etc.), conversion (ability to switch between various representations: graphical, geometrical, algebraic, natural language), type of justifications (“proof” by example, proof by algebra, proof by explanation, “proof” by incorrect rule), numerical calculation. First, the Pepite software automatically codes 80% of the answers4. For each student, an XML file stores the students’ answers and the system coding for each exercise. A graphical interface enables the teacher to check or correct the system coding. Second, a detailed report of the student’s cognitive profile is built by collecting the same criteria across the different exercises to have a higher-level view on the student’s activity. It is expressed by success rates on three dimensions (usage of algebra, translation from one representation to another, algebraic calculation) and by the student’s strong points and weak points on these three dimensions. Third, the student’s profile is used to evaluate a level of competence in each dimension with the objective to situate the student in a group of students with “equivalent” cognitive profile. By equivalent we mean that they will benefit from the same
Figure 2. Pépite automatic coding of a student’s answer on six dimensions
A Structured Set of Design Patterns for Learners’ Assessment
165
learning activities. For instance, a student is “Usage of algebra level 1, Translation level 2, algebraic calculation level 2” when she used algebra to justify, to generalize and to formulate equations, she sometimes articulated relations between variables with algebraic expressions and linked algebraic expressions to another representation, she showed abilities in algebraic calculation in simple and well known situations but she still uses some incorrect rules.
Experience 3 The Logic-ITA The Logic-ITA (Merceron, & Yacef, 2004) is a web-based tutoring system to practice formal proofs in propositional logic. It is based on the Cognitive Load Theory – the practice of many exercises should help students to build solving problems schemata, see (Sweller, van Merrienboer, & Paas, 1998). The Logic-ITA has been used by hundreds of students from Sydney University in their second year of studies. It is offered as an extra resource to a face-to-face course. The system has a database of exercises, an exercise generator, and students can also enter their own exercises. A formal proof exercise consists of a set of formulas called premises and a special formula called the conclusion. Solving an exercise is a step-by-step process where students have to derive new formulas, using premises or already derived formulas and applying logic rules to them until they reach the conclusion. The system integrates a module with expertise on logic. It automatically evaluates each step of the student’s solution. In particular it checks that the logic rule chosen by the student is applicable and that the result of its application does match the formula entered by the student. The system provides the student with a contextualized feedback and gives her a hint in case of mistake. Students practice as they wish, training is not assessed nor marked. There is neither a fixed number nor a fixed set of exercises made by all students. For each exercise attempted by a student, the system records in a database the
Figure 3. Screenshot of the Logic-ITA while a student is about to reach the conclusion
166
Delozanne, Le Calvez, Merceron, and Labat
time and date, all the mistakes made, all the logic rules that were correctly used, the number of steps entered and whether the student has successfully or not completed the exercise. The database can be queried and mined to find pedagogically relevant information. A Structured Set of DP In this section, we present a subset of DPULS DPs on Learner’s Assessment we derived from our experience. The DPULS format was defined to build automatic browsing and searching tools. We present the DPs in a simplified format to ease the communication. Each DP has a name that describes both a problem and a solution (Alexander et al., 1977; Meszaros, & Doble, 1998). Combining all the DP names forms a pattern language; you can use the DP names to describe your design to solve problems in your context. In the Learner's Assessment category we structured our set of DPs in five hierarchies. In each hierarchy, the DP at the top deals with a problem and all the DPs below are alternative patterns to solve the same problem with different or complementary approaches. Bottom level patterns are more specialized than upper DPs. Our five hierarchies are not stand-alone; patterns can be combined to solve a design problem. Let us consider Sarah’s scenario. To provide Sarah with an overview on students' activity during the lab session, Aminata, a member of the LMS design team starts with the DP “overview of the activity of a group of learners on a set of exercises,” (DP LA4, top of the fourth hierarchy). She reads the DP solution description and finds that there are several kinds of overview (lower DPs in the fourth hierarchy) according to the user's objective when asking for an overview. She decides to get more information about the users' needs. Then, in the related patterns section, she notices that this DP has some pre-requisites. This DP needs the results of the DP “overview of a learner's activity across a set of exercises,” (DP LA2, top of the second hierarchy). Indeed the group evaluation is based on each individual’s evaluation in an individual learning system. Likewise, there are several alternative or complementary overviews on a student's activity (lower DPs in the second hierarchy) and this DP uses the results of the DP “analysis of a learner's solution on a single exercise” (DP LA1, top of the first hierarchy). This DP gives several solutions to derive information from the students' logs.
Figure 4 shows the subset of DPULS DPs focusing on Learners’ Assessment. In the following section, we detail only the patterns mentioned in the above scenario to help Aminata solve her problem. In each pattern we illustrate the results with the three AIDA experiences: Combien?, Pépite, Logic-ITA.
A Structured Set of Design Patterns for Learners’ Assessment
167
DP LA1 Multidimensional Analysis of a Learner’s Solution to a single Exercise DP LA1.1 Pattern matching to analyze the learner’s solution DP LA1.2 Specific software to analyze the learner’s solution DP LA1.3 Human assessor to check the automatic analysis of the learner’s solution DP LA2 Overview on a learner’s activity across a set of exercises DP LA2.1 The Learner’s strong and weak points DP LA 2.2 The Learner’s situation on a predefined scale of skills or competence DP LA2.3 The Learner’s Progression in an Individual Learning Activity DP LA2.4 The Learner’s Autonomy in an Individual Learning Activity DP LA2.5 The Learner’s Performance in an Individual Learning Activity DP LA3 Overview of the activity of a group of learners on a single exercise DP LA4 Overview of the activity of a group of learners on a set of exercises DP LA4.1 Automatic clustering DP LA4.2 Relations between errors, success or usage DP LA4.2.1 Association rules DP LA5 Playing around with learning resources DP LA5.1 Browsing Use of a MCQ
Figure 4. The structured set of the Learners' Assessment DPs (in bold the subset discussed here)
DP LA1 Multidimensional Analysis of a Learner’s Solution to a Single Exercise Abstract: This pattern provides several approaches to automatically assess a learner’s solution to an online solved problem. You can merely assess the correctness of the solution or enrich the assessment by other dimensions such as strategy used, abilities, hesitations, categorization of errors etc. Context: The learner answered a single question or solved a single exercise. The answer was recorded as well as usage data (time spent, actions, help requests, etc.). The local analysis of the learner’s answer can be immediate (e.g., if your system provides feedback) or delayed (e.g., in a diagnosis system). Problem: How to automatically assess a learner's solution to a problem or one step of a problem solution? Or if it is not possible, how to support human assessment? The problem is how can the system analyze, correct, comment on or classify the learner’s answer? If your system asks learners to solve complex problems, your system will let them build their own solution. In that case it is often impossible to predict the exact form of the learner’s answer because of the excessive combination of possibilities or because of learners’ cognitive diversity. Solution: If your objective is to assess the correctness of a learner’s answer then you can provide an indicator like a grade or a code for success, failure or partial failure. If your objective is to provide feedback or to have a cognitive diagnosis, you may need a deeper characterization of the learn-
168
Delozanne, Le Calvez, Merceron, and Labat
er’s answer. For instance, you may need information on the strategy used by the learner or on the skills put in evidence by the learner’s solution or on the categories of mistakes made by the learner or on the learner’s hesitations. This multidimensional characterization of the learner’s solution is often domain dependant. The analysis builds a composite indicator – it is often a set of codes identifying the learner’s solving process and a list of errors. Results: If you implement this pattern, you will characterize the learner’s answer with some of the following items: • A grade to a learner's answer to an exercise. The grade can be a mark like A, B, C , D, or a message like correct, partially correct, incorrect; • Cognitive characteristics of the learner’s answer; • A set of codes (identifying the learner’s solving process); • A list of errors. Discussion: You may need an expert (experienced teacher, a cognitive psychologist or an educational researcher) to define a model of competence, a task model and/or a typology of common errors linked to the problem to be solved by the learner. Examples • Pépite automatically computes a code to assess the answers on up to six dimensions: correctness, meaning of letters (unknown, variable, generalized number, abbreviation or label), algebraic calculation (usage of parenthesis, etc.), translation between various representations (graphic, geometric, algebraic, natural language), type of justifications (“proof” by example, proof by algebra, proof by explanation, “proof” by incorrect rule), numerical calculation (Figure 2). For instance an answer qualified as “partially correct, incorrect use of parenthesis and algebraic justification” is coded by “T2 M31 R1.” Similarly “T3 M4 R3” stands for “incorrect, incorrect identification of + and x, justification by numerical example.” • Combien? computes an immediate feedback provided to the student on the correctness or on the types of errors (on average, twenty types of errors for each of the fifteen classes of problems). Combien? also stores usage data: action timestamp, action type (e.g., student’s input, asking for help, asking for a random drawing), action parameters. • The Logic-ITA computes an immediate feedback to the student on the correctness or the error type. It also stores timestamps, a list of the errors made and a list of the rules correctly used. Related Patterns: DP LA1.1, DP LA1.2, DP LA1.3.
A Structured Set of Design Patterns for Learners’ Assessment
169
DP LA1.1 Pattern Matching to Analyze Learner’s Solution Abstract: In some types of problems a pattern matching approach can help you to assess the learner’s solution. Context: It is the same as in LA1. This solution is relevant when an analysis grid is available for this exercise, providing patterns of answers, for instance when expected answers are multiple choices, arithmetic or algebraic expressions. Problem: How to use a pattern matching approach to help you analyze a learner’s solution? Requisite: You need indicators built from a pedagogical or didactical or cognitive analysis. For instance: • A grid of correct answers. When there is a single way to express a solution in the system, an analysis grid gives correct and wrong answers. For Multiple Choice Questions, your system has to be provided with a grid of correct answers. • Pattern of good answers or common learners’ answers. When there are several ways to express a solution, a pattern gives a general model of this solution. Solution: For a class of problems, a pedagogical, cognitive or didactical analysis provides you with a set of criteria to carry out a multidimensional characterization of a pattern of solutions. Thus when you can match the learner’s answer with one pattern of solution, you know how to characterize the solution. Results: See LA1. Discussion: For open questions, it is hard work to provide patterns of solutions nevertheless it is sometimes possible. Example: A very simple example in Pépite is: if E = 6P is a good answer, the system also accepts E = 6 * P, P = E / 6 etc. Thus, E= 6*P is a pattern for a correct answer: every algebraic expression equivalent to E= 6*P is correct and assessed as “correct, translating in algebra, correct use of letters”. But P = 6 E or E + P > 6 are incorrect and assessed as “incorrect, translation by abbreviating, use of letters as labels.” Related Patterns: This pattern is a specialization of DP LA 1. DP LA1.2 Specific Software to Analyze the Learner’s Solution Abstract: Domain specific software can be used to help you analyze a learner’s solution. Context: It is the same as LA1. In some specific domains like programming language, formal logic, mathematics, specific software (e.g., a compiler, a problem solver, an expert system, a Computer Algebra System) assesses the correctness and eventually gives information about errors. Problem: How to use specific software to analyze a learner’s solution? Solution: You can use the specific application to check the accurateness
170
Delozanne, Le Calvez, Merceron, and Labat
of the solution until the solution is totally correct. Some applications provide error messages and thus you can use them to characterize errors. Results: See LA1. Examples: Combien? and Logic-ITA use an expert system: • In Combien? the learner builds a solution to a combinatorics exercise. Each step of the solution is analyzed by the system using a “targeted detection of error” (Giroire, Le Calvez, Tisseau, Duma, Urtasun, 2002). If the learner’s construction cannot lead to a correct solution, Combien? provides hints to help the student achieve a correct solution. • In the Logic-ITA, an exercise is a formal proof. The learner has to derive the conclusion from the premises using logical rules and producing intermediary formulas. The expert system checks whether each intermediary formula entered by the learner is correct and provides appropriate feedback. In case of a mistake, a hint is given to help the learner correct the mistake and enter a correct formula. The mistake is stored by the system. Otherwise the system stores the correct use of the logic rule. Related Patterns: This pattern is a specialization of DP LA 1.
DP LA 1.3 Human Assessor to Check the Automatic Analysis of the Learner’s Solution Abstract: Either the teacher herself assesses the answer, or the teacher completes, or verifies, or corrects the system's assessment of the learner’s answer. Context: It is the same as LA1. This solution is relevant if the learners’ solution is composite, or if learners answered in their own words, or if the diagnosis expertise is not yet formalized, or if it is for teacher’s training purpose. Problem: How to assess the learner’s solution when the automatic diagnosis failed or has a low level of confidence? Solution: Your system provides a human assessor with a list of exercises where the automatic analysis failed or is not 100% reliable. Then it provides an interface to assist the human assessor. Results: See DP LA1. Discussion: If a large number of learners are enrolled in your course or if the teachers are very busy (and it is often the case) this solution is unrealistic because it is time consuming. But it is successful if you need a very accurate assessment of individual learners in a research context or a teacher development context for example. Examples: Pépite does not fully assess the solution when learners use natural language and it provides a software tool in order to allow teachers to correct, verify or complete the automatic diagnosis. Related Patterns: This pattern is a specialization of DP LA1
A Structured Set of Design Patterns for Learners’ Assessment
171
DP LA 2: Overview of a Learner’s Activity Across a Set of Exercises Abstract: This pattern offers several approaches to provide different stakeholders with a general view of a learner’s work across a set of exercises or a whole course. Context: The learner is asked to solve complex problems in an individual learning system or a diagnosis system. In both cases, the system collects the learner’s answers and assesses them. The objectives of this assessment are various, for example to group learners for remediation, to make learners aware of their strong points and weaknesses or to situate themselves on a predefined scale of competence. Problem: The problem is: how can the system give a general view of the learner’s work successes, failures or cognitive profile? Strategic decisionmaking requires a high level description of a learner’s activity. For instance, in order to tailor learning situations, teachers need a synthetic view on the learner's learning activity or an account of the learner’s evolution. A classification of the learner on a predefined scale of competence may be useful to organize working groups or for learners to situate themselves according to expected skills. Thus, you want to define the main features that summarize the learner’s competence. In a diagnosis system, the general view is an instantaneous picture of the learner’s competence, in a learning environment the learner’s evolution over time can be analyzed. Requisite: See DP LA 1 results. Solution: To solve this problem you may collect the learner’s answers to a set of questions, exercises or to a whole course. You must first carry out the analysis of each answer on every exercise. Then, you define the different dimensions you need (with teachers or researchers) and, finally, you build an overview according to your stakeholders’ objectives. For example, one may decide to determine the learner’s strong points and weaknesses, or to situate the learner on a scale of competence, or to report on the evolution of a particular skill during the course. Results: A synthetic analysis of the learner’s competence. Discussion: It is crucial to define the dimensions of the overview and to pay careful attention to your stakeholders’ needs and how they will use the overview. Examples: • Combien? and Logic-ITA summarize items such as the global time spent, the number of exercises attempted, succeeded, failed, the errors made, the rules correctly used etc.; • Pépite builds a cognitive profile of the learner’s competence in algebra (see DP LA 2.1 results). Related Patterns: DP LA1, DP LA2.1
172
Delozanne, Le Calvez, Merceron, and Labat
DP LA. 2.1 The Learner’s Strong and Weak Points Abstract: To highlight strong points and weak points in a learner’s performance on a set of exercises is a frequent way to build an overview of a learner’s work. Context: It is the same as LA2. The learner was asked to solve several exercises involving different skills. Problem: How to define a learner’s strong points and weaknesses? Requisite: See DP LA 1 results. Solution: After assessing each answer to each exercise, you can have a cross analysis on a whole session or on a whole period of time. First, you calculate the success rates for each skill involved in the set of exercises (or in the course). Then, for each skill or error, you calculate the number of occurrences. Sometimes you may also need the context where the skills were obvious or where errors occurred. In that case you need a categorization of the exercises or a categorization of errors. Results: • List of skills and success rates on these skills; • List of errors and number or context of these error occurrences. Discussion: It is very important to have both strong points and weaknesses, and not only the learner’s errors. If you make a report, highlighting strong points encourages the student, and even the teacher. If you want to help the student to progress, teaching strategies may be different according to the learner’s mastered skills. Example: Pépite offers an overview by providing a level on three dimensions (usage of algebra, translation from a representation to another, algebraic calculation) and, for each dimension, it provides strong points (a list of mastered skills with success rates and the context where they become obvious). It also provides weak points (a list of categorized errors with their frequency and contexts of occurrences). Related Patterns: This pattern is a specialization of DP LA2. DP LA. 4 Overview of the Activity of a Group of Learners on a Set of Exercises Abstract: This pattern provides teachers with a general view of a group of learners’ work on a set of exercises. Context: A set of exercises, either linked to a specific course or composing a diagnosis system, has been completed by a group of learners. Problem: How can your system produce a general view of a group of learners' work on a whole set of exercises? Strategic decision-making may require a synthetic view of learners' work on a whole activity. For instance, this view can help to organize work groups in a classroom. It can help teachers to reflect about the quality and adequacy of exercises and course mater-
A Structured Set of Design Patterns for Learners’ Assessment
173
ial for example if it shows that some mistakes co-occur. It provides instructional designers or teachers with information that could help them improve their teaching or their design (See DP MV1). Requisite: See DP LA1 Results. Solution: If your objective is to group learners by abilities with respect to a grid of competence or with respect to learning objectives, then you will produce a map of the class. This map will be a classification of learners into different groups. You may predefine stereotypes. A stereotype is a means to identify groups of students who have the same characteristics. These characteristics can be as simple as a high mark, an average mark and a low mark. Or a stereotype may be more accurate and may describe learners who master skills C1, C2 and C3 and who do not master skills C4 and C5. Then you map each learner in the group she belongs to according to the analysis results you have got for each exercise and the relation you have defined between exercises and skills. You may also let the system find groups for you. This may be done by using an automatic clustering algorithm from the Data Mining field. If your objective is to get an overview of learners’ performance, you can produce statistics or charts that group exercises by chapters or abilities. If your objective is to detect associations such as 'if learners make mistake A, then they will also make mistake B,' or 'if learners fail on exercise E, then they will also fail on exercise F', then you may use an association rule algorithm from the Data Mining field. Results: If you implement this pattern, you will characterize the activity of your group of students with some of the following items: • A grade book for the whole class and statistics on all students’ performance; • A map of the class, grouping learners by abilities; • Links between mistakes that often occur together, between exercises often failed or succeeded together. Discussion: Stereotypes can be very simple (low achieving, regular, high achieving students), multidimensional (ranking students on a multidimensional scale for instance in Second Language) or describing usage (player, systematic learner, butterfly, etc.) Examples: • In Pépite, stereotypes are used to classify learners by group of abilities and to offer a cognitive map of the class, grouping students by stereotypes. • With the Logic-ITA, automatic clustering is used to find out whether failing learners can be split into different groups for better remediation. Related Patterns: This pattern is specialized by DP LA4.1 and DP LA4.2. If your objective is to improve the course, see the DP MV1 hierarchy (DPULS, 2005, 6).
174
Delozanne, Le Calvez, Merceron, and Labat
DP LA 4.1 Automatic clustering Abstract: Use automatic clustering to get an overview of a group of learners’ activity on a set of exercises. Context: It is the same as LA4. When you have a characterization of each learner by a set of attributes like the exercises she passed or failed, the mistakes made, the abilities she masters or lacks, then you can use automatic clustering to split the whole group of learners into different homogeneous groups. Problem: How to get an overview of a group of learners using automatic clustering? You want to organize groups in a classroom. Whether it is better to work with homogeneous or heterogeneous groups is a pedagogical decision. In both cases, you have first to build homogeneous groups; for example, a group should be formed with learners who are similar for some characteristics. However, you do not have any predefined stereotypes, or you want to explore whether some other grouping would be sensible. You may try automatic clustering. Requisite: As for LA4. Solution: For each learner you have some analysis provided by the system on each answer submitted for each exercise. You concatenate these analyses, or you summarize them to obtain a characterization for each learner. For example, each learner can be characterized by the fail or pass obtained on each exercise. Another simple characterization would be the total number of mistakes made on all attempted exercises. From these characteristics, you select the ones that should be used for the automatic clustering algorithm and you run it. The automatic clustering algorithm produces clusters of learners. All learners belonging to one cluster are similar with respect to the characteristics that you have chosen to run the algorithm. Results: Thus you will obtain a map of the class with students grouped in clusters. Discussion: To use automatic clustering, you need some expertise in the Data Mining field. Choosing the right characteristics to run the algorithm and to interpret the resulting clustering is a crucial and difficult point. Example: In the Logic-ITA experience, we have used automatic clustering to explore whether failing learners can be grouped in homogeneous clusters. Failing learners are learners who attempt exercises without completing them successfully. As a characteristic we have used the number of mistakes made. The result was two clusters – learners making many mistakes and learners making few mistakes. In the cluster of learners making many mistakes, one could identify learners with a “guess and try” strategy, using logic rules one after the other till they hit the right rule to solve an exercise. Related Patterns: This pattern is a specialization of DP LA4.
A Structured Set of Design Patterns for Learners’ Assessment
DISCUSSION
175
How to evaluate a set of DPs? This is a hot issue in DP communities (Avgeriou et al., 2004; Buschmann et al., 1996; Chung et al., 2004; Salingaros, 2000; E-LEN project; Todd, Kemp, & Philips, 2004). Most Pattern sets described in the literature are validated through a peer review process. Some authors suggest that a pattern can be accepted by a community if it is used in three experiences other than the one that proposed the pattern (Buschmann et al., 1996). In this section, we discuss how our DP set accounts for solutions adopted in other experiences. We checked the validity of our DPs in a three step process. In a first step, we discussed the patterns within the AIDA team. In a second step, the DPs have been discussed and refined within the DPULS consortium. In a third step, we looked at the AIED’05 workshop on “Usage Analysis” (Choquet, C., Luengo, V. & Yacef, K., 2005) contributions that deal with assessment to investigate whether these experiences match our approach. We worked out the above patterns to generalize some success stories in the AIDA team namely the three systems presented in the second section and three others: Diane, Math Logs and Java Course. Diane is a diagnosis system. It identifies adequate or erroneous strategies, and cognitive mechanisms involved in the solving process of arithmetic problems by children from elementary schools (8-10 years old) (Hakem, Sander, & Labat, 2005). DP LA1 and DP LA2 account for Diane’s approach to Cognitive Diagnosis. Math Logs provides researchers with information about undergraduate students’ performance on mathematical exercises displayed on the Wims platform. The mathematical expressions entered by learners are checked by Computer Algebra Systems (CAS) as MUPAD, PARI, Maxima (accounted by DP LA 1.2). Math Logs displays to teachers average grades, average time spent by type of exercises, and indicators on the evolution of grades over time. It detects students’ usage strategies such as focussing on easier exercises or preferring challenging exercises (DP LA2, LA3 and LA4). The Java Course is an online course for introductory programming in Java. Students write programs and submit them for compilation and execution. Their programming and compiling errors are stored in a Database (DP LA1.2). For each exercise and each student, the system displays whether the exercise has been attempted, passed, the number of mistakes made and provides an overview for a chapter and for the whole set (DP LA2). From the AIED’05 workshop on Usage Analysis of Learning Systems (Choquet et al. 2005), we selected six papers dealing with assessment. Kumar (2005) presents a C++ Programming Tutor. In this system, each exercise involves exactly one learning objective. The system automatically checks whether the student’s answer is correct, partially correct, incorrect, missed or not attempted (DP LA1.1). Then an overview of each student’s work is produced.
176
Delozanne, Le Calvez, Merceron, and Labat
This overview gives for each learning objective the fraction of exercises solved correctly, partially correctly etc. (DP LA2). For a whole class, the average performance for each exercise is computed (DP LA3) and, finally, the average performance of the class on each learning objective is also calculated (DP LA4). Designers use the latter to refine the templates to automatically generate the exercises according to the objectives (DP MV1 “Material Improvement”). Feng and Heffernan (2005) describe Assistment, a math web-based tutor that provides the teacher with a report on the student’s activity and the student with assistance when she is stuck in solving problems. This assistance is given by scaffolding questions that are intended to help the student but also to help the diagnosis system to understand which knowledge components are involved in the failure when the problem involves more than one component (DP LA1). Then the system builds a grade book providing the teacher with each student’s results (DP LA2) and with an overview of the class (DP LA 4). Another overview of the class is given by a Class Summary Report and a Class Progress Report. Then an analysis of items helps determine what the difficult points are for students and improve both teaching and materials (DP MV1 “Material Improvement”). This system design is a very clever example of the best practices we wanted to account for in our DP Set. Nicaud, Chaachoua, Bittar, & Bouhineau, (2005) model the student‘s behavior in algebra calculation in Aplusix, a system to learn elementary algebra. The diagnosis is a two step process. The first phase is a local diagnosis of each student's transformation of an algebraic expression. From a library of correct and incorrect rules, the system determines the sequence of rules used by students during the transformation. The diagnosis system also characterizes the type of exercise and the algebraic context in a “Local Behavior Vector (LDV)” (DP LA1, for one step). The second phase uses a lattice of conceptions to build an overview on a student’s conceptions from the different LDV on a set of algebraic transformations (DP LA2). Aplusix offers to teachers a map of conceptions for the whole class (DP LA4). Heraud, Marty, France, & Carron, (2005), Stefanov & Stefanova (2005), and Muehlenbrock (2005) do not focus on learners’ assessment. They describe approaches to build an overview of a learner from different perspectives. They work with a trace of a student’s actions in an e-learning platform. The trace records success and task completion on exercises, time stamp and course consultation or helps (DP LA 1). Stefanov & Stefanova (2005), use a learning object approach in Virtuoso to provide students, teachers and designers with statistics on students’ weak points and strong points (DPLA 2.1), on success rates or failures to each learning objects (DP LA3) to improve the material (DP hierarchy MV). Muehlenbrock (2005) uses decision trees to classify students in three categories: low, medium or high achieving students (DP LA4). To have an overview on the student’s activity on a session (DP LA2) the system described by Heraud et al. (2005)
A Structured Set of Design Patterns for Learners’ Assessment
177
compares the student’s trace to the prescribed learning scenario. It displays the trace in a band showing the time spent for each task along with the prescribed time. The system shadows the prescribed line to provide what they call “a shadow bar” when there is a discrepancy between the trace and the prescribed time. A human assessor called “trace composer” when aware of this shadow zone in the trace can investigate other sources of information like the student’s logs in the server, the learner’s workstation or human observer. This experience suggests introducing “a human assessor” pattern in DP LA2 hierarchy. It also suggests to refining the hierarchy with usage data while so far we have focused on accurate cognitive information. This review shows that our approach accounts for solutions adopted in many learning systems at least in the domains of Programming and Mathematics Tutors. Further investigation is needed to see whether our DPs are pertinent for other domains, though it seems that it is the case. For instance Language Standardized Tests (e.g., TOEFL, TOEIC, IELTS5, TFI) use an approach similar to our DPs. On the basis of students’ scores, they classify students on a predefined scale of competence in listening, reading, writing and speaking skills. This fits well with the multidimensional analysis of DP LA1. In this section we discussed whether our DP collection on Learner’s assessment matches the state of art in the domain. Our objective was to capitalize on success stories and to help designers build successful e-learning systems. We estimate that we validated the first point but it is premature to validate the second. So far only our students used it. For technical reasons our DP set will be public soon on the Web. In particular it will be used in the Virtual Doctoral School of The Kaleidoscope Network6. This will provide interesting feedback and hopefully the Pattern Language will be enriched. CONCLUSION
In this article, we focussed on DPs dealing with Learners’ Assessment, a subset of the DP elaborated in the DPULS consortium (DPULS, 2005, 6). One outcome of our work is to put in evidence basic principles for designers whose objective is to provide teachers with an overview of the activity of a group of learners on a set of exercises, (DP LA4). A fundamental step is to obtain a multidimensional analysis of a learner's solution on a single exercise, (DP LA1). DP LA1 is specialized by other DPs depending on how the student’s solution is analyzed (pattern matching, use of some specific software, human assessment). It can be used to obtain both an overview of a learner's activity across a set of exercises, DP LA2, and an overview of the activity of a group of learners on a single activity, DP LA3. The two latter DPs are used by DP LA4. In case DP LA1 is applied to exercises that are characterized by precisely defined skills, DP LA4 gives a way to track students' problem-solving abilities.
178
Delozanne, Le Calvez, Merceron, and Labat
These DPs are high level DPs. A future work will consist in writing more specific DPs to describe how to implement the different solutions. To support LA4 we would like to work out patterns on data mining techniques. A complementary approach to build an overview is used by Pépite, Aplusix, Assistment or Second Language assessment. It consists in situating a student in predefined classes based on cognitive or educational research and not only on achievement. Further investigation is needed to study what part of the assessment is domain independent in order to be implemented in e-learning platforms. On the opposite, we think that domain specific patterns would give more accurate guidance to designers. For instance, our DP set could be completed with patterns that collect practices on displaying assessment results to different actors in specific domains. So far this work has demonstrated that DP is a very fruitful approach to generalize from different design experiences and to capitalize on them. It was a powerful integrative project because we had to distill out of our specific projects what was invariant in making good design solutions. “Patterns are very much alive and evolving” (Alexander et al., 1977). We hope that we will have feedback from practitioners to develop and enrich this first DP language on learners’ assessment. References Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I., & Angel, S. (1977). A pattern language: Towns, buildings, construction. New York: Oxford University Press. Avgeriou, P., Papasalouros, A., Retalis, S., & Skordalakis, M., (2003). Towards a pattern language for learning management systems. Educational Technology & Society, 6(2), 11-24. Avgeriou, P., Vogiatzis, D., Tzanavari, A., & Retalis, S., (2004). Towards a pattern language for adaptive web-based educational systems. Advanced Technology for Learning, 1(4), ACTA Press,. Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P. & Stal, M., (1996). Pattern-oriented software architecture. Volume 1: A System of Patterns. John Wiley & Sons. Choquet, C., Luengo, V. & Yacef, K., Eds. (2005). Proceedings of "Usage Analysis in Learning Systems" workshop, held in conjunction with the 12th Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands, Retrieved December 2005 at :http://hcs.science.uva.nl/AIED2005/W1proc.pdf Choquet, C., (2004). JEIRP design patterns for recording and analysing usage of learning systems proposal. Retrieved March 2006 from http://www.noe-kaleidoscope.org/pub/activities/jeirp/activity.php?wp=33 Chung, E.S., Hong, J. I., Lin, J., Prabaker, M.K., Landay, J.A., & Liu, A. (2004). Development and evaluation of emerging design patterns for ubiquitous computing. In Proceedings of Designing Interactive Systems (DIS2004), Boston, MA. pp. 233-242. Combien (2000). The Combien? Project site, http://www.lip6.fr/combien/
A Structured Set of Design Patterns for Learners’ Assessment
179
Delozanne, E., Prévit, D., Grugeon, B., & Jacoboni, P. (2003). Supporting teachers when diagnosing their students in algebra. Workshop Advanced Technologies for Mathematics Education, supplementary Proceedings of Artificial Intelligence in Education, Sydney, July 2003, IOS Press, Amsterdam, 461-470. Delozanne, E., Vincent, C., Grugeon, B., Gélis, J.-M., Rogalski, J., & Coulange, L. (2005). From errors to stereotypes: Different levels of cognitive models in school algebra, Proceedings of E-LEARN05, Vancouver, 24-28 October 2005. Deng, J., Kemp, E., & Todd, E. G., (2005). Managing UI pattern collections. CHINZ’05, Auckland, NZ. DPULS 2-7 (2005). Deliverables. Retrieved January 2006 from http://www.noekaleidoscope.org/intra/activity/deliverables/ deliverable 2: Merceron, A. Report on partners’ experiences. deliverable 3: David, J.P. State of art of tracking and analyzing usage. deliverable 4: Pozzi, F. The set of recurrent problems and description of solutions. deliverable 5: Verdejo, M.F., & Celorrio, C. The design pattern language structure. deliverable 6: Delozanne, E., Labat, J.-M., Le Calvez F., & Merceron, A. The structured set of design patterns. deliverable 7: Iksal, S., Alexieva, A., Beale, R., Byrne, W., Dupont, F., Londsdale, P., & Milen, P. The design pattern browser. Duval, P., Merceron, A., Scholl M., & Wargon, L., (2005). Empowering Learning Objects: an experiment with the Ganesha platform. In Proceedings of the World Conference on Educational Multimedia, Hypermedia and Telecommunications ED-MEDIA 2005, Montreal, Canada, P. Kommers and G. Richards Ed., 2005(1), pp. 4592-4599, AACE Digital Library (http://www.aace.org). E-LEN (n.d.). the E-LEN patterns site, http://www2.tisip.no/E-LEN/patterns_info.php (consulted December 2005). Feng, M., & Heffernan, N, (2005). Informing teachers live about student learning: Reporting in assistment system. Proceedings of "Usage Analysis in Learning Systems" workshop, held in conjunction with the 12th Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands, pp.25-32. Retrieved December 2005 at http://hcs.science.uva.nl/AIED2005/ W1proc.pdf Giroire, H., Le Calvez, F., Tisseau, G., Duma, J., & Urtasun, M., (2002). Targeted Detection: Application to Error Detection in a Pedagogical System, Proceedings of ITS'2002, Biarritz, p.998. Goodyear, P., Avgeriou, P., Baggetun, R., Bartoluzzi, S., Retalis, S., Ronteltap, F., & Rusman, E., (2004). Towards a pattern language for networked learning. Proceedings of Networked Learning 2004, Lancaster UK. Hakem, K., Sander, E., & Labat, J-M., (2005). DIANE, a diagnosis system for arithmetical problem solving. Proceedings of Artificial Intelligence in EDucation 2005, Looi, McCalla, Bredeweg, Breuker eds, IOS Press, Amsterdam, Holland, pp. 258-265. Heraud, J. M., Marty, J. C., France, L., & Carron, T., (2005). Helping the Interpretation of Web Logs: Application to Learning Scenario Improvement, Proceedings of "Usage Analysis in Learning Systems" workshop, held in conjunction with the 12th Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands, pp. 41-48. Retrieved December 2005 at http://hcs.science.uva.nl/AIED2005/W1proc.pdf
180
Delozanne, Le Calvez, Merceron, and Labat
Kumar, A., (2005). Usage Analysis in Tutors for C++ Programming, Proceedings of "Usage Analysis in Learning Systems" workshop, held in conjunction with the 12th Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands, pp.57-64. Retrieved December 2005 at :http://hcs.science.uva.nl/AIED2005/W1proc.pdf Le Calvez, F., Giroire, H., Duma, J., Tisseau, G., & Urtasun, M. (2003). Combien? A software to teach learners how to solve combinatorics exercises. Workshop Advanced Technologies for Mathematics Education, supplementary Proceedings of Artificial Intelligence in Education, Sydney, Australia, IOS Press, Amsterdam, pp. 447-453. Merceron, A., & Yacef, K., (2004). Mining student data captured from a web-based tutoring tool: Initial exploration and results. in Journal of Interactive Learning Research Special Issue on Computational Intelligence in Web-Based Education, 15(4), 319-346. Meszaros, G., & Doble, J., (1998). A pattern language for pattern writing. In Pattern Languages of Program Design 3 (Software Patterns Series), Addison-Wesley, Retrieved December 2005 at http://hillside.net/patterns/writing/patterns.htm Muehlenbrock, M., (2005). Automatic action analysis in an interactive learning environment. Proceedings of "Usage Analysis in Learning Systems" workshop, held in conjunction with the 12th Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands, pp.7380. Retrieved December 2005 at http://hcs.science.uva.nl/AIED2005/W1proc.pdf. Nicaud, J.-F., Chaachoua, H., Bittar, & M., Bouhineau, D. (2005). Student’s modelling with a lattice of conceptions in the domain of linear equations and inequations. Proceedings of "Usage Analysis in Learning Systems" workshop, held in conjunction with the 12th Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands, pp.81-88. Retrieved December 2005 at :http://hcs.science.uva.nl/AIED2005/W1proc.pdf. Pepite (2000). The Pépite Project site, http://pepite.univ-lemans.fr PPP (PPP, n.d.). The Pedagogical Patterns Project site, http://www.pedagogicalpatterns.org/ (consulted December 2005) Salingaros, N., (2000). The structure of pattern languages. Architectural Research Quarterly, 4, 149-161. Schmidt, D., Stal, M., Rohnert, & H., Buschmann, F., (2000). Pattern-oriented software architecture, vol.2 . Patterns for Concurrent and Networked Objects. Wiley. Stefanov K., & Stefanova E., (2005). Analysis of the Usage of the Virtuoso System, Proceedings of "Usage Analysis in Learning Systems" workshop, held in conjunction with the 12th Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands, pp. 97-104. Retrieved December 2005 at :http://hcs.science.uva.nl/AIED2005/W1proc.pdf Sweller, J., van Merrienboer, J. G., & Paas, F. G. W. C., (1998). Cognitive architecture and Instructional Design. Educational Psychology Review, 10(3), 251-295. Todd, E., Kemp, E., & Philips, C., (2004). What makes a good user interface pattern language? The 5th Australasian User Interface Conference, Dunedin, Australian Computer Society, pp. 91-100. Vandebrouck, F., Cazes, C., Gueudet, G., & Hersant, M., (2005). Problem solving and web resources at tertiary level. Proceedings of the 4th Conference of the European society for Research in Mathematics Education, CERME 2005, Barcelone Spain. van Duyne, D. K, Landay, J., & Hong, J., (2003). The design of sites: Patterns, principles and process for crafting a customer centered web experience. Addison-Wesley. van Welie, M.. (2004). Patterns in interaction design. Retrieved March 2006 from http://www.welie.com/index.html.
A Structured Set of Design Patterns for Learners’ Assessment
181
Notes http://www.noe-kaleidoscope.org/ (consulted March 2006) The DPULS consortium involved researchers and practitioners from six European countries and from different domains: educational design, educational research, teachers, engineers, LMS managers and LMS designers. 3 Aida is a consortium of Research Laboratories focusing on e-learning problems in the Paris area. The French Ministry of Research funds it. Aida belonged to the DPULS consortium. 4 It codes every answer expressed by multiple choices and by one algebraic expression, most answers expressed by several algebraic expressions, and some answers in students’ own words. 5 International Language Testing System 6 Kaleidoscope Virtual Doctoral school: http://www.noe-kaleidoscope.org/pub/activities/backbone/activity. php?wp=79 (consulted March 2006) 1 2