JISC Transforming Assessment and Feedback For Institutional Change

One of the mini-projects which we’re running as part of the TRAFFIC project is one to look at providing some kind of standard template for assignment briefs.

Right away I have to admit that this isn’t the right terminology to use, as the words ‘standard’ and ‘template’ seem to have the effect of lighting the blue touchpaper and launching us into all kinds of debate about academic freedom. An alternative expression has been suggested: ‘communication of assessment arrangements’, but this seems to propel the discussion in the direction of ‘admin tail wagging academic dogs’. Really, what we want to do is to find a consistent way of capturing what students are expected to do for their assignments but I have a feeling that the expression ‘Assignment Capture’ might send us down another distracting dead end. Unless anyone thinks that might catch on?

Anyhow, in the spirit of capturing what goes on already and of collecting examples of good practice to share, part of this mini-project involves interviewing staff and students about their experiences of working with assignment briefs. Much of this work is being undertaken by Iqra Ali, an MMU graduate with recent experience of this herself. Any MMU people who’d like to contribute, please contact me directly or leave a comment below and we’ll get in touch. We’d also be interested in looking at examples from other institutions, fellow project teams…

At the moment we are just hoping to develop the rough template we have up already in our Assessment web pages and then find a way of incorporating this into our nascent programme/unit database. This is a tall order even if we manage to agree a consistent structure for the brief, but it is a strong aspiration.

Just returned from an intensive 3 day workshop in Oslo with European colleagues looking at electronic standards for the transfer of assessment results. The working group, which included representatives from Norway, Italy, Greece, Poland and the UK, reviewed transcript approaches used in different member states and identified the need for a core electronic standard that would facilitate learner mobility. The group had previously contributed to a set of European Norms and CEN Workshop Agreements (CWA) that laid foundations for such a core, particularly European Learner Mobility Achievement Information (EN 15981), Metadata for Learning Opportunities (EN 15982) and an Educational Credit Information Model (CWA 16077). However, software vendors had identified challenges implementing these specifications for transcripts, particularly the lack of a unifying specification from which code could be auto-generated and practical implementation guidelines. The workshop set itself the ambitious aim of addressing this barrier to software development.

Outputs from the “ELMO” workshop are available on a public wiki, and developers may be particularly interested in the XML Schema Document developed in the workshop:
http://code.google.com/p/elmo-schemas/source/browse/#svn%2Ftrunk

The XML Schema Document produced was simplified to remove namespace references to the various specifications on which it drew in order to maximise auto code-generation across a range of development environments and reduce confusion for developers. During the workshop auto code-generation was tested with a variety of development environments and all manner of quirks were identified and addressed. For instance, various combinations of xs:include and xs:import were tried before a solution was found that worked in all available development environments.

More work is now required on testing the ELMO XSD with a wider range of development environments and real transcript data, but hopefully the Oslo workshop will have helped to reduce barriers for developers working on electronic transcripts of learner achievement, and provides a concrete illustration of the structure of an electronic Diploma Supplement.

Next week I’ll be representing the TRAFFIC project at a meeting in Oslo to devise actions for promoting a standards-based approach to the handling of assessment results.

The meeting follows up work started by the Metadata for Learning Opportunities (MLO) and European Learner Mobility Achievement Information (EuroLMAI) initiatives to establish common semantics for the representation of courses of study and students’ achievements on those courses. Agreement on semantics is an important first step for developing standard mechanisms for information exchange, but further work is required on the practicalities, for instance on application profiles, XML bindings and reference web services that demonstrate the standards in action. The initiative to link the MLO and EuroLMAI initiatives has been accepted as a New Work Item (NWI) by the CEN Workshop for Learning Technology (CEN WS-LT), who have agreed to work with vendor representatives from the RS3G (Rome Student Systems and Standards Group) to facilitate implementation and adoption. The Oslo meeting will bring together those who worked on the MLO and EuroLMAI initiatives with vendor representatives and system implementers.

To prepare for the meeting, I have collated high-level use-cases for web services that have emerged in TRAFFIC so far. I have deliberately omitted those associated with QTI results handling as these are being articulated more fully elsewhere (see for instance Southampton University’s work on QTI Delivery Integration).

• getAssessmentDeadlinesForStudent – often filtered by module, this service provides calendar-like information about the due-dates for scheduled assessment submissions and events for a particular student. These due-dates may be adjusted if a student has had an approved extension. Assessment details typically include an identifier, title and may include a personalised URL for online submission or printing a personalised cover-sheet for a paper submission. The exchange should be sufficiently secure to prevent another learner from following the URL. After work has been submitted, the service may return updated information acknowledging successful submission.
• getAssessmentDeadlinesForStaff – often filtered by module, this service provides calendar-like information about due-dates for scheduled assessment submissions and events for which a staff member is responsible. Assessment details typically include an identifier, title and may include summary information about the number of submissions and a personalised URL for viewing further information about the submissions. The exchange should be sufficiently secure to ensure the privacy and accuracy of the transaction.
• getAssessmentDeadlinesByFilter – various use-cases have emerged for presenting assessment deadlines for a particular module (so that a module learner can confirm their accuracy or they can be used to help prospective learners make an informed choice about studying an module) and for a particular campus or across the whole university (to appreciate when submission peaks might create challenges for infrastructure such as drop-in PCs, printing and submission facilities). Filtering could be required on various criteria: eg by week, by owning department, by campus, etc.
• getAssessmentMarksForStudent – often filtered by module, this service provides a user with marks for specific elements of assessment. The status of marks returned may require some explanation to the user. For instance, the service could be used to provide a student with a statement of the marks that will be submitted about their work for consideration by an examination board. In such a case, these marks could differ from those appearing on a transcript as they could be modified through examination board moderation or the application of penalties for work submitted after its due-date. The service needs to be sufficiently secure to ensure accuracy, authenticity and privacy of a student’s results.
• getAssessmentMarksForModule – this service provides an export of marks from one system for processing by another. For instance, it could be provided by a VLE and called to upload marks to a Student Records System. The service needs to be sufficiently secure to ensure the accuracy, authenticity and privacy of marks retrieved. Data returned typically includes an identifier for the piece of curriculum (module) on which the assessment was undertaken, an identifier for the element of assessment within the module, an identifier for the student who engaged in the assessment and an outcome (mark or grade). Additional information may identify whether this is a first or subsequent attempt at the assessment element. Attention is drawn to work done with the IMS Enterprise specification for bulk transfer of marks between different corporate systems
• recordAssessmentSubmission – this service typically records submission of a piece of assessment. Data passed to the service includes a curriculum (module) identifier where the assessment took place, an identifier for the element of assessment, an identifier for the student who submitted the work and a date-time stamp for when the submission took place. Calls should be authentic, accurate and private.
• getTranscriptForPerson – typically filtered for achievements within a particular date range and offered by a particular provider, this service summarizes final assessment outcomes sometimes with sufficient contextual information so that those other than the learner can interpret the outcomes. Considerable work has been undertaken in this area and attention is drawn for instance to CETIS work on the technical architecture for the UK Higher Education Achievement Report

Further use-cases for services emerged in the recent demonstration of Exeter’s e-Assessment extensions for Moodle, for instance a service to return a list of valid Assessment Element identifiers for a given module so that a common identifier could be established between Moodle and the Student Records System, which would facilitate subsequent transfer of marks (via a variant of the getAssessmentMarksForModule web-service). This list is not offered as exhaustive, merely as a starting point for specifying a set of re-usable services for electronic assessment handling that will aid scoping and definition work in Oslo next week.

To assist with visualising service usage models for the assessment-related services described above, screenshots from systems currently in use to support the TRAFFIC project are included:

Personalised assessment submission and marks information in Moodle

Screenshot example of a personalised assessment feed

Assessment submission information for a particular week for a particular campus

Screenshot example of an assessment submission schedule