Web class lists

Soon after the launch of the web timetables we were asked by the timetabling project team to investigate the possibility of a self-service feature for staff to produce class lists on demand, as central production and distribution of these was a potential bottleneck.

Scientia colleagues directed us towards an SWS report that produced class lists for activities:

/reporting/spreadsheet?objectclass=activity&Template=Class%20List%20Activity%20SS&idtype=name&identifier=

Unfortunately the report presented its data as a series of tables with students sorted by surname in each. This gave us two problems to solve:

  1. How to give staff a usable interface for generating class lists for activities
  2. How to order data returned from the report so that it provided a useful class list

Giving staff an interface for selecting class lists

We had already developed a page that would proxy the SWS programmeOfStudy report to produce either a list or grid view of the timetable for any course. This page included a reference to every activity that had been scheduled, which could be passed to the SWS activity report. However, the timetabling team explained that not all of these would have class lists and we knew staff would be disappointed if some links appeared broken. After discussing the problem with Scientia, their consultants produced a custom programmeOfStudy report template that listed all activities for a course with the number of students allocated to each activity. We knew that every activity with a student allocation greater than zero would have a class list, so we had the data necessary to limit link creation to only those that would return a class list. However, the data needed to be scraped from HTML tables.

We decided that our proxy page would need to make two HTTP GET requests: the first would get the course timetable and inject CSS and some form objects (for varying display range and format); the second would get the activity list for the class and process the HTML to produce an array of activities with class lists. Before displaying the (re-formatted) output from the first course timetable request we could then iterate over the array of activities with class lists replacing the activity name in the HTML with a link to the class list page for that activity. When displayed on the screen staff would then have a class list with clickable links for every activity for which a class list was available.

We encountered problems loading the activity list report as XHTML, so we had to load as a string, extract the text between the body tags and remove some invalid characters and elements before we could process it as XML. However, once loaded, we were able to pick out all the activities that had class lists using XPath as the activity references were in the first column of rows without a @class attribute in tables of @class ‘spreadsheet’ which had a non-zero value in the 8th column!

//table[@class=’spreadsheet’]/tr[not(string(@class)) and not(td[8]=’0′)]/td[1]

The page to which we directed staff then had to solve the second problem of displaying useful class lists.

Providing useful class lists

After considerable experimentation, we found we needed to load the class list report as a string and extract certain parts as XML so that we could apply an XSL Transform to sort the students by surname and produce a report that staff would find useful. We decided to load only the parent table as XML, but before loading we removed invalid characters and blank rows. We were then able to apply an XSL Transform to produce the XHTML class list report sorted by surname (which appeared immediately after a space in the first column of rows that didn’t have a @class attribute):

<xsl:apply-templates select="tr[not(string(@class))]">
<xsl:sort select="substring-after(td[1],' ')"/>
</xsl:apply-templates>

The final part of the solution was to provide staff with a list of course years for which class lists would be available. As class lists are only available where students have personal timetables, we asked for the student’s course code and name to be added to the view that indicated whether a student had a personal timetable. A SELECT DISTINCT on courses where there were students with individual timetables gave us the list we needed, which we included as an RSS feed to a staff-only page on timetabling’s intranet site.

Web timetables

As regular readers will be aware, MMU’s move to Moodle is part of a larger initiative designed to make a step change improvement in the student experience. The initiative, known as EQAL – Enhancing the Quality of Assessment for Learning, coordinates work on curriculum rules, quality processes, learning systems and admin systems, and includes introduction of new institution-wide systems for coursework submission (using a custom development) and timetabling (using Scientia). The need for easy-to-access, up-to-date, definitive timetable information has been a recurrent request in focus group and survey work, and featured strongly in our survey of students’ mobile information needs. This post describes how we used a service-oriented approach to give students access to timetable information.

We had originally planned to use Scientia’s Enterprise Reporting Database component to make timetable information available in Moodle and via students’ mobiles, but in late summer we learned that this could not be deployed for the start of term, so an intense period of re-thinking began.

Scientia includes a web publishing feature, known as SWS – Scientia Web Services, which produces HTML reports of student and course timetables specified via querystring parameters. Whilst incredibly fast, output from SWS lacks semantic mark-up, and makes extensive use of tables to control screen layout. We realised that this would impose limitations on what we could achieve, but we also became aware of two further complications:

  • The scale and complexity of scheduling the entire MMU timetable meant that for performance reasons the scheduling problem was split into three geographically-distinct timetables: Crewe, Didbsury/Gaskell and all other Manchester campuses
  • Each separate timetable had an SWS report server that ran on a non-standard port that wasn’t available over our guest wireless network (which we expected a lot of students to use)

We therefore needed to find a method of publishing web timetables that would work over wifi and on and off campus, and would hide from our students the unfortunate complexities created by having 3 separate timetable systems.

We were also aware that some but not all students would have personal timetables, and the set of students with personal timetables would change as they were allocated to particular seminars and taught groups. The aim of the EQAL project for September 2011 was to ensure that all level 3 and 4 students received personal timetables. However, some parts of the institution had committed to supplying the data necessary to give students on levels 5 and 6 personal timetables too. Our web timetable architecture therefore needed to ensure that every student saw either a personal timetable (if one were available) or their course timetable, and that this information stayed up to date.

We worked closely with colleagues on the timetabling project and with Scientia consultants to develop an architecture that ensured each student could access the best available timetable for them. A lookup table was created that indicated for any given student ID their campus (so we knew which SWS server to generate a report from), their course, and whether they had an individual timetable. Queries of this lookup table were wrapped as a simple RSS web service that returned for any given user ID, a customised URL for displaying their web timetable.

The customised URL included the port number of the SWS server that held the student’s timetable and specified whether the course or personal timetable report should be produced. The page then acted as a port 80 proxy to the SWS server so that the output could be displayed off campus or over wifi. The page also injected some CSS and some form elements to improve readability and make it easy to change the date and day range displayed.

Our architecture for displaying web timetables is shown below:

WebTimetables SWS Architecture
WebTimetables SWS Architecture

We added a custom WebPart to our myMMU SharePoint portal that called the timetable RSS web service for the authenticated user and set the href property of the timetable icon to take the student to the appropriate personal or course timetable for them.

Our Moodle mega mashup: part 2

In a previous post we described the background and rationale for creating a mega mashup web service that would aggregate relevant information at unit or programme level for the authenticated user and present it in a custom Moodle block.

For reasons of synergy with the skills of our developers, we chose to develop the mega mashup as a Microsoft Windows Communication Foundation (WCF) web service. WCF allows a service contract to be specified in which a template is provided for the web-service call and a class is specified for the data returned. We developed two main web services: one returning System.ServiceModel.Syndication.Atom10FeedFormatter output and the other returning System.IO.Stream output. The latter is produced by a function that applies a stylesheet transform to the output of the function that produces the former. The template for the two web-service calls is specified as returning WebMessageFormat.Xml for a UriTemplate of

getInfo?user={user}&courseid={courseid}&area={area}&userRole={userRole}&userMailDomain={userMailDomain}&format={format}&developer={developer}&dtm={dtm}&token={token}

Where
user = authenticated moodle user
courseid = moodle course id
area = moodle course short code (in most cases will be a Unit or Programme identifier)
userRole = role of the current authenticated moodle user
userMailDomain = mail domain from the profile of the authenticated moodle user
format = determines whether Atom or an XHTML fragment is returned
developer = identifier to track the originator of calls to the web service
dtm = date-time stamp
token = security token

Our web service uses these parameters to form useful URLs for and retrieve data from a number of systems, grouping the responses using the source property of the Atom item into the following categories:

  • Moodle News – Link to Moodle Announcements for the course
  • Moodle Events – Link to Moodle Calendar for the course
  • Live Mail – SSO link to Live@Edu for students with an @stu.mmu.ac.uk email domain
  • Guidance – SSO link to coursework receipting system guidance
  • Hand-ins – Feed from and SSO links to coursework receipting system
  • Moodle Assessments – Link to Moodle assessments for the course
  • Guidance – SSO link to coursework receipting system guidance
  • To Buy – Items tagged as ‘to buy’ on the associated TalisAspire reading list
  • Essential – Items tagged as ‘essential’ on the associated TalisAspire reading list
  • Further – Items tagged as ‘further’ on the associated TalisAspire reading list
  • Podcasts – RSS feed of a search of podcasts tagged with the course code on Podcast Producer server
  • Files – SSO links to items returned in a search of EQUELLA for files tagged with course code
  • Exam Papers – SSO links to past exam papers returned in a search of EQUELLA for files tagged with course code
  • Timetable – Link to Scientia timetable that can be displayed for the current user

Classifying the source of each item in the Atom feed with a value from one of these categories allows the items to be grouped in the transform to XHTML, eg:

<li>
Coursework (<xsl:value-of select="count(//atom:entry[contains(atom:source/atom:title,'Hand-ins')])"/>)
<ul>
<xsl:for-each select="//atom:entry[contains(atom:source/atom:title,'Hand-ins')]" >
<xsl:sort select="atom:updated" order="ascending"/>
<xsl:call-template name="showAtomEntryForAssignment" />
</xsl:for-each>
</ul>
</li>

Our custom Moodle block then displays the XHTML fragment returned by the mega-mashup web service:

Moodle right-hand block
Moodle right-hand block

Over the summer, in the build up to going live with 40,000 users, we embarked on a comprehensive testing regime, stress-testing the web service using concurrency simulations from http://loadimpact.com/. We found our web services performed better on Windows 2003 than Windows 2008, so built our farm using Windows 2003. We also discovered that our Apple podcast server introduced unacceptable latency under extreme loads, so we chose to remove this from the mega mashup web service, as this is called on every visit to a Moodle course page, and decided instead to offer podcasts in a custom block for those Moodle areas that needed it. To speed up editing of Moodle pages, we also decided to disable calls to the mega mashup web service when the page was in Edit Mode, and instead just show a place-holder for the mashed-up content.

As load started to pick up for real in the weeks before the start of term we found that the mega mashup web service was backing up waiting for the Moodle web services to return detailed information about Moodle news, events and assignments. We therefore decided to remove the calls to the Moodle web services from the mega mashup web service, and just return links to the relevant Moodle pages for the current course id. This improved performance, but we still found performance problems under extreme load. We traced the problem to a wait cycle in which the server running the web services that provided users with single-sign-on from SharePoint into Moodle was backing up. As this server was also running the web services for the right hand block, Moodle was waiting to render pages, which meant resources weren’t being freed to service the single-sign-on requests. The wait cycle built up until performance ground to a halt. Once the cause had been identified, the single-sign-on and right-hand-block web services were moved onto different servers, the vicious wait cycle was broken, and performance returned to normal.

Now, even though it is aggregating latest data from Talis Aspire, EQUELLA and our coursework submission server on every view of a Moodle course page, our mega mashup web service is meeting performance targets and providing the information our staff and students have been asking for. We are extremely grateful to our partners at ULCC for helping us to deliver our vision of an integrated Core+ VLE, particularly as we have just heard that our Moodle server has been accessed by 35,588 distinct users who have performed over 5 million activities so far, with a peak concurrency of 1,477 users in a 5 minute period – not bad considering WebCT Vista was only switched off as MMU’s institutional VLE on August 31!