Our Moodle mega mashup: part 2

In a previous post we described the background and rationale for creating a mega mashup web service that would aggregate relevant information at unit or programme level for the authenticated user and present it in a custom Moodle block.

For reasons of synergy with the skills of our developers, we chose to develop the mega mashup as a Microsoft Windows Communication Foundation (WCF) web service. WCF allows a service contract to be specified in which a template is provided for the web-service call and a class is specified for the data returned. We developed two main web services: one returning System.ServiceModel.Syndication.Atom10FeedFormatter output and the other returning System.IO.Stream output. The latter is produced by a function that applies a stylesheet transform to the output of the function that produces the former. The template for the two web-service calls is specified as returning WebMessageFormat.Xml for a UriTemplate of

getInfo?user={user}&courseid={courseid}&area={area}&userRole={userRole}&userMailDomain={userMailDomain}&format={format}&developer={developer}&dtm={dtm}&token={token}

Where
user = authenticated moodle user
courseid = moodle course id
area = moodle course short code (in most cases will be a Unit or Programme identifier)
userRole = role of the current authenticated moodle user
userMailDomain = mail domain from the profile of the authenticated moodle user
format = determines whether Atom or an XHTML fragment is returned
developer = identifier to track the originator of calls to the web service
dtm = date-time stamp
token = security token

Our web service uses these parameters to form useful URLs for and retrieve data from a number of systems, grouping the responses using the source property of the Atom item into the following categories:

  • Moodle News – Link to Moodle Announcements for the course
  • Moodle Events – Link to Moodle Calendar for the course
  • Live Mail – SSO link to Live@Edu for students with an @stu.mmu.ac.uk email domain
  • Guidance – SSO link to coursework receipting system guidance
  • Hand-ins – Feed from and SSO links to coursework receipting system
  • Moodle Assessments – Link to Moodle assessments for the course
  • Guidance – SSO link to coursework receipting system guidance
  • To Buy – Items tagged as ‘to buy’ on the associated TalisAspire reading list
  • Essential – Items tagged as ‘essential’ on the associated TalisAspire reading list
  • Further – Items tagged as ‘further’ on the associated TalisAspire reading list
  • Podcasts – RSS feed of a search of podcasts tagged with the course code on Podcast Producer server
  • Files – SSO links to items returned in a search of EQUELLA for files tagged with course code
  • Exam Papers – SSO links to past exam papers returned in a search of EQUELLA for files tagged with course code
  • Timetable – Link to Scientia timetable that can be displayed for the current user

Classifying the source of each item in the Atom feed with a value from one of these categories allows the items to be grouped in the transform to XHTML, eg:

<li>
Coursework (<xsl:value-of select="count(//atom:entry[contains(atom:source/atom:title,'Hand-ins')])"/>)
<ul>
<xsl:for-each select="//atom:entry[contains(atom:source/atom:title,'Hand-ins')]" >
<xsl:sort select="atom:updated" order="ascending"/>
<xsl:call-template name="showAtomEntryForAssignment" />
</xsl:for-each>
</ul>
</li>

Our custom Moodle block then displays the XHTML fragment returned by the mega-mashup web service:

Moodle right-hand block

Moodle right-hand block

Over the summer, in the build up to going live with 40,000 users, we embarked on a comprehensive testing regime, stress-testing the web service using concurrency simulations from http://loadimpact.com/. We found our web services performed better on Windows 2003 than Windows 2008, so built our farm using Windows 2003. We also discovered that our Apple podcast server introduced unacceptable latency under extreme loads, so we chose to remove this from the mega mashup web service, as this is called on every visit to a Moodle course page, and decided instead to offer podcasts in a custom block for those Moodle areas that needed it. To speed up editing of Moodle pages, we also decided to disable calls to the mega mashup web service when the page was in Edit Mode, and instead just show a place-holder for the mashed-up content.

As load started to pick up for real in the weeks before the start of term we found that the mega mashup web service was backing up waiting for the Moodle web services to return detailed information about Moodle news, events and assignments. We therefore decided to remove the calls to the Moodle web services from the mega mashup web service, and just return links to the relevant Moodle pages for the current course id. This improved performance, but we still found performance problems under extreme load. We traced the problem to a wait cycle in which the server running the web services that provided users with single-sign-on from SharePoint into Moodle was backing up. As this server was also running the web services for the right hand block, Moodle was waiting to render pages, which meant resources weren’t being freed to service the single-sign-on requests. The wait cycle built up until performance ground to a halt. Once the cause had been identified, the single-sign-on and right-hand-block web services were moved onto different servers, the vicious wait cycle was broken, and performance returned to normal.

Now, even though it is aggregating latest data from Talis Aspire, EQUELLA and our coursework submission server on every view of a Moodle course page, our mega mashup web service is meeting performance targets and providing the information our staff and students have been asking for. We are extremely grateful to our partners at ULCC for helping us to deliver our vision of an integrated Core+ VLE, particularly as we have just heard that our Moodle server has been accessed by 35,588 distinct users who have performed over 5 million activities so far, with a peak concurrency of 1,477 users in a 5 minute period – not bad considering WebCT Vista was only switched off as MMU’s institutional VLE on August 31!

This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Our Moodle mega mashup: part 2

  1. Sheila MacNeill says:

    Hi Mark

    Another informative post – really interesting to find out the exact causes of the “vicious wait cycle” (like that phrase!). Glad to hear that everything working now and that it’s standing up to some heavy usage.

    Sheila

  2. Pingback: University-wide coursework submission system | JISC Transforming Assessment and Feedback For Institutional Change

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>