Evaluation and reflection

The W2C project set out in July 2010 to demonstrate the potential of Widgets, Web-services and Cloud-sourced services for delivering an institutional vision of convenient, integrated and extensible learning systems. We hoped this work would:

  1. enhance the capability of our VLE to meet existing and emerging user requirements
  2. enhance user satisfaction of our learning systems
  3. heighten awareness about patterns of student engagement and success
  4. facilitate discovery of learning resources that others have found useful
  5. accelerate development of a community of practice for educational widgets
  6. improve, by providing practice-informed feedback, key sources of information available to others in the sector contemplating similar endeavours

After 18 months of intensive work it is important to assess our progress against these ambitious aims and to reflect on new directions and lessons that have emerged from the activity.

Target #1: to enhance the capability of our VLE to meet existing and emerging user requirements

MMU’s move to Moodle was shaped by extensive focus group work on students’ expectations of a good online learning environment. The importance of a one-stop-shop for administrative information came through as a critical hygiene factor (a requirement that has subsequently been reinforced by our research into students’ mobile needs). We consequently put a lot of effort into aggregating frequently-requested administrative information such as module timetables, assessment deadlines and reading lists, and devised a mechanism to publish a personalised, contextualised feed of this information in Moodle (see our mega-mashup post). This institution-wide innovation was launched on September 1, 2011 when Moodle became our institutional VLE. Hit data for the service powering the mega-mashup demonstrates that this service has quickly become a key part of institutional infrastructure!

  • Between 01/09/11 to 11/12/11, our main web-services server has responded to 11,841,793 requests
  • That’s an average of 116,096 hits a day
  • or 4,837 per hour!
  • Our getInfo (mega mashup) service accounts for 3,138,539 of those hits
  • Our getTimetables service accounts for 2,556,593 of those hits

The graph below provides a breakdown of the most popular web service calls in the first 100 days.

Web service hits in first 100 days
Web service hits in first 100 days

Total web services hits - breakdown by hour
Total web services hits - breakdown by hour
Total web services hits - breakdown by day
Total web services hits - breakdown by day

Target #2: to enhance user satisfaction of our learning systems

For 5 years, MMU has been testing and refining an online system for gathering student satisfaction. Integrated with the student records system, the system asks each student a set of standard questions about their course and modules. In the 2010/11 academic year, the system was piloted in three faculties (Humanities, Law and Social Science; Business School and Science and Engineering). For 2011/12 it has been deployed university-wide. Questions have remained constant across the two years and two closed questions are of interest for assessing the extent to which Moodle is delivering frequently-requested administrative information:

  • The course is well organised and running smoothly
  • University resources are appropriate to my learning needs

Response rates on this year’s survey have been staggering. Over 10,000 students have submitted feedback and have provided over 56,000 comments.

For the pilot faculties we have data before and after the Moodle launch for these two questions and have been able to break down interim responses by year of course to test for any improvements.

N Organisation Resources
2010/11 Year 1 1262 3.67 3.73
2010/11 Year 2 654 3.66 3.61
2010/11 Year 3 402 3.64 3.52
2010/11 Year 4 72 3.78 3.57
2011/12 Year 1 3003 3.75 3.87
2011/12 Year 2 1542 3.69 3.73
2011/12 Year 3 975 3.73 3.65
2011/12 Year 4 185 3.82 3.66
%increase Year 1 2.2% 3.8%
%increase Year 2 0.8% 3.3%
%increase Year 3 2.5% 3.7%
%increase Year 4 1.1% 2.5%

Clearly our work with Moodle has been part of the much larger EQAL change initiative, so we cannot claim these improvements flow directly from the W2C work, but it is encouraging to see satisfaction levels rise in the two key areas we’ve focused on.

Target #3: to heighten awareness about patterns of student engagement and success

During the course of W2C, MMU launched a major project to transform its annual monitoring arrangements into a Continuous Monitoring and Improvement (CMI) system so our work around student engagement and success was directed towards supporting this. The CMI system will receive feeds from a number of sources, including Moodle tracking data and the online student satisfaction survey mentioned earlier. Consistent use of curriculum codes and person identifiers has been a critical part of W2C’s work for the Moodle deployment. It has enabled our mega-mashup, and will be playing a key role in joining data about engagement, satisfaction and success.

Since its launch in September 2011, Tutors have been using Moodle’s native last-login feature to raise awareness of students who haven’t been engaging with a particular module. However, programme leaders told us they were struggling to assemble a composite picture of their students’ engagement across all Moodle areas. We discussed the problem with our partners at ULCC and decided to commission a custom Engagement block that reports each student’s total hits for the last four weeks and allows the list to be filtered by groups set up in the course where the block is deployed. This would allow a programme leader to see engagement broken down by year of study (as our sync logic automatically enrols students on Moodle programme areas into groups for particular years of study).

The volume of activity on MMU’s moodle server (5 million in the first month, 6.5 million in the second) led to a number of iterations of the tool to establish a satisfactory response time, but we are now in the final stages of testing and lessons learned will doubtless be of use to ULCC’s other clients.

Target 4: to facilitate discovery of learning resources that others have found useful

As part of MMU’s “core+” VLE we have joined Moodle with EQUELLA and Talis Aspire and have been working closely with tutors to increase the number and structure of useful resources being signposted from Moodle. Resource lists are now mandatory when modules are approved, and MMU has approved a resource list policy that requires tutors to be clear about which items are recommended for purchase, which are essential and which are further reading. Take up of Aspire has been given a big boost as all first year modules were re-written for MMU’s EQAL initiative, and colleagues are now re-writing second and final year modules. To ensure that students’ experience of resources is as seamless as possible we have worked with colleagues at Talis Aspire to ensure single-sign-on access from Moodle via Aspire to resources stored in EQUELLA.

The next stage in our Talis Aspire project will be to exploit the potential of the shared service platform to explore recommendation services: eg, tutors like you also recommended this resource. We hope that our consistent use of curriculum codes will make it easy to add information, such as module JACS code and FHEQ level, to improve the accuracy of this recommendation service and we have already had some interesting dicussions with the Talis developers about this. As part of the wider Continuous Monitoring and Improvement (CIM) project we are looking to make resource usage data available to tutors when they plan their resource list for the coming academic year.

Target #5: to accelerate development of a community of practice for educational widgets

Although we have contributed to the widget development community and documented lessons learned, particularly about Wookie and our rapid development of a location-aware widget at the CETIS widget bash, our Distributed VLE model has prioritised cloud and web services for institution-wide deployment. We see this very much as laying the foundations for deployment via a variety of channels. Moodle, SharePoint and the CampusM mobile app have been our priority channels, but we may extend these channels to include widget deployment in future.

Target #6: to improve, by providing practice-informed feedback, key sources of information available to others in the sector contemplating similar endeavours

JISC funding for this work has provided us with the extra bit of space not only to innovate, but to share lessons learned. We have had great follow-up conversations about our institution-wide service-oriented “core+” VLE approach with colleagues from Bradford, City, Coventry, Nottingham, Oxford Brookes, Plymouth, Queen Mary University of London, Royal Holloway, Southampton, Staffs, University of the Arts London, Wolverhampton – and sincere apologies to anyone we’ve missed off the list! We really hope this blog and our presentations, such as recent ones to the Neil Stewart Associates Student Experience event and the UCISA Future is Mobile event, have been useful.


W2C has been an ambitious project, within an ambitious change programme. We have demonstrated that value can be added to a core Virtual Learning Environment through web services. By building strong strategic partnerships, we have been able to deliver a “core+” VLE through shared services for Moodle, EQELLA and TalisAspire and have blended this core with content from corporate systems. Through consistent tagging we have been able to aggregate relevant content around the user, and have proved that the service-oriented approach can work at institutional scale. We hope our Distributed VLE architecture and this account of its evolution will be of interest to others.

Our DVLE architecture
Our DVLE architecture

One thought on “Evaluation and reflection”

  1. Well done everyone! Think this project is a real benchmarked for mobile deployment based on real user needs and sustainability through key linkages to wider institutional processes. Can’t wait to see what you do next, and of course your take on the CETIS DLE models;-)

Leave a Reply