Evaluation and reflection

The W2C project set out in July 2010 to demonstrate the potential of Widgets, Web-services and Cloud-sourced services for delivering an institutional vision of convenient, integrated and extensible learning systems. We hoped this work would:

  1. enhance the capability of our VLE to meet existing and emerging user requirements
  2. enhance user satisfaction of our learning systems
  3. heighten awareness about patterns of student engagement and success
  4. facilitate discovery of learning resources that others have found useful
  5. accelerate development of a community of practice for educational widgets
  6. improve, by providing practice-informed feedback, key sources of information available to others in the sector contemplating similar endeavours

After 18 months of intensive work it is important to assess our progress against these ambitious aims and to reflect on new directions and lessons that have emerged from the activity.

Target #1: to enhance the capability of our VLE to meet existing and emerging user requirements

MMU’s move to Moodle was shaped by extensive focus group work on students’ expectations of a good online learning environment. The importance of a one-stop-shop for administrative information came through as a critical hygiene factor (a requirement that has subsequently been reinforced by our research into students’ mobile needs). We consequently put a lot of effort into aggregating frequently-requested administrative information such as module timetables, assessment deadlines and reading lists, and devised a mechanism to publish a personalised, contextualised feed of this information in Moodle (see our mega-mashup post). This institution-wide innovation was launched on September 1, 2011 when Moodle became our institutional VLE. Hit data for the service powering the mega-mashup demonstrates that this service has quickly become a key part of institutional infrastructure!

  • Between 01/09/11 to 11/12/11, our main web-services server has responded to 11,841,793 requests
  • That’s an average of 116,096 hits a day
  • or 4,837 per hour!
  • Our getInfo (mega mashup) service accounts for 3,138,539 of those hits
  • Our getTimetables service accounts for 2,556,593 of those hits

The graph below provides a breakdown of the most popular web service calls in the first 100 days.

Web service hits in first 100 days
Web service hits in first 100 days

Total web services hits - breakdown by hour
Total web services hits - breakdown by hour
Total web services hits - breakdown by day
Total web services hits - breakdown by day

Target #2: to enhance user satisfaction of our learning systems

For 5 years, MMU has been testing and refining an online system for gathering student satisfaction. Integrated with the student records system, the system asks each student a set of standard questions about their course and modules. In the 2010/11 academic year, the system was piloted in three faculties (Humanities, Law and Social Science; Business School and Science and Engineering). For 2011/12 it has been deployed university-wide. Questions have remained constant across the two years and two closed questions are of interest for assessing the extent to which Moodle is delivering frequently-requested administrative information:

  • The course is well organised and running smoothly
  • University resources are appropriate to my learning needs

Response rates on this year’s survey have been staggering. Over 10,000 students have submitted feedback and have provided over 56,000 comments.

For the pilot faculties we have data before and after the Moodle launch for these two questions and have been able to break down interim responses by year of course to test for any improvements.

N Organisation Resources
2010/11 Year 1 1262 3.67 3.73
2010/11 Year 2 654 3.66 3.61
2010/11 Year 3 402 3.64 3.52
2010/11 Year 4 72 3.78 3.57
2011/12 Year 1 3003 3.75 3.87
2011/12 Year 2 1542 3.69 3.73
2011/12 Year 3 975 3.73 3.65
2011/12 Year 4 185 3.82 3.66
%increase Year 1 2.2% 3.8%
%increase Year 2 0.8% 3.3%
%increase Year 3 2.5% 3.7%
%increase Year 4 1.1% 2.5%

Clearly our work with Moodle has been part of the much larger EQAL change initiative, so we cannot claim these improvements flow directly from the W2C work, but it is encouraging to see satisfaction levels rise in the two key areas we’ve focused on.

Target #3: to heighten awareness about patterns of student engagement and success

During the course of W2C, MMU launched a major project to transform its annual monitoring arrangements into a Continuous Monitoring and Improvement (CMI) system so our work around student engagement and success was directed towards supporting this. The CMI system will receive feeds from a number of sources, including Moodle tracking data and the online student satisfaction survey mentioned earlier. Consistent use of curriculum codes and person identifiers has been a critical part of W2C’s work for the Moodle deployment. It has enabled our mega-mashup, and will be playing a key role in joining data about engagement, satisfaction and success.

Since its launch in September 2011, Tutors have been using Moodle’s native last-login feature to raise awareness of students who haven’t been engaging with a particular module. However, programme leaders told us they were struggling to assemble a composite picture of their students’ engagement across all Moodle areas. We discussed the problem with our partners at ULCC and decided to commission a custom Engagement block that reports each student’s total hits for the last four weeks and allows the list to be filtered by groups set up in the course where the block is deployed. This would allow a programme leader to see engagement broken down by year of study (as our sync logic automatically enrols students on Moodle programme areas into groups for particular years of study).

The volume of activity on MMU’s moodle server (5 million in the first month, 6.5 million in the second) led to a number of iterations of the tool to establish a satisfactory response time, but we are now in the final stages of testing and lessons learned will doubtless be of use to ULCC’s other clients.

Target 4: to facilitate discovery of learning resources that others have found useful

As part of MMU’s “core+” VLE we have joined Moodle with EQUELLA and Talis Aspire and have been working closely with tutors to increase the number and structure of useful resources being signposted from Moodle. Resource lists are now mandatory when modules are approved, and MMU has approved a resource list policy that requires tutors to be clear about which items are recommended for purchase, which are essential and which are further reading. Take up of Aspire has been given a big boost as all first year modules were re-written for MMU’s EQAL initiative, and colleagues are now re-writing second and final year modules. To ensure that students’ experience of resources is as seamless as possible we have worked with colleagues at Talis Aspire to ensure single-sign-on access from Moodle via Aspire to resources stored in EQUELLA.

The next stage in our Talis Aspire project will be to exploit the potential of the shared service platform to explore recommendation services: eg, tutors like you also recommended this resource. We hope that our consistent use of curriculum codes will make it easy to add information, such as module JACS code and FHEQ level, to improve the accuracy of this recommendation service and we have already had some interesting dicussions with the Talis developers about this. As part of the wider Continuous Monitoring and Improvement (CIM) project we are looking to make resource usage data available to tutors when they plan their resource list for the coming academic year.

Target #5: to accelerate development of a community of practice for educational widgets

Although we have contributed to the widget development community and documented lessons learned, particularly about Wookie and our rapid development of a location-aware widget at the CETIS widget bash, our Distributed VLE model has prioritised cloud and web services for institution-wide deployment. We see this very much as laying the foundations for deployment via a variety of channels. Moodle, SharePoint and the CampusM mobile app have been our priority channels, but we may extend these channels to include widget deployment in future.

Target #6: to improve, by providing practice-informed feedback, key sources of information available to others in the sector contemplating similar endeavours

JISC funding for this work has provided us with the extra bit of space not only to innovate, but to share lessons learned. We have had great follow-up conversations about our institution-wide service-oriented “core+” VLE approach with colleagues from Bradford, City, Coventry, Nottingham, Oxford Brookes, Plymouth, Queen Mary University of London, Royal Holloway, Southampton, Staffs, University of the Arts London, Wolverhampton – and sincere apologies to anyone we’ve missed off the list! We really hope this blog and our presentations, such as recent ones to the Neil Stewart Associates Student Experience event and the UCISA Future is Mobile event, have been useful.

Reflection

W2C has been an ambitious project, within an ambitious change programme. We have demonstrated that value can be added to a core Virtual Learning Environment through web services. By building strong strategic partnerships, we have been able to deliver a “core+” VLE through shared services for Moodle, EQELLA and TalisAspire and have blended this core with content from corporate systems. Through consistent tagging we have been able to aggregate relevant content around the user, and have proved that the service-oriented approach can work at institutional scale. We hope our Distributed VLE architecture and this account of its evolution will be of interest to others.

Our DVLE architecture
Our DVLE architecture

Sharing at the UCISA Future is Mobile event

To support the communications strategy for our W2C project we agreed to present with our Moodle partner, ULCC, at UCISA’s Future is Mobile event. After my 6am start, you’ll see from the tweets I’ve repeated below that the day didn’t go exactly to plan!

I wasn’t too lucky with breakfast:

  • And the prize for being up early for #UCISAmob … the dust + crumbs marking the end of the cereal #notmuchchewonatuesday

An unscientific device survey on the train revealed slightly surprising results:

  • From my train seat the device scores on the doors are: iPhone=3 blackberry=2 windows=2

Then the trouble started: a train in front brought down the overhead power lines leaving a number of twitter colleagues wondering what time we might arrive in Birmingham:

  • think @leoappleton ‘s train broke it @markpower is behind him. I’m behind Mark + @DMStephenson is behind me. It’s like an 11+ qu
  • We’re gonna need a bigger boat yfrog.com/hwlcfdqj

We were eventually put on a coach heading back North, sharing data about mobile developments and hoping we might catch up with others

  • The northern breakaway contingent of #UCISAmob (me + @markpower) is heading for Stoke: is the kettle on @FleurP ?

I arrived back in Manchester just in time to record a voice-over for the slides that I’d intended to present, which I uploaded to YouTube for my co-presenter:

With a stroke of genius for a “Future of Mobile” event, Frank then played the audio back via his iPhone to accompany the slides, which went down well with the audience!

  • Wow…not only the future is mobile…the now is too…a mobile device is presenting #ucisamob
  • Presenter didnt get to event due to issues, recorded sound file & sent 2 colleague, now playing back via an iPhone & mic #genius #UCISAmob
  • A presentation by phone. THAT’S mobile. #ucisamob
  • @thestubbs talking about researching mobile needs – via @fstoner ‘s iPhone 🙂 mobile technology at work at #UCISAmob !
  • @thestubbs Listening to you speak on @fstoner ‘s iPhone at #ucisamob via live stream. #awesomenessindeed
  • round of applause for @fstoner for toughing it out and holding his iPhone up to the microphone for so long #ucisamob
  • @fstoner presenter stuck on a train? There’s an App for that! #UCISAmob

In the course of the day I’d used my mobile to:

  1. check the agenda, delegate list and location of the event
  2. listen to music
  3. get news about the train failure from the train operator and friends on other trains
  4. keep up remotely with the conference I’d been planning to attend (I narrowly missed the chance to vote while switching from train to coach)
  5. discuss and arrange how I was going to present remotely

And, of course, Frank used his mobile to present my voice from YouTube!

On reflection, these are pretty similar to the categories we’d asked student to use to categorise their use of mobile to support their studies:

  • Checking deadlines/timetables: 66%
  • Discussing/arranging work: 66%
  • Listening to music/blocking noise: 38%
  • Accessing learning materials: 35%
  • Looking up references: 28%
  • Taking notes: 28%
  • Producing coursework: 8%
  • Voting/interacting in class: 7%
  • Commenting on learning: 6%

And, as I write this, the W2C blog and the video made in lieu of my presence are still being retweeted – and to think earlier in the day I’d been using the hashtag #UCISAstationary instead of #UCISAmob !

Mobile device ownership

Between October and December 2010, W2C ran an online survey via the MMU student portal asking students to help the university shape its mobile development priorities by answering questions about devices owned, usage of those devices and priorities for mobile content. High level summary stats from that survey were published as an RSS feed on the right hand side of this blog and survey data informed our interpretation of 100 device-led interviews we carried out through W2C.

Recently, to support some equal opportunities work, we were able to match survey responses against age range and gender categories to give us greater insight into variety in need across our diverse student community. Our sample is inevitably biased towards users of technology as it required students to follow a link from the student portal, however for a subset of our sample (964 of 982) we have been able to see how device ownership varies with gender and broad age ranges.

N % of sample Android Blackberry iPhone iPod Touch
F.<20 230 24% 7% 26% 9% 34%
M.<20 178 18% 11% 18% 22% 35%
F.20-29 246 26% 6% 18% 16% 22%
M.20-29 174 18% 11% 18% 26% 34%
F.30-39 41 4% 2% 15% 20% 22%
M.30-39 40 4% 13% 10% 13% 20%
F.40+ 28 3% 4% 11% 11% 14%
M.40+ 26 3% 4% 4% 23% 8%
2010 mobile device ownership by age range and gender
2010 mobile device ownership by age range and gender

Interpretation of this data must be done with caution. The sampling strategy meant that respondents were not representative of the institution as a whole. Addition of percentages implied by the graph is misleading as 164 students had an iPod Touch and a smartphone. Nonetheless, the graph draws attention to some interesting phenomena:

  • Blackberry was particularly popular with females under 20
  • iPhone was particularly popular with males over 40
  • Android was a lot more popular amongst 20-40 year males than females

We hope to repeat this kind of analysis with subsequent surveys as it provides a useful reminder of diversity in device ownership and hints at the importance of cost and social norms in device ownership decisions.