Our Moodle mega mashup: part 1

In our learning technologies review we developed a vision of a core VLE integrated with our corporate systems, ideally available to students to use with the devices and services of their choosing, and extended through tools that the institution arranges, recommends or recognises. Following extensive options appraisal we decided that our core VLE would comprise Moodle, Equella and Talis Aspire, and developed our “core plus” model to manage expectations about funding and support for learnng technologies within MMU:

Core Plus VLE Model

This post focuses in particular on how we deliver our core VLE and integrate it with corporate systems. First we set out threshold standards for information that should be available within the VLE, then we describe our web service architecture for delivering that information.

Threshold standards for Core VLE information

Our learning technologies review was informed by focus groups and survey work undertaken during three year action-research fieldwork for a PhD. Findings from focus groups and surveys with students presented clear requirements for definitive administrative information – calls that have been echoed in W2C’s recent mobile survey. In Summer 2010, as part of a broader initiative on establishing threshold standards for the student learning experience at MMU, it was agreed that areas should be created in Moodle for every Unit (module) and Programme a student studies and every Unit should normally display:

  • Unit Title, Code and Teaching Team
  • Recent Unit news and events
  • Assignment Hand-in Deadlines
  • Reading List (ideally organized in terms of items to buy, essential and further reading)
  • Links to online learning materials: digitized chapters, podcasts, etc
  • Personal Unit Timetable for the user for the next month
  • A week-by-week teaching schedule

Gathering threshold VLE information

We decided that we could bring this information together with:

  • a consistent tagging convention, based on Unit and Programme codes from the student records system; and
  • a web services mashup

Within MMU, Units on which students enrol are identified using a combination of three identifiers:

  • Unit Specification Code
  • Academic Year
  • Unit Occurrence Code

All users, whether staff or students, are enrolled within Moodle using their unique Active Directory identifier.

Combinations of these four identifiers give us search terms for querying MMU’s corporate systems to gather our threshold standard Unit information.

Threshold From User ID Unit Spec Acad Year Unit Occ Note
Unit Title and Code Student Records X X X Cached in VLE
Teaching Team Student Records Extension X X X Synched to VLE
News + Events VLE X X X X
Assignment Hand-ins Coursework Receipting System X X X X
Reading List Talis Aspire X X
Learning Materials Files uploaded to Moodle X X X X Useful to flag recently added
Learning Materials Equella Learning Resources Collection X X
Learning Materials Equella CLA Collection X X
Learning Materials Equella Exam Papers Collection X
Learning Materials Apple Episode Producer Podcasts X X
Personal Timetable Scientia X X X X

Publishing threshold standard information

Through focus groups with staff we devised principles for a standard three column layout for every Moodle area:

  • Left = Navigation
  • Centre = Title, teaching team contacts + weekly content or topics
  • Right = Remaining threshold information

In consultation with our hosting partner, ULCC, we decided that Title and Teaching Team contacts could best be delivered by customization of the Moodle 1.9 course/view.php page, so we contracted ULCC developers to make the necessary changes to our Moodle site. For the remaining threshold information for display in the right hand column we decided, after some deliberation about widgets, that a custom Moodle block, which presented the output from a web-service, would be our preferred solution. We decided on a custom Moodle block, rather than a widget within a block, for the simplicity of entirely server-side presentation of content, avoiding IFRAMES, and the ease of applying UI preferences (such as font size increases) to all content on the page.

To power our custom Moodle block, we designed a mega mashup web service that could return either an ATOM feed or the ATOM feed styled with an XSL Transform to produce a valid XHTML page (for testing) or a valid XHTML fragment for display within the Moodle block. We chose ATOM, rather than RSS, as our syndication feed format for the increased sophistication offered on the SOURCE for each syndication item. We decided that we would use the SOURCE element to categorise items for display, and intend (eventually) to have valid SOURCE URLs to allow fragments of the feed to be accessed, rather than consuming the aggregated feed in its entirety.

We will describe the mega-mashup web service in more detail in a follow-up post

Working with partners: Talis Aspire and Equella

A distinguishing feature of our distributed VLE is integration of a range of complementary learning technologies: Moodle, Equella, Talis Aspire, campusM, etc, based on:

  • web services; and
  • consistent naming of data within different systems.

In consultation with library colleagues we devised a workflow for including digitized materials (stored in Equella), in reading lists, held in Talis Aspire and publicised in Moodle.
Digitizated Materials Workflow
As part of new Unit (module) approval procedures introduced to support MMU’s EQAL curriculum transformation initiative, tutors are required to provide reading lists that distinguish items to buy, essential and further reading. Where essential or further reading can be delivered digitally, tutors are encouraged to use electronic sources and, within the terms of the institution’s Copyright License Agreement, tutors and library staff have been identifying chapters and articles for digitization. Library colleagues are agreeing a format for the “notes for librarians” field that will enable clear digitization instructions to be captured against item entries on Unit reading lists. All Talis Aspire lists are reviewed prior to publication. If digitization requests are encountered then the chapter or article will be scanned, uploaded to Equella, tagged with the Unit code and, finally, the Talis Aspire list item will be updated with the Equella URL. To make the outcome of this workflow as easy as possible for students, we wanted the Talis Aspire link presented in Moodle to be single-sign-on.

In an earlier post, we described a Talis Aspire integration scenario that went beyond the sample code provided for Talis Aspire. We wanted to supply a Unit (module) code and an academic year identifier and retrieve titles and URLs for the items on the reading list for that unit and display the content in Moodle. We prototyped a solution for this which initially parsed the RDF for the unit code to identify the URI for the relevant list and then parsed its XHTML representation. As prototyping began to move towards production deployment we started looking for performance improvements in our code.

On Tuesday, May 24, colleagues from Talis Aspire were on site for a project catch-up. We agreed a new naming convention for identifying versions of lists for particular academic years, provided data for a new Unit hiearchy that reflected this convention, and raised our two technical challenges with Chris and Ian:

  • providing single-sign-on links to Equella resources on Talis Aspire lists when those lists are displayed in Moodle
  • speeding up access to list item titles and URLs for a given Unit code and academic year

Chris demonstrated a new feature of Talis Aspire, which allowed javascript widgets to be added to Talis Aspire pages which could interact with the page’s content or extract data from its calling URL.

After 35 minutes of agile development, Chris had produced and deployed to our Talis Aspire tenancy a widget that could retrieve a single-sign-on token from the querystring of a list item page, and append this to the reading list item URL used to access material in Equella; Alex had modified the web service used to publish data in Moodle from Talis Aspire and Equella so that links to Talis Aspire items were appended with a short-lived token that would grant single-sign-on access to Equella.

Steve, Chris + Alex in agile dev mode

In just 35 minutes, powered by enthusiasm, coffee and a large tub of chocolate, we had achieved our aim of seamless access from Moodle via Talis Aspire to digitized journals and chapters stored in Equella!

Chris also mentioned that Talis Aspire supported CSV representations of lists, as well as the XHTML representations we had been parsing in our web service. We found the CSV representation gave a noticeable performance improvement of over 200ms per call, and have incorporated this change into the production codebase.

Our distributed VLE depends on integrating solutions from a number of different partners, and the kind of working relationship described in this post is a key enabler for delivering our DVLE vision.

Responses from 100 device-led student interviews

Between January and March 2011, one hundred students were interviewed about their use of technology and places of study at Manchester Metropolitan University (MMU). The interviews took place at locations across MMU and, as far as possible, students were selected in order to give a course and gender balance that reflected MMU as a whole. Students were asked to talk about the technology that they had with them. Those who took part were rewarded with a £5 print credit voucher.

Detailed analysis of the results and reflection on questions will follow, but this post presents preliminary analysis of headline results for students’ use of technology in their studies. Please note accuracy of data is yet to be validated by the research team, and use of specialist equipment, for instance Apple Mac computers available to Design students, is yet to be factored in. These results are presented as a preliminary indication only and should not be considered safe for citation.

A link to the interview schedule is available here.

Of the 100 students interviewed, 98 had brought a mobile phone with them and 45 had brought a laptop (or netbook). Questions concerned use of the devices *for* learning and study.

Frequency of Technologies Used In Learning 20110610

Used in learning Mobile Laptop Desktop
10+ times/day 15 4 0
6-9 times/day 2 4 0
3-5 times/day 13 13 2
1-2 times/day 17 6 7
5-6 times/wk 6 8 8
3-4 times/wk 14 6 10
1-2 times/wk 15 1 17
less than 1/wk 6 2 6
Total 88 44 50
Locations for Technology Use In Learning 20110610
Accessed from Mobile Laptop Desktop
uni 79 31 40
home 81 43 21
work 32 4 1
train/bus 67 13 n/a
café/pub 57 15 n/a
Technologies Used In Learning 20110610
Supporting learning with Mobile Laptop Desktop
calls 62 2 0
texts 76 0 0
e-mail 49 42 42
social networking 45 37 31
web 47 43 44
uni portal 31 43 40
uni VLE 25 42 38
e-books/journals 11 18 22
blogs 13 17 15
youTube 21 36 25
podcasts 9 17 9
music 11 14 7
films 2 20 7
TV 3 24 12
Apps 21 0 0
flickr 4 5 1
taking photos 28 2 1
taking video 10 1 0
games 5 1 0
dictionary 2 0 0

Initial Observations:

If the number of mobiles used for email and web is taken as an indicator of smartphone ownership amongst the students interviewed, then the figure of 49/98 (50%) is practically identical to the figure obtained in the online survey undertaken in October 2010 (496/982 = 50.5%).

Despite popular opinion that students use social networking rather than email for communication, more of those interviewed used email for learning-related communication:

Supporting learning with Mobile Laptop Desktop
e-mail 49 42 42
social networking 45 37 31

Interestingly, email use was more common on mobiles than on laptops or desktops.

Reflection on questions and responses in the online survey and interviews:

For future studies, it could be useful for research instruments to elicit responses for different (but sometimes overlapping) categories of technology-use for learning:

  1. Discussing and arranging course work
  2. Accessing course deadlines, timetables, briefs and feedback
  3. Discovering and accessing learning materials
  4. Producing and submitting course work
  5. Maintaining a personal study environment

An initial skim of the qualitative interview data suggests that mobile messaging, particularly texts and blackberry messenger (BBM), is used extensively for category#1.

Category#2 access to e-admin information to support learning emerged as the top priority for institutional mobile development in the larger (982 respondent) online survey

Prompts in this questionnaire could have done more to elicit responses about category#3 and category#4 use, but it is interesting to see in both the quantitative data and the qualitative responses to technology and study space questions that a number of students place an importance on playing music while studying (category#5).

As ever with W2C, we look forward to feedback and further ideas generated by this post.

Running with Wookie

When we first looked into getting involved with the Widget revolution we wanted scalable widgets that could enhance the student learning experience and be deployed on a range of different platforms.

Walking with Wookie

Er… thought you said Running with…

Well, when we first looked into getting involved with the Widget revolution we wanted scalable widgets that could enhance the student learning experience and be deployed on a range of different platforms. Indeed, we still do. We are in the process of deploying Moodle, hosted by the University of London Computing Centre (ULCC), as our institutional VLE, and were attracted to the potential of widgets as a way to enhance the VLE and be available on mobile devices. We realized that if widgets were to be part of our core offering, we’d need a widget server that could handle multi-thousands of hits in a short time interval, so began exploring Wookie with that in mind.

I proceeded to put a test platform together on our existing webserver and opted for the Wookie install that utilises Tomcat and MySQL so that we could potentially load-test the platform at a similar level to our existing web platforms.

I have to say the process wasn’t attractive or easy. “Dependency hell” took over fairly soon with trying to determine which JDK to run, out of the three that lived on our system (an Ubuntu derivative), making sure we had the right JDBC and that it could talk to MySQL (also having to be set up correctly). That said, much of this is really dependent on the platform you wish to run Wookie on, the variant or distro and so on. Having revisited the Wookie trunk in recent weeks I can say great strides have been made in only a year in making Wookie easier to install. You still really have to beware of the various Java JDK’s out there – stick with the Oracle one would seem to be safe advice.

The development team are fantastic, frenetic and focused – I cannot recall too many other software / platform development projects that I have been involved with that have released so many upgrades, patches and fixes over such a short period. Wookie itself is still a project under Apache incubation, in theory a kind of Beta state. This means you need to expect a certain amount of work to get the software going on a system of your own. In my struggles I was able to visit Paul and Scott (and also Scott’s interesting personal page ) in Bolton and get a couple of errors sorted out to enable our install to function. This actually fed back into the process at the time – a script hadn’t performed as it should, was corrected, and the nice new working version made its way back into the build. Just goes to show how fast communities can rectify problems!

Result – a working copy of Wookie on a server base.

Problems with the Server-based install

At some point I opted to patch our system (as you should, once everyone else has done it and found the bits that break!), ANT – an essential component climbed a couple of revision levels and promptly broke the Wookie install script. This is kind of expected behaviour when working at the beta end of things, but it came at a time when we needed to start working on our widgets: not good! The team have been working to sort out all sorts of similar issues throughout the year, but with increasing interest in developing our own widgets to deploy into our enhanced VLE platform (check out Mark Stubbs’ recent post Writing Widgets) we revisited a server-side copy of Wookie. ANT is still not working – what to do?

Running with Wookie – use the Wookie standalone install

[pullquote]Useful  links

[/pullquote]

 

Given that our project really needs to focus on the development of Widgets, a rethink was needed on applying efforts towards writing Widgets, and not messing with Wookie. Whilst visiting the developers, Scott and Paul, in Bolton about some issues regarding our server-based installation I noticed that the folks there are essentially all developing against local standalone copies of Wookie.

I decided to give it a try. Simply put, it’s the fastest and easiest way to get Wookie going. I have no stats to tell you how robust the standalone copy is when operated within a typical server situation, but thus far, for our tests, it has remained up and viable.

So – what are the crucial benefits of using the standalone copy?

Firstly, ease of install. It can be as quick as this:

[codesyntax lang=”bash” lines=”no”]

svn co http://svn.apache.org/repos/asf/incubator/wookie/trunk
ant run

[/codesyntax]

an example of a Running Wookie Instance
an example of a Running Wookie Instance in MSDOS cmd shell

Several lines of text later after SVN has copied you the latest trunk build of Wookie, and ant has performed a great many tasks including downloading IVY, Wookie will instantiate and start

logging stuff to the screen.

 

There are some pre-requisites here though:

  • An accessible JRE (Java Runtime Environment) on your machine
  • a copy of Apache ANT (for windows – you can check out WinANT which takes out some of the grief in getting this going and setting environment paths and so on, and has the added bonus that you can get an archived copy of WinANT 4.1 – the last 1.7.x release before ANT migrated to the 1.8 series that leaves your Wookie broken and downhearted).
  • Subversion
Wookie first menu
Wookie's first menu

Once it’s up and running you will have a nice fast localised copy of Wookie running with all the default settings (so the Admin user and password are java & java respectively).

You can fiddle with some settings in the config files that reside in the ‘root’ level of the trunk directory. Best place to get information on these is the wookie site.

 

Flushed with this success, I determined to remove the schizoid and out of date copy of Wookie we have running on the LRT server to see if a standalone copy might work better. The LRT server – like others at the MMU – is behind the MMU firewall. At the time I built it we were not allowed to be added to the bypass group that permits servers to communicate with the WWW without having to go via a proxy – which usually makes a mess of all sorts of things on a server level. So – the original install of Wookie had entries in the config to add the proxy in by hand – and – alas – down the line, it was just possibly getting in the way of examining accurate widget behaviour.

Nowadays the whole server is in the bypass group, so in theory a fresh install and a standalone server should have widgets that can communicate with resources out on the web without judicious proxy messing.

Wookie's Demo Widgets
Wookie's Demo Widgets

Et Voila – it works. We now have a ‘standalone copy’ running on a server hosting widgets that now actually tell me what the weather is in Manchester (and no, it’s actually sunny!).

Running with Wookie – Tips for maintaining a standalone install

The problem is, that we are running this is on an actual server. It’s designed to sit there without someone nurse-maiding an instance of a program running in a terminal or shell, any reboot, any premature death of that shell (like killing the SSH shell from my machine to the server in which I actually launched Wookie) is going to see Wookie die a death. On a reboot, it’s not going to come back alive.

The standalone doesn’t act like a standard daemon in Linux. Once you hit ant run – its going to run and stay resident in the shell window you typed that command in – logs to that window and no escape from it without killing wookie with a control-c.

So – how do we manage?

SCREEN is a wonderful tool that has, after some years, now wormed its way onto standard distros for *NIX and even MacOS X. It allows you to run virtual terminals that stay resident in memory, have full TTY access and a witty set of escape sequences that, at a base level, allow you to essentially kick a terminal into life, run a command and virtually exit the terminal. Next time you log in you can reuse the Screen command and attach to that very same terminal you left – and, barring a reboot of the box – it will still be there with your program running merrily away and not the least bit bothered that you wandered off and left it.

Check this command out:

[codesyntax lang=”bash” lines=”no” blockstate=”expanded”]

screen -S "wookie-running" -d -m ant run -Drun.args="initDB=false"

[/codesyntax]

Woah neddy! thats a lot of arguments, here is an explanation:

  • screen – the command we need
  • -S “wookie-running” our instance of screen will have a nice english language name to allow us to refer to it by
  • -d -m as a couplet – these commands allow screen to do its stuff, run your command, and automatically come back out to the terminal you are currently in – auto-detaching itself in other words. The cunning part here is that your stuff is still running in the background in memory
  • all the rest is the standard wookie command to run without destroying our existing DB

You can nip back into your virtual terminal by the following command:

[codesyntax lang=”bash” lines=”no” blockstate=”expanded”]

screen -r wookie-running

[/codesyntax]

except you need to swap wookie-running out to whatever name you used for the screen sessing with the -S command. Hey presto – you will shoot right into a log of log lines Wookie has been dutifully dumping to the Standard IO of the terminal.

Getting back out? A doddle, hold control A and hit c

This will take you back to your original terminal without killing the session. If you want to kill the session, hit control D, or control C to kill Wookie then then Control D to get out of the virtual screen.

You might take to this method of doing things, I’ll stick some links in to good resources relating to Screen as soon as possible, it is extremely powerful and indeed programmable too. I will leave you with one final very useful command in case you do decide to do a few screens:

How to see how many screens you have established:

[codesyntax lang=”bash” lines=”no” blockstate=”expanded”]

lrt ~ # screen -ls

There are screens on:

20486.wookie-running    (03/21/2011 10:58:06 PM)        (Detached)

17838.stats     (03/01/2011 10:25:58 AM)        (Detached)

15863.backup    (02/24/2011 09:30:26 AM)        (Detached)

3 Sockets in /var/run/screen/S-root.

[/codesyntax]

Finally, we haven’t really solved the issue of making sure Wookie is running following a server reboot. This isn’t a massive problem, so for the moment I will just suggest that the screen command can be put into a standard init shell script. It must either be properly pathed, or execute a ‘cd’ command to the Wookie trunk before executing the screen portion. I will trial this shortly and put the additional steps into this article.

Lessons Learned

Really, if you are testing something, it’s not always best to go for the platform you think you will need to run it in anger. Luckily for us we had time to adapt, chuck out the stuff we didn’t need and came back to the problem from a fresh angle.

If running the server itself is your main angle (maybe you are a sysadmin), then invest some time on the Wookie pages and the email group finding out the best combination of dependencies – JDK, ANT, JDBC etc… really consider if you even need MySQL vs the built-in Derby engine. These choices may change depending on the extent of your target audience and the numbers of hits you expect Wookie to be able to cope with. If you expect to hang a major system off the back of this – remember – it’s in Beta/ Incubation, hiccups are going to happen from time to time despite best efforts of the developers.

If, on the other hand, you are really more interested in developing the widgets themselves, just go with the easiest option and use the Wookie standalone. It’s far less messy, and you can be up and running developing widgets within ten minutes or so.

Caveats and other Witty Considerations

I have used the term standalone here to indicate the difference in how the platform behaves, rather than the notion that you could use Wookie in this way off the net. When you type ant run, ANT is going to try to nip off and find updates to IVY, and it will fail miserably if you aren’t on the net, or are sitting behind a proxy and haven’t set the correct proxy variable. Here’s a couple of pointers:

Running ANT in offline mode

[codesyntax lang=”bash” lines=”no”]

ant run -Doffline="true"

[/codesyntax]

 

Setting the Proxy variable for ANT (behind a proxy)

[codesyntax lang=”php” lines=”no”]

ant -run -autoproxy

[/codesyntax]

Read a whole lot more about Ant and proxy here

Due Thanks to…

Scott, Ross, Paul and Co

Please dip in with comments, this post was put together over a few days including two great days at the Bolton CETIS Widget Bash, so most likely there are some lapses of concentration! Corrections gratefully received.

Now, give it a go, give the guys a go and get active. Get hold of Wookie, read about Widgets, grab some, play with them, join the mailing list [info further down the page].

http://ant.apache.org/manual/proxy.html

Widget development

Over the last few months, W2C has been laying foundations for its extensible VLE by developing web services to publish frequently-asked-for information from university systems. We are now ready to test the potential of W3C Widgets as a mechanism for publishing the output of these web-services on a range of platforms, and have just written our first widget. This post describes our experience.

To de-scope security issues from our first development, we decided to develop a widget to publish public data available from our PC Availability web-service. First we reviewed published examples of widgets and found particularly helpful WIDE’s step-by-step guide to developing a Calendar web-service and Scott Wilson’s recent post on progressive enhancement of mobile content. Based on this initial review, we decided that our widget should use:

  • jQuery – javascript library (to consume our web-service)
  • jQuery-mobile – javascript library and CSS (to render the content)
  • A simple Controller model to organise our javascript functions

and we knew we’d need:

  • a Wookie server – Steve’s working on a post about lessons learned from getting this going!
  • a folder structure in which to develop our PC Availability widget, comprising
    • config.xml
    • pc.html
    • scripts folder, containing:
      • latest copy of jQuery.js (minimized version)
      • latest copy of jQuery-mobile.js (minimized version)
      • our javascript: pc.js
    • style folder, containing:
      • latest copy of jQuery-mobile.css (minimized version)
      • folder of jQuery-mobile images

Our files contained the following:

[codesyntax lang=”xml” title=”config.xml”]

<widget xmlns="http://www.w3.org/ns/widgets"
   id="http://mmu.ac.uk/pc"
   version="1.0"
   height="480"
   width="320"
   >
  <name>PC Availability</name>
  <description>A sample widget to display PC availability</description>
  <content src="pc.html"/>
  <author>Mark Stubbs</author>
</widget>

[/codesyntax]
Our config.xml file defines a simple “PC Availability” widget that will be loaded from pc.html. Default size has been set as 320×480 for mobile rendering.

[codesyntax lang=”xml” title=”pc.html”]

<html lang="en">
  <head>
	<meta charset="utf-8" />
	<title>PC Availability</title>
	<link rel="stylesheet" href="scripts/jquery.mobile-1.0a3.min.css" type="text/css" />
	<script type="text/javascript" src="scripts/jquery-1.5.1.min.js"></script>
	<script type="text/javascript" src="scripts/jquery.mobile-1.0a3.min.js"></script>
	<script type="text/javascript" src="scripts/pc.js"></script>
  </head>
  <body onLoad="Controller.init()">
    <div data-role="page" id="home">
      <div data-role="header">
        <h4>PC Availability</h4>
      </div>
      <div data-role="content" class="ui-content">
        <ul data-role="listview" id="rooms-listview" data-theme="d" data-inset="true">
        </ul>
      </div><!-- /content -->
    </div><!-- /page -->
  </body>
</html>

[/codesyntax]
Our pc.html file loads the jQuery-mobile 1.03a stylesheet; the jQuery 1.5.1 minimized library; the jQuery-mobile 1.0a3 minimized library and our own pc.js javascript. The body tag contains an onLoad handler that calls a javascript function we have defined for the Controller class called init(). The xhtml within the page is organized to present information as a single page (using the div data-role=”page” tag), which has a header (div data-role=”header”) and content (div data-role=”content”) section. Within the header section, the h4 tag is used for the page title. Within the content section, an unordered list tag is included (with an id of “rooms-listview”), which will contain list items for current PC Availability at each drop-in space within the university. The ul tag is styled with some jQuery-mobile markup: data-role=”listview” data-theme=”d” and data-inset=”true”. Further information about controlling the presentation of list data in jQuery-mobile is available at http://jquerymobile.com/test/#docs/content/../api/../lists/lists-themes.html.

 

[codesyntax lang=”javascript” title=”pc.js”]

var Controller = {
	init:function() {
		Controller.update();
	},

	update:function() {

		var loc = Widget.proxify("http://testapis.ad.mmu.ac.uk/icts/Service1.svc/getPcAvailability");

		$.ajax({
		        type: "GET",
			url: loc,
			dataType: "xml",
			timeout: 1000,
			complete: Controller.parseResponse
		});
	},

	parseResponse:function(response) {

		var rooms = $("#rooms-listview");
		rooms.empty();
		$(response.responseXML).find("room").each(function () {
			rooms.append($("<li/>").text($(this).attr("location") + ": "
			+ $(this).attr("free") + "/"
			+ $(this).attr("seats") + " free"));
		});
		rooms.listview("refresh");
	}
}

[/codesyntax]

Our scripts/pc.js file defines the Controller class referenced in the body onLoad handler of pc.html. The Controller class has 3 functions:

  1. init()

    initializes by simply calling the update function

  2. update()

    uses Wookie’s Widget.proxify to get a URL within the same domain (using Wookie’s proxy to avoid cross-domain scripting restrictions) that can be used to call the PC Availability web-service, which it then does using the jQuery $.ajax function; this calls the Controller’s parseResponse function when it completes, passing across the httpResponse received from the web-service call.

  3. parseResponse(response)

    gets a handle for the rooms-listview element, clears its content and then iterates over the “room” elements in the xml returned from the web-service. A function is defined for handling each room element that is found. That function appends to the rooms-listview element a new list-item and sets the text of the list-item to be the location followed by the number of seats free, followed by the total. The parseResponse function completes by refreshing the rooms-listview with this new content.

On our Windows 7 development machine, we selected the pc.html, config.xml and the scripts and styles folders and right-clicked on the “send to Compressed (zipped) folder” option. We then renamed the resulting file “pc.wgt“. From the Wookie Adninistration menu we clicked the “Add new widget”, browsed for our pc.wgt file and clicked the “Publish” button to add it. Our PC Availability widget (minus an icon, as we hadn’t specified one in the config.xml file) was then available in the widget gallery for testing. We added the address of our PC Availability web service to the Wookie white list (so that the Widget.proxify command would work for the URL we’d specified) and then clicked on the demo button in the gallery, which produced this in Firefox:

Screen-shot of PC Availaibility widget running in Wookie using Firefox 3.6.15
Screen-shot of PC Availaibility widget running in Wookie using Firefox 3.6.15

We then tried in IE8 and the widget was rendered without any CSS and a javascript error. After much searching we found this post about IE support in jQuery, which suggests that work still needs to be done for IE 7/8 and Windows Phone 7, but that Scott Jehl had produced an “at your own risk” workaround that must be loaded after the jQuery library and before the jQuery-mobile library loads. We inserted the javascript inline between the two library calls, and the widget then rendered in IE8:

Screen-shot of PC Availability Widget rendered in Wookie using IE8.0.7600
Screen-shot of PC Availability Widget rendered in Wookie using IE8.0.7600

Apart from not having rounded corners, there was no obvious difference between the rendering of the two in different browsers (content differs as expected from a dynamic feed).

We then tried in a number of other browsers:

  • Safari on web and iPhone worked
  • the Android browser worked on a phone and a Galaxy Tab
  • the native browser failed to display anything on a Blackberry or a Windows Mobile 7 phone
  • Opera mobile worked on a Blackberry

Our initial foray into the world of widgets was pretty positive, although support for popular Windows browsers would need to move beyond an “at your own risk” hack for the interesting progressive enhancement approach to be viable for production deployment. We hope this post will encourage others to have a go and share their thoughts.

Understanding how our fees web-services are being used

After intensive testing, fee status information was released via the myMMU portal to all first year students on November 17, 2010. Naturally, there has been considerable interest in understanding the impact of this new system in terms of take-up and in the broader context of feedback from students and front-line FLS staff dealing with financial queries.
Here is how we sorted the records to get the statistics

Alex’s earlier post described how LRT developers worked with colleagues in Financial and Legal Services (FLS) to give students access to a personalised traffic light summary of their financial standing across three categories: tuition, accommodation and other fees. Information displayed in the myMMU SharePoint portal was provided by a getFeeStatus web-service and used in a WebPart that enabled students to use a second web-service getFeeEmail if they wished to receive a detailed financial statement by email.

After intensive testing, fee status information was released via the myMMU portal to all first year students on November 17, 2010. Naturally, there has been considerable interest in understanding the impact of this new system in terms of take-up and in the broader context of feedback from students and front-line FLS staff dealing with financial queries.

So, here are some initial statistics gathered from the server running the REST web-services:

Total accesses Nov 17th 2010 to January 15th 2011:

All logs from November:   1,754,573 for all objects and pages on the web-services server

Students who have viewed the Traffic Lights summary page:  9084

Students who follow through from the Traffic Lights summary page to request a detailed financial statement:  1132

How did we go about getting these stats?

The getFeeStatus and getFeeEmail REST web-services run on a Microsoft IIS server which, like all other MS IIS servers, keeps hit logs of people and machines accessing pages hosted on it: URL u accessed at time t using browser b from internet address i – and the really good news is that being REST services, not only will the name of the web-service called be logged, but all the querystring calling parameters will also appear in the URL … that’s going to come in handy later.

So the process of determining usage of the traffic light web-services starts and even ends (for this part of the analysis) on the IIS server, and by necessity it involves getting grubby with raw web server log files, rooting out nuggets of information pertinent to the things we’re interested in.

Sure – we could run the logs (all x Gigabytes of them) through a good log analyser like AWStats or the freeware Funnel Web Analyzer Kieron used for the graphs on a previous post.

These are great at providing generic page and object hits, but they tend to deal with ‘top 10 links served in November’ type scenarios. Useful, but not really what we need to discover how many students made use of a particular service with particular parameters within a particular timeframe.

To do this we decided to use a suite of geek-level powertools on a Linux box. Of course, we could use a suite of geek-level tools on a Windows box, but we’d have to find them first. Yes, yes… there is Powershell, but just read on – and let me know if Powershell could cope!

Prelims – pour yourself a good strong coffee!

Firstly, we had to copy the raw logs over from the IIS box to an ancillary box – why? First of all so we can use the aforementioned Linux power tools (bash commands.) Secondly, log processing is inherently processor and memory intensive – why ruin the service you are trying to report on.  Admittedly we perhaps shouldn’t have used the web server hosting this blog as it slowed it down a bit!  But we did.

Now, let’s take a look at a few typical lines from an IIS log file; if you try this on an Apache based log file (all of these servers adhere to the Common Log Format, which, alas, allows for many different variations of the actual information stored in certain agreed keys – so you need to examine the fields in your file really carefully to make sure you are storing the right stuff. Apache normally does this out of the box, IIS has certain interesting data turned-off by default. No idea why – just take some time and make sure you configure the logging correctly on your server – stats could be dribbling away into the ether!)

[codesyntax lang=”apache”]

2011-01-14 11:56:07 W3SVC1797328370 149.170.241.162 GET /convertid/Service1.svc/getIdByMmuId8 id=5503xxxx 80 - 149.170.241.162 - 200 0 0
2011-01-14 11:56:07 W3SVC1797328370 149.170.241.162 GET /finance/Service1.svc/getFeeStatus person=550xxxx0&dtm=1295006167&developer=mymmu&format=rss&token=ee5b3e4bc033f093bd2eecec7331812f 80 - 149.170.247.8 - 200 0 0
2011-01-14 11:56:09 W3SVC1797328370 149.170.241.162 GET /srs/Service1.svc/getCurrentEnrolments person=0838xxxx 80 - 149.170.247.8 - 200 0 0
2011-01-14 11:56:13 W3SVC1797328370 149.170.241.162 GET /vle/Service1.svc/getWebCtAreas format=rss&dtm=12950xxxx2&developer=mymmu&token=057ee448560dedadbddfc842eaf838f1&person=08186066 80 - 149.170.247.8 - 200 0 0
2011-01-14 11:56:13 W3SVC1797328370 149.170.241.162 GET /convertid/Service1.svc/getIdByMmuId8 id=0818xxxx 80 - 149.170.241.162 - 200 0 0

[/codesyntax]

Gibberish isn’t it? Well, no. It just looks complicated because it has really been designed to be read by machines and witty processing routines. The fields are delineated (made distinct from other fields) by spaces. So – the first field on each line is the date field containing ‘2011-01-14’ and the 6th field starts ‘/convertid’ and so on. Each line is a record, an instance of  single request to the server for a single object or action. This is an important point to remember – each record is not a record of a single person hitting your page as practically all pages on a server contain more objects than just the page itself.

(NB: student IDs in the example data have been anonymised by replacing half of the ID number with xxxx.)

Another important thing to remember for this exercise, is that the records are distinct, regular and structured. In effect – this is the same as having a database of materials that we can process looking for regular pieces of information that match a pattern that we are interested in. As the data is regular we can know that we shouldn’t be getting any weird anomalies that would distort any processing and give errant data. That the data is distinct means we know that we can count up instances of the records we find and know that the count actually represents something.

Continue reading “Understanding how our fees web-services are being used”

Device led student interviews

During January and February the Project Team (members of Learning Research & Information Services and the Centre for Research in Library & Information Management) will be carrying out 100 short impromptu interviews with MMU students. The aim of the research is to find out about students’ use of technology in their studies. A pilot has already been run and the team are currently tweaking some of the questions based on the responses and feedback received. What’s distinctive about the interviews is that the questions are device led i.e. asking students to show us the technology device/s they are carrying with them there and then (whether it be a notebook, Kindle, mobile phone etc) and ask how these devices are used in their learning (including what they use them for, where and how frequently). We’ll also be asking how well students feel MMU supports their use of technology and how it could be improved. By using this method it is anticipated we will gather some very meaningful research – to be utilised at the next stage of the W2C research… student design workshops (more info to follow!)

Easy Access Financial Info

LRT along with Financial and Legal Services (F&LS) have recently developed a new web application that allows students to see an overview of their financial information whenever they log in to the myMMU SharePoint portal. This has been designed to highlight urgent messages about overdue fees, provide an easy-to-understand breakdown of which fees are due, and allow students to request a detailed financial statement to be emailed to them.

Part of the need for such an application is to increase the flow of financial information between F&LS and individual students. Increasing student engagement and enhancing the student experience were key reasons for this development. By displaying this information on the front page of the portal, students are immediately made aware of any financial issues that may need urgent resolution. The development of this Portal initiative, along with current Credit Control procedures including telephone, texts, letters and emails, provides the student with up to date information.

This joint venture was created as a result of listening to students and delivering a concept that is easy to understand, informative and adds value to the wider University. The concept alerts students to fee status information every time they log in to check email or access the VLE. This application consists of a front-end implemented in SharePoint Web Parts, and a web services backend.

SharePoint Web Parts

Urgent News Overdue MessageThe message above is displayed on the front screen of the myMMU portal whenever there are overdue unpaid fees.  This is at the top of the page to make it as visible as possible. The red traffic light signals a problem and grabs the attention of the user, or if there are unpaid but not yet due fees an amber traffic light is displayed instead. No message is displayed on this page if there are no overdue fees.

Finance Webpart Showing Paid and Overdue FeesThe second part is more detailed information which displays fees by category. The traffic lights are used again to make it easy for users to see which fees are a problem. The button underneath the traffic lights allow students to request a financial statement, which generates an automatic email to students by F&LS through an overnight process.

Financial Web Services

All financial information is held within the University’s SAP finance system. Part of the challenge was to ensure complete security and privacy around personal financial data. Discussions between LRT and F&LS quickly agreed on a service-oriented approach in which nightly reports inside the SAP system would feed through summarised fee status information for each student in a database table. This table was visible to a web service written by the F&LS team.  A second web service would allow students to request a full financial statement be emailed to their registered email address. To accelerate development and testing, LRT decided to create proxy web services that would eventually call the financial web services but could return test data in the interim.

The following steps were taken in consultation with F&LS in order to meet security and privacy requirements:

  • Sensitive financial data were summarised by a process within SAP and exported to a table as student id and fee status indicators only, no value data was visible.
  • A database user with read-only access to the summarised table was set up to make queries for the finance web service
  • A token-based protocol, using a one-way hash of parameters with a timestamp and shared secret, was agreed to enable the financial web services to confirm they had been called by a legitimate user before responding
  • Firewall settings ensured that only trusted machines could access the financial web services

The token-based protocol was also used for the proxy web services, which forwarded calling parameters and tokens to the financial web services and returned responses received.

Usage Statistics

Since the service was launched on Wednesday 17th November 2010:

  • There have been 157627 hits to the finance web service on testapis (as of 4pm on 30th November 2010). Note that this does include all 30,000+ students, even students who aren’t first year students (who will not be able to see any financial info) will create a hit on testapis, which will return a null value.
  • There have been 790 requests for financial statements using the traffic lights webpart. Only first year students who have unpaid fees can request a statement, so this is a significant number of roughly ten thousand MMU students who are in their first year of study.
Web Services Hits
Hits for testapis web services

Services Architecture

The services are delivered to the student through MMU’s SharePoint portal, my.mmu.ac.uk. The front page gives students a short announcement when they login if any fees are overdue through a in-house custom “Urgent News” web part, with a link for more information. When a user clicks on this link they are taken to a different SharePoint page with another custom “Traffic Lights” web part with more detailed information.

The overall architecture is shown in the diagrams below:

Urgent News Architecture
Finance Webservice Architecture for Urgent News Web Part
Traffic Lights Architecture
Finance Webservice Architecture for Traffic Lights Web Part

These web parts call LRT’s proxy webservice on testapis.mmu.ac.uk with parameters containing student ID, the one-way hash authentication token, and return format. The return type can be XML, RSS, or Atom which allowed us to develop a single web service that could be consumed by the two web parts, and should make it easier to extend this service to different applications in future.

When testapis receives a request it checks the authentication token is valid. It checks the student records system to retrieve the student’s institution ID (which identifies students on the finance systems) and to check that the student is on the first year of their studies. This is shown as step 2 and 3 in the diagram. If the student is not a first year student, null is returned from the service, and nothing is displayed on the SharePoint page. The webservice uses C#.NET and REST architecture.

Testapis calls the finance webservice and forwards the authentication token, and the institution ID, queries the SAP database fee summary table, and returns this as XML as step 7 in the architecture diagrams. An example of this XML is below:

[codesyntax lang=”xml” lines=”normal”]

<getFinancialStandingResponse>
	<student>xxxxxxxx</student>
	<tuition>2</tuition>
	<accommodation>1</accommodation>
	<other>0</other>
	<responseDate>20.09.2010, Time : 10:56:10</responseDate>
</getFinancialStandingResponse>

[/codesyntax]

Testapis receives this – if the return format is XML it formats the date and removes the student ID, before passing the data back to the calling service in step 8 of the Traffic Lights diagram:

[codesyntax lang=”xml” lines=”normal”]

<getFeeStatus>
	<tuition>1</tuition>
	<accommodation>3</accommodation>
	<other>0</other>
	<updated>30/11/2010 04:37 GMT</updated>
</getFeeStatus>

[/codesyntax]

On the other hand if the requested format is RSS, testapis uses the abstract .NET SynchronisationFeed class to create a different feed item for each of the fee types – tuition, accomodation, and other. This contains a message that is readable and useful for students rather than the numbers indicated above:

[codesyntax lang=”xml” lines=”normal”]

<rss version="2.0">
	<channel>
		<title>Fee Status</title>
		<link>http://www.finance.mmu.ac.uk/students</link>
		<description>Information on your current MMU fee status</description>
		<item>
			<guid isPermaLink="false">2</guid>
			<link>http://www.finance.mmu.ac.uk/students</link>
			<title>Tuition Fees paid</title>
			<description>Processed invoices to-date already paid.  Thank you.</description>
			<updated>2010-11-30T04:37:00Z</updated>
		</item>
		<item>
			<guid isPermaLink="false">1</guid>
			<link>http://www.finance.mmu.ac.uk/students</link>
			<title>Urgent: Accommodation Fees Overdue!</title>
			<description>Invoices are overdue for payment (if applicable these will also include any invoices paid on your behalf by your employer/sponsor).  Please contact your Faculty Finance Office if you are paying your own fees and have not agreed an instalment plan.</description>
			<updated>2010-11-30T04:37:00Z</updated>
		</item>
		<item>
			<guid isPermaLink="false">3</guid>
			<link>http://www.finance.mmu.ac.uk/students</link>
			<title>Other Fees</title>
			<description>Financial status is not applicable at this moment.</description>
			<updated>2010-11-30T04:37:00Z</updated>
		</item>
	</channel>
</rss>

[/codesyntax]

The SynchronisationFeed is converted into RSS (or Atom depending on what format was requested) before being passed back to the caller in step 8 of the Urgent News. Finally the student (who is a fully enrolled first year) is shown the custom web parts below:

Urgent News Overdue Message

Finance Webpart Showing Paid and Overdue Fees
Our next step is to find appropriate ways to make this information available to students on their mobiles. We have previously considered the possibility of a token-based approach for delivering private and personalised information to W3C widgets (2010-09-14) and we hope to explore the potential of the approach for this sensitive data in consultation with F&LS colleagues.

Credits

Web Parts and XSLT coded by Ludovic Tolhurst-Cleaver & Mark Stubbs
XML schema developed by LRT & F&LS
Finance Web Service and Traffic Lights Graphics developed by F&LS
LRT Proxy Web Service developed by Alexander Lee