Dealing With Unpredictable Web Services
The idea of bringing together services from a large range of places is great. Students and staff can log into one place like my.mmu.ac.uk and get all their information in one place. In Moodle, information is delivered in a block, that brings in information and resources in the context of the online study areas they are looking at.
But it also means that when there is an unexpected problem with one of these services, the site that it is delivered through can grind to a halt. The portal to access everything stops working, and suddenly users can access nothing.
One of the services has been unpredictable lately, perhaps because it’s being used far more than it was in the past. We call this service through a .NET SOAP request which has a timeout setting. When the server slows, the timeout should kick in and the call should be cancelled. It doesn’t though… it sits their and waits forever for a response… then probably due to exponentially increasing requests, and old requests not being cleared, the server falls over completely. Then many other services grind to a halt.
So what should we do to fix it? I thought about opening it in a new thread, then killing it if it took too long. But there are so many problems with that! Memory leaks and unpredictable results (not good on a server which provides many of our integrated services) … and it doesn’t stop the service calling the service when it’s struggling, which will still make it fall over!
So what I did was to use the web.config file to store information between calls of the service. Every five minutes, one unlucky user out of many will have to wait a bit longer while the service checks the dodgy server is ok. It writes the response and the time of the call to web.config.
When a new call is made it checks the last time the service was checked in the config file, if it was within 5 minutes and it’s ok, it calls the service. If it’s more than five mins, it does another check to see whether it’s available. Ta da, a reliable service built on an unreliable server (touch wood!)
- Configuration myConfiguration = System.Web.Configuration.WebConfigurationManager.OpenWebConfiguration("~");
-
- string lastrequest = "";
- try
- {
- //set it to not found - will get changed later if it is found!
- lastrequest = ConfigurationSettings.AppSettings["problemservice.lastrequest"];
- }
- catch (Exception)
- {
- myConfiguration.AppSettings.Settings["problemservice.update"].Value = "errornotfound";
- }
-
- if (string.IsNullOrEmpty(lastrequest))
- {
- myConfiguration.AppSettings.Settings.Remove("problemservice.lastrequest");
- myConfiguration.AppSettings.Settings.Add("problemservice.lastrequest", DateTimeToTimestamp(DateTime.Now));
- }
- else
- {
-
- DateTime dtm = DateTime.MinValue;
- if (!DateTime.TryParseExact(lastrequest, "yyyyMMddHHmmssffff", null, System.Globalization.DateTimeStyles.None, out dtm))
- {
- }
- if (dtm < DateTime.Now.AddMinutes(-5))
- {
- HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(
- "http://problemserver.mmu.ac.uk");
-
- httpRequest.Timeout = 5000;
- HttpWebResponse httpResponse;
- HttpStatusCode statusCode;
- try
- {
- httpResponse = (HttpWebResponse)httpRequest.GetResponse();
- statusCode = httpResponse.StatusCode;
- }
- catch (Exception)
- {
- statusCode = HttpStatusCode.ServiceUnavailable;
- }
- myConfiguration.AppSettings.Settings.Remove("problemservice.status");
- myConfiguration.AppSettings.Settings.Add("problemservice.status", statusCode.ToString());
-
- if (statusCode != HttpStatusCode.OK)
- {
- /*send an email notification to the admin that the server is unavaiable*/
- }
- myConfiguration.AppSettings.Settings["problemservice.update"].Value = "updated";
- myConfiguration.AppSettings.Settings.Remove("problemservice.lastrequest");
- myConfiguration.AppSettings.Settings.Add("problemservice.lastrequest", DateTimeToTimestamp(DateTime.Now));
- }
- else
- {
- myConfiguration.AppSettings.Settings["problemservice.update"].Value = "notupdated";
- }
- }
- myConfiguration.Save();
-
- string problemserviceStatus = myConfiguration.AppSettings.Settings["problemservice.status"].Value;
-
- if (problemserviceStatus == "OK")
- {
- /*Call the SOAP service!*/
- }
Using Technology To Build A Green Community At MMU
The Catalyst project is a cross-disciplinary project that looks to get academics and communities to work together to develop new technologies to meet these challenges. It is an exciting project and the world definitely needs more projects like this. Our challenges in society are not limited to one field as everything in real life is connected – so working with people outside your field is critical. At MMU and we have been discussing ideas that might fit into a Catalyst proposal.
It is clear that environmental issues do not respect the organisational boundaries that society has created. One of the greatest challenges of today is to break out of the usual structures that we operate in on a day to day basis in order to solve them. Current modes of thinking and working don’t make cross-organisational work easy, but major cultural changes will not take place unless new ideas are shared across broader society.
The concept of an online collaborative ideas space was born at a meeting between staff and students at Manchester Metropolitan University who are leading on environmental issues. MMU is implementing bold plans to reduce emissions[1] and students are running successful environmental campaigns[2], so this project could help both groups work together as a community to resolve the difficult challenge of facilitating behavioural change. The ideas space would encourage the initiation and discussion of ideas for environmental projects, and the best ideas could then be taken forward into real projects and potentially provide students with impressive evidence of wider achievement for their Higher Education Achievement Report.
Initially then, we wanted an online system to discuss and share project ideas between staff and students, and maybe other communities in the area of MMU too. Potential technologies we could use to do this would be Google Moderator, or Get Satisfaction. Both systems allow online crowdsourcing of ideas, voting, and discussions. I was impressed by the White House’s use of Google Moderator which shows that even though it is free the technology behind it is very stable, and the API looks strong. Get Satisfaction on the other hand had a track record of large scale use from Microsoft to crowdsource ideas for new products and features, and it grabbed discussions about your ideas from social networks too, which would be a big win. The downside was that to get full API access it would be necessary to buy the $600pm licence… so doing interesting things with the data would require major financial investment.
1. MMU’s Carbon Management Plan: http://www.mmu.ac.uk/environment/pdf/carbon-management-plan.pdf
2. MMUnion’s Student Switch Off Campaign: http://www.mmu.ac.uk/environment/eventsandcampaigns/joinin/switch-off.php
UPDATE:
Unfortunately, MMU’s idea didn’t quite fit into what Catalyst was looking for, specifically as our idea was looking to use existing technologies as opposed to developing new ones. We’ll definitely keep an eye on the Catalyst Project though as there will be other opportunities to be involved later. In the meantime, people are still keen to build a staff and student community on green issues, so hopefully we can do something on our own for now… and also it’s worth mentioning that developing the concept of a Storytelling Plate and making play-doh food at at the Catalyst meeting was great fun!
Notes from Moodle Moot 2011
As MMU has been a long term WebCT user for many years, this was the first time I have had the chance to go to a Moodle Moot, and thoroughly enjoyed it! There was little of the bad moods that cloud Blackboard conferences, as it’s more of a community and also because of anticipation around Moodle 2.0. As a result it was much more enjoyable. Videos from the conference are at http://mootuk11.org.uk/videos/
Unconference
The start was a little stressful as Steve and I had little time to prepare for presenting at the conference, but well worth doing. Several people were praising our architechture of aiming to have all data being passed through an API server as REST web services, and asked us for more details. I’m not sure I’ll have time to answer questions here about how we did it – but hope to do this in a future blog post. Slides from the presentation are at http://www.slideshare.net/nibb/lrt-moodlemootuk11-unconf-presentation
I missed the breakout sessions while working on the presentation and answering questions afterwards. I did have some interesting discussions. Most questions were around whether you needed to use .NET – which you emphatically don’t! There are benefits to using .NET as a API server, such using LINQ to process XML using SQL-type queries, and Parallel LINQ in .NET 4.0, although there is a PHP alternative called – funnily enough – PHPLINQ that I found after being asked by Eamonn Costello (@eam0) at Dublin University.
Paul Thornton (@jivinace) at Exeter asked about how we set up the API server. It seemed their setup is similar to the way MMU’s old setup with a system in between the student records system and Moodle which manages staff enrolments and allows people to create or request new course. When Moodle was planned, the difference was that the Moodle project was in the scope of a directorate project. This gave us the ability to talk to people high up in MMU about what data we to rethink integrations across systems to find the best way to use data. I was also grateful to Mike Hughes (@mhughes2k) for positive comments and good links to GIT.
Someone recommended PHPStorm as a powerful PHP editor… looks great, but at £74 a licence I might give it a miss and see if I can find a free alternative.
Plugins and Modules
Kent talked about their export tool from WebCT to Moodle, which we have tested ourselves at MMU. It is a good option if you want information out of WebCT, but don’t expect it to look well designed, as all the resources appear in a ‘scroll of death’ list. But we hope to use it at MMU so we have access to WebCT information after our WebCT licence runs out in September. They also developed a couple of good plugins, such as:
- Simple file upload block, which reduces the number of clicks needed for people to upload a file
- A CLA upload block
- A rollover block, which makes it easier and faster for teachers to copy content over from the equivalent area in the previous year. This currently isn’t public but Steve Coppin was considering it.
The OU mentioned the annotation tool which sounded amazing but didn’t catch the name of it. Additionally, South Devon have a public Git hub at github.com/sdc
Moodle 2.0
On the subject of Moodle 2.0 I found out some useful stuff at the OU presentation by Ross Mackensie. They have huge usage stats – 700,000 transactions per day!
They decided to make future upgrades easier by not making hacks but working with the Moodle devs to make changes that are accepted into the core code. This should benefit the community lots too. The OU are working to redevelop several things, but one of these is the groups, roles and permissions, which could be interesting. The OU are concerned about performance – which can be 2 – 3 times slower than 1.9. They are running 1.9 and 2.0 in parallel for a time, while migrating courses in groups, which will enable them to see reliability and performance before overcommitting. Their mobile access will be designed around stats that showed people want mobile browser access, rather than a mobile app. Twitter: twitter/rossmackensie and website: bit.ly/rossmac.
Martin Dougiamas AKA Mr. Moodle talked about Moodle 2.0. There will be a lot better systems in place to allow developers to contribute code to the main Moodle codebase, which should fewer institutions build custom hacks, and the common code base will improve significantly. There is a focus on web services for 2.1 that supposedly will allow access to any data in the system. There will be a mobile app at some point in the next year or so, which will be task focused and will look different from the web version. The plugin repository will change, and it will focus on allowing ‘trusted’ developers to review the plugins to show how reliable and secure they are, and it will be easier to find plugins for your version.
Some other links on Moodle 2.0:
- Moodle 1.9 > 2.0 development checklist http://bit.ly/flqU2z
- Intro to Moodle 2.0 presentation http://slidesha.re/hZWnlZ
Some notes from Ross’s presentation that I made:
Moodle 2.0
- Security
- Performance
- Media
- Integration
- Usability
- Modularity – designed to be customised
- http://hub.moodle.org/
Moodle 2.1 – June
- Web Services
- what web services???
- Quiz performance
- Basic LTI
Moodle 2.2 +
- Grading + Rubrics
- Competency Tracking
- Forum
- Lesson
- SCORM
Mobile App
- Task focused
- V different from standard web
- SEcure
- Messages as notifications
- iPhone > Android
New plugin repository
- Official reviews
- Easier to upgrade
- Easier to find plugins for your version
- Security reviews
- Trusted reviewers
- New developer wiki
- One wiki per release to ensure ‘currentness’
Cool Stuff About Linq
Linq is a shorthand way to process data natively in .NET in a SQL-like manner. It means with a few lines of code of Linq can do what could take tens of lines of indecipherable loops and if statements otherwise. In coding terms you will save a lot of time. Although it’s also generally faster for the code to run – the Linq types and statements are optimised to process large amounts of data.
So, here are some other facts about Linq:
- Linq essentially creates a view between the code you are running and the data itself, which you can modify and manipulate.
- Linq views will be able to cope with changing structures of data – as it doesn’t care about datatypes and where the data comes from.
- Don’t use the Linq To SQL libraries apart from to do basic testing. They are inefficient, and were never designed for production use. The ADO.NET Entity Data Model libraries should be used for this. Although ADO.NET before NET 4.0 isn’t great…
- LinqPad is a cool application that enables you to learn and test snippets of Linq code outside of the C# environment: http://www.linqpad.net/ (the paid version has Intellisense too)
- Use var datatype generally when returning data from a Linq query. Vars are flexible and only hold the data that is put into it, unlike other classes which may hold unnecessary data that you are not interested in. As such they save a lot of memory.
- Linq queries always return IEnumerable types – the class that lists, collections, arrays, and enums are based on, and can be casted into these if necessary (as well as the var datatype as above)
- If debugging Linq, you can pull the Linq code out and drop it into a separate function.
- Some operations like Count and Distinct have different syntax to SQL. For example c.Orders.Distinct.Count() would return the number of distinct orders.
- To convert a query into a list, you can wrap the query in brackets and do .ToList().
- You can use Linq to update databases and XML with update commands. Use the SubmitChanges() function to do this.
- PasteXMLasXelement is a very powerful plugin for VS. You can copy XML from a text editor, and paste in VS using the new menu option, and it will automatically set up the appropriate data structures in C# code, instatiating the right classes for elements, namespaces, attributes, etc. It can really change the way you code with XML.
- You can download libraries to Linq to anything or Linq to everything! Linq to Facebook, Linq to Amazon, Linq to AD, Linq to LDAP, within a few lines of code you can search their databases – check out the possibilities at http://rshelton.com/archive/2008/07/11/list-of-linq-providers.aspx)
- It’s possible to override operators such as WHERE in your own classes, so you can tell Linq which variables you want to be looked at by a Where command. It’s also possible to override operators like equals, Like, etc.
- Examples of Linq code can be found at 101 linq samples http://msdn.microsoft.com/en-us/vbasic/bb688088.aspx
Moodle Wonderland
I was surprised to enjoy the Moodle Wonderland more than other conferences I have been to. There was a modest and friendly atmosphere to the conference, no doubt due to the mix of people, from colleges, the NHS, businesses, as well as universities.
- Traffic lights give students general indications of their attendance and punctuality, or given on a unit level per-class basis.
- Teachers can see information about attendance on units, and the HoD can also see impressively detailed colour coded information about which units are performing badly or well.
- Teachers use the portfolio to set personal targets for students to improve which are available to the student.
- Previous achievements automatically dragged into the CV (e.g. GCSEs)
- Having student profiles means the student can put up their own information, which staff can see. In one case it helped a teacher understand why a student was performing badly, as they were training for a national swimming competition.
- When students are not attending, staff are shown a “recommended” traffic light status for the student, but can override this if there are exceptional reasons.
- Students and staff can compare this against previous year’s attendance.
I wonder if this could be a new project for MMU, in the next stage of the learning tech review…
Meeting with Andy Hardman
To find out what services we could develop in LRT to help SIPs, I had a meeting with Andy Hardman. Andy’s team in Geoffrey Manton have always been great to work with and I wanted to talk to them about what we could do to improve services to students with enrolment or VLE problems:
- Improve problem resolution at SIPs
- Increase communication between SIPs and the ICTS Helpline
- Give SIPs access to more knowledge and resources
- Allow students to identify for themselves what problems they have
- Stop situations where students keep being sent back and forth between SIPs and ICTS
The main idea I had for improving these areas was a wiki. This would work as a single knowledgebase specifically for enrolment and the VLE. So this would contain:
- Links to websites and resources
- Explanation of the link between the VLE and QLS
- Simple how-to guides
- Contact details – for SIPs, ICTS, local elearning champions, and programme offices
One of the most important for me would be contact details. Most problems are simple but because there is little communcation between different offices, they seem more complex than they actually are.
Urgent Enrolment Problem Web Part
However, after meeting with Andy Hardman I felt that the most important thing we could do in LRT is to improve the enrolment web part on my.mmu.ac.uk – which isn’t very prominent at the moment. It is currently on the MyAccounts page but there wouldn’t necessarily any reason for students to go to this page at the moment. So to improve it the service should be pro-active, highlighting to students on a prominent page if there may be a problem with their enrolments.
Easily achievable services, reporting an error to students if they are:
- Not enrolled on anything
- Not enrolled on any units, but is enrolled on a programme
More difficult but still achievable services, reporting an error to students if they are:
- Enrolled as PR or REG
- Student is set as Cancelled or Not Shown
Complex and difficult services, reporting an error to students if they are:
- Students are on the wrong unit
- Students are on an old code
Points 1 and 2 can be achieved fairly quickly through the existing web services. The points 3 and 4 are more complex as they require rewriting the web service. However if we can achieve points 1 and 2 then this should make a big difference to the SIPs. Instead of being stuck with a problem, students will go to a SIP already knowing what their enrolment problem is.
Point 5 will be hard to achieve – there is no way of knowing automatically if a student is on the right units. They need to be proactive and check themselves, or if they have a problem (such as with WebCT) they need to be directed to check. Once they have checked they can see they are on the wrong code and then go to the SIP.
Point 6 is difficult, there is no way of knowing if a student has been put on an old code. It’s possible to check if there are other students on that code (if they are the only student on it then it’s probably not a valid unit!) but only the programme offices know definitively which are valid codes.
So currently I am working on points 1 and 2, and have been in contact with Student Services about developing this service.
Some other suggestions that came up in the meeting:
- A service to link the Coursework Reciepting system into QLS
- Service to show the students with exceptional circumstances for assignment deadlines (later explained to me that this depends teachers entering into QLS the assignments they are actually teaching!)
- A reminder (in my.mmu.ac.uk?) for students to set up a “Secret Question” at the start of term so they don’t get locked out of their account
- A twitter account for relaying important annoncements to SIPs and ICTS etc.
- FAQs sent around to an email list of involved parties
- Training sessions for SIP staff on Moodle
- Because of the EQAL project there will be 450 Uniwide language codes… and thus a Moodle area for each one!
- Some arcitechure units will be difficult to made to work in EQAL… as they can be taught in many years, if students are on placement or not, can have different points attached to it, etc…
- Geoffrey Manton to feed into guides on using Turnitin
Meeting with Olivia Fox
After the Moodle Wonderland conference I caught up with Olivia from City University’s Learning Development Centre… fellow WebCT users in the past, and now fellow Moodlers.
The LDC alternative to spreasheets
They have a great office which is well kitted out, and I loved their alternative to using spreasheets to show what type of projects each member of staff had been working on:
So I picked up a few hints and tips:
- Use Jira to vote for new plugins to determine what staff want.
- Jira can also be used to consult staff on what help / FAQs they need.
- Check usage stats of tools to also determine what staff need training on.
- For the front page they had links to help files, but couldn’t embed videos. If we figure it out we should let them know…
- Have guides for intros, and then more advanced ones.
- The database activity is great, and is fully searchable. This could be used to help staff and students find content, as there is no other way to search within a course except for within forums.
- Use the Books activity to develop tutorials… but it’s a lot of effort if you try to make them look pretty!
- If you get enrolment data sorted for the move over, everyone will love Moodle
Gotchas:
- Staff don’t realise that the “news forum” plugin sends a message to all students by default! It drove students mad.
- Staff also didn’t realise that students can’t reply to the “news forum”
- The forums generally created a lot of issues.
- Groupings / groups… Groupings can be used to give access to files, groups are just confusing.
Plugins:
- The new official Turnitin plugin has fewer potential for user error than the UK one (such as using spaces and symbols in filenames), and also has anonymous marking facilities.
- The bulk feedback upload is enormously useful for admin staff. Otherwise they have to upload one for each individual student.
- The Single File Upload assignment has no mechanism to give students feedback! So she recommended the Multiple File Upload assignment.
- There is an attendance plugin they are looking at as it is required for Law courses to keep this information.
- They had a collapsable “My Courses” list… but this may have been a custom theme that City University created. It made the page look a lot better, and created a separate tab for previous year’s studies.
- The quickmail plugin is useful for sending a few students a message.
WebCT Chat and Chatzy
The WebCT chat tools haven’t been used a great deal throughout the university, however one exception has been the virtual writing school in the English department. Since the beginning of the year they have found that the chat logs seem to miss some students. So LRT has been looking into alternative chat tools for them.
After looking at the options, Chatzy seemed to be the best option, as it is simple, clear, and doesn’t require any Java. It works across browsers and allows a good amount of admin control and management. Purchasing a room at a minimum of $12 allows a custom URL and allows it to be embedded in websites such as a WebCT page. We subsequently found that the embedding doesn’t work across browsers but James from the English school still thought it was a good solution.
Updates From My Personal Blog- MMU W2C Blog
- Sharing at the JISC DVLE brownbag session January 11, 2012 mark
- Evaluation and reflection December 14, 2011 mark
- Sharing at the UCISA Future is Mobile event December 6, 2011 mark