University of Chichester Benchmarking

David Booth attended the cohort meeting at Woburn House on Monday 29th October. (John Scriven apologies for absence) The morning session began with updates from all the institutions about their progress so far.  I gave a brief update from the University of Chichester:

1)    The roll-out of the online survey for staff & students

2)    Paper-based questionnaires for staff & students situated in strategic places at both Bishop Otter & Bognor Regis campuses

3)    Interviews of key personnel – Senior Management & Heads of Schools

4)    Big Brother Video Booths at Bishop Otter and Bognor Regis campuses (many institutions showed great interest in this and wanted to know about how we set them up, I did promise to send out details as soon as possible)

5)    focus groups 

One of the general themes that came out of the general roundup of updates was assigning scores to the twenty core and selected supplementary criteria.  Preparation for the scoring meetings was emphasised – prepare and prescore being the operative words.  Either using an easily accessible ‘table’ with commentary, evidence, cross reference to interlinked documents on the institutional network; or another version would be a spreadsheet ordering and grouping criteria with the evidence accessed in different folders.

There was an emphasis on benchmarking being an internal reflective exercise, and several institutions made the point that this is Art not Science.  Discussion centred on various approaches to scoring slices as well as the institutional scoring.  Scoring would not reveal the entire picture; thus the importance of the narrative to accompany the report. 

The HE Academy is not comparing institutions, but looking for signs of excellence in elearning.  The importance of the final report was emphasised … to raise the profile of elearning; providing evidence for strategic influence and strategic development.  The final report will become a catalyst for change, engaging all stakeholders involved in the enterprise.

Jane Plenderleith, gave a briefing from EDSuT and mentioned a ‘summary report’ to be delivered to HE Academy by 15^th Jan 2008 – respondents were encouraged to ‘be candid’.   EDSuT have produced a draft programme for the Benchmarking Phase 2 meeting scheduled to take place in Bristol on 24 January 2008. It marks the culmination of the formal Benchmarking exercise supported by the Academy and JISC, and provides a forum for a review of Phase 2 in particular and the benchmarking exercise in general.  

 In the afternoon session Paul Bacsich made some observations about the importance of keeping blogging, the use (or non-use of Helga in its ‘community’ sense); look to the final report and what follows on from Benchmarking in the future … 

The importance of getting senior management to buy in to elearning and maintain the sustainability of the project is a vital element in the process of the project.  So that there is a continuous improvement, in mid-end January 2008 a reflection exercise how each institution can take forward the benchmarking experience through the involvement of senior management into a culture of change alongside staff development and the encouragement of future elearning initiatives. 

There were several other ‘issues’ discussed during what was an eventful and rewarding cohort meeting: ‘learning object repositories’ in focusing and articulating much better ‘bids’ for funding future elearning projects in the HE sector (maybe a further blog is in order to discuss in more detail) and my favourite comment – I believe it was Paul Bacsich – who suggested looking at scoring criteria through constellations of different perspectives. 

David Booth

Today, having completed last-minute revisions to the questionnaires and finalised arrangements for the Big Brother Video Booth (yes, really!), CLT launched Chichester’s e-Learning Benchmarking Project. A small celebration was held with mince pies and elderflower cordial.

Although there are several weeks of interviews and survey analysis in the next couple of months, there was quite a sense of achievement and thanks are due to all those within and without the University who have helped so far.

The choice of Bristol University’s online survey software has simplified the task of producing effective web-based versions of our questionnaires. The statistical tools available will ensure that analysis is much easier. As an added inducement, we are offering a prize draw for both surveys, with prizes of vouchers totaling several hundred pounds so I’m hoping staff and students will enjoy taking part in the exercise both during and after the event.

The decision to use the annual Learning & Teaching conference in January to showcase the Benchmarking reports, followed by an e-Learning Semester should make sure that e-Learning is high on everyone’s agenda throughout the whole academic year.

John

The last two meetings have finalised the research tools we are going to use. They are:

• Analysis of key documents
• Staff survey
• Student survey
• Structured interviews of key staff
• Focus groups
• Big Brother Diary Room videos (organised by the Student Union on both campuses)

It has been decided to use the online survey system developed by the University of Bristol. Chichester already has a licence and the package offers a clear methodology for creating quite complex questions as well as excellent means of analysing large data sets. Even in our modestly-sized institution, the prospect of a potential 5,000+ paper questionnaires would prove daunting. Having piloted the staff and student surveys, we can certainly recommend the system – more details here: http://www.survey.bris.ac.uk/

The timetable, stretched across 4 months, has begun to look like this:

This structure provides the flexibility needed, as well as adding variety in the range of tasks undertaken. More importantly, it spreads the workload to avoid receiving a deluge of data at any particular time. We are hoping to launch at the start of October, although the final date will depend on other events at the University – we certainly don’t want this to be relegated to second place!

John

Last weekend saw David & myself travelling to University College Falmouth for the second of our cohort meetings – hence my panorama above, which shows that we were experiencing better weather than the previous two months have produced – even warm enough for a swim.

Dinner on Saturday proved extremely useful in renewing contacts the BELA team, as well as colleagues from Anglia Ruskin, Bolton, Northumbria, St Mary’s, UC Creative Arts, Worcester and of course our excellent hosts at Falmouth. The meeting on the Monday was useful in clarifying the tasks still remaining and to hear the progress of the other institutions. Questions concerning evidence gathering and data analysis were also answered by BELA.

In the afternoon we broke into smaller interest groups to form what I felt to be the most useful session, joining colleagues from Northumbria and Anglia Ruskin to discuss the bigger picture of Benchmarking, what it will mean to our institutions and how to cope with potential problems. The session was ably chaired by Bruce Carter from BELA.

The meeting as a whole was certainly worth the long journey and we are better prepared to begin the exercise in the coming semester (albeit with some trepidation at the amount of work involved).

This post will be expanded when I’ve recovered from driving back to Southsea last night – David’s train journey to Worthing was apparently due to be routed via London….

John

This meeting will be almost the last until September so the brief agenda was to complete necessary actions before the break.  We began by checking through the questionnaires and correcting typos, etc.  Version 2 of this document can be viewed under the ‘Documents’ tab above.  We also finalised the list of those stakeholders who will need to be interviewed.  They are:

• Vice-Chancellor
• Pro Vice-Chancellors
• Heads of Schools
• Director of Information Services
• Manager of IT Services
• Head of the Centre for Learning & Teaching
• Head of Marketing & Admissions
• Head of Planning & Campus Services
• President/Vice-President Students’ Union 

David & I will look after this task. 

This was followed by a discussion of timetables and activities for the autumn semester.  At the same time as the BM exercise, the e-Learning Semester will be running.  Activities designed to keep the profile of e-Learning raised will need to be finalised.  As we need to discuss this with Phil, as Head of the Centre for Learning & Teaching, we’ll have a short impromptu meeting when he returns.

The final section considered roles during the semester.  It was decided to hold a meeting of the Centre for Learning & Teaching in September to brief the whole team.  The School reps will be essential for coordinating the administration of the questionnaires.

Tasks remaining in August:

• Finalise timetable of events (BM exercise)
• Finalise timetable of events (e-Learning Semester) 
• Develop structured interview questions
• Bring Project Plan up-to-date
• Develop online questionnaire
• Build results database
• Collect documents for analysis
• Review P&M criteria list to include possible additions
• Review decision on the use of slices

At least the sun is shining again….

John

No meeting this week as Phil is in France. Completed some editing of the questionnaires – need to test them on a pilot group in September. Took care of the blogs and I’m now sorting out an agenda for next week’s meeting, as I anticipate a ticking over period during August.

The EDSuT evaluation team use Helga as a social learning environment for the BM exercise, which uses ELGG.  So far it hasn’t been quite so overtly ’social’ but this seems likely to change.  ELGG has a lot of potential as a support environment for e-Portfolios and appears to work well with Moodle.  I’d like to explore the potential of the triumvirate of applications at some stage – Moodle, Elgg & Drupal. I have Drupal on my server, following David’s comments and it’s looking good – however in the world of Content Management Systems, it’s difficult to identify exactly which ones have a future in HE. Just looked at Open Source CMS and there are about 50 to try! Think we all could do with a Which! guide to software….

John

As promised, we spent today looking at our edited versions of the questionnaires. Phil & I had distributed ours via email and David had looked at a questionnaire from the University of Plymouth as a comparison. With one master copy projected on the wall, we were able to discuss and agree edits and I was able to take away rough copies of the final version, together with last minute notes of things to be added. To a large extent we were in agreement – problems only occured in deciding what to leave out, as we want to avoid making it appear too large a task. We were also pleased to see Bev today, who dragged herself away from PhD writing, as it’s always useful to have a perspective from our full-time academic colleagues.

The final (almost) versions will be included with this posting so that all of the team can check through these and final (really final) versions produced in September/October. We also need to decide on appropriate prizes for the draws.

I’ve just received a reply from the two sabattical student officers, Matt Gover and Alvin Ramsamy, indicating their approval of the questionnaire. Matt also suggested we use Pulped (our student newspaper) for the exercise and possibly use the Zee Bar for our Big Brother video booth….

(Eerily quiet at this time of the year!)

John

Documents

Student Questionnaire
Staff Questionnaire

This week’s meeting concentrated on developing tools to use to obtain evidence within our institution. For this we are truly indebted to the University of Chester for their sterling work in producing both staff and student questionnaires. These have been useful in determining which criteria were satisfactorily covered by documentary evidence, interviews and focus groups and which would need to be investigated by other means.

As we are concerned that the Benchmarking exercise produces both a reliable and a valid snapshot of e-learning at Chichester (clearly a waste of time if it isn’t!), we are aware that we need to get the best information from the end users, both staff and students. Today, we went through Chester’s questionnaires, deciding which questions were useful, which weren’t relevant and which areas needed expanding. It seems that all questions are relevant and there are some additional items needed, so this week’s homework will be producing our own revisions so that we can finalise details next week.

David and I have also met the two sabattical student officers for next year, in order to brief them on the project.  We were pleased to get some positive and enthusiastic feedback from them and look forward to working with them next semester.

John

A more general narrative about the cohort meeting has been posted on the HE Academy/ JISC weblog by Paul Bacsich. It gives a flavour of the discussion topics on the agenda – covered in John Scriven’s earlier posting. Although further explanation may be in order to explain the use of carpets in elearning … ‘There was then a group Q&A session on carpets, criteria wording, supplementary criteria and slicing issues …’ A Pick&Mix glossary is available for understanding those terms that educationalists metaphorically love to conjure. So, in order to learn about carpets please follow the link here.

To add to John’s previous posting, it was an excellent meeting and comforting to know that we are progressing very well, it was also the last sunny day – free from rain – I can remember.

David Booth

The P&M cohort are fuelled only by expensive French water!

A suprisingly dry day in London, made all the more pleasant by a comfortable train (until recently, an impossibility in the former Southern region). So nice that I walked to Tavistock Square in time to find they’d run out of coffee….

Still, the meeting was productive, with Paul Bacsich introducing the documents he’d just made available – P&M criteria now much clearer. After a round-the-table session collecting feedback from all the universities present, I think David & I felt we had a better grasp of P&M – and also that we were up to speed with the others involved. Rather than duplicate the proceedings, I shall take the liberty of adding BELA’s documentation from the Helga site (see Documents below).

After lunch, we arranged another meeting with Dick Hill at Chichester on September 24th. That’s just two weeks after the meeting we’re having in Falmouth.

Meetings like this, efficiently run, with a good buffet and pleasant company, generate the necessary good feeling that helps to make the hard work worth while. (I even managed a couple of guitar shops in Charing Cross Rd on the way home!)

John

Documents

BELA report – cohort mtng 9th July 2007

John