One way to approach evaluation can be to try to define what would indicate that an intervention or change has been successful. This could be successful from the perspective of a project team, the intended recipients of the intervention, or those paying for it, among others. In designing the evaluation of ocTEL (open course in Technology Enhanced Learning), I've spoken with the course team, reviewed proposals and planning documents, but it's also been interesting to see how other people have defined success for MOOCs  (Massive Open Online Courses).

There is frequent mention of completion rates, and whether or not they are a useful indicator. Dominik Lukeš provides an insightful post on how complexity science demonstrates that, given the effects of magnitude, drop out rates in (Massive) MOOCs and more traditional (small) classes aren't really comparable. Dominik goes on to reflect on his own MOOC experiences. He notes that xMOOCs (Coursera, edX) focused on knowledge and skills, with peer interaction simply being a support for these. While the cMOOCs (Connectivist) he took part in were more interactive and exploratory, with peer relationships being a strong motivator for many.

Reading his post made me think that to evaluate a MOOC perhaps we need to identify some models of participation. Indeed, in his research into one MOOC, Colin Milligan identifies:
“lurkers who purposefully did not engage with other course participants … passive participants, who expected ‘to be taught’, and viewed the course as a source of information … more active participants, who set their own goals, established connections with other learners and linked these connections with their existing personal learning network.”

So there are some not entirely unexpected patterns of participation, but what factors would influence these in a MOOC? Perhaps, number of learners, duration of the course (days or weeks), motivations of learners, processes for recognition or accreditation, what tools were used for interaction, whether the MOOC is about knowledge or 'the experience'.

At the end of his post, Dominik concludes that "we need to consider the impact dropping out of a course or education has on the individual. And the impact the dropping out of a large numbers of individuals will have on the whole group." (My emphasis.) This impact on the individual is picked up in a more recent post from Sheila MacNeill on Badges? Certificates? What counts as succeeding in MOOCs? Sheila highlights how in talking to participants from #oldsmooc and #edcmooc they all felt they had "gained something from even cursory participation". Indeed, being able to participate in a way that suits their own needs, sometimes strategically, sometimes without worrying about dropping out seemed to be key. I think Andy Powell sums it up in the comments, where he says "your [personal] measures of success depend on your reasons for taking a MOOC in the first place". Though learners have to be able to identify those reasons, and that comes up later.

For ocTEL, the evaluation will focus around a number of themes. These aim to address the individual level, but are also very much informed by the original intent of ocTEL, ie to help participants better understand how to use technology to enhance their teaching in Higher Education. The themes and associated evaluation questions are thus:

Impact on staff competency (value in practice)
1. To what extent has ocTEL helped participants make better use of technology in their teaching?
2. Has ocTEL helped participants solve their own problems, thereby adding long-term value?
  - To what extent have participants’ expected outcomes been achieved?
  - How valuable were the outcomes to participants?

There is also an underlying question of how prepared or how effective participants are as self-directed learners, which is likely to be influenced by prior experience of learning in MOOCs, and ability to set goals around problems to solve & ability to identify expected outcomes. In a post on Designing MOOCs, Colin Milligan also talks about the effect of having pre-existing networks and a certain level of digital literacy.

ocTEL Content and Design
4. What was the quality of ocTEL’s content and how well was it designed?
  - How useful is the content and design of ocTEL?
  - Are/were participants' level of experience of Technology Enhanced Learning sufficiently accounted for in the content and design of ocTEL?

Labels: #ocTEL, evaluation, MOOC, motivation

The Jisc Institutional Approaches to Curriculum Design programme set out to develop innovative technology-supported approaches to curriculum design, approval and review. The programme was perhaps unique in providing four-years of funding, and this duration presented some challenges in planning for evaluation. Added to that the focus on technology-supported curriculum design was not well documented in the literature, again a challenge when planning evaluation activity. Nonetheless, Jisc made clear that project teams should undertake in-depth baseline and ongoing evaluation activity, although the actual process or methodology for doing so was up to project teams to decide. Indeed, the projects went on to use a wide range of approaches.

I was the 'evaluation critical friend and supporter' to projects, and have recently posted an overview of the philosophy or approach to evaluation that different projects applied on the Jisc Design Studio. (The latter is a wiki containing a huge range of resources connected to the wider e-learning programme.) The outline of What the Curriculum Design projects tell us about evaluation also includes a selection of evaluation-related advice drawn from projects' experiences. This points to advice on where to look if you need help with:

Labels: #jisccdd, Curriculum Design, evaluation, researchmethods, researchtools

This year, rather than sending physical Christmas cards, I’m making a donation to the Al Ahli Arab Hospital in Gaza. Just to maintain the Christmas cheer though, here's a very merry card, courtesy of Calum.

I have recently started working with the ocTEL team looking at how to evaluate the Massive Open Online Course or MOOC they'll be running next year. (ocTEL stands for Open Course in Technology Enhanced Learning, there's an outline of the course on the ocTEL blog.) There is a *huge* amount of discussion on MOOCs at the moment from tips for those taking a MOOC to the future of MOOCs and HE. This article from the Chronicle provides a rather US centric timeline of developments and responses to MOOCs. There is also the recent launch of the slightly mysterious Futurelearn in the UK. Some very interesting discussion on this on the ALT mailing list. The list is members only, but quoting from a recent list posting by Diana Laurillard "Everyone in the field knows there's nothing new in MOOCs" but we do need to meet the "massive demand for education, across all sectors from primary to lifelong, all over the world... It can't be done without technology... It's time to start looking seriously at what those models could be."

For me at the moment, the focus is on evaluating just one MOOC, which is still under construction. Nonetheless when I started asking the ocTEL team what they hope the MOOC will provide, one of the themes that emerged quite strongly is the desire for a sustainable community to come out of the MOOC. This has led me back to work I undertook over ten years ago on the theory and practice of online learning communities. At that time, I was influenced by Etienne Wenger's insights into Communities of Practice and Jenny Preece's perspective of designing usability and supporting sociability in online communities. Figure 1 summarises the dimensions of practice Wenger attributes to 'community', and Figure 2 shows the relative similarities in Preece's online community key features.


These models will be reviewed and elements drawn into the ocTEL evaluation framework. I'll also be looking at the concept of sustainability in relation to community. This has been addressed before by Bell et al in their 2007 book chapter entitled Evaluation: a link in the chain of sustainability. They highlight how the lack of persistence could come down to coordination failure or when the"costs of participation exceed the perceived benefits".

So perhaps there is nothing new in MOOCs, or maybe it's just a reminder to put old lessons into practice.

References
Bell, F. et al (2007) Evaluation: a link in the chain of sustainability. In: Lambropoulos, N., Zaphiris, P. (eds.) User-Centered Design of Online Learning Communities. Idea Group Inc.
Preece, J. (2000) Online Communities: Designing Usability, Supporting Sociability. Chichester, John Wiley and Sons.
Wenger, E. (1998) Communities of Practice. Learning, meaning and identity. Cambridge University Press, Cambridge.

Labels: #ocTEL, community of practice, education, evaluation, MOOC, online community

At the recent JISC Assessment & Feedback programme meeting, I ran a session with the Strand B projects in which we revisited some questions I first put to them a year ago. Thus, instead of 'What do you want to happen as a result of your project’s assessment and feedback innovation?' we discussed what has happened. And, rather than 'How will you know the intended outcomes have been achieved?' we discussed the indicators and evidence that projects have actually gathered over the last year. These are particularly relevant given that Strand B projects are all about Evidence and Evaluation of assessment and feedback related innovations.

The questions were really just to kick start discussion, although the Strand B project teams are such a keen group they didn't need much encouragement! In fact, we had a very open discussion, and what emerged were some of the issues and benefits of evaluating large-scale changes in assessment and feedback using technology, as well as some interesting findings.

All the project teams want to gather a balanced view of the changes being implemented within their institutions, but many had issues with collecting data from 'reluctant users'. In other words, individuals who are reluctant to use a given technology can also be difficult to involve in the evaluation process. This is by no means unique to this context, or to evaluation. Indeed, some projects found that reluctant users also tended to be less likely to take up training opportunities, something that might only be picked up later, when difficulties with using the technology arose. This really underpins Ros Smith's reflections from the programme meeting on the need to open a dialogue with course teams, so that implementing these kinds of changes is as much about working with people and cultures as with technology. Being ready to capture the views of those who are having difficulties, or offering a light touch evaluation alternative for reluctant users might be options that provide a more balanced stakeholder perspective.

For some projects, the evaluation process itself had provided the push for lecturers to engage with online assessment and feedback tools. In one case, a lecturer who had previously noted that 'my students don't want me to use this approach' took part in a focus group. During this, the lecturer heard direct from students that they did want to use online tools for assessment. Needless to say the project team were delighted that the lecturer went on to trial the tools.

Effective training of staff was also picked up as essential, particularly as how lecturers go on to communicate use of tools to students, influences student uptake and use. This led on to discussions about the importance of training students, and how evaluation activity can help in understanding how well students interpret feedback. Essentially, ensuring that students are gaining the most from the feedback process itself, and not having difficulties with the tools used to support the process.

What surprised a number of projects was how the evaluations had picked up strong emotional reactions to assessment and feedback both from students and staff. There is a wider literature that looks at "Assessment as an 'emotional practice'" (Steinberg, 2008) and this is underpinned by studies into the links between learning identities, power and social relationships (such as this paper by Higgins, 2000). While the Strand B projects might not have set out to study emotional reactions, it seems there will be some interesting findings in this area.

The importance of relationships was also reflected in findings of a mismatch between students and lecturers in terms of perceptions of the intimacy afforded by online and hard copy assessment and feedback. Staff felt closer to students and more in a dialogue with them when marking hard copy. They wanted to sign or add a personal note to a physical piece of paper. While students felt more enabled to engage in a dialogue online, perhaps because this was felt to be less intimidating.

During the meeting we also discussed the methods and tools projects have been using for their evaluations, but that will be the subject of another blog post.

Labels: #jiscassess, assessment, emotion, evaluation, evidence, feedback