Adapted from an original post on the JISC Curriculum Design and Delivery Blog

Over the last three and a half years I've been working with a group of Higher Education projects who are implementing large scale changes to curriculum design. This JISC funded programme is in its final stages, so the focus is on finalising evaluation activity, but also how best to share what has been learned by project teams. This led to a webinar that was primarily about Telling projects' stories. The ever insightful Helen Beetham led the first half of the webinar and started by asking everyone to suggest films that their project’s reminded them of. Suggestions ranged widely, from Steptoe and Son, which I guess might be a tale of partnership, careful reuse, some deception along the way, and the value of insurance (or JISC funding?) To Monsoon Wedding, which picked up on the experience of “many interweaving stories and different (design) languages being spoken that (hopefully) all come together at the end“. While the programme was seen as something of an epic, it is perhaps not surprising that many people also talked about personal transformations, the influence of changing contexts, and cultural change within organisations.

Capturing the story of cultural or organisational change was, however, seen as no easy task. A number of approaches were discussed. For example, some projects are interviewing stakeholders to capture individual’s perceptions of change. This was seen as useful as it provided a space for individuals to stop and reflect, something that often isn’t possible when everyone is busy working on the project. During the webinar, I made some suggestions as to the kinds of questions it would be interesting to discuss with stakeholders.

This included background questions on the individual’s role, how he or she became involved, and the amount of effort put in to the project. Stakeholders could also be encouraged to reflect on local contextual factors that were important to the project, including those factors that acted as drivers, and any barriers they had encountered. Questions about how life within the institution had changed since the start of the project could pick up on a number of themes including changes to cultural norms, access to information, or the language used regarding curriculum design. Stakeholders’ perceptions of projects’ main achievements, and the impact they think projects will have in the longer-term would also be part of the story of the change. Thinking about the future, Paul Bartholomew from the T-SPARC project suggested it would also be valuable to consider what “opportunities have been missed, or remain unexploited“.

If a number of projects take this kind of approach, it will be interesting to see whether there are any commonalities from these individual stories, within projects and across the programme.

A number of attendees saw changes in language as a particularly interesting facet of cultural change. Rebecca Galley from the OU Learning Design Initiative felt “shifts in language and focus in strategy and curriculum design documentation … and stories of individual’s transformation” should both be captured.

Other options for capturing organisational change included looking at working practices, awareness of curriculum design processes and procedures, the level of discussion of curriculum design at all levels, evidence of joined up working, and whether issues or barriers identified at the baseline stage had shifted.

There was a recognition that sometimes changes to the system can make people feel a bit uncomfortable possibly because, as intended, the system is delivering a greater level of scrutiny. These kinds of negative positive outcomes can be more difficult to sell. Highlighting that ’satisfaction’ measures can sometimes mask the real outcome.

Making these cultural shifts visible and telling the story of change in an appealing manner is a challenge. Projects within the curriculum design programme have already made good use of video to bring in different voices, and no doubt there will be more voices shared in this way. The institutional narratives that form part of projects’ final reports will also be valuable in enabling projects to reflect on their role within the development of their institutions over the last three and a half years.

Just to show that some things don’t change, Steptoe and Son, or at least the Rag and bone man, is still alive and well in Croydon. Photo by Peter G Trimming, May 2011.

Labels: Cultural Change, Curriculum Design, evaluation, HE, JISC, Stories, transformational change

Thanks to John Gargani for sending this EVALentine’s Day card. John's artwork is inspired by a 1973 Valentine’s Day love stamp, the gingerbread heart is simply inspired by my love of chocolate!

As John suggests, why not find out more about evaluation from the American Evaluation Association, the AEA tip a day is a good place to start. Or closer to home for me, is the UK Evaluation Society.

I recently spent some time working up an approach to analyse a substantial number of responses to a public consultation. Due to a change in policy, the analysis did not go ahead. However, I think I learned a few things in the work up stage, and it seemed worth sharing these.

Getting at the data

Upload/Download (Photo credit: meganpru)

In this case, the first hurdle would be retrieving and collating over 300 individual responses that were due to be posted on the web. In terms of managing this kind of quantity of data, the way that the responses are made available can make a huge difference. Direct links to individual responses on a single page makes it possible to automate the download process, as opposed to side links to individual download pages. Something like the add-on DownloadThemAll, which can be used to filter and capture specific types of file links (e.g. pdf, doc, gif), as well as speeding up the downloading itself, can really help here. Thanks to Martin Hawksey for pointing that one out!

How individual responses are named is also worth considering. In some cases, numbers are used that are not necessarily related to the individual respondents or their responses. Respondents’ names might be given alongside download links, or it may be necessary to open the response to find out. Generally, the names are needed, if only to sort responses according to the 'type' of respondent. It may be possible to rename during the download process, and it’s worth experimenting with DownloadthemAll for this, as renaming manually is quite time-consuming. An alternative would be to generate an ID table, which matches response numbers to respondents.

Checking you're going in the right direction
It's likely you'll have an understanding of the kinds of terms or key words you want to focus the analysis on. This would probably be derived from the original consultation document itself, and will be influenced by the interests of the organisation sponsoring the consultation analysis. With access to a large team or limitless time, it would be possible to review these key terms iteratively as you go through the detailed qualitative analysis. However, assuming you don't have this luxury, there are options to check your key terms are (mainly) the right ones.

For a start, automated word analysis can be undertaken using a CAQDAS package*, such as the MAXDictio component of MAXQDA. This could either involve a complete analysis of all words used across all responses, or just those respondents of particular interest. Prominent terms can then be pulled out of the resulting frequency table (example on the right). Alternatively, the pre-identified key terms can be used. This automated step essentially checks if the identified terms matches the key terms being used by respondents.

The original text surrounding these terms can be ‘jumped’ to within MAXQDA. Manual coding is then required to specify the relevant sentence or paragraph that provides the context for the key term. Although a degree of autocoding of the surrounding text is possible so long as response files aren't pdfs.

* Other software provides similar functionality, and the University of Surrey provide a useful overview on choosing a CAQDAS package. Dedoose is a web-based tool that is new to me, but apparently it can be used on Android and tablet devices.

Part II - Targeting the analysis (coming soon!)

Labels: CAQDAS, Consultation Analysis, Qualitative Analysis, researchmethods, researchtools

Late last year, I completed an Insight Report with Seb Schmoller from ALT. The report was part of a study looking at the extent that the Learning Design Support Environment project (LDSE) had proved its concept. In particular, the LDSE team were interested in whether learning design support tool for teachers could support collaborative and sustainable development of innovative and effective applications of learning technologies. And if so, under what circumstances.

For the purposes of the study, a "learning design support tool" was defined as a software application that assists teachers in planning and designing their students’ learning (i.e. teachers' "design practice"). This could range from individual activities to whole lessons, to larger units of learning, such as modules or courses. Learning design support tools are also known as pedagogic planners. The LDSE project tool is called the Learning Designer.

If learning design is new to you, the Learning Design toolbox on Cloudworks has an extensive listing of activities and resources, including the Learn about Learning Design Guide (doc). While the latest issue of e-learning papers (no. 27) focuses on designing for learning.

The approach for our study involved an adapted version of scenario planning, and started with interviews with members of the LDSE team. The transcripts from these were used to develop an online questionnaire that asked respondents to rate key scenarios and uses of learning design support tools, the value of a range of features for different stakeholders, and likely influences on uptake for different stakeholders.

Nine scenarios, or potential contexts of use, were derived from the interviews, published literature, and wider discussions with LDSE team members. Respondents to the subsequent survey were asked to rate the likelihood and value of the range of scenarios. The most favourable contexts in terms of perceived likelihood and value were the scenarios where learning design support tools are used in a staff development context.

When asked about the value of different features of learning design support tools, ‘A learning design tool that is informed by models of what makes a good learning experience’ was unique in being seen as having considerable value across all stakeholder groups (learners, teaching staff, departments, and institutions).

The full findings of the study are available to download from the LDSE project website.

Labels: elearning, LDSE, Learning Design, Scenario planning

Labels: assessment, evaluation, JISC, slideshare