Design and management of an international, collaborative evaluation programme
Early in 2013 a trio from the CE team won a tender to conduct an international pilot programme evaluation in two phases for the British Council. The first phase involved desk research, first hand observation and exit surveys for around 300 people attending the workshops, mostly in wider Europe and Africa. For the second phase we needed to devise a profile group that would track around 20 participants for several months after they’d attended, to assess the impact of the experience on them and their working practices.
Working with our clients we identified five key challenges for the project:
Stuck on the surface. Asking questions in a way that would bring out the Second Wave of real stories of impact (or lack of impact), even where these stories are hard for people to notice for themselves because at quite a deep tacit level of know-how or instinctive action.
Exclusion. Working across Europe and Former Soviet Union, the evaluation would have to negotiate a range of languages - and many would not speak English. With this in mind, we would have to find a balance between use of English as a lingua franca and ensuring that those who had stories to tell but could not speak English were not excluded.
Lack of strategic purpose. To ensure that the project was evaluated in a strategic way that could make meaningful recommendations for the future, rather than simply an exercise in marketing or satisfying funders.
Bridging people, places and time. We were keen to see this evaluation not only as a standalone initiative, but a means to create relationships, momentum and international collaboration in an international volunteer profile group. This was even more challenging since we knew the profile group would remain across different countries - people who’d never met and would never meet.
Hold Empty Space. This pattern was particularly important to ensure we accessed deeper insights and strategic recommendations - resisting the temptation to jump directly to statistics and forms, but instead to create open opportunities for discovery and reflection with everyone involved in process. This was done through a range of techniques including use interview timelines, open interviews, open blog formats, evaluator journals.
Triangulation. Throughout, we sought to enrich networks an insights by designing processes and discussions in a way that established the new triangles of space made possible by the work. So, for example, two participants in an audience development workshop come up with a new idea because of the stimulus provided by the trainer; or, in the case of the profile group design, the relationship between British Council, trainers and participants or between central and local British Council and participants each become triangles of space to make visible and explore. This is an extension of thinking about Paperspaces as a ‘third’ space that sits between participants and makes visible and records their discussions.
Love the small things. The evaluation was designed carefully to accumulate small details throughout. This was reflected in the way we undertook detailed with a profile group, breaking down larger questions about impact into very small sub-questions, to illuminate some aspect of the creative, practical, collaboration and felt impact of the experience. We've also brought these tiny sightings and exchanges into a reporting dynamic, so that the reporting quite deliberately a process that unfolds and engages, as much as a product.
Wordpress, voting polls, forms, free blogging themes, Twitter hashtags, Wordpress widgets, Emailer software, local Arts Managers.