188体育

1. Investigating public perceptions of assessment reliability

Ipsos MORI undertook two workshops, each with 36 members of the public, including teachers, students and parents. For these workshops the facilitators provided stimulus material about issues relating to the reliability of assessments, on which they attempted to gauge the group members鈥� views. A major finding from this work was the workshop participants鈥� apparent distinction between inevitable sources of variation in assessment marks and grades, and what they saw as preventable errors in marks and grades. Overall, participants seemed able to cope with the idea that 鈥榯est-related errors鈥� arise and are to some extent inevitable; they were less tolerant of assessment errors arising from blatant human mistakes. The use of the word 鈥榚rror鈥� itself was also seen as problematic, especially when used to cover all aspects of unreliability - because it tends to produce negative reactions and assumptions of culpability on the part of assessors and awarding organisations.

2. Communicating with the public about assessment reliability

Blue Rubicon鈥檚 research (conducted exclusively with Ofqual staff) attempted to address the issue of how Ofqual could communicate most effectively about assessment reliability issues with the general public. Blue Rubicon is a 鈥榗ommunications messaging consultancy鈥�, and so is skilled at helping organisations frame communications strategies. In Ofqual鈥檚 case the question of helping the public understand the concept of reliability was seen as an important, yet problematic, challenge. A major recommendation from this strand of work was that the word 鈥榲ariation鈥� was a more constructive way of communicating about the uncertainties surrounding assessment results than either 鈥榬eliability鈥� or 鈥榚rror鈥�. This constituted one of a number of recommendations on how Ofqual staff and representatives might communicate about this issue with members of the public.

3. The implications of the research

Rather than give an exhaustive account of these two specialist strands of work within the Ofqual reliability programme, the conference paper reflects both on the implications of this work when viewed alongside the outcomes from other aspects of the programme, and also on the public鈥檚 current perception of assessment results in the UK. The media are responsible for a large part of the communication in this area, and in recent years they have thrived on so-called crises and disasters, whether these are to do with the annual production of assessment results at both GCSE and A level or with national curriculum assessments. Clearly, the media鈥檚 concerns are different from those of awarding organisations, naturally tending to raise doubts about assessment results, whereas awarding organisations will generally seek to increase the public鈥檚 confidence in the assessment results 鈥� which the agencies are responsible for producing.

The paper outlines the considerable technical knowledge about the limitations of assessment scores and grades as precise measures of achievement, and the very real difficulties faced by those who seek to communicate such insights to the public. It reiterates the view that awarding organisations should take central responsibility for estimating the degree of variability in assessment results. The paper concludes, positively, by asserting that a wide variety of media and analogies could be used to get the message of the complexity of the technical data across, and by holding out the hope that following such a strategy could make the UK public 鈥榤ore sophisticated鈥� users of assessment results.

Had the presentation returned to its provocative 鈥楴o news is good news?鈥� title, it might have concluded that there are ways of conveying news about the limitations of assessment results that are not necessarily 鈥榖ad news鈥�. The presentation does, however, suggest that Ofqual is determined to 鈥榦ut鈥� this issue, which has, for many, been a 鈥榮keleton in the cupboard鈥� for far too long.