|
Nigel,
thanks for the stimulating comments in your review. It's interesting
that the concerns you raise about CBM are all about use in summative
assessment. Our main objective has been to improve students' habits of
thought and study while learning (formative assessment), and it was
surprising to me, when CBM was used in UCL exams, that the concerns you
express about summative assessment simply didn't materialise.
You ask about standard setting. As we show in the paper, CBM can
generate a score (CBS) that is on average equivalent (for both T/F and
best-of-5 MCQs) to the % correct above chance. I think you may have
misunderstood this "% correct above chance" concept - known perhaps
more simply at Imperial as "% knowledge". This has nothing to do with
confidence ratings, but is simply a way of expressing the conventional
score ("number correct"), scaled so that complete guesses would give 0%
and perfect scores 100%. For example, on a T/F exam guesses would yield
on average 50% correct, so a passmark of 75% correct answers
corresponds to "50% knowledge" (half way between guesses and
perfection). Similarly with best-of-5 exam Qs, guesses would on average
give 20% correct, so it is 60% correct that corresponds to "50%
knowledge". Whatever passmark you set based on criterion referenced
standard setting procedures, if you scale this to "% knowledge" then
the equivalent CBS passmark will be the same, as shown in the empirical
relationship in Fig. 2.
Yes, a weak but confident student may occasionally do well through
luck, as with any assessment scheme. The nature of CBM means however
that both in theory and in practice a weak student will lose by
claiming confidence where it is not justified. A strong student will
also lose through inappropriate diffidence (too often choosing low
certainty levels). But experience shows that practised students become
well calibrated to use certainty levels appropriately (Gardner-Medwin
& Gahan, 2003: http://www.ucl.ac.uk/~ucgbarg/tea/caa03.doc ). The
students who do best under CBM, with the same number of correct
answers, are those with better than average ability to identify which
of their answers are reliable and which are based on uncertain
knowledge or reasoning. Knowledge, after all, is not just a matter of
being right but of knowing that you are right.
The query about whether CBM enhances stress is interesting, and I
hadn't encountered it before. It doesn't seem likely to be a major
student concern, since in a 2005 survey, less than 1/3 of UCL students
with experience of CBM in year 1, 2 exams voted to drop CBM from exams.
Imperial College
cautiously allowed extra time when it first introduced CBM into
formative exams, but it became clear that extra time was not needed. It
is an interesting fact about the brain that when it comes up with an
answer, this seems to come packaged with a certainty judgement. The
judgement may be wrong, which can in many walks of life be a disaster.
But it can be refined or corrected by reflection. CBM helps to train
students to make such judgements carefully, correctly and realistically
and rewards them accordingly.
Does CBM assess reasoning? Certainly CBM can be used with questions
that require reasoning. CBM puts a premium on care in such reasoning.
Many of our incoming students - I see this especially in maths for
medical students - have got in the habit of giving answers with very
little thought. Since they are highly selected students, these answers
are often right. But at university, this simply isn't good enough. If
it were possible, we would of course always like to assess and comment
on the quality or steps in a student's reasoning, and not just base a
mark on the final answer and confidence expressed for whether it is
right. But this requires 'hand' marking, and is not always possible.
CBM motivates extra care in reasoning, realistic self-appraisal and
checking. There is surely no way that reflection about the quality of
reasoning is in any way different or less valuable than reasoning
itself.
Thanks again - Tony G-M
|
