NB You must allow POPUPS from UCL. Click here to check.
Email to email@example.com if anything doesn't work.
(London Agreed Protocol for Teaching)
uses Certainty-Based Marking (CBM - see below) in the presentation of learning resources.
It was developed in part with a HEFCE grant to encourage uptake of CBM.
It is freely available, and can be used with material hosted elsewhere.
Tony Gardner-Medwin (homepage)
Confidence-Based, or Certainty-Based Marking (CBM)
Experiments making use of confidence judgements in marking tests are many decades old. They have repeatedly been shown to stimulate
more careful thinking and learning than simple (right/wrong) marking, and to provide more reliable
assessment. LAPT aims to provide the best possible implementation of CBM as a learning tool, for open use.
is Certainty-Based Marking (CBM)?
Why use CBM?
- After each answer, you indicate your degree
of certainty that your answer will be marked as correct.
- In LAPT, this is on a 3-point scale: 1 (low), 2 (mid) or 3 (high)
- We deliberately do not use words like 'sure' or 'very sure'
because these mean different things to different people
- The mark scheme and the risk of penalties will guide your
choice of C level: see below.
does it work?
- To make you think about how reliable your answer is
- To encourage you to try to understand the issues, not just to
react immediately to a question
- To encourage you to think laterally: other pieces of knowledge
may help to validate or question your answer
- To challenge you - if you won't risk losing marks if wrong, then
you don't really know the answer.
- If you are a careful thinker, but not very confident, you will
gain in confidence
- It is more fair - a thoughtful and confident correct answer
deserves more marks than a lucky hunch
- You need to pay attention if you make confident wrong answers -
to think, reflect and learn!
- Efficient study requires that you constantly question how your
ideas arise, and how reliable they are
- Certainty levels 1, 2, 3 always give you marks 1, 2, or 3
when you are correct
- If you are wrong, then unless you opted for C=1 you will lose
marks: -2 at C=2 and -6 at C=3
When should I use the
different certainty levels?
- When you're sure, you obviously do best with C=3 . But you stand
to lose 6 (twice the gain), if you are actually wrong!
- If you are very unsure, you can avoid any risk of a penalty by
- In between, you are best to use C=2: you stand to gain just
2 marks if correct, or lose 2 if you are wrong
- People are good at weighing up risks like this, but you will need
to think about whether you can justify your answer
- Look at the graph below, and you'll see how your average mark
depends on your choice of C level.
- suppose you think you only have a 50% chance of being right:
The highest graph for 50% on the bottom scale is black, for C=1. So you
will expect to boost your marks on average most by opting for low
- if you think you can justify your answer well, with >80%
chance of being correct, then the red graph is highest, for C=3. Opt
for high certainty.
- Note that you are always best to be honest when indicating your
certainty: if you claim high certainty (C=3: red) when you don't
think it is justified, you will expect to do badly - with very likely a
negative mark on average. If you understand the topic well, and think
your answer is, say, 90% likely to be right, then you will lose out if
you opt for C=1 or C=2 rather than C=3. You will do best if you can
distinguish which answers are reliable and which uncertain.
"Confidence" or Certainty"?
I switched from talking about CBM as "Confidence-based marking" to "Certainty-based marking" in 2005.
"Certainty" seems to carry less inappropriate baggage than "Confidence" - which sometimes
suggests to people that confident or brash personalities are being rewarded. This is not so!
Those who are rewarded are those who can distinguish between reliable and unreliable answers.
Successful people acknowledge uncertainty when they encounter it, and this is how they gain in self-confidence.
In the context of "How certain are you that this answer is right?" or "How confident are you that
this is right?", the two terms are equivalent. But "Certainty-Based Marking" is (I hope!)
less open to misinterpretation.
I have resurrected for the web an early classic review of some of the relevant concepts by
Andrew Ahlgren (1969)
- When your certainty is the same as for the last question,
pressing ENTER saves using the mouse, and speeds things
- Pressing ENTER again will take you on to the next
- Your browser will show which button ENTER will
mimic. If you want to do different things, of course use the mouse.
- To go back to earlier or skipped questions, use
the INDEX. Repeat attempts will be recorded as such.
- Don't ever use your browser's BACK or REFRESH
facilities. These may abort your exercise.
Problems, queries, suggestions: mail to
Top of Page