If you’re a member of the Decision Analysis choir, you can stop reading now. Otherwise, have you ever taken or given an almanac exercise? If not, you should! Here’s what it is…
The almanac exercise is a probability assessment game well known within some DA circles, but as I keep reminding my choirboy friends, not everybody comes from ADA/SDG/DFI/EES/UT/etc.
The facilitator giving the exercise starts with five quantitative facts, things that could be looked up in an almanac* but wouldn’t be known off hand to most participants. They are usually printed on a piece of paper in a table, like this:
| Assess the Range of Uncertainty for: | Low | High |
| 1. Equatorial diameter of Jupiter, in miles | 2. Air distance from Melbourne to Madrid, in km | 3. Birth date of Confucious | 4. Cost of St Paul's Cathedral, in pounds sterling | 5. Height of the Gateway Arch (St. Louis), in feet |
These facts are easy enough to find with a search engine, but the participants are told not to do research, but rather to assess a range given their current state of knowledge. The language can be chosen to taste, but it is usually expressed as either a P10-P90 range or a centered 80% confidence interval.
If you’re following along, feel free to do the five assessments above now.
Have five ranges written down? Now Google the answers (or email me) and see how you did. If you’re like most people, you got most of them “wrong” – that is, the actual value was not between your Low and High. If you were well calibrated, you would have got about 4 out of 5 “right”.
There are two points to this exercise:
- First, it clearly illustrates that uncertainty is in your head: these are facts about which there isn’t much uncertainty, but you don’t know, so there’s uncertainty for you.
- Second, it provides a graphic depiction of overconfidence bias. People may readily admit that they “have no clue”, but still, when they write down a range, it is too narrow.
The benefit of this game is that if you go back and do it again, you’ll find that you’re much better calibrated. It can be incorporated into formal probability assessment training, or used by itself as a quick introduction to the subject. Every little bit helps, and overconfidence bias is arguably probability assessment public enemy #1.
Real knowledge is to know the extent of one’s ignorance.
– Confucious
So go forth and assess with confidence (just not too much).
*No, I haven’t seen an actual printed almanac in decades, but that’s what this exercise is called. Keep in mind Decision Analysis is ~33 years older than Google.
**I like to have at least one “local” question, so replace this with a fact about a local landmark, but don’t make it too easy and make sure there is an undisputed answer.