The well-calibrated Bayesian [pdf]

The Well-Calibrated Bayesian A. P. DAWID* Suppose that a forecaster sequentially assigns probabil- ities to events. He is well calibrated if, for example, of those events to which he assigns a probability 30 percent, the long-run proportion that actually occurs turns out to be 30 percent. We prove a theorem to the effect that a coherent Bayesian expects to be well calibrated, and con- sider its destructive implications for the theory of coherence. KEY WORDS: Calibration; Coherence; Martingale; Probability forecasting; Subjectivism; Weather forecasting. 1. INTRODUCTION Subjective probability forecasting is now well estab- lished among meteorologists, particularly in the United States (Murphy and Winkler 1977). Weather forecasters routinely make predictions such as "the precipitation probability for Denver today is 30 percent"; they have also experimented with credible interval temperature forecasts of the form "the probability is 75 percent that today's maximum temperature in Denver will be between 63° and 67°F. " The probabilities quoted refer to the fore- casters' subjective "degree of belief," given their infor- mation at the time of the forecast. This information may include the "objective forecast" output from a climato- logical analysis, or a computer forecasting system; how- ever, no explicit modeling process need be involved in arriving at forecast probabilities. Such probability forecasting fits neatly into the general Bayesian world-view as conceived by de Finetti (1975). The coherent subjectivist Bayesian can be shown to have a joint probability distribution over all conceivably ob- servable quantities. Forecasti11$ then is merely a matter of summarizing the conditional distribution of quantities still unobserved, given current information. In this article we shall, for definiteness, talk mainly in terms of weather forecasting, but it should be understood that the scope of the discussion is much wider, taking in all applications in which a subjectivist makes repeated probability fore- casts. For added definiteness, and with the usual non- sexist understanding, we shall refer to the forecaster in the masculine. Probability forecasts can be judged by several criteria (Murphy and Epstein 1967).

In this article we concentrate exclusively on the criterion of calibration (sometimes * A. P. Dawid is Professor of Probability and Statistics, Department of Statistical Science, University College London, London WClE 6BT, England. The author is grateful to the editor, the associate editor, a referee, and Professor M. Stone, for providing valuable comments and relevant references. termed reliability). Suppose that, in a long (conceptually infinite) sequence of weather forecasts, we look at all those days for which the forecast probability of precipi- tation was, say, close to some given value 00 and (assum- ing these form an infinite sequence) determine the long- run proportion p of such days on which the forecast event (rain) in fact occurred. The plot of p against 00 is termed the forecaster's empirical calibration curve. If the curve is the diagonal, p = 00, the forecaster may be termed (empirically) well calibrated. A parallel concept holds for credible interval forecasts: these are well calibrated if, for example, the long-run proportion of forecast 75 per- cent credible intervals that succeed in covering the actual value ofthe predicted quantity turns out to be 75 percent. The calibration criterion has some similarity with the frequency definition of probability, but does not require a background of repeated trials under constant condi- tions. In particular, it is rarely appropriate to interpret a subjective probability forecast as an estimate of some underlying "objective" probability; it is usually better considered as an estimate of (the indicator of) the forecast event itself. Thus we do not have to concern ourselves with the "true ,,' probability of rain on a given day. Rob- erts (1968) has attempted to interpret such a concept by supposing that one could select a subset of all days that could be regarded, at the time of forecast, as identical in all relevant respects, and consider the limiting relative frequency of rain on such days as the "true" probability for anyone of them. However, it is doubtful whether such a selection is practically meaningful, or whether different forecasters would agree on it. The calibration approach avoids these difficulties. Murphy and Winkler (1977) show that experienced weather forecasters are, on the whole, well calibrated.

Although this is not by itself a sufficient condition for their forecasts to be "good" (it would hold, for example, for a forecaster who invariably gave the long-term relative frequency of rain as his precipitation probability), it has often been taken to be a minimal desirable property. Fur- ther empirical studies of calibration have been reported by Lichtenstein, Fischhoff, and Phillips (1977), who ob- tain some poorly calibrated responses. A common suggestion (e.g., in Cox 1958) is that the probability statements of a poorly calibrated forecaster , should be transformed before use. Thus, if a forecaster's empirical calibration curve at a quoted value of 30 percent has p = 20 percent, then the consumer of this forecast © Journal of the American Statistical Association September 1982, Volume 77, Number 379 Theory and Methods Section 605 606 might perhaps best assume that the "true probability" of rain is 20 percent. We shall not consider here the gen- eral problem of how to use someone else's forecasts; for this, see for example Morris (1974), and Lindley, Tver- sky, and Brown (1979). In this article, we investigate the forecaster's view of his own calibration, and show, in particular, that if he is coherent then he expects to be well calibrated. We also discuss the problems that this creates for the theory of coherence. 2. INDEPENDENCE AND FEEDBACK Previous theoretical studies of the calibration property (Morris 1977; Harrison 1977) have mostly been concerned with the assessment of "seemingly unrelated" uncertain events or quantities: for example the height of the Eiffel Tower, the freezing point of mercury, the population of the USSR, and so on. It appears natural for the forecaster to regard such quantities as probabilistically indepen- dent. Supposing this, consider now the dilemma faced by such a forecaster who learns that of a very large number N of such forecasts that he has made 80 percent (say) exceed their assessed medians. This is an event whose subjective probability may be calculated from the binom- ial distribution with probability!, and will be vanishingly small; yet it occurred.

In the light of such a conflict it might seem appropriate to borrow from the logic of ;ig- nificance testing and reject the basis for the probability assessments. The moral that Harrison (1977) draws is that the naive approach, in which "seemingly unrelated" quantities were regarded as subjectively independent, is unacceptable; assessments for such quantities are, in fact, related merely by virtue of the fact that they are being made by the same assessor. Defining.a potentially miscalibrated individual as one who is not sure whether his future subjective probability assignments will agree with observed frequency, Harrison goes so far as to con- clude that "such a person will never perceive two events as (probabilistically) independent." These investigations seem too specialized in contexts such as weather forecasting, where, for example, the cri- terion of "seeming unrelatedness" of precipitation for different days is clearly inapplicable. We shall follow a different path, taking advantage of the sequential nature of the weather forecaster's task. The forecaster does not op~rat~ by.giving, on 31 December, his individual pre- CIpItatIOn forecasts for every day of the coming year, and then retiring: each day he forecasts for tomorrow drawing on his accumulated experience of all that ha~ passed up to today, including, in particular, the outcomes of those oftoday's events for which he supplied forecasts yesterday. It is such sequential forecasts with feedback that will form our principal subject of study. Our mathematical structure is as follows. Forecasts are made sequentially on days 0, 1, 2, ... , each referring to events or quantities that will become known on the following day. We denote by (l}3; the totality of events known to the forecaster on day i; thus (l}3o ~ (l}31 ~ .... Joumal of the American Statistical Association, September 1982 The forecaster has an arbitrary subjective probability distribution IT defined over (l}3"" = Vi~o (l}3;. The proba- bility forecasts he makes on day i are for events or quan- tities in (l}3;+ I, and are calculated from his current con- ditional distribution IT(' I (l}3;). With this formulation, the problem of the badly cali- brated forecaster is much more serious.

For suppose X; is a (l}3;-measurable quantity (i = 1, 2, ... ), for example the maximum temperature in Denver on day i, and let m; be the median of the forecaster's distribution for Xi, as assessed on day i - 1. Let Si denote the event "Xi> m;". Then, by definition, IT(Sil(l}3i-l) = !. Since (l}3i-1 cont~ins (SI, S2, ... ,Si-d, it readily follows that, ac- cordmg to IT, IT(Si) = ! and the (Si) are independent (Pratt 1962). Once again, we appear to have a conflict if, over many days, 80 percent, say, of the (Xi) exceed their as- sessed medians. However, the only assumption made above was that IT be coherent, so that IT obeys the laws of probability theory. That is, any coherent sequential forecaster must completely discount the possibility that he might be miscalibrated, however strong the evidence against him might be. In other words, in our sequential setup, Harrison's potentially miscalibrated individual cannot be coherent. We return to this point in Section 6. 3. A GENERAL CALIBRATION THEOREM In this section we present a very general result that extends the above connections between coherence and calibration. Once again we suppose the forecasts are made sequentially according to a fixed probability dis- tribution IT, but make no other assumptions. For each day i we have an arbitrary associated event Si E (l}3;, for example, the event of precipitation on day i. We denote the indicator of Si by Yi , and introduce 1'; = IT(Si I (l}3;_ d = E( Yi I (l}3i-I), the probability forecast of Si on day (i - 1). One way of comparing forecasts with reality is to pick out some fairly arbitrary test set of days, and in it compare (a) the proportion p of days whose associated events in fact occur with (b) the average forecast probability 'IT for those days.