This is apparently the case at the UK Met Office, according to documents uncovered by the BBC that appear to show the agency taking steps to reduce the ability of the public to evaluate its forecasts.
The BBC explains that after its embarrassing 2009 prediction of a "barbeque summer" was followed by one of the wettest summers in a century,
its seasonal prediction for last winter was also awry, failing to signal sufficiently the long and severe cold spell.The idea that forecasts, once made, leave an agency "hostage to fortune" is no doubt true, but it can also be crucially important information for the public to evaluate the reliability of future forecasts. The criticism that the Met Office has faced in recent years results from too much credibility being placed in their forecasts. Highlighting the true state of predictive capabilities can help decision makers to understand better the uncertainties in forecasting, even if that means low credibility for the forecasters.
An internal executive paper noted the impact as follows:
"Unfortunately, less 'intelligent' (and potentially hostile) sections of the press, competitors and politicos have been able to maintain a sustained attack on the Met Office ... The opprobrium is leaking across to areas where we have much higher skill such as in short range forecasting and climate change - our brand is coming under pressure and there is some evidence we are losing the respect of the public."This report argued that one downside of the seasonal forecasts was that they remained on the website and could easily be later compared to reality. It said:
"One of the weaknesses of the presentation of seasonal forecasts is that they were issued with much media involvement and then remain, unchanged, on our website for extended lengths of time - making us a hostage to fortune if the public perception is that the forecast is wrong for a long time before it is updated."In contrast it noted that the "medium range forecast (out to 15 days ahead) is updated daily on the website which means that no single forecast is ever seen as 'wrong' because long before the weather happens, the forecast has been updated many times."
I saw this exact same dynamic at work in the US NWS in 1997 when I served on the agencies Disaster Survey team following the floods in the Red River of the North (for details see this paper in PDF). For the previous 16 years the NWS had issued a seasonal forecast for spring floods, and at Grand Forks the average error in this outlook had been 10%. But the NWS didn't tell anyone about its forecast accuracy. At the time NWS officials told me that they didn't advertise this information (e.g., by putting it on the web) because they didn't want the public to lose confidence in their forecasts by seeing their track record. In 1997 the seasonal outlook at Grand Forks was for a 49 foot river crest. The city build its defenses to 51 feet. The flood crested at -- you can't make this up -- 54 feet, or 10% higher than the prediction and almost exactly the average error of the past forecasts. The levees were overtopped and massive flooding resulted (the entire story is told here in PDF).
Believe it or not, but an understanding of forecasts "busts" can be a decision makers best friend. In our book on Prediction we argued that weather forecasts are so valuable not because they are always correct, but because decision makers inevitably have so much experience with them that they understand and can calibrate their accuracy based on experience. Perhaps the inevitability of such evaluations based on experience helps to explain why the weather community has fully embraced forecast evaluation in a way that the climate community has not. Such experience is not possible with seasonal forecasts (or those forecasts made on longer time scales) making understanding and communication of forecast uncertainties (and areas of ignorance) all the more important.
As difficult as it might seem, forecasters should embrace open evaluation of their forecasts, even (and especially) when they go bust. Better use of their products -- and potentially better decisions -- will result.
5 comments:
I used to post on the now defunct Climate Audit bulletin board about the Met's consistent overestimation of the global temperature in their annual forecosts. Consistent with your observation, the Met expunged all evidence of such forescasts a while back.
Fortunately for all that is right and just with the world, the Wayback Machine (archive.org) crawls the Met Office web site.
Annual predictions can be resurected, such as this one.
http://web.archive.org/web/20070111051630/www.metoffice.gov.uk/research/seasonal/global/pdf/global_temp_2007.pdf
Other annual forecasts can be called up by typing the portion of the above url starting with "www.metoffice.gov.uk" into the Wayback Machine's search box and replacing 2007 with the appropriate year.
Message to the Met: You can run but you can't hide.
I think the MetO were playing an averages game; and when the UK was in its "modern climate" phase of mild winters and BBQ summers their forecasts looked good and they reinforced the AGW message. However now we appear to be in the UK "post-modern climate" phase of late summer monsoon and Arctic winters the MetO forecasts are shown to not to be skilful. But the MetO must have known their seasonal forecasts were not skilful, so their behaviour in reference to the AGW message was pretty cynical.
On a related note the apparent post-modern climate, which is most readily apparent in the new behaviour of the jet stream, has shown up all AGW predictions as being weak. No one cares if global mean average temperatures rise a few degrees, we care if our local climate becomes adverse; something that cannot be determined until GCMs have regional skill - which I understand they do not (in fact I believe they have no skill in projecting synoptic scale features in response to forcing, so this is obvious).
"The opprobrium is leaking across to areas where we have much higher skill such as in short range forecasting and climate change.."
Higher skill... as in climate change. And they tell themselves these things with a straight face.
The final stage of AGW mania is complete failure of basic missions.
Decarbonization has slowed even as efforts by AGW promoters to control CO2 have increased.
CO2 has increased during the same effort.
Now the basic weather prediction services are showing to be so compromised by CO2 obsession that they are failing in their basic job.
UK is joined by the NZ weather service in being compromised by AGW politics. Where else is this occurring?
Your error lies in treating all forecasts as equally difficult or equally skilled - which they are not. For example, from personal experience I can easily forecast passenger numbers at my local rail station at 2pm will be small. If pressed I could suggest "Less than 15"
I can also forecast the passenger numbers at 6pm will be large. If pressed I could suggest "More than 60".
Even if 70 or 80 passengers get off at 6pm, does that make a forecast of "large" incorrect? But if I suggest 60, then EVERY other number will make my forecast wrong. So of course a very specific forecast is more likely to be wrong than right. There is nothing sinister there.
Climate change is about forecasting gas emissions/accumulations based on fuzzy numbers - i.e. using "large" rather than any specific number. Knowing large numbers will flood out of a station at 6:30pm allows me to avoid that area at that time, and similarly, knowing that gas emissions will be "small" or "large" allows decisions makers to decide how much effort needs to be put in to tackle climate change.
Finally: "No one cares if global mean average temperatures rise a few degrees, we care if our local climate becomes adverse;" Please go to your "local" food store and count how much of the stuff on the shelves is really actually local to your area. If that supply stops coming, it won't be long before you realise the importance of global mean temperatures. We really are "all in this together".
Post a Comment