Monday, August 11, 2014

William M Briggs is no futurist. About forecasts, scenarios and projections


I followed one of the items that "caught the eye" of Judith Curry at the weekend. It was an article by denier statistician to the stars, William M. Briggs. Yes, he's the bloke who coauthored a dumb article with Christopher Monckton of all people. That in itself tells you more than you need to know about him.

Anyway, William M. Briggs might shine brightly for some stars but he's no futurist. William wrote an article (archived here) under the headline:
There Is No Difference Between A Forecast, A Scenario, or A Projection

He's wrong of course. And not just in having a superfluous comma. Even I make that mistake, from time, to time :)

I thought I'd take a peek at what he wrote underneath his headline to see just how wrong he would get.  This is what I found:
This is a tad incoherent...
Well, his first clause was correct but that's about all he got right.


Here's more:
People trying to escape the implication of a bad forecast often claim their forecast wasn’t a forecast but a projection or scenario. The implication is that a bad forecast means a (possibly beloved) theory is no good. Therefore, if the forecast wasn’t a forecast, but a projection or scenario, the theory can still be admired (or funded).
This won’t do. Forecasts are scenarios are projections. And bad forecast-scenario-projections means bad theories.

There is a lot wrong with the above. I'll list a few things:
  1. If a forecast is bad it's bad.
  2. A projection isn't a forecast.
  3. A scenario isn't a forecast.
  4. A projection isn't a scenario.
  5. A forecast isn't a scenario.

So what is the difference between a forecast, a projection and a scenario? It's not even subtle.


What is a forecast?


A forecast is an informed prediction, usually based on statistics or on previous observations of what happened under the same conditions. It is an estimation (not a guess) of what is going to happen in the future to something or the other.

A weather forecast is an informed estimate of what the weather will be like in a particular locality over the coming day or maybe anything up to a week. It is based on current meteorological conditions combined with a knowledge of atmospheric science. For example, if radar images show a storm moving across the country it is likely that the storm will affect locations ahead of it in its current trajectory. The weather bureau will issue a severe storm alert. It will forecast the storm and give an estimate of its path and time of arrival.

Today there's a frost warning out for parts of Victoria, Australia. The frost hasn't happened yet but the models used to forecast weather indicate it's likely to happen overnight. That's a forecast. It's not conditional upon anything except known science and current conditions. The current conditions are as good as current observations, which these days are pretty accurate. The known science is better than it was a decade ago and much better than it was a century ago. Plus there has been a vast improvement in recent years in the tools used to estimate how the weather will progress. Computers are a lot more powerful than they used to be.


What is a projection?


A projection isn't a forecast. It is not a statement of likelihood without any conditions except current observations and known science. A projection starts to bring in "what ifs". A projection typically looks further ahead in time and therefore it has to make assumptions about more and more variables. In the case of climate projections, assumptions are made about variables such as human behaviour, technological advances, government policies and international agreements. It's not just that any one variable could work out differently. The variables can affect each other so the interplay of differences in variables can have a multiplier effect.

Good projections will list the assumptions. They don't need to be extensive. For example, with climate projections, the various variables could be represented by an assumption about future emissions of carbon dioxide. The future emissions of carbon dioxide will depend on a lot of things, such as economic activity, the rate of adoption of renewable energy technology, the rate of adoption of energy efficient appliances, government policy to curb carbon emissions and so on. But all of these can be collapsed into an assumption about future emissions.  That allows a projection of other effects for different assumptions about future emissions. Such as how quickly ice will melt, how quickly seas will rise, how quickly global surface temperatures will rise and by what amount. None of the projections are right or wrong in themselves. However they can be right or wrong in the context of the assumptions. In other words, if the projection is that temperatures will rise by 3C if  emissions are such that CO2 doubles, then that will be evident some time after CO2 doubles - if emissions stop after doubling.


What is a scenario?


Scenarios are a bit different from projections and forecasts. Scenarios are imaginary futures. They are typically used by the military when developing strategies. They are also used by corporations in planning. The idea is to imagine a future and describe it in terms that have meaning for the context in which they'll be used.

A not-for-profit providing services for the homeless might think of a future in which there is an economic downturn, combined with local disasters (such as a major fire or earthquake) and a "small government" government that has reduced or cut completely any support for the homeless or for low income housing. It might also come up with other scenarios. The scenarios don't have to be completely realistic. They are most certainly not forecasts. Nor are they projections. By their nature they will be somewhat simplistic. They are used to test the limits for decision-making purposes. To test the impact of decisions that are made today on what could be the result of those decisions if today's conditions changed tomorrow.

If scenarios are described in sufficient detail, they can be used to make projections. They can't be used to make predictions or forecasts. That would be shifting into magical thinking territory. There are too many variables embodied in a scenario to be as certain about the impact as one would be about a forecast. However they can be used as the basis for projecting what a future would be like under that scenario.

In the example I chose about the homeless, one might liken it to the great depression when the economy took a dive, government was small and the main roads were filled with swaggies (or hobos) looking to make a few bob in the next town or the one after. Some good folk helped by serving soup to desperate people. It made a difference.


Climate projections and scenarios


So what does that mean for climate projections and scenarios? The IPCC in its latest report didn't use the word scenarios. It used the word "pathways". It's not that different. There was effort put into describing and modeling different scenarios as integrated assessment models or IAMS, none of which are very realistic. They included assumptions about economic activity, technological advances, population trends etc. However at least when they are simplified down they allow projections of climate change. Without them, then climate projections would be based on even more rubbery assumptions.

The science of climate is well understood. More than sufficiently well to know that we must cut emissions of carbon dioxide. It is more difficult to forecast how human society will respond over the next few years. Will we rise to the challenge of mitigation? Will we adopt renewable energy technology quickly enough? What about other factors that could impact the future? What if there is another economic meltdown or another world war or a major pandemic or a massive volcanic eruption? How would they affect future emissions of carbon dioxide?


My forecast, prediction and scenario


I don't know if William M Briggs really doesn't know the difference between a forecast, a projection and a scenario or if he's just pretending to be dumb. He did say that his article was the basis of an abstract he was submitting to the upcoming AGU Fall meet later this year. If it's anything like his blog article, I imagine a scenario in which his abstract elicits a grunt before being dropped into the waste paper basket. I predict that scenario will come to pass. In fact I forecast that his abstract won't win a guernsey. In the unlikely event that it does, I forecast it will gain nothing but mockery from anyone who matters.

Let's see how that forecast, prediction and scenario pan out :)

20 comments:

  1. The serial or Oxford comma is not considered superfluous in many style books.

    ReplyDelete
    Replies
    1. Except that isn't how it was used. The example given by Oxford Dictionaries was:

      "These items are available in black and white, red and yellow, and blue and green."

      The comma was to make it clear the options were pairs of colours not single colours.

      If William had written: "There Is No Difference Between A Forecast and A Projection, An Estimate and A Scenario, or A Prediction and A Projection" it would have made about as much sense as his article. But at least he could have got away with using all the commas :)

      Delete
    2. I'm no grammarian or linguistic expert. I was merely being petty, adding some gratuitous snark :)

      I constantly lose marks because I often break up a sentence into two where I shouldn't, in the interest of readability. For example, the last sentence in my comment above should be a continuation of the previous one.

      Delete
    3. Oh, I misread the text. You are correct Kevin. It is indeed permitted in a list of three or more. Can't say I like it as it was used though.

      I'll leave my comments for the pedants to weigh in further to argue the case of the comma, if they want to. Also it's a good demonstration that you shouldn't trust everything you read on a blog. Or everything you read at HotWhopper. At least, not without checking for yourself.

      Delete
    4. This is not the serial comma (which is not only accepted but generally encouraged) , but comma-related and funny enough to post anyway:

      A panda walks into a bar. He orders a sandwich, eats it, then draws a gun and fires two shots in the air.

      "Why? Why are you behaving in this strange, un-panda-like fashion?" asks the confused waiter, as the panda walks towards the exit. The panda produces a badly punctuated wildlife manual and tosses it over his shoulder.

      "I'm a panda," he says, at the door. "Look it up."

      The waiter turns to the relevant entry and, sure enough, finds an explanation.

      "Panda. Large black-and-white bear-like mammal, native to China. Eats, shoots and leaves."

      - "Eats, Shoots & Leaves", Lynne Truss 2004

      Delete
    5. As we are on about commas, can someone explain the nonsensical comma often inserted nowadays between the subject and verb, e.g., a perpetrator would have titled this post "William M Briggs, is no futurist".

      Not as bad as "eats, shoots" because it doesn't change the meaning but why do it?

      Delete
  2. Yeah, William M. Briggs: Numerologist to the Stars was recently featured at tamino's place where he was basically making the same category error, referring to *historical* yearly Berkeley Earth global land temp anomalies as "predictions".

    Needless to say, appearing as the subject of one of tamino's posts is generally not a good thing. It usually winds up getting you ripped a new one.

    ReplyDelete
    Replies
    1. Tamino does not mince words. (Not like that nice Sou person). A couple of quotes from that article to give a flavour of what he thinks of Briggs:

      "Briggs is just playing word games in an infantile example of novice sophistry. He wants you to believe that since he has called them “predictions” and claimed the come from some evil “models” they can’t be trusted. It’s not the “tricky” part, it’s the “tricksy” part."



      "Yes, folks, that’s it. Create a bogey-man made of straw, call it “uncertainty,” apply no analysis and no data, all just by waving of hands. That’s all he’s got."

      Delete
  3. Mr. Briggs' AGU abstract is here:

    https://agu.confex.com/agu/fm14/preliminaryview.cgi/Paper25185

    ReplyDelete
    Replies
    1. Thanks, Raymond. Was my forecast (1) incorrect or is there yet to be a vetting process to see which of the submitted abstracts are accepted?

      I wonder if his paper will be any better than the abstract or his blog article. We will have to wait till the end of the year to see if my forecast (2) is correct.

      William seems to be quite confused. And I don't get his climate forecast being sensitive to the price of oil analogy. It looks to me that he is the one who is mixing up forecasts and projections. Playing word games without understanding the subject matter. It's not the first time.

      Delete
    2. We'll find out in October I guess:

      http://fallmeeting.agu.org/2014/scientific-program/

      "Letters of notification regarding abstract submission acceptances will be distributed in early October 2014. The online program will also be posted in early October 2014."

      More submissions, including more from Cato Institute (Pat 'n Chip) arguing that "all the models are wrong".

      https://agu.confex.com/agu/fm14/preliminaryview.cgi/Session3166

      Delete
    3. Almost all AGU abstracts are "accepted"; there's always a "General" session for those that can't be categorized. This particular abstract is a mess, so it might not make the cut. On balance, I hope it does, and Briggs get the in-person feedback he deserves.

      Delete
    4. PL, I am afraid the feedback will be that hardly anyone looks at his poster.

      Of course, it could be fun if he is invited to give a presentation, and then people start asking questions (read: an expert points out he's wrong, confused, and whatnot). I don't think this will happen, though, since selecting abstracts for presentation is usually the task of the chairman, who would then be humiliated along with the presenter. I've seen it happen once at a conference. It was ugly.

      Delete
    5. Rejection will serve his purposes better than acceptance, I suspect. Perhaps it's time for a NAGU (Not the AGU) Conference?

      Delete
    6. It's fairly common to transfer an abstract to a more appropriate session but it's rare for an abstract to be rejected outright. Rejection is a big enough deal that it has to be decided at a high level, not by the convener (session chair). The guidelines for session conveners states:

      "Conveners do not have the authority to reject an abstract. If an abstract is not appropriate for the session, the convener may return the abstract to the appropriate Program Committee member, or recommend rejection of the abstract to the Program Committee."

      The Program Committee is the high-level body that supervises sessions for an entire discipline such as Atmospheric Sciences or Hydrology. So rejection is not something done routinely.

      Delete
    7. "Perhaps it's time for a NAGU (Not the AGU) Conference?"

      didn't they just hold that in Vegas?

      Delete
  4. I got into a discussion on Bishop-Hill about this (I will say my encounters there have been more pleasant than I was expecting. To be fair, I wasn't expecting much so I can't really tell if it's because people have been pleasant or just not as unpleasant as I was expecting :-) ). The discussion seemed to vary from making snarky remarks about projections being predictions that you don't believe, projections becoming predictions (which is kind right), and projections being the same as predictions for verification purposes. It was moderately interesting, partly because some seemed to understand what it meant, but couldn't quite bring themselves to admit it. I think that would make the IPCC projections seem more reasonable and that's probably unacceptable to some.

    ReplyDelete
    Replies
    1. "It's tough to make predictions, especially about the future."

      Apparently It's easier to make projections, because when they turn out to be off they don't 'count' as a failure.

      (snarky remarks being a specialty of mine ;)

      Delete
    2. "Apparently It's easier to make projections, because when they turn out to be off they don't 'count' as a failure."

      That is some sort of logic failure. The difficulty or ease of making a projection is nothing to do with its results. The difficulty/ease of making a projection is purely in the construction.





      Delete
  5. The definition of projection according to the Australian Bureau of Statistics

    "A projection indicates what the future changes in a population would be if the assumptions about future trends actually occur".

    http://www.abs.gov.au/websitedbs/a3121120.nsf/home/statistical+language+-+estimate+and+projection



    ReplyDelete

Instead of commenting as "Anonymous", please comment using "Name/URL" and your name, initials or pseudonym or whatever. You can leave the "URL" box blank. This isn't mandatory. You can also sign in using your Google ID, Wordpress ID etc as indicated. NOTE: Some Wordpress users are having trouble signing in. If that's you, try signing in using Name/URL. Details here.

Click here to read the HotWhopper comment policy.