.

Wednesday, February 7, 2018

The spectacular failure of the 2007 climate "bet" by denier J. Scott Armstrong

Sou | 1:20 AM Go to the first of 20 comments. Add a comment
Today at WUWT there's a rambling, indecipherable article about some bet that a science denier called J. Scott Armstrong unsuccessfully tried to make with Al Gore way back in 2007. It's a tale of a failed denier prediction, and worse. Having failed so spectacularly, J. Scott Armstrong is doubling down and betting on a drop of up to 4.5 °C in global temperature over the next decade.

Armstrong was wanting to bet that there'd be no change in global average surface temperatures between 2008 and 2017. He figured, wrongly, that Al Gore would bet there would be warming. Al Gore didn't take the bet. Why would he deal with a nincompoop denier like J Scott Armstrong.

Armstrong's first draft of the bet was a bit weird. The essence of it was this:
Al Gore is invited to select any currently available fully disclosed climate model to produce the forecasts (without human adjustments to the model’s forecasts). Scott Armstrong’s forecasts will be based on the naive (no-change) model; that is, for each of the ten years of the challenge, he will use the most recent year’s average temperature at each station as the forecast for each of the years in the future. The naïve model is a commonly used benchmark in assessing forecasting methods and it is a strong competitor when uncertainty is high or when improper forecasting methods have been used.

Specifically, the challenge will involve making forecasts for ten weather stations that are reliable and geographically dispersed. An independent panel composed of experts agreeable to both parties will designate the weather stations. Data from these sites will be listed on a public web site along with daily temperature readings and, when available, error scores for each contestant.

Starting at the beginning of 2008, one-year ahead forecasts then two-year ahead forecasts, and so on up to ten-year-ahead forecasts of annual “mean temperature” will be made annually for each weather station for each of the next ten years. Forecasts must be submitted by the end of the first working day in January. Each calendar year would end on December 31.

The criteria for accuracy would be the average absolute forecast error at each weather station. Averages across stations would be made for each forecast horizon (e.g., for a six-year ahead forecast). Finally, simple unweighted averages will be made of the forecast errors across all forecast horizons. For example, the average across the two-year ahead forecast errors would receive the same weight as that across the nine-year-ahead forecast errors. This unweighted average would be used as the criterion for determining the winner.

Terms of the challenge can be modified by mutual agreement.



Version 2 of Armstrong's "bet" from 2008


The following year, after it was clear that Al Gore wasn't playing, he changed it to this, where he made it clear he was basing the bet on the Hadley Centre data:
Al Gore is invited to select any currently available fully disclosed climate model to produce the forecasts (without human adjustments to the model’s forecasts). Scott Armstrong’s forecasts will be based on the naive (no-change) model; that is, for each of the ten years of the challenge, he will use the most recent year’s average temperature at each station as the forecast for each of the years in the future.

Details on the 10-year bet would be handled with discussions between me the Hadley Centre. I would ask an independent board to aid in this process of finding an appropriate design and to monitor the progress of the bet. You would be kept up to date, and you would have the right to ask the board to consider changing aspects of the design.

Version ? of Armstrong's "bet" from 2016


More recently he changed it from surface temperature to lower troposphere temperature as recorded by UAH. I don't know exactly when he decided to give up surface for lower troposphere, but it was very recently going by Wayback Machine records, some time between 15 September 2015 and 2 March 2016, probably January 2016.

Never mind. J Scott Armstrong and his ally in denial, Kesten Green, didn't do at all well with their bet that there'd be no change in global temperature, whether surface or lower troposphere. Here are some comparisons.


Not the best bet


First of all, here is what would have looked like if Armstrong had chosen NASA's GISTemp. The brown lines are what was observed since 2008 and the red line is the "no change" from 2008:

Figure 1 | Global average surface temperature from 2008 to 2017 inclusive. Data source: GISS NASA

The linear trend for the ten year period is 4.35 °C/century. The actual difference between 2008 and 2017 is 0.38 °C.

Below is the same plot for HadCRUT, Armstrong's initial choice.

Figure 2 | Global average surface temperature from 2008 to 2017 inclusive. Data source: Met Office Hadley Centre

The linear trend for the ten year period is 3.68 °C/century. The actual difference between 2008 and 2017 is 0.28 °C.

Then there's the 2016 revision of the "bet" that wasn't. The change in UAH lower troposphere temperatures over the period. Version 6 of UAH was released in April 2015, and it's the only version that covers the period from January 2008 to December 2017. Version 5.2 would have been the current version at the time  the bet was conceived, and it looks as if that was last published in 2010. Version 5.6 stops at July 2017.

Figure 3 | Global average lower troposphere temperature from 2008 to 2017 inclusive. Data source: University of Alabama Huntsville


The linear trend for the ten year period is 4.48 °C/century. The actual difference between 2008 and 2017 is 0.48 °C over the period.

So, by whatever measure, J Scott Armstrong lost his would-be bet. I wonder why Anthony Watts posted that strange article, given he's a global warming denier. His article had a question as the headline: Tipping point 10 years on: Who won the Armstrong-Gore ‘bet’ on the climate?


Observed change is higher than IPCC's 2007 projected per century trend


The WUWT article kept referring to an "IPCC 3°C-per-century projection". The IPCC report of 2007 (the year Armstrong tried to make the bet) listed different projections, depending on what scenario we humans chose to follow. If we were to follow the scenario preferred by deniers, the best estimate of temperature change was projected as 4 C higher in 2090-2099 relative to 1980-1999. If we were to follow the scenario preferred by the other 90% of the human race (no change in atmospheric CO2), the best estimate was projected as 0.6 °C higher in 2090-2099 relative to 1980-1999. This is from the AR4 2007 IPCC report:



I guess that J Scott Armstrong and Kesten Green (his partner in crime) were expecting somewhere between the A1B scenario and the A2 scenario.


Implausible simulations from the denier duo


Quite weirdly, despite the ongoing increase in atmospheric CO2, the article at WUWT claimed that there was a 70% chance that there wouldn't be any increase in global temperature over the past ten years or so. I don't know who did the simulation, but I wouldn't give them any prizes:
A 150-year simulation of the bet suggested that his chance of winning was only about 70%.

Shifting goal posts wasn't wise


The WUWT article also claimed:
In the end, the bet was offered, and monitored, on the basis of satellite temperature data from the University of Alabama at Huntsville (UAH).
As I pointed out earlier, the "end" looks to have been about eight and a half years after the original ten year bet was proposed. Some sad end :( He should have stuck with HadCRUT, because that showed the smallest increase and the lowest trend over the period.

The article waffles on about cumulative absolute errors and other gumph, and the most telling part was this graphic inserted somewhere near the bottom of the article:

Figure 4. Graphic from WUWT showing just how bad the Armstrong projection turned out to be.

I wouldn't trust it a great deal, given it's from WUWT. However even if it were an accurate representation of what it purports to be, the chart suggests that UAH temperature is pretty close to the 3 °C per century trend. (As shown in Figure 3 above, the linear trend for UAH v6 is substantially higher than 3 °C per century, at 4.48 °C/century.)

It can only get worse for J. Scott Armstrong


Even more weird is that apparently J. Scott Armstrong is going to hang out for significant global cooling over the coming decade. From the WUWT article:
Longer is better for assessing climate forecasts, and so theclimatebet.com site will monitor the “bet” in line with Scott Armstrong’s offer to extend the challenge for another ten years by sticking with the original 2007 annual average global temperature as the starting point. Extending the bet is intended to help further publicize the important role of scientific validation of forecasts that influence public policy. Policymakers should reject forecasts that fail to reduce errors compared to an appropriate no-change benchmark.

From the WUWT comments


Nick Stokes injected some comments that attempted to help Anthony's fans work out what the article was trying to explain. Nick wrote in answer to someone's question:
February 6, 2018 at 1:41 am
“who, if anyone, would have won the bet?”
See below. The UAH slope for the period was 0.45°C/Decade. Armstrong/Green would have lost on any accounting. By a mile.

After more of Nick Stokes' explanatory contributions, paqyfelyc made a very odd comment. I can only deduce that paqyfelyc didn't understand the first thing about the WUWT article or changes in global temperature:
February 6, 2018 at 2:25 am
@Nick
As usual, you are writing before thinking.
Do you believe anyone in his sane mind really cares what you write? Not even your fellow warmunist cultists do. 

knr doesn't know that Al Gore didn't make any bet with J Scott Armstrong:
February 6, 2018 at 12:50 am
All good points which mean ‘nothing’, for the author is fighting on the wrong battlefield. St Gore claims were very about science nor facts, they never are.
It’s PR and BS all the way, that and ego with personal enrichment. And the really bad news is, he won those bets.

Michael Stevens summed things up nicely:
February 6, 2018 at 1:39 am
Blimey…

As did zazove:
February 6, 2018 at 3:43 am
So in a clumsy attempt at character assassination Mr Green gives himself an uppercut. All good fun.

What WUWT article would be complete without a "thought" from one of Anthony's pet conspiracy theorists. toorightmate wrote:
February 6, 2018 at 4:15 am
Kesten Green,
Be very careful.
Look what has happened to the late Bob Carter, Jennifer Marohasey, Prof Peter Ridd, Willie Soon, Judith Currie and numerous other wise people who have strived for the truth to be told and for the science to be objectively debated.
This left wing monster is something to be very fearful of. 

References and further reading


The Global Warming Challenge - Scott Armstrong vs. Al Gore - the original attempted "bet" proposed by J. Scott Armstrong, June 2007

Speaking of fossil fuel funding... - a HotWhopper article about more from J. Scott Armstrong and Kesten Green, October 2013



20 comments:

Jammy Dodger said...


I tried to read the WUWT article but it really is a rambling word salad that uses a lot of words to confuse the whole thing. The article and the comments just shout desperation. I am amazed Watts thought it was a good idea to publish it.

Millicent said...

I am confused: any climate change denier who wants to bet on no global warming can do so at "Cool Futures" courtesy of Monckton et al. Its almost as if they all think "Cool Futures" is yet another pile of denier doggy do.

Mark B said...

Nick Stokes has an article up on his site exploring some of the "unusual" mathematics employed by Armstrong & Green:
https://moyhu.blogspot.com/2018/02/weirdness-from-armstronggreen-and.html

Nick Stokes said...

Yes, it's an amazingly poorly written article (by Kesten Green). But they are well connected. Thir fantasy has has a run on Fox News and The Australian.

AFAICS, in claiming a win, they have miscalculated the OLS trend by a factor of three. Sou is right at about 4.35 °C/Cen (I get 4.5 on a monthly basis). They say 1.53 °C/Cen. Pretty odd for "forecasters".

Russell Seitz / Bright Water said...

Even Delingpole seems to have given up on this one- He thinks there's more action to be had parsing museum labels

Sou said...

Nick, I got the same as you for UAH v6. (You picked up the trend I estimated for GISTemp. I also included HadCRUT.)

Deeenngee said...

For fun I've produced a scoreboard of climate bets. I've included all the ones that I can find where terms were agreed by both parties and made public.

The current score is 7-0 to the IPCC-friendly / Consensus / Warmist side - that's four completed and three still underway.

The closest to a win for the Contrarians / Coolists was Whitehouse vs Annan. Whitehouse narrowly won using HADCrut3 (which was in use at the time), but Annan would arguably have won with HADCrut4. So I've called that a draw for now.

I'm happy to post somewhere if anyone's interested.

Jammy Dodger said...

Deeenngee, go for it. You can post the pertinent details here.

I am interested because I find it hard to believe the anti-Warmists scored one, even arguably!

Sou said...

I agree with Jammy, that would be good, Deeenngee. I know a couple of people who'd like to see a compilation :D

Deeenngee said...


OK, posted at https://deeenngee.blogspot.co.uk/

Let me know what you think!

Sou said...

That's great - thank you, Deeenngee.

Deeenngee said...

It's a bit crude and wonky at the moment - I need to find the best way to insert a table on Blogger which doesn't go weird on the published page...refinements to follow.

Jammy Dodger said...


Looks good Deeengee.

Perhaps some pointers to the actual bets and a line or two of your analysis would enhance it.

Victor Venema said...

"Scott Armstrong’s forecasts will be based on the naive (no-change) model; that is, for each of the ten years of the challenge, he will use the most recent year’s average temperature at each station as the forecast for each of the years in the future."

That is a good PR bet. The people who are willing to pretend to be stupid for their political cause can claim the bet is about no warming.

Actually the bet is about a lot of warming. By each time taking the last year, the warming is baked into the bet. If Armstrong really did not expect any warming he could have taken the temperature of the first year as the comparison temperature.

With a warming that is on global average about 2 degrees per century (which is 0.02 °C per year) and making the bet about station data, which has a noise level of about 1 °C, the bet is actually coin flipping and Armstong has a good chance of wining while the Earth is warming fast.

P.S. Those WUWT comments are atrocious. In the Age of Trump they seem to have given up any pretense to be more than a tribe of uglies. Not caring about whether what you say makes sense works for Trump. Looks like they got inspired.

Layzej said...

Hi Deeenngee, I'm not sure you can call the Annan/Whitehouse bet a draw. James lost £100 on that bet. He would have won if they had used anything other than HADCRU3, but they did not and he did not: http://julesandjames.blogspot.com/2012/01/not-so-fast.html?showComment=1332378320991#c407845517170362161

Layzej said...

On the other hand, Annan also has a $10,000 bet that he has just won (but has yet to collect on): https://www.theguardian.com/environment/2005/aug/19/climatechange.climatechangeenvironment

Deeenngee said...

Layzej - thanks for those links and I see from the first that James Annan paid up. I guess that's conceding under wager etiquette and as far as we know David Whitehouse didn't void the bet when HadCRUT4 came out shortly after! On that basis I've changed the scoreboard to a win for the Coolists. I've kept & amended the footnote though and generally improved the layout.

Millicent said...

Off topic, but the latest version of reality from the fossil fuel industry and its muppets is that global warming is happening but all those experts who have looked at its effect are wrong because Scott Pruitt knows better.

That's a load off then.

swimbouy said...

Deeennggee,

If you could, it would be nice to include links to blogs that discuss the bets. Here, for example, is Chris Hope's one: http://www.chrishopepolicy.com/2016/07/making-a-safe-bet-on-dangerous-climate-change/ and I think the progress of Rob Honeycutts one is described on SkS. I think it could be an interesting and useful resource.

Another avenue would be to list, like the one at WUWT, bets that weren't taken up. James Annan tried to make a bet with Terence Mills over a really dodgy statistic prediction, but Mills clearly didn't have much confidence in his own prediction (despite it being reported in The Times (UK).

These bets are typically over GMST trend; there might be some over arctic sea ice extent/volume too ...

Matthew Morse said...

I'm a little late to this, but I wanted to add some comments about Kesten Green's statistical flim-flammery in the WUWT article. This is in addition to other criticisms, such as cherry picking the data set and modifying the rules of the "bet" after the "bet" began, as others have discussed.

1. One fundamental assumption of linear regression is that the noise is uncorrelated. This is obviously not true for this data. What this means is that any error computations are inherently incorrect. Therefore none of the conclusions that Kesten Green comes to have any validity.

2. This one will take some explanation. The method typically used for linear regression is known as Ordinary Least Squares (OLS). Mathematically, what the method does is choose the model which minimizes the Mean Square Error (MSE) computed by comparing the model fit to the observed data. When assessing the predictions of a model, it is reasonable to compute the MSE comparing the predicted values from the model against the new observations.

However, MSE is not the only possible way to evaluate the model fit. There are various other ways to compute this, including using absolute error instead of square error. Green links to an article which argues that when choosing a model based on prediction, absolute error is superior to square error. This is an interesting claim and I would like to spend some time going over the linked article. I do admit, however, to a little suspicion about that article because it turns out that J. Scott Armstrong is one of the coauthors.

One obvious consequence of using absolute error as opposed to square error is that the impact of the 2016 El Niño on the error calculations is greatly reduced. This has obvious appeal to Green because it means that the penalty for not predicting the 2016 El Niño is much smaller when computing absolute error rather than square error.

3. We also need to examine the choice of models. Armstrong's model is that temperature is constant. As a choice of constant temperature, Green uses the average temperature over 2007.

For the "IPCC/Gore" model, Green uses a trend of 3 degrees per century. However, the trend also needs an origin, and Green uses the 2007 average temperature as the origin. I am not convinced that this is a good choice for the origin, but I also don’t have a recommendation for a better choice.

It turns out that the absolute error is very sensitive to the choice of the origin. Reducing the temperature of the origin by 0.01 degrees decreases the absolute error by around .7 degrees. Given that the absolute error for the 3 degree trend prediction is only 2.6 degrees greater than the absolute error for the constant prediction, a change of around 0.04 degrees in the temperature at the origin would reverse Green's conclusions. (Note that the measured temperature for December 2007 was less than -0.2 degrees. This is more than 0.35 degrees below the temperature Green used, and has nearly as much justification as a starting temperature as Green's choice.)

Incidentally, the comments on Nick’s post have a discussion of Green’s use of the origin in the regression fit and resulting conclusion that the slope is 1.5 degrees per century, rather than Nick and Sou’s computation of 4.5 degrees per century.

In summary, Green's analysis does not correct for autocorrelation, which invalidates all of the conclusions of the analysis. Leaving that aside, the analysis relies on absolute error rather than square error. While this may be justifiable, it can't be denied that absolute error gives much better results for Armstrong's predictions than square error would. Leaving that aside, the choice of the origin for the 3 degree per century prediction is arbitrary. Green's conclusions are highly sensitive to the choice of origin. Taken together, it's clear that Green's analysis has no value.