Wondering Willis Eschenbach's hilarious hilarity - auto-correlation of deniers

Wondering Willis Eschenbach thinks he has made a startling discovery (archived here).  His penny dropped, his light switched on and he found it all too, too hilarious.  This is what he wrote today at WUWT:

I read a curious statement on the web yesterday, and I don’t remember where. If the author wishes to claim priority, here’s your chance. The author said (paraphrasing):

If you’re looking at any given time window on an autocorrelated time series, the extreme values are more likely to be at the beginning and the end of the time window.

“Autocorrelation” is a way of measuring how likely it is that tomorrow will be like today. For example, daily mean temperatures are highly auto-correlated. If it’s below freezing today, it’s much more likely to be below freezing tomorrow than it is to be sweltering hot tomorrow, and vice-versa. 

So far so good. I gather that what Willis is saying is that in red noise, the pattern of frequency of extremes is sinusoidal. Willis went ahead to test it with a "large number of pseudo-random datasets".  He wrote:

The easiest way to test such a statement is to do what’s called a “Monte Carlo” analysis. You make up a large number of pseudo-random datasets which have an autocorrelation structure similar to some natural autocorrelated dataset. This highly autocorrelated pseudo-random data is often called “red noise”. Because it was handy, I used the HadCRUT global surface air temperature dataset as my autocorrelation template. 

He put up some results in the following chart.

Figure 1. HadCRUT3 monthly global mean surface air temperature anomalies (black), after removal of seasonal (annual) swings. Cyan and red show two “red noise” (autocorrelated) random datasets. Source: WUWT

He found one "pseudo-random" data set that more or less followed HadCRUT and another that was completely different.  Willis didn't say how many sets he chose from or how many of these sets were similar to his blue and red "red noise" sets.  For example, how many of his chopped data sets followed HadCRUT as closely as the red one in his chart above? What are the chances? Willis didn't say.

He did some more analysis, chopping two large sets of data into sets that contained 2000 data points. What he found that there were more extremes in what he called his pseudo-random data at the beginning and end of the series. In other words, a sinusoidal pattern as he mooted above.

Figure 2. Histogram of the location (from 1 to 2000) of the extreme values in the 2,000 datapoint chunks of “red noise” pseudodata. Source: WUWT

These "extremes" at both ends included both high and low extremes, not high extremes at one end and low extremes at the other end (based on Willis' comment here).

Willis was full of mirth, writing:

If you take a random window on a highly autocorrelated “red noise” dataset, the extreme values (minimums and maximums) are indeed more likely, in fact twice as likely, to be at the start and the end of your window rather than anywhere in the middle.

I’m sure you can see where this is going … you know all of those claims about how eight out of the last ten years have been extremely warm? And about how we’re having extreme numbers of storms and extreme weather of all kinds?

That’s why I busted out laughing. If you say “we are living today in extreme, unprecedented times”, mathematically you are likely to be right, even if there is no trend at all, purely because the data is autocorrelated and “today” is at one end of our time window!

How hilarious is that? We are indeed living in extreme times, and we have the data to prove it!

Now it's true what he says. In a short time series like ten years, one doesn't expect to see the coldest year in the last century as well as the hottest year in the last century. Just the same, Willis comes across as being really disingenuous or dumb or doesn't understand what is causing global warming, this is what he wrote further down:

Typically, we consider the odds of being in extreme times to be equal across the time window. But as Fig. 2 shows, that’s not true. As a result, we incorrectly consider the occurrence of recent extremes as evidence that the bounds of natural variation have recently been overstepped (e.g. “eight of the ten hottest years”, etc.).

This finding shows that we need to raise the threshold for what we are considering to be “recent extreme weather” … because even if there are no trends at all we are living in extreme times, so we should expect extreme weather.

That first sentence isn't true in regard to expectations of climate extremes. Although I expect it depends on who the "we" are.  In regard to extreme weather, it depends on what weather you are talking about.  Extreme heat waves of the same parameters are not likely to be equal across a long time window.  It is expected that heat waves will continue to become more extreme as time goes by relative to a static baseline.  Extreme cold waves on the other hand, will continue to be less likely as time goes by relative to the same static baseline.

His last sentence to my mind doesn't follow. He wrote: "...because even if there are no trends at all we are living in extreme times, so we should expect extreme weather." If there were no trend (that is, a signal of zero trend), then the auto-correlation would also not have any trend. There would not be extreme weather at any particular time. The weather is tending to be more extreme as climate change kicks in. But it's not because of auto-correlation. (It could be that Willis is assuming that no matter where on the time series one is, there will be more extremes and maybe he thinks all those extremes will be hot. That auto-correlation isn't just noise - that it's the signal.)

One question is: how does he equate his "extremes" expectation with the "pause" that deniers go on about? Did the extremes stop being "extreme" 16, 18, 20 or 30 years ago or whenever it is that deniers reckon the "pause" started?

I have another question. What about if you go back to 1969 and look backwards from there? Up to the mid-1940s there was a period of increasing extremes, but then the temperatures stopped rising for a while. What happened to the extreme times and extreme weather?

Data Source: NASA GISTemp

Sure there is some auto-correlation in temperature data. However the increasing extremes has less to do with auto-correlation than to the the build up of energy on Earth because of all the greenhouse gases we continue to pour into the air.

What is auto-correlation?

I'll let Tamino tell you about auto-correlation and how to allow for it in climate data. He discusses auto-correlation as nearby (in time) noise values - not the signal:

Lots of time series, especially in geophysics, exhibit the phenomenon of autocorrelation. This means that not just the signal (if nontrivial signal is present), even the noise is more complicated than the simple kind in which each noise value is independent of the others. Specifically, nearby (in time) noise values tend to be correlated, hence the term “autocorrelation.”

There are other articles by Tamino on the subject, such as this one. Science of Doom has also written an article on auto-correlation. David Appell found references in a pdf file here (that talks about how to allow for it) and here when he was working through what autocorrelation means as far as surface temperature trends go.

Recent extremes and natural variation

As far as Willis' claim that "we incorrectly consider the occurrence of recent extremes as evidence that the bounds of natural variation have been overstepped" - he's wrong on that score, too. The way the evidence is interpreted is not incorrect (or not necessarily incorrect).  Proper attribution studies do allow for auto-correlation when trying to extract the signal from the noise. In any case, it is through studies of what is causing the earth to get hotter that we know whether extremes are caused by natural variation.

It is a fact that some studies to determine the likelihood of an extreme consider it in terms of probabilities but they are also based in science. Otherwise, the scientists would be saying - "Nothing has changed yet we had a year that on the balance of probabilities, should only occur once in every 13,000 years. We can't explain it (except for auto-correlation)."

Instead they say "Earth is warming. Australia last year had an average temperature that should only occur once in every 13,000 years if only natural factors were in play. We can explain it. It's because of the build up of greenhouse gases."

We need to raise the threshold

When Willis wrote: "we need to raise the threshold" he was spot on, but not for the reason he claims.  It's because the "new normal" is higher than it was before, because of global warming. It's got nothing to do with auto-correlation.

Willis has a point in that in some of the public's mind, extremes are compared to the weather of the twentieth century.  However climate is changing at such a rapid pace (in geological terms) and energy is building up so quickly that another way of looking at extremes is to consider the extent to which they can be considered extreme in the light of rapidly *increasing* energy and global surface temperature.  That is, the baseline isn't a flat line, it's an upward sloping line. The signal line is an upward trend.

Perhaps a reader who is well-versed in statistics can comment.  Willis seems to me to be confusing the noise and the signal with his article on auto-correlation.  Even to this lay person it's not conceivable that Earth could continue to get hotter just because it got hotter last decade. There has to be a physical reason. Noise is noise, the chance of red noise going forever in the same direction is remote.

All of which makes Willis' hilarity hilarious.

The dog is the weather

Which brings us to climate vs weather.

.

Incorrect Assumption

Willis ended up with this (my bold italics):

In any case, I propose that we call this the “Extreme Times Effect”, the tendency of extremes to cluster in recent times simply because the data is autocorrelated and “today” is at one end of our time window … and the corresponding tendency for people to look at those recent extremes and incorrectly assume that we are living in extreme times.

In my view it's Willis who is making incorrect assumptions. We are heading toward more and more extremes as climate change kicks in. That's not statistics, that's physics, chemistry, biology and climate science.

Footnote: I am not claiming any expertise in statistics here. I am simply pointing to other reasons for Willis' jumping to wrong conclusions. If anyone wants to weigh in from a stats perspective, feel free.

From the WUWT comments

The auto-correlation in the comments section is more apparent at WUWT than in the sample I've selected below.

bobbyv says (did Richard really say that?):

April 24, 2014 at 4:14 pm
I think this goes to what Lindzen says – one would expect our times to be warmest in a warming climate.

John Phillips talks about the most recent string of the past fifty years or so and says:

April 24, 2014 at 4:24 pm
Making much ado about many of the years within the most recent string of years being near the recent extremes was one of the first disingenuous tactics of the CAGW alarmists. Even when warming stops, they can continue that scam for many years to come. 

Theo Goodwin got his second sentence right when he says:

April 24, 2014 at 5:04 pmWonderful explanation of a wonderful insight, Willis. Just what we expect from you.

Willis Eschenbach repeats his erroneous erroneous claim and says:

April 24, 2014 at 5:21 pm
Steve from Rockwood says: April 24, 2014 at 5:12 pm My gut feeling is you have only proved your time series is band-limited both in low and high frequencies.

Thanks, Steve, and you may be right about the cause. However, I wasn’t speculating on or trying to prove the underlying causes of the phenomenon.

Instead, I was commenting on the practical effects of the phenomenon, one of which is that we erroneously think we are living in extreme times.
w.

RobL asks not a bad question and says:

April 24, 2014 at 5:41 pm
Is the effect stronger for shorter series? Eg what about a 160 point long series (to reflect the hottest year on record claims), or 16 point long series (to reflect hottest decade)

Frederick Michael talks about proximity of data points and says:

April 24, 2014 at 5:59 pm
The “red noise” or “Brownian motion” assumption is essential to finding a closed form solution. In my example of adding the N+1th point, knowing the value of the Nth point needs to be complete knowledge. (This is sometimes called “memoryless.”) If there are longer autocorrelations (trends, periodicity, etc.) the problem gets harder, and all bets are off on the endpoint effect — it could grow or disappear.

And adds more, Frederick Michael says:

April 24, 2014 at 6:57 pm
I think the term “red noise” is throwing folks off here. Willis is talking about pure Brownian motion. That is known as red noise but thinking about this in terms of spectrum is a rabbit trail. Willis is speaking of a series with no periodicity. 

gymnosperm seems to have concluded that global warming is real and we're not going to be heading for an ice age any time soon, except she or he is wrong about the last 17 years (1995, 19 years ago, was warmer than 1999 and 2000):

April 24, 2014 at 8:18 pm
There is another reason for ” it was the n hottest of the instrumental record”. The instrumental record is an S form with the hottest years at the top. Any year in the last 17 is guaranteed to be one of the top 17.

Humans have a natural tendency to “autocorrelate”. It is a perennial search for portents. 

Mike Jonas says:

April 24, 2014 at 9:08 pm
Willis – Good thinking, nice work! Following on from your post, I thought I would investigate the notion that nine of the last 10 years being the warmest “ever” was unprecedented. Answer : NO. It also happened back in 1945 and 1946.