James Risbey and co: Another perspective on surface temperature observations and climate models

The new Risbey paper that so puzzled Anthony Watts and Bob Tisdale and caused them to make public fools of themselves yet again, was not an evaluation of climate models. It wasn't an evaluation of model's ability to emulate ENSO. The research was answering the following question:

How well have climate model projections tracked the actual evolution of global mean surface air temperature?

Their answer was:

These tests show that climate models have provided good estimates of 15-year trends, including for recent periods and for Pacific spatial trend patterns.

Perennially Puzzled Bob Tisdale gets it wrong again

Bob Tisdale writing at WUWT (archived here) gets so much wrong in his article about the Risbey paper that it would take several articles to demolish every item. I'm not about to go to all that trouble. Let me just list a few things he got wrong:

• Bob confuses weather and climate. He mistakes climate models for weather forecasts.
• Bob is a greenhouse effect denier and rejects the conservation of energy. He thinks global warming is caused by El Niño. He has never come up with an explanation of where the energy comes from nor why he thinks El Niño has suddenly decided to cause global warming when it never did in hundreds of thousands of years before.
• Bob mistakenly thinks the Risbey paper was an evaluation of CMIP5 models. It wasn't. It said nothing about the worth of the models nor about their ability to model ENSO.
• Bob misunderstands the way "best" and "worst" are used in the paper.
• Bob misunderstands the way the researchers did the study. He thinks they selected four "models" and he and Anthony Watts made a big point of that. But they were mistaken. I describe below how the research team selected subsets of model runs for consecutive fifteen year windows.
• Bob confuses replication of the research with checking arithmetic. He mistakenly thinks the work cannot be replicated. It can. All the information needed to do so is provided in the paper itself. You don't even have to start from scratch. The research team tells you which models have SST data to compare with Niño 3.4. It will take a lot of work, particularly downloading all the model runs from the publicly available data (http://cmip-pcmdi.llnl.gov/cmip5/). The observations are readily available to the public as well, including GISTemp, HadCRUT4 and Cowtan and Way. Niño 3.4 values can be got from NOAA, sea surface temperature data from HadISST.
• plus he makes lots more errors, some of which I describe below.

If I had to pick one mistake out of all the mistakes Bob made, apart from not understanding basic thermodynamics and conservation of energy, perhaps Bob's biggest mistake is that he thinks CMIP5 climate models are designed to model day to day and year to year real world weather for the next several centuries. They aren't. That's an impossible task. It would mean being able to accurately predict not only random weather fluctuations but also every action that could affect weather. Such as how many aeroplanes are going to be flying where and when. Where and when the next volcanic eruption will be and how energetic it will be and what will be the composition of the stuff that blows out of it. How the sun will vary over time. Plus being able to find a computer of a size, and people to code, every single possible present and future interaction between the air, the land, animals, plants, the oceans, the ice, clouds, rivers, lakes, trees, the sun and outer space.  Humans are good and computers are powerful, but not that good and not that powerful. It's not just random fluctuations and disturbances in nature. We also affect the weather. Scientists model climate with those big computer models, not day to day weather.

I've written more below about the difference between models that are used to make weather forecasts and models used for climate projections - with some examples and links to further reading.

Climate models and natural internal variability - if in phase it's pure chance

Before talking about any of the hows and whys of Bob Tisdale getting it wrong, let me follow up my article from a couple of days ago and talk more about the Risbey paper itself and climate models in general. If you want to read more about climate models, I recommend the article by Scott K. Johnson at Ars Technica.

The abstract and opening paragraph of Risbey14 is important to understand if you are wanting to know what the paper is about. In particular these sentences:

Some studies and the IPCC Fifth Assessment Report suggest that the recent 15-year period (1998–2012) provides evidence that models are overestimating current temperature evolution. Such comparisons are not evidence against model trends because they represent only one realization where the decadal natural variability component of the model climate is generally not in phase with observations.

Climate vs weather

The bit I put in bold italics is what this paper is all about. Some people wrongly think that climate models designed to make long term projections should also reflect natural internal variability happening at exactly the same time as it happens in reality. Why they have that expectation is anyone's guess. As you know, even weather forecasts that are primed with the most recent weather observations can only predict weather a few days out at most before chance takes over and they head off in all sorts of random different directions. Climate models that don't have recent observations plugged in, but rely on physics, will include natural variability but that random natural variability will only occasionally be in sync with reality and then purely by chance. It's the effects of long term forcings like increasing greenhouse gases that are evident in climate projections and that's what we are most interested in from them. In other words, long term climate models are for projections of climate not weather.

How realistic are climate model projections?

The Risbey paper was looking to see if climate model projections were overestimating warming or not. It took a different approach to that taken by other studies. Other studies have looked at the question from different angles. Stephan Lewandowsky has explained three previous approaches in his article about the Risbey paper. You can read about them there, there is no need for me to repeat them here.

I will repeat the following from Stephan's article though, because it's a point that science deniers ignore. Observations remain within the envelope of climate model projections. As Stephan showed, this is illustrated by a chart from a paper by Doug Smith in Nature Climate Change last year.

Source: Smith13 via ShapingTomorrowsWorld

The chart above shows three things. First that the overall trend is up. It's getting hotter. Secondly observations are within the envelope of model projections. Thirdly that over time temperature goes up and down and doesn't go up at a steady pace. The bottom part of the chart is the trend per decade. It goes up and down but has mostly been in positive territory since 1970.

Predicting weather and climate

The Risbey paper sets the scene by describing the difference between a climate forecast and a climate projection, then the difference between weather variability and climate. It describes a climate forecast as attempting to take account of the correct phase of natural internal climate variations whereas climate projections do not.

Apart from any practical considerations, there are good reasons for this distinction. For climate projections looking ahead from decades to centuries, it doesn't matter much when natural internal climate variations come and go on a year to year basis. The interest is in the long term overall picture. For the next several decades and centuries, the interest will be in how much surface temperatures will rise, how quickly ice will melt, how soon and by how much seas will rise and where they will rise the most etc.

Even for projections over decades, we are most interested in regional climate change. Not interannual fluctuations such as ENSO variations so much as long term changes in the patterns of rainfall and temperature. ENSO affects weather. It will happen without climate change. The more important questions for the long term are things like: Will a region be getting wetter or drier? Will it be subject to more or less drought? Will the annual pattern of precipitation change, which will affect agricultural production, water supply management, flood control measures?

Climate models aren't yet able to be relied upon for projecting regional patterns of climate change with great accuracy but they can provide a guide.

The point is different models are adapted and used for different purposes. There are models used to predict short term weather. Some forecasts are fully automated (computer-generated) and others have human input. They are only useful for looking ahead seven to ten days. They are good for guiding decisions on whether to pack an umbrella, plant a crop, be alert for floods or scheduling construction tasks.

There are models that are constructed or adapted to make medium term weather forecasts, like those used by the Bureau of Meteorology to predict ENSO looking out over a few weeks to months. They are useful for making farm management decisions, utility companies making decisions on water storages (store or release water from dams), putting in a rainwater tank if it looks as if the next few months will be dry etc. They are forecasting weather not climate.

Then there are models adapted or constructed for longer term regional outlooks and models developed to look at the world as a whole.

Energy moves around between the surface, the ocean and the atmosphere

Climate models are used for more than just surface temperature. It's surface temperature that probably gets most attention in the media and on denier blogs. The world as a whole is warming up very quickly. Different parts of the system heat up at different times at different rates. Sometimes the air heats up more quickly than others. Different depths in the oceans heat up at different rates at different times too. Ice melts, but not at a steady pace. All that is because all these different parts of the system are connected. Heat flows between them. Anyone who goes swimming will have experienced the patches of warm water and patches of cold water in lakes, rivers and the sea. Just as heat is uneven on a small scale, it's uneven on a large scale.

I know some of you will be wondering why I'm taking readers back to climate kindergarten. Well if you read some of the comments to the previous article you can guess why. And if you manage to wade through even a part of Bob Tisdale's article at WUWT and the comments beneath, you'll get an even better appreciation.

CMIP5 projections are based on climate models not weather forecasts

All that is a prelude to the Risbey paper. It was looking at whether or not the recent global surface temperature trends are any different to what can reasonably be expected from model projections, given natural internal variability in the climate.

So the first thing to understand is that the CMIP5 climate model projections used by the IPCC will not generally model internal variability in phase with that observed over the model run. They do include internal variability but it's a stochastic property. It's purely a matter of chance when any model will show an El Niño spike or a La Niña dip in temperature for example. Whether a particular spike or dip lines up with what happens is pure chance. Sometimes a model run will be in phase with the natural variability observed and other times it won't. It's not important. Over time the natural internal variability cancels out. It's the long term trends that are of interest here, not whether an El Niño or La Niña happens at a particular time.

The Risbey approach

I call it the Risbey approach because James Risbey from Australia's CSIRO was the lead author. However I believe it was Stephan Lewandowsky who came up with the idea to take a look. Stephan thought it would be interesting to see what would be the effect on observations if the modeled natural internal variation was in phase with those observed.

Recall the point about climate models incorporating internal natural variation, but not necessarily in sync with when it happens in reality.  So what the team did was to look at windows of fifteen year periods and scan for model runs that were most closely aligned with observations, taking ENSO as the main measure of internal natural variability.

From the perspective of interannual internal variability, the factor that affects surface temperature as much or perhaps more than any other is ENSO. El Niño warms the atmosphere and La Niña has a cooling effect on the atmosphere. What happens is that heat is shifted between the ocean and the air. If there was no global warming trend, the surface temperature would go up and down with ENSO, leaving a long term trend of zero. (I've written a long article about ENSO, which includes references to good authorative sources.)

This brings us to the opening paragraph of the Risbey paper. On a short time frame, to see if models are reasonable, one needs to look at models that forecast. In other words, models that include natural internal variation that is in phase with what actually happens. Short to medium term forecast models do this by initialising them with the most recent observations or incorporating of the latest observations.  Most climate projection models are independent of recent observations. They are based on physics not live readings of what is happening. So if they happen to be in phase with internal natural variability at any time, it will only be by chance.

If you want to allow for natural internal variability and compare models with observations, one way you can do that is to look at model runs that happen to have a period of natural internal variability in phase with observations for the period of interest. That's what the Risbey team did.

Risbey14 looked at individual model runs and compared them with observations. For each fifteen year period they selected the model runs that were in phase with real world observations in relation to ENSO. They started with 1950 to 1964, then 1951 to 1965, then 1952 to 1966 etc. After selecting the models most closely aligned with ENSO phases observed for one fifteen year period, they moved up a year and looked at the next fifteen year period and so on. As well as that they were able to select, for each fifteen year period, the models that were most out of phase with ENSO.

Here is a conceptual visualisation of what they did:

Don't take too much notice of the actual data. It's a composite CMIP5 with GISTemp. And I'm not suggesting they eyeballed like that. They didn't. The above is just to get across the concept of what was done. In practice, the researchers selected model runs on the basis of their similarity in timing with real world observations of Niño 3.4 and related spatial patterns of sea surface temperature in the Pacific. [Correction: Stephan Lewandowsky has advised that during the selection phase they didn't look at the spatial patterns beyond Niño 3.4, which makes sense in the context of Figure 5 discussed below.] The image above is just so you get the idea that they looked at fifteen year periods starting with the period from 1950 to 1964.

The meaning of "Best" and "Worst"

That leads me to the discussion of "best" and "worst" that you may have read about and that Bob Tisdale got so wrong.

The research was not evaluating models. There is labeling on Figure 5 in the paper, which uses the words "best" and "worst". However there is no suggestion that the four "best" models are in any way superior to the four "worst" models in terms of what they are designed to do, which is future projections of climate. The word "best" denoted the subset of model runs in any fifteen year period that were most in phase with observations for ENSO. Conversely the word "worst" denoted the model runs that were least in phase with ENSO observations.

The paper compares the spatial pattern of temperature variation for the period 1998 to 2012. It compares models most in phase with the ENSO regime with observations as below. The real world observations are on the right: (click to view larger size)

Source: Figure 5 Risbey14

It also compares models most out of phase with the ENSO regime with observations:

Source: Figure 5 Risbey14

The point being made was that the model runs most in phase with the real world ENSO observations had a "PDO-like" spatial pattern of cooling in the east Pacific. Look at the above charts close to the equator near South America.  In the top left hand chart, while not as cool, the overall pattern is closer to the observations on the right. In the bottom chart, the warming is smudged all over and it doesn't show the cooler east Pacific. Figure 5 showed that the model runs most out of phase showed a "more uniform El Niño-like warming in the Pacific".

Bob Tisdale was thrown by the word "best" and "worst". He also seemed to think the climate model runs should have been identical to the real world. He's wrong because he doesn't understand what CMIP5 climate models are for or how they work. As the paper states, when you select only the model runs in phase for the period, you get much closer spatial similarity, not just a closer match for the surface temperature as a whole. When you lump all the climate model runs together, the multi-model trends average out the internal variability. Models will cancel out the effect of each other when you lump them all together, because on shorter time scales, it's only by chance that some have internal variability in phase with the real world.

When it comes to "best" and "worst", the paper was only referring to the extent to which selected model runs were in phase with the real world. There is no suggestion in the paper that any one model was any better as a model than any other. The research was not an evaluation of climate models. In fact, different models were in and out of phase in different fifteen year windows. Just because by pure chance a model run was in phase with ENSO over a particular time period did not mean that same model run was in phase with ENSO in other time periods.

Models will go in and out of phase with the real world over time. That randomness is intrinsic to climate models. The models are designed to respond to forcings like increasing greenhouse gases and changes in solar radiation. They exhibit internal variation too, because the physics is built in. But it won't generally be at exactly the same time as it happens in the real world. (If there were long term models that could do that, then most weather bureaus would be out of business.)

Now, if you have a large enough sample, then in any fifteen year period some model runs will line up pretty well with natural fluctuations in the real world. Just by chance. So if you want to see a short period of time, like say the last fifteen years, you can see if any model runs line up with ENSO over that period and if they do, how close are they overall to global surface temperature observations.

That's probably the nuts and bolts of the paper. Comparing all model runs for a short period like the last few years won't tell you a whole lot about whether the models are realistic or not. That's because individual model runs won't necessarily be in sync with the real world in regard to natural variability. They are expected to reflect the dominant climate forcings but not the exact timing of ENSO for example.

By looking just at model runs that just happen to be in sync (by chance), you can see how much they vary from the real world observations. This study showed that once you allow for the fact that models won't be in phase with natural variability, then the models are even closer to real world observations. I'll finish with the two figures showing trends, that Stephan had in his article. The one on the left is from the "in-phase" model runs compared with observations. The one on the right is those least "in-phase" compared with observations. As always, click to enlarge.

Source: Risbey14 via Shaping Tomorrow's World

Further reading:

• For another perspective, with links to related papers, Dana Nuccitelli has an article on the subject at the UK Guardian.
• Here's the link again to Stephan Lewandowsky's article, which also has links to related papers.
• I wrote previously about Gavin Schmidt's paper on this topic, taking a different approach.
• Watch this space for more.

Acknowledgement: Thanks to James Risbey for answering my naive questions so promptly. Thanks to Stephan Lewandowsky for describing the work so well. Any mistakes in this article are mine. It wasn't checked by any of the paper's authors before publication. Since then, I've made one correction, thanks to Stephan L. And thanks to all those who made available a copy of the paper so I could write this article - RN, AS, JR and SL.

Note: updated with more links to WUWT. Sou 11:14 pm 24 July 2014

---

James S. Risbey, Stephan Lewandowsky, Clothilde Langlais, Didier P. Monselesan, Terence J. O’Kane & Naomi Oreskes. "Well-estimated global surface warming in climate projections selected for ENSO phase." Nature Climate Change (2014) doi:10.1038/nclimate2310