.

Friday, July 18, 2014

A question of accuracy - US COOP temperature data

Sou | 3:10 PM Go to the first of 7 comments. Add a comment

I came across an exchange at WUWT today. It was on yet another WUWT article about global surface temperature. The article itself avoided the obvious big picture and honed in on the detail (archived here). (The obvious being that global surface temperatures are rising very rapidly as CO2 emissions increase.) But that's not what this article is about.

What I'm writing about is in the comments. The exchange was about US temperature data, (which for some people is the same as global temperature data. The USA, believe it or not, comprises only a small proportion of the world's surface.). This topic seems to occupy the hive mind of WUWT to an inordinate and unwarranted degree.


Steven Mosher quoted Evan Jones, who is doing the bulk of the work for Anthony Watts, preparing for the paper that Anthony promised umpteen years ago, and says:
July 17, 2014 at 6:50 pm
“Absolutely (and highly regrettably) true. I have adjusted data, and dropped stations from the record for moves, for TOBS, and adjusted for MMTS conversion. Both times, the trend was warmer than when I started, too, I might add. But the idea is to get it right.”
Problem is you cannot trust the metadata without a cross check.
For example. The record may show NO move, but the actual agency that controlled the site ( not noaa) will show a move.
That is why you have to cross check for the following
1. undocumented changes in TOB
2. in documented changes in instrument
3. un documented stations moves
You can do this with break point analysis. its simple. you WITHHOLD the metadata.
you then run your breakpoint code. the breakpoint code will predict
changes at time a, b, c, d . you then look at your metadata and see how many you corrrectly
identify.
Trusting metadata is probably the worse mistake one could make. I hope you didnt trust it.
in other words just because a record fails to identify a change does not mean there wasnt a change
you need cross checks. from other metadata sources and from the time series itself

Steve Mosher was arguing that it pays to check that the metadata is correct, using other means to test for breakpoints. Anthony Watts responded to Steve Mosher with an inline comment: 
REPLY: “The record may show NO move, but the actual agency that controlled the site ( not noaa) will show a move.”
Mosher, this is ridiculous. All the COOP stations are managed by NOAA, local NWS WSFO’s there is no other “agency” with station metadata. Don’t make stuff up. – Anthony

The comment and particularly the response by Anthony got me to researching the history of the US cooperative observer program (COOP). The fact that the word "cooperative" is the source of the acronym suggested that the data relies on the cooperation of observers. Observers include volunteers all across the country. They report the data as the next comment from Evan Jones describes.

Then evanmjones joins in and says:
July 17, 2014 at 8:19 pm
The NOAA TOBS data comes right off the B-91 or B-44 (etc.) forms. I don’t see how you can get a more reliable record than that. The originals are available as PDF.
We prefer to find the cleanest set of stations we can possibly find and adjust as little as possible.

NOAA provides instructions to observers on how to complete the forms.

It strikes me that errors could creep in because:
  • the weather station itself might have got out of whack
  • the person filling out the form could make an error
  • there could be errors in entering the hand-written details into the database
  • any combination of the above.

In addition, data has to be adjusted to account for different times of observation, morning vs afternoon.

What it all means that all three people could be correct to a greater or lesser degree - Steven Mosher, Anthony Watts and Evan Jones. It seems to me they are talking at cross purposes. For example, Steve with his "(not noaa)" comment seemed to be talking about stations other than those controlled by NOAA. This is supported by his later comment here, in which he agreed that COOP stations came under NOAA but that it doesn't mean that the data is always accurate, writing (extract):
With Coop stations yes. But even there you cannot simply TRUST the metadata without a cross check.
years ago when your access was cut off to NOAA metadata I did an FOIA of noaa to get the mails surrounding the B-91 and the metadata. The mails indicate that the station history and metadata wasnt accurate. So as with everything one cant simply trust the records. In short. metadata has been cleansed, adjusted, “fixed” reconciled. Dont assume its perfect. It may be. but assuming its perfect will lead to mistakes.
(It's a shame Steve doesn't pay attention to capitalization or paragraph breaks. I fixed it a bit and I hope you get the gist.)

Now there have probably been numerous studies to determine the accuracy of the recorded data (which may have a reporting error or a recording error or both). For example, this 2005 paper  by Brian N. Belcher and Arthur T. DeGaetano (A method to infer time of observation at US Cooperative Observer Network stations using model analyses) included in the introduction (my bold italics):
Since 1890, daily observations of maximum and minimum temperatures and precipitation have been recorded at numerous locations across the USA as part of the US National Weather Service’s Cooperative Observer Network (CON). Given the volunteer nature of the CON, over 11 000 participating stations have specific, but different, observation times (OBTs). When deciding on an appropriate OBT for their particular station, each observer takes into consideration times that are suggested by the National Weather Service, as well as times that best fit into their daily schedule. Although these observation times are generally consistent from day to day and recorded by the observer on a monthly basis, this information is sometimes undocumented or incorrectly archived.

The paper was considering different techniques for checking for errors in time of observation. They ran some numbers and wrote:
As presented during the evaluation of this technique, some OBT estimates were inconsistent with the corresponding reported observation times in digital metadata files. After closer inspection of these inconsistent estimates within two diverse regions during a particular season (57 stations in region NE and 108 stations in region WC during Autumn 2003), each station falls into one of three general categories:
  1. The reported observation time in the metadata file was incorrect and could be verified by an inconsistency with the observation time reported in the publication of Climatological Data (October 2003). For the two-category method, approximately 72% (41 out of 57) of the OBT estimates in region NE and 27% (29 out of 108) of OBT estimates in region WC that were initially labelled as incorrect were consistent with OBTs reported in Climatological Data. For the three-category method, about 10% fewer incorrect OBT estimates fall in this category.
  1. Visual comparisons between the RUC simulated maximum temperature time series and observations from two nearby stations (1 ‘AM’ and 1 ‘non-AM’) confirm that the OBT estimates are likely correct (Figure 1). Although subjective, this verification method confirmed about one-third of the OBT estimates that could not be verified by Climatological Data. 
  1. OBT estimation was likely incorrect given the Climatological Data OBT and visual comparison. About 15% of the OBT estimates that were initially labelled as incorrect in region NE fall in this category (45% in region WC).

You may be interested in this document on COOP, from the American Association of State Climatologists, which was prepared in 2007. It was written as an open letter to Congress and states in part:
The slow collapse of this 116-year old network has been documented in numerous professional reports during the past two decades. This network faces extinction because the data still are manually acquired, much like in 1890, by volunteer observers using equipment that is either obsolete or obsolescent. It has proven almost impossible to replace aging observers. Data quality continues to diminish, access to the data (recorded on paper) remains arduous, and the corps of maintenance professionals has been decimated because federal personnel have been shifted to other duties.

There have been improvements since then. The point being made is that just because COOP is part of the National Weather Service and NOAA doesn't automatically mean that all the metadata is accurate - or that every item of temperature and other weather data is accurate.

I also think it's strange that Anthony Watts and Evan Jones place so much faith in the hand-written records of thousands of observers, which went through at least one but probably multiple transcriptions before ending up in final digitised form - but don't appear to place much faith in modern analysis of temperature data, designed to weed out errors and inconsistencies and develop a comprehensive record of surface temperature trends for US regions and the whole USA.

Thing is, relying on raw data as reported by volunteer observers since the year dot, almost certainly will not give as accurate a picture of US temperature trends as the composite record, which includes necessary corrections and adjustments.



Belcher, Brian N., and Arthur T. DeGaetano. "A method to infer time of observation at US Cooperative Observer Network stations using model analyses." International journal of climatology 25, no. 9 (2005): 1237-1251.

7 comments:

  1. It is the general opinion of the homogenization community that station history data is very important, but cannot be relied on as single information source of inhomogeneities. Depending on the kind of problems in a certain nation, the rules for when a record number is kept or changed, the quality of the stations histories an the power of the statistical breakpoint methods there is a large range in the number of breakpoints for which there is confirming information in the metadata. A typical number would be that about half of the breaks found by statistical homogenization (comparison of one station to its neighbours) can be confirmed by the station history.

    It is ironic that the climate pseudosceptics like to cite Hansen a lot: “Our analysis shows that “data homogenization for [temperature] stations moved from downtowns to suburbs can lead to a significant overestimate of rising trends of surface air temperature.”

    Here Hansen was talking about homogenization using only station history information. He explained why that is dangerous, which is one reason why the normal procedure is to apply statistical homogenization methods as well.

    I guess the average reader of this blog will recognise that this question about whether metadata should be used or not is not the only politically convenient inconsistency of the pseudosceptics.

    ReplyDelete
    Replies
    1. Thanks for that information, Victor. I'd have thought the percentage would be higher than that.

      This is probably a silly question, but would that mean that changes to the station aren't recorded half the time, or are there other reasons for half the inhomogeneities.

      Delete
    2. Sou,

      outside the United States metadata records are pretty much unavailable (at least digitally and publicly). Even within the States they are known to be incomplete. Retention of metadata has been far worse than that of data (which itself leaves a lot to be desired ...). When Uncle Victor changed the thermometer out back in 1960 (Victor isn't really that old btw ...) he didn't think that you or I may care in 2014. Which I guess if we are being fair to old Uncle Victor wasn't an unreasonable assumption. For most stations we are lucky that we have good and correct lat/lon/elev ...

      Delete
    3. Thank you, Peter. And a warm welcome to HotWhopper. I am watching your project with interest.

      http://www.surfacetemperatures.org/

      Delete
  2. Victor, I'm sure you know the rules of Climateball (TM) by now (heh). Just because you detect a TOBS change at a station due to a discontinuity in the actual data that is consistent with that, doesn't mean you have to allow for it. If the volunteer who monitors the station didn't document the TOBS change, then it didn't happen!

    Their mantra is: "It's not CO2, it's not CO2, it's not CO2...". *Anything* - any little crack, or gap, or inconsistency in the science - they can come up with that supports their ideology will be latched onto and held as firmly as Linus holds his security blanket (Charlie Brown ref there for the non-Yanks :-)

    ReplyDelete
  3. I also think it's strange that Anthony Watts and Evan Jones place so much faith in the hand-written records of thousands of observers, which went through at least one but probably multiple transcriptions before ending up in final digitised form - but don't appear to place much faith in modern analysis of temperature data, designed to weed out errors and inconsistencies and develop a comprehensive record of surface temperature trends for US regions and the whole USA.

    They are desperate anti-government nuts. They are alienated by real progress and results in data handling, science and systematisation, all of it funded by TAXES and feeding information to or through GOVERNMENT . They are wandering beyond 'citizen science' to be like survivalists in their all-encompassing paranoia, idealised independence and rejection of institutions. Because AGW is a government plot, of course. It just must be...

    We could call them 'survivalist scientists', doing 'survival science'.

    ReplyDelete
    Replies
    1. I like that turn of phrase, Nick: 'survival science'. I am *so* stealing that :-)

      Delete

Instead of commenting as "Anonymous", please comment using "Name/URL" and your name, initials or pseudonym or whatever. You can leave the "URL" box blank. This isn't mandatory. You can also sign in using your Google ID, Wordpress ID etc as indicated. NOTE: Some Wordpress users are having trouble signing in. If that's you, try signing in using Name/URL. Details here.

Click here to read the HotWhopper comment policy.