A question of accuracy - US COOP temperature data

I came across an exchange at WUWT today. It was on yet another WUWT article about global surface temperature. The article itself avoided the obvious big picture and honed in on the detail (archived here). (The obvious being that global surface temperatures are rising very rapidly as CO2 emissions increase.) But that's not what this article is about.

What I'm writing about is in the comments. The exchange was about US temperature data, (which for some people is the same as global temperature data. The USA, believe it or not, comprises only a small proportion of the world's surface.). This topic seems to occupy the hive mind of WUWT to an inordinate and unwarranted degree.


Steven Mosher quoted Evan Jones, who is doing the bulk of the work for Anthony Watts, preparing for the paper that Anthony promised umpteen years ago, and says:

July 17, 2014 at 6:50 pm
“Absolutely (and highly regrettably) true. I have adjusted data, and dropped stations from the record for moves, for TOBS, and adjusted for MMTS conversion. Both times, the trend was warmer than when I started, too, I might add. But the idea is to get it right.”

Problem is you cannot trust the metadata without a cross check.

For example. The record may show NO move, but the actual agency that controlled the site ( not noaa) will show a move.

That is why you have to cross check for the following
1. undocumented changes in TOB
2. in documented changes in instrument
3. un documented stations moves

You can do this with break point analysis. its simple. you WITHHOLD the metadata.
you then run your breakpoint code. the breakpoint code will predict
changes at time a, b, c, d . you then look at your metadata and see how many you corrrectly
identify.

Trusting metadata is probably the worse mistake one could make. I hope you didnt trust it.
in other words just because a record fails to identify a change does not mean there wasnt a change

you need cross checks. from other metadata sources and from the time series itself

Steve Mosher was arguing that it pays to check that the metadata is correct, using other means to test for breakpoints. Anthony Watts responded to Steve Mosher with an inline comment: 

REPLY: “The record may show NO move, but the actual agency that controlled the site ( not noaa) will show a move.”

Mosher, this is ridiculous. All the COOP stations are managed by NOAA, local NWS WSFO’s there is no other “agency” with station metadata. Don’t make stuff up. – Anthony

The comment and particularly the response by Anthony got me to researching the history of the US cooperative observer program (COOP). The fact that the word "cooperative" is the source of the acronym suggested that the data relies on the cooperation of observers. Observers include volunteers all across the country. They report the data as the next comment from Evan Jones describes.

Then evanmjones joins in and says:

July 17, 2014 at 8:19 pm
The NOAA TOBS data comes right off the B-91 or B-44 (etc.) forms. I don’t see how you can get a more reliable record than that. The originals are available as PDF.
We prefer to find the cleanest set of stations we can possibly find and adjust as little as possible.

NOAA provides instructions to observers on how to complete the forms.

• Here are the Instructions For WS Form B91.
• Here is a word document of WS Form B44.

It strikes me that errors could creep in because:

• the weather station itself might have got out of whack
• the person filling out the form could make an error
• there could be errors in entering the hand-written details into the database
• any combination of the above.

In addition, data has to be adjusted to account for different times of observation, morning vs afternoon.

What it all means that all three people could be correct to a greater or lesser degree - Steven Mosher, Anthony Watts and Evan Jones. It seems to me they are talking at cross purposes. For example, Steve with his "(not noaa)" comment seemed to be talking about stations other than those controlled by NOAA. This is supported by his later comment here, in which he agreed that COOP stations came under NOAA but that it doesn't mean that the data is always accurate, writing (extract):

With Coop stations yes. But even there you cannot simply TRUST the metadata without a cross check.
years ago when your access was cut off to NOAA metadata I did an FOIA of noaa to get the mails surrounding the B-91 and the metadata. The mails indicate that the station history and metadata wasnt accurate. So as with everything one cant simply trust the records. In short. metadata has been cleansed, adjusted, “fixed” reconciled. Dont assume its perfect. It may be. but assuming its perfect will lead to mistakes.

(It's a shame Steve doesn't pay attention to capitalization or paragraph breaks. I fixed it a bit and I hope you get the gist.)

Now there have probably been numerous studies to determine the accuracy of the recorded data (which may have a reporting error or a recording error or both). For example, this 2005 paper  by Brian N. Belcher and Arthur T. DeGaetano (A method to infer time of observation at US Cooperative Observer Network stations using model analyses) included in the introduction (my bold italics):

Since 1890, daily observations of maximum and minimum temperatures and precipitation have been recorded at numerous locations across the USA as part of the US National Weather Service’s Cooperative Observer Network (CON). Given the volunteer nature of the CON, over 11 000 participating stations have specific, but different, observation times (OBTs). When deciding on an appropriate OBT for their particular station, each observer takes into consideration times that are suggested by the National Weather Service, as well as times that best fit into their daily schedule. Although these observation times are generally consistent from day to day and recorded by the observer on a monthly basis, this information is sometimes undocumented or incorrectly archived.

The paper was considering different techniques for checking for errors in time of observation. They ran some numbers and wrote:

As presented during the evaluation of this technique, some OBT estimates were inconsistent with the corresponding reported observation times in digital metadata files. After closer inspection of these inconsistent estimates within two diverse regions during a particular season (57 stations in region NE and 108 stations in region WC during Autumn 2003), each station falls into one of three general categories:

1. The reported observation time in the metadata file was incorrect and could be verified by an inconsistency with the observation time reported in the publication of Climatological Data (October 2003). For the two-category method, approximately 72% (41 out of 57) of the OBT estimates in region NE and 27% (29 out of 108) of OBT estimates in region WC that were initially labelled as incorrect were consistent with OBTs reported in Climatological Data. For the three-category method, about 10% fewer incorrect OBT estimates fall in this category.

1. Visual comparisons between the RUC simulated maximum temperature time series and observations from two nearby stations (1 ‘AM’ and 1 ‘non-AM’) confirm that the OBT estimates are likely correct (Figure 1). Although subjective, this verification method confirmed about one-third of the OBT estimates that could not be verified by Climatological Data. 

1. OBT estimation was likely incorrect given the Climatological Data OBT and visual comparison. About 15% of the OBT estimates that were initially labelled as incorrect in region NE fall in this category (45% in region WC).

You may be interested in this document on COOP, from the American Association of State Climatologists, which was prepared in 2007. It was written as an open letter to Congress and states in part:

The slow collapse of this 116-year old network has been documented in numerous professional reports during the past two decades. This network faces extinction because the data still are manually acquired, much like in 1890, by volunteer observers using equipment that is either obsolete or obsolescent. It has proven almost impossible to replace aging observers. Data quality continues to diminish, access to the data (recorded on paper) remains arduous, and the corps of maintenance professionals has been decimated because federal personnel have been shifted to other duties.

There have been improvements since then. The point being made is that just because COOP is part of the National Weather Service and NOAA doesn't automatically mean that all the metadata is accurate - or that every item of temperature and other weather data is accurate.

I also think it's strange that Anthony Watts and Evan Jones place so much faith in the hand-written records of thousands of observers, which went through at least one but probably multiple transcriptions before ending up in final digitised form - but don't appear to place much faith in modern analysis of temperature data, designed to weed out errors and inconsistencies and develop a comprehensive record of surface temperature trends for US regions and the whole USA.

Thing is, relying on raw data as reported by volunteer observers since the year dot, almost certainly will not give as accurate a picture of US temperature trends as the composite record, which includes necessary corrections and adjustments.

---

Belcher, Brian N., and Arthur T. DeGaetano. "A method to infer time of observation at US Cooperative Observer Network stations using model analyses." International journal of climatology 25, no. 9 (2005): 1237-1251.