NOAA and temperature data - it must be a conspiracy.

Update: Nick Stokes of Moyhu has written two articles that demonstrate what would happen if Anthony Watts had his way and stations with no data were ignored completely. See here and here. Anthony Watts might change his tune if he read them.

Sou 3:22 pm 1 July 2014 AEST

---

This article is about the kerfuffle that erupted between a bunch of science deniers. It all started when Steve Goddard accused NOAA and NASA of "fabricating data" (archived here).  The lie was spread all over the right wing media. Politifact and Climate Crocks among others pointed out he was wrong. Steve didn't explain how the two agencies did this. All he did was put up an animated chart that he claimed showed that the US temperature was warmer in the 1930s than at any time since. He wrote:

Prior to the year 2000, NASA showed US temperatures cooling since the 1930′s, and 1934 much warmer than 1998....Right after the year 2000, NASA and NOAA dramatically altered US climate history, making the past much colder and the present much warmer. The animation below shows how NASA cooled 1934 and warmed 1998, to make 1998 the hottest year in US history instead of 1934. This alteration turned a long term cooling trend since 1930 into a warming trend.

Steve doesn't say what data was "fabricated". Why should he? He's not a fact checker. Quite the opposite. He's in the denial business of making up stuff to stop any action to mitigate global warming.

This article is another one that's too long :( Click read more if you're on the home page.

Why Steve Goddard is wrong

The NOAA has spent quite a bit of effort explaining the various adjustments over the years. You can click here for one very comprehensive overview of the changes, together with links to relevant papers and data. There's a shorter FAQ paper on the US temperature record here as a pdf file. When data is missing or incorrect it can be replaced, using a calculated estimate based on surrounding temperatures as recorded. It's not simply an average either. Raw data could be missing altogether (eg if the observer missed some readings) or as a marked discontinuity (eg as a result of moving the weather station).

If you want to know why earlier USA records are more likely to be adjusted higher than lower, then read Nick Stokes new article on Time of Observation corrections. Read this one too. It's got a chart that shows up the problem nicely. And then there's Victor Venema's classic article on the subject. (This mainly applies only to the USA.)

Sensible people ignore Steve Goddard.

Judith Curry uses the fiasco as an excuse for spreading her FUD

Judith Curry wrote in her usual denialist fashion using words like "uneasiness" (definition: Judith hasn't checked the facts which gives her an "out" to spread FUD) and "astonishing" (definition: similar to the definition of "uneasiness"). Judith wrote, for example:

Apart from the astonishing scientific and political implications of what could be a major bug in the USHCN dataset, there are some interesting insights and lessons from this regarding the technical skeptical blogosphere. 

Thing is, of course, there is no "major bug". Steve Goddard is full of it as usual, flinging his wild and baseless allegations hither and yon and delighted that they've been picked up by the right-wing media.

Jump, NOAA, jump higher

Judith seems to think that NOAA needs to respond to every bit of silliness written by deniers on their little blogs. She wrote about an article by another science denier (archived here), Paul Homewood, who is a frequent "guest" at WUWT, who made a silly song and dance about a single weather station in Luling Texas, which, it turns out, had malfunctioned. Judith didn't wait, she can't help herself as usual and wrote:

Homewood’s post sheds light on Goddard’s original claim regarding the data drop out (not just stations that are no longer reporting, but reporting stations that are ‘estimated’). I infer from this that there seems to be a real problem with the USHCN data set, or at least with some of the stations. Maybe it is a tempest in a teacup, but it looks like something that requires NOAA’s attention. As far as I can tell, NOAA has not responded to Goddard’s allegations. Now, with Homewood’s explanation/clarification, NOAA really needs to respond.

Judith extrapolated from the record of a single weather station at Luling Texas, which malfunctioned so the data was dropped from the record for a bit and replaced by temperature estimated by an algorithm using the record from surrounding stations. But even the hint from a science denier that "something must be wrong" was enough for Judith to proclaim that "there seems to be a real problem with the USHCN data set". Although she does mildly qualify her pronouncement with a "at least some of the stations" and her "Maybe it is a tempest in a teacup" remark.

The blog reaction to Steve Goddard's idiocy provided an interesting insight into the mind of science deniers. Anthony Watts is boasting how he and Judith (a real live scientist) were emailing each other all day. Judith is writing about Anthony Watts as if he's got something of value to add. And both of them are expecting NOAA to jump through hoops for people who run piddly denier blogs when the blog owners say "jump".

NOAA estimates missing data

The only thing that anyone eventually found was that a number of stations that purportedly had records were included as estimates in the NOAA data instead of actual values. Anthony reckons this is a dreadful state of affairs. But is it? Government agencies in the USA were stripped of staff for a long time because the Republicans refused to pay them. What is the priority for double checking records, when it will make virtually no difference anyway, compared to other priorities competing for scarce resources. I bet both Anthony Watts and Judith Curry approved of the budget cuts. Anthony claims that this affects readings in Texas and Kansas. But what is the effect on the national or even regional data?  It lies somewhere between zero and minimal. Turns out that only 13 Texas stations out of 188 49, over one period of time or another in the past few years, were estimated (from nearby records).

Anthony Watts should leave well enough alone

Anthony really got his knickers in a twist and wrote a lot of nonsense. He made a real mess of things. First he wrote about how Steve Goddard was wrong about "fabrication". Then he wrote about how Steve Goddard was right about "fabrication" - or at least that's how it appeared to many of his denier readers. Then in the comments he gets a whole heap wrong and adds some new claims of his own - which he fails to substantiate. It looks to me as if he's trying hard to reclaim his ground as a science denier. Doing penance and trying to repair the damage to his reputation from when he dared point out that Steve Goddard was writing a load of codswallop.

Here's an exchange between Anthony Watts and Nick Stokes, who says:

June 28, 2014 at 1:49 pm
” Along with that is his latest followup, showing the problem isn’t limited to Texas”
But what was the problem in Texas? I did a post on Luling here. When you look at the local anomaly plots there is a very obvious inhomogeneity. The NOAA software detected this, and quarantined the data, exactly as it should. It then turned out, via comments of mesoman who had worked on this very site, that there was a faulty cable causing readings to be transmitted low, and this was fixed on Jan 2014.
So, you might say, good for the computer, it got it right, and Paul H was wrong. A bit of introspection from Paul Homewood and co re how they had been claiming malfeasance etc? But no, no analysis at all – instead they are on to the next “problem” in Kansas. And so the story goes – first we had problems in Texas, now in Kansas.

REPLY:
Despite what you think you can’t “estimate” the characteristics of temperature from effects of a faulty cable. In Lulings’s case, just throw out the data, don’t imagine you are smart enough to be able to predict the resistance changes that occur from rain, heat, humidity, dust, etc. as they affect it or the next lawnmower bangs into it. As you’ll note, the test “mesoman” did say the temperatures were fluctuating when he did his test to determine what was wrong. he said the data was unstable.

Can you predict what the temperature will be in a thermistor that has a faulty connection at any given moment? Can you predict what the low and high temperatures it will produce will be on any given day when compared to the vagaries of weather it experiences?
Is is patently absurd to try to salvage data from a faulty instrument, especially when you have one nearby also recording the temperature.

THROW OUT THE DATA – DON’T TRY TO FIX IT.

Imagine forensic science trying to get away with this stuff. I’m reminded of the famous line from The Green Mile The Shawshank Redemption “how can you be so obtuse?”.
-Anthony

Anthony Watts got so irate that he shouted at Nick. He also showed he knows nothing about surface temperature records, suggesting wrongly that NOAA tried to "salvage data from a faulty instrument".

When Nick Stokes pointed out that the NOAA did indeed throw out the faulty data, Anthony shouts some more, this time about blood and accuses Nick of being "obtuse":

June 28, 2014 at 2:12 pm
...Throw out the data? That’s exactly what they did. They replaced it with an estimate based on neighboring stations. Not on trying to repair the Luling data. In the NCDC method which uses absolute temperatures, you have to have an estimate for each station, otherwise you get into the Godard spike issues.

I notice that John N-G said there were 13 stations in Texas that have had to replace measured data in recent years, for various periods. I believe Texas has 188 stations in total.

REPLY: Great, you should be a legal adviser in court.
Judge: The Blood samples tainted! You: OK. THROW IT OUT AND REPLACE IT WITH SOME BLOOD from …THAT GUY, OVER THERE! NO, Wait, lets get blood from the nearest five guys that look like him and mix it together. Yeah that’s a reasonable estimate.
You can’t ever assume your estimates will model reality.
Again, how can you be so obtuse? – Anthony 

(Note: Nick's since advised in the comments there are 49 stations in Texas, not 188)

Anyone who's read any of Anthony's blog knows who is obtuse, and it's not Nick Stokes. There's more. Nick Stokes says:

June 28, 2014 at 2:39 pm
“Wait, lets get blood from the nearest five guys that look like him and mix it together. Yeah that’s a reasonable estimate.”

They are computing a spatial average, based on stations. Infilling with neighboring data doesn’t change anything. It just, in the final sum, changes the weighting. The neighboring stations get a bit more weight to cover the area near Luling. 

As I showed in the shaded plots, there is plenty of data in the region. It doesn’t depend on Luling. Using a neighbour-based estimate is just the way of getting the arithmetic to work properly. With anomalies you could just leave Luling out completely. With absolute values, you have to do something extra, so that the climatology of the omitted Luling doesn’t create Goddard spike type distortions. Estimating from neighbor values is the simplest way to do it properly.

This is where Anthony Watts first makes the unsubstantiated claim that "80%" of the US temperature network is "compromised by bad siting". In other words, he's now shifted to claiming that virtually all the US temperature data is worthless.

REPLY: Oh Nick, puhlease. When 80% of your network is compromised by bad siting, what makes you think those neighboring stations have any data thats worth a damn? You are adjusting bad data with…drum roll….more bad data. And that’s why homogenization fails here. It’s perfectly mathematically legitimate, but its useless when the data you are using to adjust with is equally crappy or crappier than the data you want to “fix”.

The problem with climate science is they really have no handle on just how bad the surface network is. I do. Evan does, John N-G does. Even Mosher has a bit of a clue.
You can’t make a clean estimated signal out of a bunch of muddied signals, ever.
Now its well past your bedtime is Australia. Maybe that is why you aren’t thinking clearly -Anthony

This, of course, means that Anthony's argument falls apart. Why bother fixing a weather station if they are all wrong anyway? Why not simply discard 80% of the entire record or forget about US temperature data altogether.  He finishes up by lecturing Nick Stokes on staying up past what Anthony mistakenly thinks is his bedtime.

Nick picks up on Anthony's contradictory stance. Nick Stokes says:

June 28, 2014 at 3:00 pm
“You can’t make a clean estimated signal out of a bunch of muddied signals, ever.”

Then there’s no point in discussing analysis, is there? But it is the job of NOAA and USHCN to interpret the data, as best they can, even if you think it is worthless. And I think at Luling they did everything right. They picked up a problem, quarantined the data, and got the best estimate available with the remaining data.

“Now its well past your bedtime is Australia. Maybe that is why you aren’t thinking clearly”

When it’s afternoon in California, the sun is over the Pacific somewhere. It’s 8am here.

This time Anthony elaborates on his unsubstantiated claim about "compromised" network, and claims that it has a "warm bias". He replies to Nick Stokes:

REPLY: Right you are, I thought you’d been up all night based on your commentary elsewhere. I also thought you lived in Perth. Obviously not.

Estimating data is the issue, and again when you use let’s say the six nearest stations, and statistically as we have shown at least 80% of them are unacceptably sited, resulting in a warm bias (and that’s not just my opinion that’s from Leroy 99 and 2010, and NCDC’s use of that to setup USCRN), that means your signal is going to be biased, full of the mud from the other stations.

It renders the idea of a useful estimate pointless.

And if you are too obtuse to see that, then yes, there’s nothing else to discuss -Anthony

If you extrapolate what Anthony is arguing, you soon realise that firstly he's arguing that 80% of the records are wrong. He is arguing that there is a "warm bias". He's not provided any evidence for that being so. Nor does he indicate for how long that "warm bias" will last. He's also basically saying that if it's 20C in all places closely surrounding place A one cannot assume that it's 20C in place A.


I looked at Anthony's claim that there is a warm bias compared to the Climate Reference Network (USCRN), well I don't see it. The overlap between the new USCRN, the newish ClimDiv and the older USHCN records is so tight that you can't pick one from another. (The USCRN data starts in 2004.)

Data source: NCDC/NOAA

This is confirmed by the NCDC on this page. Check out the chart. The differences are so minute they are almost indistinguishable.

Interpolating sea ice extent and maintaining the DJIA

Anthony finally comes up with what he thinks is a killer argument. Anthony Watts says:

June 28, 2014 at 4:40 pm
By Nick Stokes thinking, we could use FILNET to make up for missing ice in the Arctic extent maps by interpolating from nearby ice readings and “infill” where ice is missing. We know there’s supposed to be ice there, so let’s just infill it from surrounding ice data, even if its “rotten ice”.

Arctic problem solved. Polar bears saved!

No, wait, that would be wrong….and equally ridiculous.

Making up data where there is none, especially for years for long dead weather stations, is just wrong. If it were financial data, say companies that went bankrupt and closed, and fell off the Dow-Jones Industrial average, but somebody decided that they could “fill in” that missing company data to keep the “continuity” of the DJIA data set over the years, you can bet that somebody would be hauled off to jail within a day or two by the SEC.

Fixing a few missing datapoints in a month with FILNET to make the record useable is one thing, wholesale reanimation of dead weather stations for years is something else altogether.

Except that is exactly what happens with missing data within an image of sea ice. This is how it's done at NSIDC (my bold italics):

There are instances of missing data. In some cases whole days (or weeks or months) are missing. In other cases, large swaths or wedges of missing data exist within an image, along with scattered pixels of missing data throughout the grid. The scattered pixels of missing data, resulting generally from mapping the orbital data to the SSM/I grid, were filled by applying a spatial linear interpolation scheme on the brightness temperature maps. The larger areas of missing data, resulting from gaps between orbital swaths (generally at low latitudes on daily maps) or from partial coverage or missing days, were filled by temporal interpolation on the sea ice concentration maps. 

It's only when there is no data at all that the (time) gap is left unfilled:

No data at all were available for the period from 02 December 1987 through 12 January 1988. This gap was not filled by temporal linear interpolation; instead it was left as missing data.

BTW, Anthony's analogy with the Dow Jones index is Freudian, don't you think. I wonder is Anthony aware that the companies that make up the index do change over time? What does he think happens when there's a stock split? Perhaps Anthony thinks that the DJIA still comprises the original twelve industrial stocks rather than the current thirty.

Infilling missing temperature data doesn't change the overall picture

Zeke Hausfather chimes in and tries to get back on point, which is not that Steve Goddard is correct because he's not. Nor is Anthony Watts. Zeke Hausfather says:

June 28, 2014 at 6:04 pm
If you don’t like infilling, don’t use it. It doesn’t change the result, almost by definition, since infilling mimics spatial interpolation: http://rankexploits.com/musings/wp-content/uploads/2014/06/USHCN-infilled-noninfilled.png

The interesting issue currently is that some stations that report apparently valid raw data are being replaced with estimated data. The cause seems to be that the NCDC data file is missing the X flag, which indicates that the data was too inhomogeneous at the time (e.g. between two station moves) to figure out what is going on. The folks at NCDC are looking into it, as the number of stations that fall into this category seems to be a bit high, at least in my opinion.

Also, the confusion here was on Anthony’s part rather than mine; I always knew that NCDC used infilling to ensure that there were 1218 reports per month in the homogenized dataset. I personally think infilling is silly, since its not really needed (as any sort of reasonable spatial interpolation will produce the same result). But I understand its something of a legacy product to ensure consistency for folks who want to calculate average absolute temperatures.

Anthony wasn't "confused", just mistaken

Anthony tries to make out he wasn't "confused", after confusing all his denier readers and says he just didn't know in his reply to Zeke's comment above:

REPLY: Confusion is the wrong word, I simply didn’t know that NCDC was reanimating dead weather stations for the final dataset. I agree, it is silly.
However I disagree that it doesn’t make a difference, because the majority 80%+ of stations are non-compliant siting-wise. A small minority are compliant, and the infilling and homogenization obliterates their signal, and those stations are by definition, the most free from bias. As we have shown, compliant stations have a lower trend than non complaint stations, and a far lower trend than final adjusted data.

Basically what NCDC is doing here is mathematically favoring the signal of the crappiest stations – Anthony

Deniers don't self-correct - that's left up to other people

Let's complete the circle. Deniers all claim to just want to get to the truth. Judith Curry wrote (excerpts):

Who do I include in the technical skeptical blogosphere?  Tamino, Moyhu, Blackboard, Watts, Goddard, ClimateAudit, Jeff Id, Roman M.  There are others, but the main discriminating factor is that they do data analysis, and audit the data analysis of others. ... 

...However, the main point is that this group is rapidly self-correcting – the self-correcting function in the skeptical technical blogosphere seems to be more effective (and certainly faster) than for establishment climate science.

Have any of the people involved in spreading FUD corrected their articles? Is anyone under the mistaken impression that deniers care about anything other than trying to prove that physics and chemistry and biology are all "wrong"?

• Steve Goddard (denier) - No. His "fabricated" claim is still up there and he's repeated it, despite being proven wrong and wrong and wrong again. He's a conspiracy nutter.
• Anthony Watts (denier) - No. He's left a complete mess on his blog. He's kissed and tried to make up with his brother-in-denial Steve Goddard but didn't make plain that Steve Goddard was wrong about "fabrication". And along the way he blustered and bluffed and added unsubstantiated claims of a "warm bias".
• Judith Curry (denier) - No. As usual, she's just left her normal unsubstantiated fare "doubt and uncertainty" mixed with "astonishing" and "political ramifications".
• Paul Homewood (denier) - No. He's left his "something nefarious is going on" article unchanged. He did an update but only mentioned that the Texan station moved a bit. He didn't point out that the weather station was not reporting correctly.

The only people who published correct information and showed a keenness to expose the facts were Zeke Hausfather and Nick Stokes (who don't deny science) - Nick Stokes investigated the record for Luling Texas, saw that it was anomalous. Found out the reason why it went askew and published an update.

So out of all the above, it's only Nick Stokes (and Zeke Hausfather) who bothered to write about the real situation. The science deniers are only interested in peddling denial, doubt and disinformation.