.

Monday, August 31, 2015

On reproducibility: Replicated errors and double standards from Anthony Watts at WUWT

Sou | 2:32 PM Go to the first of 24 comments. Add a comment
The most recent article from Anthony Watts, who runs a climate conspiracy blog called wattsupwiththat, or WUWT, demonstrates once again that he doesn't understand the first thing about scientific research. It also provides another lesson in the telltale techniques of climate science denial, and the double standards of Anthony Watts. 

What the WUWT article (archived here) is about is an article in the New York Times (without a link), which in turn is an article about a paper in Science about the lack of reproducibility of many psychology papers. That is, Anthony published a "guest essay" about a New York Times article about a paper, topped and tailed with some irrelevant silly comments from Anthony himself. (Anthony doesn't write much himself these days. He's probably conscious that most of what he writes is too silly for public consumption so he copies and pastes press releases and denier blog articles from elsewhere, and relies on "guest essays" of dubious quality from his readers.)

If you skip to the end of the WUWT article, you'll notice an irony that suggests that Anthony is not only ignorant but lacking in critical thinking skills. Or maybe he regards his readers as lacking in intelligence. At the end of his copy and paste he added:
Related stories where Stephan Lewandowsky, Naomi Oreskes, and John Cook produce some questionable, and perhaps irreproducible psychological “science”:

Lewandowsky and Cook – back from the dead with another smear paper

A disturbance in the farce: Another hateful and pointless paper from Stephan Lewandowsky and Naomi Oreskes

And then there’s Rasmus Benstad’s recent laughable paper that five journals rejected, before he and his psyops crew found a journal that would publish what other the five journals rejected. Yeah, that’s what you might call “robusted” science.
The telltale technique Anthony used was Technique No. 2, the logical fallacy. He was claiming that a paper that argued that a large proportion of psychological studies were not reproducible, meant that two particular psych studies and one study showing that selected science papers weren't replicable, were "questionable, and perhaps irreproducible". That does not follow. It's a non sequitur, a logical fallacy.

Take Anthony's examples one by one.



Anthony Watts' first fail - a bout of recurrent fury


The first paper he refers to is Recurrent Fury, which you can read about here. Not only would that work most likely be reproducible, it was validated not once but arguably four times in the same paper! Anthony thinks it's a "smear" paper because it's about the evolution of conpiracy theories in the blogosphere. Something that you'll see daily at blogs like Anthony's WUWT. The paper reports how it:
If you read the paper, you'll also see how it cites other papers and builds on work that has been done before. A great example of how psychological studies can be validated and taken even further.


Anthony Watts' second fail - rejecting seepage of a "pause"?


The second paper that Anthony referred to was about seepage of denier memes into the scientific literature.  You can read about that paper here with links to more here. It would not be at all difficult to replicate that work. It validates itself in the scientific literature on climate. It makes you wonder if Anthony Watts is:
  • Denying that there has been a "pause" in global surface temperature
  • Accepting it was a slowdown, not a "hiatus"
  • Rejecting the notion that deniers like him claimed that global warming has "stopped" or that it started cooling 16, 17, 18, 19 and 23 years ago!
  • Decided his denial has absolutely no influence on climate science.

Anthony Watts' third fail is a spectacular replication of his mistakes


Anthony left his most spectacular fail to last. His evidence that many psychological studies can't be replicated is a paper demonstrating how specific climate science papers that disputed global warming couldn't be replicated! Let me put that another way. The paper that Anthony cites as "evidence" of non-replicability is a paper about non-replicability!

You can read about that paper at the Guardian in an article with the title:


You can also read about it at realclimate.org - Let’s learn from mistakes, something that deniers are seemingly quite unable to do.


Reproducibility of science


The paper that was being talked about in today's WUWT article was about reproducibility of papers on psychology. (It builds on work that was flagged in another paper from a couple of years ago.) From the abstract:
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. Scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence. Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error.
The study reported that it evaluated reproducibility using:
  1. significance and P values, 
  2. effect sizes, 
  3. subjective assessments of replication teams, and 
  4. meta-analysis of effect sizes. 
And the authors stated that although ninety-seven percent of original studies had statistically significant results, only thirty-six percent of replications had statistically significant results. The authors discussed scientific pressures for publication in terms of innovation vs replication. And how the pressure is on innovation, that simple replication isn't likely to result in publication. You'll need to read the paper to get a better feel for the work and what it signifies. 

I'll add that psychology is probably one of the most difficult sciences - unravelling the complexity of the human mind and human behaviour.


What Anthony claimed from the paper


At the top of his article, Anthony wrote:
Shades of Lewandowsky and Cook: When psychological science isn’t so sciency
Sound familiar? A study revealing a stunning lack of reproducibility in psychological science triggers another instance of reluctance to share data with any but friends, and an “adjustment” of  data to fit a theory.

First of all, the paper he wrote about didn't "trigger" any such thing. It was only published last week and the work that Anthony cited was published well before that. What it did trigger was another bout of conspiracy ideation at WUWT.

Nor does it signal a "reluctance to share data".  Could it possibly be that where Anthony Watts was complaining loud and long that a paper on conspiracy ideation did provide evidence to blog comments directly, now he's complaining that the replacement paper was anonymised? Surely not! (The data for Recurrent Fury is available to responsible researchers on application.)



Footnote about the WUWT author: one of Anthony's own "anonymous cowards"


Anthony's article was by "guest blogger" Frank Lee MeiDere. It didn't take much to discover that he is a WUWT reader and a blogger who says of himself:
I trust you'll realize that's not my real name, but I'm an English prof at a Toronto college and I figure a low profile is in order.

That same blogger on his own blog says: "while it’s possible that I may not like you, it’s far more probable that either as myself or as Frank Lee MeiDere, I just don’t give a damn."

So it turns out that "Frank Lee MeiDere" is one of Anthony Watts' pet anonymous cowards. Anthony denounces and despises pseudonyms, but only when they are used by pro-science people. I've no problem with pseudonyms, needless to say. What this episode does demonstrate once more are the double standards of Anthony Watts, and his inconsistency.


From the WUWT comments


Now since the blog owner, Anthony Watts, hasn't learnt anything at all from his mistakes, what are the chances that he attracts any readers who are capable of critical thinking or learning? Nil? Next to nil? You can see for yourself in the comments:


Pat Ch has a hypothesis which is not based on any research, let alone reproducible research. One might call it conspiracy ideation:
August 30, 2015 at 5:11 pm
Green energy, climate, sociology, anthropology, psychology. What do they all have in common? They are controlled by third rate scientists who are really left wing ideologues.

Chip Javert claims some expertise in the scientific method and declares all political, psychological and climate science null and void. Again, his hypothesis is not based on any research let alone reproducible research. Again, it is more properly compared to conspiracy ideation:
August 30, 2015 at 4:18 pm
Cogsys
Nice rant, but most of it simply is eye-wash.
Political science, psychology and “global warming” do not use the scientific method, and, thus, are not sciences (they are “fields of observation”).
The scientific method is not forgotten (popular internet material, written by non-scientifically trained journalists IS NOT SCIENCE). I would refer you to the recent search for the Higgs Boson – that definitely was not decided by a “consensus” vote; nor was the technology in your smart phone, PC, GPS…
If you choose to further argue this point, knowing your educational background would be interesting (mine: Ga Tech physics).

Allencic might be referring to the paper that Anthony cited, but I think not. It's more likely that he's got a (conspiracy) theory about mainstream science:
August 30, 2015 at 4:02 pm
Anyone who thinks replication of results doesn’t matter (such as most so-called “climate science”) needs to go back and re-read the story of “Cold Fusion.” Good intentions, crappy and false results. 

wickedwenchfan is either a long-standing Poe or the resident extremist at WUWT, so as to make the regular nutters appear not quite so extreme: 
August 30, 2015 at 3:34 pm
The world of physics is sadly no better. If it were you wouldn’t be hearing about absurd magic like Dark Matter, Dark Energy, Black Holes, Snowball Comets or Back Radiation!

MarkW doesn't "believe in" physics, engineering, biology, chemistry, economics, finance, ecology or any other field that uses computer models to explore and extend knowledge. And he doesn't "believe" that climate scientists go out in the field and collect and analyze actual data! I'm surprised that he owns a computer:
August 30, 2015 at 7:03 pm
Climate science is not one of the hard sciences. Never has been. Hard sciences go out into the field and collect and analyze actual data. They don’t use statistical “tricks” to compensate for missing data, or data that has been polluted by confounding affects.
And they certainly don’t use computer models for “proof” of anything. 

You'd think this would have been predicted (it was - pers comm). First deniers complain that evidence was made public in the supplement to Recursive Fury, and threatened the journal till it withdrew the paper. Now they complain that it's only made available to approved researchers on request - in Recurrent Fury, the anonymised version! Read about it at Shaping Tomorrow's World. Louis Hunt wrote:
August 30, 2015 at 2:11 pm
Lewandowsky’s original findings were replicated by whom? He refused to provide his data to researchers, so how could anyone replicate his findings unless he did it himself?

I'll finish with a comment from Anthony's "anonymous coward", Frank Lee MeiDere. He could be writing about any and almost everyone who posts their various random "thoughts' at WUWT - bald statements unsupported by evidence, with not even a reference:
August 30, 2015 at 1:27 pm
I taught at a city college for almost 10 years, and virtually every student started my class believing that a couple of references from second or third-level sources constituted proof. More disturbing, however, was their attitude that the references didn’t really matter; what counted was what they (the students) believed was right. 


References and further reading


Open Science Collaboration (Brian A. Nosek). "Estimating the reproducibility of psychological science." Science 28 August 2015: Vol. 349 no. 6251 DOI: 10.1126/science.aac4716 (subs req'd)

Open Science Collaboration (Brian A. Nosek). "An open, large-scale, collaborative effort to estimate the reproducibility of psychological science." Perspectives on Psychological Science 7, no. 6 (2012): 657-660. doi: 10.1177/1745691612462588 (open access)

Many Psychology Findings Not as Strong as Claimed, Study Says - article by Benedict Carey in The New York Times

Benestad, Rasmus E., Dana Nuccitelli, Stephan Lewandowsky, Katharine Hayhoe, Hans Olav Hygen, Rob van Dorland, and John Cook. "Learning from mistakes in climate research." Theoretical and Applied Climatology (2015): 1-5. doi:10.​1007/​s00704-015-1597-5 (open access)
Lewandowsky, Stephan, John Cook, Klaus Oberauer, Scott Brophy, Elisabeth A. Lloyd, and Michael Marriott. (2015). "Recurrent fury: Conspiratorial discourse in the blogosphere triggered by research on the role of conspiracist ideation in climate denial." Journal of Social and Political Psychology, 3 (1). doi: 10.5964/jspp.v3i1.443. (open access)

Restoring Recurrent Fury - by Stefan Lewandowsky at Shaping Tomorrow's World

Stephan Lewandowsky, Naomi Oreskes, James S. Risbey, Ben R. Newell and Michael Smithson. "Seepage: Climate change denial and its effect on the scientific community." Global Environmental Change, 2015 doi:10.1016/j.gloenvcha.2015.02.013 (open access)


From the HotWhopper archives

24 comments:

  1. I'd like to (anonymously) point out this apparent discrepancy, or perhaps, contradiction;

    Sou says: "... one of Anthony Watts' pet anonymous cowards...".

    But Sou goes on to say, "... I've no problem with pseudonyms, needless to say...."

    ReplyDelete
    Replies
    1. No contradiction, marke. Anthony Watts calls pro-science people who use pseudonyms "anonymous cowards". I don't.

      Delete
  2. By the way, Sou, do you have any views on;

    The Hood Robin Syndrome
    Willis Eschenbach / August 29, 2015

    On WUWT as a guest article, of course.
    I come here to get an alternative view of things, and was hoping this one would come up, as it is on a topic we have discussed here at some length on previous occasions.

    ReplyDelete
    Replies
    1. No. I have just skimmed it now as a result of your comment but haven't done any analysis. It seems to be about how some policies work against the least wealthy in the USA, and favour middle income and wealthy people.

      The carbon price in Australia was a better model. It put a price on carbon, but for people on low and middle incomes, they got all the cost back - in some cases more than all the cost. It was managed separately from consolidated revenue so was relatively transparent. Unfortunately the Abbott government dismantled it and replaced it with a non-transparent model, hidden in consolidated revenue. It's sent us backwards in regard to mitigation efforts.

      When governments stop using taxpayers to prop up fossil fuels, then we might start to make progress.

      http://reneweconomy.com.au/2015/australia-still-subsidising-fossil-fuels-at-rate-of-1712-per-person-a-year-33164

      Delete
    2. marke -

      IMO, it's an interesting article in that it refers to an interesting and I think useful study that helps to give perspective on the impact of some specific energy-related policies.

      Unfortunately, it's also full of the kind of sanctimony and righteousness that can typically be found in Willis' articles - where he tries to leverage and exploit "da poor" to score cheap points in the climate wars.

      As Sou alludes to, I would think that a full treatment of the article would talk about the distribution of benefits from fossil fuel subsidies, the full range of impacts on the poor from BAU w/r/t emissions, as well as be careful to not make facile implications that the benefits distributed from energy tax credits would otherwise be distributed to the world's poor - particularly given that the political audience to which the article is directed are typically among those who would be most likely to object to a distribution of those benefits to the world's poor.

      What was your take on the article?

      Delete
    3. And since the study was focused only on the impact of the tax subsidies, it necessarily ignored the distributional effects of other programs included in the ARRA (most of the tax subsidies were part of or were enlarged as part of the ARRA) to assist lower income people with things like weatherization. So they were not looking at the distributional effects of the entire alternative energy and conservation program in the ARRA, only the part which was not intended to have a downward distributional effect.

      Delete
    4. Rattus -

      Yeah, I did a quick glance to see if they quantified the impact of weatherization (since they mentioned it as part of the incentives) ...but didn't come across any...but thought I needed to look harder.

      I didn't realize that they didn't even look at the distribution of those benefits in the original study....which seems a bit odd since they spoke of those benefits in the intro....

      1) Did they give any rationale for their selectivity of focus?
      2) Do you know of any analyses that look at the distribution of those other benefits?

      Delete
    5. They only looked at tax subsidies (it was what they were interested in, probably Republicans in the B. School) and not at direct subsidies. My landlord took advantage of the weatherization subsidy and that ended up saving me about 25% on my heating costs the next winter.

      Delete
    6. Rattus -

      Just saw your comments over at WUWT. Well done (although I have no idea why you bothered). Not surprisingly, Willis skipped over a chance to actually discuss the issues you raised and instead focused on proving his moral superiority (I wonder why he spends so much time doing that?).

      Perhaps since marke wanted to discuss the article, he'll respond on the issues you raised over at WUWT? Maybe even he'd be willing to comment on the sanctimony dripping from Willis' response to your comments? That would be a refreshing change from a "skeptic."

      Delete
    7. Joshua, Rattus,

      My take is that it is a fine example of poorly planned, knee jerk policy making/politics-playing with unforeseen (yet foreseeable) consequences.

      Mind you, not so many years ago, I thought this direct subsidizing of renewal energy setups at a household level was the ideal way to increase take-up, and that a decentralized grid was going to be the answer to all things. But, I was just musing from the sidelines, and it was not my job to foresee the foreseeable. Those whose job it was, either overlooked it, or, more likely, brushed it under the rug in the interests of the political expediency of the time.

      In Willis' comment thread, I agree with Rattus that income equality was not the intention of the policy, but I'd also assume that creating an obvious inequality was not a very useful outcome.

      I also agree with Rattus that all policy is a matter of intentions. Whether those intentions are achieved (or indeed were what we expected them to be, as at times political expediency trumps realism) or whether other unforeseen consequences come to predominate are another matter.

      Re weatherization (in the UK): I understand this subsidy was cancelled due to an unexpectedly low uptake. It is surprising to me that people would ignore such an opportunity, but again, there are those out there whose full time job is, supposedly, to make these things work.

      I make no comment on Willis' sanctimony, other than to note I quite often get assailed by a rather religious minded righteousness in these very pages for questioning data and/or proposed solutions. (ie, There is plenty of it going around).

      Delete
    8. curse teh typos ... "..is another matter. .."

      Delete
    9. marke -

      ==> "but I'd also assume that creating an obvious inequality was not a very useful outcome."

      How did the program create an inequality?

      The problem with the sanctimonious rhetoric in Willis' article isn't that anyone is assailed, IMO - it's how it undermines the reasonableness of his analysis. It leave his analysis shallow, vapid even.

      And taking the time to say "they do it too" doesn't seem at all like a "no comment" to me.

      Delete
    10. marke -

      ==> "but I'd also assume that creating an obvious inequality was not a very useful outcome."

      How did the program create an inequality?

      The problem with the sanctimonious rhetoric in Willis' article isn't that anyone is assailed, IMO - it's how it undermines the reasonableness of his analysis. It leave his analysis shallow, vapid even.

      And taking the time to say "they do it too" doesn't seem at all like a "no comment" to me.

      Delete
    11. Josh ... it was no comment ... other than...
      not ... no comment.

      Delete
    12. How did the program create an inequality?

      Sorry, I should have said; "exacerbate existing inequalities".

      Delete
    13. marke -

      I dunno. "No comment...other than..." -a comment where you say they do it to, seems to me like you're letting yourself off the hook.

      ==> "Sorry, I should have said; "exacerbate existing inequalities".

      Correctly noted...but still, how meaningful do they "exacerbate existing inequalities?"

      A reasonable question is whether there is meaningful opportunity cost, but the answer to that question involves a complicated speculation about an unknowable counterfactual. As such, when you see the kind of rhetorical leverage applied by putative "skeptics" like Willis, then don't you ask yourself about why they are so susceptible to such blatant biases?

      Delete
  3. I think a carbon tax is preferable to any CET scheme I have seen, though I suspect there may be better methods of equalization than refunding based on income.

    I note from your linked article, the US$1,260 per capita 'subsidy' per year (for Australia) discussed mainly consists of uncosted externalities rather than any major direct payment of subsidies....

    (ie .... the report projects global energy subsidies at $US5.3 trillion in 2015, or 6.5 per cent of total GDP – most of which, it says, arises from countries setting energy taxes below levels that fully reflect the environmental and health damage associated with energy consumption.)

    ReplyDelete
  4. I bet Frank Lee thought he had come up with a brilliant pseudonym (see Private Eye letters page ad nauseam). I believe it comes from a film. I found some footage of Ronald Reagan auditioning for the male lead in that film (scroll to 6:30):

    https://m.youtube.com/watch?v=LLBwP5EhftE

    ReplyDelete
  5. I snuck this comment in:

    Frank Lee MeiDere, Do you give a damn? :D

    ReplyDelete
    Replies
    1. At WUWT I mean. No one has noticed it yet.

      Delete
    2. Oh, Hurrah!

      R the ANon

      Delete
  6. Somehow, methinks, we could come up with a long list of psychological research articles that Anthony would just happen to exclude from his "concerns" about reproducibility. Perhaps, for example, articles written by Jose Duarte?

    Of course that the conclusions of the articles he'd exclude would happen to align with his own opinions would be pure coincidence.

    ReplyDelete
  7. I'm amused that anyone who runs a blog which includes articles that you originally printed on a blog with your real name all over it which links to your 12 page resume "magazine" expects to maintain any anonymity.

    That Kit thinks this with his self-advertised experience in digital media is a bit, well, special...

    ReplyDelete
    Replies
    1. Sorry, lost a bit in editing - the above pertains to Mr MaiDere, the "anonymous coward", in case that's not obvious.

      Delete

Instead of commenting as "Anonymous", please comment using "Name/URL" and your name, initials or pseudonym or whatever. You can leave the "URL" box blank. This isn't mandatory. You can also sign in using your Google ID, Wordpress ID etc as indicated. NOTE: Some Wordpress users are having trouble signing in. If that's you, try signing in using Name/URL. Details here.

Click here to read the HotWhopper comment policy.