tag:blogger.com,1999:blog-2313427464944392482.post3617555400112442842..comments2024-03-25T05:30:23.847+11:00Comments on HotWhopper: Ridiculous Richard Tol sez 12,000 is a strange number...Souhttp://www.blogger.com/profile/08818999735123752034noreply@blogger.comBlogger50125tag:blogger.com,1999:blog-2313427464944392482.post-45004354975034923142014-07-22T02:09:56.471+10:002014-07-22T02:09:56.471+10:00I'll mention this here as it seems to have esc...I'll mention this here as it seems to have escaped most except at DesMogBlog which produced a noteworthy assessment of Tol's position in <a href="http://www.desmogblog.com/2014/07/14/richard-tol-dons-cloak-climate-denial" rel="nofollow">Richard Tol Dons Cloak of Climate Denial</a>, where a <br /><br />'...small gathering of climate science deniers, including Conservative MP and member of the UK House of Commons Select Committee on Climate change, oilman Peter Lilley, and Conservative MP for Monmouth, David Davies, met in a small room buried on the third floor of the UK’s House of Commons in London last week.'<br /><br />And one Piers Corbyn was up to stupid stunts:<br /><br />'Tol’s talk was weirdly interrupted by whooshing sounds from the back of the room where Piers “sunspots” Corbyn was inflating a huge plastic globe.'<br /><br />This shower would surely belong in a circus were it not for the Danse Macabre that will be enacted because of their dragging effect on the taking affirmative action that will help mitigation.Lionel Ahttp://lionels.orpheusweb.co.uk/noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-39194115332270544562014-06-11T06:57:37.472+10:002014-06-11T06:57:37.472+10:00Given that only 59% of Cook's rankings are wro...Given that only 59% of Cook's rankings are wrong, I can understand why it's so hard for shub to come up with an example of one miscategorized abstract.numerobisnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-89213388736609434482014-06-11T04:33:16.187+10:002014-06-11T04:33:16.187+10:00"I don't know where you got your numbers ..."I don't know where you got your numbers from ..."<br /><br />Sou, the numbers come from Cook's data.<br /><br />If you consider disagreement ratings, volunteers rated 1869 abstracts as '3' the first time. The same volunteers rated 1510 of the same abstracts as '4' the second time, an inconsistency of 81%.<br /><br />Volunteers rated 1089 abstracts as '4' the first time. The same volunteers rated 904 of the same abstracts as '3' the second time, i.e., an inconsistency of 83%.<br /><br />The above is one-tailed. You get similar results the other way around.<br /><br />The data presented graphically: http://nigguraths.wordpress.com/?attachment_id=4219<br /><br />What are the odds of Cook's method identifying a '3', assuming their final as true positive? 0.41(1187/2910). Would you get a test for cancer if you were told the chances of ruling out cancer were worse than a coin toss? <br /><br />Since Cook's group cannot reliably identify '3', it cannot reliably come to the '97%' conclusion.shubhttp://nigguraths.wordpress.comnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-27643620309729067132014-06-11T00:50:52.975+10:002014-06-11T00:50:52.975+10:00It's not much but this is worthwhile:
http://...It's not much but this is worthwhile:<br /><br />http://ruptresearch.weebly.com/9-untitled.html<br /><br />It is a piece of research on examiner fatigue which is much more relevant because the assessment of the abstracts is somewhat like marking exam scripts. For Shub, the final two sentences say:<br />"Finally, despite the fatigue experienced in this OSCE and regardless of the method in which the examiners scored student performance, all examiners were able to concentrate over time. These results suggest that an examiner can concentrate despite experiencing fatigue. "<br /><br />Not saying that's the end of the matter but it is another nail. Finding it difficult to find more room to bang in more nails.Catmandohttps://www.blogger.com/profile/12313870265499015076noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-90883942425162562602014-06-11T00:33:36.660+10:002014-06-11T00:33:36.660+10:00Shub, go look up the Cochrane Database of Systemat...Shub, go look up the Cochrane Database of Systematic Reviews. People working on systematic reviews routinely rate 1000s of abstracts. Of course you and Tol wouldn't understand this, because you're both systematic review amateurs (well, Tol is; I doubt you even know what a review is). You're putting your stock in someone who can't even get their own systematic review of 12 papers right (maybe he got tired, poor dear). Why?Captain Flashheartnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-34286933372622276602014-06-11T00:24:13.163+10:002014-06-11T00:24:13.163+10:00Shub, survey *interviewers* are quite different fr...Shub, survey *interviewers* are quite different from survey *takers*. This is a crucial difference, and also the main reason why the available literature on survey fatigue is largely irrelevant, as that focuses on survey *takers* getting tired of "yet another survey" or a "too long survey" which they have limited interest in. The survey interviewers, especially in tis particular case, do have a special interest in the survey. Moreover, in this particular case it was short surveys and time availability was completely up to the 'respondent', unlike with yet another known survey fatigue issue: having to respond to the interviewer's questions rather quickly. No walking away for a few minutes.<br /><br />And as noted by many, not only is there a potential "fatigue", there also is "experience" that increase with time. I remember my own time spent on reading scientific papers, easily spending 2-3 hours to understand a 16-page paper. Currently I can do this in 30 minutes if I really want to understand the details, 5 minutes to just do a simple BS check and get the main story.Marconoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-86909892797924796562014-06-10T23:30:54.682+10:002014-06-10T23:30:54.682+10:00Shub you've made foul allegations that the res...Shub you've made foul allegations that the researchers were dishonest and just filled in any response they liked.<br /><br />You're wrong about the false equivalence you've made. The researchers weren't taking a survey, they were evaluating abstracts and categorising them. The equivalent of a market researcher analysing responses to open ended questions at best. You've not provided any evidence of anything. Just made up stuff.<br /><br />I don't know where you got your numbers from or why you think they are relevant. Even if they were (though I doubt it), categories 3 (implicit endorsement) and 4 (no position) could often be line ball calls. The majority of first and second ratings agreed (66%). When they didn't they were either reconciled between the researchers or went to a third independent person. You might be working off Richard's numbers, which are up the creek. From the Cook response:<br /><br /><i>The assertion of rater drift is based on analysis of the average endorsement level, using ordinal labels from 1 to 7. However, average endorsement level is not an appropriate statistic for making inferences about consensus percentages. C14 replicates T14’s analysis using the more appropriate consensus percentage calculated for 50-, 100- and 500-abstract windows. We find no evidence of the claimed rater drift. Consensus among initial ratings in a window falls outside the 95% confidence interval 2.8%, 3.2% and 1.7% of the time for 50-, 100- and 500-abstract windows respectively.</i><br /><br />What you have failed to come up with is a single bit of evidence to show that 97% of scientific papers that attribute a cause to the current rapid warming is in error. Now why not focus on that? <br /><br />If you spent an hour a day doing it all by yourself you'd be able to finish the job in about six months. If you worked with a mate, you'd get it done in three months. Work with two mates each for an hour or so a day, you'd get them all categorised in less than two months. Then you could double it and do each others if you wanted to. Or you could rope in another couple of mates and get it done in no time at all. The researchers have prepared the abstracts for you and handed them to you on a platter, saving you a lot of work.<br /><br />Instead you pretend "something must be wrong" without the slightest shred of evidence - all you're good for is bluster and sleazy innuendo. What a lazy, good for nothing so-and-so you are. A credit to the denier illiterati - and that's about it.Souhttps://www.blogger.com/profile/08818999735123752034noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-48555080993897767672014-06-10T23:22:26.101+10:002014-06-10T23:22:26.101+10:00Tol is sinking along the same demented trajectory ...Tol is sinking along the same demented trajectory as Curry. Interesting phenomenon.cRR Kampenhttps://www.blogger.com/profile/07571285063752477448noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-25745270356002587962014-06-10T23:05:04.757+10:002014-06-10T23:05:04.757+10:00Marco, Shub isn't in the business of doing res...Marco, Shub isn't in the business of doing research. He's in the smear, uncertainty, doubt business - much less demanding and "no answer" is always the right answer in deniersville.<br /><br />Here's <a href="https://www.google.com.au/search?q=survey+fatigue&oq=survey+fatigue&aqs=chrome..69i57.6008j0j7&sourceid=chrome&es_sm=122&ie=UTF-8" rel="nofollow">a link for you</a>. They are all about respondents not interviewees. Here's a sample where I've replaced "respondents" with "scientific abstracts", which if there was any equivalence (there isn't) would be the equivalent. It's an article about how to make sure the researchers can make the survey interesting to the abstracts, so the abstracts don't suffer survey fatigue and either not respond or get silly responses from the abstracts or have the abstracts complain that they are too tired to take part.<br /><br /><i>From <a href="http://www.surveygizmo.com/survey-blog/5-basic-ways-to-avoid-survey-fatigue/" rel="nofollow">surveygizmo</a>:<br />...we see so many surveys fail to collect useful data simply because they’re not designed to keep the scientific abstracts interested.</i>Souhttps://www.blogger.com/profile/08818999735123752034noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-83277502591216198652014-06-10T23:00:42.868+10:002014-06-10T23:00:42.868+10:00What dishonesty are you talking about, Sou? None o...What dishonesty are you talking about, Sou? None of my comments above imply dishonesty.<br /><br />[1] From the '24' document:<br /><br />"The raters performed the function of a survey interviewer in the process of rating abstracts". <br /><br />In surveys, the questions are the passive agents and the survey takers are subject to fatigue. In Cook's project, the abstracts are passive agents and the survey interviewers are subject to fatigue.<br /><br />The literature on survey fatigue is entirely relevant here.<br /><br />[2] "The researchers would have got more proficient over time."<br /><br />Available data contradicts this. If true, ratings would progressively get more consistent. What is actually seen is the opposite: the second ratings are less than 50% consistent with the first. In other words, the probability of distinctly identifying a consensus category is worse than a coin toss. Among disagreeing ratings, ~80% belong to exact opposite ratings, i.e., 80% of '3' is identified as '4' and 80% of '4' as '3'. <br /><br />The passage of time did not improve rating accuracy.shubhttp://nigguraths.wordpress.comnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-16921986615844722132014-06-10T22:29:39.347+10:002014-06-10T22:29:39.347+10:00First of all, it wasn't a survey. It was an an...First of all, it wasn't a survey. It was an analysis of abstracts. Shub is mixing up different things, you'll have noticed. First the ridiculous "sleepy" allegation. Second the separate allegation that people got worse at analysing abstracts as they did more. And finally that the researchers were dishonest. <br /><br />The researchers would have got more proficient over time. They worked at their own pace, not to a clock. <br /><br />As for this:<br /><i>A single person's first ratings for say 50 abstracts would get second ratings from x different volunteers</i><br /><br />That would all make it even less likely that dishonest ratings that Shub makes up out of thin air would have carried through the whole process. <br /><br />Shub isn't interested in checking whether there was 0.036% error or 0.0036% error - he's just making mischief. He's been around climate blogs long enough to know that almost all scientific papers on the topic support AGW, just like Cook13 and similar studies show.<br /><br />As for the implied allegation of dishonesty - well, we all know who are the dishonest people wanting to smear on no basis at all. People like Shub would be beneath contempt if they were worth that much. Since they aren't I won't even waste my contempt.<br /><br />No more fake made up allegations, Shub. Next step is the door.Souhttps://www.blogger.com/profile/08818999735123752034noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-74440038295704315982014-06-10T22:17:48.276+10:002014-06-10T22:17:48.276+10:00Shub, perhaps you should look for "survey fat...Shub, perhaps you should look for "survey fatigue" yourself. You might learn something. Most, if not all, of the available literature is not relevant to Cook et al. Tol also hilariously refers to literature that is completely irrelevant. Then again, he also manages to clearly misinterpret Andy Skuce's comment. It's what happens when you are out on a mission: everything is evidence that you are right, even if an objective analysis shows it is not.Marconoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-44962049112460934452014-06-10T21:45:40.783+10:002014-06-10T21:45:40.783+10:00"Say each researcher was tired 10% of the tim..."Say each researcher was tired 10% of the time and of that 10% of the time they miscategorised 50% of the abstracts. Yes, that's pushing the envelope hugely, but stay with me. What are the chances that the second person rating those same abstracts independently was also tired rating those very same abstracts."<br /><br />That's not how abstracts were rated. A single person's first ratings for say 50 abstracts would get second ratings from x different volunteers. Which is why you need rater ids to access abstracts rated by individual volunteers. Furthermore, if such data is made available, it would still be realized that the same set of n abstracts were rated by two different raters at discontinuous time periods. <br /><br />There are further issues. The abstracts retrieved contains no consensus information for most part. There would be no way of separating a fatigued volunteer dumping abstracts into '4' (the no position) versus their 'true' ratings being '4'. <br /><br />There would be no difference between two raters, rating an abstract/minute, with the first one falling asleep on his keyboard with his forehead on the '4' key for 70 minutes, then waking up and completing the remainder, and a diligent rater who stayed awake for 100 minutes to rate abstracts. Contrary to Tol, I actually believe Cook et al - most of these abstracts have no signal. They think it is ~70%, I calculate this close to 90%.<br /><br />The above means that rater ids and rating time stamps are needed to assess fatigue.shubhttp://nigguraths.wordpress.comnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-11206841878617528142014-06-10T21:07:55.435+10:002014-06-10T21:07:55.435+10:00We await your re-do with considerable interest the...We await your re-do with considerable interest then. But you won't, and absolutely everyone here, including yourself, knows why.<br /><br />As for the rest - and how did Schollenberger get hold of this material again? What's the actual issue here; is it really the one it's convenient for you to claim it is? How, precisely would <i>you</i> respond, or would you expect your institution to respond, in the circumstances?<br /><br />And did you read Tom's comment, or what?<br /><br />Rater fatigue; <i>pfffft</i>! The 97%, plus-or-minus not-much, consensus is rock solid, and if that wasn't important, and didn't gall you to the nth degree, you wouldn't be here.billnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-72422889233253312592014-06-10T20:34:23.507+10:002014-06-10T20:34:23.507+10:00Fellows,
I am a layperson, not a lawyer. As far as...Fellows,<br />I am a layperson, not a lawyer. As far as I am concerned, the University of Queensland and John Cook are the same w.r.t this episode. UQ would not have gone after Shollenberger had it not been for Cook. Cook has the capability to call off UQ's legal threats. The data belongs to Cook. There is zero intellectual property in the data set apart from its value as scientific data. So, this line from UQ's lawyer:<br /><br />"any and all activities involving the use or disclosure in any manner of the IP"<br /><br />as part of a legal threat, is in effect a threat against analysis.<br /><br />For the record, I told Shollenberger that he Cook should give the go ahead for such data to be released, as it is his data. Shollenberger agrees as well, which is why he did not simply release it.<br /><br />Fatigue is a real concern in research of this kind. If you perform repetitive tasks of any kind, you would know. Go to Google Scholar and put in 'survey fatigue'.shubhttp://nigguraths.wordpress.comnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-46691967056076521182014-06-10T18:55:17.694+10:002014-06-10T18:55:17.694+10:00Sorry, Sou, but here I think it is OK to state tha...Sorry, Sou, but here I think it is OK to state that John Cook will sue, when it is in reality his employer. If John wouldn't tell them, I don't see them do anything about the release of that data.<br /><br />So, that makes both of you correct in that John Cook will (not) sue.Marconoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-27289608223827743362014-06-10T18:17:27.210+10:002014-06-10T18:17:27.210+10:00Nanny goat is spewing pellets. John Cook isn't...Nanny goat is spewing pellets. John Cook isn't suing anybody. <br /><br />Truly amazing how goats will eat anything and then "poof," out it come from the other end, all over the landscape. Phew. dbostromhttps://www.blogger.com/profile/13885863615343906724noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-73622491967806714112014-06-10T16:50:49.472+10:002014-06-10T16:50:49.472+10:00I agree - the UQ letter is not, to any reasonable ...I agree - the UQ letter is not, to any reasonable mind, what shub is describing (or should I say 'purveying'?) above.<br /><br />Actual evidence of this assertion, please; <i>It is this data that Cook has threatened to sue if anyone reveals to the outside world.</i> (Which rather implies Cook will be suing the data, we might add, but we all know what is being suggested.)<br /><br />billnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-37823926377460638832014-06-10T16:42:15.719+10:002014-06-10T16:42:15.719+10:00Yes, Marco. That's what I think Shub may be re...Yes, Marco. That's what I think Shub may be referring to. As I've been saying, that isn't from John Cook. So either Shub knows something that no-one else knows or he's mistaken. There is a third possibility, of course.Souhttps://www.blogger.com/profile/08818999735123752034noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-51306845198996796672014-06-10T16:10:21.395+10:002014-06-10T16:10:21.395+10:00Sou and others, there has been a letter from U Que...Sou and others, there has been a letter from U Queensland that Brandon Shollenberger better not release the data he obtained illegally:<br />http://davidappell.blogspot.dk/2014/05/the-university-of-queensland-letter.html<br />(also on WUWT, but I don't want to link there)Marconoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-32570135812866380982014-06-10T14:50:17.771+10:002014-06-10T14:50:17.771+10:00Good point, Sou; it would surely be smeared across...Good point, Sou; it would surely be smeared across WUWT like ice-cream on a baby's face. Apart from the whining and victimhood (always popular with the Denialati) it would be groundwork for the Mann v Steyn case's likely outcome.Cugelnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-17672170893810433222014-06-10T13:34:31.038+10:002014-06-10T13:34:31.038+10:00I don't believe that John Cook has threatened ...I don't believe that John Cook has threatened to sue anyone. Shub hasn't provided any evidence nor indicated who John Cook could have threatened, when or where or by what means (private email, public comment). It would have been all over WUWT for starters - and it's not.Souhttps://www.blogger.com/profile/08818999735123752034noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-25767086515838860442014-06-10T13:08:44.921+10:002014-06-10T13:08:44.921+10:00And let's pause to not forget that shub has fa...And let's pause to not forget that shub has failed to provide evidence of his 'threatened to sue' - beyond the privacy issues - claim. <br /><br />I submit that you appear to be trying to establish a (convenient) 'truth' in the minds of the susceptible - possibly including yourself - via repetition rather than (inconvenient) evidence. Prove me wrong.<br /><br />Also, the obvious thing for Deniers in general to do is to re-do the study yourselves, and yet, no matter how noisy, none of you has done so. I put it to you that this is because you are only too well-aware of what the result would be. Inconvenient. Hence all the cries of 'I can see smoke! I can see smoke!'<br /><br />Again: prove me wrong.<br /><br />billnoreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-39833537850827828002014-06-10T12:13:39.589+10:002014-06-10T12:13:39.589+10:00Shub is concerned that individual raters got tired...Shub is concerned that individual raters got tired. At what point did they get tired? If they rated 50 abstracts in a day were they tired for the last one abstract? The last two abstracts? The last ten abstracts? All 50 abstracts? Did tiredness mean that they miscategorised an abstract? <br /><br />Say each researcher was tired 10% of the time and of that 10% of the time they miscategorised 50% of the abstracts. Yes, that's pushing the envelope hugely, but stay with me. What are the chances that the second person rating those same abstracts independently was also tired rating those very same abstracts. Let's say on the balance of probabilities it was 10% of the time and 50% of those were miscategorised. Now given there are seven categories, that would mean that there could have been the same miscategorisation on 1/7 of that 50% of 10% or whatever.<br /><br />What's that come out at - around 0.036% of the abstracts could have been miscategorised through "tiredness". Well within the error margins given by Cook13. And that's with hugely inflated estimates! <br /><br />That's why I refer to denialism as utter nuttery!Souhttps://www.blogger.com/profile/08818999735123752034noreply@blogger.comtag:blogger.com,1999:blog-2313427464944392482.post-85977725518251893382014-06-10T11:47:58.043+10:002014-06-10T11:47:58.043+10:00I'm not aware of John Cook threatening to sue ...I'm not aware of John Cook threatening to sue anyone. I've no idea what Shub is referring to if he's not interpreting the letter to the script kiddie from UQ. Shub - who did John Cook threaten and when? Who else hacked a private website and stole data from the research apart from the script kiddie who boasts about it?<br /><br />Also, the "abstracts got tired" is dumb dumb dumb. Richard not only got his facts screwed up and confused the raters with the ratees, he couldn't even analyse the data he was given. Research shows, interviewers (the closest parallel to the raters) get more proficient with time, not less. Perhaps Richard is projecting and he's blaming his mistakes in his economics papers on the fact he got "tired". Should we disregard all science on the grounds that researchers got tired? A lot of science is repetitive observation and interpretation.<br /><br />The research isn't measuring how tired the abstracts got, it's a classification of the content of the abstracts. The only sound way to check that is for other people to categorise the abstracts. <br /><br />Oh, wait - that's already been done by the authors of those very same abstracts. Hmmm..they didn't categorise the abstracts they categorised the content of the papers themselves. And guess what - 97% again!<br /><br />Why do deniers like squirrels so much? Why do they never do any research themselves? <br /><br />Did you know that fewer than 3% of scientific papers published about AGW are from deniers? With all the noise they make you'd think more of them would try to put their money where their mouth is. Too busy sounding off about squirrels I guess.Souhttps://www.blogger.com/profile/08818999735123752034noreply@blogger.com