Thursday, August 05, 2010

Google Scholar illiteracy in the PNAS


A recent paper published in the PNAS, "Expert credibility in climate change" is being used as propaganda to claim that 97% of all climate scientists agree with the IPCC and the need for government action on climate change. An analysis of this paper does not support these conclusions.

Note: All Google Scholar numerical result totals will change over time and sometimes every time you search.

PNAS reviewers and author's William R. L. Anderegg, James W. Prall, Jacob Harold and Stephen H. Schneider are apparently Google Scholar illiterate since searching for just the word "climate" with an author's name will bring results from non-peer-reviewed sources such as books, magazines, newspapers, patents, citations, duplicate listings and all sorts of other erroneous results. Such as 16,000 from the Guardian, 52,000 from Newsweek and 115,000 from the New York Times. There is no "peer-reviewed journal only" search option in Google Scholar.


Cherry Picking:

It is clear the authors cherry picked away skeptics using subjective criteria,
"we imposed a 20 climate-publications minimum to be considered a climate researcher."
So if a scientist published only 19 or less papers on the climate he is not considered an "expert". They did this intentionally as they noted,
"researchers with fewer than 20 climate publications comprise ≈80% the UE group."
Volume of publications does not indicate scientific truth nor does it denote expertise. It cannot be ignored that skeptics extensively publish peer-reviewed papers so they have to use this propaganda to subjectively define "experts". An objective criteria for determining if an author has done climate research would be if an author has or has not published a paper on the climate. Expertise is simply an opinion and who is considered an expert will change based on who you talk to.


Climate Patents:

By default Google Scholar is set to search both "articles and patents" yet no mention of searching only for articles is in the paper.



So why were they searching for patents and how is a patent that contains the search word "climate" a relevant "climate publication"?


Verification:

An attempt to reproduce the results using their methods was unsuccessful,
"we collected the number of climate-relevant publications for all 1,372 researchers from Google Scholar (search terms: "author:fi-lastname climate")"
Using their search terms: "author:fi-lastname climate" I searched Google Scholar for the infamous CRU director Phil Jones,

author:P-Jones climate

Results: 6,580

The first result listed is "Very high resolution interpolated climate surfaces for global land areas" by author Peter G. Jones of the International Center for Tropical Agriculture, Cali, Colombia.

The third result listed is "Organizational climate: A review of theory and research" in Psychological Bulletin by author Allan P. Jones of the Department of Psychology, University of Houston.

The seventh result listed is "Psychological climate: Dimensions and relationships of individual and aggregated work environment perceptions" in Organizational Behavior and Human Performance by author Allan P. Jones of the Department of Psychology, University of Houston.

Clearly these were not papers by Phil Jones of the CRU on climate change.

Looking on Prall's list of IPCC AR4 Working Group 1 Authors referenced from their Supplemental Information you see Phil Jones listed with 724 climate publications not the 6,580 that I found using their search method. A link is provided under "GS queries" for Phil Jones labeled "CLIM", clicking on this link brings a surprising revelation, the search term is changed to "author:PD-Jones climate". When their paper explicitly said "author:fi-lastname climate" and no mention is made of including the middle initial. It appears Prall added the middle initial arbitrarily to the authors on the list further undermining the consistency of their results. Using this search term I again searched Google Scholar,

author:PD-Jones climate

Results: 5,370

The sixth result down is "Climate since A. D. 1500", a 1992 book by Phil Jones not a peer-reviewed paper.

Chapter 13 from the same book is found later in the same results as a separate listing, "13 Climatic variations in the longest instrumental records", thus counting the same book twice.

The book's introduction is also found later in the same results "Climate since AD 1500: Introduction", now counting the same book three times but it gets worse,

Citations for this book are counted over 20 times in Google Scholar, further inflating the erroneous results. No mention of turning off citations is in their paper as this feature is on by default in Google Scholar and in the "CLIM" link from Prall's page.

The climate total number of 724 for Phil Jones on Prall's list is unverifiable using the methods outlined in their paper and appears to be made up.

It is clear they used the total number of climate publications because this is explicitly stated in their paper,
"We ranked researchers based on the total number of climate publications authored."
However no verification of these results was done by the authors as they only mention,
"We verified, however, author identity for the four top-cited papers by each author."
It appears they only verified the top four results for their "citation analysis" not for the total amount of results using the search word "climate" for each of the 1372 authors. As demonstrated here, without complete verification it is not possible to draw any meaningful conclusions that include erroneous results.

Conclusion: the study is worthless due to Google Scholar illiteracy and Cherry Picking.


References:
Google Scholar at the Academy (National Post, Canada)

2 comments:

marcodbeast said...

From their supplementals:

Supporting Information
Anderegg et al. 10.1073/pnas.1003187107
SI Materials and Methods
We compiled a database of 1,372 climate researchers and classified
each researcher as either convinced by the evidence (CE)
for anthropogenic climate change or unconvinced by the evidence
(UE) for ACC. We compiled these CE researchers comprehensively
(i.e., all names listed) from the following lists: IPCC
AR4 Working Group I Contributors (coordinating lead authors,
lead authors, and contributing authors; 619 names listed), 2007
Bali Declaration (212 signers listed), Canadian Meteorological
and Oceanographic Society (CMOS) 2006 statement (120 names
listed), CMOS 2008 statement (130 names listed), and 37 signers
of open letter protesting The Great Global Warming Swindle film
errors. After removing duplicate names across these lists, we had
a total of 903 names.
We define UE researchers as those who have signed reputable
statements strongly dissenting from the views of the IPCC. We
compiled UE names comprehensively from the following 12 lists:
1992 statement from the Science and Environmental Policy
Project (46 names), 1995 Leipzig Declaration (80 names), 2002
letter to Canadian Prime Minister Jean Chretien (30 names),
2003 letter to Canadian Prime Minister Paul Martin (46 names),
2006 letter to Canadian Prime Minister Stephen Harper (61
names), 2007 letter to U.N. Secretary General Ban Ki-Moon (100
names), 2007 TV film The Great Global Warming Swindle interviewees
(17 names), NIPCC: 2008 Heartland Institute document
“Nature, Not Human Activity, Rules the Climate," ed. S.
Fred Singer (24 listed contributors), 2008 Manhattan Declaration
from a conference in New York City (206 names listed as
qualified experts), 2009 newspaper ad by the Cato Institute
challenging President Obama’s stance on climate change (115
signers), 2009 Heartland Institute document “Climate Change
Reconsidered: 2009 Report of the Nongovernmental Panel on
Climate Change (NIPCC)” (36 authors), and 2009 letter to the
American Physical Society (61 names). After removing duplicate
names across these lists, we had a total of 472 names.

Where's Google Scholar?

Andrew said...

Try reading the paper,

"Between December 2008 and July 2009, we collected the number of
climate-relevant publications for all 1,372 researchers from Google Scholar"