Climate Alarmism at Science Magazine?

I feel obliged to begin this piece with a disclaimer.  I am not a climate sceptic.

That said, I believe that the case for climate change, and action on climate change, should be made with good science and honest arguments.  And in that spirit, there's a paper in Science (along with Nature, one of the worlds most prestigious science journals) last week that's bothering me quite a lot.  The abstract says this:

Terrestrial net primary production (NPP) quantifies the amount of atmospheric carbon fixed by plants and accumulated as biomass. Previous studies have shown that climate constraints were relaxing with increasing temperature and solar radiation, allowing an upward trend in NPP from 1982 through 1999. The past decade (2000 to 2009) has been the warmest since instrumental measurements began, which could imply continued increases in NPP; however, our estimates suggest a reduction in the global NPP of 0.55 petagrams of carbon. Large-scale droughts have reduced regional NPP, and a drying trend in the Southern Hemisphere has decreased NPP in that area, counteracting the increased NPP over the Northern Hemisphere. A continued decline in NPP would not only weaken the terrestrial carbon sink, but it would also intensify future competition between food demand and proposed biofuel production.

In other words, the claim is that, whereas in the past, increasing amounts of carbon dioxide was causing land plants to fix more carbon (and by implication grow more and better), now land plants are overall fixing less carbon because the planet was getting hotter and drier in the 2000s.

That's obviously scary, since all 6.5 billion (and on the way to 9 billion) of us humans have to eat from those plants, and if they are drying up and doing less well because of climate change, that's clearly a trend in the wrong direction.  And sure enough, various newspapers, magazines, and blogs picked up this paper. Here's the UK Guardian:

Rising temperatures in the past decade have reduced the ability of the world's plants to soak up carbon from the atmosphere, scientists said today.

Large-scale droughts have wiped out plants that would have otherwise absorbed an amount of carbon equivalent to Britain's annual man-made greenhouse gas emissions.

Scientists measure the amount of atmospheric carbon dioxide absorbed by plants and turned into biomass as a quantity known as the net primary production. NPP increased from 1982 to 1999 as temperatures rose and there was more solar radiation.

But the period from 2000 to 2009 reverses that trend – surprising some scientists. Maosheng Zhao and Steven Running of the University of Montana estimate that there has been a global reduction in NPP of 0.55 gigatonnes (Gt). In comparison, the UK's contribution to annual worldwide carbon dioxide emissions was 0.56Gt in 2007, while global aviation industry made up around 0.88Gt (3%) of the world total of 29.3Gt that year, according to UN data.

and here's Climate Progress, with the characteristically blaring headline Science shocker: Drought drives decade-long decline in plant growth, which concludes:

And so we have another potential of an amplifying feedback in the carbon cycle.

This new study is the latest in a series of truly alarming scientific studies published just this year that suggest we may be closer to dangerous carbon-cycle tipping points and irreversible thresholds than anyone realized...

Ok, so I go to read the paper itself with a view to blogging about the latest data point in the overall scary trend of humans poking the angry beast with sticks.  Now, before we go on, let's take apart the paper abstract above - what are the main claims this paper is making?  The first sentence says:

Terrestrial net primary production (NPP) quantifies the amount of atmospheric carbon fixed by plants and accumulated as biomass.

Ok, so that's just defining what we're going to be talking about here - the total amount of carbon which plants extract out of the air and fix into their own tissues (as opposed to immediately using in their own respiration: including that would make it gross primary production instead of net primary production).

Next we get to the first major claim of the paper:

Previous studies have shown that climate constraints were relaxing with increasing temperature and solar radiation, allowing an upward trend in NPP from 1982 through 1999. The past decade (2000 to 2009) has been the warmest since instrumental measurements began, which could imply continued increases in NPP; however, our estimates suggest a reduction in the global NPP of 0.55 petagrams of carbon.

Ok, so NPP used to be trending up, now it's trending down. We even have a number for how much it's trending down: 0.55 petagrams between 2000 and 2009 (petagrams = Gigatonnes, billions of metric tonnes of carbon).  So that's the big claim of the paper.  Whereas the world's plants used to be going from strength to strength, at least as measured by total mass, now they are starting to get in trouble.

Note that there's no error bar quoted on that "0.55", a point we will come back to.

Now the rest of the abstract:

Large-scale droughts have reduced regional NPP, and a drying trend in the Southern Hemisphere has decreased NPP in that area, counteracting the increased NPP over the Northern Hemisphere.

So the reason for this trend reversal is droughts, especially in the southern hemisphere. And finally:

A continued decline in NPP would not only weaken the terrestrial carbon sink, but it would also intensify future competition between food demand and proposed biofuel production.

This is scientist speak for BAD AND SCARY.  A) if plants stop absorbing so much carbon because of climate change, our own emissions will be buffered less, giving rise to a nasty vicious circle of escalating CO₂ in the atmosphere, and B) we humans need/want those plants for own ends, and it's not going to be so great if there are fewer of them.  Hence the Climate Progress headline of "Science Shocker", etc.

Ok, so I get into the body of the paper (which unfortunately you can't read unless you have a subscription or buy the paper from Science), and in due course I come to the argument for why there is now a reduction in NPP (the 0.55 petagrams, remember):

Satellite data can generally provide realistic information on vegetation dynamics, including land cover change (5, 6), disturbances, and recovery (7), which may help to reduce uncertainties in carbon-budget estimates. In this study, we investigate terrestrial NPP and climate variability over the past decade (2000 to 2009) by analyzing satellite data from the Moderate Resolution Imaging Spectroradiometer (MODIS) on board NASA’s Terra satellite and global climate data.

We used the global MODIS NPP algorithm (8) [see supporting online material (SOM) text S1] to examine spatially explicit NPP changes from 2000 through 2009. We used collection 5 (C5) 8-day composite 1-km fraction of photosynthetically active radiation (FPAR) and leaf area index (LAI) data from the MODIS sensor (9) as remotely sensed vegetation property dynamic inputs to the algorithm. Data gaps in the 8-day temporal MODIS FPAR/LAI caused by cloudiness were filled with information from accompanying quality-assessment fields (SOM text S2) (10). For daily meteorological data required to drive the algorithm, we used a reanalysis data set from National Center for Environmental Prediction (NCEP) (SOM text S3) (11). A Palmer Drought Severity Index (PDSI) (12) at 0.5° resolution was used as a surrogate of soil moisture (13) to measure environmental water stress by combining information from both evaporation and precipitation (SOM text S4). A lower PDSI generally implies a drier climate.

Global NPP slightly decreased for the past decade by –0.55 Pg C (Fig. 1). Interannual variations of the global NPP were negatively correlated with the global atmospheric CO2 growth rates (correlation coefficient r = –0.89, p < 0.0006) (Fig. 1) (14), suggesting that global terrestrial NPP is a major driver of the interannual CO2 growth rate. Carbon isotopic measurements have indicated that the exchange of CO2 with terrestrial ecosystems is the dominant cause of the CO2 interannual growth rate (15). Though NPP is a part of carbon exchange between the land and atmosphere, the strong correlation may imply that the process of heterotrophic respiration depends ultimately on the substrate supply from NPP (16), as shown by the strong correlation between gross primary production (GPP) and ecosystem respiration derived from eddy-flux towers (17). Although also a major contributor, global fire emissions do not correlate as strongly as NPP with interannual CO2 growth rate (SOM text S5) (18).

My emphasis.  The bolded sentence is the main argument in the paper for why the 0.55 reduction is true, and it pretty much entirely consists of a reference to Figure 1.  Ok, so Figure 1 looks this:

And the caption in the paper in the paper reads: Interannual variations from 2000 through 2009 in anomalies of annual total global terrestrial NPP (green circles) and inverted global atmospheric CO2 annual growth rate [red squares and (14)]. Global average annual total NPP is 53.5 Pg C/yr.

Ok, so we're looking at the green curve.  And the claim is it's trending down by 0.55 in the units on the left hand side.  Now, there's a major problem here to my way of thinking.  The authors didn't give us a trend line, but one thing that's immediately clear is that the bouncing around from year to year in that green curve is quite large compared to any overall trend in it.  So that raises the question: are we really sure that it's genuinely trending down?  Or is it just bouncing around and happened by chance to be a bit lower at the end than the beginning?  To give you some feel for this, I took the numbers off their graph (which might be slightly inaccurate but that will be small compared to the noise in the data), and made my own graph, along with a trend line:

The trend line is in black.  My numbers come out to a slope of -0.05 GT/year, which is -0.045 -0.45 over the nine years.  That's the reduction in NPP.  However, to illustrate the concern about the robustness of this trend, suppose it had happened that the satellite was launched a year later, so the 2000 data point was missing:

Oops!  Now it's going up!  So in other words, this "trend" of "reduction" in NPP is extremely sensitive to exactly which data points we look at.  It's not a robust trend, but rather is sensitive to noise.  Suddenly our 0.05/year reduction has become a 0.03/year increase.

Now, there is a whole long history in science of how to assess this kind of thing - to test whether a particular number or trend is statistically significant, as opposed to the situation where it could well just be a fluke.  It's a normal part of doing science that one analyzes the statistical significance of trends, and analyzes the likely range of uncertainty around a particular estimate.  However, in this paper, there is no analysis in the paper of whether the reduction in NPP is statistically significant, and, as I noted, no error bar is provided for the "0.55 petagram" reduction.  

I find it very strange for a paper published in a top journal to provide no evidence that the main claim in the abstract is actually statistically significant.

So I'm driven to doing my own analysis.

The simplest possible method is based on the theory of ordinary least squares.  I entered the data above (including the 2000 point) into an online regression calculator here, and it tells us:

Look at the "Slope" part.  The slope is -0.05 (in agreement with what Excel came up with in the graph above), but the standard error in the slope is 0.1 - in other words the negative slope is less than the expected uncertainty in the slope, and the result is very much not statistically significant.  Roughly what this means is that, given this level of year-to-year noise, even if there was no real trend up or down, we would expect to get a trend reading between -0.1 and +0.1 about 2/3 of the time.  So the -0.05 trend is entirely consistent with just being noise, and is not good evidence of a reduction in NPP.

I should stress here that we are not doing advanced theoretical ecology here.  This is the kind of stuff scientists learn in their first undergrad stats class, or in my case, in first year undergraduate physics labs.  Uncertainty analysis, OLS regressions for straight lines, etc, is absolutely basic stuff if one is a scientist.

So, I read the rest of the paper to see if I'm missing something, and when it seems like I'm not (the rest of the paper is mainly concerned with the drought/NPP connection and is quite interesting), I email the authors of the paper.  I start off with Maosheng Zhao, the listed correspondence author, who is a research scientist with the Numerical Terradynamic Simulation Group at University of Montana, Missoula.  My initial email was as follows:

Hi:

I blog about energy/resource issues at

http://earlywarn.blogspot.com/

I read your Science paper on the decline in global NPP in the 2000-2009 timeframe.

I notice that you don't seem to have any analysis of whether the decline (the green line in your Figure 1) is statistically significant given the substantial year-to-year fluctuations. A quick OLS suggests that the slope of that data is -0.05 +/- 0.1 GT/yr, which certainly doesn't suggest statistical significance for the decline. Taking account of auto-correlations would seem like it could only make it less significant.

Am I missing something? Did you assess the statistical significance of the trend somehow?

Thanks,

Stuart Staniford.

So, given that this is not my field, I was kind of expecting that they'd come back and point out that I'd missed the importance of the paragraph on page XYZ in the supplemental material, which explained how they analyzed the trend a different way and it really is significant.  Instead, Maosheng gets back to me essentially saying that yes, they agree with me that the data isn't statistically significant, but this is all the data that's available from the satellite.  He and I go back and forth a couple of times, and then he brings in his boss Steve Running, the second author on the paper, who is a Professor at UMT, and the Director of the Numerical Terradynamics Simulation Group.  He gave me permission to quote his emails verbatim, so here is the first one:

Hello Stuart;

We have a continuing difficulty in Earth science when doing time series
research of how to separate real system interannual variability from
measurement error. And the problem is even worse at global scales where
there is only one dataset from only one instrument. The large real world
variability generates a very high statistical threshold of significance for
defining changes in trend trajectory. Because terrestrial carbon sink issues
are of such high policy significance, we felt that this new global trend was
an important message that could not wait longer to give a first look. And
obviously Science magazine agreed. It will probably take another decade of
data to answer this question to full statistical satisfaction, which will
then be with a new satellite, as they don't last that long. So then we will
have to wrestle with changing sensor characterstics too. There are also a
number of other papers from other groups with different methodologies
showing similar trends in related variables, adding to our confidence that
these trends, for now, are real.

Sincerely,

Steve Running

To which I responded:

Hi Steve:

I appreciate you guys taking the time to respond.

I completely understand the "this is all the data we have and it's an important problem" line of reasoning, and that that might well justify publication of a paper, but I still find it very surprising that the paper lacks an explicit discussion of the statistical limitations of the evidence supporting the main claim of the abstract (and certainly, as a longtime subscriber, my surprise extends to the Science editorial and refereeing process). By my calculation, just on the data in the paper, you've got something like a 5-10% chance of being wrong about the "there is a trend break" part, and more like a one third chance of being wrong about the "NPP is now trending down" part. That's weaker evidence than most fields of science will accept as clear evidence of an important claim. The abstract is not phrased as "our evidence is statistically weak but consistent with other papers", but more like "we discovered that this trend has reversed" - that's why it's in Science, right, not some other minor journal that I would never have paid attention to?

I intend to blog about this over the weekend. May I quote your emails publicly, or would you prefer that I just paraphrase the gist?

Thanks,

Stuart.

And he responded:

Stuart;

Some research findings are so important that society really cannot afford to
wait another 10+yr for 95% or 99% statistical confidence. We (and I suppose
Science) felt this result was one of them. And recognize that we are not
advocating this result, merely reporting what we measured and why we think
it is happening. I actually hope in 10-20yr that some young scientist proves
we are wrong, and that NPP trends have turned back up. Humanity will be much
better off if that occurs.

And yes, if you feel my comments are interesting enough, you can quote them
directly, I speak publicly on these topics all the time. For Maosheng you
need to get his permission.

Steve Running

Ok.  So here we have a statistically non-robust result, that the authors are well aware is not statistically robust, being published because it's of "high policy significance".  However, and critically, the authors included no discussion whatsoever of the statistical limitations of the evidence.  The "-0.55" in the abstract is not "-0.55 +/- 1.1" or something like that to give the reader a heads up that there is a lot of uncertainty here.  There is no calculation of the "p-value" of that trend (how likely it was to occur by chance), even though the rest of the paper is littered with p-values of subsidiary results.  They know perfectly well how to calculate this, they know it's not statistically significant, but they chose to put their readers in a position where we have to take the data off the graph and do our own statistical analysis to realize what's really going on.

And the refereeing and editorial process at Science allowed the paper to be published like that.

I think that sucks.

Just to summarize, here's some things I'm not saying:

• I'm not saying that climate change isn't real, or humans aren't causing it.
• I'm not saying that trends in NPP aren't incredibly important, they are.
• I'm not saying NPP is still trending up - the evidence in the paper is rather equivocal and it's hard to say for sure.
• I'm not saying a paper on this satellite data shouldn't have been published, or that they should have waited ten years to do so.
• I'm not even saying it shouldn't have been published in Science.

Instead, what I'm saying is:

• The paper should have been much more explicit that this is a very preliminary result with weak statistical evidence - weaker than is normally accepted in scientific publications.  There should have been an explicit error bar on the trend.

Maybe that would have resulted in less press and blogosphere attention.  So be it.  In the end, in the great debate with all manner of moneyed interests and massive human inertia, the only things climate scientists have going for them are the truth and the moral high ground.  Those things shouldn't be thrown away lightly.