Climate of the Past
www.clim-past.net
1814-9324
1814-9332
3
3
2007
10.5194/cp-3-397-2007
http://www.clim-past.net/3/397/2007/
http://www.clim-past.net/3/397/2007/cp-3-397-2007.html
http://www.clim-past.net/3/397/2007/cp-3-397-2007.pdf
397
409
2007-07-11
On the verification of climate reconstructions
G. Bürger
gerd.buerger@met.fu-berlin.de
FU-Berlin, Institut für Meteorologie; Carl-Heinrich-Becker-Weg 6–10, 12165 Berlin, Germany
The skill of proxy-based reconstructions of Northern hemisphere
temperature is reassessed. Using an almost complete set of proxy and
instrumental data of the past 130 years a multi-crossvalidation is
conducted of a number of statistical methods, producing a distribution
of verification skill scores. Among the methods are multiple
regression, multiple inverse regression, total least squares, RegEM,
all considered with and without variance matching. For all of them the
scores show considerable variation, but previous estimates, such as a
50% reduction of error (<i>RE</i>), appear as outliers and more realistic
estimates vary about 25%. It is shown that the overestimation of
skill is possible in the presence of strong persistence (trends). In
that case, the classical "early" or "late" calibration sets are
not representative for the intended (instrumental, millennial)
domain. As a consequence, <i>RE</i> scores are generally inflated, and the
proxy predictions are easily outperformed by stochastic, a priori
skill-less predictions.
<br><br>
To obtain robust significance levels the multi-crossvalidation is
repeated using stochastic predictors. Comparing the score
distributions it turns out that the proxies perform significantly
better for almost all methods. The scores of the stochastic predictors
do not vanish, nonetheless, with an estimated 10% of spurious skill
based on representative samples. I argue that this residual score is
due to the limited sample size of 130 years, where the memory of the
processes degrades the independence of calibration and validation
sets. It is likely that proxy prediction scores are similarly inflated
and have to be downgraded further, leading to a final overall skill
that for the best methods lies around 20%.
<br><br>
The consequences of the limited verification skill for millennial
reconstructions is briefly discussed.
Aldrich, J.: Correlations Genuine and Spurious in Pearson and Yule, Discussion Paper Series In Economics And Econometrics 9502, Economics Division, School of Social Sciences, University of Southampton, available at: http://ideas.repec.org/p/stn/sotoec/9502.html, 1995.
Bhansali, R J. and Kokoszka, P S.: Estimation of the long memory parameter: a review of recent developments and an extension, in: Selected proceedings of the symposium on inference for stochastic processes. IMS Lecture notes and monograph series, edited by: Basawa., I., Heyde, C C., and Taylor, R., 125–150, Institute of Mathematical Statistics, Ohio, USA, 2001.
Briffa, K R.: Annual climate variability in the holocene: interpreting the message of ancient trees, Quat. Sci. Rev., 19, 87–105, 2000.
Briffa, K R., Jones, P D., Pilcher, J R., and Hughes, M K.: Reconstructing Summer Temperatures in Northern Fennoscandinavia Back to A.D.1700 Using Tree Ring Data from Scots Pine, Arctic and Alpine Research, 385–94, 1988.
Briffa, K R., Bartholin, T S., Eckstein, D., Jones, P D., Karlen, W., Schweingruber, F H., and Zetterberg, P.: A 1,400-year tree-ring record of summer temperatures in fennoscandia, Nature, 346, 434–439, 1990.
Briffa, K R., Jones, P D., and Schweingruber, F H.: Tree-ring density reconstructions of summer temperature patterns across western north america since 1600, J. Climate, 5, 735–754, 1992.
Briffa, K R., Osborn, T J., Schweingruber, F H., Harris, I C., Jones, P D., Shiyatov, S G., and Vaganov, E A.: Low-frequency temperature variations from a northern tree ring density network, J. Geophys. Res., 106(D3), 2929–2941, 2001.
Brockwell, P J. and Davis, R A.: Time series: theory and methods, 2nd edition, Springer Series in Statistics, 1998.
Bürger, G. and Cubasch, U.: Are multiproxy climate reconstructions robust?, Geophys. Res. Lett., 32, L23711, doi:10.1029/2005GL0241550, 2005.
Bürger, G., Fast, I., and Cubasch, U.: Climate reconstruction by regression – 32 variations on a theme, Tellus A, 227–35, 2006.
Cattin, P.: Estimation of the predictive power of a regression model, J. Appl. Psychol., 65(4), 407–414, 1980.
Cook, E R., Briffa, K R., and Jones, P D.: Spatial regression methods in dendroclimatology: a review and comparison of two techniques, Int. J. Clim., 14, 379–402, 1994.
Cook, E R., Buckley, B M., D'Arrigo, R D., and Peterson, M J.: Warm-season temperatures since 1600 bc reconstructed from tasmanian tree rings and their relationship to large-scale sea surface temperature anomalies., Clim. Dyn., 16, 79–91, 2000.
Cooley, W W. and Lohnes, P R.: Multivariate data analysis, New York: Wiley, 1971.
Crowley, T J. and Lowery, T S.: How warm was the medieval warm period?, Ambio, 29, 54, 2000.
Dempster, A., Laird, N., and Rubin, D.: Maximum likelihood estimation from incomplete data via the EM algorithm, J. Royal Statist. Soc., B, 39, 1–38, 1977.
Efron, B.: Bootstrap methods: another look at the jackknife, Annals of Statistics, 17, 1–26, 1979.
Efron, B. and Gong, G.: A Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation, American Statistician, 36–48, 1983.
Esper, J., Cook, E R., and Schweingruber, F H.: Low frequency signals in long tree-ring chronologies for reconstructing past temperature variability, Science, 295, 2250–2253, 2002.
Esper, J., Frank, D C., Wilson, R. J S., and Briffa, K R.: Effect of scaling and regression on reconstructed temperature amplitude for the past millennium, Geophys. Res. Lett., 32(7), L07711, doi:10.1029/2004GL021236, 2005.
Evans, M N., Kaplan, A., and Cane, M A.: Pacific sea surface temperature field reconstruction from coral delta o-18 data using reduced space objective analysis, Paleoceanography, 17, 1007, doi:10.1029/2000PA000590, 2002.
Frank, I E., Friedman, J H.: A Statistical View of Some Chemometrics Regression Tools, Technometrics, 35, 109, 109–148, 1993.
Fritts, H C.: Tree rings and climate, Academic Press, 1976.
Fritts, H C. and Guiot, J.: Methods of calibration, verification, and reconstruction, in: Methods Of Dendrochronology. Applications In The Environmental Sciences, edited by: Cook, E R. and Kairiukstis, L A., 163–217, Kluwer Academic Publishers, 1990.
Geweke, J. and Porter-Hudak, S.: The estimation and application of long-memory time series models, J. Time Series Analysis, 4, 221–238, 1983.
Golub, G H. and Loan, C. F V.: Matrix computations (3rd ed.), Johns Hopkins University Press, Baltimore MD USA, 1996.
Guiot, J., Nicault, A., Rathgeber, C., Edouard, J L., Guibal, E., Pichard, G., and Till, C.: Last-millennium summer-temperature variations in western europe based on proxy data, Holocene, 15, 500, 2005.
Hoerl, A E.: Application of ridge analysis to regression problems, Chem. Eng. Prog., 58, 54–59, 1962.
Hosking, J.: Modeling persistence in hydrological time series using fractional differencing, Water Resour. Res., 20(12), 1898–1908, 1984.
Huybers, P.: Comment on "Hockey sticks, principal components, and spurious significance", Geophys. Res. Lett., L20705, doi:10.1029/2005GL023395, 2005.
IPCC: Climate change 2001: the scientific basis. contribution of working group I to the third assessment report of the intergovernmental panel on climate change, Cambridge University Press, Cambridge, 2001.
Working Group I Report "The Physical Science Basis". The Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (in press).
Jones, P D., Briffa, K R., Barnett, T P., and Tett, S. F B.: High-Resolution Palaeoclimatic Records for the Last Millennium: Interpretation, Integration and Comparison with General Circulation Model Control-Run Temperatures, Holocene, 8, 455–71, 1998.
Geisser, S..: The Predictive Sample Reuse Method with Applications, Journal of The American Statistical Association, 70, 320–328, 1975.
Little, R. J A. and Rubin, D B.: Statistical analysis with missing data, Wiley, 1987.
Lorenz, E N.: Empirical orthogonal functions and statistical weather prediction, Sci. Rept. No. 1, Dept. of Met., M. I. T., p. 49pp, 1956.
Luterbacher, J., Xoplaki, E., Dietrich, D., Rickli, R., Jacobeit, J., Beck, C., Gyalistras, D., Schmutz, C., and Wanner, H.: Reconstruction of sea level pressure fields over the eastern north atlantic and europe back to 1500, Climate Dynamics, 18, 545–561, 2002.
Luterbacher, J., Dietrich, D., Xoplaki, E., Grosjean, M., and Wanner, H.: European Seasonal and Annual Temperature Variability, Trends, and Extremes Since 1500, Science, 303, 1499–1503, 2004.
Mann, M E. and Rutherford, S.: Climate reconstruction using "Pseudoproxies", Geophys. Res. Lett., p. 139, 2002.
Mann, M E., Bradley, R S., and Hughes, M K.: Global-scale temperature patterns and climate forcing over the past six centuries, Nature, 779–87, 1998.
Mann, M E., Bradley, R S., and Hughes, M K.: Northern hemisphere temperatures during the past millennium: inferences, uncertainties, and limitations, Geophys. Res. Lett., 759–762, 1999.
Mann, M E., Rutherford, S., Wahl, E., and Ammann, C.: Testing the Fidelity of Methods Used in Proxy-Based Reconstructions of Past Climate, J. Climate, 4097–107, 2005.
McIntyre, S. and McKitrick, R.: Corrections to the Mann et al (1998) proxy data base and northern hemispheric average temperature series, Energy Environ., 14(6), 751–771, 2003.
McIntyre, S. and McKitrick, R.: Hockey sticks, principal components and spurious significance, Geoph. Res. Let., 32, L03710, doi:10.1029/2004GL021750, 2005a.
McIntyre, S. and McKitrick, R.: Reply to comment by huybers on \textquotesingle\textquotesinglehockey sticks, principal components, and spurious significance\textquotesingle\textquotesingle, Geophys. Res. Lett., 32, L20713, doi:10.1029/2005GL023586, 2005b.
Moberg, A., Sonechkin, D M., Holmgren, K., Datsenko, N M., and Karlen, W.: Highly variable northern hemisphere temperatures reconstructed from low- and high-resolution proxy data, Nature, 433, 617, 2005.
Murphy, A. H.: The Finley Affair: A Signal Event in the History of Forecast Verification, Weather and Forecasting, 11(1), 3, 1996.
Nash, J E. and Sutcliffe, J V.: River flow forecasting through conceptual models - Part I - A discussion of principles, J. Hydrol., 10, 282–290, 1970.
Nicholson, G E.: Prediction in future samples, in: Contributions to Probability and Statistics, edited by: Olkin, I., 322–330, 1960.
Overpeck, J., Hughen, K., Hardy, D., Bradley, R., Case, R., Douglas, M., Finney, B., Gajewski, K., Jacoby, G., Jennings, A., Lamoureux, S., Lasca, A., MacDonald, G., Moore, J., Retelle, M., Smith, S., Wolfe, A., and Zielinski, G.: Arctic Environmental Change of the Last Four Centuries, Science, 278, 1251–1256, 1997.
Pearson, K.: Mathematical Contributions to the Theory of Evolution – On a Form of Spurious Correlation Which May Arise When Indices Are Used in the Measurement of Organs, Proc. R. Soc., 60, 489–498, 1897.
Raju, N S., Bilgic, R., Edwards, J E., and Fleer, P F.: Methodology review: estimation of population validity and cross-validity, and the use of equal weights in prediction, J Appl. Psychol. Measurement, 21(4), 291–305, 1997.
Robinson, P M.: Log-Periodogram Regression of Time Series with Long Range Dependence, Annals of Statistics, 23, 1048–1072.
Rubin, D B.: Inference and missing data, Biometrika, 63, 581–592, 1976.
Rutherford, S., Mann, M E., Delworth, T L., and Stouffer, R J.: Climate field reconstruction under stationary and nonstationary forcing, J. Climate, 16, 462–479, 2003.
Rutherford, S., Mann, M E., Osborn, T J., Bradley, R S., Briffa, K R., Hughes, M K., and Jones, P D.: Northern Hemisphere Surface Temperature Reconstructions: Sensitivity to Methodology, Predictor Network, Target Season and Target Domain, J. Climate, 2308–29, 2005.
Schneider, T.: Analysis of incomplete climate data: estimation of mean values and covariance matrices and imputation of missing values, J. Climate, 14, 853–871, 2001.
Seber, G. A F. and Lee, A J.: Linear regression analysis (wiley series in probability and statistics), Wiley-Interscience, 2 edn., 2003.
von Storch, H., Zorita, E., Jones, J M., Dmitriev, Y., and Tett, S. F B.: Reconstructing Past Climate from Noisy Data, Science, 679–82, 2004.
Wahl, E R. and Ammann, C M.: Robustness of the Mann, Bradley, Hughes reconstruction of Northern hemisphere surface temperatures: Examination of criticisms based on the nature and processing of proxy climate evidence, Climatic Change, in press, 2007.
Wahl, E R., Ritson, D M., and Ammann, C M.: Comment on "Reconstructing past climate from noisy data", Science, 312, p. 529b, http://www.sciencemag.org/cgi/content/abstract/312/5773/529b, doi:10.1126/science.1120866, 2006.
Wilks, D S.: Statistical methods in the atmospheric sciences. an introduction, Academic Press, San Diego, 1995.
Yule, G U.: Why do we sometimes get nonsense-correlations between time-series? – A study in sampling and the nature of time-series, J. Roy. Stat. Soc., 1–29, 1926.