Persönlicher Status und Werkzeuge

Sprachwahl

Software ROC/AUC-calculation

ProgramEvaluating the predictive performance of species distribution models

ROC_AUC is a Delphi program for assessing the predictive performance of habitat models. The program calculates the area under Receiver Operating Characterstic curves (Beck & Shultz 1986; Hanley & McNeil 1982; Hosmer & Lemeshow 2000) and estimates optimal cutoff-values. It can be used to assess the transferability of habitat models as described in Schröder (2000) and Bonn & Schröder (2001).

Features
  • construction of the Receiver Operating Characterstic (ROC) showing the predictive performance of a habitat model as a trade off between sensitivity and specificity
  • calculation of the Area Under the ROC-Curve (AUC) as a threshold-independent measure of predicitve performance with bootstrapped confidence intervals calculated with the percentil method referring to Buckland et al. (1997) and Augustin et al. (1998)
  • calculation of a variety of threshold-dependent performance criteria as described in Fielding & Bell (1997), e.g. Cohen's Kappa, correct classification rate, sensitivity and specificity
  • optimisation of cut-off values regarding 
    i) maximised Kappa (Pkappa), 
    ii) minimised difference between sensitivity and specificty (Pfair in Schröder & Richter 1999) 
    iii) maximised correct classification rate (Popt, calculated from the ROC as described in Zweig and Campbell (1993) taking into account different costs of false positive or false negative predictions) and 
    iv) P = 0.5
  • calculation of classification matrices regarding the optmised cut-off values
  • summary of the most important criteria regarding the optmised cut-off values
  • import of ASCII-input data comprising 
    i) observed incidence {0,1} and 
    ii) estimated occurrence probabilities [0,1]; 
    please make sure that your PC uses the decimal point "." instead of the comma "," as the character for separating decimal places (in case of German PCs: "Einstellungen -> Systemsteuerung -> Ländereinstellungen -> Dezimaltrennzeichen").
  • additionally import via the clipboard (copy & paste from spreadsheets)
  • creation of ASCII-output comprising 
    i) all data needed to plot ROC and 
    ii) all performance criteria for cut-off values ranging from 0 to 1 as well as a summary of predictive performance criteria regarding the optmised cut-off values
ReferencesAugustin, NH, Mugglestone, MA and Buckland, ST (1998) The role of simulation in modelling spatially correlated data. Environmetrics 9: 175-196.
Beck, JR & Shultz, EK (1986) The use of ROC curves in test performance evaluation. Archives of pathology and laboratory medicine, 110, 13-20. 
Bonn, A & Schröder, B (2001) Habitat models and their transfer for single- and multi-species groups: a case study of carabids in an alluvial forest. Ecography, 24, 483-496. 
Buckland, ST, Burnham, KP & Augustin, NH (1997) Model selection: an integral part of inference. - Biometrics 53: 603-618. 
Fielding, AH & Bell, JF (1997) A review of methods for the assessment of prediction errors in conservation presence-absence models. Environmental Conservation, 24, 38-49. 
Hanley, JA & McNeil, BJ (1982) The meaning and use of the area under a ROC curve. Radiology, 143, 29-36. 
Hosmer, DW & Lemeshow, S (2000) Applied logistic regression, 2nd edn. Wiley, New York. 
Schröder, B (2000) Zwischen Naturschutz und Theoretischer Ökologie: Modelle zur Habitateignung und räumlichen Populationsdynamik für Heuschrecken im Niedermoor. PhD - Thesis (in German), TU Braunschweig, Braunschweig. 
Schröder, B & Richter, O (1999/2000) Are habitat models transferable in space and time? Journal of Nature Conservation, formerly Zeitschrift für Ökologie und Naturschutz, 8, 195-205. 
Zweig, MH & Campbell, G (1993) Receiver-Operating Characteristic (ROC) Plots: a fundamental evaluation tool in clinical medicine. Clinical chemistry, 39, 561-577.
Downloadnew version (08. Dec. 2006): roc_auc_dec06.zip (291 kB)
Note: Please send an email (boris.schroeder[at]tum.de) to add your name on to the user list so that you can be notified of changes and updates.

Bug fixes: AUC confidence bounds for percentile and normal method corrected


bootstrap_distr.png


screen_all75.png


screen_all.png