General Performance Score

Terminology and derivations
from a confusion matrix
condition positive (P) the number of real positive cases in the data condition negative (N) the number of real negative cases in the data true positive (TP) A test result that correctly indicates the presence of a condition or characteristic true negative (TN) A test result that correctly indicates the absence of a condition or characteristic false positive (FP) A test result which wrongly indicates that a particular condition or attribute is present false negative (FN) A test result which wrongly indicates that a particular condition or attribute is absent sensitivity, recall, hit rate, or true positive rate (TPR) $T P R = \frac{T P}{P} = \frac{T P}{T P + F N} = 1 - F N R$ specificity, selectivity or true negative rate (TNR) $T N R = \frac{T N}{N} = \frac{T N}{T N + F P} = 1 - F P R$ precision or positive predictive value (PPV) $P P V = \frac{T P}{T P + F P} = 1 - F D R$ negative predictive value (NPV) $N P V = \frac{T N}{T N + F N} = 1 - F O R$ miss rate or false negative rate (FNR) $F N R = \frac{F N}{P} = \frac{F N}{F N + T P} = 1 - T P R$ fall-out or false positive rate (FPR) $F P R = \frac{F P}{N} = \frac{F P}{F P + T N} = 1 - T N R$ false discovery rate (FDR) $F D R = \frac{F P}{F P + T P} = 1 - P P V$ false omission rate (FOR) $F O R = \frac{F N}{F N + T N} = 1 - N P V$ Positive likelihood ratio (LR+) $L R + = \frac{T P R}{F P R}$ Negative likelihood ratio (LR-) $L R - = \frac{F N R}{T N R}$ prevalence threshold (PT) $P T = \frac{\sqrt{F P R}}{\sqrt{T P R} + \sqrt{F P R}}$ threat score (TS) or critical success index (CSI) $T S = \frac{T P}{T P + F N + F P}$ Prevalence $\frac{P}{P + N}$ accuracy (ACC) $A C C = \frac{T P + T N}{P + N} = \frac{T P + T N}{T P + T N + F P + F N}$ balanced accuracy (BA) $B A = \frac{T P R + T N R}{2}$ F1 score is the harmonic mean of precision and sensitivity: $F_{1} = 2 \times \frac{P P V \times T P R}{P P V + T P R} = \frac{2 T P}{2 T P + F P + F N}$ phi coefficient (φ or r_φ) or Matthews correlation coefficient (MCC) $M C C = \frac{T P \times T N - F P \times F N}{\sqrt{(T P + F P) (T P + F N) (T N + F P) (T N + F N)}}$ Fowlkes–Mallows index (FM) $F M = \sqrt{\frac{T P}{T P + F P} \times \frac{T P}{T P + F N}} = \sqrt{P P V \times T P R}$ informedness or bookmaker informedness (BM) $B M = T P R + T N R - 1$ markedness (MK) or deltaP (Δp) $M K = P P V + N P V - 1$ Diagnostic odds ratio (DOR) $D O R = \frac{L R +}{L R -}$ Sources: Fawcett (2006),^[1] Piryonesi and El-Diraby (2020),^[2] Powers (2011),^[3] Ting (2011),^[4] CAWCR,^[5] D. Chicco & G. Jurman (2020, 2021, 2023),^[6]^[7]^[8] Tharwat (2018).^[9] Balayla (2020)^[10]

The General Performance Score ( $G P S$ )^[11] is a family of metrics to assess the performance of Machine Learning models in classification problems. It is defined for binary and multiclass classification problems, and is suitable for any $K \times K$ , $K \geq 2$ confusion matrix. The $G P S$ is defined as the harmonic mean of a set of different performance metrics $p_{1}, . . ., p_{n}$ obtained from the confusion matrix:

$G P S (p_{1}, . . ., p_{n}) = \frac{n}{\sum_{i = 1}^{n} \frac{1}{p_{i}}}$ .

If the performance metrics $p_{1}, . . ., p_{n}$ range in [0,1], $G P S$ also takes values in that interval. It is equal to 1 when all performance metrics achieve their maximum of 1, and it is equal to 0 if one of them (or more) is 0. $G P S$ punishes low values of the performance metrics.

$G P S$ allows the analyst to adapt the performance metric according to the specific domain and the problem requirements.

Binary classification

Some particular examples of the $G P S$ for binary classification are:

The $G P S$ parameterised with Precision and Recall is exactly F1-score: $G P S (P r e c i s i o n, R e c a l l) = G P S (P P V, T P R) = F_{1}^{+} = 2 \cdot \frac{P P V \cdot T P R}{P P V + T P R}$ .
The $G P S$ parameterised with Specificity and Negative Predictive Value is the F1-score from the perspective of the negative class, that is, $F_{1}^{-}$ ^[12]: $G P S (S p e c i f i c i t y, N e g a t i v e P r e d i c t i v e V a l u e) = G P S (T N R, N P V) = F_{1}^{-} = 2 \cdot \frac{N P V \cdot T N R}{N P V + T N R}$ . $F_{1}^{-}$ takes into consideration the proportion of correctly classified points from the negative class and the success when predicting an instance as from the negative class.
$G P S (S e n s i t i v i t y, S p e c i f i c i t y) = G P S (T P R, T N R) = 2 \cdot \frac{T N R \cdot T P R}{T N R + T P R}$ .
Taking into account all the elements of the confusion matrix, the Unified Performance Measure ( $U P M$ )^[12] measure is obtained. That is, the harmonic mean of Precision (PPV), Recall (TPR), Negative Predictive Value (NPV) and Specificity (TNR) is equal to the $U P M$ measure: $G P S (P P V, T P R, T N R, N P V) = U P M = 4 \cdot \frac{P P V \cdot T P R \cdot T N R \cdot N P V}{P P V \cdot T P R \cdot N P V + P P V \cdot T P R \cdot T N R + N P V \cdot T N R \cdot P P V + N P V \cdot T N R \cdot T P R}$ . Note that this formula is exactly the same as the $G P S$ parameterised with $F_{1}^{+}$ and $F_{1}^{-}$ : $G P S (P P V, T P R, T N R, N P V) = G P S (F_{1}^{+}, F_{1}^{-}) = 2 \cdot \frac{F_{1}^{+} \cdot F_{1}^{-}}{F_{1}^{+} + F_{1}^{-}}$ . $U P M$ takes values in the interval [0,1], being 1 the perfect score, 0.5 reveals randomness and 0 reflecting that at least one of the four considered measures is exactly 0. This 0 implies indeed that one (or both) of the classes has all its observations wrongly classified. $U P M$ is suitable for both balanced and imbalanced scenarios^[13]. For the case of imbalanced datasets, $U P M$ outperforms the Matthews Correlation Coefficient (MCC).

Multiclass classification

In the case of multiclass classification with $K$ classes, a first step is to obtain $K$ binary confusion matrices. For example, the one vs rest technique can be applied. Let $q$ be the desired performance metric for the multiclass classification problem (for example, $q = U P M$ ) and let $(q_{1}, . . ., q_{K})$ be the evaluated performance metric $q$ in each of the $K$ binary confusion matrices, the multiclass $G P S$ (parameterised with $q$ ) is:

$G P S_{q} = G P S (q_{1}, . . ., q_{K}) = \frac{K \cdot \prod_{k = 1}^{K} q_{k}}{\sum_{k^{'} = 1}^{K} \prod_{k = 1, k \neq k^{'}} q_{k}}$ .

References

↑ Fawcett, Tom (2006). "An Introduction to ROC Analysis" (PDF). Pattern Recognition Letters. 27 (8): 861–874. doi:10.1016/j.patrec.2005.10.010.
↑ Piryonesi S. Madeh; El-Diraby Tamer E. (2020-03-01). "Data Analytics in Asset Management: Cost-Effective Prediction of the Pavement Condition Index". Journal of Infrastructure Systems. 26 (1): 04019036. doi:10.1061/(ASCE)IS.1943-555X.0000512.
↑ Powers, David M. W. (2011). "Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation". Journal of Machine Learning Technologies. 2 (1): 37–63.
↑ Ting, Kai Ming (2011). Sammut, Claude; Webb, Geoffrey I., eds. Encyclopedia of machine learning. Springer. doi:10.1007/978-0-387-30164-8. ISBN 978-0-387-30164-8. Search this book on
↑ Brooks, Harold; Brown, Barb; Ebert, Beth; Ferro, Chris; Jolliffe, Ian; Koh, Tieh-Yong; Roebber, Paul; Stephenson, David (2015-01-26). "WWRP/WGNE Joint Working Group on Forecast Verification Research". Collaboration for Australian Weather and Climate Research. World Meteorological Organisation. Retrieved 2019-07-17.
↑ Chicco D.; Jurman G. (January 2020). "The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation". BMC Genomics. 21 (1): 6-1–6-13. doi:10.1186/s12864-019-6413-7. PMC 6941312 Check |pmc= value (help). PMID 31898477.
↑ Chicco D.; Toetsch N.; Jurman G. (February 2021). "The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation". BioData Mining. 14 (13): 1-22. doi:10.1186/s13040-021-00244-z. PMC 7863449 Check |pmc= value (help). PMID 33541410 Check |pmid= value (help).
↑ Chicco D.; Jurman G. (2023). "The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification". BioData Mining. 16 (1). doi:10.1186/s13040-023-00322-4. PMC 9938573 Check |pmc= value (help).
↑ Tharwat A. (August 2018). "Classification assessment methods". Applied Computing and Informatics. doi:10.1016/j.aci.2018.08.003.
↑ Balayla, Jacques (2020). "Prevalence threshold (ϕe) and the geometry of screening curves". PLoS One. 15 (10). doi:10.1371/journal.pone.0240215.
↑ De Diego, Isaac Martín; Redondo, Ana R.; Fernández, Rubén R.; Navarro, Jorge; Moguerza, Javier M. (2022-01-31). "General Performance Score for classification problems". Applied Intelligence. 52 (10): 12049–12063. doi:10.1007/s10489-021-03041-7. ISSN 0924-669X. Unknown parameter |s2cid= ignored (help)
↑ ^12.0 ^12.1 Redondo, Ana R.; Navarro, Jorge; Fernández, Rubén R.; de Diego, Isaac Martín; Moguerza, Javier M.; Fernández-Muñoz, Juan José (2020), Unified Performance Measure for Binary Classification Problems, Lecture Notes in Computer Science, 12490, Cham: Springer International Publishing, pp. 104–112, doi:10.1007/978-3-030-62365-4_10, ISBN 978-3-030-62364-7, retrieved 2023-01-05 Unknown parameter |s2cid= ignored (help)
↑ Fernández, Alberto; García, Salvador; Galar, Mikel; Prati, Ronaldo C.; Krawczyk, Bartosz; Herrera, Francisco (2018). Learning from Imbalanced Data Sets. doi:10.1007/978-3-319-98074-4. ISBN 978-3-319-98073-7. Unknown parameter |s2cid= ignored (help) Search this book on

This article "General Performance Score" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:General Performance Score. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.

[1] Fawcett, Tom (2006). "An Introduction to ROC Analysis" (PDF). Pattern Recognition Letters. 27 (8): 861–874. doi:10.1016/j.patrec.2005.10.010.

[2] Piryonesi S. Madeh; El-Diraby Tamer E. (2020-03-01). "Data Analytics in Asset Management: Cost-Effective Prediction of the Pavement Condition Index". Journal of Infrastructure Systems. 26 (1): 04019036. doi:10.1061/(ASCE)IS.1943-555X.0000512.

[3] Powers, David M. W. (2011). "Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation". Journal of Machine Learning Technologies. 2 (1): 37–63.

[4] Ting, Kai Ming (2011). Sammut, Claude; Webb, Geoffrey I., eds. Encyclopedia of machine learning. Springer. doi:10.1007/978-0-387-30164-8. ISBN 978-0-387-30164-8. Search this book on

[5] Brooks, Harold; Brown, Barb; Ebert, Beth; Ferro, Chris; Jolliffe, Ian; Koh, Tieh-Yong; Roebber, Paul; Stephenson, David (2015-01-26). "WWRP/WGNE Joint Working Group on Forecast Verification Research". Collaboration for Australian Weather and Climate Research. World Meteorological Organisation. Retrieved 2019-07-17.

[6] Chicco D.; Jurman G. (January 2020). "The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation". BMC Genomics. 21 (1): 6-1–6-13. doi:10.1186/s12864-019-6413-7. PMC 6941312 Check |pmc= value (help). PMID 31898477.

[7] Chicco D.; Toetsch N.; Jurman G. (February 2021). "The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation". BioData Mining. 14 (13): 1-22. doi:10.1186/s13040-021-00244-z. PMC 7863449 Check |pmc= value (help). PMID 33541410 Check |pmid= value (help).

[8] Chicco D.; Jurman G. (2023). "The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification". BioData Mining. 16 (1). doi:10.1186/s13040-023-00322-4. PMC 9938573 Check |pmc= value (help).

[9] Tharwat A. (August 2018). "Classification assessment methods". Applied Computing and Informatics. doi:10.1016/j.aci.2018.08.003.

[10] Balayla, Jacques (2020). "Prevalence threshold (ϕe) and the geometry of screening curves". PLoS One. 15 (10). doi:10.1371/journal.pone.0240215.

[11] De Diego, Isaac Martín; Redondo, Ana R.; Fernández, Rubén R.; Navarro, Jorge; Moguerza, Javier M. (2022-01-31). "General Performance Score for classification problems". Applied Intelligence. 52 (10): 12049–12063. doi:10.1007/s10489-021-03041-7. ISSN 0924-669X. Unknown parameter |s2cid= ignored (help)

[:0-12] 12.0 ^12.1 Redondo, Ana R.; Navarro, Jorge; Fernández, Rubén R.; de Diego, Isaac Martín; Moguerza, Javier M.; Fernández-Muñoz, Juan José (2020), Unified Performance Measure for Binary Classification Problems, Lecture Notes in Computer Science, 12490, Cham: Springer International Publishing, pp. 104–112, doi:10.1007/978-3-030-62365-4_10, ISBN 978-3-030-62364-7, retrieved 2023-01-05 Unknown parameter |s2cid= ignored (help)

[13] Fernández, Alberto; García, Salvador; Galar, Mikel; Prati, Ronaldo C.; Krawczyk, Bartosz; Herrera, Francisco (2018). Learning from Imbalanced Data Sets. doi:10.1007/978-3-319-98074-4. ISBN 978-3-319-98073-7. Unknown parameter |s2cid= ignored (help) Search this book on

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]