ISUP (Inter Subject Unified Performance)

Analytical performance in blood glucose monitoring (BGM) - New single figure characterization ‘ISUP’ to complement the currently used ‘MARD’

Several approaches to BGM exists. Some 30 years ago, home-test kits were introduced, and over the last 5 years the continuous glucose monitoring (CGM) systems, e.g. from Dexcom, Abbott and Medtronic, have entered mainstream use, where statistical and graphical representations have been developed to summarize the analytical quality with regard to their primary purpose; aiding the diabetic patient in controlling his/her blood glucose levels.

Since the introduction of CGMs, mean absolute relative difference (MARD) has become the most popular figure of merit for performance assessment. In practice, this figure is supplemented by scatter plots in a so-called consensus error grid, where measured values are plotted against reference values (measured by a more accurate reference method) and certain zones are indicated to identify the most critical points in relation to therapeutic decisions. The consensus error grid has been used in the examples below.

It is broadly accepted that the sole use of MARD for assessing the accuracy of glucose monitors may lead to flawed conclusions and, for this reason, it has been suggested to supplement the MARD with reliability metrics, like the precision absolute relative difference (PARD) and MARD reliability index. As an alternative evaluation of sensor performance, a new figure of merit has been proposed, the so-called inter-subject unified performance (ISUP) parameter, which compounds statistical performance and aspects of the consensus error grid plot.

Inter-subject unified performance (ISUP)

The ISUP value is a composite measure that evaluates the performance of a regression model. The ISUP is, by definition, a general performance measure that can be utilized to compare the predictive capability of regression models, irrespective of the origin of the input data and the (continuous) quantity to be predicted. Yet, the ISUP has been developed to quantify the analytical performance of glucose monitoring systems that are routinely used by subjects with diabetes. The challenging aspect in quantifying the performance of glucose monitoring systems owes to the fact that subjects with diabetes show very different temporal behavior and range of glucose values. This may turn single-parameter measures, like root-mean-square-error (RMSE), MARD, and slope, to misleading figures of merit, particularly when compared between subjects.

The composite ISUP parameter is evaluated on cross- or independent validated results and is designed to weight the presence of samples in zone A and B of the consensus error grid, together with a low MARD, a high slope and a high correlation coefficient. Furthermore, the ISUP takes into account the difference in the average glucose level between subjects by weighting with the median of the blood glucose concentrations.

It is noted that the usability of a regression model is application specific, but past studies used a threshold on the ISUP of 0.5 to classify good and bad performing regression models.

Formula

The ISUP of an subject specific regression model is calculated with the following formula:

$ISUP={(A+B-MARD)\times a\times R^{2} \over y_{median}\displaystyle }$

where A and B are percent samples lying in zone A and B of a consensus error grid, MARD is the mean absolute relative difference between estimated (y_p) and reference (y) glucose values, a is the slope of the least-squares fitted linear polynomial, R² is the correlation coefficient between y_p and y, and y_median  is the median of the reference glucose values.

MARD is calculated with the following formula:

$MARD={\frac {1}{n}}\sum {\frac {|y_{p}-y|}{y}}\times 100\%$

where n is the number of paired glucose measurements.

Example

In the figures below, the following three scenarios are presented:

• Scenario A: Subject with well-regulated diabetes. Glucose level from ~4 - ~12 mM.
• Scenario B: Subject with very variating glucose level from ~4 - ~28 mM.
• Scenario C: Subject with mainly hyperglycemic periods. Glucose level from ~9 - ~30 mM.

Scenario A. Subject with well-regulated diabetes. Glucose level from ~4 - ~12 mM.

Scenario B. Subject with very variating glucose level from ~4 - ~28 mM.

Scenario C. Subject with mainly hyperglycemic periods. Glucose level from ~9 - ~30 mM.

From a visual inspection of the consensus error grids, it seems reasonable to conclude that scenario A and B feature well-performing regression models, while that is not the case for scenario C. With an ISUP threshold of 0.5, we arrive at the same conclusion (see Table 1). Though the conclusions of the three scenarios may seem trivial, the examples illustrate the issue of using single-parameter evaluations, like MARD and samples in zone A and B, that lead to the opposite conclusion. The misleading conclusions are a result of both the MARD and samples in zone A and B being vulnerable to deviations in predictions for low glucose levels, while the opposite is true for high glucose levels.

Table 1. Results

Scenario	A	B	C
ISUP	3.62	6.41	-0.01
MARD (%)	30.67	14.9	23.4
A+B (%)	93	100	96

Inter-Subject Unified Performance (ISUP)

This article "ISUP (Inter Subject Unified Performance)" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:ISUP (Inter Subject Unified Performance). Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.

---