Jorge Quitério Search
Technology

Figures of merit for digital twins

The term "digital twin" is often used loosely. Defining figures of merit, namely accuracy, detection rate, precision and latency, makes it an engineering object that can be compared and improved. A stylised anomaly detector shows the trade-off between detection rate and precision.

The term "digital twin" is used inconsistently across vendors and is often a marketing label.

For the final assignment of MIT's Management of Technology programme I defined a technology concept and its roadmap: a digital twin of a satellite ground station, a virtual model fed with live data that mirrors the station's power, radio-frequency and health status. The most instructive part was defining, quantitatively, what makes one digital twin better than another.

This article proposes four figures of merit for a twin, shows why two of them are in tension, and illustrates the trade-off with a simulation.

The relevant standards and literature are as follows.

Grieves introduced the digital twin concept in product lifecycle management in the early 2000s and described it in a 2014 white paper on virtual factory replication.

Kritzinger and colleagues (2018) reviewed the manufacturing literature and proposed the classification used below: digital model, digital shadow and digital twin, distinguished by how data flows between the physical and virtual objects.

ISO/PAS 19450:2015 publishes Object-Process Methodology (OPM), a conceptual modelling language described by Dori (2002), which represents a system as objects and the processes that transform them, in diagrams and equivalent plain-English sentences.

de Weck (2022) frames technology management around figures of merit (FOMs): measurable quantities that capture what users value.

Hyndman and Koehler (2006) reviewed forecast accuracy measures and documented the weaknesses of the mean absolute percentage error, which matter for the accuracy figure below.

Model, shadow and twin

  • Digital model: data moves between physical and virtual by hand.
  • Digital shadow: data flows automatically from the physical system to the model.
  • Digital twin: data flows automatically in both directions, so the model can act on, or advise, the physical system.

Many products sold as twins are shadows. This is not a criticism, since a good shadow is very useful, but it changes what should be measured.

A modern car shows the distinction. The dashboard that displays tyre pressures and warns of a fault is a digital shadow: data flows from the car to the display. A navigation system that receives traffic data and changes the route the car takes closes the loop, which is the defining feature of a twin.

Scoping with OPM

Before any measurement, the system should be described. OPM forces clarity about which objects are mirrored, which processes are simulated and where each data stream comes from. A twin whose scope is not written down cannot be evaluated, because nobody agrees what it is supposed to reproduce.

Four figures of merit

Accuracy. How close the twin's values are to the measured ones. Over nn comparisons between a modelled value Dm,iD_{m,i} and the actual Da,iD_{a,i}:

ACC=(1−1n∑i=1n∣Dm,i−Da,i∣∣Da,i∣)×100%, \text{ACC} = \left(1 - \frac{1}{n}\sum_{i=1}^{n} \frac{|D_{m,i} - D_{a,i}|}{|D_{a,i}|}\right) \times 100\% ,

that is, 100% minus the mean absolute percentage error. Hyndman and Koehler's caution applies: the measure becomes unstable when actual values are near zero, so it should be used only for quantities well away from zero, such as power levels in watts, with a scaled error otherwise.

Detection rate. Of the real anomalies, what share did the twin flag? With aa detected out of bb:

ADR=ab×100%, \text{ADR} = \frac{a}{b} \times 100\%,

which statisticians call recall.

Precision. Of the alarms raised, what share were real? Detection rate alone rewards a twin that alarms on everything. Operators lose trust in a tool with many false alarms faster than in one that occasionally misses something, so precision belongs next to recall. Car alarms illustrate the point: they sound so often without cause that passers-by rarely react to them.

Latency. How long after an event the twin reflects it: treflected−teventt_{\text{reflected}} - t_{\text{event}}. An accurate twin that is ten minutes late cannot support operational decisions.

Lifecycle figures such as availability and cost per monitored asset complete the picture.

Availability of comparable figures

When I tried to compare commercial products on these figures, I could not. None of the vendors I looked at published comparable accuracy, detection or latency figures. They described features, not performance. This indicates an immature market. Buyers should therefore request figures of merit measured in a defined way on their own data.

The stylised detector below illustrates this: detection rate and precision trade off through the alarm threshold, and when anomalies are rare, even a good detector produces mostly false alarms unless its score separates anomalies from normal readings very well.

This is a synthetic, stylised detector: anomaly scores for normal readings and for real anomalies are bell curves d′d' apart.

Suggested settings:

  1. Default settings. With anomalies in 1% of readings, d′=2.5d' = 2.5 and a threshold of 2, the twin catches about 69% of anomalies, but only about 24% of its alarms are real: three false alarms for every true one.
  2. Higher threshold. Raise the threshold to 3. Precision climbs to about 70%, and detection falls to about 31%. The twin itself is unchanged; only the operating point on the trade-off has moved.
  3. Rarer anomalies. Back at threshold 2, lower the share of anomalies to 0.1%. Detection is unchanged, but precision falls to about 3%. A twin evaluated on a test set rich in anomalies appears considerably better than it performs in operation.
  4. Better model. Raise d′d' to 4. Both numbers improve at once. Only a better model, rather than a different threshold, improves both measures.

The formulas. With normal scores N(0,1)N(0,1) and anomaly scores N(d′,1)N(d',1), a threshold tt gives

ADR=1−Φ(t−d′),FPR=1−Φ(t),precision=π ADRπ ADR+(1−π) FPR, \text{ADR} = 1 - \Phi(t - d'), \qquad \text{FPR} = 1 - \Phi(t), \qquad \text{precision} = \frac{\pi\,\text{ADR}}{\pi\,\text{ADR} + (1-\pi)\,\text{FPR}} ,

with π\pi the share of anomalies. Raising tt lowers both ADR and FPR, so precision rises while recall falls. Precision depends on π\pi and recall does not, which is why the same detector looks so different on a test bench and in operation. For π=0.01\pi = 0.01, d′=2.5d' = 2.5, t=2t = 2: ADR=1−Φ(−0.5)≈0.69\text{ADR} = 1 - \Phi(-0.5) \approx 0.69, FPR≈0.023\text{FPR} \approx 0.023, precision ≈0.0069/(0.0069+0.0225)≈0.24\approx 0.0069/(0.0069 + 0.0225) \approx 0.24.

With defined figures of merit, a digital twin becomes an engineering object that can be compared and improved. Recommendations for buyers and developers:

  • Write down the scope first, with a model of the system, so everyone agrees what is mirrored.
  • Report accuracy, detection rate, precision and latency together, each with how it was measured.
  • Measure precision at the anomaly rate expected in operation. An enriched test set overstates it.
  • Ask vendors for these numbers on the buyer's own data. The response is itself informative about the vendor.

The same base-rate arithmetic governs security scan gates, as I showed in a signed and scanned container pipeline for Docker hosts.

References

  1. M. Grieves, Digital Twin: Manufacturing Excellence through Virtual Factory Replication, white paper, 2014.
  2. W. Kritzinger, M. Karner, G. Traar, J. Henjes and W. Sihn, "Digital Twin in manufacturing: A categorical literature review and classification", IFAC-PapersOnLine, 51(11), 1016-1022, 2018.
  3. ISO/PAS 19450:2015, Automation systems and integration - Object-Process Methodology.
  4. D. Dori, Object-Process Methodology: A Holistic Systems Paradigm, Springer, 2002.
  5. O. L. de Weck, Technology Roadmapping and Development: A Quantitative Approach to the Management of Technology, Springer, 2022.
  6. R. J. Hyndman and A. B. Koehler, "Another look at measures of forecast accuracy", International Journal of Forecasting, 22(4), 679-688, 2006.