Are well-calibrated users effective users? Associations between calibration of trust and performance on an automation-aided task.

OBJECTIVE

We present alternative operationalizations of trust calibration and examine their associations with predictors and outcomes.

BACKGROUND

It is thought that trust calibration (correspondence between aid reliability and user trust in the aid) is a key to effective human-automation performance. We propose that calibration can be operationalized in three ways. Perceptual accuracy is the extent to which the user perceives the aid's reliability accurately at one point in time. Perceptual sensitivity and trust sensitivity reflect user adjustment of perceived reliability and trust as the aid's actual reliability changes over time.

METHOD

One hundred fifty-five students completed an X-ray screening task with an automated screener. Awareness of the aid's accuracy trajectory and error type was examined as predictors, and task performance and aid failure detection were examined as outcomes.

RESULTS

Awareness of accuracy trajectory was significantly associated with all three operationalizations of calibration, but awareness of error type was not when considered in conjunction with accuracy trajectory. Contrary to expectations, only perceptual accuracy was significantly associated with task performance and failure detection, and combined, the three operationalizations accounted for only 9% and 4% of the variance in these outcomes, respectively.

CONCLUSION

Our results suggest that the potential importance of trust calibration warrants further examination. Moderators may exist.

APPLICATION

Users who were better able to perform the task unaided were better able to identify and correct aid failure, suggesting that user task training and expertise may benefit human-automation performance.