An Uncertainty-based Human-in-the-loop System for Industrial Tool Wear Analysis (2007.07129v2)

Abstract: Convolutional neural networks have shown to achieve superior performance on image segmentation tasks. However, convolutional neural networks, operating as black-box systems, generally do not provide a reliable measure about the confidence of their decisions. This leads to various problems in industrial settings, amongst others, inadequate levels of trust from users in the model's outputs as well as a non-compliance with current policy guidelines (e.g., EU AI Strategy). To address these issues, we use uncertainty measures based on Monte-Carlo dropout in the context of a human-in-the-loop system to increase the system's transparency and performance. In particular, we demonstrate the benefits described above on a real-world multi-class image segmentation task of wear analysis in the machining industry. Following previous work, we show that the quality of a prediction correlates with the model's uncertainty. Additionally, we demonstrate that a multiple linear regression using the model's uncertainties as independent variables significantly explains the quality of a prediction ((R^2=0.718)). Within the uncertainty-based human-in-the-loop system, the multiple regression aims at identifying failed predictions on an image-level. The system utilizes a human expert to label these failed predictions manually. A simulation study demonstrates that the uncertainty-based human-in-the-loop system increases performance for different levels of human involvement in comparison to a random-based human-in-the-loop system. To ensure generalizability, we show that the presented approach achieves similar results on the publicly available Cityscapes dataset.