Does Higher Prediction Accuracy Guarantee Better Human-AI Collaboration?
The renaissance of AI has led to revolutionary breakthroughs in numerous applications. It has shown great potentials to automate daily workflows of professionals and to collaborate with users as a team. For example, doctors referencing the diagnoses recommended by AI before making final decisions can be more effective and accurate than entirely rely on one of them. To make these AI more capable of supporting human decision making, it seems to be widely believed that we simply need a more accurate AI model. But the truth might be surprisingly counter-intuitive.
It’s all about human factors.
let’s start with the basics. We know that to achieve better accuracies, AI models need to be updated with new data or algorithms over time, which is called training in ML terms. These updates usually increase the accuracy of model prediction, as they were designed for, but they also introduce new problems if we take into consideration the interaction between humans and AI — How well would human users be able to adapt to the updated AI? How would users’ mental models, which were established by interacting with the original models, influence the acceptance of the updated ones?
Updates that increase AI performance may actually hurt team performance.
As stated clearly in a paper [1] published in a top-tier AI conference AAAI 2019 by researchers from the University of Washington, Microsoft Research and the University of Michigan. The results show that if the updates to AI models are incompatible with the original ones, even with increased accuracies, the collaborative performance will drop significantly.
But wait a second…what do authors mean by incompatible?
In software engineering, an update is backward compatible if the updated system can support legacy software. By analogy, we define that an update to an AI component is locally compatible with a user’s mental model if it does not introduce new errors and the user, even after the update, can safely trust the AI’s recommendations.
In the context of cognitive psychology, humans would gradually develop their mental models while interacting with artifacts. In this paper, more specifically, the mental model indicates which inputs the users expect the AI will process correctly. If an AI model is updated incompatibly, which might be common for the neural nets due to its black-box nature, chances are that it will violate users’ expectations of correctness — we’ll dive deeper into this.
To investigate the effects of updates on AI models, the authors conducted an experiment where participants collaborated with an imperfect AI model to maximize scores in a classification game. To boost the score, users must gradually learn when to trust the AI (i.e. build mental models on when the AI will succeed or fail at the classification). After 75 cycles, the researchers updated the AI models to be 5% more accurate and the update could be either compatible or incompatible. Remember that an update is compatible if it introduces no new errors, vice versa. Therefore, if the update is compatible, the users wouldn’t make mistakes by adopting the same trust strategies as they had before updates!
The results surprisingly showed that a more accurate but incompatible classifier results in lower team performance than a less accurate but compatible classifier (no update). On the other hand, compatible updates did improve team performance. The implication here is NEVER update your model with an INCOMPATIBLE update!
The authors explained that it might have been due to that incompatible updates sacrificed the team score while workers had to re-learn the new error boundaries where the new models would fail.
So, key takeaways — does higher prediction accuracy guarantee better human-AI collaboration? The answer is not necessary as we need to take into consideration human factors such as users’ mental models. However, if we update AI models with careful regards to compatibility, we can ensure a better human-AI team performance. Nowadays, we see many AI papers omit the importance of humanity in the context of AI and focus only on boosting accuracy. This paper stood out by providing a different human-centered perspective to AI. However, it’s also noteworthy that the experiments in this paper are somehow toy-like, using relatively naive classification tasks. It would be interesting to see relevant experiments to be conducted on real-world tasks, for instance, speech recognition on cellphones. I always feel uncertain about how well my phone could recognize my spoken words, hence at times prevent me from using speech assistant services.
The article only summarizes a small part of the original paper referenced, please refer to the manuscript for more details such as the tradeoff between the performance and the compatibility and Do current ML classifiers produce compatible updates?
Reference: [1] Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S. Weld, Walter S. Lasecki, Eric Horvitz. Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. In AAAI 2019,
I have compiled a paper list on Human-AI Interaction with wonderful co-curators across institutes over the world. Click the link below to check more papers related to this topic, and, don’t forget to star it! https://github.com/bwang514/awesome-HAI
