Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction (2303.09224v1)

Abstract: Recent work has proposed AI models that can learn to decide whether to make a prediction for an instance of a task or to delegate it to a human by considering both parties' capabilities. In simulations with synthetically generated or context-independent human predictions, delegation can help improve the performance of human-AI teams -- compared to humans or the AI model completing the task alone. However, so far, it remains unclear how humans perform and how they perceive the task when they are aware that an AI model delegated task instances to them. In an experimental study with 196 participants, we show that task performance and task satisfaction improve through AI delegation, regardless of whether humans are aware of the delegation. Additionally, we identify humans' increased levels of self-efficacy as the underlying mechanism for these improvements in performance and satisfaction. Our findings provide initial evidence that allowing AI models to take over more management responsibilities can be an effective form of human-AI collaboration in workplaces.

• The paper demonstrates that AI delegation enhances human task performance compared to independent human or AI workflows.
• The study’s randomized experiment with 196 participants shows increased accuracy and higher task satisfaction under delegation conditions.
• Mediation analysis reveals that improved self-efficacy is a key driver for the enhanced performance and satisfaction observed.

Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction

Introduction

In "Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction" (2303.09224), the authors investigate the dynamics of collaboration between humans and AI models, particularly focusing on how AI delegation impacts human task performance and satisfaction. The paper acknowledges recent advancements in AI capabilities, which in certain domains, such as medical diagnostics, have surpassed human expertise. Despite these breakthroughs, human judgment remains superior in specific contexts due to factors like side information inaccessible to AI. The paper explores AI delegation—a method allowing AI models to allocate decision-making tasks to humans rather than make predictions themselves—and how this impacts the performance and perception of human collaborators.

Figure 1: A schematic overview of the AI model. During training, the AI model learns to make a prediction for a task from available ground truth labels. Additionally, human predictions allow the AI model to learn human capabilities simultaneously. After deployment, the AI model decides to make a prediction or delegate an instance to the human, depending on expected prediction correctness probabilities.

The research proposes that allowing AI models to delegate more management responsibilities could enhance human-AI collaboration, particularly in high-volume decision-making environments like content moderation or medical diagnostics.

Research Questions and Hypotheses

The authors develop several hypotheses, primarily focused on whether AI delegation improves task performance and satisfaction compared to independent human and AI workflows. They posit that:

1. H1: AI delegation enhances human task performance relative to humans and AI operating independently.
2. H2: AI delegation improves human task satisfaction relative to humans working independently.
3. H3: Self-efficacy mediates the effect of AI delegation on task performance; AI delegation increases self-efficacy, which subsequently enhances task performance.
4. H4: Self-efficacy mediates the effect of AI delegation on task satisfaction; AI delegation increases self-efficacy, which enhances task satisfaction.

These hypotheses are tested using an experimental framework involving image classification tasks and an AI model that estimates human error predictions to optimize task delegation.

Figure 2: Research model: Proposed effects of AI delegation on human task performance and satisfaction.

Methodology

The paper employs a randomized experiment involving 196 participants who are tasked with classifying images under varying conditions of AI involvement.

Data:

The researchers utilize a modified subset of the ImageNet dataset, specifically including consistent visual distortions to balance performance levels between humans and AI. Participants classify images from various classes, enabling the evaluation of task performance cohesively across different experimental groups.

AI Model Development:

The AI model described employs a dual-component system: an image classification model and a human error prediction model. Delegation is based on confidence levels; the AI delegates tasks if the predicted confidence of human success exceeds that of the AI model.

Figure 3: Interface of the image classification task, exemplified by the delegation condition. Participants are informed about AI delegation, and view images already classified by the AI.

Experimental Design:

Participants are distributed across three groups:

• An AI delegation group (with notified delegation)
• A hidden delegation group (not informed about delegation)
• A human-alone control group.

Performance is evaluated by calculating classification accuracy, task satisfaction, and self-efficacy.

Results

The findings robustly support the initial hypotheses:

1. Task Performance: Participants in both AI delegation groups performed better than those working independently. No significant performance difference was found between the two delegation conditions, suggesting that awareness of delegation did not influence outcomes.

Figure 4: Task performance, task satisfaction, and self-efficacy of the participants, split by conditions.

1. Task Satisfaction: Satisfaction levels were higher in the delegation groups, with self-efficacy emerging as a critical mediator for these perceptions.

   Figure 5: Overview of the direct and indirect effect of AI delegation on task performance and task satisfaction.

2. Self-efficacy: The mediation analyses confirmed that increased self-efficacy drove improvements in both task performance and satisfaction.

Discussion and Implications

The paper’s results suggest that AI delegation, by considering both human and AI capabilities, can enhance task performance and satisfaction without additional communication about task assignment rationale. This has implications for applying AI delegation in environments such as the gig economy and in fields where task specialization is necessary. AI's role as both manager and performer in human-AI teams could revolutionize task allocation strategies, optimizing collaboration across various domains.

Conclusion

This research validates AI delegation as an effective strategy for enhancing human-AI team performance and satisfaction, emphasizing the role of self-efficacy in these improvements. Future directions may explore personalization of delegation strategies, transparency enhancements in AI task assignments, and the broader applications of AI delegation in specialized fields. The strategic deployment of AI as a managerial entity in task distribution could reshape conventional workflows, underscoring the need for further exploration in diverse practical scenarios.