Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making (2403.16812v2)

Abstract: In AI-assisted decision-making, humans often passively review AI's suggestion and decide whether to accept or reject it as a whole. In such a paradigm, humans are found to rarely trigger analytical thinking and face difficulties in communicating the nuances of conflicting opinions to the AI when disagreements occur. To tackle this challenge, we propose Human-AI Deliberation, a novel framework to promote human reflection and discussion on conflicting human-AI opinions in decision-making. Based on theories in human deliberation, this framework engages humans and AI in dimension-level opinion elicitation, deliberative discussion, and decision updates. To empower AI with deliberative capabilities, we designed Deliberative AI, which leverages LLMs as a bridge between humans and domain-specific models to enable flexible conversational interactions and faithful information provision. An exploratory evaluation on a graduate admissions task shows that Deliberative AI outperforms conventional explainable AI (XAI) assistants in improving humans' appropriate reliance and task performance. Based on a mixed-methods analysis of participant behavior, perception, user experience, and open-ended feedback, we draw implications for future AI-assisted decision tool design.

• The paper presents a novel Human-AI Deliberation framework that integrates LLMs with domain-specific models to bridge human and AI perspectives.
• It outlines a multi-layer architecture—communication, control, and knowledge layers—for dynamic opinion updating during deliberative discussions.
• The study demonstrates improved decision accuracy and reduced over-reliance on AI, as evidenced by enhanced performance in graduate admissions tasks.

Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making

The paper "Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making" (2403.16812) addresses significant challenges in AI-assisted decision-making processes. It proposes a novel paradigm termed Human-AI Deliberation, designed to foster analytical reasoning and facilitate the exchange of divergent opinions between humans and AI systems. The framework introduces the concept of Deliberative AI, leveraging the capabilities of LLMs to bridge communication between human users and domain-specific models. Through an illustrative paper on graduate admissions, this research explores the impact of Human-AI Deliberation on decision accuracy, reliance on AI, and user perceptions.

Figure 1: An illustration of Human-AI Deliberation. (A) In traditional AI-assisted decision-making, when humans disagree with AI's suggestions (and only find parts of AI's reasons applaudable), it is difficult for humans to decide whether and how much to adopt AI's suggestion. (B) In our proposed Human-AI Deliberation, we provide opportunities for the human and the AI model to deliberate on conflicting opinions by discussing related evidence and arguments. Then, AI and humans can update their thoughts (when find it necessary) and reach final predictions.

Introduction

AI technologies have increasingly been implemented across various decision-making domains, such as criminal justice, admissions, and healthcare, to enhance decision support. However, two critical challenges persist within the traditional AI-assisted decision-making framework: a failure to trigger necessary analytical thinking in human decision-makers, leading to inappropriate levels of reliance on AI recommendations, and limited human-AI interaction when conflicts arise in decision rationale.

Figure 2: The framework for Human-AI Deliberation. (A) Illustrates the WoE concept in decision-making, showcasing how decision-makers assess evidence across dimensions to shape opinions and arrive at a final decision. (B) Presents the Architecture for Human-AI Deliberation, with key activities (shown in grey boxes) and potential design space (shown in dashed-line boxes).

Human-AI Deliberation is proposed as an innovative solution based on human deliberation theories, facilitating an interactive process that encourages discussion of evidence at the dimension level to resolve conflicting opinions. The framework leverages theories such as Weight of Evidence (WoE) and provides a model-augmented AI—Deliberative AI—that supports natural dialogues with users by combining LLMs with domain-specific models.

Figure 3: The Architecture of Deliberative AI. Our design integrates both a domain-specific model and a LLM, enabling the AI to engage in natural communication with humans while also harnessing domain knowledge from the specialized model.

Implementation of Deliberative AI

The implementation of Deliberative AI is divided into three layers: Communication, Control, and Knowledge Layers, each playing a vital role in facilitating naturalistic and informative interactions between human users and the AI.

Communication Layer: Central to this layer is the capability of LLMs to analyze user intentions and discourse constructs. The system evaluates human inputs across various dimensions, enabling AI to engage constructively and update its opinions dynamically. This layer is comprised of three components: (1) Intention Analyzer, (2) Deliberation Facilitator, and (3) Argument Evaluator.

Control Layer: This layer acts as a mediator between the descriptive outputs of the DS-model and the user-focused responses of the LLM. Key components include the Dialogue Controller, which orchestrates the discussion flow; the Knowledge Extractor, which provides LLM with data from domain-specific models; and the Opinion Update Controller, enabling AI to adapt its stance in response to a human's argument strength during the deliberation.

Figure 4: An illustration of how Deliberative AI processes humans' inputs and how it generates outputs. The prompts used are simplified in this figure for illustration purposes. For the complete prompts, please check our supplementary materials.

Implementation of Human-AI Deliberation

Task and Dataset

A graduate admission decision-making task served as the illustrative setup for implementing the proposed framework. The task utilized a synthesized dataset simulating student profiles at a U.S. public university, replete with attributes like GPA, major, recommendation letters, and associated decision labels (accept or reject).

To build a domain-specific model, a multi-category linear regression model was trained on 70% of the dataset. This DS-model integrates with an LLM—GPT-3.5-turbo in this paper—to enable real-time generation and interactive communication.

Deliberative AI Implementation

Human-AI Thought Representation and Alignment: The system begins by gathering structured thoughts from humans and the AI. While AI employs SHAP to determine feature contributions in tabular data, humans provide answers that encompass reasoning and subjective perspectives. This common ground facilitates an effective basis for deliberative discussions.

Implementation Details:

• Intention Analyzer: GPT-3.5-turbo interprets user input, identifying intent and dimension relevant to the discussion.
• Deliberation Facilitator: Prompts guide the Generative Pre-trained Transformer (GPT) to engage in meaningful interactions by evaluating user inputs for coherence, relevance, and ensuring correspondence with domain-specific models.

  Figure 3: The Architecture of Deliberative AI. Our design integrates both a domain-specific model and a LLM, enabling the AI to engage in natural communication with humans while also harnessing domain knowledge derived from the specialized model.

Control Layer: Functions by querying domain-specific information as inputs to the LLM for deliberative discussions, maintaining message consistency through the Regulator, and guiding the conversation flow with the Dialogue Controller to ensure an organized exchange between the human and AI. The Opinion Update Controller enables dynamic updates to AI opinions in response to user engagement.

Exploratory User Study in Graduate Admission Prediction

An exploratory evaluation was conducted in a graduate admissions context, involving 174 participants with graduate admission experience recruited on Prolific. Participants' reliance, task performance, perceptions, and user experiences with Deliberative AI were evaluated, comparing two baseline conditions: traditional XAI assistants and humans functioning independently.

Figure 5: Task performance in different conditions. The error bars represent 95\% confidence interval. (

: $p$<0.05, *: $p$<0.01, *: $p$<0.001).

Task Setup and Conditions

The paper involved a graduate admission task, where participants assessed applicant profiles and predicted admission chances. The task was selected due to its familiar nature in AI-assisted decision-making research and existing complexities. To ensure genuine engagement, challenging task cases were curated where both humans and AI showed moderate accuracy.

Three paper conditions were evaluated: Deliberative AI (DAI), Explainable AI (XAI), and Human Alone. Deliberative AI uniquely required participants to engage in discussions around conflicting opinions with AI.

Figure 6: The conversation flow for the deliberative discussion.

Exploratory Study Outcomes

The results of the exploratory evaluation highlighted several key findings regarding the impact of Human-AI Deliberation in AI-assisted decision-making. Deliberative AI demonstrated improved decision accuracy and reduced over-reliance on AI's incorrect recommendations compared to traditional XAI setups.

Figure 5: Task performance in different conditions. The error bars represent 95% confidence interval. (

: $p$<0.05, *: $p<0.01$, *: $p<0.001$).

Participants engaged more deeply during deliberations, as reflected by lower agreement and switch fractions (Figure 7) and engaged in multiple rounds of dialogue even when not prompted by AI, enhancing their understanding and helping them identify flaws in AI recommendations.

Figure 7: Participants' reliance and the appropriateness of their reliance. (A) Participants' reliance on AI's suggestions was measured by agreement fraction and switch fraction. (B) The appropriateness of participants' reliance on AI's suggestions, including under-reliance (the ratio where participants did not use a correct AI suggestion) and over-reliance (the ratio where participants used an incorrect AI suggestion). The error bars represent 95% confidence interval. (

: $p<0.05$, *: $p<0.01$, *: $p<0.001$).

Implications for AI-Assisted Decision-Making

Human-AI Deliberation presents opportunities for enhancing decision accuracy and ensuring more suitable reliance on AI by fostering the exchange of perspectives between humans and AI. In situations where tasks challenge both humans and AI, Deliberative AI demonstrates a potential to enhance overall task outcomes compared to traditional XAI.

Figure 8: Learning curve measured by decision accuracy in each task round. The error bars represent 95\% confidence interval.

Despite the increased cognitive demand, Deliberative AI encourages deeper independent thinking by compelling users to reflect on conflicting perspectives and reduce anchoring biases. As participants identified AI's limitations and developed their perspectives (Figure 7), appropriate reliance was promoted, manifesting in increased decision accuracy even when faced with challenging tasks (Figure 5). Furthermore, Human-AI Deliberation instills a learning effect, as evidenced by the gradual improvement in participants' performance over successive decision rounds (Figure 8), suggesting potential long-term benefits for decision-making contexts akin to graduate admissions.

Figure 7: Participants' reliance and the appropriateness of their reliance. (A) Participants' reliance on AI's suggestions was measured by agreement fraction and switch fraction. (B) The appropriateness of participants' reliance on AI's suggestions, including under-reliance (the ratio where participants did not use a correct AI suggestion) and over-reliance (the ratio where participants used an incorrect AI suggestion). The error bars represent 95% confidence interval. (

: $p<0.05$, *: $p<0.01$, *: $p<0.001$).

Conclusion

The research delineates substantial implications for integrating deliberation into AI-assisted decision-making frameworks, illustrating tangible benefits for tasks characterized by multifaceted challenges. Future studies should aim to refine AI's dynamic update capabilities to bolster transparency while enlarging the scope of deliberative aids to encompass additional decision domains. Furthermore, potential customization features in deliberative discussions could serve as a catalyst for deeper user engagement and sustained collaborative improvements in decision-making efficacy.