Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models
Abstract: Questions in open-domain question answering are often ambiguous, allowing multiple interpretations. One approach to handling them is to identify all possible interpretations of the ambiguous question (AQ) and to generate a long-form answer addressing them all, as suggested by Stelmakh et al., (2022). While it provides a comprehensive response without bothering the user for clarification, considering multiple dimensions of ambiguity and gathering corresponding knowledge remains a challenge. To cope with the challenge, we propose a novel framework, Tree of Clarifications (ToC): It recursively constructs a tree of disambiguations for the AQ -- via few-shot prompting leveraging external knowledge -- and uses it to generate a long-form answer. ToC outperforms existing baselines on ASQA in a few-shot setup across the metrics, while surpassing fully-supervised baselines trained on the whole training set in terms of Disambig-F1 and Disambig-ROUGE. Code is available at https://github.com/gankim/tree-of-clarifications.
Summary
- The paper introduces Tree of Clarifications, a retrieval-augmented framework that recursively expands disambiguated questions, verifies their relevance, and synthesizes evidence-grounded long-form answers.
- On the ASQA benchmark, the full five-shot GPT-3 system achieves 33.7 Disambig-F1 and 36.6 DR, outperforming the strongest supervised baseline by 7.3 and 2.9 points, respectively.
- Self-verification provides the largest filtering benefit by reducing 12,838 generated disambiguations to 4,239 and increasing Answer-F1 from 40.9 to 59.3, though the approach remains costly and untested beyond ASQA and GPT-3.
Problem formulation and motivation
“Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented LLMs” (2310.14696) addresses open-domain question answering when a user’s query admits multiple legitimate interpretations. The paper adopts the task formulation of ASQA, in which a system does not ask the user to resolve the ambiguity. Instead, it must identify the relevant disambiguated questions (DQs), answer them, and integrate the answers into a coherent long-form response.
This formulation is more demanding than conventional factoid QA for two reasons. First, ambiguity can arise along several dimensions simultaneously. The question “what country has the most medals in Olympic history,” for example, may vary according to medal type, Olympic season, or another implicit comparison criterion. Second, identifying plausible interpretations often requires world knowledge that is not reliably encoded in the LLM’s parametric memory. A system can therefore produce linguistically plausible but incomplete or factually unsupported disambiguations even when it generates a fluent final answer.
The proposed solution, Tree of Clarifications (ToC), combines retrieval-augmented prompting with recursive exploration of interpretations. It is explicitly designed for few-shot inference rather than task-specific fine-tuning. The central claim is that ambiguity should be explored structurally: instead of generating a flat list of interpretations in a single prompt, the system recursively expands individual DQs, allowing each branch to retrieve evidence targeted to its own interpretation.
Framework architecture
ToC consists of retrieval-augmented clarification (RAC), a recursive tree structure (TS), self-verification-based pruning, and final answer generation. The framework first retrieves relevant Wikipedia passages using ColBERT and Bing, combines the retrieved material, and reranks it with Sentence-BERT. The top passages are inserted into a few-shot prompt that asks the LLM to generate DQ–answer pairs for the original ambiguous question.
The resulting DQs become child nodes in a clarification tree. Each child is expanded recursively by rerunning RAC with passages reranked for that particular query. This query-conditioned retrieval is important: the evidence relevant to one interpretation may differ substantially from the evidence relevant to another. ToC uses breadth-first search by default, prioritizing breadth of interpretation over exhaustive depth along a single branch. Expansion terminates after reaching a predefined number of valid nodes, reaching the maximum depth, or failing to produce useful expansions repeatedly.
Figure 1: ToC retrieves evidence for the ambiguous question, recursively generates and filters disambiguated questions, and produces a long-form answer from the retained nodes.
The tree representation operationalizes the paper’s view that ambiguity is multidimensional. A root question may first be divided according to entities, temporal conditions, categories, or measurement criteria; subsequent branches can then refine those dimensions independently. This is a substantive architectural distinction from prompting methods that ask a LLM to enumerate interpretations in one step.
Retrieval-augmented clarification
RAC provides the factual substrate for both disambiguation and answer generation. The retrieval pipeline collects more than 200 passages by combining ColBERT and Bing results, reranks them, and supplies the highest-ranked passages to the LLM. Five dynamically selected in-context examples are also included, chosen by nearest-neighbor search over question representations.
The paper’s ablation results establish that retrieval is not merely useful for final answer support; it is particularly important for generating the DQs themselves. The GPT-3 baseline obtains a Disambig-F1 of 24.2 and a Disambiguation-ROUGE score of 29.5. Adding RAC raises these values to 31.1 and 35.1, respectively. Removing the retrieval systems reduces Disambig-F1 to 25.6, nearly eliminating the improvement over the closed-book baseline. The implication is that external evidence primarily improves factual coverage and interpretation accuracy rather than only lexical realization.
The two retrieval sources are complementary. On a 100-example sample, the combined reranked system achieves answer coverage of 64.2 at the top 10 passages and 80.1 at the top 100, compared with 56.8 and 73.4 for reranked ColBERTv2 and 62.7 and 72.8 for reranked Bing. These figures support the system design, but they also qualify the claim that retrieval solves the knowledge problem: even the combined retrieval pool fails to cover approximately one-fifth of disambiguated answers at depth 100 under the paper’s answer-coverage proxy.
Recursive exploration and pruning
The TS component improves on one-shot RAC by expanding each generated DQ independently. In the main ablation, RAC alone reaches 31.1 Disambig-F1, 39.6 ROUGE-L, and 35.1 DR. Adding the tree structure increases Disambig-F1 to 32.4 and DR to 36.0. The gains are modest but directionally consistent, indicating that recursive exploration improves factual disambiguation more reliably than surface-level answer overlap.
ToC then applies self-verification to remove nodes that are irrelevant to the original AQ. The verifier receives the root question, the target DQ’s answer, and an answer-containing passage, and predicts whether the target answer could answer the original question. This criterion is intended to reject factually correct but scope-shifting branches. For instance, a node asking who hosted the 2018 World Cup may be factually supported by its passage while remaining irrelevant to a question about the 2022 host.
Pruning produces the strongest reported result. The full system, GPT-3 + RAC + TS with pruning, obtains:
| System | Disambig-F1 | ROUGE-L | DR |
|---|---|---|---|
| GPT-3 baseline | 24.2 | 36.0 | 29.5 |
| GPT-3 + RAC | 31.1 | 39.6 | 35.1 |
| GPT-3 + RAC + TS | 32.4 | 40.0 | 36.0 |
| GPT-3 + RAC + TS + pruning | 33.7 | 39.7 | 36.6 |
The final system surpasses the best fully supervised baseline, T5-Large with JPR, by 7.3 points in Disambig-F1 and 2.9 points in DR, despite using only five in-context examples rather than the complete training set. Relative to the prior few-shot GPT-3 baseline, the improvements are 8.4 points in Disambig-F1 and 7.0 points in DR. The result is notably asymmetric: pruning slightly lowers ROUGE-L from 40.0 to 39.7 while improving factual disambiguation. This supports the paper’s broader observation that lexical overlap is an inadequate proxy for factual completeness in long-form QA.
The intrinsic pruning analysis provides a direct explanation for this improvement. Without pruning, the system generates 12,838 DQs with Answer-F1 of 40.9. Deduplication alone reduces the set to 10,598 DQs but lowers Answer-F1 to 40.1. Self-verification retains 4,239 DQs and raises Answer-F1 to 59.3, an improvement of 18.4 points. Thus, the benefit does not arise from merely controlling redundancy; it comes from filtering nodes according to their factual coherence with the root question.
Final answer construction
After tree expansion and pruning, ToC aggregates up to ten retained disambiguations in breadth-first order and prioritizes five passages containing their answers. A final GPT-3 prompt receives the original AQ, the selected DQs and answers, and the retrieved evidence. It is instructed to produce a detailed response that explicitly covers the multiple interpretations.
This final stage separates interpretation discovery from response synthesis. The system does not require the model to infer every ambiguity and compose the answer in a single generation. Instead, it exposes intermediate structured evidence to the final generator. The ablation in which disambiguations are removed from the few-shot examples reduces ROUGE-L from 39.6 to 37.3, showing that the intermediate DQ–answer format itself contributes to answer completeness. The result also indicates that answer generation depends on more than passage retrieval: the model benefits from being shown how ambiguity should be decomposed and summarized.
The framework’s computational cost is bounded operationally rather than asymptotically. ToC uses fewer than 20 language-model calls per question, terminates after obtaining ten valid nodes, and can stop after three consecutive failed expansions. Self-verification calls use shorter prompts than RAC calls because they contain a single relevant passage and no few-shot exemplars. The paper therefore argues that ToC provides state-of-the-art performance with a call count comparable to ensemble-style methods, although this comparison does not include API cost, latency, retrieval overhead, or the variance introduced by multiple generations.
Evaluation and interpretation of results
The evaluation uses ASQA, a 6,316-example benchmark constructed from ambiguous questions in AmbigNQ and Natural Questions. The split contains 4,353 training, 948 development, and 1,015 test examples. The primary metrics are Disambig-F1, which measures factual correctness of answers extracted for reference DQs; ROUGE-L, which measures lexical overlap with reference long-form answers; and DR, the geometric mean of Disambig-F1 and ROUGE-L.
The performance profile favors factual coverage over lexical similarity. The fully supervised T5-Large closed-book model has only 7.4 Disambig-F1 but 33.5 ROUGE-L, whereas the strongest ToC model has 33.7 Disambig-F1 and 39.7 ROUGE-L. This discrepancy reinforces the paper’s reliance on Disambig-F1 and DR as more informative measures of whether the answer addresses the intended interpretations. Nevertheless, Disambig-F1 is itself dependent on an extractive reading-comprehension model trained on SQuADv2, so it is not an entirely model-independent assessment of factual correctness.
The comparison between fully supervised and few-shot systems also supports a specific methodological conclusion. Standard few-shot GPT-3 performs worse than fully supervised baselines across all reported metrics, but retrieval, recursive exploration, and pruning reverse that relationship on the factual metrics. The result should not be interpreted as evidence that few-shot prompting generally supersedes supervised learning: the comparison is restricted to ASQA, uses a particular GPT-3 configuration, and evaluates a framework with substantially more inference-time computation.
Limitations and open questions
The empirical evidence is limited to ASQA and a GPT-3 backbone, so the paper does not establish generalization across model scales, architectures, retrieval corpora, or ambiguity types. The framework is described as model-agnostic, but this property is not experimentally demonstrated. In particular, the behavior of self-verification may depend on the LLM’s ability to judge entailment and scope consistency, and the paper does not provide a systematic calibration analysis of the verifier.
ToC also does not explicitly classify whether a question is ambiguous. It attempts recursive clarification even for unambiguous questions, which can yield duplicate or irrelevant DQs. Treating failure to generate valid disambiguations as evidence of unambiguity is only an operational heuristic, not a validated ambiguity detector. This limitation matters because unnecessary branching increases both noise and inference cost.
The pruning mechanism is vulnerable to verifier errors. The qualitative analysis includes incorrect pruning decisions, including cases where a proposed answer is relevant to a narrow interpretation but the verifier fails to recognize the relationship. Conversely, a factually supported answer may pass verification despite subtly changing the original scope. Since pruning substantially improves results, errors at this stage can directly affect both recall of valid interpretations and precision of the final answer.
Finally, the method incurs nontrivial computational and monetary costs through repeated retrieval, reranking, generation, and verification. The reported upper bound of fewer than 20 LLM calls per question does not establish practical efficiency under production constraints. The paper also reports that chain-of-thought prompting did not improve pilot results, but it does not isolate whether this reflects insufficient prompting, a limitation of the backbone, or the greater importance of document grounding for this task.
Conclusion
ToC frames ambiguous long-form QA as structured interpretation search rather than single-pass response generation. Its combination of retrieval-grounded DQ generation, breadth-first recursive expansion, self-verification, and evidence-based synthesis improves factual disambiguation on ASQA. The strongest result is a five-shot GPT-3 system achieving 33.7 Disambig-F1 and 36.6 DR, exceeding fully supervised baselines on both metrics. The evidence attributes this improvement primarily to retrieval and self-verification, while also showing that lexical overlap alone does not adequately measure progress. The principal open question is whether the same gains persist across datasets and model families when ambiguity detection, verifier calibration, and inference cost are evaluated explicitly.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is this paper about?
This paper presents a system called Tree of Clarifications (ToC). It helps artificial intelligence answer questions that could mean several different things.
For example, consider the question:
“Which country has won the most Olympic medals?”
This question is unclear because “medals” might mean:
- Gold, silver, or bronze medals
- Medals from the Summer or Winter Olympics
- Medals won by athletes or by countries in a particular time period
Instead of asking the user to explain what they mean, ToC tries to think of the possible meanings and gives one complete answer covering them.
2. What questions are the researchers trying to answer?
The researchers wanted to find out whether a LLM could give better answers to unclear questions if it:
- Finds different possible meanings of the original question.
- Uses outside information, such as Wikipedia and web search, rather than relying only on what it already knows.
- Explores meanings step by step, like branches growing from a tree.
- Removes confusing or unrelated interpretations before writing the final answer.
- Produces a long, complete answer that discusses all the useful interpretations.
The main research question was essentially:
Can a LLM answer ambiguous questions more accurately by retrieving information and exploring several possible interpretations in a tree-shaped process?
3. How does the method work?
ToC combines a LLM with a search system. This combination is called retrieval-augmented generation.
A simple analogy is a student writing a research answer:
- The student first searches for useful sources.
- Then the student lists the different ways the question could be understood.
- Next, the student checks which interpretations actually match the question.
- Finally, the student writes a complete answer using the reliable information.
ToC follows several main steps.
Step 1: Find useful information
The system searches for passages from Wikipedia using two tools:
- ColBERT, an information-retrieval system that searches for text with meanings similar to the question.
- Bing, a web search engine.
The results are then ranked so that the most useful passages appear first. This gives the LLM information to work with and reduces the chance that it will invent facts, a problem often called a hallucination.
Step 2: Create clearer versions of the question
The LLM uses the retrieved passages to turn the original ambiguous question into several more specific questions.
For example:
| Original question | Possible clearer questions |
|---|---|
| Which country has won the most Olympic medals? | Which country has won the most gold medals? |
| Which country has won the most silver medals? | |
| Which country has won the most bronze medals? | |
| Which country has won the most medals in the Summer Olympics? |
Each clearer question is called a disambiguated question, or DQ.
Step 3: Build a tree of interpretations
The system does not stop after creating the first set of clearer questions. It can examine each one and create more specific questions from it.
This forms a structure like a family tree:
1 2 3 4 5 6 7 8 |
Original ambiguous question ├── Interpretation 1 │ ├── More specific interpretation │ └── More specific interpretation ├── Interpretation 2 │ ├── More specific interpretation │ └── More specific interpretation └── Interpretation 3 |
The system usually explores many branches at the same time. This is called breadth-first search. It is similar to checking all the children in the first level of a family tree before looking at grandchildren.
Step 4: Remove unhelpful branches
Some generated questions may be factually correct but unrelated to the original question.
For example, if the question is:
“Who will host the 2022 World Cup?”
The system might generate:
“Who hosted the 2018 World Cup?”
The answer “Russia” is true, but it does not answer the original question about 2022. ToC uses self-verification, meaning the LLM checks whether each branch still matches the original question. Unhelpful branches are removed, or pruned.
Step 5: Write the final answer
After keeping the useful branches, ToC gives the LLM the selected questions, answers, and supporting passages. The model then writes one long answer that explains the different meanings of the original question.
4. What did the researchers find?
The researchers tested ToC on ASQA, a dataset containing 6,316 ambiguous questions and long answers. They compared ToC with other systems, including:
- LLMs answering without outside sources
- Systems trained on the entire training dataset
- Systems given only a few examples in their prompts
The results showed that adding each part of ToC improved performance.
| System | Disambig-F1 | ROUGE-L | Overall DR |
|---|---|---|---|
| GPT-3 baseline | 25.0 | 31.8 | 28.2 |
| GPT-3 with retrieved information | 31.1 | 39.6 | 35.1 |
| Retrieval plus tree structure | 32.4 | 40.0 | 36.0 |
| Retrieval, tree, and pruning | 33.7 | 39.7 | 36.6 |
These scores measure different qualities:
- Disambig-F1 checks whether the facts in the answer are correct.
- ROUGE-L checks how much the answer resembles a trusted reference answer.
- DR combines the first two measures into one overall score.
The full ToC system achieved the best results. Compared with the previous few-shot system, it improved the factual score by 8.4 points and the overall score by 7.0 points.
It even performed better than some systems trained on the complete training dataset. In particular, it exceeded the best fully supervised system by 7.3 points in Disambig-F1 and 2.9 points in Disambig-ROUGE.
The pruning step was especially useful. Before pruning, the generated clearer questions had an Answer-F1 score of 40.9. After self-verification, the score increased to 59.3. This means the system removed many branches that were wrong or did not really belong.
The researchers also found that combining Bing with ColBERT was better than using either search method alone. Together, they found passages containing answers for more of the possible interpretations.
5. Why are these findings important?
People often ask questions without realizing that their words can have multiple meanings. A computer that gives only one interpretation might provide an incomplete or misleading answer.
ToC is important because it can:
- Recognize several possible meanings automatically
- Use reliable outside information to check facts
- Avoid asking the user extra questions
- Give a complete answer in one response
- Reduce irrelevant or incorrect interpretations
This could be useful in search engines, educational tools, virtual assistants, and research systems. For example, a student asking about “the largest planet” might receive information that explains whether “largest” means by diameter, mass, or another measurement.
However, the system also has limitations. It requires several language-model calls for each question, which can make it slower and more expensive. It was tested mainly on one dataset, ASQA, so it is not yet clear whether it works equally well for every subject or with every LLM. It can also create unnecessary interpretations when a question is already clear.
Conclusion
The paper’s main idea is to treat an ambiguous question like a tree. The original question is the trunk, different possible meanings are the branches, and more specific interpretations grow from those branches. The system searches for evidence, checks each interpretation, removes bad branches, and then writes a complete answer.
Overall, the research shows that LLMs answer unclear questions more accurately when they combine web-based evidence, step-by-step exploration, and self-checking. This approach could help future AI systems give answers that are more complete, factual, and useful.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
- Generalizability beyond ASQA is untested. The framework is evaluated on a single benchmark of 6,316 Wikipedia-derived questions, leaving its performance on conversational, multilingual, domain-specific, non-Wikipedia, and real user-generated questions unresolved.
- Performance on genuinely unambiguous questions is not established. ToC does not explicitly detect ambiguity and may generate irrelevant or redundant interpretations when a question has only one valid reading; the paper does not quantify the frequency or impact of these errors.
- The framework’s ability to identify all valid interpretations remains uncertain. Evaluation primarily measures whether generated answers match annotated disambiguations, but does not directly assess recall of the complete interpretation space or distinguish missing interpretations from legitimately omitted ones.
- The definition of “all possible” interpretations is underspecified. The paper does not provide a principled criterion for deciding which semantic dimensions—such as time, location, entity type, measurement criterion, or event scope—should be expanded or when exploration is complete.
- The fixed tree-search configuration is not systematically justified. The effects of breadth-first search, maximum depth, maximum valid nodes, termination after three failed expansions, and the target of ten retained disambiguations are not evaluated through comprehensive sensitivity analyses.
- Alternative tree-search strategies remain unexplored. Depth-first search, best-first search, beam search, adaptive expansion, Monte Carlo methods, and diversity-aware search are not compared beyond a qualitative claim that depth-first search is suboptimal.
- Self-verification may inherit the generator’s errors. The same or a closely related LLM is prompted to judge whether a generated answer is relevant, but the paper does not establish how reliably this verifier detects subtle scope shifts, unsupported claims, or plausible-sounding factual errors.
- Pruning recall is not measured. The reported Answer-F1 after pruning shows that retained nodes are more accurate on average, but the study does not report how many correct ground-truth interpretations are incorrectly discarded.
- The pruning decision is evaluated with an incomplete proxy. Self-verification uses the answer to a target DQ and a selected passage, rather than directly evaluating whether the DQ itself represents a valid interpretation of the root question; this may fail for questions whose answers are correct but whose interpretation is invalid, or vice versa.
- The interaction between retrieval errors and clarification errors is not isolated. The paper shows aggregate retrieval coverage and final QA scores but does not determine whether failures primarily arise from missing evidence, incorrect reranking, poor DQ generation, erroneous pruning, or final answer synthesis.
- Retrieval coverage remains incomplete. Even the combined retriever covers only 80.1% of disambiguated answers at top-100 on a random sample of 100 examples, leaving unresolved how ToC behaves when evidence for an interpretation is absent from the retrieved corpus.
- The retrieval evaluation is based on a small, potentially nonrepresentative sample. The intrinsic retrieval analysis uses only 100 ASQA examples and does not report confidence intervals or variation across question types.
- Dependence on Wikipedia and Bing is not investigated. The approach may be sensitive to Wikipedia’s coverage, page structure, revision history, and Bing’s availability or ranking behavior; performance with other corpora and retrieval providers is unknown.
- Temporal and changing-fact questions are not addressed. Because retrieval is performed from external sources without an explicit time-awareness or snapshot protocol, the reliability of answers about future events, historical knowledge states, and rapidly changing facts remains unclear.
- Source quality and contradictory evidence are not handled explicitly. The framework does not describe how it resolves disagreement among retrieved passages, evaluates source credibility, or prevents a low-quality passage from grounding an incorrect disambiguation.
- The model and prompting choices are narrowly tested. Generalizability across LLM families, parameter scales, instruction-tuned models, open-weight models, and models with different context-window sizes is left for future work.
- The comparison with baselines may not control all implementation differences. ToC uses retrieval augmentation and multiple model calls, whereas several prompting baselines are evaluated in a closed-book setting; the contribution of retrieval, additional computation, and tree search is therefore not fully disentangled.
- No statistical significance or run-to-run variance is reported. The paper presents single aggregate scores, although API-based LLM generation and retrieval can vary with sampling, model revisions, prompt ordering, and dynamically selected demonstrations.
- The evaluation is reported mainly on the development set. The main table explicitly describes development-set results, and the paper does not provide a corresponding held-out test-set comparison for ToC, limiting confidence in the claimed state-of-the-art performance.
- The reliance on automatic metrics leaves answer quality partially unresolved. ROUGE-L is known to be inadequate for long-form QA, while D-F1 depends on an auxiliary RoBERTa model extracting answers from generated text; factuality, completeness, coherence, and citation correctness are not comprehensively assessed by human evaluators.
- The answer-generation stage is not independently analyzed. It remains unclear how much of the final improvement comes from better DQs and answers versus the final aggregation prompt, passage selection, answer ordering, or the instruction to produce at least three sentences.
- Redundancy and diversity of generated interpretations are not quantified. The paper reports deduplication ablations but does not define or measure semantic diversity, coverage of distinct ambiguity dimensions, or the rate of near-duplicate DQs in the final output.
- The use of retrieved passages may introduce irrelevant or misleading interpretations. The paper demonstrates failure cases qualitatively but does not quantify how often retrieval expands ambiguity beyond what a user would reasonably intend.
- User preference and interaction trade-offs are unexplored. The assumption that a comprehensive answer is preferable to asking a clarification question is not tested with users, and the paper does not examine whether users value exhaustive coverage, brevity, or selective clarification in different contexts.
- Computational and monetary costs are insufficiently characterized. Although the paper reports fewer than 20 LLM calls per question, it does not provide latency, token usage, API cost, energy consumption, or comparisons with interactive clarification and supervised alternatives.
- Scalability to larger ambiguity trees is unknown. The framework’s behavior when a question has many valid interpretations, when retrieved evidence is highly redundant, or when the target number of nodes must be increased has not been evaluated.
- Robustness to adversarial or noisy retrieved content is untested. The paper does not examine prompt injection, contradictory passages, entity-name collisions, malformed documents, or passages designed to induce the model to generate irrelevant disambiguations.
- Potential benchmark and demonstration leakage is not examined. Dynamically selected few-shot examples and externally retrieved text could overlap semantically or textually with evaluation items, but no leakage analysis is reported.
- The method’s applicability to multilingual and culturally dependent ambiguity is unknown. All experiments and prompts appear to be English-centric, leaving unresolved whether the tree structure and self-verification procedure transfer across languages and culture-specific interpretations.
- The proposed improvements to reranking and pruning remain unimplemented. The paper suggests stronger answer-sentence-selection rerankers and alternative pruning methods but does not establish whether these changes improve final performance or alter the cost–quality trade-off.
- The relationship between ambiguity resolution and factual correctness is not fully separated. A system can generate a correct answer to an incorrectly formulated DQ, or a correct interpretation with an incorrect answer; the evaluation does not fully disentangle these two error types.
Practical Applications
Immediate Applications
- Ambiguity-aware search and question answering for consumer information services (software, search, customer support) Deploy ToC as a retrieval-augmented layer over existing search engines, enterprise knowledge bases, or conversational assistants. For a query such as “Who has won the most World Series?”, the system can identify relevant interpretations—such as most titles as a player versus as a coach—and provide a single answer covering both. Potential products/workflows: search-result answer panels, FAQ assistants, help-desk copilots, and chat interfaces that return structured disambiguations alongside cited evidence. Dependencies and assumptions: reliable document retrieval, current indexed sources, sufficient LLM context capacity, and human-designed safeguards against irrelevant or historically outdated interpretations. The paper evaluates only ASQA, so production quality outside this benchmark must be validated.
- Enterprise knowledge-base assistants (business software, legal operations, finance, engineering, healthcare administration) Organizations can apply the framework to internal documents when employee questions contain underspecified terms, dates, entities, or metrics. For example, “What was the revenue last year?” could be expanded into interpretations by fiscal year, geographic region, business unit, or accounting definition before producing a cited summary. Potential workflow: retrieve passages from policies, reports, or manuals; recursively generate candidate interpretations; prune interpretations unsupported by the source documents; produce a long-form response with links to evidence. Dependencies and assumptions: access-controlled retrieval, document quality, domain-specific terminology, privacy protection, and review procedures for high-stakes answers.
- Customer-service and technical-support triage (telecommunications, software, retail, utilities) A support bot can use ToC to interpret vague requests such as “My account is blocked” or “How do I reset the device?” across different products, account states, operating systems, or failure modes. The system can provide a consolidated response or route the user to the appropriate troubleshooting path. Potential tools: ambiguity-aware ticket classification, automated troubleshooting trees, and agent-assistance systems that show alternative interpretations before drafting a response. Dependencies and assumptions: integration with customer records and product documentation; strict controls are needed to prevent the model from inferring sensitive account information or recommending unsafe actions.
- Citation-grounded educational tutoring (education and academia) Educational assistants can use recursive clarification to explain questions that have multiple valid interpretations. A question about “the causes of the revolution,” for example, could be separated by country, period, political versus economic causes, or scholarly perspective. Potential workflow: retrieve approved textbooks and academic sources, generate disambiguated subquestions, and present a comparative answer with source attribution. This can also help instructors create discussion prompts, reading guides, and formative assessments. Dependencies and assumptions: curated educational corpora, age-appropriate language, teacher oversight, and mechanisms for distinguishing genuine scholarly disagreement from model-generated alternatives.
- Research literature discovery and synthesis (academic research, libraries, scientific information services) ToC can help researchers refine broad literature-search questions into multiple dimensions, such as population, intervention, time period, methodology, or outcome measure. A literature assistant could retrieve relevant papers and produce a structured synthesis rather than treating the initial query as having one fixed meaning. Potential products: ambiguity-aware scholarly search, systematic-review query expansion, and research assistants that expose alternative interpretations and supporting passages. Dependencies and assumptions: access to full-text literature, high-quality scholarly retrieval, domain-specific validation, and safeguards against confusing related but non-equivalent research questions.
- Policy and public-information communication (government, public administration, civic technology) Public agencies can apply the method to citizen questions whose answers depend on jurisdiction, eligibility status, date, or program type—for example, “Who qualifies for the housing benefit?” The system could retrieve official regulations and explain the relevant interpretations in plain language. Potential workflow: connect ToC to official policy repositories, require source citations, log the retrieved evidence, and escalate uncertain cases to civil servants. Dependencies and assumptions: authoritative and current sources, jurisdiction detection, legal review, multilingual support, and a rule that generated text is informational rather than a binding administrative decision.
- Personal information and planning assistants (daily life, productivity) General-purpose assistants can use the framework for everyday questions involving hidden dimensions, such as “What is the best route?” or “How much should I save?” The system could distinguish time, cost, accessibility, risk, location, or goal before presenting options. Potential tools: travel planning, budgeting assistants, shopping comparison systems, and calendar or task-management interfaces that show the assumptions behind recommendations. Dependencies and assumptions: accurate real-time data, user preferences, transparent uncertainty, and consent for using personal information. The method should not be treated as a substitute for professional advice in medical, legal, or financial contexts.
- Document-grounded answer quality control (software engineering and AI operations) The self-verification and pruning mechanism can be incorporated into existing retrieval-augmented generation pipelines as a lightweight relevance and scope check. Candidate answers can be tested against the original question and an answer-containing passage before inclusion in the final response. Potential workflow: retrieve evidence, generate several candidate interpretations, run entailment or LLM-based verification, remove scope-shifting candidates, and retain citations and audit logs. Dependencies and assumptions: verification models can themselves make errors; therefore, high-impact deployments require deterministic checks, independent retrieval, confidence thresholds, and human review.
Long-Term Applications
- High-stakes clinical information assistants (healthcare) A mature version of ToC could clarify patient or clinician questions according to condition, age, treatment stage, contraindications, or clinical guideline version. It might generate a structured explanation of how recommendations differ across interpretations. Potential products: evidence-grounded clinical search, patient education systems, and clinician-facing guideline assistants. Dependencies and risks: prospective clinical evaluation, authoritative medical databases, privacy compliance, temporal versioning of guidelines, bias assessment, and mandatory clinician oversight. The current benchmark does not establish medical safety or diagnostic reliability.
- Legal and regulatory interpretation systems (law, compliance, policy) The tree structure could map an ambiguous legal question across jurisdiction, effective date, statutory definition, exception, and type of regulated entity. A system could then summarize relevant provisions and identify where interpretations diverge. Potential workflow: retrieve statutes, regulations, case law, and internal policies; generate candidate readings; prune unsupported interpretations; present sources and unresolved ambiguity to a legal professional. Dependencies and risks: legal-domain retrieval, authoritative source maintenance, citation-level verification, jurisdictional reasoning, privilege and confidentiality controls, and clear limits against autonomous legal advice.
- Financial research and compliance copilots (finance and accounting) Financial users could ask broad questions such as “What was the company’s profit?” and receive separate interpretations for net income, operating profit, adjusted earnings, fiscal period, or consolidated versus regional results. Potential tools: earnings-call analysis, accounting-policy comparison, investment-research assistants, and compliance-question triage. Dependencies and risks: structured financial data, precise metric definitions, market-date alignment, regulatory compliance, and independent validation. Incorrectly combining financial concepts could lead to material decisions or reporting errors.
- Robotics and embodied-agent instruction following (robotics, manufacturing, autonomous systems) ToC’s recursive exploration could help robots handle underspecified commands such as “place the object on the table” by considering object identity, table location, placement orientation, and safety constraints. In human-robot collaboration, the system could either present alternatives or select a plan supported by visual and environmental evidence. Potential products: ambiguity-aware robot task planners, warehouse assistants, and household robotic interfaces. Dependencies and risks: grounding retrieved language in sensor data, real-time latency, reliable world models, physical safety verification, and explicit confirmation before irreversible actions. This application requires research beyond the paper’s text-only QA setting.
- Scientific and engineering multi-hop reasoning systems (research, energy, aerospace, industrial engineering) The ToC tree could be generalized from ambiguous questions to multi-step technical problems, exploring alternative hypotheses, parameterizations, or causal pathways while using retrieved evidence at each step. Potential tools: engineering design assistants, energy-system analysis, experimental planning, and technical incident investigation. Dependencies and assumptions: domain-specific retrieval, symbolic or numerical solvers, calibrated confidence estimates, reproducible reasoning traces, and evaluation against expert-authored problems. The paper explicitly identifies generalization to multi-hop QA as future work rather than demonstrating it.
- Multilingual and cross-cultural public-service assistants (government, education, global commerce) A future system could clarify ambiguity arising from translation, local terminology, cultural references, or different administrative systems. It could produce separate interpretations rather than silently choosing one translation or jurisdiction. Potential products: multilingual government portals, international customer support, translation-aware search, and educational access tools. Dependencies and risks: multilingual retrieval quality, culturally representative data, human evaluation by native speakers, localization of policies, and protection against language-specific hallucinations.
- Interactive clarification systems that combine silent expansion with user feedback (human-computer interaction) The current framework avoids interrupting users, but future systems could dynamically decide whether to answer comprehensively or ask a targeted clarification question. The tree could rank interpretations by evidence, user history, and task cost, presenting only the most useful alternatives. Potential workflow: detect whether a query is ambiguous, estimate the cost of answering all interpretations, ask one concise clarification when necessary, or provide a summarized multi-interpretation answer when interruption is undesirable. Dependencies and assumptions: calibrated ambiguity detection, user-preference modeling, interaction studies, accessibility testing, and policies for handling uncertainty without overwhelming users.
- Scalable, lower-cost deployment through model compression and selective expansion (AI infrastructure) Because ToC can require multiple LLM calls—up to roughly 20 per question in the reported setup—future systems could use small models for retrieval, deduplication, and preliminary pruning, reserving larger models for difficult branches and final synthesis. Caching, parallel breadth-first expansion, and adaptive depth limits could reduce latency and cost. Potential tools: cost-aware RAG orchestration, branch-budget controllers, confidence-based early stopping, and domain-specific rerankers. Dependencies: stronger pruning accuracy, reliable confidence calibration, latency benchmarks, privacy-preserving infrastructure, and evidence that compressed or smaller models preserve factual correctness.
- Auditable decision-support and knowledge-management platforms (industry, academia, policy) The tree itself can become an audit artifact showing which interpretations were considered, which passages supported them, and why some branches were pruned. This would support review, reproducibility, and organizational learning. Potential products: answer provenance dashboards, policy-analysis workbenches, research-synthesis logs, and model-risk monitoring systems. Dependencies and risks: standardized provenance formats, immutable source snapshots, interpretable verification criteria, retention policies, and recognition that an LLM-generated reasoning tree is evidence of the system’s process—not proof that every omitted interpretation was invalid.
Glossary
- Ablation study: An experiment that removes or modifies components of a system to measure their individual contributions. “Table~\ref{table:ablation} displays the ablation study for measuring the contributions of each proposed component.”
- AmbigNQ: A dataset of questions annotated for ambiguity and their possible interpretations. “It is a long-form QA dataset built upon the 6K ambiguous questions identified from AmbigNQ~\cite{min2020ambigqa}.”
- Answer coverage: The proportion of target answers contained in a set of retrieved passages. “Inspired by \citet{min2021joint}, we devise an evaluation proxy, answer coverage, for measuring the quality of retrieved passages in ambiguous QA tasks.”
- Answer-F1: An F1-based metric measuring the accuracy of a generated answer to a disambiguated question. “For validating intermediate nodes, we additionally use Answer-F1 that measures the accuracy of generated short answers in disambiguation.”
- Autoregressive: A generation method in which each output is produced using previously generated outputs. “\citet{min2021joint} propose the tree-decoding algorithm to autoregressively rerank passages in ambiguous QA.”
- BERT: A bidirectional Transformer LLM commonly used for contextual text representations. “ColBERT is a recent dense retriever that has effective and efficient zero-shot search quality.”
- Breadth-first search (BFS): A tree or graph traversal strategy that explores all nodes at one depth before moving to the next depth. “We choose the breadth-first search (BFS) by default, hence the resulting tree could cover the broader interpretations.”
- Chain-of-thought prompting: A prompting technique that encourages a LLM to generate intermediate reasoning steps. “Concurrently, extending chain of thoughts~\cite{wei2022chain} prompting, \citet{yao2023tree} apply the tree architecture to reasoning tasks for deductive or mathematical problems.”
- Closed-book setup: A question-answering setting in which the model answers without retrieving external documents. “In the closed-book setup, GPT-3 shows competitive performances with T5-large with JPR in D-F1 score, showing LLM's strong reasoning ability over its inherent knowledge.”
- ColBERT: A retrieval model that uses contextualized token-level representations and late interaction for efficient passage search. “We first retrieve relevant Wikipedia documents for the AQ by using two retrieval systems, ColBERT~\cite{khattab2020colbert} and Bing search engine.”
- Dense retriever: A retrieval model that represents queries and documents as continuous vectors for semantic similarity search. “ColBERT is a recent dense retriever that has effective and efficient zero-shot search quality.”
- Disambiguation-ROUGE (DR): The geometric mean of ROUGE-L and Disambig-F1, used as an overall performance measure. “Disambiguation-ROUGE (DR) score is computed as the geometric mean of ROUGE-L and Disambig-F1 to measure the overall performance.”
- Disambiguated question (DQ): A reformulation of an ambiguous question that specifies one particular interpretation. “Then, leveraging the passages, DQs for the AQ are recursively generated via few-shot prompting and pruned as necessary.”
- Disambig-F1: An F1 metric measuring the factual correctness of answers generated for disambiguated questions. “Disambig-F1 (D-F1) measures the factual correctness of generated predictions.”
- Document-grounded: Based on information explicitly provided in retrieved or reference documents. “It might indicate the disambiguation process requires external knowledge, which shows the importance of document-grounded or retrieval-augmented systems.”
- Few-shot prompting: Guiding a LLM by including a small number of input-output examples in the prompt. “It recursively constructs a tree of DQs for the AQ---via few-shot prompting leveraging external knowledge---and uses it to generate a long-form answer.”
- Factoid answer: A short answer expressing a specific factual item, such as a name, date, or quantity. “LLM generates all possible DQs and their corresponding answers.”
- Fine-tuning: Further training a pretrained model on task-specific data. “However, their approaches require fine-tuning models on the large-scale train set.”
- Geometric mean: A type of average calculated by multiplying values and taking the corresponding root. “(3) DR score is the geometric mean of two scores, which assesses the overall performance.”
- Hallucination: A language-model output that is unsupported, fabricated, or factually incorrect. “thereby potentially increasing the risk of hallucinations from LLMs.”
- In-context learning: Conditioning a model on examples supplied in the input prompt rather than updating its parameters. “For in-context learning setup, we dynamically choose -shot examples with the nearest neighbor search.”
- Intrinsic evaluation: Evaluation of an intermediate component or property independently of the complete end-to-end task. “We report intrinsic evaluation for each retrieval system in Appendix~\ref{sec:additional-experiment}.”
- Late interaction: A retrieval architecture that computes fine-grained interactions between query and document representations after encoding them separately. “Colbert: Efficient and effective passage search via contextualized late interaction over bert.”
- Lexical overlap: The degree to which two texts share words or other surface-level textual units. “(2) ROUGE-L (R-L) measures the lexical overlap between long-form answers from references and predictions.”
- Long-form question answering: Question answering that produces extended, explanatory responses rather than short answers. “a long-form QA benchmark for AQs.”
- Model-agnostic: Designed to work with different underlying model architectures or model types. “It has a model-agnostic structure that could effectively explore diverse paths of recursive reasoning.”
- Multi-hop QA: Question answering that requires combining information from multiple facts, passages, or reasoning steps. “such as multi-hop QA.”
- Nearest-neighbor search: A method for finding items whose vector representations are most similar to a query representation. “For in-context learning setup, we dynamically choose -shot examples with the nearest neighbor search.”
- Open-domain question answering (ODQA): Question answering in which the system may retrieve evidence from a broad document collection rather than a fixed context. “In open-domain question answering (ODQA), users often ask ambiguous questions (AQs), which can be interpreted in multiple ways.”
- Parametric knowledge: Information encoded in a model’s learned parameters during training. “Moreover, the results could be bounded by inherent parametric knowledge of LLM.”
- Passage reranking: Reordering retrieved passages according to their relevance to a query. “After collecting a passage set for the AQ, we rerank and choose top- passages and augment them to a prompt.”
- Pruning: Removing nodes, questions, or candidates judged unhelpful or irrelevant. “To remove unhelpful nodes, we design a pruning method, inspired by current studies for self-verification~\cite{kadavath2022language, cole2023selectively}.”
- Prompt engineering: Designing or optimizing prompts to influence a LLM’s behavior on a task. “On the other hand, \citet{amplayo2022query} propose a prompt engineering method to adapt LLMs to the ASQA benchmark.”
- Recall-then-verify: A framework that first retrieves potentially relevant candidates and then checks their validity. “Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework.”
- Recursive reasoning: Reasoning that repeatedly applies a procedure to the outputs of earlier reasoning steps. “It has a model-agnostic structure that could effectively explore diverse paths of recursive reasoning.”
- Retrieval-augmented clarification (RAC): The process of using retrieved passages to generate and validate clarifications of an ambiguous question. “We first devise retrieval-augmented clarification (RAC; Sec.~\ref{subsec:question_clarification}), a basic component that clarifies AQ and generates DQs based on relevant passages.”
- Retrieval-augmented generation: A method that supplies retrieved external information to a generative LLM before answer generation. “Experiments demonstrate that our proposed use of LLMs with retrieval-augmentation and guidance to pursue diverse paths of clarification results in the new state-of-the-art on ASQA.”
- ROUGE-L: An evaluation metric based on the longest common subsequence between generated and reference texts. “First, ROUGE-L (R-L) measures the lexical overlap between long-form answers from references and system-generated predictions.”
- Self-consistency: An ensemble-style approach that samples multiple reasoning paths and selects or aggregates their outputs. “Compared to the existing ensemble methods such as self-consistency~\cite{wei2022chain} which cannot be directly adopted to the generative task, ToC achieves a state-of-the-art performance with a comparable number of LLM calls.”
- Self-verification: A model-based procedure for checking whether a generated answer is factually compatible with the original question and evidence. “We perform self-verification by prompting LLMs to determine whether the current node would be pruned or not.”
- SentenceBERT: A BERT-based model that produces semantically meaningful sentence embeddings for similarity comparison. “We use SentenceBERT~\cite{reimers2019sentence} pre-trained on MS-Marco as the reranker backbone.”
- Soft prompt tuning: Adapting a LLM by learning continuous prompt representations rather than changing the model’s main parameters. “PaLM w/ Soft Prompt Tuning”
- State of the art: The highest reported performance achieved by existing methods on a task or benchmark. “advancing the state-of-the-art on the ASQA benchmark.”
- Tree structure (TS): A hierarchical representation in which each node contains a question-answer interpretation and its descendants represent further clarifications. “ToC explores various fine-grained interpretations, represented as a tree structure (TS; Sec.~\ref{subsec:tree_structure}) by recursively performing RAC and pruning unhelpful DQs.”
- Zero-shot search: Retrieval performed without task-specific examples or supervised adaptation for the target task. “ColBERT is a recent dense retriever that has effective and efficient zero-shot search quality.”