Visual Question Answering Metrics

• Visual Question Answering Metrics are evaluation frameworks that quantify model outputs by comparing predicted answers against human consensus and semantic similarity.
• They employ diverse protocols including exact-match, embedding-based similarity (e.g., Wu-Palmer, MaSSeS), and visual grounding tests to assess performance.
• These metrics drive model improvements by highlighting strengths and limitations in handling ambiguity, robustness to corruptions, calibration, and generative responses.

Visual Question Answering (VQA) metrics define and quantify the performance of models that generate natural language answers in response to visual stimuli and open-ended questions. Given the multimodal and ambiguous nature of VQA, a variety of evaluation protocols have been proposed to address aspects such as answer correctness, semantic plausibility, inter-annotator variability, visual grounding, calibration, robustness, and answerability. These metrics vary significantly depending on the form of the answer (short-form, long-form, open vs. multiple-choice) and evaluation context (in-distribution, robust, trustworthy, or generative settings). This article systematically surveys the design, mathematical foundations, and comparative advantages of established and emerging VQA metrics, emphasizing their roles and limitations in current research.

1. Accuracy, Consensus, and Early Metrics

The canonical VQA metric, established by Antol et al., 2015, is based on consensus-driven, exact-string matching, motivated by brevity of answers (typically one to three words):

$$\mathrm{Accuracy}_{\mathrm{VQA}}(\hat{w}) = \min\left( \frac{\# \text{human answers matching } \hat{w}}{3}, 1 \right)$$

Given ten human responses, the system’s answer receives fractional credit up to 1 if at least three annotators agree, reflecting partial consensus while retaining the computational simplicity of exact matching. This metric is applicable to both open-ended and multiple-choice settings—with the latter using a curated answer set per question. Preprocessing (lowercasing, digit normalization, punctuation removal) standardizes both sets. Strengths include efficiency, interpretability, and robust behavior for factoid and binary questions.

However, the metric systematically disregards synonymy (“sofa” vs. “couch”), semantic paraphrasing, and multi-word or compositional answers. It is insensitive to the distribution of human responses beyond the count for the predicted answer, and can over-credit ambiguous or conflicting consensus clusters (e.g., 6 “yes” vs. 4 “no,” both answers eligible for max credit).

WUPS (Kafle et al., 2016), or Wu–Palmer Similarity, was introduced to soften penalties for near-synonyms by leveraging WordNet taxonomy, but it is restricted to single-token, concept-referential answers and prone to spurious similarities (e.g., “black”/“white”). Multiple-match scoring from DAQUAR uses either average or min-consensus across several ground-truths, but faces similar taxonomic and coverage limitations.

2. Subjectivity, Semantic Fidelity, and MaSSeS

Recognizing these limitations, Jolly et al. introduced MaSSeS—a multi-component evaluation regime integrating majority voting, subjectivity, and semantic similarity (Jolly et al., 2018). The MaSSeS score decomposes as follows:

• Majority (Ma): Relativizes the machine answer’s annotator frequency $f(\hat w)$ against the majority answer’s frequency $f^*$:

$$\mathrm{Ma}(\hat w) = \frac{f(\hat w)}{f^*} \in [0,1]$$

Full credit only for true majority; partial for non-majorities inversely proportional to their popularity.

• Subjectivity (S): Models inter-annotator agreement using the normalized Earth Mover’s Distance (Wasserstein-1) between empirical and uniform answer distributions $\mathbf{p}, \mathbf{u}$:

$$S = 1 - \frac{W_1(\mathbf{p}, \mathbf{u})}{W_{\max}}$$

High $S$ indicates low subjectivity (tight consensus).

• Semantic Similarity (SeS): Clusters semantically similar answers using embedding-based cosine similarity (e.g., with GloVe or FastText), merges clusters exceeding a threshold $t$, and recomputes the subjectivity measure over clustered frequencies.
• Final score:

$$\mathrm{MaSSeS}_t(\hat w) = \mathrm{Ma}(\hat w) \cdot \mathrm{SeS}_t$$

This yields a continuous [0,1] score, sensitive to both surface and semantic answer agreement, and robust to tautological synonyms. MaSSeS is superior to the VQA3+ stepwise metric in discriminative capacity, especially in high-variability datasets (e.g., VizWiz), and adjusts credit when models predict semantically equivalent but non-majority forms.

3. Faithful Visual Grounding and Explanation-Sensitive Metrics

Standard VQA accuracy does not guarantee the model relies on question-relevant image content. Faithful and Plausible Visual Grounding (FPVG) (Reich et al., 2023) is a diagnostic metric to quantify if model answers are supported by the annotated relevant image regions and not by irrelevant content:

Given a dataset with question-relevant regions annotated, three forward passes are performed:

1. All objects/features present: baseline prediction.
2. Only relevant objects: sufficiency (answer should remain).
3. Only irrelevant objects: comprehensiveness (answer should flip).

$$\mathrm{FPVG}_j = \mathrm{Eq}(a_{j,all}, a_{j,rel}) \land \neg \mathrm{Eq}(a_{j,all}, a_{j,irrel})$$

The global FPVG+ rate is the fraction of questions where both properties are satisfied. Models with high FPVG avoid spurious image-question shortcuts, enforce visual grounding, and show improved robustness. Empirical results on GQA show that even SOTA models achieve only 20–36% FPVG rates, revealing a persistent grounding deficit.

4. Robustness, Calibration, and Uncertainty-Aware Metrics

Recent work foregrounds the need to evaluate VQA models under realistic corruptions and across selective-answering scenarios.

• Robustness Under Visual Corruptions: (Ishmam et al., 2024) defines the Visual Robustness Error (VRE), aggregating five error-based sub-metrics over corruption types (blur, noise, etc.) and severities ($f(\hat w)$0 clean, $f(\hat w)$1 increasing). These include:
  • First-Drop: Immediate sensitivity to mild corruption.
  • Range: Fold-increase in error up to severe corruption.
  • Slope: Linear rate of error increase.
  • Average Error ($f(\hat w)$2): Mean error at all severities.
  • Delta ($f(\hat w)$3): Average error relative to the clean baseline.

Normalization and model- or corruption-averaging allows for fair comparison across broad operating contexts. VRE unifies these into a single composite score, capturing trade-offs between clean performance and robustness to distribution shift.

• Calibration and Selective Answering: (Eisenschlos et al., 2024) introduces metrics for confidence calibration, crucial in safety-critical and assistive settings. Key quantities include:
  • Coverage@Acc: Maximal fraction of queries answered at or above a target accuracy when thresholded on model confidence.
  • Expected Calibration Error (ECE): Discrepancy between predicted confidence and observed correctness.
  • Avg BLEU Calibration Score: Average pairwise BLEU agreement among model output samples, capturing both mode likelihood and semantic spread.

Sampling-based calibration outperforms likelihood alone for text-only models; multimodal grounding further improves reliability. Selective answering via calibrated thresholds can maintain high accuracy on “triggered” answers.

5. Beyond Short Answers: Generative, Long-Form, and LLM-Based Metrics

VQA contexts with long-form or generative answers render traditional accuracy metrics unusable (Chen et al., 2023).

• NLG-Inspired Metrics: For multi-sentence or paragraph VQA, reference-based text metrics are adopted:
  • ROUGE-L: Longest common subsequence relevance.
  • METEOR: Unigram match with synonym/stem normalization.
  • BERTScore: Contextual embedding similarity.
  • CLIP-S / RefCLIP-S: Image-text grounding and hybrid reference similarity.

Manual human evaluation or LLM-based correctness prompts (e.g., LLaMA2 scoring: 0/0.5/1) are used to capture nuanced semantic acceptance, with METEOR and BERTScore achieving the highest correlation with human ratings.

• LLM-Assisted Evaluation: The LAVE metric (Mañas et al., 2023) embeds VQA answer scoring in a structured LLM prompt, providing reference answers and a candidate, then asking the LLM for a human-like rationale and correctness rating (1–3). LAVE outperforms all previous automatic metrics in alignment with human judgment (ρ≈0.65–0.69 vs. ~0.6 for VQA-Acc). LAVE’s strengths include synonym handling, robustness to answer verbosity, paraphrasing, and coverage of ambiguous or OOD settings.

6. Holistic and Task-Integrated VQA Metrics

Task-integrated VQA metrics deploy multi-criteria frameworks to benchmark and diagnose models beyond raw accuracy (Väth et al., 2021):

• Bias Metrics: Quantify modality dependence—image-bias and question-bias—by measuring the stability of model predictions when one modality is randomly altered.
• Robustness to Adversarial Textual Noise: SEAR (Sensitivity to Edits and Rephrases) rules generate adversarial question variants and measure answer invariance.
• Noise Perturbation Robustness: Models are perturbed in image, feature, and question-embedding spaces to test for prediction stability.
• Uncertainty: Monte Carlo dropout at inference yields predictive entropy quantifying model uncertainty.

Tabular results over representative models show tightly specialized models succeeding only in their respective training domains, with robust multi-modal fusion architectures less prone to trivial biases and attaining lower uncertainty.

7. Application of VQA Metrics in Other Multimodal Tasks

VQA-based metrics have been exported to other domains such as text-to-image generation (Miyamoto et al., 2024). Here, alignment assessment frameworks operate by:

• Generating binary (yes/no) questions from the input prompt via a LLM (e.g., ChatGPT).
• Deploying a VQA model (e.g., BEIT-3) to answer these questions over generated images.
• Aggregating alignment as the proportion of correct answers, optionally weighted with non-reference image quality assessment (NR-IQA) scores.

This composite approach allows for fine-grained, object- and attribute-level fidelity auditing, with tunable emphasis (α) on alignment vs. perceptual quality.

---

In sum, the landscape of VQA evaluation metrics encompasses exact-match and consensus protocols, embedding-based and semantic similarity measures, human and LLM-assisted judgment, calibration and robustness diagnostics, and visual grounding scores. There is increasing recognition that no single metric suffices: rigorous VQA assessment integrates multiple axes—correctness, alignment, robustness, faithfulness, and calibration—tailored to answer-type and deployment scenario (Agrawal et al., 2015, Jolly et al., 2018, Ishmam et al., 2024, Mañas et al., 2023, Eisenschlos et al., 2024, Reich et al., 2023, Väth et al., 2021, Chen et al., 2023, Miyamoto et al., 2024, Kafle et al., 2016).