AWBench: Benchmarking Suites Overview

• AWBench is a collection of diverse benchmarking suites designed to evaluate automated weak supervision, plasma simulations, and character animation with tailored experimental protocols.
• Each suite employs rigorous metrics—ranging from statistical performance profiles to video quality scores—to ensure reproducible and reliable results across challenging tasks.
• Its modular design and comprehensive datasets drive innovations in low-annotation regimes, enabling fair comparisons and methodological improvements in multiple technical domains.

AWBench refers to multiple distinct benchmarking suites in contemporary research, each serving to rigorously evaluate technical methods in unique scientific and engineering domains. Notably, AWBench is (1) a comprehensive benchmark for automated weak supervision algorithms ("AutoWS-Bench-101") (Roberts et al., 2022), (2) a rigorous benchmarking suite for plasma simulation in the context of the AWAKE experiment (Lotov, 2017), and (3) an evaluation suite for generality and quality in character image animation ("Animate in the Wild Benchmark") (Luo et al., 29 Jan 2026). Each incarnation of AWBench is tailored to the specific challenges and methodological nuances of its respective field.

1. Objectives and Motivations

AWBench as a concept embodies the need for reliable, extensible, and representative benchmarks in domains where evaluation is fundamentally challenging due to the absence of plentiful ground truth annotations, the complexity of physics simulation, or the diversity of generated content.

In automated weak supervision, AWBench ("AutoWS-Bench-101") addresses the central question: given only a limited ground-truth label budget (specifically, 100 labels per task), should practitioners use automated weak supervision methods to synthesize additional labels, or should they rely instead on few-shot or zero-shot methods powered by modern foundation models? The suite supplies a unified experimental pipeline (via WRENCH) that supports systematic comparison of these approaches over a heterogeneous set of challenging tasks (Roberts et al., 2022).

In the context of high-fidelity plasma simulation for the AWAKE experiment, AWBench isolates two cornerstone tests: one probing the long-term integrity of kinetic plasma solvers in the linear regime, and another benchmarking self-modulation phenomena in a nonlinear, multi-scale regime (Lotov, 2017). The intent is to enable robust cross-code validation on physically significant scenarios with analytically tractable properties.

Within character image animation, AWBench ("Animate in the Wild Benchmark") was introduced to overcome the limited diversity of prior benchmarks—most of which cover only human or narrowly anthropomorphic domains by curating a spectrum of subjects, motion granularities, and cross-domain (e.g., human→cartoon) pairings—thus facilitating the development and fair evaluation of universal, highly generalizable animation models (Luo et al., 29 Jan 2026).

2. Benchmark Suite Design and Composition

AutoWS-Bench-101 ("AWBench" for Automated Weak Supervision)

• Task Diversity: Encompasses 10 datasets from three domains: image (MNIST, CIFAR-10, Spherical MNIST, Permuted MNIST), "diverse" (ECG arrhythmia, Navier-Stokes turbulence, EMBER malware, all tabular/time series), and text (YouTube spam, Yelp, IMDb).
• Data Structure: Each task provides an unlabeled pool ($n$ examples) and a validation pool with $m=100$ labeled examples, distributed class-stratified when possible.
• Feature Extractors: Raw features, PCA (top 100 components), ResNet-18 (ImageNet), and CLIP zero-shot logits for images; BERT embeddings for text.
• Pipeline: Supports modular combination of feature extractors $\varphi$, LF synthesis algorithms ($A_{\text{WS}}^\varphi$), and label-model aggregators (LM).

AWAKE Simulation Benchmark ("AWBench" in Plasma Physics)

• Test 1: Small-Amplitude Plasma Wave
  • Geometry: 2D axisymmetric, comoving frame $\xi=z-ct$, cylindrical coordinates.
  • Setup: Uniform electron plasma with immobile ions; short, low-amplitude, analytically parameterized proton driver.
  • Diagnostics: On-axis field $E_z$, period elongation, amplitude drift, wave noise induced by macro-particle representation.
• Test 2: Seeded Self-Modulation
  • Geometry: Same as Test 1; plasma column of $L=10$ m.
  • Setup: Hard-cut, long proton bunch with controlled energy/angle spread; test for self-modulation exponential growth and phase evolution.
  • Diagnostics: Max accelerating field $E_{z,\max}(z)$, local field maxima $E_{z,\text{loc}}(z,\xi)$, direct comparison to theoretical amplitudes.

Animate in the Wild Benchmark ("AWBench" for Character Animation)

• Subject Diversity: Three core classes—humans, animals (multiple species), and pure cartoons (e.g., "Tom & Jerry").
• Motion Granularities: Encompasses face-only, upper-body, full-body, multi-subject interactions.
• Dataset Scale: $N_{\text{drive}}=100$ driving videos, $N_{\text{ref}}=200$ reference images, $\bar{T}\approx85$ frames per driving video. The standard pairwise evaluation uses $N_{\text{pairs}}=100$ primarily human$\to$human and human$\to$cartoon transfers.
• Annotations: Metadata for subject type and motion; no ground-truth frames for cross-identity pairs, necessitating protocol-specific evaluation.

3. Evaluated Methods and Baselines

AutoWS-Bench-101

• LF Synthesis Algorithms:
  • Snuba: Weak classifier search on feature subsets, supports unipolar and multipolar labeling functions.
  • Interactive Weak Supervision (IWS): Greedy LF addition above a validation precision threshold ($\alpha$), with optional human-in-the-loop curation.
  • GOGGLES: Affinity coding using cosine metrics, hierarchical (generative) clustering, GMM-based cluster$\to$class assignment.
• Foundation Model Integration: CLIP logits serve as extra LFs or as primary features.
• Baselines:
  • Few-shot supervised logistic regression ($n=100$ labels).
  • Semi-supervised label propagation over the $k$-NN feature graph.
  • Zero-shot CLIP prediction using class prompt embeddings.

AWAKE Simulation Codes

• Primary Codebase: LCODE, a quasi-static, kinetic, 2D axisymmetric code.
• Diagnostics: High-resolution (down to $dr=d\xi=0.0025\, c/\omega_p$), up to tens of CPU-hours per run for fine-tuned benchmarking.

Character Animation

• Compared Methods:
  • Animate-X++, MTVCrafter, DreamActor-M1, Wan2.2-Animate, own variants of DreamActor-M2.
• Evaluation: All models are evaluated zero-shot (train on external data only), never with AWBench samples in the training set.

4. Evaluation Protocols and Metrics

AutoWS-Bench-101

• Performance Metrics:
  • Accuracy, macro-F1, and method coverage.
  • Performance profiles (fraction of tasks within $\tau$-factor of best).
  • Statistical reporting over random seeds; signed coverage-accuracy tradeoff analysis.

AWAKE Simulation

• Key Quantities:
  • Wakefield amplitude drift ($<0.1\%$ over $3000\,\omega_p^{-1}$ for high-res runs).
  • Estimated noise for macro-particle drivers: $E_{az}\sim 0.1 E_0/\sqrt{N}$.
  • Matching to analytic theory for amplitude and nonlinear period elongation.

Animate in the Wild Benchmark

• Automatic Metrics (Video-Bench protocol [Han et al., CVPR 2025]):
  • Imaging Quality (IQ), Motion Smoothness (MS), Temporal Consistency (TC), Appearance Consistency (AC); each rated $[1,5]$ by neural evaluators.
• Human Studies:
  • Parallel 12-person assessments on the same axes.

Summary Table of AWBench Video Animation Metrics

Method	IQ (Auto)	MS (Auto)	TC (Auto)	AC (Auto)	IQ (Human)	MC (Human)	AC (Human)
Animate-X++	3.45	3.42	4.15	3.21	3.18±0.23	2.95±0.29	2.86±0.34
MTVCrafter	3.71	3.81	4.02	3.53	3.35±0.26	3.26±0.28	3.07±0.36
DreamActor-M1	4.17	3.92	4.21	4.06	3.96±0.21	3.72±0.26	3.54±0.31
Wan2.2-Animate	4.05	4.06	4.17	3.92	3.91±0.20	3.83±0.25	3.51±0.30
Pose-based DreamActor-M2	4.68	4.53	4.61	4.28	4.23±0.19	4.18±0.24	4.12±0.28
End-to-End DreamActor-M2	4.72	4.56	4.69	4.35	4.27±0.18	4.24±0.23	4.20±0.29

5. Empirical Findings and Comparative Analyses

AutoWS-Bench-101

• CLIP-based features significantly improve performance of AutoWS (and zero-shot) methods for in-distribution images but lead to reduced coverage or underperformance out-of-distribution.
• Few-shot logistic regression is consistently robust, outperforming AutoWS baselines in most high-dimensional image and text benchmarks.
• Multipolar Snuba is favored for multiclass, low-class-number settings; unipolar offers better coverage in larger label spaces.
• Human-in-the-loop curation in IWS can, on some datasets, increase final accuracy without sacrificing coverage, but not universally.

AWAKE Simulation

• High-resolution runs numerically converge to analytic predictions for wakefield amplitude and nonlinear period adjustments at high accuracy. Lower resolution settings trade off computation for reproducibility within practical tolerances.
• Test 2 demonstrates exponential growth in seeded self-modulation as predicted by analytical theory, confirming the capacity of kinetic codes to accurately reproduce nonlinear beam-plasma effects over physically relevant scales.

Character Animation

• AWBench supports side-by-side comparison of methods across diverse subject and motion domains. Both pose-based and end-to-end DreamActor-M2 variants achieve maximal scores across neural and human evaluators, indicating strong generalization over diverse subject and motion types (Luo et al., 29 Jan 2026).

6. Ablations, Design Trade-offs, and Key Lessons

• In AutoWS-Bench-101, increasing LF cardinality offers diminishing marginal gains beyond moderate settings ($D\approx4$). Expanding seed label budgets above 100 does not improve performance for some methods, indicating algorithmic constraints as the primary limitation.
• Foundation model (CLIP) integration produces a trade-off: per-example accuracy rises on familiar (in-distribution) classes, but coverage collapses in out-of-distribution domains, necessitating careful selection and profiling of components in practical pipelines.
• In character animation, the absence of ground-truth frames for cross-identity cases enforces reliance on perceptual and human-aligned video quality metrics, highlighting the unique challenges of generalizing animation beyond anthropomorphic subjects.

7. Significance and Public Impact

AWBench, across its distinct manifestations, provides benchmark suites crucial for reproducible, scalable, and representative evaluation in: (a) automated weak supervision, (b) full-physics simulation of beam-plasma systems in accelerator physics, and (c) universal character animation. In all cases, AWBench frameworks have established new standards for evaluating algorithmic performance under realistic, cross-domain, and low-annotation regimes, guiding both methodological innovation and fair comparison (Roberts et al., 2022, Lotov, 2017, Luo et al., 29 Jan 2026).