MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation

Abstract: LLMs have recently advanced text-driven 3D generation, yet Text-to-CAD remains far from supporting industrial product design. Existing benchmarks focus primarily on generating single-part CAD models and evaluate them using geometric similarity metrics that fail to capture functionality, manufacturability, and assemblability. To address this gap, we introduce MUSE, a Text-to-CAD benchmark focused on complex, editable boundary representation (B-Rep) assemblies. MUSE pairs practical design instances with structured Design Specifications and evaluates generated models through a three-stage protocol: code check, geometric check, and design-intent alignment. The final stage uses design-specific rubrics to assess functionality, manufacturability, and assemblability, moving beyond shape matching toward practical design quality. To enable scalable evaluation, we use a rubric-based visual LLM (VLM) judge and validate its reliability through human annotation. Experiments on closed-source and open-source LLMs reveal a clear failure cascade from executable code to valid geometry and finally to engineering-ready design, with even the strongest models achieving limited success on fine-grained engineering criteria. Together, MUSE provides a realistic benchmark and evaluation framework for advancing Text-to-CAD from geometric generation toward true engineering design. Our project website, including the leaderboard, dataset, and code, is available at https://dong7313.github.io/muse-benchmark/.

• The paper introduces MUSE, a benchmark of 106 expert-curated B-Rep assembly tasks with structured Design Specifications covering component graphs, parameters, materials, and manufacturing processes.
• The paper evaluates CAD generation through a three-stage pipeline—code execution, geometric validity, and rubric-based design assessment—and finds closed-source models average 42.87% execution, 30.71% geometric validity, and 20.30% final design-intent scores.
• The paper shows that multi-component spatial reasoning is the main bottleneck, with overlap-free geometry failing sharply, while judge reliability reaches a Pearson correlation of 0.835 at the design-instance level against human annotations.

MUSE is a Text-to-CAD benchmark that reframes the task from generating geometrically plausible single parts to producing engineering-usable B-Rep assemblies (2605.28579). The authors argue that existing datasets and metrics, which are built on modeling histories (command sequences or CadQuery-style scripts) and evaluated with geometric similarity measures such as Chamfer Distance, cannot determine whether a generated model is a viable design. A chair that differs visually from the reference may still fulfill the seating function, while a visually faithful chair may be unstable, unmanufacturable, or unassemblable. MUSE addresses both gaps—data and evaluation—by pairing practical design instances with structured Design Specifications and a three-stage evaluation protocol that culminates in rubric-based assessment of functionality, manufacturability, and assemblability.

Benchmark design and formalization

Each MUSE instance is defined by a Design Specification $$\mathcal{S} = \langle \mathcal{D}, \mathcal{G}, \Omega, \mathcal{M} \rangle$$, comprising a design description, a Physical Assembly Graph, a valid parameter space, and a manufacturing plan. The Physical Assembly Graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ is an undirected graph whose vertices are physical components (e.g., a seat panel or support leg) and whose edges are physical interfaces. A generated model $C$ is topologically valid when its induced assembly graph $\Phi(C)$ is isomorphic to $\mathcal{G}$; the valid parameter space $\Omega$ then constrains continuous component parameters (width, height, thickness) so that topologically equivalent outputs remain semantically correct—a single assembly graph can yield a chair, bench, or bed depending on parameterization. The manufacturing plan $\mathcal{M}$ grounds $\Omega$ in material properties (e.g., minimum thickness, load-bearing capacity) and process constraints (e.g., 3D printing wall-thickness limits, laser-cutting sheet-thickness caps), drawn from engineering knowledge tables covering CNC milling, laser cutting, 3D printing, molding, and joint types from interlocking to bonding.

This specification format is substantially richer than prior textual inputs: where Text2CAD-style prompts run 80–120 tokens and CADFusion descriptions 20–40 tokens, MUSE specifications exceed 2,500 tokens and are explicitly assemblable.

Dataset construction

The dataset comprises 106 design instances built through a human-in-the-loop pipeline. Professional designers first construct seed models in STEP format and convert their modeling procedures into CadQuery scripts. Claude Opus 4.7 then augments these scripts along stylistic and functional-repurposing axes (e.g., adapting a traditional wooden chair to industrial aesthetics, or evolving a chair into a shoe rack). For evaluation evidence, the pipeline extracts engineering views directly from the geometry kernel—boundaries, contours, and hidden lines rendered as dashed paths—normalized onto a 2×2 grid of top, front, right, and isometric views. The authors argue these SVG-style drawings avoid the perspective distortion, occlusion, and incomplete view coverage that plague rendered images used in prior VLM-based CAD verification. Finally, GPT-5.5 synthesizes structured Design Specifications conditioned on expert-written few-shot examples, with all specifications reviewed and corrected by human experts.

The distribution emphasizes assemblable designs: CNC milling is the largest manufacturing category, timber and PLA dominate materials, and connection methods span one-piece, interlocking, nailing, and snap-fit assemblies.

Three-stage evaluation protocol

Evaluation is funnel-structured: a sample must pass each stage before advancing, receiving zero on all downstream metrics otherwise. Stage 1 executes the generated CadQuery script in a sandbox and requires successful STEP export. Stage 2 applies four binary geometric checks on the exported B-Rep: watertightness, manifold topology, self-intersection freedom, and overlap freedom across distinct solids. Stage 3 converts surviving outputs to engineering views and assesses design-intent alignment with a VLM judge under a design-specific rubric generated per instance from the Design Specification, reference views, reference code, and engineering knowledge tables. The rubric decomposes into six binary sub-criteria: Assembly-ready (inferred component graph must match the target graph), Connectable (joint type, location, direction, and constraint behavior), Well-toleranced (seams and clearances consistent with the specified process precision), Functional (must-have and nice-to-have functions preserved over $\Omega$), Robust (support posture and load-transfer path), and Manufacturable (geometry compatible with material and process constraints). Providing reference views alongside generated views is intended to anchor the judge and reduce hallucinated verdicts.

Experimental results

Fifteen LLMs were evaluated—nine closed-source (GPT-5.5, GPT-4o, Claude Opus 4.7, Claude 3.7 Sonnet, Gemini 3.1 Pro, GLM-5.1/4.7-Flash, MiniMax-M2.7/M2.5) and six open-source. Existing Text-to-CAD baselines were not run: the authors state their task settings are architecturally incompatible with multi-component assemblies under manufacturing constraints, and the only assembly-level baseline, ArtiCAD, lacks public code—a concession that leaves MUSE without a direct comparison point to prior assembly-aware systems.

The central finding is a consistent failure cascade. Averaged over closed-source models, sandbox success is 42.87%, geometric validity 30.71%, and the final design-intent score 20.30% (judged by Gemini 3.1 Pro); open-source averages fall from 22.33% execution to 3.72% final score. Within geometry checking, Overlap Free degrades far more sharply than watertightness or manifoldness—for example, GPT-4o drops from 36.79% execution to 13.21% overlap-free—indicating that multi-component spatial reasoning, not code synthesis, is the primary bottleneck once scripts execute. GPT-5.5 is the strongest model overall, with 77.36% sandbox success and 68.87% geometric validity. Notably, judge identity materially shifts absolute design-intent scores: under GPT-5.5 as judge, Claude Opus 4.7 and Gemini 3.1 Pro reach roughly 53–54% final scores versus roughly 39–43% under Gemini 3.1 Pro judging, so cross-paper comparisons should note this dependence.

On fine-grained criteria, even the best closed-source models achieve only about 19–21% on functionality, manufacturability, and assemblability under Gemini judging (GPT-5.5 reaches 48–54% on individual sub-criteria), while open-source models remain around 3–4%. The authors also show that code capability does not predict CAD quality: qwen-2.5-72b attains the highest open-source execution rate (50%) yet scores only 20.75% geometric validity, whereas llama-3.1-70b converts a lower 31.13% execution rate into 23.58% validity.

Judge reliability

To validate the VLM judge, 20 design instances were annotated by four human annotators across two rounds, yielding 624 sub-criterion judgments. Gemini 3.1 Pro shows the strongest agreement, with Pearson correlation of 0.713 at the sub-criterion level, 0.762 at the criterion level, and 0.835 at the design-instance level (95% bootstrap CIs reported); GPT-5.5 and GPT-4o follow at 0.659 and 0.620 respectively. The authors position these figures against prior LLM-as-judge results: the sub-criterion correlation is comparable to FLASK (0.673–0.732) and exceeds G-Eval's GPT-4 Spearman correlation (0.514), and the instance-level agreement of roughly 0.83 approaches MT-Bench's 85% GPT-4–human agreement. This supports scalable automatic evaluation, though the correlation remains imperfect at the finest granularity.

Limitations and open questions

The authors concede two substantive limitations. First, no benchmark model has been physically manufactured; designs are validated only by professional designers, so real-world manufacturing issues may be underrepresented. Second, the evaluation does not model the physical assembly process: the Connectable criterion checks geometric joint feasibility but not assembly ordering, and the Robust criterion relies on qualitative LLM assessment rather than physics-based quantitative analysis. An additional open question concerns judge sensitivity: the substantial spread in design-intent scores between the Gemini-3.1-Pro and GPT-5.5 judges for identical generations is reported but not analyzed, and the benchmark's 106 instances are small relative to single-part datasets. Comparison with ArtiCAD also remains pending until its implementation is available.

Conclusion

MUSE contributes a benchmark of 106 expert-curated, specification-driven B-Rep assembly design tasks and a three-stage, rubric-based evaluation protocol with a human-validated VLM judge. Its results document a monotone degradation from executable code to valid geometry to engineering-ready design across fifteen LLMs, with the strongest model achieving only limited success on functionality, manufacturability, and assemblability. The benchmark makes explicit that Text-to-CAD has been easier to measure as shape generation than to solve as design generation, and it provides the measurement infrastructure for progress on the latter.