InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (2402.04717v1)

Abstract: Comprehending natural language instructions is a charming property for 3D indoor scene synthesis systems. Existing methods directly model object joint distributions and express object relations implicitly within a scene, thereby hindering the controllability of generation. We introduce InstructScene, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 3D scene synthesis. The proposed semantic graph prior jointly learns scene appearances and layout distributions, exhibiting versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 3D scene synthesis, we curate a high-quality dataset of scene-instruction pairs with large language and multimodal models. Extensive experimental results reveal that the proposed method surpasses existing state-of-the-art approaches by a large margin. Thorough ablation studies confirm the efficacy of crucial design components. Project page: https://chenguolin.github.io/projects/InstructScene.

• The paper introduces a novel framework that uses a semantic graph prior to decouple scene layout from object semantics.
• It leverages a layout decoder with conditional diffusion models to generate realistic 3D scenes with enhanced instruction recall and fidelity.
• The approach excels in zero-shot tasks like stylization and completion, advancing applications in interior design and virtual reality.

Summary of "InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior"

The paper introduces InstructScene, a novel generative framework aimed at improving the controllability and fidelity of 3D indoor scene synthesis using natural language instructions. This work addresses the challenge of synthesizing realistic 3D scenes influenced by abstract object relationships, which has traditionally posed difficulties for existing methods that implicitly model relationships through object distributions within scenes.

Methodology

InstructScene integrates two core components:

1. Semantic Graph Prior: A model that learns high-level object appearances and layout distributions conditioned on user instructions. This component decouples scene layout attributes from object semantics using a discrete graph-based representation, effectively learning latent structures that facilitate intricate scene synthesis tasks.
2. Layout Decoder: Converts semantic graphs into concrete layout configurations, embodying objects with respect to specified positions and orientations. This decoder relies on conditional diffusion models, which separate discrete and continuous scene attributes for efficient network optimization.

The notable element of this framework is its reliance on semantic graphs as priors, transforming scene attributes into a structured latent space that simplifies complex object relationships into interpretable and controllable elements. The semantic graph encompasses categorical attributes like object class, spatial relations, and quantized feature indices derived from multimodal-aligned models, ensuring style consistency and thematic coherence.

Dataset and Experimental Results

To benchmark text-driven 3D scene synthesis, the authors curated a dataset of scene-instruction pairs. This involved generating descriptive captions and spatial relations using automated tools refined by LLMs. Through extensive experiments on this dataset, InstructScene demonstrated notable improvements in generation controllability, as evident by increased instruction recall (iRecall) rates, alongside competitive fidelity measured by Fréchet Inception Distance (FID).

Zero-shot Applications

The proposed framework also excels in zero-shot tasks like stylization, re-arrangement, completion, and unconditional scene generation. Thanks to the discrete and mask-based design of the graph prior, InstructScene can effectively adapt to these tasks by treating unknown scene attributes as intermediate states, leveraging learned knowledge for diverse generative applications.

Theoretical and Practical Implications

Theoretically, InstructScene presents a significant shift towards explicit relational modeling in scene synthesis, emphasizing the utility of semantic graphs for disentangled and conditional generation. Practically, this approach is poised to advance applications such as interactive interior design, 3D content creation for immersive environments, and virtual reality, where detailed control over object placements and styles is paramount.

Future Prospects

Future research directions suggested include scaling the framework to encompass larger and more diverse datasets, potentially incorporating fully generative models for individual objects, and exploring the integration of LLMs to enhance instruction comprehension and object-generation synergy further.

In conclusion, InstructScene represents a comprehensive framework advancing the field of 3D scene synthesis by seamlessly blending semantic understanding with object-level control, paving the way for new possibilities in AI-driven scene generation.