Graphic Design with Large Multimodal Model

Enhancing Graphic Design Automation with Graphist: A Large Multimodal Model Approach

Introduction

The paper presents a novel AI approach to graphic design, introducing the Hierarchical Layout Generation (HLG) task that addresses ordering and spatial arrangement of multimodal input elements without predetermined sequence. This advances the previous Graphic Layout Generation (GLG) methods by allowing unordered element input, fostering creativity and reducing user workload.

Overview of Graphist

Graphist is a pioneering model built upon large multimodal models (LMMs), specifically designed to handle the HLG task. It utilizes a sequence generation framework to process input elements represented as RGB-A images and outputs JSON draft protocols describing the layout. Graphist integrates components like an RGB-A Encoder, Visual Shrinker, and a base LMM derived from established models such as Qwen1.5-0.5B/7B.

Key Contributions

1. Introduction of HLG Task: The hierarchical approach in layout generation bypasses the need for pre-determined layer ordering, broadening the scope for AI in practical graphic design applications.
2. Development of Graphist: As an end-to-end trainable model leveraging LMMs, Graphist efficiently processes multimodal inputs (images and text) and outputs detailed layout specifications in JSON format.
3. New Evaluation Metrics: The introduction of Inverse Order Pair Ratio (IOPR) and GPT-4V Eval helps in quantitatively assessing the performance of the layout generation models, with Graphist showing leading performance across these metrics.

Experimental Results

Graphist strongly outperforms existing methods, including state-of-the-art models like Flex-DM and GPT-4V in both HLG and GLG tasks. It excels particularly in maintaining layer order and enhancing the aesthetic quality of graphic compositions, corroborated by high scores in new evaluation metrics. Additionally, the model shows adaptability across various graphical tasks, as demonstrated in tests on diverse datasets like Crello and CGL-V2.

Future Implications and Developments

The introduction and successful implementation of Graphist suggest significant potential for further exploration into more effective integration of AI in graphic design. This ranges from refining model architecture to improving task-specific performance. Continuous advancements in LMMs could bolster Graphist’s capabilities, potentially democratizing high-quality design creation and expanding practical applications in industry.

Conclusion

This paper marks a substantial advancement in automated graphic design technologies. Graphist, leveraging the power of LLMs, addresses critical challenges in layout generation, demonstrating superior performance and flexibility compared to existing models. The introduction of the HLG task, alongside innovative evaluation metrics, sets a new benchmark in the field, pushing the boundaries of what AI can achieve in creative industries.