SketchyScene: Richly-Annotated Scene Sketches (1808.02473v1)

Abstract: We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of advancing research on sketch understanding at both the object and scene level. The dataset is created through a novel and carefully designed crowdsourcing pipeline, enabling users to efficiently generate large quantities of realistic and diverse scene sketches. SketchyScene contains more than 29,000 scene-level sketches, 7,000+ pairs of scene templates and photos, and 11,000+ object sketches. All objects in the scene sketches have ground-truth semantic and instance masks. The dataset is also highly scalable and extensible, easily allowing augmenting and/or changing scene composition. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches and showing how the new dataset enables several applications including image retrieval, sketch colorization, editing, and captioning, etc. The dataset and code can be found at https://github.com/SketchyScene/SketchyScene.

• The paper introduces SketchyScene, the first large-scale scene sketch dataset with over 29,000 sketches and detailed annotations for object and scene understanding.
• It employs a novel crowdsourcing pipeline that leverages reference photos to ensure high-quality, diverse sketch synthesis.
• Applications demonstrated include sketch-based image retrieval, colorization, and dynamic scene synthesis that advance both theoretical and practical AI research.

An Expert Overview of the SketchyScene Dataset for Scene Sketch Understanding

The development and contribution of the SketchyScene dataset marks a significant advancement in the domain of computer vision, specifically targeting sketch understanding at a scene level. The SketchyScene dataset is the first of its kind, large-scale collection dedicated to scene sketches—aiming to propel research in understanding sketches at both the object and scene levels.

Dataset Construction and Statistics

SketchyScene was constructed using a novel crowdsourcing pipeline that ingeniously balances efficiency with the fidelity of scene sketches. The dataset houses over 29,000 scene-level sketches, 7,000+ scene templates paired with reference photos, and 11,000+ object sketches. Each object within these scenes has been meticulously annotated with ground-truth semantic and instance masks. This detailed level of annotation ensures that the dataset holds great utility for developing models capable of nuanced sketch understanding.

The innovative crowdsourcing approach employed significantly enhances the quality and diversity of the data. Workers engaged in sketch scene synthesis were aided by reference images, which bolstered the authenticity and variability of the sketches. The object-oriented synthesis mechanism employed makes SketchyScene not only rich in its current form but also extensible, allowing for dynamic augmentation and expansion by changing scene composition or object sketches.

Applications and Practical Uses

The utility of the SketchyScene dataset extends beyond semantic segmentation. The authors demonstrate its robustness through various applications including sketch-based scene image retrieval, sketch colorization, editing, and captioning. For instance, in the domain of retrieval systems, the dataset is used to develop a scene-level Sketch-Based Image Retrieval (SBIR) application that complements conventional methods. This integration exemplifies the potential for sketches to facilitate advanced image processing tasks.

In sketch colorization, the dataset's demonstration allows for semantic-informed color assignment, showcasing a practical utilization in real-world applications such as children’s educational tools. The architectural design of SketchyScene also supports dynamic scene synthesis—a compelling application for creating animated sequences from static sketches.

Comparisons and Challenges in Sketch Segmentation

The paper methodically evaluates the performance of multiple baseline models, such as FCN-8s, SegNet, DeepLab-v2, and DeepLab-v3, on the SketchyScene dataset. Notably, DeepLab-v2 and -v3 emerged superior, though the challenges aligned with sketch-specific constraints, such as sparse visual cues and occlusions, remain significant. The paper suggests further research in segment-specific model design, possibly integrating perceptual grouping principles to ameliorate the identified limitations.

Theoretical and Practical Implications

The SketchyScene dataset offers insightful implications for both theoretical exploration and practical advancements in AI. The comprehensive annotations provided lend significant potential to developing more sophisticated models capable of understanding and interpreting intricate scene sketches. As scene sketch understanding matures, it could fundamentally alter how human-computer interaction is conceptualized, offering more intuitive platforms for artistic and educational applications.

Speculation on Future Developments

Future directions could include enhancing the dataset with additional annotations, such as text captions and scene-level descriptions, paving the way for applications like text-driven scene sketch generation. Moreover, integrating machine learning models capable of handling the complex dynamics of sketches could evolve sketch understanding beyond static images to include temporal progression and interactivity.

In conclusion, the SketchyScene dataset represents a valuable asset in the sketch understanding domain, providing the tools needed to explore new frontiers in artificial intelligence. By leveraging such comprehensive datasets, researchers can explore the subtleties of visual understanding, ultimately bridging the gap between human creativity and machine interpretation.