Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images

Recent advances in computer vision have shown promising results in using synthetic data to enhance visual recognition models, especially when only a small number of real images are available for training. Synthetic data refers to artificially generated images, typically created to augment real datasets for model training. The conventional approach has been to fine-tune diffusion models on ImageNet and employ the fine-tuned models to generate synthetic images conditioned on ImageNet class labels. However, this strategy comes with challenges.

One of the key issues is the complexity added by the fine-tuning process, which not only requires considerable computational resources but also must be repeated for different datasets, increasing the computational burden significantly. Additionally, when synthetic images begin to outnumber real ones in the training data, the recognition models tend to perform poorly due to the domain gap—the notable differences in data distribution between synthetic and real images.

In light of this, researchers have proposed a new approach that avoids the fine-tuning of generative models. By utilizing pre-trained diffusion models and leveraging advanced techniques, they have developed a scalable framework for generating a vast number of synthetic images to train recognition models.

The framework introduced three innovative techniques:

1. Label Ambiguity Resolution (LAR): LAR addresses the issue of class names with multiple meanings. For example, the word "crane" could refer to a bird or machinery. To avoid semantic misalignment, where synthetic images could represent the wrong concept, LAR uses LLMs to provide context to the class names, ensuring that the generated synthetic images match the intended subject.
2. Diversification Methods: The framework introduces contextualized diversification (CD) and stylized diversification (SD) to enhance the variety of synthetic images. CD combines LLM-derived contextual elements, such as different environments or camera angles, with the class name to form rich and varied prompts. SD, on the other hand, adds distinct visual styles to the images—such as turning a photo into a painting or sketch—through prompts adjusted for style.
3. Domain Adaptation Techniques: To mitigate the risk of overfitting due to domain shifts between real and synthetic images, the framework incorporates separate auxiliary batch normalization layers for recognition models that specifically process synthetic images. It also maintains a balanced number of real and synthetic images in training batches to prevent the model from overfitting to the synthetic domain.

Notably, the approach without generative finetuning has shown significant improvements over previous fine-tuning methodologies. Experiments revealed consistent performance enhancements as the volume of synthetic data scaled up to six times the size of the original ImageNet dataset, in stark contrast to prior work where recognition performance declined. Models using this novel pipeline also demonstrated robust out-of-domain generalization.

In summary, this new method presents a compelling alternative to the traditional fine-tuning pipeline for synthetic image generation. It promises an efficient and effective way to capitalize on synthetic data for large-scale visual recognition training, fostering better model performance and adaptability.