Cross-Domain Few-Shot Learning by Representation Fusion (2010.06498v2)

Abstract: In order to quickly adapt to new data, few-shot learning aims at learning from few examples, often by using already acquired knowledge. The new data often differs from the previously seen data due to a domain shift, that is, a change of the input-target distribution. While several methods perform well on small domain shifts like new target classes with similar inputs, larger domain shifts are still challenging. Large domain shifts may result in high-level concepts that are not shared between the original and the new domain, whereas low-level concepts like edges in images might still be shared and useful. For cross-domain few-shot learning, we suggest representation fusion to unify different abstraction levels of a deep neural network into one representation. We propose Cross-domain Hebbian Ensemble Few-shot learning (CHEF), which achieves representation fusion by an ensemble of Hebbian learners acting on different layers of a deep neural network. Ablation studies show that representation fusion is a decisive factor to boost cross-domain few-shot learning. On the few-shot datasets miniImagenet and tieredImagenet with small domain shifts, CHEF is competitive with state-of-the-art methods. On cross-domain few-shot benchmark challenges with larger domain shifts, CHEF establishes novel state-of-the-art results in all categories. We further apply CHEF on a real-world cross-domain application in drug discovery. We consider a domain shift from bioactive molecules to environmental chemicals and drugs with twelve associated toxicity prediction tasks. On these tasks, that are highly relevant for computational drug discovery, CHEF significantly outperforms all its competitors. Github: https://github.com/ml-jku/chef

• The paper introduces CHEF, a method that fuses multi-layer representations to significantly improve few-shot learning across diverse domains.
• The methodology leverages Hebbian ensemble learning to efficiently adapt network parameters without full-scale backpropagation.
• Empirical results on datasets such as miniImagenet, CropDisease, and ChestX demonstrate CHEF's superior performance over traditional models.

Cross-Domain Few-Shot Learning by Representation Fusion

The paper presents a methodological advancement in the field of few-shot learning, where the challenge lies in adapting models trained on one data domain to perform well on a significantly different domain, often referred to as a domain shift. This challenge is particularly pronounced when the distribution of the input-target pair is altered between the source and target domains, necessitating the development of strategies like the one proposed in this paper—Cross-domain Hebbian Ensemble Few-shot learning (CHEF).

Key Contributions

CHEF introduces a novel approach termed representation fusion, which aims to unify and merge information from various abstraction layers of a deep neural network. This approach is crucial in dealing with cross-domain few-shot learning scenarios where large domain shifts are present. The CHEF algorithm employs ensemble learning with Hebbian learners that operate across different layers of the neural network, which are typically trained on rich datasets, to infer on new domains with limited data.

The paper demonstrates that representation fusion significantly boosts performance in few-shot learning tasks, outperforming state-of-the-art methods, particularly in settings with large domain shifts. This was validated with empirical results from miniImagenet and tieredImagenet datasets with small domain shifts, and more notably, with larger shifts in data domains such as CropDisease, EuroSAT, ISIC, and ChestX datasets.

Methodology

The CHEF framework consists of several novel elements:

• Representation Fusion: This is the linchpin of the method, leveraging multiple levels of abstract data representation to infer more robust predictions in new domains.
• Hebbian Learning in Ensemble: By using Hebbian learning rules, the model adapts the parameters of individual learners without requiring backpropagation throughout the entire network, thus gaining computational efficiency and flexibility.
• Layer Ensemble Strategy: Different layers of the neural network contribute through their individual learners, and their outputs are aggregated to form a final prediction, enabling adaptability across varying domain shifts.

Experimental Results

CHEF showcased state-of-the-art performance across multiple few-shot learning benchmarks. Particularly, it established new benchmarks in cross-domain tasks, demonstrating remarkable adaptability and efficacy in real-world applications such as drug discovery, where the prediction of molecular properties and toxicities in new chemical spaces was significantly enhanced.

The experimentation highlighted that the ensemble approach not only improved overall accuracy but was also robust across a diverse range of configurations and datasets. Importantly, it outperformed traditional methods such as SVM and RF in contexts with limited training data, underscoring its potential for practical applications where data is scarce.

Implications and Future Directions

The findings imply that the principle of representation fusion may be a promising direction for future developments in machine learning, particularly in tackling the challenges associated with transfer learning and domain adaptation. The success of CHEF could pave the way for more adaptive AI systems that can seamlessly transition between tasks with minimal retraining, potentially revolutionizing fields that rely heavily on rapid model deployment and adaptation, such as healthcare, autonomous driving, and environmental monitoring.

Future research could explore enhancing the CHEF methodology through integration with other learning paradigms such as self-supervised learning or incorporating more advanced ensemble algorithms that could further refine the fusion of representations. The adaptability and efficiency of Hebbian learners within this framework also present a fertile ground for optimizing neural network architectures for better performance in dynamic and continually changing environments.