Learning Discrete Structures for Graph Neural Networks (1903.11960v4)

Abstract: Graph neural networks (GNNs) are a popular class of machine learning models whose major advantage is their ability to incorporate a sparse and discrete dependency structure between data points. Unfortunately, GNNs can only be used when such a graph-structure is available. In practice, however, real-world graphs are often noisy and incomplete or might not be available at all. With this work, we propose to jointly learn the graph structure and the parameters of graph convolutional networks (GCNs) by approximately solving a bilevel program that learns a discrete probability distribution on the edges of the graph. This allows one to apply GCNs not only in scenarios where the given graph is incomplete or corrupted but also in those where a graph is not available. We conduct a series of experiments that analyze the behavior of the proposed method and demonstrate that it outperforms related methods by a significant margin.

• The paper presents a bilevel optimization approach that jointly learns discrete edge distributions and GCN parameters in a semi-supervised setting.
• The paper demonstrates that the LDS model outperforms traditional GCNs on datasets with noisy, missing, or incomplete graph structures.
• The paper integrates probabilistic edge modeling with gradient-based hyperparameter optimization, extending the applicability of GNNs in challenging scenarios.

Learning Discrete Structures for Graph Neural Networks

Graph Neural Networks (GNNs) are a prominent category of machine learning models that leverage the structural relationships between data points, represented as graphs. A common challenge in applying GNNs arises when the required graph structure is either missing or unreliable due to noise or incomplete data. Addressing this issue, the paper "Learning Discrete Structures for Graph Neural Networks" proposes a method to simultaneously learn the graph structure and the parameters of Graph Convolutional Networks (GCNs), a particular class of GNNs, by solving a bilevel optimization problem.

The authors propose a bilevel learning framework that treats the existence of edges within a graph as random variables with discrete distributions, specifically Bernoulli distributions. The approach involves optimizing the parameters of these distributions alongside the parameters of the GCN in a semi-supervised learning setting. The outer problem focuses on minimizing the validation error over the edge distributions, while the inner problem targets minimizing the training error on the GCN parameters given a sampled graph structure.

Numerical Results and Implications

The experimental results of the proposed method, termed as Learning Discrete Structures (LDS), display superior accuracy compared to traditional GCN models, especially in scenarios with incomplete graphs. The paper presents results showing that LDS not only enhances accuracy on real-world datasets with noise and missing edges but also surpasses baseline models when no graph data is initially available, constructing effective graphs from scratch based on input features.

From a practical perspective, the implications of LDS are significant. It enables the utilization of GNN techniques in domains where graph data might be scarce or entirely unavailable, thus broadening the applicability of graph-based learning models. Furthermore, the ability of LDS to outperform even when the entire original dataset graph is available suggests it learns additional meaningful relational data not present in the initial structure.

Theoretical Contributions

The theoretical contribution lies in the novel application of bilevel optimization to jointly learn discrete graph structures and GNN parameters. By employing a probabilistic model for edge prediction and optimizing edge distribution parameters, the method marries the discrete nature of graphs with the continuous optimization landscape typical of neural networks. Additionally, the adaptation of gradient-based hyperparameter optimization to discrete hyperparameters marks a significant advancement, especially in efficiently handling high-dimensional search spaces bounded by constraints stemming from graph structures.

This paper addresses a pertinent issue in GNN deployment and contributes a method with both theoretical insights and practical utility. Its approach to using bilevel optimization reflects a broader trend towards sophisticated optimizations in AI methods that consider multi-layered, interdependent objectives. Future advancements could aim to further reduce computational costs and enhance scalability, as well as exploring extensions that integrate with diverse types of neural architectures beyond GCNs.

In conclusion, "Learning Discrete Structures for Graph Neural Networks" provides a robust framework for tackling the absence or corruption of graph structures, offering a path forward for leveraging GNNs in a broader array of applications. The implications for AI model training, deployment, and accuracy enhancement are profound, charting a promising avenue for future research in the domain of graph-based machine learning.