TabularFM: An Open Framework For Tabular Foundational Models (2406.09837v2)

Abstract: Foundational models (FMs), pretrained on extensive datasets using self-supervised techniques, are capable of learning generalized patterns from large amounts of data. This reduces the need for extensive labeled datasets for each new task, saving both time and resources by leveraging the broad knowledge base established during pretraining. Most research on FMs has primarily focused on unstructured data, such as text and images, or semi-structured data, like time-series. However, there has been limited attention to structured data, such as tabular data, which, despite its prevalence, remains under-studied due to a lack of clean datasets and insufficient research on the transferability of FMs for various tabular data tasks. In response to this gap, we introduce a framework called TabularFM, which incorporates state-of-the-art methods for developing FMs specifically for tabular data. This includes variations of neural architectures such as GANs, VAEs, and Transformers. We have curated a million of tabular datasets and released cleaned versions to facilitate the development of tabular FMs. We pretrained FMs on this curated data, benchmarked various learning methods on these datasets, and released the pretrained models along with leaderboards for future comparative studies. Our fully open-sourced system provides a comprehensive analysis of the transferability of tabular FMs. By releasing these datasets, pretrained models, and leaderboards, we aim to enhance the validity and usability of tabular FMs in the near future.

• The paper introduces an open framework integrating curated datasets and modular generative models to enable rigorous pretraining and transferability studies in tabular data.
• The paper demonstrates that pretrained CTGAN and STVAE variants consistently outperform scratch-trained models by approximately 10 metric points across benchmarks.
• The paper highlights limitations such as dataset scale and meta-information integration while laying a foundation for future advancements in tabular FMs.

TabularFM: An Open Framework for Tabular Foundational Models

Motivation and Context

Pretrained foundational models (FMs) have yielded strong generalization advances in domains such as text and vision, but tabular data—despite its ubiquity and practical significance—remains comparatively under-explored in FM research. Key challenges include tabular data heterogeneity, lack of standardized, high-quality benchmarks, and open questions regarding cross-table transferability and model design. TabularFM systematically addresses these deficiencies by providing large-scale cleaned datasets, pretrained generative models, transferability studies, and a modular open-source framework for rigorous experimentation with tabular FMs.

Figure 1: TSNE representation of the top 10 tabular data domains, showing clear structure in the projected space and enabling analysis for domain-level transferability.

Datasets: Curation and Domain Splitting

TabularFM constructs its experimental foundation on two corpora: Kaggle and GitTables. From >1 million candidate tables, rigorous quality filtering (file format, usability score, metadata completeness, and value-type constraints) yields 1,435 from Kaggle and 1,258 from GitTables. Notably, extensive data cleaning—column filtering, missing data imputation, noise/ID/timestamp exclusion—is critical given the highly noisy distribution of web-acquired tabular resources.

Multiple splitting protocols are introduced. Beyond standard random splits, TabularFM establishes a domain-based partition using $k$-means clustering over BERT-encoded table names, yielding domain-homogeneous test sets for probing out-of-domain transfer. This is essential for scientific analysis of model generalization and transferability beyond i.i.d. settings.

TabularFM Framework and Model Architectures

The TabularFM framework supports end-to-end processing from data acquisition through preprocessing, model training, and evaluation with extensible modules for (i) data transformation, (ii) generative model pretraining and fine-tuning, and (iii) transferability-centric benchmarking.

Supported Generative Models

• CTGAN: Conditional GAN for tabular data, employing WGAN-GP objectives, conditional vectors per column, and PacGAN-style batches to mitigate mode collapse.
• TVAE: VAE adapted for tabular modality, coupling Gaussian mixture-based normalization and ELBO optimization.
• STVAE: A modification of TVAE eliminating dataset-specific trainable std-dev parameters, hence directly optimizing MSE—a design intended to increase inter-table transferability.
• STVAEM: Extends STVAE with per-column signature embeddings, concatenated via column-name encoding from large-scale pretrained LLMs.
• GReaT: Transformer decoder models (distilled GPT-2 baseline), utilizing language modeling over serialized, textified table rows.

Each model is integrated with data transformation pipelines: categorical columns are one-hot encoded; numerical columns are normalized via Gaussian mixture modeling; for transformer models, data is serialized as natural-language phrases e.g., "Age is 26 and Gender is M". This modularization supports reproducible comparison and experimentation.

Experimental Protocol and Evaluation Metrics

TabularFM evaluates transferability by pretraining models on curated pretraining sets, then fine-tuning and evaluating on distinct validation/test splits against a baseline of models trained from scratch. For model and data comparison, synthetic versus real data is systematically assessed using:

• Column Shape Similarity: KS statistic for numericals, TVD for categoricals.
• Column Trend Similarity: Pearson correlation for numerical pairs, TVD over contingency tables for categorical/heterogeneous pairs.

Overall scores are mean-aggregated to enable rigorous, interpretable statistical comparison (e.g., using Mann-Whitney U tests to assess significance).

Results and Empirical Analysis

Consistently, CTGAN and STVAE variants pretrained on large tables outperform models trained from scratch by approximately 10 points in overall metrics, both on random and domain splits. However, adding meta-information (STVAEM) only marginally improves transferability. Transformers pretrained solely on text (GReaT) perform better than any tabular-pretrained transformer, and surprisingly, domain-adaptive finetuning sometimes slightly degrades transformer performance. This implies existing tabular datasets may be insufficiently scaled for further transformer pretraining to outcompete LLMs trained on massive text.

The transferability of pretrained generative models is also demonstrated in convergence acceleration and superior fit to empirical data distributions:

Figure 2: Training/validation loss for STVAE, with pretrained initialization yielding optimized solutions faster and to lower final loss than training from scratch.

Figure 3: Column-wise distributions for pre-trained STVAE vs. scratch-trained STVAE reveal superior tail modeling and distributional fidelity post-pretraining.

Performance improvements are robust across a range of network sizes and learning rates:

Figure 4: Learning rate sensitivity: Pretrained CTGAN models outpace scratch-trained models across learning rates in validation trials.

Figure 5: Larger CTGAN architectures yield improved validation performance, with pretraining consistently outperforming training from scratch.

Transferability, Generality, and Limits of Knowledge Capture

Analysis of column-level transferability reveals that columns corresponding to general semantics (e.g., Age, Gender, Disease) benefit most from pretraining, while columns containing highly specific or domain-unique semantics (e.g., budget codes) do not:

Figure 6: Wordclouds for columns with high vs. low transferability highlight semantic generality as a key factor.

Moreover, pretrained models are able to capture and transfer meaningful correlations, such as public-health and clinical patterns, across unrelated tables:

Figure 7: Visualization of correlation trends where pretrained models exhibit significant improvements or deficits relative to scratch-trained models.

Limitations and Open Problems

Despite clear evidence for transferability gains, several limitations are identified:

• Current training focuses exclusively on numerical and categorical columns, omitting richer data types (dates, free text, timeseries).
• The scale and diversity of current tabular datasets are still insufficient, especially for transformer FMs, whose pretraining appears bounded by dataset size.
• Pretraining with additional meta-information (signature/column embeddings) yields at best incremental improvements, indicating a need for more expressive or context-sensitive metadata integration.
• Comparison to larger-scale LLMs remains difficult without more expansive tabular pretraining corpora.

Implications and Future Directions

The systematic findings of TabularFM establish that:

• Generative models such as CTGAN and STVAE, when pretrained on large, diverse corpora, exhibit substantially improved sample quality, faster convergence, and modest cross-domain generalization capacity in tabular data synthesis.
• Transformers, despite their column permutation invariance and high baseline performance when pretrained on text, do not automatically benefit from further pretraining on modest tabular corpora unless tabular FM datasets can be scaled much further.
• Evaluation and benchmarking in tabular foundation models now becomes standardized thanks to the provided open framework, cleaned datasets, pretrained checkpoints, and reproducible leaderboards.
• The field remains in early stages relative to canonical NLP/Vision FMs: challenges in data curation, transfer across heavily heterogeneous domains, and appropriate use of meta-information must be addressed for next-generation tabular FMs.

Conclusion

TabularFM significantly advances the systematic study of foundational models for tabular data by providing curated datasets, pretrained generative model architectures, and rigorous benchmarks for transferability. The experimental findings robustly support the efficacy of pretraining (especially for GANs and VAEs) in tabular domains and illustrate the nuanced limits of transfer in transformer-based approaches given current dataset scale. The openly released framework, models, and benchmarks form an essential foundation for future state-of-the-art research in tabular FM design and deployment. Further scaling, richer architecture innovations, and broader data-type support represent important areas for upcoming advancement.