Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Benchpress: A Scalable and Versatile Workflow for Benchmarking Structure Learning Algorithms (2107.03863v4)

Published 8 Jul 2021 in stat.ML, cs.LG, stat.AP, and stat.CO

Abstract: Describing the relationship between the variables in a study domain and modelling the data generating mechanism is a fundamental problem in many empirical sciences. Probabilistic graphical models are one common approach to tackle the problem. Learning the graphical structure for such models is computationally challenging and a fervent area of current research with a plethora of algorithms being developed. To facilitate the benchmarking of different methods, we present a novel Snakemake workflow, called Benchpress for producing scalable, reproducible, and platform-independent benchmarks of structure learning algorithms for probabilistic graphical models. Benchpress is interfaced via a simple JSON-file, which makes it accessible for all users, while the code is designed in a fully modular fashion to enable researchers to contribute additional methodologies. Benchpress currently provides an interface to a large number of state-of-the-art algorithms from libraries such as BDgraph, BiDAG, bnlearn, causal-learn, gCastle, GOBNILP, pcalg, r.blip, scikit-learn, TETRAD, and trilearn as well as a variety of methods for data generating models and performance evaluation. Alongside user-defined models and randomly generated datasets, the workflow also includes a number of standard datasets and graphical models from the literature, which may be included in a benchmarking study. We demonstrate the applicability of this workflow for learning Bayesian networks in five typical data scenarios. The source code and documentation is publicly available from http://benchpressdocs.readthedocs.io.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (97)
  1. Anaconda (2016). “\pkgAnaconda Software Distribution.”
  2. “Fast Scalable and Accurate Discovery of DAGs Using the Best Order Score Search and Grow-Shrink Trees.” In 37th Conference on Neural Information Processing Systems (NeurIPS 2023).
  3. Atay-Kayis A, Massam H (2005). “A Monte Carlo Method for Computing the Marginal Likelihood in Nondecomposable Gaussian Graphical Models.” Biometrika, 92(2), 317–335.
  4. Carvalho CM (2006). Structure and Sparsity in High-Dimensional Multivariate Analysis. Ph.D. thesis, Duke University.
  5. Chickering DM (1995). “A Transformational Characterization of Equivalent Bayesian Network Structures.” In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, UAI’95, pp. 87–98. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
  6. Chickering DM (2002). “Optimal Structure Identification with Greedy Search.” Journal of machine learning research, 3(11), 507–554.
  7. “Large-Sample Learning of Bayesian Networks is NP-Hard.” Journal of Machine Learning Research, 5, 1287–1330.
  8. “The \pkgBayesys Data and Bayesian Network Repository.” Queen Mary University of London, pp. 2–2.
  9. “Large-Scale Empirical Validation of Bayesian Network Structure Learning Algorithms with Noisy Data.” International Journal of Approximate Reasoning, 131, 151–188.
  10. Probabilistic Networks and Expert Systems: Exact Computational Methods for Bayesian Networks. Information Science and Statistics. Springer-Verlag New York. ISBN 9780387987675.
  11. Cussens J (2011). “Bayesian Network Learning with Cutting Planes.” In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence, UAI’11, pp. 153–160. AUAI Press, Arlington, Virginia, USA.
  12. Cussens J (2020). “\pkgGOBNILP: Learning Bayesian Network Structure with Integer Programming.” In International Conference on Probabilistic Graphical Models, pp. 605–608. PMLR.
  13. Dawid AP, Lauritzen SL (1993). “Hyper Markov Laws in the Statistical Analysis of Decomposable Graphical Models.” The Annals of Statistics, 21(3), 1272–1317.
  14. Diestel R (2005). Graph Theory (Graduate texts in mathematics), volume 173. Springer Heidelberg.
  15. Duarte E, Solus L (2021). “A New Characterization of Discrete Decomposable Models.” arXiv preprint arXiv:2105.05907.
  16. Edwards D (2012). Introduction to Graphical Modelling. Springer-Verlag Science & Business Media.
  17. Elmasri M (2017). “On Decomposable Random Graphs.” arXiv preprint arXiv:1710.03283.
  18. Elwert F (2013). “Graphical Causal Models.” In Handbook of causal analysis for social research, pp. 245–273. Springer-Verlag.
  19. “The Generalized Pairs Plot.” Journal of Computational and Graphical Statistics, 22(1), 79–91.
  20. “Sparse Inverse Covariance Estimation with the Graphical Lasso.” Biostatistics, 9(3), 432–441.
  21. Friedman N (2004). “Inferring Cellular Networks using Probabilistic Graphical Models.” Science, 303(5659), 799–805.
  22. “Where is the Impact of Bayesian Networks in Learning.” In International Joint Conference on Artificial Intelligence. Citeseer.
  23. Friedman N, Koller D (2003). “Being Bayesian About Network Structure. A Bayesian Approach to Structure Discovery in Bayesian Networks.” Machine Learning, 50(1), 95–125.
  24. “Using Bayesian Networks to Analyze Expression Data.” Journal of computational biology, 7(3-4), 601–620.
  25. Geiger D, Heckerman D (1994). “Learning Gaussian Networks.” In Proceedings of the Tenth international conference on Uncertainty in artificial intelligence, pp. 235–243. Morgan Kaufmann Publishers Inc.
  26. “Parameter Priors for Directed Acyclic Graphical Models and the Characterization of Several Probability Distributions.” The Annals of Statistics, 30(5), 1412–1440.
  27. “The dual PC algorithm and the role of Gaussianity for structure learning of Bayesian networks.” International Journal of Approximate Reasoning, 161, 108975.
  28. Giudici P, Castelo R (2003). “Improving Markov Chain Monte Carlo Model Search for Data Mining.” Machine learning, 50(1), 127–158.
  29. Giudici P, Green PJ (1999). “Decomposable Graphical Gaussian Model Determination.” Biometrika, 86(4), 785–801.
  30. Glymour C, Scheines R (1986). “Causal Modeling with the \pkgTETRAD Program.” Synthese, 68(1), 37–63.
  31. “\pkgcausalAssembly: Generating Realistic Production Data for Benchmarking Causal Discovery.” arXiv preprint arXiv:2306.10816.
  32. Green PJ, Thomas A (2013). “Sampling Decomposable Graphs using a Markov Chain on Junction Trees.” Biometrika, 100(1), 91–110.
  33. Hauser A, Bühlmann P (2012). “Characterization and Greedy Learning of Interventional Markov Equivalence Classes of Directed Acyclic Graphs.” The Journal of Machine Learning Research, 13(1), 2409–2464.
  34. “Counting and Sampling Labeled Chordal Graphs in Polynomial Time.” arXiv preprint arXiv:2308.09703.
  35. “Learning Bayesian Networks: The Combination of Knowledge and Statistical Data.” Machine learning, 20(3), 197–243.
  36. Jack Kuipers PS, Moffa G (2022). “Efficient Sampling and Structure Learning of Bayesian Networks.” Journal of Computational and Graphical Statistics, 31(3), 639–650.
  37. Jennings D, Corcoran JN (2018). “A Birth and Death Process for Bayesian Network Structure Inference.” Probability in the Engineering and Informational Sciences, 32(4), 615–625.
  38. “Causal Inference using Graphical Models with the \proglangR Package \pkgpcalg.” Journal of Statistical Software, 47(11), 1–26.
  39. “A survey of Bayesian Network structure learning.” Artificial Intelligence Review, pp. 1–94.
  40. Koller D, Friedman N (2009). Probabilistic Graphical Models - Principles and Techniques. MIT Press. ISBN 978-0-262-01319-2.
  41. Koski T, Noble J (2012). “A Review of Bayesian Networks and Structure Learning.” Mathematica Applicanda, 40(1), 51–103.
  42. Köster J, Rahmann S (2012). “\pkgSnakemake—A Scalable Bioinformatics Workflow Engine.” Bioinformatics, 28(19), 2520–2522.
  43. Kuipers J, Moffa G (2017). “Partition MCMC for Inference on Acyclic Digraphs.” Journal of the American Statistical Association, 112(517), 282–299.
  44. Kuipers J, Moffa G (2022). “The Interventional Bayesian Gaussian Equivalent Score for Bayesian Causal Inference with Unknown Soft Interventions.” arXiv preprint arXiv:2205.02602.
  45. “Addendum on the Scoring of Gaussian Directed Acyclic Graphical Models.” The Annals of Statistics, 42(4), 1689–1691.
  46. “Links Between Psychotic and Neurotic Symptoms in the General Population: An Analysis of Longitudinal British National Survey Data using Directed Acyclic Graphs.” Psychological Medicine, 49(3), 388–395.
  47. “Mutational Interactions Define Novel Cancer Subgroups.” Nature communications, 9(1), 1–10.
  48. “\pkgSingularity: Scientific Containers for Mobility of Compute.” PloS one, 12(5), e0177459.
  49. “Greedy relaxations of the sparsest permutation algorithm.” In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pp. 1052–1062. PMLR.
  50. “Towards FAIR Principles for Research Software.” 3(1), 37–59.
  51. Lauritzen SL (1996). Graphical Models. Oxford University Press. ISBN 0-19-852219-3.
  52. Lenkoski A (2013). “A Direct Sampler for G-Wishart Variates.” Stat, 2(1), 119–128.
  53. “Bayesian Graphical Models for Discrete Data.” International Statistical Review / Revue Internationale de Statistique, 63(2), 215–232.
  54. Meek C (1997). Graphical Models: Selecting Causal and Statistical Models. Ph.D. thesis, PhD thesis, Carnegie Mellon University.
  55. Meinshausen N (2008). “A Note on the Lasso for Gaussian Graphical Model Selection.” Statistics and Probability Letters, 78(7), 880 – 884.
  56. Merkel D (2014). “\pkgDocker: Lightweight Linux Containers for Consistent Development and Deployment.” Linux journal, 2014(239), 2.
  57. “Using Directed Acyclic Graphs in Epidemiological Research in Psychosis: An Analysis of the Role of Bullying in Psychosis.” Schizophrenia Bulletin, 43(6), 1273–1279.
  58. Mohammadi R, Wit EC (2019). “BDgraph: An R Package for Bayesian Structure Learning in Graphical Models.” Journal of Statistical Software, Articles, 89(3), 1–30.
  59. “Assumption violations in causal discovery and the robustness of score matching.” arXiv preprint arXiv:2310.13387.
  60. “A Manifesto for Reproducible Science.” Nature Human Behaviour, 1(1), 0021.
  61. “A Hybrid Causal Search Algorithm for Latent Variable Models.” In Conference on Probabilistic Graphical Models, pp. 368–379.
  62. “Bayesian Learning of Weakly Structural Markov Graph Laws Using Sequential Monte Carlo Methods.” Electron. J. Statist., 13(2), 2865–2897.
  63. “Sequential Sampling of Junction Trees for Decomposable Graphs.” Statistics and Computing, 32(5), 1–18.
  64. Onisko A (2003). Probabilistic Causal Models in Medicine: Application to Diagnosis in Liver Disorders. Institute of Biocybernetics and Biomedical Engineering. Ph.D. thesis, PhD dissertation. Polish Academy of Science.
  65. Pearl J (1995). “Causal Diagrams for Empirical Research.” Biometrika, 82(4), 669–688.
  66. Pearl J (1997). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Representation and Reasoning Series. Morgan Kaufmann. ISBN 9781558604797.
  67. “\pkgScikit-learn: Machine Learning in \proglangPython.” the Journal of machine Learning research, 12, 2825–2830.
  68. “Foundations of \proglangJSON Schema.” In Proceedings of the 25th International Conference on World Wide Web, pp. 263–273. International World Wide Web Conferences Steering Committee.
  69. Ramsey JD (2021). “Improving Accuracy of Permutation DAG Search using Best Order Score Search.” arXiv preprint arXiv:2108.10141.
  70. “\pkgalgcomparison: Comparing the Performance of Graphical Structure Learning Algorithms with \pkgTETRAD.” Journal of Machine Learning Research, 21(238), 1–6.
  71. “Learning Chordal Markov Networks via Branch and Bound.” Advances in Neural Information Processing Systems 30 (NIPS 2017).
  72. Raskutti G, Uhler C (2018). “Learning directed acyclic graph models based on sparsest permutations.” Stat, 7(1).
  73. “Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy to Game.” In M Ranzato, A Beygelzimer, Y Dauphin, P Liang, JW Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 27772–27784. Curran Associates, Inc.
  74. Russell S, Norvig P (2002). Artificial intelligence: a modern approach. Pearson. ISBN 0-13-461099-7.
  75. “Causal protein-signaling networks derived from multiparameter single-cell data.” Science, 308(5721), 523–529.
  76. “Approximate structure learning for large Bayesian networks.” Machine Learning, 107(8-10), 1209–1227.
  77. “Learning Bayesian Networks with Thousands of Variables.” In Advances in neural information processing systems, pp. 1864–1872.
  78. Scutari M (2010). “Learning Bayesian Networks with the \pkgbnlearn \proglangR Package.” Journal of Statistical Software, Articles, 35(3), 1–22.
  79. “Who Learns Better Bayesian Network Structures: Accuracy and Speed of Structure Learning Algorithms.” International Journal of Approximate Reasoning, 115, 235–253.
  80. “Learning Bayesian Networks from Big Data with Greedy Search: Computational Complexity and Efficient Implementation.” Statistics and Computing, 29(5), 1095–1108.
  81. “On Sensitivity of the MAP Bayesian Network Structure to the Equivalent Sample Size Parameter.” The Conference on Uncertainty in Artificial Intelligence.
  82. “Consistency guarantees for greedy permutation-based causal inference algorithms.” Biometrika, 108(4), 795–814.
  83. Spirtes P, Glymour CN (1991). “An Algorithm for Fast Recovery of Sparse Causal Graphs.” Social science computer review, 9(1), 62–72.
  84. Studenỳ M, Cussens J (2017). “Towards Using the Chordal Graph Polytope in Learning Decomposable Models.” International Journal of Approximate Reasoning, 88, 259–281.
  85. “Bayesian Structure Learning and Sampling of Bayesian Networks with the \proglangR Package \pkgBiDAG.” Journal of Statistical Software, 105(9), 1–31.
  86. Teyssier M, Koller D (2012). “Ordering-Based Search: A Simple and Effective Algorithm for Learning Bayesian Networks.” arXiv preprint arXiv:1207.1429.
  87. “Algorithms for Large Scale Markov Blanket Discovery.” In FLAIRS conference, volume 2, pp. 376–380.
  88. “The Max-Min Hill-Climbing Bayesian Network Structure Learning Algorithm.” Machine learning, 65(1), 31–78.
  89. Verma T, Pearl J (1991). Equivalence and Synthesis of Causal Models. UCLA, Computer Science Department.
  90. “Permutation-based Causal Inference Algorithms with Interventions.” In I Guyon, UV Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, R Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  91. Waskom ML (2021). “\pkgseaborn: Statistical Data Visualization.” Journal of Open Source Software, 6(60), 3021.
  92. “Welcome to the \pkgtidyverse.” Journal of Open Source Software, 4(43), 1686.
  93. “New Insights and Faster Computations for the Graphical Lasso.” Journal of Computational and Graphical Statistics, 20(4), 892–900.
  94. Wormald NC (1985). “Counting Labelled Chordal Graphs.” Graphs and Combinatorics, 1(1), 193–200.
  95. “\pkggCastle: A \proglangPython Toolbox for Causal Discovery.” arXiv preprint arXiv:2307.16405.
  96. “DAGs with NO TEARS: Continuous optimization for structure learning.” In Advances in Neural Information Processing Systems, pp. 9472–9483.
  97. “\pkgCausal-learn: Causal Discovery in \proglangPython.” arXiv preprint arXiv:2307.16405.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now