Review
Non-parametric Bayesian networks: Improving theory and reviewing applications

https://doi.org/10.1016/j.ress.2015.07.027Get rights and content

Highlights

  • The paper gives an overview of the current NPBNs methodology.

  • We extend the NPBN methodology by relaxing the conditions of one of its fundamental theorems.

  • We propose improvements of the data mining algorithm for the NPBNs.

  • We review the professional applications of the NPBNs.

Abstract

Applications in various domains often lead to high dimensional dependence modelling. A Bayesian network (BN) is a probabilistic graphical model that provides an elegant way of expressing the joint distribution of a large number of interrelated variables. BNs have been successfully used to represent uncertain knowledge in a variety of fields. The majority of applications use discrete BNs, i.e. BNs whose nodes represent discrete variables. Integrating continuous variables in BNs is an area fraught with difficulty. Several methods that handle discrete-continuous BNs have been proposed in the literature. This paper concentrates only on one method called non-parametric BNs (NPBNs). NPBNs were introduced in 2004 and they have been or are currently being used in at least twelve professional applications. This paper provides a short introduction to NPBNs, a couple of theoretical advances, and an overview of applications. The aim of the paper is twofold: one is to present the latest improvements of the theory underlying NPBNs, and the other is to complement the existing overviews of BNs applications with the NPNBs applications. The latter opens the opportunity to discuss some difficulties that applications pose to the theoretical framework and in this way offers some NPBN modelling guidance to practitioners.

Introduction

Understanding and representing multivariate distributions along with their dependence structure is a highly active area of research. A large body of scientific work treating multivariate models is available. This paper advocates graphical models in general and Bayesian networks (BNs) in particular to represent high dimensional distributions with complex dependence structures.

A BN consists of a directed acyclic graph (DAG) and a set of (conditional) distributions. Each node in the graph corresponds to a random variable and the arcs represent direct qualitative dependence relationships. The absence of arcs guarantees a set of (conditional) independence facts. The direct predecessors (successors) of a node are called parents (children). A marginal distribution is specified for each node with no parents, and a conditional distribution is associated with each child node. The (conditional) distributions serve as the quantitative information about the strength of the dependencies between the variables involved. The graph with the conditional independence statements encoded by it, together with the (conditional) distributions represent the joint distribution over the random variables denoted by the nodes of the graph. The quantitative information needed can be either retrieved from data, when available, or from experts.

The relatively simple visualization of the complicated relationships among the random variables is one of the most appealing features of a BN model. The main use of BNs is to update distributions given observations. This is referred to as inference in BNs.

BNs have been successfully used to represent uncertain knowledge, in a consistent probabilistic manner, in a variety of fields [62]. Nevertheless, most of the applications use discrete BNs, i.e BNs whose nodes represent discrete random variables. These models specify marginal distributions for the nodes with no parents, and conditional probability tables for child nodes. Inference in discrete BNs can be performed using either exact algorithms (e.g. [68], [51]), or approximate algorithms (e.g. [31], [47]). Despite their popularity, discrete BNs suffer from severe limitations. Applications involving high complexity in data-sparse environments are severely limited by the excessive assessment burden which leads to rapid, informal and indefensible quantification. Furthermore, this type of representation, using only discrete variables, is inadequate for many problems of practical importance. Discrete BNs have been extensively studied, hence we omit their presentation here, but refer to Weber et al. [62] and references therein. Many domains require reasoning about the joint behaviour of a mixture of discrete and continuous variables. These domains are often called hybrid domains. Hence, BNs involving both discrete and continuous variables will be called hybrid BNs (HBNs). Working with HBNs proves considerably more challenging than working with their discrete counterparts. Several methods for HBNs are available and discussed in the literature (e.g. [38], [34]). We will only briefly mention some of them, insist on the most recent ones, and present in more detail the method called non-parametric BNs (NPBNs) that is overlooked in many HBNs review papers. The emphasis here is on the application of the NPBN methods in different fields of science and engineering in the course of approximately 10 years since NPBNs were first presented.

A study which is perhaps the most comprehensive overview study of BNs applications [62] incorporated information about most methods available for working with HBNs up to the date of its publication. Given the amount of research available on the theoretical and practical aspects of BNs, it is not surprising that even such an impressively large study overlooked a couple of methods and their applications. For instance only one application of NPBNs was mentioned, without any specification of the methodology applied. We would like to complement their presentation of NPBN applications.

We aim to provide a short introduction to the NPBNs, present a couple of theoretical refinements and an overview of their use in practice. Moreover a number of difficulties and challenges occur when working with HBNs in general, and with NPBNs in particular. We address these challenges from the practitioner׳s perspective, and in this way we provide guidance in using the NPBN framework.

The remainder of the paper is organized as follows. Section 2 provides some background on methods available for HBNs. Section 3 describes the NPBN model and gives details about quantification and inference using NPBNs. The main theorem of NPBNs is reformulated and proved for more general settings. Two NPBN structure learning algorithms are detailed. The first one represents the initially proposed algorithm used in most of the applications presented later in the paper, whereas the second one is a new proposal that shows promise. NPBNs have been or are currently being used in at least twelve professional applications. Section 4 reviews these applications. Two of the applications are described in more detail with the intention of presenting some more theoretical details on real examples. The other applications are only very briefly presented, with more details given in the Appendix. Conclusions and directions for future research are given in Section 5.

Section snippets

Hybrid Bayesian networks

One of the first ideas in dealing with HBNs is the use of the conditional Gaussian model [51], [55], [37]. The price of the discrete-normal HBNs is the restriction (of the continuous part) to the joint normal distribution. Some of the exact algorithms for inference designed for discrete BNs were extended to discrete-normal HBNs. Approximation algorithms are also available (e.g. [38]).

Nevertheless uncertainty distributions need not conform to a particular parametric form. When the joint

Non-parametric Bayesian networks

The NPBN methodology was initially developed for purely continuous BNs. NPBNs associate nodes with random variables for which no marginal distribution assumption is made, and arcs with one-parameter conditional copulae [30], parameterized by Spearman׳s rank correlations. The (conditional) copulae are assigned to the arcs of the NPBN according to a protocol that depends on a (non-unique) ordering of the parent nodes. The main result of NPBNs states that a particular choice of conditional

Applications of NPBNs

This section introduces twelve of the largest projects that use the NPBN methodology. Nine of these applications are divided according to the different subjects they cover: risk analysis, reliability of structures, properties of materials. Two applications of temporal/dynamic NPBNs: one that tackles a parameter estimation problem, and another related to traffic prediction, are coupled in Section 4.4. The last part of this section presents three ongoing research projects. In all NPBNs, the

Conclusions

We have mentioned a number of approaches to deal with HBNs, amongst which MoTBFs, eBN, and NPBNs. This paper concentrates on the NPBN methodology, hence NPBNs were presented in more detail. The presentation tried to cover three aspects: the quantification of the models, the inference, and the real-life applications. Unlike other approaches, the NPBN methodology does not build on classical BN methods, but rather proposes a new way of dealing with non-parametric continuous distributions in BNs.

References (68)

  • T. Bedford et al.

    Vines—a new graphical model for dependent random variables

    Ann Stat

    (2002)
  • Blauw S. Constructing non-parametric Bayesian networks from data [Bachelor thesis]. The Netherlands: TU Delft;...
  • R. Clemen et al.

    Assessing dependenciessome experimental results

    Manag Sci

    (2000)
  • B. Cobb

    Influence diagrams for capacity planning and pricing under uncertainty

    J Manag Account Res

    (2009)
  • Cobb B. Detecting online credit card fraud with hybrid influence diagrams. Working Paper,...
  • R. Cooke

    Experts in uncertaintyopinion and subjective probability in science

    Environmental ethics and science policy series

    (1991)
  • Cover T, Thomas J. Elements of information theory. New York: John Wiley Sons;...
  • D.-J. Delgado-Hernández et al.

    A continuous Bayesian network for earth dams׳ risk assessmentan application

    Struct Infrastruct Eng

    (2014)
  • Embrechts P, Lindskog F, McNeil A. Modelling dependence with copulas and applications to risk management. In: Handbook...
  • Evensen G, Hove J, Meisingset H, Reiso E, Seim K, Espelid O. Using the EnKF for assisted history matching of a North...
  • A. Fernandez et al.

    Incorporating prior knowledge when learning mixtures of truncated basis functions from data

    Front Artif Intell Appl

    (2013)
  • Gheorghe M. Non parametric Bayesian belief nets versus Ensemble Kalman Filter in reservoir simulation [M.Sc. thesis],...
  • Greene M, Liu Y, Chen W, Liu W. Computational uncertainty analysis in multiresolution materials via stochastic...
  • Haak Wvd, Rothkrantz L, Wiggers P, Heijligers B, Bakri T, Vukovic D. Modeling Traffic Information using Bayesian...
  • A. Hanea et al.

    Hybrid method for quantifying and analyzing Bayesian belief nets

    Qual Reliab Eng Int

    (2006)
  • Hanea A, Kurowicka D, Cooke R. The population version of Spearman׳s rank correlation coefficient in the case of ordinal...
  • A.M. Hanea et al.

    Non-parametric Bayesian networks for parameter estimation in reservoir simulationa graphical take on the ensemble Kalman filter (Part I)

    Comput Geosci

    (2013)
  • Hanea AM, Harrington W. Ordinal PM2.5 data mining with non-parametric continuous Bayesian belief nets. Inf Processes J,...
  • Hanea D. Human risk of fire: building a decision support tool using Bayesian networks [Ph.D. thesis]. Technische...
  • Huibregtse E, Morales Nápoles O, Hellebrandt L, Paprotny D, de Wit S. Climate change in asset management of...
  • Jager W. Eliciting and representing joint distributions from experts: quantification of a human performance model for...
  • Jesionek P, Cooke R. Generalized method for modeling dose–response relations—application to BENERIS project. Technical...
  • Joe H. Multivariate models and dependence concepts. London: Chapman & Hall;...
  • M. Jordan et al.

    An introduction to variational methods for graphical models

    Mach Learn J

    (1999)
  • Cited by (0)

    View full text