Graph neural networks in particle physics

Measurements in particle physics are commonly done in large accelerator facilities (CERN KEK, Fermilab, etc.), using detectors with sizes on the order of tens of meters, which capture millions of high-dimensional measurements each second. These detectors are composed of multiple sub-detectors—tracking detector, calorimeters, muon detector, etc.—each using a different technology to measure the trace of particles. The data in particle physics are therefore heterogeneous. Detectors in astrophysics are typically bigger, with size up to kilometers (IceCube, Antares, etc.) constructed around a single measurement technology, the data are therefore homogeneous. In both cases, the measurements are inherently sparse in space, due to the design of the geometry of the sensors. The measurements therefore do not a-priori fit homogeneous, grid-like data structures.

Deep learning is often applied on high level features derived from particle physics data [1]. This can improve over more classical data analysis methods, but does not use the full potential of deep learning, which can be effective when operating on lower level information.

Some data in particle physics can be fractionally interpreted as images and hence computer vision techniques (CNNs) are being applied with improved performances [16–21]. However, image representations face some limitations with irregular geometry of detectors or sparsity of the projections applied. Because of the inherent loss of information, image representations may constrain the amount of information that can be extracted from the data.

Measurement and reconstructed objects can be viewed as sequences, with an order imposed from theoretical or experimental understanding of the data. Methods otherwise applied to natural language processing (e.g. RNNs, LSTMs [22] and GRUs [23]) have thus been explored [24, 25]. While the ordering used can usually be justified experimentally, it is often imposed and therefore constrains how the data are presented to models. This ordering can also be learned [26] in some cases, using prior experimental knowledge of the physics process at stake. This is however not always the case and one may expect that the imposed ordering will reduce the learning performance—ordering that is not required as we will see in the following. For example [27] shows evidence that a permutation invariant network outperforms a sequence based algorithm that uses the exact same input features, for the same classification task.

At many levels the data are, by definition, sets (unordered collection) of items. If one considers relation between items (geometrical, or physical) a set transforms into a graph with the addition of an adjacency matrix. There is a-priori less limitation in applying deep learning on this intrinsic representation of the data, than at the other levels mentioned above. A variety of HEP data and their formulation as graphs is illustrated in figure 2.

Zoom In Zoom Out Reset image size

We concentrate in this review on the applications of GNNs to HEP. We argue why graphs are a very useful data representation, and review key architectures. Common traits in graph construction and model architecture will be linked to the specific requirements of the HEP problems under consideration. By providing a normalized description of the models through the formalism introduced in [13] we hope to make the adoption and further development of GNNs for HEP simpler.

This review paper is organized as follows. An overview of the field of geometrical deep learning is given in section 2. Existing applications to particle physics are reviewed in 3. General guidelines for formulating HEP tasks for GNNs are given in section 4. In particular we go in the details of the different approaches in building the graph connectivity in section 4.2, the various model architecture adopted in section 4.3. This paper concludes with a discussion on the various approaches and the remaining open questions in section 5.

Deep learning has been central to the past decade's advances in machine learning and artificial intelligence [28, 29], and can be understood as the confluence of several key factors. First, large neural networks can express very complex functions. Second, valuable information in big data can be encoded into the parameters of large neural networks via gradient-based training procedures. Third, parallel computer hardware can perform such training in hours or days, which is efficient enough for many important use cases. Fourth, well-designed software frameworks, such as TensorFlow [30] and PyTorch [31], lower the technical bar to developing and distributing deep learning applications, making powerful machine learning tools broadly accessible to practitioners.

Fully connected, convolutional, and recurrent layers have been the primary building blocks in modern deep learning, each of which carries different inductive biases, which incentivize or constrain the learning algorithm to prioritize one solution over another. For example, convolutional layers share their underlying kernel function across spatial dimensions of the input signal, while recurrent layers share across the temporal dimension of the input. These building blocks are most suitable for approximating functions on vectors, grids, and sequences, but when a problem involves data with richer structure, these modules are not always convenient or effective to apply. For example, consider learning functions over sets of particles—while it is possible to order them, for example sorting by the transverse momentum p_T of the particle, the imposed ordering in not unique, and it fails to reflect that particles are fundamentally unordered. The aforementioned deep learning modules do not have appropriate inductive biases to exploit this richer graphical structure.

Graph-structured data are ubiquitous across science, engineering, and many other problem domains. A graph is defined, minimally, as a set of nodes as well as a set of edges adjacent to pairs of nodes. Richer varieties and special cases include: trees, where there is exactly one sequence of edges connecting any two nodes; directed graphs, where the two nodes associated with an edge are ordered; attributed graphs, which include node-level, edge-level, or graph-level attributes; multigraphs, where more than one edge may exist between a pair of nodes; hypergraphs, where more than two nodes are associated with an edge; etc. Crucially, graphs are a natural and powerful way of representing many complex systems [11–15], e.g. trees for representing evolution of species, or the hierarchical structure of sentences; lattices and meshes for representing regular and irregular discretizations of space, respectively; dynamic networks for representing traffic on roads and social relationships over time.

GNNs [11–13, 32] are a class of deep learning architectures which implement strong relational inductive biases for learning functions that operate on graphs. They implement a form of parameterized message-passing whereby information is propagated across the graph, allowing sophisticated edge-, node-, and graph-level outputs to be computed. Within a GNN there are one or more standard neural network building blocks, typically fully connected layers, which implement the message computations and propagation functions. The first GNNs [32, 33] were developed and applied for network analysis, especially on internet data, and were trained not with the back-propagation algorithm, but with fixed point iteration via the Almeida–Pineda algorithm [34, 35]. Li et al's [36]'s gated graph sequence neural networks helped integrate more recent deep learning innovations into GNNs, adding RNN modules for improving multiple rounds of message-passing and optimizing their parameters by the back-propagation learning rule [28, 29].

In recent years, the field of GNNs has grown very rapidly, with applications to science and engineering. For example, graph convolution has been used for molecular fingerprinting [37]. Message-passing neural networks [12], which provided a general formulation of GNNs which captured a number of previous methods, were introduced for quantum chemistry. Interaction networks [38] and graph networks [13] have been developed for learning to simulate increasingly complex physical systems [38–41].

GNNs are situated within the broader family of what Bronstein et al [11] term geometric deep learning, which, aside from GNNs, captures related deep learning methods which apply to data structures beyond vectors, tensors, sequences, etc. Their survey explores graph signal processing and how it can be connected to deep learning, with substantial discussion on how the general principles of CNNs applied to Euclidean signals can be transferred to graph-structured signals. Key examples of spectral graph convolution approaches are [42–44], which applied neural networks to the eigenvalues and eigenvectors of the graph Laplacian.

Much work on GNs has focused on learning physical simulation [38–40, 45], similar to Lagrangian methods for particle-based simulation in engineering and graphics. The system is represented as a set of particle vertices, whose interactions are represented by edges and computed via learned functions. Recent work by [41] highlights how far this sub-field has advanced: they trained models to predict systems of thousands of particles, which represent fluids, solids, sand, and 'goop', and show generalization to orders of magnitude more particles and longer trajectories than experienced during training. Because GNs are highly parallelizable on modern deep learning hardware (GPUs, TPUs, FPGAs), their approach scaled well, and its speed was on par with heavily engineered state-of-the-art fluid simulation engines, despite that they did not optimize for speed in their work.

Recently GNs have been extended by adding inductive biases derived from physics, adjusting their architectures to be consistent with Hamiltonian [46] and Lagrangian mechanics [47], which can improve performance and generalization on various physical prediction problems. Other recent work [7] has shown symbolic physical laws can be extracted from the learned functions within a GN.

Here we focus on the graph network (GN) formalism [13], which generalizes various GNNs, as well as other methods (e.g. Transformer-style self-attention [48]). GNs are graph-to-graph functions, whose output graphs have the same node and edge structure as the input. Adopting [13]'s formalism, a graph can be represented by, $G = ({\mathbf{u}}, V, E)$, with N_v vertices and N_e edges. The ${\mathbf{u}}$ represents graph-level attributes. The set of nodes (or vertices) are $V = \{\mathbf{v}_i\}_{i = 1:N_v}$, where ${\mathbf{v}}_i$ represents the ith node's attributes. The set of edges are $E = \{\left({\mathbf{e}}_k, r_k, s_k\right)\}_{k = 1:N_e}$, where ${\mathbf{e}}_k$ represents the kth edge's attributes, and r_k and s_k are the indices of the two (receiver and sender, respectively) nodes connected by the k-th edge.

A GN's stages of processing are as follows.

$$\begin{align} \hspace{-9pc}\begin{array}{l@{}lll} \mathbf{e}^{\prime}_k & = \phi^e\left(\mathbf{e}_k, \mathbf{v}_{r_k}, \mathbf{v}_{s_k}, \mathbf{u} \right) \\ \mathbf{v}^{\prime}_i & = \phi^v\left({\bar{\mathbf e}}^{\prime}_i, \mathbf{v}_i, \mathbf{u}\right) \\ \mathbf{u}^{\prime} & = \phi^u\left({\bar{\mathbf e}}^{\prime}, {\bar{\mathbf v}}^{\prime}, \mathbf{u}\right) \end{array}\quad \begin{array}{l@{}lll} {\bar{\mathbf e}}^{\prime}_i & = \rho^{e \rightarrow v}\left({E^{\prime}_i}\right) \hspace{48pt} \triangleright {\text{Edge block}} \\ {\bar{\mathbf e}}^{\prime} & = \rho^{e \rightarrow u}\left({E^{\prime}} \right) \hspace{48pt} \triangleright {\text{Vertex block}}\\ {\bar{\mathbf v}}^{\prime} & = \rho^{v \rightarrow u}\left( {V^{\prime}} \right) \hspace{48pt} \triangleright {\text{Global block}} \end{array} \end{align} \tag{ 1 }$$

A GN block contains 6 internal functions: 3 update functions (φ^e , φ^v , and φ^u ) and 3 aggregation functions ($\rho^{e \rightarrow v}$, $\rho^{e \rightarrow u}$, and $\rho^{v \rightarrow u}$). The GN formalism is not a specific model architecture, it does not determine what exactly those functions are. The update functions are functions of fixed size input and fixed size output, and the aggregation functions take in a variable-sized set of inputs (such as a set of edges connected to a particular node) and output a fixed size representation of the input set. This is illustrated in figure 3.

Zoom In Zoom Out Reset image size

The edge block computes one output for each edge, $\mathbf{e}^{\prime}_k$, and aggregates them by their corresponding receiving node, ${\bar{\mathbf e}}^{\prime}_i$, where $E^{\prime}_i$ is the set of edges incident on the ith node. The vertex block computes one output for each node, $\mathbf{v}^{\prime}_i$. The edge- and node-level outputs are all aggregated in order to compute the global block. The output of the GN is the set of all edge-, node-, and graph-level outputs, $G^{\prime} = (\mathbf{u}^{\prime}, V^{\prime}, E^{\prime})$. See figure 4(a).

Zoom In Zoom Out Reset image size

In practice the φ^e , φ^v , and φ^u are often implemented as a simple trainable neural network, e.g. a FC. The $\rho^{e \rightarrow v}$, $\rho^{e \rightarrow u}$, and $\rho^{v \rightarrow u}$ functions are typically implemented as permutation invariant reduction operators, such as element-wise sums, means, or maximums. The ρ functions must be permutation invariant if the GN block is to maintain permutation equivariance.

Some key benefits of GNs are that they are generic: if a problem can be expressed as requiring a graph to be mapped to another graph or some summary output, GNs are often suitable. They also tend to generalize well to graphs not experienced during training, because the learning is focused on the edge- and node-level—in fact if the global block is omitted, the GN is not even aware of the full graph in any of its computations, as the edge and node blocks take only their respective localities as input. Yet when multiple GN blocks are arranged in deep or recurrent configurations, as in figure 4(b), information can be processed and propagated across the graph's structure, to allow more complex, long-range computations to be performed.

The GN formalism is a general framework which can capture a variety of other GNN architectures. Such architectures can be expressed by removing or rearranging internal components of the general GN block in figure 4, and implementing the various φ and ρ functions using specific functional forms. For example, one very popular GNN architecture is the graph convolutional network (GCN) [49]. Using the GN formalism [12, 13], a GCN can be expressed as,

$$\begin{align*} \mathbf{e}^{\prime}_k & = \phi^e\left(\mathbf{e}_k, \mathbf{v}_{s_k} \right) = \mathbf{e}_k \mathbf{v}_{s_k} \ , \qquad\text{where } \mathbf{e}_k = \frac{1}{\sqrt{\text{degree}(r_k)\text{degree}(s_k)}} \\ {\bar{\mathbf e}}^{\prime}_i & = \rho^{e\rightarrow v}\left(E^{\,^{\prime}}_i\right) = \sum_{\{k \, \vert \, r_k = i\}} \mathbf{e}^{\prime}_k \\ \mathbf{v}^{\prime}_i & = \phi^v\left({\bar{\mathbf e}}^{\prime}_i \right) = \sigma \left({\bar{\mathbf e}}^{\prime}_i W\right) \end{align*}$$

Figure 5 shows the correspondence between the GCN and the GN depicted in figure 4.

Zoom In Zoom Out Reset image size

In section 4 we will discuss the considerations taken into account when deciding how to choose the actual implementation of the GNs internal functions. The choice of the specific architecture is motivated by the relationships that exist between the elements in the input data and the task one is trying to solve with the model.

3.1.1. Jet classification

Here we use the term event for the capture by an experiment of the full history of a physics process. In astroparticle, for example it is the collection of signals that covers the interaction of an high energy particle interacting with the atmosphere. The jet tagging task presented in the previous section is part of a full event identification in collider physics. Event classification is the task of predicting or inferring the physics process at the origin of the recorded data.

The authors of [64] applied a graph convolution method for the classification of the signal in the IceCube detector, to determine if a muon originated from a cosmic neutrino, or from a cosmic ray showering in the earth atmosphere. The adjacency matrix of a fully connected graph of the detector sensors is constrained to a Gaussian kernel on the physical distance, with a learnable locality parameter. Node features are updated by application of the adjacency matrix and non-linear activation. The graph property is extracted from the sum over the latent features of the nodes of the graph. This GNN model yields a signal-to-background ratio about three times as big as the baseline analysis of such signal.

In [65], the message passing neural network architecture from [12] is used over a fully connected graph composed of the final state particles, and the missing transverse energy. Messages are computed from the node features and a distance in the azimuth-rapidity plane first, then in the node latent space for later iterations. Such messages are passed across the graph in two iterations, and each node receives a categorization. The node-averaged value is used to predict the event category. The model is compared to densely connected models, and is showing superior performance when comparing the $S/\sqrt{B}$ analysis significance. From the same authors, in [66, 67], a similar architecture is applied to event classification for other signal topologies, demonstrating the versatility of the method.

3.3.1. Pileup mitigation

3.3.2. Calorimeter reconstruction

3.4.1. Charged particle tracking

3.4.2. Secondary vertex reconstruction

The first step is to decide what function one wants to learn with the GNN. In some applications this is trivial—for example jet, event or particle classification. In those cases a GNN is used to learn some representation of the node or the entire graph/set and a standard classifier is trained on that representation.

For tasks such as segmentation or clustering, there is a choice between formulating the task as edge classification or something like the object condensation method which uses node representations to formulate a partition of the input set. The object condensation method has an important advantage, in that it computes relationships between objects (the attractive or repulsive potential) only while training the algorithm, in the computation of the loss function. An edge classifier will learn an edge representation and use that to classify edges. The number of edges can be large, increasing the computation and memory requirements of the algorithm. The determination of the set partition in the object condensation method is a simple function of the node representation, which greatly reduces those requirements.

None of the work presented in section 3 is using a mapping of the input onto the edges of the graph. Because an edge can only link two nodes—while a node can be connected to as many edges as desirable—construction of such graph would require a specific structure of the input. One such use case could be in situations where observations arise from two concurrent measurements, such as hit position in stereo strip detectors. The detector is composed of two rectangular modules with a thin strip of sensors along one dimension, and the modules are tilted with respect to each other by a couple of degrees so as to have the strip sensors overlapping and hence creating a grid. With strip measurement positioned on the nodes, the important information would be located on the edges, as a combination of two such hits. Other examples in network communication might also be relevant.

In most particle physics applications, the nature of the relationships between different elements in the set are not clear cut (as it would be for a molecule or a social network). Therefore a decision needs to be made about how to construct a graph from the set of inputs. Different graph construction methods are illustrated in figure 6. Depending on the task, one might even want to avoid creating any pairwise relationships between nodes. If the objects have no pairwise conditional dependence—a DeepSet [53] architecture with only node and global properties might be more suitable. Edges in the graph serve 3 roles:

• (a)  
  The edges are communication channels among the nodes.
• (b)  
  Input edge features can indicate a relationship between objects, and can encode physics motivated variables about that relationship (such as ΔR between objects).
• (c)  
  Latent edges store relational information computed during message-passing, allowing the network to encode such variables it sees relevant for the task.

Zoom In Zoom Out Reset image size

In cases where the input sets are small ($N_v \sim \mathcal{O}(10)$ ) the typical and easiest choice is to form a fully connected graph, allowing the network to learn which object relationships are important. In larger sets, as the number of edges between all nodes increases as $N_e \propto ({N_v})^2$, the computational load of using a neural network to create an edge representation or compute attention weights becomes prohibitive. One possible work-around is to choose a fixed edge feature that is easy to pre-compute—such as distance between detector modules.

If an edge-level computation is required, it is necessary to only form some edges. Edges can be formed based on a relevant metric such as the ΔR between particles in a detector, or the physical distance between detector modules. Given a distance measure between nodes, some criterion for connecting them needs to be formulated, such as connecting k-nearest neighbors in the feature space.

The node features used to connect edges can also be based on a learned representation. This is sometimes referred to as dynamic graph construction, and used by the EdgeConv [54] and GravNet [70] architectures, for example. We will discuss this in more detail in section 4.3, showing the connection between the idea of dynamic graph construction and attention mechanisms.

When the graph is constructed dynamically, such as using the node representation to connect edges between k-nearest neighbors, the gradient of the neural network parameters is only affected by those nodes that have actually been connected. Since the indexing of node-neighborhood is non differentiable, its parameters cannot be learn with gradient descent, but can be optimized on hyper-parameter search.

In initial stages of the training, the edge formation is essentially random, allowing the network to explore which node representations should be closer together in the latest space. During later stages of the training, one may wish to encourage further exploration by the network. One possible way to do this is to inject random edges—for example besides connecting nodes to k-nearest neighbors in latest space, connecting an additional small number of random connections to nodes further away in the latent space.

A recent paper [77] introduces a reinforcement learning agent which traverses an input graph to reach nodes which should be connected by new edges. Its policy is optimized for some downstream task performance, so that the nodes it chooses to connect with new edges improve the task performance.

Designing the model architecture should reflect a logical combination of the inputs towards the learning task. In the language of the GN formalism (section 2.2), we need to select a concrete implementation of the GN block update and aggregation functions φ and ρ, and decide how to configure their sequence inside the GN block. Additionally we need to decide which kinds of GN blocks we want to combine and how to stack them together. As explained in section 2.2, different architectures such as Graph Convolution Networks, Graph Attention Networks, are specific choices for constructing a GNN—but they are all equivalent in the sense that their output is a graph with learned node/edge/graph representations which are then used to perform the actual task.

4.3.1. GN block functions

The key question here is what logical steps one would take to form the GN block output in a way that serves the task, and which parts of this logical process should be modeled with neural networks? The most general GN block (as shown in figure 4(a)) could have all of its update functions implemented as neural networks, which allows the most flexibility in the learning processes. This flexibility might not be required for the task, and it might carry computational costs that we wish to keep to a minimum. Therefore its probably better to start with a simple architecture, and only add complexity gradually, until the algorithms performance is satisfactory.

Figure 7 shows two examples of possible configurations, either creating an edge representation before aggregating edges and forming a node update, or using global aggregation before a node update. Both configurations result in an updated node representation, but one of them is based on a sum of pair-wise representations, and the other on a global sum of node representations—the information content is the same, but the inductive bias is different. For example, the authors of [72] assumed that the jet-tagging efficiency is heavily affected by the ΔR between neighboring jets—therefore an edge update step created a representation of pair-wise interaction between jets, which was then summed for each jet to create the updated node representation. In contrast the authors of [53] used a DeepSet architecture, where each node representation is created independently from its neighbors, the node representations are then summed to create the graph representation, with each node representation weighted by the particles energy.

Zoom In Zoom Out Reset image size

4.3.2. Attention mechanisms

Another important component that can be used in defining the $\rho^{e\rightarrow v}$ and $\rho^{v,e\rightarrow u}$ aggregation functions is using attention mechanisms, as illustrated in figure 8. The term attention is rooted in the perceptual psychology and neuroscience literatures, where it refers to the phenomenon and mechanisms by which a subset of incoming sensory information is selected for more extensive processing, while other information is deprioritized or filtered out.

Zoom In Zoom Out Reset image size

The key consideration for defining and adding an attention mechanism is whether different parts of the input data are more important than others. For example, in classifying jets, some particles that originate from a secondary decay are an important footprint of a particular class of jets—therefore those particles may be more important for the classification task. There are a few different implementations of attention mechanisms. They all share the basic concept of using a neural network or a pre-defined function to compute weights which represent the relative importance of different elements in a set. In the GN block ρ functions, these weights are used to create weighted sums of the representations of the different elements.

Here we want to draw attention to the connection between attention mechanisms and dynamic graph construction. Figure 9 shows the structure of two architectures discussed in section 3, the EdgeConv, GravNet layers. These are both GN block implementations, they take as input a set of nodes (without explicit edges) and output an updated node representation. Both begin with a node embedding stage, which creates a node representation without exchanging information between the nodes. This node embedding (or only part of its feature vector, in the case of GravNet) is interpreted as a position of the node in a latent euclidean space, and edges are formed between k-nearest neighbors. This can be thought of as a fully connected graph with an attention mechanism that assigns a weight of 1 to nodes within the set of k-nearest neighbors, and 0 otherwise. The advantage of this procedure over using a neural network to compute attention weights is the much lower computational cost of both computing the edge attention weight and the subsequent edge-related operations.

Zoom In Zoom Out Reset image size

It is worth noting that the GarNet layer [70] can be described as a form of multi-headed self-attention mechanism [48]. The GarNet layer interprets the node embedding as s different 'distances' (with s being the dimension of the embedding). These distances are attention weights over each node of the graph, and they are used to compute s different weighted sums—these are the s different heads of the attention mechanism. The weighted sums are propagated back to the nodes again via attention weights of each node to each of the s attention heads. The reason GarNet is computationally affordable without a hard cutoff—such as k-nearest neighbors—is that φ^v , the node embedding function, is the only one computed with a neural network. The attention weights are all computed with pre-defined functions given the node embedding (specifically, the function is $\textrm{exp}(-|w|)$ where w is the attention weight).

4.3.3. Stacking GN blocks

A stack of GN blocks (as described in figure 4(b)) serves two purposes. First, in the same way that stacked layers in any neural network architecture (such as a CNN) can be thought of as gradually constructing a high level representation of the data, GN blocks arranged sequentially serve the same purpose for constructing the node/edge and graph representations. Therefore, additional GN blocks increase the depth of the model and its expressive power.

Second, after one iteration of message passing in a single GN block, the node has only exchanged information with its immediate connected neighbors. This is illustrated in figure 10. Multiple iterations with a GN block (either the same block applied multiple times, or different blocks applied in a sequence) increase each nodes receptive field, as the representation of its neighboring nodes was previously updated with information from their neighbors. Often skip or residual connections, which combine the input with the output, are used to prevent corruption of the updated representations, and preservation of the gradient signal, over many message passing steps, as is common in CNNs and RNNs.

Zoom In Zoom Out Reset image size

Following the discussion of the GravNet and EdgeConv layers in section 4.3 and their relation to attention mechanisms, another class of models which are closely related to GNNs, and which perform a type of soft structural prediction, are Transformer architectures, based on the self-attention mechanism [48]. In GNN language, a Transformer computes normalized edge weights in a complete graph (i.e. a graph with edges connecting all pairs of nodes), and passes messages along the edges in proportion to these weights, analogous to a hybrid of graph attention networks [63] and GCNs [49].

In GN notation, described in [13] and used explicitly in graph attention networks [63], the Transformer uses a φ^e which produces both a vector message and a scalar unnormalized weight, and the $\rho^{e\rightarrow v}$ function normalizes the weights before computing a weighted sum of the message vectors. This allows a set of input items to be treated as nodes in a graph, without observed input edges, and the edge structure to be inferred and used within the architecture for message-passing. Different variants of attention mechanisms are a way to give different weights in the pooling operations $\rho^{e\rightarrow v}$, $\rho^{v,e\rightarrow u}$, as illustrated if figure 8. The implementation of attention should reflect the nature of the interaction between the objects in the set, as they relate to the task.

The Reformer [78] architecture overcomes the quadratic computational and memory costs that challenge traditional Transformer-based methods, by projecting nodes into a learned high-dimensional embedding space where nearest neighbors are efficiently computed to inform a sparse graph over which to pass messages. The recent Linformer [79] method is similar, but with a low rank approximation to the soft adjacency matrix.

Importantly, the GN does not predict structural changes directly. However, many recent papers use GNs (or other GNNs) to decide how to modify a graph's structure. For example, [80] and [81] are autogressive graph generators, which use a GN or Transformer to predict whether a new vertex should be added to a graph (by the graph-level output), and which existing vertices to connect it to with edges (by the vertex-level outputs). The GraphRNN [82], and Graphite [83] are generative models over edges that use an RNN for sequential prediction, and GraphGAN [84] is an analogous method based on generative adversarial networks. [85]'s Neural Relational Inference treats the existence of edges as latent random variables, and trains a posterior edge inference front-end via variational autoencoding. In [86] and [87], a GN is used to guide the policy of a reinforcement learning agent and build graphs that represent physical scenes. The DiffPool [88] architecture (illustrated in figure 11 is an attention-based soft edge prediction mechanism, but over hierarchies of graphs, where lower-level ones are pooled to higher-level ones.

Zoom In Zoom Out Reset image size

Generative models of graphs have not been explored much in particle physics, though some unpublished work is on-going. The need for computational resource for simulation in particle physics is almost as large as the requirements for event reconstruction. There is a breadth of efforts on using machine learning as surrogate simulators in particle physics. For the reasons exposed in section 1 that data in particle physics can often be represented as graphs, it is natural to investigate the use of generative models using graphs as a possible solution. Models under development are for example predicting energy deposition in the cells of a calorimeter or the particle candidates obtained from a particle flow reconstruction algorithm. In all cases, the generated quantities are naturally represented as a set or graph, with fixed or variable size.

An important consideration for building and efficiently training GNNs on hardware is whether to use dense or sparse implementations of the graph's edges. The number of edges in a graph usually defines the memory and speed bottleneck, because there are typically more edges than nodes and the φ^e function is applied the most times. A dense adjacency matrix supports fast, parallel matrix multiplication to compute $E^{\prime}$, which, for example, is exploited in speed-efficient GCN- and Transformer-style models. The downside is that the adjacency matrix's memory footprint is quadratic in the number of nodes. Alternatively, using sparse adjacency matrices allows the memory to scale linearly in the number of edges, which allows much larger graphs to be processed. But the sparse indexing operations required to implement sparse matrix multiplication can incur greater time costs than their dense counterparts—this is an active area of development for both software and hardware acceleration. However, sparse operations are a key bottleneck in current deep learning hardware, and should next generation hardware substantially improve their speed, this would potentially improve the relative advantage of sparse edge implementations of GNNs. In a computing environment in HEP, one cannot expect to have access to dedicated accelerators (GPU, TPU, FPGA, etc.)—although work is going in the direction of building the infrastructure—and one needs to keep into consideration the time for running the model in production.

Neural networks that operate on sets are increasing in popularity in HEP tasks, both in event reconstruction, and in physics analysis. These neural networks are performing well in proof-of-concept studies, either surpassing or matching existing state of the art techniques. They have not yet been tested in the field with real detector data. It is important to understand that all of the models use the same basic building blocks to perform their tasks, and the most important consideration in designing the architecture for these neural networks is to correctly model the nature of interaction between the objects in the input set. It is probably the best practice to start with a simple graph model and architecture then build up on additional complexity geared towards incorporating scientific understanding of the physical process at stake.