betaFeedback

WorksPhysics and Astronomy

DeepTreeGAN: Fast Generation of High Dimensional Point Clouds

Abstract

In High Energy Physics, detailed and time-consuming simulations are used for particle interactions with detectors. To bypass these simulations with a generative model, the generation of large point clouds in a short time is required, while the complex dependencies between the particles must be correctly modelled. Particle showers are inherently tree-based processes, as each particle is produced by the decay or detector interaction of a particle of the previous generation. In this work, we present a novel Graph Neural Network model (DeepTreeGAN) that is able to generate such point clouds in a tree-based manner. We show that this model can reproduce complex distributions, and we evaluate its performance on the public JetNet dataset.

Introduction

In particle physics, detailed, near-perfect simulations of the underlying physical processes, the measurements and the details of the experimental apparatus are state of the art. However, accurate simulations of large and complex detectors, especially calorimeters, using current Monte Carlo-based tools such as those implemented in the Geant4 [1] toolkit are computationally intensive. Currently, more than half of the worldwide Large Hadron Collider (LHC) grid resources are used for the generation and processing of simulated data [2]. For the future High-Luminosity upgrade of the LHC (HL-LHC [3]), the computational requirements will exceed the computational resources without significant speedups, e.g. for the CMS experiment by 2028 [4] projecting the current technologies. This will become even more demanding for future high-granularity calorimeters, e.g. the CMS HGCAL [5] with its complex geometry and extremely large number of channels. To address these challenges, fast simulation approaches based on Deep Learning, including Generative Adversarial Networks (GAN) [6] and Variational Autoencoders (VAE) [7], have been explored early [8, 9, 10, 11, 12, 13] and deployed recently [14].

The majority of these approaches represent the data as voxels living on three-dimensional grids. But especially for high-granularity calorimeters like the HGCAL, the calorimeter cells are highly irregular and the data is often sparse. In such a situation, it is beneficial to represent the data as Point Clouds (PC), e.g. tuples of cell energies and three-dimensional coordinates, i.e. four-dimensional point clouds [15, 16, 17]. For modelling calorimeter showers, which are cascades of particles, it seems natural to model the creation of such point clouds as a tree-like process. Such an approach can be implemented by Graph Neural Networks (GNN) [18] and Graph Convolutions [19]. For modelling PC data representing three-dimensional objects, a similar approach [20] called tree-GAN [20] exists. For our use case, especially for modelling the large dynamic range of the energy dimension, this approach is not sufficient. Here we report on the development of a largely extended Graph GAN approach, which we named DeepTreeGAN that can be applied to multiple areas of PC generation. To demonstrate the abilities of our model, we present a first application to the JetNet dataset introduced in the next section.

JetNet and MPGAN

In this study, the JetNet [21] datasets are used. Each dataset contains PYTHIA [22] jets with an energy of about 1 TeV, with each jet containing up to 30 or 150 constituents (here: 30). The datasets differ in the jet-initiating partons. Here, datasets for top quark, light quarks and gluon-initiated jets are studied [23, 24]. Each dataset contains about 170k individual jets split as 110k/10k/50k for training/test/validation where the validation dataset is used for our results. The jet constituents, the particles, are clustered with a cone radius of R=0.8R = 0.8. These particles are considered to be massless and can therefore be described by their 3-momenta or equivalently by transverse momentum pTp_T, pseudorapidity η\eta, and azimuthal angle ϕ\phi. In the JetNet datasets, these variables are given relative to the jet momentum: ηirel:=ηi−ηjet\eta_i^{\mathrm{rel}} := \eta_i - \eta^{\mathrm{jet}}, ϕirel:=ϕi−(ϕjet mod 2π)\phi_i^{\mathrm{rel}} := \phi_i - (\phi^{\mathrm{jet}} \bmod2\pi), and pT,irel:=pT,i/pT,jetp_{T,i}^{\mathrm{rel}} := p_{T,i}/p_{T,\mathrm{jet}}, where ii runs over the particles in a jet. Calculating the invariant mass from these relative quantities, for example, for the jet mass, implies mrel=mjet/pT,jetm^{\mathrm{rel}} = m_{\mathrm{jet}}/p_{T,\mathrm{jet}}. The JetNet library [25] provides several metrics that are used in this study. Additionally, the authors provide a baseline model called MPGAN [26].

Architecture

The generator of this GAN constructs point clouds by starting with a random vector as the root of a tree and then attaching one level of leaves after the other. The output of the generator is the last level of the tree. Starting with the root node, a Branching Layer (Figure 1) takes the current leaves from the tree, maps each leaf to a given number nodes nin_i and attaches these nodes as new leaves to the tree. Then further Branching Layers are applied until the desired number ∏ini\prod_i n_i of nodes (here: 30 nodes) is reached. For these projections, multiple feed-forward neural networks (FFNs) are used.

Figure 1. Branching Layer. In this example, the nodes of the 2nd level of the tree are produced and attached as the new leaves. With the nodes from level 1 (dark green / orange) as the parents, the children in level 2 (blue+green / red+yellow) are generated independently: The vector of the parent is split into one part for each of the branches. To each of these vectors, the Global Feature (2) and the global condition vector (here: number of constituents) are concatenated. The Branching FFN maps the vectors to the same size as their parents. Then the feature vector of the parents is added to each of its children. With the new children added as leaves to the tree, all the levels of the tree are stacked up and passed through a Dimensionality Reduction FFN. With each Branching Layer, the number of nodes increases (1 → 2 → 23 → 23*5) and the number of features decreases (64 → 33 → 20 → 3).

In between the Branching Layers a Message Passing Layer (MPL) passes the information down the tree. In this step, each of the nodes is updated with information from its ancestors. This Ancestor MPL is described in Figure 2.

Figure 2. Ancestor Message Passing Layer. The edges in the graph are constructed so that each node receives messages from each of its ancestors as well as itself. In the given example node 4 is updated with the aggregate of the messages from nodes 1, 2 and 4. As a message-passing algorithm, GINConv [27] – as implemented in PyTorch Geometric [28] – is chosen. For GINConv, the messages are the features of the source node (here: the ancestor). These messages are aggregated by summing over all messages addressed to the target node. These aggregates are added to the feature vector of the target node (scaled with a learnable weight) and passed through an FFN. The nodes are then updated with the output of the FFNs plus, as a residual connection, the original feature vector of the nodes themselves.

A Global Feature Layer is introduced that condenses the information of the current leaves in a single vector: The feature vectors of the leaves are passed through a first FFN, summed up and passed through a second FFN. The resulting vector is then used – together with a global condition vector – to condition the following Branching Layer and Ancestor MPL. The global condition vector contains only the number of constituents in the event. For each of the levels of the tree, first the Global Feature Layer is executed, then the Branching Layer and then the Ancestor MPL.

The FFNs of this model consist of 3 hidden layers of 100 nodes without bias. The first two hidden layers are followed by a Batch Normalization [29] layer and LeakyReLU activation with a negative slope of 0.1. As a critic, the MPGAN critic [26] is used.

Training

Generator and Critic are trained using the Hinge [30] loss, with a ratio of 1 to 2 gradient steps. The optimizer for the Generator is Adam with β1=0.9,β2=0.999\beta_1 = 0.9,\beta_2 = 0.999, a weight decay of 10−410^{-4} and a learning rate of 10−510^{-5}. The optimizer for the critic is Adam with β1=0.0,β2=0.9\beta_1 = 0.0,\beta_2 = 0.9, a weight decay of 10−410^{-4} and a learning rate of 3∗10−53 * 10^{-5}.

Comparison to tree-GAN

Compared to the model proposed in [20], DeepTreeGAN introduces some major changes: The branching (Figure 1) uses two FFNs instead of matrix multiplication: one for increasing the number of nodes, the other for reducing the number of features. As the new leaves have the same size as their ancestors, a single Dimensionality Reduction FFN can be applied to all nodes of the tree. After the branching, both models implement a graph convolution step, passing information from the ancestors to their children. In tree-GAN, the leaves are updated using the leaf itself and its ancestors, each multiplied with a separate trainable matrix. In DeepTreeGAN, all nodes in the tree have the same dimension at all times. Thus, all nodes can be updated at the same time, using a single FFN in the Ancestor MPL (Figure 2).

Preprocessing & Postprocessing

For the training, ηrel\eta^{\mathrm{rel}} and ϕrel\phi^{\mathrm{rel}} are scaled separately to a normal distribution. The pTrelp_{\mathrm{T}}^{\mathrm{rel}} distribution is transformed using the Box-Cox transformation with standardizing implemented in [31]. To produce samples, the inverse scaling is applied to the output of the generator.

Results

The variables of the constituents of the top quark dataset are shown in Figure 3. The generated distributions of the constituents closely match the distribution of the data. The jet variables shown in Figure 4 are significantly harder to model. Our model successfully reproduces the double peak structure in the relative jet mass, showing a significant advantage over tree-GAN as shown in [21 Figure 3].

Figure 3. The global particle distributions for the top-quark test dataset, as described in § 1.1.

Figure 4. The distributions of the jet variables in the top-quark test dataset.

In Table 1 the achieved metrics are shown. While our model shows an advantage over the MPGAN baseline on the top quark dataset and an at least similar performance on the light quarks’ dataset, the performance is worse for the gluon dataset. At the current stage, the generator always produces the full 30 constituents for each of the events. Thus, the performance for events with less than 30 constituents is limited, as is frequently the case for the gluon jets. The models and code are available on GitHub2.

DatasetModelW1M×103W^{\mathrm{M}}_{1} \times10^{3}W1P×103W^{\mathrm{P}}_{1} \times10^{3}W1EFP×105W^{\mathrm{EFP}}_{1} \times10^{5}FPND
GluonLimit0.7±0.20.7 \pm0.20.44±0.090.44 \pm0.090.63±0.070.63 \pm0.07<10−3< 10^{-3}
MPGAN0.7±0.20.7 \pm0.20.9±0.30.9 \pm0.30.7±0.20.7 \pm0.20.120.12
DeepTreeGAN1.8±0.21.8 \pm0.21.6±0.61.6 \pm0.62.1±0.22.1 \pm0.20.340.34
Light QuarksLimit0.5±0.10.5 \pm0.10.5±0.10.5 \pm0.10.46±0.040.46 \pm0.04<10−3< 10^{-3}
MPGAN0.7±0.20.7 \pm0.24.9±0.54.9 \pm0.50.7±0.40.7 \pm0.40.350.35
DeepTreeGAN0.9±0.20.9 \pm0.21.7±0.31.7 \pm0.30.9±0.20.9 \pm0.20.150.15
Top QuarksLimit0.51±0.070.51 \pm0.070.55±0.070.55 \pm0.071.1±0.11.1 \pm0.1<10−3< 10^{-3}
MPGAN0.6±0.20.6 \pm0.22.3±0.32.3 \pm0.32±12 \pm10.370.37
DeepTreeGAN0.6±0.10.6 \pm0.11.1±0.51.1 \pm0.51.3±0.31.3 \pm0.30.140.14

Table 1. Comparison of the proposed DeepTreeGAN to the MPGAN baseline with metrics introduced in [21]. Significant advantages for a model in a given metric are highlighted bold. The ‘Limit’ row gives the measured in-sample distance by sampling datasets from the training dataset and calculating the metrics. Note that the MPGAN model was selected for W1MW^{\mathrm{M}}_{1}, and the testing sample was used to compute the given metrics. The W1W_{1} metrics are calculated with 10000 samples, FPND is calculated with 50000 samples.

Conclusion

This paper presents a new model (DeepTreeGAN) for generating large point clouds using a novel upscaling method for point clouds. It successfully replicates the complex distributions of the JetNet datasets with high fidelity (§ 3) and demonstrating its suitability for generating high-dimensional point clouds. The model’s tree-based structure promises good scaling behavior for even larger point clouds, such as particle showers in high-granularity calorimeters. However, the preliminary version presented here uses the MPGAN critic that scales with O(points2)O(\mathrm{points}^{2}) and thus, requires a new critic with better scaling behavior. In addition, the generator will need to be able to generate a variable number of points for each event, especially for the JetNet datasets with up to 150 particles and the calorimeter use case. This will be the subject of future publications.

Acknowledgement

Moritz Scham is funded by Helmholtz Association’s Initiative and Networking Fund through Helmholtz AI (grant number: ZT-I-PF-5-3). Benno Käch is funded by Helmholtz Association’s Initiative and Networking Fund through Helmholtz AI (grant number: ZT-I-PF-5-64). This research was supported in part through the Maxwell computational resources operated at Deutsches Elektronen-Synchrotron DESY (Hamburg, Germany). The authors acknowledge support from Deutsches Elektronen-Synchrotron DESY (Hamburg, Germany), a member of the Helmholtz Association HGF.

References

  1. [1]S. Agostinelli et al. (GEANT4), Nucl. Instrum. Meth. A 506, 250 (2003)
  2. [2]J. Albrecht et al. (HEP Software Foundation), Computing and Software for Big Science 3, 7 (2019), 1712.06982
  3. [3]G. Apollinari, O. Brüning, T. Nakamoto, L. Rossi, CERN Yellow Report pp. 1–19 (2015), chapter 1 in High-Luminosity Large Hadron Collider (HL-LHC) : Preliminary Design Report, 1705.08830
  4. [4]C.O. Software, Computing, Tech. rep., Geneva (2022), https://cds.cern.ch/record/2815292
  5. [5]The CMS collaboration (CMS), The CMS HGCAL detector for HL-LHC upgrade, in 5th Large Hadron Collider Physics Conference (2017), 1708.08234
  6. [6]I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Commun. ACM 63, 139 (2020), 1406.2661
  7. [7]D.P. Kingma, M. Welling (2013), 1312.6114
  8. [8]L. de Oliveira, M. Paganini, B. Nachman, Comput. Software Big Sci. 4 (2017), 1701.05927
  9. [9]M. Paganini, L. de Oliveira, B. Nachman, Phys. Rev. Lett. 120, 042003 (2018), 1705.02355
  10. [10]M. Paganini, L. de Oliveira, B. Nachman, Phys. Rev. D 97, 014021 (2018), 1712.10321
  11. [11]M. Erdmann, L. Geiger, J. Glombitza, D. Schmidt, Comput. Softw. Big Sci. 2, 4 (2018), 1802.03325
  12. [12]M. Erdmann, J. Glombitza, T. Quast, Comput. Softw. Big Sci. 3, 4 (2019), 1807.01954
  13. [13]F. Carminati, A. Gheata, G. Khattak, P. Mendez Lorenzo, S. Sharan, S. Vallecorsa, J. Phys. Conf. Ser. 1085, 032016 (2018)
  14. [14]G. Aad et al., Comput. Softw. Big Sci. 6, 7 (2022)
  15. [15]B. Käch, D. Krücker, I. Melzer-Pellmann, M. Scham, S. Schnake, A. Verney-Provatas, Jetflow: Generating jets with conditioned and mass constrained normalising flows (2022), 2211.13630arxiv.org/abs/2211.13630
  16. [16]B. Käch, D. Krücker, I. Melzer-Pellmann, Point cloud generation using transformer encoders and normalising flows (2022), 2211.13623arxiv.org/abs/2211.13623
  17. [17]S. Schnake, D. Krücker, K. Borras, Generating calorimeter showers as point clouds (2022), https://ml4physicalsciences.github.io/2022/files/NeurIPS_ML4PS_2022_77.pdf
  18. [18]F. Scarselli, M. Gori, A.C. Tsoi, M. Hagenbuchner, G. Monfardini, IEEE Transactions on Neural Networks 20, 61 (2009)
  19. [19]T.N. Kipf, M. Welling, IEEE Transactions on Neural Networks. 5, 61 (2016), 1609.02907
  20. [20]D.W. Shu, S.W. Park, J. Kwon, 3d point cloud generative adversarial network based on tree structured graph convolutions (2019), 1905.06292arxiv.org/abs/1905.06292
  21. [21]R. Kansal, J. Duarte, H. Su, B. Orzari, T. Tomei, M. Pierini, M. Touranakou, J.R. Vli-mant, D. Gunopulos (2022), 2106.11535
  22. [22]T. Sjostrand, S. Mrenna, P.Z. Skands, Comput. Phys. Commun. 178, 852 (2008), 0710.3820
  23. [23]R. Kansal et al., Jetnet (2022), https://doi.org/10.5281/zenodo.6975118
  24. [24]R. Kansal et al., Jetnet150 (2022), https://doi.org/10.5281/zenodo.6975117
  25. [25]R. Kansal, JetNet library for machine learning in high energy physics (2022), https://doi.org/10.5281/zenodo.7140150
  26. [26]R. Kansal, MPGAN code, https://github.com/rkansal47/MPGAN/tree/neuri ps21
  27. [27]K. Xu, W. Hu, J. Leskovec, S. Jegelka, How powerful are graph neural networks? (2019), 1810.00826arxiv.org/abs/1810.00826
  28. [28]M. Fey, J.E. Lenssen, Fast Graph Representation Learning with PyTorch Geometric, in ICLR Workshop on Representation Learning on Graphs and Manifolds (2019)arxiv.org/abs/1903.02428
  29. [29]S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift (2015), 1502.03167
  30. [30]J.H. Lim, J.C. Ye, Geometric gan (2017), 1705.02894arxiv.org/abs/1705.02894
  31. [31]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg et al., Journal of Machine Learning Research 12, 2825 (2011)

Paper details

vlnc:W_a77enq7v
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents