betaFeedback

WorksComputer Science

Information Estimation with Discrete Diffusion

Abstract

Information-theoretic measures, such as Mutual Information (MI), play a crucial role in understanding non-linear relationships between random variables and are widely used across scientific disciplines. Yet, their use on real-world discrete data remains challenging. Existing methods typically rely on embedding discrete data into a continuous space and apply neural estimators originally designed for continuous distributions. This process requires careful engineering for both the embedding model and estimator architecture, but suffers from issues related to high data dimensionality. In this work, we introduce InfoSEDD, a discrete diffusion-based approach that bridges information-theoretic estimation and generative modeling such that they can be used to compute Kullback-Leibler divergences. Backed by Continuous Time Markov Chains theory principles, the design of InfoSEDD is lightweight and scalable and allows seamless integration with pretrained models. We showcase the versatility of our approach through applications on motif discovery in genetic promoter data, semantic-aware model selection in text summarization, and entropy estimation in Ising models. Finally, we construct consistency tests on real-world textual and genomics data. Our experiments demonstrate that InfoSEDD outperforms alternatives that rely on the "embedding trick". Our results position InfoSEDD as a robust and scalable tool for information-theoretic analysis of discrete data.

Introduction

Information theoretic measures represent a powerful tool to understand non-linear relationships between random variables [74, 48] and find a wide range of applications in scientific fields [37, 22]. Mutual information (MI), in particular, has become an established metric in machine learning [7, 79, 6, 59, 33], both for training models [2, 15, 91] and at inference time [1, 36].

Estimating information theoretic quantities remains an open problem, and different paradigms for their estimation have emerged. Classical parametric and non-parametric methods [64, 52, 41, 28] have been recently superseded by variational approaches [5, 55, 57, 65, 88, 44, 25] and neural estimators. Notably, several approaches rely on optimizing a lower bound [60, 6, 59, 78, 70, 43, 10]. However, McAllester and Stratos (2020) challenge the applicability of this kind of estimator, especially with high MI scenarios, as they require an amount of samples that grows exponentially with the ground truth MI. To overcome this limitation, a new class of estimators has emerged [27, 11, 39]. Unfortunately, these methods focus on continuous data and, despite their practical importance, few estimators for high-dimensional discrete distributions have been proposed in the literature. While classical estimators for discrete random variables exist [63], their accuracy rapidly decreases with increasing data dimensionality. Applications that would benefit from scalable estimators of MI include, among others, DNA or peptide sequencing [54 Xia et al.], text summarization [18] and neuroscience [13], to name a few examples. Consequently, the development of new estimation techniques is of paramount importance for the broader scientific community.

A common workaround to deal with high-dimensional, discrete data is to embed it in a continuous space and use neural estimators conceived for continuous distributions. One recent example is [42], where it is shown that the embeddings of pretrained language models can provide meaningful representations to estimate information theoretic quantities in unstructured data. However, such a process may not fully capture the discrete nature of the underlying data and might suffer from several limitations, such as the necessity to consider application-specific embeddings. For example, we show that MINDE [26], which is a strong MI estimator in continuous domains, like image data, struggles with discrete data.

In this work, we build on the growing literature of diffusion-based estimators [40, 27], and present INFO-SEDD, a novel method for estimating information theoretic quantities of discrete data using Continuous Time Markov Chains (CTMCs) [47]. These stochastic processes have recently seen a surge in popularity for applications such as generative language modeling [47, 71, 56]. Their fundamental working principle is the reversal of a perturbation process which starts with clean data from a given distribution and that converges to uninformative noise. The workhorse of these approaches is the score function, which contains information about the probability distributions associated with the CTMCs at different time instants. Our proposed method, INFO-SEDD, builds upon such mathematical framework, extending it via Dynkin’s lemma [32], and leverages score functions to compute key information-theoretic metrics, such as MI between two random variables, and the entropy of a given distribution. By carefully selecting perturbation processes, our approach requires training only a single parametric model to compute MI across arbitrary subsets of variables. Furthermore, INFO-SEDD seamlessly integrates with pretrained models, without requiring ad-hoc procedures to work with discrete data.

To rigorously evaluate our method, we perform a large number of experiments both on synthetic and real data, and compare INFO-SEDD to state-of-the-art methods. We design a synthetic benchmark with ground truth MI that presents challenges such as high data dimensionality and high MI scenarios. Our results demonstrate that INFO-SEDD is both robust and consistently outperforms existing estimation methods. We then focus on two application domains in which we compare INFO-SEDD to competitors, which all require the “embedding trick” discussed above. First, we tackle the text summarization domain, evaluate the consistency of MI estimators, and study if MI represents a meaningful signal to perform model selection, as INFO-SEDD estimates are well aligned with human metrics for evaluating text summarization. We also present results in the domain of genomics, whereby we evaluate the consistency of MI estimators, and show that INFO-SEDD can be used to detect motif location in DNA sequencing data. Overall, our results indicate that INFO-SEDD outperforms alternatives for MI estimation, and pave the way to exploit information measures as meaningful signals for a variety of downstream tasks involving high-dimensional discrete data.

Methodology

We begin by defining the relation between CTMCs over discrete state spaces [3] and the computation of Kullback-Leibler (KL) divergences. First, in § 2.1, we provide a brief introduction to the fundamentals of CTMCs, emphasizing their time-reversal properties and parametric approximations [47]. Then, in Section 2.2, we demonstrate how these processes can be adapted for divergence estimation, specifically by analyzing two processes that share the same generator but differ in their initial conditions.

Preliminaries

Consider a CTMC X⃗t\vec{X}_t, t∈[0,T]t \in[0,T], defined over a finite state space X={1,…,N}\mathcal{X} = \{1,\ldots,N\} and specified by the infinitesimal generators Q⃗t:[0,T]→RN×N\vec{Q}_t : [0,T] \to\mathbb{R}^{N \times N}, where the diagonal entries satisfy Q→t(a,a)=−∑a≠bQ→t(a,b)\overrightarrow{Q}_t(a,a)=-\sum_{a\ne b}\overrightarrow{Q}_t(a,b), with Q→t(a,b)≥0\overrightarrow{Q}_t(a,b)\ge0, a≠ba\ne b. As established by Anderson [3], the time evolution of the probability distribution Pr(Xt=x):=p→t(x)Pr(X_t=x)\mathrel{\mathop:}=\overrightarrow{p}_t(x) satisfies the following Ordinary Differential Equation (ODE):

p→t=p→0+∫0tQ→sp→s ds,\overrightarrow{p}_t=\overrightarrow{p}_0+\int_0^t\overrightarrow{Q}_s\overrightarrow{p}_s\,\mathrm{d}s,

where the initial conditions of the process p→0\overrightarrow{p}_0 determine the distribution p→t\overrightarrow{p}_t at any time tt. A key property of CTMCs is that their time-reversed counterpart, recently used for generative modeling purposes (Lou et al., [47]), also follows a CTMC, but with a different set of transition matrices. More precisely, defining the time-reversed process as p←t:=p→T−t\overleftarrow{p}_t\mathrel{\mathop:}=\overrightarrow{p}_{T-t}, the reverse process evolves according to the following ODE (Lou et al., [47]; Sun et al., [80]):

p←t=p→T+∫0tQ←sp←s ds,\overleftarrow{p}_t=\overrightarrow{p}_T+\int_0^t\overleftarrow{Q}_s\overleftarrow{p}_s\,\mathrm{d}s,

where the reverse-time transition matrices Q←t\overleftarrow{Q}_t relate to the forward transition matrices as follows:

Q←t(b,a)=(p→T−t(b)p→T−t(a)Q→T−t(a,b))(1−δ(a,b))+(−∑b≠aQ←t(b,a))δ(a,a).(1)\overleftarrow{Q}_t(b,a)=\left(\frac{\overrightarrow{p}_{T-t}(b)}{\overrightarrow{p}_{T-t}(a)}\overrightarrow{Q}_{T-t}(a,b)\right)(1-\delta(a,b))+\left(-\sum_{b\ne a}\overleftarrow{Q}_t(b,a)\right)\delta(a,a). \tag*{(1)}

Under appropriate technical conditions on Q←t\overleftarrow{Q}_t (Lou et al., [47]), the terminal distribution p→T\overrightarrow{p}_T converges to a known reference distribution π\pi, which is independent of the initial distribution p→0\overrightarrow{p}_0. This property enables sampling from p→0\overrightarrow{p}_0 by simulating a CTMC with appropriately chosen generators (Sun et al., [80]; Kelly, [38]). However, other than simple and uninteresting scenarios, exact knowledge of the quantities p→T−t(b)p→T−t(a)\frac{\overrightarrow{p}_{T-t}(b)}{\overrightarrow{p}_{T-t}(a)} is out of reach. A practical solution is to substitute in this numerical integration a parametric function sθp(a,t)bs_\theta^p(a,t)_b, whose parameters are optimized according to the Diffusion Weighted Denoising Score Entropy (DWDSE) loss function (Equation 10, Section 3.2 of Lou et al. [47]). Whenever the context is clear, we simplify the notation for the parametric score in the remainder of the paper and denote it with sθp(X→t)xs_\theta^p(\overrightarrow{X}_t)_x instead of sθp(X→t,t)xs_\theta^p(\overrightarrow{X}_t,t)_x.

KL Divergences via CTMC

Next, we show how to extend the CTMC framework to compute the KL divergence between two probability distributions p→0\overrightarrow{p}_0 and q→0\overrightarrow{q}_0 defined over the same support χ\chi, expressed as KL⁡[p→0∥q→0]\operatorname{KL}\left[\overrightarrow{p}_0\|\overrightarrow{q}_0\right]. To achieve this, we construct two Markov chains that differ only in their initial conditions: one initialized from p→0\overrightarrow{p}_0, and the other initialized from q→0\overrightarrow{q}_0. The KL divergence can be expressed as:

KL⁡[p→0∥q→0]=E[log⁡p→0(X←T)q→0(X←T)]=E[log⁡p←T(X←T)q←T(X←T)]=E[E[log⁡p←T(X←T)q←T(X←T)∣X←0]].(2)\operatorname{KL}\left[\overrightarrow{p}_0\|\overrightarrow{q}_0\right]=\mathbb{E}\left[\log\frac{\overrightarrow{p}_0(\overleftarrow{X}_T)}{\overrightarrow{q}_0(\overleftarrow{X}_T)}\right]=\mathbb{E}\left[\log\frac{\overleftarrow{p}_T(\overleftarrow{X}_T)}{\overleftarrow{q}_T(\overleftarrow{X}_T)}\right]=\mathbb{E}\left[\mathbb{E}\left[\left.\log\frac{\overleftarrow{p}_T(\overleftarrow{X}_T)}{\overleftarrow{q}_T(\overleftarrow{X}_T)}\right|\overleftarrow{X}_0\right]\right]. \tag*{(2)}

The last term in Equation (2) can be rewritten using Dynkin’s formula (Hanson, [32]), which states that for a generic function f:χ×[0,T]→Rf:\chi\times[0,T]\to\mathbb{R}, we have:

References

  1. [1]Alexander A Alemi and Ian Fischer. Gilbo: One metric to measure them all. Advances in Neural Information Processing Systems, 31, 2018.arxiv.org/abs/1802.04874
  2. [2]Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2016.arxiv.org/abs/1612.00410
  3. [3]William J Anderson. Continuous-time Markov chains: An applications-oriented approach. Springer Science & Business Media, 2012.
  4. [4]Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993, 2021.arxiv.org/abs/2107.03006
  5. [5]David Barber and Felix Agakov. The im algorithm: a variational approach to information maximization. Advances in neural information processing systems, 16(320):201, 2004.
  6. [6]Mohamed Ishmael Belghazi, Aristide Baratın, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, 2018.
  7. [7]Anthony J Bell and Terrence J Sejnowski. An information-maximization approach to blind separation and blind deconvolution. Neural computation, 7(6):1129–1159, 1995.
  8. [8]Virginie Bernard, Véronique Brunaud, and Alain Lecharny. Tc-motifs at the tata-box expected position in plant genes: a novel class of motifs involved in the transcription regulation. BMC genomics, 11:1–15, 2010.
  9. [9]Gyan Bhanot. The metropolis algorithm. Reports on Progress in Physics, 51(3):429, 1988.DOI
  10. [10]Rob Brekelmans, Sicong Huang, Marzyeh Ghassemi, Greg Ver Steeg, Roger Baker Grosse, and Alireza Makhzani. Improving mutual information estimation with annealed and energy-based bounds. In International Conference on Learning Representations, 2022.
  11. [11]Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, and Alexey Frolov. Mutual information estimation via normalizing flows. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=JiQXsLvDls.arxiv.org/abs/2403.02187
  12. [12]Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266–28279, 2022.
  13. [13]Barry Chai, Dirk Walther, Diane Beck, and Li Fei-Fei. Exploring functional connectivities of the human brain using multivariate information analysis. Advances in neural information processing systems, 22, 2009.
  14. [14]Hongrui Chen and Lexing Ying. Convergence analysis of discrete diffusion model: Exact implementation through uniformization. Journal of Machine Learning, 4(2):108–127, 2025.arxiv.org/abs/2402.08095
  15. [15]Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.arxiv.org/abs/1606.03657
  16. [16]Lukasz Cincio, Jacek Dziarmaga, Marek M Rams, and Wojciech H Zurek. Entropy of entanglement and correlations induced by a quench: Dynamics of a quantum phase transition in the quantum Ising model. Physical Review A—Atomic, Molecular, and Optical Physics, 75(5):052321, 2007.DOI
  17. [17]Thomas Cover and Roger King. A convergent gambling estimate of the entropy of english. IEEE Transactions on Information Theory, 24(4):413–421, 1978.
  18. [18]Maxime Darrin, Philippe Formont, Jackie Cheung, and Pablo Piantanida. COSMIC: Mutual information for task-agnostic summarization evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, August 2024.
  19. [19]Edo Dotan, Gal Jaschek, Tal Pupko, and Yonatan Belinkov. Effect of tokenization on transformers for biological sequences. Bioinformatics, 40(4):btae196, 2024.DOI
  20. [20]René Dreos, Giovanna Ambrosini, Rouyada Cavin Périer, and Philipp Bucher. Epd and epdnew, high-quality promoter resources in the next-generation sequencing era. Nucleic acids research, 41(D1):D157–D164, 2013.DOI
  21. [21]Bell Raj Eapen. Genomic tokenizer: Toward a biology-driven tokenization in transformer models for dna sequences. bioRxiv, pages 2025–04, 2025.DOI
  22. [22]Andrew W. Eckford, Kenneth A. Loparo, and Peter J. Thomas. Finite-state channel models for signal transduction in neural systems. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 6300–6304. IEEE, March 2016. doi: 10.1109/icassp.2016.7472889. URL http://dx.doi.org/10.1109/ICASSP.2016.7472889.
  23. [23]Olivier Elemento, Noam Slonim, and Saeed Tavazoie. A universal framework for regulatory element discovery across all genomes and data types. Molecular cell, 28(2):337–350, 2007.pubmed.ncbi.nlm.nih.gov/17964271
  24. [24]Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409, 2021.arxiv.org/abs/2007.12626
  25. [25]Marco Federici, David Ruhe, and Patrick Forré. On the effectiveness of hybrid mutual information estimation. arXiv preprint arXiv:2306.00608, 2023.arxiv.org/abs/2306.00608
  26. [26]Giulio Franzese, Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filippone, and Pietro Michiardi. How much is enough? a study on diffusion times in score-based generative models. Entropy, 2023.DOI
  27. [27]Giulio Franzese, Mustapha Bounoua, and Pietro Michiardi. Minde: Mutual information neural diffusion estimation. In International Conference on Learning Representations, volume 2024, pages 16685–16716, 2024.arxiv.org/abs/2310.09031
  28. [28]Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for strongly dependent variables. In Artificial intelligence and statistics, pages 277–286. PMLR, 2015.arxiv.org/abs/1411.2003
  29. [29]Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual information for discrete-continuous mixtures. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1709.06212
  30. [30]B. Goebel, Z. Dawy, J. Hagenauer, and J.C. Mueller. An approximation to the distribution of finite sample size mutual information estimates. In IEEE International Conference on Communications, 2005. ICC 2005. 2005, volume 2, pages 1102–1106 Vol. 2, 2005. doi: 10.1109/ICC.2005.1494518.DOI
  31. [31]Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25, 2023.DOI
  32. [32]Floyd B Hanson. Applied stochastic processes and control for jump-diffusions: modeling, analysis and computation. SIAM, 2007.DOI
  33. [33]R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.arxiv.org/abs/1808.06670
  34. [34]Peter Holderrieth, Marton Hayasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky TQ Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary markov processes. In International Conference on Learning Representations, volume 2025, pages 52153–52219, 2025.arxiv.org/abs/2410.20587
  35. [35]Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.DOI
  36. [36]Sicong Huang, Alireza Makhzani, Yanshuai Cao, and Roger Grosse. Evaluating lossy compression rates of deep generative models. In International Conference on Machine Learning. PMLR, 2020.arxiv.org/abs/2008.06653
  37. [37]Jan Karbowski. Information thermodynamics: From physics to neuroscience. Entropy, 26(9):779, September 2024. ISSN 1099-4300. doi: 10.3390/e26090779. URL http://dx.doi.org/10.3390/e26090779.
  38. [38]Frank P Kelly. Reversibility and stochastic networks. PF Kelly—New York: Willy, 1981.
  39. [39]Sergei Kholkin, Ivan Butakov, Evgeny Burnaev, Nikita Guschin, and Alexander Korotin. Infobridge: Mutual information estimation via bridge matching. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=y8KzU9SKpv.arxiv.org/abs/2502.01383
  40. [40]Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theoretic diffusion. In International Conference on Learning Representations, 2022.arxiv.org/abs/2302.03792
  41. [41]Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69(6):066138, 2004.pubmed.ncbi.nlm.nih.gov/15244698
  42. [42]Kyung eun Lee and Wonjong Rhee. A benchmark suite for evaluating neural mutual information estimators on unstructured datasets. Advances in Neural Information Processing Systems, 37:46319–46338, 2024.arxiv.org/abs/2410.10924
  43. [43]Nunzio A Letizia, Nicola Novello, and Andrea M Tonello. Copula density neural estimation. IEEE Transactions on Neural Networks and Learning Systems, 2025.DOI
  44. [44]Nunzio Alessandro Letizia, Nicola Novello, and Andrea M Tonello. Mutual information estimation via f-divergence and data derangements. Advances in Neural Information Processing Systems, 37:105114–105150, 2024.arxiv.org/abs/2305.20025
  45. [45]Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880. Association for Computational Linguistics, July 2020.arxiv.org/abs/1910.13461
  46. [46]Maxwell W Libbrecht and William Stafford Noble. Machine learning applications in genetics and genomics. Nature Reviews Genetics, 16(6):321–332, 2015.DOI
  47. [47]Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2310.16834
  48. [48]David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.DOI
  49. [49]Michael W Macy, Boleslaw K Szymanski, and Janusz A Holyst. The ising model celebrates a century of interdisciplinary contributions. npj Complexity, 1(1):10, 2024.DOI
  50. [50]Aditya Malusare, Harish Kothandaraman, Dipesh Tamboli, Nadia A Lanman, and Vaneet Aggarwal. Understanding the natural language of dna using encoder–decoder foundation models with byte-level precision. Bioinformatics Advances, 4(1):vbae117, 2024.pubmed.ncbi.nlm.nih.gov/38410643
  51. [51]David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, 2020.
  52. [52]Young-Il Moon, Balaji Rajagopalan, and Upamanyu Lall. Estimation of mutual information using kernel density estimators. Physical Review E, 52(3):2318, 1995.pubmed.ncbi.nlm.nih.gov/9963673
  53. [53]Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gülçehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Stefan Riezler and Yoav Goldberg, editors, Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290. Association for Computational Linguistics, August 2016.arxiv.org/abs/1602.06023
  54. [54]Garin Newcomb and Khalid Sayood. Use of average mutual information and derived measures to find coding regions. Entropy, 23(10):1324, 2021.DOI
  55. [55]XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances in Neural Information Processing Systems, 2007.DOI
  56. [56]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. Advances in Neural Information Processing Systems, 38:50608–50646, 2026.arxiv.org/abs/2502.09992
  57. [57]Sebastian Nowozin, Botond Cs eke, and Ryota Tomioka. F-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 271–279, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.arxiv.org/abs/1606.00709
  58. [58]Lars Onsager. Crystal statistics. i. a two-dimensional model with an order-disorder transition. Physical Review, 65(3-4):117, 1944.DOI
  59. [59]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. Advances in neural information processing systems, 2018.arxiv.org/abs/1807.03748
  60. [60]George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1705.07057
  61. [61]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  62. [62]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.arxiv.org/abs/1201.0490
  63. [63]Assaf Pinchas, Irad Ben-Gal, and Amichai Painsky. A comparative analysis of discrete entropy estimators for large-alphabet problems. Entropy, 26(5):369, 2024.DOI
  64. [64]Stephen M Pizer, E Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. Adaptive histogram equalization and its variations. Computer vision, graphics, and image processing, 39(3):355–368, 1987.
  65. [65]Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, 2019.arxiv.org/abs/1905.06922
  66. [66]Lifeng Qiao, Peng Ye, Yuchen Ren, Weiqiang Bai, Chaoqi Liang, Xinzhu Ma, Nanqing Dong, and Wanli Ouyang. Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna. Advances in Neural Information Processing Systems, 37:66080–66107, 2024.arxiv.org/abs/2412.13716
  67. [67]Arvind Rao, Alfred O Hero, David J States, and James Douglas Engel. Motif discovery in tissue-specific regulatory sequences using directed information. EURASIP Journal on Bioinformatics and Systems Biology, 2007:1–13, 2007.DOI
  68. [68]Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, 2006. ISBN 0-262-18253-X.DOI
  69. [69]Yinuo Ren, Haoxuan Chen, Grant Rotskoff, and Lexing Ying. How discrete and continuous diffusion models meet: Comprehensive analysis of discrete diffusion models via a stochastic integral framework. In International Conference on Learning Representations, volume 2025, pages 42904–42941, 2025.
  70. [70]Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 2020.arxiv.org/abs/2006.12204
  71. [71]Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184, 2024.arxiv.org/abs/2406.07524
  72. [72]Yair Schiff, Chia Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range DNA sequence modeling. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=mk3A5IUdnh8.arxiv.org/abs/2403.03234
  73. [73]Elad Schneidman, Michael J Berry, Ronen Segev, and William Bialek. Weak pairwise correlations imply strongly correlated network states in a neural population. Nature, 440(7087):1007–1012, 2006.pubmed.ncbi.nlm.nih.gov/16625187
  74. [74]Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.
  75. [75]David Sherrington and Scott Kirkpatrick. Solvable model of a spin-glass. Physical review letters, 35(26):1792, 1975.
  76. [76]Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37:103131–103167, 2024.arxiv.org/abs/2406.04329
  77. [77]Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations, 2019a.arxiv.org/abs/1910.06222
  78. [78]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019b.arxiv.org/abs/1907.05600
  79. [79]Karl Stratos. Mutual information maximization for simple and accurate part-of-speech induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.arxiv.org/abs/1804.07849
  80. [80]Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=BYWwSY2G5s.arxiv.org/abs/2211.16750
  81. [81]Ryosuke Takahira, Kumiko Tanaka-Ishii, and Łukasz Dębowski. Entropy rate estimates for natural language—a new extrapolation of compressed large-scale corpora. Entropy, 18(10):364, 2016.DOI
  82. [82]Andrew E Teschendorff and Steve Horvath. Epigenetic ageing clocks: statistical methods and emerging computational challenges. Nature Reviews Genetics, pages 1–19, 2025.DOI
  83. [83]Ramzan Kh Umarov and Victor V Solovyev. Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks. PloS one, 12(2):e0171410, 2017.DOI
  84. [84]AW van der Vaart and Jon A Wellner. Weak convergence and empirical processes with applications to statistics. Journal of the Royal Statistical Society-Series A Statistics in Society, 160(3):596–608, 1997.
  85. [85]Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.arxiv.org/abs/1411.5726
  86. [86]Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272, 2020. doi: 10.1038/s41592-019-0686-2.DOI
  87. [87]Sean Whalen, Jacob Schreiber, William S Noble, and Katherine S Pollard. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics, 23(3):169–181, 2022.DOI
  88. [88]Gerhard Wunder, Benedikt Groß, Rick Fritschek, and Rafael F. Schaefer. A reverse jensen inequality with result with application to mutual information estimation. In 2021 IEEE Information Theory Workshop (ITW), 2021.arxiv.org/abs/2111.06676
  89. [90]Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SkeHuCVFDr.arxiv.org/abs/1904.09675
  90. [91]Shengjia Zhao, Jiaming Song, and Stefano Ermon. A lagrangian perspective on latent variable generative models. In Proc. 34th Conference on Uncertainty in Artificial Intelligence, 2018.

Paper details

vlnc:W_6ukt6jm3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents