Information-theoretic measures, such as Mutual Information (MI), play a crucial role in understanding non-linear relationships between random variables and are widely used across scientific disciplines. Yet, their use on real-world discrete data remains challenging. Existing methods typically rely on embedding discrete data into a continuous space and apply neural estimators originally designed for continuous distributions. This process requires careful engineering for both the embedding model and estimator architecture, but suffers from issues related to high data dimensionality. In this work, we introduce InfoSEDD, a discrete diffusion-based approach that bridges information-theoretic estimation and generative modeling such that they can be used to compute Kullback-Leibler divergences. Backed by Continuous Time Markov Chains theory principles, the design of InfoSEDD is lightweight and scalable and allows seamless integration with pretrained models. We showcase the versatility of our approach through applications on motif discovery in genetic promoter data, semantic-aware model selection in text summarization, and entropy estimation in Ising models. Finally, we construct consistency tests on real-world textual and genomics data. Our experiments demonstrate that InfoSEDD outperforms alternatives that rely on the "embedding trick". Our results position InfoSEDD as a robust and scalable tool for information-theoretic analysis of discrete data.
Introduction
Information theoretic measures represent a powerful tool to understand non-linear relationships between random variables [74, 48] and find a wide range of applications in scientific fields [37, 22]. Mutual information (MI), in particular, has become an established metric in machine learning [7, 79, 6, 59, 33], both for training models [2, 15, 91] and at inference time [1, 36].
Estimating information theoretic quantities remains an open problem, and different paradigms for their estimation have emerged. Classical parametric and non-parametric methods [64, 52, 41, 28] have been recently superseded by variational approaches [5, 55, 57, 65, 88, 44, 25] and neural estimators. Notably, several approaches rely on optimizing a lower bound [60, 6, 59, 78, 70, 43, 10]. However, McAllester and Stratos (2020) challenge the applicability of this kind of estimator, especially with high MI scenarios, as they require an amount of samples that grows exponentially with the ground truth MI. To overcome this limitation, a new class of estimators has emerged [27, 11, 39]. Unfortunately, these methods focus on continuous data and, despite their practical importance, few estimators for high-dimensional discrete distributions have been proposed in the literature. While classical estimators for discrete random variables exist [63], their accuracy rapidly decreases with increasing data dimensionality. Applications that would benefit from scalable estimators of MI include, among others, DNA or peptide sequencing [54 Xia et al.], text summarization [18] and neuroscience [13], to name a few examples. Consequently, the development of new estimation techniques is of paramount importance for the broader scientific community.
A common workaround to deal with high-dimensional, discrete data is to embed it in a continuous space and use neural estimators conceived for continuous distributions. One recent example is [42], where it is shown that the embeddings of pretrained language models can provide meaningful representations to estimate information theoretic quantities in unstructured data. However, such a process may not fully capture the discrete nature of the underlying data and might suffer from several limitations, such as the necessity to consider application-specific embeddings. For example, we show that MINDE [26], which is a strong MI estimator in continuous domains, like image data, struggles with discrete data.
In this work, we build on the growing literature of diffusion-based estimators [40, 27], and present INFO-SEDD, a novel method for estimating information theoretic quantities of discrete data using Continuous Time Markov Chains (CTMCs) [47]. These stochastic processes have recently seen a surge in popularity for applications such as generative language modeling [47, 71, 56]. Their fundamental working principle is the reversal of a perturbation process which starts with clean data from a given distribution and that converges to uninformative noise. The workhorse of these approaches is the score function, which contains information about the probability distributions associated with the CTMCs at different time instants. Our proposed method, INFO-SEDD, builds upon such mathematical framework, extending it via Dynkin’s lemma [32], and leverages score functions to compute key information-theoretic metrics, such as MI between two random variables, and the entropy of a given distribution. By carefully selecting perturbation processes, our approach requires training only a single parametric model to compute MI across arbitrary subsets of variables. Furthermore, INFO-SEDD seamlessly integrates with pretrained models, without requiring ad-hoc procedures to work with discrete data.
To rigorously evaluate our method, we perform a large number of experiments both on synthetic and real data, and compare INFO-SEDD to state-of-the-art methods. We design a synthetic benchmark with ground truth MI that presents challenges such as high data dimensionality and high MI scenarios. Our results demonstrate that INFO-SEDD is both robust and consistently outperforms existing estimation methods. We then focus on two application domains in which we compare INFO-SEDD to competitors, which all require the “embedding trick” discussed above. First, we tackle the text summarization domain, evaluate the consistency of MI estimators, and study if MI represents a meaningful signal to perform model selection, as INFO-SEDD estimates are well aligned with human metrics for evaluating text summarization. We also present results in the domain of genomics, whereby we evaluate the consistency of MI estimators, and show that INFO-SEDD can be used to detect motif location in DNA sequencing data. Overall, our results indicate that INFO-SEDD outperforms alternatives for MI estimation, and pave the way to exploit information measures as meaningful signals for a variety of downstream tasks involving high-dimensional discrete data.
Methodology
We begin by defining the relation between CTMCs over discrete state spaces [3] and the computation of Kullback-Leibler (KL) divergences. First, in § 2.1, we provide a brief introduction to the fundamentals of CTMCs, emphasizing their time-reversal properties and parametric approximations [47]. Then, in Section 2.2, we demonstrate how these processes can be adapted for divergence estimation, specifically by analyzing two processes that share the same generator but differ in their initial conditions.
Preliminaries
Consider a CTMC Xt, t∈[0,T], defined over a finite state space X={1,…,N} and specified by the infinitesimal generators Qt:[0,T]→RN×N, where the diagonal entries satisfy Qt(a,a)=−∑a=bQt(a,b), with Qt(a,b)≥0, a=b. As established by Anderson [3], the time evolution of the probability distribution Pr(Xt=x):=pt(x) satisfies the following Ordinary Differential Equation (ODE):
pt=p0+∫0tQspsds,
where the initial conditions of the process p0 determine the distribution pt at any time t. A key property of CTMCs is that their time-reversed counterpart, recently used for generative modeling purposes (Lou et al., [47]), also follows a CTMC, but with a different set of transition matrices. More precisely, defining the time-reversed process as pt:=pT−t, the reverse process evolves according to the following ODE (Lou et al., [47]; Sun et al., [80]):
pt=pT+∫0tQspsds,
where the reverse-time transition matrices Qt relate to the forward transition matrices as follows:
Under appropriate technical conditions on Qt (Lou et al., [47]), the terminal distribution pT converges to a known reference distribution π, which is independent of the initial distribution p0. This property enables sampling from p0 by simulating a CTMC with appropriately chosen generators (Sun et al., [80]; Kelly, [38]). However, other than simple and uninteresting scenarios, exact knowledge of the quantities pT−t(a)pT−t(b) is out of reach. A practical solution is to substitute in this numerical integration a parametric function sθp(a,t)b, whose parameters are optimized according to the Diffusion Weighted Denoising Score Entropy (DWDSE) loss function (Equation 10, Section 3.2 of Lou et al. [47]). Whenever the context is clear, we simplify the notation for the parametric score in the remainder of the paper and denote it with sθp(Xt)x instead of sθp(Xt,t)x.
KL Divergences via CTMC
Next, we show how to extend the CTMC framework to compute the KL divergence between two probability distributions p0 and q0 defined over the same support χ, expressed as KL[p0∥q0]. To achieve this, we construct two Markov chains that differ only in their initial conditions: one initialized from p0, and the other initialized from q0. The KL divergence can be expressed as:
The last term in Equation (2) can be rewritten using Dynkin’s formula (Hanson, [32]), which states that for a generic function f:χ×[0,T]→R, we have:
References
[1]Alexander A Alemi and Ian Fischer. Gilbo: One metric to measure them all. Advances in Neural Information Processing Systems, 31, 2018.arxiv.org/abs/1802.04874
[2]Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2016.arxiv.org/abs/1612.00410
[3]William J Anderson. Continuous-time Markov chains: An applications-oriented approach. Springer Science & Business Media, 2012.
[4]Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993, 2021.arxiv.org/abs/2107.03006
[5]David Barber and Felix Agakov. The im algorithm: a variational approach to information maximization. Advances in neural information processing systems, 16(320):201, 2004.
[6]Mohamed Ishmael Belghazi, Aristide Baratın, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, 2018.
[7]Anthony J Bell and Terrence J Sejnowski. An information-maximization approach to blind separation and blind deconvolution. Neural computation, 7(6):1129–1159, 1995.
[8]Virginie Bernard, Véronique Brunaud, and Alain Lecharny. Tc-motifs at the tata-box expected position in plant genes: a novel class of motifs involved in the transcription regulation. BMC genomics, 11:1–15, 2010.
[9]Gyan Bhanot. The metropolis algorithm. Reports on Progress in Physics, 51(3):429, 1988.DOI
[10]Rob Brekelmans, Sicong Huang, Marzyeh Ghassemi, Greg Ver Steeg, Roger Baker Grosse, and Alireza Makhzani. Improving mutual information estimation with annealed and energy-based bounds. In International Conference on Learning Representations, 2022.
[11]Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, and Alexey Frolov. Mutual information estimation via normalizing flows. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=JiQXsLvDls.arxiv.org/abs/2403.02187
[12]Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266–28279, 2022.
[13]Barry Chai, Dirk Walther, Diane Beck, and Li Fei-Fei. Exploring functional connectivities of the human brain using multivariate information analysis. Advances in neural information processing systems, 22, 2009.
[14]Hongrui Chen and Lexing Ying. Convergence analysis of discrete diffusion model: Exact implementation through uniformization. Journal of Machine Learning, 4(2):108–127, 2025.arxiv.org/abs/2402.08095
[15]Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.arxiv.org/abs/1606.03657
[16]Lukasz Cincio, Jacek Dziarmaga, Marek M Rams, and Wojciech H Zurek. Entropy of entanglement and correlations induced by a quench: Dynamics of a quantum phase transition in the quantum Ising model. Physical Review A—Atomic, Molecular, and Optical Physics, 75(5):052321, 2007.DOI
[17]Thomas Cover and Roger King. A convergent gambling estimate of the entropy of english. IEEE Transactions on Information Theory, 24(4):413–421, 1978.
[18]Maxime Darrin, Philippe Formont, Jackie Cheung, and Pablo Piantanida. COSMIC: Mutual information for task-agnostic summarization evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, August 2024.
[19]Edo Dotan, Gal Jaschek, Tal Pupko, and Yonatan Belinkov. Effect of tokenization on transformers for biological sequences. Bioinformatics, 40(4):btae196, 2024.DOI
[20]René Dreos, Giovanna Ambrosini, Rouyada Cavin Périer, and Philipp Bucher. Epd and epdnew, high-quality promoter resources in the next-generation sequencing era. Nucleic acids research, 41(D1):D157–D164, 2013.DOI
[21]Bell Raj Eapen. Genomic tokenizer: Toward a biology-driven tokenization in transformer models for dna sequences. bioRxiv, pages 2025–04, 2025.DOI
[22]Andrew W. Eckford, Kenneth A. Loparo, and Peter J. Thomas. Finite-state channel models for signal transduction in neural systems. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 6300–6304. IEEE, March 2016. doi: 10.1109/icassp.2016.7472889. URL http://dx.doi.org/10.1109/ICASSP.2016.7472889.
[23]Olivier Elemento, Noam Slonim, and Saeed Tavazoie. A universal framework for regulatory element discovery across all genomes and data types. Molecular cell, 28(2):337–350, 2007.pubmed.ncbi.nlm.nih.gov/17964271
[24]Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409, 2021.arxiv.org/abs/2007.12626
[25]Marco Federici, David Ruhe, and Patrick Forré. On the effectiveness of hybrid mutual information estimation. arXiv preprint arXiv:2306.00608, 2023.arxiv.org/abs/2306.00608
[26]Giulio Franzese, Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filippone, and Pietro Michiardi. How much is enough? a study on diffusion times in score-based generative models. Entropy, 2023.DOI
[27]Giulio Franzese, Mustapha Bounoua, and Pietro Michiardi. Minde: Mutual information neural diffusion estimation. In International Conference on Learning Representations, volume 2024, pages 16685–16716, 2024.arxiv.org/abs/2310.09031
[28]Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for strongly dependent variables. In Artificial intelligence and statistics, pages 277–286. PMLR, 2015.arxiv.org/abs/1411.2003
[29]Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual information for discrete-continuous mixtures. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1709.06212
[30]B. Goebel, Z. Dawy, J. Hagenauer, and J.C. Mueller. An approximation to the distribution of finite sample size mutual information estimates. In IEEE International Conference on Communications, 2005. ICC 2005. 2005, volume 2, pages 1102–1106 Vol. 2, 2005. doi: 10.1109/ICC.2005.1494518.DOI
[31]Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25, 2023.DOI
[32]Floyd B Hanson. Applied stochastic processes and control for jump-diffusions: modeling, analysis and computation. SIAM, 2007.DOI
[33]R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.arxiv.org/abs/1808.06670
[34]Peter Holderrieth, Marton Hayasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky TQ Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary markov processes. In International Conference on Learning Representations, volume 2025, pages 52153–52219, 2025.arxiv.org/abs/2410.20587
[35]Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.DOI
[36]Sicong Huang, Alireza Makhzani, Yanshuai Cao, and Roger Grosse. Evaluating lossy compression rates of deep generative models. In International Conference on Machine Learning. PMLR, 2020.arxiv.org/abs/2008.06653
[37]Jan Karbowski. Information thermodynamics: From physics to neuroscience. Entropy, 26(9):779, September 2024. ISSN 1099-4300. doi: 10.3390/e26090779. URL http://dx.doi.org/10.3390/e26090779.
[38]Frank P Kelly. Reversibility and stochastic networks. PF Kelly—New York: Willy, 1981.
[39]Sergei Kholkin, Ivan Butakov, Evgeny Burnaev, Nikita Guschin, and Alexander Korotin. Infobridge: Mutual information estimation via bridge matching. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=y8KzU9SKpv.arxiv.org/abs/2502.01383
[40]Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theoretic diffusion. In International Conference on Learning Representations, 2022.arxiv.org/abs/2302.03792
[41]Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69(6):066138, 2004.pubmed.ncbi.nlm.nih.gov/15244698
[42]Kyung eun Lee and Wonjong Rhee. A benchmark suite for evaluating neural mutual information estimators on unstructured datasets. Advances in Neural Information Processing Systems, 37:46319–46338, 2024.arxiv.org/abs/2410.10924
[43]Nunzio A Letizia, Nicola Novello, and Andrea M Tonello. Copula density neural estimation. IEEE Transactions on Neural Networks and Learning Systems, 2025.DOI
[44]Nunzio Alessandro Letizia, Nicola Novello, and Andrea M Tonello. Mutual information estimation via f-divergence and data derangements. Advances in Neural Information Processing Systems, 37:105114–105150, 2024.arxiv.org/abs/2305.20025
[45]Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880. Association for Computational Linguistics, July 2020.arxiv.org/abs/1910.13461
[46]Maxwell W Libbrecht and William Stafford Noble. Machine learning applications in genetics and genomics. Nature Reviews Genetics, 16(6):321–332, 2015.DOI
[47]Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2310.16834
[48]David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.DOI
[49]Michael W Macy, Boleslaw K Szymanski, and Janusz A Holyst. The ising model celebrates a century of interdisciplinary contributions. npj Complexity, 1(1):10, 2024.DOI
[50]Aditya Malusare, Harish Kothandaraman, Dipesh Tamboli, Nadia A Lanman, and Vaneet Aggarwal. Understanding the natural language of dna using encoder–decoder foundation models with byte-level precision. Bioinformatics Advances, 4(1):vbae117, 2024.pubmed.ncbi.nlm.nih.gov/38410643
[51]David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, 2020.
[52]Young-Il Moon, Balaji Rajagopalan, and Upamanyu Lall. Estimation of mutual information using kernel density estimators. Physical Review E, 52(3):2318, 1995.pubmed.ncbi.nlm.nih.gov/9963673
[53]Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gülçehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Stefan Riezler and Yoav Goldberg, editors, Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290. Association for Computational Linguistics, August 2016.arxiv.org/abs/1602.06023
[54]Garin Newcomb and Khalid Sayood. Use of average mutual information and derived measures to find coding regions. Entropy, 23(10):1324, 2021.DOI
[55]XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances in Neural Information Processing Systems, 2007.DOI
[56]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. Advances in Neural Information Processing Systems, 38:50608–50646, 2026.arxiv.org/abs/2502.09992
[57]Sebastian Nowozin, Botond Cs eke, and Ryota Tomioka. F-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 271–279, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.arxiv.org/abs/1606.00709
[58]Lars Onsager. Crystal statistics. i. a two-dimensional model with an order-disorder transition. Physical Review, 65(3-4):117, 1944.DOI
[59]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. Advances in neural information processing systems, 2018.arxiv.org/abs/1807.03748
[60]George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1705.07057
[61]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
[62]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.arxiv.org/abs/1201.0490
[63]Assaf Pinchas, Irad Ben-Gal, and Amichai Painsky. A comparative analysis of discrete entropy estimators for large-alphabet problems. Entropy, 26(5):369, 2024.DOI
[64]Stephen M Pizer, E Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. Adaptive histogram equalization and its variations. Computer vision, graphics, and image processing, 39(3):355–368, 1987.
[65]Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, 2019.arxiv.org/abs/1905.06922
[66]Lifeng Qiao, Peng Ye, Yuchen Ren, Weiqiang Bai, Chaoqi Liang, Xinzhu Ma, Nanqing Dong, and Wanli Ouyang. Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna. Advances in Neural Information Processing Systems, 37:66080–66107, 2024.arxiv.org/abs/2412.13716
[67]Arvind Rao, Alfred O Hero, David J States, and James Douglas Engel. Motif discovery in tissue-specific regulatory sequences using directed information. EURASIP Journal on Bioinformatics and Systems Biology, 2007:1–13, 2007.DOI
[68]Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, 2006. ISBN 0-262-18253-X.DOI
[69]Yinuo Ren, Haoxuan Chen, Grant Rotskoff, and Lexing Ying. How discrete and continuous diffusion models meet: Comprehensive analysis of discrete diffusion models via a stochastic integral framework. In International Conference on Learning Representations, volume 2025, pages 42904–42941, 2025.
[70]Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 2020.arxiv.org/abs/2006.12204
[71]Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184, 2024.arxiv.org/abs/2406.07524
[72]Yair Schiff, Chia Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range DNA sequence modeling. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=mk3A5IUdnh8.arxiv.org/abs/2403.03234
[73]Elad Schneidman, Michael J Berry, Ronen Segev, and William Bialek. Weak pairwise correlations imply strongly correlated network states in a neural population. Nature, 440(7087):1007–1012, 2006.pubmed.ncbi.nlm.nih.gov/16625187
[74]Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.
[75]David Sherrington and Scott Kirkpatrick. Solvable model of a spin-glass. Physical review letters, 35(26):1792, 1975.
[76]Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37:103131–103167, 2024.arxiv.org/abs/2406.04329
[77]Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations, 2019a.arxiv.org/abs/1910.06222
[78]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019b.arxiv.org/abs/1907.05600
[79]Karl Stratos. Mutual information maximization for simple and accurate part-of-speech induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.arxiv.org/abs/1804.07849
[80]Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=BYWwSY2G5s.arxiv.org/abs/2211.16750
[81]Ryosuke Takahira, Kumiko Tanaka-Ishii, and Łukasz Dębowski. Entropy rate estimates for natural language—a new extrapolation of compressed large-scale corpora. Entropy, 18(10):364, 2016.DOI
[82]Andrew E Teschendorff and Steve Horvath. Epigenetic ageing clocks: statistical methods and emerging computational challenges. Nature Reviews Genetics, pages 1–19, 2025.DOI
[83]Ramzan Kh Umarov and Victor V Solovyev. Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks. PloS one, 12(2):e0171410, 2017.DOI
[84]AW van der Vaart and Jon A Wellner. Weak convergence and empirical processes with applications to statistics. Journal of the Royal Statistical Society-Series A Statistics in Society, 160(3):596–608, 1997.
[85]Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.arxiv.org/abs/1411.5726
[86]Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272, 2020. doi: 10.1038/s41592-019-0686-2.DOI
[87]Sean Whalen, Jacob Schreiber, William S Noble, and Katherine S Pollard. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics, 23(3):169–181, 2022.DOI
[88]Gerhard Wunder, Benedikt Groß, Rick Fritschek, and Rafael F. Schaefer. A reverse jensen inequality with result with application to mutual information estimation. In 2021 IEEE Information Theory Workshop (ITW), 2021.arxiv.org/abs/2111.06676
[90]Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SkeHuCVFDr.arxiv.org/abs/1904.09675
[91]Shengjia Zhao, Jiaming Song, and Stefano Ermon. A lagrangian perspective on latent variable generative models. In Proc. 34th Conference on Uncertainty in Artificial Intelligence, 2018.