Entropy estimation via normalizing flow

Ziqiao Ao, Jinglai Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

541 Downloads (Pure)

Abstract

Entropy estimation is an important problem in information theory and statistical science. Many popular entropy estimators suffer from fast growing estimation bias with respect to dimensionality, rendering them unsuitable for high dimensional problems. In this work we propose a transformbased method for high dimensional entropy estimation, which consists of the following two main ingredients. First by modifying the k-NN based entropy estimator, we propose a new estimator which enjoys small estimation bias for samples that are close to a uniform distribution. Second we design a normalizing flow based mapping that pushes samples toward a uniform distribution, and the relation between the entropy of the original samples and the transformed ones is also derived. As a result the entropy of a given set of samples is estimated by first transforming them toward a uniform distribution and then applying the proposed estimator to the transformed samples. Numerical experiments demonstrate the effectiveness of the method for high dimensional entropy estimation problems.
Original languageEnglish
Title of host publicationProceedings of the 36th AAAI Conference on Artificial Intelligence
Subtitle of host publicationAAAI-22 Technical Track 9 on Reasoning under Uncertainty
Place of PublicationPalo Alto, California USA
PublisherAssociation for the Advancement of Artificial Intelligence
Pages9990-9998
Number of pages9
Volume9
ISBN (Print) 978-1-57735-876-3
DOIs
Publication statusPublished - 28 Jun 2022
Event36th AAAI Conference on Artificial Intelligence - Vancouver, Canada
Duration: 22 Feb 20221 Mar 2022

Publication series

NameProceedings of the AAAI Conference on Artificial Intelligence
Number9
Volume36
ISSN (Print)2159-5399
ISSN (Electronic)2374-3468

Conference

Conference36th AAAI Conference on Artificial Intelligence
Abbreviated titleAAAI-22
Country/TerritoryCanada
CityVancouver
Period22/02/221/03/22

Fingerprint

Dive into the research topics of 'Entropy estimation via normalizing flow'. Together they form a unique fingerprint.

Cite this