TY - GEN
T1 - Entropy estimation via normalizing flow
AU - Ao, Ziqiao
AU - Li, Jinglai
PY - 2022/6/28
Y1 - 2022/6/28
N2 - Entropy estimation is an important problem in information theory and statistical science. Many popular entropy estimators suffer from fast growing estimation bias with respect to dimensionality, rendering them unsuitable for high dimensional problems. In this work we propose a transformbased method for high dimensional entropy estimation, which consists of the following two main ingredients. First by modifying the k-NN based entropy estimator, we propose a new estimator which enjoys small estimation bias for samples that are close to a uniform distribution. Second we design a normalizing flow based mapping that pushes samples toward a uniform distribution, and the relation between the entropy of the original samples and the transformed ones is also derived. As a result the entropy of a given set of samples is estimated by first transforming them toward a uniform distribution and then applying the proposed estimator to the transformed samples. Numerical experiments demonstrate the effectiveness of the method for high dimensional entropy estimation problems.
AB - Entropy estimation is an important problem in information theory and statistical science. Many popular entropy estimators suffer from fast growing estimation bias with respect to dimensionality, rendering them unsuitable for high dimensional problems. In this work we propose a transformbased method for high dimensional entropy estimation, which consists of the following two main ingredients. First by modifying the k-NN based entropy estimator, we propose a new estimator which enjoys small estimation bias for samples that are close to a uniform distribution. Second we design a normalizing flow based mapping that pushes samples toward a uniform distribution, and the relation between the entropy of the original samples and the transformed ones is also derived. As a result the entropy of a given set of samples is estimated by first transforming them toward a uniform distribution and then applying the proposed estimator to the transformed samples. Numerical experiments demonstrate the effectiveness of the method for high dimensional entropy estimation problems.
UR - https://www.aaai.org/Library/AAAI/aaai-library.php
UR - https://aaai.org/Press/Proceedings/proceedings.php
UR - https://aaai.org/Conferences/AAAI-22/
UR - https://ojs.aaai.org//index.php/aimagazine/about/submissions
UR - https://www.aaai.org/Library/AAAI/aaai22contents.php
U2 - 10.1609/aaai.v36i9.21237
DO - 10.1609/aaai.v36i9.21237
M3 - Conference contribution
SN - 978-1-57735-876-3
VL - 9
T3 - Proceedings of the AAAI Conference on Artificial Intelligence
SP - 9990
EP - 9998
BT - Proceedings of the 36th AAAI Conference on Artificial Intelligence
PB - Association for the Advancement of Artificial Intelligence
CY - Palo Alto, California USA
T2 - 36th AAAI Conference on Artificial Intelligence
Y2 - 22 February 2022 through 1 March 2022
ER -