### Abstract

Two new information-theoretic methods are introduced for establishing Poisson approximation inequalities. First, using only elementary information-theoretic
techniques it is shown that, when $S_n=\sum_{i=1}^nX_i$ is the sum of the (possibly dependent) binary random variables $X_1,X_2,\ldots,X_n$, with $E(X_i)=p_i$ and $E(S_n)=\la$, then
$$ D(P_{S_n}\|\Pol) \leq \sum_{i=1}^n p_i^2 \;+\;
\Big[\sum_{i=1}^nH(X_i) \,-\,H(X_1,X_2,\ldots, X_n)\Big],$$
where $D(P_{S_n}\| Po(\la))$ is the relative entropy between the distribution of $S_n$ and the Poisson distribution. The first term in this bound measures the individual smallness of the $X_i$ and the second term measures their dependence. A general method is outlined for obtaining corresponding bounds
when approximating the distribution of a sum of general discrete random variables by an infinitely divisible distribution.
Second, in the particular case when the $X_i$ are independent, the following sharper bound is established,
$$ D(P_{S_n}\|\Pol)\leq \frac{1}{\lambda} \sum_{i=1}^n \frac{p_i^3}{1-p_i},$$
and it is also generalized to the case when the $X_i$ are general integer-valued random variables. Its proof is based on the derivation of a subadditivity property for a new discrete version of the Fisher information, and uses a recent logarithmic Sobolev inequality for the Poisson distribution.

Translated title of the contribution | Entropy and the Law of Small Numbers |
---|---|

Original language | English |

Pages (from-to) | 466 - 472 |

Number of pages | 7 |

Journal | IEEE Transactions on Information Theory |

Volume | 51 (2) |

DOIs | |

Publication status | Published - Feb 2005 |

### Bibliographical note

Publisher: IEEE - Institute for Electrical Electronics Engineering## Fingerprint Dive into the research topics of 'Entropy and the Law of Small Numbers'. Together they form a unique fingerprint.

## Cite this

Kontoyiannis, I., Harremoes, P., & Johnson, OT. (2005). Entropy and the Law of Small Numbers.

*IEEE Transactions on Information Theory*,*51 (2)*, 466 - 472. https://doi.org/10.1109/TIT.2004.840861