Generalized Entropy Concentration for Counts (1611.00297v7)

Abstract: The phenomenon of entropy concentration provides strong support for the maximum entropy method, MaxEnt, for inferring a probability vector from information in the form of constraints. Here we extend this phenomenon, in a discrete setting, to non-negative integral vectors not necessarily summing to 1. We show that linear constraints that simply bound the allowable sums suffice for concentration to occur even in this setting. This requires a new, `generalized' entropy measure in which the sum of the vector plays a role. We measure the concentration in terms of deviation from the maximum generalized entropy value, or in terms of the distance from the maximum generalized entropy vector. We provide non-asymptotic bounds on the concentration in terms of various parameters, including a tolerance on the constraints which ensures that they are always satisfied by an integral vector. Generalized entropy maximization is not only compatible with ordinary MaxEnt, but can also be considered an extension of it, as it allows us to address problems that cannot be formulated as MaxEnt problems.