Skip to main content
SpringerLink
Log in

Some Information Theoretic Ideas Useful in Statistical Inference

  • Published:
Methodology and Computing in Applied Probability Aims and scope Submit manuscript

Abstract

In this paper we discuss four information theoretic ideas and present their implications to statistical inference: (1) Fisher information and divergence generating functions, (2) information optimum unbiased estimators, (3) information content of various statistics, (4) characterizations based on Fisher information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • A. A. Abdushukurov, and L. V. Kim, “Lower Cramer–Rao and Bhattacharya bounds for randomly censored observations,” Journal of Soviet Mathematics vol. 38 pp. 2171–2185, 1987.

    Article  MATH  Google Scholar 

  • Z. Abo-Eleneen, and H. N. Nagaraja, “Fisher information in an order statistic and its concomitant,” Annals of the Institute of Statistical Mathematics vol. 54 pp. 667–680, 2002.

    Article  MATH  MathSciNet  Google Scholar 

  • J. Aczel, “Characterizing information measures: approaching the end of an era.” In B. Bouchon and R. R. Yager (eds.), Uncertainty on Knowledge-Based Systems. Lecture Notes in Computer Science, pp. 359–384, Springer: Berlin Heidelberg New York, 1986.

    Google Scholar 

  • J. Aczel, and Z. Daroczy, On Measures of Information and their Characterizations, Academic: New York, 1975.

    MATH  Google Scholar 

  • B. Arnold, N. Balakrishnan, and H. N. Nagaraja, Records, Wiley: New York, 1998.

    MATH  Google Scholar 

  • M. J. Bayarri, M. H. DeGroot, and P. K. Goel, “Truncation, information and the coefficient of variation.” In L. Gleser, M. Perlman, S. J. Press and A. Sampson (eds.), Contributions to Probability and Statistics. Essays in Honor of Ingram Olkin, pp. 412–428, Springer: Berlin Heidelberg New York, 1989.

    Google Scholar 

  • R. Beran, “Minimum Hellinger distance estimates for parametric models,” The Annals of Statistics vol. 5 pp. 445–463, 1977.

    MATH  MathSciNet  Google Scholar 

  • D. Blackwell, Comparison of experiments. Proceedings of the second Berkeley Symposium on Mathematical Statistics and Probability, University of California Press, pp. 93–102, 1951.

  • T. M. Cover, and J. A. Thomas, Elements of Information Theory, Wiley: New York, 1991.

    MATH  Google Scholar 

  • N. Cressie, and T. R. C. Read, “Multinomial goodness of fit tests,” Journal of the Royal Statistical Society, Series B vol. 46 pp. 440–464, 1984.

    MATH  MathSciNet  Google Scholar 

  • B. Ebanks, P. Sahoo, and W. Sander, Characterizations of Information Measures, World Scientific: Singapore, 1998.

    Google Scholar 

  • B. Efron, and I. M. Johnstone, “Fisher’s information in terms of the hazard rate,” The Annals of Statistics vol. 18 pp. 38–62, 1990.

    MATH  MathSciNet  Google Scholar 

  • K. Ferentinos, and T. Papaioannou, “New parametric measures of information,” Information and Control vol. 51 pp. 193–208, 1981.

    Article  MATH  MathSciNet  Google Scholar 

  • K. Ferentinos, and T. Papaioannou, “Information in experiments and sufficiency,” Journal of Statistical Planning and Inference vol. 6 pp. 309–317, 1982.

    Article  MATH  MathSciNet  Google Scholar 

  • I. Gertsbakh, and A. Kagan, “Characterization of the Weibull distribution by properties of the Fisher information under type I censoring,” Statistics and Probability Letters vol. 42 pp. 99–105, 1999.

    Article  MATH  MathSciNet  Google Scholar 

  • S. Golomb, “The information generating function of a probability distribution,” IEEE Transactions on Information Theory vol. IT-12 pp. 75–77, 1966.

    Article  MathSciNet  Google Scholar 

  • S. Guiasu, and C. Reischer, “The relative information generating function,” Information Sciences vol. 35 pp. 235–241, 1985.

    Article  MATH  MathSciNet  Google Scholar 

  • R. D. Gupta, R. C. Gupta, and P. G. Sankaran, “Some characterization results based on factorization of the (reversed) hazard rate function,” Communications in Statistics—Theory and Methods vol. 33 pp. 3009–3031, 2004.

    Article  MATH  MathSciNet  Google Scholar 

  • G. Hofmann, and H. N. Nagaraja, “Fisher information in record data,” Metrika vol. 57 pp. 177–193, 2003.

    Article  MathSciNet  Google Scholar 

  • M. Hollander, F. Proschan, and J. Sconing, “Measuring information in right-censored models,” Naval Research Logistics vol. 34 pp. 669–681, 1987.

    MATH  MathSciNet  Google Scholar 

  • M. Hollander, F. Proschan, and J. Sconing, “Information, censoring and dependence,” Institute of Mathematical Statistics, Lecture Notes—Monograph Series, Topics in Statistical Dependence vol. 16 pp. 257–268, 1990.

    MathSciNet  Google Scholar 

  • S. Iyengar, P. Kvam, and H. Singh, “Fisher information in weighted distributions,” The Canadian Journal of Statistics vol. 27 pp. 833–841, 1999.

    MATH  MathSciNet  Google Scholar 

  • A. Kagan, “A discrete version of the Stam inequality and a characterization of the Poisson distribution,” Journal of Statistical Planning and Inference vol. 92 pp. 7–12, 2001.

    Article  MATH  MathSciNet  Google Scholar 

  • J. N. Kapur, Maximum-Entropy Models in Science and Engineering, Wiley: New York, 1989.

    MATH  Google Scholar 

  • S. Kullback, Information Theory and Statistics, Peter Smith: USA, 1959.

    MATH  Google Scholar 

  • W. Nelson, Applied Life Data Analysis, Wiley: New York, 1982.

    Book  MATH  Google Scholar 

  • T. Papaioannou, “On distances and measures of information: a case of diversity.” In C. A. Charalambides, M. V. Koutras and N. Balakrishnan (eds.), Probability and Statistical Models with applications, pp. 503–515, Chapman and Hall: London, UK, 2001.

    Google Scholar 

  • T. Papaioannou, and K. Ferentinos, “On two forms of Fisher’s measure of information,” Communications in Statistics—Theory and Methods vol. 34 pp. 1461–1470, 2005.

    MATH  MathSciNet  Google Scholar 

  • T. Papaioannou, and O. Kempthorne, On statistical information theory and related measures of information. ARL. Tech. Report N o 71-0059, Aerospace Research Laboratories, Wright-Patterson A.F.B., Ohio, 1971.

  • V. Papathanasiou, “Some characteristic properties of the Fisher information matrix via Cacoullos-type inequalities,” Journal of Multivariate Analysis vol. 44 pp. 256–265, 1993.

    Article  MATH  MathSciNet  Google Scholar 

  • S. Park, and G. Zheng, “Equal Fisher information in order statistics,” Sankhya: The Indian Journal of Statistics, Series B vol. 66 pp. 20–34, 2004.

    MathSciNet  Google Scholar 

  • T. Rothenberg, “Identification in parameter models,” Econometrica vol. 39 pp. 577–591, 1971.

    Article  MATH  MathSciNet  Google Scholar 

  • A. Stam, “Some inequalities satisfied by the quantities of information of Fisher and Shannon,” Information and Control vol. 2 pp. 101–112, 1959.

    Article  MathSciNet  Google Scholar 

  • Ch. Tsairidis, K. Ferentinos, and T. Papaioannou, “Information and random censoring,” Information Sciences vol. 92 pp. 159–174, 1996.

    Article  MATH  MathSciNet  Google Scholar 

  • Ch. Tsairidis, K. Zografos, K. Ferentinos, and T. Papaioannou, “Information in quantal response data and random censoring,” Annals of the Institute of Statistical Mathematics vol. 53 pp. 528–542, 2001.

    Article  MATH  MathSciNet  Google Scholar 

  • I. Vincze, “Some questions concerning the probabilistic concept of information,” AMS and IMS Selected Translations vol. V pp. 373–380, 1965.

    Google Scholar 

  • P. W. Vos, “Minimum f-divergence estimators and quasi-likelihood functions,” Annals of the Institute of Statistical Mathematics vol. 44 pp. 261–279, 1992.

    Article  MATH  MathSciNet  Google Scholar 

  • R. Wijman, “On the attainment of the Cramer–Rao lower bound,” The Annals of Statistics vol. 1 pp. 538–542, 1973.

    MathSciNet  Google Scholar 

  • S. Zacks, The Theory of Statistical Inference, Wiley: New York, 1971.

    Google Scholar 

  • G. Zheng, “A characterization of the factorization of hazard function by the Fisher information under Type II censoring with application to the Weibull family,” Statistics and Probability Letters vol. 52 pp. 249–253, 2001.

    Article  MATH  MathSciNet  Google Scholar 

  • G. Zheng, and J. L. Gastwirth, “Where is the Fisher information in an ordered sample?,” Statistica Sinica vol. 10 pp. 1267–1280, 2000.

    MATH  MathSciNet  Google Scholar 

  • G. Zheng, and J. L. Gastwirth, “On the Fisher information in randomly censored data,” Statistics and Probability Letters vol. 52 pp. 421–426, 2001.

    Article  MATH  MathSciNet  Google Scholar 

  • G. Zheng, and J. L. Gastwirth, “Do tails of symmetric distributions contain more Fisher information about the scale parameter?,” Sankhya: The Indian Journal of Statistics, Series B vol. 64 pp. 289–300, 2002.

    MathSciNet  Google Scholar 

  • G. Zheng, and J. L. Gastwirth, “Fisher information in ordered randomly censored data with applications to characterizations problems,” Statistica Sinica vol. 13 pp. 507–517, 2003.

    MATH  MathSciNet  Google Scholar 

  • K. Zografos, and K. Ferentinos, “An information theoretic argument for the validity of the exponential model,” Metrika vol. 41 pp. 109–119, 1994.

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Statistics and Insurance Science, University of Piraeus, 18534, Piraeus, Greece

    Takis Papaioannou

  2. Department of Mathematics, University of Ioannina, 45110, Ioannina, Greece

    Kosmas Ferentinos

  3. Department of Social Administration, Democritus University of Thrace, 69100, Komotini, Greece

    Charalampos Tsairidis

Authors
  1. Takis Papaioannou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kosmas Ferentinos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charalampos Tsairidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Takis Papaioannou.

Additional information

This paper was written while the first author was visiting the University of Cyprus, Department of Mathematics and Statistics.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Papaioannou, T., Ferentinos, K. & Tsairidis, C. Some Information Theoretic Ideas Useful in Statistical Inference. Methodol Comput Appl Probab 9, 307–323 (2007). https://doi.org/10.1007/s11009-007-9017-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11009-007-9017-7

Keywords

AMS 2000 Subject Classification

Search

Navigation