Skip to main content
SpringerLink
Log in

Capability-Aware Information Aggregation in Peer-to-Peer Grids

Methods, Architecture, and Implementation

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

Information aggregation is the process of summarizing information across the nodes of a distributed system. We present a hierarchical information aggregation system tailored for Peer-to-Peer Grids which typically exhibit a high degree of volatility and heterogeneity of resources. Aggregation is performed in a scalable yet efficient way by merging data along the edges of a logical self-healing tree with each inner node providing a summary view of the information delivered by the nodes of the corresponding subtree. We describe different tree management methods suitable for high-efficiency and high-scalability scenarios that take host capability and stability diversity into account to attenuate the impact of slow and/or unstable hosts. We propose an architecture covering all three phases of the aggregation process: Data gathering through a highly extensible sensing framework, data aggregation using reusable, fully isolated reduction networks, and application-sensitive data delivery using a broad range of propagation strategies. Our solution combines the advantages of approaches based on Distributed Hash Tables (DHTs) (i.e., load balancing and self-maintenance) and hierarchical approaches (i.e., respecting administrative boundaries and resource limitations). Our approach is integrated into our Peer-to-Peer Grid platform Cohesion. We substantiate its effectiveness through performance measurements and demonstrate its applicability through a graphical monitoring solution leveraging our aggregation system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Internet Systems Consortium: Internet domain survey—host count. http://www.isc.org/index.pl?/ops/ds/ (2008), January

  2. van Renesse, R.: The importance of aggregation. In: Future Directions in Distributed Computing. Lecture Notes in Computer Science, vol. 2584, pp. 87–92. Springer, Heidelberg (2003), April

    Chapter  Google Scholar 

  3. Risson, J., Moorsa, T.: Survey of research towards robust peer-to-peer networks: Search methods. Comput. Netw. 50(17), 3485–3521 (2006)

    Article  MATH  Google Scholar 

  4. Schulz, S., Blochinger, W., Held, M., Dangelmayr, C.: COHESION—a microkernel based desktop grid platform for irregular task-parallel applications. Future Gener. Comput. Syst. 24(5), 354–370 (2008)

    Article  Google Scholar 

  5. Montresor, A., Jelasity, M., Babaoglu, O.: Robust aggregation protocols for large-scale overlay networks. In: Proceedings of the 2004 International Conference on Dependable Systems and Networks (DSN’04), pp. 19–28 (2004)

  6. Massie, M.L., Chun, B.N., Culler, D.E.: The Ganglia distributed monitoring system: design, implementation, and experience. Parallel Comput. 30(7), 817–840 (2004)

    Article  Google Scholar 

  7. Van Renesse, R., Birman, K.P., Vogels, W.: Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining. ACM Trans. Comput. Syst. 21(2), 164–206 (2003)

    Article  Google Scholar 

  8. Yalagandula, P., Dahlin, M.: A scalable distributed information management system. In: Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications, pp. 379–390 (2004)

  9. Madden, S., Franklin, M.J., Hellerstein, J.M., Hong, W.: Tag: a tiny aggregation service for ad-hoc sensor networks. In: OSDI ’02: Proceedings of the 5th symposium on Operating systems design and implementation, pp. 131–146. ACM, New York, NY, USA (2002)

    Google Scholar 

  10. Kondo, D., Taufer, M., Brooks, C.L., Casanova, H., Chien, A.A.: Characterizing and evaluating desktop grids: An empirical study. In: Proc. of 18th International Parallel and Distributed Processing Symposium (IPDPS’04), p. 26. Sante Fe, New Mexico (2004)

  11. Anderson, D.P.: BOINC: a system for public-resource computing and storage. In: Proc. of the 5th IEEE/ACM International Workshop on Grid Computing, pp. 365–372. Pittsburgh, USA (2004), November

  12. Chien, A., Calder, B., Elbert, S., Bhatia, K.: Entropia: architecture and performance of an enterprise desktop grid system. J. Parallel Distrib. Comput. 63, 597–610 (2003)

    Article  Google Scholar 

  13. Shudo, Y. T. K., Sekiguchi, S.: P3: P2P-based middleware enabling transfer and aggregation of computational resources. In: Proc. Cluster Computing and Grid 2005 (Fifth Int’l Workshop on Global and Peer-to-Peer Computing), pp. 259–265. Cardiff, UK (2005)

  14. Verbeke, J., Nadgir, N., Ruetsch, G., Sharapov, I.: Framework for peer-to-peer distributed computing in a heterogeneous, decentralized environment. In: Proc. of the Third International Workshop on Grid Computing (GRID ’02), pp. 1–12. Springer, London, UK (2002)

    Google Scholar 

  15. Anderson, D.P., Fedak, G.: The computational and storage potential of volunteer computing. In: Proc. of the Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2006), pp. 73–80. Singapore (2006)

  16. Foster, I., Iamnitchi, A.: On death, taxes, and the convergence of peer-to-peer and grid computing. In: 2nd International Workshop on Peer-to-Peer Systems (IPTPS 03, pp. 118–128 (2003)

  17. Blochinger, W., Dangelmayr, C., Schulz, S.: Aspect-oriented parallel discrete optimization on the cohesion desktop grid platform. In: Proc. of the Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2006), pp. 49–56. Singapore (2006), May

  18. Schulz, S., Blochinger, W.: An integrated approach for managing peer-to-peer desktop grid systems. In: Proc. of the Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2007), Rio de Janeiro, Brazil, (2007), May

  19. OSGi Alliance: OSGiTM—The Dynamic Module System for JavaTM. http://www.osgi.org. Accessed February 2008

  20. Ganesh, A. J., Kermarrec, A.-M., Massoulié, L.: Peer-to-peer membership management for gossip-based protocols. IEEE Trans. Comput., 52(2), 139–149 (2003)

    Article  Google Scholar 

  21. Das, A., Gupta, I., Motivala, A.: SWIM: scalable weakly-consistent infection-style process group membership protocol. In: Proc. of the International Conference on Dependable Systems and Networks (DSN 2002), 23-26 June 2002, Bethesda, MD, USA, pp. 303–312. IEEE Computer Society, (2002), February

  22. Kurzyniec, D., Wrzosek, T., Drzewiecki, D., Sunderam, V.: Towards self-organizing distributed computing frameworks: the H2O approach. Parallel Process. Lett., 13(2), 273–290 (2003)

    Article  MathSciNet  Google Scholar 

  23. Bhagwan, R., Savage, S., Voelker, G.: Understanding availability. In: Proc. of the 2nd International Workshop on Peer-to-Peer Systems (IPTPS ’03), pp. 256–267, (2003), February

  24. Wolski, R., Spring, N., Hayes, J.: Predicting the cpu availability of time-shared unix systems on the computational grid. In: Proc. of the 8th IEEE International Symposium on High Performance Distributed Computing, pp. 105–112. Washington, DC, USA (1999)

  25. Standard Performance Evaluation Corporation: Spec cpu2006 results, August (2008)

  26. Demers, A., Greene, D., Houser, C., Irish, W., Larson, J., Shenker, S., Sturgis, H., Swinehart, D., Terry, D.: Epidemic algorithms for replicated database maintenance. SIGOPS Oper. Syst. Rev., 22(1):8–32, January (1988)

    Article  Google Scholar 

  27. Birman, K.P., Hayden, M., Ozkasap, O., Xiao, Z., Budiu, M., Minsky, Y.: Bimodal multicast. ACM Trans. Comput. Syst., 17(2), 41–88 (1999)

    Article  Google Scholar 

  28. Hyperic Inc.: SIGAR (System Information Gatherer and Reporter). http://www.hyperic.com/products/sigar.html. Accessed February 2008

  29. Sun Microsystems Inc.: Java Management Extensions (JMX) Remote API (JSR-160). http://jcp.org/en/jsr/detail?id=160. Accessed February 2008

  30. Distributed Management Task Force Inc.: Common Information Model (CIM) Standards. http://www.dmtf.org/standards/cim/. Accessed February 2008

  31. Wolski, R., Spring, N., Hayes, J.: The network weather service: a distributed resource performance forecasting service for metacomputing. Future Gener. Comput. Syst., 15(5–6), 757–768 (1999)

    Article  Google Scholar 

  32. Waldspurger, C. A., Weihl, W. E.: Lottery scheduling: flexible proportional-share resource management. In: Operating Systems Design and Implementation, pp. 1–11 (1994)

  33. Case, J.D., Fedor, M., Schoffstall, M.L., Davin, J.: Simple Network Management Protocol (SNMP). RFC 1157 (Historic) (1990), May

  34. Markovic, S.: JRobin, a Java port of RRDTool. https://rrd4j.dev.java.net. Accessed February 2008

  35. Cohesion Project Website. http://www.cohesion.de. Accessed February 2008

  36. yWorks GmbH: yFiles. http://www.yworks.com. Accessed February 2008

  37. Kempe, D., Dobra, A., Gehrke, J.: Gossip-based computation of aggregate information. In: FOCS ’03: Proceedings of the 44th Annual IEEE Symposium on Foundations of Computer Science, pp. 482–491, Washington, DC, USA. IEEE Computer Society (2003)

    Chapter  Google Scholar 

  38. Sun Microsystems Inc.: Java Management Extensions (JMX) Specification (JSR-3). http://jcp.org/en/jsr/detail?id=3. Accessed February 2008

Download references

Author information

Authors and Affiliations

  1. Wilhelm-Schickard-Institute, University of Tübingen, Sand 14, D-72076, Tübingen, Germany

    Sven Schulz, Wolfgang Blochinger & Hannes Hannak

Authors
  1. Sven Schulz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wolfgang Blochinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hannes Hannak
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sven Schulz.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schulz, S., Blochinger, W. & Hannak, H. Capability-Aware Information Aggregation in Peer-to-Peer Grids. J Grid Computing 7, 135–167 (2009). https://doi.org/10.1007/s10723-008-9114-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-008-9114-z

Keywords

Search

Navigation