Skip to main content
SpringerLink
Log in

A Markov Decision Process based flow assignment framework for heterogeneous network access

  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

We consider a scenario where devices with multiple networking capabilities access networks with heterogeneous characteristics. In such a setting, we address the problem of efficient utilization of multiple access networks by devices via optimal assignment of traffic flows with given utilities to different networks. We develop and analyze a device middleware functionality that monitors network characteristics and employs a Markov Decision Process (MDP) based control scheme that in conjunction with stochastic characterization of the available bit rate and delay of the networks generates an optimal policy for allocation of flows to different networks. The optimal policy maximizes, under available bit rate and delay constraints on the access networks, a discounted reward which is a function of the flow utilities. The flow assignment policy is periodically updated and is consulted by the flows to dynamically perform network selection during their lifetimes. We perform measurement tests to collect traces of available bit rate and delay characteristics on Ethernet and WLAN networks on a work day in a corporate work environment. We implement our flow assignment framework in ns-2 and simulate the system performance for a set of elastic video-like flows using the collected traces. We demonstrate that the MDP based flow assignment policy leads to significant enhancement in the QoS provisioning (higher rate allocation, lower packet delays and packet loss rates) for the flows and better access network utilization, as compared to policies that allocate flows to different networks using greedy approaches or heuristics like average available bit rate on the networks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Vidales, P., Baliosion, J., Serrat, J., Mapp, G., Stejano, F., & Hopper, A. (2005). Autonomic system for mobility support in 4G networks. IEEE Journal on Selected Areas in Communications, 23(12), 2288–2304.

    Article  Google Scholar 

  2. IEEE 802.21. http://www.ieee802.org/21/.

  3. Cuevas, A., Moreno, J. I., Vidales, P., & Einsiedler, H. (2006). The IMS platform: A solution for next generation network operators to be more than bit pipes. In IEEE Communications Magazine, Issue on Advances of Service Platform Technologies.

  4. Bertsekas, D. (2001). Dynamic programming and optimal control (Vol. 2, 2nd ed.). Belmont, MA: Athena Scientific.

    MATH  Google Scholar 

  5. The Network Simulator (ns-2). http://www.isi.edu/ns/.

  6. Yaiche, H., Mazumdar, R., & Rosenburg, C. (2000). A game theoretic framework for bandwidth allocation and pricing in broadband networks. IEEE/ACM Transactions on Networking (TON), 8(5), 667–678.

    Article  Google Scholar 

  7. Alpcan, T., & Başar, T. (2005). A utility-based congestion control scheme for internet-style networks with delay. IEEE Transactions on Networking, 13(6), 1261–1274.

    Article  Google Scholar 

  8. Alpcan, T., & Başar, T. (2003). Global stability analysis of an end-to-end congestion control scheme for general topology networks with delay. In Proc. of the 42nd IEEE Conference on Decision and Control (pp. 1092–1097). Maui, HI.

  9. Chebrolu, K., & Rao, R. (2002). Communication using multiple wireless interfaces. In Proc. IEEE Wireless Communications and Networking Conference (WCNC 2002) (Vol. 1, pp. 327–331).

  10. Fodor, G., Furuksar, A., & Lundsjo, J. (2004). On access selection techniques in always best connected networks. In Proc. ITC Specialist Seminar on Performance Evaluation of Wireless and Mobile Systems.

  11. Zemlianov, A., & de Veciana, G. (2005). Cooperation and decision-making in a Wireless multi-provider setting. In Proc. IEEE INFOCOM (pp. 1–14).

  12. Shakkottai, S., Altman, E., & Kumar, A. (2005). The case for non-cooperative multihoming of users to access points in IEEE 802.11 WLANs. In Proc. IEEE INFOCOM.

  13. Furuksar, A., & Zander, J. (2005). Multiservice allocation for multiaccess wireless systems. IEEE Transactions on Wireless Communications, 4, 174–184.

    Article  Google Scholar 

  14. Mariz, D., Cananea, I., Sadok, D., & Fodor, G. (2006). Simulative analysis of access selection algorithms for multi-access networks. In Proc. IEEE Internation Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM 2006).

  15. Thompson, N., He, G., & Luo, H. (2006). Flow scheduling for end-host multihoming. In Proc. IEEE INFOCOM.

  16. Kumar, D., Altman, E., & Kelif, J.-M. (2007). Globally optimal user-network association in an 802.11 WLAN and 3G UMTS hybrid cell. In Proc. 20th International Telegraph Congress. Ottawa, Canada.

  17. Alpcan, T., Singh, J. P., & Başar, T. (2007). A robust flow control framework for heterogeneous network access. In Proc. 5th International Symposium on Modeling and Optimization in Mobile, Ad-hoc, and Wireless Networks (WiOpt).

  18. Zhu, X., Agrawal, P., Singh, J. P., Alpcan, T., & Girod, B. (2007). Rate allocation for multi-user video streaming over heterogeneous access networks. In Proc. ACM Multimedia.

  19. Singh, J. P., Alpcan, T., Zhu, X., & Agrawal, P. (2007). Towards heterogeneous network convergence: Policies and middleware architecture for efficient flow assignment, rate allocation and rate control for multimedia applications. In Workshop on Middleware for Next-generation Converged Networks and Application (MNCNA), ACM/IFIP/USENIX 8th International Middleware Conference.

  20. Guo, F., Chen, J., Li, W., & Chiueh, T. (2004). Experiences in building a multihoming load balancing system. In: Proc. IEEE INFOCOM.

  21. Konrad, A., Zhao, B., Joseph, A., & Ludwig, R. (2003). A markov-based channel model algorithm for wireless networks. Wireless Networks (pp. 189–199).

  22. Salmatian, K., & Vaton, S. (2001). Hidden Markov Modeling for network communication channels. In Proc. ACM SIGMETRICS.

  23. Froyland, G. (2001). Extracting dynamical behaviour via Markov models. In Nonlinear Dynamics and Statistics: Proceedings, Newton Institute, Cambridge, 1998 (pp. 283–324). Birkhauser.

  24. StuhlMuller, K., Farber, N., Link, M., & Girod, B. (2000). Analysis of video transmission over lossy channels. IEEE Journal on Selected Areas in Communication, 18(6), 1012–1032.

    Article  Google Scholar 

  25. Navratil, J., & Cottrell, R. L., Abing. http://www-iepm.slac.stanford.edu/tools/abing/.

  26. Laboratory, U. N. R., CapProbe. http://www.cs.ucla.edu/NRL/CapProbe/.

Download references

Acknowledgements

Research Supported by Deutsche Telekom AG. An earlier version of the work appeared in the IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks, June 18–21, 2007, Helsinki, Finland.

Author information

Authors and Affiliations

  1. Deutsche Telekom Laboratories, 5050 El Camino Real, Los Altos, CA, 94022, USA

    Jatinder Pal Singh

  2. Deutsche Telekom Laboratories, Ernst-Reuter Platz 7, 10587, Berlin, Germany

    Tansu Alpcan

  3. Department of Electrical Engineering, Stanford University, Stanford, CA, 94305, USA

    Piyush Agrawal

  4. Lehman Brothers Inc., Mumbai, India

    Varun Sharma

Authors
  1. Jatinder Pal Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tansu Alpcan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Piyush Agrawal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Varun Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jatinder Pal Singh.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Singh, J.P., Alpcan, T., Agrawal, P. et al. A Markov Decision Process based flow assignment framework for heterogeneous network access. Wireless Netw 16, 481–495 (2010). https://doi.org/10.1007/s11276-008-0148-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-008-0148-8

Keywords

Search

Navigation