Skip to main content
SpringerLink
Log in

Coflow scheduling and placement for packet-switched optical datacenter networks

  • Original Paper
  • Published:
Photonic Network Communications Aims and scope Submit manuscript

Abstract

Data-parallel computing applications (DPCAs) (e.g., MapReduce, web search, etc.) are driving the need of scalable, low-latency, high-speed, and energy-efficient datacenters, because a DPCA consists of a series of heavy-computation stages within a datacenter, and each stage contains multiple parallel flows that must be completed before next stage starts, referred to as “Coflow”. These parallel flows are grouped as a Coflow. Coflow is a networking abstraction to convey application-level communication requirements by exposing rich semantics of DPCAs to underlying networks, e.g., latency of data transmission between two computation stages, known as “Coflow Completion Time” (CCT). Packet-switched optical network (PSON) is a practical intra-datacenter interconnect solution for DPCAs, as it is designed as a low-complexity and scalable one-stage switching architecture, using advanced optical networking technologies, such as Arrayed Waveguide Grating Routers and wavelength-division multiplexing. In this work, we study how to minimize CCT in PSON-enabled datacenters by placing senders and receivers of Coflows to proper transceiver nodes and scheduling data transmission wisely, for which we propose a Coflow-aware placement and scheduling algorithm, consisting of Min-Priority placement algorithm and Priority-aware scheduling algorithm. They are designed to cooperate with each other to jointly minimize CCT. Numerical simulations demonstrate the benefits of joint design of Coflow placement and scheduling algorithm, compared to state-of-the-art scheduling and placement algorithms designed without correlation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Index, C.G.C.: Forecast and Methodology, 2015–2020. Cisco Systems Inc, San Jose (2016)

    Google Scholar 

  2. Ananthanarayanan, G., et al.: Coordinated memory caching for parallel jobs. NSDI 20 12  (2012)

  3. Zahariaetal, M., et al.: Resilient distributed datasets: a fault-tolerant P2 abstraction for in-memory cluster computing. NSDI 2 (2012)

  4. Chowdhury, M., et al.: Coflow: a networking abstraction for cluster applications. In: 11th ACM Workshop on Hot Topics in Networks (2012)

  5. Dogar, F., et al.: Decentralized task-aware scheduling for datacenter networks. ACM SIGCOMM Comput. Commun. Rev. 44(4), 431–442 (2014)

    Article  Google Scholar 

  6. Chowdhury, M., et al.: Efficient coflow scheduling with varys. ACM SIGCOMM Comput. Commun. 44(4), 443–454 (2014)

    Article  Google Scholar 

  7. Chowdhury, M., et al.: Efficient coflow scheduling without prior knowledge. ACM SIGCOMM Comput. Commun. Rev. 45(4), 393–406 (2015)

    Article  Google Scholar 

  8. Huang, X., et al.: Exploiting inter-flow relationship for coflow placement in datacenters. In: Proceedings of the 1st Asia-Pacific Workshop on Networking, pp. 113–119, Hong Kong, China (2017)

  9. Munir, A., et al.: Network scheduling aware task placement in datacenters. In: Proceedings of the 12th International on Conference on Emerging Networking Experiments and Technologies, Irvine, pp. 221–235 (2016)

  10. https://goo.gl/F5GZ8g

  11. Kachris, C., et al.: A survey on optical interconnects for data centers. IEEE Commun. Surv. Tutor. 14(4), 1021–1036 (2012)

    Article  Google Scholar 

  12. R. Proietti, et al. “40 Gb∕s 8 × 8 low-latency optical switch for data centers,” Optical Fiber Communication Conf., Los Angeles, CA, 2011.

  13. Xi, K., et al.: Petabit optical switch for datacenter networks. Tech. Rep., Polytechnic Institute of New York University, New York (2010)

  14. Liboiron-Ladouceur, O., et al.: Energy-efficient design of a scalable optical multiplane interconnection architecture. IEEE J. Sel. Top. Quantum Electron. 17(2), 377–383 (2010)

    Article  Google Scholar 

  15. Wang, L., et al.: Scheduling with machine-learning-based flow detection for packet-switched optical data center networks. J. Opt. Commun. Netw. 10(4), 365–375 (2018)

    Article  Google Scholar 

  16. Chowdhury, M., et al.: Leveraging endpoint flexibility in data-intensive clusters. ACM SIGCOMM Comput. Commun. Rev. 43(4), 231–242 (2013)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIP) [B0117-16-1008, Development of datacenter Optical Networking Core Technologies for Photonic Frame-based Packet Switching]. It was also supported in part by National Science Foundation Grant No. 1716945.

Author information

Authors and Affiliations

  1. University of California Davis, Davis, CA, USA

    Lin Wang, Massimo Tornatore & Biswanath Mukherjee

  2. Facebook, Inc, Cambridge, MA, USA

    Xinbo Wang

  3. Politecnico Di Milano, Milano, Italy

    Massimo Tornatore

  4. Electronics and Telecommunications Research Institute, Gwangju, Korea

    Kwangjoon Kim

Authors
  1. Lin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinbo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Massimo Tornatore
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kwangjoon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Biswanath Mukherjee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Lin Wang or Massimo Tornatore.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, L., Wang, X., Tornatore, M. et al. Coflow scheduling and placement for packet-switched optical datacenter networks. Photon Netw Commun 43, 156–164 (2022). https://doi.org/10.1007/s11107-021-00958-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11107-021-00958-4

Keywords

Search

Navigation