Ki Suh Lee
ABSTRACT
In this paper, we present Datacenter Time Protocol (DTP), a clock synchronization protocol that does not use packets at all, but is able to achieve nanosecond precision. In essence, DTP uses the physical layer of network devices to implement a decentralized clock synchronization protocol. By doing so, DTP eliminates most non-deterministic elements in clock synchronization protocols. Further, DTP uses control messages in the physical layer for communicating hundreds of thousands of protocol messages without interfering with higher layer packets. Thus, DTP has virtually zero overhead since it does not add load at layers 2 or higher layers. It does require replacing network devices, which can be done incrementally. We demonstrate that the precision provided by DTP is bounded by 25.6 nanoseconds for directly connected nodes, and in general, is bounded by 4TD where D is the longest distance between any two servers in a network in terms of number of hops and T is the period of the fastest clock (≈ 6.4ns). Moreover, in software, a DTP daemon can access the DTP clock with usually better than 4T (≈ 25.6ns) precision. As a result, the end-to-end precision can be better than 4T D + 8T nanoseconds. By contrast, the precision of the state of the art protocol is not bounded: The precision is hundreds of nanoseconds when a network is idle and can decrease to hundreds of microseconds when a network is heavily congested.
Supplemental Material
- 1.Bluespec. www.bluespec.com.Google Scholar
- 2.Broadcom. http://http://www.broadcom.com/products/Switching/Data-Center.Google Scholar
- 3.DE5-Net FPGA development kit. http://de5-net.terasic.com.tw.Google Scholar
- 4.Endace DAG network cards. http://www.endace.com/endace-dag-high-speed-packet-capture-cards.html.Google Scholar
- 5.Exablaze. https://exablaze.com/.Google Scholar
- 6.Fibre channel. http://fibrechannel.org.Google Scholar
- 7.Highly accurate time synchronization with ConnectX-3 and Timekeeper. http://www.mellanox.com/pdf/whitepapers/WP_Highly_Accurate_Time_Synchronization.pdf.Google Scholar
- 8.IEEE Standard 1588-2008. http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=4579757.Google Scholar
- 9.IEEE Standard 802.3-2008. http://standards.ieee.org/about/get/802/802.3.html.Google Scholar
- 10.Intel 64 and IA-32 architectures software developer manuals. http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html.Google Scholar
- 11.iperf. https://iperf.fr.Google Scholar
- 12.ITU-T Rec. G.8262. http://www.itu.int/rec/T-REC-G.8262.Google Scholar
- 13.Mellanox. www.mellanox.com.Google Scholar
- 14.Open compute project. http://www.opencompute.org.Google Scholar
- 15.Stratix V FPGA. http://www.altera.com/devices/fpga/stratix-fpgas/stratix-v/stxv-index.jsp.Google Scholar
- 16.Timekeeper. http://www.fsmlabs.com/timekeeper.Google Scholar
- 17.IEEE 1588 PTP and Analytics on the Cisco Nexus 3548 Switch. http://www.cisco.com/c/en/us/products/collateral/switches/nexus-3000-series-switches/white-paper-c11-731501.html, 2014.Google Scholar
- 18.Al-Fares, M., Loukissas, A., and Vahdat, A. A scalable, commodity data center network architecture. In Proceedings of the ACM SIGCOMM Conference on Data Communication (2008). Google ScholarDigital Library
- 19.Broadcom. Ethernet time synchronization. http://www.broadcom.com/collateral/wp/StrataXGSIV-WP100-R.pdf.Google Scholar
- 20.Chapman, J. T., Chopra, R., and Montini, L. The DOCSIS timing protocol (DTP) generating precision timing services from a DOCSIS system. In Proceedings of the Spring Technical Forum (2011).Google Scholar
- 21.Cochran, R., Marinescu, C., and Riesch, C. Synchronizing the Linux System Time to a PTP Hardware Clock. In Proceedings of the International IEEE Symposium on Precision Clock Synchronization for Measurement Control and Communication (2011).Google ScholarCross Ref
- 22.Corbett, J. C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J. J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P., Hsieh, W., Kanthak, S., Kogan, E., Li, H., Lloyd, A., Melnik, S., Mwaura, D., Nagle, D., Quinlan, S., Rao, R., Rolig, L., Saito, Y., Szymaniak, M., Taylor, C., Wang, R., and Woodford, D. Spanner: Google's globally-distributed database. In Proceedings of the 10th USENIX conference on Operating Systems Design and Implementation (2012). Google ScholarDigital Library
- 23.Costa, P., Ballani, H., Razavi, K., and Kash, I. R2C2: A network stack for rack-scale computers. In Proceedings of the ACM Conference on SIGCOMM (2015). Google ScholarDigital Library
- 24.Cristian, F. Probabilistic clock synchronization. Distributed Computing 3 (September 1989), 146–158.Google ScholarDigital Library
- 25.Davis, M., Villain, B., Ridoux, J., Orgerie, A.-C., and Veitch, D. An IEEE-1588 Compatible RADclock. In Proceedings of International IEEE Symposium on Precision Clock Synchronization for Measurement, Control and Communication (2012).Google ScholarCross Ref
- 26.Edwards, T. G., and Belkin, W. Using SDN to Facilitate Precisely Timed Actions on Real-time Data Streams. In Proceedings of the Third Workshop on Hot Topics in Software Defined Networking (2014). Google ScholarDigital Library
- 27.Freedman, D. A., Marian, T., Lee, J. H., Birman, K., Weatherspoon, H., and Xu, C. Exact temporal characterization of 10 Gbps optical wide-area network. In Proceedings of the 10th ACM SIGCOMM Conference on Internet measurement (2010). Google ScholarDigital Library
- 28.Froehlich, S., Hack, M., Meng, X., and Zhang, L. Achieving precise coordinated cluster time in a cluster environment. In Proceedings of International IEEE Symposium on Precision Clock Synchronization for Measurement, Control and Communication (2008).Google ScholarCross Ref
- 29.Gusella, R., and Zatti, S. The Accuracy of the Clock Synchronization Achieved by TEMPO in Berkeley UNIX 4.3BSD. IEEE Transactions on Software Engineering 15, 7 (July 1989), 847–853. Google ScholarDigital Library
- 30.Jasperneite, J., Shehab, K., and Weber, K. Enhancements to the time synchronization standard IEEE-1588 for a system of cascaded bridges. In Proceedings of the IEEE International Workshop in Factory Communication Systems (2004).Google ScholarCross Ref
- 31.Kachris, C., Bergman, K., and Tomkos, I. Optical Interconnects for Future Data Center Networks. Springer, 2013. Google ScholarDigital Library
- 32.King, M., Hicks, J., and Ankcorn, J. Software-driven hardware development. In Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (2015). Google ScholarDigital Library
- 33.Kopetz, H., and Ochsenreiter, W. Clock synchronization in distributed real-time systems. IEEE Transactions on Computers C-36 (Aug 1987), 933–940. Google ScholarDigital Library
- 34.Lamport, L., and Melliar-Smith, P. M. Byzantine Clock Synchronization. In Proceedings of the Third Annual ACM Symposium on Principles of Distributed Computing (1984). Google ScholarDigital Library
- 35.Lapinski, M., Wlostowki, T., Serrano, J., and Alvarez, P. White Rabbit: a PTP Application for Robust Sub-nanosecond Synchronization. In Proceedings of the International IEEE Symposium on Precision Clock Synchronization for Measurement Control and Communication (2011).Google ScholarCross Ref
- 36.Lee, K. S., Wang, H., and Weatherspoon, H. SoNIC: Precise Realtime Software Access and Control of Wired Networks. In Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (2013). Google ScholarDigital Library
- 37.Lewandowski, W., Azoubib, J., and Klepczynski, W. J. GPS: primary tool for time transfer. Proceedings of the IEEE 87 (January 1999), 163–172.Google ScholarCross Ref
- 38.Li, H. IEEE 1588 time synchronization deployment for mobile backhaul in China Mobile, 2014. Keynote speech in the International IEEE Symposium on Precision Clock Synchronization for Measurement Control and Communication.Google Scholar
- 39.Lipinski, M., Wlostowski, T., Serrano, J., Alvarez, P., Cobas, J. D. G., Rubini, A., and Moreira, P. Performance results of the first White Rabbit installation for CNGS time transfer. In Proceedings of the International IEEE Symposium on Precision Clock Synchronization for Measurement Control and Communication (2012).Google ScholarCross Ref
- 40.Mallada, E., Meng, X., Hack, M., Zhang, L., and Tang, A. Skewless Network Clock Synchronization. In Proceedings of the 21st IEEE International Conference on Network Protocols (2013).Google ScholarCross Ref
- 41.Mills, D. L. Internet time synchronization: the network time protocol. IEEE transactions on Communications 39 (October 1991), 1482–1493.Google Scholar
- 42.Mizrahi, T., and Moses, Y. Software Defined Networks: It's about time. In Proceedings of the IEEE International Conference on Computer Communications (2016).Google ScholarCross Ref
- 43.Moreira, P., Serrano, J., Wlostowski, T., Loschmidt, P., and Gaderer, G. White Rabbit: Sub-Nanosecond Timing Distribution over Ethernet. In Proceedings of the International IEEE Symposium on Precision Clock Synchronization for Measurement Control and Communication (2009).Google ScholarCross Ref
- 44.Ogden, B., Fadel, J., and White, B. IBM system z9 109 technical introduction. Google ScholarDigital Library
- 45.Ohly, P., Lombard, D. N., and Stanton, K. B. Hardware assisted precision time protocol. design and case study. In Proceedings of the 9th LCI International Conference on High-Performance Clustered Computing (2008).Google Scholar
- 46.Pásztor, A., and Veitch, D. PC Based Precision Timing Without GPS. In Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (2002). Google ScholarDigital Library
- 47.Perry, J., Ousterhout, A., Balakrishnan, H., Shah, D., and Fugal, H. Fastpass: A centralized "zero-queue" datacenter network. In Proceedings of the ACM Conference on SIGCOMM (2014). Google ScholarDigital Library
- 48.Schneider, F. B. Understanding Protocols for Byzantine Clock Synchronization. Tech. Rep. TR87-859, Cornell University, August 1987. Google ScholarDigital Library
- 49.Sobeih, A., Hack, M., Liu, Z., and Zhang, L. Almost Peer-to-Peer Clock Synchronization. In Proceedings of IEEE International Parallel and Distributed Processing Symposium (2007).Google ScholarCross Ref
- 50.Veitch, D., Babu, S., and Pàsztor, A. Robust Synchronization of Software Clocks Across the Internet. In Proceedings of the 4th ACM SIGCOMM Conference on Internet Measurement (2004). Google ScholarDigital Library
- 51.Zarick, R., Hagen, M., and Bartos, R. The impact of network latency on the synchronization of real-world IEEE 1588-2008 devices. In Proceedings of the International IEEE Symposium on Precision Clock Synchronization for Measurement Control and Communication (2010).Google ScholarCross Ref
- 52.Zarick, R., Hagen, M., and Bartos, R. Transparent clocks vs. enterprise ethernet switches. In Proceedings of the International IEEE Symposium on Precision Clock Synchronization for Measurement, Control and Communication (2011).Google ScholarCross Ref
- 53.Zeng, H., Zhang, S., Ye, F., Jeyakumar, V., Ju, M., Liu, J., McKeown, N., and Vahdat, A. Libra: Divide and conquer to verify forwarding tables in huge networks. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (2014). Google ScholarDigital Library
Index Terms
- Globally Synchronized Time via Datacenter Networks
Recommendations
Globally Synchronized Time via Datacenter Networks
Synchronized time is critical to distributed systems and network applications in a datacenter network. Unfortunately, many clock synchronization protocols in datacenter networks such as NTP and PTP are fundamentally limited by the characteristics of ...
Green spine switch management for datacenter networks
Energy consumption for datacenter has grown significantly and the trend is still growing due to the increasing popularity of cloud computing. Datacenter networks (DCNs), however, are starting to consume a greater portion of overall energy in comparison ...
Comments