Abstract
The emergence of blockchains has fueled the development of resilient systems that can deal with Byzantine failures due to crashes, bugs, or even malicious behavior. Recently, we have also seen the exploration of sharding in these resilient systems, this to provide the scalability required by very large data-based applications. Unfortunately, current sharded resilient systems all use system-specific specialized approaches toward sharding that do not provide the flexibility of traditional sharded data management systems.
To improve on this situation, we fundamentally look at the design of sharded resilient systems. We do so by introducing BYSHARD, a unifying framework for the study of sharded resilient systems. Within this framework, we show how two-phase commit and two-phase locking---two techniques central to providing atomicity and isolation in traditional sharded databases---can be implemented efficiently in a Byzantine environment, this with a minimal usage of costly Byzantine resilient primitives. Based on these techniques, we propose eighteen multi-shard transaction processing protocols. Finally, we practically evaluate these protocols and show that each protocol supports high transaction throughput and provides scalability while each striking its own trade-off between throughput, isolation level, latency, and abort rate. As such, our work provides a strong foundation for the development of ACID-compliant general-purpose and flexible sharded resilient data management systems.
- Mustafa Al-Bassam, Alberto Sonnino, Shehar Bano, Dave Hrycyszyn, and George Danezis. 2017. Chainspace: A Sharded Smart Contracts Platform. http://arxiv.org/abs/1708.03778Google Scholar
- Mohammad Javad Amiri, Divyakant Agrawal, and Amr El Abbadi. 2019. CAPER: A Cross-application Permissioned Blockchain. Proc. VLDB Endow. 12, 11 (2019), 1385--1398. Google ScholarDigital Library
- Mohammad Javad Amiri, Divyakant Agrawal, and Amr El Abbadi. 2020. SharPer: Sharding Permissioned Blockchains Over Network Clusters. https://arxiv.org/abs/1910.00765v2Google Scholar
- Elli Androulaki, Artem Barger, Vita Bortnikov, Christian Cachin, Konstantinos Christidis, Angelo De Caro, David Enyeart, Christopher Ferris, Gennady Laventman, Yacov Manevich, Srinivasan Muralidharan, Chet Murthy, Binh Nguyen, Manish Sethi, Gari Singh, Keith Smith, Alessandro Sorniotti, Chrysoula Stathakopoulou, Marko Vukolić, Sharon Weed Cocco, and Jason Yellick. 2018. Hyperledger Fabric: A Distributed Operating System for Permissioned Blockchains. In Proceedings of the Thirteenth EuroSys Conference. ACM, 30:1--30:15. Google ScholarDigital Library
- Vijayalakshmi Atluri, Elisa Bertino, and Sushil Jajodia. 1997. A theoretical formulation for degrees of isolation in databases. Inform. Software Tech. 39, 1 (1997), 47--53. Google ScholarCross Ref
- Hal Berenson, Phil Bernstein, Jim Gray, Jim Melton, Elizabeth O'Neil, and Patrick O'Neil. 1995. A Critique of ANSI SQL Isolation Levels. SIGMOD Rec. 24, 2 (1995), 1--10. Google ScholarDigital Library
- Christian Berger and Hans P. Reiser. 2018. Scaling Byzantine Consensus: A Broad Analysis. In Proceedings of the 2nd Workshop on Scalable and Resilient Infrastructures for Distributed Ledgers. ACM, 13--18. Google ScholarDigital Library
- Burkhard Blechschmidt. 2018. Blockchain in Europe: Closing the Strategy Gap. Technical Report. Cognizant Consulting. https://www.cognizant.com/whitepapers/blockchain-in-europe-closing-the-strategy-gap-codex3320.pdfGoogle Scholar
- Christian Cachin and Marko Vukolic. 2017. Blockchain Consensus Protocols in the Wild (Keynote Talk). In 31st International Symposium on Distributed Computing, Vol. 91. Schloss Dagstuhl, 1:1--1:16. Google ScholarCross Ref
- Michael Casey, Jonah Crane, Gary Gensler, Simon Johnson, and Neha Narula. 2018. The Impact of Blockchain Technology on Finance: A Catalyst for Change. Technical Report. International Center for Monetary and Banking Studies. https://www.cimb.ch/uploads/1/1/5/4/115414161/geneva21_1.pdfGoogle Scholar
- Miguel Castro and Barbara Liskov. 2002. Practical Byzantine Fault Tolerance and Proactive Recovery. ACM Trans. Comput. Syst. 20, 4 (2002), 398--461. Google ScholarDigital Library
- Christie's. 2018. Major Collection of the Fall Auction Season to be Recorded with Blockchain Technology. https://www.christies.com/presscenter/pdf/9160/RELEASE_ChristiesxArtoryxEbsworth_9160_1.pdfGoogle Scholar
- Cindy Compert, Maurizio Luinetti, and Bertrand Portier. 2018. Blockchain and GDPR: How blockchain could address five areas associated with GDPR compliance. Technical Report. IBM Security. https://public.dhe.ibm.com/common/ssi/ecm/61/en/61014461usen/security-ibm-security-solutions-wg-white-paper-external-61014461usen-20180319.pdfGoogle Scholar
- Miguel Correia, Giuliana Santos Veronese, Nuno Ferreira Neves, and Paulo Verissimo. 2011. Byzantine Consensus in Asynchronous Message-Passing Systems: A Survey. Int. J. Crit. Comput.-Based Syst. 2, 2 (2011), 141--161. Google ScholarDigital Library
- Hung Dang, Tien Tuan Anh Dinh, Dumitrel Loghin, Ee-Chien Chang, Qian Lin, and Beng Chin Ooi. 2019. Towards Scaling Blockchain Systems via Sharding. In Proceedings of the 2019 International Conference on Management of Data. ACM, 123--140. Google ScholarDigital Library
- Tien Tuan Anh Dinh, Rui Liu, Meihui Zhang, Gang Chen, Beng Chin Ooi, and Ji Wang. 2018. Untangling Blockchain: A Data Processing View of Blockchain Systems. IEEE Trans. Knowl. Data Eng. 30, 7 (2018), 1366--1385. Google ScholarCross Ref
- D. Dolev. 1981. Unanimity in an unknown and unreliable environment. In 22nd Annual Symposium on Foundations of Computer Science. IEEE, 159--168. Google ScholarDigital Library
- Danny Dolev. 1982. The Byzantine generals strike again. J. Algorithms 3, 1 (1982), 14--30. Google ScholarCross Ref
- Wayne W. Eckerson. 2002. Data quality and the bottom line: Achieving Business Success through a Commitment to High Quality Data. Technical Report. The Data Warehousing Institute, 101communications LLC.Google Scholar
- Muhammad El-Hindi, Carsten Binnig, Arvind Arasu, Donald Kossmann, and Ravi Ramamurthy. 2019. BlockchainDB: A Shared Database on Blockchains. Proc. VLDB Endow. 12, 11 (2019), 1597--1609. Google ScholarDigital Library
- Ethereum Foundation. 2017. BTC Relay: A bridge between the Bitcoin blockchain & Ethereum smart contracts. http://btcrelay.orgGoogle Scholar
- Lan Ge, Christopher Brewster, Jacco Spek, Anton Smeenk, and Jan Top. 2017. Blockchain for agriculture and food: Findings from the pilot study. Technical Report. Wageningen University. https://www.wur.nl/nl/Publicatie-details.htm?publicationId=publication-way-353330323634Google Scholar
- William J. Gordon and Christian Catalini. 2018. Blockchain Technology for Healthcare: Facilitating the Transition to Patient-Driven Interoperability. Comput. Struct. Biotechnol. J. 16 (2018), 224--230. Google ScholarCross Ref
- Jim Gray. 1978. Notes on Data Base Operating Systems. In Operating Systems, An Advanced Course. Springer-Verlag, 393--481. Google ScholarDigital Library
- GSM Association. 2017. Blockchain for Development: Emerging Opportunities for Mobile, Identity and Aid. https://www.gsma.com/mobilefordevelopment/wp-content/uploads/2017/12/Blockchain-for-Development.pdfGoogle Scholar
- Suyash Gupta, Jelle Hellings, and Mohammad Sadoghi. 2021. Fault-Tolerant Distributed Transactions on Blockchain. Morgan & Claypool. Google ScholarCross Ref
- Suyash Gupta, Sajjad Rahnama, Jelle Hellings, and Mohammad Sadoghi. 2020. ResilientDB: Global Scale Resilient Blockchain Fabric. Proc. VLDB Endow. 13, 6 (2020), 868--883. Google ScholarDigital Library
- Suyash Gupta, Sajjad Rahnama, and Mohammad Sadoghi. 2020. Permissioned Blockchain Through the Looking Glass: Architectural and Implementation Lessons Learned. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS). IEEE, 754--764. Google ScholarCross Ref
- Theo Haerder and Andreas Reuter. 1983. Principles of Transaction-Oriented Database Recovery. ACM Comput. Surv. 15, 4 (1983), 287--317. Google ScholarDigital Library
- Jelle Hellings, Daniel P. Hughes, Joshua Primero, and Mohammad Sadoghi. 2020. Cerberus: Minimalistic Multi-shard Byzantine-resilient Transaction Processing. https://arxiv.org/abs/2008.04450Google Scholar
- Jelle Hellings and Mohammad Sadoghi. 2019. Brief Announcement: The Fault-Tolerant Cluster-Sending Problem. In 33rd International Symposium on Distributed Computing (DISC 2019). Schloss Dagstuhl, 45:1--45:3. Google ScholarCross Ref
- Maurice Herlihy. 2018. Atomic Cross-Chain Swaps. In Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing. ACM, 245--254. Google ScholarDigital Library
- Maurice Herlihy. 2019. Blockchains from a Distributed Computing Perspective. Commun. ACM 62, 2 (2019), 78--85. Google ScholarDigital Library
- Maurice Herlihy, Barbara Liskov, and Liuba Shrira. 2019. Cross-Chain Deals and Adversarial Commerce. Proc. VLDB Endow. 13, 2 (2019), 100--113. Google ScholarDigital Library
- Thomas N. Herzog, Fritz J. Scheuren, and William E. Winkler. 2007. Data Quality and Record Linkage Techniques. Springer. Google ScholarDigital Library
- Matt Higginson, Johannes-Tobias Lorenz, Björn Münstermann, and Peter Braad Olesen. 2017. The promise of blockchain. Technical Report. McKinsey&Company. https://www.mckinsey.com/industries/financial-services/our-insights/the-promise-of-blockchainGoogle Scholar
- Maged N. Kamel Boulos, James T. Wilson, and Kevin A. Clauson. 2018. Geospatial blockchain: promises, challenges, and scenarios in health and healthcare. Int. J. Health. Geogr 17, 1 (2018), 1211--1220. Google ScholarCross Ref
- Jae Kwon and Ethan Buchman. 2019. Cosmos Whitepaper: A Network of Distributed Ledgers. https://cosmos.network/cosmos-whitepaper.pdfGoogle Scholar
- Leslie Lamport. 2001. Paxos Made Simple. ACM SIGACT News 32, 4 (2001), 51--58. Distributed Computing Column 5. Google ScholarDigital Library
- Laphou Lao, Zecheng Li, Songlin Hou, Bin Xiao, Songtao Guo, and Yuanyuan Yang. 2020. A Survey of IoT Applications in Blockchain Systems: Architecture, Consensus, and Traffic Modeling. ACM Comput. Surv. 53, 1, Article 18 (2020), 32 pages. Google ScholarDigital Library
- Satoshi Nakamoto. [n.d.]. Bitcoin: A Peer-to-Peer Electronic Cash System. https://bitcoin.org/en/bitcoin-paperGoogle Scholar
- Arvind Narayanan and Jeremy Clark. 2017. Bitcoin's Academic Pedigree. Commun. ACM 60, 12 (2017), 36--45. Google ScholarDigital Library
- Senthil Nathan, Chander Govindarajan, Adarsh Saraf, Manish Sethi, and Praveen Jayachandran. 2019. Blockchain Meets Database: Design and Implementation of a Blockchain Relational Database. Proc. VLDB Endow. 12, 11 (2019), 1539--1552. Google ScholarDigital Library
- Faisal Nawab and Mohammad Sadoghi. 2019. Blockplane: A Global-Scale Byzantizing Middleware. In 35th International Conference on Data Engineering (ICDE). IEEE, 124--135. Google ScholarCross Ref
- Dick O'Brie. 2017. Internet Security Threat Report: Ransomware 2017, An ISTR Special Report. Technical Report. Symantec. https://www.symantec.com/content/dam/symantec/docs/security-center/white-papers/istr-ransomware-2017-en.pdfGoogle Scholar
- The Council of Economic Advisers. 2018. The Cost of Malicious Cyber Activity to the U.S. Economy. Technical Report. Executive Office of the President of the United States. https://www.whitehouse.gov/wp-content/uploads/2018/03/The-Cost-of-Malicious-Cyber-Activity-to-the-U.S.-Economy.pdfGoogle Scholar
- M. Tamer Özsu and Patrick Valduriez. 2020. Principles of Distributed Database Systems. Springer. Google ScholarDigital Library
- Michael Pisa and Matt Juden. 2017. Blockchain and Economic Development: Hype vs. Reality. Technical Report. Center for Global Development. https://www.cgdev.org/publication/blockchain-and-economic-development-hype-vs-realityGoogle Scholar
- PwC. 2016. Blockchain - an opportunity for energy producers and consumers? https://www.pwc.com/gx/en/industries/energy-utilities-resources/publications/opportunity-for-energy-producers.htmlGoogle Scholar
- Thomas C. Redman. 1998. The Impact of Poor Data Quality on the Typical Enterprise. Commun. ACM 41, 2 (1998), 79--82. Google ScholarDigital Library
- David Reinsel, John Gantz, and John Rydning. 2018. Data Age 2025: The Digitization of the World, From Edge to Core. Technical Report. IDC. https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdfGoogle Scholar
- Dale Skeen. 1982. A Quorum-Based Commit Protocol. Technical Report. Cornell University. Google Scholar
- Symantec. 2018. Internet Security Threat Report, Volume 32. https://www.symantec.com/content/dam/symantec/docs/reports/istr-23-2018-en.pdfGoogle Scholar
- Gerard Tel. 2001. Introduction to Distributed Algorithms (2nd ed.). Cambridge University Press. Google ScholarDigital Library
- Maarten van Steen and Andrew S. Tanenbaum. 2017. Distributed Systems (3th ed.). Maarten van Steen. https://www.distributed-systems.net/Google Scholar
- Gavin Wood. [n.d.]. Ethereum: a secure decentralised generalised transaction ledger. https://gavwood.com/paper.pdf EIP-150 revision.Google Scholar
- Gavin Wood. 2016. Polkadot: vision for a heterogeneous multi-chain framework. https://polkadot.network/PolkaDotPaper.pdfGoogle Scholar
- Yang Xiao, Ning Zhang, Wenjing Lou, and Y. Thomas Hou. 2020. A Survey of Distributed Consensus Protocols for Blockchain Networks. IEEE Commun. Surv. Tutor 22, 2 (2020), 1432--1465. Google ScholarCross Ref
- Maofan Yin, Dahlia Malkhi, Michael K. Reiter, Guy Golan Gueta, and Ittai Abraham. 2019. HotStuff: BFT Consensus with Linearity and Responsiveness. In Proceedings of the ACM Symposium on Principles of Distributed Computing. ACM, 347--356. Google ScholarDigital Library
- Victor Zakhary, Divyakant Agrawal, and Amr El Abbadi. 2020. Atomic Commitment across Blockchains. Proc. VLDB Endow. 13, 9 (2020), 1319--1331. Google ScholarDigital Library
Index Terms
- ByShard: sharding in a byzantine environment
Recommendations
ByShard: sharding in a Byzantine environment
AbstractThe emergence of blockchains has fueled the development of resilient systems that deal with Byzantine failures due to crashes, bugs, or even malicious behavior. Recently, we have also seen the exploration of sharding in these resilient systems, ...
SHAFT: Supporting Transactions with Serializability and Fault-Tolerance in Highly-Available Datastores
ICPADS '15: Proceedings of the 2015 IEEE 21st International Conference on Parallel and Distributed Systems (ICPADS)Guaranteeing transaction semantics in a highly available and fault tolerant manner is desirable to application developers. Besides, it is a very valuable feature for database-backed applications. In this paper, we propose SHAFT to support transactions ...
Comments