Stavros Harizopoulos
Daniel J. Abadi
Michael Stonebraker
ABSTRACT
Online Transaction Processing (OLTP) databases include a suite of features - disk-resident B-trees and heap files, locking-based concurrency control, support for multi-threading - that were optimized for computer technology of the late 1970's. Advances in modern processors, memories, and networks mean that today's computers are vastly different from those of 30 years ago, such that many OLTP databases will now fit in main memory, and most OLTP transactions can be processed in milliseconds or less. Yet database architecture has changed little.
Based on this observation, we look at some interesting variants of conventional database systems that one might build that exploit recent hardware trends, and speculate on their performance through a detailed instruction-level breakdown of the major components involved in a transaction processing database system (Shore) running a subset of TPC-C. Rather than simply profiling Shore, we progressively modified it so that after every feature removal or optimization, we had a (faster) working system that fully ran our workload. Overall, we identify overheads and optimizations that explain a total difference of about a factor of 20x in raw performance. We also show that there is no single "high pole in the tent" in modern (memory resident) database systems, but that substantial time is spent in logging, latching, locking, B-tree, and buffer management operations.
- Agrawal, R., Carey, M. J., and Livny, M. "Concurrency control performance modeling: alternatives and implications." ACM Trans. Database Syst. 12(4), Dec. 1987. Google Scholar
Digital Library
- Aguilera, M., Merchant, A., Shah, M., Veitch, A. C., and Karamanolis, C. T. "Sinfonia: a new paradigm for building scalable distributed systems." In Proc. SOSP, 2007. Google ScholarDigital Library
- Aho, A. V., Hopcroft, J. E., and Ullman, J. D. "The Design and Analysis of Computer Algorithms." Addison-Wesley Publishing Company, 1974. Google ScholarDigital Library
- Ailamaki, A., DeWitt, D. J., Hill, M. .D., and Wood, D. A. "DBMSs on a Modern Processor: Where Does Time Go?" In Proc. VLDB, 1999, 266--277. Google ScholarDigital Library
- Ailamaki, A. "Database Architecture for New Hardware." Tutorial. In Proc. VLDB, 2004. Google ScholarDigital Library
- Anon et al. "A Measure of Transaction Processing Power." In Datamation, February 1985.Google Scholar
- Baulier, J. D., Bohannon, P., Khivesara, A., et al. "The DataBlitz Main-Memory Storage Manager: Architecture, Performance, and Experience." In The VLDB Journal, 1998.Google Scholar
- Bitton, D., DeWitt, D. J., and Turbyfill, C. "Benchmarking Database Systems, a Systematic Approach." In Proc. VLDB, 1983. Google ScholarDigital Library
- Bitton, D., Hanrahan, M., and Turbyfill, C. "Performance of Complex Queries in Main Memory Database Systems." In Proc. ICDE, 1987. Google ScholarDigital Library
- Boncz, P. A., Manegold, S., and Kersten, M. L. "Database Architecture Optimized for the New Bottleneck: Memory Access." In Proc. VLDB, 1999. Google ScholarDigital Library
- Brewer, E. A. "Towards robust distributed systems (abstract)." In Proc. PODC, 2000. Google ScholarDigital Library
- Bugnion, E., Devine, S., and Rosenblum, M. "Disco: running commodity operating systems on scalable multiprocessors." In Proc. SOSP, 1997. Google ScholarDigital Library
- Carey, M. J., DeWitt, D. J., Franklin, M. J. et al. "Shoring up persistent applications." In Proc. SIGMOD, 1994. Google ScholarDigital Library
- Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., and Gruber, R. E. "Bigtable: A Distributed Storage System for Structured Data." In Proc. OSDI, 2006. Google ScholarDigital Library
- Dean, J. and Ghemawat, S. "MapReduce: Simplified Data Processing on Large Clusters." In Proc. OSDI, 2004. Google ScholarDigital Library
- DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., and Vogels, W. "Dynamo: amazon's highly available key-value store." In Proc. SOSP, 2007. Google ScholarDigital Library
- DeWitt, D. J., Ghandeharizadeh, S., Schneider, D. A., Bricker, A., Hsiao, H., and Rasmussen, R. "The Gamma Database Machine Project." IEEE Transactions on Knowledge and Data Engineering 2(1):44--62, March 1990. Google ScholarDigital Library
- Eich, M. H. "MARS: The Design of A Main Memory Database Machine." In Proc. of the 1987 International workshop on Database Machines, October, 1987. Google ScholarDigital Library
- Garcia-Molina, H. and Salem, K. "Main Memory Database Systems: An Overview." IEEE Trans. Knowl. Data Eng. 4(6): 509--516 (1992). Google ScholarDigital Library
- Gray, J. and Reuter, A. "Transaction Processing: Concepts and Techniques." Morgan Kaufmann Publishers, Inc., 1993. Google ScholarDigital Library
- Gribble, S. D., Brewer, E. A., Hellerstein, J. M., and Culler, D .E. "Scalable, Distributed Data Structures for Internet Service Construction." In Proc. OSDI, 2000. Google ScholarDigital Library
- Helland, P. "Life beyond Distributed Transactions: an Apostate's Opinion." In Proc. CIDR, 2007.Google Scholar
- Herlihy, M. P. and Moss, J. E. B. "Transactional Memory: architectural support for lock-free data structures." In Proc. ISCA, 1993. Google ScholarDigital Library
- Kung, H. T. and Robinson, J. T. "On optimistic methods for concurrency control." ACM Trans. Database Syst. 6(2):213--226, June 1981. Google ScholarDigital Library
- Lau, E. and Madden, S. "An Integrated Approach to Recovery and High Availability in an Updatable, Distributed Data Warehouse." In Proc. VLDB, 2006. Google ScholarDigital Library
- Lehman, T. J. and Carey, M. J. "A study of index structures for main memory database management systems." In Proc. VLDB, 1986. Google ScholarDigital Library
- Liskov, B., Ghemawat, S., Gruber, R., Johnson, P., Shrira, L., and Williams, M. "Replication in the harp file system." In Proc. SOSP, pages 226--238, 1991. Google ScholarDigital Library
- McWherter, D. T., Schroeder, B., Ailamaki, A., and Harchol-Balter, M. "Priority Mechanisms for OLTP and Transactional Web Applications." In Proc. ICDE, 2004. Google ScholarDigital Library
- Mohan, C., Haderle, D., Lindsay, B., Pirahesh, H., and Schwarz, P. "ARIES: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging." ACM Trans. Database Syst. 17(1):94--162, 1992. Google ScholarDigital Library
- Mohan, C. "ARIES/KVL: A Key-Value Locking Method for Concurrency Control of Multiaction Transactions Operating on B-Tree Indexes." 1989, Research Report RJ 7008, Data Base Technology Institute, IBM Almaden Research Center.Google Scholar
- Mohan, C. and Levine, F. "ARIES/IM: An Efficient and High Concurrency Index Management Method Using Write-Ahead Logging." 1989, Research Report RJ 6846, Data Base Technology Institute, IBM Almaden Research Center.Google Scholar
- Mucci, P. J., Browne, S., Deane, C., and Ho, G. "PAPI: A Portable Interface to Hardware Performance Counters." In Proc. Department of Defense HPCMP Users Group Conference, Monterey, CA, June 1999.Google Scholar
- Rao, J. and Ross, K. A. "Cache Conscious Indexing for Decision-Support in Main Memory." In Proc. VLDB, 1999. Google ScholarDigital Library
- Rao, J. and Ross, K. A. "Making B+- trees cache conscious in main memory." In SIGMOD Record, 29(2):475--486, June 2000. Google ScholarDigital Library
- Stoica, I., Morris, R., Karger, D. R., Kaashoek, M. F., and Balakrishnan, H. "Chord: A Scalable Peer-to-peer Lookup Protocol for Internet Applications." In Proc. SIGCOMM, 2001. Google ScholarDigital Library
- Stonebraker, M., Abadi, D. J., Batkin, A., Chen, X., Cherniack, M., Ferreira, M., Lau, E., Lin, A., Madden, S., O'Neil, E., O'Neil, P., Rasin, A., Tran, N., and Zdonik, S. "C-Store: A Column-oriented DBMS." In Proc. VLDB, 2005. Google ScholarDigital Library
- Stonebraker, M., Madden, S., Abadi, D. J., Harizopoulos, S., Hachem, N., and Helland, P. "The End of an Architectural Era (It's Time for a Complete Rewrite)." In Proc. VLDB, 2007. Google ScholarDigital Library
- Oracle TimesTen. http://www.oracle.com/timesten/index.html. 2007.Google Scholar
- The Transaction Processing Council. TPC-C Benchmark (Rev. 5.8.0), 2006. http://www.tpc.org/tpcc/spec/tpcc_current.pdfGoogle Scholar
- Whitney, A., Shasha, D., and Apter, S. "High Volume Transaction Processing Without Concurrency Control, Two Phase Commit, SQL or C." In Proc. HPTPS, 1997.Google Scholar
Index Terms
- OLTP through the looking glass, and what we found there
Recommendations
Hekaton: SQL server's memory-optimized OLTP engine
SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of DataHekaton is a new database engine optimized for memory resident data and OLTP workloads. Hekaton is fully integrated into SQL Server; it is not a separate system. To take advantage of Hekaton, a user simply declares a table memory optimized. Hekaton ...
OLTP through the looking glass, and what we found there
Making Databases WorkOnline Transaction Processing (OLTP) databases include a suite of features---disk-resident B-trees and heap files, locking-based concurrency control, support for multi-threading---that were optimized for computer technology of the late 1970's. Advances ...
Micro-architectural Analysis of In-memory OLTP
SIGMOD '16: Proceedings of the 2016 International Conference on Management of DataMicro-architectural behavior of traditional disk-based online transaction processing (OLTP) systems has been investigated extensively over thepast couple of decades. Results show that traditional OLTP mostly under-utilize the available micro-...
Comments