ABSTRACT
Distributed applications and web services, such as online stores or social networks, are expected to be scalable, available, responsive, and fault-tolerant. To meet these steep requirements in the face of high round-trip latencies, network partitions, server failures, and load spikes, applications use eventually consistent datastores that allow them to weaken the consistency of some data. However, making this transition is highly error-prone because relaxed consistency models are notoriously difficult to understand and test.
In this work, we propose a new programming model for distributed data that makes consistency properties explicit and uses a type system to enforce consistency safety. With the Inconsistent, Performance-bound, Approximate (IPA) storage system, programmers specify performance targets and correctness requirements as constraints on persistent data structures and handle uncertainty about the result of datastore reads using new consistency types. We implement a prototype of this model in Scala on top of an existing datastore, Cassandra, and use it to make performance/correctness tradeoffs in two applications: a ticket sales service and a Twitter clone. Our evaluation shows that IPA prevents consistency-based programming errors and adapts consistency automatically in response to changing network conditions, performing comparably to weak consistency and 2-10× faster than strong consistency.
- Fusion ticket. http://fusionticket.org.Google Scholar
- Scala in the enterprise. http://www.scala-lang.org/old/node/1658, March 2009.Google Scholar
- Amazon Web Services, Inc. Elastic compute cloud (ec2) cloud server & hosting -- aws. https://aws.amazon.com/ec2/,2016.Google Scholar
- Apache Software Foundation. Cassandra. http://cassandra.apache.org/, 2015.Google Scholar
- Apache Software Foundation. Apache spark - lightning-fast cluster computing. http://spark.apache.org/, 2016a.Google Scholar
- Apache Software Foundation. Apache thrift. https://thrift.apache.org/, 2016b.Google Scholar
- Lisa Baertlein. Ellen's Oscar 'selfie' crashes Twitter, breaks record. http://www.reuters.com/article/2014/03/03/us-oscars-selfie-idUSBREA220C320140303, March 2014.Google Scholar
- Peter Bailis, Ali Ghodsi, Joseph M. Hellerstein, and Ion Stoica. Bolt-on causal consistency. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD '13, pages 761--772, New York, NY, USA, 2013. ACM. ISBN 978-1-4503-2037-5. doi:10.1145/2463676.2465279. Google ScholarDigital Library
- Valter Balegas, Sérgio Duarte, Carla Ferreira, Rodrigo Rodrigues, Nuno Preguiça, Mahsa Najafzadeh, and Marc Shapiro. Putting consistency back into eventual consistency. In Proceedings of the Tenth European Conference on Computer Systems, EuroSys, pages 6:1--6:16, New York, NY, USA, 2015a. ACM. ISBN 978-1-4503-3238-5. doi:10.1145/2741948.2741972. Google ScholarDigital Library
- Valter Balegas, Diogo Serra, Sergio Duarte, Carla Ferreira, Marc Shapiro, Rodrigo Rodrigues, and Nuno Preguiça. Extending eventually consistent cloud databases for enforcing numeric invariants. 34th International Symposium on Reliable Distributed Systems (SRDS 2015), September 2015b. Google ScholarDigital Library
- Basho Technologies, Inc. Riak. http://docs.basho.com/riak/latest/, 2015.Google Scholar
- Nalini Belaramani, Mike Dahlin, Lei Gao, Amol Nayate, Arun Venkataramani, Praveen Yalagandula, and Jiandan Zheng. Practi replication. In Proceedings of the 3rd Conference on Networked Systems Design & Implementation - Volume 3, NSDI'06, pages 5--5, Berkeley, CA, USA, 2006. USENIX Association. URL http://dl.acm.org/citation.cfm?id=1267680.1267685. Google ScholarDigital Library
- Nalini Belaramani, Jiandan Zheng, Amol Nayate, Robert Soulé, Mike Dahlin, and Robert Grimm. Pads: A policy architecture for distributed storage systems. In Proceedings of the 6th USENIX Symposium on Networked Systems Design and Implementation, NSDI'09, pages 59--73, Berkeley, CA, USA, 2009. USENIX Association. URL http://dl.acm.org/citation.cfm?id=1558977.1558982. Google ScholarDigital Library
- Philip A. Bernstein, Alan Fekete, Hongfei Guo, Raghu Ramakrishnan, and Pradeep Tamma. Relaxed currency serializability for middle-tier caching and replication. In Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data, Chicago, IL, USA, June 2006. ACM. Google ScholarDigital Library
- James Bornholt, Todd Mytkowicz, and Kathryn S. McKinley. Uncertain: A First-Order Type for Uncertain Data. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS 14, ASPLOS. Association for Computing Machinery (ACM), 2014. doi:10.1145/2541940.2541958. Google ScholarDigital Library
- Brett Boston, Adrian Sampson, Dan Grossman, and Luis Ceze. Probability type inference for flexible approximate programming. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, pages 470--487, 2015. doi:10.1145/2814270.2814301. Google ScholarDigital Library
- Eric A. Brewer. Towards robust distributed systems. In Keynote at PODC (ACM Symposium on Principles of Distributed Computing). Association for Computing Machinery (ACM), 2000. doi:10.1145/343477.343502. Google ScholarDigital Library
- Travis Brown. Scala at scale at Twitter (talk). http://conferences.oreilly.com/oscon/open-source-2015/public/schedule/detail/42332, July 2015.Google Scholar
- Michael Carbin, Sasa Misailovic, and Martin C. Rinard. Verifying quantitative reliability for programs that execute on unreliable hardware. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2013, pages 33--52, 2013. doi:10.1145/2509136.2509546. Google ScholarDigital Library
- Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM symposium on Cloud computing - SoCC 10. Association for Computing Machinery (ACM), 2010. doi:10.1145/1807128.1807152. Google ScholarDigital Library
- Hayley C. Cuccinello. 'star wars' presales crash ticketing sites, set record for fandango. http://www.forbes.com/sites/hayleycuccinello/2015/10/20/star-wars-presales-crash-ticketing-sites-sets-record-for-fandango/, October 2015.Google Scholar
- Datastax, Inc. How are consistent read and write operations handled? http://docs.datastax.com/en/cassandra/3.x/cassandra/dml/dmlAboutDataConsistency.html, 2016.Google Scholar
- Dorothy E. Denning and Peter J. Denning. Certification of programs for secure information flow. Communications of the ACM, 20 (7): 504--513, July 1977. Google ScholarDigital Library
- Docker, Inc. Docker. https://www.docker.com/, 2016.Google Scholar
- Robert Escriva, Bernard Wong, and Emin Gün Sirer. HyperDex. In Proceedings of the ACM SIGCOMM Conference. Association for Computing Machinery (ACM), August 2012. doi:10.1145/2342356.2342360.Google Scholar
- Brady Forrest. Bing and google agree: Slow pages lose users. Radar, June 2009. http://radar.oreilly.com/2009/06/bing-and-google-agree-slow-pag.html.Google Scholar
- Dieter Gawlick and David Kinkade. Varieties of Concurrency Control in IMS/VS Fast Path. IEEE Database Engineering Bulletin, 8 (2): 3--10, 1985.Google Scholar
- Google, Inc. Compute engine --- google cloud platform. https://cloud.google.com/compute/, 2016.Google Scholar
- Susan Hall. Employers can't find enough scala talent. http://insights.dice.com/2014/04/04/employers-cant-find-enough-scala-talent/, March 2014.Google Scholar
- Maurice P. Herlihy and Jeannette M. Wing. Linearizability: a correctness condition for concurrent objects. ACM Transactions on Programming Languages and Systems, 12 (3): 463--492, July 1990. doi:10.1145/78969.78972. Google ScholarDigital Library
- Hyperdex. Hyperdex. http://hyperdex.org/, 2015.Google Scholar
- Avinash Lakshman and Prashant Malik. Cassandra: A decentralized structured storage system. SIGOPS Oper. Syst. Rev., 44 (2): 35--40, April 2010. ISSN 0163-5980. doi:10.1145/1773912.1773922. Google ScholarDigital Library
- Lamport. How to make a multiprocessor computer that correctly executes multiprocess programs. IEEE Transactions on Computers, C-28 (9): 690--691, September 1979. doi:10.1109/tc.1979.1675439. Google ScholarDigital Library
- Cheng Li, Daniel Porto, Allen Clement, Johannes Gehrke, Nuno Preguiça, and Rodrigo Rodrigues. Making geo-replicated systems fast as possible, consistent when necessary. In Presented as part of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI 12), pages 265--278, Hollywood, CA, 2012. USENIX. ISBN 978-1-931971-96-6. URL https://www.usenix.org/conference/osdi12/technical-sessions/presentation/li. Google ScholarDigital Library
- Lightbend Inc. Akka. http://akka.io/, 2016.Google Scholar
- Greg Linden. Make data useful. Talk, November 2006. http://glinden.blogspot.com/2006/12/slides-from-my-talk-at-stanford.html.Google Scholar
- Jed Liu, Tom Magrino, Owen Arden, Michael D. George, and Andrew C. Myers. Warranties for faster strong consistency. In USENIX Symposium on Networked Systems Design and Implementation (NSDI'14), pages 503--517, Seattle, WA, April 2014. USENIX Association. ISBN 978-1-931971-09-6. URL https://www.usenix.org/conference/nsdi14/technical-sessions/presentation/liu_jed. Google ScholarDigital Library
- Cade Metz. How Instagram Solved Its Justin Bieber Problem, November 2015. URL http://www.wired.com/2015/11/how-instagram-solved-its-justin-bieber-problem/.Google Scholar
- Sasa Misailovic, Michael Carbin, Sara Achour, Zichao Qi, and Martin C. Rinard. Chisel: reliability- and accuracy-aware optimization of approximate computational kernels. In Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2014, pages 309--328, 2014. doi:10.1145/2660193.2660231. Google ScholarDigital Library
- Ramon E. Moore. Interval analysis. Prentice-Hall, 1966.Google Scholar
- Andrew C. Myers. Jflow: Practical mostly-static information flow control. In Proceedings of the 26th ACM Symposium on Principles of Programing Languages (POPL '99), San Antonio, TX, USA, January 1999. ACM. Google ScholarDigital Library
- Dao Nguyen. What it's like to work on buzzfeed's tech team during record traffic. http://www.buzzfeed.com/daozers/what-its-like-to-work-on-buzzfeeds-tech-team-during-record-t, February 2015.Google Scholar
- Chris Olston, Boon Thau Loo, and Jennifer Widom. Adaptive precision setting for cached approximate values. In Proceedings of the 1999 ACM SIGMOD International Conference on Management of Data, Santa Barbara, CA, USA, May 1999. ACM. Google ScholarDigital Library
- Patrick E. O'Neil. The escrow transactional method. ACM Transactions on Database Systems, 11 (4): 405--430, December 1986. doi:10.1145/7239.7265. Google ScholarDigital Library
- outworkers ltd. Phantomby outworkers. http://outworkers.github.io/phantom/, March 2016.Google Scholar
- Christian Plattner and Gustavo Alonso. Ganymed: Scalable replication for transactional web applications. In Proceedings of the International Middleware Conference, Toronto, Ontario, Canada, October 2004. Google ScholarDigital Library
- Dan R. K. Ports, Austin T. Clements, Irene Zhang, Samuel Madden, and Barbara Liskov. Transactional consistency and automatic management in an application data cache. In Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI '10), Vancouver, BC, Canada, October 2010. USENIX. Google ScholarDigital Library
- Nuno Preguiça, J. Legatheaux Martins, Miguel Cunha, and Henrique Domingos. Reservations for conflict avoidance in a mobile database system. In Proceedings of the 1st international conference on Mobile systems, applications and services - MobiSys 03, MobiSys. Association for Computing Machinery (ACM), 2003. doi:10.1145/1066116.1189038. Google ScholarDigital Library
- Calton Pu and Avraham Leff. Replica control in distributed systems: An asynchronous approach. In Proceedings of the 1991 ACM SIGMOD International Conference on Management of Data, Denver, CO, USA, May 1991. ACM. Google ScholarDigital Library
- Andreas Reuter. Concurrency on high-traffic data elements. ACM, New York, New York, USA, March 1982.Google Scholar
- Uwe Röhm, Klemens Böhm, Hans-Jörg Schek, and Heiko Schuldt. FAS --- a freshness-sensitive coordination middleware for a cluster of OLAP components. In Proceedings of the 28th International Conference on Very Large Data Bases (VLDB '02), Hong Kong, China, August 2002. Google ScholarDigital Library
- Andrei Sabelfeld and Andrew C. Myers. Language-based information-flow security. IEEE Journal on Selected Areas in Communications, 21 (1): 1--15, January 2003. Google ScholarDigital Library
- Adrian Sampson, Werner Dietl, Emily Fortuna, Danushen Gnanapragasam, Luis Ceze, and Dan Grossman. Enerj: approximate data types for safe and general low-power computation. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2011, pages 164--174, 2011. doi:10.1145/1993498.1993518. Google ScholarDigital Library
- Salvatore Sanfilippo. Redis. http://redis.io/, 2015a.Google Scholar
- Salvatore Sanfilippo. Design and implementation of a simple Twitter clone using PHP and the Redis key-value store. http://redis.io/topics/twitter-clone, 2015b.Google Scholar
- Marc Shapiro, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. Conflict-free Replicated Data Types. In Proceedings of the 13th International Conference on Stabilization, Safety, and Security of Distributed Systems, SSS, pages 386--400, 2011. ISBN 978-3-642-24549-7. Google ScholarDigital Library
- KC Sivaramakrishnan, Gowtham Kaki, and Suresh Jagannathan. Declarative programming over eventually consistent data stores. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation - PLDI 2015, PLDI. Association for Computing Machinery (ACM), 2015. doi:10.1145/2737924.2737981. Google ScholarDigital Library
- Yair Sovran, Russell Power, Marcos K. Aguilera, and Jinyang Li. Transactional storage for geo-replicated systems. In ACM Symposium on Operating Systems Principles - SOSP'11, SOSP. Association for Computing Machinery (ACM), 2011. doi:10.1145/2043556.2043592. Google ScholarDigital Library
- Jeremy Stribling, Yair Sovran, Irene Zhang, Xavid Pretzer, Jinyang Li, M. Frans Kaashoek, and Robert Morris. Flexible, wide-area storage for distributed systems with WheelFS. In Proceedings of the 6th USENIX Symposium on Networked Systems Design and Implementation (NSDI), NSDI'09, pages 43--58, Berkeley, CA, USA, 2009. USENIX Association. URL http://dl.acm.org/citation.cfm?id=1558977.1558981. Google ScholarDigital Library
- D.B. Terry, A.J. Demers, K. Petersen, M.J. Spreitzer, M.M. Theimer, and B.B. Welch. Session guarantees for weakly consistent replicated data. In Proceedings of 3rd International Conference on Parallel and Distributed Information Systems, PDIS. Institute of Electrical & Electronics Engineers (IEEE), 1994. doi:10.1109/pdis.1994.331722. Google ScholarDigital Library
- Douglas B. Terry, Vijayan Prabhakaran, Ramakrishna Kotla, Mahesh Balakrishnan, Marcos K. Aguilera, and Hussam Abu-Libdeh. Consistency-based service level agreements for cloud storage. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles - SOSP 13. ACM Press, 2013. doi:10.1145/2517349.2522731. Google ScholarDigital Library
- The Linux Foundation. netem. http://www.linuxfoundation.org/collaborate/workgroups/networking/netem, November 2009.Google Scholar
- Twitter, Inc. Finagle. https://twitter.github.io/finagle/, March 2016.Google Scholar
- Werner Vogels. Eventually consistent. Communications of the ACM, 52 (1): 40, January 2009. doi:10.1145/1435417.1435432. Google ScholarDigital Library
- Chao Xie, Chunzhi Su, Manos Kapritsos, Yang Wang, Navid Yaghmazadeh, Lorenzo Alvisi, and Prince Mahajan. Salt: Combining acid and base in a distributed database. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 495--509, Broomfield, CO, October 2014. USENIX Association. ISBN 978-1-931971-16-4. URL https://www.usenix.org/conference/osdi14/technical-sessions/presentation/xie. Google ScholarDigital Library
- Chao Xie, Chunzhi Su, Cody Littley, Lorenzo Alvisi, Manos Kapritsos, and Yang Wang. High-Performance ACID via Modular Concurrency Control. In ACM Symposium on Operating Systems Principles (SOSP), SOSP, pages 276--291, 2015. ISBN 978-1-4503-2388-8. doi:10.1145/2517349.2522729. Google ScholarDigital Library
- Haifeng Yu and Amin Vahdat. Design and evaluation of a conit-based continuous consistency model for replicated services. ACM Transactions on Computer Systems (TOCS), 20 (3): 239--282, 2002. Google ScholarDigital Library
Index Terms
- Disciplined Inconsistency with Consistency Types
Recommendations
Adaptive trade-off between consistency and performance in data replication
Replication is widely adopted in modern Internet applications and distributed systems to improve the reliability and performance. Though maintaining the strong consistency among replicas can guarantee the correctness of application behaviors, however, ...
On Mixing Eventual and Strong Consistency: Acute Cloud Types
In this article we study the properties of distributed systems that mix eventual and strong consistency. We formalize such systems through <italic>acute cloud types</italic> (ACTs), abstractions similar to conflict-free replicated data types (CRDTs), ...
Multi-consistency Data Replication
ICPADS '10: Proceedings of the 2010 IEEE 16th International Conference on Parallel and Distributed SystemsReplication is a technique widely used in parallel and distributed systems to provide qualities such as performance, scalability, reliability and availability to their clients. These qualities comprise the non-functional requirements of the system. But ...
Comments