Marco Canini
Jon Crowcroft
ABSTRACT
Better reproducibility of networking research results is currently a major goal that the academic community is striving towards. This position paper makes the case that improving the extent and pervasiveness of reproducible research can be greatly fostered by organizing a yearly international contest. We argue that holding a contest undertaken by a plurality of students will have benefits that are two-fold. First, it will promote hands-on learning of skills that are helpful in producing artifacts at the replicable-research level. Second, it will advance the best practices regarding environments, testbeds, and tools that will aid the tasks of reproducibility evaluation committees by and large.
Supplemental Material
- AT&T's SDN Network Design Challenge. http://about.att.com/innovationblog/051515acallouttoacad.Google Scholar
- Cisco Networking Academy Contest. http://www.academynetriders.com/.Google Scholar
- Congestion-Control Contest. https://web.stanford.edu/class/cs344g/contest.html.Google Scholar
- DARPA Grand Challenge. http://www.darpa.mil/grandchallenge/.Google Scholar
- Jubilee 15th Challenge24 Contest Networking. http://ch24.org/networking.html.Google Scholar
- ONUG Grand Challenge Hackathon. https://opennetworkingusergroup.com/onug-grand-challenge-hackathon/.Google Scholar
- OpenFlow Driver Competition. https://www.opennetworking.org/competition.Google Scholar
- Reproducing Network Research. https://reproducingnetworkresearch.wordpress.com/.Google Scholar
- Stanford CS 244 -- Programming Assignment 3. http://web.stanford.edu/class/cs244/pa3.html.Google Scholar
- VideoNet -- A benchmark initiative for all things video. http://videonet.team/.Google Scholar
- Wireless Battle of the Mesh. http://battlemesh.org/.Google Scholar
- METRICS Bootcamp. http://metrics-itn.eu/metrics-bootcamp/, 2014.Google Scholar
- New gold standard established for open and reproducible research. http://www.cam.ac.uk/research/news/new-gold-standard-established-for-open-and-reproducible-research, 2015.Google Scholar
- R. F. Boisvert. Incentivizing Reproducibility. Commun. ACM, 59(10), Oct. 2016. Google ScholarDigital Library
- O. Bonaventure. The january 2017 issue. SIGCOMM Comput. Commun. Rev., 47(1), Jan. 2017. Google ScholarDigital Library
- B. Clark, T. Deshane, E. Dow, S. Evanchik, M. Finlayson, J. Herne, and J. N. Matthews. Xen and the Art of Repeated Research. In ATEC, 2004. Google ScholarDigital Library
- C. Collberg and T. A. Proebsting. Repeatability in Computer Systems Research. Commun. ACM, 59(3), Mar. 2016. Google ScholarDigital Library
- J. Crowcroft, M. Fidler, K. Nahrstedt, and R. Steinmetz. Future Internet (Dagstuhl Seminar 13131). Dagstuhl Reports, 3(3), 2013.Google Scholar
- S. Edwards, X. Liu, and N. Riga. Creating Repeatable Computer Science and Networking Experiments on Shared, Public Testbeds. SIGOPS Oper. Syst. Rev., 49(1), Jan. 2015. Google ScholarDigital Library
- E. Eide. Foreword: Special Issue on Repeatability and Sharing of Experimental Artifacts. SIGOPS Oper. Syst. Rev., 49(1), Jan. 2015. Google ScholarDigital Library
- J. Freire, P. Bonnet, and D. Shasha. Computational Reproducibility: State-of-the-art, Challenges, and Database Research Opportunities. In SIGMOD, 2012. Google ScholarDigital Library
- P. B. Godfrey. What are Networking's Grand Challenges? http://conferences2.sigcomm.org/co-next/2015/#!/keynote, 2015. CoNEXT keynote.Google Scholar
- N. Handigol, B. Heller, V. Jeyakumar, B. Lantz, and N. McKeown. Reproducible Network Experiments Using Container-based Emulation. In CoNEXT, 2012. Google ScholarDigital Library
- H. Howard, M. Schwarzkopf, A. Madhavapeddy, and J. Crowcroft. Raft Refloated: Do We Have Consensus? SIGOPS Oper. Syst. Rev., 49(1), Jan. 2015. Google ScholarDigital Library
- S. Krishnamurthi and J. Vitek. The Real Software Crisis: Repeatability As a Core Value. Commun. ACM, 58(3), Mar. 2015. Google ScholarDigital Library
- I. Manolescu, L. Afanasiev, A. Arion, J. Dittrich, S. Manegold, N. Polyzotis, K. Schnaitter, P. Senellart, S. Zoupanos, and D. Shasha. The Repeatability Experiment of SIGMOD 2008. SIGMOD Rec., 37(1), Mar. 2008. Google ScholarDigital Library
- D. Pediaditakis, C. Rotsos, and A. W. Moore. Faithful Reproduction of Network Experiments. In ANCS, 2014. Google ScholarDigital Library
- J. Rexford. The Networking Philosopher's Problem. SIGCOMM Comput. Commun. Rev., 41(3), July 2011. Google ScholarDigital Library
- R. Ricci, G. Wong, L. Stoller, K. Webb, J. Duerig, K. Downie, and M. Hibler. Apt: A Platform for Repeatable Research in Computer Science. SIGOPS Oper. Syst. Rev., 49(1), Jan. 2015. Google ScholarDigital Library
- Q. Scheitle, M. Wählisch, O. Gasser, T. C. Schmidt, and G. Carle. Towards an Ecosystem for Reproducible Research in Computer Networking. In Reproducibility, 2017. Google ScholarDigital Library
- A. Sivaraman, K. Winstein, P. Varley, J. a. Batalha, A. Goyal, S. Das, J. Ma, and H. Balakrishnan. Protocol Design Contests. SIGCOMM Comput. Commun. Rev., 44(3), July 2014. Google ScholarDigital Library
- V. Stodden, M. McNutt, D. H. Bailey, E. Deelman, Y. Gil, B. Hanson, M. A. Heroux, J. P. Ioannidis, and M. Taufer. Enhancing reproducibility for computational methods. Science, 354(6317):1240--1241, 2016.Google ScholarCross Ref
- L. Yan and N. McKeown. Learning Networking by Reproducing Research Results. SIGCOMM Comput. Commun. Rev., 47(2), Apr. 2017. Google ScholarDigital Library
Index Terms
- Learning Reproducibility with a Yearly Networking Contest
Comments