Ethan Fast
Alex Aiken
Michael S. Bernstein
Daphne Koller
ABSTRACT
Large online courses often assign problems that are easy to grade because they have a fixed set of solutions (such as multiple choice), but grading and guiding students is more difficult in problem domains that have an unbounded number of correct answers. One such domain is derivations: sequences of logical steps commonly used in assignments for technical, mathematical and scientific subjects. We present DeduceIt, a system for creating, grading, and analyzing derivation assignments in any formal domain. DeduceIt supports assignments in any logical formalism, provides students with incremental feedback, and aggregates student paths through each proof to produce instructor analytics. DeduceIt benefits from checking thousands of derivations on the web: it introduces a proof cache, a novel data structure which leverages a crowd of students to decrease the cost of checking derivations and providing real-time, constructive feedback. We evaluate DeduceIt with 990 students in an online compilers course, finding students take advantage of its incremental feedback and instructors benefit from its structured insights into course topics. Our work suggests that automated reasoning can extend online assignments and large-scale education to many new domains.
- Coursera support documentation. http://support.coursera.org.Google Scholar
- Bennett, R. E., and Bejar, I. I. Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice 17, 4 (1998), 9--17.Google ScholarCross Ref
- Bennett, R. E., Steffen, M., Singley, M. K., Morley, M., and Jacquemin, D. Evaluating an automatically scorable, open-ended response type for measuring mathematical reasoning in computer-adaptive tests. Journal of Educational Measurement 34, 2 (1997), pp. 162--176.Google ScholarCross Ref
- Burstall, R. Proveeasy: Helping people learn to do proofs. In Proc. ENTCS 2000 (2000), 16--32.Google Scholar
- Cheang, B., Kurnia, A., Lim, A., and Oon, W.-C. On automated grading of programming assignments in an academic institution. Comput. Educ. 41, 2 (2003), 121--131. Google ScholarDigital Library
- Clavel, M., Durán, F., Eker, S., Lincoln, P., Mart-Oliet, N., Meseguer, J., and Quesada, J. Maude as a metalanguage. In Proc. WRLA 1998 15 (1998).Google Scholar
- Corbett, A., and Anderson, J. Knowledge tracing: Modeling the acquisition of procedural knowledge. In Proc. UMUAI 1994 (1994), 253--278.Google Scholar
- Corbett, A. T., and Anderson, J. R. Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes. In Proc. CHI '01 (2001). Google ScholarDigital Library
- Gallien, T., and Oomen-Early, J. Personalized versus collective instructor feedback in the online courseroom: Does type of feedback affect student satisfaction, academic performance and perceived connectedness with the instructor? International Journal on E-Learning 7, 3 (2008), 463--476.Google Scholar
- Hearst, M. The debate on automated essay grading. Intelligent Systems and their Applications, IEEE 15, 5 (2000), 22--37. Google ScholarDigital Library
- Heffernan, N. T., Koedinger, K. R., and Razzaq, L. Expanding the model-tracing architecture: A 3rd generation intelligent tutor for algebra symbolization. Int. J. Artif. Intell. Ed. (2008), 153--178. Google ScholarDigital Library
- Hernan-Losada, I., Pareja-Flores, C., and Velazquez-Iturbide, A. Testing-based automatic grading: A proposal from bloom's taxonomy. In Proc. ICALT 2008 (2008), 847--849. Google ScholarDigital Library
- Huang, S.-W., and Fu, W.-T. Enhancing reliability using peer consistency evaluation in human computation. In Proc. CSCW 2013 (2013), 639--648. Google ScholarDigital Library
- Kaindl, H., and Kainz, G. Bidirectional heuristic search reconsidered. Journal of Artificial Intelligence Research 7 (1997), 283--317. Google ScholarCross Ref
- Kaufmann, M., and Moore, J. S. An industrial strength theorem prover for a logic based on common lisp. IEEE Trans. Softw. Eng. 23, 4 (1997), 203--213. Google ScholarDigital Library
- Kirsh, D., and Maglio, P. P. On Distinguishing Epistemic from Pragmatic Action. Cognitive Science 18, 4 (1994), 513--549.Google ScholarCross Ref
- Kulkarni, C., Pang, K., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., and Klemmer, S. Peer and self assessment in massive online design classes. ACM TOCHI (2013).Google Scholar
- Lapets, A., Skowyra, R., Bassem, C., Kfoury, A., and Bestavros, A. Towards an infrastructure for integrated accessible formal reasoning environments. In Proc. UITP 2012.Google Scholar
- Mart-Oliet, N., and Meseguer, J. Rewriting logic: Roadmap and bibliography. J. Log. Algebr. Program. 81 (2001).Google Scholar
- Nielsen, J. Usability Engineering. Morgan Kaufmann, 1993. Google ScholarDigital Library
- Nipkow, T., Wenzel, M., and Paulson, L. C. Isabelle/HOL: a proof assistant for higher-order logic. Springer-Verlag, Berlin, Heidelberg, 2002. Google ScholarCross Ref
- Pappano, L. Massive open online courses are multiplying at a rapid pace. http://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html.Google Scholar
- Paulin-Mohring, C. Inductive definitions in the system coq rules and properties. TLCA 1993 (1993). Google ScholarDigital Library
- Paulson, L. C. The foundation of a generic theorem prover. Journal of Automated Reasoning 5 (1989). Google ScholarDigital Library
- Ritter, S., Towle, B., Murray, R., Hausmann, R., and Connelly, J. A cognitive tutor for geometric proof. In Prof. ITS 2010 (2010), 453--453. Google ScholarDigital Library
- Schleimer, S., Wilkerson, D. S., and Aiken, A. Winnowing: local algorithms for document fingerprinting. In Proc. ACM SIGMOD 2003 (2003), 76--85. Google ScholarDigital Library
- Suppes, P. The next generation of interactive theorem provers. 7th International Conference on Automated Deduction 170 (1984), 303--315. Google ScholarDigital Library
- Suppes, P. Student use of an interactive theorem prover. Contemporary Mathematics 29 (1984).Google Scholar
- Tosic, M., and Nejkovic, V. Trust-based peer assessment for virtual learning systems. In Proc. SocInfo 2010 (2010), 176--191. Google ScholarDigital Library
- VanLehn, K. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist 46, 4 (2011), 197--221.Google ScholarCross Ref
- Windsteiger, W. Theorema 2.0: A graphical user interface for a mathematical assistant system. CEUR Workshop Proceedings (2012), 73--81.Google Scholar
Index Terms
- Crowd-scale interactive formal reasoning and analytics
Recommendations
Paraconsistent Circumscription: First-Order Case
ISMVL '95: Proceedings of the 25th International Symposium on Multiple-Valued LogicAbstract: In this paper we describe paraconsistent circumscription by application of predicate circumscription in a paraconsistent logic. In addition to circumscribe the predicates, we also circumscribe the inconsistency. Paraconsistent circumscription ...
Coinductive Verification of Program Optimizations Using Similarity Relations
Formal verification methods have gained increased importance due to their ability to guarantee system correctness and improve reliability. Nevertheless, the question how proofs are to be formalized in theorem provers is far from being trivial, yet very ...
Building Automated Theorem Provers for Infinitely-Valued Logics with Satisfiability Modulo Theory Solvers
ISMVL '12: Proceedings of the 2012 IEEE 42nd International Symposium on Multiple-Valued LogicThere is a relatively large number of papers dealing with complexity and proof theory issues of infinitely-valued logics. Nevertheless, little attention has been paid so far to the development of efficient solvers for such logics. In this paper we show ...
Comments