Advance in Detecting Key Concepts as an Expert Model: Using Student Mental Model Analyzer for Research and Teaching (SMART)

Abstract

While key concepts embedded within an expert’s textual explanation have been considered an aspect of expert model, the complexity of textual data makes determining key concepts demanding and time consuming. To address this issue, we developed Student Mental Model Analyzer for Teaching and Learning (SMART) technology that can analyze an expert’ textual explanation to elicit an expert concept map from which key concepts are automatically derived. SMART draws on four graph-based metrics (i.e., clustering coefficient, betweenness, PageRank, and closeness) to automatically filter key concepts from experts’ concept maps. This study investigated which filtering method extract key concepts most accurately. Using 18 expert textual data, we compared the accuracy levels of those four competing filtering methods by referring to four accuracy measures (i.e., precision, recall, F-measure, and N-similarity). The results showed the PageRank filtering method outperformed the other methods in all accuracy measures. For example, on average, PageRank derived 79% of key concepts as accurately as human experts. SMART’s automatic filtering methods can help human experts save time when building an expert model, and it can validate their decision making on a list of key concepts.

Appendix: Examples of Expert Textual Explanations (Human-Judged Key Concepts were Bolded in the Texts)

1.1 Model 1

Evaluation is a fundamental component of the instructional design. Evaluation is the process of determining merit, worth, and value of things. The types of evaluation include confirmatory evaluation, formative evaluation, and summative evaluation. For example, formative evaluation supports the process of improvement, focusing on learner ability. Summative evaluation focuses on the overall effectiveness, usefulness, or worth of the instruction. This chapter introduces several evaluation models. Stufflebeam proposes the CIPP model that stands for context, input, process, and product evaluation. CIPP model influenced program planning, program structuring, implementation decisions. In the CIPP model, an evaluator often participates in a project as a member of the project team. From a broad perspective, Rossi views that evaluation can include needs assessment, theory assessment, implementation assessment, impact, and efficiency assessment. Chen proposes theory-driven evaluation in which evaluators and stakeholders work together. The important role of an evaluator is to help articulate, evaluate, and improve the program theory including an action model and change model. Kirkpatrick suggests that training evaluation should exam four levels of the outcomes including reaction, learning, behavior, and business results. Brinkerhoff emphasizes the use of success case to evaluate a program. He suggests that an organization can gain profits by applying knowledge learned from success cases. Lastly, Patton views that the use of evaluation findings is critical, and thus his evaluation model focuses on producing evaluation use. The utility of evaluation is judged by the degree of use. The use of evaluation findings can increase when stakeholders become active participants in the evaluation process

1.2 Model 2

Technology implementations usually begin with an identified instructional need. An instructional need was likely not fully identified due to the insufficient study of how instructional practices in the classroom were being conducted already without the technology. One big issue is defining what a successful integration or change in instructional practice actually is. While teachers in the situation may have felt that they knew this already, the assumptions inherent in a design situation need to be articulated and checked if the assumptions are not to distort the design space by which instructional practices are manipulated. Teachers didn’t have enough professional development using the technology in classroom teaching and learning, on ways to integrate use into their teaching, and best practices with regard to effective educational use. Teacher professional development that discusses not just technical know-how but also pedagogy could help teachers realize how to do things differently that takes full advantage of the affordances of the tablets. Training as a professional development should be extensive including teacher beliefs and attitude. Teacher beliefs play a role in adopting new practices and changing their instructional practice. Teachers may not believe that students learn with laptops, and thus do not use laptops in their instruction. The only support teachers had during implementation was technical support; Teachers lacked a mentor who could assist them as instructional issues arose throughout the year. Mentoring on additional and advanced uses of the technology in the classroom is critical for teachers to increase their skills and maintain their motivation in utilizing the technology. In addition, a mentor could help teachers to maintain the belief that these efforts will have positive results. There are concerns that the environment does not support change. An ongoing supportive environment where teachers initially learn how to use the technology, how to use the technology with their content, and how to continue to develop their expertise in the technology and incorporating it to the classroom is critical The environment could include a culture that does not support the desired performance. For example, the lack of incentives to make effective use of a new technology could also contribute to a lack of use. The intervention seems to have been applied to this community rather than involving teachers from the beginning as collaborators in its design and modification. Teachers were not involved in the decision to implement the new media; thus, they did not fully “buy into” the plan

1.3 Model 3¹

Atoms, the basic building blocks of matter, are made of three basic components: protons, neutrons and electrons. The protons and neutrons cluster together to form the nucleus, the central part of the atom, and the electrons orbit about the nucleus. Protons and electrons both carry an electrical charge. The charges they carry are opposite to each other; protons carry a positive electrical charge while electrons carry a negative electrical charge. Neutrons are neutrally charged - they carry no charge at all Electricity is the movement of charged particles, usually electrons, from one place to another. Materials that electricity can move through easily are called conductors. Most metals, such as iron, copper and aluminum, are good conductors of electricity. Other materials, such as rubber, wood and glass, block the flow of electricity. Materials which prevent the flow of electricity are called insulators. Electrical cords are usually made with both conductors and insulators. Electricity flows through a conductor in the center of the cord. A layer of insulation surrounds the conductor and prevents the electricity from ‘leaking’ out Objects usually have equal numbers of positive and negative charges, but it isn’t too hard to temporarily create an imbalance. One way scientists can create an imbalance is with a machine called a Van de Graaff generator. It creates a large static charge by placing electrons on a metal dome using a motor and a big rubber band. Since like charges repel, the electrons push away from each other as they collect on the do me. Eventually, too many electrons are placed on the dome and they leap off, creating a spark that looks like a bolt of lightning Have you ever received a shock after having walked across a carpet? This shock was caused by extra electrons you collected while walking across the carpet. Your body became like the dome of the Van de Graaff generator, full of extra electrons looking for a way to get away. The path back to the carpet was blocked by the shoes you were wearing, but they were able to move through your hand and into the object that you touched, causing the shock. So, the next time you shuffle across a carpet and shock your friend on the ear, tell them you were just trying to be a Van de Graaff generator!