Development and Evaluation of a Generative Artificial Intelligence and Human Collaborative Multiple Choice Item Generation Model

Project no. 32503 · Status: Kapanmış (Yayın bekleniyor) · 2024

Summary

The project aims to develop a model that aims to produce multiple-choice items following the principles of question writing for different levels of objectives, cognitive levels, and difficulty from the Secondary School Mathematics curriculum by using educational materials and the existing question pool with the cooperation of artificial intelligence and construction, and to share methodological information about how this model can be developed with researchers. ChatGPT was used as an artificial intelligence model in the project. For the training and evaluation of the model, a multiple-choice item writing principles evaluation form was developed and its psychometric properties were determined and it was shown to be valid and reliable. Trained and untrained models were compared in terms of item quality level scores and the appropriateness of these items to the learning outcome, cognitive level, and item difficulty. The item quality score of the untrained model was significantly lower than the trained model. Similarly, while the untrained model showed 92% success performance in item production according to item gain, the trained model showed 96% performance. For other item properties, much more dramatic performance differences were found: the trained model performed 92% while the untrained model performed 72%. When the model performances were compared in terms of conformity to item difficulty levels, the untrained model performed with a success level of 56%. The trained model showed 90% success performance in writing questions appropriate to the item difficulty level. When the appropriateness of the items to the cognitive level is analyzed, the trained model performed with 92% success performance while the untrained model performed with 72% performance. The trained model was shared openly on ChatGPT as a GPT named IteMatic. Keywords: Artificial intelligence and human collaboration, Artificial intelligence in education, Item generation with artificial intelligence, Artificial intelligence and multiple-choice items, Artificial intelligence in measurement and evaluation

View details