Pengembangan Automated Short Answer Grading untuk Bahasa Indonesia berbasis Transformers dengan Contrastive Learning
Aldo Arya Saka Mukti, Syukron Abu Ishaq Alfarozi, S.T., Ph.D; Dr. Ir. Sri Suning Kusumawardani, S.T., M.T.
2023 | Skripsi | TEKNOLOGI INFORMASI
The rapid development of technology has impacted various sectors, including education. These developments have enabled e-Learning to thrive, especially during the Covid-19 pandemic. Evaluating student performance and understanding in e-Learning is typically done through quizzes in multiple-choice or essay formats. However, these evaluations, especially in essay grading, still require manual effort. This can lead to exhaustion and introduce bias and inconsistency into the scoring process. To address this issue, one possible solution is to develop an automated short-answer grading system.
This research explores one of the deep learning models, which is a large language model that has a general understanding of language. This model is then subjected to a finetuning process. Specifically, this study employs a transformer model, especially BERT, with contrastive learning method to develop an automated short-answer scoring system and compare its performance with similar systems. The model is composed of two components, namely the model body which utilizes BERT variation, which is 'bert-base-multilingual-cased', and the model head which employs logistic regression. The body model is structured in a siamese architecture.
The results of this research demonstrate an improvement in model performance by utilizing grid search for hyperparameter and contrastive score limit optimization with 10-fold cross-validation. When compared to the pretrained BERT (baseline model) and BERT with cosine similarity finetuning (finetuning model), the reduction in prediction errors, measured by the MAE metric, is 21.72% and 9.90%, while for the RMSE metric, it is 17.79% and 13.80%. The transformers-based model with contrastive learning achieves metrics of 0.191 for MAE and 0.231 for RMSE. These findings indicate the potential of using the contrastive learning method in transformers models to develop an automated short-answer scoring system.
Kata Kunci : contrastive learning, transformers, sistem penilai jawaban pendek otomatis, e-Learning, deep learning