CLASSIFICATION OF VISUAL-VERBAL COGNITIVE STYLE IN MULTIMEDIA LEARNING USING EYE-TRACKING AND MACHINE LEARNING

ALOYSIUS GONZAGA P S, Dr. Eng. Sunu Wibirama, S.T., M.Eng.;Teguh Bharata Adji, S.T., M.T., M.Eng.

2020 | Tesis | MAGISTER TEKNOLOGI INFORMASI

Multimedia learning didefinisikan sebagai membangun representasi mental dari kata-kata dan gambar. Dalam multimedia learning, perbedaan gaya kognitif menunjukkan strategi pembelajaran yang berbeda. Gaya kognitif visual dan verbal memberikan pengaruh pada perilaku, preferensi, dan bahkan hasil belajar. Eye-tracking telah digunakan dalam studi-studi sebelumnya tentang aktivitas kognitif selama multimedia learning. Namun, metrik eye-tracking terbatas sebagai pengukuran validasi. Data pergerakan mata hanya diproses dalam metode statistik deskriptif atau representasi visual dalam bentuk heat map. Sejauh pengetahuan penulis, belum ada penelitian yang mengimplementasikan machine learning pada data eye-tracking untuk kasus klasifikasi otomatis gaya kognitif visual-verbal. Banyak studi sudah menerapkan metode machine learning untuk memfasilitasi metode pengolahan data yang efektif terhadap data pergerakan mata. Tren ini memberikan peluang pengembangan bagian dari multimedia learning adaptif. Oleh karena itu, penelitian ini mengusulkan pendekatan baru untuk secara otomatis mengklasifikasikan gaya kognitif visual-verbal dengan memanfaatkan pendekatan machine learning yang dikombinasikan dengan data eye-tracking. Dalam penelitian ini, 70 peserta terdiri dari 35 pelajar visual dan 35 pelajar verbal diminta untuk mempelajari suatu topik menggunakan stimulus. Stimulus terdiri dari kombinasi gambar dan teks. Gerakan mata peserta direkam menggunakan pelacak mata. Fitur dari metrik eye-tracking dipilih dengan menggunakan metode SVM-Recursive Feature Elimination dan SelectKBest. Tujuh algoritma klasifikasi: K-Nearest Neighbors, Linear Discriminant Analysis, Logistic Regression, Gaussian Naive Bayes, Random Forest, Support Vector Machine, dan Multi-layer Perceptron dilatih dan diuji menggunakan dataset metrik pergerakan mata. Algoritme-algoritme ini telah umum digunakan dalam banyak studi mengenai machine learning dan diimplementasikan kepada dataset eye-tracking. Berdasarkan hasil percobaan, Multi-layer Perceptron---ditingkatkan dengan tiga fitur yang dipilih dari SelectKBest---memperoleh 83% akurasi klasifikasi. Hasil ini dapat digunakan sebagai pedoman dasar untuk pengembangan multimedia learning adaptif, peningkatan dalam metode pengolahan data gerakan mata, dan klasifikasi otomatis gaya kognitif visual-verbal.

Multimedia learning is defined as building mental representations from words and pictures. In multimedia learning, the difference in cognitive style indicates different learning strategies. The cognitive style of visual and verbal exert influence on behavior, preferences, and even learning outcomes. Eye-tracking has been used in previous studies of cognitive activities during multimedia learning. However, eye-tracking metrics were limited as validation measurements. Eye movement data were merely processed in a statistical descriptive method or visual representation in the form of a heat map. To the best knowledge of authors, there is no study that implements machine learning on eye-tracking data for automatic classification of visual-verbal cognitive style. Meanwhile, recent studies have been implemented machine learning to facilitate robust data processing method toward eye movement data. These trends provide opportunities in a part of adaptive multimedia learning development. Hence, this study proposes a new approach to automatically classify the visual-verbal cognitive style by utilizing a machine learning approach combined with eye-tracking data. In this study, 70 participants consist of 35 visual learners and 35 verbal learners were asked to learn a topic using a stimulus. The stimulus consists of a combination of image and text. The eye movements of the participants were recorded using an eye tracker. Features from eye-tracking metrics were selected by using SVM-Recursive Feature Elimination and SelectKBest method. Seven shallow classifiers: K-Nearest Neighbors, Linear Discriminant Analysis, Logistic Regression, Gaussian Naive Bayes, Random Forest, Support Vector Machine, and Multi-layer Perceptron were trained and tested using the eye movement dataset. These algorithms have been commonly used in many machine learning studies and implemented toward eye-tracking data. Based on experimental results, Multi-layer Perceptron---enhanced with three selected features from SelectKBes---gained 83% of classification accuracy. This result can be used as a baseline guide for the development of adaptive multimedia learning in the aspect of the data processing method with eye movement data.

Kata Kunci : cognitive style, visual-verbal, eye-tracking, multimedia learning, machine learning