Comparative Study of LoRA Methods For Text Classification Using RoBERTa

Varisan Falih Ramadhan, Yunita Sari, S.Kom., M.Sc., Ph.D.

2026 | Skripsi | ILMU KOMPUTER

Since their introduction in 2017, transformer models have always been considered the state-of-the-art models for Natural Language Processing tasks including text classification. However, the need for an extensive computational environment to train them has posed a great barrier in the realm of machine learning and deep learning. To address this issue, researchers have proposed Parameter-Efficient Fine-Tuning (PEFT) methods that allow the training of only a small percentage of the total trainable parameters while maintaining state-of-the-art performance. Despite the existence of various studies on PEFT, there is a limited number of comparative studies of Low-Rank Adaptation (LoRA) and its variants specifically focusing on text classification and their performance across different scenarios including binary text classification, multiclass classification datasets and in terms of dataset size including small, medium, and large. This experiment addresses the gap by evaluating four different variants of LoRA on four different data subsets with two different experimental scenarios: classification type (binary and multiclass) and dataset size (small, medium, and large) to provide a benchmark on their performance. The results showed that LoRA and RsLoRA yielded the highest performance metrics, with LoRA achieving accuracies of over 90?ross all subsets and RsLoRA achieving accuracies of above 93?ross all subsets. In terms of time efficiency, LoRA and RsLoRA were 10-15?ster in training duration than their counterparts. Ultimately, this study concludes there is not a single optimal method under the fixed hyperparameters as the differences in performance results were negligible and the pairwise statistical significance tests achieved a variation of results.

Since their introduction in 2017, transformer models have always been considered the state-of-the-art models for Natural Language Processing tasks including text classification. However, the need for an extensive computational environment to train them has posed a great barrier in the realm of machine learning and deep learning. To address this issue, researchers have proposed Parameter-Efficient Fine-Tuning (PEFT) methods that allow the training of only a small percentage of the total trainable parameters while maintaining state-of-the-art performance. Despite the existence of various studies on PEFT, there is a limited number of comparative studies of Low-Rank Adaptation (LoRA) and its variants specifically focusing on text classification and their performance across different scenarios including binary text classification, multiclass classification datasets and in terms of dataset size including small, medium, and large. This experiment addresses the gap by evaluating four different variants of LoRA on four different data subsets with two different experimental scenarios: classification type (binary and multiclass) and dataset size (small, medium, and large) to provide a benchmark on their performance. The results showed that LoRA and RsLoRA yielded the highest performance metrics, with LoRA achieving accuracies of over 90?ross all subsets and RsLoRA achieving accuracies of above 93?ross all subsets. In terms of time efficiency, LoRA and RsLoRA were 10-15 faster in training duration than their counterparts. Ultimately, this study concludes there is not a single optimal method under the fixed hyperparameters as the differences in performance results were negligible and the pairwise statistical significance tests achieved a variation of results.

Kata Kunci : Transformer, Natural Language Processing, Text Classification, Parameter-Efficient Fine-Tuning (PEFT), Low-Rank Adaptation (LoRA)