Analisis Performa Algoritma Demonstration-Guided Reinforcement Learning untuk Tugas Needle Pick Menggunakan Simulator SurRoL

Firda Salmah Atiatul Maula, Dr.Eng. Ir. Igi Ardiyanto, S.T., M.Eng., IPM., ASEAN Eng., SMIEEE. ; Ahmad Ataka Awwalur Rizqi , S.T., Ph.D

2026 | Skripsi | S1 TEKNIK BIOMEDIS

Operasi bedah merupakan prosedur medis yang membutuhkan tingkat presisi dan konsentrasi yang tinggi. Namun, durasi operasi yang panjang dan beban kerja yang tinggi dapat menyebabkan kelelahan pada dokter bedah sehingga berpotensi menurunkan kualitas tindakan medis. Salah satu solusi yang dapat dikembangkan untuk membantu otomatisasi tugas bedah adalah pemanfaatan robot bedah yang mampu mempelajari tugas manipulasi melalui pendekatan pembelajaran mesin. Pada bidang robotika, Reinforcement Learning (RL) memiliki kemampuan untuk mempelajari kebijakan secara mandiri, tetapi sering mengalami kesulitan pada lingkungan dengan sparse reward. Sementara itu, Imitation Learning (IL) mampu mempercepat pembelajaran melalui demonstrasi pakar, tetapi memiliki keterbatasan dalam kemampuan generalisasi. Untuk mengatasi permasalahan tersebut, dikembangkan pendekatan Demonstration-guided Reinforcement Learning (DGRL) yang menggabungkan keunggulan RL dan IL. Penelitian ini bertujuan untuk menganalisis performa pendekatan DGRL pada tugas Needle Pick menggunakan robot dVRK dalam lingkungan simulasi SurRoL.

Penelitian dilakukan dengan membandingkan algoritma RL, IL, dan DGRL yang terdiri atas SAC, DDPG, BC, SQIL, DDPGBC, dan DEX. Evaluasi dilakukan pada environment Needle Pick di simulator SurRoL hingga 100.000 environment steps menggunakan lima random seed yang berbeda. Kinerja setiap algoritma dianalisis berdasarkan metrik Success Rate dan Average Episode Reward. Selain itu, dilakukan studi sensitivitas hyperparameter pada algoritma DEX untuk mengevaluasi pengaruh parameter exploration weight (?) dan jumlah k-nearest neighbors (k) terhadap performa pembelajaran agen.

Hasil penelitian menunjukkan bahwa algoritma berbasis DGRL menghasilkan performa terbaik dengan Success Rate mencapai 0, 97, melampaui algoritma RL dan IL yang diuji. DDPGBC menunjukkan performa paling optimal berdasarkan nilai Average Episode Reward dan tingkat stabilitas pelatihan yang tinggi. Hasil studi sensitivitas pada algoritma DEX menunjukkan bahwa variasi parameter exploration weight ? = 5 dan konfigurasi k-nearest neighbors k = 5 memberikan kontribusi terbaik terhadap akselerasi konvergensi dan performa akhir agen. Dengan demikian, dapat disimpulkan bahwa pendekatan Demonstration-guided Reinforcement Learning efektif dalam mengatasi permasalahan sparse reward pada tugas Needle Pick dan mampu menghasilkan kebijakan kontrol yang lebih optimal dibandingkan pendekatan Reinforcement Learning maupun Imitation Learning secara terpisah.

Surgery is a medical procedure that requires a high degree of precision and concentration. However, long operating times and heavy workloads can cause fatigue in surgeons, which has the potential to compromise the quality of medical care. One solution that can be developed to help automate surgical tasks is the use of surgical robots capable of learning manipulation tasks through machine learning. In the field of robotics, Reinforcement Learning (RL) has the ability to learn policies independently, but often faces difficulties in environments with sparse rewards. Meanwhile, Imitation Learning (IL) can accelerate learning through expert demonstrations but has limitations in its generalization ability. To address these issues, a Demonstration-guided Reinforcement Learning (DGRL) approach was developed that combines the strengths of RL and IL. This study aims to analyze the performance of the DGRL approach on the Needle Pick task using the dVRK robot in the SurRoL simulation environment. 

The research was conducted by comparing RL, IL, and DGRL algorithms, specifically SAC, DDPG, BC, SQIL, DDPGBC, and DEX. The evaluation was performed on the Needle Pick environment in the SurRoL simulator for up to 100, 000 environment steps using five different random seeds. The performance of each algorithm was analyzed based on the Success Rate and Average Episode Reward metrics. Additionally, a hyperparameter sensitivity study was conducted on the DEX algorithm to evaluate the effect of the exploration weight parameter (?) and the number of k-nearest neighbors (k) on the agent’s learning performance. 

The results show that the DGRL-based algorithm achieved the best performance with a Success Rate of 0.97, outperforming the tested RL and IL algorithms. DDPGBC demonstrated the most optimal performance based on the Average Episode Reward and a high level of training stability. Sensitivity analysis results for the DEX algorithm indicate that a exploration weight of ? = 5 and a k-nearest neighbors configuration of k = 5 contribute most effectively to accelerating convergence and improving the agent’s final performance. Thus, it can be concluded that the Demonstration-guided Reinforcement Learning approach is effective in addressing the sparse reward problem in the Needle Pick task and is capable of producing more optimal control policies compared to Reinforcement Learning or Imitation Learning approaches used separately.

Kata Kunci : Demonstration-guided Reinforcement Learning, Needle Pick, SurRoL