Deteksi Serangan DDoS Pada Layanan IoT dengan Pendekatan Deep Reinforcement Learning

syaifuddin, Prof. Dr. Ir. Ridi Ferdiana, S.T., M.T., IPM. ; Widyawan, S.T., M. Sc., Ph.D.

2026 | Disertasi | S3 Teknik Elektro

Perangkat Internet of Things (IoT) memiliki keterbatasan sumber daya, menjadikannya rentan terhadap serangan Distributed Denial of Service (DDoS) yang bertujuan mengganggu ketersediaan layanan pada ekosistem IoT. Metode deteksi konvensional berbasis Machine Learning sering kali bersifat statis dan bergantung pada dataset historis. Ketergantungan ini menyebabkan penurunan kinerja yang signifikan ketika sistem dihadapkan pada pola serangan baru (zero-day). Untuk mengatasi sifat statis tersebut dan memberikan kemampuan adaptasi pembelajaran berkelanjutan, disertasi ini mengusulkan sistem deteksi menggunakan pendekatan Deep Reinforcement Learning (DRL). Sistem yang dikembangkan memanfaatkan dataset primer IoT-DH, yang dikumpulkan menggunakan honeypot RIoTPot untuk menangkap lalu lintas serangan autentik pada berbagai protokol IoT (MQTT, CoAP, HTTP, Telnet, SSH, dan ICMP). Beberapa varian DRL (Q-Learning, DQN, Double DQN, dan Dueling DQN) dievaluasi sebagai algoritma pembanding. Dari evaluasi tersebut, Dueling DQN terpilih sebagai model dasar (baseline) karena keunggulan arsitekturnya dalam mengevaluasi kondisi jaringan. Model Dueling DQN tersebut kemudian ditingkatkan performanya melalui penerapan Prioritized Experience Replay (PER), Adaptive Reward Shaping (ARS), dan Huber Loss guna meningkatkan stabilitas serta efisiensi pembelajaran. Hasil pengujian pada dataset primer (IoT-DH) menunjukkan bahwa model Enhanced Dueling DQN mencapai metrik performa tertinggi dibandingkan dengan algoritma lain yang dievaluasi, dengan nilai F1-Score sebesar 0,981. Guna memvalidasi ketahanan dan kemampuan generalisasi, algoritma yang diusulkan dievaluasi lebih lanjut menggunakan sejumlah dataset publik standar. Hasil pengujian lintas-dataset menunjukkan kinerja yang konsisten dengan rata-rata akurasi sebesar 97,68?n rata-rata F1-Score mencapai 0,9753. Capaian performa yang tinggi membuktikan kemampuan adaptasi model secara efektif terhadap skenario serangan zero-day, berbanding terbalik dengan metode konvensional yang secara empiris mengalami degradasi kinerja secara signifikan. Secara keseluruhan, temuan dari penelitian ini mengindikasikan bahwa arsitektur pendeteksi yang dikembangkan memiliki potensi untuk diimplementasikan secara ringan dan efisien pada perangkat edge atau gateway dengan sumber daya komputasi terbatas.

Internet of Things (IoT) devices are characterized by limited computational resources, making them vulnerable to Distributed Denial of Service (DDoS) attacks aimed at disrupting service availability within the IoT ecosystem. Conventional detection methods based on Machine Learning are often static and heavily dependent on historical datasets. This dependency leads to significant performance degradation when the system is confronted with novel attack patterns (zero-day). To address this limitation and enable continuous adaptive learning, this dissertation proposes a detection system based on a Deep Reinforcement Learning (DRL) approach. The developed system utilizes a primary dataset, IoT-DH, collected using the RIoTPot honeypot to capture authentic attack traffic across multiple IoT protocols, including MQTT, CoAP, HTTP, Telnet, SSH, and ICMP. Several DRL variants (Q-Learning, DQN, Double DQN, and Dueling DQN) were evaluated as comparative algorithms. Based on this evaluation, Dueling DQN was selected as the baseline model due to its architectural advantage in assessing network conditions. The Dueling DQN model was further enhanced through the implementation of Prioritized Experience Replay (PER), Adaptive Reward Shaping (ARS), and Huber Loss to improve learning stability and efficiency. Experimental results on the primary dataset (IoT-DH) show that the Enhanced Dueling DQN model achieves the highest performance among all evaluated algorithms, with an F1-Score of 0.981. To validate robustness and generalization capability, the proposed algorithm was further evaluated using several standard public datasets. Cross-dataset evaluation demonstrates consistent performance, achieving an average accuracy of 97.68% and an average F1-Score of 0.9753. The high performance indicates the model’s ability to effectively adapt to zero-day attack scenarios, in contrast to conventional methods which empirically exhibit significant performance degradation. Overall, the findings of this research suggest that the proposed detection architecture has strong potential for lightweight and efficient implementation on edge or gateway devices with limited computational resources.

Kata Kunci : Distributed Denial of Service (DDoS), Deep Reinforcement Learning (DRL), Internet of Things (IoT), Adaptive Detection, Honeypot