Toward Secure Federated Learning for Internet of Things using Deep Reinforcement Learning (DRL)-enabled Reputation Mechanism

Abstract

Federated Learning (FL) has emerged to leverage datasets from multiple devices to improve the performance of a Machine Learning (ML) model while providing privacy preservation for devices. The training data is collected at the devices, called workers, which collaboratively train a global learning model and share their local model updates with a central entity or server without sharing their actual data. Hierarchical Federated Learning (HFL) framework was introduced to improve FL communication efficiency where devices are clustered and seek model consensus with the support of edge servers (e.g., base stations). The devices in a cluster submit their local model updates to their assigned local edge server for aggregation at each iteration. The edge servers transmit the aggregated model updates to a centralized server and establish a global consensus. However, both FL and HFL can be susceptible to various adversarial attacks that target their security and privacy. In particular, the workers within a cluster can upload unreliable local model updates, leading to the corruption of the main FL task. Workers may intentionally contribute unreliable local updates by launching poisoning attacks or unintentionally by updating low-quality models caused by high device mobility, limited device resources, or unstable network connection. Consequently, identifying reliable and trustworthy workers becomes critical for FL security. In this thesis, the concept of reputation is adopted as a metric to evaluate workers' reliability and trustworthiness in the FL and HFL frameworks. In addition, Deep Reinforcement Learning (DRL)-based reputation mechanism is proposed for optimal selection and evaluation of reliable FL workers. Due to the dynamic nature of worker behavior in the FL environment, the DRL-based algorithm Deep Deterministic Policy Gradient (DDPG) is employed to improve the FL model's accuracy and stability. Moreover, a Multi-Agent DDPG (MADDPG)-based reputation model is proposed to enhance the accuracy and stability of HFL. We compare the performance of the DDPG-based model with a conventional reputation model and Deep Q-Networks (DQN)-based reputation model. Our simulation results demonstrate that the DDPG-based model can improve the FL accuracy by more than 30\% under various scenarios and achieves better convergence compared to other methods. Moreover, comparing the MADDPG-based reputation model to the conventional reputation model and the single-agent DDPG-based reputation model reveals that MADDPG enhances the accuracy, stability, and convergence of the HFL global model.

Date of Award	2023
Original language	American English
Awarding Institution	HBKU College of Science and Engineering