Distributed artificial intelligence systems are increasingly vital, yet their reliance on centralized data aggregation poses significant privacy and scalability challenges. A recent study has delved into an alternative paradigm: fully decentralized federated learning leveraging peer-to-peer gossip protocols, combined with robust client-side differential privacy measures. This research moves beyond conventional server-client models to explore how data privacy mechanisms interact with network communication structures in a truly distributed environment.
Deconstructing Decentralized Learning
The investigation implemented both traditional centralized Federated Averaging (FedAvg) and a novel decentralized Gossip Federated Learning approach from the ground up. Central to the decentralized model is the removal of the single aggregation server, replaced by a peer-to-peer gossip mechanism where clients directly exchange model updates. To ensure user data confidentiality, client-side differential privacy was integrated by introducing precisely calibrated noise into local model updates.
Controlled experiments were conducted using non-IID MNIST data, a common benchmark for federated learning, to realistically simulate data heterogeneity among clients. The experimental setup carefully managed execution environments and dependencies to ensure reproducibility. A compact neural network model, balancing expressive power with computational efficiency, was defined to process the partitioned training dataset across multiple client nodes.
The Privacy-Utility Tug-of-War
A core aspect of the research involved examining the inherent trade-offs between data privacy protections and learning system efficiency. Utility functions were developed to manage model parameters, enabling operations like combination, difference calculation, scaling, and aggregation across participating clients. The differential privacy mechanism, crucial for this trade-off, sanitized updates through L2 norm clipping and the introduction of Gaussian noise, with its magnitude determined by the chosen privacy budget (epsilon).
Each client independently executed a local training loop on its private data. Concurrently, a standardized evaluation routine measured test loss and accuracy for any given model state, effectively decoupling training from data ownership. This robust framework enabled a direct comparison between centralized and decentralized architectures under varying privacy constraints.
Comparing Centralized FedAvg and Decentralized Gossip
The centralized FedAvg algorithm served as the baseline, with a subset of clients performing local training and sending their differentially private updates to a central aggregator. This process tracked model performance over communication rounds, offering insights into convergence under different privacy budgets.
In contrast, the decentralized Gossip Federated Learning model operated on a peer-to-peer basis, exchanging information over a predefined network topology, such as a ring or an Erdos-Renyi graph. This approach simulated repeated local training and pairwise parameter averaging among clients without relying on a central server. The analysis focused on understanding how privacy noise propagates through these distributed communication patterns and its subsequent impact on model convergence.
Key Findings and Implications
Experiments were systematically run across a spectrum of privacy levels (varying epsilon values) for both centralized and decentralized training strategies. The results revealed distinct convergence trends and final accuracy levels, starkly illustrating the privacy-utility trade-off. While centralized FedAvg often demonstrated faster convergence under less stringent privacy constraints, gossip-based federated learning demonstrated greater resilience against noisy updates, albeit at the expense of slower convergence rates.
The study underscored that stronger privacy guarantees invariably slow down the learning process in both settings. However, this effect was notably amplified in decentralized topologies, primarily due to the delayed mixing of information. This suggests that designing effective privacy-preserving federated systems necessitates a holistic approach, where the aggregation topology, communication patterns, and privacy budgets are considered interdependently, rather than as isolated design choices.
This article is a rewritten summary based on publicly available reporting. For the original story, visit the source.
Source: MarkTechPost
