Learning to Reflect: Hierarchical Multi-Agent Reinforcement Learning for CSI-Free mmWave Beam-Focusing
2026-03-07 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address challenges in using reconfigurable intelligent surfaces (RIS) for wireless communication, which usually require costly channel information. They propose a new method using hierarchical multi-agent reinforcement learning (HMARL) that replaces detailed channel data with user location info to guide system control. Their approach splits decision-making into a high-level user allocation and low-level beam focusing, improving signal strength and scaling well with more users. Tests show the method works reliably even with localization errors and varying hardware sizes, making it a practical way to improve millimeter-wave wireless systems without heavy data overhead.
Reconfigurable Intelligent SurfacesChannel State InformationMillimeter-wave SystemsMulti-Agent Reinforcement LearningProximal Policy OptimizationCentralized Training Decentralized ExecutionUser LocalizationRay-TracingBeam-FocusingSignal-to-Noise Ratio
Authors
Hieu Le, Oguz Bedir, Mostafa Ibrahim, Jian Tao, Sabit Ekin
Abstract
Reconfigurable Intelligent Surfaces promise to transform wireless environments, yet practical deployment is hindered by the prohibitive overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization. This paper proposes a Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework for the control of mechanically reconfigurable reflective surfaces in millimeter-wave (mmWave) systems. We introduce a "CSI-free" paradigm that substitutes pilot-based channel estimation with readily available user localization data. To manage the massive combinatorial action space, the proposed architecture utilizes Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) paradigm. The proposed architecture decomposes the control problem into two abstraction levels: a high-level controller for user-to-reflector allocation and decentralized low-level controllers for low-level focal point optimization. Comprehensive ray-tracing evaluations demonstrate that the framework achieves 2.81-7.94 dB RSSI improvements over centralized baselines, with the performance advantage widening as system complexity increases. Scalability analysis reveals that the system maintains sustained efficiency, exhibiting minimal per-user performance degradation and stable total power utilization even when user density doubles. Furthermore, robustness validation confirms the framework's viability across varying reflector aperture sizes (45-99 tiles) and demonstrates graceful performance degradation under localization errors up to 0.5 m. By eliminating CSI overhead while maintaining high-fidelity beam-focusing, this work establishes HMARL as a practical solution for intelligent mmWave environments.