Query Selection with Proper Latent-Representation Similarity in Preference-Based Reinforcement Learning
Abstract
Preference-based reinforcement learning (PbRL) aims to distill a reward function from binary human preference feedback on pairs of query trajectories, which often incurs significant costs. Therefore, the efficiency and effectiveness of PbRL largely depend on the quality of query trajectories. However, existing query sampling methods have not adequately considered the impact of trajectory pair feature similarity. We observe that both excessively high and low similarity between trajectory pairs can negatively affect the overall performance of PbRL. To address this, we propose a novel two-stage query selection strategy that exploits structural differences in the latent representations of query trajectories. Specifically, in the first stage, we design a reward-guided adjacent cluster sampling module to obtain a set of candidate query samples, in which fundamental feature differences and high entropy between trajectory pairs are realized. In the second stage, an entropy-based similarity filtering mechanism is applied on the candidate query samples to remove excessively similar trajectory pairs, thereby maximizing the information value and forming the final query set. To validate the performance, we implement our sampling strategy based on the QPA algorithm. Empirical validation on DMControl and Metaworld benchmarks demonstrates that our method achieves higher query efficiency and 15.92% improvement in policy performance over baseline methods. Our implementation is available at https://github.com/leowang018/QSPLS .