Context shift reduction for offline meta-reinforcement learning

Y Gao, R Zhang, J Guo, F Wu, Q Yi… - Advances in …, 2024 - proceedings.neurips.cc
Y Gao, R Zhang, J Guo, F Wu, Q Yi, S Peng, S Lan, R Chen, Z Du, X Hu, Q Guo, L Li, Y Chen
Advances in Neural Information Processing Systems, 2024proceedings.neurips.cc
Offline meta-reinforcement learning (OMRL) utilizes pre-collected offline datasets to
enhance the agent's generalization ability on unseen tasks. However, the context shift
problem arises due to the distribution discrepancy between the contexts used for training
(from the behavior policy) and testing (from the exploration policy). The context shift problem
leads to incorrect task inference and further deteriorates the generalization ability of the
meta-policy. Existing OMRL methods either overlook this problem or attempt to mitigate it …
Abstract
Offline meta-reinforcement learning (OMRL) utilizes pre-collected offline datasets to enhance the agent's generalization ability on unseen tasks. However, the context shift problem arises due to the distribution discrepancy between the contexts used for training (from the behavior policy) and testing (from the exploration policy). The context shift problem leads to incorrect task inference and further deteriorates the generalization ability of the meta-policy. Existing OMRL methods either overlook this problem or attempt to mitigate it with additional information. In this paper, we propose a novel approach called Context Shift Reduction for OMRL (CSRO) to address the context shift problem with only offline datasets. The key insight of CSRO is to minimize the influence of policy in context during both the meta-training and meta-test phases. During meta-training, we design a max-min mutual information representation learning mechanism to diminish the impact of the behavior policy on task representation. In the meta-test phase, we introduce the non-prior context collection strategy to reduce the effect of the exploration policy. Experimental results demonstrate that CSRO significantly reduces the context shift and improves the generalization ability, surpassing previous methods across various challenging domains.
proceedings.neurips.cc
以上显示的是最相近的搜索结果。 查看全部搜索结果

Google学术搜索按钮

example.edu/paper.pdf
搜索
获取 PDF 文件
引用
References