Prioritized Semantic Learning for Zero-shot Instance Navigation
xinyu sun*, Lizhao Liu, Hongyan Zhi, Ronghe Qiu, Junwei Liang*
;
Abstract
"We study zero-shot instance navigation, in which the agent navigates to a specific object without using object annotations for training. Previous object navigation approaches apply the image-goal navigation () task (go to the location of an image) for pretraining, and transfer the agent to achieve object goals using a vision-language model. However, these approaches lead to issues of semantic neglect, where the model fails to learn meaningful semantic alignments. In this paper, we propose a () method to improve the semantic understanding ability of navigation agents. Specifically, a semantic-enhanced PSL agent is proposed and a strategy is introduced to select goal images that exhibit clear semantic supervision and relax the reward function from strict exact view matching. At inference time, a scheme is designed to preserve the same granularity level of the goal-semantic as training. Furthermore, for the popular HM3D environment, we present an Instance Navigation () task that requires going to a specific object instance with detailed descriptions, as opposed to the Object Navigation () task where the goal is defined merely by the object category. Our PSL agent outperforms the previous state-of-the-art by 66% on zero-shot in terms of success rate and is also superior on the new task. Code will be released at https://github.com/XinyuSun/PSL-InstanceNav."
Related Material
[pdf]
[supplementary material]
[DOI]