[2023-11-27]: Our paper Can Vision-Language Models Think from a First-Person Perspective? has been released.

Vision-language models (VLMs) have recently shown promising results in traditional downstream tasks. Evaluation studies have emerged to assess their abilities, with the majority focusing on the third-person perspective, and only a few addressing specific tasks from the first-person perspective. However, the capability of VLMs to "think" from a first-person perspective, a crucial attribute for advancing autonomous agents and robotics, remains largely unexplored. To bridge this research gap, we introduce EgoThink, a novel visual question-answering benchmark that encompasses six core capabilities with twelve detailed dimensions. The benchmark is constructed using selected clips from egocentric videos, with manually annotated question-answer pairs containing first-person information. To comprehensively assess VLMs, we evaluate eighteen popular VLMs on EgoThink. Moreover, given the open-ended format of the answers, we use GPT-4 as the automatic judge to compute single-answer grading. Experimental results indicate that although GPT-4V leads in numerous dimensions, all evaluated VLMs still possess considerable potential for improvement in first-person perspective tasks. Meanwhile, enlarging the number of trainable parameters has the most significant impact on model performance on EgoThink. In conclusion, EgoThink serves as a valuable addition to existing evaluation benchmarks for VLMs, providing an indispensable resource for future research in the realm of embodied artificial intelligence and robotics.

We specifically design six categories with twelve fine-grained dimensions from the first-person perspective for quantitative evaluation.

• Object: What is around me? Recognizing objects in the real world is a preliminary ability of the human visual system. Images from a first-person or egocentric perspective pay more attention to the objects surrounding the subject or in hands. Moreover, we further divide the object category into three fine-grained dimensions: (1) Existence, predicting whether there is an object as described in the images; (2) Attribute, detecting properties or characteristics (e.g., color) of an object; (3) Affordance, predicting potential actions that a human can apply to an object.
• Activity: What am I doing? Activity recognition is to automatically recognize specific human activities in video frames or still images. From the egocentric perspective, we mainly focus on actions or activities based on object-hand interaction.
• Localization: Where am I? In reality, localization is a critical capability for navigation and scene understanding in the real world. Here we investigate the localization capability from two aspects, Location and Spatial Relationship. Location indicates detecting the scene surrounding the subject. Spatial reasoning contains allocentric and egocentric perspectives. We focus on the egocentric perspective, i.e., the position of the object with respect to the subject.
• Reasoning: What about the situation around me? During the complex decision-making process, reasoning lies everywhere in our lives. Here we mainly focus on Counting, Comparison, and Situated Reasoning. Due to the first-person perspective, we generally count or compare objects in our hands or surrounding ourselves. As for situated reasoning, we employ cases that cannot be answered directly from the information in the images and require further reasoning processes.
• Forecasting: What will happen to me? Forecasting is a critical skill in the real world. From an egocentric view, forecasting always predicts the future of object-state transformation or hand-object interactions.
• Planning: How will I do? In reality, planning is an important capability to deal with complex problems, typically applied in Navigation and Assistance. Navigation is going to a goal location from a start position, while assistance is offering instructions to solve daily problems.

The ability to think from a first-person perspective is not adequately addressed by current evaluation benchmarks for VLMs. On one hand, most of these benchmarks (six out of nine, as listed in Table 1) focus solely on the third-person perspective. On the other hand, those benchmarks that do consider the first-person perspective only encompass a limited range of capabilities. For instance, EgoTaskQA examines spatial, temporal, and causal aspects, whereas EgoVQA is limited to object, action, and person aspects. Therefore, there is a clear need to develop a comprehensive benchmark to evaluate the first-person capabilities of VLMs more effectively.

Table 1: Comparison of recent comprehensive evaluation benchmarks of VLMs and our proposed benchmark EgoThink. Third and first indicate third-person and first-person perspectives. Datasets/Handcraft/LLMs denote existing datasets, manual annotation, and automatic generation by LLMs. PS/MC/OE indicate pairwise scoring, multi-choice, and open-ended question-answering, respectively.