Search Results for author:
Found 7 papers, 6 papers with code
Learning Manipulation by Predicting Interaction
To this end, we propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction (MPI) and enhances the visual representation. Given a pair of keyframes representing the initial and final states, along with language instructions, our algorithm predicts the transition frame and detects the interaction object, respectively.
SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation
In this paper, we propose SAM-E, a novel architecture for robot manipulation by leveraging a vision-foundation model for generalizable scene understanding and sequence imitation for long-term action reasoning.
Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs
Generalizable articulated object manipulation is essential for home-assistant robots.
Robust Cross-Modal Knowledge Distillation for Unconstrained Videos
However, such semantic consistency from the synchronization is hard to guarantee in unconstrained videos, due to the irrelevant modality noise and differentiated semantic correlation.
Balanced Audiovisual Dataset for Imbalance Analysis
We surprisingly find that: the multimodal models with existing imbalance algorithms consistently perform worse than the unimodal one on specific subsets, in accordance with the modality bias.
Revisiting Pre-training in Audio-Visual Learning
Specifically, we explore the effects of pre-trained models on two audio-visual learning scenarios: cross-modal initialization and multi-modal joint learning.
TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World
Experimental results indicate that the models incorporating large language models (LLM) can generate more diverse responses, while the model utilizing knowledge graphs to introduce external knowledge performs the best overall.
