Search Results for author:
Found 62 papers, 41 papers with code
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation.
No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation
To reduce the reliance on large-scale datasets, recent works in 3D segmentation resort to few-shot learning.
CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching
We further attribute this phenomenon to the diffusion model's insufficient condition utilization, which is caused by its training paradigm.
Parsing All Adverse Scenes: Severity-Aware Semantic Segmentation with Mask-Enhanced Cross-Domain Consistency
The SPM module incorporates a Severity Perception mechanism, guiding a Mask operation that enables our model to learn highly consistent features from the augmented scenes.
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs.
OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning
To evaluate the effectiveness of our general framework OneTracker, which is consisted of Foundation Tracker and Prompt Tracker, we conduct extensive experiments on 6 popular tracking tasks across 11 benchmarks and our OneTracker outperforms other models and achieves state-of-the-art performance.
Ranked #15 on
Rgb-T Tracking
on LasHeR
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX.
Ranked #5 on
Video Question Answering
on MVBench
ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation
By fine-tuning the injected adapters, we preserve the inherent common sense and reasoning ability of the MLLMs while equipping them with the ability for manipulation.
A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
They endow Large Language Models (LLMs) with powerful capabilities in visual understanding, enabling them to tackle diverse multi-modal tasks.
Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation
To tackle these issues, we propose a continual self-supervised method, Adaptive Distribution Masked Autoencoders (ADMA), which enhances the extraction of target domain knowledge while mitigating the accumulation of distribution shifts.
Language-Assisted 3D Scene Understanding
The scale and quality of point cloud datasets constrain the advancement of point cloud learning.
Gradient-based Parameter Selection for Efficient Fine-Tuning
With the growing size of pre-trained models, full fine-tuning and storing all the parameters for various downstream tasks is costly and infeasible.
3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V
In this work, we present a new visual prompting method called 3DAxiesPrompts (3DAP) to unleash the capabilities of GPT-4V in performing 3D spatial tasks.
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings.
Ranked #2 on
Visual Question Answering
on BenchLMM
Improving Compositional Text-to-image Generation with Large Vision-Language Models
However, compositional text-to-image models frequently encounter difficulties in generating high-quality images that accurately align with input texts describing multiple objects, variable attributes, and intricate spatial relationships.
MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning
In this paper, we present a method to fine-tune open-source language models, enabling them to use code for modeling and deriving math equations and, consequently, enhancing their mathematical reasoning abilities.
Ranked #4 on
Math Word Problem Solving
on SVAMP
(using extra training data)
NTO3D: Neural Target Object 3D Reconstruction with Segment Anything
NTO3D lifts the 2D masks and features of SAM into the 3D neural field for high-quality neural target object 3D reconstruction.
PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation
Our dataset poses new challenges in panoramic VOS and we hope that our PanoVOS can advance the development of panoramic segmentation/tracking.
RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision
3D occupancy prediction holds significant promise in the fields of robot perception and autonomous driving, which quantifies 3D scenes into grid cells with semantic labels.
ImageBind-LLM: Multi-modality Instruction Tuning
During training, we adopt a learnable bind network to align the embedding space between LLaMA and ImageBind's image encoder.
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following
We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video.
Ranked #5 on
3D Question Answering (3D-QA)
on 3D MM-Vet
Less is More: Towards Efficient Few-shot 3D Semantic Segmentation via Training-free Networks
However, the prior pre-training stage not only introduces excessive time overhead, but also incurs a significant domain gap on `unseen' classes.
3D Semantic Segmentation
Few-shot 3D semantic segmentation
+1
Dynamic Embedding Size Search with Minimum Regret for Streaming Recommender System
The high-throughput data requires the model to be updated in a timely manner for capturing the user interest dynamics, which leads to the emergence of streaming recommender systems.
Revisiting Event-based Video Frame Interpolation
We therefore propose to incorporate RGB information in an event-guided optical flow refinement strategy.
Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models
Video Question Answering (VideoQA) has been significantly advanced from the scaling of recent Large Language Models (LLMs).
Ranked #3 on
Temporal/Casual QA
on NExT-QA
(using extra training data)
ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
Note that, our method can be regarded as a novel transfer paradigm for large-scale models, delivering promising results in adaptation to continually changing distributions.
Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation
In this paper, we propose MUTR, a Multi-modal Unified Temporal transformer for Referring video object segmentation.
Personalize Segment Anything Model with One Shot
Driven by large-data pre-training, Segment Anything Model (SAM) has been demonstrated as a powerful and promptable framework, revolutionizing the segmentation models.
Ranked #1 on
Personalized Segmentation
on PerSeg
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset.
Ranked #6 on
Visual Question Answering (VQA)
on InfiMM-Eval
Instruction Following
Optical Character Recognition (OCR)
+7
Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement
The popularity of Contrastive Language-Image Pre-training (CLIP) has propelled its application to diverse downstream vision tasks.
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model.
Ranked #2 on
Music Question Answering
on MusicQA
PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection
Masked Autoencoders learn strong visual representations and achieve state-of-the-art results in several independent modalities, yet very few works have addressed their capabilities in multi-modality settings.
Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysis
We present a Non-parametric Network for 3D point cloud analysis, Point-NN, which consists of purely non-learnable components: farthest point sampling (FPS), k-nearest neighbors (k-NN), and pooling operations, with trigonometric functions.
Ranked #1 on
Training-free 3D Part Segmentation
on ShapeNet-Part
3D Point Cloud Classification
Training-free 3D Part Segmentation
+1
Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking
To alleviate this, previous methods simply replace the pixel reconstruction targets of 75% masked tokens by encoded features from pre-trained image-image (DINO) or image-language (CLIP) contrastive learning.
Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
Our CaFo incorporates CLIP's language-contrastive knowledge, DINO's vision-contrastive knowledge, DALL-E's vision-generative knowledge, and GPT-3's language-generative knowledge.
Nearest Neighbors Meet Deep Neural Networks for Point Cloud Analysis
Performances on standard 3D point cloud benchmarks have plateaued, resulting in oversized models and complex network design to make a fractional improvement.
Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training
In this paper, we explore how the 2D modality can benefit 3D masked autoencoding, and propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training.
Starting From Non-Parametric Networks for 3D Point Cloud Analysis
We present a Non-parametric Network for 3D point cloud analysis, Point-NN, which consists of purely non-learnable components: farthest point sampling (FPS), k-nearest neighbors (k-NN), and pooling operations, with trigonometric functions.
SparseMAE: Sparse Training Meets Masked Autoencoders
In this paper, we aim to reduce model complexity from large vision transformers pretrained by MAE with assistant of sparse training.
TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning
In this paper, we propose the learning scheme of Target Inner-Geometry from the LiDAR modality into camera-based BEV detectors for both dense depth and BEV features, termed as TiG-BEV.
Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders
Pre-training by numerous image data has become de-facto for robust 2D representations.
Ranked #2 on
3D Point Cloud Linear Classification
on ModelNet40
(using extra training data)
Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation
Since pseudo labels are noisy and unreliable, these methods suffer from catastrophic forgetting and error accumulation when dealing with dynamic data distributions.
iQuery: Instruments as Queries for Audio-Visual Sound Separation
To generalize to a new instrument or event class, drawing inspiration from the text-prompt design, we insert an additional query as an audio prompt while freezing the attention mechanism.
PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
In this paper, we first collaborate CLIP and GPT to be a unified 3D open-world learner, named as PointCLIP V2, which fully unleashes their potential for zero-shot 3D classification, segmentation, and detection.
Ranked #2 on
3D Open-Vocabulary Instance Segmentation
on STPLS3D
CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention
Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with great transferability, which achieves promising accuracy for zero-shot classification.
Ranked #4 on
Training-free 3D Point Cloud Classification
on ScanObjectNN
(using extra training data)
Training-free 3D Point Cloud Classification
Transfer Learning
+1
Collaboration of Pre-trained Models Makes Better Few-shot Learner
In this paper, we propose CoMo, a Collaboration of pre-trained Models that incorporates diverse prior knowledge from various pre-training paradigms for better few-shot learning.
Frozen CLIP Models are Efficient Video Learners
Video recognition has been dominated by the end-to-end learning paradigm -- first initializing a video recognition model with weights of a pretrained image model and then conducting end-to-end training on videos.
Ranked #26 on
Action Classification
on Kinetics-400
(using extra training data)
Exploring Resolution and Degradation Clues as Self-supervised Signal for Low Quality Object Detection
Image restoration algorithms such as super resolution (SR) are indispensable pre-processing modules for object detection in low quality images.
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
On top of that, the performance of Tip-Adapter can be further boosted to be state-of-the-art on ImageNet by fine-tuning the cache model for 10$\times$ fewer epochs than existing methods, which is both effective and efficient.
Can Language Understand Depth?
To our best knowledge, we are the first to conduct zero-shot adaptation from the semantic language knowledge to quantified downstream tasks and perform zero-shot monocular depth estimation.
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
By fine-tuning on downstream tasks, Point-M2AE achieves 86. 43% accuracy on ScanObjectNN, +3. 36% to the second-best, and largely benefits the few-shot classification, part segmentation and 3D object detection with the hierarchical pre-training scheme.
Ranked #4 on
3D Point Cloud Linear Classification
on ModelNet40
(using extra training data)
POS-BERT: Point Cloud One-Stage BERT Pre-Training
We propose to use the dynamically updated momentum encoder as the tokenizer, which is updated and outputs the dynamic supervision signal along with the training process.
MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection
In this paper, we introduce the first DETR framework for Monocular DEtection with a depth-guided TRansformer, named MonoDETR.
3D Object Detection From Monocular Images
Autonomous Driving
+4
Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning
Especially, we develop a variant of ViT for 3D point cloud feature extraction, which also achieves comparable results with existing backbones when combined with our framework, and visualization of the attention maps show that our model does understand the point cloud by combining the global shape information and multiple local structural information, which is consistent with the inspiration of our representation learning method.
VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts
Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning.
PointCLIP: Point Cloud Understanding by CLIP
On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D.
Ranked #3 on
3D Open-Vocabulary Instance Segmentation
on STPLS3D
3D Open-Vocabulary Instance Segmentation
Few-Shot Learning
+6
DSPoint: Dual-scale Point Cloud Recognition with High-frequency Fusion
We reverse the conventional design of applying convolution on voxels and attention to points.
Ranked #36 on
3D Part Segmentation
on ShapeNet-Part
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification.
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
Large-scale contrastive vision-language pre-training has shown significant progress in visual representation learning.
Dual-stream Network for Visual Recognition
Transformers with remarkable global representation capacities achieve competitive results for visual tasks, but fail to consider high-level local pattern information in input images.
End-to-End Object Detection with Adaptive Clustering Transformer
In this paper, a novel variant of transformer named Adaptive Clustering Transformer(ACT) has been proposed to reduce the computation cost for high-resolution input.
