Nips2024 多模态论文热点分析

type

date

slug

summary

·Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding

We present a novel framework for OCR-free document understanding based on pretrained Multimodal Large Language Models (MLLMs). Our approach employs multiscale visual features to effectively handle various font sizes within document images. To address the increasing costs of considering the multi-scale visual inputs for the pretrained MLLMs, we propose a hierarchical visual feature aggregation module designed to reduce the number of input tokens to LLMs. Our approach leverages feature pyramid hierarchy with cross-attentive pooling, effectively handling the trade-off between information loss and efficiency without being affected by varying document image sizes.Additionally, we introduce a novel instruction tuning task that aims to enhance model readability by incorporating text positional information within images, which is robust to text truncation issue. Through comprehensive experiments, we demonstrate the efficacy of our framework in achieving outstanding document understanding performance on various tasks.

文档理解、OCR-free、多尺度特征、层次聚合

用于无 OCR 文档理解的分层视觉特征聚合

提出了一种基于多模式大语言模型(MLLMS)的无 OCR 文档理解框架。该方法利用多尺度视觉特征来有效地处理文档图像中的各种字体大小，针对预先训练的 MLLMS 考虑多尺度视觉输入的代价不断增加的问题，提出了一种层次化视觉特征聚合模块，旨在减少 LLMS 的输入标记数。该方法利用特征金字塔层次结构和交叉注意池，在不受文档图像大小影响的情况下，有效地处理了信息损失和效率之间的权衡；此外，我们还引入了一种新的指令调优任务，旨在通过在图像中加入文本位置信息来增强模型的可读性，该任务对文本截断问题具有很强的鲁棒性。通过全面的实验，我们证明了我们的框架在各种任务上取得了出色的文档理解性能。

（文档理解，高效感知）

M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation

This paper presents $M^3$GPT, an advanced \textbf{M}ultimodal, \textbf{M}ultitask framework for \textbf{M}otion comprehension and generation. $M^3$GPT operates on three fundamental principles. The first focuses on creating a unified representation space for various motion-relevant modalities. We employ discrete vector quantization for multimodal control and generation signals, such as text, music and motion/dance, enabling seamless integration into a large language model (LLM) with a single vocabulary.The second involves modeling model generation directly in the raw motion space. This strategy circumvents the information loss associated with discrete tokenizer, resulting in more detailed and comprehensive model generation. Third, $M^3$GPT learns to model the connections and synergies among various motion-relevant tasks. Text, the most familiar and well-understood modality for LLMs, is utilized as a bridge to establish connections between different motion tasks, facilitating mutual reinforcement. To our knowledge, $M^3$GPT is the first model capable of comprehending and generating motions based on multiple signals. Extensive experiments highlight $M^3$GPT's superior performance across various motion-relevant tasks and its powerful zero-shot generalization capabilities for extremely challenging tasks.

动作生成、多模态统一、多任务学习、零样本泛化

$M#3$GPT：用于运动理解和生成的高级多模式、多任务框架

本文提出了一个高级的多通道、多任务框架--$M^3$GPT，用于理解和生成文本操作。$M^3$GPT 遵循三个基本原则。第一个重点是为各种运动相关的模态创建一个统一的表示空间。我们将离散矢量量化用于文本、音乐和动作/舞蹈等多模式控制和生成信号，使其能够无缝集成到具有单一词汇的大型语言模型(LLM)中。第二，直接在原始运动空间中建模模型生成。该策略避免了与离散标记器相关的信息损失，从而产生更详细和更全面的模型生成。第三，$M^3$GPT 学习对各种与运动相关的任务之间的联系和协同作用进行建模。语篇是 LLMS 最熟悉和最被理解的情态形式，它被用作在不同的动作任务之间建立联系的桥梁，促进了相互加强。据我们所知，$M^3$GPT 是第一个能够理解和生成基于多个信号的运动的模型。广泛的实验突出了$M^3$GPT 在各种与运动相关的任务中的卓越性能，以及其针对极具挑战性的任务的强大的零射泛化能力。

（运动理解和生成）

·Training-Free Visual Prompt Learning for Multimodal Large Language Models

In this work, we propose a training-free method to inject visual referring into Multimodal Large Language Models (MLLMs) through learnable visual token optimization. We observe the relationship between text prompt tokens and visual tokens in MLLMs, where attention layers model the connection between them. Our approach involves adjusting visual tokens from the MLP output during inference, controlling which text prompt tokens attend to which visual tokens. We optimize a learnable visual token based on an energy function, enhancing the strength of referential regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referential abilities into MLLMs. Our method support referring with box, mask, scribble and point. The results demonstrate that our method exhibits controllability and interpretability.

视觉指代、无训练优化、多形式标注

多模式大型语言模型的免培训视觉提示学习

在这项工作中，我们提出了一种免训练的方法，通过可学习的视觉标记优化将视觉引用注入多模式大型语言模型（MLLM）。我们观察 MLLM 中文本提示标记和视觉标记之间的关系，其中注意力层对它们之间的连接进行建模。我们的方法涉及在推理期间调整 MLP 输出的视觉标记，控制哪些文本提示标记关注哪些视觉标记。我们基于能量函数优化可学习的视觉标记，增强注意力地图中参考区域的强度。这使得可以进行详细的区域描述和推理，而不需要大量的培训成本或模型再培训。我们的方法为将参考能力集成到 MLLM 中提供了一个有希望的方向。我们的方法支持引用框、面具、涂鸦和点。结果表明我们的方法具有可控性和可解释性。

（提示学习、后训练）

Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight

This paper introduces Chain-of-Sight, a vision-language bridge module that accelerates the pre-training of Multimodal Large Language Models (MLLMs). Our approach employs a sequence of visual resamplers that capture visual details at various spacial scales.This architecture not only leverages global and local visual contexts effectively, but also facilitates the flexible extension of visual tokens through a compound token scaling strategy, allowing up to a 16x increase in the token count post pretraining.Consequently, Chain-of-Sight requires significantly fewer visual tokens in the pre-training phase compared to the fine-tuning phase. This intentional reduction of visual tokens during pre-training notably accelerates the pre-training process, cutting down the wall-clock training time by $\sim$73\%.Empirical results on a series of vision-language benchmarks reveal that the pre-train acceleration through Chain-of-Sight is achieved without sacrificing performance, matching or surpassing the standard pipeline of utilizing all visual tokens throughout the entire training process. Further scaling up the number of visual tokens for pre-training leads to stronger performances, competitive to existing approaches in a series of benchmarks. Codes and models will be released.

预训练加速、多尺度视觉、令牌扩展

通过视线链加速多模式 LLM 的预训练

本文介绍了一种视觉-语言桥梁模块--视链，它加速了多通道大语言模型(MLLMS) 的预训练。我们的方法使用一系列视觉重采样器来捕捉不同空间尺度上的视觉细节。该架构不仅有效地利用了全局和局部视觉上下文，而且通过复合令牌缩放策略促进了视觉令牌的灵活扩展，使得训练前的令牌计数增加了 16 倍。因此，与微调阶段相比，视链在预训练阶段需要的视觉令牌显著减少。这种在训练前有意减少视觉标记的方法显著加快了训练前的过程，将挂钟训练时间缩短了 73 美元。一系列视觉语言基准的经验结果表明，通过视链进行训练前的加速是在不牺牲性能的情况下实现的，在整个训练过程中匹配或超过了使用所有视觉标记的标准流水线。在一系列基准中，进一步扩大训练前视觉标记的数量将导致更强的表现，与现有方法具有竞争力。代码和型号将会公布。

（预训练）

·Visual Perception by Large Language Model’s Weights

Existing Multimodal Large Language Models (MLLMs) follow the paradigm that perceives visual information by aligning visual features with the input space of Large Language Models (LLMs), and concatenating visual tokens with text tokens to form a unified sequence input for LLMs. These methods demonstrate promising results on various vision-language tasks but are limited by the high computational effort due to the extended input sequence resulting from the involvement of visual tokens. In this paper, instead of input space alignment, we propose a novel parameter space alignment paradigm that represents visual information as model weights. For each input image, we use a vision encoder to extract visual features, convert features into perceptual weights, and merge the perceptual weights with LLM's weights. In this way, the input of LLM does not require visual tokens, which reduces the length of the input sequence and greatly improves efficiency. Following this paradigm, we propose VLoRA with the perceptual weights generator. The perceptual weights generator is designed to convert visual features to low-rank perceptual weights since the visual information is redundant. Due to the low-rank property, our generated perceptual weights exhibit a form similar to LoRA. The experimental results show that our VLoRA achieves comparable performance on various benchmarks for MLLMs, while significantly reducing the computational costs for both training and inference.

视觉感知、参数空间对齐、低秩权重、计算优化

大型语言模型权重的视觉感知

现有的多通道大语言模型遵循的是通过将视觉特征与大语言模型的输入空间对齐，并将视觉标记与文本标记连接以形成大语言模型的统一序列输入来感知视觉信息的范例。这些方法在不同的视觉语言任务上显示出良好的结果，但由于视觉标记的参与导致输入序列的扩展而限制了较高的计算工作量。在本文中，我们提出了一种新的参数空间对齐范式，将视觉信息表示为模型权重，而不是输入空间对齐。对于每一幅输入图像，我们使用视觉编码器来提取视觉特征，将特征转换为感知权重，并将感知权重与 LLM 的权重进行合并。这样，LLM 的输入不需要视觉标记，减少了输入序列的长度，大大提高了效率。遵循这一范式，我们提出了具有知觉权重生成器的 Vlora。由于视觉信息是冗余的，感知权重生成器被设计为将视觉特征转换为低等级感知权重。由于低阶性质，我们生成的知觉权重呈现出类似于 LORA 的形式。实验结果表明，我们的 Vlora 在 MLLMS 的各种基准上取得了相当的性能，同时显著降低了训练和推理的计算代价。

（高效视觉感知）

·Dense Connector for MLLMs

Do we fully leverage the potential of visual encoder in Multimodal Large Language Models (MLLMs)? The recent outstanding performance of MLLMs in multimodal understanding has garnered broad attention from both academia and industry. In the current MLLM rat race, the focus seems to be predominantly on the linguistic side. We witness the rise of larger and higher-quality instruction datasets, as well as the involvement of larger-sized LLMs. Yet, scant attention has been directed towards the visual signals utilized by MLLMs, often assumed to be the final high-level features extracted by a frozen visual encoder. In this paper, we introduce the Dense Connector - a simple, effective, and plug-and-play vision-language connector that significantly enhances existing MLLMs by leveraging multi-layer visual features, with minimal additional computational overhead. Furthermore, our model, trained solely on images, showcases remarkable zero-shot capabilities in video understanding as well. Experimental results across various vision encoders, image resolutions, training dataset scales, varying sizes of LLMs (2.7B→70B), and diverse architectures of MLLMs (e.g., LLaVA and Mini-Gemini) validate the versatility and scalability of our approach, achieving state-of-the-art performance on across 19 image and video benchmarks. We hope that this work will provide valuable experience and serve as a basic module for future MLLM development. Code is available at https://anonymous.4open.science/r/DCNIPS.

视觉接口、多层视觉特征、即插即用、跨模态通用性

MLLM 的密集连接器

我们是否充分利用了多模式大型语言模型(MLLMS)中的可视编码器的潜力？近年来，最大似然模型在多模式理解方面的出色表现引起了学术界和产业界的广泛关注。在目前的 MLLM 激烈竞争中，焦点似乎主要集中在语言方面。我们见证了更大、更高质量的指令数据集的崛起，以及更大规模的 LLM 的参与。然而，很少有人注意到 MLLMS 所利用的视觉信号，这些信号通常被认为是由冻结的视觉编码器提取的最终高级特征。在本文中，我们介绍了密集连接器-一种简单、有效和即插即用的视觉语言连接器，它通过利用多层视觉功能显著增强了现有的 MLLMS，而额外的计算开销最小。此外，我们的模型仅针对图像进行培训，在视频理解方面也展示了非凡的零镜头能力。在不同视觉编码器、图像分辨率、训练数据集比例、不同大小的 LLMS(2.7B→70B)和不同架构的 MLLMS(例如 LLaVA 和 Mini-Gemini)上的实验结果验证了我们方法的通用性和可扩展性，在 19 个图像和视频基准上获得了最先进的性能。我们希望这项工作将提供宝贵的经验，并作为未来 MLLM 发展的基本模块。代码可在 https://anonymous.4open.science/r/DC-NIPS.上找到

（高效视觉感知）

· Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models

Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn. Different from prior work, which uses text-to-image models to enable LMs to draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning. \name can also use specialist vision models during the sketching process (e.g., draw bounding boxes with object detection models, draw masks with segmentation models), to further enhance visual perception and reasoning. We experiment on a wide range of math tasks (including geometry, functions, graph, chess) and complex visual reasoning tasks. Sketchpad substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%). We will release all code and data.

视觉推理、草图辅助、工具增强

视觉画板：草图作为多模式语言模型的视觉思维链

人类画画是为了促进推理：我们在解决几何问题时画辅助线；在地图上进行推理时我们画标记和圈；我们用草图来放大我们的想法，释放我们有限的工作记忆。然而，在当前的多通道语言模型(LMS)中，这样的动作是缺失的。目前的思想链和工具使用范式只使用文本作为中间推理步骤。在这项工作中，我们介绍了 SketchPad，这是一个为多通道 LMS 提供可视 SketchPad 的框架，以及在 SketchPad 上绘制的工具。 LM 根据它绘制的视觉文物进行规划和推理。与以往使用文本到图像模型来使 LMS 能够绘制不同的是，SketchPad 使 LMS 能够用线、框、标记等绘制，更接近于人类的草图，更好地促进推理。\NAME 还可以在素描过程中使用专业视觉模型 (例如，使用对象检测模型绘制边界框，使用分割模型绘制蒙版)，以进一步增强视觉感知和推理。我们在广泛的数学任务(包括几何、函数、图形、国际象棋)和复杂的视觉推理任务上进行实验。SketchPad 在没有绘制草图的强大基础模型上显著提高了所有任务的性能，在数学任务上的平均收益为 12.7%，在视觉任务上的平均收益为 8.6%。带有 SketchPad 的 GPT-40 在所有任务上设置了新的艺术状态，包括 V*BASE(80.3%)、眨眼空间推理(83.9%)和视觉对应(80.8%)。我们将公布所有代码和数据。

（视觉思维链）

·Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs

Large Language Models (LLMs) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. They are the de facto building block for Large Multimodal Models (LMMs), yet, we still lack a proper understanding of their success. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation with the attempt to understand their generalization beyond textual inputs. Our work provides the following findings. Perceptual tokens (1) are easily distinguishable from textual ones inside LLMs, with significantly different representations (e.g. live in different narrow cones), and complete translation to textual tokens does not exists. Yet, (2) both perceptual and textual tokens activate similar LLM weights. Despite their differences, (3) perceptual tokens are implicitly aligned to textual tokens inside LLMs, we call this the implicit multimodal alignment effect (IMA), and argue that this is linked to architectural design, helping LLMs to generalize. This provide more evidence to believe that the generalization of LLMs to multimodal inputs is mainly due to their architecture. These findings lead to several implications. This work provides several implications. (1) We find a positive correlation between the implicit alignment score and the task performance, suggesting that this could act as a proxy metric for model evaluation and selection. (2) A negative correlation exists regarding hallucinations (e.g. describing non-existing objects in images), revealing that this problem is mainly due to misalignment between the internal perceptual and textual representations. (3) Perceptual tokens change slightly throughout the model, thus, we propose different approaches to skip computations (e.g. in FFN layers), and significantly reduce the inference cost. (4) Due to the slowly changing embeddings across layers, and the high overlap between textual and multimodal activated weights, we compress LLMs by keeping only 1 subnetwork (called alpha-SubNet) that works well across a wide range of multimodal tasks. The code will be made public.

多模态对齐, 表征分析, 模型压缩

隐式多模式对齐：冻结 LLM 到多模式输入的推广

大型语言模型(LLM)在多模式任务上表现出了令人印象深刻的性能，没有任何多模式的微调。它们是大型多通道模型(LMM)事实上的基石，然而，我们仍然缺乏对它们成功的正确理解。在这项工作中，我们将冻结的 LLM 暴露在图像、视频、音频和文本输入中，并分析它们的内部表示，试图理解它们在文本输入之外的泛化。我们的工作提供了以下发现。感知标记(1)在 LLMS 中很容易与文本标记区分开来，具有显著的不同表征(例如，生活在不同的窄锥体中)，并且不存在到文本标记的完整翻译。然而，(2)知觉标记和文本标记都激活了相似的 LLM 权重。尽管不同，(3) 感知标记与文本标记在 LLMS 中是隐式对齐的，我们称之为隐式多通道对齐效应 (IMA)，并认为这与建筑设计有关，有助于 LLMS 的泛化。这提供了更多的证据，证明最小二乘模型对多模式输入的泛化主要是由于它们的体系结构。这些发现带来了几点启示。这项工作提供了几点启示。(1)内隐对齐分数与任务绩效之间存在显著的正相关，可以作为模型评价和选择的替代指标。(2)幻觉(如在图像中描述不存在的物体)存在负相关，表明这一问题主要是由于内部知觉表征和文本表征之间的错位造成的。(3)感知标记在整个模型中略有变化，因此，我们提出了不同的方法来跳过计算(例如在 FFN 层)，并显著降低了推理代价。(4)由于各层之间的嵌入变化缓慢，以及文本和多通道激活权重之间的高度重叠，我们只保留 1 个子网络(称为阿尔法子网)来压缩 LLMS，该子网络在广泛的多通道任务中工作得很好。代码将被公之于众。

（探究多模态对齐）

Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy

Diplomacy is one of the most sophisticated activities in human society. The complex interactions among multiple parties/ agents involve various abilities like social reasoning, negotiation arts, and long-term strategy planning. Previous AI agents surely have proved their capability of handling multi-step games and larger action spaces on tasks involving multiple agents. However, diplomacy involves a staggering magnitude of decision spaces, especially considering the negotiation stage required. Recently, LLM agents have shown their potential for extending the boundary of previous agents on a couple of applications, however, it is still not enough to handle a very long planning period in a complex multiagent environment. Empowered with cutting-edge LLM technology, we make the first stab to explore AI's upper bound towards a human-like agent for such a highly comprehensive multi-agent mission by combining three core and essential capabilities for stronger LLM-based societal agents: 1) strategic planner with memory and reflection; 2) goal-oriented negotiate with social reasoning; 3) augmenting memory by self-play games to self-evolving without any human in the loop.

多智能体博弈, 自我进化, 社会推理

Richelieu：自我发展的基于法学硕士的人工智能外交代理

外交是人类社会最复杂的活动之一。多方/代理人之间的复杂互动涉及各种能力，如社会推理、谈判艺术和长期战略规划。以前的人工智能代理肯定已经证明了他们在涉及多个代理的任务中处理多步骤游戏和更大的动作空间的能力。然而，外交涉及数量惊人的决策空间，特别是考虑到所需的谈判阶段。最近，LLM 代理在一些应用上显示出了扩展以前代理边界的潜力，但在复杂的多代理环境中，它仍然不足以处理非常长的规划周期。在尖端 LLM 技术的授权下，我们通过结合三个核心和基本能力，首次探索了人工智能在这样一个高度全面的多智能体任务中向类人智能体的上限：1)具有记忆和反思的战略规划者；2)具有社会推理的目标导向型谈判； 3)通过自我游戏增强记忆，在没有任何人类参与的情况下自我进化。

agents

Slot-VLM: Object-Event Slots for Video-Language Modeling

Video-Language Models (VLMs), powered by the advancements in Large Language Models (LLMs), are charting new frontiers in video understanding. A pivotal challenge is the development of an effective method to encapsulate video content into a set of representative tokens to align with LLMs. In this work, we introduce Slot-VLM, a new framework designed to generate semantically decomposed video tokens, in terms of object-wise and event-wise visual representations, to facilitate LLM inference. Particularly, we design an Object-Event Slots module, \ieno, OE-Slots, that adaptively aggregates the dense video tokens from the vision encoder to a set of representative slots. In order to take into account both the spatial object details and the varied temporal dynamics, we build OE-Slots with two branches: the Object-Slots branch and the EventSlots branch. The Object-Slots branch focuses on extracting object-centric slots from features of high spatial resolution but low frame sample rate, emphasizing detailed object information. The Event-Slots branch is engineered to learn event-centric slots from high temporal sample rate but low spatial resolution features. These complementary slots are combined to form the vision context, serving as the input to the LLM for effective video reasoning. Our experimental results demonstrate the effectiveness of our Slot-VLM, which achieves the state-of-the-art performance on video question-answering.

视频表征, 时空分解, 槽位注意力

Slot-VLM：用于视频语言建模的对象事件插槽

在大型语言模型(LLM)的推动下，视频语言模型(VLM)正在开辟视频理解的新领域。一个关键的挑战是开发一种有效的方法来将视频内容封装到一组具有代表性的令牌中，以与 LLM 保持一致。在这项工作中，我们引入了一个新的框架 Slot-VLM，它被设计用来生成语义分解的视频令牌，在对象和事件的视觉表示方面，以便于 LLM 推理。具体地说，我们设计了一个对象事件时隙模块，它自适应地将来自视觉编码器的密集视频令牌聚集到一组具有代表性的时隙中。为了同时考虑对象的空间细节和动态变化的时间特性，我们建立了具有两个分支的 OE-槽：对象-槽分支和事件-槽分支。对象槽分支专注于从高空间分辨率但低帧采样率的特征中提取以对象为中心的槽，强调详细的对象信息。事件时隙分支被设计为从高时间采样率但低空间分辨率特征中学习以事件为中心的时隙。这些互补的时隙被组合在一起形成视觉上下文，作为 LLM 的输入以进行有效的视频推理。实验结果证明了该算法的有效性，在视频问答中达到了最好的性能。

视频事件理解

·Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model

In the realm of Multimodal Large Language Models (MLLMs), vision-language connector plays a crucial role to link the pre-trained vision encoders with Large Language Models (LLMs). Despite its importance, the vision-language connector has been relatively less explored. In this study, we aim to propose a strong vision-language connector that enables MLLM to simultaneously achieve high accuracy and low computation cost. We first reveal the existence of the visual anchors in Vision Transformer and propose a cost-effective search algorithm to progressively extract them. Building on these findings, we introduce the Anchor Former (AcFormer), a novel visionlanguage connector designed to leverage the rich prior knowledge obtained from these visual anchors during pretraining, guiding the aggregation of information. Through extensive experimentation, we demonstrate that the proposed method significantly reduces computational costs by nearly two-thirds, while simultaneously outperforming baseline methods. This highlights the effectiveness and efficiency of AcFormer.

多模态对齐, 视觉锚点, 高效计算

视觉传递器是多模式大型语言模型的强大信息聚集器

在多模式大语言模型(MLLMS)领域中，视觉-语言连接器在连接预先训练的视觉编码器和大语言模型(LLMS)方面起着至关重要的作用。尽管视觉-语言连接很重要，但人们对它的研究相对较少。在这项研究中，我们的目标是提出一种强大的视觉语言连接器，使 MLLM 能够同时达到高精度和低计算代价。我们首先揭示了视觉变形中视觉锚点的存在，并提出了一种低成本的搜索算法来逐步提取视觉锚点。基于这些发现，我们引入了 Anchor Former(AcFormer)，这是一种新型的视觉-语言连接器，旨在利用在预培训期间从这些视觉锚获得的丰富先验知识，指导信息的聚合。通过大量的实验，我们证明了所提出的方法显著降低了近三分之二的计算代价，同时性能优于基线方法。这凸显了 AcFormer 的有效性和高效性。

模态融合、连接器、高效

·Unleash Region Understanding in Intermediate Layers for MLLM-based Referring Expression Generation

This paper studies the Multi-modal Large Language Model (MLLM) based Referring Expression Generation (REG) task, which aims to generate an unambiguous text description that applies to exactly one object or region in the image. MLLM-based REG models naturally inherit the hallucination issues from MLLM. We found there is a tradeoff between the detailed description and accurate targeting of referring objects. Providing precise object descriptions necessitates generating sentences with more details, which will also increase the probability of the introducing of hallucinations. To address this issue, we proposed a training-free method, named as ``unleash-then-eliminate'', that first elicits the latent information in intermediate layers, then adopts a cycle-consistency-based decoding method to alleviate the production of hallucinations. We conduct extensive experiments on the RefCOCOg and PHD benchmarks, outperforming existing methods on both semantic and hallucination-related metrics, demonstrating its effectiveness for this task.\footnote{Code will be made available in \url{https://anonymous.4open.science/status/NeurIPS24-DE0B}.}

指代表达生成, 幻觉消除, 中间层特征

在中间层释放区域理解以实现基于 MLLM 的引用表达生成

本文研究了基于多模式大语言模型的指称表达式生成任务，其目的是生成一个明确的文本描述，该文本描述恰好适用于图像中的一个对象或区域。基于 MLLM 的 REG 模型自然继承了 MLLM 的幻觉问题。我们发现，在对所指对象的详细描述和准确定位之间存在权衡。提供精确的物体描述需要生成更详细的句子，这也会增加引入幻觉的可能性。为了解决这个问题，我们提出了一种无需训练的方法，称为 “释放-然后消除”，该方法首先在中间层提取潜在信息，然后采用基于循环一致性的解码方法来缓解幻觉的产生。我们在 RefCOCOg 和 PHD 基准上进行了广泛的实验，在语义和幻觉相关的指标上都优于现有的方法，证明了它对这项任务的有效性。 \脚注{代码将在\url{https://anonymous.4open.science/status/NeurIPS24-DE0B}.}中提供

REG、幻觉

Grounding Multimodal Large Language Models in Actions

Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, including both continuous and discrete actions. For continuous actions, a set of learned tokenizations that capture an action at various resolutions allows for sufficient modeling precision, yielding the best performance on downstream tasks. For discrete actions, semantically aligning these actions with the native output token space of the MLLM leads to the strongest performance. We arrive at these lessons via a thorough study of seven action grounding approaches on five different environments, encompassing over 114 embodied tasks.

动作 Grounding, 多模态对齐, 实体智能

将多模式大型语言模型作为行动的基础

多模式大型语言模型（MLLM）已在包括并行人工智能在内的许多领域展示了广泛的功能。在这项工作中，我们研究如何将 MLLM 最好地嵌入不同的实施例及其相关的动作空间，包括连续和离散动作。对于连续动作，一组学习到的标记化可以以各种分辨率捕获动作，从而可以获得足够的建模精度，从而在下游任务中获得最佳性能。对于离散动作，将这些动作与 MLLM 的本地输出令牌空间进行语义对齐会产生最强的性能。我们通过对五种不同环境下的七种行动基础方法（涵盖超过 114 项具体任务）进行彻底研究来得出这些教训。

动作表征理解

·Automated Multi-level Preference for MLLMs

Current multimodal Large Language Models (MLLMs) suffer from "hallucination", occasionally generating responses that are not grounded in the input images. To tackle this challenge, one promising path is to utilize reinforcement learning from human feedback (RLHF), which steers MLLMs towards learning superior responses while avoiding inferior ones. We rethink the common practice of using binary preferences (i.e., superior, inferior), and find that adopting multi-level preferences (e.g., superior, medium, inferior) is better for two benefits: 1) It narrows the gap between adjacent levels, thereby encouraging MLLMs to discern subtle differences. 2) It further integrates cross-level comparisons (beyond adjacent-level comparisons), thus providing a broader range of comparisons with hallucination examples. To verify our viewpoint, we present the Automated Multi-level Preference (AMP) framework for MLLMs. To facilitate this framework, we first develop an automated dataset generation pipeline that provides high-quality multi-level preference datasets without any human annotators. Furthermore, we design the Multi-level Direct Preference Optimization (MDPO) algorithm to robustly conduct complex multi-level preference learning. Additionally, we propose a new hallucination benchmark, MRHal-Bench. Extensive experiments across public hallucination and general benchmarks, as well as our MRHal-Bench, demonstrate the effectiveness of our proposed method.Code is available at https://anonymous.4open.science/r/amp-nips.

偏好对齐，多级偏好学习, 幻觉消除, 自动数据生成

MLLM 的自动化多级别偏好

当前的多模式大型语言模型(MLLM)存在“幻觉”，偶尔会产生与输入图像无关的反应。为了应对这一挑战，一条很有希望的途径是利用人类反馈的强化学习 (RLHF)，它引导 MLLMS 学习更好的响应，同时避免更差的响应。我们重新思考了使用二元偏好(即，上级、下级)的常见做法，并发现采用多级偏好(例如，上级、中级、下级)更有两个好处：1)它缩小了相邻级别之间的差距，从而鼓励 MLLM 区分细微的差异。2)它进一步整合了跨级别的比较(超越相邻级别的比较)，从而提供了更广泛的带有幻觉例子的比较。为了验证我们的观点，我们提出了 MLLMS 的自动多层次偏好(AMP)框架。为了方便这个框架，我们首先开发了一个自动数据集生成管道，它提供高质量的多层次偏好数据集，而不需要任何人工注释员。此外，我们设计了多层直接偏好优化(MDPO)算法，以稳健地进行复杂的多层偏好学习。此外，我们还提出了一个新的幻觉基准--MRHal-BENCH。在公共幻觉和一般基准以及我们的 MRHal 工作台上进行的广泛实验证明了我们所建议的方法的有效性。代码可在 https://anonymous.4open.science/r/amp-nips.上获得

幻觉问题、增强图像依赖

Unified Generative and Discriminative Training for Multi-modal Large Language Models

In recent times, Vision-Language Models (VLMs) have been trained under two predominant paradigms. Generative training has enabled Multimodal Large Language Models (MLLMs) to tackle various complex tasks, yet issues such as hallucinations and weak object discrimination persist. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval, yet struggles with complex scenarios requiring fine-grained semantic differentiation. This paper addresses these challenges by proposing a unified approach that integrates the strengths of both paradigms. Considering interleaved image-text sequences as the general format of input samples, we introduce a structure-induced training strategy that imposes semantic relationships between input samples and the MLLM’s hidden state. This approach enhances the MLLM’s ability to capture global semantics and distinguish fine-grained semantics. By leveraging dynamic sequence alignment within the Dynamic Time Warping framework and integrating a novel kernel for fine-grained semantic differentiation, our method effectively balances generative and discriminative tasks. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the-art results in multiple generative tasks, especially those requiring cognitive and discrimination abilities. Additionally, our method surpasses discriminative benchmarks in interleaved and fine-grained retrieval tasks. By employing a retrieval-augmented generation strategy, our approach further enhances performance in some generative tasks within one model, offering a promising direction for future research in vision-language modeling.

统一训练范式, 生成判别融合, 语义对齐

多模式大型语言模型的统一生成和区分训练

近年来，视觉语言模型(VLM)在两种主要范式下进行了训练。生成性训练使多通道大型语言模型(MLLMS)能够处理各种复杂的任务，但幻觉和弱对象辨别等问题仍然存在。以 CLIP 等模型为代表的鉴别性训练，在零镜头图文分类和检索方面表现出色，但在需要细粒度语义区分的复杂场景中举步维艰。本文通过提出一种整合了两种范式的优点的统一方法来应对这些挑战。考虑到交织的图文序列作为输入样本的一般格式，我们引入了一种结构诱导训练策略，该策略将输入样本与 MLLM 的隐含状态之间的语义关系强加于输入样本之间。这种方法增强了 MLLM 捕获全局语义和区分细粒度语义的能力。通过利用动态时间规整框架中的动态序列比对，并集成了一个新的用于细粒度语义区分的核，我们的方法有效地平衡了生成性任务和鉴别性任务。广泛的实验证明了我们的方法的有效性，在多个生成任务中获得了最先进的结果，特别是那些需要认知和辨别能力的任务。此外，在交错和细粒度的检索任务中，我们的方法超过了判别性基准。通过使用检索-增强生成策略，我们的方法进一步提高了在一个模型内的一些生成任务的性能，为未来视觉语言建模的研究提供了一个很有前途的方向。

·LOVA3: Learning to Visual Question Answering, Asking and Assessment

Question answering, asking, and assessment are three innate human traits crucial for understanding the world and acquiring knowledge. By enhancing these capabilities, humans can more effectively utilize data, leading to better comprehension and learning outcomes. However, current Multimodal Large Language Models (MLLMs) primarily focus on question answering, often neglecting the full potential of questioning and assessment skills. In this study, we introduce LOVA3, an innovative framework named ``Learning tO Visual Question Answering, Asking and Assessment,'' designed to equip MLLMs with these additional capabilities. Our approach involves the creation of two supplementary training tasks GenQA and EvalQA, aiming at fostering the skills of asking and assessing questions in the context of images. To develop the questioning ability, we compile a comprehensive set of multimodal foundational tasks. For assessment, we introduce a new benchmark called EvalQABench, comprising 64,000 training samples (split evenly between positive and negative samples) and 5,000 testing samples. We posit that enhancing MLLMs with the capabilities to answer, ask, and assess questions will improve their multimodal comprehension and lead to better performance. We validate our hypothesis by training an MLLM using the LOVA3 framework and testing it on 10 multimodal benchmarks. The results demonstrate consistent performance improvements, thereby confirming the efficacy of our approach.

多维问答能力, 问题生成, 答案评估

LOVA 3：学习视觉问题回答、提问和评估

问题回答、提问和评估是人类与生俱来的三个特征，对理解世界和获取知识至关重要。通过增强这些能力，人类可以更有效地利用数据，导致更好的理解和学习结果。然而，目前的多通道大型语言模型主要关注问题回答，往往忽略了提问和评估技能的全部潜力。在这项研究中，我们引入了 LOVA3，这是一个名为“学习视觉问答、提问和评估”的创新框架，旨在为 MLLMS 配备这些额外的能力。我们的方法包括创建两个补充培训任务 GenQA 和 EvalQA，旨在培养在图像背景下提问和评估问题的技能。为了培养提问能力，我们编制了一套全面的多通道基础任务。在评估方面，我们引入了一个名为 EvalQABtch 的新基准，包括 64,000 个训练样本(正负样本平均分配)和 5,000 个测试样本。我们假设，增强 MLMS 的回答、提问和评估问题的能力将提高他们的多模式理解能力，并导致更好的性能。我们使用 LOVA3 框架训练 MLLM，并在 10 个多模式基准上进行测试，以验证我们的假设。结果表明，性能得到了持续的提高，从而证实了我们方法的有效性。

基准、思维学习

·Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models

Large Multimodal Model (LMM) is a hot research topic in the computer vision area and has also demonstrated remarkable potential across multiple disciplinary fields. A recent trend is to further extend and enhance the perception capabilities of LMMs. The current methods follow the paradigm of adapting the visual task outputs to the format of the language model, which is the main component of a LMM. This adaptation leads to convenient development of such LMMs with minimal modifications, however, it overlooks the intrinsic characteristics of diverse visual tasks and hinders the learning of perception capabilities. To address this issue, we propose a novel LMM architecture named Lumen, a Large multimodal model with versatile vision-centric capability enhancement. We decouple the LMM's learning of perception capabilities into taskagnostic and task-specific stages. Lumen first promotes fine-grained vision-language concept alignment, which is the fundamental capability for various visual tasks. Thus the output of the task-agnostic stage is a shared representation for all the tasks we address in this paper. Then the task-specific decoding is carried out by flexibly routing the shared representation to lightweight task decoders with negligible training efforts. Comprehensive experimental results on a series of vision-centric and VQA benchmarks indicate that our Lumen model not only achieves or surpasses the performance of existing LMM-based approaches in a range of vision-centric tasks while maintaining general visual understanding and instruction following capabilities.

视觉能力增强, 任务解耦, 语义对齐

Lumen：释放大型多模式模型的多功能以视觉为中心的功能

大型多通道模型(LMM)是计算机视觉领域的研究热点，也显示出跨越多学科领域的巨大潜力。最近的一个趋势是进一步扩展和增强 LMM 的感知能力。目前的方法遵循使视觉任务输出适应语言模型的格式的范例，语言模型是 LMM 的主要组成部分。这种适应导致了这类 LMM 的发展，但它忽略了不同视觉任务的内在特征，阻碍了感知能力的学习。为了解决这一问题，我们提出了一种新的 LMM 体系结构 Lumen，这是一个具有通用的以视觉为中心的能力增强的大型多通道模型。我们将 LMM 的感知能力学习分离为任务不可知阶段和任务特定阶段。流明首先促进细粒度的视觉-语言概念对齐，这是各种视觉任务的基本能力。因此，任务不可知阶段的输出是我们在本文中讨论的所有任务的共享表示。然后，通过将共享表示灵活地路由到轻量级任务解码器来执行特定于任务的解码，训练工作量可以忽略不计。在一系列视觉中心和 VQA 基准上的综合实验结果表明，我们的 Lumen 模型不仅在一系列以视觉为中心的任务中达到或超过了现有基于 LMM 的方法的性能，而且保持了一般的视觉理解和指令跟随能力。

视觉理解、视觉中心

·Octopus: A Multi-modal LLM with Parallel Recognition and Sequential Understanding

A mainstream of Multi-modal Large Language Models (MLLMs) have two essential functions, i.e., visual recognition (e.g., grounding) and understanding (e.g., visual question answering). Presently, all these MLLMs integrate visual recognition and understanding in a same sequential manner in the LLM head, i.e., generating the response token-by-token for both recognition and understanding. We think unifying them in the same sequential manner is not optimal for two reasons: 1) parallel recognition is more efficient than sequential recognition and is actually prevailing in deep visual recognition, and 2) the recognition results can be integrated to help high-level cognition (while the current manner does not). Such motivated, this paper proposes a novel “parallel recognition → sequential understanding” framework for MLLMs. The bottom LLM layers are utilized for parallel recognition and the recognition results are relayed into the top LLM layers for sequential understanding. Specifically, parallel recognition in the bottom LLM layers is implemented via object queries, a popular mechanism in DEtection TRansformer, which we find to harmonize well with the LLM layers. Empirical studies show our MLLM named Octopus improves accuracy on popular MLLM tasks and is up to 5× faster on visual grounding tasks.

视觉识别与理解、并行识别, 序列理解, 双阶段架构

Octopus：具有并行识别和顺序理解的多模式 LLM

主流的多模式大语言模型有两个基本功能，即视觉识别(如基础)和理解(如视觉问答)。目前，所有这些 MLLM 在 LLM 头中以相同的顺序方式集成视觉识别和理解，即逐个标记地生成用于识别和理解的响应标记。我们认为将它们以相同的顺序方式统一并不是最优的，原因有两个：1)并行识别比顺序识别效率更高，实际上在深度视觉识别中占优势；2)识别结果可以被整合，以帮助高级认知(而目前的方式不是这样)。在此背景下，本文提出了一种新的支持多语言学习的并行识别和顺序理解框架(Parly Recognition→Sequence Underach)。利用底层 LLM 进行并行识别，并将识别结果传递到上层 LLM 进行顺序理解。具体地说，底层 LLM 层的并行识别是通过对象查询来实现的，这是检测变压器中的一种流行机制，我们发现这种机制与 LLM 层很好地协调。实验研究表明，我们的名为 Octopus 的 MLLM 提高了流行的 MLLM 任务的准确率，并且在视觉接地任务中的速度提高了 5 倍。

视觉识别、理解

·MoVA: Adapting Mixture of Vision Experts to Multimodal Context

As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e.g., the CLIP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the MoVA, a powerful and novel MLLM, adaptively routing and fusing taskspecific vision experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts. This benefits from the powerful model function understanding ability of the large language model (LLM). In the fine-grained stage, we elaborately conduct the mixture-ofvision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge from various experts. This coarse-to-fine paradigm effectively leverages representations from experts based on multimodal context and model expertise, further enhancing the generalization ability. We conduct extensive experiments to evaluate the effectiveness of the proposed approach. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of challenging multimodal benchmarks.

视觉理解推理、视觉专家混合, 自适应路由, 粗细粒度融合

MoVA：将混合视觉专家适应多模式环境

视觉编码器作为多通道大语言模型的关键组成部分，其性能的好坏直接影响着多通道大语言模型对不同图像内容的理解。虽然一些大规模的预训练视觉编码器，如 CLIP 和 DINOv2 中的视觉编码器，已经带来了良好的性能，但我们发现仍然没有单一的视觉编码器能够主导各种图像内容的理解，例如，CLIP 视觉编码器在一般图像理解方面表现出色，但在文档或图表内容上的性能较差。为了缓解裁剪视觉编码器的偏差，我们首先深入研究了不同预训练视觉编码器的固有行为，然后提出了一种强大而新颖的 MLLM，它采用从粗到精的机制自适应地路由和融合特定任务的视觉专家。在粗粒度阶段，我们设计了一种上下文感知的专家路由策略，根据用户指令、输入图像和视觉专家的专业知识动态选择最合适的视觉专家。这得益于大型语言模型(LLM)强大的模型函数理解能力。在细粒度阶段，我们精心地进行了混合视觉专家适配器(MOV-Adapter)来提取和融合来自不同专家的特定任务知识。这种从粗略到精细的范例有效地利用了专家基于多模式背景和模型专业知识的表示，进一步增强了泛化能力。我们进行了大量的实验来评估所提出的方法的有效性。在没有任何花哨的情况下，Mova 可以在一系列具有挑战性的多模式基准测试中实现比当前最先进的方法显著的性能提升。

·Understanding Information Storage and Transfer in Multi-Modal Large Language Models

Understanding the mechanisms of information storage and transfer in Transformer-based models is important for driving model understanding progress. Recent work has studied these mechanisms for Large Language Models (LLMs), revealing insights on how information is stored in a model's parameters and how information flows to and from these parameters in response to specific prompts. However, these studies have not yet been extended to Multi-modal Large Language Models (MLLMs). Given their expanding capabilities and real-world use, we start by studying one aspect of these models -- how MLLMs process information in a factual visual question answering task. We use a constraint-based formulation which views a visual question as having a set of visual or textual constraints that the model's generated answer must satisfy to be correct (e.g. What movie directed by \emph{the director in this photo} has won a \emph{Golden Globe}?). Under this setting, we contribute i) a method that extends causal information tracing from pure language to the multi-modal setting, and ii) \emph{VQA-Constraints}, a test-bed of 9.7K visual questions annotated with constraints. We use these tools to study two open-source MLLMs, LLaVa and multi-modal Phi-2. Our key findings show that these MLLMs rely on MLP and self-attention blocks in much earlier layers for information storage, compared to LLMs whose mid-layer MLPs are more important. We also show that a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. We validate these mechanisms by introducing MultEdit a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs by targeting these causal blocks. We will publicly release our dataset and code.

模型解释、信息流分析、因果追踪

理解多模式大型语言模型中的信息存储和传输

了解基于变压器的模型中的信息存储和传递机制对于推动模型理解的进展具有重要意义。最近的工作研究了大型语言模型(LLM)的这些机制，揭示了信息如何存储在模型的参数中，以及信息如何响应特定提示在这些参数之间流动。然而，这些研究尚未扩展到多通道大语言模型(MLLMS)。考虑到它们的扩展能力和现实世界的使用，我们首先研究这些模型的一个方面--MLLMS 如何在一个实际的可视问题回答任务中处理信息。我们使用基于约束的公式，它认为视觉问题具有一组视觉或文本约束，模型生成的答案必须满足这些约束才能正确(例如，由\emph{这张照片中的导演}执导的哪部电影获得了\emph{金球奖}？)。在此背景下，我们提出了一种将因果信息追踪从纯语言扩展到多通道环境的方法，以及基于约束的 9.7K 可视问题的测试平台 EMPH。我们使用这些工具研究了两个开源的 MLLMS，LLaVa 和多模式 Phi-2。我们的主要发现表明，这些多层记忆模型依赖于更早的层中的 MLP 和自我注意块进行信息存储，而不是那些中间层 MLP 更重要的 LLMS。我们还表明，由视觉编码器输出的一致的小的视觉标记子集负责将信息从图像传输到这些因果块。我们通过引入一个模型编辑算法来验证这些机制，该算法可以纠正错误，并通过针对这些因果块将新的长尾信息插入到 MLLMS 中。我们将公开发布我们的数据集和代码。

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Emotion perception is crucial in applications like human-computer interaction, education, and counseling. However, traditional single-modality approaches fall short in real-world scenarios with multimodal emotional data, and Multimodal Large Language Models (MLLMs) struggle with audio integration and recognition facial micro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. Furthermore , we propose Emotion-LLaMA, a model that integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA enhances both emotional recognition and reasoning capabilities. Extensive evaluations show Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on the MER2023 challenge, and the highest UAR (45.59) and WAR ( 59.37) in zero-shot evaluations on the DFEW dataset. Code and models are available at the supplementary materials.

情感识别、多模态融合、指令微调

描述-LLaMA：具有指令调优的多模式情感识别和推理

情绪感知在人机交互、教育和咨询等应用中至关重要。然而，传统的单通道方法在多通道情感数据的真实场景中并不适用，而多通道大语言模型(MLLMS)在音频集成和面部微表情识别方面存在困难。为了解决这个问题，我们引入了 MERR 数据集，它包含 28,618 个粗粒度样本和 4,487 个细粒度注释样本，涉及不同的情感类别。此数据集使模型能够从不同的场景中学习并推广到现实世界的应用程序。此外，我们还提出了情感骆驼模型，该模型通过特定于情感的编码器整合了音频、视觉和文本输入。通过将特征对齐到共享空间，并使用带有指令调整的改进的大羊驼模型，情感大羊驼增强了情感识别和推理能力。广泛的评估显示，Emove-Allama 的表现优于其他 MLLM，在 EMER 上获得了线索重叠(7.83)和标签重叠(6.25)的最高分数，在 MER2023 挑战赛上获得了 0.9036 的 F1 分数，在 DFEW 数据集的零命中评估中获得了最高的 UAR(45.59)和 WAR(59.37)。代码和型号可在补充资料中找到。

·VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

We present VisionLLM v2, an end-to-end generalist multimodal large model (MLLM) that unifies visual perception, understanding, and generation within a single framework. Unlike traditional MLLMs limited to text output, VisionLLM v2 significantly broadens its application scope. It excels not only in conventional visual question answering (VQA) but also in open-ended, cross-domain vision tasks such as object localization, pose estimation, and image generation and editing. To this end, we propose a new information transmission mechanism termed ``super link'', as a medium to connect MLLM with taskspecific decoders. It not only allows flexible transmission of task information and gradient feedback between the MLLM and multiple downstream decoders but also effectively resolves training conflicts in multi-tasking scenarios. In addition, to support the diverse range of tasks, we carefully collected and combed training data from hundreds of public vision and vision-language tasks. In this way, our model can be joint-trained end-to-end on hundreds of vision language tasks and generalize to these tasks using a set of shared parameters through different user prompts, achieving performance comparable to task-specific models. We believe VisionLLM v2 will offer a new perspective on the generalization of MLLMs.

通用视觉理解、多任务学习、端到端训练

VisionLLM v2：用于数百项视觉语言任务的端到端通用多模式大型语言模型

我们提出了 VisionLLm v2，这是一个端到端的通用型多通道大型模型(MLLM)，它将视觉感知、理解和生成统一在一个框架中。与传统的仅限于文本输出的 MLLMS 不同，VisionLLm v2 显著拓宽了其应用范围。它不仅在传统的视觉问答(VQA)中表现出色，而且在开放的、跨域的视觉任务中也表现出色，例如对象定位、姿势估计以及图像生成和编辑。为此，我们提出了一种新的信息传输机制，称为“超级链接”，作为连接 MLLM 和特定任务解码器的媒介。它不仅允许 MLLM 和多个下行解码器之间灵活地传输任务信息和梯度反馈，而且有效地解决了多任务场景中的训练冲突。此外，为了支持多样化的任务，我们仔细收集和梳理了数百项公共视觉和视觉语言任务的培训数据。这样，我们的模型就可以对数百个视觉语言任务进行端到端的联合训练，并通过不同的用户提示使用一组共享参数来推广到这些任务，获得了与特定任务模型相当的性能。我们相信，VisionLLm v2 将为 MLLMS 的推广提供一个新的视角。

·Wings: Learning Multimodal LLMs without Text-only Forgetting

Multimodal large language models (MLLMs), initiated with a trained LLM, first align images with text and then fine-tune on multimodal mixed inputs. However, the MLLM catastrophically forgets the text-only instructions, which do not include images and can be addressed within the initial LLM. In this paper, we present Wings, a novel MLLM that excels in both text-only dialogues and multimodal comprehension. Analyzing MLLM attention in multimodal instructions reveals that text-only forgetting is related to the attention shifts from pre-image to post-image text. From that, we construct extra modules that act as the boosted learner to compensate for the attention shift. The complementary visual and textual learners, like "wings" on either side, are connected in parallel within each layer's attention block. Initially, image and text inputs are aligned with visual learners operating alongside the main attention, balancing focus on visual elements. Textual learners are later collaboratively integrated with attention-based routing to blend the outputs of the visual and textual learners. We design the Low-Rank Residual Attention (LoRRA) to guarantee high efficiency for learners. Our experimental results demonstrate that Wings outperforms equally-scaled MLLMs in both text-only and visual question-answering tasks. On a newly constructed Interleaved Image-Text (IIT) benchmark, Wings exhibits superior performance from text-only-rich to multimodal-rich question-answering tasks.

多模态训练、灾难性遗忘、注意力机制、多模态平衡

翅膀：学习多模式 LLM，而不会忘记纯文本

多模式大语言模型(MLLMS)由训练有素的 LLM 启动，首先将图像与文本对齐，然后对多模式混合输入进行微调。然而，MLLM 灾难性地忘记了纯文本指令，这些指令不包括图像，并且可以在初始 LLM 中寻址。在本文中，我们介绍了 Wings，一种新颖的 MLLM，它在纯文本对话和多通道理解方面都表现出色。对多通道教学中 MLLM 注意的分析表明，只有文本遗忘与注意从图像前向图像后转移有关。在此基础上，我们构建了额外的模块，作为增强的学习者来补偿注意力转移。相辅相成的视觉和文本学习者，就像两边的“翅膀”，在每一层的注意块内平行连接。最初，图像和文本输入与视觉学习者在主要注意力的同时操作，平衡对视觉元素的关注。文本学习者随后被协作地与基于注意力的路径相结合，以混合视觉学习者和文本学习者的输出。我们设计了低等级剩余注意力 (LORRA)来保证学习者的高效率。我们的实验结果表明，Wings 在纯文本和视觉问答任务中的表现都优于同等规模的 MLLMS。在一个新构建的交错图文(IIT)基准测试中，Wings 在从纯文本丰富到多模式丰富的问答任务中表现出了卓越的性能。

·FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion

As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce ``FuseMoE'', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks.

多模态融合 模型结构 缺失数据处理

FuseMoE：多模式融合的专家混合变形金刚

随着关键领域的机器学习模型越来越多地应对多模式数据，它们面临着处理多种模式（由于元素缺失而通常不完整）以及收集样本的时间不规则性和稀疏性的双重挑战。成功利用这些复杂的数据，同时克服高质量训练样本的稀缺性，是提高这些模型预测性能的关键。我们引入了“FuseMoE”，这是一个集成了创新门控功能的混合专家框架。FueMoE 旨在集成多种模式，可以有效管理模式缺失和不规则采样数据轨迹的场景。理论上，我们独特的门控功能有助于提高收敛率，从而在多个下游任务中实现更好的性能。FueMoE 在现实世界中的实际用途得到了一系列不同具有挑战性的预测任务的验证。

·Towards Neuron Attributions in Multi-Modal Large Language Models

As Large Language Models (LLMs) demonstrate impressive capabilities, demystifying their internal mechanisms becomes increasingly vital. Neuron attribution, which attributes LLM outputs to specific neurons to reveal the semantic properties they learn, has emerged as a key interpretability approach. However, while neuron attribution has made significant progress in deciphering text-only LLMs, its application to Multimodal LLMs (MLLMs) remains less explored. To address this gap, we propose a novel Neuron Attribution method tailored for MLLMs, termed NAM. Specifically, NAM not only reveals the modality-specific semantic knowledge learned by neurons within MLLMs, but also highlights several intriguing properties of neurons, such as cross-modal invariance and semantic sensitivity. These properties collectively elucidate the inner workings mechanism of MLLMs, providing a deeper understanding of how MLLMs process and generate multi-modal content. Through theoretical analysis and empirical validation, we demonstrate the efficacy of NAM and the valuable insights it offers. Furthermore, leveraging NAM, we introduce a multi-modal knowledge editing paradigm, underscoring the practical significance of our approach for downstream applications of MLLMs.

可解释性 神经元分析 多模态LLM

多模式大语言模型中的神经元属性

随着大型语言模型(LLM)显示出令人印象深刻的能力，揭开它们的内部机制变得越来越重要。神经元属性，它将 LLM 的输出归因于特定的神经元，以揭示它们学习的语义属性，已经成为一种关键的可解释性方法。然而，尽管神经元属性在破译纯文本 LLMS 方面取得了重大进展，但其在多模式 LLMS(MLLMS)中的应用仍鲜有人探索。为了解决这一问题，我们提出了一种新的神经元属性方法，称为 NAM。具体地说，NAM 不仅揭示了 MLLMS 中神经元学习的特定通道的语义知识，而且突出了神经元的一些有趣的属性，如跨通道不变性和语义敏感性。这些性质共同阐明了 MLLMS 的内部工作机制，提供了对 MLLMS 如何处理和生成多模式内容的更深层次的理解。通过理论分析和实证验证，我们论证了 NAM 的有效性及其提供的有价值的见解。此外，利用 NAM，我们引入了多模式知识编辑范式，强调了我们的方法对 MLLMS 下游应用的实际意义。

·KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension

Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced their capabilities in vision-language tasks like image captioning and question answering. However, these models lack proficiency in the fine-grained perceptual task, i.e., keypoint comprehension. To bridge this gap, we introduce the novel challenge of Semantic Keypoint Comprehension, which aims to understand keypoints within different human-AI interaction contexts, including keypoint semantic understanding, visual prompt-based keypoint detection, and textual prompt-based keypoint detection. Moreover, we introduce KptLLM, a unified multimodal model that utilizes an identify-then-detect strategy to effectively address these challenges. KptLLM underscores the initial discernment of semantics in keypoints, followed by the precise determination of their positions through a chain-of-thought process. With several carefully designed modules, KptLLM adeptly handles various modality inputs, facilitating the interpretation of both semantic contents and keypoint locations. Our extensive experiments demonstrate KptLLM's superiority in various keypoint detection benchmarks and its unique semantic capabilities in interpreting keypoints. Codes and models will be released to facilitate future research.

内容关键点理解、关键点检测、语义理解、思维链

KpLLM：揭示大型语言模型在关键点理解中的力量

多模式大型语言模型(MLLMS)的最新进展显著增强了它们在图像字幕和问题回答等视觉语言任务中的能力。然而，这些模型缺乏对细粒度知觉任务，即关键点理解的熟练程度。为了弥补这一差距，我们引入了语义关键点理解的新挑战，旨在理解不同人机交互上下文中的关键点，包括关键点语义理解、基于视觉提示的关键点检测和基于文本提示的关键点检测。此外，我们还引入了 KptLLM，这是一个统一的多通道模型，它利用先识别后检测的策略来有效地应对这些挑战。KptLLM 强调了对关键点的语义的初始识别，然后通过思维链过程精确地确定它们的位置。通过几个精心设计的模块，KptLLM 熟练地处理各种通道输入，促进了语义内容和关键点位置的解释。我们的大量实验表明，KptLLM 在各种关键点检测基准测试中的优势以及在解释关键点方面的独特语义能力。代码和模型将公布，以方便未来的研究。