Research and development in multimodal large models are advancing rapidly, but despite significant progress, there has always been a gap in capabilities between open-source and commercial models. The OpenGVLab team from Shanghai AI Laboratory, in partnership with Tsinghua University and SenseTime, recently released a new open-source multimodal large language model project called InternVL 1.5. This project aims to challenge the dominance of commercial model giants like GPT-4V and raises the question of how far the power of open source can go.

Today, InternVL has been the top 1 Huggingface trending model for both image feature extraction and VQA.

As the latest generation of the open-source vision-languaged model(VLM) or multimodal large language models (MLLM), InternVL 1.5 achieves performance parity with top commercial models like GPT-4V and demonstrates outstanding technical advantages in the open-source domain. The power of the open-source community is once again affirmed, driving not only technological advancement but also the construction of a tech ecosystem that emphasizes sharing, cooperation, and innovation.

Technical report:

https://arxiv.org/pdf/2404.16821

Demo（Try it！）

https://huggingface.co/spaces/OpenGVLab/InternVL

Code:

https://github.com/OpenGVLab/InternVL

3 Highlights of InternVL 1.5

Powerful Visual Encoder

InternVL 1.5, through its unique visual encoder, InternViT-6B, employs a continuous learning strategy to greatly enhance the depth and breadth of visual understanding. This strategy enables InternViT-6B to achieve seamless transfer and reuse among various large language models, strengthening the model’s ability to parse complex visual content and exhibit more precise recognition and interpretation capabilities in image-intensive tasks.

Dynamic High Resolution

InternVL 1.5 introduces a brand-new dynamic high-resolution strategy for image processing.This feature supports input resolutions of up to 4K, optimizing the presentation of image details and improving the model’s expressiveness and accuracy on high-resolution images, while also ensuring efficient computation. This revolutionary feature significantly enhances the overall image processing performance and is expected to be a game-changer in the field of image processing.

High-Quality Bilingual Dataset

InternVL 1.5 integrates a wide range of high-quality bilingual datasets covering English and Chinese, significantly improving the model’s operational flexibility and accuracy in multilingual environments. In addition, through the data translation pipeline developed by open-source large language models, InternVL 1.5 can automatically expand to more languages, showing tremendous potential in global applications.

InternVL 1.5 VS GPT4V

Benchmark Evaluation

To evaluate InternVL 1.5’s performance, the research team conducted extensive evaluations in 18 multimodal benchmark tests. These benchmarks cover various aspects, including OCR-related, general multimodal, mathematical, and multi-turn dialogues. InternVL 1.5 performs well on multiple key dimensions, narrowing the gap between open-source models and GPT-4V, especially reaching the SOTA level in tasks such as OCR, MMB, SEED, and Math.

Evaluation on real case

General QA

InternVL 1.5 was tested with everyday user questions to evaluate its performance in general question answering. In the first example, InternVL 1.5 demonstrated a good understanding of the image and correctly explained the movement in the image. It also distinguished the positioning of two individuals, slightly outperforming GPT-4V in detail. In the second example, InternVL 1.5 identified the action as imitating Bruce Lee’s posture, while GPT-4V failed to answer this question.

OCR-related QA

Compare the OCR ability of the InternVL 1.5 and GPT-4V with two small cases. In the first test, attention was paid to the model’s understanding of Chinese scenes. The results show that GPT-4V performs poorly in extracting all useful information from the image, while InternVL 1.5 can more accurately identify and parse details in the image. The second test focuses on graph understanding. GPT-4V and InternVL 1.5 can effectively parse graph data and structure, identifying the highest and lowest years, but GPT-4V fails to answer the final difference accurately. In contrast, InternVL 1.5 accurately answers the difference as “1580 billion RMB”.

Scientific Understanding QA

In the field of multimodal large language models, models perform poorly in complex scenarios involving domain-specific knowledge and logical reasoning. However, in the first question shown in the figure below, the InternVL 1.5 model not only accurately analyzes the elements in the image but also provides precise answers, while GPT-4V demonstrates its unique insights in inferring trends in amino acid transport. In the second example, both models can accurately answer and provide in-depth analysis from the perspective of aerodynamics, demonstrating their efficient capabilities in handling scientific problems. These achievements also demonstrate the comparability of InternVL 1.5 and GPT-4V in scientific understanding and reasoning abilities.

Object Localization

Accurate object localization techniques become crucial in multimodal tasks. In the following tests, the InternVL 1.5 model performs exceptionally well. It not only quickly and accurately identifies the specific position of the basketball player but also describes the posture of the corresponding player in detail. It can also accurately interpret dynamic interactions in complex scenes, such as correctly identifying the man’s right hand pointing and understanding the meaning the man wants to convey based on the actual scene.

Conclusion

InternVL 1.5 is not just a project but also an important contribution of the open-source community to high-level multimodal artificial intelligence technology. In the commercial field, multimodal models like GPT-4V have become industry benchmarks. In this context, InternVL 1.5 demonstrates that the open-source community can not only keep pace with the commercial field but also lead in some aspects.

InternVL 1.5 significantly improves the performance of open-source multimodal models and narrows the gap with commercial models. This open-source model is expected to drive the development of the multimodal field and deserves further exploration and optimization.

As a completely open-source project, InternVL 1.5 encourages everyone interested to participate and jointly promote the future development of AI. By visiting the online demo of InternVL 1.5 (https://internvl.opengvlab.com), researchers and developers worldwide can now experience the charm of this technology and participate in this exciting open-source project.

Demo（Try it！）

https://huggingface.co/spaces/OpenGVLab/InternVL

Opensource link: https://github.com/OpenGVLab/InternVL

Follow us: https://twitter.com/opengvlab

Open-source vision-language model now comparable to GPT-4V was originally published in AI Advances on Medium, where people are continuing the conversation by highlighting and responding to this story.

1. Background

With the substantial advances in big data and computational power, large language models (LLMs) have emerged as a central point of interest in industry and academia. To broaden their applicability across diverse contexts, multimodal LLMs have recently become a fast-evolving field.

Concurrently, various evaluation benchmarks are curated to assess their visual comprehension performance across different domains. However, most of them only contain images from daily life. To deeply assess the multimodal logical thinking prowess of MLLMs, the capability to solve mathematical problems involving diagrams is a critical measure.

Nevertheless, an appropriate mathematical benchmark for MLLMs still exists. Previous efforts in the field, e.g., GeoQA, MathVista, and MMMU, suffer from several issues under our analysis. Therefore, we present MathVerse, a holistic and specialized visual math benchmark crafted to evaluate the multimodal mathematical reasoning skills of MLLMs.

Paper：https://arxiv.org/pdf/2403.14624.pdf

Project：https://mathverse-cuhk.github.io/

Code：https://github.com/ZrrSkywalker/MathVerse

Dataset：https://huggingface.co/datasets/AI4Math/MathVerse

PS： The work ranks 1st in the Huggingface Daily Paper🔥🔥🔥. It is also trending in X with over 10K views.

2. Key Observation

Through our comprehensive observation and analysis, we identify three primary issues in current mathematical benchmarks for evaluating MLLMs:

(1) Do MLLMs truly see the math diagrams in evaluation?

This is the most fundamental question concerning assessing visual math problem-solving. Figure 1 (a) showcases three examples from current benchmarks. We observe that their texts contain too much duplicate information (highlighted in red), also depicted in the diagram. This redundancy may provide MLLMs with a shortcut to resolve the problem by mostly reading the text rather than interpreting the diagram. Our hypothesis gains support from the experiment in Figure 1 (b). For 40 randomly sampled problems from each benchmark, we remove redundant texts from the question, challenging MLLMs to capture the corresponding information exclusively from visual inputs.

The results reveal a significant drop in accuracy among most MLLMs (the blue column), even falling below the scores without taking diagrams as input (the grey column).

This outcome suggests that MLLMs primarily depend on textual cues rather than the visual diagrams themselves to solve these problems in evaluation. Given this, we demonstrate that current visual math benchmarks might need to be more comprehensive to assess the genuine multimodal mathematical reasoning capabilities of MLLMs.

(2) Is it equitable to assess solely by the final answer?

Most existing multimodal benchmarks directly compare model outputs with ground truths to derive a binary evaluation result. While this approach may suffice for general visual contexts, it must improve in math problems requiring intricate step-by-step reasoning. In Figure 2, we examine three model outputs. Although they all arrive at incorrect answers in the end, they demonstrate varying levels of precision in the intermediate reasoning processes. Merely categorizing these outputs as Incorrect fails to capture the nuanced differences in the reasoning quality of MLLMs.

(3) Do they specialize in mathematical reasoning evaluation?

GeoQA narrowly targets specific aspects of plane geometry. This limits evaluating broader mathematical capabilities, e.g., functions and solid geometry. Instead, MathVista expands its scope by including a wide array of peripheral tasks (19 out of 28), encompassing natural images, statistic plots, and charts, which do not directly evaluate professional math skills. Furthermore, the math problems in MMMU are of college-level complexity with extensive domain-specific knowledge, potentially hindering MLLMs from fully demonstrating their reasoning capacity.

3. MathVerse Benchmark

(1) Data Composition and Categorization

MathVerse comprises 2,612 visual math problems, which contribute to the final created 15K test samples. These test samples cover 3 fundamental math subjects: plane geometry, solid geometry, and functions, as well as 12 fine-grained categories. With meticulous annotation and review, the high-quality data in MathVerse provides a robust and comprehensive benchmark for evaluating the capability of multimodal mathematical reasoning.

(2) How can the MLLMs’ capability of mathematical image understanding be assessed?

We first define three distinct categories for the textual information within the questions:

1) Descriptive Information refers to the diagram’s directly observable and clearly portrayed content. It depicts the elemental figure composition, spatial arrangement, and annotated entities, such as the presence of geometric shapes or intersection points of functions. This information is repetitive of the visual components in the diagram and is thus regarded as redundant information for problem-solving.

2) Implicit Property involves information that requires a higher level of visual perception but less mathematical knowledge to discern from the diagram. It signifies solid visual conditions for problem-solving, such as parallelism and perpendicularity between lines, similarity, congruence among triangles, and the category and periodicity of functions.

3) Essential Condition denotes the specific numerical or algebraic measurements, indispensable conditions to derive the solution, and cannot be derived from the visual diagram. This category encompasses precise values of angles, lengths, and function expressions.

Based on the three categories, expert annotators systematically remove different textual information within questions and incrementally incorporate the critical elements into diagrams, resulting in six different question versions. This approach can progressively reduce textual redundancy and information content, thereby increasingly compelling MLLMs to capture mathematical conditions from the visual input. With this curated problem set, we can provide a holistic evaluation of the genuine visual comprehension of MLLMs and whether it can facilitate multimodal mathematical reasoning.

(3) How can the intermediate solution steps of MLLM be evaluated fine-grained?

Compared to visual question-answering in general scenarios, solving MLLMs for mathematical problems requires nuanced, step-by-step CoT reasoning. To this end, we propose a CoT evaluation strategy to thoroughly assess their mathematical CoT skills in visual contexts. It involves two prompting phases with GPT4 and GPT4V: Key-step Extraction and Multi-step Scoring, as shown in the following graph.

This evaluation strategy focuses on the correctness of the final answer and emphasizes the logical coherence and depth of reasoning during the problem-solving process. Through this approach, we can more accurately reveal the true capabilities of MLLMs in solving complex mathematical problems, especially in how they construct problem-solving solutions step by step. This is crucial for understanding the thinking patterns, reasoning abilities, and how they process and interpret the integration of visual and mathematical information of MLLMs.

4. Experiments and Conclusion

As shown in the table below, we conducted experiments on 17 MLLMs on MathVerse. ‘CoT-E’ or ‘w/o’ denotes whether to employ the proposed CoT evaluation strategy.

Based on the evaluation result, we come to the following conclusions:

(1) MLLMs rely more on textual information for problem-solving rather than observing mathematical images.

(2) Apart from GPT-4V and ShareGPT4V, most MLLMs can achieve higher scores solely through text, without the input of images. This proves that low-quality visual encoding currently plays a negative role in problem-solving.

(3) MLLMs need help to accurately interpret the primary conditions and questions from images.

(4) Closed-source models have better multimodal mathematical problem-solving capabilities than open-source models.

(5) Comparing G-LLaVA and LLaVA-1.5 shows that fine-tuning models with mathematical training data can enhance specific problem-solving abilities but also reduce their generalization capabilities.

(6) Compared to binary evaluation, The Chain of Thought (CoT) evaluation can more comprehensively reflect the model’s logical reasoning ability.

Does Your Multi-model LLM Truly See The Diagrams In Visual Math Problems? was originally published in AI Advances on Medium, where people are continuing the conversation by highlighting and responding to this story.

• Large Scale: InternVid is one of the largest publicly available video-text datasets, containing more than 7 million videos with a total duration of nearly 760,000 hours.
• Diverse Content: Video content spans various domains, including everyday life, sports, entertainment, education, and more, catering to the needs of different research and applications.
• High Quality: Both the videos and texts have been carefully selected and processed to ensure the high quality of the dataset, which provides rich captions, video-text similarity scores, and video aesthetics scores.

The Starting Point of InternVid

Learning transferable video-text representations is crucial for video understanding, especially in practical applications such as autonomous driving, intelligent surveillance, and visual search. OpenAI’s release of the Sora model has recently marked significant progress in text-to-video generation. Sora not only breaks through the limitations of video coherence but also maintains consistency across multi-angle camera transitions and demonstrates a profound understanding of the logic of the real world. This breakthrough opens new possibilities for multimodal contrastive learning in the video-language domain.

Although Sora is not yet available for public use, its achievements in video generation, akin to GPT-3 in the text domain, suggest that the realization of general artificial intelligence may arrive sooner than anticipated. The development of InternVid is set against this backdrop, aiming to advance video understanding and generation by providing a large-scale dataset that can facilitate the training and evaluation of models capable of handling the complexities of real-world video-text interactions.

However, a key reason currently limiting exploration is the need for high-quality video-language datasets for large-scale pre-training. Current research relies on datasets such as HowTo100M [1], HD-VILA [2], and YT-Temporal [3, 4], where the text is generated using Automatic Speech Recognition (ASR). Despite the vast scale of these datasets, they often need higher semantic relevance between the videos and their corresponding text descriptions. Such data is not well-suited for generation tasks like text-to-video synthesis, and on the other hand, enhancing this relevance (for example, by aligning videos with captions to improve their match) would significantly benefit downstream tasks, such as video retrieval and video question-answering.

To address the challenge of scaling up video-language modeling while maintaining high video-text correspondence, we propose a large-scale video-centric dataset, InternVid, as shown in Figure 1. ASR transcripts barely describe the visual elements in the videos, whereas generated captions cover more visual content. This dataset includes highly relevant video-text pairs, comprising over 7 million videos, totaling 760,000 hours, generating 234 million video clips, covering 16 scenes and approximately 6,000 actions. We employ a multi-scale approach to generate captions to enhance the video-text match. On a coarse scale, we describe the middle frame of each video. On a fine scale, we generate frame-by-frame captions and summarize them with a language model.

Through InternVid, we learned about a video representation model called ViCLIP, which achieves zero-shot solid performance. For text-to-video generation, we have curated an aesthetic subset of InternVid, as InternVid-Aes, covering 18 million video clips. Together with WebVid-10M [5], InternVid can significantly enhance the generative capabilities of diffusion-based video generation models.

II. Dataset Collection

We have identified three critical factors for building this dataset: significant temporal information, rich and diverse semantics, and video-text correlation. We collected raw videos based on query terms for actions and activities to ensure high temporality. To enrich and diversify semantics, we crawled popular videos from various categories and deliberately increased the proportion of data from different countries and languages. To enhance video-text relevance, we used image description and language models to generate video descriptions from annotations of specific frames. Next, we will detail the dataset construction process and discuss its statistics and characteristics.

Diversity of the Dataset. Crawled popular videos from various categories, and various videos were collected from 16 popular categories, with videos from multiple percentages. To ensure diversity, we selected videos from countries with different languages rather than relying on a dominant language environment. The countries sampled include the United Kingdom, the United States, Australia, Japan, South Korea, China, Russia, and France. In terms of duration, each video averages 351.9 seconds in length. Half (49%) of the videos are at most five minutes, while a quarter (26%) of the videos range from five to ten minutes. 8% of the videos are at most 20 minutes in duration. Among the curated videos, 85% are of high resolution (720P), with the remaining 15% varying in resolution from 360P to 720P. Although videos of lower resolution may not perform as well as high-resolution videos in content generation tasks, they can still be used for video-language representation learning as long as appropriate captions accompany them.

InternVid demonstrates diversity at the segmented clip level with varying clip durations and caption lengths. Aesthetic scores and video-text similarities are evenly distributed. Most clips are between 0 and 10 seconds long, accounting for 85% of all clips. Approximately half of the clip captions contain 10 to 20 words, while a third contains fewer than 10. About 11% of the clips have long captions with more than 20 words.

Original Data Collection. We collected videos from YouTube, considering the diversity and richness of its data and its availability for academic use. We obtained 7 million YouTube video links with an average duration of 6.4 minutes. To ensure the uniqueness of our dataset, we created a database of YouTube IDs and excluded any videos already available in publicly available datasets before April 2023. On the one hand, we selected popular channels and their corresponding trending or highly-rated videos, obtaining 2 million videos from categories such as news and gaming. On the other hand, we created a list of verbs related to actions/activities. We also obtained 5.1 million videos with this list, selecting videos at the top of search results.

Video Retrieval Keywords. We defined approximately 6.1K action phrases from ATUS [6], public video datasets, and text corpora. These were then refined using a language model and manually created. We utilized ATUS actions from 2017 to 2022, merging them and removing duplicates. For reference to public video data, we leveraged datasets such as Kinetics [7], SomethingSomething[8,9], UCF101 [10], etc. This provided us with 1103 action labels. Additionally, we accessed several grounding datasets. We used language models to extract actions and their corresponding objects (where applicable) from the corpora, forming phrases, of which 5001 actions passed manual inspection. In total, we collected 6104 video query terms for searching videos online.

Collection Strategy. To ensure the quality of our dataset, we established specific crawling rules. We only collected videos between 10 seconds and 30 minutes, with resolutions ranging from 360P to 720P. In this process, we prioritized the highest available resolution. We collected videos along with their audio, captions, and titles to provide a comprehensive multimodal dataset.

Multi-scale Video Caption Generation. To generate scalable, rich, and diverse video captions, we employed a multi-scale approach encompassing two distinct captioning strategies, as illustrated in Figure 4. At a finer scale, we streamlined the video captioning process by concentrating on common objects, actions, and scenes within video clips. We intentionally overlooked intricate details, such as nuanced facial expressions, movements, and other subtle elements. At a coarser scale, captions were crafted solely for the central frames of the videos. Considering our focus on short video clips filtered through scene segmentation (around 10 seconds), most between 10 seconds and 30 minutes videos predominantly feature consistent objects without significant visual changes. This approach circumvents the need to address identity preservation issues from an image perspective when handling videos. Technically, we utilized the lightweight image captioning model Tag2Text [11] for finer-scale captioning, which describes videos on a per-frame basis at low frames per second (fps). For the coarser scale, we leveraged BLIP2 [12] to caption the central frames of the clips. These individual frame captions were then aggregated into an integrated video caption using pre-trained language models [13,14].

III. Multimodal Representation Model ViCLIP

Based on InternVid and CLIP[15], we propose ViCLIP, a video-text contrastive learning model based on ViT-L. This model adopts video occlusion and contrastive loss methods to learn transferable video-text representations, as shown in Figure 5. By introducing DeepSpeed and FlashAttention, ViCLIP was trained for 3 days on 64 NVIDIA A100 GPUs. Most.

We train ViCLIP on five subsets of InternVid and evaluate its performance on multiple popular video-related benchmarks, including zero-shot and thoroughly fine-tuned settings. These five subsets are randomly sampled InternVid-10M, InternVid-50M, InternVid-200M, InternVid-10M-DIV, which assumes that the diversity of training video clips is more critical than the quantity, and InternVid-10M-FLT which assumes that text-video match is more important. We compare InternVid-10M and InternVid-10M-DIV/-FLT with WebVid10M while using InternVid-50M and InternVid-200M to further validate the indicative data of video-language contrastive learning. The specific performance tables can be referred to in the original text, and Figure 6 lists the performance of the retrieval task.

IV. Exploration of Text-to-Video Generation

Additionally, we provide a subset filtered for specific video-text relationships and visual aesthetics, InternVid-Aes, which aids in generating high-resolution, watermark-free videos. Using InternVid-Aes can significantly enhance the visual and quantitative results of a simple text-to-video baseline model. These resources offer a powerful tool for researchers and practitioners interested in multimodal video understanding and generation.

To quantitatively evaluate our model, we conducted zero-shot text-to-video experiments. We randomly selected2020 videos from the UCF-101 dataset and 2990 videos from the MSRVTT dataset, testing the CLIPSIM, IS, FID, and ViCLIP shows significant performance improvements in zero-shot settings compared to previous Video CLIP variants and FVD metrics in Figure 8.

IV. Download

The InternVid dataset can be accessed through the following:

🤗 Hugging Face 🤗

• Dataset meta info: https://huggingface.co/datasets/OpenGVLab/InternVid
• ViCLIP weights: https://huggingface.co/OpenGVLab/ViCLIP

Currently, on HuggingFace, the InternVid-10M-FLT (default), InternVid-10M-DIV, and InternVid-18M-Aes have been open-sourced. By selecting different subsets, you can preview them, which provide video IDs, start and end timestamps, video captions, aesthetic scores, and video-text match scores. You can download them in conjunction with yt-dlp or video2dataset.

To download the metafile, you can also use the following command (please log in to the website to obtain permissions first):

pip install -U huggingface_hub
hugging face-cli download --repo-type dataset --token hf_***(your_huggingface_token) --
resume-download --local-dir-use-symlinks False OpenGVLab/InternVid --local-dir 
InternVid_metainfo

We also provide information covering the source languages, which you can download at https://huggingface.co/datasets/OpenGVLab/InternVid-10M-FLT-INFO.

Github:

• Dataset information and Model Zoo: https://github.com/OpenGVLab/InternVideo/tree/main/Data/InternVid
• ViCLIP training code: https://github.com/OpenGVLab/ViCLIP

Citation

The release of InternVid will stimulate research and development in the field of video understanding and generation. In the future, the OpenGVLab team will continue to update and refine InternVid, and is committed to providing researchers and developers with higher quality datasets and tools.

If you have used the InternVid dataset or ViCLIP in your research or applications, we welcome you to cite the following references:

@inproceedings{wang2023internvid,
  title={InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation},
  author={Wang, Yi and He, Yinan and Li, Yizhuo and Li, Kunchang and Yu, Jiashuo and Ma, Xin and Chen, Xinyuan and Wang, Yaohui and Luo, Ping and Liu, Ziwei and Wang, Yali and Wang, Limin and Qiao, Yu},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2023}
}

Contact us

If you have any questions or suggestions regarding the InternVid dataset, please feel free to contact the OpenGVLab-InternVideo team:

• Email: gvx-sh@pjlab.org.cn

For any issues or inquiries, the team can provide further assistance and support. If you encounter any difficulties with the dataset or have ideas for improvements, don’t hesitate to reach out. video understanding and generation. In the future, the OpenGVLab team will continue to update and refine InternVid and is committed to providing researchers and developers with higher-quality Your feedback is valuable for the continuous development and enhancement of the dataset.

Reference

[1] Miech A, Zhukov D, Alayrac J B, et al. Howto100m: Learning a text-video embedding by watching a hundred million narrated video clips[C]//ICCV 2019: 2630–2640.

[2] Xue H, Hang T, Zeng Y, et al. Advancing high-resolution video-language representation with large-scale video transcriptions[C]//CVPR 2022: 5036–5045.

[3] Zellers R, Lu X, Hessel J, et al. Merlot: Multimodal neural script knowledge models[J]. NeurIPS 2021, 34: 23634–23651.

[4] Zellers R, Lu J, Lu X, et al. Merlot reserve: Neural script knowledge through vision and language and sound[C]//CVPR 2022: 16375–16387.

[5] Bain M, Nagrani A, Varol G, et al. Frozen in time: A joint video and image encoder for end-to-end retrieval[C]//ICCV 2021: 1728–1738.

[6] https://www.bls.gov/tus/

[7] Carreira J, Zisserman A. Quo vadis, action recognition? a new model and the kinetics dataset[C]//CVPR 2017: 6299–6308.

[8] Goyal R, Ebrahimi Kahou S, Michalski V, et al. The” something something” video database for learning and evaluating visual common sense[C]//ICCV 2017: 5842–5850.

[9] Mahdisoltani F, Berger G, Gharbieh W, et al. On the effectiveness of task granularity for transfer learning[J]. arXiv preprint arXiv:1804.09235, 2018.

[10] Soomro K, Zamir A R, Shah M. UCF101: A dataset of 101 human actions classes from videos in the wild[J]. arXiv preprint arXiv:1212.0402, 2012.

[11] Huang X, Zhang Y, Ma J, et al. Tag2text: Guiding vision-language model via image tagging[J]. arXiv preprint arXiv:2303.05657, 2023.

[12] Zheng L, Chiang W L, Sheng Y, et al. Judging llm-as-a-judge with mt-bench and chatbot arena[J]. NeurIPS 2024, 36.

[13] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//ICML 2023: 19730–19742.

[14] Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. Journal of machine learning research, 2020, 21(140): 1–67.

[15] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//ICML 2021: 8748–8763.

InternVid: Video-Text Dataset to Empowering Video Creation and Understanding was originally published in AI Advances on Medium, where people are continuing the conversation by highlighting and responding to this story.

VideoMamba is a pioneering SSM-based model for video understanding🎥:
Visual Domain Scalability;
Short-term Action Sensitivity;
Long-term Video Superiority;
Modality Compatibility.

Motivation

The core objective for video understanding lies in mastering spatiotemporal representations, which inherently presents two formidable challenges: the large spatiotemporal redundancy within short video clips and the complex spatiotemporal dependencies among long contexts. The once-dominant 3D CNNs and video transformers, which use convolution and self-attention mechanisms, addressed these two issues. In our previous work, UniFormer[1], we attempted to combine convolution and self-attention seamlessly. It could simultaneously address both problems but fell short for long videos. The popularity of Gemini[2] and Sora[3] shifted the research focus towards understanding and generating long videos, urgently demanding more efficient representation learning.

Fortunately, the NLP field has seen the emergence of several efficient operators in the past two years, such as S4[4], RWKV[5], and RetNet[6]. Mamba[7] stands out with its selective state space model (S6), which is capable of long-term dynamic modeling with linear complexity. This led to the adaptation of a series of vision tasks, such as Vision Mamba[8] and VMamba[9], proposing a multi-directional SSM mechanism for processing 2D images. These models compete with attention-based architectures and significantly reduce memory consumption.

Given videos’ inherently longer token sequences, the question arises: Is Mambaequally effective for video understanding? The answer is yes.

We proposed an efficient video understanding architecture, VideoMamba, which is solely based on the State Space Model (SSM), and conducted extensive experiments to show its excellent properties, including (1) Visual Domain Scalability, (2) Short-term Action Sensitivity, (3) Long-term Video Superiority; (4) Modality Compatibility. These enable VideoMamba to achieve remarkable results across a range of video benchmarks, especially in long-video benchmarks, providing an efficient solution for comprehensive video understanding in the future.

Paper: https://arxiv.org/abs/2403.06977

Code: https://github.com/OpenGVLab/VideoMamba

Model: https://huggingface.co/OpenGVLab/VideoMamba

Online Demo: https://huggingface.co/spaces/OpenGVLab/VideoMamba

Methods

Architecture

Before formally introducing VideoMamba’s structure, let’s first examine the Mamba block for 1D sequences and the bidirectional Mamba block for vision tasks. We will not delve into the underlying principles of SSM and Mamba here, but those who are interested can learn through YouTube videos at https://www.youtube.com/watch?v=8Q_tqwpTpVU.

The bidirectional Mamba (B-Mamba) block builds upon the normal Mamba by introducing the extra SSM for the reverse sequence, enabling better modeling of 2D sequences and thus enhancing the perceptual capability for visual inputs. Based on the B-Mamba block, our VideoMamba follows the design of ViT[10], introducing the [CLS] token and spatial position encoding. Specifically for video modeling, we utilize 3D patch embedding and spatial position encoding.

To apply the B-Mamba block for spatiotemporal tokens, we expand the original 2D scanning to different bidirectional 3D scanning methods:

Among them, the spatial-first scanning is the simplest and the most effective. Based on this, we propose three model sizes: VideoMamba-Ti, VideoMamba-S, and VideoMamba-M.

In our experiments, increasing the size of VideoMamba’s model is prone to overfitting, resulting in the larger models performing even worse than the smaller ones. To address this, we proposed a Self-Distillation strategy, using the pretrained smaller model as a teacher to guide the training of the larger model, effectively avoiding model overfitting with only a minimal additional cost.

Masked Modeling

Recently, VideoMAE[11] introduced masked modeling, significantly enhancing the model’s capacity for fine-grained temporal understanding. UMT[12] further proposes an efficient mask alignment strategy, significantly reducing training costs and enabling the model to handle various single-modality and multi-modality tasks robustly. To enhance VideoMamba’s temporal sensitivity and verify its compatibility with the text modality, we follow UMT to introduce CLIP-ViT as a teacher for two-stage training.

Unlike the multi-layer alignment, due to architectural differences between VideoMamba and ViT, we only align the last layer of the models. Considering the Mamba block’s preference for continuous tokens, we design the row masking strategies:

Additionally, we explore attention masking to preserve meaningful adjacency among tokens, leveraging the inherent strengths of the 1D convolution within the B-Mamba block for improved performance.

Experiments

Scale Up

We first conduct experiments on ImageNet as follows:

It is evident that without Self-Distillation (SD), both VideoMamba-M and VideoMamba-B tend to overfit towards the end of the training, with VideoMamba-B experiencing particularly severe overfitting. However, with SD, VideoMamba-M converges as expected and is notably superior to the teacher model, VideoMamba-S. To prevent the potential over-guidance of the teacher model, we introduce an Early Stop strategy, i.e., removing the distillation guidance, but experiments still need improvement. The complete ImageNet comparison is as follows:

VideoMamba outperforms the convolution-based ConvNeXt and attention-based ViT compared to other isotropic models. As the model scale and resolution increase, performance consistently improves.

Short-term Video Understanding

On K400 and SthSthV2, we also observe VideoMamba’s excellent scalability, significantly outperforming attention-based video models such as TimeSformer and ViViT and performing comparably to UniFormer, which combines convolution and self-attention. Furthermore, VideoMamba’s performance significantly improves after introducing masked training, notably surpassing the ViT-based UMT on the fine-grained action dataset SthSthV2.

Further ablation experiments show that the spatial-first scanning approach yields the best results. Unlike on ImageNet, where performance gradually increases with resolution, on video datasets, the resolution has a limited impact on performance, whereas the frame number significantly affects it. For masked modeling, row masking is superior to random masking strategies, and attention masking strategies are the most effective; aligning the last layer yields the best results; appropriate masking ratios and Droppath rate can significantly enhance training effectiveness.

Long-term Video Understanding

We evaluate VideoMamba’s understanding of long-duration videos on the Breakfast, COIN, and LVU datasets. Compared to previous feature-based methods, VideoMamba requires only 32 to 64 sparse frames to achieve significantly superior performance, while the model size is smaller.

Multi-modality Video Understanding

We connected VideoMamba with BERT to construct a multimodal model, used large-scale multimodal data for pre-training, and evaluated its performance on several video-text retrieval tasks. Experiments reveal that VideoMamba can also serve as a multimodal visual encoder. Larger pre-training data can continuously improve multimodal capacity. Compared with UMT based on ViT, VideoMamba excels on datasets containing long videos (ANet and DiDeMo) and more complex scenarios (LSMDC).

Conclusion

We propose VideoMamba for efficient video understanding based solely on the stateSpace Model. Comprehensive experiments have shown that VideoMamba possesses a series of excellent characteristics for video understanding. We hope it can pave the way for future representation learning of long videos.

Paper: https://arxiv.org/abs/2403.06977

Code: https://github.com/OpenGVLab/VideoMamba

Model: https://huggingface.co/OpenGVLab/VideoMamba

Online Demo: https://huggingface.co/spaces/OpenGVLab/VideoMamba

Welcome to follow us on Twitter: https://twitter.com/opengvlab

References

[1] UniFormer: https://github.com/Sense-X/UniFormer

[2] Gemini: https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/

[3] Sora: https://openai.com/sora

[4] S4: https://github.com/state-spaces/s4

[5] RWKV: https://www.rwkv.com/

[6] RetNet: https://github.com/microsoft/unilm/tree/master/retnet

[7] Mamba: https://github.com/state-spaces/mamba

[8] Vision Mamba: https://github.com/hustvl/Vim

[9] VMamba: https://github.com/MzeroMiko/VMamba

[10] ViT: https://github.com/google-research/vision_transformer

[11] VideoMAE: https://github.com/MCG-NJU/VideoMAE

[12] UMT: https://github.com/OpenGVLab/unmasked_teacher

VideoMamba: State Space Model for Efficient Video Understanding was originally published in AI Advances on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to follow us on Twitter: https://twitter.com/opengvlab

The study of artificial general intelligence (AGI) systems that can match human intelligence and excel in any task across domains represents the ultimate goal in the field of artificial intelligence. Benefiting from the advancements of Large Language Models (LLMs), Multi-modal Large Language Models (MLLMs) have demonstrated impressive capabilities in a variety of Vision-Language tasks, suggesting new avenues for achieving AGI. However, most popular MLLMs are limited to understanding the image as a whole instead of a specific region or instance in it.

As an effective method to improve interaction efficiency, the capabilities of grounding and referring (i.e., adopting bounding boxes in responses) have attracted increasing attention and have been widely integrated into current Grounded MLLMs. Such capabilities empower users and models to directly point to objects or image regions without the necessity of elaborate textual descriptions for reference and enable the model to provide visual responses (e.g., bounding boxes), supporting more vision-language tasks such as region captioning, referring expression comprehension, and referring question answering. However, without the support of large-scale instance-level visual understanding data, the generalization ability of these models is still limited. Besides, due to the lack of appropriate modeling methods and suitable training data for relation knowledge, these models also struggle to comprehend the inter-object relations within images accurately.

To address these challenges, we propose the All-Seeing project. From the data aspect, this project proposes (1) the All-Seeing Dataset (AS-1B), which is constructed for pretraining and consists of over 1.2 billion region annotations in various formats, such as semantic tags, question-answering pairs, and detailed captions; (2) the All-Seeing Dataset v2 (AS-V2), which is constructed for instruction tuning and consists of 127K high-quality samples, focusing on relation comprehension.

In terms of the model perspective, we propose (1) the All-Seeing Model (ASM), a unified location-aware image-text foundation model supporting both image-text retrieval and generation tasks, demonstrating impressive zero-shot capability. (2) the All-Seeing Model v2 (ASMv2), which unfiies the formulation of text generation, object localization, and relation comprehension, demonstrating powerful performance across various tasks, including Open-ended Scene Graph Generation and other general image-level and region-level vision-language tasks..

Notably, ASMv2 achieves SoTA performance on a total of 20 multimodal benchmarks, including 13 image-level benchmarks and 7 region-level benchmarks. In addition, benefiting from the ability to ground relations between objects, ASMv2 can also be applied to the Scene Graph Generation task, on which it demonstrates competitive performance.

Paper：https://arxiv.org/abs/2308.01907

Code：https://github.com/OpenGVLab/all-seeing/tree/main/all-seeing

Huggingface：https://huggingface.co/collections/OpenGVLab/all-seeing-project-65e19865ffaa2b2de9161c8c

Paper：https://arxiv.org/abs/2402.19474

Code：https://github.com/OpenGVLab/all-seeing/tree/main/all-seeing-v2

Huggingface：https://huggingface.co/collections/OpenGVLab/all-seeing-project-65e19865ffaa2b2de9161c8c

Dataset

All-Seeing Dataset

AS-1B is the first large-scale region-level vision-language dataset, consisting of over 1 billion region-text pairs, 3.5 million open-world concepts, and 100 billion tokens of region-related question-answering and caption. Each region is annotated with comprehensive information, including semantic tags, question-answer pairs, and captions. Compared with the previous visual recognition datasets like ImageNet and COCO, visual understanding datasets like Visual Genome and Laion-5B, the proposed AS-1B dataset stands out due to its rich and diverse instance-level annotation and corresponding detailed object concepts and descriptions.

AS-1B is constructed based on a semi-automatic data engine, which greatly reduces the annotation cost.

Specifically, the process of the data engine begins by generating noisy pseudo data using well-trained off-the-shelf foundation models from diverse fields. Subsequently, a subset of these pseudo data are sampled to be verified and corrected by human annotators. After that, we pre-train our All-Seeing-Model (ASM) with the noisy pseudo data and finetune it with the human-verified data. Then we re-annotate the data with the aid of ASM. The process of annotation, verification, and fine-tuning are repeated to iteratively refine the annotation quality. By employing this ``data-human-model’’ cycle, we can generate a large number of region-level annotations with exceptional quality.

The fully verified subset of AS-1B, termed AS-Core, is released separately.

All-Seeing Dataset V2

We propose a novel task, termed Relation Conversation (ReC), which unifies the formulation of text generation, object localization, and relation comprehension. Based on the unified formulation, we construct the AS-V2 dataset, which consists of 127K high-quality relation conversation samples, to unlock the ReC capability for Multi-modal Large Language Models (MLLMs).

As depicted in the Figure below, ReC requires the model to generate the text response while linking all mentioned objects as well as the subjects and objects of each predicate in the response to the corresponding regions in the image simultaneously.

We utilize GPT-4V to construct AS-V2 based on COCO images and their caption, location, and relation annotations. The key idea is to query GPT-4V to generate responses while linking the objects and predicates mentioned in the generated response to specific regions within the image, referring to the given location annotations and relation annotations.

Circular-based Relation Probing Evaluation

To evaluate the relation comprehension ability of existing MLLMs, we construct a benchmark called Circular-based Relation Probing Evaluation (CRPE), which is the first benchmark that covers all elements of the relation triplets (subject, predicate, object), providing a systematic platform for the evaluation of relation comprehension ability.

CRPE is formulated as single-choice questions and consists of four splits: Existence, Subject, Predicate, and Object. The Existence split evaluates the object recognition ability while the remaining splits are designed to evaluate the relation comprehension capability.

Additionally, to evaluate the dependency on language priors, we include abnormal data in CRPE, which depict relations that are rare but reasonable in the real world.

Model

All-Seeing Model

Our All-Seeing Model (ASM) is a unified location-aware image-text foundation model, which supports both generative tasks (e.g., Image Captioning and Region Captioning) and discriminative tasks (e.g., Region Recognition) and demonstrates powerful performance on these tasks.

This model comprises three key designs:

(1) a location-aware image tokenizer extracting features from both the image and region levels based on the input image and bounding box, respectively.

(2) a trainable task prompt that is incorporated at the beginning of the vision and text tokens to guide the model in distinguishing between discriminative and generative tasks.

(3) a LLM-based decoder that is utilized to extract vision and text features for discriminative tasks, as well as to auto-regressively generate response tokens for generative tasks.

In the case of the discriminative task, a trainable align token is appended to the input sequence to gather the overall representation, and its embedding is then used in the matching process.

All-Seeing Model v2

All-Seeing Model v2 (ASMv2) is a powerful Multi-modal Large Language Model (MLLM), which integrates the Relation Conversation (ReC) ability while maintaining powerful general capabilities. This model can deal with three types of tasks, including (1) Relation Conversation, which requires the model to link all objects and predicates mentioned in the response to the corresponding regions in the image; (2) Open-ended Scene Graph Generation, which requires the model to generate a scene graph based on the given image in an open-ended manner; (3) General Multimodal Conversation, which requires the model to generate a text response given the image and user query. Compared to ASM, ASMv2 places more emphasis on the recognization and comprehension of objects and the relations between them within images during multimodal dialogue processes.

ASMv2 follows the model architecture of LLaVA, comprising a vision encoder (CLIP-L-336), a vision-language connector (2-layer MLP), and a language model (Vicuna-13B). Unlike ASM, ASMv2 directly understands regions in images based on the bounding boxes provided in the text, eliminating the need for additional RoI Align operations to assist in understanding region information.

Specifically, our relation conversation marks the object and predicate in the sentence using <ref></ref> and <pred></pred>, respectively. Each marked object is followed by a bounding box, indicating its localization. Similarly, each predicate is followed by two bounding boxes, which specifically refer to the subjects and objects of the predicate. All bounding boxes are normalized to integer values within the range [0, 1000) and formatted as <box>[[x1, y1, x2, y2]]</box>. Notably, models trained on ReC can be naturally adapted to the Scene Graph Generation task. The grounded objects serve as the nodes in the scene graph while the grounded predicates serve as the edges.

Experiment

The experimental results show that ASMv2 achieves state-of-the-art performance on a total of 20 multimodal benchmarks, including 13 image-level benchmarks and 7 region-level benchmarks. Specifically, ASMv2 outperforms LLaVA-1.5 by 6.7 points on MMBench and improves by 89.7 points on MME.

On region-level benchmarks, ASMv2 demonstrates an average improvement of approximately 1.7 points over Qwen-VL in the Referring Expression Comprehension task. Additionally, ASMv2 achieves state-of-the-art performance on Region Captioning and Referring Question Answering tasks.

ASMv2 also exhibits outstanding performance in the Scene Graph Generation task. Notably, compared to traditional scene graph generation models, ASMv2 not only generates scene graphs in an open-ended manner, thereby having the potential to generalize to previously unseen predicate labels, but also maintains the general multimodal capabilities of MLLMs, thus broadening the applicability in real-world scenarios.

On the CRPE benchmark, ASMv2 also demonstrates state-of-the-art performance. For example, ASMv2 achieves an overall accuracy of 52.04, which is significantly higher than 43.14 of LLaVA-1.5 and 27.27 of Qwen-VL. These results demonstrate that ASMv2 can comprehend the relations between the objects within the image better, benefiting from the training of relation conversation data.

Paper：https://arxiv.org/abs/2308.01907

Code：https://github.com/OpenGVLab/all-seeing

Welcome to follow us on Twitter! https://twitter.com/opengvlab

The All-Seeing Project: Towards Panoptic Visual Recognization and General Relation Comprehension… was originally published in AI Advances on Medium, where people are continuing the conversation by highlighting and responding to this story.

Existing post-training quantization algorithms for large language models rely on manually set quantization parameters and lack a gradient optimization process, leading to significant performance degradation when facing low-bit quantization. Although quantization-aware training is effective in determining the best quantization parameters, it requires a large amount of additional training overhead and data. The computational burden of large language models themselves further hinders the application of quantization-aware training in the quantization of large language models. This raises the question: Can we achieve the performance of quantization-aware training while maintaining the training time and data efficiency of post-training quantization?

To address the optimization problem of quantization parameters in post-training quantization of large language models, researchers from Shanghai AI Lab, The University of Hong Kong, and The Chinese University of Hong Kong have proposed OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models. This algorithm simultaneously supports the quantization of weights and activations in large language models and covers various quantization bit settings.

arXiv Paper：https://arxiv.org/abs/2308.13137

OpenReview Paper：https://openreview.net/forum?id=8Wuvhh0LYW

Code：https://github.com/OpenGVLab/OmniQuant

Overall Framework

As shown in the figure above, OmniQuant is a differentiable quantization technology for large language models (LLM) that supports both weight and activation quantization. It achieves high-performance quantized models while maintaining the efficiency of training time and data in post-training quantization. For example, OmniQuant can complete the update of quantization parameters for LLaMA-7B to LLaMA70B models on a single A100–40GB GPU within 1–16 hours. To achieve this goal, OmniQuant adopts a Block-wise quantization error minimization framework. At the same time, OmniQuant designs two novel strategies to increase learnable quantization parameters, including learnable weight clipping (LWC) to reduce the difficulty of quantizing weights, and a learnable equivalent transformation (LET) to further shift the challenge of quantization from activations to weights. Additionally, all learnable parameters introduced by OmniQuant can be fused and eliminated after quantization, allowing the quantized model to be deployed on multiple platforms, including GPUs, Android, iOS, etc., based on existing tools.

Block-wise Quantization Error Minimization

OmniQuant proposes a new optimization process that uses Block-wise quantization error minimization and optimizes additional quantization parameters in a differentiable manner. The optimization objective is formulated as follows:

where F represents the mapping function of a transformer block in LLM, W and X are the full-precision weights and activations, respectively, QW(·) and Qa(·) represent the weight and activation quantizers, respectively, and $\Theta_1$ and $\Theta_2$ are the quantization parameters in learnable weight clipping (LWC) and learnable equivalent transformation (LET). OmniQuant quantizes the parameters in a transformer block sequentially and then moves to the next.

Learnable Weight Clipping (LWC)

Equivalent transformation migrates the scale between model weights and activations. The learnable equivalent transformation used in OmniQuant makes the model weight distribution change continuously during the parameter optimization process. Previous methods that directly learn weight clipping thresholds [1,2] are only suitable for situations where the weight distribution does not change drastically, otherwise, they will struggle to converge. Based on this issue, unlike previous methods that directly learn weight clipping thresholds, LWC optimizes the clipping strength in the following way:

where [·] denotes the floor operation. N is the target bit-width. Wq and W represent the quantized and full-precision weights, respectively. h is the normalization factor for weights, and z is the zero-point value. The clipping (clamp) operation limits the quantized values within the range of N-bit integers, i.e., [N-1]. In the above formula, $\gamma\in[0,1]$ and $\beta\in[0,1]$ are the learnable clipping strengths for the upper and lower bounds of weights, respectively. Thus, in the optimization objective function,

Learnable Equivalent Transformation (LET)

In addition to LWC, which optimizes weights for quantization through the clipping threshold, OmniQuant further reduces the difficulty of activating quantization through LET. Considering that outliers in LLM activations exist in specific channels, previous methods such as SmoothQuant[3], Outlier Suppression+[4] shift the difficulty of quantization from activations to weights through mathematical equivalent transformations. However, manually chosen or greedily searched equivalent transformation parameters limit the performance of quantized models. Thanks to the introduction of Block-wise quantization error minimization, OmniQuant’s LET can determine the optimal equivalent transformation parameters in a differentiable manner. Inspired by Outlier Suppression+~\citep{outlier-plus}, it adopts channel-level scaling and channel-level shifting to manipulate the activation distribution, providing an effective solution for the outlier problem in activations. Specifically, OmniQuant explores equivalent transformations in linear layers and attention operations.

Equivalent Transformation in Linear Layers:

The linear layer accepts an input token sequence$\mathbf{X} \in \mathbb{R}^{T\times C_{in}}$, where T is the token length, and computes the product of the weight matrix $\mathbf{W} \in \mathbb{R}^{C_{in} \times C_{out}}$and the bias vector $\mathbf{B}\in\mathbb{R}^{1 \times C_{out}}$ . The mathematically equivalent linear layer expression is:

Here, Y represents the output, $\mathbf{s} \in \mathbb{R} ^{1\times C_{in}}$and $\mathbf{\delta} \in \mathbb{R}^{1\times C_{in}}$ are the channel-level scaling and shifting parameters, respectively, and X W B are the equivalent activations, weights, and biases, respectively, with $\oslash$ and ⊙ representing element-wise division and multiplication. Through this equivalent transformation, activations are converted into a form more amenable to quantization, at the cost of increasing the difficulty of weight quantization. In this sense, LWC can improve the model quantization performance achieved by LET because it makes weights more quantizable. Finally, OmniQuant quantizes the transformed activations and weights as follows:

where Qa is a standard MinMax quantizer, and Qw is a MinMax quantizer with learnable weight clipping (i.e., the proposed LWC).

Equivalent Transformation in Attention Operations:

In addition to linear layers, attention operations also account for a significant portion of LLM computations. Moreover, the autoregressive inference mode of LLMs requires storing key-value (KV) caches for each token, leading to substantial memory demands for long sequences. Therefore, OmniQuant also considers quantizing the Q/K/V matrices in self-attention computations to low bits. Specifically, the learnable equivalent transformation in the self-attention matrix can be written as:

where $s_a\in \mathbb{R}^{1\times C_{out}}$is the scaling factor. The quantized computation in the self-attention is expressed as P=(Qa(Q)Qa(KT)). OmniQuant also uses the MinMax quantization scheme as Qa to quantize the Q K matrices. Thus, the final optimization objective function includes $\Theta_2={\delta,s,s_a}$.

## Pseudocode

Experiments

It can be seen that OmniQuant consistently outperforms previous LLM weight-only quantization methods across various LLM models (OPT, LLaMA-1, LLaMA-2) and diverse quantization configurations (including W2A16, W2A16g128, W2A16g64, W3A16, W3A16g128, W4A16, and W4A16g128). These experiments demonstrate the versatility of OmniQuant, which can adapt to various quantization configurations. For example, although AWQ [5] is particularly effective for group-wise quantization, OmniQuant shows superior performance in both channel-wise and group-wise quantization. Moreover, as the number of quantization bits decreases, the performance advantage of OmniQuant becomes more pronounced.

In the settings of weight-activation quantziation, the experiments mainly focus on W6A6 and W4A4 quantization. The W8A8 quantization is excluded from the experimental setup because the previous SmoothQuant almost achieves lossless W8A8 model quantization compared to the full-precision model. The figure above shows the experimental results of OmniQuant on the LLaMA model with both weights and activations quantized. Notably, in the W4A4 quantization for different models, OmniQuant significantly improves the average accuracy, with an increase ranging from +4.99% to +11.80%. Especially in the LLaMA-7B model, OmniQuant even surpasses the recent quantization-aware training method LLM-QAT [6] with a significant gap of +6.22%. This improvement proves the effectiveness of introducing additional learnable parameters, which is more beneficial than the global weight adjustments used in quantization-aware training.

Furthermore, models quantized using OmniQuant can be seamlessly deployed on MLC-LLM [7]. The figure above shows the memory requirements and inference speeds of the LLaMA series quantized models on the NVIDIA A100–80G. Weights Memory (WM) represents the storage of quantized weights, while Running Memory (RM) indicates the memory during inference, which is higher due to retaining some activations. Inference speed is measured by generating 512 tokens. It is evident that compared to the 16-bit full-precision model, the quantized models significantly reduce memory usage. Moreover, W4A16g128 and W2A16g128 quantization almost double the inference speed.

It is worth noting that MLC-LLM [7] also supports the deployment of OmniQuant-quantized models on other platforms, including Android and iOS phones. As shown in the figure, the recent application Private LLM utilizes the OmniQuant algorithm to achieve memory-efficient deployment of LLMs on iPhones, iPads, macOS, and other platforms.

Conclusion

OmniQuant is an advanced algorithm for large language model quantization that pushes quantization down to low-bit formats. The core principle of OmniQuant is to retain the original full-precision weights while adding learnable quantization parameters. It optimizes the quantization compatibility of weights and activations using learnable weight clipping and equivalent transformations. By fusing gradient updates, OmniQuant maintains training time efficiency and data efficiency comparable to existing PTQ methods. Additionally, OmniQuant ensures hardware compatibility, as its added trainable parameters can be fused into the original model without any extra overhead.

Reference

[1] Pact: Parameterized clipping activation for quantized neural networks.

[2] LSQ: Learned step size quantization.

[3] Smoothquant: Accurate and efficient post-training quantization for large language models.

[4] Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling.

[5] Awq: Activation-aware weight quantization for llm compression and acceleration.

[6] Llm-qat: Data-free quantization aware training for large language models.

[7]MLC-LLM：https://github.com/mlc-ai/mlc-llm