分类： 未分类

The Evolution and Evaluation of Large Multimodal Models: A Critical Examination of Current Benchmarks
Artificial intelligence has entered a new era with the advent of Large Multimodal Models (LMMs), which combine language and vision processing to create Visual Foundation Agents. These agents represent a leap toward general artificial intelligence, capable of performing diverse tasks that were once the exclusive domain of humans. Yet, despite their promise, the benchmarks used to evaluate these models remain inadequate, failing to fully test their capabilities in complex, real-world scenarios. This article explores the current state of LMMs, the limitations of existing benchmarks, and the path forward for more rigorous evaluation frameworks.

The Rise of Visual Foundation Agents

LMMs have emerged as a transformative force in AI, integrating text and visual data to perform tasks ranging from image recognition to interactive decision-making. Unlike traditional models that specialize in a single domain, these agents are designed to generalize across multiple applications, mimicking human-like understanding. However, their rapid development has outpaced the tools used to measure their performance.
Current benchmarks often focus on narrow tasks, such as image classification or text generation, without accounting for the dynamic, multifaceted environments these models must navigate. For instance, while a model might excel at identifying objects in static images, it may struggle when asked to interpret a sequence of actions in a video or make real-time decisions in a simulated world. This gap between capability and evaluation highlights the need for more comprehensive testing frameworks.

The Shortcomings of Existing Benchmarks

1. Limited Scope and Real-World Relevance

Most benchmarks fail to replicate the unpredictability of real-world scenarios. Tasks are often simplified, with clean datasets and controlled conditions that don’t reflect the noise and complexity of actual applications. For example, a model trained to recognize household items in curated images might falter when confronted with cluttered, poorly lit environments.
The introduction of VisualAgentBench (VAB) by THUDM represents a step toward addressing this issue. VAB includes five diverse environments—VAB-OmniGibson, VAB-Minecraft, VAB-Mobile, VAB-WebArena-Lite, and VAB-CSS—each designed to test different aspects of LMM performance. From navigating virtual worlds to completing web-based tasks, these environments provide a more holistic assessment. Yet, even VAB has limitations, as it still operates within simulated settings rather than real-world deployments.

2. The Role of Statistical Features in Visual Tasks

A critical factor in LMM performance is their ability to process high-level statistical features in visual data. Research by Morgenstern and Hansmann-Roth demonstrates that models relying on these features achieve greater accuracy in tasks like shape recognition and object detection. For instance, the feature distribution learning method enables models to distinguish between relevant and irrelevant visual cues, improving their robustness.
However, current benchmarks often overlook this aspect, focusing instead on raw accuracy rather than the underlying mechanisms of visual interpretation. Future benchmarks should incorporate tests that evaluate how well models learn and utilize statistical features, ensuring they can adapt to varied and ambiguous inputs.

3. Performance Metrics and the Gap Between Benchmarks and Reality

While LMMs show promise in controlled evaluations, their real-world performance remains inconsistent. In 2025, OpenAI’s gpt-40-2024-05-13 achieved a 36.2% success rate in the VAB benchmark, with GPT-4 vision preview trailing at 31.7%. These figures, while impressive, pale in comparison to the 41.98% success rate of fully funded Kickstarter projects—a metric that reflects real-world decision-making and adaptability.
This discrepancy underscores the need for benchmarks that better simulate practical applications. For example, testing models in dynamic environments where they must interact with humans or adapt to unforeseen challenges would provide a more accurate measure of their readiness for deployment.

The Future of LMM Evaluation

The software market is projected to grow by 4.87% annually, reaching $896.20 billion by 2029, driven by innovations in AI and automation. This growth necessitates advanced testing methodologies to ensure reliability and safety. The software testing market, valued at over $51.8 billion in 2023, is expected to expand at a 7% CAGR, reflecting the increasing demand for robust evaluation tools.
To keep pace with LMM development, benchmarks must evolve in three key ways: