The Visual Perception Challenge: Why Top Multimodal Models Are Still Failing

The Kimi team recently open-sourced PerceptionBench, a new benchmark designed to stress-test the core visual perception skills of multimodal AI models. Moving beyond holistic evaluation, this benchmark adopts a dissective approach, examining foundational capabilities in isolation.

The Test: Ten Atomic Skills, Zero Reasoning Required

PerceptionBench avoids complex, multi-step problems. Instead, it isolates ten atomic visual capabilities for individual assessment:

  • Visual Relations: Understanding spatial and action relationships between objects.
  • Counting: Accurately enumerating specific items in an image.
  • Attribute Recognition: Identifying color, shape, material, etc.
  • Depth & 3D Understanding: Perceiving spatial layout and three-dimensionality.
  • Localization: Pinpointing the location of elements within an image.
  • Comparison : Discriminating similarities and differences between visual elements.
  • Fine-Grained Recognition: Distinguishing subtle sub-category differences (e.g., dog breeds).
  • Context Integration: Using global scene context to interpret local details.
  • OCR: Reading text within images.
  • Hallucination Identification: Detecting if a model "sees" non-existent content.

Its 3,000 questions are curated from model failure cases across 42 existing datasets, each manually verified to test only one core skill, eliminating the need for external knowledge or chain-of-thought reasoning.

The Results: A Universal Struggle, Hallucinations Prevail

The benchmark evaluated 16 state-of-the-art multimodal models, revealing a stark reality: not a single model achieved an overall accuracy above 60%. Even the most advanced systems show unreliable performance on basic perceptual tasks.

GPT-5.6-Sol topped the list with 59.7% accuracy. It was closely followed by Kimi K3 (58.5%), Claude-Fable-5 (57.2%), Gemini-3.1-Pro (56.2%), and GPT-5.5 (55.8%), forming a tight cluster just below the 60% threshold.

The report highlights that visual hallucination remains the most critical weakness across all models. The tendency to confidently generate descriptions of objects or details not present in the image fundamentally undermines real-world reliability.

Implications: Perception as the Critical Bottleneck

PerceptionBench serves as a sobering checkpoint for multimodal AI research. It demonstrates that despite impressive performance on complex reasoning tasks, the underlying visual perception engine remains fundamentally flawed. If a model cannot reliably answer "what is there," any sophisticated understanding built upon that foundation is precarious.

By open-sourcing this diagnostic tool, the Kimi team provides the community with a means to move beyond aggregate scores. The focus must shift to methodically addressing specific, atomic perceptual shortcomings in data, architecture, and training. The path toward AI that truly sees and understands is still being paved.