Frontier AI Models Struggle with Basic Vision: A Critical Bottleneck for ARC-2 Puzzles

The Surprising Perception Gap in Advanced AI

While a five-year-old can effortlessly count holes in a shape, cutting-edge artificial intelligence models, backed by billions in computational power, frequently fail at this seemingly simple task. This unexpected challenge in basic visual perception has emerged as a significant hurdle in the development of solvers for the ARC-AGI-2 benchmark, a problem set designed to test advanced AI reasoning.

Following an initial exploration into a multi-agent Socratic reasoning system for ARC-AGI-2, recent findings delve deeper into what researchers are calling a critical perception problem, rather than a mere reasoning deficit. This insight suggests a fundamental shift in how the AI community should approach solving these complex puzzles.

Unpacking ARC-2: A Four-Skill Framework

Successful navigation of ARC-AGI-2 puzzles necessitates a blend of distinct capabilities. Researchers have identified four crucial skills:

• Perception: The ability to accurately interpret grid elements, including object identification, feature counting, boundary recognition, and spatial relationships.
• Reasoning: Given precise perceptual data, the model's capacity to deduce underlying transformation rules.
• Execution: The precise application of derived rules to construct the correct output.
• Verification: The model's capacity to self-assess the correctness of its solution, often by testing rules against training examples.

While much AI research has historically concentrated on enhancing reasoning capabilities (Skill 2), current investigations highlight that execution (Skill 3) is frequently undermined by deficiencies in perception (Skill 1).

The Execution Challenge: When AI Cannot 'See'

To isolate and examine the execution skill, an experiment was devised where advanced models were provided with explicit, correct transformation rules for a specific ARC-2 task (e3721c99). This puzzle requires identifying gray objects, counting their holes, and recoloring them based on a legend. The aim was to determine if models could accurately apply these instructions, even when given the 'answer'. Several frontier models, including Gemini 2.5 Pro, GPT-4o, and Claude Sonnet 4, were tested, with Gemini 2.5 Pro demonstrating the strongest performance in this particular scenario.

Experimental Insights: Input Modality Matters

Initial tests using text-only input (numeric arrays representing grids) resulted in widespread failures, with models unable to correctly process objects or even validate against training examples. Shifting to an image-only input yielded even poorer results, with models failing to reconstruct basic grid structures. The breakthrough arrived when combining text and image inputs. This multimodal approach significantly improved performance, enabling models to pass validation, though errors in counting holes or preserving object boundaries persisted.

Further refinements to visual representation through enhanced images—doubling pixel size, adding gridlines, and including coordinate labels—led to additional accuracy gains, suggesting that image quality and structural clarity are paramount for effective parsing by models.

The Power of Explicit Verbalization

A pivotal experiment involved introducing an intermediate step: asking the model to articulate its plan in natural language before generating the output grid. This 'explicit reasoning step' revealed distinct perception errors, such as misidentifying or merging objects and inaccurate hole counting. Crucially, when this verbal plan was subsequently used to guide grid construction, the output dramatically improved, achieving 92.4% similarity. This suggests that forcing models to verbalize their interpretation can solidify their internal representation and lead to more consistent execution.

Key Takeaways and Future Directions

This research underscores several critical insights:

• Text and image inputs offer complementary information; neither alone suffices for complex spatial reasoning.
• The quality and structure of visual inputs profoundly impact a model's ability to 'parse' images.
• Explicit verbalization of a plan prior to execution significantly enhances accuracy and reveals perceptual errors.
• Even with advanced capabilities, fundamental visual perception, like counting holes, remains a challenging area for current frontier models.

Future work will focus on generalizing these techniques across the full ARC-2 evaluation set and exploring how different puzzle types might require distinct perception strategies. The broader implications extend to applications requiring precise spatial understanding, such as document analysis, UI automation, and robotics, all of which depend on robust foundational perception.