The Illusion of Thinking: Understanding the Limitations of Reasoning Models
Recent advancements in language models have led to the development of Large Reasoning Models (LRMs), which differ from traditional LLMs by simulating step-by-step thinking processes before generating responses. Although these models have shown better results in reasoning benchmarks, their deeper capabilities and limitations remain poorly understood, especially since current evaluation methods mostly focus on final answers in math and code tasks and often suffer from data contamination.
To address these gaps, the authors designed controlled puzzle environments that allow for precise manipulation of logical complexity while keeping the underlying structure consistent. This setup makes it possible to examine not just final answers but also the internal reasoning steps, providing a clearer look into how these models function. Experiments revealed that LRMs perform well up to a point, but when the task becomes too complex, their accuracy collapses—even when given sufficient computational resources.
By comparing LRMs with standard LLMs under equal conditions, researchers identified three performance zones: standard LLMs outperform LRMs in simple tasks; LRMs show an advantage in moderately complex tasks due to extended reasoning; but both models fail in highly complex tasks. Moreover, LRMs struggle with exact calculations and consistent logic, often failing to apply clear algorithms across problems.
Country: Global Keywords: Artificial intelligence, mind, large reasoning models, language model