7. The Wall

7. The Wall

Remember The Wall?

Do you remember when there’s was a lot of articles about how AI companies were “hitting the wall.” Several major companies were having trouble training their next-generation models, with only incremental improvements in performance found.

All the controversy started around November, 2024, with a flurry of media articles expressing concern about the big models. There was a two-fold indication that AI progress was “plateauing” or “hitting a wall.” The two main indicators were:

• Underwhelming progress in new frontier models
• Inference-based reasoning (“test time compute”)

The GPT “o1” model released in September 2024 wasn’t a bigger, more heavily-trained model with trillions more weights. Instead, it was a model that improved intelligence by doing multiple steps of inference, rather than one smarter step in an uber-trained model. This algorithm for “multi-step reasoning” was known as “chain-of-thought” and used repeated calls to process queries, before merging them together into the one final response.

Why did this change to multi-step inference for reasoning support the “wall” theory? Well, inference is a slow process when it runs, and “o1” was therefore slow for users — the line of logic was that OpenAI wouldn’t tolerate using this slow method if they could do it faster with one request to a bigger model. It almost seemed like a kind of “workaround” to hide training failures.

Hence, wall.

Secondly, there were also rumors about the big AI players are having difficulty training much better next-gen models. In particular, there were indicators that the GPT-5 release was having trouble gaining extra capabilities compared to GPT-4. Instead of OpenAI launching GPT-5 sooner rather than later, we were given “o1” with its multiple steps.

Obviously, training trillion-parameter models is a specialist field, and it’s evolving fast, with literally billions of dollars in funding being applied there. But open source models seemed to be keeping up with the leading commercial vendors (albeit, after a lag), which tends to indicate that there’s only incremental progress in reasoning capabilities, and the commercial vendors don’t have a huge “secret sauce” algorithmic advantage in training. Some of the constraints include:

• Shortage of new high-quality training data (text).
• Complexity of software algorithms to train ever-bigger LLMs.
• Sheer volume of training data needed for multimodal LLMs (audio, images, and video).
• Capital cost of GPUs to crunch all that.
• Apparent lack of a new algorithmic advance in one-shot reasoning.
• Fundamental limitations of the way that LLMs and Transformers work.

There was also plenty of evidence to the contrary. OpenAI CEO Sam Altman posted on X that “there is no wall.” And there were certainly signs that many of the bigger players were still gearing up to use NVIDIA Blackwell GPUs for even bigger training runs. And there were two multi-billion dollar fund raises in just that month. So, the plateau was perhaps only a temporary thing.

Certainly, there was (and still is) a lot of research happening in training and in making LLMs better at reasoning in general. Some of the newer areas include:

• Newer GPU hardware for training (e.g., Blackwell).
• Faster software training algorithms (optimizing both computations and inter-GPU network traffic).
• Resiliency improvements to training (both software and hardware).
• Synthetic training data and derivative data.
• Multi-step reasoning algorithms are smarter (if slower).
• Long context processing seems to be a solved problem now.
• Inference optimization research (makes each step of multi-step reasoning faster).
• Next-gen architectures beyond LLMs (e.g., SSMs, Mamba, Hyena, and hybrid versions).

Nevertheless, the question at the time was whether there was a progress wall.

Wall Obliterated

For a while there, it looked like there really was a wall. What a wonderful outcome for AI, because then all those billions of dollars in funding for training trillion-parameter models could now be redirected to making AI useful and safe, instead of training for obscure benchmarks in advanced theoretical equations.

It’s nice to have a dream.

Actually, no, the new new thing became inference at the end of 2024. This is exemplified by the recent OpenAI “o1” (“Strawberry”) model release in September, 2024, which was based on the “Chain-of-Thought” reasoning strategy. The basic idea was to run multiple LLM inference steps for every user question, rather than a “one-shot” attempt to answer the user. Over multiple repeating steps, the model could reassess its own output, and then converge on a much smarter answer.

It worked great. A little slow, but great.

But then the pendulum swung back the other way with the release of the DeepSeek R1 reasoning model in January, 2025. Instead of multiple steps of inference, it used “longer answers” as a way for the model to reason its way to an answer. It turned out that you could train an LLM to do reasoning better just by training it on a lot of such longer sequences, which it could them mimic. The result was smarter answers at a much lower token cost than multi-step reasoning. Who knew that “talking to yourself” would be an AI strategy for reasoning.

So, the wall debate was over largely before it got running, but it wasn’t OpenAI that provided the sledgehammer. The release of the DeepSeek R1 model in January 2025 by a Chinese startup has largely ended the wall debate. Their model beat OpenAI’s o1 model on multiple benchmarks (but not all), and did so using only single-step inference (whereas OpenAI o1 is multi-step inference). Hence, they used training advances to train a much smarter model than other single-step models.

Bye-bye, wall!!

Apparently, OpenAI knew this all the whole time, and I believe them. A research paper emerged from OpenAI within weeks of the DeepSeek release, where they presented research showing the advances possible in single-step reasoning.

Anyway, the point of all this is that there are now dozens of reasoning models, both commercial and open-source, that you can use to do anything. You can choose between one-step or two-step models.

On the other hand, there are still concerns about running out of training data, and also the long duration that it has taken for OpenAI to release the GPT-5 version. Unless all of these concerns are resolved, there’s likely to be some more media articles resurrecting the theory of the wall, sometime in the near future.

References on the AI Wall

Articles and papers covering concerns about recent “wall” concerns about AI progress:

1. Deirdre Bosa, Jasmine Wu, Dec 11 2024, The limits of intelligence — Why AI advancement could be slowing down, https://www.cnbc.com/2024/12/11/why-ai-advancement-could-be-slowing-down.html
2. The Information, Nov 2024, OpenAI Shifts Strategy as Rate of GPT AI Improvement Slows, https://www.theinformation.com/articles/openai-shifts-strategy-as-rate-of-gpt-ai-improvements-slows
3. Bloomberg, Nov 2024, OpenAI, Google and Anthropic are Struggling to Build More Advanced AI, https://www.bloomberg.com/news/articles/2024-11-13/openai-google-and-anthropic-are-struggling-to-build-more-advanced-ai
4. Gary Marcus, Nov 25, 2024, A new AI scaling law shell game? Scaling laws ain’t what they used to be, https://garymarcus.substack.com/p/a-new-ai-scaling-law-shell-game
5. Kyle Orland, 13 Nov 2024, What if AI doesn’t just keep getting better forever? New reports highlight fears of diminishing returns for traditional LLM training. https://arstechnica.com/ai/2024/11/what-if-ai-doesnt-just-keep-getting-better-forever/
6. Will Lockett, Nov 2024, Apple Calls BS On The AI Revolution, They aren’t late to the AI game; they are just the only sceptical big tech company, https://medium.com/predict/apple-calls-bullshit-on-the-ai-revolution-ae38fdf83392
7. Sam Altman, Nov 14, 2024, X post: there is no wall, https://x.com/sama/status/1856941766915641580
8. Shirin Ghaffary, December 6, 2024, Tech CEOs Say It’s Getting Harder to Build Better AI Systems: The comments follow a renewed debate over whether AI is hitting a scaling wall, https://www.bloomberg.com/news/newsletters/2024-12-05/tech-ceos-say-it-s-getting-harder-to-build-better-ai-systems
9. Maxwell Zeff, November 20, 2024, Current AI scaling laws are showing diminishing returns, forcing AI labs to change course, https://techcrunch.com/2024/11/20/ai-scaling-laws-are-showing-diminishing-returns-forcing-ai-labs-to-change-course/ ("at least 10 to 20x gains in model performance ...intelligent prompting, UX decisions, and passing context at the right time into the models...")
10. Joe Procopio, Dec 17, 2024, We’ve Hit The “AI Wall.” Here’s What That Means For the Tech Industry, https://ehandbook.com/weve-hit-the-ai-wall-here-s-what-that-means-for-the-tech-industry-97f543a68e77
11. Lan Chu, Jan 2025, Is AI progress slowing down? https://levelup.gitconnected.com/is-ai-progress-slowing-down-69d4f1215e49
12. Jano le Roux, Jan 2025, Why AI’s Growth Will Hit A Wall Very Very Soon, https://medium.com/swlh/why-ais-growth-will-hit-a-wall-very-very-soon-f6c138b7cfcb

Research and articles that apparently triggered the end of the wall concerns:

1. Duncan Anderson, Jan 2025, The wall that wasn’t: Benchmark results for the latest AI models suggest that any “scaling wall” has already been breached and we’re on the path to AGI, https://medium.com/barnacle-labs/the-wall-that-wasnt-62c617f66ad4
2. Kyle Wiggers, January 27, 2025, Viral AI company DeepSeek releases new image model family, https://techcrunch.com/2025/01/27/viral-ai-company-deepseek-releases-new-image-model-family/
3. Manish Singh, January 27, 2025, DeepSeek ‘punctures’ AI leaders’ spending plans, and what analysts are saying, https://techcrunch.com/2025/01/27/deepseek-punctures-tech-spending-plans-and-what-analysts-are-saying/
4. Rafe Brena, Jan 31, 2025, AI Isn’t ‘Hitting A Wall.” Here Is Why: What does DeepSeek have to do with it? https://pub.towardsai.net/ai-isnt-hitting-a-wall-here-is-why-e75fe86e47f1
5. Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contributors, Oleg Mürk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, Wenda Zhou, 3 Feb 2025, Competitive Programming with Large Reasoning Models, https://arxiv.org/abs/2502.06807 (OpenAI’s paper on o3 that has similar conclusions to what DeepSeek showed about Reinforcement Learning for reasoning models, namely that “scaling general-purpose reinforcement learning” still works.)

Research showing that there are still lingering concerns about AI’s growth trajectory:

1. Jeremy Kahn, February 26, 2025, The $19.6 billion pivot: How OpenAI’s 2-year struggle to launch GPT-5 revealed that its core AI strategy has stopped working, https://fortune.com/2025/02/25/what-happened-gpt-5-openai-orion-pivot-scaling-pre-training-llm-agi-reasoning/
2. Parshin Shojaee, Maxwell Horton, Iman Mirzadeh, Samy Bengio, Keivan Alizadeh, June 2025, The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity, Apple, https://machinelearning.apple.com/research/illusion-of-thinking, PDF: https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf
3. Dr. Ashish Bamania, June 2025, Apple’s New Research Shows That LLM Reasoning Is Completely Broken: A deep dive into Apple research that exposes the flawed thinking process in state-of-the-art Reasoning LLMs, https://ai.gopubby.com/apples-new-research-shows-that-llm-reasoning-is-completely-broken-47b5be71a06a

• Online: Table of Contents • PDF: Free PDF book download • Buy: The Sweetest Lesson: Your Brain vs AI

The Sweetest Lesson: Your Brain Versus AI: new book on AI intelligence theory: Your brain is 50 times bigger than the best AI engines. Truly intelligent AI will require more compute! Another case of the bitter lesson? Maybe it's the opposite of that: the sweetest lesson. Get your copy from Amazon: The Sweetest Lesson